Handle different token types when tokenizing, and use this to filter out other tokens according to config.