Changed the way that the different token types are filtered.

moved from a central nested if-then-else tier