This change cleans up the token extraction pipeline. Conditionals are
no longer redundant, and the filtering is done in one file instead of 2.
removes filter-tokens from main pipeline