CLN: minimize tokenizer passes
Merge pull request #6432 from cpcloud/eval-consolidate-tokenization

CLN: minimize tokenizer passes