PERF: refactor tokenizer to give compiler branching hints.