Merged tokenizer enhancements.