Consolidated tokenizer