Tokenizer code duplication refactoring - phase 1
Tokenizer code duplication refactoring - phase 2