Refactoring tokenizer code