refactored tokenizer for clarity