refactored tokenizer.process