Refactored tokenizer to act as stream of tokens