refactored tokenizer to use a non-reversed stream