Removes duplication in new tokenizer.