refactoring out of tokenizer