refactor tokenizer with tests