delete a little code repetition in tokenizer