deleta a little code repetition in tokenizer