reorder conditions in tokenize function