Separating out token logic from tokenizer.