Moving to tokenize

Addressed the previous issues mentioned and a couple of other mistakes.