Separating state transitions in the Tokenizer into functions as an attempt to improve readability. Not sure if succeeded...