Finished tokenizer, 100% test coverage :-)
Cleaned up tokenizer EOF handling.