add some comments to the tokenizer, note where things were done to solve bugs and remove a relic of the old version of parsing key-value pairs