Complete tokenizer code and add some very simple tests.