Finished up the base of the Tokenizer.
Merged in Tokenizer.java conflict fix.