I think everything works now...
Basically, I made some assumptions about what peek() and next() do in the Tokenizer when originally writing this code, and wrote a lot of things incorrectly as a result of that. I was very confused in the debugging process and eventually traced things to peek() and next(), and now I believe I've corrected most if not all of the issues.