mozilla/rhino — pull request #97 by mikelehen (2012-11-11 19:37:03+00:00)
Title: (Potential) fix for parsing of keywords (e.g. "null") at EOF to return correct length.

[This should be code-reviewed by somebody familiar with the rhino token parsing as I don't know if this is the correct way to fix this.]

Prior to this change, TokenStream would return an incorrect length for keywords ("null", "true", etc.) encountered immediately prior to the end-of-file. I added a test to ParserTest to reproduce the issue and made a potential fix.

My fix is to not call ungetChar() if getChar() returned EOF_CHAR, since getChar() does not increment "cursor" when encountering EOF, but ungetChar() always decrements "cursor" thus leading to the incorrect length.