SoarGroup/Soar — pull request #178 by garfieldnate (2014-07-25 16:54:00+00:00)
Title: Refactor lexer and lexing logic

Previously lexer state was stored in the agent and was accessed and modified globally from there. This series of commits organizes the lexer into a class and removes state from the agent. The global state was accessed by many files, so many files were changed to instantiate a new Lexer. None of these commits causes tests to fail. 43aceb8 is a large change, but I couldn't really break it into smaller changes.

The gp test commit just got stuck in this branch because I started by looking at the parser.

There are some TODO items that I didn't want to include in this PR:

* Modernization (char Bool)/formatting from 9.5 branch will probably need to be redone
* An agent is still required to create a Lexer because print requires an agent. Refactor printing to be separable from agent, and then change Lexer to take a printer instead (like in jSoar)
* `MAX_LEXEME_LENGTH` and `MAX_LINE_LENGTH` are not related to actual maximum lexeme length; this was a bug before, but now it has been pushed out of the lexer into agent, and whatever other files use it. This is a potential buffer overflow bug if anyone writes a production with long lexemes. See ca021aa.

Feel free to suggest changes. Hopefully all will be well on Jenkins...