remove dead lex_alias code

Reroll of 5a8ce580ff.

Remove the code related to the `lex_alias` variable in agent.h,
including the `expansion_node` and `alias_struct` structs. This code
is uncommented (except for a date and originating file), and the logic
in `get_lexeme` is unreachable because `lex_alias` is always NIL. The
original functionality seems to have been to skip actual string lexing
and instead programatically provide a list of tokens, perhaps for
testing the parser.
remove top_dir_stack

Reroll of acd969576e.

The `top_dir_stack` variable in agent.h is not used anywhere, so
remove it along with related code and the `dir_stack_struct` struct.

I believe that this was not meant to be accessed by client code because
it's usage involves manual memory allocation and deallocation.
remove fake right paren logic

Reroll of c132b2fb3b.

The functions to fake a right paren in the lexer were never used,
so removed all related code.
remove dollar-related lexing

Reroll of 9e851ee3cf.

The dollar (`$`) character is no longer used for any special lexemes,
nor is it handled by the parser, so remove the special processing for
it (apparently last touched in 1994).
factor out whitespace and comment skipping

Reroll of 93c2c0dc68.

Create a new function for skipping comments and whitespace in the
production lexer, and move the proper logic into it.
remove loading error code

Reroll of 88368ed8a7.

Remove `load_errors_quit`, which was not used, and the possibly related
`respond_to_load_errors` calls, which were all commented out.
remove unused code related to lexer file stacks

Reroll of 1d5695bc99.

`load_file` is the only function to call `stop_lex_from_file`, and it
does so right after the start function; `load_file` is never used, so
remove it and the stop function.

`start_lex_from_file`, then, is only called once, during lexer
initialization. `reading_from_top_level`, then, is always true, so
remove it and simplify all of the conditional blocks where it is used.
remove alternate_input_exit

Reroll of bf1c2dfe2b.

`alternate_input_exit` was always true, so remove it and simplify
referring conditional blocks. This includes the code in `get_next_char`
for reading from a file.

Also comment/TODO the function that prints the lexer file location; this
has been useless ever since Soar stopped using file input here, but
its uselessness has not been obvious.
rename alt vars to lex vars

Reroll of be2b475375.

Rename the `alternate_input_...` variables to reflect the fact that
they are the only input for the lexer. It seems that previously files
were the main input to the lexer, and that the `alternate` variables
were a way to put off lexing the file and instead lex an input string.
remove __SC__ sections

Reroll of 8d5184bef0.

Symantech C++ is long gone, and the code in these sections seem
outdated; one of them contains unused variables, like other `__SC__`
sections were deleted, and the other looks for `soar>` in the current
lexeme while a number is being read.
remove file usage in lexer

Reroll of 093b45996f.

Move `allow_ids` and `parentheses_level` into agent, and remove the
structs and logic for reading from files and stdin, which is no longer
done in the lexer. Also remove the memory deallocation required to
match the first call to `start_lex_from_file` inside of lexer.
remove input suffix

Reroll of 7f2d777c1c.

`lexer_input_suffix` was always provided so that the last lexeme given
by the lexer would be `R_PAREN_LEXEME`. The reason for this has probably
been gone for a long time (older Soar syntax?), but it was still required
because the parser and other code that relied on the lexer (the print
command and semantic memory) compensated by assuming that lexing would
finish with a right paren. Remove the input suffix and fix related code
so that lexing will always finish with EOF and client code will expect
it.
put lexer input access in one place

Reroll of aaeae21ccc.

Lexer input setting also requires that `current_char` be set to
whitespace; put this in a function and use it everywhere the lexer
is set up.
remove unused print_prompt_flag variable

Reroll of a4aa5a8d08.
make lexer code into a class

Reroll of 43aceb885c.

Rewrite the lexer code as `Lexer`, removing global state stored
in the agent and instantiating an object wherever lexing is required.
Global lexer state was used extensively and required considerable
untangling, thus the large change set.
move creation of lexer in cli_print

Reroll of 3e22cce822.

Move Lexer creation into `read_pattern_and_get_matching_wmes` for
better reusability.
clean/document lexer

Reroll of bfe84fc365.

Organize the lexer, removing unneeded defines and grouping related code.
Add doxygen comments to everything in lexer.h.
make determine_type_of_constituent_string private

Reroll of 4230293f8b.

This method only changes the type of `current_lexeme` and makes more
sense as a private method. The only place that used it was
`get_lexeme_from_string`; this function looked for quoted (`||`)
constants and then instantiated a `Lexer`. The lexer, however, will find
quoted symbolic constants from a simple call to `get_lexeme`, so this
code is extra and can be deleted in favor of `get_lexeme`.
avoid null pointer crash in failed test

Reroll of 618e24a858.

During some editing, `pOutputLink` was null in this test and caused the
unit tests to crash half-way through instead of failing and finishing.
change lexeme_info struct to soar::Lexeme class

Reroll of 4400abd011.