Prepare lexer for hooking up to generated parser.

Use single character tokens as-is. Make token enums private in a
separate header file, and give them better names.