Base library for a lexer that can be used in Top-Down, Recursive Descent Parsers.