Primary lexer features and unittests.

Fully commented and unit tested.