Added unit tests for a future version of AbstractLexer class.