base lexer implementation