Refining the lexer API and error handling