lexer API update, all tests pass