refining the lexer down to core functions