Rewrite the lexer to use a common function to make tokens.