Remove unnecessary code from lexer

After #408 landed, there is a lot of junk in the lexer which is not actually needed. Removing this could see some marginal performance increase, and also simplify things nicely.