Remove unnecessary code from lexer (DEPENDS ON #414)

After #408 landed, there is a lot of junk in the lexer which is not actually needed. Removing this could see some marginal performance increase, and also simplify things nicely.