Fixed some lexing issues that were causing unicode errors.

I improved the lexer and made it easier to understand and less error-prone. I removed a line that forced an encoding and caused errors on unicode input. I updated some related scripts.