dabeaz/ply — pull request #33 by mdboom (2013-05-16 02:09:50+00:00)
Title: Support defining lexers in source files with a magic coding comment

On Python 3, if the lexer is defined in a source file with a magic coding comment (e.g. `# -*- coding: utf-8 -*-`), and `locale.getpreferredencoding()` doesn't match, `LexerReflect.validate_file` will throw a `UnicodeDecodeError`.

To reproduce set the locale to something ascii, and import a lexer where the file contains some Unicode, such as this [gist](https://gist.github.com/mdboom/5588912):

```
LC_LANG=C python lex_unicode.py
```

I couldn't find a stdlib function to handling opening a file with magic coding comments, so I instead chose to just work with module objects (rather than filenames), and use the module objects to get the already decoded source lines.