github/linguist — pull request #855 by g-nix (2013-12-23 08:51:37+00:00)
Title: Rebased onto 2.10.7: Fix handling of invalid UTF-8 (and other character encoding errors).

So I've gone ahead and rebased this ...

But can I ask, um, what your leaning towards here? Should I close this and reopen an issue? If you think this will be resolved another way? Or just close if this is a wontfix? It's totally fine however you choose, its your project after all... I just get a little antsy with a pull just sitting open while new revisions get released, I guess.

Am I just crazy? Does no-one else get a bunch of Unicode decoding errors when they try to run this over any significant amount of code?

Addresses a number of encoding errors, mostly by:
- For non-ASCII/UTF-8, convert text to UTF-8, replacing missing characters prior to spitting into lines and/or parsing.
- For ASCII/UTF-8, convert to UTF-16, then back, replacing invalid characters. (This is necessary because Ruby won't convert to/from the same encoding.)
- Workaround for incorrect (or maybe just extremely obscure) encodings reported by 'charlock'.
See changes in [blob_helper.rb](https://github.com/pullreq/linguist/blob/master/lib/linguist/blob_helper.rb), etc.
- Includes the following new test cases for the above, all taken from real repositories here on Github:
- [Python/shtest-encoding.py](https://raw.github.com/llvm-mirror/llvm/master/utils/lit/tests/shtest-encoding.py) (invalid UTF-8, error in blob helper)
- [Text/btParallelConstraintSolver.h](https://raw.github.com/kripken/emscripten/master/tests/bullet/src/BulletMultiThreaded/btParallelConstraintSolver.h) (invalid UTF-8, error in tokenizer)
- [JavaScript/lang-vb.js](https://raw.github.com/nodesocket/commando/master/js/code-pretty/lang-vb.js) (no eqivalent character in UTF-8 from Windows-1252)
- [JavaScript/xor-sanity.js](https://raw.github.com/mozilla-servo/mozjs/master/js/src/jit-test/tests/jaeger/xor-sanity.js) (bad encoding reported: IBM424_rtl)