github/linguist — pull request #845 by g-nix (2013-12-17 04:28:39+00:00)
Title: Fix handling of invalid UTF-8 (and other character encoding errors).

This pull request is a proposal to fix #830, and closes #829.

Addresses a number of encoding errors, mostly by:
- For non-ASCII/UTF-8, convert text to UTF-8, replacing missing characters prior to spitting into lines and/or parsing.
- For ASCII/UTF-8, convert to UTF-16, then back, replacing invalid characters. (This is necessary because Ruby won't convert to/from the same encoding.)
- Workaround for incorrect (or maybe just extremely obscure) encodings reported by 'charlock'.
See changes in [blob_helper.rb](https://github.com/pullreq/linguist/blob/master/lib/linguist/blob_helper.rb), etc.
- Includes the following new test cases for the above, all taken from real repositories here on Github:
- [Python/shtest-encoding.py](https://raw.github.com/llvm-mirror/llvm/master/utils/lit/tests/shtest-encoding.py) (invalid UTF-8, error in blob helper)
- [Text/btParallelConstraintSolver.h](https://raw.github.com/kripken/emscripten/master/tests/bullet/src/BulletMultiThreaded/btParallelConstraintSolver.h) (invalid UTF-8, error in tokenizer)
- [JavaScript/lang-vb.js](https://raw.github.com/nodesocket/commando/master/js/code-pretty/lang-vb.js) (no eqivalent character in UTF-8 from Windows-1252)
- [JavaScript/xor-sanity.js](https://raw.github.com/mozilla-servo/mozjs/master/js/src/jit-test/tests/jaeger/xor-sanity.js) (bad encoding reported: IBM424_rtl)

- Also includes a small patch to fix verbose debugging in classifier.rb -- it was broken at https://github.com/github/linguist/commit/6a8de63d2d98514f370b0610adeadfdcaf35039e.