github/linguist — pull request #841 by g-nix (2013-12-16 12:22:57+00:00)
Title: Fix handling of invalid UTF-8 byte sequences in Ruby 1.9+.

This pull request is a proposal to fix #830, and closes #829.

Basically:
- explicitly convert text to UTF-8, replacing invalid characters prior to spitting into lines and/or parsing.
See changes in [blob_helper.rb](https://github.com/pullreq/linguist/blob/master/lib/linguist/blob_helper.rb) and [sample.rb](https://github.com/pullreq/linguist/blob/master/lib/linguist/sample.rb).
- Adds a test case (from LLVM's [lit](http://llvm.org/docs/CommandGuide/lit.html) testsuite) as [samples/Python/invalid-encoding.py](https://github.com/pullreq/linguist/blob/samples/Python/invalid-encoding.py).

Tested with Ruby 1.8.7p358 and 2.0.0p353 on Darwin.