github/linguist — pull request #1872 by geoff-codes (2014-12-13 02:21:13+00:00)
Title: Quickly auto-detect and auto-convert character encodings to UTF-8, no "Charlock" needed.

This is an improved reimplementation of #855, something I was hacking on a year ago.

The long and short of it: replace @brianmario / [charlock_holmes](https://github.com/brianmario/charlock_holmes) with a fast, pure Ruby encoding detector for common encodings. It certainly doesn't do all that the ICU library can do — it won't detect legacy multi-byte encodings, Big5, EBCDIC. But it also doesn't have to `dlopen()` a ~25Mb C++ library/module when invoked. And if I had to take a guess, I'd say this probably covers ±99.9% of text files hosted on GitHub.

While I don't have exhaustive benchmarks like last time (@arfon), nor is the module really written as it should be (it currently just monkey patches String), I believe it's at least 50% faster than Charlock (as its used in linguist) and returns far fewer errors, where just about anything that's not UTF-8 gives a `'split': invalid byte sequence in UTF-8 (ArgumentError)`.

I've put the code for this in a [small gem](https://github.com/geoff-codes/charlotte). The implementation is [here](https://github.com/geoff-codes/charlotte/blob/master/lib/charlotte.rb). Let me know if the comments/code are unclear; I'd be happy elaborate. The repo also has a couple quick and dirty test scripts to compare against charlock and rchardet.

This PR isn't really meant to be merged as is — I've just quickly replaced "charlock" with "charlotte" to demonstrate. It works as-is, but I think a major refactor of blob_helper.rb, file_blob.rb, etc. would be in order if this idea were to be adopted. So perhaps it's more perhaps an idea for a topic branch?

Also, it would probably be wise to bring `linguist` in line with [github/lib/github/encoding.rb](https://github.com/github/github/blob/master/lib/github/encoding.rb), [gist/lib/gists/encoding.rb](https://github.com/github/gist/blob/master/lib/gists/encoding.rb), [gitrpc/lib/gitrpc/encoding.rb](https://github.com/github/gitrpc/blob/master/lib/gitrpc/encoding.rb), and kin.
Although I don't believe that code is presently open source.

So maybe this is something to talk about on or sometime after Monday, @tmm1? :wink:

:octocat: — Geoff