github/linguist — pull request #1245 by alindeman (2014-06-03 16:26:42+00:00)
Title: Handle case where newline chars don't transcode to detected encoding

We've seen cases where binary files are detected as encodings such as ISO-8859-8-I. This usually happens when the binary files are short, so while the detector is mistaken, there is also not very much data for use in the detection algorithm in the first place so it's understandable that the detector was wrong.

In these cases, the code to convert ASCII newline characters to encodings such as ISO-8859-8-I fails because there is no conversion between them.

We now simply assume that the data is all one line in those cases. In reality the data is binary, but this obviously difficult to detect reliably.

For review: @github/linguist