customink/gigo — pull request #1 by ryanfields (2013-05-13 23:19:11+00:00)
Title: Refactor + Open discussion regarding CharDet's inaccuracies.

I set out to accomplish one goal, and ended up killing two birds with one stone. There is a significant refactor here that has merit regardless of any decisions reached about CharDet. That said, my primary goal here is to provide concrete proof that CharDet is a risky fallback option behind ActiveSupport::Multibyte.

Running the test cases for this branch will yield one failing test. When asked to detect the encoding for a string whose actual encoding is UTF-8, but whose "force_encoding" is US-ASCII, CharDet tells us that the encoding is ISO-8859-2 and the resulting transcode results in garbage.

I'm calling out this scenario because it is an ongoing occurrence in RFE. We have at least one case where RFE and the backend both write to the same column. RFE writes UTF-8. The backend writes ISO-8859-1. When I attempted to mitigate this with CharDet, it detected them both as ISO-8859-2, resulting in garbage after transcoding RFE's data, and a technically identical string in the backend's data.

Thankfully, ActiveSupport::Multibyte seems to do the right thing in all of my test cases, so CharDet's likelihood of ever being used is extremely low. However, in the cases where it does wind up in use, it produces less favorable results than a simple blind transcoding with no attempt at detection.