rails/rails — pull request #10353 by burke (2013-04-26 02:07:15+00:00)
Title: Use ruby's native String#encode to implement `tidy_bytes`

There is debatably a regression here, in exchange for reduced complexity and a three-order-of-magnitude performance improvement.

This couldn't be done when the code was written initially because Rails couldn't rely on the presence of these encoding methods without introducing a dependency on 1.9+.

The behaviour of this proposed version is identical to the current version in any case where a string has an internally-consistent encoding.

It differs where:

1. A string is not a valid UTF-8 string (because it contains at least one ISO-8859-1/CP1252 byte); and
2. The string also contains at least one UTF-8 multibyte character.

In this case, the UTF-8 characters will be reinterpreted as CP1252 bytes. This is in contrast to the current strategy which attempts to resolve a mish-mash of the two disparate encodings.

I normally wouldn't propose something like this, but this is a particularly slow method, and letting ruby handle it natively actually cuts 99.7% off of its runtime. It is my experience that the case this regresses on is not common, but I may be wrong on this.

Thoughts?