rails/rails — pull request #10355 by burke (2013-04-26 04:43:51+00:00)
Title: Use Ruby's native Encoding functionality to implement `tidy_bytes`

The current implementation is quite slow. This leverages some of the transcoding abilities built into Ruby 1.9 instead. It is roughly 96% faster.

The roundtrip through UTF_8_MAC here is because ruby won't let you transcode from UTF_8 to UTF_8. I chose the closest encoding I could find as an intermediate.

These test cases seem quite thorough and it passes all of them: https://github.com/rails/rails/blob/master/activesupport/test/multibyte_chars_test.rb#L630-L661

### Benchmarking

```ruby
require 'benchmark'

kilobyte = "1" * 1024

u = ActiveSupport::Multibyte::Unicode

res = Benchmark.realtime do
10_000.times do
u.tidy_bytes(kilobyte)
end
end

puts "tidy_bytes took #{res / 10}ms"
```

Save as `./activesupport/bm.rb`, cd to `./activesupport`, and run:

```
ruby -I./lib -ractive_support/multibyte/unicode bm.rb
```

Output before proposed change:

```
tidy_bytes took 1.952ms
```

After proposed change:

```
tidy_bytes took 0.0668ms
```