puppetlabs/puppetdb — pull request #347 by cprice-puppet (2012-11-12 22:56:13+00:00)
Title: (#17216) Compose UTF-8 cleaning methods for ruby 1.8

We've tried doing UTF8 cleaning for ruby 1.8 a few different
ways. Doing it via IConv, we don't do a good job of handling
characters whose codepoints would exceed the legal maximum for UTF-8.
Doing it via our manual ruby scrubbing code is slower and doesn't
catch overlong encodings. Since this code really shouldn't even
exist in the first place we've decided to simply compose the two
scrubbing methods for now, rather than trying to add manual detection
of overlong encodings. It'd be a non-trivial chunk of code, and
it'd have to do a lot of bitwise arithmetic (which Ruby
is not blazingly fast at).

Between our existing ruby 1.8 manual cleaning stuff and IConv, I
think we cover all of the known problem cases. This commit
simply changes the ruby 1.8 code path to call both of those