aaw/mongoid_fulltext — pull request #46 by dblock (2012-03-07 15:05:18+00:00)
Title: Bug: cannot search for "blabla%desert".

The current implementation of remove_accents says:

"If a string is encoded in UTF-8, we strip the accents using NFKD normalization (via an external library, `unicode_utils`. If a string is encoded in ASCII-8BIT, we assume it has been passed via a URL, for instance we might have "%C3%A9" which is how an "e-accute" ("é") gets passed through a web-browser. These are then changed to their UTF-8 equivalents (via the `CGI` gem) - and then finally stripped, as before."

That's an assumption that works for web requests, but doesn't work for indexing data. For example, you can have an object with a title "Untitled (cal%desert)" which will then fail indexing because CGI.unescape transforms %d and ends up with an invalid UTF-8 byte sequence.

I think the burden of CGI.unescape should be with the caller who integrates this into a web service. Removed it and beefed up specs, including some other encoding issues on call.