os97673/regexp_m17n — pull request #15 by kreeger (2014-08-01 15:22:01+00:00)
Title: Add solution for UTF-8 regex issue using String#unpack.

(This pull request supersedes #5.)

Here's my fix — it worked for nearly every case except for `UTF-7` and `ISO-2022-JP-2` because those encodings are defined as **dummy encodings**. Here is the exception I got from **inside the test case**.

```
Encoding::ConverterNotFoundError: code converter not found (UTF-8 to ISO-2022-JP-2)
```

[Ruby 2.1.0 doesn't support](http://stackoverflow.com/questions/23852059/convert-from-iso-2022-jp-2-to-utf-8-in-ruby-2-1) those encodings. [Originally I tried falling back on `iconv`](https://github.com/kreeger/regexp_m17n/commit/cc0f852fbaf6586f468a8402d184d4ac2cc541fd), which converted the strings just fine. This worked great in MRI, and with my fix to the `non_empty?` method, the tests passed! However, `iconv` doesn't play nicely with JRuby (since it's a C-based extension), so I took it out and instead skipped any conversions where Ruby has them defined as dummy encodings. These conversions aren't implemented yet in Ruby:

- JRuby doesn't convert from `UTF-8` to `UTF-7`.
- MRI doesn't convert from `UTF-8` to `UTF-7` or `ISO-2022-JP-2`.

Granted, if *true* strings come in with that encoding already assigned to them, this test should pass just fine. These tests, however, were failing **inside of the test** before they could ever get to the `non_empty?` method. Once the test itself was fixed to skip any offending conversions, the test still fails with the original exception.

```
Encoding::CompatibilityError: incompatible encoding regexp match (US-ASCII regexp
with UTF-16BE string)
```

Adding `.unpack('C*').pack('U*')` to the string before it's "regexed" solves the issue straight away. `String#unpack` decodes the string into a series of raw code values (as an array of 8-bit unsigned integers), and then `Array#pack` combines them all again into UTF-8 characters. I used this solution quite extensively at my last job, and it preserved unique characters during the conversion.