logstash/logstash — pull request #1058 by jordansissel (2014-02-13 06:22:36+00:00)
Title: Fix string encoding problems from the geoip library

Strings that come out of GeoIP are sometimes UTF-8, sometimes
ISO-8859-1, sometimes falsely-labeled ASCII-8BIT. For example:

[15] pry(#)> (to_hash["geoip"].to_a + to_hash["pants"].to_a).collect { |k,v| p k => (v.encoding rescue v.class) }
{"ip"=>#}
{"country_code2"=>#}
{"country_code3"=>#}
{"country_name"=>#}
{"continent_code"=>#}
{"region_name"=>#}
{"city_name"=>#}
{"timezone"=>#}
{"real_region_name"=>#}
{"number"=>#}
{"asn"=>#}

In testing, I found that the strings with ASCII-8BIT encoding were
actually mislabeled and were ISO8859-1.

This patch converts any non-UTF-8 encoding, hopefully correctly, to
UTF-8. I also fixed a bug in the tests.

- Tests written in #1054 now pass. Hurray!
- Fixes LOGSTASH-1354, LOGSTASH-1372, and LOGSTASH-1853