elasticsearch/logstash — pull request #1173 by colinsurprenant (2014-03-15 01:44:29+00:00)
Title: Utf8fields

All your utf8 are belong to us.

- utf8 cleanup for all plugins which created non utf8 event fields.
- support for imap message codec and headers utf8 transcoding

I do not include the accessor encoding temporaty validation code which allowed to do the global utf8 verification/cleanup, I'll just include it here for reference, this will go in another PR when correctly implemented and support for enabling/disabling, somehow, need to figure out.

`lib/logstash/util/accessor.rb` in PR #1163

```ruby
def set(accessor, value)
target, key = lookup(accessor)

if value.is_a?(String)
if (value.encoding != Encoding::UTF_8)
raise("unexpected encoding for value=#{value}, encoding=#{value.encoding.inspect}")
end
value.force_encoding(Encoding::UTF_8) if (value.encoding != Encoding::UTF_8)
unless value.valid_encoding?
raise("invalid encoding for value=#{value}, encoding=#{value.encoding.inspect}")
# @logger.warn("Received an event that has a different character encoding than you configured.", :text => data.inspect[1..-2], :expected_charset => @charset)
target[key] = value.inspect[1..-2]
else
target[key] = value
end
else
target[key] = value
end
end
```