Maybe:

1. charset => "auto" would try other encoding and know performance
implications,
2. default of UTF-8 if not specified,
3. if option set and doesn't match can either default back to UTF-8 or
fail because encoding set wasn't correct?

On 10/4/13, Jordan Sissel <notifications@github.com> wrote:
>> @@ -32,6 +32,9 @@ def decode(data)
>>        # into UTF-8 so we can use it with JSON, etc.
>>        data = data.encode("UTF-8", :invalid => :replace, :undef =>
>> :replace)
>>      end
>> +
>> +    data = data.chars.select{|i| i.valid_encoding?}.join
>
> This causes data loss, and logstash aims to never have a default situation
> that intentionally loses data.
>
> In my opinion, the correct action to take is to know what encoding your data
> uses and tell logstash what that encoding is using the charset setting on
> this codec.
>
> Put another way - If you get UTF-8 encoding errors, it is because your data
> is *NOT* UTF-8 and logstash defaults to expecting UTF-8; if your data is not
> UTF-8 (signalled by a UTF-8 encoding error) then you MUST tell logstash what
> the correct encoding is.
>
> I'm open to making a non-default option that makes logstash try other
> encodings if the logstash is told the data is UTF-8 but it is not valid, but
> it's worth noting this will have performance implications as well as causing
> data loss - making it an option is OK with me.
>
> ---
> Reply to this email directly or view it on GitHub:
> https://github.com/logstash/logstash/pull/698/files#r6780663

-- 
Sent from my mobile device