logstash/logstash — pull request #584 by yaauie (2013-08-21 06:41:25+00:00)
Title: LOGSTASH-952: Allow configurable drop isolated events on repeat failure.

Currently `Stud::Buffer` will keep retrying flushing the buffer of events
on failure, but some events will never succeed (esp. due to ES index-
type errors).

This patch provides an option (`drop_on_repeated_failure`) to allow
some data loss when an event fails to be indexed repeatedly. It does
so by bisecting the bulk request into smaller requests as they fail;
when a single event fails to index 3 times in a row, it will be
dropped and the event will be logged as a string.

I don't believe this P/R is quite ready to merge; for one thing it needs
better granularity on what exceptions will cause the bisect-and-retry.
It could also use discussion on what is needed for configurability.

Should this be handled upstream in `Stud::Buffer`?

See [LOGSTASH-952](https://logstash.jira.com/browse/LOGSTASH-952)