elasticsearch/logstash-forwarder — pull request #180 by driskell (2014-03-22 09:57:59+00:00)
Title: Features/protocol improvements and server client improvements

This will fully implement ACK verification and partial ACKs between Logstash-forwarder and Logstash. Old LF clients connecting to LS will still work fine, but without the improvements.

I've not yet got around to checking the verification and partial ACK in client.rb yet but hope to do so soon. But LS to LS lumberjack connections should still work fine, worst case without improvements.

I'm aware protocol will be replaced soon but I needed to fix this for my production environment to prevent duplicate logs during LS congestion.

* Fix LogStash "too many open files" error due to constant reconnecting of LF clients due to timeouts and a blocked Logstash pipeline (redis congestion etc.)
* Fix the timeouts too - LogStash will send partial ACKs to keep LF informed things are just slow - LF will send a window update during inactivity to keep firewall routes open etc
* Fix LogStash disconnecting other clients if a rogue client connects and sends plaintext or offers an invalid certificate etc.
* Implement logging to Logstash stdout of any connection failures, including which client the failure was for, to aid in finding problematic clients. This will require a change to lumberjack input plugin in LogStash to pass through @logger (its optional so will work without the change)