logstash/logstash — pull request #967 by sethp-jive (2014-01-20 23:54:08+00:00)
Title: Single-thread the TCP input's server

Should address https://logstash.jira.com/browse/LOGSTASH-922

We ran into that issue on our systems, where we had multiple thousands of TCP connections in TIME_WAIT state (indicating that our client process had sent its data and closed its end of the connection) and a logstash thread to match. The root problem was that our output was sluggish and the queue was filling up, but logstash wasn't applying back pressure well. Since the TCP input was accepting new connections as fast as it could fork threads, we were extending the SizedQueue into the kernel's run queue up to the process' thread limit by way of blocked threads.

After applying this patch, we saw only hundreds of outstanding connections – TCP implementations offer relatively poor control over the sizing of the connect queue from the application level – but only a single TCP thread that was blocked on the SizedQueue. In other words, this change allows us to implement back pressure more effectively (catching connection failures in the client and retrying with a back off) by refusing connections earlier when we got backed up and without killing the logstash process or overloading the scheduler.