Ah, I see – we're using the TCP input differently than it was intended. Rather than establishing a single persistent connection per client we have more ephemeral clients that connect, send a few events, and disconnect.

In our use-case logstash is more than happy to keep accepting new clients, reading their message(s) into the thread's local storage, and then blocking trying to put into the SizedQueue. It's related to LOGSTASH-922 (though differently than I had thought) because if we didn't back up and crash, @client_threads would keep growing without bound and eventually OOM the process. This patch does address both issues, but does require the previous client to behave in this way (closing its end quickly) before logstash will move on to the next connection to read.

To make both cases work, we could limit the rate at which connections are accepted if the SizedQueue is full, but that still leaves the system vulnerable to connection bursts and extends the number of messages lost by the number of connections we accept before we catch the slowdown. A better solution might be to keep the input single-threaded and switch to nonblocking socket calls (which as a side effect would have to handle the now-erronious case where a single long message gets broken up across subsequent sysread calls) so we can only accept and read as fast as we can put into the queue.

What do you think?