facebook/scribe — Issue #52: quitting scribe with scribe_ctrl stop can lead to errors
Author: zviadm
Created: 2012-01-31 04:53:32+00:00
State: open

lets say you have following scribe setup:

you have many nodes (~500 of them) which send data over network to central scribe hub.
scribe hub then writes data to hdfs.

if you try to:
scribe_ctrl stop, the scribe daemon running on the scribe-hub machine after following seconds this error starts spamming in scribe.log:

Thrift: Tue Jan 31 04:39:42 2012 thriftServerEventHandler: accept() Too many open files

I am assuming this is because scribe_ctrl stop can take a while when scribe-hub is writing data to hdfs and the buffer size is large, so before it finishes way too many connections are created due to node machines trying to send data to the hub.