box/ClusterRunner — pull request #15 by josephharrington (2014-11-03 18:58:22+00:00)
Title: Close all file descriptors on server stop

This is a fix for an issue in ClusterRunner where even if the Tornado
IOLoop has been stopped, web requests will not be rejected. In fact,
those requests would just hang open indefinitely until the Python
process actually ended.

This had led to some strange effects where if a slave process was
hanging due to a different issue, the master would have connections
open to it. Later when we would manually kill the hanging slave
process, the master connections would finally fail and bring down the
master as well.

The fix is to explicitly close the Tornado IOLoop with an argument to
close all file descriptors. I couldn't find good documentation on why
this fixes the issue, but I verified the fix through manual testing.