box/ClusterRunner — pull request #48 by josephharrington (2014-12-02 18:08:33+00:00)
Title: Fix log rotation deadlock

There was a bug in log rotation added in #38. This issue would cause
the app to freeze up completely when it tried to rotate the first log
file. (Since the default log size is 50mb, we didn't hit this for
several days.)

The cause of the freeze was a deadlock in the log handler. Since we
added a new log statement (logging the app summary) during log
rotation, we now have a recursive logger call that tries to reacquire
a lock in the recursive call.

The simplest fix was to change the log handler lock from a Lock to an
RLock (reentrant lock). This allows the lock to be reacquired by the
same thread, and will prevent the recursive call from deadlocking.