burke/zeus — pull request #323 by antifuchs (2013-05-07 23:42:31+00:00)
Title: Gather file changes within a short time window for restarts

This change addresses the `git checkout` problem where many files change in quick succession. Previously, this would trigger multiple restarts per slave node (each in potentially slightly-incompatible states), which would very often result in locked-up processes.

The high-level things that happen here are:

* Introduce a command-line switch for a process tree node state tracking log (to make it easier do diagnose similar problems)
* Collect changes to files within a time window (defaulting to 300ms from first to last) and trigger a restart for the process tree based on all these changes.

With this change (and on my SSD), I couldn't reproduce hung processes yet. I would really really like to find a not-essentially-delay based approach to this, but don't think there's a way in this day&age's file systems. Also, it may be necessary to tweak the delay here for slower update situations. But really, the completely right way to fix this would be to make sure slave nodes in all states don't mess up (and I've pretty much given up on doing that this week).

@lastobelus - I would appreciate if you could test this change (ask me if you want a binary)! (: