spotify/luigi — pull request #460 by daveFNbuck (2014-09-22 20:49:24+00:00)
Title: Allows configuration of wait between tasks finishing

With the latest worker refactoring, workers with fewer running tasks than available processes end up pinging the scheduler once per second for work. My scheduler has occasionally stopped responding since updating to this code path, and I believe the increased requests are the culprit. This applies worker-wait-interval to the timeout when checking for complete tasks to avoid hitting the scheduler so hard.