spotify/luigi — pull request #461 by daveFNbuck (2014-09-22 22:59:34+00:00)
Title: Allows workers to keep alive only for unique ready tasks

With resource limitations, it's very easy to have a lot of workers sitting around just waiting for resources. This is problematic for people using locks to prevent scheduling, as it will stop new tasks from getting scheduled. To limit this behavior, this PR adds an option to keep alive only if there is a pending task with all requirements done which no other worker can work on. So older workers whose tasks are all being waited on by newer workers can just die. I require the unique tasks to be ready because all of my workers have a unique WrapperTask at the root, and I don't want to wait for those. Having a unique ready task tends to mean the same thing as having a unique pending task other than the root one for me, so this works well for my scheduling patterns.

Since this won't work well for all scheduling patterns, it has to be activated via a config parameter, worker-require-unique-ready as well as enabling worker-keep-alive.