spotify/luigi — pull request #513 by daveFNbuck (2014-10-27 21:21:47+00:00)
Title: Takes number of workers into account in scheduler resource planning

I've noticed recently that sometimes I'll have a job with thousands of pending
tasks scheduled all using the same resource, but it's not running anything. All
of the tasks are ready and nothing else is using the resource.

It turned out that another worker had several tasks using the same resource
scheduled with higher priority. It only had one worker and wasn't going to get
to them anytime soon, but the greedy scheduling assigned all the resources to
this worker so it could sit for hours preparing to use them.

In order to alleviate this issue and allow resources to be used more freely,
this commit only allows workers to reserve resources when they are actually
about to use them.