spotify/luigi — pull request #496 by daveFNbuck (2014-10-13 21:01:04+00:00)
Title: Improves resource management in scheduler.

I've noticed recently that my scheduler has been limiting a lot of my tasks to
run 1 at a time when the resource limitations should allow 4 simultaneous jobs.
This happens when tasks have multiple resources. Some of my tasks would have
a resource for writing to mysql and one for making a hive query, while others
would only have one for making a hive query. I have 1 write resource, and 4
hive resources, so I should be able to run 1 mysql write job and 3 other hive
jobs that don't write to mysql. Unfortunately, the mysql write jobs have higher
priority, so the end up getting scheduled in the greedy scheduling and take all
of the hive resources without being able to actually run more than one.

I added a test case for this as well as a few other similar issues and re-wrote
get_work to keep track of the greedy resources and the actual resources
separately. A task must fit in both of these to be scheduled, but this gives
more flexibility to allow tasks to run that don't interfere with the running of
higher priority tasks. Running 3 of the hive-only resource jobs doesn't
interfere with running the next hive plus mysql write task when the current one
is finished.