spotify/luigi — pull request #445 by daveFNbuck (2014-09-12 18:49:14+00:00)
Title: Allows specification of resource limitations for tasks

This allows specification of resource limitations, which are enforced by the scheduler. This was driven by two needs I had. The first was to limit the number of simultaneous tasks running impala queries, because impala does not manage high task load well. The second was to prevent tasks that write to the same output from running simultaneously.

To accomplish the first, we add a resource limitation to client.cfg, like

[resources]
impala: 3

the server reads the entire resources section and turns it into the dictionary {'impala': 3}, meaning that we can have at most 3 units of impala resource used at any given time. Now, in any task that uses an impala resource, I just add resources = {'impala': 1} (or perhaps resources = {'impala': 2} if it's a particularly nasty query. If I want to add impala resources to the resource dicts of all classes inheriting from a base class, this can be done in _resources. Now the scheduler will enforce that the total sum of impala resources being run never exceeds 3.

For the second use case, consider a case where we overwrite a hive table daily using the latest version in mysql. We don't want to simultaneously run two such copies (for example, if we missed yesterday's copy and therefore have 2 scheduled) so we add a resource like {'_copy_to_this_table_unique_resource': 1}. We do not need to add this to client.cfg, as the scheduler will assume that any undefined resource has one unit available. This allows us to add copy tasks for new tables without having to edit client.cfg and restard luigid.

Scheduling had to be modified to prevent workers with low priority tasks from hogging all the resources and preventing high priority tasks from ever getting run. We do this by building up a hypothetical schedule where we greedily assign one task to each worker (ignoring currently running tasks) and only allow tasks that are consistent with this assignment. This heuristic has worked pretty well internally at Houzz, usually allowing full resource usage while getting high priority tasks done first.