spotify/luigi — pull request #570 by daveFNbuck (2014-12-11 23:12:12+00:00)
Title: Groups tasks that supersede each other together in the scheduler

Many tasks supersede older versions of the same job. For example, if I do an
hourly sqoop job to copy data from a production db to a hive table, completing
the 8am copy means I don't need to run the 7am copy. If both the 7am copy and
the 8am copy are pending and ready, I should run the 8am copy and consider them
both complete.

This commit achieves these goals by allowing jobs to define two optional
properties. The first is supersedes_bucket. Tasks with the same supersedes
bucket (if not None) are considered by the scheduler to supersede each other.
The scheduler will mark all earlier versions of the task DONE when one of them
completes. It us up to the user to ensure that earlier versions of the task
become complete when a later version is run.

Earlier tasks are identified by having a lower supersedes_priority. The
scheduler will prefer to schedule tasks with higher supersedes_priority.
Supersedes priority can be any type, but most be comparable between tasks in
the same bucket.