spotify/luigi — pull request #467 by daveFNbuck (2014-09-23 22:16:48+00:00)
Title: Allows co-scheduling of tasks without dependency

Currently, the only way to guarantee that task B runs whenever we schedule task A is to make A depend on B. This is not always feasible. For example, it may already be the case that B depends on A. We may also want to schedule them together without requiring that one runs before the other. For this usage, I've added a co_schedule function to Task. If A.co_schedule() returns B, then B will be scheduled whenever A is without affecting either of their dependencies.

The main use case I have is to schedule automatic cleanups. Currently, I have a setup where I'll have one mapreduce task M, with many consumers C1,...,Cn that each depend on M. I then have a purge task P that depends on C1,...,Cn which removes all of M's output after all the consumers have used it. This works fine normally, as I regularly schedule P going forward. It causes problems when someone creates or changes a consumer Ci and has to do a backfill. When Ci is run, we end up running M but not P, so we're left with a lot of garbage at the end of the backfill which won't necessarily be cleaned up by my regularly scheduled tasks. To fix this, we set M to co-schedule P. Now when I backfill Ci, P will be scheduled as well and I won't be left with any cruft.