mistercrunch/Airflow — pull request #56 by mistercrunch (2014-12-12 05:52:12+00:00)
Title: Adding wait_for_downstream as BaseOperator attribute

wait_for_downstream is complementary to depends_on_past. In this case the task will not only wait for the last run to be done, but also for the direct dependents on the last run to be done.

Why?

Because I inherited a bunch of self destructive pipelines that not only can't be parallelized on time (depends_on_past would take care of that), but where the dependents fail if I start running the next run because the table the task delivers get dropped in the next execution.