bunbun/ruffus — pull request #4 by jakebiesinger (2013-01-13 06:49:31+00:00)
Title: Trickle jobs

Phew! That was a lot more convoluted than I thought that it would be, but I think I've covered all the bases and am ready to submit this pull request. It *finally* passes all the unittests, including a new set that tests specifically for "trickling".

The solution was quite complicated thanks to a fair few edge cases I hadn't thought of, but the idea is simple: just maintain a set of output files that have been completed, then allow tasks to be "considered" when their ancestors have all had at least one job submitted. Regardless of the overall task completion state, jobs are submitted when their inputs are complete.

Several job categories are considered "blocking" and will not allow any downstream trickling until all jobs in the task are complete:
1. jobs with indeterminate (globbed) output
2. jobs with an explicit `@follows` list which is incomplete
3. jobs whose incomplete ancestors fit into any of the categories above

Also, tasks with globbed input have to wait for the completion of upstream tasks before being considered.

It would be great to have the indefinitely-running task you mentioned before and I think it would actually be pretty easy to add, given the changes outlined here. The code that determines if a task is really complete would just need to be special cased (never remove a forever-running task).

Have a look at the code, particularly check out the "Files changed" mode above which collapses all the (many) commits into a single diff.