bunbun/ruffus — pull request #3 by jakebiesinger (2013-01-10 00:11:02+00:00)
Title: Job completion fix

Hey Leo,

Opened a pull request before, but I had apparently merged it from some other branch? Not sure...Anyway here is the PR again:

I think my job_completion branch is *almost* ready to be merged. This branch allows you to specify one of several checksum levels:

```
CHECKSUM_FILE_TIMESTAMPS = 0 # only rerun when the file timestamps are out of date (classic mode)
CHECKSUM_HISTORY_TIMESTAMPS = 1 # also rerun when the history shows a job as being out of date
CHECKSUM_FUNCTIONS = 2 # also rerun when function body has changed
CHECKSUM_FUNCTIONS_AND_PARAMS = 3 # also rerun when function parameters or function body change
```

A sqlite database `.ruffus_history.sqlite` is kept in the directory where the job is run (`.`) which is just a persistent dictionary. Keys are file names, values are `ruffus.ruffus_utility.JobHistoryChecksum` objects and include `mtime` and two separate MD5 checksums: one for the function body and one for any "extra parameters" passed in to the function by ruffus.

The default checksum is changed from level 0 (classic mode) to level 1 (only look at timestamps from job history). I could change this so current users don't have to rerun their pipelines, but iirc, it's sufficient to do a `pipeline_run(..., touch_files_only=True)` to bring all the `mtime`s on the files and in the db up to date.

I has some issues before with multiple-in/multiple-out files, but it looks like those have been fixed, thanks to some more unittests that helped me pin down the issues.

I wonder if you could grab this branch and do some tests in your own code? It feels like a major feature and I don't want to push broken code on the users :).

So *don't* merge yet-- the pull request is just to have a place we can discuss issues and you can inspect the code in a place that's easy on the eyes.