cooperative-computing-lab/cctools — pull request #344 by batrick (2014-01-21 07:53:42+00:00)
Title: Chirp with Job Execution

[This PR is ready to be reviewed but not merged. I still need to polish it some more I think and maybe add 1 or 2 more minor features. I still need to document the new job interface.]

This pull request adds a job execution system to Chirp. There are 6 new RPCs: create, commit (commit a created job), wait, kill, status, and reap (commit a wait). [See batch_job_chirp.c for an example usage.]

Makeflow execution should work out-of-the-box:

./makeflow -T chirp --debug=all -o >(tee makeflow.debug) --auth=unix --working-dir=chirp://localhost:9000/ example.makeflow

[Only on-demand GC in Makeflow is NYI. Anything else missing is a bug.]

Batch job has also been significantly reworked in this PR. It requires a thorough review. Most changes are not API visible. There is a new FS layer to batch job which is required for Chirp. It only implements what is necessary.

It was discussed that Batch Job should have more verbose `struct batch_job_info` (output of batch_job_wait). In particular, the structure should support expressing errors in the execution engine but not the job itself. I have not modified that yet pending more discussion.

Please have a look @btovar @dthain.