cooperative-computing-lab/cctools — pull request #369 by dpandiar (2014-02-07 18:42:48+00:00)
Title: Fixes to failure handling in WQ.

This pull request will result in WQ handling its failures as follows:

1. Missing or inaccessible input file in local filesystem (app-level failure). The task will be returned to the application on encountering the first missing file. The task result will be set to INPUT_MISSING. Any transferred input files of the task that were marked to be cached will be left at the remote cache.

2. Missing or inaccessible output file at worker (app-level failure). The task will be returned after attempting to retrieve all the output files. The task result will be set to OUTPUT_MISSING. All output files of the task will be removed from the remote cache.

3. Failure to store the output file in local filesystem (app-level failure). Same as 2.

4. The stdout exceeds the 1GB limit set by WQ (app-level failure). Same as 2. Except the result will be set to STDOUT_MISSING.

5. Failure to allocate memory to store stdout (app-level failure). Same as 4.

6. Failure to stat/open/lseek a file of a task (app-level failure). Same as 1 if input file. Same as 2 if output file.

7. Failure to allocate memory (app-level or wq failure depending on circumstance). Won't segfault.
If the memory allocation concerns a task, the task is returned with the result set to either INPUT or OUTPUT_MISSING. If the memory allocation concerns a worker, the worker is disconnected and removed.

8. Failure to transmit input file to worker due to network failure (wq failure). Worker is disconnected and removed. Task retried.

9. Failure to transfer the output file from worker to local filesystem due to network failure (wq failure). Worker is disconnected and removed. Task retried.

10. Failure to transfer the stdout from worker (wq failure) due to network failure. Worker is disconnected and removed. Task retried.

11. Broken connection to worker (wq failure). Worker is disconnected and removed. Task retried.

12. Invalid protocol message (wq failure). Same behavior as before.

13. Incompatible master/worker (wq failure). Same behavior as before.

14. Failed to authenticate with password (wq failure). Same behavior as before.

15. Suspended worker (wq failure). Same behavior as before.

@btovar