Repository: DataBiosphere/toil

Issue:
Inconsistent job graph in case of post-job failures
After tracking down a strange failure in a large workflow, I noticed that the job graph can get into an inconsistent state if there is a failure in the `deferredFunctionManager` code (e.g. when using caching job store). There is no way to recover from this, so I think this is a pretty serious bug.

Scenario:
1. The job itself completes successfully, and [`_serialiseExistingJob`](https://github.com/DataBiosphere/toil/blob/bd8c6d16eca1cba8a54d45ee2359e49aa9b0badf/src/toil/job.py#L1431) is called, which sets [`jobGraph.command` to None](https://github.com/DataBiosphere/toil/blob/bd8c6d16eca1cba8a54d45ee2359e49aa9b0badf/src/toil/job.py#L1342).
2. An exception is raised in `deferredFunctionManager` after the job is done. For example in `chunkedFileUpload` function (see logs below).
3. The exception is caught and [`_terminateEvent` is set](https://github.com/DataBiosphere/toil/blob/e6dea9602ca05dcf28b190be97673fd57f69022f/src/toil/fileStores/cachingFileStore.py#L1742), but it's ignored upstream.
4. The jobgraph (with command set to None from Step 1) is [updated](https://github.com/DataBiosphere/toil/blob/bd8c6d16eca1cba8a54d45ee2359e49aa9b0badf/src/toil/job.py#L1389) since there are no exceptions.
5.  The failure is correctly caught via `_terminateEvent`.
6. The leader retries the job, but this time, `jobGraph.command` is None, so it just [ignores running the job](https://github.com/DataBiosphere/toil/blob/bd8c6d16eca1cba8a54d45ee2359e49aa9b0badf/src/toil/worker.py#L348).
7. The leader thinks that the job completed successfully and proceeds with the dependent tasks, which end up failing.

@adamnovak do you have any suggestions about the most robust way of fixing this?

Logs from original failed job (I introduced a dummy exception to make this reproducible):
```
Exception in thread Thread-9:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/dist-packages/toil/fileStores/cachingFileStore.py", line 1718, in startCommitThread
    self._executePendingUploads(con, cur)
  File "/usr/local/lib/python3.6/dist-packages/toil/fileStores/cachingFileStore.py", line 764, in _executePendingUploads
    self.jobStore.updateFile(fileID, filePath)
  File "/usr/local/lib/python3.6/dist-packages/toil/jobStores/aws/jobStore.py", line 566, in updateFile
    info.upload(localFilePath, not self.config.disableJobStoreChecksumVerification)
  File "/usr/local/lib/python3.6/dist-packages/toil/jobStores/aws/jobStore.py", line 1093, in upload
    headers=headers)
  File "/usr/local/lib/python3.6/dist-packages/toil/jobStores/aws/utils.py", line 218, in uploadFromPath
    version = chunkedFileUpload(f, bucket, fileID, file_size, headers, partSize)
  File "/usr/local/lib/python3.6/dist-packages/toil/jobStores/aws/utils.py", line 231, in chunkedFileUpload
    raise Exception('fail')
Exception: fail

[2020-07-22T00:02:47+0000] [Thread-10 ] [D] [toil.fileStores.cachingFileStore] Committing file uploads asynchronously
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/toil/worker.py", line 384, in workerScript
    raise RuntimeError("The termination flag is set")
RuntimeError: The termination flag is set
[2020-07-22T00:02:47+0000] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host ip-10-0-131-223.ec2.internal
[2020-07-22T00:02:47+0000] [Thread-10 ] [D] [toil.fileStores.cachingFileStore] Committing file deletes and job state changes asynchronously
[2020-07-22T00:02:47+0000] [Thread-10 ] [D] [toil.jobStores.aws.jobStore] Updating job 4f58974a-37c5-4dc2-9e8d-d71db741a511
[2020-07-22T00:02:47+0000] [MainThread] [D] [toil.jobStores.aws.jobStore] Loaded job 4f58974a-37c5-4dc2-9e8d-d71db741a511
```

Logs from retry (should still fail, but ends up "succeeding")
```
[2020-07-22T00:02:50+0000] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
...
[2020-07-22T00:02:50+0000] [MainThread] [D] [toil.deferred] Running for file /var/lib/toil/node-8646ddba-7ebc-467c-ab31-87e55a076f82-b04db54e196814e250a043d45f15abca/deferred/func3_4hacr7
[2020-07-22T00:02:50+0000] [MainThread] [D] [toil.jobStores.aws.jobStore] Loaded job 4f58974a-37c5-4dc2-9e8d-d71db741a511
[2020-07-22T00:02:50+0000] [MainThread] [D] [toil.worker] Parsed job wrapper
[2020-07-22T00:02:50+0000] [MainThread] [D] [toil.worker] Wrapper has no user job to run.
[2020-07-22T00:02:50+0000] [MainThread] [D] [toil.worker] Cleaned up any references to completed successor jobs
[2020-07-22T00:02:50+0000] [MainThread] [D] [toil.jobStores.aws.jobStore] Updating job 4f58974a-37c5-4dc2-9e8d-d71db741a511
[2020-07-22T00:02:50+0000] [MainThread] [D] [toil.worker] No user job to run, so finishing
[2020-07-22T00:02:50+0000] [MainThread] [I] [toil.worker] Worker log can be found at /var/lib/toil/node-8646ddba-7ebc-467c-ab31-87e55a076f82-b04db54e196814e250a043d45f15abca/tmp9n8uc32e. Set --cleanWorkDir to retain this log
```

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-590)
┆Issue Number: TOIL-590



Discussion:
I *think* `_serialiseExistingJob` is supposed to pickle the job but not actually commit the changes. See the description for `_serialiseJobGraph`:

https://github.com/DataBiosphere/toil/blob/bd8c6d16eca1cba8a54d45ee2359e49aa9b0badf/src/toil/job.py#L1260-L1265

The commit back to the job store is I think supposed to happen here (at least with caching on?):

https://github.com/DataBiosphere/toil/blob/e6dea9602ca05dcf28b190be97673fd57f69022f/src/toil/fileStores/cachingFileStore.py#L1730-L1731

After that, the only thing to do is cleanup, so if we fail the job won't need to rerun (but we might want to retry the cleanup?).

It looks to me like your steps 2 and 3 shouldn't be happening until after step 4, because `startCommit` kicks off `startCommitThread` for the current job, which needs to happen in order for 2 to happen. Also, 2 doesn't involve `DeferredFunctionManager` in your example; instead the issue is with "deferred" uploading of files by the file store, which doesn't involve the deferred function system (because we are reusing the word when we shouldn't).

The `startCommitThread` thread is supposed to [commit the pending uploads](https://github.com/DataBiosphere/toil/blob/e6dea9602ca05dcf28b190be97673fd57f69022f/src/toil/fileStores/cachingFileStore.py#L1718) before it [commits the completion of the job](https://github.com/DataBiosphere/toil/blob/e6dea9602ca05dcf28b190be97673fd57f69022f/src/toil/fileStores/cachingFileStore.py#L1723) to avoid exactly this sort of problem. I don't really see how we're managing to get around that.

Now, the way in which these things can interleave in a different order is if one job chains to another job. The first job can be finished, with its uploads happening in the background, while the second job is running (because the cache is there to fulfill reads for the second job). But if the first job's background uploads fail, we're *supposed* to not commit the second job's results, and rerun from that first job. Maybe something is going wrong with notions of "current" job that is allowing the second job's view of what the first job ought to look like to make it to the job store, even when the first job's commit step failed?

Your example definitely has two instances of `startCommitThread` running. `Thread-9` crashes but then `Thread-10` runs again and commits the changes to the job. Maybe we're able to start the commit thread multiple times if the first one fails fast enough, when we really just meant to ensure that the async commit has been kicked off, and a failure of the first attempt lets the second attempt skip over actually doing the uploads and jump straight to updating the job state?

Thanks for the explanation, @adamnovak! I think I now know what's causing the bug. There are indeed two threads as you mentioned. The first one is called [here](https://github.com/DataBiosphere/toil/blob/bd8c6d16eca1cba8a54d45ee2359e49aa9b0badf/src/toil/worker.py#L371) while the second one is called [here](https://github.com/DataBiosphere/toil/blob/bd8c6d16eca1cba8a54d45ee2359e49aa9b0badf/src/toil/job.py#L1389).

The first one fails (and is not supposed to update `jobState` anyway), but it has already updated the state from `uploadable` to `uploading` in the DB. The 2nd thread waits for the first one to complete (i.e. `self.commitThread` is not None), and when it starts, it sees that there are no files in the `uploadable` state, so `_executePendingUploads` returns immediately without any errors.

I think we should somehow revert the state back to `uploadable` when the first thread fails. Any ideas how best to do this? Alternatively, we could just remove the [first thread](https://github.com/DataBiosphere/toil/blob/bd8c6d16eca1cba8a54d45ee2359e49aa9b0badf/src/toil/worker.py#L371). I don't think it'd really buy us much?

P.S. I have disabled chaining in my case, so it's not related to that.
Yeah, I don't see a good reason to kick off the commit before we leave the executor context *and* as part of leaving the executor context. 

All the caching database logic for picking up and retrying uploads that failed is based around finding failed *processes*. The mechanism we use can't really identify that a *thread* in the *current* process dropped an upload on the floor. I think we need to just handle the errors: hava a try/except [here](https://github.com/DataBiosphere/toil/blob/bd8c6d16eca1cba8a54d45ee2359e49aa9b0badf/src/toil/fileStores/cachingFileStore.py#L764) that sets the state back to uploadable, and similarly for pending deletions, before re-raising.



Resolving patch:
diff --git a/src/toil/fileStores/cachingFileStore.py b/src/toil/fileStores/cachingFileStore.py
--- a/src/toil/fileStores/cachingFileStore.py
+++ b/src/toil/fileStores/cachingFileStore.py
@@ -761,7 +761,13 @@ def _executePendingUploads(self, con, cur):
 
             # Upload the file
             logger.debug('Actually executing upload for file %s', fileID)
-            self.jobStore.updateFile(fileID, filePath)
+            try:
+                self.jobStore.updateFile(fileID, filePath)
+            except:
+                # We need to set the state back to 'uploadable' in case of any failures to ensure
+                # we can retry properly.
+                self._staticWrite(con, cur, [('UPDATE files SET state = ? WHERE id = ? AND state = ?', ('uploadable', fileID, 'uploading'))])
+                raise
 
             # Count it for the total uploaded files value we need to return
             uploadedCount += 1
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -367,9 +367,6 @@ def workerScript(jobStore, config, jobName, jobStoreID, redirectOutputToLogFile=
 
                             job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
 
-                            # When the job succeeds, start committing files immediately.
-                            fileStore.startCommit(jobState=False)      
-
                 # Accumulate messages from this job & any subsequent chained jobs
                 statsDict.workers.logsToMaster += fileStore.loggingMessages
 


Test patch: