Repository: DataBiosphere/toil

Issue:
Restarting a workflow raises NoSuchJob exceptions
Toil is trying to run jobs that don't exist when restarting a failed pipeline.

The job that is correctly resumed is a follow-on of the root job, if that matters. I can provide a snapshot of a jobStore which triggers this bug if that would help.

```
Repairing job: 1/X/jobPiXIW2
Repairing job: c/C/jobTbIFOx
---TOIL WORKER OUTPUT LOG---
---TOIL WORKER OUTPUT LOG---
Traceback (most recent call last):
  File "/cluster/home/jcarmstr/toil/src/toil/worker.py", line 208, in main
    jobWrapper = jobStore.load(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 111, in load
    self._checkJobStoreId(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 329, in _checkJobStoreId
    raise NoSuchJobException(jobStoreID)
NoSuchJobException: The job 'c/C/jobTbIFOx' does not exist
Traceback (most recent call last):
  File "/cluster/home/jcarmstr/venv/bin/_toil_worker", line 8, in <module>
    load_entry_point('toil==3.2.0a2', 'console_scripts', '_toil_worker')()
  File "/cluster/home/jcarmstr/toil/src/toil/worker.py", line 482, in main
    jobWrapper = jobStore.load(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 111, in load
    self._checkJobStoreId(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 329, in _checkJobStoreId
    raise NoSuchJobException(jobStoreID)
toil.jobStores.abstractJobStore.NoSuchJobException: The job 'c/C/jobTbIFOx' does not exist
---TOIL WORKER OUTPUT LOG---
```



Discussion:
This is probably related to #760.

@benedictpaten is this something you'd expect during a restart?

No, this is a bug. It is happening within the worker itself, suggesting that the cleanup process that fixes the dag of jobs at the beginning of a restart is flawed. 

Here is some more detailed debug output. This job tree has two jobs left: the root job `x/c/...` and its follow-on `J/p/...`. `J/p` is scheduled first, then the root job `x/c` is scheduled, then `J/p` is pulled off the root job's stack again and scheduled, and it obviously fails to run as it's already been deleted.

```
Jobstore directory is: /hive/users/jcarmstr/tmp/jobStoreTarSyr
Checking job graph connectivity...
2 jobs reachable from root.
reset the retry count
Repairing job: J/p/jobNRU0_W
new stack size 1 is not old stack size 2295
Repairing job: x/c/jobUbROqr
Updating status of job: J/p/jobNRU0_W with result status: 0
Jobstore directory is: /hive/users/jcarmstr/tmp/jobStoreTarSyr
---TOIL WORKER OUTPUT LOG---
Jobstore directory is: /hive/users/jcarmstr/tmp/jobStoreTarSyr
Updating status of job: x/c/jobUbROqr with result status: 0
Job: x/c/jobUbROqr has 1 successors to schedule
seeing if we should schedule job J/p/jobNRU0_W
Jobstore directory is: /hive/users/jcarmstr/tmp/jobStoreTarSyr
---TOIL WORKER OUTPUT LOG---
Traceback (most recent call last):
  File "/cluster/home/jcarmstr/toil/src/toil/worker.py", line 208, in main
    jobWrapper = jobStore.load(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 111, in load
    self._checkJobStoreId(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 329, in _checkJobStoreId
    raise NoSuchJobException(jobStoreID)
NoSuchJobException: The job 'J/p/jobNRU0_W' does not exist
Traceback (most recent call last):
  File "/cluster/home/jcarmstr/venv/bin/_toil_worker", line 8, in <module>
    load_entry_point('toil==3.2.0a2', 'console_scripts', '_toil_worker')()
  File "/cluster/home/jcarmstr/toil/src/toil/worker.py", line 482, in main
    jobWrapper = jobStore.load(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 111, in load
    self._checkJobStoreId(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 329, in _checkJobStoreId
    raise NoSuchJobException(jobStoreID)
toil.jobStores.abstractJobStore.NoSuchJobException: The job 'J/p/jobNRU0_W' does not exist
Updating status of job: x/c/jobUbROqr with result status: 0
Job: x/c/jobUbROqr is empty, we are scheduling to clean it up
Jobstore directory is: /hive/users/jcarmstr/tmp/jobStoreTarSyr
---TOIL WORKER OUTPUT LOG---
Jobstore directory is: /hive/users/jcarmstr/tmp/jobStoreTarSyr
```

I'm assigning to myself, since I can reproduce this very consistently. I think I can find time to track this down sometime tomorrow.

@joelarmstrong, realistically speaking, what are the chances of you taking care of this in the next two weeks? Trying to get an realistic idea if this should be assigned to someone else.

Hey Hannes,

Sorry for the delayed response. I tracked the root cause down a while ago (failed child jobs can run before their parent on resume) but still need to implement a fix. It's top on my agenda for tomorrow.

Cool! Thank you.


Resolving patch:
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -366,6 +366,7 @@ def _updatePredecessorStatus(self, jobStoreID):
                 assert self.toilState.successorCounts[predecessorJob.jobStoreID] >= 0
                 if self.toilState.successorCounts[predecessorJob.jobStoreID] == 0: #Job is done
                     self.toilState.successorCounts.pop(predecessorJob.jobStoreID)
+                    predecessorJob.stack.pop()
                     logger.debug('Job %s has all its non-service successors completed or totally '
                                  'failed', predecessorJob.jobStoreID)
                     assert predecessorJob not in self.toilState.updatedJobs
@@ -821,7 +822,7 @@ def innerLoop(jobStore, config, batchSystem, toilState, jobBatcher, serviceManag
                     #List of successors to schedule
                     successors = []
                     #For each successor schedule if all predecessors have been completed
-                    for successorJobStoreID, memory, cores, disk, preemptable, predecessorID in jobWrapper.stack.pop():
+                    for successorJobStoreID, memory, cores, disk, preemptable, predecessorID in jobWrapper.stack[-1]:
                         #Build map from successor to predecessors.
                         if successorJobStoreID not in toilState.successorJobStoreIDToPredecessorJobs:
                             toilState.successorJobStoreIDToPredecessorJobs[successorJobStoreID] = []


Test patch: