Repository: databiosphere/toil
Issue: Restarting a workflow raises NoSuchJob exceptions

Toil is trying to run jobs that don't exist when restarting a failed pipeline.

The job that is correctly resumed is a follow-on of the root job, if that matters. I can provide a snapshot of a jobStore which triggers this bug if that would help.

```
Repairing job: 1/X/jobPiXIW2
Repairing job: c/C/jobTbIFOx
---TOIL WORKER OUTPUT LOG---
---TOIL WORKER OUTPUT LOG---
Traceback (most recent call last):
  File "/cluster/home/jcarmstr/toil/src/toil/worker.py", line 208, in main
    jobWrapper = jobStore.load(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 111, in load
    self._checkJobStoreId(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 329, in _checkJobStoreId
    raise NoSuchJobException(jobStoreID)
NoSuchJobException: The job 'c/C/jobTbIFOx' does not exist
Traceback (most recent call last):
  File "/cluster/home/jcarmstr/venv/bin/_toil_worker", line 8, in <module>
    load_entry_point('toil==3.2.0a2', 'console_scripts', '_toil_worker')()
  File "/cluster/home/jcarmstr/toil/src/toil/worker.py", line 482, in main
    jobWrapper = jobStore.load(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 111, in load
    self._checkJobStoreId(jobStoreID)
  File "/cluster/home/jcarmstr/toil/src/toil/jobStores/fileJobStore.py", line 329, in _checkJobStoreId
    raise NoSuchJobException(jobStoreID)
toil.jobStores.abstractJobStore.NoSuchJobException: The job 'c/C/jobTbIFOx' does not exist
---TOIL WORKER OUTPUT LOG---
```


Changed files:
src/toil/leader.py

diff --git a/src/toil/leader.py b/src/toil/leader.py
index 1c2a1993..fc6f8a8c 100644
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -366,6 +366,7 @@ class JobBatcher:
                 assert self.toilState.successorCounts[predecessorJob.jobStoreID] >= 0
                 if self.toilState.successorCounts[predecessorJob.jobStoreID] == 0: #Job is done
                     self.toilState.successorCounts.pop(predecessorJob.jobStoreID)
+                    predecessorJob.stack.pop()
                     logger.debug('Job %s has all its non-service successors completed or totally '
                                  'failed', predecessorJob.jobStoreID)
                     assert predecessorJob not in self.toilState.updatedJobs
@@ -821,7 +822,7 @@ def innerLoop(jobStore, config, batchSystem, toilState, jobBatcher, serviceManag
                     #List of successors to schedule
                     successors = []
                     #For each successor schedule if all predecessors have been completed
-                    for successorJobStoreID, memory, cores, disk, preemptable, predecessorID in jobWrapper.stack.pop():
+                    for successorJobStoreID, memory, cores, disk, preemptable, predecessorID in jobWrapper.stack[-1]:
                         #Build map from successor to predecessors.
                         if successorJobStoreID not in toilState.successorJobStoreIDToPredecessorJobs:
                             toilState.successorJobStoreIDToPredecessorJobs[successorJobStoreID] = []