Repository: DataBiosphere/toil

Issue:
Cactus slurm support is broken in 5.10
I've been using https://github.com/DataBiosphere/toil/commit/356261d5ac84913d4c4f897d990c285e2a7e11f6 without much problem (expect for the extra delay for each job).  When I upgrade to 5.10, the delay is gone, but Cactus stops suddenly after a few jobs

to reproduce, do something like:
```
git clone https://github.com/ComparativeGenomicsToolkit/cactus.git --branch t2t_apes_glenn --recursive
cd cactus
virtualenv -p python3 venv
. venv/bin/activate
export PATH=$(pwd)/bin:$PATH
make -j 8
pip install -U .
pip install toil
cactus ./js ./examples/evolverMammals.txt em.hal --batchSystem slurm --consCores 8
```

The entire log for the above command
```
[2023-05-18T18:50:06-0700] [MainThread] [I] [toil.statsAndLogging] Enabling realtime logging in Toil
[2023-05-18T18:50:06-0700] [MainThread] [I] [toil.statsAndLogging] Cactus Command: /private/home/hickey/dev/yon/cactus/venv/bin/cactus ./js ./examples/evolverMammals.txt em.hal --batchSystem slurm --consCores 8
[2023-05-18T18:50:06-0700] [MainThread] [I] [toil.statsAndLogging] Cactus Commit: b9f74df17db1edeb3dde16bb092340daf38f0df7
[2023-05-18T18:50:06-0700] [MainThread] [I] [toil.statsAndLogging] Tree: ((simHuman_chr6:0.144018,(simMouse_chr6:0.084509,simRat_chr6:0.091589)mr:0.271974)Anc1:0.020593,(simCow_chr6:0.18908,simDog_chr6:0.16303)Anc2:0.032898)Anc0;
[2023-05-18T18:50:06-0700] [MainThread] [I] [toil.statsAndLogging] Importing https://raw.githubusercontent.com/ComparativeGenomicsToolkit/cactusTestData/master/evolver/mammals/loci1/simCow.chr6
[2023-05-18T18:50:06-0700] [MainThread] [I] [toil.statsAndLogging] Importing https://raw.githubusercontent.com/ComparativeGenomicsToolkit/cactusTestData/master/evolver/mammals/loci1/simDog.chr6
[2023-05-18T18:50:07-0700] [MainThread] [I] [toil.statsAndLogging] Importing https://raw.githubusercontent.com/ComparativeGenomicsToolkit/cactusTestData/master/evolver/mammals/loci1/simHuman.chr6
[2023-05-18T18:50:07-0700] [MainThread] [I] [toil.statsAndLogging] Importing https://raw.githubusercontent.com/ComparativeGenomicsToolkit/cactusTestData/master/evolver/mammals/loci1/simMouse.chr6
[2023-05-18T18:50:07-0700] [MainThread] [I] [toil.statsAndLogging] Importing https://raw.githubusercontent.com/ComparativeGenomicsToolkit/cactusTestData/master/evolver/mammals/loci1/simRat.chr6
[2023-05-18T18:50:07-0700] [MainThread] [W] [toil.common] Batch system does not support auto-deployment. The user script ModuleDescriptor(dirPath='/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages', name='cactus.progressive.cactus_progressive', fromVirtualEnv=True) will have to be present at the same location on every worker.
[2023-05-18T18:50:08-0700] [MainThread] [I] [toil.job] Saving graph of 1 jobs, 1 non-service, 1 new
[2023-05-18T18:50:08-0700] [MainThread] [I] [toil.job] Processing job 'progressive_workflow' kind-progressive_workflow/instance-uurk8yl0 v0
[2023-05-18T18:50:08-0700] [MainThread] [I] [toil] Running Toil version 5.10.0-21422a3440f8a6d5e9d2f1c9695c4fbc57fa5372 on host phoenix.prism.
[2023-05-18T18:50:08-0700] [MainThread] [I] [toil.realtimeLogger] Starting real-time logging.
[2023-05-18T18:50:08-0700] [MainThread] [I] [toil.leader] Issued job 'progressive_workflow' kind-progressive_workflow/instance-uurk8yl0 
v1 with job batch system ID: 0 and disk: 2.0 Gi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2023-05-18T18:50:10-0700] [MainThread] [I] [toil.leader] 1 jobs are running, 0 jobs are issued and waiting to run
[2023-05-18T18:50:11-0700] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-ezqyigp
5 v1 with job batch system ID: 1 and disk: 4.3 Mi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2023-05-18T18:50:11-0700] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-fwub4n9
8 v1 with job batch system ID: 2 and disk: 4.1 Mi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2023-05-18T18:50:11-0700] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-ocg9wv7
t v1 with job batch system ID: 3 and disk: 4.4 Mi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2023-05-18T18:50:11-0700] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-ujun2f3
o v1 with job batch system ID: 4 and disk: 4.0 Mi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2023-05-18T18:50:11-0700] [MainThread] [I] [toil.leader] Issued job 'sanitize_fasta_header' kind-sanitize_fasta_header/instance-wo03bs6
_ v1 with job batch system ID: 5 and disk: 4.1 Mi, memory: 2.0 Gi, cores: 1, accelerators: [], preemptible: False
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.062006: Running the command: "cactus_sanitizeFastaHeaders /dat
a/tmp/0a0358daf11a586aa564d94f4b8800ce/f194/e30b/tmptl43fu7i/simDog_chr6.fa simDog_chr6"
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.062570: Running the command: "cactus_sanitizeFastaHeaders /dat
a/tmp/0a0358daf11a586aa564d94f4b8800ce/bc15/2eaa/tmpe9fbpe3q/simHuman_chr6.fa simHuman_chr6"
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.062637: Running the command: "cactus_sanitizeFastaHeaders /dat
a/tmp/0a0358daf11a586aa564d94f4b8800ce/d1fc/9011/tmpy8h2w0fz/simMouse_chr6.fa simMouse_chr6"
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.062697: Running the command: "cactus_sanitizeFastaHeaders /dat
a/tmp/0a0358daf11a586aa564d94f4b8800ce/3040/17e8/tmpkcvg0576/simRat_chr6.fa simRat_chr6"
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.062791: Running the command: "cactus_sanitizeFastaHeaders /dat
a/tmp/0a0358daf11a586aa564d94f4b8800ce/090a/f647/tmpg3a1pyne/simCow_chr6.fa simCow_chr6"
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.072270: Successfully ran: "cactus_sanitizeFastaHeaders /data/t
mp/0a0358daf11a586aa564d94f4b8800ce/f194/e30b/tmptl43fu7i/simDog_chr6.fa simDog_chr6" in 0.0094 seconds and 2.7 Mi memory
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.072348: Successfully ran: "cactus_sanitizeFastaHeaders /data/t
mp/0a0358daf11a586aa564d94f4b8800ce/090a/f647/tmpg3a1pyne/simCow_chr6.fa simCow_chr6" in 0.0089 seconds and 2.6 Mi memory
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.072473: Successfully ran: "cactus_sanitizeFastaHeaders /data/t
mp/0a0358daf11a586aa564d94f4b8800ce/d1fc/9011/tmpy8h2w0fz/simMouse_chr6.fa simMouse_chr6" in 0.009 seconds and 2.7 Mi memory
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.072513: Running the command: "cactus_analyseAssembly simDog_ch
r6.sanitized.fa"
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.072590: Running the command: "cactus_analyseAssembly simCow_ch
r6.sanitized.fa"
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.072702: Running the command: "cactus_analyseAssembly simMouse_
chr6.sanitized.fa"
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.072742: Successfully ran: "cactus_sanitizeFastaHeaders /data/t
mp/0a0358daf11a586aa564d94f4b8800ce/3040/17e8/tmpkcvg0576/simRat_chr6.fa simRat_chr6" in 0.0093 seconds and 2.7 Mi memory
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.072775: Successfully ran: "cactus_sanitizeFastaHeaders /data/t
mp/0a0358daf11a586aa564d94f4b8800ce/bc15/2eaa/tmpe9fbpe3q/simHuman_chr6.fa simHuman_chr6" in 0.0093 seconds and 2.7 Mi memory
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.072975: Running the command: "cactus_analyseAssembly simRat_ch
r6.sanitized.fa"
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.073008: Running the command: "cactus_analyseAssembly simHuman_
chr6.sanitized.fa"
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.080793: Successfully ran: "cactus_analyseAssembly simCow_chr6.
sanitized.fa" in 0.0078 seconds and 2.6 Mi memory
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.080841: Successfully ran: "cactus_analyseAssembly simHuman_chr
6.sanitized.fa" in 0.0075 seconds and 2.6 Mi memory
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.081031: Successfully ran: "cactus_analyseAssembly simDog_chr6.
sanitized.fa" in 0.0082 seconds and 2.7 Mi memory
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.081082: Successfully ran: "cactus_analyseAssembly simRat_chr6.
sanitized.fa" in 0.0078 seconds and 2.6 Mi memory
[2023-05-18T18:50:13-0700] [MainThread] [I] [toil-rt] 2023-05-18 18:50:13.081085: Successfully ran: "cactus_analyseAssembly simMouse_chr
6.sanitized.fa" in 0.008 seconds and 2.6 Mi memory
[2023-05-18T18:50:13-0700] [Thread-5 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] Got message from job at time 05-18-2023 18:
50:13: Assembly stats for simHuman_chr6: Input-sample: simHuman_chr6.sanitized.fa Total-sequences: 1 Total-length: 601863 Proportion-rep
eat-masked: 0.074751 ProportionNs: 0.000000 Total-Ns: 0 N50: 601863 Median-sequence-length: 601863 Max-sequence-length: 601863 Min-seque
nce-length: 601863

[2023-05-18T18:50:13-0700] [Thread-5 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] Got message from job at time 05-18-2023 18:
50:13: Assembly stats for simDog_chr6: Input-sample: simDog_chr6.sanitized.fa Total-sequences: 1 Total-length: 593897 Proportion-repeat-
masked: 0.045009 ProportionNs: 0.000000 Total-Ns: 0 N50: 593897 Median-sequence-length: 593897 Max-sequence-length: 593897 Min-sequence-
length: 593897

[2023-05-18T18:50:13-0700] [Thread-5 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] Got message from job at time 05-18-2023 18:
50:13: Assembly stats for simCow_chr6: Input-sample: simCow_chr6.sanitized.fa Total-sequences: 1 Total-length: 602619 Proportion-repeat-
masked: 0.054570 ProportionNs: 0.000000 Total-Ns: 0 N50: 602619 Median-sequence-length: 602619 Max-sequence-length: 602619 Min-sequence-
length: 602619

[2023-05-18T18:50:13-0700] [Thread-5 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] Got message from job at time 05-18-2023 18:
50:13: Assembly stats for simRat_chr6: Input-sample: simRat_chr6.sanitized.fa Total-sequences: 1 Total-length: 647215 Proportion-repeat-
masked: 0.075276 ProportionNs: 0.000000 Total-Ns: 0 N50: 647215 Median-sequence-length: 647215 Max-sequence-length: 647215 Min-sequence-
length: 647215

[2023-05-18T18:50:13-0700] [Thread-5 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] Got message from job at time 05-18-2023 18:
50:13: Assembly stats for simMouse_chr6: Input-sample: simMouse_chr6.sanitized.fa Total-sequences: 1 Total-length: 636262 Proportion-rep
eat-masked: 0.056477 ProportionNs: 0.000000 Total-Ns: 0 N50: 636262 Median-sequence-length: 636262 Max-sequence-length: 636262 Min-seque
nce-length: 636262


Workflow Progress  33%|██████████████████████▋                                             | 2/6 (0 failures) [00:05<00:11, 0.37 jobs/s]
[2023-05-18T18:50:15-0700] [MainThread] [I] [toil.realtimeLogger] Stopping real-time logging server.
[2023-05-18T18:50:15-0700] [MainThread] [I] [toil.realtimeLogger] Joining real-time logging server thread.
Traceback (most recent call last):
  File "/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages/toil/jobStores/fileJobStore.py", line 229, in load_job
    with open(jobFile, 'rb') as fileHandle:
FileNotFoundError: [Errno 2] No such file or directory: '/private/home/hickey/dev/yon/cactus/js/jobs/kind-sanitize_fasta_header/instance-ezqyigp5/job'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/home/hickey/dev/yon/cactus/venv/bin/cactus", line 8, in <module>
    sys.exit(main())
  File "/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages/cactus/progressive/cactus_progressive.py", line 432, in main
    hal_id = toil.start(Job.wrapJobFn(progressive_workflow, options, config_node, mc_tree, og_map, input_seq_id_map))
  File "/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages/toil/common.py", line 1064, in start
    return self._runMainLoop(rootJobDescription)
  File "/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages/toil/common.py", line 1516, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages/toil/leader.py", line 251, in run
    self.innerLoop()
  File "/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages/toil/leader.py", line 752, in innerLoop
    self._gatherUpdatedJobs(updatedJobTuple)
  File "/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages/toil/leader.py", line 708, in _gatherUpdatedJobs
    self.process_finished_job(bsID, exitStatus, wall_time=wallTime, exit_reason=exitReason)
  File "/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages/toil/leader.py", line 1141, in process_finished_job
    return self.process_finished_job_description(issued_job, result_status, wall_time, exit_reason, batch_system_id)
  File "/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages/toil/leader.py", line 1171, in process_finished_job_description
    self.toilState.reset_job(job_store_id)
  File "/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages/toil/toilState.py", line 175, in reset_job
    new_truth = self.__job_store.load_job(job_id)
  File "/private/home/hickey/dev/yon/cactus/venv/lib/python3.10/site-packages/toil/jobStores/fileJobStore.py", line 236, in load_job
    raise NoSuchJobException()
TypeError: NoSuchJobException.__init__() missing 1 required positional argument: 'jobStoreID'

```

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1339)
┆Issue Number: TOIL-1339



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/kubernetes.py b/src/toil/batchSystems/kubernetes.py
--- a/src/toil/batchSystems/kubernetes.py
+++ b/src/toil/batchSystems/kubernetes.py
@@ -306,7 +306,7 @@ def drop_boring(here: Dict[str, Any]) -> None:
                 del here[k]
 
         drop_boring(root_dict)
-        return cast(str, yaml.dump(root_dict))
+        return yaml.dump(root_dict)
 
     @overload
     def _api(
diff --git a/src/toil/jobStores/fileJobStore.py b/src/toil/jobStores/fileJobStore.py
--- a/src/toil/jobStores/fileJobStore.py
+++ b/src/toil/jobStores/fileJobStore.py
@@ -233,7 +233,7 @@ def load_job(self, job_id):
             # This is the good case; the delete arrived in time.
             # If it didn't, we might go on to re-execute the already-finished job.
             # Anyway, this job doesn't really exist after all.
-            raise NoSuchJobException()
+            raise NoSuchJobException(job_id)
 
         # Pass along the current config, which is the JobStore's responsibility.
         job.assignConfig(self.config)


Test patch: