Repository: DataBiosphere/toil

Issue:
Problem with --batchSystem Slurm option
I'm trying to run a CWL workflow on a Slurm queueing system. 
I am able to succesfully run a CWL workflow on a local machine (batchSystem singlemachine):
```
$ toil-cwl-runner --workDir /mnt/home/tjarosiewicz/tmp/ --jobStore test example.cwl example-job.yaml 
INFO:cwltool:Resolved 'example.cwl' to 'file:///mnt/home/tjarosiewicz/workflows/example.cwl'
WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (40).
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (201217310720).
WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (203102003134464).
INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
INFO:toil.leader:Issued job 'file:///mnt/home/tjarosiewicz/workflows/example.cwl' echo E/p/job8c_ibarm with job batch system ID: 0 and cores: 1, disk: 3.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/tmp/tmptrn1ykx7'.
INFO:toil.worker:Redirecting logging to /mnt/home/tjarosiewicz/tmp/toil-cc520ff8-24a4-46d0-896d-479c31d802f8-7a1a9ca2-0214-47f8-b07e-6c3685a89053/tmp9bsoxv63/worker_log.txt
INFO:toil.leader:Job ended successfully: 'file:///mnt/home/tjarosiewicz/workflows/example.cwl' echo E/p/job8c_ibarm
INFO:toil.leader:Finished toil run successfully.
INFO:toil.common:Successfully deleted the job store: FileJobStore(/tmp/tmptrn1ykx7)

```
I am able to succesfully run a Toil workflow (using Toil Python module) on a local machine:
```
$ python helloworld.py
WARNING:toil.batchSystems.singleMachine:Limiting maxCores to CPU count of system (40).
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (201217310720).
WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (203103171248128).
INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
INFO:toil.leader:Issued job 'helloWorld' 6/n/job7cpnl9am with job batch system ID: 0 and cores: 1, disk: 3.0 G, and memory: 2.0 G
DEBUG:toil.jobStores.fileJobStore:Path to job store directory is '/mnt/home/tjarosiewicz/workflows/toilRun'.
INFO:toil.worker:Redirecting logging to /mnt/home/tjarosiewicz/tmp/toil-dd5907f8-63ba-425c-ab15-eb9406d8dca1-7a1a9ca2-0214-47f8-b07e-6c3685a89053/tmpvmuy5qdy/worker_log.txt
INFO:toil.leader:Job ended successfully: 'helloWorld' 6/n/job7cpnl9am
INFO:toil.leader:Finished toil run successfully.
INFO:toil.common:Successfully deleted the job store: FileJobStore(/mnt/home/tjarosiewicz/workflows/toilRun)
Hello, world!, here's a message: You did it!!!!!
```

When I try to run above workflows through Slurm queueing system I get errors about job submission. 

1) With `toil-cwl-runner`:
```
$ toil-cwl-runner --batchSystem Slurm --workDir /mnt/home/tjarosiewicz/tmp/ --jobStore test  example.cwl example-job.yaml 
INFO:cwltool:Resolved 'example.cwl' to 'file:///mnt/home/tjarosiewicz/workflows/example.cwl'
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (201217310720).
WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (203101904568320).
INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
INFO:toil.leader:Issued job 'file:///mnt/home/tjarosiewicz/workflows/example.cwl' echo 8/X/job1slhcxw_ with job batch system ID: 0 and cores: 1, disk: 3.0 G, and memory: 2.0 G
Exception in thread Thread-401:
Traceback (most recent call last):
  File "/mnt/home/tjarosiewicz/anaconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/mnt/home/tjarosiewicz/venv/lib/python3.6/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 234, in run
    activity |= self.createJobs(newJob)
  File "/mnt/home/tjarosiewicz/venv/lib/python3.6/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 138, in createJobs
    batchJobID = with_retries(self.submitJob, subLine)
  File "/mnt/home/tjarosiewicz/venv/lib/python3.6/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 42, in with_retries
    return operation(*args, **kwargs)
  File "/mnt/home/tjarosiewicz/venv/lib/python3.6/site-packages/toil/batchSystems/slurm.py", line 71, in submitJob
    result = int(output.strip().split()[-1])
IndexError: list index out of range

```
2) With Toil as a Python module; I have added following line to the hello world code:
`options.batchSystem = "Slurm"`
```
$ python helloworld.py
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (201217310720).
WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (203102687854592).
WARNING:toil.common:Batch system does not support auto-deployment. The user script ModuleDescriptor(dirPath='/mnt/home/tjarosiewicz/workflows', name='agent', fromVirtualEnv=False) will have to be present at the same location on every worker.
INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
INFO:toil.leader:Issued job 'helloWorld' d/Q/jobwvjxxov7 with job batch system ID: 0 and cores: 1, disk: 3.0 G, and memory: 2.0 G
Exception in thread Thread-401:
Traceback (most recent call last):
  File "/mnt/home/tjarosiewicz/anaconda3/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/mnt/home/tjarosiewicz/venv/lib/python3.6/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 234, in run
    activity |= self.createJobs(newJob)
  File "/mnt/home/tjarosiewicz/venv/lib/python3.6/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 138, in createJobs
    batchJobID = with_retries(self.submitJob, subLine)
  File "/mnt/home/tjarosiewicz/venv/lib/python3.6/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 42, in with_retries
    return operation(*args, **kwargs)
  File "/mnt/home/tjarosiewicz/venv/lib/python3.6/site-packages/toil/batchSystems/slurm.py", line 71, in submitJob
    result = int(output.strip().split()[-1])
IndexError: list index out of range
```

I have managed to fix this issue by modifying the 173 line in `.../toil/batchSystems/slurm.py`
from 
`sbatch_line = ['sbatch', '-Q', '-J', 'toil_job_{}'.format(jobID)]`
to
`sbatch_line = ['sbatch', '-J', 'toil_job_{}'.format(jobID)]`

With Toil CWL runner:
```
$ toil-cwl-runner --batchSystem Slurm --workDir /mnt/home/tjarosiewicz/tmp/ --jobStore test example.cwl example-job.yaml 
INFO:cwltool:Resolved 'example.cwl' to 'file:///mnt/home/tjarosiewicz/workflows/example.cwl'
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (201217310720).
WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (203094966140928).
INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
INFO:toil.leader:Issued job 'file:///mnt/home/tjarosiewicz/workflows/example.cwl' echo i/k/jobdhlfuse_ with job batch system ID: 0 and cores: 1, disk: 3.0 G, and memory: 2.0 G
INFO:toil.leader:Job ended successfully: 'file:///mnt/home/tjarosiewicz/workflows/example.cwl' echo i/k/jobdhlfuse_
INFO:toil.leader:Finished toil run successfully.
INFO:toil.common:Successfully deleted the job store: FileJobStore(/mnt/home/tjarosiewicz/workflows/test)
```

With Toil Python module:
```
$ python helloworld.py 
WARNING:toil.batchSystems.singleMachine:Limiting maxMemory to physically available memory (201217310720).
WARNING:toil.batchSystems.singleMachine:Limiting maxDisk to physically available disk (203094390472704).
WARNING:toil.common:Batch system does not support auto-deployment. The user script ModuleDescriptor(dirPath='/mnt/home/tjarosiewicz/workflows', name='helloworld', fromVirtualEnv=False) will have to be present at the same location on every worker.
INFO:toil:Running Toil version 3.20.0-cf34ca3416697f2abc816b2538f20ee29ba16932.
INFO:toil.leader:Issued job 'helloWorld' 9/X/jobio7nz7hm with job batch system ID: 0 and cores: 1, disk: 3.0 G, and memory: 2.0 G
INFO:toil.leader:Job ended successfully: 'helloWorld' 9/X/jobio7nz7hm
INFO:toil.leader:Finished toil run successfully.
INFO:toil.common:Successfully deleted the job store: FileJobStore(/mnt/home/tjarosiewicz/workflows/toilRun)
Hello, world!, here's a message: You did it!!!!!
```

Both examples are taken from the tutorial; https://toil.readthedocs.io/en/latest/gettingStarted/quickStart.html#running-a-basic-cwl-workflow

I haven't noticed other people having issues with Slurm batch system so maybe it is a problem with my Slurm configuration?
I will be happy to learn if there are other ways of dealing with this problem.

Cheers, 
Tobiasz

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-411)
┆Issue Number: TOIL-411



Discussion:
Yeah, it definitely looks like that Q should not be there, since the docs say it suppresses the output line with the job ID, and since we always try to parse the job ID from the `sbatch` output. I'm not sure how this would possibly be working for anyone else.

Can you submit your modification as a PR?

Resolving patch:
diff --git a/src/toil/batchSystems/slurm.py b/src/toil/batchSystems/slurm.py
--- a/src/toil/batchSystems/slurm.py
+++ b/src/toil/batchSystems/slurm.py
@@ -170,7 +170,7 @@ def _getJobDetailsFromScontrol(self, slurmJobID):
 
         def prepareSbatch(self, cpu, mem, jobID):
             #  Returns the sbatch command line before the script to run
-            sbatch_line = ['sbatch', '-Q', '-J', 'toil_job_{}'.format(jobID)]
+            sbatch_line = ['sbatch', '-J', 'toil_job_{}'.format(jobID)]
 
             if self.boss.environment:
                 argList = []


Test patch: