Repository: DataBiosphere/toil

Issue:
arguments passed using TOIL_SLURM_ARGS not recognized
Hi, 

I am trying to modify the default Slurm parameter via TOIL_SLURM_ARGS. As far as I can tell the env variable looks OK. 

[]export TOIL_SLURM_ARGS="-t 1:00:00 --qos=lassance"
[]env | grep TOIL
TOIL_SLURM_ARGS=-t 1:00:00 --qos=lassance

However, from the error message I get, things do not seem to be parsed properly ( I don't undertand why the valueError is raised as neither "--mem" or "--cpus-per-task" are present in the args )
  
Exception in thread Thread-200:
Traceback (most recent call last):
  File "/n/home01/lassance/.conda/envs/ENV_PROGRESSIVECACTUS/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 209, in run
    activity |= self.createJobs(newJob)
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 115, in createJobs
    subLine = self.prepareSubmission(cpu, memory, jobID, command)
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/slurm.py", line 61, in prepareSubmission
    return self.prepareSbatch(cpu, memory, jobID) + ['--wrap={}'.format(command)]
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/slurm.py", line 188, in prepareSbatch
    raise ValueError("Some resource arguments are incompatible: {}".format(nativeConfig))
ValueError: Some resource arguments are incompatible: -t 1:00:00 --qos=lassance

Any help troubleshooting this would be greatly appreciated. 

Thanks

JML



arguments passed using TOIL_SLURM_ARGS not recognized
Hi, 

I am trying to modify the default Slurm parameter via TOIL_SLURM_ARGS. As far as I can tell the env variable looks OK. 

[]export TOIL_SLURM_ARGS="-t 1:00:00 --qos=lassance"
[]env | grep TOIL
TOIL_SLURM_ARGS=-t 1:00:00 --qos=lassance

However, from the error message I get, things do not seem to be parsed properly ( I don't undertand why the valueError is raised as neither "--mem" or "--cpus-per-task" are present in the args )
  
Exception in thread Thread-200:
Traceback (most recent call last):
  File "/n/home01/lassance/.conda/envs/ENV_PROGRESSIVECACTUS/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 209, in run
    activity |= self.createJobs(newJob)
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 115, in createJobs
    subLine = self.prepareSubmission(cpu, memory, jobID, command)
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/slurm.py", line 61, in prepareSubmission
    return self.prepareSbatch(cpu, memory, jobID) + ['--wrap={}'.format(command)]
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/slurm.py", line 188, in prepareSbatch
    raise ValueError("Some resource arguments are incompatible: {}".format(nativeConfig))
ValueError: Some resource arguments are incompatible: -t 1:00:00 --qos=lassance

Any help troubleshooting this would be greatly appreciated. 

Thanks

JML





Discussion:
@lassancejm looks like a bug in the code - `if 'mem' or...` will always be True! This is on master, right? I should have time to fix this tomorrow. Thanks for reporting it!
@cket Yes, it is on the master; thanks for making the necessary correction.
I changed the  line 187         
             ` if "--mem" or "--cpus-per-task" in nativeConfig:`
 by: 
              `if ("--mem" or "--cpus-per-task") in nativeConfig:`
Now it works ... as long as I pass only one argument via TOIL_SLURM_ARGS:
              [OK] TOIL_SLURM_ARGS="--time=01:00:00"
              [FAIL] TOIL_SLURM_ARGS="--time=01:00:00 --partition=general"

error message
toil.batchSystems.slurm ERROR sbatch command failed with code 1: sbatch: error: Invalid time limit specification
Exception in thread Thread-200:
Traceback (most recent call last):
  File "/n/home01/lassance/.conda/envs/ENV_PROGRESSIVECACTUS/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 209, in run
    activity |= self.createJobs(newJob)
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 119, in createJobs
    batchJobID = self.submitJob(subLine)
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/slurm.py", line 72, in submitJob
    raise e
CalledProcessError: Command '['sbatch', '-Q', '-J', 'toil_job_0', '--mem=8192', '--cpus-per-task=1', '--time=01:00:00 --partition=general', '--wrap=_toil_worker file:/n/home01/lassance/Comparative-Annotation-Toolkit/test_install/toil/chaining/jobStore U/E/jobR3GZY6']' returned non-zero exit status 1

I am guessing that appending nativeConfig to sbatch_line as is causing the problem (line 190)

`sbatch_line.extend([nativeConfig])`
should be replaced by 
` sbatch_line.extend(nativeConfig.split())`


Thanks




@lassancejm looks like a bug in the code - `if 'mem' or...` will always be True! This is on master, right? I should have time to fix this tomorrow. Thanks for reporting it!
@cket Yes, it is on the master; thanks for making the necessary correction.
I changed the  line 187         
             ` if "--mem" or "--cpus-per-task" in nativeConfig:`
 by: 
              `if ("--mem" or "--cpus-per-task") in nativeConfig:`
Now it works ... as long as I pass only one argument via TOIL_SLURM_ARGS:
              [OK] TOIL_SLURM_ARGS="--time=01:00:00"
              [FAIL] TOIL_SLURM_ARGS="--time=01:00:00 --partition=general"

error message
toil.batchSystems.slurm ERROR sbatch command failed with code 1: sbatch: error: Invalid time limit specification
Exception in thread Thread-200:
Traceback (most recent call last):
  File "/n/home01/lassance/.conda/envs/ENV_PROGRESSIVECACTUS/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 209, in run
    activity |= self.createJobs(newJob)
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/abstractGridEngineBatchSystem.py", line 119, in createJobs
    batchJobID = self.submitJob(subLine)
  File "/n/home01/lassance/.local/lib/python2.7/site-packages/toil/batchSystems/slurm.py", line 72, in submitJob
    raise e
CalledProcessError: Command '['sbatch', '-Q', '-J', 'toil_job_0', '--mem=8192', '--cpus-per-task=1', '--time=01:00:00 --partition=general', '--wrap=_toil_worker file:/n/home01/lassance/Comparative-Annotation-Toolkit/test_install/toil/chaining/jobStore U/E/jobR3GZY6']' returned non-zero exit status 1

I am guessing that appending nativeConfig to sbatch_line as is causing the problem (line 190)

`sbatch_line.extend([nativeConfig])`
should be replaced by 
` sbatch_line.extend(nativeConfig.split())`


Thanks





Resolving patch:
diff --git a/src/toil/batchSystems/slurm.py b/src/toil/batchSystems/slurm.py
--- a/src/toil/batchSystems/slurm.py
+++ b/src/toil/batchSystems/slurm.py
@@ -184,10 +184,10 @@ def prepareSbatch(self, cpu, mem, jobID):
             nativeConfig = os.getenv('TOIL_SLURM_ARGS')
             if nativeConfig is not None:
                 logger.debug("Native SLURM options appended to sbatch from TOIL_SLURM_RESOURCES env. variable: {}".format(nativeConfig))
-                if "--mem" or "--cpus-per-task" in nativeConfig:
+                if ("--mem" in nativeConfig) or ("--cpus-per-task" in nativeConfig):
                     raise ValueError("Some resource arguments are incompatible: {}".format(nativeConfig))
 
-                sbatch_line.extend([nativeConfig])
+                sbatch_line.extend(nativeConfig.split())
 
             return sbatch_line
 


Test patch: