Repository: DataBiosphere/toil

Issue:
slurm.py error, undefeind stdout - stdout should be stderr in code
There's a potential bug in slurm.py that leads to toil-cwl-runner failing. Line 127 declares stderr but line 129 references stdout leading to an error message (NameError: name 'stdout' is not defined)

Line 129 from [slurm.py](https://github.com/DataBiosphere/toil/blob/0aec2884aa829d4e1c9a888a4959b762bca39168/src/toil/batchSystems/slurm.py#L129) should be 

```python
            for line in stderr:
```

not

```python
            for line in stdout:
```

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-634)
┆Issue Number: TOIL-634

slurm.py string decode error.
Related to issue [3132](https://github.com/DataBiosphere/toil/issues/3132), line 131 of slurm.py throws the same error message.

>   File "/xxx/venv_CWL/lib/python3.7/site-packages/toil/batchSystems/slurm.py", line 131, in _getJobDetailsFromScontrol
>     values = line.decode('utf-8').strip().split()
> AttributeError: 'str' object has no attribute 'decode'

ps - I'm new to raising issues on git (plus using toil + CWL with slurm) so any hints on improving the issues raised welcomed.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-635)
┆Issue Number: TOIL-635



Discussion:

Resolving patch:
diff --git a/src/toil/batchSystems/slurm.py b/src/toil/batchSystems/slurm.py
--- a/src/toil/batchSystems/slurm.py
+++ b/src/toil/batchSystems/slurm.py
@@ -11,22 +11,12 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-
-from __future__ import absolute_import
-from __future__ import division
-from builtins import str
-from past.utils import old_div
 import logging
 import os
 from pipes import quote
-import time
 import math
 from toil.lib.misc import call_command, CalledProcessErrorStderr
 
-# Python 3 compatibility imports
-from six.moves.queue import Empty, Queue
-from six import iteritems
-
 from toil.batchSystems import MemoryString
 from toil.batchSystems.abstractGridEngineBatchSystem import AbstractGridEngineBatchSystem
 
@@ -114,9 +104,9 @@ def _getJobDetailsFromSacct(self, slurmJobID):
                     # A non-zero signal may indicate e.g. an out-of-memory killed job
                     status = 128 + signal
                 logger.debug("sacct exit code is %s, returning status %d", exitcode, status)
-                return (state, status)
+                return state, status
             logger.debug("Did not find exit code for job in sacct output")
-            return (None, None)
+            return None, None
 
         def _getJobDetailsFromScontrol(self, slurmJobID):
             args = ['scontrol',
@@ -124,16 +114,16 @@ def _getJobDetailsFromScontrol(self, slurmJobID):
                     'job',
                     str(slurmJobID)]
 
-            stderr = call_command(args)
+            stdout = call_command(args)
             job = dict()
             for line in stdout:
                 logger.debug("%s output %s", args[0], line)
-                values = line.decode('utf-8').strip().split()
+                values = line.strip().split()
 
                 # If job information is not available an error is issued:
                 # slurm_load_jobs error: Invalid job id specified
                 # There is no job information, so exit.
-                if len(values)>0 and values[0] == 'slurm_load_jobs':
+                if len(values) > 0 and values[0] == 'slurm_load_jobs':
                     return (None, None)
 
                 # Output is in the form of many key=value pairs, multiple pairs on each line
@@ -158,7 +148,7 @@ def _getJobDetailsFromScontrol(self, slurmJobID):
             except KeyError:
                 rc = None
 
-            return (state, rc)
+            return state, rc
 
         """
         Implementation-specific helper methods
@@ -179,9 +169,9 @@ def prepareSbatch(self, cpu, mem, jobID, jobName):
 
             if mem is not None:
                 # memory passed in is in bytes, but slurm expects megabytes
-                sbatch_line.append('--mem={}'.format(old_div(int(mem), 2 ** 20)))
+                sbatch_line.append(f'--mem={math.ceil(mem / 2 ** 20)}')
             if cpu is not None:
-                sbatch_line.append('--cpus-per-task={}'.format(int(math.ceil(cpu))))
+                sbatch_line.append(f'--cpus-per-task={math.ceil(cpu)}')
 
             stdoutfile = self.boss.formatStdOutErrPath(jobID, 'slurm', '%j', 'std_output')
             stderrfile = self.boss.formatStdOutErrPath(jobID, 'slurm', '%j', 'std_error')


Test patch: