Repository: DataBiosphere/toil

Issue:
Failure when running CWL on slurm
I'm having trouble getting a simple CWL definition to execute on slurm.  Can someone help me understand what I'm messing up?

The error is ` IOError: [Errno 2] No such file or directory: 'file:/working/cwarth/cft/toil/jobStore/results.txt'`

This looks very similar to #1108 when using the GridEngine batch system, but the workaround of using ` --jobStore ./jobStore` does not help.
&nbsp;
      * [`cwltoil` fails when run with `--batchSystem slurm`](#cwltoil_w_slurm)
      * [version info](#version)
      * [`hello.cwl` and `hello.yml` test scripts](#sources)

Working Examples:
      * [Toil script without CWL works with Slurm,](#toil_wo_cwl)
      * [`cwltool` can run CWL without Slurm](#cwltool_wo_slurm)
      * [`cwltoil` can run CWL without Slurm](#cwltoil_wo_slurm)

<h2 id="cwltoil_w_slurm">`cwltoil` fails when run with `--batchSystem slurm`</h2>

```
$ cwltoil --logDebug --batchSystem slurm --disableCaching --jobStore ./jobStore --no-container hello.cwl hello.yml                                                                                                                                                                                                             
stoat 2016-11-01 10:59:50,870 MainThread INFO toil.lib.bioio: Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'.
stoat 2016-11-01 10:59:54,277 MainThread INFO toil.lib.bioio: Root logger is at level 'DEBUG', 'toil' logger at level 'DEBUG'.
stoat 2016-11-01 10:59:54,718 MainThread INFO toil.jobStores.fileJobStore: Path to job store directory is '/working/cwarth/cft/toil/jobStore'.
stoat 2016-11-01 10:59:54,844 MainThread INFO toil.jobStores.abstractJobStore: The workflow ID is: '3c805a37-44aa-4c4d-84f0-38faa5370313'
stoat 2016-11-01 10:59:55,293 MainThread INFO toil.common: Using the slurm batch system
stoat 2016-11-01 10:59:55,293 MainThread DEBUG toil.batchSystems.slurm: slurm results file is file:/working/cwarth/cft/toil/jobStore/results.txt
Traceback (most recent call last):
  File "/working/cwarth/cft/toil/venv/bin/cwltoil", line 11, in <module>
    load_entry_point('toil', 'console_scripts', 'cwltoil')()
  File "/working/cwarth/cft/toil/venv/src/toil/src/toil/cwl/cwltoil.py", line 665, in main
    outobj = toil.start(wf1)
  File "/working/cwarth/cft/toil/venv/src/toil/src/toil/common.py", line 591, in start
    self._batchSystem = self.createBatchSystem(self.config)
  File "/working/cwarth/cft/toil/venv/src/toil/src/toil/common.py", line 758, in createBatchSystem
    return batchSystemClass(**kwargs)
  File "/working/cwarth/cft/toil/venv/src/toil/src/toil/batchSystems/slurm.py", line 282, in __init__
    self.slurmResultsFileHandle = open(self.slurmResultsFile, 'w')
IOError: [Errno 2] No such file or directory: 'file:/working/cwarth/cft/toil/jobStore/results.txt'

```

<h2 id="version">version info</h2>

```
$ python --version
Python 2.7.6
(venv)cwarth@stoat ~/src/matsen/cft/toil (master) 
$ pip freeze | grep toil
toil==3.5.0a1
```

`toil` is installed directly from github with,

```
pip install -e git+https://github.com/BD2KGenomics/toil#egg=toil[cwl]
```

<h2 id="sources">`hello.cwl` and `hello.yml` test scripts</h2>

```
$ cat hello.cwl
cwlVersion: v1.0
class: CommandLineTool
baseCommand: echo
inputs:
  message:
    type: string
    inputBinding:
      position: 1
outputs: 
  output:
    type: stdout
stdout: $(inputs.message).ext
```

```
$ cat hello.yml
message: foo.bar
```


<h2 id="toil_wo_cwl">Toil script without CWL works with Slurm,</h2>

```
$ python HelloWorld.py --batchSystem slurm --disableCaching --logDebug ./jobStore
$ python HelloWorld.py  --logDebug --batchSystem slurm --disableCaching --jobStore ./jobStore --logFile logfile --no-container 
stoat 2016-11-01 11:09:49,520 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
stoat 2016-11-01 11:09:49,521 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
stoat 2016-11-01 11:09:49,972 MainThread INFO toil.jobStores.fileJobStore: Path to job store directory is '/working/cwarth/cft/toil/toilWorkflow'.
stoat 2016-11-01 11:09:50,021 MainThread INFO toil.jobStores.abstractJobStore: The workflow ID is: '9e858ae4-6ec4-45c2-bc48-285d0db66efa'
stoat 2016-11-01 11:09:50,334 MainThread INFO toil.common: Using the single machine batch system
stoat 2016-11-01 11:09:50,334 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (16).
stoat 2016-11-01 11:09:50,334 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (67378266112).
stoat 2016-11-01 11:09:50,335 MainThread INFO toil.common: Created the workflow directory at /tmp/toil-9e858ae4-6ec4-45c2-bc48-285d0db66efa
stoat 2016-11-01 11:09:50,335 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxDisk to physically available disk (4560371712).
stoat 2016-11-01 11:09:50,336 MainThread INFO toil.batchSystems.singleMachine: Setting up the thread pool with 160 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 16.
stoat 2016-11-01 11:09:50,390 MainThread INFO toil.common: No user script to hot-deploy.
stoat 2016-11-01 11:09:50,421 MainThread INFO toil.common: Written the environment for the jobs to the environment file
stoat 2016-11-01 11:09:50,421 MainThread INFO toil.common: Caching all jobs in job store
stoat 2016-11-01 11:09:50,453 MainThread INFO toil.common: 0 jobs downloaded.
stoat 2016-11-01 11:09:50,810 MainThread INFO toil.realtimeLogger: Real-time logging disabled
stoat 2016-11-01 11:09:51,185 MainThread INFO toil.leader: (Re)building internal scheduler state
stoat 2016-11-01 11:09:51,186 MainThread INFO toil.leader: Checked batch system has no running jobs and no updated jobs
stoat 2016-11-01 11:09:51,187 MainThread INFO toil.leader: Found 1 jobs to start and 0 jobs with successors to run
stoat 2016-11-01 11:09:51,187 MainThread INFO toil.leader: Starting the main loop
stoat 2016-11-01 11:09:51,229 Thread-1 INFO toil.batchSystems.singleMachine: Executing command: '/working/cwarth/cft/toil/venv/bin/_toil_worker file:/working/cwarth/cft/toil/toilWorkflow q/x/jobJt2wuN'.
stoat 2016-11-01 11:09:59,293 MainThread INFO toil.leader: No jobs left to run so exiting.
stoat 2016-11-01 11:09:59,294 MainThread INFO toil.leader: Finished the main loop
stoat 2016-11-01 11:09:59,294 MainThread INFO toil.leader: Waiting for stats and logging collator thread to finish ...
stoat 2016-11-01 11:09:59,645 Thread-162 INFO toil.leader: Got message from job at time 11-01-2016 11:09:59: Job q/x/jobJt2wuN/g/tmpx6jOcL.tmp used 0.00% (0.0 B [0B] used, 3.0 GB [3221225472B] requested) at the end of its run.
stoat 2016-11-01 11:09:59,833 MainThread INFO toil.leader: ... finished collating stats and logs. Took 0.538393974304 seconds
stoat 2016-11-01 11:09:59,833 MainThread INFO toil.leader: Waiting for service manager thread to finish ...
stoat 2016-11-01 11:10:00,188 MainThread INFO toil.leader: ... finished shutting down the service manager. Took 0.355026006699 seconds
stoat 2016-11-01 11:10:00,188 MainThread INFO toil.leader: Finished toil run successfully
stoat 2016-11-01 11:10:00,276 MainThread INFO toil.common: Attempting to delete the job store
stoat 2016-11-01 11:10:00,936 MainThread INFO toil.common: Successfully deleted the job store
Hello, world!, here's a message: woot
```

<h2 id="cwltool_wo_slurm">`cwltool` can run CWL without Slurm</h2>

```
$ cwltool --no-container hello.cwl  hello.yml                                                                                                                                                                                                                                                                                                                         
/working/cwarth/cft/toil/venv/bin/cwltool 1.0.20160714182449
[job hello.cwl] /tmp/tmpaiOU7c$ echo \
    foo.bar > /tmp/tmpaiOU7c/results.txt
Final process status is success
{
    "output": {
        "checksum": "sha1$f9299c817b423a41aae4bbce4b17f5edcb9c9282", 
        "basename": "results.txt", 
        "location": "file:///working/cwarth/cft/toil/results.txt", 
        "path": "/working/cwarth/cft/toil/results.txt", 
        "class": "File", 
        "size": 8
    }
}
```

<h2 id="cwltoil_wo_slurm">`cwltoil` can run CWL without Slurm</h2>

```
$ cwltoil --no-container hello.cwl hello.yml                                                                                                                                                                                                                                                                                                                              
stoat 2016-11-01 11:06:36,753 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
stoat 2016-11-01 11:06:39,447 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
stoat 2016-11-01 11:06:39,945 MainThread INFO toil.jobStores.fileJobStore: Path to job store directory is '/tmp/tmpuMpTs9'.
stoat 2016-11-01 11:06:39,946 MainThread INFO toil.jobStores.abstractJobStore: The workflow ID is: '2dbb34d1-ce9f-4a64-99dd-1955980412f8'
stoat 2016-11-01 11:06:40,483 MainThread INFO toil.common: Using the single machine batch system
stoat 2016-11-01 11:06:40,483 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (16).
stoat 2016-11-01 11:06:40,483 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (67378266112).
stoat 2016-11-01 11:06:40,484 MainThread INFO toil.common: Created the workflow directory at /tmp/toil-2dbb34d1-ce9f-4a64-99dd-1955980412f8
stoat 2016-11-01 11:06:40,484 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxDisk to physically available disk (4560850944).
stoat 2016-11-01 11:06:40,484 MainThread INFO toil.batchSystems.singleMachine: Setting up the thread pool with 160 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 16.
stoat 2016-11-01 11:06:40,565 MainThread INFO toil.common: User script ModuleDescriptor(dirPath='/working/cwarth/cft/toil/venv/src/toil/src', name='toil.cwl.cwltoil', fromVirtualEnv=True) belongs to Toil. No need to hot-deploy it.
stoat 2016-11-01 11:06:40,565 MainThread INFO toil.common: No user script to hot-deploy.
stoat 2016-11-01 11:06:40,566 MainThread INFO toil.common: Written the environment for the jobs to the environment file
stoat 2016-11-01 11:06:40,566 MainThread INFO toil.common: Caching all jobs in job store
stoat 2016-11-01 11:06:40,566 MainThread INFO toil.common: 0 jobs downloaded.
stoat 2016-11-01 11:06:40,571 MainThread INFO toil.realtimeLogger: Real-time logging disabled
stoat 2016-11-01 11:06:41,021 MainThread INFO toil.leader: (Re)building internal scheduler state
stoat 2016-11-01 11:06:41,022 MainThread INFO toil.leader: Checked batch system has no running jobs and no updated jobs
stoat 2016-11-01 11:06:41,023 MainThread INFO toil.leader: Found 1 jobs to start and 0 jobs with successors to run
stoat 2016-11-01 11:06:41,024 MainThread INFO toil.leader: Starting the main loop
stoat 2016-11-01 11:06:41,075 Thread-1 INFO toil.batchSystems.singleMachine: Executing command: '/working/cwarth/cft/toil/venv/bin/_toil_worker file:/tmp/tmpuMpTs9 L/S/job0TZ6_5'.
stoat 2016-11-01 11:06:51,240 MainThread INFO toil.leader: No jobs left to run so exiting.
stoat 2016-11-01 11:06:51,240 MainThread INFO toil.leader: Finished the main loop
stoat 2016-11-01 11:06:51,241 MainThread INFO toil.leader: Waiting for stats and logging collator thread to finish ...
stoat 2016-11-01 11:06:51,541 Thread-162 INFO toil.leader: Got message from job at time 11-01-2016 11:06:51: Job L/S/job0TZ6_5/g/tmpUbWKa4.tmp used 0.00% (0.0 B [0B] used, 2.0 GB [2147483648B] requested) at the end of its run.
stoat 2016-11-01 11:06:51,541 Thread-162 INFO toil.leader: Got message from job at time 11-01-2016 11:06:51: Successfully deleted local copies of file with ID 'R/L/tmpSVgIYD.tmp'.
stoat 2016-11-01 11:06:51,542 MainThread INFO toil.leader: ... finished collating stats and logs. Took 0.301622867584 seconds
stoat 2016-11-01 11:06:51,543 MainThread INFO toil.leader: Waiting for service manager thread to finish ...
stoat 2016-11-01 11:06:52,026 MainThread INFO toil.leader: ... finished shutting down the service manager. Took 0.4829621315 seconds
stoat 2016-11-01 11:06:52,026 MainThread INFO toil.leader: Finished toil run successfully
stoat 2016-11-01 11:06:52,430 MainThread INFO toil.jobStores.abstractJobStore: Unable to import 'toil.jobStores.azureJobStore'. You may want to try reinstalling Toil with additional extras.
stoat 2016-11-01 11:06:52,755 MainThread INFO toil.jobStores.abstractJobStore: Unable to import 'toil.jobStores.googleJobStore'. You may want to try reinstalling Toil with additional extras.
stoat 2016-11-01 11:06:53,166 MainThread INFO toil.jobStores.abstractJobStore: Unable to import 'toil.jobStores.aws.jobStore'. You may want to try reinstalling Toil with additional extras.
{
    "output": {
        "checksum": "sha1$f9299c817b423a41aae4bbce4b17f5edcb9c9282", 
        "basename": "results.txt", 
        "location": "file:///tmp/tmpoMG_XW/results.txt", 
        "path": "/working/cwarth/cft/toil/results_2.txt", 
        "class": "File", 
        "size": 8
    }
stoat 2016-11-01 11:06:53,188 MainThread INFO toil.common: Attempting to delete the job store
stoat 2016-11-01 11:06:53,190 MainThread INFO toil.common: Successfully deleted the job store
}
```


Discussion:
FWIW, I patched `batchSystems/slurm.py` to remove the `file:` prefix from the resultsFileName, just like the `batchSystems/gridengine.py` does.   There might be another workaround because back in June @guillermo-carrasco was able to get CWL jobs working with slurm (#918) without this patch.  

```
$ git diff --no-prefix
diff --git src/toil/batchSystems/slurm.py src/toil/batchSystems/slurm.py
index c0ec8d9..cf0beeb 100644
--- src/toil/batchSystems/slurm.py
+++ src/toil/batchSystems/slurm.py
@@ -258,6 +258,17 @@ class SlurmBatchSystem(BatchSystemSupport):
     def supportsHotDeployment(cls):
         return False

+    def _getResultsFileName(self, toilPath):
+        """
+        Get a path for the batch systems to store results.
+        slurm batch support cannot currently handle filenames prefixed
+        with 'file:', so strip it off.
+        """
+        prefix = 'file:'
+        assert toilPath.startswith(prefix)
+        return os.path.join(toilPath[len(prefix):], "results.txt")
+
+
     def __init__(self, config, maxCores, maxMemory, maxDisk):
         super(SlurmBatchSystem, self).__init__(config, maxCores, maxMemory, maxDisk)
         self.slurmResultsFile = self._getResultsFileName(config.jobStore)
```

@JakeNarkizian fixed #1108 for GridEngine in 455603de9b59a6852948ae926b315fdd286f1e90.

Regressions like this are the reason I insist on removing the duplication between SGE, Slurm and TORQUE. See https://github.com/BD2KGenomics/toil/pull/1195.

@JakeNarkizian can you copy your fix to Slurm?

I think the right thing may be to override `_getResultsFileName()` as I did in `slurm.py` rather than fixup the results afterward as in `gridengine.py`.  There may be other consumers of _getResultsFileName()` (tomorrow, if not today) that also need the fixed result.

> override _getResultsFileName 

Agreed


Resolving patch:
diff --git a/src/toil/batchSystems/abstractBatchSystem.py b/src/toil/batchSystems/abstractBatchSystem.py
--- a/src/toil/batchSystems/abstractBatchSystem.py
+++ b/src/toil/batchSystems/abstractBatchSystem.py
@@ -286,10 +286,13 @@ def getRescueBatchJobFrequency(cls):
 
     def _getResultsFileName(self, toilPath):
         """
-        Get a path for the batch systems to store results. GridEngine
-        and LSF currently use this.
+        Get a path for the batch systems to store results. GridEngine, slurm,
+        and LSF currently use this and only work if locator is file.
         """
-        return os.path.join(toilPath, "results.txt")
+        # Use  parser to extract the path and type
+        locator, filePath = Toil.parseLocator(toilPath)
+        assert locator == "file"
+        return os.path.join(filePath, "results.txt")
 
     @staticmethod
     def workerCleanup(info):
diff --git a/src/toil/batchSystems/gridengine.py b/src/toil/batchSystems/gridengine.py
--- a/src/toil/batchSystems/gridengine.py
+++ b/src/toil/batchSystems/gridengine.py
@@ -222,9 +222,7 @@ def supportsHotDeployment(cls):
 
     def __init__(self, config, maxCores, maxMemory, maxDisk):
         super(GridengineBatchSystem, self).__init__(config, maxCores, maxMemory, maxDisk)
-        prefix = 'file:'
-        assert config.jobStore.startswith(prefix)
-        self.gridengineResultsFile = self._getResultsFileName(config.jobStore[len(prefix):])
+        self.gridengineResultsFile = self._getResultsFileName(config.jobStore)
         # Reset the job queue and results (initially, we do this again once we've killed the jobs)
         self.gridengineResultsFileHandle = open(self.gridengineResultsFile, 'w')
         # We lose any previous state in this file, and ensure the files existence


Test patch:
diff --git a/src/toil/test/batchSystems/batchSystemTest.py b/src/toil/test/batchSystems/batchSystemTest.py
--- a/src/toil/test/batchSystems/batchSystemTest.py
+++ b/src/toil/test/batchSystems/batchSystemTest.py
@@ -79,11 +79,12 @@ def createBatchSystem(self):
         def supportsWallTime(self):
             return False
 
-        @staticmethod
-        def _createDummyConfig():
+        @classmethod
+        def createConfig(cls):
             """
             Returns a dummy config for the batch system tests.  We need a workflowID to be set up
-            since we are running tests without setting up a jobstore.
+            since we are running tests without setting up a jobstore. This is the class version
+            to be used when an instance is not available.
 
             :rtype: toil.common.Config
             """
@@ -92,6 +93,15 @@ def _createDummyConfig():
             config.workflowID = str(uuid4())
             return config
 
+        def _createConfig(self):
+            """
+            Returns a dummy config for the batch system tests.  We need a workflowID to be set up
+            since we are running tests without setting up a jobstore.
+
+            :rtype: toil.common.Config
+            """
+            return self.createConfig()
+
         @classmethod
         def setUpClass(cls):
             super(hidden.AbstractBatchSystemTest, cls).setUpClass()
@@ -99,7 +109,7 @@ def setUpClass(cls):
 
         def setUp(self):
             super(hidden.AbstractBatchSystemTest, self).setUp()
-            self.config = self._createDummyConfig()
+            self.config = self._createConfig()
             self.batchSystem = self.createBatchSystem()
             self.tempDir = self._createTempDir('testFiles')
 
@@ -271,6 +281,29 @@ def testJobConcurrency(self):
                 _, maxValue = getCounters(counterPath)
                 self.assertEqual(maxValue, self.cpuCount / coresPerJob)
 
+    class AbstractGridEngineBatchSystemTest(AbstractBatchSystemTest):
+        """
+        An abstract class to reduce redundancy between Grid Engine, Slurm, and other similar batch
+        systems
+        """
+
+        def _createConfig(self):
+            config = super(hidden.AbstractGridEngineBatchSystemTest, self)._createConfig()
+            # can't use _getTestJobStorePath since that method removes the directory
+            config.jobStore = 'file:' + self._createTempDir('jobStore')
+            return config
+
+        def testResultFile(self):
+            """
+            Tests that the result file name is formatted properly
+            """
+            # noinspection PyUnresolvedReferences
+            fileName = self.batchSystem._getResultsFileName(self.config.jobStore)
+            filePath, _ = os.path.split(fileName)  # removes file so dir matches config.jobStore
+            locator = self.config.jobStore
+            self.assertTrue(locator.startswith('file:'))
+            self.assertEqual(locator[len('file:'):], filePath)
+
 
 @needs_mesos
 class MesosBatchSystemTest(hidden.AbstractBatchSystemTest, MesosTestSupport):
@@ -394,7 +427,7 @@ def test(self):
                     if jobs >= 1 and minCores <= coresPerJob < maxCores:
                         self.assertEquals(maxCores, float(maxCores))
                         bs = SingleMachineBatchSystem(
-                            config=hidden.AbstractBatchSystemTest._createDummyConfig(),
+                            config=hidden.AbstractBatchSystemTest.createConfig(),
                             maxCores=float(maxCores),
                             # Ensure that memory or disk requirements don't get in the way.
                             maxMemory=jobs * 10,
@@ -488,8 +521,8 @@ class ParasolBatchSystemTest(hidden.AbstractBatchSystemTest, ParasolTestSupport)
     def supportsWallTime(self):
         return True
 
-    def _createDummyConfig(self):
-        config = super(ParasolBatchSystemTest, self)._createDummyConfig()
+    def _createConfig(self):
+        config = super(ParasolBatchSystemTest, self)._createConfig()
         # can't use _getTestJobStorePath since that method removes the directory
         config.jobStore = self._createTempDir('jobStore')
         return config
@@ -555,17 +588,11 @@ def _getBatchList(self):
 
 
 @needs_gridengine
-class GridEngineBatchSystemTest(hidden.AbstractBatchSystemTest):
+class GridEngineBatchSystemTest(hidden.AbstractGridEngineBatchSystemTest):
     """
     Tests against the GridEngine batch system
     """
 
-    def _createDummyConfig(self):
-        config = super(GridEngineBatchSystemTest, self)._createDummyConfig()
-        # can't use _getTestJobStorePath since that method removes the directory
-        config.jobStore = 'file:'+ self._createTempDir('jobStore')
-        return config
-
     def createBatchSystem(self):
         from toil.batchSystems.gridengine import GridengineBatchSystem
         return GridengineBatchSystem(config=self.config, maxCores=numCores, maxMemory=1000e9,
@@ -579,17 +606,11 @@ def tearDown(self):
             os.unlink(f)
 
 @needs_slurm
-class SlurmBatchSystemTest(hidden.AbstractBatchSystemTest):
+class SlurmBatchSystemTest(hidden.AbstractGridEngineBatchSystemTest):
     """
     Tests against the Slurm batch system
     """
 
-    def _createDummyConfig(self):
-        config = super(SlurmBatchSystemTest, self)._createDummyConfig()
-        # can't use _getTestJobStorePath since that method removes the directory
-        config.jobStore = self._createTempDir('jobStore')
-        return config
-
     def createBatchSystem(self):
         from toil.batchSystems.slurm import SlurmBatchSystem
         return SlurmBatchSystem(config=self.config, maxCores=numCores, maxMemory=1000e9,