Repository: DataBiosphere/toil

Issue:
CWL toil workflow input file name collision
If two inputs to a cwl workflow step have the same name there will be a collision, for example:

```
2016-03-30 23:49:00,930 WARNING:toil.leader: f/z/jobmQtiTU: ---TOIL WORKER OUTPUT LOG---
2016-03-30 23:49:00,930 WARNING:toil.leader: f/z/jobmQtiTU: INFO:rdflib:RDFLib Version: 4.2.1
2016-03-30 23:49:00,930 WARNING:toil.leader: f/z/jobmQtiTU: INFO:toil.jobStores.fileJobStore:Jobstore directory is: /Users/ellrott/workspaces/mc3_cwl/test_workflow/test_store
2016-03-30 23:49:00,930 WARNING:toil.leader: f/z/jobmQtiTU: Traceback (most recent call last):
2016-03-30 23:49:00,930 WARNING:toil.leader: f/z/jobmQtiTU:   File "/Users/ellrott/workspaces/mc3_cwl/venv/lib/python2.7/site-packages/toil-3.2.0a2-py2.7.egg/toil/worker.py", line 283, in main
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:     fileStore=fileStore)
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:   File "/Users/ellrott/workspaces/mc3_cwl/venv/lib/python2.7/site-packages/toil-3.2.0a2-py2.7.egg/toil/job.py", line 1298, in _execute
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:     returnValues = self.run(fileStore)
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:   File "/Users/ellrott/workspaces/mc3_cwl/venv/lib/python2.7/site-packages/toil-3.2.0a2-py2.7.egg/toil/cwl/cwltoil.py", line 213, in run
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:     adjustFiles(cwljob, lambda x: getFile(fileStore, inpdir, *x, index=index))
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:   File "/Users/ellrott/workspaces/mc3_cwl/venv/lib/python2.7/site-packages/cwltool/process.py", line 139, in adjustFiles
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:     adjustFiles(rec[d], op)
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:   File "/Users/ellrott/workspaces/mc3_cwl/venv/lib/python2.7/site-packages/cwltool/process.py", line 137, in adjustFiles
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:     rec["path"] = op(rec["path"])
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:   File "/Users/ellrott/workspaces/mc3_cwl/venv/lib/python2.7/site-packages/toil-3.2.0a2-py2.7.egg/toil/cwl/cwltoil.py", line 213, in <lambda>
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:     adjustFiles(cwljob, lambda x: getFile(fileStore, inpdir, *x, index=index))
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:   File "/Users/ellrott/workspaces/mc3_cwl/venv/lib/python2.7/site-packages/toil-3.2.0a2-py2.7.egg/toil/cwl/cwltoil.py", line 123, in getFile
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU:     raise Exception("Conflicting filesStoreID %s and %s both trying to link to %s" % (index.get(dstPath, None), fileStoreID, dstPath))
2016-03-30 23:49:00,931 WARNING:toil.leader: f/z/jobmQtiTU: Exception: Conflicting filesStoreID n/1/tmpxw47b3.tmp and L/C/tmpCnIrNm.tmp both trying to link to /Users/ellrott/workspaces/mc3_cwl/test_workflow/tmpzJ7dNP/localTempDir/tn7MTGE/inp/out.txt
```

Example Workflow (works in cwltool, fails in cwltoil):

```
- class: CommandLineTool
  id: "#echo"
  inputs:
    - id: text
      type: string
      inputBinding: {}

  outputs:
    - id: "fileout"
      type: File
      outputBinding:
        glob: out.txt

  baseCommand: echo
  stdout: out.txt

- class: CommandLineTool
  id: "#cat"
  inputs:
    - id: file1
      type: File
      inputBinding:
        position: 1
    - id: file2
      type: File
      inputBinding:
        position: 2

  outputs:
    - id: "fileout"
      type: File
      outputBinding:
        glob: out.txt

  baseCommand: cat
  stdout: out.txt

- class: Workflow
  id: "#collision"

  inputs:
    - id: "#input_1"
      type: string

    - id: "#input_2"
      type: string

  outputs:
    - id: "#fileout"
      type: File
      source: "#cat_step.fileout"

  steps:
    - id: "#echo_1"
      run: "#echo"
      inputs:
        - id: "#echo_1.text"
          source: "#input_1"
      outputs:
        - id: "#echo_1.fileout"

    - id: "#echo_2"
      run: 
        import: "#echo"
      inputs:
        - id: "#collision.echo_2.text"
          source: "#input_2"
      outputs:
        - id: "#echo_2.fileout"

    - id: "#cat_step"
      run: 
        import: "#cat"
      inputs:
        - id: file1
          source: "#echo_1.fileout"
        - id: file2
          source: "#echo_2.fileout"
      outputs:
        - id: "#cat_step.fileout"
```

Sample input:

```
{
  "input_1" : "Hello World",
  "input_2" : "How are you?"
}
```



Discussion:
Hi @kellrott, thanks I'll take a look at it.  It should probably stage each input file to separate directory to avoid name conflicts like that.


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -39,7 +39,7 @@
         'encryption': [
             'pynacl==0.3.0'],
         'cwl': [
-            'cwltool==1.0.20160226205952']},
+            'cwltool==1.0.20160408133555']},
     package_dir={'': 'src'},
     packages=find_packages('src', exclude=['*.test']),
     entry_points={
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -1,6 +1,7 @@
 # Implement support for Common Workflow Language (CWL) for Toil.
 #
 # Copyright (C) 2015 Curoverse, Inc
+# Copyright (C) 2016 UCSC Computational Genomics Lab
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -21,7 +22,7 @@
 import cwltool.main
 import cwltool.workflow
 import cwltool.expression
-from cwltool.process import adjustFiles, shortname
+from cwltool.process import adjustFiles, shortname, adjustFilesWithSecondary
 from cwltool.aslist import aslist
 import schema_salad.validate as validate
 import schema_salad.ref_resolver
@@ -111,10 +112,25 @@ def resolve_indirect(d):
     else:
         return res
 
-def getFile(fileStore, dir, fileStoreID, fileName, index=None, copy=False):
+def getFile(fileStore, dir, fileTuple, index=None, copy=False, primary=None, rename_collision=False):
+    fileStoreID, fileName = fileTuple
+
+    if rename_collision is False:
+        if primary:
+            dir = os.path.dirname(primary)
+        else:
+            dir = tempfile.mkdtemp(dir=dir)
+
     srcPath = fileStore.readGlobalFile(fileStoreID)
     dstPath = os.path.join(dir, fileName)
     if srcPath != dstPath:
+        if rename_collision:
+            n = 1
+            while os.path.exists(dstPath):
+                n += 1
+                stem, ext = os.path.splitext(dstPath)
+                stem = "%s_%s" % (stem, n)
+                dstPath = stem + ext
         if copy:
             shutil.copyfile(srcPath, dstPath)
         else:
@@ -128,7 +144,8 @@ def getFile(fileStore, dir, fileStoreID, fileName, index=None, copy=False):
 
 def writeFile(fileStore, index, x):
     if x not in index:
-        index[x] = (fileStore.writeGlobalFile(x), x.split('/')[-1])
+        rp = os.path.realpath(x)
+        index[x] = (fileStore.writeGlobalFile(rp), os.path.basename(x))
     return index[x]
 
 class StageJob(Job):
@@ -174,7 +191,8 @@ def run(self, fileStore):
         cwljob = resolve_indirect(self.cwljob)
 
         index={}
-        adjustFiles(cwljob, lambda x: getFile(fileStore, self.outdir, *x, index=index, copy=True))
+        adjustFilesWithSecondary(cwljob, functools.partial(getFile, fileStore, self.outdir, index=index, copy=True, rename_collision=True))
+
         with open(os.path.join(self.outdir, "cwl.output.json"), "w") as f:
             json.dump(cwljob, f, indent=4)
         return True
@@ -210,7 +228,7 @@ def run(self, fileStore):
         # Copy input files out of the global file store.
 
         index={}
-        adjustFiles(cwljob, lambda x: getFile(fileStore, inpdir, *x, index=index))
+        adjustFilesWithSecondary(cwljob, functools.partial(getFile, fileStore, inpdir, index=index))
 
         logging.getLogger("cwltool").setLevel(logging.DEBUG)
 
@@ -294,6 +312,10 @@ def run(self, fileStore):
 
         self.vfinputs = cwljob
 
+        shortscatter = [shortname(s) for s in scatter]
+        cwljob = {k: self.valueFromFunc(k, v) if k not in shortscatter else v
+                    for k,v in cwljob.items()}
+
         if scatterMethod == "dotproduct":
             for i in xrange(0, len(cwljob[shortname(scatter[0])])):
                 copyjob = copy.copy(cwljob)


Test patch:
diff --git a/src/toil/test/cwl/conftest.py b/src/toil/test/cwl/conftest.py
new file mode 100644
--- /dev/null
+++ b/src/toil/test/cwl/conftest.py
@@ -0,0 +1,17 @@
+# Copyright (C) 2015 UCSC Computational Genomics Lab
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# https://pytest.org/latest/example/pythoncollection.html
+
+collect_ignore = ["spec"]
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -14,26 +14,33 @@
 from __future__ import absolute_import
 import json
 import os
+import subprocess
 
 from toil.test import ToilTest, needs_cwl
 
 @needs_cwl
 class CWLTest(ToilTest):
-    def test_run_revsort(self):
+
+    def _tester(self, cwlfile, jobfile, outDir, expect):
         from toil.cwl import cwltoil
-        outDir = self._createTempDir()
         rootDir = self._projectRootPath()
         outputJson = os.path.join(outDir, 'cwl.output.json')
         try:
             cwltoil.main(['--outdir', outDir,
-                          os.path.join(rootDir, 'src/toil/test/cwl/revsort.cwl'),
-                          os.path.join(rootDir, 'src/toil/test/cwl/revsort-job.json')])
+                          os.path.join(rootDir, cwlfile),
+                          os.path.join(rootDir, jobfile)])
             with open(outputJson) as f:
                 out = json.load(f)
         finally:
             if os.path.exists(outputJson):
                 os.remove(outputJson)
-        self.assertEquals(out, {
+        self.assertEquals(out, expect)
+
+    def test_run_revsort(self):
+        outDir = self._createTempDir()
+        self._tester('src/toil/test/cwl/revsort.cwl',
+                     'src/toil/test/cwl/revsort-job.json',
+                     outDir, {
             # Having unicode string literals isn't necessary for the assertion but makes for a
             # less noisy diff in case the assertion fails.
             u'output': {
@@ -41,3 +48,15 @@ def test_run_revsort(self):
                 u'size': 1111,
                 u'class': u'File',
                 u'checksum': u'sha1$b9214658cc453331b62c2282b772a5c063dbd284'}})
+
+
+    def test_run_conformance(self):
+        rootDir = self._projectRootPath()
+        cwlSpec = os.path.join(rootDir, 'src/toil/test/cwl/spec')
+        if os.path.exists(cwlSpec):
+            subprocess.call(["git", "fetch"], cwd=cwlSpec)
+        else:
+            subprocess.check_call(["git", "clone", "https://github.com/common-workflow-language/common-workflow-language.git", cwlSpec])
+        subprocess.check_call(["git", "checkout", "4daac6db94ccf626c4509fe0c3e645ceda23f50a"], cwd=cwlSpec)
+        subprocess.check_call(["git", "clean", "-f", "-x", "."], cwd=cwlSpec)
+        subprocess.check_call(["./run_test.sh", "RUNNER=cwltoil", "DRAFT=draft-3"], cwd=cwlSpec)
diff --git a/src/toil/test/cwl/revsort.cwl b/src/toil/test/cwl/revsort.cwl
--- a/src/toil/test/cwl/revsort.cwl
+++ b/src/toil/test/cwl/revsort.cwl
@@ -3,7 +3,7 @@
 #
 class: Workflow
 description: "Reverse the lines in a document, then sort those lines."
-cwlVersion: "cwl:draft-3.dev1"
+cwlVersion: "cwl:draft-3"
 
 # Requirements specify prerequisites and extensions to the workflow.
 # In this example, DockerRequirement specifies a default Docker container
@@ -56,7 +56,7 @@ steps:
       - { id: "#rev.input", source: "#input" }
     outputs:
       - { id: "#rev.output" }
-    run: { "@import": revtool.cwl }
+    run: { "$import": revtool.cwl }
 
   - id: "#sorted"
     inputs:
@@ -64,4 +64,4 @@ steps:
       - { id: "#sorted.reverse", source: "#reverse_sort" }
     outputs:
       - { id: "#sorted.output" }
-    run: { "@import": sorttool.cwl }
+    run: { "$import": sorttool.cwl }