Repository: DataBiosphere/toil

Issue:
Document has DockerRequirement under 'requirements' but use_container is false.
Using the most recent git [1] and a local cwltool [2], I get the following error when running a simple cwl [3]

Error:

```
(p2) [jeremiah@wireless-s1-so-150-52-240 ~]$ cwltoil ~/code/cocleaning-cwl/tools/md5sum.cwl ./ex.yml 
wireless-s1-so-150-52-240.uchicago.edu 2016-10-07 17:16:28,867 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
wireless-s1-so-150-52-240.uchicago.edu 2016-10-07 17:16:29,195 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
wireless-s1-so-150-52-240.uchicago.edu 2016-10-07 17:16:29,196 MainThread INFO toil.jobStores.fileJobStore: Path to job store directory is '/tmp/tmpNJ72E9'.
wireless-s1-so-150-52-240.uchicago.edu 2016-10-07 17:16:29,196 MainThread INFO toil.jobStores.abstractJobStore: The workflow ID is: '76b2b9e5-c80a-42da-a3a3-706e3905fff3'
Traceback (most recent call last):
  File "/home/jeremiah/.virtualenvs/p2/bin/cwltoil", line 11, in <module>
    load_entry_point('toil', 'console_scripts', 'cwltoil')()
  File "/home/jeremiah/code/toil/src/toil/cwl/cwltoil.py", line 653, in main
    builder = t._init_job(job, basedir=basedir)
  File "/home/jeremiah/.virtualenvs/p2/lib/python2.7/site-packages/cwltool/process.py", line 451, in _init_job
    raise WorkflowException("Document has DockerRequirement under 'requirements' but use_container is false.  DockerRequirement must be under 'hints' or use_container must be true.")
cwltool.errors.WorkflowException: Document has DockerRequirement under 'requirements' but use_container is false.  DockerRequirement must be under 'hints' or use_container must be true.

```

[1]
https://github.com/BD2KGenomics/toil/commit/196ba62d468963d57dc36be07e382c247551af62

[2]
https://github.com/jeremiahsavage/cwltool/releases/tag/1.0_gdc_e

[3]
md5sum.cwl:

```
#!/usr/bin/env cwl-runner

cwlVersion: v1.0

requirements:
  - class: DockerRequirement
    dockerPull: ubuntu:xenial-20160809
  - class: InlineJavascriptRequirement

class: CommandLineTool

inputs:
  - id: INPUT
    type: File
    inputBinding:
      position: 0

outputs:
  - id: OUTPUT
    type: File
    outputBinding:
      glob: $(inputs.INPUT.nameroot + ".md5")

stdout: $(inputs.INPUT.nameroot + ".md5")

baseCommand: [md5sum]
```

ex.yml:

```
INPUT:
  class: File
  location: /home/jeremiah/XX_UUID_XX.db
```


Discussion:
I don't think we can expect Toil's CWL runner to work with any version of `cwltool`. In fact, Toil's `setup.py` explicitly pins the `cwltool` requirement to 1.0.20160714182449.

What's the reasoning behind upgrading `cwltool` to `1.0_gdc_e`?

Sorry to confuse the issue. I see this problem even with the pinned version of `cwltool`

```
(toil)ubuntu@frontend001:~$ cwltool 
/home/ubuntu/.virtualenvs/toil/bin/cwltool 1.0.20160714182449

CWL document required, try --help for details
(toil)ubuntu@frontend001:~$ cwltoil ~/cocleaning-cwl/tools/md5sum.cwl ex.yml 
frontend001 2016-10-07 23:43:49,327 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
frontend001 2016-10-07 23:43:50,026 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
frontend001 2016-10-07 23:43:50,028 MainThread INFO toil.jobStores.fileJobStore: Path to job store directory is '/tmp/tmpRM2Mdv'.
frontend001 2016-10-07 23:43:50,028 MainThread INFO toil.jobStores.abstractJobStore: The workflow ID is: '965e75e7-8d1b-405a-ac4e-2395b1d75410'
Traceback (most recent call last):
  File "/home/ubuntu/.virtualenvs/toil/bin/cwltoil", line 11, in <module>
    load_entry_point('toil', 'console_scripts', 'cwltoil')()
  File "/home/ubuntu/toil/src/toil/cwl/cwltoil.py", line 653, in main
    builder = t._init_job(job, basedir=basedir) #, use_container=use_container)
  File "/home/ubuntu/.virtualenvs/toil/local/lib/python2.7/site-packages/cwltool/process.py", line 420, in _init_job
    raise WorkflowException("Document has DockerRequirement under 'requirements' but use_container is false.  DockerRequirement must be under 'hints' or use_container must be true.")
cwltool.errors.WorkflowException: Document has DockerRequirement under 'requirements' but use_container is false.  DockerRequirement must be under 'hints' or use_container must be true.
```

I am separately attempting to see if toil can run with other `cwltool` due to issues such as https://github.com/common-workflow-language/cwltool/pull/170 which are not yet fixed in upstream, but are patched in a local branch.

With the patch https://github.com/BD2KGenomics/toil/pull/1201 :

```
(toil)ubuntu@frontend001:~$ cwltool 
/home/ubuntu/.virtualenvs/toil/bin/cwltool 1.0.20160714182449

CWL document required, try --help for details
(toil)ubuntu@frontend001:~$ cwltoil ~/cocleaning-cwl/tools/md5sum.cwl ex.yml 
frontend001 2016-10-07 23:51:50,914 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
frontend001 2016-10-07 23:51:51,609 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
frontend001 2016-10-07 23:51:51,611 MainThread INFO toil.jobStores.fileJobStore: Path to job store directory is '/tmp/tmpaDQIig'.
frontend001 2016-10-07 23:51:51,611 MainThread INFO toil.jobStores.abstractJobStore: The workflow ID is: '8b80a9bf-be75-4a8b-b03a-095cf71cc3eb'
frontend001 2016-10-07 23:51:51,617 MainThread INFO toil.jobStores.abstractJobStore: Unable to import 'toil.jobStores.azureJobStore'. You may want to try reinstalling Toil with additional extras.
frontend001 2016-10-07 23:51:51,619 MainThread INFO toil.jobStores.abstractJobStore: Unable to import 'toil.jobStores.googleJobStore'. You may want to try reinstalling Toil with additional extras.
frontend001 2016-10-07 23:51:51,621 MainThread INFO toil.jobStores.abstractJobStore: Unable to import 'toil.jobStores.aws.jobStore'. You may want to try reinstalling Toil with additional extras.
frontend001 2016-10-07 23:51:51,640 MainThread INFO toil.common: Using the single machine batch system
frontend001 2016-10-07 23:51:51,640 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (8).
frontend001 2016-10-07 23:51:51,640 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (50651033600).
frontend001 2016-10-07 23:51:51,640 MainThread INFO toil.common: Created the workflow directory at /tmp/toil-8b80a9bf-be75-4a8b-b03a-095cf71cc3eb
frontend001 2016-10-07 23:51:51,640 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxDisk to physically available disk (3327827968).
frontend001 2016-10-07 23:51:51,641 MainThread INFO toil.batchSystems.singleMachine: Setting up the thread pool with 80 workers, given a minimum CPU fraction of 0.100000 and a maximum CPU value of 8.
frontend001 2016-10-07 23:51:51,661 MainThread INFO toil.common: Written the environment for the jobs to the environment file
frontend001 2016-10-07 23:51:51,661 MainThread INFO toil.common: Caching all jobs in job store
frontend001 2016-10-07 23:51:51,661 MainThread INFO toil.common: 0 jobs downloaded.
frontend001 2016-10-07 23:51:51,664 MainThread INFO toil.realtimeLogger: Real-time logging disabled
frontend001 2016-10-07 23:51:51,667 MainThread INFO toil.leader: (Re)building internal scheduler state
frontend001 2016-10-07 23:51:51,667 MainThread INFO toil.leader: Checked batch system has no running jobs and no updated jobs
frontend001 2016-10-07 23:51:51,667 MainThread INFO toil.leader: Found 1 jobs to start and 0 jobs with successors to run
frontend001 2016-10-07 23:51:51,668 MainThread INFO toil.leader: Starting the main loop
frontend001 2016-10-07 23:51:51,668 Thread-1 INFO toil.batchSystems.singleMachine: Executing command: '/home/ubuntu/.virtualenvs/toil/bin/_toil_worker file:/tmp/tmpaDQIig T/d/jobh2g2hn'.
frontend001 2016-10-07 23:52:02,939 MainThread INFO toil.leader: No jobs left to run so exiting.
frontend001 2016-10-07 23:52:02,940 MainThread INFO toil.leader: Finished the main loop
frontend001 2016-10-07 23:52:02,940 MainThread INFO toil.leader: Waiting for stats and logging collator thread to finish ...
frontend001 2016-10-07 23:52:03,190 Thread-82 INFO toil.leader: Got message from job at time 10-07-2016 23:52:03: Successfully deleted local copies of file with ID 'R/Z/tmpL_83YW.tmp'.
frontend001 2016-10-07 23:52:03,190 Thread-82 INFO toil.leader: Got message from job at time 10-07-2016 23:52:03: Successfully deleted local copies of file with ID 'A/W/tmpzM9mU1.tmp'.
frontend001 2016-10-07 23:52:03,191 MainThread INFO toil.leader: ... finished collating stats and logs. Took 0.25093793869 seconds
frontend001 2016-10-07 23:52:03,192 MainThread INFO toil.leader: Waiting for service manager thread to finish ...
frontend001 2016-10-07 23:52:03,671 MainThread INFO toil.leader: ... finished shutting down the service manager. Took 0.479404211044 seconds
frontend001 2016-10-07 23:52:03,672 MainThread INFO toil.leader: Finished toil run successfully
{
    "OUTPUT": {
        "checksum": "sha1$31be2d1a5a62e193e2404510996f896a0309196b", 
        "basename": "picard-tools-2.5.0.md5", 
        "location": "file:///tmp/tmpwdXJo9/picard-tools-2.5.0.md5", 
        "path": "/home/ubuntu/picard-tools-2.5.0_2_3_4.md5", 
        "class": "File", 
        "size": 110
    }
frontend001 2016-10-07 23:52:03,683 MainThread INFO toil.common: Attempting to delete the job store
frontend001 2016-10-07 23:52:03,686 MainThread INFO toil.common: Successfully deleted the job store
```


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -47,7 +47,7 @@ def runSetup():
                 'gcs_oauth2_boto_plugin==1.9',
                 botoRequirement],
             'cwl': [
-                'cwltool==1.0.20160714182449']},
+                'cwltool==1.0.20161221171240']},
         package_dir={'': 'src'},
         packages=find_packages(where='src',
                                # Note that we intentionally include the top-level `test` package for
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -26,8 +26,10 @@
 import cwltool.workflow
 import cwltool.expression
 import cwltool.builder
+import cwltool.resolver
+import cwltool.stdfsaccess
 from cwltool.pathmapper import adjustFiles
-from cwltool.process import shortname, adjustFilesWithSecondary, fillInDefaults
+from cwltool.process import shortname, adjustFilesWithSecondary, fillInDefaults, compute_checksums
 from cwltool.utils import aslist
 import schema_salad.validate as validate
 import schema_salad.ref_resolver
@@ -121,6 +123,9 @@ def resolve_indirect(d):
         return res
 
 def getFile(fileStore, dir, fileTuple, index=None, export=False, primary=None, rename_collision=False):
+    # File literal outputs with no path, from writeFile
+    if fileTuple is None:
+        raise cwltool.process.UnsupportedRequirement("CWL expression file inputs not yet supported in Toil")
     fileStoreID, fileName = fileTuple
 
     if rename_collision is False:
@@ -158,6 +163,10 @@ def writeFile(writeFunc, index, x):
     # Toil fileStore references are tuples of pickle and internal file
     if isinstance(x, tuple):
         return x
+    # File literal outputs with no path, we don't write these and will fail
+    # with unsupportedRequirement when retrieving later with getFile
+    elif x.startswith("_:"):
+        return None
     else:
         if x not in index:
             if not urlparse.urlparse(x).scheme:
@@ -171,6 +180,13 @@ def writeFile(writeFunc, index, x):
                 raise
         return index[x]
 
+def computeFileChecksums(fs_access, f):
+    # File literal inputs with no path, no checksum
+    if isinstance(f, dict) and f.get("location", "").startswith("_:"):
+        return f
+    else:
+        return compute_checksums(fs_access, f)
+
 def addFilePartRefs(p):
     """Provides new v1.0 functionality for referencing file parts.
     """
@@ -221,7 +237,7 @@ def __init__(self, tool, cwljob, **kwargs):
         builder.timeout = 0
         builder.resources = {}
         req = tool.evalResources(builder, {})
-        self.cwltool = tool
+        self.cwltool = remove_pickle_problems(tool)
         # pass the default of None if basecommand is empty
         unitName = self.cwltool.tool.get("baseCommand", None)
         if isinstance(unitName, (list, tuple)):
@@ -251,7 +267,7 @@ def run(self, fileStore):
         os.mkdir(tmpdir)
 
         # Copy input files out of the global file store.
-        index={}
+        index = {}
         adjustFilesWithSecondary(cwljob, functools.partial(getFile, fileStore, inpdir, index=index))
 
         # Run the tool
@@ -266,6 +282,8 @@ def run(self, fileStore):
                                                   **opts)
         cwltool.builder.adjustDirObjs(output, locToPath)
         cwltool.builder.adjustFileObjs(output, locToPath)
+        cwltool.builder.adjustFileObjs(output, functools.partial(computeFileChecksums,
+                                                                 cwltool.stdfsaccess.StdFsAccess(outdir)))
         # Copy output files into the global file store.
         adjustFiles(output, functools.partial(writeFile, fileStore.writeGlobalFile, {}))
 
@@ -419,6 +437,16 @@ def addChild(self, c):
     def hasChild(self, c):
         return self.j.hasChild(c)
 
+def remove_pickle_problems(obj):
+    """doc_loader does not pickle correctly, causing Toil errors, remove from objects.
+    """
+    if hasattr(obj, "doc_loader"):
+        obj.doc_loader = None
+    if hasattr(obj, "embedded_tool"):
+        obj.embedded_tool = remove_pickle_problems(obj.embedded_tool)
+    if hasattr(obj, "steps"):
+        obj.steps = [remove_pickle_problems(s) for s in obj.steps]
+    return obj
 
 class CWLWorkflow(Job):
     """Traverse a CWL workflow graph and schedule a Toil job graph."""
@@ -428,6 +456,7 @@ def __init__(self, cwlwf, cwljob, **kwargs):
         self.cwlwf = cwlwf
         self.cwljob = cwljob
         self.executor_options = kwargs
+        self.cwlwf = remove_pickle_problems(self.cwlwf)
 
     def run(self, fileStore):
         cwljob = resolve_indirect(self.cwljob)
@@ -553,6 +582,25 @@ def run(self, fileStore):
                                                 "StepInputExpressionRequirement",
                                                 "ResourceRequirement")
 
+def unsupportedInputCheck(p):
+    """Check for file inputs we don't current support in Toil:
+
+    - Directories
+    - File literals
+    """
+    if p.get("class") == "Directory":
+        raise cwltool.process.UnsupportedRequirement("CWL Directory inputs not yet supported in Toil")
+    if p.get("contents") and (not p.get("path") and not p.get("location")):
+        raise cwltool.process.UnsupportedRequirement("CWL File literals not yet supported in Toil")
+
+def unsupportedDefaultCheck(tool):
+    """Check for file-based defaults, which don't get staged correctly in Toil.
+    """
+    for inp in tool["in"]:
+        if isinstance(inp, dict) and "default" in inp:
+            if isinstance(inp["default"], dict) and inp["default"].get("class") == "File":
+                raise cwltool.process.UnsupportedRequirement("CWL default file inputs not yet supported in Toil")
+
 def main(args=None, stdout=sys.stdout):
     parser = ArgumentParser()
     Job.Runner.addToilOptions(parser)
@@ -593,7 +641,8 @@ def main(args=None, stdout=sys.stdout):
         cwllogger.setLevel(options.logLevel)
 
     try:
-        t = cwltool.load_tool.load_tool(options.cwltool, cwltool.workflow.defaultMakeTool)
+        t = cwltool.load_tool.load_tool(options.cwltool, cwltool.workflow.defaultMakeTool,
+                                        resolver=cwltool.resolver.tool_resolver)
     except cwltool.process.UnsupportedRequirement as e:
         logging.error(e)
         return 33
@@ -612,19 +661,9 @@ def main(args=None, stdout=sys.stdout):
     else:
         job = {}
 
-    def unsupportedCheck(p):
-        """Check for file inputs we don't current support in Toil:
-
-        - Directories
-        - File literals
-        """
-        if p.get("class") == "Directory":
-            raise cwltool.process.UnsupportedRequirement("CWL Directory inputs not yet supported in Toil")
-        if p.get("contents") and (not p.get("path") and not p.get("location")):
-            raise cwltool.process.UnsupportedRequirement("CWL File literals not yet supported in Toil")
     try:
-        cwltool.builder.adjustDirObjs(job, unsupportedCheck)
-        cwltool.builder.adjustFileObjs(job, unsupportedCheck)
+        cwltool.builder.adjustDirObjs(job, unsupportedInputCheck)
+        cwltool.builder.adjustFileObjs(job, unsupportedInputCheck)
     except cwltool.process.UnsupportedRequirement as e:
         logging.error(e)
         return 33
@@ -663,8 +702,15 @@ def importDefault(tool):
             outobj = toil.restart()
         else:
             basedir = os.path.dirname(os.path.abspath(options.cwljob or options.cwltool))
-            builder = t._init_job(job, basedir=basedir)
+            builder = t._init_job(job, basedir=basedir, use_container=use_container)
             (wf1, wf2) = makeJob(t, {}, use_container=use_container, preserve_environment=options.preserve_environment, tmpdir=os.path.realpath(outdir))
+            try:
+                if isinstance(wf1, CWLWorkflow):
+                    [unsupportedDefaultCheck(s.tool) for s in wf1.cwlwf.steps]
+            except cwltool.process.UnsupportedRequirement as e:
+                logging.error(e)
+                return 33
+
             cwltool.builder.adjustDirObjs(builder.job, locToPath)
             cwltool.builder.adjustFileObjs(builder.job, locToPath)
             adjustFiles(builder.job, lambda x: "file://%s" % os.path.abspath(os.path.join(basedir, x))
@@ -678,7 +724,12 @@ def importDefault(tool):
 
         outobj = resolve_indirect(outobj)
 
-        adjustFilesWithSecondary(outobj, functools.partial(getFile, toil, outdir, index={}, export=True, rename_collision=True))
+        try:
+            adjustFilesWithSecondary(outobj, functools.partial(getFile, toil, outdir, index={},
+                                                               export=True, rename_collision=True))
+        except cwltool.process.UnsupportedRequirement as e:
+            logging.error(e)
+            return 33
 
         stdout.write(json.dumps(outobj, indent=4))
 


Test patch:
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -88,7 +88,7 @@ def test_run_conformance(self):
             subprocess.call(["git", "fetch"], cwd=cwlSpec)
         else:
             subprocess.check_call(["git", "clone", "https://github.com/common-workflow-language/common-workflow-language.git", cwlSpec])
-        subprocess.check_call(["git", "checkout", "1d5714dc434ffd6ac45ec64f1535475fa163be09"], cwd=cwlSpec)
+        subprocess.check_call(["git", "checkout", "87d982f7793fe08d4f4af5555d551c272eaa809c"], cwd=cwlSpec)
         subprocess.check_call(["git", "clean", "-f", "-x", "."], cwd=cwlSpec)
         try:
             subprocess.check_output(["./run_test.sh", "RUNNER=cwltoil", "DRAFT=v1.0"], cwd=cwlSpec,