Repository: DataBiosphere/toil

Issue:
cwltoil fails when providing default and input value to nested workflows
`cwltoil` fails when an input is passed to an input in a nested workflow that also have a default value specified.

Environment:
```
$ uname -a
Linux docker-vm 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ pip3 freeze | grep toil
toil==3.24.0
```

Scripts:
```
$ head -n-0 wf.cwl sub.cwl tool.cwl
==> wf.cwl <==
cwlVersion: v1.0
class: Workflow
requirements:
  SubworkflowFeatureRequirement: {}
inputs:
  inp: string?
steps:
  tool:
    run: sub.cwl
    in:
      inp: inp
    out: []
outputs: []

==> sub.cwl <==
cwlVersion: v1.0
class: Workflow
requirements:
  SubworkflowFeatureRequirement: {}
inputs:
  inp: string?
steps:
  tool:
    run: tool.cwl
    in:
      inp: inp
    out: []
outputs: []

==> tool.cwl <==
cwlVersion: v1.0
class: CommandLineTool
baseCommand: echo
inputs:
  inp:
    type: string
    default: "Hello"
outputs: []
```

Running `cwltoil wf.cwl` works, but not `cwltoil wf.cwl --inp Hello`.

```
$ cwltoil --logWarning wf.cwl --inp Hello
docker-vm 2020-02-26 00:51:15,461 MainThread WARNING cwltool: Workflow checker warning:
sub.cwl:6:3:  Source 'inp' of type ["null", "string"] may be incompatible
sub.cwl:11:7:   with sink 'inp' of type "string"
docker-vm 2020-02-26 00:51:15,482 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (4).
docker-vm 2020-02-26 00:51:15,484 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (16819580928).
docker-vm 2020-02-26 00:51:15,485 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxDisk to physically available disk (19015532544).
docker-vm 2020-02-26 00:51:16,917 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'CWLWorkflow' kind-CWLWorkflow/instancedr0axrjw
docker-vm 2020-02-26 00:51:16,917 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/ubuntu/.local/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=False).
docker-vm 2020-02-26 00:51:16,918 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/ubuntu/.local/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=False).
docker-vm 2020-02-26 00:51:16,918 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/ubuntu/.local/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=False).
docker-vm 2020-02-26 00:51:16,918 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    WARNING:toil.resource:Can't globalize module ModuleDescriptor(dirPath='/home/ubuntu/.local/lib/python3.6/site-packages', name='toil.cwl.cwltoil', fromVirtualEnv=False).
docker-vm 2020-02-26 00:51:16,918 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    Traceback (most recent call last):
docker-vm 2020-02-26 00:51:16,918 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw      File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/worker.py", line 366, in workerScript
docker-vm 2020-02-26 00:51:16,919 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
docker-vm 2020-02-26 00:51:16,919 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw      File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/job.py", line 1392, in _runner
docker-vm 2020-02-26 00:51:16,919 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw        returnValues = self._run(jobGraph, fileStore)
docker-vm 2020-02-26 00:51:16,919 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw      File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/job.py", line 1329, in _run
docker-vm 2020-02-26 00:51:16,920 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw        return self.run(fileStore)
docker-vm 2020-02-26 00:51:16,920 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw      File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 953, in run
docker-vm 2020-02-26 00:51:16,920 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw        jobobj[key][0]] is None):
docker-vm 2020-02-26 00:51:16,920 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    TypeError: tuple indices must be integers or slices, not str
docker-vm 2020-02-26 00:51:16,920 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    ERROR:toil.worker:Exiting the worker because of a failed job on host docker-vm 
docker-vm 2020-02-26 00:51:16,921 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CWLWorkflow' kind-CWLWorkflow/instancedr0axrjw with ID kind-CWLWorkflow/instancedr0axrjw to 1
docker-vm 2020-02-26 00:51:18,215 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'CWLWorkflow' kind-CWLWorkflow/instancedr0axrjw
docker-vm 2020-02-26 00:51:18,216 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    Traceback (most recent call last):
docker-vm 2020-02-26 00:51:18,216 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw      File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/worker.py", line 366, in workerScript
docker-vm 2020-02-26 00:51:18,216 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
docker-vm 2020-02-26 00:51:18,216 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw      File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/job.py", line 1392, in _runner
docker-vm 2020-02-26 00:51:18,217 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw        returnValues = self._run(jobGraph, fileStore)
docker-vm 2020-02-26 00:51:18,217 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw      File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/job.py", line 1329, in _run
docker-vm 2020-02-26 00:51:18,217 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw        return self.run(fileStore)
docker-vm 2020-02-26 00:51:18,217 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw      File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 953, in run
docker-vm 2020-02-26 00:51:18,217 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw        jobobj[key][0]] is None):
docker-vm 2020-02-26 00:51:18,217 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    TypeError: tuple indices must be integers or slices, not str
docker-vm 2020-02-26 00:51:18,217 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    ERROR:toil.worker:Exiting the worker because of a failed job on host docker-vm
docker-vm 2020-02-26 00:51:18,218 MainThread WARNING toil.leader: kind-CWLWorkflow/instancedr0axrjw    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'CWLWorkflow' kind-CWLWorkflow/instancedr0axrjw with ID kind-CWLWorkflow/instancedr0axrjw to 0
docker-vm 2020-02-26 00:51:18,219 MainThread WARNING toil.leader: Job 'CWLWorkflow' kind-CWLWorkflow/instancedr0axrjw with ID kind-CWLWorkflow/instancedr0axrjw is completely failed
Traceback (most recent call last):
  File "/home/ubuntu/.local/bin/cwltoil", line 11, in <module>
    sys.exit(main())
  File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/cwl/cwltoil.py", line 1357, in main
    result = toil.start(wf1)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/common.py", line 800, in start
    return self._runMainLoop(rootJobGraph)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/common.py", line 1070, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/toil/leader.py", line 246, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException
```

If `default` is removed in `tool.cwl` it also works, so there seem to be a collision when both an input is provided and default value is specified. This only happens in a nested workflow. Running the workflow directly works fine:

```
$ cwltoil --logError sub.cwl --inp Hello
{}
```

Everything works as expected in `cwltool`:
```
$ pip3 freeze | grep cwltool
cwltool==2.0.20200224214940
$ cwltool --quiet wf.cwl --inp Hello
WARNING Workflow checker warning:
sub.cwl:6:3:  Source 'inp' of type ["null", "string"] may be incompatible
sub.cwl:11:7:   with sink 'inp' of type "string"

{}
```

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-517)
┆Issue Number: TOIL-517



Discussion:

Resolving patch:
diff --git a/src/toil/cwl/conftest.py b/src/toil/cwl/conftest.py
--- a/src/toil/cwl/conftest.py
+++ b/src/toil/cwl/conftest.py
@@ -1,4 +1,4 @@
-# Copyright (C) 2015-2016 Regents of the University of California
+# Copyright (C) 2015-2020 Regents of the University of California
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -1,8 +1,7 @@
-""" Implement support for Common Workflow Language (CWL) for Toil."""
-#
+"""Implemented support for Common Workflow Language (CWL) for Toil."""
 # Copyright (C) 2015 Curoverse, Inc
-# Copyright (C) 2016 UCSC Computational Genomics Lab
-# Copyright (C) 2019 Seven Bridges
+# Copyright (C) 2016-2020 Regents of the University of California
+# Copyright (C) 2019-2020 Seven Bridges
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -18,10 +17,6 @@
 
 # For an overview of how this all works, see discussion in
 # docs/architecture.rst
-
-from builtins import range
-from builtins import object
-import abc
 import argparse
 import os
 import tempfile
@@ -30,16 +25,23 @@
 import logging
 import copy
 import functools
-import shutil
-from typing import Text, Mapping, MutableSequence, MutableMapping
-import hashlib
 import uuid
 import datetime
+from typing import (Text,
+                    Mapping,
+                    MutableSequence,
+                    MutableMapping,
+                    List,
+                    Dict,
+                    Union,
+                    Any,
+                    Iterator,
+                    TextIO,
+                    Set)
 
 # Python 3 compatibility imports
 from six import iteritems, string_types
 from six.moves.urllib import parse as urlparse
-import six
 
 from schema_salad import validate
 from schema_salad.schema import Names
@@ -56,19 +58,20 @@
 import cwltool.command_line_tool
 import cwltool.provenance
 
-from toil.jobStores.abstractJobStore import NoSuchJobStoreException
+from toil.jobStores.abstractJobStore import NoSuchJobStoreException, NoSuchFileException
 from toil.fileStores import FileID
+from toil.fileStores.abstractFileStore import AbstractFileStore
 from cwltool.loghandler import _logger as cwllogger
 from cwltool.loghandler import defaultStreamHandler
 from cwltool.pathmapper import (PathMapper, adjustDirObjs, adjustFileObjs,
                                 get_listing, MapperEnt, visit_class,
                                 normalizeFilesDirs)
 from cwltool.process import (shortname, fill_in_defaults, compute_checksums,
-                             add_sizes)
+                             add_sizes, Process)
 from cwltool.secrets import SecretStore
 from cwltool.software_requirements import (
     DependenciesConfiguration, get_container_from_software_requirements)
-from cwltool.utils import aslist
+from cwltool.utils import aslist, convert_pathsep_to_unix
 from cwltool.mutation import MutationManager
 
 from toil.job import Job, Promise
@@ -76,7 +79,10 @@
 from toil.version import baseVersion
 
 # Define internal jobs we should avoid submitting to batch systems and logging
-CWL_INTERNAL_JOBS = ("CWLJobWrapper", "CWLWorkflow", "CWLScatter", "CWLGather",
+CWL_INTERNAL_JOBS = ("CWLJobWrapper",
+                     "CWLWorkflow",
+                     "CWLScatter",
+                     "CWLGather",
                      "ResolveIndirect")
 
 # The job object passed into CWLJob and CWLWorkflow
@@ -94,8 +100,8 @@ def cwltoil_was_removed():
     raise RuntimeError('Please run with "toil-cwl-runner" instead of "cwltoil" (which has been removed).')
 
 
-class IndirectDict(dict):
-    """Tag to indicate a dict is an IndirectDict that needs to resolved."""
+class UnresolvedDict(dict):
+    """Tag to indicate a dict contains promises that must be resolved."""
     pass
 
 
@@ -108,7 +114,7 @@ class SkipNull:
     pass
 
 
-def filter_skip_null(name, value):
+def filter_skip_null(name: str, value: Any) -> Any:
     """
     Recursively filter out SkipNull objects from 'value'.
 
@@ -119,7 +125,7 @@ def filter_skip_null(name, value):
     :param value: port output value object
     """
     err_flag = [False]
-    value = _filter_skip_null(name, value, err_flag)
+    value = _filter_skip_null(value, err_flag)
     if err_flag[0]:
         cwllogger.warning(
             "In %s, SkipNull result found and cast to None. \n"
@@ -128,14 +134,10 @@ def filter_skip_null(name, value):
     return value
 
 
-def _filter_skip_null(name, value, err_flag):
+def _filter_skip_null(value: Any, err_flag: List[bool]) -> Any:
     """
     Private implementation for recursively filtering out SkipNull objects from 'value'.
 
-    :param name: Name of port producing this value.
-                 Only used when we find an unhandled null from a conditional step
-                 and we print out a warning. The name allows the user to better
-                 localize which step/port was responsible for the unhandled null.
     :param value: port output value object
     :param err_flag: A pass by reference boolean (passed by enclosing in a list) that
                      allows us to flag, at any level of recursion, that we have encountered
@@ -145,9 +147,9 @@ def _filter_skip_null(name, value, err_flag):
         err_flag[0] = True
         value = None
     elif isinstance(value, list):
-        return [_filter_skip_null(name, v, err_flag) for v in value]
+        return [_filter_skip_null(v, err_flag) for v in value]
     elif isinstance(value, dict):
-        return {k: _filter_skip_null(name, v, err_flag) for k, v in value.items()}
+        return {k: _filter_skip_null(v, err_flag) for k, v in value.items()}
     return value
 
 
@@ -156,7 +158,10 @@ class Conditional:
     Object holding conditional expression until we are ready to evaluate it.
     Evaluation occurs at the moment the encloses step is ready to run.
     """
-    def __init__(self, expression=None, outputs=None, requirements=None):
+    def __init__(self,
+                 expression: Union[str, None] = None,
+                 outputs: Union[dict, None] = None,
+                 requirements: List[Dict[str, Any]] = None):
         """
         Instantiate a conditional expression.
 
@@ -171,7 +176,7 @@ def __init__(self, expression=None, outputs=None, requirements=None):
         self.outputs = outputs
         self.requirements = requirements
 
-    def is_false(self, job):
+    def is_false(self, job: Union[dict, None]) -> bool:
         """
         Determine if expression evaluates to False given completed step inputs
 
@@ -184,7 +189,7 @@ def is_false(self, job):
         expr_is_true = cwltool.expression.do_eval(
             self.expression,
             {shortname(k): v for k, v in iteritems(
-                resolve_indirect(job))},
+                resolve_dict_w_promises(job))},
             self.requirements,
             None,
             None,
@@ -196,7 +201,7 @@ def is_false(self, job):
         raise cwltool.errors.WorkflowException(
                 "'%s' evaluated to a non-boolean value" % self.expression)
 
-    def skipped_outputs(self):
+    def skipped_outputs(self) -> dict:
         """
         Generate a dictionary of SkipNull objects corresponding to the output
         structure of the step.
@@ -217,11 +222,11 @@ def sn(n):
         return outobj
 
 
-class ResolveSource(object):
+class ResolveSource:
     """
     Apply linkMerge and pickValue operators to values coming into a port.
     """
-    def __init__(self, name, input, source_key, promises):
+    def __init__(self, name: str, input: dict, source_key: str, promises: dict):
         """
         Construct a container object that carries what information it can about the input
         sources and the current promises, ready for evaluation when the time comes.
@@ -246,30 +251,29 @@ def __init__(self, name, input, source_key, promises):
             # '#' in the name will be returned as a
             # CommentedSeq list by the yaml parser.
             s = str(source_names[0])
-            self.promise_tuples = (shortname(s), promises[s].rv())
+            self.promise_tuples = (shortname(s), promises[s].rv())  # type: ignore
 
-    def resolve(self):
+    def resolve(self) -> Any:
         """
         First apply linkMerge then pickValue if either present
 
         :return: dict
         """
         if isinstance(self.promise_tuples, list):
-            result = self.link_merge([v[1][v[0]] for v in self.promise_tuples])
+            result = self.link_merge([v[1][v[0]] for v in self.promise_tuples])  # type: ignore
         else:
             value = self.promise_tuples
-            result = value[1].get(value[0])
+            result = value[1].get(value[0])  # type: ignore
 
-        result = self.pick_value(result)
+        result = self.pick_value(result)  # type: ignore
         result = filter_skip_null(self.name, result)
         return result
 
-    def link_merge(self, values):
+    def link_merge(self, values: dict) -> Union[list, dict]:
         """
         Apply linkMerge operator to `values` object
 
         :param values: dict: result of step
-        :return: dict
         """
         link_merge_type = self.input.get("linkMerge", "merge_nested")
 
@@ -277,7 +281,7 @@ def link_merge(self, values):
             return values
 
         elif link_merge_type == "merge_flattened":
-            result = []
+            result = []  # type: ignore
             for v in values:
                 if isinstance(v, MutableSequence):
                     result.extend(v)
@@ -289,11 +293,11 @@ def link_merge(self, values):
             raise validate.ValidationException(
                 "Unsupported linkMerge '%s' on %s." % (link_merge_type, self.name))
 
-    def pick_value(self, values):
+    def pick_value(self, values: Union[List, Any]) -> Any:
         """
         Apply pickValue operator to `values` object
 
-        :param values: dict
+        :param values: Intended to be a list, but other types will be returned without modification.
         :return:
         """
         pick_value_type = self.input.get("pickValue")
@@ -333,12 +337,12 @@ def pick_value(self, values):
                 "Unsupported pickValue '%s' on %s" % (pick_value_type, self.name))
 
 
-class StepValueFrom(object):
+class StepValueFrom:
     """
     A workflow step input which has a valueFrom expression attached to it, which
     is evaluated to produce the actual input object for the step.
     """
-    def __init__(self, expr, source, req):
+    def __init__(self, expr: str, source: Any, req: List[Dict[str, Any]]):
         """
         Instantiate an object to carry as much information as we have access to right now
         about this valueFrom expression
@@ -352,7 +356,7 @@ def __init__(self, expr, source, req):
         self.context = None
         self.req = req
 
-    def resolve(self):
+    def resolve(self) -> Any:
         """
         The valueFrom expression's context is the value for this input. Resolve the promise.
 
@@ -361,7 +365,7 @@ def resolve(self):
         self.context = self.source.resolve()
         return self.context
 
-    def do_eval(self, inputs):
+    def do_eval(self, inputs: Dict[str, cwltool.expression.JSON]) -> Any:
         """
         Evaluate the valueFrom expression with the given input object
 
@@ -372,11 +376,11 @@ def do_eval(self, inputs):
             self.expr, inputs, self.req, None, None, {}, context=self.context)
 
 
-class DefaultWithSource(object):
+class DefaultWithSource:
     """
     A workflow step input that has both a source and a default value.
     """
-    def __init__(self, default, source):
+    def __init__(self, default: Any, source: Any):
         """
         Instantiate an object to handle a source that has a default value
 
@@ -386,7 +390,7 @@ def __init__(self, default, source):
         self.default = default
         self.source = source
 
-    def resolve(self):
+    def resolve(self) -> Any:
         """
         Determine the final input value when the time is right (when the source can be resolved)
 
@@ -399,33 +403,33 @@ def resolve(self):
         return self.default
 
 
-class JustAValue(object):
+class JustAValue:
     """
     A simple value masquerading as a 'resolve'-able object
     """
 
-    def __init__(self, val):
+    def __init__(self, val: Any):
         self.val = val
 
-    def resolve(self):
+    def resolve(self) -> Any:
         return self.val
 
 
-def resolve_indirect(pdict):
+def resolve_dict_w_promises(dict_w_promises: dict) -> dict:
     """
-    Resolve the contents of an indirect dictionary (containing promises) and
+    Resolve the contents of an unresolved dictionary (containing promises) and
     evaluate expressions to produce the dictionary of actual values.
 
-    :param pdict: input pdict for these values
+    :param dict_w_promises: input dict for these values
     :return:
     """
-    if isinstance(pdict, IndirectDict):
-        first_pass_results = {k: v.resolve() for k, v in pdict.items()}
+    if isinstance(dict_w_promises, UnresolvedDict):
+        first_pass_results = {k: v.resolve() for k, v in dict_w_promises.items()}
     else:
-        first_pass_results = {k: v for k, v in pdict.items()}
+        first_pass_results = {k: v for k, v in dict_w_promises.items()}
 
     result = {}
-    for k, v in pdict.items():
+    for k, v in dict_w_promises.items():
         if isinstance(v, StepValueFrom):
             result[k] = v.do_eval(inputs=first_pass_results)
         else:
@@ -433,7 +437,7 @@ def resolve_indirect(pdict):
     return result
 
 
-def simplify_list(maybe_list):
+def simplify_list(maybe_list: Any) -> Any:
     """Turn a length one list loaded by cwltool into a scalar.
     Anything else is passed as-is, by reference."""
     if isinstance(maybe_list, MutableSequence):
@@ -450,92 +454,110 @@ class ToilPathMapper(PathMapper):
     and the the location of the file inside the Docker container.
     """
 
-    def __init__(self, referenced_files, basedir, stagedir,
-                 separateDirs=True,
-                 get_file=None,
-                 stage_listing=False):
+    def __init__(self,
+                 referenced_files: list,
+                 basedir: str,
+                 stagedir: str,
+                 separateDirs: bool = True,
+                 get_file: Union[Any, None] = None,
+                 stage_listing: bool = False):
         self.get_file = get_file
         self.stage_listing = stage_listing
         super(ToilPathMapper, self).__init__(
             referenced_files, basedir, stagedir, separateDirs=separateDirs)
 
-    def visit(self, obj, stagedir, basedir, copy=False, staged=False):
-        # type: (Dict[Text, Any], Text, Text, bool, bool) -> None
-        tgt = os.path.join(stagedir, obj["basename"])
+    def visit(self,
+              obj: Dict[str, Any],
+              stagedir: str,
+              basedir: str,
+              copy: bool = False,
+              staged: bool = False) -> None:
+        tgt = convert_pathsep_to_unix(os.path.join(stagedir, obj["basename"]))
         if obj["location"] in self._pathmap:
             return
         if obj["class"] == "Directory":
             if obj["location"].startswith("file://"):
-                resolved = schema_salad.ref_resolver.uri_file_path(
-                    obj["location"])
+                resolved = schema_salad.ref_resolver.uri_file_path(obj["location"])
             else:
                 resolved = obj["location"]
             self._pathmap[obj["location"]] = MapperEnt(
-                resolved, tgt,
-                "WritableDirectory" if copy else "Directory", staged)
-            if obj["location"].startswith("file://") \
-                    and not self.stage_listing:
+                resolved, tgt, "WritableDirectory" if copy else "Directory", staged)
+
+            if obj["location"].startswith("file://") and not self.stage_listing:
                 staged = False
             self.visitlisting(
                 obj.get("listing", []), tgt, basedir, copy=copy, staged=staged)
+
         elif obj["class"] == "File":
-            loc = obj["location"]
+            path = obj["location"]
             if "contents" in obj and obj["location"].startswith("_:"):
-                self._pathmap[obj["location"]] = MapperEnt(
-                    obj["contents"], tgt, "CreateFile", staged)
+                self._pathmap[path] = MapperEnt(
+                    obj["contents"],
+                    tgt,
+                    "CreateFile",  # TODO: Allow "WritableFile" here; see base class
+                    staged)
             else:
-                resolved = self.get_file(loc) if self.get_file else loc
+                resolved = self.get_file(path) if self.get_file else path
                 if resolved.startswith("file:"):
-                    resolved = schema_salad.ref_resolver.uri_file_path(
-                        resolved)
-                self._pathmap[loc] = MapperEnt(
+                    resolved = schema_salad.ref_resolver.uri_file_path(resolved)
+                self._pathmap[path] = MapperEnt(
                     resolved, tgt, "WritableFile" if copy else "File", staged)
                 self.visitlisting(obj.get("secondaryFiles", []),
                                   stagedir, basedir, copy=copy, staged=staged)
 
 
 class ToilCommandLineTool(cwltool.command_line_tool.CommandLineTool):
-    """Subclass the cwltool command line tool to provide the custom
-    Toil.PathMapper.
+    """Subclass the cwltool command line tool to provide the custom Toil.PathMapper."""
+    def make_path_mapper(self,
+                         reffiles: List[Any],
+                         stagedir: str,
+                         runtimeContext: cwltool.context.RuntimeContext,
+                         separateDirs: bool) -> cwltool.pathmapper.PathMapper:
+        return ToilPathMapper(reffiles,
+                              runtimeContext.basedir,
+                              stagedir,
+                              separateDirs,
+                              runtimeContext.toil_get_file)
+
+
+def toil_make_tool(toolpath_object: MutableMapping[str, Any],
+                   loading_context: cwltool.context.LoadingContext) -> cwltool.command_line_tool.CommandLineTool:
     """
-
-    def make_path_mapper(self, reffiles, stagedir, runtimeContext,
-                         separateDirs):
-        return ToilPathMapper(
-            reffiles, runtimeContext.basedir, stagedir, separateDirs,
-            runtimeContext.toil_get_file)
-
-
-def toil_make_tool(toolpath_object, loading_context):
-    """Factory function passed to load_tool() which creates instances of the
+    Factory function passed to load_tool() which creates instances of the
     custom ToilCommandLineTool.
-
     """
-
-    if isinstance(toolpath_object, Mapping) \
-            and toolpath_object.get("class") == "CommandLineTool":
+    if isinstance(toolpath_object, Mapping) and toolpath_object.get("class") == "CommandLineTool":
         return ToilCommandLineTool(toolpath_object, loading_context)
     return cwltool.workflow.default_make_tool(toolpath_object, loading_context)
 
 
 class ToilFsAccess(cwltool.stdfsaccess.StdFsAccess):
-    """Custom filesystem access class which handles toil filestore references.
-
-    """
-
-    def __init__(self, basedir, file_store=None):
+    """Custom filesystem access class which handles toil filestore references."""
+    def __init__(self, basedir: str, file_store: AbstractFileStore = None):
         self.file_store = file_store
         super(ToilFsAccess, self).__init__(basedir)
 
-    def _abs(self, path):
+    def exists(self, path: str) -> bool:
+        # toil's _abs() throws errors when files are not found and cwltool's _abs() does not
+        try:
+            return os.path.exists(self._abs(path))
+        except NoSuchFileException:
+            return False
+
+    def _abs(self, path: str) -> str:
+        """
+        Return a local absolute path for a file (no schema).
+
+        Overwrites cwltool.stdfsaccess.StdFsAccess._abs() to account for toil specific schema.
+        """
+        # Note: toil's _abs() throws errors when files are not found and cwltool's _abs() does not
         if path.startswith("toilfs:"):
             return self.file_store.readGlobalFile(FileID.unpack(path[7:]))
         return super(ToilFsAccess, self)._abs(path)
 
 
-def toil_get_file(file_store, index, existing, file_store_id):
+def toil_get_file(file_store: AbstractFileStore, index: dict, existing: dict, file_store_id: str) -> str:
     """Get path to input file from Toil jobstore."""
-
     if not file_store_id.startswith("toilfs:"):
         return file_store.jobStore.getPublicUrl(file_store.jobStore.importFile(file_store_id))
     src_path = file_store.readGlobalFile(FileID.unpack(file_store_id[7:]))
@@ -544,86 +566,90 @@ def toil_get_file(file_store, index, existing, file_store_id):
     return schema_salad.ref_resolver.file_uri(src_path)
 
 
-def write_file(writeFunc, index, existing, x):
-    """Write a file into the Toil jobstore.
+def write_file(writeFunc: Any, index: dict, existing: dict, file_uri: str) -> str:
+    """
+    Write a file into the Toil jobstore.
 
     'existing' is a set of files retrieved as inputs from toil_get_file. This
     ensures they are mapped back as the same name if passed through.
 
+    Returns a toil uri path to the object.
     """
     # Toil fileStore reference
-    if x.startswith("toilfs:"):
-        return x
+    if file_uri.startswith("toilfs:"):
+        return file_uri
     # File literal outputs with no path, we don't write these and will fail
     # with unsupportedRequirement when retrieving later with getFile
-    elif x.startswith("_:"):
-        return x
+    elif file_uri.startswith("_:"):
+        return file_uri
     else:
-        x = existing.get(x, x)
-        if x not in index:
-            if not urlparse.urlparse(x).scheme:
-                rp = os.path.realpath(x)
+        file_uri = existing.get(file_uri, file_uri)
+        if file_uri not in index:
+            if not urlparse.urlparse(file_uri).scheme:
+                rp = os.path.realpath(file_uri)
             else:
-                rp = x
+                rp = file_uri
             try:
-                index[x] = "toilfs:" + writeFunc(rp).pack()
-                existing[index[x]] = x
+                index[file_uri] = "toilfs:" + writeFunc(rp).pack()
+                existing[index[file_uri]] = file_uri
             except Exception as e:
-                cwllogger.error("Got exception '%s' while copying '%s'", e, x)
+                cwllogger.error("Got exception '%s' while copying '%s'", e, file_uri)
                 raise
-        return index[x]
+        return index[file_uri]
 
 
-def uploadFile(uploadfunc, fileindex, existing, uf, skip_broken=False):
-    """Update a file object so that the location is a reference to the toil file
+def uploadFile(uploadfunc: Any,
+               fileindex: dict,
+               existing: dict,
+               file_metadata: dict,
+               skip_broken: bool = False) -> None:
+    """
+    Update a file object so that the location is a reference to the toil file
     store, writing it to the file store if necessary.
-
     """
-
-    if uf["location"].startswith("toilfs:") or uf["location"].startswith("_:"):
+    if file_metadata["location"].startswith("toilfs:") or file_metadata["location"].startswith("_:"):
         return
-    if uf["location"] in fileindex:
-        uf["location"] = fileindex[uf["location"]]
+    if file_metadata["location"] in fileindex:
+        file_metadata["location"] = fileindex[file_metadata["location"]]
         return
-    if not uf["location"] and uf["path"]:
-        uf["location"] = schema_salad.ref_resolver.file_uri(uf["path"])
-    if uf["location"].startswith("file://") and not os.path.isfile(uf["location"][7:]):
+    if not file_metadata["location"] and file_metadata["path"]:
+        file_metadata["location"] = schema_salad.ref_resolver.file_uri(file_metadata["path"])
+    if file_metadata["location"].startswith("file://") and not os.path.isfile(file_metadata["location"][7:]):
         if skip_broken:
             return
         else:
             raise cwltool.errors.WorkflowException(
-                "File is missing: %s" % uf["location"])
-    uf["location"] = write_file(
-        uploadfunc, fileindex, existing, uf["location"])
+                "File is missing: %s" % file_metadata["location"])
+    file_metadata["location"] = write_file(
+        uploadfunc, fileindex, existing, file_metadata["location"])
 
 
-def writeGlobalFileWrapper(file_store, fileuri):
-    """Wrap writeGlobalFile to accepts file:// URIs"""
+def writeGlobalFileWrapper(file_store: AbstractFileStore, fileuri: str) -> str:
+    """Wrap writeGlobalFile to accept file:// URIs"""
     return file_store.writeGlobalFile(
         schema_salad.ref_resolver.uri_file_path(fileuri))
 
 
 class ResolveIndirect(Job):
-    """A helper Job which accepts an indirect dict (containing promises) and
+    """A helper Job which accepts an unresolved dict (containing promises) and
     produces a dictionary of actual values.
 
     """
 
-    def __init__(self, cwljob):
+    def __init__(self, cwljob: dict):
         super(ResolveIndirect, self).__init__()
         self.cwljob = cwljob
 
-    def run(self, file_store):
-        return resolve_indirect(self.cwljob)
-
+    def run(self, file_store: AbstractFileStore) -> dict:
+        return resolve_dict_w_promises(self.cwljob)
 
-def toilStageFiles(file_store, cwljob, outdir, index, existing, export,
-                   destBucket=None):
-    """Copy input files out of the global file store and update location and
-    path."""
 
-    def _collectDirEntries(obj):
-    # type: (Union[Dict[Text, Any], List[Dict[Text, Any]]]) -> Iterator[Dict[Text, Any]]
+def toilStageFiles(file_store: AbstractFileStore,
+                   cwljob: Union[Dict[Text, Any], List[Dict[Text, Any]]],
+                   outdir: str,
+                   destBucket: Union[str, None] = None) -> None:
+    """Copy input files out of the global file store and update location and path."""
+    def _collectDirEntries(obj: Union[Dict[Text, Any], List[Dict[Text, Any]]]) -> Iterator[Dict[Text, Any]]:
         if isinstance(obj, dict):
             if obj.get("class") in ("File", "Directory"):
                 yield obj
@@ -668,7 +694,7 @@ def _collectDirEntries(obj):
             with open(p.target, "wb") as n:
                 n.write(p.resolved.encode("utf-8"))
 
-    def _check_adjust(f):
+    def _check_adjust(f: dict) -> dict:
         f["location"] = schema_salad.ref_resolver.file_uri(
             pm.mapper(f["location"])[1])
         if "contents" in f:
@@ -681,39 +707,48 @@ def _check_adjust(f):
 class CWLJobWrapper(Job):
     """Wrap a CWL job that uses dynamic resources requirement.  When executed, this
     creates a new child job which has the correct resource requirement set.
-
     """
-
-    def __init__(self, tool, cwljob, runtime_context, conditional):
-        super(CWLJobWrapper, self).__init__(
-            cores=1, memory=1024*1024, disk=8*1024)
+    def __init__(self,
+                 tool: ToilCommandLineTool,
+                 cwljob: dict,
+                 runtime_context: cwltool.context.RuntimeContext,
+                 conditional: Union[Conditional, None] = None):
+        super(CWLJobWrapper, self).__init__(cores=1, memory=1024*1024, disk=8*1024)
         self.cwltool = remove_pickle_problems(tool)
         self.cwljob = cwljob
         self.runtime_context = runtime_context
         self.conditional = conditional
 
-    def run(self, file_store):
-        cwljob = resolve_indirect(self.cwljob)
+    def run(self, file_store: AbstractFileStore) -> Any:
+        cwljob = resolve_dict_w_promises(self.cwljob)
         fill_in_defaults(
-            self.cwltool.tool['inputs'], cwljob,
-            self.runtime_context.make_fs_access(
-                self.runtime_context.basedir or ""))
-        realjob = CWLJob(self.cwltool, cwljob, self.runtime_context, conditional=self.conditional)
+            self.cwltool.tool['inputs'],
+            cwljob,
+            self.runtime_context.make_fs_access(self.runtime_context.basedir or ""))
+        realjob = CWLJob(
+            tool=self.cwltool,
+            cwljob=cwljob,
+            runtime_context=self.runtime_context,
+            conditional=self.conditional)
         self.addChild(realjob)
         return realjob.rv()
 
 
 class CWLJob(Job):
     """Execute a CWL tool using cwltool.executors.SingleJobExecutor"""
+    def __init__(self,
+                 tool: ToilCommandLineTool,
+                 cwljob: dict,
+                 runtime_context: cwltool.context.RuntimeContext,
+                 conditional: Union[Conditional, None] = None):
 
-    def __init__(self, tool, cwljob, runtime_context, step_inputs=None, conditional=None):
         self.cwltool = remove_pickle_problems(tool)
         self.conditional = conditional or Conditional()
 
         if runtime_context.builder:
-            builder = runtime_context.builder
+            self.builder = runtime_context.builder
         else:
-            builder = cwltool.builder.Builder(
+            self.builder = cwltool.builder.Builder(
                 job=cwljob,
                 files=[],
                 bindings=[],
@@ -731,12 +766,13 @@ def __init__(self, tool, cwljob, runtime_context, step_inputs=None, conditional=
                 debug=False,
                 js_console=False,
                 force_docker_pull=False,
-                loadListing=u'',
-                outdir=u'',
-                tmpdir=u'',
-                stagedir=u''
+                loadListing='',
+                outdir='',
+                tmpdir='/tmp',  # TODO: use actual defaults here
+                stagedir='/var/lib/cwl'  # TODO: use actual defaults here
             )
-        req = tool.evalResources(builder, runtime_context)
+
+        req = tool.evalResources(self.builder, runtime_context)
         # pass the default of None if basecommand is empty
         unitName = self.cwltool.tool.get("baseCommand", None)
         if isinstance(unitName, (MutableSequence, tuple)):
@@ -748,7 +784,8 @@ def __init__(self, tool, cwljob, runtime_context, step_inputs=None, conditional=
             displayName = None
 
         super(CWLJob, self).__init__(
-            cores=req["cores"], memory=int(req["ram"]*(2**20)),
+            cores=req["cores"],
+            memory=int(req["ram"]*(2**20)),
             disk=int((req["tmpdirSize"]*(2**20))+(req["outdirSize"]*(2**20))),
             unitName=unitName,
             displayName=displayName)
@@ -761,13 +798,14 @@ def __init__(self, tool, cwljob, runtime_context, step_inputs=None, conditional=
             # an identifier
             pass
         self.runtime_context = runtime_context
-        self.step_inputs = step_inputs or self.cwltool.tool["inputs"]
+        self.step_inputs = self.cwltool.tool["inputs"]
         self.workdir = runtime_context.workdir
 
-    def run(self, file_store):
-        cwljob = resolve_indirect(self.cwljob)
+    def run(self, file_store: AbstractFileStore) -> Any:
+        cwljob = resolve_dict_w_promises(self.cwljob)
         fill_in_defaults(
-            self.step_inputs, cwljob,
+            self.step_inputs,
+            cwljob,
             self.runtime_context.make_fs_access(""))
 
         if self.conditional.is_false(cwljob):
@@ -795,8 +833,8 @@ def run(self, file_store):
         # cleaned up by Toil.
         tmp_outdir_prefix = os.path.join(file_store.getLocalTempDir(), "tmp-out")
 
-        index = {}
-        existing = {}
+        index = {}  # type: ignore
+        existing = {}  # type: ignore
         # Prepare the run instructions for cwltool
         runtime_context = self.runtime_context.copy()
         runtime_context.basedir = os.getcwd()
@@ -805,14 +843,18 @@ def run(self, file_store):
         runtime_context.tmpdir_prefix = file_store.getLocalTempDir()
         runtime_context.make_fs_access = functools.partial(
             ToilFsAccess, file_store=file_store)
+
         runtime_context.toil_get_file = functools.partial(
             toil_get_file, file_store, index, existing)
 
         process_uuid = uuid.uuid4()
         started_at = datetime.datetime.now()
         # Run the tool
-        (output, status) = cwltool.executors.SingleJobExecutor().execute(
-            self.cwltool, cwljob, runtime_context, cwllogger)
+        output, status = cwltool.executors.SingleJobExecutor().execute(
+            process=self.cwltool,
+            job_order_object=cwljob,
+            runtime_context=runtime_context,
+            logger=cwllogger)
         ended_at = datetime.datetime.now()
         if status != "success":
             raise cwltool.errors.WorkflowException(status)
@@ -821,6 +863,7 @@ def run(self, file_store):
             get_listing, cwltool.stdfsaccess.StdFsAccess(outdir),
             recursive=True))
 
+        # write the outputs into the jobstore
         adjustFileObjs(output, functools.partial(
             uploadFile, functools.partial(writeGlobalFileWrapper, file_store),
             index, existing))
@@ -835,19 +878,22 @@ def run(self, file_store):
         return output
 
 
-def makeJob(tool, jobobj, step_inputs, runtime_context, conditional):
-    """Create the correct Toil Job object for the CWL tool (workflow, job, or job
+def makeJob(tool: Process,
+            jobobj: dict,
+            runtime_context: cwltool.context.RuntimeContext,
+            conditional: Union[Conditional, None]) -> tuple:
+    """
+    Create the correct Toil Job object for the CWL tool (workflow, job, or job
     wrapper for dynamic resource requirements.)
 
+    :return: "wfjob, followOn" if the input tool is a workflow, and "job, job" otherwise
     """
-
     if tool.tool["class"] == "Workflow":
         wfjob = CWLWorkflow(tool, jobobj, runtime_context, conditional=conditional)
         followOn = ResolveIndirect(wfjob.rv())
         wfjob.addFollowOn(followOn)
-        return (wfjob, followOn)
+        return wfjob, followOn
     else:
-        # get_requirement
         resourceReq, _ = tool.get_requirement("ResourceRequirement")
         if resourceReq:
             for req in ("coresMin", "coresMax", "ramMin", "ramMax",
@@ -856,18 +902,21 @@ def makeJob(tool, jobobj, step_inputs, runtime_context, conditional):
                 if isinstance(r, string_types) and ("$(" in r or "${" in r):
                     # Found a dynamic resource requirement so use a job wrapper
                     job = CWLJobWrapper(tool, jobobj, runtime_context, conditional=conditional)
-                    return (job, job)
-        job = CWLJob(tool, jobobj, runtime_context, conditional=conditional)
-        return (job, job)
+                    return job, job
+        job = CWLJob(tool, jobobj, runtime_context, conditional=conditional)  # type: ignore
+        return job, job
 
 
 class CWLScatter(Job):
-    """Implement workflow scatter step.  When run, this creates a child job for
+    """
+    Implement workflow scatter step.  When run, this creates a child job for
     each parameterization of the scatter.
-
     """
-
-    def __init__(self, step, cwljob, runtime_context, conditional):
+    def __init__(self,
+                 step: cwltool.workflow.WorkflowStep,
+                 cwljob: dict,
+                 runtime_context: cwltool.context.RuntimeContext,
+                 conditional: Union[Conditional, None]):
         super(CWLScatter, self).__init__()
         self.step = step
         self.cwljob = cwljob
@@ -875,45 +924,52 @@ def __init__(self, step, cwljob, runtime_context, conditional):
         self.conditional = conditional
 
     def flat_crossproduct_scatter(self,
-                                  joborder,
-                                  scatter_keys,
-                                  outputs, postScatterEval):
+                                  joborder: dict,
+                                  scatter_keys: list,
+                                  outputs: list,
+                                  postScatterEval: Any) -> None:
         scatter_key = shortname(scatter_keys[0])
         for n in range(0, len(joborder[scatter_key])):
-            jo = copy.copy(joborder)
-            jo[scatter_key] = joborder[scatter_key][n]
+            updated_joborder = copy.copy(joborder)
+            updated_joborder[scatter_key] = joborder[scatter_key][n]
             if len(scatter_keys) == 1:
-                jo = postScatterEval(jo)
-                (subjob, followOn) = makeJob(
-                    self.step.embedded_tool, jo, None, self.runtime_context,
+                updated_joborder = postScatterEval(updated_joborder)
+                subjob, followOn = makeJob(
+                    tool=self.step.embedded_tool,
+                    jobobj=updated_joborder,
+                    runtime_context=self.runtime_context,
                     conditional=self.conditional)
                 self.addChild(subjob)
                 outputs.append(followOn.rv())
             else:
                 self.flat_crossproduct_scatter(
-                    jo, scatter_keys[1:], outputs, postScatterEval)
+                    updated_joborder, scatter_keys[1:], outputs, postScatterEval)
 
     def nested_crossproduct_scatter(self,
-                                    joborder, scatter_keys, postScatterEval):
+                                    joborder: dict,
+                                    scatter_keys: list,
+                                    postScatterEval: Any) -> list:
         scatter_key = shortname(scatter_keys[0])
         outputs = []
         for n in range(0, len(joborder[scatter_key])):
-            jo = copy.copy(joborder)
-            jo[scatter_key] = joborder[scatter_key][n]
+            updated_joborder = copy.copy(joborder)
+            updated_joborder[scatter_key] = joborder[scatter_key][n]
             if len(scatter_keys) == 1:
-                jo = postScatterEval(jo)
-                (subjob, followOn) = makeJob(
-                    self.step.embedded_tool, jo, None, self.runtime_context,
+                updated_joborder = postScatterEval(updated_joborder)
+                subjob, followOn = makeJob(
+                    tool=self.step.embedded_tool,
+                    jobobj=updated_joborder,
+                    runtime_context=self.runtime_context,
                     conditional=self.conditional)
                 self.addChild(subjob)
                 outputs.append(followOn.rv())
             else:
                 outputs.append(self.nested_crossproduct_scatter(
-                    jo, scatter_keys[1:], postScatterEval))
+                    updated_joborder, scatter_keys[1:], postScatterEval))
         return outputs
 
-    def run(self, file_store):
-        cwljob = resolve_indirect(self.cwljob)
+    def run(self, file_store: AbstractFileStore) -> list:
+        cwljob = resolve_dict_w_promises(self.cwljob)
 
         if isinstance(self.step.tool["scatter"], string_types):
             scatter = [self.step.tool["scatter"]]
@@ -928,19 +984,19 @@ def run(self, file_store):
         valueFrom = {shortname(i["id"]): i["valueFrom"]
                      for i in self.step.tool["inputs"] if "valueFrom" in i}
 
-        def postScatterEval(io):
-            shortio = {shortname(k): v for k, v in iteritems(io)}
+        def postScatterEval(job_dict: dict) -> Any:
+            shortio = {shortname(k): v for k, v in iteritems(job_dict)}
             for k in valueFrom:
-                io.setdefault(k, None)
+                job_dict.setdefault(k, None)
 
-            def valueFromFunc(k, v):
+            def valueFromFunc(k: str, v: Any) -> Any:
                 if k in valueFrom:
                     return cwltool.expression.do_eval(
                         valueFrom[k], shortio, self.step.requirements,
                         None, None, {}, context=v)
                 else:
                     return v
-            return {k: valueFromFunc(k, v) for k, v in list(io.items())}
+            return {k: valueFromFunc(k, v) for k, v in list(job_dict.items())}
 
         if scatterMethod == "dotproduct":
             for i in range(0, len(cwljob[shortname(scatter[0])])):
@@ -948,9 +1004,11 @@ def valueFromFunc(k, v):
                 for sc in [shortname(x) for x in scatter]:
                     copyjob[sc] = cwljob[sc][i]
                 copyjob = postScatterEval(copyjob)
-                (subjob, follow_on) = makeJob(
-                    self.step.embedded_tool, copyjob, None,
-                    self.runtime_context, conditional=self.conditional)
+                subjob, follow_on = makeJob(
+                    tool=self.step.embedded_tool,
+                    jobobj=copyjob,
+                    runtime_context=self.runtime_context,
+                    conditional=self.conditional)
                 self.addChild(subjob)
                 outputs.append(follow_on.rv())
         elif scatterMethod == "nested_crossproduct":
@@ -972,17 +1030,18 @@ def valueFromFunc(k, v):
 
 
 class CWLGather(Job):
-    """Follows on to a scatter.  This gathers the outputs of each job in the
+    """
+    Follows on to a scatter.  This gathers the outputs of each job in the
     scatter into an array for each output parameter.
-
     """
-
-    def __init__(self, step, outputs):
+    def __init__(self,
+                 step: cwltool.workflow.WorkflowStep,
+                 outputs: Union[Mapping, MutableSequence]):
         super(CWLGather, self).__init__()
         self.step = step
         self.outputs = outputs
 
-    def allkeys(self, obj, keys):
+    def allkeys(self, obj: Union[Mapping, MutableSequence], keys: Set) -> None:
         if isinstance(obj, Mapping):
             for k in list(obj.keys()):
                 keys.add(k)
@@ -990,7 +1049,7 @@ def allkeys(self, obj, keys):
             for l in obj:
                 self.allkeys(l, keys)
 
-    def extract(self, obj, k):
+    def extract(self, obj: Union[Mapping, MutableSequence], k: str) -> list:
         if isinstance(obj, Mapping):
             return obj.get(k)
         elif isinstance(obj, MutableSequence):
@@ -1001,7 +1060,7 @@ def extract(self, obj, k):
         else:
             return []
 
-    def run(self, file_store):
+    def run(self, file_store: AbstractFileStore) -> Any:
         outobj = {}
 
         def sn(n):
@@ -1016,27 +1075,24 @@ def sn(n):
         return outobj
 
 
-class SelfJob(object):
+class SelfJob:
     """Fake job object to facilitate implementation of CWLWorkflow.run()"""
-
-    def __init__(self, j, v):
+    def __init__(self, j: cwltool.workflow.Workflow, v: dict):
         self.j = j
         self.v = v
 
-    def rv(self):
+    def rv(self) -> Any:
         return self.v
 
-    def addChild(self, c):
+    def addChild(self, c: str) -> Any:
         return self.j.addChild(c)
 
-    def hasChild(self, c):
+    def hasChild(self, c: str) -> Any:
         return self.j.hasChild(c)
 
 
-def remove_pickle_problems(obj):
-    """doc_loader does not pickle correctly, causing Toil errors, remove from
-       objects.
-    """
+def remove_pickle_problems(obj: cwltool.workflow.Workflow) -> cwltool.workflow.Workflow:
+    """Doc_loader does not pickle correctly, causing Toil errors, remove from objects."""
     if hasattr(obj, "doc_loader"):
         obj.doc_loader = None
     if hasattr(obj, "embedded_tool"):
@@ -1047,12 +1103,12 @@ def remove_pickle_problems(obj):
 
 
 class CWLWorkflow(Job):
-    """Traverse a CWL workflow graph and create a Toil job graph with appropriate
-    dependencies.
-
-    """
-
-    def __init__(self, cwlwf, cwljob, runtime_context, conditional=None):
+    """Traverse a CWL workflow graph and create a Toil job graph with appropriate dependencies."""
+    def __init__(self,
+                 cwlwf: cwltool.workflow.Workflow,
+                 cwljob: dict,
+                 runtime_context: cwltool.context.RuntimeContext,
+                 conditional: Union[Conditional, None] = None):
         super(CWLWorkflow, self).__init__()
         self.cwlwf = cwlwf
         self.cwljob = cwljob
@@ -1060,8 +1116,8 @@ def __init__(self, cwlwf, cwljob, runtime_context, conditional=None):
         self.cwlwf = remove_pickle_problems(self.cwlwf)
         self.conditional = conditional or Conditional()
 
-    def run(self, file_store):
-        cwljob = resolve_indirect(self.cwljob)
+    def run(self, file_store: AbstractFileStore):
+        cwljob = resolve_dict_w_promises(self.cwljob)
 
         if self.conditional.is_false(cwljob):
             return self.conditional.skipped_outputs()
@@ -1074,7 +1130,7 @@ def run(self, file_store):
         promises = {}
 
         # `jobs` dict from step id to job that implements that step.
-        jobs = {}
+        jobs = {}  # type: ignore
 
         for inp in self.cwlwf.tool["inputs"]:
             promises[inp["id"]] = SelfJob(self, cwljob)
@@ -1104,18 +1160,17 @@ def run(self, file_store):
                             if "source" in inp:
                                 jobobj[key] = \
                                     ResolveSource(
-                                        name="%s/%s" % (step.tool["id"], key),
+                                        name=f'{step.tool["id"]}/{key}',
                                         input=inp,
                                         source_key="source",
                                         promises=promises)
 
                             if "default" in inp:
-                                jobobj[key] = DefaultWithSource(
+                                jobobj[key] = DefaultWithSource(  # type: ignore
                                     copy.copy(inp["default"]), jobobj.get(key))
 
-                            if "valueFrom" in inp \
-                                    and "scatter" not in step.tool:
-                                jobobj[key] = StepValueFrom(
+                            if "valueFrom" in inp and "scatter" not in step.tool:
+                                jobobj[key] = StepValueFrom(  # type: ignore
                                     inp["valueFrom"], jobobj.get(key, JustAValue(None)),
                                     self.cwlwf.requirements)
 
@@ -1125,16 +1180,17 @@ def run(self, file_store):
                             requirements=self.cwlwf.requirements)
 
                         if "scatter" in step.tool:
-                            wfjob = CWLScatter(step, IndirectDict(jobobj),
+                            wfjob = CWLScatter(step,
+                                               UnresolvedDict(jobobj),
                                                self.runtime_context,
                                                conditional=conditional)
                             followOn = CWLGather(step, wfjob.rv())
                             wfjob.addFollowOn(followOn)
                         else:
-                            (wfjob, followOn) = makeJob(
-                                step.embedded_tool, IndirectDict(jobobj),
-                                step.tool["inputs"],
-                                self.runtime_context,
+                            wfjob, followOn = makeJob(
+                                tool=step.embedded_tool,
+                                jobobj=UnresolvedDict(jobobj),
+                                runtime_context=self.runtime_context,
                                 conditional=conditional)
 
                         jobs[step.tool["id"]] = followOn
@@ -1144,13 +1200,13 @@ def run(self, file_store):
                             for s in aslist(inp.get("source", [])):
                                 if (isinstance(
                                         promises[s], (CWLJobWrapper, CWLGather)
-                                ) and not promises[s].hasFollowOn(wfjob)):
-                                    promises[s].addFollowOn(wfjob)
+                                ) and not promises[s].hasFollowOn(wfjob)):  # type: ignore
+                                    promises[s].addFollowOn(wfjob)  # type: ignore
                                     connected = True
                                 if (not isinstance(
                                         promises[s], (CWLJobWrapper, CWLGather)
-                                ) and not promises[s].hasChild(wfjob)):
-                                    promises[s].addChild(wfjob)
+                                ) and not promises[s].hasChild(wfjob)):  # type: ignore
+                                    promises[s].addChild(wfjob)  # type: ignore
                                     connected = True
                         if not connected:
                             # the workflow step has default inputs only & isn't
@@ -1159,7 +1215,7 @@ def run(self, file_store):
                             self.addChild(wfjob)
 
                         for out in step.tool["outputs"]:
-                            promises[out["id"]] = followOn
+                            promises[out["id"]] = followOn  # type: ignore
 
                 for inp in step.tool["inputs"]:
                     for source in aslist(inp.get("source", [])):
@@ -1182,7 +1238,7 @@ def run(self, file_store):
                     source_key="outputSource",
                     promises=promises))
 
-        return IndirectDict(outobj)
+        return UnresolvedDict(outobj)
 
 
 cwltool.process.supportedProcessRequirements = (
@@ -1194,14 +1250,44 @@ def run(self, file_store):
     "StepInputExpressionRequirement", "ResourceRequirement")
 
 
-def visitSteps(t, op):
-    if isinstance(t, cwltool.workflow.Workflow):
-        for s in t.steps:
-            op(s.tool)
-            visitSteps(s.embedded_tool, op)
+def visitSteps(cmdline_tool: Union[cwltool.command_line_tool.CommandLineTool, cwltool.workflow.Workflow],
+               op: Any) -> None:
+    if isinstance(cmdline_tool, cwltool.workflow.Workflow):
+        for step in cmdline_tool.steps:
+            op(step.tool)
+            visitSteps(step.embedded_tool, op)
+
+
+def remove_unprocessed_secondary_files(unfiltered_secondary_files: dict) -> list:
+    """
+    Interpolated strings and optional inputs in secondary files were added to CWL in version 1.1.
 
+    The CWL libraries we call do successfully resolve the interpolated strings, but add the resolved
+    fields to the list of unresolved fields so we remove them here after the fact.
 
-def main(args=None, stdout=sys.stdout):
+    We also remove any secondary files here not containing 'toilfs:', which means that it was not
+    successfully imported into the toil jobstore.  The 'required' logic seems to be handled deeper in
+    cwltool.builder.Builder(), and correctly determines which files should be imported.  Therefore we
+    remove the files here and if this file is SUPPOSED to exist, it will still give the appropriate file
+    does not exist error, but just a bit further down the track.
+    """
+    intermediate_secondary_files = []
+    final_secondary_files = []
+    # remove secondary files still containing interpolated strings
+    for sf in unfiltered_secondary_files['secondaryFiles']:
+        sf_bn = sf.get('basename', '')
+        if ('$(' not in sf_bn) and ('${' not in sf_bn):
+            intermediate_secondary_files.append(sf)
+    # remove secondary files that are not present in the filestore
+    # i.e. 'file://' only gets converted to 'toilfs:' upon a successful import
+    for sf in intermediate_secondary_files:
+        sf_loc = sf.get('location', '')
+        if sf_loc.startswith('toilfs:'):
+            final_secondary_files.append(sf)
+    return final_secondary_files
+
+
+def main(args: Union[List[str]] = None, stdout: TextIO = sys.stdout) -> int:
     """Main method for toil-cwl-runner."""
     cwllogger.removeHandler(defaultStreamHandler)
     config = Config()
@@ -1219,7 +1305,7 @@ def main(args=None, stdout=sys.stdout):
                         help="Enable loading and running development versions of CWL")
     parser.add_argument("--quiet", dest="logLevel", action="store_const",
                         const="ERROR")
-    parser.add_argument("--basedir", type=str)
+    parser.add_argument("--basedir", type=str)  # TODO: Might be hard-coded?
     parser.add_argument("--outdir", type=str, default=os.getcwd())
     parser.add_argument("--version", action='version', version=baseVersion)
     dockergroup = parser.add_mutually_exclusive_group()
@@ -1377,8 +1463,8 @@ def main(args=None, stdout=sys.stdout):
     outdir = os.path.abspath(options.outdir)
     tmp_outdir_prefix = os.path.abspath(options.tmp_outdir_prefix)
 
-    fileindex = {}
-    existing = {}
+    fileindex = {}  # type: ignore
+    existing = {}  # type: ignore
     conf_file = getattr(options,
                         "beta_dependency_resolvers_configuration", None)
     use_conda_dependencies = getattr(options, "beta_conda_dependencies", None)
@@ -1455,6 +1541,7 @@ def main(args=None, stdout=sys.stdout):
             initialized_job_order = cwltool.main.init_job_order(
                 job_order_object, options, tool, jobloader, sys.stdout,
                 secret_store=runtime_context.secret_store)
+
             fs_access = cwltool.stdfsaccess.StdFsAccess(options.basedir)
             fill_in_defaults(
                 tool.tool["inputs"], initialized_job_order, fs_access)
@@ -1464,25 +1551,27 @@ def path_to_loc(obj):
                     obj["location"] = obj["path"]
                     del obj["path"]
 
-            def import_files(tool):
-                visit_class(tool, ("File", "Directory"), path_to_loc)
-                visit_class(tool, ("File", ), functools.partial(
-                    add_sizes, fs_access))
-                normalizeFilesDirs(tool)
-                adjustDirObjs(tool, functools.partial(
+            def import_files(inner_tool):
+                visit_class(inner_tool, ("File", "Directory"), path_to_loc)
+                visit_class(inner_tool, ("File",), functools.partial(add_sizes, fs_access))
+                normalizeFilesDirs(inner_tool)
+                adjustDirObjs(inner_tool, functools.partial(
                     get_listing, fs_access, recursive=True))
-                adjustFileObjs(tool, functools.partial(
+                adjustFileObjs(inner_tool, functools.partial(
                     uploadFile, toil.importFile, fileindex, existing,
-                    skip_broken=True))
+                    skip_broken=True))  # actually import files into the jobstore
 
-            tool.visit(import_files)
+            import_files(tool.tool)
 
             for inp in tool.tool["inputs"]:
                 def set_secondary(fileobj):
                     if isinstance(fileobj, Mapping) and fileobj.get("class") == "File":
                         if "secondaryFiles" not in fileobj:
+                            # inits all secondary files with 'file://' schema
+                            # later changed to 'toilfs:' when imported into the jobstore
                             fileobj["secondaryFiles"] = [{"location": cwltool.builder.substitute(fileobj["location"],
-                                                         sf["pattern"]), "class": "File"}
+                                                          sf["pattern"]),
+                                                          "class": "File"}
                                                          for sf in inp["secondaryFiles"]]
 
                     if isinstance(fileobj, MutableSequence):
@@ -1492,26 +1581,48 @@ def set_secondary(fileobj):
                 if shortname(inp["id"]) in initialized_job_order and inp.get("secondaryFiles"):
                     set_secondary(initialized_job_order[shortname(inp["id"])])
 
+            runtime_context.use_container = use_container
+            runtime_context.tmp_outdir_prefix = os.path.realpath(tmp_outdir_prefix)
+            runtime_context.job_script_provider = job_script_provider
+            runtime_context.force_docker_pull = options.force_docker_pull
+            runtime_context.no_match_user = options.no_match_user
+            runtime_context.no_read_only = options.no_read_only
+            runtime_context.basedir = options.basedir
+
+            # We instantiate an early builder object here to populate indirect secondaryFile references
+            # using cwltool's library because we need to resolve them before toil imports them into the filestore.
+            # A second builder will be built in the job's run method when toil actually starts the cwl job.
+            builder = tool._init_job(initialized_job_order, runtime_context)
+            builder.bind_input(
+                    tool.inputs_record_schema,
+                    initialized_job_order,
+                    discover_secondaryFiles=True,
+            )
+
+            # files with the 'file://' uri are imported into the jobstore and changed to 'toilfs:'
             import_files(initialized_job_order)
+
             visitSteps(tool, import_files)
 
+            for job_name in initialized_job_order:
+                if isinstance(initialized_job_order[job_name], list):
+                    for job_params in initialized_job_order[job_name]:
+                        if isinstance(job_params, dict):
+                            if 'secondaryFiles' in job_params:
+                                job_params['secondaryFiles'] = remove_unprocessed_secondary_files(job_params)
+
             try:
-                runtime_context.use_container = use_container
-                runtime_context.tmp_outdir_prefix = os.path.realpath(
-                    tmp_outdir_prefix)
-                runtime_context.job_script_provider = job_script_provider
-                runtime_context.force_docker_pull = options.force_docker_pull
-                runtime_context.no_match_user = options.no_match_user
-                runtime_context.no_read_only = options.no_read_only
-                (wf1, _) = makeJob(tool, {}, None, runtime_context, conditional=None)
+                wf1, _ = makeJob(tool=tool,
+                                 jobobj={},
+                                 runtime_context=runtime_context,
+                                 conditional=None)
             except cwltool.process.UnsupportedRequirement as err:
                 logging.error(err)
                 return 33
-
             wf1.cwljob = initialized_job_order
             outobj = toil.start(wf1)
 
-        outobj = resolve_indirect(outobj)
+        outobj = resolve_dict_w_promises(outobj)
 
         # Stage files. Specify destination bucket if specified in CLI
         # options. If destination bucket not passed in,
@@ -1520,9 +1631,6 @@ def set_secondary(fileobj):
             toil,
             outobj,
             outdir,
-            fileindex,
-            existing,
-            export=True,
             destBucket=options.destBucket)
 
         if runtime_context.research_obj is not None:
@@ -1558,12 +1666,11 @@ def remove_at_id(doc):
     return 0
 
 
-def find_default_container(args, builder):
+def find_default_container(args: argparse.Namespace, builder: cwltool.builder.Builder) -> str:
     default_container = None
     if args.default_container:
         default_container = args.default_container
     elif args.beta_use_biocontainers:
-        default_container = get_container_from_software_requirements(
-            args, builder)
+        default_container = get_container_from_software_requirements(args, builder)
 
     return default_container


Test patch:
diff --git a/src/toil/test/cwl/conftest.py b/src/toil/test/cwl/conftest.py
--- a/src/toil/test/cwl/conftest.py
+++ b/src/toil/test/cwl/conftest.py
@@ -1,4 +1,4 @@
-# Copyright (C) 2015-2016 Regents of the University of California
+# Copyright (C) 2015-2020 Regents of the University of California
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -1,3 +1,4 @@
+# Copyright (C) 2015-2020 Regents of the University of California
 # Copyright (C) 2015 Curoverse, Inc
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -11,8 +12,6 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-from __future__ import absolute_import
-from __future__ import print_function
 import json
 import os
 import sys
@@ -23,10 +22,8 @@
 import zipfile
 import pytest
 import uuid
-from future.moves.urllib.request import urlretrieve
-from six.moves import StringIO
-from six import u as str
-from six import text_type
+from urllib.request import urlretrieve
+from io import StringIO
 
 pkg_root = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))  # noqa
 sys.path.insert(0, pkg_root)  # noqa
@@ -39,11 +36,10 @@
 
 
 @needs_cwl
-class CWLTest(ToilTest):
+class CWLv10Test(ToilTest):
     def setUp(self):
         """Runs anew before each test to create farm fresh temp dirs."""
-        from builtins import str as normal_str
-        self.outDir = os.path.join('/tmp/', 'toil-cwl-test-' + normal_str(uuid.uuid4()))
+        self.outDir = f'/tmp/toil-cwl-test-{str(uuid.uuid4())}'
         os.makedirs(self.outDir)
         self.rootDir = self._projectRootPath()
         self.cwlSpec = os.path.join(self.rootDir, 'src/toil/test/cwl/spec')
@@ -96,7 +92,7 @@ def revsort(self, cwl_filename, tester_fn):
                   self._expected_revsort_output(self.outDir))
 
     def download(self,inputs, tester_fn):
-        input_location = os.path.join('src/toil/test/cwl',inputs)
+        input_location = os.path.join('src/toil/test/cwl', inputs)
         tester_fn('src/toil/test/cwl/download.cwl',
                   input_location,
                   self._expected_download_output(self.outDir))
@@ -196,11 +192,7 @@ def test_run_conformance(self, batchSystem=None, caching=False):
             # check output -- if we failed but only have unsupported features, we're okay
             p = re.compile(r"(?P<failures>\d+) failures, (?P<unsupported>\d+) unsupported features")
 
-            error_log = e.output
-            if isinstance(e.output, bytes):
-                # py2/3 string handling
-                error_log = e.output.decode('utf-8')
-
+            error_log = e.output.decode('utf-8')
             for line in error_log.split('\n'):
                 m = p.search(line)
                 if m:
@@ -300,39 +292,99 @@ def _expected_seqtk_output(outDir):
         # Having unicode string literals isn't necessary for the assertion but
         # makes for a less noisy diff in case the assertion fails.
         loc = 'file://' + os.path.join(outDir, 'out')
-        loc = str(loc) if not isinstance(loc, text_type) else loc  # py2/3
         return {
-            u'output1':  {
-                u'location': loc,
-                u'checksum': u'sha1$322e001e5a99f19abdce9f02ad0f02a17b5066c2',
-                u'basename': u'out',
-                u'class': u'File',
-                u'size': 150}}
+            'output1':  {
+                'location': loc,
+                'checksum': 'sha1$322e001e5a99f19abdce9f02ad0f02a17b5066c2',
+                'basename': 'out',
+                'class': 'File',
+                'size': 150}}
 
     @staticmethod
     def _expected_revsort_output(outDir):
         # Having unicode string literals isn't necessary for the assertion but
         # makes for a less noisy diff in case the assertion fails.
         loc = 'file://' + os.path.join(outDir, 'output.txt')
-        loc = str(loc) if not isinstance(loc, text_type) else loc  # py2/3
         return {
-            u'output': {
-                u'location': loc,
-                u'basename': u'output.txt',
-                u'size': 1111,
-                u'class': u'File',
-                u'checksum': u'sha1$b9214658cc453331b62c2282b772a5c063dbd284'}}
+            'output': {
+                'location': loc,
+                'basename': 'output.txt',
+                'size': 1111,
+                'class': 'File',
+                'checksum': 'sha1$b9214658cc453331b62c2282b772a5c063dbd284'}}
 
     @staticmethod
     def _expected_download_output(outDir):
         # Having unicode string literals isn't necessary for the assertion but
         # makes for a less noisy diff in case the assertion fails.
         loc = 'file://' + os.path.join(outDir, 'output.txt')
-        loc = str(loc) if not isinstance(loc, text_type) else loc  # py2/3
         return {
-            u'output': {
-                u'location': loc,
-                u'basename': u'output.txt',
-                u'size': 0,
-                u'class': u'File',
-                u'checksum': u'sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709'}}
+            'output': {
+                'location': loc,
+                'basename': 'output.txt',
+                'size': 0,
+                'class': 'File',
+                'checksum': 'sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709'}}
+
+
+@needs_cwl
+class CWLv11Test(ToilTest):
+    @classmethod
+    def setUpClass(cls):
+        """Runs anew before each test to create farm fresh temp dirs."""
+        cls.outDir = f'/tmp/toil-cwl-v1_1-test-{str(uuid.uuid4())}'
+        os.makedirs(cls.outDir)
+        cls.rootDir = cls._projectRootPath()
+        cls.cwlSpec = os.path.join(cls.rootDir, 'src/toil/test/cwl/spec_v11')
+        cls.test_yaml = os.path.join(cls.cwlSpec, 'conformance_tests.yaml')
+        # TODO: Use a commit zip in case someone decides to rewrite master's history?
+        url = 'https://github.com/common-workflow-language/cwl-v1.1.git'
+        commit = 'a22b7580c6b50e77c0a181ca59d3828dd5c69143'
+        p = subprocess.Popen(f'git clone {url} {cls.cwlSpec} && cd {cls.cwlSpec} && git checkout {commit}', shell=True)
+        p.communicate()
+
+    def tearDown(self):
+        """Clean up outputs."""
+        if os.path.exists(self.outDir):
+            shutil.rmtree(self.outDir)
+        unittest.TestCase.tearDown(self)
+
+    @slow
+    @pytest.mark.timeout(2400)
+    # Cannot work until we fix https://github.com/DataBiosphere/toil/issues/2801
+    @pytest.mark.xfail
+    def test_run_conformance_with_caching(self):
+        self.test_run_conformance(caching=True)
+
+    @slow
+    @pytest.mark.timeout(2400)
+    def test_run_conformance(self, batchSystem=None, caching=False):
+        try:
+            # TODO: we do not currently run tests: 213, 227, 228, 229, 242, 244, 246, 249
+            selected_tests = '1-212,214-226,230-241,243,245,247-248,250-253'
+            cmd = [f'cwltest',
+                   f'--tool=toil-cwl-runner',
+                   f'--test={self.test_yaml}',
+                   f'--timeout=2400',
+                   f'--basedir={self.cwlSpec}',
+                   f'-n={selected_tests}']
+            if batchSystem:
+                cmd.extend(["--batchSystem", batchSystem])
+            if caching:
+                cmd.extend(['--', '--disableCaching="False"'])
+            subprocess.check_output(cmd, cwd=self.cwlSpec, stderr=subprocess.STDOUT)
+        except subprocess.CalledProcessError as e:
+            only_unsupported = False
+            # check output -- if we failed but only have unsupported features, we're okay
+            p = re.compile(r"(?P<failures>\d+) failures, (?P<unsupported>\d+) unsupported features")
+
+            error_log = e.output.decode('utf-8')
+            for line in error_log.split('\n'):
+                m = p.search(line)
+                if m:
+                    if int(m.group("failures")) == 0 and int(m.group("unsupported")) > 0:
+                        only_unsupported = True
+                        break
+            if not only_unsupported:
+                print(error_log)
+                raise e