Repository: DataBiosphere/toil

Issue:
Toil tries to move directories by copy instead of through the FileStore in CWL conformance tests
As discovered in #2770, this CWL conformance test will pass with job chaining enabled on Toil master:

```
wget https://github.com/common-workflow-language/common-workflow-language/archive/a062055fddcc7d7d9dbc53d28288e3ccb9a800d8.zip
unzip a062055fddcc7d7d9dbc53d28288e3ccb9a800d8.zip
mv common-workflow-language-a062055fddcc7d7d9dbc53d28288e3ccb9a800d8/ test/
toil-cwl-runner --disableCaching=True --jobStore ./chain --logDebug --realTimeLogging --outdir=/tmp/tmpb5MyuT2 test/v1.0/v1.0/iwdr_with_nested_dirs.cwl test/v1.0/v1.0/empty.json
```

But it will fail when chaining is disabled:

```
toil-cwl-runner --disableCaching=True --disableChaining --jobStore ./nochain --logDebug --realTimeLogging --outdir=/tmp/tmpb5MyuT2 test/v1.0/v1.0/iwdr_with_nested_dirs.cwl test/v1.0/v1.0/empty.json
```

It is failing because, when setting up the second job cwltool ends up trying to recursively copy a directory form the first job's local temp directory. With chaining on this directory happens to still exist, but with chaining off it will have already been cleaned up. In general, two jobs might run on completely different machines, so cwltool needs to be shipping the directory tree around via Toil's FileStore.

The exceptions look like this:

```
Traceback (most recent call last):
  File "/Users/anovak/Documents/Workspace/toil/venv35/lib/python3.5/site-packages/cwltool/executors.py", line 182, in run_jobs
    job.run(runtime_context)
  File "/Users/anovak/Documents/Workspace/toil/venv35/lib/python3.5/site-packages/cwltool/job.py", line 445, in run
    symlink=True, secret_store=runtimeContext.secret_store)
  File "/Users/anovak/Documents/Workspace/toil/venv35/lib/python3.5/site-packages/cwltool/process.py", line 253, in stage_files
    shutil.copytree(entry.resolved, entry.target)
  File "/Users/anovak/Documents/Workspace/toil/venv35/lib/python3.5/shutil.py", line 309, in copytree
    names = os.listdir(src)
FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/ws/t7b9wm2j2b7fy4_xyq0n5vh00000gn/T/toil-dfba1124-76bd-45d9-ada7-12e2e8d2020f-106618767760070/tmp5tgq0qpe/3f3d9b8e-b1cd-4ff7-ac5a-2dd0a8c05afc/tcc4hil93/tmp-outnngwbct7/deeply'
```

@mr-c Is there something you can do about this in cwltool? Or is there something wrong in cwltoil that is causing cwltool to try and do these local copies when it needs to be shipping around tarballs or rebuilding the directory structure instead?

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-424)
┆Issue Number: TOIL-424



Discussion:
@adamnovak Good find!

> Is there something you can do about this in cwltool?

No, managing data movement is not the domain of the CWL reference runner, `cwltool`. Toil (or `cwltoil.py` in this case) is responsible for transportation of data, `Directory` or otherwise.  `cwltool` doesn't move data around except by soft link or the occasional copy when a writable input is needed.

> Or is there something wrong in cwltoil that is causing cwltool to try and do these local copies when it needs to be shipping around tarballs or rebuilding the directory structure instead?

Probably. We don't have code to move `Directory` as a tarball. I thought we were moving each `File` in the CWL `Directory` into a toil file store using `cwltoil.py:uploadFile()` and updating the `location` field to use the `toilfs` UR, as see at `cwltoil.py:585` but evidently we're missing something
I'm going to see if I can make progress on this today.
I think the problem is maybe that there aren't actually any files, and we don't seem to upload directories.

Here's the upload code:

https://github.com/DataBiosphere/toil/blob/1ae5547fe93c73f68e3eb7f5d3dd05aeceb413ab/src/toil/cwl/cwltoil.py#L939-L946

The relevant functions we're calling are here: https://github.com/common-workflow-language/cwltool/blob/445f6b3a214f0c48317b25f7af278ae919ce5403/cwltool/pathmapper.py#L56-L66

We visit and upload all the files, but we visit the directories and run the [undocumented `get_listing()`](https://github.com/common-workflow-language/cwltool/blob/445f6b3a214f0c48317b25f7af278ae919ce5403/cwltool/pathmapper.py#L154), which looks to just scan the directories for their contents. We never actually do anything for directories themselves, independent of their contained files.

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -82,6 +82,8 @@
 from toil.common import Config, Toil, addOptions
 from toil.version import baseVersion
 
+logger = logging.getLogger(__name__)
+
 # Define internal jobs we should avoid submitting to batch systems and logging
 CWL_INTERNAL_JOBS = ("CWLJobWrapper",
                      "CWLWorkflow",
@@ -131,7 +133,7 @@ def filter_skip_null(name: str, value: Any) -> Any:
     err_flag = [False]
     value = _filter_skip_null(value, err_flag)
     if err_flag[0]:
-        cwllogger.warning(
+        logger.warning(
             "In %s, SkipNull result found and cast to None. \n"
             "You had a conditional step that did not run, "
             "but you did not use pickValue to handle the skipped input." % name)
@@ -310,7 +312,7 @@ def pick_value(self, values: Union[List, Any]) -> Any:
             return values
 
         if not isinstance(values, list):
-            cwllogger.warning(
+            logger.warning(
                 "pickValue used but input %s is not a list." % self.name)
             return values
 
@@ -469,6 +471,10 @@ def __init__(self,
         self.stage_listing = stage_listing
         super(ToilPathMapper, self).__init__(
             referenced_files, basedir, stagedir, separateDirs=separateDirs)
+            
+    def mapper(self, src: str) -> MapperEnt:
+        result = super(ToilPathMapper, self).mapper(src)
+        return result
 
     def visit(self,
               obj: Dict[str, Any],
@@ -476,6 +482,7 @@ def visit(self,
               basedir: str,
               copy: bool = False,
               staged: bool = False) -> None:
+              
         tgt = convert_pathsep_to_unix(os.path.join(stagedir, obj["basename"]))
         if obj["location"] in self._pathmap:
             return
@@ -489,6 +496,7 @@ def visit(self,
 
             if obj["location"].startswith("file://") and not self.stage_listing:
                 staged = False
+                
             self.visitlisting(
                 obj.get("listing", []), tgt, basedir, copy=copy, staged=staged)
 
@@ -507,7 +515,7 @@ def visit(self,
                     obj,
                     "location",
                     validate.ValidationException,
-                    cwllogger.isEnabledFor(logging.DEBUG),
+                    logger.isEnabledFor(logging.DEBUG),
                 ):
                     deref = self.get_file(path) if self.get_file else ab
                     if deref.startswith("file:"):
@@ -529,6 +537,7 @@ def visit(self,
                     self._pathmap[path] = MapperEnt(
                         deref, tgt, "WritableFile" if copy else "File", staged
                     )
+            
             self.visitlisting(
                 obj.get("secondaryFiles", []),
                 stagedir,
@@ -582,6 +591,7 @@ def _abs(self, path: str) -> str:
 
         Overwrites cwltool.stdfsaccess.StdFsAccess._abs() to account for toil specific schema.
         """
+        
         # Used to fetch a path to determine if a file exists in the inherited cwltool.stdfsaccess.StdFsAccess,
         # (among other things) so this should not error on missing files.
         # See: https://github.com/common-workflow-language/cwltool/blob/beab66d649dd3ee82a013322a5e830875e8556ba/cwltool/stdfsaccess.py#L43
@@ -627,10 +637,44 @@ def write_file(writeFunc: Any, index: dict, existing: dict, file_uri: str) -> st
                 index[file_uri] = "toilfs:" + writeFunc(rp).pack()
                 existing[index[file_uri]] = file_uri
             except Exception as e:
-                cwllogger.error("Got exception '%s' while copying '%s'", e, file_uri)
+                logger.error("Got exception '%s' while copying '%s'", e, file_uri)
                 raise
         return index[file_uri]
 
+def prepareDirectoryForUpload(directory_metadata: dict,
+                              skip_broken: bool = False) -> None:
+    """
+    Prepare a Directory object to be uploaded.
+
+    Assumes listings are already filled in.
+
+    Makes sure the directory actually exists, and rewrites its location to be
+    something we can use on another machine.
+    
+    Since Files and sub-Directories are already tracked by the directory's
+    listing, we just need some sentinel path to represent the existence of a
+    directory coming from Toil and not the local filesystem.
+    """
+    if directory_metadata["location"].startswith("toilfs:") or directory_metadata["location"].startswith("_:"):
+        # Already in Toil; nothing to do
+        return
+    if not directory_metadata["location"] and directory_metadata["path"]:
+        directory_metadata["location"] = schema_salad.ref_resolver.file_uri(directory_metadata["path"])
+    if directory_metadata["location"].startswith("file://") and not os.path.isdir(directory_metadata["location"][7:]):
+        if skip_broken:
+            return
+        else:
+            raise cwltool.errors.WorkflowException(
+                "Directory is missing: %s" % directory_metadata["location"])
+                
+    # The metadata for a directory is all we need to keep around for it. It
+    # doesn't have a real location. But each directory needs a unique location
+    # or cwltool won't ship the metadata along. cwltool takes "_:" as a signal
+    # to make directories instead of copying from somewhere. So we give every
+    # directory a unique _: location and cwltool's machinery Just Works.
+    directory_metadata["location"] = "_:" + directory_metadata["location"]
+    
+    logger.debug("Sending directory at %s", directory_metadata["location"])
 
 def uploadFile(uploadfunc: Any,
                fileindex: dict,
@@ -641,6 +685,7 @@ def uploadFile(uploadfunc: Any,
     Update a file object so that the location is a reference to the toil file
     store, writing it to the file store if necessary.
     """
+    
     if file_metadata["location"].startswith("toilfs:") or file_metadata["location"].startswith("_:"):
         return
     if file_metadata["location"] in fileindex:
@@ -656,6 +701,8 @@ def uploadFile(uploadfunc: Any,
                 "File is missing: %s" % file_metadata["location"])
     file_metadata["location"] = write_file(
         uploadfunc, fileindex, existing, file_metadata["location"])
+        
+    logger.debug("Sending file at: %s", file_metadata["location"])
 
 
 def writeGlobalFileWrapper(file_store: AbstractFileStore, fileuri: str) -> str:
@@ -876,6 +923,14 @@ def populate_env_vars(self, cwljob: dict) -> dict:
         return required_env_vars
 
     def run(self, file_store: AbstractFileStore) -> Any:
+        
+        # Adjust cwltool's logging to conform to Toil's settings.
+        # We need to make sure this happens in every worker process before we
+        # do CWL things.
+        cwllogger.removeHandler(defaultStreamHandler)
+        cwllogger.setLevel(logger.getEffectiveLevel())
+        
+        
         cwljob = resolve_dict_w_promises(self.cwljob)
         fill_in_defaults(
             self.step_inputs,
@@ -939,6 +994,8 @@ def run(self, file_store: AbstractFileStore) -> Any:
         adjustDirObjs(output, functools.partial(
             get_listing, cwltool.stdfsaccess.StdFsAccess(outdir),
             recursive=True))
+            
+        adjustDirObjs(output, prepareDirectoryForUpload)
 
         # write the outputs into the jobstore
         adjustFileObjs(output, functools.partial(
@@ -1357,6 +1414,7 @@ def remove_unprocessed_secondary_files(unfiltered_secondary_files: dict) -> list
 
 def main(args: Union[List[str]] = None, stdout: TextIO = sys.stdout) -> int:
     """Main method for toil-cwl-runner."""
+    # Remove cwltool logger's stream handler so it uses Toil's
     cwllogger.removeHandler(defaultStreamHandler)
     config = Config()
     config.cwl = True
@@ -1526,6 +1584,8 @@ def main(args: Union[List[str]] = None, stdout: TextIO = sys.stdout) -> int:
     use_container = not options.no_container
 
     if options.logLevel:
+        # Make sure cwltool uses Toil's log level.
+        # Applies only on the leader.
         cwllogger.setLevel(options.logLevel)
 
     outdir = os.path.abspath(options.outdir)


Test patch:
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -171,8 +171,6 @@ def path_with_bogus_rev():
 
     @slow
     @pytest.mark.timeout(2400)
-    # Cannot work until we fix https://github.com/DataBiosphere/toil/issues/2801
-    @pytest.mark.xfail
     def test_run_conformance_with_caching(self):
         self.test_run_conformance(caching=True)
 
@@ -206,48 +204,36 @@ def test_run_conformance(self, batchSystem=None, caching=False):
     @slow
     @needs_lsf
     @unittest.skip
-    # Cannot work until we fix https://github.com/DataBiosphere/toil/issues/2801
-    @pytest.mark.xfail
     def test_lsf_cwl_conformance_with_caching(self):
         return self.test_run_conformance(batchSystem="LSF", caching=True)
 
     @slow
     @needs_slurm
     @unittest.skip
-    # Cannot work until we fix https://github.com/DataBiosphere/toil/issues/2801
-    @pytest.mark.xfail
     def test_slurm_cwl_conformance_with_caching(self):
         return self.test_run_conformance(batchSystem="Slurm", caching=True)
 
     @slow
     @needs_torque
     @unittest.skip
-    # Cannot work until we fix https://github.com/DataBiosphere/toil/issues/2801
-    @pytest.mark.xfail
     def test_torque_cwl_conformance_with_caching(self):
         return self.test_run_conformance(batchSystem="Torque", caching=True)
 
     @slow
     @needs_gridengine
     @unittest.skip
-    # Cannot work until we fix https://github.com/DataBiosphere/toil/issues/2801
-    @pytest.mark.xfail
     def test_gridengine_cwl_conformance_with_caching(self):
         return self.test_run_conformance(batchSystem="gridEngine", caching=True)
 
     @slow
     @needs_mesos
     @unittest.skip
-    # Cannot work until we fix https://github.com/DataBiosphere/toil/issues/2801
-    @pytest.mark.xfail
     def test_mesos_cwl_conformance_with_caching(self):
         return self.test_run_conformance(batchSystem="mesos", caching=True)
 
     @slow
     @needs_parasol
     @unittest.skip
-    # Cannot work until we fix https://github.com/DataBiosphere/toil/issues/2801
-    @pytest.mark.xfail
     def test_parasol_cwl_conformance_with_caching(self):
         return self.test_run_conformance(batchSystem="parasol", caching=True)
 
@@ -360,8 +346,8 @@ def test_run_conformance_with_caching(self):
     @pytest.mark.timeout(2400)
     def test_run_conformance(self, batchSystem=None, caching=False):
         try:
-            # TODO: we do not currently pass tests: 213, 236, 242, 243, 244, 245, 246, 249
-            selected_tests = '1-212,214-235,237-241,247-248,250-253'
+            # TODO: we do not currently pass tests: 213, 236, 242, 243, 244, 245, 246
+            selected_tests = '1-212,214-235,237-241,247-253'
             cmd = [f'cwltest',
                    f'--tool=toil-cwl-runner',
                    f'--test={self.test_yaml}',
@@ -422,8 +408,8 @@ def test_run_conformance_with_caching(self):
     @pytest.mark.timeout(2400)
     def test_run_conformance(self, batchSystem=None, caching=False):
         try:
-            # TODO: we do not currently pass tests: 213, 236, 242, 243, 244, 245, 246, 249
-            selected_tests = '1-212,214-235,237-241,247-248,250-276'
+            # TODO: we do not currently pass tests: 213, 236, 242, 243, 244, 245, 246
+            selected_tests = '1-212,214-235,237-241,247-276'
             cmd = [f'cwltest',
                    f'--tool=toil-cwl-runner',
                    f'--test={self.test_yaml}',