Repository: DataBiosphere/toil

Issue:
CWL output directories from task globs don't get their listings filled in when bypassing the file store
 If I use this CWL:
```
cwlVersion: v1.2
class: CommandLineTool
requirements:
  MultipleInputFeatureRequirement: {}
  InlineJavascriptRequirement: {}
hints:
  DockerRequirement:
    dockerPull: docker.io/ubuntu:20.04
inputs:
  - id: tmp_prefix
    type: string
outputs:
  spaceTx_converted:
    type: Directory
    outputBinding:
      glob: $("tmp/" + inputs.tmp_prefix + "/2_tx_converted/")
baseCommand: [ bash, -c ]
arguments:
 - |
   mkdir -p $("tmp/" + inputs.tmp_prefix + "/2_tx_converted/");
   touch $("tmp/" + inputs.tmp_prefix + "/2_tx_converted/test.txt");
```
And I run it with `toil-cwl-runner --bypass-file-store --logDebug src/toil/test/cwl/glob_dir.cwl --tmp_prefix 12345`, then I get an output JSON without a listing, and 2_tx_converted in the current directory is not populated with its file after the workflow ends. If I drop --bypass-file-store, or if I use cwltool, I get a Directory at the end that actually has a listing showing the file, and the file seems to actually be made.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1455)
┆Issue Number: TOIL-1455

CWL output directories from task globs don't get their listings filled in when bypassing the file store
 If I use this CWL:
```
cwlVersion: v1.2
class: CommandLineTool
requirements:
  MultipleInputFeatureRequirement: {}
  InlineJavascriptRequirement: {}
hints:
  DockerRequirement:
    dockerPull: docker.io/ubuntu:20.04
inputs:
  - id: tmp_prefix
    type: string
outputs:
  spaceTx_converted:
    type: Directory
    outputBinding:
      glob: $("tmp/" + inputs.tmp_prefix + "/2_tx_converted/")
baseCommand: [ bash, -c ]
arguments:
 - |
   mkdir -p $("tmp/" + inputs.tmp_prefix + "/2_tx_converted/");
   touch $("tmp/" + inputs.tmp_prefix + "/2_tx_converted/test.txt");
```
And I run it with `toil-cwl-runner --bypass-file-store --logDebug src/toil/test/cwl/glob_dir.cwl --tmp_prefix 12345`, then I get an output JSON without a listing, and 2_tx_converted in the current directory is not populated with its file after the workflow ends. If I drop --bypass-file-store, or if I use cwltool, I get a Directory at the end that actually has a listing showing the file, and the file seems to actually be made.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1455)
┆Issue Number: TOIL-1455



Discussion:
This seems to be happening because cwltool's executor doesn't fill in the listings of Directory outputs unless the `outdir` on the `runtime_context` is a string. If running `toil-cwl-runner` without `--outdir` and with `--bypass-file-store`, we don't set `outdir` on the root runtime context, and we also don't set it on the runtime context we show to cwltool for each job.

A workaround seems to be to use the `--outdir` option.
This seems to be happening because cwltool's executor doesn't fill in the listings of Directory outputs unless the `outdir` on the `runtime_context` is a string. If running `toil-cwl-runner` without `--outdir` and with `--bypass-file-store`, we don't set `outdir` on the root runtime context, and we also don't set it on the runtime context we show to cwltool for each job.

A workaround seems to be to use the `--outdir` option.

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -3714,6 +3714,7 @@ def main(args: Optional[List[str]] = None, stdout: TextIO = sys.stdout) -> int:
         find_default_container, options
     )
     runtime_context.workdir = workdir  # type: ignore[attr-defined]
+    runtime_context.outdir = outdir
     runtime_context.move_outputs = "leave"
     runtime_context.rm_tmpdir = False
     runtime_context.streaming_allowed = not options.disable_streaming


Test patch:
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -226,15 +226,21 @@ def test_cwl_cmdline_input(self):
         # If the workflow runs, it must have had options
         cwltoil.main(args, stdout=st)
 
-    def _tester(self, cwlfile, jobfile, expect, main_args=[], out_name="output"):
+    def _tester(self, cwlfile, jobfile, expect, main_args=[], out_name="output", output_here=False):
         from toil.cwl import cwltoil
 
         st = StringIO()
         main_args = main_args[:]
+        if not output_here:
+            # Don't just dump output in the working directory.
+            main_args.extend(
+                [
+                    "--outdir",
+                    self.outDir
+                ]
+            )
         main_args.extend(
             [
-                "--outdir",
-                self.outDir,
                 os.path.join(self.rootDir, cwlfile),
                 os.path.join(self.rootDir, jobfile),
             ]
@@ -378,6 +384,24 @@ def test_run_colon_output(self):
             out_name="result",
         )
 
+    def test_glob_dir_bypass_file_store(self):
+        try:
+            # We need to output to the current directory to make sure that
+            # works.
+            self._tester(
+                "src/toil/test/cwl/glob_dir.cwl",
+                "src/toil/test/cwl/empty.json",
+                self._expected_glob_dir_output(os.getcwd()),
+                main_args=["--bypass-file-store"],
+                output_here=True 
+            )
+        finally:
+            # Clean up anything we made in the current directory.
+            try:
+                shutil.rmtree(os.path.join(os.getcwd(), "shouldmake"))
+            except FileNotFoundError:
+                pass
+
     @needs_aws_s3
     def test_download_s3(self):
         self.download("download_s3.json", self._tester)
@@ -634,6 +658,31 @@ def _expected_download_output(outDir):
             }
         }
 
+    @staticmethod
+    def _expected_glob_dir_output(out_dir):
+        dir_loc = "file://" + os.path.join(out_dir, "shouldmake")
+        file_loc = os.path.join(dir_loc, "test.txt")
+        return {
+            "shouldmake": {
+                "location": dir_loc,
+                "basename": "shouldmake",
+                "nameroot": "shouldmake",
+                "nameext": "",
+                "class": "Directory",
+                "listing": [
+                    {
+                        "class": "File",
+                        "location": file_loc,
+                        "basename": "test.txt",
+                        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
+                        "size": 0,
+                        "nameroot": "test",
+                        "nameext": ".txt"
+                    }
+                ]
+            }
+        }
+
     @classmethod
     def _expected_load_contents_output(cls, out_dir):
         """
diff --git a/src/toil/test/cwl/glob_dir.cwl b/src/toil/test/cwl/glob_dir.cwl
new file mode 100644
--- /dev/null
+++ b/src/toil/test/cwl/glob_dir.cwl
@@ -0,0 +1,15 @@
+cwlVersion: v1.2
+class: CommandLineTool
+inputs: []
+outputs:
+  shouldmake:
+    type: Directory
+    outputBinding:
+      glob: "shouldmake"
+baseCommand: [ bash, -c ]
+arguments:
+ - |
+   mkdir -p shouldmake;
+   touch shouldmake/test.txt;
+
+