Repository: DataBiosphere/toil

Issue:
toil-wdl-runner fails when --outputDirectory does not already exist
Since https://github.com/DataBiosphere/toil/pull/4792, we now fail if the output directory doesn't exist, because we try to make directories in it with a plain `os.mkdir()` to put files in.

I ran:
```
toil-wdl-runner \
        --jobStore /data/tmp/anovak/hprc_assembly_NA20905/jobstore \
        --stats \
        --clean=never \
        --batchSystem single_machine \
        --maxCores 4 \
        /private/home/juklucas/github/hpp_production_workflows/QC/wdl/tasks/produce_fai.wdl \
        /private/groups/hprc/assembly/batch4/hifiasm_input_jsons/NA20905_test.json \
        --outputDirectory /private/groups/patenlab/anovak/trash/produce_fai_outputs \
        --outputFile /private/groups/patenlab/anovak/trash/NA20905_produce_fai_outputs.json \
        --runLocalJobsOnWorkers \
        --retryCount 1 \
        --disableProgress
```

And I got:
```
[2024-03-11T07:51:46-0700] [MainThread] [I] [toil.wdl.wdltoil] Imported file:///private/groups/hprc/assembly/batch3/HG00621/analysis/assembly/HG00621.mat.fa.gz
[2024-03-11T07:51:46-0700] [MainThread] [I] [toil.wdl.wdltoil] Imported file:///private/groups/hprc/assembly/batch3/HG00621/analysis/assembly/HG00621.mat.fa.gz
[2024-03-11T07:51:47-0700] [MainThread] [I] [toil] Running Toil version 5.10.0a1-4eb18fd2fc44029a8a833fd2bdfbeacb3209c1a9 on host mustard.
[2024-03-11T07:51:48-0700] [MainThread] [I] [toil.leader] 0 jobs are running, 0 jobs are issued and waiting to run
[2024-03-11T07:51:48-0700] [MainThread] [I] [toil.leader] Issued job 'WDLTaskJob' runProduceFai.produceFai.command kind-WDLTaskJob/instance-lmpbbn6k v1 with job batch system ID: 2 and disk: 29.8 Gi, memory: 3.7 Gi, cores: 2.0, accelerators: [], preemptible: False
[2024-03-11T07:53:15-0700] [Thread-3 (statsAndLoggingAggregator)] [I] [toil.statsAndLogging] runProduceFai.produceFai.stderr follows:
=========>
	++ basename /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/HG00621.mat.fa.gz
	+ FILENAME=HG00621.mat.fa.gz
	+ PREFIX=HG00621.mat.fa
	+ gunzip -c /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/HG00621.mat.fa.gz
	+ samtools faidx HG00621.mat.fa
<=========
[2024-03-11T07:53:20-0700] [MainThread] [I] [toil.leader] Finished toil run successfully.
[2024-03-11T07:53:20-0700] [MainThread] [C] [toil.wdl.wdltoil] Could not run workflow
[Errno 2] No such file or directory: '/private/groups/patenlab/anovak/trash/produce_fai_outputs/1baf98b6-8452-46aa-b45f-d79a9c79ee3b'
```

The fancy `wdl_error_reporter` reports `FileNotFoundError` oddly for directories we can't make; it seems to be reporting the directory we are trying to make and not the directory that doesn't exist.

┆Issue is synchronized with this [Jira Story](https://ucsc-cgl.atlassian.net/browse/TOIL-1517)
┆Issue Number: TOIL-1517



Discussion:

Resolving patch:
diff --git a/src/toil/wdl/wdltoil.py b/src/toil/wdl/wdltoil.py
--- a/src/toil/wdl/wdltoil.py
+++ b/src/toil/wdl/wdltoil.py
@@ -558,11 +558,22 @@ def devirtualze_to(filename: str, dest_dir: str, file_source: Union[AbstractFile
         """
         Download or export a WDL virtualized filename/URL to the given directory.
 
-        Makes sure sibling files stay siblings and files with the same name don't clobber each other. Called from within this class for tasks, and statically at the end of the workflow for outputs.
+        The destination directory must already exist.
 
-        Returns the local path to the file.
+        Makes sure sibling files stay siblings and files with the same name
+        don't clobber each other. Called from within this class for tasks, and
+        statically at the end of the workflow for outputs.
+
+        Returns the local path to the file. If it already had a local path
+        elsewhere, it might not actually be put in dest_dir.
         """
 
+        if not os.path.isdir(dest_dir):
+            # os.mkdir fails saying the directory *being made* caused a
+            # FileNotFoundError. So check the dest_dir before trying to make
+            # directories under it.
+            raise RuntimeError(f"Cannot devirtualize {filename} into nonexistent directory {dest_dir}")
+
         # TODO: Support people doing path operations (join, split, get parent directory) on the virtualized filenames.
         # TODO: For task inputs, we are supposed to make sure to put things in the same directory if they came from the same directory. See <https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md#task-input-localization>
         if is_url(filename):
@@ -2899,6 +2910,9 @@ def devirtualize_output(filename: str) -> str:
             'devirtualize' a file using the "toil" object instead of a filestore.
             Returns its local path.
             """
+            # Make sure the output directory exists if we have output files
+            # that might need to use it.
+            os.makedirs(output_directory, exist_ok=True)
             return ToilWDLStdLibBase.devirtualze_to(filename, output_directory, toil, execution_dir) 
 
         # Make all the files local files


Test patch:
diff --git a/src/toil/test/wdl/wdltoil_test.py b/src/toil/test/wdl/wdltoil_test.py
--- a/src/toil/test/wdl/wdltoil_test.py
+++ b/src/toil/test/wdl/wdltoil_test.py
@@ -116,6 +116,14 @@ def test_MD5sum(self):
         assert os.path.exists(result['ga4ghMd5.value'])
         assert os.path.basename(result['ga4ghMd5.value']) == 'md5sum.txt'
 
+    def test_missing_output_directory(self):
+        """
+        Test if Toil can run a WDL workflow into a new directory.
+        """
+        wdl = os.path.abspath('src/toil/test/wdl/md5sum/md5sum.1.0.wdl')
+        json_file = os.path.abspath('src/toil/test/wdl/md5sum/md5sum.json')
+        subprocess.check_call(self.base_command + [wdl, json_file, '-o', os.path.join(self.output_dir, "does", "not", "exist"), '--logDebug', '--retryCount=0'])
+
     @needs_singularity_or_docker
     def test_miniwdl_self_test(self, extra_args: Optional[List[str]] = None) -> None:
         """Test if the MiniWDL self test runs and produces the expected output."""