Repository: DataBiosphere/toil

Issue:
"File staging conflict" error is suppressed when using a container; files get overwritten
This is the same issue as described here; https://github.com/common-workflow-language/cwltool/issues/1403

Using the same CWL files as in that Issue with `cwltool`, when you run Toil with a CWL that uses `InitialWorkDirRequirement` like this;

```
  InitialWorkDirRequirement:
    listing:
      - entryname: some_dir
        writable: true
        entry: "$({class: 'Directory', listing: inputs.input_files})"
```

 you get a File staging conflict error:

```
$ toil-cwl-runner workflow.cwl input.json
...
...
[2021-02-08T11:58:02-0500] [MainThread] [I] [toil.worker] ---TOIL WORKER OUTPUT LOG---
[2021-02-08T11:58:02-0500] [MainThread] [I] [toil] Running Toil version 5.2.0-047d0c4f2949c576c80e452a0807c5be6355c63d on host server.
[2021-02-08T11:58:02-0500] [MainThread] [I] [toil.worker] Working on job 'CWLJob' bash run.sh kind-CWLJob/instance-vjb1bunf
[2021-02-08T11:58:02-0500] [MainThread] [I] [toil.worker] Loaded body Job('CWLJob' bash run.sh kind-CWLJob/instance-vjb1bunf) from description 'CWLJob' bash run.sh kind-CWLJob/instance-vjb1bunf
[2021-02-08T11:58:02-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] Failed job accessed files:
[2021-02-08T11:58:02-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CWLJob/instance-unfjem45/file-cd83ea62575a44098a4fdb15a6dd6790/output.txt' to path '/tmp/node-efa42d50-baaa-41db-b0b9-d26bc000e945-ae225bf2-14b6-4c6a-9f5b-1dfe49b07a51/tmpgz22ltyt/67f17b0c-d724-45b0-8185-498cb56a6ced/tmpm3b65eu7.tmp'
[2021-02-08T11:58:02-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CWLJob/instance-7gxq1o32/file-e570db2d16284f1fb9359a454f04ba2a/output.txt' to path '/tmp/node-efa42d50-baaa-41db-b0b9-d26bc000e945-ae225bf2-14b6-4c6a-9f5b-1dfe49b07a51/tmpgz22ltyt/67f17b0c-d724-45b0-8185-498cb56a6ced/tmpeef58r6u.tmp'
[2021-02-08T11:58:02-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CWLJob/instance-unfjem45/file-cd83ea62575a44098a4fdb15a6dd6790/output.txt' to path '/tmp/node-efa42d50-baaa-41db-b0b9-d26bc000e945-ae225bf2-14b6-4c6a-9f5b-1dfe49b07a51/tmpgz22ltyt/67f17b0c-d724-45b0-8185-498cb56a6ced/tmp266g1809.tmp'
[2021-02-08T11:58:02-0500] [MainThread] [W] [toil.fileStores.abstractFileStore] Downloaded file 'files/for-job/kind-CWLJob/instance-7gxq1o32/file-e570db2d16284f1fb9359a454f04ba2a/output.txt' to path '/tmp/node-efa42d50-baaa-41db-b0b9-d26bc000e945-ae225bf2-14b6-4c6a-9f5b-1dfe49b07a51/tmpgz22ltyt/67f17b0c-d724-45b0-8185-498cb56a6ced/tmpnjzcsd2f.tmp'
Traceback (most recent call last):
  File "/home/conda/lib/python3.7/site-packages/toil/worker.py", line 394, in workerScript
    job._runner(jobGraph=None, jobStore=jobStore, fileStore=fileStore, defer=defer)
  File "/home/conda/lib/python3.7/site-packages/toil/job.py", line 2359, in _runner
    returnValues = self._run(jobGraph=None, fileStore=fileStore)
  File "/home/conda/lib/python3.7/site-packages/toil/job.py", line 2280, in _run
    return self.run(fileStore)
  File "/home/conda/lib/python3.7/site-packages/toil/cwl/cwltoil.py", line 1222, in run
    logger=cwllogger,
  File "/home/conda/lib/python3.7/site-packages/cwltool/executors.py", line 150, in execute
    self.run_jobs(process, job_order_object, logger, runtime_context)
  File "/home/conda/lib/python3.7/site-packages/cwltool/executors.py", line 257, in run_jobs
    job.run(runtime_context)
  File "/home/conda/lib/python3.7/site-packages/cwltool/job.py", line 566, in run
    secret_store=runtimeContext.secret_store,
  File "/home/conda/lib/python3.7/site-packages/cwltool/process.py", line 287, in stage_files
    % (targets[entry.target].resolved, entry.resolved, entry.target)
cwltool.errors.WorkflowException: File staging conflict, trying to stage both /tmp/node-efa42d50-baaa-41db-b0b9-d26bc000e945-ae225bf2-14b6-4c6a-9f5b-1dfe49b07a51/tmpgz22ltyt/67f17b0c-d724-45b0-8185-498cb56a6ced/tmpm3b65eu7.tmp and /tmp/node-efa42d50-baaa-41db-b0b9-d26bc000e945-ae225bf2-14b6-4c6a-9f5b-1dfe49b07a51/tmpgz22ltyt/67f17b0c-d724-45b0-8185-498cb56a6ced/tmpeef58r6u.tmp to the same target /tmp/node-efa42d50-baaa-41db-b0b9-d26bc000e945-ae225bf2-14b6-4c6a-9f5b-1dfe49b07a51/tmpgz22ltyt/67f17b0c-d724-45b0-8185-498cb56a6ced/tjxzkadyk/tmp-oute3hzotys/some_dir/output.txt
[2021-02-08T11:58:02-0500] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host server
```

However if you add a container requirement;


```
  DockerRequirement:
    dockerPull: ubuntu:latest
```

The File staging conflict does not occur, and the files overwrite each other upon being staged in the directory

```
$ toil-cwl-runner --singularity workflow.cwl input.json
...

[2021-02-08T12:05:28-0500] [MainThread] [I] [toil.leader] Finished toil run successfully.

Workflow Progress 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 (0 failures) [00:50<00:00, 0.20 jobs/s]
{
    "output_file": {
        "location": "file:///home/_test_cwl_initialDir/output.txt",
        "basename": "output.txt",
        "nameroot": "output",
        "nameext": ".txt",
        "class": "File",
        "checksum": "sha1$24c5362ecd1bfafba85f185f28b12c121d03ee12",
        "size": 8
    }
}[2021-02-08T12:05:28-0500] [MainThread] [I] [toil.common] Successfully deleted the job store: FileJobStore(/tmp/tmp3iv1zy24)

$ cat output.txt
Sample2

```

The File staging conflict error needs to be raised in this case so that you can make sure you are not silently losing files in your workflow.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-793)
┆Issue Number: TOIL-793



Discussion:
@stevekm Thank you for raising this issue.  A cwltool fix should fix things in Toil.  We'll try to stay apprised of developments there and update accordingly.
It looks like this problem arises when the user's CWL code constructs a CWL `Directory` object with a `listing` that is not actually acceptable as a `Directory`'s listing, because it contains multiple entries with the same name and thus can't ever be physically realized on disk.

This *doesn't* happen just when you take two input `File` objects and try and e.g. pass them to a command line tool, and they happen to have the same basename. That works regardless of whether Singularity is in use, right?

`cwltool` can sometimes detect a broken `Directory` at the file staging step, and instead of showing you an arbitrary one of the files, fails the whole workflow. But when using Singularity, it instead just shows you an arbitrary one of the files. Toil gets basically the same behavior as `cwltool` has by calling into it.

I feel like really this should be detected at a different point. `InitialWorkDirRequirement` should have its own listing pre-checked before we attempt to stage it, and if the listing tree is self-contradictory we should refuse to continue. without trying to actually stage anything.

I'm also not sure this is a Toil fix, though.
I've tested this on Toil commit 5280227633703372ce06923f2164bcf1aad65a0d and without the DockerRequirement we get the file staging conflict. With the DockerRequirement we don't, but I also see both `Sample1` and `Sample2` in the output file:

```
{
    "output_file": {
        "location": "file:///Users/anovak/workspace/toil/output.txt",
        "basename": "output.txt",
        "nameroot": "output",
        "nameext": ".txt",
        "class": "File",
        "checksum": "sha1$4c092e98f5dc520a853c1f2f1db4e1b14fe2955d",
        "size": 16
    }
}[2023-04-20T16:23:00-0400] [MainThread] [I] [toil.common] Successfully deleted the job store: FileJobStore(/var/folders/0n/4y413_9s7y70lmm3yhtt3b8m0000gq/T/tmp4oyctazw)
(venv) [anovak@swords toil]% cat output.txt
Sample1
Sample2
```

So it could be that CWLTool changed something here.

Do we really want to force the file staging conflict error if cwltool is working around it? Probably for portability...
It could also be that I am testing with Docker, and we run into the problem with loss of files only on Singularity.

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -60,6 +60,7 @@
 import cwltool.builder
 import cwltool.command_line_tool
 import cwltool.context
+import cwltool.job
 import cwltool.load_tool
 import cwltool.main
 import cwltool.provenance
@@ -84,6 +85,7 @@
 )
 from cwltool.stdfsaccess import StdFsAccess, abspath
 from cwltool.utils import (
+    DirectoryType,
     CWLObjectType,
     CWLOutputType,
     adjustDirObjs,
@@ -201,6 +203,34 @@ def _filter_skip_null(value: Any, err_flag: List[bool]) -> Any:
         return {k: _filter_skip_null(v, err_flag) for k, v in value.items()}
     return value
 
+def ensure_no_collisions(directory: DirectoryType, dir_description: Optional[str] = None) -> None:
+    """
+    Make sure no items in the given CWL Directory have the same name.
+
+    If any do, raise a WorkflowException about a "File staging conflict". 
+    
+    Does not recurse into subdirectories.
+    """
+
+    if dir_description is None:
+        # Work out how to describe the directory we are working on.
+        dir_description = f"the directory \"{directory.get('basename')}\""
+
+    seen_names = set()
+
+    for child in directory.get("listing", []):
+        if "basename" in child:
+            # For each child that actually has a path to go at in its parent
+            wanted_name = cast(str, child["basename"])
+            if wanted_name in seen_names:
+                # We used this name already so bail out
+                raise cwl_utils.errors.WorkflowException(
+                    f"File staging conflict: Duplicate entries for \"{wanted_name}\""
+                    f" prevent actually creating {dir_description}"
+                )
+            seen_names.add(wanted_name)
+        
+        
 
 class Conditional:
     """
@@ -730,6 +760,10 @@ def visit(
 
             logger.debug("ToilPathMapper visiting directory %s", location)
 
+            # We want to check the directory to make sure it is not
+            # self-contradictory in its immediate children and their names.
+            ensure_no_collisions(cast(DirectoryType, obj))
+
             # We may need to copy this directory even if we don't copy things inside it.
             copy_here = False
 
@@ -968,9 +1002,23 @@ def __str__(self) -> str:
 class ToilCommandLineTool(ToilTool, cwltool.command_line_tool.CommandLineTool):
     """Subclass the cwltool command line tool to provide the custom ToilPathMapper."""
 
-    pass
-
+    def _initialworkdir(self, j: cwltool.job.JobBase, builder: cwltool.builder.Builder) -> None:
+        """
+        Hook the InitialWorkDirRequirement setup to make sure that there are no
+        name conflicts at the top level of the work directory.
+        """
 
+        super()._initialworkdir(j, builder)
+        
+        # The initial work dir listing is now in j.generatefiles["listing"]
+        # Also j.generatrfiles is a CWL Directory.
+        # So check the initial working directory.
+        logger.info('Initial work dir: %s', j.generatefiles)
+        ensure_no_collisions(
+            j.generatefiles,
+            "the job's working directory as specified by the InitialWorkDirRequirement"
+        )
+        
 class ToilExpressionTool(ToilTool, cwltool.command_line_tool.ExpressionTool):
     """Subclass the cwltool expression tool to provide the custom ToilPathMapper."""
 
@@ -1587,11 +1635,16 @@ def visit_file_or_directory_down(
         Ensures that any child File or Directory objects from the original
         listing remain as child objects, so that they will be hit by the
         recursion.
+
+        Ensures that no directory listings have name collisions.
         """
         if rec.get("class", None) == "File":
             # Nothing to do!
             return None
         elif rec.get("class", None) == "Directory":
+            # Check the original listing for collisions
+            ensure_no_collisions(cast(DirectoryType, rec))
+
             # Pull out the old listing, if any
             old_listing = cast(Optional[List[CWLObjectType]], rec.get("listing", None))
 
@@ -1607,6 +1660,9 @@ def visit_file_or_directory_down(
                 # Otherwise, we preserve the existing listing (including all
                 # its original File objects that we need to process)
 
+                # Check the new listing for collisions
+                ensure_no_collisions(cast(DirectoryType, rec))
+
             return old_listing
         return None
 


Test patch:
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -1121,6 +1121,52 @@ def test_filename_conflict_resolution(tmp_path: Path):
     assert b"Finished toil run successfully" in stderr
     assert p.returncode == 0
 
+@needs_cwl
+@needs_docker
+@pytest.mark.cwl_small_log_dir
+def test_filename_conflict_detection(tmp_path: Path):
+    """
+    Make sure we don't just stage files over each other when using a container.
+    """
+    out_dir = tmp_path / "cwl-out-dir"
+    toil = "toil-cwl-runner"
+    options = [
+        f"--outdir={out_dir}",
+        "--clean=always",
+    ]
+    cwl = os.path.join(
+        os.path.dirname(__file__), "test_filename_conflict_detection.cwl"
+    )
+    cmd = [toil] + options + [cwl]
+    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    stdout, stderr = p.communicate()
+    assert b"File staging conflict" in stderr
+    assert p.returncode != 0
+
+@needs_cwl
+@needs_docker
+@pytest.mark.cwl_small_log_dir
+def test_filename_conflict_detection_at_root(tmp_path: Path):
+    """
+    Make sure we don't just stage files over each other.
+
+    Specifically, when using a container and the files are at the root of the work dir.
+    """
+    out_dir = tmp_path / "cwl-out-dir"
+    toil = "toil-cwl-runner"
+    options = [
+        f"--outdir={out_dir}",
+        "--clean=always",
+    ]
+    cwl = os.path.join(
+        os.path.dirname(__file__), "test_filename_conflict_detection_at_root.cwl"
+    )
+    cmd = [toil] + options + [cwl]
+    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    stdout, stderr = p.communicate()
+    assert b"File staging conflict" in stderr
+    assert p.returncode != 0
+
 
 @needs_cwl
 @pytest.mark.cwl_small
diff --git a/src/toil/test/cwl/test_filename_conflict_detection.cwl b/src/toil/test/cwl/test_filename_conflict_detection.cwl
new file mode 100644
--- /dev/null
+++ b/src/toil/test/cwl/test_filename_conflict_detection.cwl
@@ -0,0 +1,36 @@
+cwlVersion: v1.2
+class: CommandLineTool
+baseCommand:
+  - ls
+
+requirements:
+  DockerRequirement:
+    dockerPull: ubuntu:20.04
+  InlineJavascriptRequirement: {}
+  InitialWorkDirRequirement:
+    listing:
+      # https://github.com/DataBiosphere/toil/issues/3449 is specifically about
+      # a directory with a conflict in it so test that.
+      - basename: subdir
+        class: Directory
+        listing:
+          - basename: innderdupe.txt
+            class: File
+            contents: |
+              Stuff
+          - basename: innderdupe.txt
+            class: File
+            contents: |
+              Other stuff
+
+inputs: {}
+
+stdout: stdout.txt
+
+outputs:
+  standard_out:
+    type: stdout
+
+
+
+
diff --git a/src/toil/test/cwl/test_filename_conflict_detection_at_root.cwl b/src/toil/test/cwl/test_filename_conflict_detection_at_root.cwl
new file mode 100644
--- /dev/null
+++ b/src/toil/test/cwl/test_filename_conflict_detection_at_root.cwl
@@ -0,0 +1,35 @@
+cwlVersion: v1.2
+class: CommandLineTool
+baseCommand:
+  - ls
+
+requirements:
+  DockerRequirement:
+    dockerPull: ubuntu:20.04
+  InlineJavascriptRequirement: {}
+  InitialWorkDirRequirement:
+    listing:
+      - entryname: duplicate.txt
+        writable: true # Unless writable we catch the problem due to overwriting a read-only file
+        entry: |
+          This is one file
+      - entryname: unique.txt
+        writable: true
+        entry: |
+          This is a different file
+      - entryname: duplicate.txt
+        writable: true
+        entry: |
+          This is another file pretending to be the first one
+
+inputs: {}
+
+stdout: stdout.txt
+
+outputs:
+  standard_out:
+    type: stdout
+
+
+
+
diff --git a/src/toil/test/cwl/test_filename_conflict_resolution.cwl b/src/toil/test/cwl/test_filename_conflict_resolution.cwl
--- a/src/toil/test/cwl/test_filename_conflict_resolution.cwl
+++ b/src/toil/test/cwl/test_filename_conflict_resolution.cwl
@@ -16,7 +16,7 @@ requirements:
           for i in range(2):
             msout = f"{root}_{i}"
             print(f"Copying: {msin} -> {msout}")
-            subprocess.check_call(['cp', '-r', '-L', '--no-preserve=mode', msin, msout])
+            subprocess.check_call(['cp', '-R', '-L', msin, msout])
 
 inputs:
   - id: msin