Repository: DataBiosphere/toil

Issue:
nested directory getting lost between steps
hi! I'm unsuccessfully trying to run our CWL pipeline with toil. It looks like somehow intermediate date products get lost. The pipeline runs fine with the CWL reference runner. My logfile is filled with errors like:

```
2017-11-26 11:19:34,177 - toil.leader - WARNING - W/2/jobLmwkno    WorkflowException: [Errno 2] No such file or directory: u'/vault-tina/gijs/prefactor-cwl/runs/run_2017-11-26-11-18-25/results/out_tmpdir2RF79v/calibrated.MS/OBSERVATION/table.lock'
```

The intermediate folder seems to be empty. My guess this is probably related to our data products being nested directory structures and maybe in combination with docker.

To reproduce this issue you could:
* clone our repository https://github.com/EOSC-LOFAR/prefactor-cwl/tree/b5c080d10c659232b46dd9dcb2d9e4c19cc9febc
* run `make toil` in the checkout

Requirements:
 * git-lfs
 * Docker
 * virtualenv






Discussion:

[output.txt](https://github.com/BD2KGenomics/toil/files/1503827/output.txt)

I posted an issue on your repo, but in attempting to use either `make run` or `make toil` I get the same error (running on Ubuntu 16.04):

`lifeisaboutfishtacos@TacoTruck:~/Desktop/LOFAR/prefactor-cwl$ make run`
`cd data && tar Jxvf L570745_uv_first10.MS.tar.xz`
`xz: (stdin): File format not recognized`
`tar: Child returned status 1`
`tar: Error is not recoverable: exiting now`
`Makefile:19: recipe for target 'data/L570745_SB000_uv_first10.MS/' failed`
`make: *** [data/L570745_SB000_uv_first10.MS/] Error 2`
this is probably because you don't have git LFS installed, which should be available during git checkout to checkout the binary blob.
Sorry, I amended the error under your repo but forgot to reproduce it here.  I have git lfs, the issue is that your repo broke its data quota unfortunately.  Do you have a different source?

`Downloading data/L570745_uv_first10.MS.tar.xz (387 MB)`
`Error downloading object: data/L570745_uv_first10.MS.tar.xz (d6acef4): Smudge error: Error downloading data/L570745_uv_first10.MS.tar.xz`
`(d6acef40bcf795c5ae94232654414fece4a20eb75494577ea630fc663673b3c1): batch response: This repository is over its data quota.`
`Purchase more data packs to restore access.`
Hey @DailyDreaming, here is a complete capture of the run with `--logDebug` 
https://drive.google.com/file/d/1Gv1TezB6YtpYJIwZVNqX3AMpjeSb3vEa/view?usp=sharing


Just uncompress that and put it in `runs` and you should be able to restart Toil with the provided jobStore
Here are the raw files from `data/`
[L570745_uv_first10.MS.tar.xz](https://drive.google.com/a/ucdavis.edu/file/d/1h-cDhNiEyjBz87vITZ2BolQbNyeboBGI/view?usp=sharing)
[L591513_SB000_uv_delta_t_4.MS.tar.xz](https://drive.google.com/a/ucdavis.edu/file/d/1xvXPpCUW8FqlP2tug82lWpKCP5iJRFrO/view?usp=sharing)
i've updated the Makefile and repo to pull the data from a FTP.
i'm running into the same issue with a different pipeline:

https://github.com/gijzelaerr/spiel

again, it is where we pass around a nested directory structure between steps.

Here's a simplified minimal test case:

``` yaml
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: Workflow

inputs: []
outputs:
  deepest_dir:
    type: File
    outputSource: second/ya

steps:
  first:
    run:
      class: CommandLineTool
      baseCommand: [ mkdir, -p, deeply/nested/dir/structure ]
      inputs: []
      outputs:
        deep_dir:
          type: Directory
          outputBinding: { glob: deeply }
    in: {}
    out: [ deep_dir ]

  second:
    run:
      class: CommandLineTool
      baseCommand: [ touch, deeply/nested/dir/structure/ya ]
      requirements:
        InitialWorkDirRequirement:
          listing: [ $(inputs.dir) ]
      inputs:
        dir: Directory
      outputs:
        ya:
          type: File
          outputBinding: { glob: deeply/nested/dir/structure/ya }
 
    in: { dir: first/deep_dir }
    out: [ ya ]
```

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -455,12 +455,6 @@ def run(self, fileStore):
                                                 cwltool.stdfsaccess.StdFsAccess(outdir),
                                                 recursive=True))
 
-        def make_dir_literal(obj):
-            if "location" in obj and obj["location"].startswith("file:"):
-                obj["location"] = "_:" + str(uuid.uuid4())
-
-        adjustDirObjs(output, make_dir_literal)
-
         adjustFileObjs(output, functools.partial(uploadFile,
                                                  functools.partial(writeGlobalFileWrapper, fileStore),
                                                  index, existing))


Test patch: