Repository: common-workflow-language/cwltool

Issue:
Output directories containing files don't get cleaned up
## Expected Behavior

cwltool should remove (or have an option to remove) intermediate output directories once they're no longer needed.

## Actual Behavior

Output directories that have files in them (whether that's because the tool generated extra files that weren't included in the output, or because cwltool left the files in place for the next step to use them) are never deleted, meaning that long workflows can consume large amounts of disk and eventually run out of space...

This looks intentional, but I don't understand why it's desirable to leave behind huge numbers of directories containing the output from intermediate steps.

https://github.com/common-workflow-language/cwltool/blob/c27774b2fcfc66497ee22e08b309fd8a542f139b/cwltool/executors.py#L98-L99

https://github.com/common-workflow-language/cwltool/blob/c27774b2fcfc66497ee22e08b309fd8a542f139b/cwltool/process.py#L375-L379

https://github.com/common-workflow-language/cwltool/blob/c27774b2fcfc66497ee22e08b309fd8a542f139b/cwltool/process.py#L799-L801

## Workflow Code

wf.cwl:
```cwl
cwlVersion: v1.0
class: Workflow

inputs:
- id: workflow_input
  type: File

steps:
- id: workflow_step_one
  run: step_one.cwl
  in:
    step_one_input: workflow_input
  out:
  - step_one_output
- id: workflow_step_two
  run: step_two.cwl
  in:
    step_two_input: workflow_step_one/step_one_output
  out:
  - step_two_output

outputs:
- id: workflow_output
  type: File
  outputSource: workflow_step_two/step_two_output
```

step_one.cwl:
```cwl
cwlVersion: v1.0
class: CommandLineTool

baseCommand: ["touch", "step1", "step1_other"]

inputs:
- id: step_one_input
  type: File

outputs:
- id: step_one_output
  type: File
  outputBinding:
    glob: "step1"
```

step_two.cwl:
```cwl
cwlVersion: v1.0
class: CommandLineTool

baseCommand: ["touch", "step2", "step2_other"]

inputs:
- id: step_two_input
  type: File

outputs:
- id: step_two_output
  type: File
  outputBinding:
    glob: "step2"
```

job.yml:
```yaml
workflow_input:
  class: File
  path: input_file # this is just an empty file
```

## Full Traceback

```console
$ cwltool --debug wf.cwl job.yml 
/Users/jh36/venv/bin/cwltool 1.0.20180819175200
Resolved 'wf.cwl' to 'file:///Users/jh36/cwl/tmpfiles/wf.cwl'
[workflow ] initialized from file:///Users/jh36/cwl/tmpfiles/wf.cwl
[workflow ] start
[workflow ] {
    "workflow_input": {
        "class": "File",
        "location": "file:///Users/jh36/cwl/tmpfiles/input_file",
        "size": 11,
        "basename": "input_file",
        "nameroot": "input_file",
        "nameext": ""
    }
}
[workflow ] job step file:///Users/jh36/cwl/tmpfiles/wf.cwl#workflow_step_two not ready
[workflow ] starting step workflow_step_one
[job step workflow_step_one] job input {
    "file:///Users/jh36/cwl/tmpfiles/wf.cwl#workflow_step_one/step_one_input": {
        "class": "File",
        "location": "file:///Users/jh36/cwl/tmpfiles/input_file",
        "size": 11,
        "basename": "input_file",
        "nameroot": "input_file",
        "nameext": ""
    }
}
[job step workflow_step_one] evaluated job input to {
    "file:///Users/jh36/cwl/tmpfiles/wf.cwl#workflow_step_one/step_one_input": {
        "class": "File",
        "location": "file:///Users/jh36/cwl/tmpfiles/input_file",
        "size": 11,
        "basename": "input_file",
        "nameroot": "input_file",
        "nameext": ""
    }
}
[step workflow_step_one] start
[job workflow_step_one] initializing from file:///Users/jh36/cwl/tmpfiles/step_one.cwl as part of step workflow_step_one
[job workflow_step_one] {
    "step_one_input": {
        "class": "File",
        "location": "file:///Users/jh36/cwl/tmpfiles/input_file",
        "size": 11,
        "basename": "input_file",
        "nameroot": "input_file",
        "nameext": ""
    }
}
[job workflow_step_one] path mappings is {
    "file:///Users/jh36/cwl/tmpfiles/input_file": [
        "/Users/jh36/cwl/tmpfiles/input_file",
        "/private/var/folders/9n/vrwpqrl91gl4ft1v0vdghcx0000ggv/T/tmpjk2r7hwx/stg5a3a6b3b-1596-489f-a871-6d6716e98192/input_file",
        "File",
        true
    ]
}
[job workflow_step_one] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "touch"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "step1"
    },
    {
        "position": [
            -1000000,
            2
        ],
        "datum": "step1_other"
    }
]
[job workflow_step_one] /private/tmp/docker_tmp6dsu5gil$ touch \
    step1 \
    step1_other
[job workflow_step_one] completed success
[job workflow_step_one] {
    "step_one_output": {
        "location": "file:///private/tmp/docker_tmp6dsu5gil/step1",
        "basename": "step1",
        "nameroot": "step1",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "http://commonwl.org/cwltool#generation": 0
    }
}
[step workflow_step_one] produced output {
    "file:///Users/jh36/cwl/tmpfiles/wf.cwl#workflow_step_one/step_one_output": {
        "location": "file:///private/tmp/docker_tmp6dsu5gil/step1",
        "basename": "step1",
        "nameroot": "step1",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "http://commonwl.org/cwltool#generation": 0
    }
}
[step workflow_step_one] completed success
[job workflow_step_one] Removing input staging directory /private/var/folders/9n/vrwpqrl91gl4ft1v0vdghcx0000ggv/T/tmpjk2r7hwx
[job workflow_step_one] Removing temporary directory /private/var/folders/9n/vrwpqrl91gl4ft1v0vdghcx0000ggv/T/tmpl9_nygt_
[workflow ] starting step workflow_step_two
[job step workflow_step_two] job input {
    "file:///Users/jh36/cwl/tmpfiles/wf.cwl#workflow_step_two/step_two_input": {
        "location": "file:///private/tmp/docker_tmp6dsu5gil/step1",
        "basename": "step1",
        "nameroot": "step1",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "http://commonwl.org/cwltool#generation": 0
    }
}
[job step workflow_step_two] evaluated job input to {
    "file:///Users/jh36/cwl/tmpfiles/wf.cwl#workflow_step_two/step_two_input": {
        "location": "file:///private/tmp/docker_tmp6dsu5gil/step1",
        "basename": "step1",
        "nameroot": "step1",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "http://commonwl.org/cwltool#generation": 0
    }
}
[step workflow_step_two] start
[job workflow_step_two] initializing from file:///Users/jh36/cwl/tmpfiles/step_two.cwl as part of step workflow_step_two
[job workflow_step_two] {
    "step_two_input": {
        "location": "file:///private/tmp/docker_tmp6dsu5gil/step1",
        "basename": "step1",
        "nameroot": "step1",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "http://commonwl.org/cwltool#generation": 0
    }
}
[job workflow_step_two] path mappings is {
    "file:///private/tmp/docker_tmp6dsu5gil/step1": [
        "/private/tmp/docker_tmp6dsu5gil/step1",
        "/private/var/folders/9n/vrwpqrl91gl4ft1v0vdghcx0000ggv/T/tmpc8es5f5v/stge5b03988-2b54-4b5a-91b2-fa54099e2b7b/step1",
        "File",
        true
    ]
}
[job workflow_step_two] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "touch"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "step2"
    },
    {
        "position": [
            -1000000,
            2
        ],
        "datum": "step2_other"
    }
]
[job workflow_step_two] /private/tmp/docker_tmpe9ti48xo$ touch \
    step2 \
    step2_other
[job workflow_step_two] completed success
[job workflow_step_two] {
    "step_two_output": {
        "location": "file:///private/tmp/docker_tmpe9ti48xo/step2",
        "basename": "step2",
        "nameroot": "step2",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "http://commonwl.org/cwltool#generation": 0
    }
}
[step workflow_step_two] produced output {
    "file:///Users/jh36/cwl/tmpfiles/wf.cwl#workflow_step_two/step_two_output": {
        "location": "file:///private/tmp/docker_tmpe9ti48xo/step2",
        "basename": "step2",
        "nameroot": "step2",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "http://commonwl.org/cwltool#generation": 0
    }
}
[step workflow_step_two] completed success
[workflow ] completed success
[workflow ] {
    "workflow_output": {
        "location": "file:///private/tmp/docker_tmpe9ti48xo/step2",
        "basename": "step2",
        "nameroot": "step2",
        "nameext": "",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "http://commonwl.org/cwltool#generation": 0
    }
}
[job workflow_step_two] Removing input staging directory /private/var/folders/9n/vrwpqrl91gl4ft1v0vdghcx0000ggv/T/tmpc8es5f5v
[job workflow_step_two] Removing temporary directory /private/var/folders/9n/vrwpqrl91gl4ft1v0vdghcx0000ggv/T/tmpk82h0lu3
Moving /private/tmp/docker_tmpe9ti48xo/step2 to /Users/jh36/cwl/tmpfiles/step2
Removing intermediate output directory /private/tmp/docker_tmpzyyjmen_
{
    "workflow_output": {
        "location": "file:///Users/jh36/cwl/tmpfiles/step2",
        "basename": "step2",
        "class": "File",
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "size": 0,
        "path": "/Users/jh36/cwl/tmpfiles/step2"
    }
}
Final process status is success
$ # these are `JobExecutor.output_dirs`
$ ls /private/tmp/docker_tmpe9ti48xo /private/tmp/docker_tmp6dsu5gil /private/tmp/docker_tmpzyyjmen_
ls: /private/tmp/docker_tmpzyyjmen_: No such file or directory
/private/tmp/docker_tmp6dsu5gil:
step1       step1_other

/private/tmp/docker_tmpe9ti48xo:
step2_other
```

## Your Environment

* cwltool version: 1.0.20180819175200


Discussion:
I can't recall the reasoning either, and I wrote it.  So maybe we should remove the empty_subtree() check from cleanIntermediate() and see what happens.

If someone could verify that removing the empty_subtree() check doesn't break anything, and make a pull request that would be great.
I ran the conformance tests and they all seem to pass - I'll make a PR shortly.
Hmm, if this behavior really does date back to 2016, I am a little puzzled it is just being reported as a problem now.  Maybe something else changed, like #850?  Or maybe it is just because people are scaling up their workflows, making disk usage an issue.
(Ok it can't be #850 because that was only merged 3 days ago and the bug is older than that).
Bonus points for using `git bisect` to narrow down when the behaviour changed, if at all..
Ok, I'll try to bisect it (fwiw, I've noticed temporary files building up since I started using cwltool, which was 2 months ago)
Don't spend a lot of effort on bisecting, if you go back 6-12 months and the behavior is the same, it has probably always been that way.

Looking forward for a PR on this issue.

Resolving patch:
diff --git a/cwltool/executors.py b/cwltool/executors.py
--- a/cwltool/executors.py
+++ b/cwltool/executors.py
@@ -5,7 +5,7 @@
 import tempfile
 import threading
 from abc import ABCMeta, abstractmethod
-from typing import Any, Dict, List, Optional, Set, Tuple
+from typing import Any, Dict, Iterable, List, Optional, Set, Tuple
 
 import psutil
 from schema_salad.validate import ValidationException
@@ -97,7 +97,11 @@ def execute(self,
                 path_mapper=runtime_context.path_mapper)
 
         if runtime_context.rm_tmpdir:
-            cleanIntermediate(self.output_dirs)
+            if runtime_context.cachedir is None:
+                output_dirs = self.output_dirs # type: Iterable[Any]
+            else:
+                output_dirs = filter(lambda x: not x.startswith(runtime_context.cachedir), self.output_dirs)
+            cleanIntermediate(output_dirs)
 
         if self.final_output and self.final_status:
 
diff --git a/cwltool/process.py b/cwltool/process.py
--- a/cwltool/process.py
+++ b/cwltool/process.py
@@ -372,9 +372,9 @@ def relink(relinked,  # type: Dict[Text, Text]
     return outputObj
 
 
-def cleanIntermediate(output_dirs):  # type: (Set[Text]) -> None
+def cleanIntermediate(output_dirs):  # type: (Iterable[Text]) -> None
     for a in output_dirs:
-        if os.path.exists(a) and empty_subtree(a):
+        if os.path.exists(a):
             _logger.debug(u"Removing intermediate output directory %s", a)
             shutil.rmtree(a, True)
 


Test patch: