Repository: common-workflow-language/cwltool

Issue:
--cachedir needs to take `stdout` field into account when calculating the hash



Discussion:
Hi @mr-c, here does `stdout` field means whatever is written to stdout should be used in calculating cache hash. We are calculating cache key here https://github.com/common-workflow-language/cwltool/blob/master/cwltool/draft2tool.py#L298 I mean to ask, how to obtain `stdout` field inside this function.
Also I personally feel that instead of calculating `keydict` of values that go inside cachekey calculation in `Job` function, we can pass it as a argument, starting from main function and propagating to all intermediate function adding value to it whenever necessary and finally resolving inside `job` function. This can allow us to avoid `parameter references` that may come inside our prepared cmdline https://github.com/common-workflow-language/cwltool/blob/master/cwltool/draft2tool.py#L274 
Is this possible?
Here are the ingredients for this issue
1) `type: stdout` without and accompanying `stdout: my_stdout_name` results in a randomly named file that `stdout` is redirected to. This isn't very reproducible! Suggested fix: make the generation of this name stable as long as the tool nor its inputs change
2) As you point out, the cache key calculation doesn't take the name of the stdout field into effect. This also need to be fixed
for 1. see https://github.com/common-workflow-language/cwltool/blob/master/cwltool/load_tool.py#L91
for 2. Change https://github.com/common-workflow-language/cwltool/blob/master/cwltool/draft2tool.py#L274 to be `keydict = {u"cmdline": cmdline, u"stdout",  ... }`

Resolving patch:
diff --git a/cwltool/draft2tool.py b/cwltool/draft2tool.py
--- a/cwltool/draft2tool.py
+++ b/cwltool/draft2tool.py
@@ -244,7 +244,6 @@ def job(self,
         # type: (...) -> Generator[Union[JobBase, CallbackJob], None, None]
 
         jobname = uniquename(kwargs.get("name", shortname(self.tool.get("id", "job"))))
-
         if kwargs.get("cachedir"):
             cacheargs = kwargs.copy()
             cacheargs["outdir"] = "/out"
@@ -270,6 +269,8 @@ def job(self,
                 cmdline = ["docker", "run", dockerimg] + cmdline
             keydict = {u"cmdline": cmdline}
 
+            if "stdout" in self.tool:
+                keydict["stdout"] = self.tool["stdout"]
             for location, f in cachebuilder.pathmapper.items():
                 if f.type == "File":
                     checksum = next((e['checksum'] for e in cachebuilder.files
diff --git a/cwltool/load_tool.py b/cwltool/load_tool.py
--- a/cwltool/load_tool.py
+++ b/cwltool/load_tool.py
@@ -6,6 +6,8 @@
 import os
 import re
 import uuid
+import hashlib
+import json
 from typing import Any, Callable, Dict, List, Text, Tuple, Union, cast
 
 import requests.sessions
@@ -88,7 +90,8 @@ def _convert_stdstreams_to_files(workflowobj):
                         if streamtype in workflowobj:
                             filename = workflowobj[streamtype]
                         else:
-                            filename = Text(uuid.uuid4())
+                            filename = Text(hashlib.sha1(json.dumps(workflowobj,
+                                        sort_keys=True).encode('utf-8')).hexdigest())
                             workflowobj[streamtype] = filename
                         out['type'] = 'File'
                         out['outputBinding'] = {'glob': filename}


Test patch: