Repository: DataBiosphere/toil

Issue:
cwl dynamic resource requirement bug
I am trying use the dynamic resource requirement feature in gridEngine which was merged in #1810 and found an issue. 
To reproduce the issue, please add the below lines to toil/src/toil/test/cwl/revtool.cwl  and run "cwltoil --no-container --retryCount 0  --batchSystem gridEngine revsort.cwl revsort-job.json"
``` cwl
requirements:
  - class: InlineJavascriptRequirement
  - class: ResourceRequirement
    ramMin: $(inputs.input.size)
```

Below is the worker log for sorttool

```python
---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.13.0-3cb535425306955028ed7604114f772990d00440.
Traceback (most recent call last):
  File "/work/toil/src/toil/worker.py", line 301, in main
    job = Job._loadJob(jobGraph.command, jobStore)
  File "/work/toil/src/toil/job.py", line 902, in _loadJob
    return cls._unpickle(userModule, fileHandle, jobStore.config)
  File "/work/toil/src/toil/job.py", line 931, in _unpickle
    runnable = unpickler.load()
  File "/work/toil/src/toil/job.py", line 1735, in __new__
    return cls._resolve(*args)
  File "/work/toil/src/toil/job.py", line 1747, in _resolve
    value = pickle.load(fileHandle)
  File "/work/toil/src/toil/job.py", line 1735, in __new__
    return cls._resolve(*args)
  File "/work/toil/src/toil/job.py", line 1747, in _resolve
    value = pickle.load(fileHandle)
  File "/work/toil/src/toil/job.py", line 1827, in __setstate__
    "this job as a child/follow-on of {jobName}.".format(jobName=jobName))
RuntimeError: (RuntimeError(RuntimeError("This job was passed a promise that wasn't yet resolved when it ran. The job 'file:///work/toil/src/toil/test/cwl/revtool.cwl' rev that fulfills this promise hasn't yet finished. This means that there aren't enough constraints to ensure the current job always runs after 'file:///work/toil/src/toil/test/cwl/revtool.cwl' rev. Consider adding a follow-on indirection between this job and its parent, or adding this job as a child/follow-on of 'file:///work/toil/src/toil/test/cwl/revtool.cwl' rev.",), <class 'toil.job.Promise'>, (u'file:/tmp/tmp14goWU', 'N/u/tmpF1_VvA.tmp')), <class 'toil.job.Promise'>, (u'file:/tmp/tmp14goWU', 'O/g/tmp_Mr8W3.tmp'))
ERROR:toil.worker:Exiting the worker because of a failed job on host host01
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///work/toil/src/toil/test/cwl/sorttool.cwl' sort J/b/jobhOqgt1 with ID J/b/jobhOqgt1 to 0

```


As per my understanding, there are two steps revtool and sorttool in the revsort workflow. Two wrapper jobs are created for both of them. The revtool_wrapper spawns the actual revtool job, but also sorttool_wrapper job is started. Since sorttool wrapper does not have its promise fulfilled the workflow fails.


Discussion:
Hello @pkp124 and thank you for your report. Is this reproducible using the latest dev version of Toil? 

http://toil.readthedocs.io/en/3.12.0/gettingStarted/install.html#building-from-source

Hello @mr-c , thanks for the quick response. I was able to reproduce the same issue on the latest commit of the master branch, although i had to rerun multiple times (~ 10) to reproduce the issue.
@pkp124 Thanks. I am also now able to reproduce and I will continue my investigation.
@pkp124 Does https://github.com/BD2KGenomics/toil/pull/2049 work for you?
@pkp124 I think we fixed it! Thanks again for alerting me to this issue and for the easy to reproduce example; that was very helpful.
@mr-c , thanks for the effort. I did some quick tests today and unfortunately, found that the error still persists :)  Again, the issue can be reproduced with the same files, by running multiple times.
Okay, @DailyDreaming @ejacox and others, I need your help.

When in toil-cwl-runner we have dynamic resource requirements we add a CWLJobWrapper to the job graph so we can attach the real job with the current memory, cpu, and disk requirements once the inputs are known.

In an attempt to address this issue I added a follow on link between the existing children of the CWLJobWrapper and the new "real" job:

https://github.com/BD2KGenomics/toil/blob/ad1a097f45d5019540972da36d44a805b1104b72/src/toil/cwl/cwltoil.py#L400

Seems like this isn't enough. Any suggestions?
One thing I noticed, the CWLJobWrapper does not seem to have any children. 
@pkp124 Yes, and it is missing the reference to its predecessors as well. Perhaps a pickling error?

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -63,7 +63,7 @@
 cwllogger = logging.getLogger("cwltool")
 
 # Define internal jobs we should avoid submitting to batch systems and logging
-CWL_INTERNAL_JOBS = ("CWLJob", "CWLJobWrapper", "CWLWorkflow", "CWLScatter", "CWLGather",
+CWL_INTERNAL_JOBS = ("CWLJobWrapper", "CWLWorkflow", "CWLScatter", "CWLGather",
                      "ResolveIndirect")
 
 # The job object passed into CWLJob and CWLWorkflow
@@ -392,9 +392,6 @@ def run(self, fileStore):
         options = copy.deepcopy(self.kwargs)
         options['jobobj'] = cwljob
         realjob = CWLJob(self.cwltool, cwljob, **options)
-        for child in self._children:
-            cwllogger.debug("CWLJobWrapper child: {}".format(child))
-            realjob.addFollowOn(child)
         self.addChild(realjob)
         return realjob.rv()
 
@@ -403,18 +400,18 @@ class CWLJob(Job):
     """Execute a CWL tool using cwltool.main.single_job_executor"""
 
     def __init__(self, tool, cwljob, **kwargs):
+        self.cwltool = remove_pickle_problems(tool)
         if 'builder' in kwargs:
             builder = kwargs["builder"]
         else:
             builder = cwltool.builder.Builder()
             builder.job = cwljob
-            builder.requirements = []
+            builder.requirements = self.cwltool.requirements
             builder.outdir = None
             builder.tmpdir = None
             builder.timeout = kwargs.get('eval_timeout')
             builder.resources = {}
         req = tool.evalResources(builder, {})
-        self.cwltool = remove_pickle_problems(tool)
         # pass the default of None if basecommand is empty
         unitName = self.cwltool.tool.get("baseCommand", None)
         if isinstance(unitName, (list, tuple)):
@@ -773,7 +770,10 @@ def run(self, fileStore):
                         for inp in step.tool["inputs"]:
                             for s in aslist(inp.get("source", [])):
                                 if not promises[s].hasChild(wfjob):
-                                    promises[s].addChild(wfjob)
+                                    if isinstance(promises[s], (CWLJobWrapper, CWLGather)):
+                                        promises[s].addFollowOn(wfjob)
+                                    else:
+                                        promises[s].addChild(wfjob)
                                     connected = True
                         if not connected:
                             # workflow step has default inputs only, isn't connected to other jobs,


Test patch:
diff --git a/src/toil/test/cwl/cwlTest.py b/src/toil/test/cwl/cwlTest.py
--- a/src/toil/test/cwl/cwlTest.py
+++ b/src/toil/test/cwl/cwlTest.py
@@ -32,6 +32,7 @@
                        needs_torque)
 
 
+
 @needs_cwl
 class CWLTest(ToilTest):
 
@@ -75,6 +76,20 @@ def test_run_revsort(self):
                 u'class': u'File',
                 u'checksum': u'sha1$b9214658cc453331b62c2282b772a5c063dbd284'}})
 
+    def test_run_revsort2(self):
+        outDir = self._createTempDir()
+        self._tester('src/toil/test/cwl/revsort2.cwl',
+                     'src/toil/test/cwl/revsort-job.json',
+                     outDir, {
+            # Having unicode string literals isn't necessary for the assertion but makes for a
+            # less noisy diff in case the assertion fails.
+            u'output': {
+                u'location': "file://" + str(os.path.join(outDir, 'output.txt')),
+                u'basename': str("output.txt"),
+                u'size': 1111,
+                u'class': u'File',
+                u'checksum': u'sha1$b9214658cc453331b62c2282b772a5c063dbd284'}})
+
     def test_run_revsort_debug_worker(self):
         outDir = self._createTempDir()
         # Having unicode string literals isn't necessary for the assertion
diff --git a/src/toil/test/cwl/revsort2.cwl b/src/toil/test/cwl/revsort2.cwl
new file mode 100644
--- /dev/null
+++ b/src/toil/test/cwl/revsort2.cwl
@@ -0,0 +1,65 @@
+#
+# This is a two-step workflow which uses "revtool" and "sorttool" defined above.
+#
+class: Workflow
+doc: "Reverse the lines in a document, then sort those lines."
+cwlVersion: v1.0
+
+# Requirements & hints specify prerequisites and extensions to the workflow.
+# In this example, DockerRequirement specifies a default Docker container
+# in which the command line tools will execute.
+hints:
+  - class: DockerRequirement
+    dockerPull: debian:8
+
+
+# The inputs array defines the structure of the input object that describes
+# the inputs to the workflow.
+#
+# The "reverse_sort" input parameter demonstrates the "default" field.  If the
+# field "reverse_sort" is not provided in the input object, the default value will
+# be used.
+inputs:
+  input:
+    type: File
+    doc: "The input file to be processed."
+  reverse_sort:
+    type: boolean
+    default: true
+    doc: "If true, reverse (decending) sort"
+
+# The "outputs" array defines the structure of the output object that describes
+# the outputs of the workflow.
+#
+# Each output field must be connected to the output of one of the workflow
+# steps using the "connect" field.  Here, the parameter "#output" of the
+# workflow comes from the "#sorted" output of the "sort" step.
+outputs:
+  output:
+    type: File
+    outputSource: sorted/output
+    doc: "The output with the lines reversed and sorted."
+
+# The "steps" array lists the executable steps that make up the workflow.
+# The tool to execute each step is listed in the "run" field.
+#
+# In the first step, the "inputs" field of the step connects the upstream
+# parameter "#input" of the workflow to the input parameter of the tool
+# "revtool.cwl#input"
+#
+# In the second step, the "inputs" field of the step connects the output
+# parameter "#reversed" from the first step to the input parameter of the
+# tool "sorttool.cwl#input".
+steps:
+  rev:
+    in:
+      input: input
+    out: [output]
+    run: revtool2.cwl
+
+  sorted:
+    in:
+      input: rev/output
+      reverse: reverse_sort
+    out: [output]
+    run: sorttool.cwl
diff --git a/src/toil/test/cwl/revtool2.cwl b/src/toil/test/cwl/revtool2.cwl
new file mode 100644
--- /dev/null
+++ b/src/toil/test/cwl/revtool2.cwl
@@ -0,0 +1,33 @@
+#
+# Simplest example command line program wrapper for the Unix tool "rev".
+#
+class: CommandLineTool
+cwlVersion: v1.0
+doc: "Reverse each line using the `rev` command"
+
+requirements:
+  InlineJavascriptRequirement: {}
+  # A lengthy process to test premature execution of descendent steps
+  ResourceRequirement:
+    ramMin: |
+      ${ var start = new Date().getTime();
+         while (new Date().getTime() < start + 10000);
+         return inputs.input.size + 1536;
+       }
+
+inputs:
+  input:
+    type: File
+    inputBinding: {}
+
+outputs:
+  output:
+    type: File
+    outputBinding:
+      glob: output.txt
+
+baseCommand: rev
+
+# Specify that the standard output stream must be redirected to a file called
+# output.txt in the designated output directory.
+stdout: output.txt