Repository: DataBiosphere/toil

Issue:
cwl dynamic resource requirement bug
I am trying use the dynamic resource requirement feature in gridEngine which was merged in #1810 and found an issue. 
To reproduce the issue, please add the below lines to toil/src/toil/test/cwl/revtool.cwl  and run "cwltoil --no-container --retryCount 0  --batchSystem gridEngine revsort.cwl revsort-job.json"
``` cwl
requirements:
  - class: InlineJavascriptRequirement
  - class: ResourceRequirement
    ramMin: $(inputs.input.size)
```

Below is the worker log for sorttool

```python
---TOIL WORKER OUTPUT LOG---
INFO:toil:Running Toil version 3.13.0-3cb535425306955028ed7604114f772990d00440.
Traceback (most recent call last):
  File "/work/toil/src/toil/worker.py", line 301, in main
    job = Job._loadJob(jobGraph.command, jobStore)
  File "/work/toil/src/toil/job.py", line 902, in _loadJob
    return cls._unpickle(userModule, fileHandle, jobStore.config)
  File "/work/toil/src/toil/job.py", line 931, in _unpickle
    runnable = unpickler.load()
  File "/work/toil/src/toil/job.py", line 1735, in __new__
    return cls._resolve(*args)
  File "/work/toil/src/toil/job.py", line 1747, in _resolve
    value = pickle.load(fileHandle)
  File "/work/toil/src/toil/job.py", line 1735, in __new__
    return cls._resolve(*args)
  File "/work/toil/src/toil/job.py", line 1747, in _resolve
    value = pickle.load(fileHandle)
  File "/work/toil/src/toil/job.py", line 1827, in __setstate__
    "this job as a child/follow-on of {jobName}.".format(jobName=jobName))
RuntimeError: (RuntimeError(RuntimeError("This job was passed a promise that wasn't yet resolved when it ran. The job 'file:///work/toil/src/toil/test/cwl/revtool.cwl' rev that fulfills this promise hasn't yet finished. This means that there aren't enough constraints to ensure the current job always runs after 'file:///work/toil/src/toil/test/cwl/revtool.cwl' rev. Consider adding a follow-on indirection between this job and its parent, or adding this job as a child/follow-on of 'file:///work/toil/src/toil/test/cwl/revtool.cwl' rev.",), <class 'toil.job.Promise'>, (u'file:/tmp/tmp14goWU', 'N/u/tmpF1_VvA.tmp')), <class 'toil.job.Promise'>, (u'file:/tmp/tmp14goWU', 'O/g/tmp_Mr8W3.tmp'))
ERROR:toil.worker:Exiting the worker because of a failed job on host host01
WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///work/toil/src/toil/test/cwl/sorttool.cwl' sort J/b/jobhOqgt1 with ID J/b/jobhOqgt1 to 0

```


As per my understanding, there are two steps revtool and sorttool in the revsort workflow. Two wrapper jobs are created for both of them. The revtool_wrapper spawns the actual revtool job, but also sorttool_wrapper job is started. Since sorttool wrapper does not have its promise fulfilled the workflow fails.


Discussion:
Hello @pkp124 and thank you for your report. Is this reproducible using the latest dev version of Toil? 

http://toil.readthedocs.io/en/3.12.0/gettingStarted/install.html#building-from-source

Hello @mr-c , thanks for the quick response. I was able to reproduce the same issue on the latest commit of the master branch, although i had to rerun multiple times (~ 10) to reproduce the issue.
@pkp124 Thanks. I am also now able to reproduce and I will continue my investigation.

Resolving patch:
diff --git a/src/toil/cwl/cwltoil.py b/src/toil/cwl/cwltoil.py
--- a/src/toil/cwl/cwltoil.py
+++ b/src/toil/cwl/cwltoil.py
@@ -385,29 +385,19 @@ class CWLJobWrapper(Job):
     def __init__(self, tool, cwljob, **kwargs):
         super(CWLJobWrapper, self).__init__(cores=1,
                                             memory=1024*1024,
-                                            disk=1)
+                                            disk=8*1024)
         self.cwltool = remove_pickle_problems(tool)
         self.cwljob = cwljob
         self.kwargs = kwargs
 
     def run(self, fileStore):
         cwljob = resolve_indirect(self.cwljob)
-
-        if 'builder' in self.kwargs:
-            builder = self.kwargs["builder"]
-        else:
-            builder = cwltool.builder.Builder()
-            builder.job = cwljob
-            builder.requirements = []
-            builder.outdir = None
-            builder.tmpdir = None
-            builder.timeout = 0
-            builder.resources = {}
         options = copy.deepcopy(self.kwargs)
-        options.update({'tool': self.cwltool,
-                        'cwljob': self.cwljob,
-                        'builder': builder})
-        realjob = CWLJob(**options)
+        options['jobobj'] = cwljob
+        realjob = CWLJob(self.cwltool, cwljob, **options)
+        for child in self._children:
+            cwllogger.debug("CWLJobWrapper child: {}".format(child))
+            realjob.addFollowOn(child)
         self.addChild(realjob)
         return realjob.rv()
 
@@ -420,11 +410,11 @@ def __init__(self, tool, cwljob, **kwargs):
             builder = kwargs["builder"]
         else:
             builder = cwltool.builder.Builder()
-            builder.job = {}
+            builder.job = cwljob
             builder.requirements = []
             builder.outdir = None
             builder.tmpdir = None
-            builder.timeout = 0
+            builder.timeout = kwargs.get('eval_timeout')
             builder.resources = {}
         req = tool.evalResources(builder, {})
         self.cwltool = remove_pickle_problems(tool)
@@ -509,7 +499,7 @@ def makeJob(tool, jobobj, **kwargs):
                     options = copy.deepcopy(kwargs)
                     options.update({
                         'tool': tool,
-                        'jobobj': jobobj})
+                        'cwljob': jobobj})
                     job = CWLJobWrapper(**options)
                     return (job, job)
         options = copy.deepcopy(kwargs)


Test patch: