Repository: DataBiosphere/toil

Issue:
Job chaining breaks service termination timing
The docs say:

> The services of a job are started before any of its successors have been run and stopped after all the successors of the job have completed successfully.

But that's not the case if a chained job starts a service. The chaining works by pushing the predecessor's "stack" onto the next job's "stack", basically making the predecessor's successors into successors of the next job. So the job graph as designed by the user is no longer the same graph that's in the job store: the chained job has extra successors. This breaks the timing of service termination.

Consider this example (follow-on edges in red, service edges dotted):
![toil-chaining-issue dot](https://cloud.githubusercontent.com/assets/4723163/26659729/b28afba2-4627-11e7-84c2-fc48da5c7fb2.png)

`chainEnd` can be chained to `chainStart` (assuming it has the same CPU, memory, etc requirements). If it's chained, then `service` will stick around not just through `job1` and `job2`, but also `job3` and `job4`. This isn't great, because if the service is, say, a very large database, it will hang around on a non-preemptable server costing lots of money for no reason.

I fixed this on my fork by just disabling chaining entirely--I've never been too convinced of its value and I'm not sure there aren't other edge cases like this still remaining to be found. I don't think it's feasible to shoehorn a fix into the chaining code as written, because it's undecidable to figure out whether the next job will run a service or not.
Job chaining breaks service termination timing
The docs say:

> The services of a job are started before any of its successors have been run and stopped after all the successors of the job have completed successfully.

But that's not the case if a chained job starts a service. The chaining works by pushing the predecessor's "stack" onto the next job's "stack", basically making the predecessor's successors into successors of the next job. So the job graph as designed by the user is no longer the same graph that's in the job store: the chained job has extra successors. This breaks the timing of service termination.

Consider this example (follow-on edges in red, service edges dotted):
![toil-chaining-issue dot](https://cloud.githubusercontent.com/assets/4723163/26659729/b28afba2-4627-11e7-84c2-fc48da5c7fb2.png)

`chainEnd` can be chained to `chainStart` (assuming it has the same CPU, memory, etc requirements). If it's chained, then `service` will stick around not just through `job1` and `job2`, but also `job3` and `job4`. This isn't great, because if the service is, say, a very large database, it will hang around on a non-preemptable server costing lots of money for no reason.

I fixed this on my fork by just disabling chaining entirely--I've never been too convinced of its value and I'm not sure there aren't other edge cases like this still remaining to be found. I don't think it's feasible to shoehorn a fix into the chaining code as written, because it's undecidable to figure out whether the next job will run a service or not.


Discussion:
I agree with @joelarmstrong, this also a problem in LSF when the logic in the batchSystem includes wall times (e.g. short running jobs accessing a short queue preference)
@DailyDreaming, do think its possible we can add a command line option to disable job chaining? In our case `LSF` kills jobs that take more than 2 hours (given that you said it was going to be a short job). As such, when toil chain jobs, things that were expected to last less than 2 hours last longer.
That seems very reasonable. @joelarmstrong just spoke with me about this and he's better situated to help on this front. He's been generously giving us his time in sharing some of the fixes he's had to patch in to toil for his own projects and I think he'll be adding this.
I agree with @joelarmstrong, this also a problem in LSF when the logic in the batchSystem includes wall times (e.g. short running jobs accessing a short queue preference)
@DailyDreaming, do think its possible we can add a command line option to disable job chaining? In our case `LSF` kills jobs that take more than 2 hours (given that you said it was going to be a short job). As such, when toil chain jobs, things that were expected to last less than 2 hours last longer.
That seems very reasonable. @joelarmstrong just spoke with me about this and he's better situated to help on this front. He's been generously giving us his time in sharing some of the fixes he's had to patch in to toil for his own projects and I think he'll be adding this.

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -124,6 +124,7 @@ def __init__(self):
 
         #Misc
         self.disableCaching = True
+        self.disableChaining = False
         self.maxLogFileSize = 64000
         self.writeLogs = None
         self.writeLogsGzip = None
@@ -261,6 +262,7 @@ def parseIntList(s):
 
         #Misc
         setOption("disableCaching")
+        setOption("disableChaining")
         setOption("maxLogFileSize", h2b, iC(1))
         setOption("writeLogs")
         setOption("writeLogsGzip")
@@ -507,6 +509,9 @@ def _addOptions(addGroupFn, config):
                 help='Disables caching in the file store. This flag must be set to use '
                      'a batch system that does not support caching such as Grid Engine, Parasol, '
                      'LSF, or Slurm')
+    addOptionFn('--disableChaining', dest='disableChaining', action='store_true', default=False,
+                help="Disables chaining of jobs (chaining uses one job's resource allocation "
+                "for its successor job if possible).")
     addOptionFn("--maxLogFileSize", dest="maxLogFileSize", default=None,
                 help=("The maximum size of a job log file to keep (in bytes), log files "
                       "larger than this will be truncated to the last X bytes. Setting "
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -18,7 +18,6 @@
 from builtins import str
 from builtins import map
 from builtins import filter
-from builtins import object
 import os
 import sys
 import copy
@@ -42,32 +41,70 @@
 from toil.common import Toil
 from toil.fileStore import FileStore
 from toil import logProcessContext
+from toil.job import Job
+from toil.lib.bioio import setLogLevel
+from toil.lib.bioio import getTotalCpuTime
+from toil.lib.bioio import getTotalCpuTimeAndMemoryUsage
 import signal
 
 logger = logging.getLogger(__name__)
 
-
-def nextOpenDescriptor():
-    """Gets the number of the next available file descriptor.
+def nextChainableJobGraph(jobGraph, jobStore):
+    """Returns the next chainable jobGraph after this jobGraph if one
+    exists, or None if the chain must terminate.
     """
-    descriptor = os.open("/dev/null", os.O_RDONLY)
-    os.close(descriptor)
-    return descriptor
-
+    #If no more jobs to run or services not finished, quit
+    if len(jobGraph.stack) == 0 or len(jobGraph.services) > 0 or jobGraph.checkpoint != None:
+        logger.debug("Stopping running chain of jobs: length of stack: %s, services: %s, checkpoint: %s",
+                     len(jobGraph.stack), len(jobGraph.services), jobGraph.checkpoint != None)
+        return None
+
+    #Get the next set of jobs to run
+    jobs = jobGraph.stack[-1]
+    assert len(jobs) > 0
+
+    #If there are 2 or more jobs to run in parallel we quit
+    if len(jobs) >= 2:
+        logger.debug("No more jobs can run in series by this worker,"
+                    " it's got %i children", len(jobs)-1)
+        return None
+
+    #We check the requirements of the jobGraph to see if we can run it
+    #within the current worker
+    successorJobNode = jobs[0]
+    if successorJobNode.memory > jobGraph.memory:
+        logger.debug("We need more memory for the next job, so finishing")
+        return None
+    if successorJobNode.cores > jobGraph.cores:
+        logger.debug("We need more cores for the next job, so finishing")
+        return None
+    if successorJobNode.disk > jobGraph.disk:
+        logger.debug("We need more disk for the next job, so finishing")
+        return None
+    if successorJobNode.preemptable != jobGraph.preemptable:
+        logger.debug("Preemptability is different for the next job, returning to the leader")
+        return None
+    if successorJobNode.predecessorNumber > 1:
+        logger.debug("The jobGraph has multiple predecessors, we must return to the leader.")
+        return None
+
+    # Load the successor jobGraph
+    successorJobGraph = jobStore.load(successorJobNode.jobStoreID)
+
+    # Somewhat ugly, but check if job is a checkpoint job and quit if
+    # so
+    if successorJobGraph.command.startswith( "_toil " ):
+        #Load the job
+        successorJob = Job._loadJob(successorJobGraph.command, jobStore)
+
+        # Check it is not a checkpoint
+        if successorJob.checkpoint:
+            logger.debug("Next job is checkpoint, so finishing")
+            return None
+
+    # Made it through! This job is chainable.
+    return successorJobGraph
 
-class AsyncJobStoreWrite(object):
-    def __init__(self, jobStore):
-        pass
-    
-    def writeFile(self, filePath):
-        pass
-    
-    def writeFileStream(self):
-        pass
-    
-    def blockUntilSync(self):
-        pass
-    
 def workerScript(jobStore, config, jobName, jobStoreID, redirectOutputToLogFile=True):
     """
     Worker process script, runs a job. 
@@ -79,29 +116,6 @@ def workerScript(jobStore, config, jobName, jobStoreID, redirectOutputToLogFile=
     """
     logging.basicConfig()
 
-    ##########################################
-    #Import necessary modules 
-    ##########################################
-    
-    # This is assuming that worker.py is at a path ending in "/toil/worker.py".
-    sourcePath = os.path.dirname(os.path.dirname(__file__))
-    if sourcePath not in sys.path:
-        sys.path.append(sourcePath)
-    
-    #Now we can import all the necessary functions
-    from toil.lib.bioio import setLogLevel
-    from toil.lib.bioio import getTotalCpuTime
-    from toil.lib.bioio import getTotalCpuTimeAndMemoryUsage
-    from toil.job import Job
-    try:
-        import boto
-    except ImportError:
-        pass
-    else:
-        # boto is installed, monkey patch it now
-        from bd2k.util.ec2.credentials import enable_metadata_credential_caching
-        enable_metadata_credential_caching()
-    
     ##########################################
     #Create the worker killer, if requested
     ##########################################
@@ -203,7 +217,6 @@ def badWorker():
     statsDict.jobs = []
     statsDict.workers.logsToMaster = []
     blockFn = lambda : True
-    cleanCacheFn = lambda x : True
     listOfJobs = [jobName]
     try:
 
@@ -211,11 +224,6 @@ def badWorker():
         logger.info("---TOIL WORKER OUTPUT LOG---")
         sys.stdout.flush()
         
-        #Log the number of open file descriptors so we can tell if we're leaking
-        #them.
-        logger.debug("Next available file descriptor: {}".format(
-            nextOpenDescriptor()))
-
         logProcessContext(config)
 
         ##########################################
@@ -275,12 +283,8 @@ def badWorker():
         ##########################################
         
         if config.stats:
-            startTime = time.time()
             startClock = getTotalCpuTime()
 
-        #Make a temporary file directory for the jobGraph
-        #localTempDir = makePublicDir(os.path.join(localWorkerTempDir, "localTempDir"))
-
         startTime = time.time()
         while True:
             ##########################################
@@ -324,58 +328,10 @@ def badWorker():
             ##########################################
             #Establish if we can run another jobGraph within the worker
             ##########################################
-            
-            #If no more jobs to run or services not finished, quit
-            if len(jobGraph.stack) == 0 or len(jobGraph.services) > 0 or jobGraph.checkpoint != None:
-                logger.debug("Stopping running chain of jobs: length of stack: %s, services: %s, checkpoint: %s",
-                             len(jobGraph.stack), len(jobGraph.services), jobGraph.checkpoint != None)
-                break
-            
-            #Get the next set of jobs to run
-            jobs = jobGraph.stack[-1]
-            assert len(jobs) > 0
-            
-            #If there are 2 or more jobs to run in parallel we quit
-            if len(jobs) >= 2:
-                logger.debug("No more jobs can run in series by this worker,"
-                            " it's got %i children", len(jobs)-1)
-                break
-            
-            #We check the requirements of the jobGraph to see if we can run it
-            #within the current worker
-            successorJobNode = jobs[0]
-            if successorJobNode.memory > jobGraph.memory:
-                logger.debug("We need more memory for the next job, so finishing")
-                break
-            if successorJobNode.cores > jobGraph.cores:
-                logger.debug("We need more cores for the next job, so finishing")
-                break
-            if successorJobNode.disk > jobGraph.disk:
-                logger.debug("We need more disk for the next job, so finishing")
+            successorJobGraph = nextChainableJobGraph(jobGraph, jobStore)
+            if successorJobGraph is None or config.disableChaining:
+                # Can't chain any more jobs.
                 break
-            if successorJobNode.preemptable != jobGraph.preemptable:
-                logger.debug("Preemptability is different for the next job, returning to the leader")
-                break
-            if successorJobNode.predecessorNumber > 1:
-                logger.debug("The jobGraph has multiple predecessors, we must return to the leader.")
-                break
-
-            # Load the successor jobGraph
-            successorJobGraph = jobStore.load(successorJobNode.jobStoreID)
-
-            # add the successor to the list of jobs run
-            listOfJobs.append(str(successorJobGraph))
-
-            # Somewhat ugly, but check if job is a checkpoint job and quit if
-            # so
-            if successorJobGraph.command.startswith( "_toil " ):
-                #Load the job
-                successorJob = Job._loadJob(successorJobGraph.command, jobStore)
-
-                # Check it is not a checkpoint
-                if successorJob.checkpoint:
-                    logger.debug("Next job is checkpoint, so finishing")
-                    break
 
             ##########################################
             #We have a single successor job that is not a checkpoint job.
@@ -385,21 +341,16 @@ def badWorker():
             #We can then delete the successor jobGraph in the jobStore, as it is
             #wholly incorporated into the current jobGraph.
             ##########################################
-            
+
+            # add the successor to the list of jobs run
+            listOfJobs.append(str(successorJobGraph))
+
             #Clone the jobGraph and its stack
             jobGraph = copy.deepcopy(jobGraph)
             
             #Remove the successor jobGraph
             jobGraph.stack.pop()
 
-            #These should all match up
-            assert successorJobGraph.memory == successorJobNode.memory
-            assert successorJobGraph.cores == successorJobNode.cores
-            assert successorJobGraph.predecessorsFinished == set()
-            assert successorJobGraph.predecessorNumber == 1
-            assert successorJobGraph.command is not None
-            assert successorJobGraph.jobStoreID == successorJobNode.jobStoreID
-
             #Transplant the command and stack to the current jobGraph
             jobGraph.command = successorJobGraph.command
             jobGraph.stack += successorJobGraph.stack
@@ -539,18 +490,28 @@ def badWorker():
 def main(argv=None):
     if argv is None:
         argv = sys.argv
-        
+
     # Parse input args
     jobName = argv[1]
     jobStoreLocator = argv[2]
     jobStoreID = argv[3]
-    
+
     ##########################################
     #Load the jobStore/config file
     ##########################################
-    
+
+    # Try to monkey-patch boto early so that credentials are cached.
+    try:
+        import boto
+    except ImportError:
+        pass
+    else:
+        # boto is installed, monkey patch it now
+        from bd2k.util.ec2.credentials import enable_metadata_credential_caching
+        enable_metadata_credential_caching()
+
     jobStore = Toil.resumeJobStore(jobStoreLocator)
     config = jobStore.config
-    
+
     # Call the worker
     workerScript(jobStore, config, jobName, jobStoreID)


Test patch:
diff --git a/src/toil/test/src/workerTest.py b/src/toil/test/src/workerTest.py
new file mode 100644
--- /dev/null
+++ b/src/toil/test/src/workerTest.py
@@ -0,0 +1,91 @@
+# Copyright (C) 2015-2018 Regents of the University of California
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import pickle
+
+from toil.common import Config
+from toil.job import Job
+from toil.jobGraph import JobGraph
+from toil.jobStores.fileJobStore import FileJobStore
+from toil.test import ToilTest
+from toil.worker import nextChainableJobGraph
+
+class WorkerTests(ToilTest):
+    """Test miscellaneous units of the worker."""
+    def setUp(self):
+        super(WorkerTests, self).setUp()
+        path = self._getTestJobStorePath()
+        self.jobStore = FileJobStore(path)
+        self.config = Config()
+        self.config.jobStore = 'file:%s' % path
+        self.jobStore.initialize(self.config)
+        self.jobGraphNumber = 0
+
+    def testNextChainableJobGraph(self):
+        """Make sure chainable/non-chainable jobs are identified correctly."""
+        def createJobGraph(memory, cores, disk, preemptable, checkpoint):
+            """Create a fake-ish Job and JobGraph pair, and return the
+            jobGraph."""
+            name = 'jobGraph%d' % self.jobGraphNumber
+            self.jobGraphNumber += 1
+
+            job = Job()
+            job.checkpoint = checkpoint
+            with self.jobStore.writeFileStream() as (f, fileStoreID):
+                pickle.dump(job, f, pickle.HIGHEST_PROTOCOL)
+            command = '_toil %s fooCommand toil True' % fileStoreID
+            jobGraph = JobGraph(command=command, memory=memory, cores=cores,
+                                disk=disk, unitName=name,
+                                jobName=name, preemptable=preemptable,
+                                jobStoreID=name, remainingRetryCount=1,
+                                predecessorNumber=1)
+            return self.jobStore.create(jobGraph)
+
+        # Identical non-checkpoint jobs should be chainable.
+        jobGraph1 = createJobGraph(1, 2, 3, True, False)
+        jobGraph2 = createJobGraph(1, 2, 3, True, False)
+        jobGraph1.stack = [[jobGraph2]]
+        self.assertEquals(jobGraph2, nextChainableJobGraph(jobGraph1, self.jobStore))
+
+        # Identical checkpoint jobs should not be chainable.
+        jobGraph1 = createJobGraph(1, 2, 3, True, False)
+        jobGraph2 = createJobGraph(1, 2, 3, True, True)
+        jobGraph1.stack = [[jobGraph2]]
+        self.assertEquals(None, nextChainableJobGraph(jobGraph1, self.jobStore))
+
+        # If there is no child we should get nothing to chain.
+        jobGraph1 = createJobGraph(1, 2, 3, True, False)
+        jobGraph1.stack = []
+        self.assertEquals(None, nextChainableJobGraph(jobGraph1, self.jobStore))
+
+        # If there are 2 or more children we should get nothing to chain.
+        jobGraph1 = createJobGraph(1, 2, 3, True, False)
+        jobGraph2 = createJobGraph(1, 2, 3, True, False)
+        jobGraph3 = createJobGraph(1, 2, 3, True, False)
+        jobGraph1.stack = [[jobGraph2, jobGraph3]]
+        self.assertEquals(None, nextChainableJobGraph(jobGraph1, self.jobStore))
+
+        # If there is an increase in resource requirements we should get nothing to chain.
+        reqs = {'memory': 1, 'cores': 2, 'disk': 3, 'preemptable': True, 'checkpoint': False}
+        for increased_attribute in ('memory', 'cores', 'disk'):
+            jobGraph1 = createJobGraph(**reqs)
+            reqs[increased_attribute] += 1
+            jobGraph2 = createJobGraph(**reqs)
+            jobGraph1.stack = [[jobGraph2]]
+            self.assertEquals(None, nextChainableJobGraph(jobGraph1, self.jobStore))
+
+        # A change in preemptability from True to False should be disallowed.
+        jobGraph1 = createJobGraph(1, 2, 3, True, False)
+        jobGraph2 = createJobGraph(1, 2, 3, False, True)
+        jobGraph1.stack = [[jobGraph2]]
+        self.assertEquals(None, nextChainableJobGraph(jobGraph1, self.jobStore))