Repository: DataBiosphere/toil

Issue:
Detection of checkpoint jobs that are not leaf vertices omits the workflow root job
The current code to detect checkpoint jobs that are not leaf vertices does not check the workflow root job.

Below is an excerpt from `toil.job.Job.checkNewCheckpointsAreLeafVertices`:
```python
for y in filter(lambda x : x.checkpoint, jobs):
    if y not in roots: # The roots are the prexisting jobs
        if len(y._children) != 0 and len(y._followOns) != 0 and len(y._services) != 0:
            raise JobGraphDeadlockException(“New checkpoint job %s is not a leaf in the job graph” % y)
```
The line in question:
```python
    if y not in roots: # The roots are the prexisting jobs
```

This line **_is correct and necessary_** during normal/ongoing operation, as well as when resuming a workflow. Yet, it also prevents detection of a workflow root vertex that is marked as a checkpoint yet is not a leaf vertex at the time the workflow is first started. (This case is illustrated in the sample workflow below.)

Both the initial/start-up case and the ongoing/resume case come through this same code path, and in general they can and likely should be handled the same. However, in this particular corner case, it creates an issue.

Two potential approaches come to mind for fixing this:
- Change the current code path to detect the difference between initial and ongoing states and handle them differently.
I don’t like this option because it adds otherwise unnecessary complexity to the common case. It would complicate both the production code and the testing. Also, I have not noticed precedent for this approach in the current code base.

- Leave the current code path as it is, and identify another place in the code that the check for this specific case (initial, root checkpoint that is not a leaf) could be performed.

The second option could be implemented as follows:

Refactor the leaf check into a new method:
```python
    @staticmethod
    def _isLeafVertex(job):
        return len(job._children) == 0 and len(job._followOns) == 0 and len(
            job._services) == 0
```

The existing code is unchanged except for this substitution:
```python
for y in filter(lambda x : x.checkpoint, jobs):
    if y not in roots: # The roots are the prexisting jobs
        if not Job._isLeafVertex(y):
            raise JobGraphDeadlockException("New checkpoint job %s is not a leaf in the job graph" % y)
```

And one if-statement is added to _serializeFirstJob (first statement in the method below)
```python
def _serialiseFirstJob(self, jobStore):
    """
    Serialises the root job. Returns the wrapping job.

    :param toil.jobStores.abstractJobStore.AbstractJobStore jobStore:
    """
    # Special case check for an initial root job that is a checkpoint yet is not a leaf vertex.
    # This is checked for all other jobs by Job.checkNewCheckpointsAreLeafVertices
    if self.checkpoint and not Job._isLeafVertex(self):
        raise JobGraphDeadlockException(
            "Root checkpoint job %s is not a leaf in the job graph" % self)

    # Create first jobGraph
    jobGraph = self._createEmptyJobGraphForJob(jobStore=jobStore, predecessorNumber=0)
    # Write the graph of jobs to disk
    self._serialiseJobGraph(jobGraph, jobStore, None, True)
    jobStore.update(jobGraph)
    # Store the name of the first job in a file in case of restart. Up to this point the
    # root job is not recoverable. FIXME: "root job" or "first job", which one is it?
    jobStore.setRootJob(jobGraph.jobStoreID)
    return jobGraph
```

Although this is a corner case, it could happen and would be a problem for users if it did.
This check is performed for every other job vertex it the workflow, it seems we should do it for the workflow root vertex as well.
I prefer to not have a check for the essentially the same condition in two different locations in the code, yet at least the two locations are in the same class (Job), rather than disparate parts of the code. To me, this seems to be the better of the two options identified.
Do you have thoughts/recommendations for better ways to fix this?

The before and after behavior of this change may be verified with the following simple workflow.
With the current code, the workflow runs to completion.
With the modification above, the exception is raised due to the root checkpoint job that is not a leaf.
```python
from argparse import ArgumentParser
from toil.job import Job

def jobFn(job, value):
    job.fileStore.logToMaster(value)

def createWorkflow():
    rootJob = Job.wrapJobFn(jobFn, "Parent", checkpoint=True) # Checkpoint root non-leaf job
    childJob = rootJob.addChildJobFn(jobFn, "Child")
    grandChildJob = childJob.addChildJobFn(jobFn, "Grandchild")
    return rootJob

def main():
    parser = ArgumentParser()
    Job.Runner.addToilOptions(parser)
    options = parser.parse_args()

    rootJob = createWorkflow()

    Job.Runner.startToil(rootJob, options)

if __name__ == '__main__':
    main()
```


Discussion:

Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -646,7 +646,7 @@ def checkNewCheckpointsAreLeafVertices(self):
         # Check for each job for which checkpoint is true that it is a cut vertex or leaf
         for y in filter(lambda x : x.checkpoint, jobs):
             if y not in roots: # The roots are the prexisting jobs
-                if len(y._children) != 0 and len(y._followOns) != 0 and len(y._services) != 0:
+                if not Job._isLeafVertex(y):
                     raise JobGraphDeadlockException("New checkpoint job %s is not a leaf in the job graph" % y)
 
     def defer(self, function, *args, **kwargs):
@@ -825,6 +825,12 @@ def _addPredecessor(self, predecessorJob):
             raise RuntimeError("The given job is already a predecessor of this job")
         self._directPredecessors.add(predecessorJob)
 
+    @staticmethod
+    def _isLeafVertex(job):
+        return len(job._children) == 0 \
+               and len(job._followOns) == 0 \
+               and len(job._services) == 0
+
     @classmethod
     def _loadUserModule(cls, userModule):
         """
@@ -1180,6 +1186,13 @@ def _serialiseFirstJob(self, jobStore):
 
         :param toil.jobStores.abstractJobStore.AbstractJobStore jobStore:
         """
+
+        # Check if the workflow root is a checkpoint but not a leaf vertex.
+        # All other job vertices in the graph are checked by checkNewCheckpointsAreLeafVertices
+        if self.checkpoint and not Job._isLeafVertex(self):
+            raise JobGraphDeadlockException(
+                'New checkpoint job %s is not a leaf in the job graph' % self)
+
         # Create first jobGraph
         jobGraph = self._createEmptyJobGraphForJob(jobStore=jobStore, predecessorNumber=0)
         # Write the graph of jobs to disk


Test patch:
diff --git a/src/toil/test/src/jobTest.py b/src/toil/test/src/jobTest.py
--- a/src/toil/test/src/jobTest.py
+++ b/src/toil/test/src/jobTest.py
@@ -13,6 +13,7 @@
 # limitations under the License.
 from __future__ import absolute_import, print_function
 import unittest
+import logging
 import os
 import random
 
@@ -25,12 +26,18 @@
 from toil.job import Job, JobGraphDeadlockException, JobFunctionWrappingJob
 from toil.test import ToilTest
 
+logger = logging.getLogger(__name__)
 
 class JobTest(ToilTest):
     """
     Tests testing the job class
     """
 
+    @classmethod
+    def setUpClass(cls):
+        super(JobTest, cls).setUpClass()
+        logging.basicConfig(level=logging.DEBUG)
+
     def testStatic(self):
         """
         Create a DAG of jobs non-dynamically and run it. DAG is:
@@ -254,6 +261,112 @@ def checkFollowOnEdgeCycleDetection(fNode, tNode):
                 and (fNode, tNode) not in childEdges and (fNode, tNode) not in followOnEdges):
                 checkFollowOnEdgeCycleDetection(fNode, tNode)
 
+    def testNewCheckpointIsLeafVertexNonRootCase(self):
+        """
+        Test for issue #1465: Detection of checkpoint jobs that are not leaf vertices
+        identifies leaf vertices incorrectly
+
+        Test verification of new checkpoint jobs being leaf verticies,
+        starting with the following baseline workflow:
+
+            Parent
+              |
+            Child # Checkpoint=True
+
+        """
+
+        def createWorkflow():
+            rootJob = Job.wrapJobFn(simpleJobFn, "Parent")
+            childCheckpointJob = rootJob.addChildJobFn(simpleJobFn, "Child", checkpoint=True)
+            return rootJob, childCheckpointJob
+
+        self.runNewCheckpointIsLeafVertexTest(createWorkflow)
+
+    def testNewCheckpointIsLeafVertexRootCase(self):
+        """
+        Test for issue #1466: Detection of checkpoint jobs that are not leaf vertices
+                              omits the workflow root job
+
+        Test verification of a new checkpoint job being leaf vertex,
+        starting with a baseline workflow of a single, root job:
+
+            Root # Checkpoint=True
+
+        """
+
+        def createWorkflow():
+            rootJob = Job.wrapJobFn(simpleJobFn, "Root", checkpoint=True)
+            return rootJob, rootJob
+
+        self.runNewCheckpointIsLeafVertexTest(createWorkflow)
+
+    def runNewCheckpointIsLeafVertexTest(self, createWorkflowFn):
+        """
+        Test verification that a checkpoint job is a leaf vertex using both
+        valid and invalid cases.
+
+        :param createWorkflowFn: function to create and new workflow and return a tuple of:
+
+                                 0) the workflow root job
+                                 1) a checkpoint job to test within the workflow
+
+        """
+
+        logger.info('Test checkpoint job that is a leaf vertex')
+        self.runCheckpointVertexTest(*createWorkflowFn(),
+                                     expectedException=None)
+
+        logger.info('Test checkpoint job that is not a leaf vertex due to the presence of a service')
+        self.runCheckpointVertexTest(*createWorkflowFn(),
+                                     checkpointJobService=TrivialService("LeafTestService"),
+                                     expectedException=JobGraphDeadlockException)
+
+        logger.info('Test checkpoint job that is not a leaf vertex due to the presence of a child job')
+        self.runCheckpointVertexTest(*createWorkflowFn(),
+                                     checkpointJobChild=Job.wrapJobFn(
+                                         simpleJobFn, "LeafTestChild"),
+                                     expectedException=JobGraphDeadlockException)
+
+        logger.info('Test checkpoint job that is not a leaf vertex due to the presence of a follow-on job')
+        self.runCheckpointVertexTest(*createWorkflowFn(),
+                                     checkpointJobFollowOn=Job.wrapJobFn(
+                                         simpleJobFn,
+                                         "LeafTestFollowOn"),
+                                     expectedException=JobGraphDeadlockException)
+
+    def runCheckpointVertexTest(self,
+                                workflowRootJob,
+                                checkpointJob,
+                                checkpointJobService=None,
+                                checkpointJobChild=None,
+                                checkpointJobFollowOn=None,
+                                expectedException=None):
+        """
+        Modifies the checkpoint job according to the given parameters
+        then runs the workflow, checking for the expected exception, if any.
+        """
+
+        self.assertTrue(checkpointJob.checkpoint)
+
+        if checkpointJobService is not None:
+            checkpointJob.addService(checkpointJobService)
+        if checkpointJobChild is not None:
+            checkpointJob.addChild(checkpointJobChild)
+        if checkpointJobFollowOn is not None:
+            checkpointJob.addFollowOn(checkpointJobFollowOn)
+
+        # Run the workflow and check for the expected behavior
+        options = Job.Runner.getDefaultOptions(self._getTestJobStorePath())
+        options.logLevel = "INFO"
+        if expectedException is None:
+            Job.Runner.startToil(workflowRootJob, options)
+        else:
+            try:
+                Job.Runner.startToil(workflowRootJob, options)
+                self.fail("The expected exception was not thrown")
+            except expectedException as ex:
+                logger.info("The expected exception was thrown: %s", repr(ex))
+
     def testEvaluatingRandomDAG(self):
         """
         Randomly generate test input then check that the job graph can be 
@@ -496,6 +609,9 @@ def cyclic(fNode, visited, stack):
                 return False
         return True
 
+def simpleJobFn(job, value):
+    job.fileStore.logToMaster(value)
+
 def fn1Test(string, outputFile):
     """
     Function appends string to output file, then returns the next ascii character of the first