Repository: DataBiosphere/toil

Issue:
Checkpoint jobs are considered "totally failed" after a successful restart
I can consistently get failures if I do the following:
1. Run a workflow containing a checkpointed subtree with a retry count of 0, wait for the workflow to get to the checkpointed subtree, and kill the workflow (causing some job failures in the checkpointed subtree)
2. Restart the workflow

On step 2, the workflow completes the checkpointed subtree fine, but then bails out because it considers the checkpointed job to be totally failed. But restarting the workflow after that works fine, and runs to completion.

If it matters, the checkpointed job in question has an associated service. I think this is probably something wrong with the "cleanup" of the jobStore when resuming a workflow.


Discussion:
You can assign to me - this is definitely my bug.

On Tue, Mar 28, 2017 at 3:19 PM, Joel Armstrong <notifications@github.com>
wrote:

> I can consistently get failures if I do the following:
>
>    1. Run a workflow containing a checkpointed subtree with a retry count
>    of 0, wait for the workflow to get to the checkpointed subtree, and kill
>    the workflow (causing some job failures in the checkpointed subtree)
>    2. Restart the workflow
>
> On step 2, the workflow completes the checkpointed subtree fine, but then
> bails out because it considers the checkpointed job to be totally failed.
> But restarting the workflow after that works fine, and runs to completion.
>
> If it matters, the checkpointed job in question has an associated service.
> I think this is probably something wrong with the "cleanup" of the jobStore
> when resuming a workflow.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/BD2KGenomics/toil/issues/1598>, or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAkOIDOkBl7mj7plGr8WJGEqxoxyRoPWks5rqYd0gaJpZM4MsREQ>
> .
>

Well, regardless of who it's assigned to, I have a minimal example which should make things a bit easier. That minimal example actually revealed another bug: it looks like checkpointed subtrees are restarted even if the retry count is set to zero. But obviously that's a little less serious.

@cket, would it be make sense if I made a PR for the unit test I have that reproduces this, maybe marking it xfail? I think the bug(s) are probably easy to find given a reproducible example, but I don't think I'll have time to get around to fixing it for a while, since it's not really a blocker.
A PR with a unit test would be great! You don't have to mark it expected fail since we will want it to be passing before we merge it in.

Resolving patch:
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -260,7 +260,13 @@ def innerLoop(self):
                             continue
 
                         # If the job is a checkpoint and has remaining retries then reissue it.
-                        elif jobGraph.checkpoint is not None and jobGraph.remainingRetryCount > 0:
+                        # The logic behind using > 1 rather than > 0 here: Since this job has
+                        # been tried once (without decreasing its retry count as the job
+                        # itself was successful), and its subtree failed, it shouldn't be retried
+                        # unless it has more than 1 try.
+                        elif jobGraph.checkpoint is not None and jobGraph.remainingRetryCount > 1:
+                            jobGraph.setupJobAfterFailure(self.config)
+                            self.jobStore.update(jobGraph)
                             logger.warn('Job: %s is being restarted as a checkpoint after the total '
                                         'failure of jobs in its subtree.', jobGraph.jobStoreID)
                             self.issueJob(JobNode.fromJobGraph(jobGraph))
@@ -905,8 +911,9 @@ def _updatePredecessorStatus(self, jobStoreID):
                     # Remove it from the set of jobs with active successors
                     self.toilState.successorCounts.pop(predecessorJob.jobStoreID)
 
-                    # Pop stack at this point, as we can get rid of its successors
-                    predecessorJob.stack.pop()
+                    if predecessorJob.jobStoreID not in self.toilState.hasFailedSuccessors:
+                        # Pop stack at this point, as we can get rid of its successors
+                        predecessorJob.stack.pop()
 
                     # Now we know the job is done we can add it to the list of updated job files
                     assert predecessorJob not in self.toilState.updatedJobs
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -259,10 +259,6 @@ def badWorker():
                 else:
                     jobGraph.command = jobGraph.checkpoint
 
-                # Reduce the retry count
-                assert jobGraph.remainingRetryCount >= 0
-                jobGraph.remainingRetryCount = max(0, jobGraph.remainingRetryCount - 1)
-
                 jobStore.update(jobGraph) # Update immediately to ensure that checkpoint
                 # is made before deleting any remaining successors
 


Test patch:
diff --git a/src/toil/test/src/checkpointTest.py b/src/toil/test/src/checkpointTest.py
new file mode 100644
--- /dev/null
+++ b/src/toil/test/src/checkpointTest.py
@@ -0,0 +1,102 @@
+# Copyright (C) 2015-2017 Regents of the University of California
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import absolute_import
+from toil.job import Job
+from toil.test import ToilTest
+from toil.leader import FailedJobsException
+from toil.jobStores.abstractJobStore import NoSuchFileException
+
+class CheckpointTest(ToilTest):
+    def testCheckpointNotRetried(self):
+        """A checkpoint job with a retryCount of 0 should not be retried."""
+        options = Job.Runner.getDefaultOptions(self._getTestJobStorePath())
+        options.retryCount = 0
+        # We set the workflow to *succeed* if the checkpointed job is
+        # retried despite the retry count of 0. So, if the workflow
+        # fails, this test succeeds.
+        # This ends up being a bit confusing, but it's the easiest way
+        # to check that the job wasn't retried short of parsing the
+        # log.
+        with self.assertRaises(FailedJobsException):
+            Job.Runner.startToil(CheckRetryCount(numFailuresBeforeSuccess=1), options)
+
+    def testCheckpointRetriedOnce(self):
+        """A checkpoint job with a retryCount of 1 should be retried exactly once."""
+        options = Job.Runner.getDefaultOptions(self._getTestJobStorePath())
+        options.retryCount = 1
+
+        # This should succeed (the checkpoint should be retried once, then succeed)
+        try:
+            Job.Runner.startToil(CheckRetryCount(numFailuresBeforeSuccess=1), options)
+        except FailedJobsException:
+            self.fail("The checkpoint job wasn't retried enough times.")
+
+        # This should fail (the checkpoint should be retried once, then fail again)
+        with self.assertRaises(FailedJobsException):
+            Job.Runner.startToil(CheckRetryCount(numFailuresBeforeSuccess=2), options)
+
+    def testCheckpointedRestartSucceeds(self):
+        """A checkpointed job should succeed on restart of a failed run if its child job succeeds."""
+        options = Job.Runner.getDefaultOptions(self._getTestJobStorePath())
+        # The child job should fail the first time
+        with self.assertRaises(FailedJobsException):
+            Job.Runner.startToil(CheckpointFailsFirstTime(), options)
+
+        # The second time, everything should work
+        options.restart = True
+        try:
+            Job.Runner.startToil(CheckpointFailsFirstTime(), options)
+        except FailedJobsException:
+            self.fail("Checkpointed workflow restart doesn't clean failures.")
+
+class CheckRetryCount(Job):
+    """Fail N times, succeed on the next try."""
+    def __init__(self, numFailuresBeforeSuccess):
+        super(CheckRetryCount, self).__init__(checkpoint=True)
+        self.numFailuresBeforeSuccess = numFailuresBeforeSuccess
+
+    def getNumRetries(self, fileStore):
+        """Mark a retry in the fileStore, and return the number of retries so far."""
+        try:
+            with fileStore.jobStore.readSharedFileStream("checkpointRun") as f:
+                timesRun = int(f.read())
+        except NoSuchFileException:
+            timesRun = 0
+        with fileStore.jobStore.writeSharedFileStream("checkpointRun") as f:
+            f.write(str(timesRun + 1))
+        return timesRun
+
+    def run(self, fileStore):
+        retryCount = self.getNumRetries(fileStore)
+        fileStore.logToMaster(str(retryCount))
+        if retryCount < self.numFailuresBeforeSuccess:
+            self.addChild(AlwaysFail())
+
+class AlwaysFail(Job):
+    def run(self, fileStore):
+        raise RuntimeError(":(")
+
+class CheckpointFailsFirstTime(Job):
+    def __init__(self):
+        super(CheckpointFailsFirstTime, self).__init__(checkpoint=True)
+
+    def run(self, fileStore):
+        self.addChild(FailOnce())
+
+class FailOnce(Job):
+    """Fail the first time the workflow is run, but succeed thereafter."""
+    def run(self, fileStore):
+        if fileStore.jobStore.config.workflowAttemptNumber < 1:
+            raise RuntimeError("first time around")