Repository: DataBiosphere/toil

Issue:
Incorrect accounting of failed jobs
I noticed that tasks started to fail on fewer retries than the value set by `--retryCount`  when I synced to head. This is because `worker.py` also [decrements the retry count when the job fails](https://github.com/DataBiosphere/toil/blob/65e61cc54dfa75995919d8212ac5bc042b70294e/src/toil/worker.py#L481). This had always been the case, but before PR #3052, the failed tasks actually exited with `0` (i.e. were successful as far as the leader was concerned), and so the leader would not decrement the retry count, and the actual retry happened b/c the job was still in the jobstore and the check [here](https://github.com/DataBiosphere/toil/blob/1ae5547fe93c73f68e3eb7f5d3dd05aeceb413ab/src/toil/leader.py#L892) would return true.

Moreover, I noticed that the Grafana dashboard shows any failed tasks as not being completed in the "running or queued jobs" graph, which results in ever-increasing tasks in the "queue". This is b/c previously, the `exitStatus` was zero, so [this code](https://github.com/DataBiosphere/toil/blob/1ae5547fe93c73f68e3eb7f5d3dd05aeceb413ab/src/toil/leader.py#L520) actually got called (the graph is `issued_jobs - completed_jobs`). With PR #3052, the exit status is no longer zero, so we never mark the failed job as being "completed".

Fix proposals:
* Remove the line [here](https://github.com/DataBiosphere/toil/blob/65e61cc54dfa75995919d8212ac5bc042b70294e/src/toil/worker.py#L481) so that the worker no longer decrements the retry count (actually, I don't think we need any of that logic other than setting `jobAttemptFailed = True`)
* Always call `logCompletedJob` in `_gatherUpdatedJobs` regardless of exit status. This is ok since any completely failed jobs are still "completed" and any transient failures will be retried, so they'll appear again in the issued queue.

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-580)
┆Issue Number: TOIL-580



Discussion:

Resolving patch:
diff --git a/src/toil/job.py b/src/toil/job.py
--- a/src/toil/job.py
+++ b/src/toil/job.py
@@ -313,7 +313,6 @@ def __init__(self, memory=None, cores=None, disk=None, preemptable=None,
         self._fileStore = None
         self._defer = None
         self._tempDir = None
-        self._succeeded = True
 
     def run(self, fileStore):
         """
diff --git a/src/toil/leader.py b/src/toil/leader.py
--- a/src/toil/leader.py
+++ b/src/toil/leader.py
@@ -517,11 +517,11 @@ def _gatherUpdatedJobs(self, updatedJobTuple):
                 cur_logger = (logger.debug if str(updatedJob.jobName).startswith(CWL_INTERNAL_JOBS)
                               else logger.info)
                 cur_logger('Job ended: %s', updatedJob)
-                if self.toilMetrics:
-                    self.toilMetrics.logCompletedJob(updatedJob)
             else:
                 logger.warning('Job failed with exit value %i: %s',
                                exitStatus, updatedJob)
+            if self.toilMetrics:
+                self.toilMetrics.logCompletedJob(updatedJob)
             self.processFinishedJob(jobID, exitStatus, wallTime=wallTime, exitReason=exitReason)
 
     def _processLostJobs(self):
diff --git a/src/toil/worker.py b/src/toil/worker.py
--- a/src/toil/worker.py
+++ b/src/toil/worker.py
@@ -478,10 +478,7 @@ def workerScript(jobStore, config, jobName, jobStoreID, redirectOutputToLogFile=
     
     if AbstractFileStore._terminateEvent.isSet():
         jobGraph = jobStore.load(jobStoreID)
-        jobGraph.setupJobAfterFailure(config)
         jobAttemptFailed = True
-        if job and jobGraph.remainingRetryCount == 0:
-            job._succeeded = False
 
     ##########################################
     #Cleanup


Test patch: