Repository: DataBiosphere/toil

Issue:
Increase default retry count
Any thoughts on this? The current default retry count is 0, which means that a task won't be rerun on failure unless the user overrides this in their app/on the command line. I don't think that's a great default, since its pretty common that people run on preemptable resources (we use resource preemption on our internal cluster, and often run on spot instances). Other systems use higher defaults, e.g., Spark defaults to retrying a task 4 times.

This would just entail a configuration change.


Discussion:
I think that, on the cloud at least, a default of 0 makes sense since retrying a particularly resource intensive job might end up costing hundreds to restart a process that seems likely to fail again.  My opinion is that, when it's a novel error, the user should look at the error before deciding on a restart, and when it's a known error, the user will know and be able to set this manually beforehand.
Can we default to 1 on batch systems?

2017-10-22 6:21 GMT+03:00 DailyDreaming <notifications@github.com>:

> I think that, on the cloud at least, a default of 0 makes sense since
> retrying a particularly resource intensive job might end up costing
> hundreds to restart a process that seems likely to fail again. My opinion
> is that, when it's a novel error, the user should look at the error before
> deciding on a restart, and when it's a known error, the user will know and
> be able to set this manually beforehand.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/BD2KGenomics/toil/issues/1933#issuecomment-338448167>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABROCKz_OlYuPRc-O8PgqmWrU7XZ1pd_ks5surTAgaJpZM4QBuzD>
> .
>



-- 
Michael R. Crusoe
Co-founder & Lead,
Common Workflow Language project <http://www.commonwl.org/>
https://impactstory.org/u/0000-0002-2961-9670
mrc@commonwl.org
+1 480 627 9108

When using spot instances, a retryCount of 5+ is almost necessary, because Toil counts spot instance failures as failures of the tasks running on it. (IMO this shouldn't be the case, but there are bigger issues to fix.)

I change the default retry count to 5 for my pipeline, but that default might not be appropriate for everyone.
I agree that spot instance failure should be handled separately, and with a bit more thought.  But yes, there are bigger issues right now.

I'll bring this up tomorrow when people are in the office.
Is the consensus 1 then? Any objections?

Resolving patch:
diff --git a/src/toil/common.py b/src/toil/common.py
--- a/src/toil/common.py
+++ b/src/toil/common.py
@@ -112,7 +112,7 @@ def __init__(self):
         self.maxDisk = sys.maxsize
 
         #Retrying/rescuing jobs
-        self.retryCount = 0
+        self.retryCount = 1
         self.maxJobDuration = sys.maxsize
         self.rescueJobsFrequency = 3600
 
@@ -245,7 +245,7 @@ def parseIntList(s):
         setOption("defaultPreemptable")
 
         #Retrying/rescuing jobs
-        setOption("retryCount", int, iC(0))
+        setOption("retryCount", int, iC(1))
         setOption("maxJobDuration", int, iC(1))
         setOption("rescueJobsFrequency", int, iC(1))
 


Test patch:
diff --git a/src/toil/test/src/resumabilityTest.py b/src/toil/test/src/resumabilityTest.py
--- a/src/toil/test/src/resumabilityTest.py
+++ b/src/toil/test/src/resumabilityTest.py
@@ -35,6 +35,7 @@ def test(self):
         """
         options = Job.Runner.getDefaultOptions(self._getTestJobStorePath())
         options.logLevel = "INFO"
+        options.retryCount = 0
         root = Job.wrapJobFn(parent)
         with self.assertRaises(FailedJobsException):
             # This one is intended to fail.