hjoliver/cylc — Issue #29: Retry on failure
Author: dpmatthews
Created: 2012-02-07 09:02:36+00:00
State: open

Cylc currently supports task retry on failure if you specify a list for "command scripting". This seems to work fine but the interface isn't ideal if your command is multi-line and you just want to repeat the same command (or a minor variant of it). There is also no support for delaying the re-try which is a feature that is widely used in our current system.

For example, there are certain tasks in our trial suites which, experience has shown, will occasionally fail but will often succeed if re-submitted after a suitable delay. This is typically due to a short outage of a service such as an archiving or database system. This is not currently straightforward to configure in cylc (other than via failure triggers which are extremely powerful but require work to set up and are probably overkill for this particular case).

I'd like to propose the following changes to task re-tries:

1) "command scripting" is always a single string (not a list).

2) "retry delay" specifies the delay in minutes before each retry. For example, "retry delay = 0, 10, 60" implies the first retry is submitted immediately, the second following a 10 min delay and the third following a 1 hr delay (i.e. total of 4 attempts).

3) An environment variable CYLC_TRY_NUM (or CYLC_TASK_TRY ?) is defined which can be used to alter behaviour on retry.
1 => initial attempt, 2 => first retry, etc.
Note that it would be good if this variable gets incremented for manual retries as well. For instance you might configure your suite such that extra diagnostic output is enabled for particular tasks if they have to be re-run.