cylc/cylc — pull request #364 by hjoliver (2013-03-15 12:29:28+00:00)
Title: 215 job submission retry (no polling of submitted tasks yet)

Addresses #215 - job submission retry, handles failure of the submission process itself.

Still to do: polling of successfully submitted tasks to ensure they aren't killed before executing. But this pull request can be merged already if you don't find problems, it is a useful enhancement as is.

1) adds a new task state, "submit-failed"

2) adds job submission retries and associated event handler:

```
[runtime]
[[foo]]
# retry delays = 10 # (the original execution retries)
[[[job submission]]]
retry delays = 0.3 # new job submission retries
[[[event handlers]]]
submission retry = "#..."
```

3) improves gcylc task state color theme handling (see commit message for reasons)

A modified ```examples/task-states``` serves as a test case: a task with the "fail" job submission method retries after a retry event handler broadcasts a new job submission method to the task, which then succeeds.