Change:
This issue was born out of the discussion from and In short: Job controller has an exponential backoff in place when failures happen to a job. As long as the parallelism is one, it's clear when a failure or success happens, the latter clears the backoff. The problem is when parallelism 1, which can lead to situation where the controller can observe successful and failed pod at the same time and won't clear the backoff leading to further retries and longer wait times (see for extensive details about the problem in e2e) Possible solutions: everything as is both success and failure are observed clear the backoff Pros and cons: Ad. 1. Aside from () I wonder if users got affected by it. Does it make sense to fix it at all? Ad. 2. This approach make sense in some situation, but the question arise, should that be controlled by a user. What if I have 9 failures and 1 success (parallelism = 10), should I still clear the backoff? Maybe we should introduce user controller field to set the failure rate in this case.
/cc
(jumping here 'cause forwarded me this). I personally think that the right approach is to reset the back-off as soon we have a successful pod run to complete. Back-off was introduced, at least in our case, to reduce the impact of bad configured PodSpec in Jobs. If in your run you do have some success, it means you've a good config and you just keep trying (not that different that what we did before ). I don't think it's worth asking user finer tune config. is already complex enough. My 2 cents.
cc
I agree with any success should immediately clear the backoff flag, since it shows the job is configured properly and the problem is only temporary. I'll assign it to myself, but anyone is open to submit a PR fixing this.
Hi I started to implement the pod success check in order to clear the backoff delay in case of at least one of the pods succeed. let me know if you are fine with this approach, and then I will go further with unit-tests and end2end tests.
is the rest of the work on this issue getting done for 1.10? Please update, and add approved-for-milestone if this is on track for 1.10.
I updated my change with missing unit-tests and created the PR:

Discussion:
} forget := false <ins> // Check if the number of jobs succeeded increased since the last check. If yes "forget" should be true // This logic is linked to the issue: https://github.com/kubernetes/kubernetes/issues/56853 that aims to // improve the Job backoff policy when parallelism > 1 and few Jobs failed but others succeed. // In this case, we should clear the backoff delay. if job.Status.Succeeded < succeeded { forget = true } </ins> // no need to update the job if the status hasn't changed since last time if job.Status.Active != active || job.Status.Succeeded != succeeded || job.Status.Failed != failed || len(job.Status.Conditions) != conditions { job.Status.Active = active

job.Status.Failed = failed if err := jm.updateHandler(&job); err != nil { <del> return false, err </del> <ins> return forget, err </ins> } if jobHaveNewFailure && !IsJobFinished(&job) { // returning an error will re-enqueue Job after the backoff period <del> return false, fmt.Errorf("failed pod(s) detected for job key %q", key) </del> <ins> return forget, fmt.Errorf("failed pod(s) detected for job key %q", key) </ins> } forget = true

fmt.Errorf("Fake error"), true, 0, 3, 0, 0, 0, 1, 3, 0, 0, nil, "", }, <del> "failed pod": { </del> <ins> "failed + succeed pods: reset backoff delay": { </ins> 2, 5, 6, false, 0, <del> fmt.Errorf("Fake error"), false, 0, 1, 1, 1, </del> <ins> fmt.Errorf("Fake error"), true, 0, 1, 1, 1, </ins> 1, 0, 1, 1, 1, nil, "", }, <ins> "only new failed pod": { 2, 5, 6, false, 0, fmt.Errorf("Fake error"), false, 0, 1, 0, 1, 1, 0, 1, 0, 1, nil, "", }, </ins> "job finish": { 2, 5, 6, false, 0, nil, true, 0, 0, 5, 0,