Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a job with and , the pod with the configuration included below continued to restart and was never marked as failed: What you expected to happen: We expected only 2 attempted pod restarts before the job was marked as failed. The pod kept restarting and was never set to failed, remained active. How to reproduce it (as minimally and precisely as possible): Create the above job and observe the number of pod restarts. Anything else we need to know?: The flag works as expected when . Environment: Kubernetes version (use ): 1.8.0 using minikube
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think this is because for , kubelet will try to restart failed pod in place, and the pod status will be even the container is failed to run, so jobcontroller get a illusion as the pod is still running. Here's my naive thoughts: for short-term to avoid this, add a , for long-term we need consider how to recognize and count this kind of Pod as failed pod. cc
Just ran into the issue as well. It might be useful to amend the documentation to warn about the fact that won't be effective if is chosen as restart policy. At least, it wasn't obvious to me and I didn't catch any relevant note in the docs. Happy to do a quick PR if that's a desirable short-term action. For Pods failing as reported in this issue, what's the difference in behavior between a restart policy and a hypothetical scenario where is respected by ? Anything other than failing Pods being replaced by newly created ones in the former case vs. existing Pods being restarted in the latter?
For OnFailure, the failed Pod will restart on the same kubelet host, so emptyDir/hostPath is still there, for Never the failed Pod got delete and a replacement will create on another(may be the same) kubelet host, if the Pod use emptyDir/hostPath to store some tmp data, the tmp data may be lost. For Pod don't use emptyDir/hostPath, I think there's no much difference for usage( will involve controller, apiserver, scheduler to do some work as it needs to create a replacement, backoffLimit policy will avoid bad jobSpec & container cause to create too many replacement consume control plane bandwidth).
Might it be a reasonable idea to move the parameter into the Pod? I'd think it could prove valuable to have an upper bound on the restarts on Pods themselves independently of Jobs, and it may solve the "how does the Cronjob controller determine that the limit has been reached" problem more elegantly if the Pod's state automatically transitions to failed once the limit is hit. WDYT?
can anyone tell me if I set the Never and the back off to 3 will it try a restart 3 times? this is very confusing
yes, it will. In fact, setting it to is the only way right now to have a back off limit be applied and effective.
I should mention that my statement is based on Kubernetes 1.8. I don't know if any fixes possibly made it into the upcoming 1.9. Then again, if there were, this issue should have been updated or at least referenced.
this is not working for a cronjob job spec (i am running 1.8.6) in the picture i still get new pods when i kill the previous on (5 created 3 should have been created and the job should have failed) any idea how to get this to actually work? !
what's your spec's ? I don't see it in your manifest.
every minute just for testing
Aren't you risking then that restarting jobs from a failed schedule and those from a succeeding schedule might be overlapping? How does it look like when you set the interval to, say, 10 minutes?
same result with 10 minutes interval
I ran a simplified version of your manifest against my Kubernetes 1.8.2 cluster with the command field replaced by so that every run fails, the schedule set to every 5 minutes, and the back-off limit reduced to 2. Here's my manifest: Initially, I get a series of three failing pods: Then it takes pretty much 5 minutes for the next batch to start up and fail: Interestingly, I only get two new containers instead of three. The next batch though comes with three again: and so does the final batch I tested: Apart from the supposed off-by-one error, things seem to work for me.
Maybe you can double-check using my manifest?
In the we've stated it should work for both restart policies, I've addressed the problem in It counts pod's restartCount, if overall (iow. all containers in all pods) number exceeds backoff limit the job will fail.
I think the backoffLimit doesn't work at all. My kubernetes version is 1.10.3
See .
NAME READY STATUS RESTARTS AGE job-backofflimit-2-9n74x 0/1 Error 0 34s job-backofflimit-2-bt796 0/1 Error 0 1m job-backofflimit-2-fgktc 0/1 Error 0 49s job-backofflimit-2-gfhlx 0/1 Error 0 1m job-backofflimit-2-gzd54 0/1 ContainerCreating 0 8s job-backofflimit-2-ljlpx 0/1 Error 0 1m job-backofflimit-2-p5fxm 0/1 Error 0 1m job-backofflimit-2-w6s5d 0/1 Error 0 21s apiVersion: batch/v1 kind: Job metadata: name: job-backofflimit-2 spec: backoffLimit: 2 template: spec: containers: name: job-with-python2 image: restartPolicy: Never v1.10.4
9 has the same problem. My cluster is 1.9.6, and i run into the same issue

Discussion:
var failureMessage string jobHaveNewFailure := failed > job.Status.Failed <ins> exceedsBackoffLimit := jobHaveNewFailure && (int32(previousRetry)+1 > *job.Spec.BackoffLimit) </ins> <del> // check if the number of failed jobs increased since the last syncJob if jobHaveNewFailure && (int32(previousRetry)+1 > *job.Spec.BackoffLimit) { </del> <ins> if exceedsBackoffLimit || pastBackoffLimitOnFailure(&job, pods) { // check if the number of pod restart exceeds backoff (for restart OnFailure only) // OR if the number of failed jobs increased since the last syncJob </ins> jobFailed = true failureReason = "BackoffLimitExceeded" <del> failureMessage = "Job has reach the specified backoff limit" </del> <ins> failureMessage = "Job has reached the specified backoff limit" </ins> } else if pastActiveDeadline(&job) { jobFailed = true failureReason = "DeadlineExceeded"

wait.Wait() } <ins> // pastBackoffLimitOnFailure checks if container restartCounts sum exceeds BackoffLimit // this method applies only to pods with restartPolicy == OnFailure func pastBackoffLimitOnFailure(job *batch.Job, pods []*v1.Pod) bool { if job.Spec.Template.Spec.RestartPolicy != v1.RestartPolicyOnFailure { return false } result := int32(0) for i := range pods { po := pods[i] if po.Status.Phase != v1.PodRunning { continue } for j := range po.Status.InitContainerStatuses { stat := po.Status.InitContainerStatuses[j] result += stat.RestartCount } for j := range po.Status.ContainerStatuses { stat := po.Status.ContainerStatuses[j] result += stat.RestartCount } } return result >= *job.Spec.BackoffLimit } </ins> // pastActiveDeadline checks if job has ActiveDeadlineSeconds field set and if it is exceeded. func pastActiveDeadline(job *batch.Job) bool { if job.Spec.ActiveDeadlineSeconds == nil || job.Status.StartTime == nil {

nil, true, 0, 0, 0, 0, 10, 0, 10, 0, 0, nil, "", }, <del> "to many job sync failure": { </del> <ins> "too many job failures": { </ins> 2, 5, 0, true, 0, nil, true, 0, 0, 0, 1, 0, 0, 0, 0, 1, &jobConditionFailed, "BackoffLimitExceeded",

}) } } <ins> func TestJobBackoffForOnFailure(t *testing.T) { jobConditionFailed := batch.JobFailed testCases := map[string]struct { // job setup parallelism  int32 completions  int32 backoffLimit int32 // pod setup jobKeyForget  bool restartCounts []int32 // expectations expectedActive          int32 expectedSucceeded       int32 expectedFailed          int32 expectedCondition       *batch.JobConditionType expectedConditionReason string }{ "too many job failures - single pod": { 1, 5, 2, true, []int32{2}, 0, 0, 1, &jobConditionFailed, "BackoffLimitExceeded", }, "too many job failures - multiple pods": { 2, 5, 2, true, []int32{1, 1}, 0, 0, 2, &jobConditionFailed, "BackoffLimitExceeded", }, "not enough failures": { 2, 5, 3, true, []int32{1, 1}, 2, 0, 0, nil, "", }, } for name, tc := range testCases { t.Run(name, func(t *testing.T) { // job manager setup clientset := clientset.NewForConfigOrDie(&restclient.Config{Host: "", ContentConfig: restclient.ContentConfig{GroupVersion: &legacyscheme.Registry.GroupOrDie(v1.GroupName).GroupVersion}}) manager, sharedInformerFactory := newJobControllerFromClient(clientset, controller.NoResyncPeriodFunc) fakePodControl := controller.FakePodControl{} manager.podControl = &fakePodControl manager.podStoreSynced = alwaysReady manager.jobStoreSynced = alwaysReady var actual *batch.Job manager.updateHandler = func(job *batch.Job) error { actual = job return nil } // job & pods setup job := newJob(tc.parallelism, tc.completions, tc.backoffLimit) job.Spec.Template.Spec.RestartPolicy = v1.RestartPolicyOnFailure sharedInformerFactory.Batch().V1().Jobs().Informer().GetIndexer().Add(job) podIndexer := sharedInformerFactory.Core().V1().Pods().Informer().GetIndexer() for i, pod := range newPodList(int32(len(tc.restartCounts)), v1.PodRunning, job) { pod.Status.ContainerStatuses = []v1.ContainerStatus{{RestartCount: tc.restartCounts[i]}} podIndexer.Add(&pod) } // run forget, err := manager.syncJob(getKey(job, t)) if err != nil { t.Errorf("unexpected error syncing job.  Got %#v", err) } if forget != tc.jobKeyForget { t.Errorf("unexpected forget value. Expected %v, saw %vn", tc.jobKeyForget, forget) } // validate status if actual.Status.Active != tc.expectedActive { t.Errorf("unexpected number of active pods.  Expected %d, saw %dn", tc.expectedActive, actual.Status.Active) } if actual.Status.Succeeded != tc.expectedSucceeded { t.Errorf("unexpected number of succeeded pods.  Expected %d, saw %dn", tc.expectedSucceeded, actual.Status.Succeeded) } if actual.Status.Failed != tc.expectedFailed { t.Errorf("unexpected number of failed pods.  Expected %d, saw %dn", tc.expectedFailed, actual.Status.Failed) } // validate conditions if tc.expectedCondition != nil && !getCondition(actual, *tc.expectedCondition, tc.expectedConditionReason) { t.Errorf("expected completion condition.  Got %#v", actual.Status.Conditions) } }) } } </ins>