Change:
Prior to 1.24.0 setting in a Job object () would kill the Job and Pods if they ran longer than the integer specified. I tested today with 1.24.0 and it's not working. The Job defined below (for testing) should be killed 10 seconds, but isn't. It's actually being terminated after about 55 seconds. The Job to be killed after 10 seconds as per . It's worked like this in 1.23.x. On a cluster running 1.24.0 with the following YAML file called . Create the Job. On Kubernetes clusters earlier than 1.24.0 it would fail after 10 seconds and the following command should show in the events 10 seconds after the Pod is created. In K8s 1.24.0 it's terminated after about a minute instead of 10 seconds. It works as expected in 1.23.6 and other point versions of 1.23. It works as expected, and documented, in all versions of 1.23 I've tested on. Docs: <details</details<detailsAll and also local clusters. </details<detailsN/A </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
/sig apps
I just searched for at locally and could not see any logic that compares running time with and marks a pod for termination. Do I need to look somewhere else?
It's here Here's the quote: "Another way to terminate a Job is by setting an active deadline. Do this by setting the field of the Job to a number of seconds. The applies to the duration of the job, no matter how many Pods are created. Once a Job reaches , all of its running Pods are terminated and the Job status will become type: Failed with reason: DeadlineExceeded." I may be misunderstanding it, but I don't think I am as it works the way described in older versions. The Job owns the Pod, you set on the Job to a value, let's say . This will cause the Job to be killed if it's still running after 10 seconds. As the Job owns the Pod(s) it cleans them up as well. This is exactly how it works in previous versions.
The job controller for sure has the code for cleanup on active deadline still in place +1 from me as well. The behavior for sure is not the same on 1.23.x vs 1.24.0 using the same job manifest. And the documentation v/s the behavior is not the same.
Seem to be a side effect of feature gate. Disabling that gives me the old behavior as in 1.23.x You can get the same behavior even in 1.23.x with the feature flag enabled.
The weird bit as you see is the pod actually completes but when it processes the completion event it notices the activeDeadline value and marks deadline Exceeded while the pod actually ran to completion
/assign
I think I have figured out why this happened. It is a side effect of feature gate (thanks ). : when we call with multiple times, the workqueue will only keep the duration that will be triggered nearest from now. (when there is no feature gate or do not enable it): job-controller mainly relies on the first time sync to add the job after its : (unless the field has been changed then it will be add after ). : the feature gate makes to be and it will be add after for the job when a pod of it has been updated. Then the duration which has been after in the first time sync will be overwrite to . And after the next syncJob, there will be no one to addAfter the job again. WDYT
I have came to the same conclusion as you are based on what is in the and . So, I did a dirty prototype of tracking the activeDeadline in an entirely different to avoid conflicts and it seem to be working well. If this is an acceptable change to and I would be more than happy to raise a Draft PR for review along with some additional UT to cover the active deadline cases
Emm... Usually it is not suggested to have multiple active queues in one controller, which will cause an object to be handled by multiple workers at the same time and also make the progress more complicated. My suggestion is do at the end of syncJob every time an unfinished job is reconciled, the duration should be . This is clear for everyone to understand, brings less change and no extra burden on the controller.
I was too trigger happy and did not think about the cases you mentioned above. Thanks for the suggestion of the approach I will took a stab at your suggestion and here is a draft of the changes to get your suggestion working. I did a quick test and that seem to work fine. There were one too many termination point in call and the has no access to the object of the key. So, instead of reading it took a deferred approach as we already had named return parameters in place on the
Other approach is to use the and fetch the job via key if the is not set to This will do an additional fetch but much simpler in change
This is essentially the same as . The solution is actually very simple, but I didn't have time to implement it Can someone take it? /priority critical-urgent /triage accepted
/assign
Let me open a PR. Do you want to just add the item back to queue or retain the check for that I have in the above ?
just add it back to the queue every time there is a job sync. It's a noop if the item is already there.

Discussion:
if feature.DefaultFeatureGate.Enabled(features.JobReadyPods) { ready = pointer.Int32(countReadyPods(activePods)) } <del> // Job first start. Set StartTime and start the ActiveDeadlineSeconds timer // only if the job is not in the suspended state. </del> <ins> // Job first start. Set StartTime only if the job is not in the suspended state. </ins> if job.Status.StartTime == nil && !jobSuspended(&job) { now := metav1.Now() job.Status.StartTime = &now <del> // enqueue a sync to check if job past ActiveDeadlineSeconds if job.Spec.ActiveDeadlineSeconds != nil { klog.V(4).Infof("Job %s has ActiveDeadlineSeconds will sync after %d seconds", key, *job.Spec.ActiveDeadlineSeconds) jm.queue.AddAfter(key, time.Duration(*job.Spec.ActiveDeadlineSeconds)*time.Second) } </del> } var manageJobErr error

finishedCondition = newCondition(batch.JobFailed, v1.ConditionTrue, "BackoffLimitExceeded", "Job has reached the specified backoff limit") } else if pastActiveDeadline(&job) { finishedCondition = newCondition(batch.JobFailed, v1.ConditionTrue, "DeadlineExceeded", "Job was active longer than specified deadline") <ins> } else if job.Spec.ActiveDeadlineSeconds != nil && !jobSuspended(&job) { syncDuration := time.Duration(*job.Spec.ActiveDeadlineSeconds)*time.Second - time.Since(job.Status.StartTime.Time) klog.V(2).InfoS("Job has activeDeadlineSeconds configuration. Will sync this job again", "job", key, "nextSyncIn", syncDuration) jm.queue.AddAfter(key, syncDuration) </ins> } var prevSucceededIndexes, succeededIndexes orderedIntervals

var alwaysReady = func() bool { return true } <del> func newJob(parallelism, completions, backoffLimit int32, completionMode batch.CompletionMode) *batch.Job { </del> <ins> func newJobWithName(name string, parallelism, completions, backoffLimit int32, completionMode batch.CompletionMode) *batch.Job { </ins> j := &batch.Job{ TypeMeta: metav1.TypeMeta{Kind: "Job"}, ObjectMeta: metav1.ObjectMeta{ <del> Name:      "foobar", </del> <ins> Name:      name, </ins> UID:       uuid.NewUUID(), Namespace: metav1.NamespaceDefault, },

return j } <ins> func newJob(parallelism, completions, backoffLimit int32, completionMode batch.CompletionMode) *batch.Job { return newJobWithName("foobar", parallelism, completions, backoffLimit, completionMode) } </ins> func newControllerFromClient(kubeClient clientset.Interface, resyncPeriod controller.ResyncPeriodFunc) (*Controller, informers.SharedInformerFactory) { sharedInformers := informers.NewSharedInformerFactory(kubeClient, resyncPeriod()) jm := NewController(sharedInformers.Core().V1().Pods(), sharedInformers.Batch().V1().Jobs(), kubeClient)

} func TestSyncPastDeadlineJobFinished(t *testing.T) { <del> clientset := clientset.NewForConfigOrDie(&restclient.Config{Host: "", ContentConfig: restclient.ContentConfig{GroupVersion: &schema.GroupVersion{Group: "", Version: "v1"}}}) </del> <ins> clientset := fake.NewSimpleClientset() </ins> manager, sharedInformerFactory := newControllerFromClient(clientset, controller.NoResyncPeriodFunc) <del> fakePodControl := controller.FakePodControl{} manager.podControl = &fakePodControl </del> manager.podStoreSynced = alwaysReady manager.jobStoreSynced = alwaysReady <del> var actual *batch.Job manager.updateStatusHandler = func(ctx context.Context, job *batch.Job) (*batch.Job, error) { actual = job return job, nil } job := newJob(1, 1, 6, batch.NonIndexedCompletion) activeDeadlineSeconds := int64(10) job.Spec.ActiveDeadlineSeconds = &activeDeadlineSeconds start := metav1.Unix(metav1.Now().Time.Unix()-15, 0) job.Status.StartTime = &start job.Status.Conditions = append(job.Status.Conditions, *newCondition(batch.JobFailed, v1.ConditionTrue, "DeadlineExceeded", "Job was active longer than specified deadline")) sharedInformerFactory.Batch().V1().Jobs().Informer().GetIndexer().Add(job) forget, err := manager.syncJob(context.TODO(), testutil.GetKey(job, t)) if err != nil { t.Errorf("Unexpected error when syncing jobs %v", err) } if !forget { t.Errorf("Unexpected forget value. Expected %v, saw %vn", true, forget) } if len(fakePodControl.Templates) != 0 { t.Errorf("Unexpected number of creates.  Expected %d, saw %dn", 0, len(fakePodControl.Templates)) } if len(fakePodControl.DeletePodName) != 0 { t.Errorf("Unexpected number of deletes.  Expected %d, saw %dn", 0, len(fakePodControl.DeletePodName)) </del> <ins> ctx, cancel := context.WithCancel(context.Background()) defer cancel() sharedInformerFactory.Start(ctx.Done()) go manager.Run(ctx, 1) tests := []struct { name         string setStartTime bool jobName      string }{ { name:         "New job created without start time being set", setStartTime: false, jobName:      "job1", }, { name:         "New job created with start time being set", setStartTime: true, jobName:      "job2", }, </ins> } <del> if actual != nil { t.Error("Unexpected job modification") </del> <ins> for _, tc := range tests { t.Run(tc.name, func(t *testing.T) { job := newJobWithName(tc.jobName, 1, 1, 6, batch.NonIndexedCompletion) activeDeadlineSeconds := int64(1) job.Spec.ActiveDeadlineSeconds = &activeDeadlineSeconds if tc.setStartTime { start := metav1.Unix(metav1.Now().Time.Unix()-1, 0) job.Status.StartTime = &start } _, err := clientset.BatchV1().Jobs(job.GetNamespace()).Create(ctx, job, metav1.CreateOptions{}) if err != nil { t.Errorf("Could not create Job: %v", err) } if err := sharedInformerFactory.Batch().V1().Jobs().Informer().GetIndexer().Add(job); err != nil { t.Fatalf("Failed to insert job in index: %v", err) } var j *batch.Job err = wait.Poll(200*time.Millisecond, 3*time.Second, func() (done bool, err error) { j, err = clientset.BatchV1().Jobs(metav1.NamespaceDefault).Get(ctx, job.GetName(), metav1.GetOptions{}) if err != nil { return false, nil } if len(j.Status.Conditions) == 1 && j.Status.Conditions[0].Reason == "DeadlineExceeded" { return true, nil } return false, nil }) if err != nil { t.Errorf("Job failed to enforce activeDeadlineSeconds configuration. Expected condition with Reason 'DeadlineExceeded' was not found in %v", j.Status) } }) </ins> } }