Change:
When a Job is deleted (via Background cascade), we have a backup logic to remove finalizers from pods This logic could also be applied for any finished Job. While investigating a case similar to , we observed that the job controller didn't delete finalizers from Pods where the Job was deleted with Foreground cascading policy. Since the Job was already marked as Finished, it only makes sense to trigger the backup logic to remove finalizers, so that we can guard against any possible bug where a Job is marked as Finished but not all pod finalizers are removed. Note that I'm not aware of any other bug in tracking with finalizers.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/remove-kind feature /kind cleanup
/wg batch
I am happy to give this a go.
/assign
Just to check if I have the logic chain right: We are talking about adding this logic If so, I should probably turn the backup-logic code into a function that then apply that in both places? I also would be unsure of where in the unit tests this would be tested, are you able to point me to how this is tested in the deletePod function? I assume we add a pod that is never included in the "uncounted" lists.
I would put it in the event handler, to be more of a "catch all" Yes, that's a good idea. I think you would have to force it quite a bit, as we have fixed all the bugs that we have found. For example, from the test: the controller, but do not start the workers (this will setup the event handlers) a job the pods the job the workers some inspiration here
Alright, thank you :))
Should the issue title be: Remove finalizers from Pods belonging to finished Jobs ?
If someone has set a custom finalizer on a Pod, even one owned by a Job, (when) would we remove that finalizer?
/retitle Remove tracking finalizers from Pods belonging to finished Jobs No, only the finalizer specific to job tracking

Discussion:
} else { // Trigger immediate sync when spec is changed. jm.enqueueSyncJobImmediately(logger, curJob) <ins> } // if curJob is finished, remove the finalizer as a backup check. if IsJobFinished(curJob) { jm.backupRemovePodFinalizers(curJob) </ins> } // check if need to add a new rsync for ActiveDeadlineSeconds

return } } <del> // Listing pods shouldn't really fail, as we are just querying the informer cache. selector, err := metav1.LabelSelectorAsSelector(jobObj.Spec.Selector) if err != nil { utilruntime.HandleError(fmt.Errorf("parsing deleted job selector: %v", err)) return } pods, _ := jm.podStore.Pods(jobObj.Namespace).List(selector) for _, pod := range pods { if metav1.IsControlledBy(pod, jobObj) && hasJobTrackingFinalizer(pod) { jm.enqueueOrphanPod(pod) } } </del> <ins> jm.backupRemovePodFinalizers(jobObj) </ins> } // enqueueSyncJobImmediately tells the Job controller to invoke syncJob

} return feature.DefaultFeatureGate.Enabled(features.JobPodFailurePolicy) && job.Spec.PodFailurePolicy != nil } <ins> func (jm *Controller) backupRemovePodFinalizers(job *batch.Job) { // Listing pods shouldn't really fail, as we are just querying the informer cache. selector, err := metav1.LabelSelectorAsSelector(job.Spec.Selector) if err != nil { utilruntime.HandleError(fmt.Errorf("parsing deleted job selector: %v", err)) return } pods, _ := jm.podStore.Pods(job.Namespace).List(selector) for _, pod := range pods { if metav1.IsControlledBy(pod, job) && hasJobTrackingFinalizer(pod) { jm.enqueueOrphanPod(pod) } } } </ins>

} } <ins> func TestBackupFinalizerRemoval(t *testing.T) { _, ctx := ktesting.NewTestContext(t) clientset := fake.NewSimpleClientset() sharedInformers := informers.NewSharedInformerFactory(clientset, controller.NoResyncPeriodFunc()) manager := NewController(ctx, sharedInformers.Core().V1().Pods(), sharedInformers.Batch().V1().Jobs(), clientset) manager.podStoreSynced = alwaysReady manager.jobStoreSynced = alwaysReady stopCh := make(chan struct{}) defer close(stopCh) podInformer := sharedInformers.Core().V1().Pods().Informer() go podInformer.Run(stopCh) cache.WaitForCacheSync(stopCh, podInformer.HasSynced) // 1. Create the controller but do not start the workers. // This is done above by initializing the manager and not calling manager.Run() yet. // 2. Create a job. job := newJob(1, 1, 1, batch.NonIndexedCompletion) _, err := clientset.BatchV1().Jobs(job.GetNamespace()).Create(ctx, job, metav1.CreateOptions{}) if err != nil { t.Fatalf("Creating job: %v", err) } pod := newPod("test-pod", job) // pod.Finalizers = append(pod.Finalizers, batch.JobTrackingFinalizer) _, err = clientset.CoreV1().Pods(pod.GetNamespace()).Create(ctx, pod, metav1.CreateOptions{}) if err != nil { t.Fatalf("Creating pod: %v", err) } job.Status.Conditions = append(job.Status.Conditions, batch.JobCondition{ Type:   batch.JobComplete, Status: v1.ConditionTrue, }) clientset.BatchV1().Jobs(job.GetNamespace()).UpdateStatus(ctx, job, metav1.UpdateOptions{}) go manager.Run(context.TODO(), 0) ctx, cancel := context.WithTimeout(context.Background(), wait.ForeverTestTimeout) defer cancel() if err := wait.PollUntilContextTimeout(ctx, 100*time.Millisecond, wait.ForeverTestTimeout, true, func(ctx context.Context) (bool, error) { p, err := clientset.CoreV1().Pods(pod.Namespace).Get(ctx, pod.Name, metav1.GetOptions{}) if err != nil { return false, err } return !hasJobTrackingFinalizer(p), nil }); err != nil { // Check for context deadline exceeded, which would mean the operation timed out if errors.Is(err, context.DeadlineExceeded) { t.Errorf("Timed out waiting for Pod to get the finalizer removed") } else { t.Errorf("Waiting for Pod to get the finalizer removed: %v", err) } } } </ins> func checkJobCompletionLabel(t *testing.T, p *v1.PodTemplateSpec) { t.Helper() labels := p.GetLabels()