Repository: kubernetes-sigs/kueue

Issue:
jobframework JobReconciler don't update PodsReady condition timely when updata status failed
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:
when enable `waitForPodsReady` feature,  submit  [batch job](https://kubernetes.io/docs/concepts/workloads/controllers/job/),  the jobframework [JobReconciler](https://github.com/kubernetes-sigs/kueue/blob/main/pkg/controller/jobframework/reconciler.go#L514) don't update PodsReady condition timely when update status failed, the error message is as follows
```
{"level":"error","ts":"2025-10-23T09:54:18.123509681Z","caller":"jobframework/reconciler.go:523","msg":"Updating workload status","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"k8s-job-20m-mxy","namespa
ce":"default"},"namespace":"default","name":"k8s-job-20m-mxy","reconcileID":"23c12b4b-786f-42f5-bd58-a10213f56e19","job":"default/k8s-job-20m-mxy","gvk":"batch/v1, Kind=Job","error":"Internal error occurred: failed calling webhook \"vwork
load.kb.io\": Post \"https://kueue-webhook-service.kueue-system.svc:443/validate-kueue-x-k8s-io-v1beta1-workload?timeout=10s\": dial tcp 169.169.109.130:443: connect: connection refused","stacktrace":"sigs.k8s.io/kueue/pkg/controller/jobf
ramework.(*JobReconciler).ReconcileGenericJob\n\t/Users/didi/ml/kueue/pkg/controller/jobframework/reconciler.go:523\nsigs.k8s.io/kueue/pkg/controller/jobframework.(*genericReconciler).Reconcile\n\t/Users/didi/ml/kueue/pkg/controller/jobfr
amework/reconciler.go:1522\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/Users/didi/ml/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controlle
r-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/Users/didi/ml/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:340\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Contro
ller[...]).processNextWorkItem\n\t/Users/didi/ml/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:300\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1\n\t/Users/didi
/ml/kueue/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:202"}
```
**What you expected to happen**:
set PodsReady condition to True timely when updata status failed

**How to reproduce it (as minimally and precisely as possible)**:

for example, in the above situation, access to the webhook sometimes works and sometimes doesn't.

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`):
- Kueue version: master
- Cloud provider or hardware configuration:
- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Install tools:
- Others:

Hints from issue comments:
@olderTaoist: /assign olderTaoist@olderTaoist: in my situation, i notice that when a [batch job](https://kubernetes.io/docs/concepts/workloads/controllers/job/) pod changes from pending to ready, but the batch job's status.ready doesn't change, the job's [PodReady](https://github.com/kubernetes-sigs/kueue/blob/main/pkg/controller/jobs/job/job_controller.go#L318) remains false. Therefore, the batch job framework should also listen for pod event information. as fellow:
```
{"level":"Level(-3)","ts":"2025-10-28T01:04:50.13980978Z","caller":"jobframework/reconciler.go:1447","msg":"Generating PodsReady condition","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"k8s-job-20m-mxy","namespace":"default"},"namespace":"default","name":"k8s-job-20m-mxy","reconcileID":"6a0451e6-f0ce-49e2-a4d4-b96bdbab9e05","job":"default/k8s-job-20m-mxy","gvk":"batch/v1, Kind=Job","Current PodsReady condition":"&Condition{Type:PodsReady,Status:False,ObservedGeneration:1,LastTransitionTime:2025-10-27 08:24:40 +0000 UTC,Reason:WaitForStart,Message:Not all pods are ready or succeeded,}","Pods are ready":false}

{"level":"Level(-3)","ts":"2025-10-28T02:10:16.313011389Z","caller":"jobframework/reconciler.go:1447","msg":"Generating PodsReady condition","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"k8s-job-20m-mxy","namespace":"default"},"namespace":"default","name":"k8s-job-20m-mxy","reconcileID":"64a3cdc0-5236-4382-a4a3-fd07067ba895","job":"default/k8s-job-20m-mxy","gvk":"batch/v1, Kind=Job","Current PodsReady condition":"&Condition{Type:PodsReady,Status:False,ObservedGeneration:1,LastTransitionTime:2025-10-27 08:24:40 +0000 UTC,Reason:WaitForStart,Message:Not all pods are ready or succeeded,}","Pods are ready":true}
```

I'm not sure if this is a bug in batch jobs.

@mimowo @PBundyra PTAL

Solution patch:
diff --git a/pkg/controller/jobframework/reconciler.go b/pkg/controller/jobframework/reconciler.go
index ef7a5a6c8..c1830f7bc 100644
--- a/pkg/controller/jobframework/reconciler.go
+++ b/pkg/controller/jobframework/reconciler.go
@@ -522,6 +522,7 @@ func (r *JobReconciler) ReconcileGenericJob(ctx context.Context, req ctrl.Reques
 			err := workload.UpdateStatus(ctx, r.client, wl, condition.Type, condition.Status, condition.Reason, condition.Message, constants.JobControllerName, r.clock)
 			if err != nil {
 				log.Error(err, "Updating workload status")
+				return ctrl.Result{}, client.IgnoreNotFound(err)
 			}
 			// update the metrics only when PodsReady condition status is true
 			if condition.Status == metav1.ConditionTrue {



Test patch:
diff --git a/pkg/controller/jobframework/reconciler_test.go b/pkg/controller/jobframework/reconciler_test.go
index 26592bd7d..315196b52 100644
--- a/pkg/controller/jobframework/reconciler_test.go
+++ b/pkg/controller/jobframework/reconciler_test.go
@@ -17,6 +17,8 @@ limitations under the License.
 package jobframework_test
 
 import (
+	"context"
+	"errors"
 	"testing"
 	"time"
 
@@ -28,6 +30,7 @@ import (
 	appsv1 "k8s.io/api/apps/v1"
 	batchv1 "k8s.io/api/batch/v1"
 	corev1 "k8s.io/api/core/v1"
+	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/utils/clock"
@@ -35,6 +38,7 @@ import (
 	"k8s.io/utils/ptr"
 	controllerruntime "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
 	"sigs.k8s.io/jobset/api/jobset/v1alpha2"
 
 	configapi "sigs.k8s.io/kueue/apis/config/v1beta2"
@@ -42,6 +46,7 @@ import (
 	mocks "sigs.k8s.io/kueue/internal/mocks/controller/jobframework"
 	"sigs.k8s.io/kueue/pkg/controller/constants"
 	"sigs.k8s.io/kueue/pkg/controller/core/indexer"
+	"sigs.k8s.io/kueue/pkg/controller/jobs/job"
 	"sigs.k8s.io/kueue/pkg/features"
 	"sigs.k8s.io/kueue/pkg/util/kubeversion"
 	utiltesting "sigs.k8s.io/kueue/pkg/util/testing"
@@ -709,3 +714,137 @@ func TestProcessOptions(t *testing.T) {
 		})
 	}
 }
+
+func TestReconcileGenericJobWithWaitForPodsReady(t *testing.T) {
+	var (
+		testLocalQueueName = kueue.LocalQueueName("default")
+		testGVK            = batchv1.SchemeGroupVersion.WithKind("Job")
+	)
+	testCases := map[string]struct {
+		workload  *kueue.Workload
+		job       GenericJob
+		wantError error
+	}{
+		"update podready condition failed": {
+			workload: utiltestingapi.MakeWorkload("job-test-job-podready-fail", metav1.NamespaceDefault).
+				Finalizers(kueue.ResourceInUseFinalizerName).
+				Label(constants.JobUIDLabel, "test-job-podready-fail").
+				ControllerReference(testGVK, "test-job-podready-fail", "test-job-podready-fail").
+				Queue(testLocalQueueName).
+				PodSets(*utiltestingapi.MakePodSet("main", 1).Obj()).
+				Conditions(metav1.Condition{
+					Type:               kueue.WorkloadAdmitted,
+					Status:             metav1.ConditionTrue,
+					Reason:             "Admitted",
+					Message:            "The workload is admitted",
+					LastTransitionTime: metav1.NewTime(time.Now()),
+				}, metav1.Condition{
+					Type:               kueue.WorkloadPodsReady,
+					Status:             metav1.ConditionFalse,
+					Reason:             kueue.WorkloadWaitForStart,
+					Message:            "Not all pods are ready or succeeded",
+					LastTransitionTime: metav1.NewTime(time.Now()),
+				}).
+				Admission(&kueue.Admission{
+					ClusterQueue: "default-cq",
+				}).
+				Obj(),
+			job: (*job.Job)(testingjob.MakeJob("test-job-podready-fail", metav1.NamespaceDefault).
+				UID("test-job-podready-fail").
+				Label(constants.QueueLabel, string(testLocalQueueName)).
+				Parallelism(1).
+				Suspend(false).
+				Containers(corev1.Container{
+					Name: "c",
+					Resources: corev1.ResourceRequirements{
+						Requests: make(corev1.ResourceList),
+					},
+				}).
+				Ready(1).
+				Obj()),
+			wantError: apierrors.NewInternalError(errors.New("failed calling webhook")),
+		},
+		"update podready condition success": {
+			workload: utiltestingapi.MakeWorkload("job-test-job-podready-success", metav1.NamespaceDefault).
+				Finalizers(kueue.ResourceInUseFinalizerName).
+				Label(constants.JobUIDLabel, "job-test-job-podready-success").
+				ControllerReference(testGVK, "test-job-podready-success", "test-job-podready-success").
+				Queue(testLocalQueueName).
+				PodSets(*utiltestingapi.MakePodSet("main", 1).Obj()).
+				Conditions(metav1.Condition{
+					Type:               kueue.WorkloadAdmitted,
+					Status:             metav1.ConditionTrue,
+					Reason:             "Admitted",
+					Message:            "The workload is admitted",
+					LastTransitionTime: metav1.NewTime(time.Now()),
+				}, metav1.Condition{
+					Type:               kueue.WorkloadPodsReady,
+					Status:             metav1.ConditionFalse,
+					Reason:             kueue.WorkloadWaitForStart,
+					Message:            "Not all pods are ready or succeeded",
+					LastTransitionTime: metav1.NewTime(time.Now()),
+				}).
+				Admission(&kueue.Admission{
+					ClusterQueue: "default-cq",
+				}).
+				Obj(),
+			job: (*job.Job)(testingjob.MakeJob("test-job-podready-success", metav1.NamespaceDefault).
+				UID("test-job-podready-success").
+				Label(constants.QueueLabel, string(testLocalQueueName)).
+				Parallelism(1).
+				Suspend(false).
+				Containers(corev1.Container{
+					Name: "c",
+					Resources: corev1.ResourceRequirements{
+						Requests: make(corev1.ResourceList),
+					},
+				}).
+				Ready(1).
+				Obj()),
+			wantError: nil,
+		},
+	}
+
+	for name, tc := range testCases {
+		t.Run(name, func(t *testing.T) {
+			ctx, _ := utiltesting.ContextWithLog(t)
+			managedNamespace := utiltesting.MakeNamespaceWrapper(metav1.NamespaceDefault).
+				Label("managed-by-kueue", "true").
+				Obj()
+			builder := utiltesting.NewClientBuilder(batchv1.AddToScheme, kueue.AddToScheme).
+				WithObjects(tc.workload, tc.job.Object(), managedNamespace).
+				WithStatusSubresource(tc.workload, tc.job.Object()).
+				WithIndex(&kueue.Workload{}, indexer.OwnerReferenceIndexKey(testGVK), indexer.WorkloadOwnerIndexFunc(testGVK)).
+				WithInterceptorFuncs(interceptor.Funcs{
+					SubResourcePatch: func(ctx context.Context, client client.Client, subResourceName string, obj client.Object, patch client.Patch, opts ...client.SubResourcePatchOption) error {
+						if _, ok := obj.(*kueue.Workload); ok && subResourceName == "status" && tc.wantError != nil {
+							return tc.wantError
+						}
+						return utiltesting.TreatSSAAsStrategicMerge(ctx, client, subResourceName, obj, patch, opts...)
+					},
+				})
+
+			cl := builder.Build()
+
+			testStartTime := time.Now().Truncate(time.Second)
+
+			fakeClock := testingclock.NewFakeClock(testStartTime)
+			options := []Option{
+				WithClock(nil, fakeClock),
+				WithWaitForPodsReady(&configapi.WaitForPodsReady{
+					Enable: true,
+				}),
+			}
+			recorder := &utiltesting.EventRecorder{}
+			r := NewReconciler(cl, recorder, options...)
+			_, err := r.ReconcileGenericJob(ctx, controllerruntime.Request{
+				NamespacedName: types.NamespacedName{
+					Name:      tc.job.Object().GetName(),
+					Namespace: tc.job.Object().GetNamespace(),
+				}}, tc.job)
+			if !errors.Is(err, tc.wantError) {
+				t.Errorf("unexpected reconcile error want %s got %s)", tc.wantError, err)
+			}
+		})
+	}
+}