Repository: kubernetes-sigs/kueue (2026-03-13T13:37:01Z)

Issue:
MultiKueue Manager Workload admits but does not get "Admitted" condition when additional Admission Checks still "Pending"
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:
If a user of MultiKueue defines additional admission checks beyond the MultiKueue admission check on the Manager Cluster queue, these checks will not be taken into account when dispatching the Remote Workloads to Workers.
As long as the additional check does not enter the "Rejected" or "Retry" state, workloads will continue to be dispatched. Eventually a remote may become Admitted and begin executing the Remote Workload. If the additional check is not "Ready" by then, the process will continue, but the Manager Workload will not be marked as "Admitted". This can proceed to the eventual success of the Remote Workload, leading to the Manager Workload reaching the Finished state without ever getting the "Admitted" condition.

**What you expected to happen**:
The workload should enter the "Admitted" state when a Remote is "Admitted". This could be achieved, for example, by:
* ensuring dispatch may not happen until all admission checks on the Manager Workload (other than the MK AC) are ready,
* propagating the Manager ACs to the Remotes, setting their state to "Ready" there once they become "Ready" on the Manager Workload.

**How to reproduce it (as minimally and precisely as possible)**:
Test verifying this behavior can be found at https://github.com/kubernetes-sigs/kueue/pull/9573

**Anything else we need to know?**:
---

**Environment**:
- Kubernetes version (use `kubectl version`):
```
Client Version: v1.34.4-dispatcher
Kustomize Version: v5.7.1
```
- Kueue version (use `git describe --tags --dirty --always`): `v0.16.0-devel-632-gb08267fe0`


Discussion:
From #7254 

> Regarding the very possibility of defining an admission check separate form the MultiKueue admission check on a manager cluster queue - even if we ensure the MK AC is always present on the workload, the current logic does not support a correct handling of such checks:
> 
> * Such a check would not affect the dispatch of a workload itself unless it enters the Rejected or Retry state. In fact, the check could remain in the Pending state, keeping the workload on the manager from being admitted (condition admitted), but the quota would be reserved and remotes would be created as usual. This could even lead to the workload finishing without being officially admitted at all. This behavior is not ideal and inconsistent with what an AC is meant to be doing: instead of keeping the workload from admission until Ready, it keeps it from admission only after it is Rejected (or Retrying after rejection).
> * Even though the workload can end up finished despite such a check never leaving the Pending state, the check could enter the Rejected or Retry state, leading to the workloads eviction and removal of all of the remote workloads.
> 
> We could resolve this in one of two ways:
> 
> * by ether adding the support for such custom checks to be used by MK or
> * by adding an additional restriction to MK CQs - when the MK AC is defined it has to be the only AC on a cluster queue.
/area multikueue
I think option 1
> ensuring dispatch may not happen until all admission checks on the Manager Workload (other than the MK AC) are ready,

is better, because:

- the decision is done in one place (on the manager cluster), semantically it's also concise - manager `dispatches`, therefor `admits`
- we don't replicate Acs to remote clusters, which we would have to cleanup afterwards

while option 2 brings risks like:
- ACs may reference manager-only context (namespaces, policies, credentials, external controllers) that don’t exist on workers
-  If AC becomes Ready/Rejected on the manager, we need to propagate the state to all workers

wdyt @mimowo
Yes, (1.) is simpler. If we need some configuration in the future, then we will add knobs. For now this is much better.
When will this bug be fixed? I am experiencing the same issue. Please fix it quickly.
cc @mszadkow @kshalot 



Patch:
diff --git a/pkg/controller/admissionchecks/multikueue/workload.go b/pkg/controller/admissionchecks/multikueue/workload.go
index 0e9e4776670..01739cfe4cd 100644
--- a/pkg/controller/admissionchecks/multikueue/workload.go
+++ b/pkg/controller/admissionchecks/multikueue/workload.go
@@ -706,6 +706,12 @@ func (w *wlReconciler) nominateAndSynchronizeWorkers(ctx context.Context, group
 		return reconcile.Result{}, err
 	}
 
+	// check nonMK AdmissionChecks are Ready before nominating clusters, to avoid nominating clusters for workloads that are not fully admitted yet.
+	if ready, ac := allNonMKAdmissionChecksReady(group.local, group.acName); !ready {
+		log.V(3).Info("Waiting for all non-MultiKueue admission checks to be ready before nominating clusters", "admissionCheck", ac)
+		return reconcile.Result{}, nil
+	}
+
 	if _, ok := group.jobAdapter.(jobframework.MultiKueueMultiWorkloadAdapter); ok && componentWorkloads != nil {
 		allExist, err := w.allExpectedWorkloadsExist(ctx, group.local, group.jobAdapter, len(componentWorkloads.Items))
 		if err != nil {
@@ -1105,3 +1111,16 @@ func cloneForCreate(orig *kueue.Workload, origin string, preemptionGated bool) *
 
 	return remoteWl
 }
+
+func allNonMKAdmissionChecksReady(wl *kueue.Workload, mkACName kueue.AdmissionCheckReference) (bool, []kueue.AdmissionCheckReference) {
+	var notReady []kueue.AdmissionCheckReference
+	for _, ac := range wl.Status.AdmissionChecks {
+		if ac.Name == mkACName {
+			continue
+		}
+		if ac.State != kueue.CheckStateReady {
+			notReady = append(notReady, ac.Name)
+		}
+	}
+	return len(notReady) == 0, notReady
+}


Test patch:
diff --git a/test/integration/multikueue/jobs_test.go b/test/integration/multikueue/jobs_test.go
index 5c7087f0bcf..faf108fe295 100644
--- a/test/integration/multikueue/jobs_test.go
+++ b/test/integration/multikueue/jobs_test.go
@@ -1192,6 +1192,202 @@ var _ = ginkgo.Describe("MultiKueue", ginkgo.Label("area:multikueue", "feature:m
 		})
 	})
 
+	ginkgo.It("Should handle a pod group admission with extra non-multikueue admission checks defined", func() {
+		var testAc, testAc2 *kueue.AdmissionCheck
+
+		ginkgo.By("creating non-multikueue ACs and adding them to the CQ", func() {
+			testAc = utiltestingapi.MakeAdmissionCheck("test-ac").
+				ControllerName("test-controller").
+				Active(metav1.ConditionTrue).
+				Obj()
+			util.MustCreate(managerTestCluster.ctx, managerTestCluster.client, testAc)
+
+			testAc2 = utiltestingapi.MakeAdmissionCheck("test-ac-2").
+				ControllerName("test-controller-2").
+				Active(metav1.ConditionTrue).
+				Obj()
+			util.MustCreate(managerTestCluster.ctx, managerTestCluster.client, testAc2)
+
+			gomega.Eventually(func(g gomega.Gomega) {
+				g.Expect(managerTestCluster.client.Get(managerTestCluster.ctx, client.ObjectKeyFromObject(managerCq), managerCq)).To(gomega.Succeed())
+				managerCq.Spec.AdmissionChecksStrategy.AdmissionChecks = append(
+					managerCq.Spec.AdmissionChecksStrategy.AdmissionChecks,
+					kueue.AdmissionCheckStrategyRule{Name: kueue.AdmissionCheckReference(testAc.Name)},
+					kueue.AdmissionCheckStrategyRule{Name: kueue.AdmissionCheckReference(testAc2.Name)},
+				)
+				g.Expect(managerTestCluster.client.Update(managerTestCluster.ctx, managerCq)).To(gomega.Succeed())
+			}, util.VeryLongTimeout, util.Interval).Should(gomega.Succeed())
+		})
+
+		groupName := "test-group"
+		podgroup := testingpod.MakePod(groupName, managerNs.Name).
+			Queue(managerLq.Name).
+			ManagedByKueueLabel().
+			KueueFinalizer().
+			KueueSchedulingGate().
+			MakeGroup(3)
+
+		for _, p := range podgroup {
+			gomega.Expect(managerTestCluster.client.Create(managerTestCluster.ctx, p)).Should(gomega.Succeed())
+		}
+
+		createdWorkload := &kueue.Workload{}
+		wlLookupKey := types.NamespacedName{Name: groupName, Namespace: managerNs.Name}
+
+		ginkgo.By("checking workload in manager is set up correctly", func() {
+			gomega.Eventually(func(g gomega.Gomega) {
+				g.Expect(managerTestCluster.client.Get(managerTestCluster.ctx, wlLookupKey, createdWorkload)).To(gomega.Succeed())
+				g.Expect(admissioncheck.FindAdmissionCheck(createdWorkload.Status.AdmissionChecks, kueue.AdmissionCheckReference(multiKueueAC.Name))).
+					To(gomega.BeComparableTo(&kueue.AdmissionCheckState{
+						Name:  kueue.AdmissionCheckReference(multiKueueAC.Name),
+						State: kueue.CheckStatePending,
+					}, cmpopts.IgnoreFields(kueue.AdmissionCheckState{}, "LastTransitionTime", "Message")))
+				g.Expect(admissioncheck.FindAdmissionCheck(createdWorkload.Status.AdmissionChecks, kueue.AdmissionCheckReference(testAc.Name))).
+					To(gomega.BeComparableTo(&kueue.AdmissionCheckState{
+						Name:  kueue.AdmissionCheckReference(testAc.Name),
+						State: kueue.CheckStatePending,
+					}, cmpopts.IgnoreFields(kueue.AdmissionCheckState{}, "LastTransitionTime", "Message")))
+				g.Expect(admissioncheck.FindAdmissionCheck(createdWorkload.Status.AdmissionChecks, kueue.AdmissionCheckReference(testAc2.Name))).
+					To(gomega.BeComparableTo(&kueue.AdmissionCheckState{
+						Name:  kueue.AdmissionCheckReference(testAc2.Name),
+						State: kueue.CheckStatePending,
+					}, cmpopts.IgnoreFields(kueue.AdmissionCheckState{}, "LastTransitionTime", "Message")))
+			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+		})
+
+		admission := utiltestingapi.MakeAdmission(managerCq.Name).
+			PodSets(
+				kueue.PodSetAssignment{
+					Name:  "bf90803c",
+					Count: ptr.To[int32](3),
+				},
+			).Obj()
+		ginkgo.By("setting workload reservation in the management cluster", func() {
+			gomega.Eventually(func(g gomega.Gomega) {
+				g.Expect(managerTestCluster.client.Get(managerTestCluster.ctx, wlLookupKey, createdWorkload)).To(gomega.Succeed())
+				gomega.Expect(createdWorkload.Spec.PodSets[0].Count).To(gomega.Equal(int32(3)))
+			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			util.SetQuotaReservation(managerTestCluster.ctx, managerTestCluster.client, wlLookupKey, admission)
+		})
+
+		ginkgo.By("checking the workload creation in the worker clusters didn't happen due to pending non-multikueue ACs", func() {
+			managerWl := &kueue.Workload{}
+			gomega.Expect(managerTestCluster.client.Get(managerTestCluster.ctx, wlLookupKey, managerWl)).To(gomega.Succeed())
+			gomega.Eventually(func(g gomega.Gomega) {
+				g.Expect(worker1TestCluster.client.Get(worker1TestCluster.ctx, wlLookupKey, createdWorkload)).To(utiltesting.BeNotFoundError())
+				g.Expect(worker2TestCluster.client.Get(worker2TestCluster.ctx, wlLookupKey, createdWorkload)).To(utiltesting.BeNotFoundError())
+			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+		})
+
+		ginkgo.By("Update first pending AC to Ready", func() {
+			gomega.Eventually(func(g gomega.Gomega) {
+				updatedWorkload := &kueue.Workload{}
+				g.Expect(managerTestCluster.client.Get(managerTestCluster.ctx, wlLookupKey, updatedWorkload)).To(gomega.Succeed())
+				acs := admissioncheck.FindAdmissionCheck(updatedWorkload.Status.AdmissionChecks, kueue.AdmissionCheckReference(testAc.Name))
+				g.Expect(acs).NotTo(gomega.BeNil())
+				acs.State = kueue.CheckStateReady
+				acs.Message = "Test AC is ready"
+				g.Expect(managerTestCluster.client.Status().Update(managerTestCluster.ctx, updatedWorkload)).To(gomega.Succeed())
+			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+		})
+
+		ginkgo.By("checking the workload creation in the worker clusters still didn't happen due to second pending AC", func() {
+			gomega.Eventually(func(g gomega.Gomega) {
+				g.Expect(worker1TestCluster.client.Get(worker1TestCluster.ctx, wlLookupKey, createdWorkload)).To(utiltesting.BeNotFoundError())
+				g.Expect(worker2TestCluster.client.Get(worker2TestCluster.ctx, wlLookupKey, createdWorkload)).To(utiltesting.BeNotFoundError())
+			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+		})
+
+		ginkgo.By("Update second pending AC to Ready", func() {
+			gomega.Eventually(func(g gomega.Gomega) {
+				updatedWorkload := &kueue.Workload{}
+				g.Expect(managerTestCluster.client.Get(managerTestCluster.ctx, wlLookupKey, updatedWorkload)).To(gomega.Succeed())
+				acs := admissioncheck.FindAdmissionCheck(updatedWorkload.Status.AdmissionChecks, kueue.AdmissionCheckReference(testAc2.Name))
+				g.Expect(acs).NotTo(gomega.BeNil())
+				acs.State = kueue.CheckStateReady
+				acs.Message = "Test AC 2 is ready"
+				g.Expect(managerTestCluster.client.Status().Update(managerTestCluster.ctx, updatedWorkload)).To(gomega.Succeed())
+			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+		})
+
+		ginkgo.By("checking the workload creation in the worker clusters", func() {
+			managerWl := &kueue.Workload{}
+			gomega.Expect(managerTestCluster.client.Get(managerTestCluster.ctx, wlLookupKey, managerWl)).To(gomega.Succeed())
+			gomega.Eventually(func(g gomega.Gomega) {
+				g.Expect(worker1TestCluster.client.Get(worker1TestCluster.ctx, wlLookupKey, createdWorkload)).To(gomega.Succeed())
+				g.Expect(createdWorkload.Spec).To(gomega.BeComparableTo(managerWl.Spec))
+				g.Expect(worker2TestCluster.client.Get(worker2TestCluster.ctx, wlLookupKey, createdWorkload)).To(gomega.Succeed())
+				g.Expect(createdWorkload.Spec).To(gomega.BeComparableTo(managerWl.Spec))
+			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+		})
+
+		ginkgo.By("setting workload reservation in worker1, AC state is updated in manager and worker2 wl is removed", func() {
+			util.SetQuotaReservation(worker1TestCluster.ctx, worker1TestCluster.client, wlLookupKey, admission)
+
+			gomega.Eventually(func(g gomega.Gomega) {
+				g.Expect(managerTestCluster.client.Get(managerTestCluster.ctx, wlLookupKey, createdWorkload)).To(gomega.Succeed())
+				acs := admissioncheck.FindAdmissionCheck(createdWorkload.Status.AdmissionChecks, kueue.AdmissionCheckReference(multiKueueAC.Name))
+				g.Expect(acs).NotTo(gomega.BeNil())
+				g.Expect(acs.State).To(gomega.Equal(kueue.CheckStateReady))
+				g.Expect(acs.Message).To(gomega.Equal(`The workload got reservation on "worker1"`))
+				g.Expect(apimeta.IsStatusConditionTrue(createdWorkload.Status.Conditions, kueue.WorkloadAdmitted)).To(gomega.BeTrue())
+			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			util.ExpectEventAppeared(managerTestCluster.ctx, managerTestCluster.client, corev1.Event{
+				Reason:  "MultiKueue",
+				Type:    corev1.EventTypeNormal,
+				Message: `The workload got reservation on "worker1"`,
+			})
+
+			gomega.Eventually(func(g gomega.Gomega) {
+				g.Expect(worker2TestCluster.client.Get(worker2TestCluster.ctx, wlLookupKey, createdWorkload)).To(utiltesting.BeNotFoundError())
+			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+		})
+
+		pods := corev1.PodList{}
+		gomega.Expect(managerTestCluster.client.List(managerTestCluster.ctx, &pods)).To(gomega.Succeed())
+
+		ginkgo.By("finishing the worker pod", func() {
+			pods := corev1.PodList{}
+			gomega.Expect(worker1TestCluster.client.List(worker1TestCluster.ctx, &pods)).To(gomega.Succeed())
+			for _, p := range podgroup {
+				gomega.Eventually(func(g gomega.Gomega) {
+					createdPod := corev1.Pod{}
+					g.Expect(worker1TestCluster.client.Get(worker1TestCluster.ctx, client.ObjectKeyFromObject(p), &createdPod)).To(gomega.Succeed())
+					createdPod.Status.Phase = corev1.PodSucceeded
+					createdPod.Status.Conditions = append(createdPod.Status.Conditions,
+						corev1.PodCondition{
+							Type:   corev1.PodReadyToStartContainers,
+							Status: corev1.ConditionFalse,
+							Reason: "",
+						},
+						corev1.PodCondition{
+							Type:   corev1.PodInitialized,
+							Status: corev1.ConditionTrue,
+							Reason: string(corev1.PodSucceeded),
+						},
+						corev1.PodCondition{
+							Type:   corev1.PodReady,
+							Status: corev1.ConditionFalse,
+							Reason: string(corev1.PodSucceeded),
+						},
+						corev1.PodCondition{
+							Type:   corev1.ContainersReady,
+							Status: corev1.ConditionFalse,
+							Reason: string(corev1.PodSucceeded),
+						},
+						corev1.PodCondition{
+							Type:   corev1.PodScheduled,
+							Status: corev1.ConditionTrue,
+							Reason: "",
+						},
+					)
+					g.Expect(worker1TestCluster.client.Status().Update(worker1TestCluster.ctx, &createdPod)).To(gomega.Succeed())
+				}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			}
+			waitForWorkloadToFinishAndRemoteWorkloadToBeDeleted(wlLookupKey, "Pods succeeded: 3/3.")
+		})
+	})
+
 	ginkgo.It("Should remove the worker's workload and job after reconnect when the managers job and workload are deleted", func() {
 		job := testingjob.MakeJob("job", managerNs.Name).
 			Queue(kueue.LocalQueueName(managerLq.Name)).