Repository: kubernetes-sigs/kueue

Issue:
Flaky E2E Test: StatefulSet integration when StatefulSet created should allow to change queue name if ReadyReplicas=0
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:

End To End Suite: kindest/node:v1.30.10: [It] StatefulSet integration when StatefulSet created should allow to change queue name if ReadyReplicas=0

```
{Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:375 with:
Expected
    <int32>: 0
to equal
    <int32>: 3 failed [FAILED] Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:375 with:
Expected
    <int32>: 0
to equal
    <int32>: 3
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/statefulset_test.go:376 @ 04/18/25 07:24:31.271
}
```

**What you expected to happen**:
No errors

**How to reproduce it (as minimally and precisely as possible)**:

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5030/pull-kueue-test-e2e-main-1-30/1913127191198044160

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`):
- Kueue version (use `git describe --tags --dirty --always`):
- Cloud provider or hardware configuration:
- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Install tools:
- Others:

Hints from issue comments:
@mbobrovskyi: /kind flake@mimowo: happend again recently https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-kueue-test-e2e-main-1-31/1925093654561558528@sohankunkerkar: Have we considered using `util.VeryLongTimeout` in this case? AFAIK, changing the queue name triggers a rolling update in the StatefulSet, as the webhook updates pod labels to ensure correct scheduling. This causes each pod to terminate and restart sequentially, which can take significantly longer than the current (`util.Longtimeout`) timeout allows.@mimowo: @sohankunkerkar I'm sure open to extending the timeout. Do you have some evidence, looking into the logs which could support the hypothesis that extending timeout would help? If so, I'm happy to accept the PR.@sohankunkerkar: Yeah, this issue surfaced during [this change](https://github.com/openshift/kueue-operator/pull/441#discussion_r2207285597) to the Kueue Operator, where we run upstream Kueue e2e tests in CI. After modifying the leader election configuration to use longer lease durations, the test started [failing](https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openshift_kueue-operator/441/pull-ci-openshift-kueue-operator-release-1.0-test-e2e-4-19/1945360009701363712/artifacts/test-e2e-4-19/e2e-kueue-upstream/build-log.txt) because it incorrectly assumed the controller would be immediately functional once the deployment was marked as available. However, in reality, the leader election was still in progress, and the controller hadn’t become active yet.@mimowo: TBH, it does not feel related, we already wait for Kueue manager to be Available before running e2e tests.

I looked at the logs of the build https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5030/pull-kueue-test-e2e-main-1-30/1913127191198044160

and the namespace for which the test failed was `sts-e2e-ljgzt` with the timeline of events:
```
  [1mSTEP:[0m Create StatefulSet [38;5;243m@ 04/18/25 07:23:45.216[0m
  [1mSTEP:[0m Checking that replicas is not ready [38;5;243m@ 04/18/25 07:23:45.251[0m
  [1mSTEP:[0m Update queue name [38;5;243m@ 04/18/25 07:23:46.252[0m
  [1mSTEP:[0m Waiting for replicas is ready [38;5;243m@ 04/18/25 07:23:46.27[0m
```
now, these are the logs of the Kueue manager in this time:

```
2025-04-18T07:23:45.212985235Z stderr F 2025-04-18T07:23:45.212782864Z	LEVEL(-2)	localqueue-reconciler	core/localqueue_controller.go:140	LocalQueue create event	{"localQueue": {"name":"sts-lq","namespace":"sts-e2e-ljgzt"}}
2025-04-18T07:23:45.215122602Z stderr F 2025-04-18T07:23:45.214952561Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:356	ClusterQueue update event	{"clusterQueue": {"name":"sts-cq"}}
2025-04-18T07:23:45.233477639Z stderr F 2025-04-18T07:23:45.233287367Z	LEVEL(-2)	localqueue-reconciler	core/localqueue_controller.go:173	Queue update event	{"localQueue": {"name":"sts-lq","namespace":"sts-e2e-ljgzt"}}
2025-04-18T07:23:45.311003525Z stderr F 2025-04-18T07:23:45.310514781Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:601	Workload create event	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq-invalid", "status": "pending"}
2025-04-18T07:23:45.311026755Z stderr F 2025-04-18T07:23:45.310628612Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:613	ignored an error for now	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq-invalid", "status": "pending", "error": "localQueue doesn't exist or inactive"}
2025-04-18T07:23:45.330689491Z stderr F 2025-04-18T07:23:45.330444059Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:680	Workload update event	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq-invalid", "status": "pending"}
2025-04-18T07:23:45.330715562Z stderr F 2025-04-18T07:23:45.330594431Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:707	ignored an error for now	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq-invalid", "status": "pending", "error": "localQueue doesn't exist or inactive"}
2025-04-18T07:23:45.360205616Z stderr F 2025-04-18T07:23:45.359839074Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:680	Workload update event	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq-invalid", "status": "pending"}
2025-04-18T07:23:45.360273227Z stderr F 2025-04-18T07:23:45.359962605Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:707	ignored an error for now	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq-invalid", "status": "pending", "error": "localQueue doesn't exist or inactive"}
2025-04-18T07:23:46.370037469Z stderr F 2025-04-18T07:23:46.368645918Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:680	Workload update event	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq-invalid", "status": "pending"}
2025-04-18T07:23:46.370074289Z stderr F 2025-04-18T07:23:46.368744408Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:707	ignored an error for now	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq-invalid", "status": "pending", "error": "localQueue doesn't exist or inactive"}
2025-04-18T07:23:46.404589184Z stderr F 2025-04-18T07:23:46.403579226Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:680	Workload update event	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq", "status": "pending", "prevQueue": "sts-lq-invalid"}
2025-04-18T07:23:46.409785995Z stderr F 2025-04-18T07:23:46.409536383Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:631	Workload delete event	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq", "status": "pending"}
2025-04-18T07:24:31.289716957Z stderr F 2025-04-18T07:24:31.289509215Z	LEVEL(-2)	localqueue-reconciler	core/localqueue_controller.go:165	LocalQueue delete event	{"localQueue": {"name":"sts-lq","namespace":"sts-e2e-ljgzt"}}
2025-04-18T07:24:31.322200867Z stderr F 2025-04-18T07:24:31.321954415Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:601	Workload create event	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq", "status": "pending"}
2025-04-18T07:24:31.322229017Z stderr F 2025-04-18T07:24:31.322085786Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:613	ignored an error for now	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq", "status": "pending", "error": "localQueue doesn't exist or inactive"}
2025-04-18T07:24:31.322246277Z stderr F 2025-04-18T07:24:31.322122026Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:356	ClusterQueue update event	{"clusterQueue": {"name":"sts-cq"}}
2025-04-18T07:24:31.334769627Z stderr F 2025-04-18T07:24:31.334540635Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:343	ClusterQueue delete event	{"clusterQueue": {"name":"sts-cq"}}
2025-04-18T07:24:31.334791317Z stderr F 2025-04-18T07:24:31.334671426Z	LEVEL(-2)	cluster-queue-reconciler	core/clusterqueue_controller.go:349	Cleared resource metrics for deleted ClusterQueue.	{"clusterQueue": {"name":"sts-cq"}}
2025-04-18T07:24:31.33515214Z stderr F 2025-04-18T07:24:31.334946709Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:680	Workload update event	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq", "status": "pending"}
2025-04-18T07:24:31.33518064Z stderr F 2025-04-18T07:24:31.335052899Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:707	ignored an error for now	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq", "status": "pending", "error": "localQueue doesn't exist or inactive"}
2025-04-18T07:24:31.349163132Z stderr F 2025-04-18T07:24:31.3489146Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:680	Workload update event	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq", "status": "pending"}
2025-04-18T07:24:31.349195592Z stderr F 2025-04-18T07:24:31.349045911Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:707	ignored an error for now	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq", "status": "pending", "error": "localQueue doesn't exist or inactive"}
2025-04-18T07:24:31.352994783Z stderr F 2025-04-18T07:24:31.352795811Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:631	Workload delete event	{"workload": {"name":"statefulset-sts-18b6e","namespace":"sts-e2e-ljgzt"}, "queue": "sts-lq", "status": "pending"}
```
So, one outstanding thing is many "ignored an error for now" errors. I'm not yet totally sure, but these are likely the reason why the workload was not picked up after queue-name change, just because the CQ was not active while the workload was updated. 

This is actually already ticketed here: https://github.com/kubernetes-sigs/kueue/issues/5310, but for now the easiest fix for the flakes is to just ensure the CQ is active by calling `ExpectClusterQueuesToBeActive`, and maybe also ExpectLocalQueuesToBeActive. We use this tactic in many other tests.

@mimowo: Let me know if this makes sense and maybe you would also like to submit a PR fixing the flake along the lines.@sohankunkerkar: @mimowo Thanks for the apt reply. I think this is a much more targeted and accurate diagnosis than my initial leader election hypothesis. 

Solution patch:
diff --git a/Makefile b/Makefile
index 94500f336..24f896f69 100644
--- a/Makefile
+++ b/Makefile
@@ -166,6 +166,11 @@ helm-verify: helm helm-lint ## run helm template and detect any rendering failur
 	$(HELM) template charts/kueue > /dev/null
 # test nondefault options (kueueviz, prometheus, certmanager)
 	$(HELM) template charts/kueue --set enableKueueViz=true --set enableCertManager=true --set enablePrometheus=true > /dev/null
+
+.PHONY: helm-unit-test
+helm-unit-test: helm
+	$(HELM) unittest charts/kueue --strict --debug
+
 .PHONY: vet
 vet: ## Run go vet against code.
 	$(GO_CMD) vet ./...
@@ -188,7 +193,7 @@ sync-hugo-version:
 
 PATHS_TO_VERIFY := config/components apis charts/kueue/templates client-go site/ netlify.toml
 .PHONY: verify
-verify: gomod-verify ci-lint fmt-verify shell-lint toc-verify manifests generate update-helm helm-verify prepare-release-branch sync-hugo-version
+verify: gomod-verify ci-lint fmt-verify shell-lint toc-verify manifests generate update-helm helm-verify helm-unit-test prepare-release-branch sync-hugo-version
 	git --no-pager diff --exit-code $(PATHS_TO_VERIFY)
 	if git ls-files --exclude-standard --others $(PATHS_TO_VERIFY) | grep -q . ; then exit 1; fi
 
diff --git a/Makefile-deps.mk b/Makefile-deps.mk
index b5f3dd09e..da5454fc8 100644
--- a/Makefile-deps.mk
+++ b/Makefile-deps.mk
@@ -81,8 +81,11 @@ yq: ## Download yq locally if necessary.
 
 HELM = $(PROJECT_DIR)/bin/helm
 .PHONY: helm
-helm: ## Download helm locally if necessary.
+helm: ## Download helm and helm-unittest locally if necessary.
 	@GOBIN=$(PROJECT_DIR)/bin GO111MODULE=on $(GO_CMD) install helm.sh/helm/v3/cmd/helm@$(HELM_VERSION)
+	@if ! $(HELM) plugin list | grep -q unittest; then \
+		$(HELM) plugin install https://github.com/helm-unittest/helm-unittest.git; \
+	fi
 
 GENREF = $(PROJECT_DIR)/bin/genref
 .PHONY: genref
diff --git a/README.md b/README.md
index d4420c898..28b353789 100644
--- a/README.md
+++ b/README.md
@@ -52,7 +52,7 @@ Read the [overview](https://kueue.sigs.k8s.io/docs/overview/) and watch the Kueu
 
 ## Installation
 
-**Requires Kubernetes 1.25 or newer**.
+**Requires Kubernetes 1.29 or newer**.
 
 To install the latest release of Kueue in your cluster, run the following command:
 
diff --git a/pkg/controller/admissionchecks/provisioning/controller.go b/pkg/controller/admissionchecks/provisioning/controller.go
index 7dfa172db..e6d2811d6 100644
--- a/pkg/controller/admissionchecks/provisioning/controller.go
+++ b/pkg/controller/admissionchecks/provisioning/controller.go
@@ -23,7 +23,6 @@ import (
 	"errors"
 	"fmt"
 	"maps"
-	"time"
 
 	corev1 "k8s.io/api/core/v1"
 	apimeta "k8s.io/apimachinery/pkg/api/meta"
@@ -45,7 +44,6 @@ import (
 
 	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta1"
 	"sigs.k8s.io/kueue/pkg/constants"
-	"sigs.k8s.io/kueue/pkg/features"
 	"sigs.k8s.io/kueue/pkg/podset"
 	"sigs.k8s.io/kueue/pkg/util/admissioncheck"
 	"sigs.k8s.io/kueue/pkg/util/api"
@@ -164,16 +162,13 @@ func (c *Controller) Reconcile(ctx context.Context, req reconcile.Request) (reco
 		return reconcile.Result{}, err
 	}
 
-	requeueAfter, err := c.syncOwnedProvisionRequest(ctx, wl, &wlInfo, checkConfig, activeOrLastPRForChecks)
+	err = c.syncOwnedProvisionRequest(ctx, wl, &wlInfo, checkConfig, activeOrLastPRForChecks)
 	if err != nil {
 		// this can also delete unneeded checks
 		log.V(2).Error(err, "syncOwnedProvisionRequest failed")
 		return reconcile.Result{}, err
 	}
 
-	if requeueAfter != nil {
-		return reconcile.Result{RequeueAfter: *requeueAfter}, nil
-	}
 	return reconcile.Result{}, nil
 }
 
@@ -229,9 +224,8 @@ func (c *Controller) syncOwnedProvisionRequest(
 	wlInfo *workloadInfo,
 	checkConfig map[kueue.AdmissionCheckReference]*kueue.ProvisioningRequestConfig,
 	activeOrLastPRForChecks map[kueue.AdmissionCheckReference]*autoscaling.ProvisioningRequest,
-) (*time.Duration, error) {
+) error {
 	log := ctrl.LoggerFrom(ctx)
-	var requeAfter *time.Duration
 	for checkName, prc := range checkConfig {
 		if prc == nil {
 			// the check is not active
@@ -255,18 +249,8 @@ func (c *Controller) syncOwnedProvisionRequest(
 				ac != nil && ac.State == kueue.CheckStatePending {
 				// if the workload is in Retry/Rejected state we don't create another ProvReq
 				attempt = getAttempt(log, req, wl.Name, checkName)
-				if features.Enabled(features.KeepQuotaForProvReqRetry) {
-					remainingTime := c.remainingTimeToRetry(req, attempt, prc)
-					if remainingTime <= 0 {
-						shouldCreatePr = true
-						attempt++
-					} else if requeAfter == nil || remainingTime < *requeAfter {
-						requeAfter = &remainingTime
-					}
-				} else {
-					shouldCreatePr = true
-					attempt++
-				}
+				shouldCreatePr = true
+				attempt++
 			}
 		} else {
 			shouldCreatePr = true
@@ -296,7 +280,7 @@ func (c *Controller) syncOwnedProvisionRequest(
 				ps, psFound := podSetMap[psName]
 				psa, psaFound := psaMap[psName]
 				if !psFound || !psaFound {
-					return nil, errInconsistentPodSetAssignments
+					return errInconsistentPodSetAssignments
 				}
 
 				ptName := getProvisioningRequestPodTemplateName(requestName, psName)
@@ -304,14 +288,14 @@ func (c *Controller) syncOwnedProvisionRequest(
 				pt := &corev1.PodTemplate{}
 				err := c.client.Get(ctx, types.NamespacedName{Namespace: wl.Namespace, Name: ptName}, pt)
 				if client.IgnoreNotFound(err) != nil {
-					return nil, err
+					return err
 				}
 				if err != nil {
 					// it's a not found, so create it
 					_, err := c.createPodTemplate(ctx, wl, ptName, ps, psa)
 					if err != nil {
 						msg := fmt.Sprintf("Error creating PodTemplate %q: %v", ptName, err)
-						return nil, c.handleError(ctx, wl, ac, msg, err)
+						return c.handleError(ctx, wl, ac, msg, err)
 					}
 				}
 
@@ -324,41 +308,21 @@ func (c *Controller) syncOwnedProvisionRequest(
 			}
 
 			if err := ctrl.SetControllerReference(wl, req, c.client.Scheme()); err != nil {
-				return nil, err
+				return err
 			}
 
 			if err := c.client.Create(ctx, req); err != nil {
 				msg := fmt.Sprintf("Error creating ProvisioningRequest %q: %v", requestName, err)
-				return nil, c.handleError(ctx, wl, ac, msg, err)
+				return c.handleError(ctx, wl, ac, msg, err)
 			}
 			c.record.Eventf(wl, corev1.EventTypeNormal, "ProvisioningRequestCreated", "Created ProvisioningRequest: %q", req.Name)
 			activeOrLastPRForChecks[checkName] = req
 		}
 		if err := c.syncProvisionRequestsPodTemplates(ctx, wl, req); err != nil {
-			return nil, err
-		}
-	}
-	return requeAfter, nil
-}
-
-func (c *Controller) remainingTimeToRetry(pr *autoscaling.ProvisioningRequest, failuresCount int32, prc *kueue.ProvisioningRequestConfig) time.Duration {
-	backoffDuration := time.Duration(*prc.Spec.RetryStrategy.BackoffBaseSeconds) * time.Second
-	maxBackoffDuration := time.Duration(*prc.Spec.RetryStrategy.BackoffMaxSeconds) * time.Second
-	var cond *metav1.Condition
-	if isFailed(pr) {
-		cond = apimeta.FindStatusCondition(pr.Status.Conditions, autoscaling.Failed)
-	} else {
-		cond = apimeta.FindStatusCondition(pr.Status.Conditions, autoscaling.BookingExpired)
-	}
-	for i := 1; i < int(failuresCount); i++ {
-		backoffDuration *= 2
-		if backoffDuration >= maxBackoffDuration {
-			backoffDuration = maxBackoffDuration
-			break
+			return err
 		}
 	}
-	timeElapsedSinceLastFailure := time.Since(cond.LastTransitionTime.Time)
-	return backoffDuration - timeElapsedSinceLastFailure
+	return nil
 }
 
 func (c *Controller) handleError(ctx context.Context, wl *kueue.Workload, ac *kueue.AdmissionCheckState, msg string, err error) error {
@@ -574,9 +538,7 @@ func (c *Controller) syncCheckStates(
 					// it is going to be retried
 					message := fmt.Sprintf("Retrying after failure: %s", apimeta.FindStatusCondition(pr.Status.Conditions, autoscaling.Failed).Message)
 					updated = updateCheckMessage(&checkState, message) || updated
-					if features.Enabled(features.KeepQuotaForProvReqRetry) {
-						updated = updateCheckState(&checkState, kueue.CheckStatePending) || updated
-					} else if wl.Status.RequeueState == nil || getAttempt(log, pr, wl.Name, check) > ptr.Deref(wl.Status.RequeueState.Count, 0) {
+					if wl.Status.RequeueState == nil || getAttempt(log, pr, wl.Name, check) > ptr.Deref(wl.Status.RequeueState.Count, 0) {
 						// We don't want to Retry on old ProvisioningRequests
 						updated = true
 						updateCheckState(&checkState, kueue.CheckStateRetry)
@@ -601,9 +563,7 @@ func (c *Controller) syncCheckStates(
 						// it is going to be retried
 						message := fmt.Sprintf("Retrying after booking expired: %s", apimeta.FindStatusCondition(pr.Status.Conditions, autoscaling.BookingExpired).Message)
 						updated = updateCheckMessage(&checkState, message) || updated
-						if features.Enabled(features.KeepQuotaForProvReqRetry) {
-							updated = updateCheckState(&checkState, kueue.CheckStatePending) || updated
-						} else if wl.Status.RequeueState == nil || getAttempt(log, pr, wl.Name, check) > ptr.Deref(wl.Status.RequeueState.Count, 0) {
+						if wl.Status.RequeueState == nil || getAttempt(log, pr, wl.Name, check) > ptr.Deref(wl.Status.RequeueState.Count, 0) {
 							updated = true
 							updateCheckState(&checkState, kueue.CheckStateRetry)
 							workload.UpdateRequeueState(wlPatch, backoffBaseSeconds, backoffMaxSeconds, c.clock)
diff --git a/pkg/features/kube_features.go b/pkg/features/kube_features.go
index 4112695fc..2c9ca6c1b 100644
--- a/pkg/features/kube_features.go
+++ b/pkg/features/kube_features.go
@@ -116,12 +116,6 @@ const (
 	// all currently available ResourceFlavors for the LocalQueue.
 	ExposeFlavorsInLocalQueue featuregate.Feature = "ExposeFlavorsInLocalQueue"
 
-	// owner: @pbundyra
-	// kep: https://github.com/kubernetes-sigs/kueue/tree/main/keps/1136-provisioning-request-support
-	//
-	// Workloads keeps allocated quota and preserves QuotaReserved=True when ProvisioningRequest fails
-	KeepQuotaForProvReqRetry featuregate.Feature = "KeepQuotaForProvReqRetry"
-
 	// owner: @dgrove-oss
 	// kep: https://github.com/kubernetes-sigs/kueue/tree/main/keps/3589-manage-jobs-selectively
 	//
@@ -228,9 +222,6 @@ var defaultVersionedFeatureGates = map[featuregate.Feature]featuregate.Versioned
 	ExposeFlavorsInLocalQueue: {
 		{Version: version.MustParse("0.9"), Default: true, PreRelease: featuregate.Beta},
 	},
-	KeepQuotaForProvReqRetry: {
-		{Version: version.MustParse("0.9"), Default: false, PreRelease: featuregate.Deprecated},
-	},
 	ManagedJobsNamespaceSelector: {
 		{Version: version.MustParse("0.10"), Default: true, PreRelease: featuregate.Beta},
 	},
diff --git a/site/content/en/docs/concepts/cohort.md b/site/content/en/docs/concepts/cohort.md
new file mode 100644
index 000000000..b8e8efdec
--- /dev/null
+++ b/site/content/en/docs/concepts/cohort.md
@@ -0,0 +1,100 @@
+---
+title: "Cohort"
+date: 2025-04-16
+weight: 4
+description: >
+  A cluster-scoped resource for organizing quotas
+---
+
+## Hello, Cohorts
+Cohorts give you the ability to organize your Quotas. ClusterQueues within the same Cohort (or same CohortTree for [Hieararchical Cohorts](#hierarchical-cohorts)) can share resources with each other. The simplest possible Cohort is the following:
+
+```yaml
+apiVersion: kueue.x-k8s.io/v1alpha1
+kind: Cohort
+metadata:
+  name: "hello-cohort"
+```
+
+A ClusterQueue may join this Cohort by referencing it:
+```yaml
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: ClusterQueue
+metadata:
+  name: "my-cluster-queue"
+spec:
+  cohort: "hello-cohort"
+```
+
+## Configuring Quotas
+
+Similarly to [ClusterQueues](/docs/concepts/cluster_queue/#flavors-and-resources) Resource quotas may be defined at the Cohort level, and consumed by ClusterQueues within the Cohort.
+
+```yaml
+apiVersion: kueue.x-k8s.io/v1alpha1
+kind: Cohort
+metadata:
+  name: "hello-cohort"
+spec:
+  resourceGroups:
+    - coveredResources: ["cpu"]
+      flavors:
+      - name: "default-flavor"
+        resources:
+        - name: "cpu"
+          nominalQuota: 12
+```
+
+In order for a ClusterQueue to borrow resources from its Cohort, it **must**
+define nominal quota for the desired Resource and Flavor -  even if this value is 0.
+
+```yaml
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: ClusterQueue
+metadata:
+  name: "my-cluster-queue"
+spec:
+  cohort: "hello-cohort"
+  resourceGroups:
+  - coveredResources: ["cpu"]
+    flavors:
+    - name: "default-flavor"
+      resources:
+      - name: "cpu"
+        nominalQuota: 0
+```
+
+## Hierarchical Cohorts
+Cohorts may be organized in a tree structure. We refer to the grouping of ClusterQueues and Cohorts that are part of the same tree as a **CohortTree**.
+
+ClusterQueues within a given CohortTree may use resources within it,
+subject to [Borrowing and Lending limits](/docs/reference/kueue.v1beta1/#kueue-x-k8s-io-v1beta1-ResourceQuota).
+These Borrowing and Lending Limits can be specified for Cohorts, as well as for ClusterQueues.
+
+Here is a simple CohortTree, with three Cohorts:
+```yaml
+apiVersion: kueue.x-k8s.io/v1alpha1
+kind: Cohort
+metadata:
+  name: "root-cohort"
+---
+apiVersion: kueue.x-k8s.io/v1alpha1
+kind: Cohort
+metadata:
+  name: "important-org"
+spec:
+  parent: "root-cohort"
+  fairSharing:
+    weight: "0.75"
+---
+apiVersion: kueue.x-k8s.io/v1alpha1
+kind: Cohort
+metadata:
+  name: "regular-org"
+spec:
+  parent: "root-cohort"
+  fairSharing:
+    weight: "0.25"
+```
+
+This example assumes that Fair Sharing is enabled. In this case, the important org will trend towards using 75% of common resources, while the regular org towards using 25%.
diff --git a/site/content/en/docs/installation/_index.md b/site/content/en/docs/installation/_index.md
index edc5963ca..07657d2d5 100644
--- a/site/content/en/docs/installation/_index.md
+++ b/site/content/en/docs/installation/_index.md
@@ -28,7 +28,7 @@ description: >
 
 Make sure the following conditions are met:
 
-- A Kubernetes cluster with version 1.25 or newer is running. Learn how to [install the Kubernetes tools](https://kubernetes.io/docs/tasks/tools/).
+- A Kubernetes cluster with version 1.29 or newer is running. Learn how to [install the Kubernetes tools](https://kubernetes.io/docs/tasks/tools/).
 - The `SuspendJob` [feature gate][feature_gate] is enabled. In Kubernetes 1.22 or newer, the feature gate is enabled by default.
 - (Optional) The `JobMutableNodeSchedulingDirectives` [feature gate][feature_gate] (available in Kubernetes 1.22 or newer) is enabled.
   In Kubernetes 1.23 or newer, the feature gate is enabled by default.
@@ -294,7 +294,6 @@ spec:
 |---------------------------------------|---------|------------|-------|-------|
 | `QueueVisibility`                     | `false` | Alpha      | 0.4   | 0.9   |
 | `QueueVisibility`                     | `false` | Deprecated | 0.9   |       |
-| `KeepQuotaForProvReqRetry`            | `false` | Deprecated | 0.9   |       |
 | `MultiplePreemptions`                 | `false` | Alpha      | 0.8   | 0.8   |
 | `MultiplePreemptions`                 | `true`  | Beta       | 0.9   | 0.9   |
 | `MultiplePreemptions`                 | `true`  | GA         | 0.10  |       |
diff --git a/site/content/en/docs/overview/_index.md b/site/content/en/docs/overview/_index.md
index 86f063320..98e37e036 100644
--- a/site/content/en/docs/overview/_index.md
+++ b/site/content/en/docs/overview/_index.md
@@ -25,7 +25,7 @@ A core design principle for Kueue is to avoid duplicating mature functionality i
 ## Features overview
 
 - **Job management:** Support job queueing based on [priorities](/docs/concepts/workload/#priority) with different [strategies](/docs/concepts/cluster_queue/#queueing-strategy): `StrictFIFO` and `BestEffortFIFO`.
-- **Advanced Resource management:** Comprising: [resource flavor fungibility](/docs/concepts/cluster_queue/#flavorfungibility), [Fair Sharing](/docs/concepts/preemption/#fair-sharing), [cohorts](/docs/concepts/cluster_queue/#cohort) and [preemption](/docs/concepts/cluster_queue/#preemption) with a variety of policies between different tenants.
+- **Advanced Resource management:** Comprising: [resource flavor fungibility](/docs/concepts/cluster_queue/#flavorfungibility), [Fair Sharing](/docs/concepts/preemption/#fair-sharing), [Cohorts](/docs/concepts/cohort) and [preemption](/docs/concepts/cluster_queue/#preemption) with a variety of policies between different tenants.
 - **Integrations:** Built-in support for popular jobs, e.g. [BatchJob](/docs/tasks/run/jobs/), [Kubeflow training jobs](/docs/tasks/run/kubeflow/), [RayJob](/docs/tasks/run/rayjobs/), [RayCluster](/docs/tasks/run/rayclusters/), [JobSet](/docs/tasks/run/jobsets/),  [AppWrappers](/docs/tasks/run/appwrappers/), [plain Pod and Pod Groups](/docs/tasks/run/plain_pods/).
 - **System insight:** Build-in [prometheus metrics](/docs/reference/metrics/) to help monitor the state of the system, and on-demand visibility endpoint for [monitoring of pending workloads](/docs/tasks/manage/monitor_pending_workloads/pending_workloads_on_demand/).
 - **AdmissionChecks:** A mechanism for internal or external components to influence whether a workload can be [admitted](/docs/concepts/admission_check/).



Test patch:
diff --git a/charts/kueue/tests/manager_test.yaml b/charts/kueue/tests/manager_test.yaml
new file mode 100644
index 000000000..875dcf28c
--- /dev/null
+++ b/charts/kueue/tests/manager_test.yaml
@@ -0,0 +1,15 @@
+suite: test manager deployment
+templates:
+  - manager/manager.yaml
+  # This is required for the annotation `charts.kueue.x-k8s.io/config-checksum`
+  - manager/manager-config.yaml
+tests:
+  - it: should set replicas correctly
+    template: manager/manager.yaml
+    set:
+      controllerManager:
+        replicas: 2
+    asserts:
+      - equal:
+          path: spec.replicas
+          value: 2
diff --git a/pkg/controller/admissionchecks/provisioning/controller_test.go b/pkg/controller/admissionchecks/provisioning/controller_test.go
index 03b053f17..3af8181de 100644
--- a/pkg/controller/admissionchecks/provisioning/controller_test.go
+++ b/pkg/controller/admissionchecks/provisioning/controller_test.go
@@ -516,36 +516,6 @@ func TestReconcile(t *testing.T) {
 			templates:            []corev1.PodTemplate{*baseTemplate1.DeepCopy(), *baseTemplate2.DeepCopy()},
 			wantRequestsNotFound: []string{"wl-check1"},
 		},
-		"KeepQuotaForProvReqRetry; when request fails and is retried": {
-			workload: baseWorkload.DeepCopy(),
-			checks:   []kueue.AdmissionCheck{*baseCheck.DeepCopy()},
-			flavors:  []kueue.ResourceFlavor{*baseFlavor1.DeepCopy(), *baseFlavor2.DeepCopy()},
-			configs:  []kueue.ProvisioningRequestConfig{*baseConfigWithRetryStrategy.Clone().RetryLimit(2).BaseBackoff(0).Obj()},
-			requests: []autoscaling.ProvisioningRequest{
-				*requestWithCondition(baseRequest, autoscaling.Failed, metav1.ConditionTrue),
-			},
-			enableGates: []featuregate.Feature{features.KeepQuotaForProvReqRetry},
-			templates:   []corev1.PodTemplate{*baseTemplate1.DeepCopy(), *baseTemplate2.DeepCopy()},
-			wantWorkloads: map[string]*kueue.Workload{
-				baseWorkload.GetName(): (&utiltesting.WorkloadWrapper{Workload: *baseWorkload.DeepCopy()}).
-					AdmissionChecks(kueue.AdmissionCheckState{
-						Name:    "check1",
-						State:   kueue.CheckStatePending,
-						Message: "Retrying after failure: ",
-					}, kueue.AdmissionCheckState{
-						Name:  "not-provisioning",
-						State: kueue.CheckStatePending,
-					}).
-					Obj(),
-			},
-			wantEvents: []utiltesting.EventRecord{
-				{
-					Key:       client.ObjectKeyFromObject(baseWorkload),
-					EventType: corev1.EventTypeNormal,
-					Reason:    "ProvisioningRequestCreated",
-					Message:   `Created ProvisioningRequest: "wl-check1-2"`,
-				},
-			}},
 		"when request fails and is retried": {
 			workload: baseWorkload.DeepCopy(),
 			checks:   []kueue.AdmissionCheck{*baseCheck.DeepCopy()},
@@ -971,92 +941,6 @@ func TestReconcile(t *testing.T) {
 					Obj(),
 			},
 		},
-		"KeepQuotaForProvReqRetry; workload retries the admission check when is not admitted and receives the provisioning request's BookingExpired condition": {
-			workload: (&utiltesting.WorkloadWrapper{Workload: *baseWorkload.DeepCopy()}).
-				Admitted(false).
-				Obj(),
-			checks:      []kueue.AdmissionCheck{*baseCheck.DeepCopy()},
-			flavors:     []kueue.ResourceFlavor{*baseFlavor1.DeepCopy(), *baseFlavor2.DeepCopy()},
-			configs:     []kueue.ProvisioningRequestConfig{*baseConfigWithRetryStrategy.Clone().RetryLimit(1).BaseBackoff(0).Obj()},
-			enableGates: []featuregate.Feature{features.KeepQuotaForProvReqRetry},
-			requests: []autoscaling.ProvisioningRequest{
-				*requestWithConditions(baseRequest,
-					[]metav1.Condition{
-						{
-							Type:   autoscaling.Failed,
-							Status: metav1.ConditionFalse,
-						},
-						{
-							Type:   autoscaling.Provisioned,
-							Status: metav1.ConditionTrue,
-						},
-						{
-							Type:   autoscaling.Accepted,
-							Status: metav1.ConditionTrue,
-						},
-						{
-							Type:   autoscaling.BookingExpired,
-							Status: metav1.ConditionTrue,
-						},
-					}),
-			},
-			wantWorkloads: map[string]*kueue.Workload{
-				baseWorkload.GetName(): (&utiltesting.WorkloadWrapper{Workload: *baseWorkload.DeepCopy()}).
-					AdmissionChecks(kueue.AdmissionCheckState{
-						Name:    "check1",
-						State:   kueue.CheckStatePending,
-						Message: "Retrying after booking expired: ",
-					}, kueue.AdmissionCheckState{
-						Name:  "not-provisioning",
-						State: kueue.CheckStatePending,
-					}).
-					Admitted(false).
-					Obj(),
-			},
-			wantRequests: map[string]*autoscaling.ProvisioningRequest{
-				ProvisioningRequestName("wl", kueue.AdmissionCheckReference(baseCheck.Name), 2): {
-					ObjectMeta: metav1.ObjectMeta{
-						Namespace: TestNamespace,
-						Name:      "wl-check1-2",
-						Labels: map[string]string{
-							constants.ManagedByKueueLabelKey: constants.ManagedByKueueLabelValue,
-						},
-						OwnerReferences: []metav1.OwnerReference{
-							{
-								Name: "wl",
-							},
-						},
-					},
-					Spec: autoscaling.ProvisioningRequestSpec{
-						PodSets: []autoscaling.PodSet{
-							{
-								PodTemplateRef: autoscaling.Reference{
-									Name: "ppt-wl-check1-2-ps1",
-								},
-								Count: 4,
-							},
-							{
-								PodTemplateRef: autoscaling.Reference{
-									Name: "ppt-wl-check1-2-ps2",
-								},
-								Count: 3,
-							},
-						},
-						ProvisioningClassName: "class1",
-						Parameters: map[string]autoscaling.Parameter{
-							"p1": "v1",
-						},
-					},
-				},
-			},
-			wantEvents: []utiltesting.EventRecord{
-				{
-					Key:       types.NamespacedName{Namespace: "ns", Name: "wl"},
-					EventType: corev1.EventTypeNormal,
-					Reason:    "ProvisioningRequestCreated",
-					Message:   `Created ProvisioningRequest: "wl-check1-2"`,
-				},
-			}},
 		"workload retries the admission check when is not admitted and receives the provisioning request's BookingExpired condition": {
 			workload: (&utiltesting.WorkloadWrapper{Workload: *baseWorkload.DeepCopy()}).
 				Admitted(false).
diff --git a/test/integration/singlecluster/controller/admissionchecks/provisioning/provisioning_test.go b/test/integration/singlecluster/controller/admissionchecks/provisioning/provisioning_test.go
index 039ca2bbc..f34ab6872 100644
--- a/test/integration/singlecluster/controller/admissionchecks/provisioning/provisioning_test.go
+++ b/test/integration/singlecluster/controller/admissionchecks/provisioning/provisioning_test.go
@@ -18,7 +18,6 @@ package provisioning
 
 import (
 	"fmt"
-	"time"
 
 	"github.com/google/go-cmp/cmp/cmpopts"
 	"github.com/onsi/ginkgo/v2"
@@ -35,7 +34,6 @@ import (
 	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta1"
 	"sigs.k8s.io/kueue/pkg/constants"
 	"sigs.k8s.io/kueue/pkg/controller/admissionchecks/provisioning"
-	"sigs.k8s.io/kueue/pkg/features"
 	"sigs.k8s.io/kueue/pkg/util/testing"
 	"sigs.k8s.io/kueue/pkg/workload"
 	"sigs.k8s.io/kueue/test/integration/framework"
@@ -896,584 +894,6 @@ var _ = ginkgo.Describe("Provisioning", ginkgo.Ordered, ginkgo.ContinueOnFailure
 		})
 	})
 
-	ginkgo.When("Feature KeepQuotaForProvReqRetry is enabled and a workload is using a pending provision admission check with BackOffLimitCount=0", func() {
-		var (
-			ns             *corev1.Namespace
-			wlKey          types.NamespacedName
-			provReqKey     types.NamespacedName
-			ac             *kueue.AdmissionCheck
-			pendingAC      *kueue.AdmissionCheck
-			prc            *kueue.ProvisioningRequestConfig
-			rf             *kueue.ResourceFlavor
-			cq             *kueue.ClusterQueue
-			lq             *kueue.LocalQueue
-			admission      *kueue.Admission
-			createdRequest autoscaling.ProvisioningRequest
-			updatedWl      kueue.Workload
-		)
-		ginkgo.JustBeforeEach(func() {
-			ginkgo.By("Enabling KeepQuotaForProvReqRetry feature", func() {
-				features.SetFeatureGateDuringTest(ginkgo.GinkgoTB(), features.KeepQuotaForProvReqRetry, true)
-			})
-
-			ns = util.CreateNamespaceFromPrefixWithLog(ctx, k8sClient, "provisioning-")
-
-			prc = baseConfigWithParameters.Clone().RetryLimit(0).Obj()
-
-			util.MustCreate(ctx, k8sClient, prc)
-
-			ac = testing.MakeAdmissionCheck("ac-prov").
-				ControllerName(kueue.ProvisioningRequestControllerName).
-				Parameters(kueue.GroupVersion.Group, "ProvisioningRequestConfig", prc.Name).
-				Obj()
-			util.MustCreate(ctx, k8sClient, ac)
-
-			pendingAC = testing.MakeAdmissionCheck("pending-ac").
-				ControllerName("dummy-controller").
-				Obj()
-			util.MustCreate(ctx, k8sClient, pendingAC)
-
-			rf = testing.MakeResourceFlavor(flavorOnDemand).NodeLabel("ns1", "ns1v").Obj()
-			util.MustCreate(ctx, k8sClient, rf)
-
-			cq = testing.MakeClusterQueue("cluster-queue").
-				ResourceGroup(*testing.MakeFlavorQuotas(flavorOnDemand).
-					Resource(resourceGPU, "5", "5").Obj()).
-				Cohort("cohort").
-				AdmissionChecks(kueue.AdmissionCheckReference(ac.Name), kueue.AdmissionCheckReference(pendingAC.Name)).
-				Obj()
-			util.MustCreate(ctx, k8sClient, cq)
-			gomega.Eventually(func(g gomega.Gomega) {
-				g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(cq), cq)).Should(gomega.Succeed())
-				g.Expect(cq.Status.Conditions).To(gomega.ContainElements(gomega.BeComparableTo(metav1.Condition{
-					Type:    kueue.ClusterQueueActive,
-					Status:  metav1.ConditionFalse,
-					Reason:  "AdmissionCheckInactive",
-					Message: "Can't admit new workloads: references inactive AdmissionCheck(s): [pending-ac].",
-				}, util.IgnoreConditionTimestampsAndObservedGeneration)))
-			}, util.Timeout, util.Interval).Should(gomega.Succeed())
-
-			lq = testing.MakeLocalQueue("queue", ns.Name).ClusterQueue(cq.Name).Obj()
-			util.MustCreate(ctx, k8sClient, lq)
-			gomega.Eventually(func(g gomega.Gomega) {
-				g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(lq), lq)).Should(gomega.Succeed())
-				g.Expect(lq.Status.Conditions).To(gomega.ContainElements(gomega.BeComparableTo(metav1.Condition{
-					Type:    kueue.LocalQueueActive,
-					Status:  metav1.ConditionFalse,
-					Reason:  "ClusterQueueIsInactive",
-					Message: "Can't submit new workloads to clusterQueue",
-				}, util.IgnoreConditionTimestampsAndObservedGeneration)))
-			}, util.Timeout, util.Interval).Should(gomega.Succeed())
-
-			wl := testing.MakeWorkload("wl", ns.Name).
-				Queue(lq.Name).
-				PodSets(
-					*testing.MakePodSet("ps1", 3).
-						Request(corev1.ResourceCPU, "1").
-						Image("image").
-						Obj(),
-					*testing.MakePodSet("ps2", 6).
-						Request(corev1.ResourceCPU, "500m").
-						Request(customResourceOne, "1").
-						Limit(customResourceOne, "1").
-						Image("image").
-						Obj(),
-				).
-				Annotations(map[string]string{
-					"provreq.kueue.x-k8s.io/ValidUntilSeconds": "0",
-					"invalid-provreq-prefix/Foo":               "Bar"}).
-				Obj()
-			util.MustCreate(ctx, k8sClient, wl)
-			util.SetWorkloadsAdmissionCheck(ctx, k8sClient, wl, kueue.AdmissionCheckReference(pendingAC.Name), kueue.CheckStateReady, false)
-
-			wlKey = client.ObjectKeyFromObject(wl)
-			provReqKey = types.NamespacedName{
-				Namespace: wlKey.Namespace,
-				Name:      provisioning.ProvisioningRequestName(wlKey.Name, kueue.AdmissionCheckReference(ac.Name), 1),
-			}
-
-			admission = testing.MakeAdmission(cq.Name).
-				PodSets(
-					kueue.PodSetAssignment{
-						Name: "ps1",
-						Flavors: map[corev1.ResourceName]kueue.ResourceFlavorReference{
-							corev1.ResourceCPU: kueue.ResourceFlavorReference(rf.Name),
-						},
-						ResourceUsage: map[corev1.ResourceName]resource.Quantity{
-							corev1.ResourceCPU: resource.MustParse("3"),
-						},
-						Count: ptr.To[int32](3),
-					},
-					kueue.PodSetAssignment{
-						Name: "ps2",
-						Flavors: map[corev1.ResourceName]kueue.ResourceFlavorReference{
-							corev1.ResourceCPU: kueue.ResourceFlavorReference(rf.Name),
-						},
-						ResourceUsage: map[corev1.ResourceName]resource.Quantity{
-							corev1.ResourceCPU: resource.MustParse("2"),
-						},
-						Count: ptr.To[int32](4),
-					},
-				).
-				Obj()
-		})
-
-		ginkgo.AfterEach(func() {
-			gomega.Expect(util.DeleteNamespace(ctx, k8sClient, ns)).To(gomega.Succeed())
-			util.ExpectObjectToBeDeleted(ctx, k8sClient, cq, true)
-			util.ExpectObjectToBeDeleted(ctx, k8sClient, rf, true)
-			util.ExpectObjectToBeDeleted(ctx, k8sClient, ac, true)
-			util.ExpectObjectToBeDeleted(ctx, k8sClient, pendingAC, true)
-			util.ExpectObjectToBeDeleted(ctx, k8sClient, prc, true)
-		})
-
-		ginkgo.It("Should reject an admission check if a workload is not Admitted, the ProvisioningRequest's condition is set to BookingExpired", func() {
-			ginkgo.By("Setting the admission check to the workload", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).Should(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-
-				util.SetWorkloadsAdmissionCheck(ctx, k8sClient, &updatedWl, kueue.AdmissionCheckReference(ac.Name), kueue.CheckStatePending, false)
-				util.SetWorkloadsAdmissionCheck(ctx, k8sClient, &updatedWl, kueue.AdmissionCheckReference(pendingAC.Name), kueue.CheckStatePending, true)
-			})
-
-			ginkgo.By("Setting the quota reservation to the workload", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).Should(gomega.Succeed())
-					gomega.Expect(util.SetQuotaReservation(ctx, k8sClient, &updatedWl, admission)).To(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Setting the provisioning request as Provisioned", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, provReqKey, &createdRequest)).To(gomega.Succeed())
-					apimeta.SetStatusCondition(&createdRequest.Status.Conditions, metav1.Condition{
-						Type:   autoscaling.Provisioned,
-						Status: metav1.ConditionTrue,
-						Reason: autoscaling.Provisioned,
-					})
-					g.Expect(k8sClient.Status().Update(ctx, &createdRequest)).Should(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Checking if one admission check is ready, and the other is pending thus workload is not admitted", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-
-					g.Expect(updatedWl.Status.AdmissionChecks).To(gomega.ContainElement(gomega.BeComparableTo(
-						kueue.AdmissionCheckState{
-							Name:  kueue.AdmissionCheckReference(ac.Name),
-							State: kueue.CheckStateReady,
-						},
-						cmpopts.IgnoreFields(kueue.AdmissionCheckState{}, "LastTransitionTime", "PodSetUpdates"))))
-					g.Expect(updatedWl.Status.AdmissionChecks).To(gomega.ContainElement(gomega.BeComparableTo(
-						kueue.AdmissionCheckState{
-							Name:  kueue.AdmissionCheckReference(pendingAC.Name),
-							State: kueue.CheckStatePending,
-						},
-						cmpopts.IgnoreFields(kueue.AdmissionCheckState{}, "LastTransitionTime", "PodSetUpdates"))))
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-					g.Expect(workload.IsAdmitted(&updatedWl)).To(gomega.BeFalse(), "The Workload should not be admitted")
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Setting the ProvisioningRequest as BookingExpired", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, provReqKey, &createdRequest)).Should(gomega.Succeed())
-					apimeta.SetStatusCondition(&createdRequest.Status.Conditions, metav1.Condition{
-						Type:   autoscaling.BookingExpired,
-						Status: metav1.ConditionTrue,
-						Reason: autoscaling.BookingExpired,
-					})
-					g.Expect(k8sClient.Status().Update(ctx, &createdRequest)).Should(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Checking if workload is deactivated, once had Rejected status in the status.admissionCheck[*] field, an event is emitted and a metric is increased", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-					g.Expect(updatedWl.Status.AdmissionChecks).To(gomega.ContainElement(gomega.BeComparableTo(
-						kueue.AdmissionCheckState{
-							Name:    kueue.AdmissionCheckReference(ac.Name),
-							State:   kueue.CheckStatePending,
-							Message: "Reset to Pending after eviction. Previously: Rejected",
-						},
-						cmpopts.IgnoreFields(kueue.AdmissionCheckState{}, "LastTransitionTime", "PodSetUpdates"))))
-					g.Expect(workload.IsActive(&updatedWl)).To(gomega.BeFalse(), "The workload should be deactivated")
-
-					ok, err := testing.HasEventAppeared(ctx, k8sClient, corev1.Event{
-						Reason:  "AdmissionCheckRejected",
-						Type:    corev1.EventTypeWarning,
-						Message: fmt.Sprintf("Deactivating workload because AdmissionCheck for %v was Rejected: ", ac.Name),
-					})
-					g.Expect(err).NotTo(gomega.HaveOccurred())
-					g.Expect(ok).To(gomega.BeTrue(), "The event should have appeared")
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-
-					g.Expect(workload.IsEvictedByDeactivation(&updatedWl)).To(gomega.BeTrue(), "The workload should be evicted by deactivation")
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-
-				util.ExpectEvictedWorkloadsTotalMetric(cq.Name, "DeactivatedDueToAdmissionCheck", 1)
-			})
-		})
-	})
-
-	ginkgo.When("FeatureGate KeepQuotaForProvReqRetry is enabled, and a workload is using a provision admission check with  BackoffLimitCount=1", func() {
-		var (
-			ns             *corev1.Namespace
-			wlKey          types.NamespacedName
-			provReqKey     types.NamespacedName
-			ac             *kueue.AdmissionCheck
-			pendingAC      *kueue.AdmissionCheck
-			prc            *kueue.ProvisioningRequestConfig
-			rf             *kueue.ResourceFlavor
-			cq             *kueue.ClusterQueue
-			lq             *kueue.LocalQueue
-			admission      *kueue.Admission
-			createdRequest autoscaling.ProvisioningRequest
-			updatedWl      kueue.Workload
-		)
-		ginkgo.JustBeforeEach(func() {
-			ginkgo.By("Enabling KeepQuotaForProvReqRetry feature", func() {
-				features.SetFeatureGateDuringTest(ginkgo.GinkgoTB(), features.KeepQuotaForProvReqRetry, true)
-			})
-
-			ns = util.CreateNamespaceFromPrefixWithLog(ctx, k8sClient, "provisioning-")
-			prc = baseConfigWithParameters.Clone().RetryLimit(1).BaseBackoff(1).Obj()
-			util.MustCreate(ctx, k8sClient, prc)
-
-			ac = testing.MakeAdmissionCheck("ac-prov").
-				ControllerName(kueue.ProvisioningRequestControllerName).
-				Parameters(kueue.GroupVersion.Group, "ProvisioningRequestConfig", prc.Name).
-				Obj()
-			util.MustCreate(ctx, k8sClient, ac)
-
-			pendingAC = testing.MakeAdmissionCheck("pending-ac").
-				ControllerName("dummy-controller").
-				Obj()
-			util.MustCreate(ctx, k8sClient, pendingAC)
-
-			rf = testing.MakeResourceFlavor("rf1").Label("ns1", "ns1v").Obj()
-			util.MustCreate(ctx, k8sClient, rf)
-
-			cq = testing.MakeClusterQueue("cluster-queue").
-				ResourceGroup(*testing.MakeFlavorQuotas(flavorOnDemand).
-					Resource(resourceGPU, "5", "5").Obj()).
-				Cohort("cohort").
-				AdmissionChecks(kueue.AdmissionCheckReference(ac.Name), kueue.AdmissionCheckReference(pendingAC.Name)).
-				Obj()
-			util.MustCreate(ctx, k8sClient, cq)
-			lq = testing.MakeLocalQueue("queue", ns.Name).ClusterQueue(cq.Name).Obj()
-			util.MustCreate(ctx, k8sClient, lq)
-			wl := testing.MakeWorkload("wl", ns.Name).
-				Queue(lq.Name).
-				PodSets(
-					*testing.MakePodSet("ps1", 3).
-						Request(corev1.ResourceCPU, "1").
-						Image("image").
-						Obj(),
-					*testing.MakePodSet("ps2", 6).
-						Request(corev1.ResourceCPU, "500m").
-						Request(customResourceOne, "1").
-						Limit(customResourceOne, "1").
-						Image("image").
-						Obj(),
-				).
-				Obj()
-			util.MustCreate(ctx, k8sClient, wl)
-			util.SetWorkloadsAdmissionCheck(ctx, k8sClient, wl, kueue.AdmissionCheckReference(pendingAC.Name), kueue.CheckStateReady, false)
-
-			wlKey = client.ObjectKeyFromObject(wl)
-			provReqKey = types.NamespacedName{
-				Namespace: wlKey.Namespace,
-				Name:      provisioning.ProvisioningRequestName(wlKey.Name, kueue.AdmissionCheckReference(ac.Name), 1),
-			}
-
-			admission = testing.MakeAdmission(cq.Name).
-				PodSets(
-					kueue.PodSetAssignment{
-						Name: "ps1",
-						Flavors: map[corev1.ResourceName]kueue.ResourceFlavorReference{
-							corev1.ResourceCPU: kueue.ResourceFlavorReference(rf.Name),
-						},
-						ResourceUsage: map[corev1.ResourceName]resource.Quantity{
-							corev1.ResourceCPU: resource.MustParse("3"),
-						},
-						Count: ptr.To[int32](3),
-					},
-					kueue.PodSetAssignment{
-						Name: "ps2",
-						Flavors: map[corev1.ResourceName]kueue.ResourceFlavorReference{
-							corev1.ResourceCPU: kueue.ResourceFlavorReference(rf.Name),
-						},
-						ResourceUsage: map[corev1.ResourceName]resource.Quantity{
-							corev1.ResourceCPU: resource.MustParse("2"),
-						},
-						Count: ptr.To[int32](4),
-					},
-				).
-				Obj()
-		})
-
-		ginkgo.AfterEach(func() {
-			gomega.Expect(util.DeleteNamespace(ctx, k8sClient, ns)).To(gomega.Succeed())
-			util.ExpectObjectToBeDeleted(ctx, k8sClient, cq, true)
-			util.ExpectObjectToBeDeleted(ctx, k8sClient, rf, true)
-			util.ExpectObjectToBeDeleted(ctx, k8sClient, ac, true)
-			util.ExpectObjectToBeDeleted(ctx, k8sClient, pendingAC, true)
-			util.ExpectObjectToBeDeleted(ctx, k8sClient, prc, true)
-		})
-
-		ginkgo.It("Should set check as Pending after the first ProvisioningRequest fails, then as Ready if the second Provisioning request succeeds", func() {
-			ginkgo.By("Setting the admission check to the workload", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).Should(gomega.Succeed())
-					util.SetWorkloadsAdmissionCheck(ctx, k8sClient, &updatedWl, kueue.AdmissionCheckReference(ac.Name), kueue.CheckStatePending, false)
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Setting the quota reservation to the workload", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).Should(gomega.Succeed())
-					g.Expect(util.SetQuotaReservation(ctx, k8sClient, &updatedWl, admission)).To(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Setting the provision request-1 as Failed", func() {
-				provReqKey := types.NamespacedName{
-					Namespace: wlKey.Namespace,
-					Name:      provisioning.ProvisioningRequestName(wlKey.Name, kueue.AdmissionCheckReference(ac.Name), 1),
-				}
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, provReqKey, &createdRequest)).Should(gomega.Succeed())
-					apimeta.SetStatusCondition(&createdRequest.Status.Conditions, metav1.Condition{
-						Type:   autoscaling.Failed,
-						Status: metav1.ConditionTrue,
-						Reason: autoscaling.Failed,
-					})
-					g.Expect(k8sClient.Status().Update(ctx, &createdRequest)).Should(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Checking the admission check is pending", func() {
-				// use consistently with short interval to make sure it does not
-				// flip to a short period of time.
-				gomega.Consistently(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-					state := workload.FindAdmissionCheck(updatedWl.Status.AdmissionChecks, kueue.AdmissionCheckReference(ac.Name))
-					g.Expect(state).NotTo(gomega.BeNil())
-					g.Expect(state.State).To(gomega.Equal(kueue.CheckStatePending))
-				}, time.Second, 10*time.Millisecond).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Setting the provision request-2 as Provisioned", func() {
-				provReqKey := types.NamespacedName{
-					Namespace: wlKey.Namespace,
-					Name:      provisioning.ProvisioningRequestName(wlKey.Name, kueue.AdmissionCheckReference(ac.Name), 2),
-				}
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, provReqKey, &createdRequest)).Should(gomega.Succeed())
-					apimeta.SetStatusCondition(&createdRequest.Status.Conditions, metav1.Condition{
-						Type:   autoscaling.Provisioned,
-						Status: metav1.ConditionTrue,
-						Reason: autoscaling.Provisioned,
-					})
-					g.Expect(k8sClient.Status().Update(ctx, &createdRequest)).Should(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Checking the admission check is ready", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-					state := workload.FindAdmissionCheck(updatedWl.Status.AdmissionChecks, kueue.AdmissionCheckReference(ac.Name))
-					g.Expect(state).NotTo(gomega.BeNil())
-					g.Expect(state.State).To(gomega.Equal(kueue.CheckStateReady))
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-		})
-
-		ginkgo.It("Should set check as Rejected, if every ProvisioningRequest retry fails", func() {
-			ginkgo.By("Setting the admission check to the workload", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).Should(gomega.Succeed())
-					util.SetWorkloadsAdmissionCheck(ctx, k8sClient, &updatedWl, kueue.AdmissionCheckReference(ac.Name), kueue.CheckStatePending, false)
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Setting the quota reservation to the workload", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).Should(gomega.Succeed())
-					g.Expect(util.SetQuotaReservation(ctx, k8sClient, &updatedWl, admission)).To(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Setting the provision request-1 as Failed", func() {
-				provReqKey := types.NamespacedName{
-					Namespace: wlKey.Namespace,
-					Name:      provisioning.ProvisioningRequestName(wlKey.Name, kueue.AdmissionCheckReference(ac.Name), 1),
-				}
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, provReqKey, &createdRequest)).Should(gomega.Succeed())
-					apimeta.SetStatusCondition(&createdRequest.Status.Conditions, metav1.Condition{
-						Type:   autoscaling.Failed,
-						Status: metav1.ConditionTrue,
-						Reason: autoscaling.Failed,
-					})
-					g.Expect(k8sClient.Status().Update(ctx, &createdRequest)).Should(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Checking the admission check is pending", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-					state := workload.FindAdmissionCheck(updatedWl.Status.AdmissionChecks, kueue.AdmissionCheckReference(ac.Name))
-					g.Expect(state).NotTo(gomega.BeNil())
-					g.Expect(state.State).To(gomega.Equal(kueue.CheckStatePending))
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Setting the provision request-2 as Failed", func() {
-				provReqKey := types.NamespacedName{
-					Namespace: wlKey.Namespace,
-					Name:      provisioning.ProvisioningRequestName(wlKey.Name, kueue.AdmissionCheckReference(ac.Name), 2),
-				}
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, provReqKey, &createdRequest)).Should(gomega.Succeed())
-					apimeta.SetStatusCondition(&createdRequest.Status.Conditions, metav1.Condition{
-						Type:   autoscaling.Failed,
-						Status: metav1.ConditionTrue,
-						Reason: autoscaling.Failed,
-					})
-					g.Expect(k8sClient.Status().Update(ctx, &createdRequest)).Should(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Checking if workload is deactivated, once had Rejected status in the status.admissionCheck[*] field, an event is emitted and a metric is increased", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-
-					g.Expect(updatedWl.Status.AdmissionChecks).To(gomega.ContainElement(gomega.BeComparableTo(
-						kueue.AdmissionCheckState{
-							Name:    kueue.AdmissionCheckReference(ac.Name),
-							State:   kueue.CheckStatePending,
-							Message: "Reset to Pending after eviction. Previously: Rejected",
-						},
-						cmpopts.IgnoreFields(kueue.AdmissionCheckState{}, "LastTransitionTime", "PodSetUpdates"))))
-					g.Expect(workload.IsActive(&updatedWl)).To(gomega.BeFalse())
-
-					ok, err := testing.HasEventAppeared(ctx, k8sClient, corev1.Event{
-						Reason:  "AdmissionCheckRejected",
-						Type:    corev1.EventTypeWarning,
-						Message: fmt.Sprintf("Deactivating workload because AdmissionCheck for %v was Rejected: ", ac.Name),
-					})
-					g.Expect(err).NotTo(gomega.HaveOccurred())
-					g.Expect(ok).To(gomega.BeTrue())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-
-					g.Expect(workload.IsEvictedByDeactivation(&updatedWl)).To(gomega.BeTrue())
-					util.ExpectEvictedWorkloadsTotalMetric(cq.Name, "DeactivatedDueToAdmissionCheck", 1)
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-		})
-
-		ginkgo.It("Should set a check as Pending if a workload is not Admitted, and the ProvisioningRequest's condition is set to BookingExpired", func() {
-			ginkgo.By("Setting the admission check to the workload", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).Should(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-
-				util.SetWorkloadsAdmissionCheck(ctx, k8sClient, &updatedWl, kueue.AdmissionCheckReference(ac.Name), kueue.CheckStatePending, false)
-				util.SetWorkloadsAdmissionCheck(ctx, k8sClient, &updatedWl, kueue.AdmissionCheckReference(pendingAC.Name), kueue.CheckStatePending, true)
-			})
-
-			ginkgo.By("Setting the quota reservation to the workload", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).Should(gomega.Succeed())
-					g.Expect(util.SetQuotaReservation(ctx, k8sClient, &updatedWl, admission)).To(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Setting the provisioning request as Provisioned", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, provReqKey, &createdRequest)).To(gomega.Succeed())
-					apimeta.SetStatusCondition(&createdRequest.Status.Conditions, metav1.Condition{
-						Type:   autoscaling.Provisioned,
-						Status: metav1.ConditionTrue,
-						Reason: autoscaling.Provisioned,
-					})
-					g.Expect(k8sClient.Status().Update(ctx, &createdRequest)).Should(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Checking if one admission check is ready, and the other is pending thus workload is not admitted", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-					g.Expect(updatedWl.Status.AdmissionChecks).To(gomega.ContainElement(gomega.BeComparableTo(
-						kueue.AdmissionCheckState{
-							Name:  kueue.AdmissionCheckReference(ac.Name),
-							State: kueue.CheckStateReady,
-						},
-						cmpopts.IgnoreFields(kueue.AdmissionCheckState{}, "LastTransitionTime", "PodSetUpdates"))))
-					g.Expect(updatedWl.Status.AdmissionChecks).To(gomega.ContainElement(gomega.BeComparableTo(
-						kueue.AdmissionCheckState{
-							Name:  kueue.AdmissionCheckReference(pendingAC.Name),
-							State: kueue.CheckStatePending,
-						},
-						cmpopts.IgnoreFields(kueue.AdmissionCheckState{}, "LastTransitionTime", "PodSetUpdates"))))
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-					g.Expect(workload.IsAdmitted(&updatedWl)).To(gomega.BeFalse(), "The workload shouldn't be admitted")
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Setting the ProvisioningRequest as BookingExpired", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, provReqKey, &createdRequest)).Should(gomega.Succeed())
-					apimeta.SetStatusCondition(&createdRequest.Status.Conditions, metav1.Condition{
-						Type:   autoscaling.BookingExpired,
-						Status: metav1.ConditionTrue,
-						Reason: autoscaling.BookingExpired,
-					})
-					g.Expect(k8sClient.Status().Update(ctx, &createdRequest)).Should(gomega.Succeed())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-
-			ginkgo.By("Checking if the admission check is pending and the request is retried", func() {
-				gomega.Eventually(func(g gomega.Gomega) {
-					g.Expect(k8sClient.Get(ctx, wlKey, &updatedWl)).To(gomega.Succeed())
-					g.Expect(updatedWl.Status.AdmissionChecks).To(gomega.ContainElement(gomega.BeComparableTo(
-						kueue.AdmissionCheckState{
-							Name:  kueue.AdmissionCheckReference(ac.Name),
-							State: kueue.CheckStatePending,
-						},
-						cmpopts.IgnoreFields(kueue.AdmissionCheckState{}, "LastTransitionTime", "PodSetUpdates", "Message"))))
-
-					provReqKey = types.NamespacedName{
-						Namespace: wlKey.Namespace,
-						Name:      provisioning.ProvisioningRequestName(wlKey.Name, kueue.AdmissionCheckReference(ac.Name), 2),
-					}
-					g.Expect(k8sClient.Get(ctx, provReqKey, &createdRequest)).To(gomega.Succeed())
-					provReqKey = types.NamespacedName{
-						Namespace: wlKey.Namespace,
-						Name:      provisioning.ProvisioningRequestName(wlKey.Name, kueue.AdmissionCheckReference(ac.Name), 1),
-					}
-					g.Expect(k8sClient.Get(ctx, provReqKey, &createdRequest)).To(testing.BeNotFoundError())
-				}, util.Timeout, util.Interval).Should(gomega.Succeed())
-			})
-		})
-	})
-
 	ginkgo.When("Workload uses a provision admission check with BackoffLimitCount=1", func() {
 		var (
 			ns             *corev1.Namespace