Repository: kubernetes-sigs/kueue

Issue:
Incorrect admission due to wrong borrowable resource when a cluster queue in cohort is deleted
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:
ClusterQueue can admit a workload incorrectly through borrowing even when a ClusterQueue in the same cohort is deleted and no borrowable resource is present.

**What you expected to happen**:
When ClusterQueue is deleted, borrowable resources of ClusterQueues in the same cohort should be updated automatically to keep the borrowable resources up-to-date.

**How to reproduce it (as minimally and precisely as possible)**:

1. Create two ClusterQueues in the same cohort and LocalQueues for respective CQs:
  ```yaml
  apiVersion: kueue.x-k8s.io/v1beta1
  kind: ResourceFlavor
  metadata:
    name: default-flavor
  spec: {}
  ---
  # Create two cluster queues in the same cohort
  # 8 CPU nominal quota and 0 CPU nominal quota respectively.
  # cluster queue with 0 CPU nominal quota will have to borrow from the other to admit workloads.
  apiVersion: kueue.x-k8s.io/v1beta1
  kind: ClusterQueue
  metadata:
    name: cluster-queue-1
  spec:
    cohort: bug-test
    namespaceSelector: {}
    resourceGroups:
    - coveredResources: ["cpu"]
      flavors:
      - name: "default-flavor"
        resources:
          - name: cpu
            nominalQuota: "8"
  ---
  apiVersion: kueue.x-k8s.io/v1beta1
  kind: ClusterQueue
  metadata:
    name: cluster-queue-2
  spec:
    cohort: bug-test
    namespaceSelector: {}
    resourceGroups:
    - coveredResources: ["cpu"]
      flavors:
      - name: "default-flavor"
        resources:
          - name: cpu
            nominalQuota: "0"
  ---
  # Create local queues for the corresponding cluster queues.
  apiVersion: kueue.x-k8s.io/v1beta1
  kind: LocalQueue
  metadata:
    name: local-queue-1
    namespace: test
  spec:
    clusterQueue: cluster-queue-1
  ---
  apiVersion: kueue.x-k8s.io/v1beta1
  kind: LocalQueue
  metadata:
    name: local-queue-2
    namespace: test
  spec:
    clusterQueue: cluster-queue-2
  
  ```
2. Delete **cluster-queue-1** which has 8 CPU nominal quota. This will result in 0 borrowable CPU and 0 nominal CPU quota for **cluster-queue-2**
3. Create a pod which requests 8 CPU to **cluster-queue-2**. This is supposed to be scheduling gated but you'll see it pending, passing the admission.
```yaml
# A pod requesting 8 CPUs
apiVersion: v1
kind: Pod
metadata:
  name: pod-8-cpu
  namespace: test
  labels:
    kueue.x-k8s.io/queue-name: local-queue-2
spec:
  containers:
  - name: sleep
    command: ["sleep", "infinity"]
    image: busybox:stable
    resources:
      requests:
        cpu: "8"
```

**Anything else we need to know?**:

We can remediate this situation by:
* Restarting the Kueue controller manager.
* Submitting another workload in cluster-queue-2 (which seems to trigger sync).

However since there could be an incorrect admission until we take this remediation, it'd be best if controller could remediate by itself.

**Environment**:
- Kubernetes version (use `kubectl version`): v1.29.4
- Kueue version (use `git describe --tags --dirty --always`): v0.12.4 (latest) and v0.11.6

Hints from issue comments:
@mimowo: cc @gabesaba @amy: I am able to replicate this issue locally. I'm going to...
1. Take a look at whether a ClusterQueue deletion triggers an event for LocalQueue
2. Take a look at when the cache representation of the tree gets synced after a ClusterQueue deletion
3. If the cache does get synced after a ClusterQueue deletion, what it looks like@amy: Adding findings here so folks can follow along / correct me as I debug:

For this line: `cqName, cqOk := r.queues.ClusterQueueForWorkload(&wl)`
I added a log here to print the values for cqName and cqOk, I get:
`clusterQueue:  cluster-queue-2, cq exists:  true`

https://github.pie.apple.com/kubernetes-sigs/kueue/blob/main/pkg/controller/core/workload_controller.go#L298
@amy: I think I found a fix! The issue is within `cache.DeleteClusterQueue` function. We need to add ` updateCohortTreeResources(parent)` after clusterqueue deletion. 

I see the pod with `SchedulingGated` locally. Will send up a PR in the upcoming days! (need a second to get approvals with my employer)@mimowo: Awesome, this sounds quite reasonable! I think ideally the fix is accompanied by an integration test.@amy: For curiosity purposes, is this an issue? https://github.com/kubernetes-sigs/kueue/issues/5937#issuecomment-3071819429

Want to make sure my solution isn't roundabout. Specifically, this means that `queues *queue.Manager` in `WorkloadReconciler` still sees this workload attached to the deleted cq. My solution separately triggers the `cache` to reconcile the resources to prevent pod admission. But the `queue` representation sees the deleted cq even after cq deletion. Which means the deletion event isn't reconciling the queue. 

Prior to my local fix, when I print its:
--delete cq 1--

1. logs from cq controller - triggers cq delete event

--submit workload--

2. logs from workload controller -  workload has reservation
3. logs from workload controller - workload queue.Manager returns deleted CQ as workload's CQ

I would expect my fix (that updates `cache` resource tree) to resolve 2. But also, 3 is weird because I would have expected the workload controller `queue` representation to have been updated. The workload should be orphaned.

@mimowo ^@mimowo: Maybe @gabesaba can answer in more detail, but AFAIK deleting the CQ from `queues` sounds like a proper fix, here are more details:

> logs from workload controller - workload has reservation

This sounds expected. Even though the quota for cohort is reduced, the Workload still has the `QuotaReserved` condition. Kueue does not have a mechanism to preempt due to reduced quota (either by lowering the numbers or deleting a CQ contributing to the cohort's quota). So, as a result CQs and cohorts may run over-committed for a while. 

We may consider supporting triggering preemptions in this case, but I would call it out-of-scope, and limit the scope of the issue to only prevent "new" admissions.

> logs from workload controller - workload queue.Manager returns deleted CQ as workload's CQ

Is the workload targeting the deleted CQ, or the other one in the cohort?
- if the other CQ (local-queue-2, as in the Issue description), then it is perfectly ok. 
- if the workload is targeting local-queue-1, then indeed the internal functions may get confused, because the `local-queue-1` is remembered at the [QueueName status field](https://github.com/kubernetes-sigs/kueue/blob/f4a78e032c0d116860d7dff15d445952a65e6904/apis/kueue/v1beta1/workload_types.go#L42). So, if `local-queue-1` exists, while `cluster-queue-1` is deleted, then it might be confusing, but we generally prevent it with finalizers. Let me know if you want to explore these scenarios more.@amy: ah ok nevermind. Yeah the workload is targeting `cluster-queue-2`, so not the deleted one. Alrighty! I'll go ahead with my original suggestion: updating `cache.DeleteClusterQueue` function.

This is interesting. I didn't realize when borrowing, that a workload borrows from a specific clusterQueue. I falsely assumed that the cohort aggregates unallocated guarantees. And that when a clusterQueue needs to borrow, that it borrows from an aggregated pool. (😂 assumption made from not reading the code)

@mimowo Re: running overcommitted, yeah... I think eventual consistency may actually be preferable here (ie not preempting in the scenario you described). So for instance, if we're running migrations with cohorts (moving quota around), it'd be nice for already running workloads to have a somewhat graceful end & finish using their granted reservation, then just block on admission.@mimowo: > This is interesting. I didn't realize when borrowing, that a workload borrows from a specific clusterQueue. I falsely assumed that the cohort aggregates unallocated guarantees. And that when a clusterQueue needs to borrow, that it borrows from an aggregated pool. (😂 assumption made from not reading the code)

Actually, we are borrowing from the aggregated pool rather than a specific CQ. In this specific case of 2 CQs it is actually the same, so sometimes we shortcut saying 'borrowing from another CQ. More generically the aggregated pool description is better. 

More precisely tbis is an aggregated pool of lendable resources. If lendingLimit is used then it might be less than nominalQuota.@amy: /assign

Solution patch:
diff --git a/pkg/cache/cache.go b/pkg/cache/cache.go
index 36d1da4fb..781947e96 100644
--- a/pkg/cache/cache.go
+++ b/pkg/cache/cache.go
@@ -476,8 +476,16 @@ func (c *Cache) DeleteClusterQueue(cq *kueue.ClusterQueue) {
 			})
 		}
 	}
+
+	parent := curCq.Parent()
+
 	c.hm.DeleteClusterQueue(cqName)
 	metrics.ClearCacheMetrics(cq.Name)
+
+	// Update cohort resources after deletion
+	if parent != nil {
+		updateCohortTreeResourcesIfNoCycle(parent)
+	}
 }
 
 func (c *Cache) AddOrUpdateCohort(apiCohort *kueue.Cohort) error {
diff --git a/pkg/cache/resource_node.go b/pkg/cache/resource_node.go
index 4c0ea3926..5e6248dc9 100644
--- a/pkg/cache/resource_node.go
+++ b/pkg/cache/resource_node.go
@@ -196,6 +196,16 @@ func updateCohortResourceNode(cohort *cohort) {
 	}
 }
 
+// updateCohortTreeResourcesIfNoCycle only updates Cohort tree resources if there's no cycle. Purpose
+// of this function is to reduce instances where errors from updateCohortTreeResources are silenced.
+// Errors from event handlers are not retried, so its better to execute updateCohortResourceNode when no error
+// would be produced.
+func updateCohortTreeResourcesIfNoCycle(cohort *cohort) {
+	if !hierarchy.HasCycle(cohort) {
+		updateCohortResourceNode(cohort.getRootUnsafe())
+	}
+}
+
 func accumulateFromChild(parent *cohort, child flatResourceNode) {
 	for fr, childQuota := range child.getResourceNode().SubtreeQuota {
 		parent.resourceNode.SubtreeQuota[fr] += childQuota - child.getResourceNode().guaranteedQuota(fr)
diff --git a/site/content/zh-CN/docs/admission-check-controllers/_index.md b/site/content/zh-CN/docs/admission-check-controllers/_index.md
index e08be8107..4cd4d2d6a 100644
--- a/site/content/zh-CN/docs/admission-check-controllers/_index.md
+++ b/site/content/zh-CN/docs/admission-check-controllers/_index.md
@@ -3,6 +3,6 @@ title: "准入检查"
 linkTitle: "准入检查"
 weight: 10
 description: >
-  This section describes the Admission Checks that are included with Kueue.
+  本节描述了 Kueue 中包含的准入检查控制器。
 ---
-You can read the [Concepts](/docs/concepts) section to learn how [Admission Checks](/docs/concepts/admission_check/) influence admission.
+您可以阅读[概念](/docs/concepts)部分了解[准入检查](/docs/concepts/admission_check/)如何影响准入。
diff --git a/site/content/zh-CN/docs/admission-check-controllers/provisioning.md b/site/content/zh-CN/docs/admission-check-controllers/provisioning.md
index 5138c3eb0..6d647e3c5 100644
--- a/site/content/zh-CN/docs/admission-check-controllers/provisioning.md
+++ b/site/content/zh-CN/docs/admission-check-controllers/provisioning.md
@@ -1,38 +1,49 @@
 ---
-title: "Provisioning Admission Check Controller"
+title: "Provisioning 准入检查控制器"
 date: 2023-10-23
 weight: 1
 description: >
-  An admission check controller providing kueue integration with cluster autoscaler.
+  一个提供 Kueue 与集群自动缩放器集成的准入检查控制器。
 ---
 
-The Provisioning AdmissionCheck Controller is an AdmissionCheck Controller designed to integrate Kueue with [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler). Its primary function is to create [ProvisioningRequests](https://github.com/kubernetes/autoscaler/blob/4872bddce2bcc5b4a5f6a3d569111c11b8a2baf4/cluster-autoscaler/provisioningrequest/apis/autoscaling.x-k8s.io/v1beta1/types.go#L41) for the workloads holding [Quota Reservation](/docs/concepts/#quota-reservation) and keeping the [AdmissionCheckState](/docs/concepts/admission_check/#admissioncheckstate) in sync.
+Provisioning 准入检查控制器(AdmissionCheck Controller)是一个专为将 Kueue 与
+[Kubernetes 集群自动缩放器](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)
+集成而设计的准入检查控制器。其主要功能是为持有[配额预留](/docs/concepts/#quota-reservation)的工作负载创建
+[ProvisioningRequests](https://github.com/kubernetes/autoscaler/blob/4872bddce2bcc5b4a5f6a3d569111c11b8a2baf4/cluster-autoscaler/provisioningrequest/apis/autoscaling.x-k8s.io/v1beta1/types.go#L41)，
+并保持 [AdmissionCheckState](/docs/concepts/admission_check/#admissioncheckstate) 的同步。
 
-The controller is part of Kueue. It is enabled by default. You can disable it by editing the `ProvisioningACC` feature gate. Check the [Installation](/docs/installation/#change-the-feature-gates-configuration) guide for details on feature gate configuration.
+该控制器是 Kueue 的一部分，默认启用。您可以通过编辑 `ProvisioningACC` 特性门控来禁用它。
+有关特性门控配置的详细信息，请查看[安装](/docs/installation/#change-the-feature-gates-configuration)指南。
 
-The Provisioning Admission Check Controller is supported on [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler) versions 1.29 and later. However, some cloud-providers may not have an implementation for it.
+Provisioning 准入检查控制器支持
+[Kubernetes 集群自动缩放器](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)
+1.29 及更高版本。但是，某些云提供商可能没有相应的实现。
 
-Check the list of supported Provisioning Classes and prerequisite for them in [ClusterAutoscaler documentation](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#supported-provisioningclasses).
+请查看[集群自动缩放器文档](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#supported-provisioningclasses)
+中支持的 Provisioning 类和先决条件列表。
 
-## Usage
+## 使用方法 {#usage}
 
-To use the Provisioning AdmissionCheck, create an [AdmissionCheck](/docs/concepts/admission_check)
-with `kueue.x-k8s.io/provisioning-request` as a `.spec.controllerName` and create a ProvisioningRequest configuration using a `ProvisioningRequestConfig` object.
+要使用 Provisioning 准入检查，需要创建一个 [AdmissionCheck](/docs/concepts/admission_check)，
+将 `kueue.x-k8s.io/provisioning-request` 作为 `.spec.controllerName`，
+并使用 `ProvisioningRequestConfig` 对象创建 ProvisioningRequest 配置。
 
-Next, you need to reference the AdmissionCheck from the ClusterQueue, as detailed in [Admission Check usage](/docs/concepts/admission_check#usage).
+接下来，您需要从 ClusterQueue 引用 AdmissionCheck，详见[准入检查使用](/docs/concepts/admission_check#usage)。
 
-See [below](#setup) for a full setup.
+完整设置请参见[下文](#setup)。
 
-## ProvisioningRequest configuration
+## ProvisioningRequest 配置 {#provisioningrequest-configuration}
 
-There are two ways to configure the ProvisioningRequests that Kueue creates for your Jobs.
+Kueue 为您的作业创建 ProvisioningRequests 有两种配置方式：
 
-- **ProvisioningRequestConfig:** This configuration in the AdmissionCheck applies to all the jobs that go through this check.
-It enables you to set `provisioningClassName`, `managedResources`, and `parameters`
-- **Job annotation**: This configuration enables you to set `parameters` to a specific job. If both the annotation and the ProvisioningRequestConfig refer to the same parameter, the annotation value takes precedence.
+- **ProvisioningRequestConfig：** AdmissionCheck 中的此配置适用于通过此检查的所有作业。
+  它允许您设置 `provisioningClassName`、`managedResources` 和 `parameters`
+- **Job 注解：** 此配置允许您为特定作业设置 `parameters`。
+  如果注解和 ProvisioningRequestConfig 都引用相同的参数，注解值优先。
 
-### ProvisioningRequestConfig
-A `ProvisioningRequestConfig` looks like the following:
+### ProvisioningRequestConfig {#provisioningrequestconfig}
+
+一个 `ProvisioningRequestConfig` 如下所示：
 
 ```yaml
 apiVersion: kueue.x-k8s.io/v1beta1
@@ -50,27 +61,32 @@ spec:
   podSetMergePolicy: IdenticalWorkloadSchedulingRequirements
 ```
 
-Where:
-- **provisioningClassName** - describes the different modes of provisioning the resources. Supported ProvisioningClasses are listed in [ClusterAutoscaler documentation](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#supported-provisioningclasses), also check your cloud provider's documentation for other ProvisioningRequest classes they support.
-- **managedResources** -  contains the list of resources managed by the autoscaling.
-- **retryStrategy.backoffLimitCount** - indicates how many times ProvisioningRequest should be retried in case of failure. Defaults to 3.
-- **retryStrategy.backoffBaseSeconds** - provides the base for calculating backoff time that ProvisioningRequest waits before being retried. Defaults to 60.
-- **retryStrategy.backoffMaxSeconds** - indicates the maximum backoff time (in seconds) before retrying a ProvisioningRequest. Defaults to 1800.
-- **podSetMergePolicy** - allows to merge similar PodSets into a single PodTemplate used by the ProvisioningRequest.
-- **podSetUpdates** - allows to update the Workload's PodSets with nodeSelectors based on the successful ProvisioningRequest.
-  This allows to restrict scheduling of the PodSets' pods to the newly provisioned nodes.
-
-#### PodSet merge policy
-
-{{% alert title="Note" color="primary" %}}
-`podSetMergePolicy` feature is available in Kueue v0.12.0 version or newer.
-
-It offers two options:
-- `IdenticalPodTemplates` - merges only identical PodTemplates
-- `IdenticalWorkloadSchedulingRequirements` - merges PodTemplates which have
-  identical fields which are considered for defining the workload scheduling
-  requirements. The PodTemplate fields which are considered as workload
-  scheduling requirements: 
+其中：
+
+- **provisioningClassName** - 描述资源配置的不同模式。
+  支持的 ProvisioningClasses 列在[集群自动缩放器文档](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#supported-provisioningclasses)中，
+  也请查看您的云提供商文档以了解他们支持的其他 ProvisioningRequest 类。
+- **managedResources** - 包含由自动缩放管理的资源列表。
+- **retryStrategy.backoffLimitCount** - 指示在失败情况下 ProvisioningRequest 应重试多少次。
+  默认为 3。
+- **retryStrategy.backoffBaseSeconds** - 提供计算 ProvisioningRequest 重试前等待退避时间的基础。
+  默认为 60。
+- **retryStrategy.backoffMaxSeconds** - 指示重试 ProvisioningRequest 前的最大退避时间（以秒为单位）。
+  默认为 1800。
+- **podSetMergePolicy** - 允许将相似的 PodSets 合并为 ProvisioningRequest 使用的单个 PodTemplate。
+- **podSetUpdates** - 允许根据成功的 ProvisioningRequest 使用 nodeSelectors 更新工作负载的 PodSets。
+  这允许将 PodSets 的 pods 调度限制到新配置的节点。
+
+#### PodSet 合并策略 {#podset-merge-policy}
+
+{{% alert title="注意" color="primary" %}}
+`podSetMergePolicy` 特性在 Kueue v0.12.0 版本或更新版本中可用。
+
+它提供两个选项：
+
+- `IdenticalPodTemplates` - 仅合并相同的 PodTemplates
+- `IdenticalWorkloadSchedulingRequirements` - 合并具有相同字段的 PodTemplates，
+  这些字段被认为是定义工作负载调度要求的。被视为工作负载调度要求的 PodTemplate 字段包括：
   - `spec.containers[*].resources.requests`
   - `spec.initContainers[*].resources.requests`
   - `spec.resources`
@@ -79,31 +95,30 @@ It offers two options:
   - `spec.affinity`
   - `resourceClaims`
 
-When the field is not set, the PodTemplates are not merged when creating the ProvisioningRequest, even if identical.
+当未设置该字段时，即使相同，在创建 ProvisioningRequest 时也不会合并 PodTemplates。
 
-For example, setting the field as either `IdenticalPodTemplates` or `IdenticalWorkloadSchedulingRequirements`, 
-allows to create a ProvisioningRequest with a single PodTemplate when using PyTorchJob as in this sample: [`sample-pytorchjob.yaml`](/docs/tasks/run/kubeflow/pytorchjobs/#sample-pytorchjob). 
+例如，将字段设置为 `IdenticalPodTemplates` 或 `IdenticalWorkloadSchedulingRequirements`，
+允许在使用 PyTorchJob 时创建具有单个 PodTemplate 的 ProvisioningRequest，
+如此示例：[`sample-pytorchjob.yaml`](/docs/tasks/run/kubeflow/pytorchjobs/#sample-pytorchjob)。
 {{% /alert %}}
 
-#### Retry strategy
+#### 重试策略 {#retry-strategy}
 
-If a ProvisioningRequest fails, it may be retried after a backoff period.
-The backoff time (in seconds) is calculated using the following formula, where `n` is the retry number (starting at 1):
+如果 ProvisioningRequest 失败，它可能在退避期后重试。
+退避时间（以秒为单位）使用以下公式计算，其中 `n` 是重试次数（从 1 开始）：
 
 ```latex
 time = min(backoffBaseSeconds^n, backoffMaxSeconds)
 ```
 
-When a ProvisioningRequest fails, the quota reserved for a Workload is released, and the Workload needs to restart the
-admission cycle.
+当 ProvisioningRequest 失败时，为工作负载预留的配额会被释放，工作负载需要重新开始准入周期。
 
-#### PodSet updates
+#### PodSet 更新 {#podset-updates}
 
-In order to restrict scheduling of the workload's Pods to the newly provisioned
-nodes you can use the "podSetUpdates" API which allows to inject node selectors
-to target the nodes.
+为了将工作负载的 Pods 调度限制到新配置的节点，您可以使用 "podSetUpdates" API，
+它允许注入节点选择器来针对这些节点。
 
-For example:
+例如：
 
 ```yaml
 podSetUpdates:
@@ -112,30 +127,34 @@ podSetUpdates:
     valueFromProvisioningClassDetail: RequestKey
 ```
 
-This snippet in ProvisioningRequestConfig instructs Kueue to update the Job's
-PodTemplate, after provisioning, to target the newly provisioned nodes which
-have the label: `autoscaling.cloud-provider.com/provisioning-request` with the
-value coming from the [ProvisiongClassDetails](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1/types.go#L169) map, under the "RequestKey" key.
+ProvisioningRequestConfig 中的此代码片段指示 Kueue 在配置后更新作业的 PodTemplate，
+以针对具有标签 `autoscaling.cloud-provider.com/provisioning-request` 的新配置节点，
+该标签的值来自 [ProvisiongClassDetails](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1/types.go#L169)
+映射中的 "RequestKey" 键。
 
-Note that, this assumes the provisioning class (which can be cloud-provider
-specific) supports setting unique node label on the newly provisioned nodes.
+请注意，这假设配置类（可能是云提供商特定的）支持在新配置的节点上设置唯一的节点标签。
 
-#### Reference
+#### 参考 {#reference}
 
-Check the [API definition](https://github.com/kubernetes-sigs/kueue/blob/main/apis/kueue/v1beta1/provisioningrequestconfig_types.go) for more details.
+有关更多详细信息，请查看
+[API 定义](https://github.com/kubernetes-sigs/kueue/blob/main/apis/kueue/v1beta1/provisioningrequestconfig_types.go)。
 
-### Job annotations
+### Job 注解 {#job-annotations}
 
-Another way to pass ProvisioningRequest's [parameters](https://github.com/kubernetes/autoscaler/blob/0130d33747bb329b790ccb6e8962eedb6ffdd0a8/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/types.go#L115) is by using Job annotations. Every annotation with the ***provreq.kueue.x-k8s.io/*** prefix will be directly passed to created ProvisioningRequest. E.g. `provreq.kueue.x-k8s.io/ValidUntilSeconds: "60"` will pass `ValidUntilSeconds` parameter with the value of `60`. See more examples below.
+传递 ProvisioningRequest 的[参数](https://github.com/kubernetes/autoscaler/blob/0130d33747bb329b790ccb6e8962eedb6ffdd0a8/cluster-autoscaler/apis/provisioningrequest/autoscaling.x-k8s.io/v1beta1/types.go#L115)
+的另一种方式是使用 Job 注解。每个带有 ***provreq.kueue.x-k8s.io/*** 前缀的注解都会直接传递给创建的 ProvisioningRequest。
+例如，`provreq.kueue.x-k8s.io/ValidUntilSeconds: "60"` 将传递值为 `60` 的
+`ValidUntilSeconds` 参数。
+请参见下面的更多示例。
 
-Once Kueue creates a ProvisioningRequest for the job you submitted, modifying the value of annotations in the job will have no effect in the ProvisioningRequest.
+一旦 Kueue 为您提交的作业创建了 ProvisioningRequest，修改作业中注解的值将不会对 ProvisioningRequest 产生影响。
 
-## Example
+## 示例 {#examples}
 
-### Setup
+### 设置 {#setup}
 
 {{< include "examples/provisioning/provisioning-setup.yaml" "yaml" >}}
 
-### Job using a ProvisioningRequest
+### 使用 ProvisioningRequest 的作业 {#job-using-a-provisioningrequest}
 
 {{< include "examples/provisioning/sample-job.yaml" "yaml" >}}
diff --git a/site/content/zh-CN/docs/concepts/workload_priority_class.md b/site/content/zh-CN/docs/concepts/workload_priority_class.md
index 9047ccda3..ffa7299c0 100644
--- a/site/content/zh-CN/docs/concepts/workload_priority_class.md
+++ b/site/content/zh-CN/docs/concepts/workload_priority_class.md
@@ -1,17 +1,17 @@
 ---
-title: "Workload Priority Class"
+title: "Workload 优先级类"
 date: 2023-10-02
 weight: 6
 description: >
-  A priority class whose value is utilized by Kueue controller and is independent from Pod's priority.
+  一种优先级类，其值被 Kueue 控制器使用，并且独立于 Pod 的优先级。
 ---
 
-A `WorkloadPriorityClass` allows you to control the [`Workload`'s](/docs/concepts/workload) priority without affecting the pod's priority.
-This feature is useful for these cases:
-- want to prioritize workloads that remain inactive for a specific duration
-- want to set a lower priority for development workloads and higher priority for production workloads
+`WorkloadPriorityClass` 允许你控制 [`Workload`](/docs/concepts/workload) 的优先级，而不会影响 Pod 的优先级。
+此功能适用于以下场景：
+- 希望优先调度那些长时间处于非活跃状态的工作负载
+- 希望为开发环境工作负载设置较低优先级，为生产环境工作负载设置较高优先级
 
-A sample WorkloadPriorityClass looks like the following:
+一个示例 WorkloadPriorityClass 如下所示：
 
 ```yaml
 apiVersion: kueue.x-k8s.io/v1beta1
@@ -19,14 +19,14 @@ kind: WorkloadPriorityClass
 metadata:
   name: sample-priority
 value: 10000
-description: "Sample priority"
+description: "示例优先级"
 ```
 
-`WorkloadPriorityClass` objects are cluster scoped, so they can be used by a job in any namespace.
+`WorkloadPriorityClass` 对象是集群范围的，因此可以被任何命名空间下的作业使用。
 
-## How to use WorkloadPriorityClass on Jobs
+## 如何在 Job 上使用 WorkloadPriorityClass
 
-You can specify the `WorkloadPriorityClass` by setting the label `kueue.x-k8s.io/priority-class`.
+你可以通过设置标签 `kueue.x-k8s.io/priority-class` 来指定 `WorkloadPriorityClass`。
 
 ```yaml
 apiVersion: batch/v1
@@ -39,12 +39,11 @@ metadata:
 spec:
 ...
 ```
-
-Kueue generates the following `Workload` for the Job above.
-The `PriorityClassName` field can accept either `PriorityClass` or
-`WorkloadPriorityClass` name as a value. To distinguish, when using `WorkloadPriorityClass`,
-a `priorityClassSource` field has the `kueue.x-k8s.io/workloadpriorityclass` value.
-When using `PriorityClass`, a `priorityClassSource` field has the `scheduling.k8s.io/priorityclass` value.
+Kueue 会为上述 Job 生成如下 `Workload`。
+`PriorityClassName` 字段可以接受 `PriorityClass` 或 `WorkloadPriorityClass` 的名称。
+为区分两者，当使用 `WorkloadPriorityClass` 时，`priorityClassSource` 字段为
+`kueue.x-k8s.io/workloadpriorityclass`；当使用 `PriorityClass` 时，`priorityClassSource`
+字段为 `scheduling.k8s.io/priorityclass`。
 
 ```yaml
 apiVersion: kueue.x-k8s.io/v1beta1
@@ -59,8 +58,8 @@ spec:
 ...
 ```
 
-For other job frameworks, you can set `WorkloadPriorityClass` using the same label.
-The Following is an example of `MPIJob`.
+对于其他作业框架，也可以通过相同标签设置 `WorkloadPriorityClass`。
+以下是 `MPIJob` 的示例：
 
 ```yaml
 apiVersion: kubeflow.org/v2beta1
@@ -74,42 +73,40 @@ spec:
 ...
 ```
 
-## The relationship between pod's priority and workload's priority
-
-When creating a `Workload` for a given job, Kueue considers the following scenarios:
-1. A job specifies both `WorkloadPriorityClass` and `PriorityClass`
-- `WorkloadPriorityClass` is used for the workload's priority.
-- `PriorityClass` is used for the pod's priority.
-2. A job specifies only `WorkloadPriorityClass`
-- `WorkloadPriorityClass` is used for the workload's priority.
-- `WorkloadPriorityClass` is not used for pod's priority.
-3. A job specifies only `PriorityClass`
-- `PriorityClass` is used for the workload's priority and pod's priority.
-
-In certain job frameworks, there are CRDs that:
-- Define multiple pod specs, where each can have their own pod priority, or
-- Define the overall pod priority in a dedicated field.
-By default kueue will take the PriorityClassName of the first PodSet having one set,
-however the integration of the CRD with Kueue can implement
-[`JobWithPriorityClass interface`](https://github.com/kubernetes-sigs/kueue/blob/e162f8508b503d20feb9b31fd0b27d91e58f2c2f/pkg/controller/jobframework/interface.go#L81-L84)
-to change this behavior. You can read the code for each job integration
-to learn how the priority class is obtained.
-
-## Where workload's priority is used
-
-The priority of workloads is used for:
-- Sorting the workloads in the ClusterQueues.
-- Determining whether a workload can preempt others.
-
-## Workload's priority values are always mutable
-
-The `Workload`'s `Priority` field is always mutable.
-If a `Workload` has been pending for a while, you can consider updating its priority to execute it earlier,
-based on your own policies.
-Workload's `PriorityClassSource` and `PriorityClassName` fields are immutable.
-
-## What's next?
-
-- Learn how to [run jobs](/docs/tasks/run/jobs)
-- Learn how to [run jobs with workload priority](/docs/tasks/manage/run_job_with_workload_priority)
-- Read the [API reference](/docs/reference/kueue.v1beta1/#kueue-x-k8s-io-v1beta1-WorkloadPriorityClass) for `WorkloadPriorityClass`
+## Pod 优先级与 Workload 优先级的关系
+
+为给定作业创建 `Workload` 时，Kueue 会考虑以下场景：
+1. 作业同时指定了 `WorkloadPriorityClass` 和 `PriorityClass`
+    - `WorkloadPriorityClass` 用于 Workload 的优先级。
+    - `PriorityClass` 用于 Pod 的优先级。
+2. 作业仅指定了 `WorkloadPriorityClass`
+    - `WorkloadPriorityClass` 用于 Workload 的优先级。
+    - `WorkloadPriorityClass` 不用于 Pod 的优先级。
+3. 作业仅指定了 `PriorityClass`
+    - `PriorityClass` 用于 Workload 和 Pod 的优先级。
+
+在某些作业框架中，CRD 可能：
+    - 定义多个 Pod 规范，每个规范可以有自己的 Pod 优先级，或
+    - 在专用字段中定义整体 Pod 优先级。
+
+默认情况下，kueue 会采用第一个设置了 PriorityClassName 的 PodSet，但 CRD 与 Kueue 的集成可以实现
+[`JobWithPriorityClass 接口`](https://github.com/kubernetes-sigs/kueue/blob/e162f8508b503d20feb9b31fd0b27d91e58f2c2f/pkg/controller/jobframework/interface.go#L81-L84) 来改变此行为。
+你可以阅读每个作业集成的代码，了解优先级类的获取方式。
+
+## Workload 优先级的用途
+
+Workload 的优先级用于：
+- 在 ClusterQueue 中对 Workload 排序。
+- 决定某个 Workload 是否可以抢占其他 Workload。
+
+## Workload 优先级值始终可变
+
+`Workload` 的 `Priority` 字段始终是可变的。
+如果某个 `Workload` 已经挂起一段时间，你可以根据自己的策略考虑提升其优先级以提前执行。
+Workload 的 `PriorityClassSource` 和 `PriorityClassName` 字段是不可变的。
+
+## 下一步？
+
+- 了解如何[运行作业](/docs/tasks/run/jobs)
+- 了解如何[使用 Workload 优先级运行作业](/docs/tasks/manage/run_job_with_workload_priority)
+- 阅读 `WorkloadPriorityClass` 的 [API 参考](/docs/reference/kueue.v1beta1/#kueue-x-k8s-io-v1beta1-WorkloadPriorityClass)



Test patch:
diff --git a/pkg/cache/cache_test.go b/pkg/cache/cache_test.go
index b5218cdb9..296edff2a 100644
--- a/pkg/cache/cache_test.go
+++ b/pkg/cache/cache_test.go
@@ -562,8 +562,11 @@ func TestCacheClusterQueueOperations(t *testing.T) {
 			},
 			wantClusterQueues: map[kueue.ClusterQueueReference]*clusterQueue{
 				"b": {
-					Name:                          "b",
-					AllocatableResourceGeneration: 1,
+					Name: "b",
+					// AllocatableResourceGeneration is 2 because it was incremented:
+					// 1. Once during initial setup when added to cohort "one"
+					// 2. Once during deletion when cohort tree resources were recalculated after "a" was deleted
+					AllocatableResourceGeneration: 2,
 					NamespaceSelector:             labels.Nothing(),
 					FlavorFungibility:             defaultFlavorFungibility,
 					Status:                        active,
diff --git a/test/integration/singlecluster/scheduler/scheduler_test.go b/test/integration/singlecluster/scheduler/scheduler_test.go
index 906615177..b9e8db1b3 100644
--- a/test/integration/singlecluster/scheduler/scheduler_test.go
+++ b/test/integration/singlecluster/scheduler/scheduler_test.go
@@ -2639,6 +2639,7 @@ var _ = ginkgo.Describe("Scheduler", func() {
 
 			ns = util.CreateNamespaceFromPrefixWithLog(ctx, k8sClient, "core-")
 		})
+
 		ginkgo.It("finds correct flavor by discarding the first one in which preemption is not possible", func() {
 			fungibility := kueue.FlavorFungibility{WhenCanBorrow: kueue.TryNextFlavor, WhenCanPreempt: kueue.TryNextFlavor}
 			preemption := kueue.ClusterQueuePreemption{WithinClusterQueue: kueue.PreemptionPolicyLowerPriority, ReclaimWithinCohort: kueue.PreemptionPolicyAny, BorrowWithinCohort: &kueue.BorrowWithinCohort{Policy: kueue.BorrowWithinCohortPolicyLowerPriority}}
@@ -2683,4 +2684,56 @@ var _ = ginkgo.Describe("Scheduler", func() {
 			}
 		})
 	})
+	ginkgo.When("Deleting ClusterQueue should update cohort borrowable resources", func() {
+		var (
+			cq1 *kueue.ClusterQueue
+			cq2 *kueue.ClusterQueue
+			lq1 *kueue.LocalQueue
+			lq2 *kueue.LocalQueue
+		)
+
+		ginkgo.AfterEach(func() {
+			gomega.Expect(util.DeleteNamespace(ctx, k8sClient, ns)).To(gomega.Succeed())
+			util.ExpectObjectToBeDeleted(ctx, k8sClient, cq1, true)
+			util.ExpectObjectToBeDeleted(ctx, k8sClient, cq2, true)
+			util.ExpectObjectToBeDeleted(ctx, k8sClient, onDemandFlavor, true)
+		})
+
+		ginkgo.It("Should prevent incorrect admission through borrowing after ClusterQueue deletion", func() {
+			ginkgo.By("Creating two ClusterQueues in the same cohort")
+			// ClusterQueue with 8 CPU nominal quota - this will be the lender
+			cq1 = testing.MakeClusterQueue("cluster-queue-1").
+				Cohort("bug-test").
+				ResourceGroup(*testing.MakeFlavorQuotas("on-demand").Resource(corev1.ResourceCPU, "8").Obj()).
+				Obj()
+			util.MustCreate(ctx, k8sClient, cq1)
+
+			// ClusterQueue with 0 CPU nominal quota - this will need to borrow
+			cq2 = testing.MakeClusterQueue("cluster-queue-2").
+				Cohort("bug-test").
+				ResourceGroup(*testing.MakeFlavorQuotas("on-demand").Resource(corev1.ResourceCPU, "0").Obj()).
+				Obj()
+			util.MustCreate(ctx, k8sClient, cq2)
+
+			ginkgo.By("Creating LocalQueues for the corresponding ClusterQueues")
+			lq1 = testing.MakeLocalQueue("local-queue-1", ns.Name).ClusterQueue(cq1.Name).Obj()
+			util.MustCreate(ctx, k8sClient, lq1)
+
+			lq2 = testing.MakeLocalQueue("local-queue-2", ns.Name).ClusterQueue(cq2.Name).Obj()
+			util.MustCreate(ctx, k8sClient, lq2)
+
+			ginkgo.By("Deleting cluster-queue-1 which has 8 CPU nominal quota")
+			gomega.Expect(util.DeleteObject(ctx, k8sClient, cq1)).To(gomega.Succeed())
+
+			ginkgo.By("Creating a workload that requests 8 CPU - should not be admitted due to no borrowable resources")
+			wl := testing.MakeWorkload("should-be-pending-wl", ns.Name).
+				Queue(kueue.LocalQueueName(lq2.Name)).Request(corev1.ResourceCPU, "8").Obj()
+			util.MustCreate(ctx, k8sClient, wl)
+
+			ginkgo.By("Verifying metrics reflect the correct state")
+			util.ExpectPendingWorkloadsMetric(cq2, 0, 1)
+			util.ExpectReservingActiveWorkloadsMetric(cq2, 0)
+			util.ExpectAdmittedWorkloadsTotalMetric(cq2, 0)
+		})
+	})
 })