Repository: kubernetes-sigs/kueue

Issue:
Fairsharing bug that prevents CQs from reclaiming guarantees
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->

**What happened**: 
We noticed an increase in guarantee violations when fairsharing is turned on where CQs are unable to reclaim their guarantees when preemption is required despite having guarantees.

There were a lot of logs that said...
- `Workload requires preemption, but there are no candidate workloads allowed for preemption` 
- `couldn't assign flavors to pod set foo: insufficient unused quota for nvidia.com/gpu in flavor bar, 5 more needed`. 
  - This cluster also only has 1 flavor.

**What you expected to happen**:
Cqs need to be able to reclaim guarantees. 

**How to reproduce it (as minimally and precisely as possible)**:
We detect this in production and don't know how to repro. But it occurs multiple times in production. 

**Anything else we need to know?**:
Here is a theory:

We do hit these logs: `logV.Info("Simulating fair preemption", "candidates", workload.References(candidates), "resourcesRequiringPreemption", preemptionCtx.frsNeedPreemption.UnsortedList(), "preemptingWorkload", klog.KObj(preemptionCtx.preemptor.Obj))`

* `candidates` is heavily populated within our logs (meaning at some point it had a lot of preemption candidates)

Tracing the code, here is our theory:
1. `fairPreemptions(...)`
2. `logV.Info("Simulating fair preemption", "candidates", workload.References(candidates), ...)`
    1. we do see a lot of candidates here in the logs but later down return no preemption candidates
3. This means `fits` within `fits, targets, retryCandidates := runFirstFsStrategy(preemptionCtx, candidates, strategies[0])` is probably false. Why?

It could be this...
1. `workloadFits()` checks...
```
if v > preemptionCtx.preemptorCQ.Available(fr) {
    return false
} 
// v is resources preemptor workload needs for flavor
// "Available returns the current capacity available, before preempting any workloads"
```
2. `Available` → `available` → `LocalAvailable`
3. `LocalAvailable` returns `guaranteedQuota() - Usage[fr]`

The issue is probably here:
```
func (r resourceNode) guaranteedQuota(fr resources.FlavorResource) int64 {
    if lendingLimit := r.Quotas[fr].LendingLimit; lendingLimit != nil {
        return max(0, r.SubtreeQuota[fr]-*lendingLimit)
    }
    return 0
}
```
1. `guaranteedQuota` isn’t actually returning you guaranteed quota. Its basing the calculation off lendingLimit... which... we do not set. 
2. `LendingLimit == nil` → `return 0`

SO. Back to this:
1. 
```
if v > preemptionCtx.preemptorCQ.Available(fr) {
        return false
}
 ```
2. v > 0 → return false
3. ❌ fails fit check

---
⭐️ Other supporting evidence: This could explain this! https://github.com/kubernetes-sigs/kueue/issues/6219
`Reclamation disappeared as a preemption reason after turning on fairshare`

**Environment**:
- Kueue version (use `git describe --tags --dirty --always`): 0.12.2

Discussion:
@amy: The other question may be... why doesn't classical preemption have this issue?
Maybe its around the difference in timing of when you add/remove usage for the workloads in relation to fit checks? @tenzen-y: @amy Could you share the ClusterQueue manifests? Especially, I would like to know preemption parameters.
Additionally, did you try to upgrade Kueue version to v0.12.3? The newer v0.12 patch version fixed flavor assignment accuracy bugs.
@tenzen-y: cc @gabesaba @mimowo @mpaliwal3: @tenzen-y  Here is a snippet from the impacted CQ. We are running v0.12.2 right now. We only have one resource flavor in this cluster.

```
spec:
  cohort: cohort-ufbw46fk3tkd
  fairSharing:
    weight: "100"
  flavorFungibility:
    whenCanBorrow: TryNextFlavor
    whenCanPreempt: TryNextFlavor
  namespaceSelector: {}
  preemption:
    borrowWithinCohort:
      maxPriorityThreshold: 80000
      policy: Never
    reclaimWithinCohort: Any
    withinClusterQueue: LowerPriority
  queueingStrategy: BestEffortFIFO
  resourceGroups:
```@gabesaba: > guaranteedQuota isn’t actually returning you guaranteed quota. Its basing the calculation off lendingLimit... which... we do not set.

This is a poorly named method. It is referring to the quota that will never be lent out, while the rest is bubbled up to the parent (see `accumulateFromChild`). It doesn't refer to the nominal quota which the CQ is guaranteed to reclaim (assuming proper policy, etc).@tenzen-y: > [@tenzen-y](https://github.com/tenzen-y) Here is a snippet from the impacted CQ. We are running v0.12.2 right now. We only have one resource flavor in this cluster.
> 
> ```
> spec:
>   cohort: cohort-ufbw46fk3tkd
>   fairSharing:
>     weight: "100"
>   flavorFungibility:
>     whenCanBorrow: TryNextFlavor
>     whenCanPreempt: TryNextFlavor
>   namespaceSelector: {}
>   preemption:
>     borrowWithinCohort:
>       maxPriorityThreshold: 80000
>       policy: Never
>     reclaimWithinCohort: Any
>     withinClusterQueue: LowerPriority
>   queueingStrategy: BestEffortFIFO
>   resourceGroups:
> ```

Thank you for sharing that. Could you share fairSharing.weight value range across all ClusterQueues (e.g, 1-100, 100-1000)?
I want to check if this clusterqueue dws is not rounded down into 0.@amy: Its weighted fairshare on the CQ itself was around 33. But the DRS in logs for this workload during tournament is 0 indicating reclamation.@amy: Updating the thread on other potential theories to follow along with/debug:
Issues with `nextTarget`. Maybe there's something wrong with the pruning logic where its somehow causing a CQ to be discarded prematurely. @gabesaba: > Updating the thread on other potential theories to follow along with/debug: Issues with `nextTarget`. Maybe there's something wrong with the pruning logic where its somehow causing a CQ to be discarded prematurely.

+1 to scrutinizing both this as well as `workloadFits`, which you mentioned earlier. Also worth looking at `findCandidates` Linking to them all here:

1) [nextTarget](https://github.com/kubernetes-sigs/kueue/blob/v0.12.2/pkg/scheduler/preemption/fairsharing/ordering.go#L127-L133)
2) [workloadFitsForFairSharing](https://github.com/kubernetes-sigs/kueue/blob/v0.12.2/pkg/scheduler/preemption/preemption.go#L494-L498)
3) [findCandidates](https://github.com/kubernetes-sigs/kueue/blob/v0.12.2/pkg/scheduler/preemption/preemption.go#L417-L420)

Since the scheduling CQ is within nominal quota and the first strategy is [LessThanOrEqualToFinalShare](https://github.com/kubernetes-sigs/kueue/blob/b5234c619aa542cbaf48dc324a8b06a2856ec826/pkg/scheduler/preemption/fairsharing/strategy.go#L35-L38), I expect [strategy](https://github.com/kubernetes-sigs/kueue/blob/v0.12.2/pkg/scheduler/preemption/preemption.go#L329) to always return true.

## Mitigation if any other user experiences similar issue:
For the CQ failing to schedule, set lending limit to 0. This will immediately put the rest of the Cohort in debt. As other workloads in Cohort finish up/are preempted (paying back the debt), this capacity would return to this CQ. This shouldn't result in overadmission: see [this testcase](https://github.com/kubernetes-sigs/kueue/blob/b5234c619aa542cbaf48dc324a8b06a2856ec826/pkg/cache/resource_test.go#L117-L153)@amy: I've augmented our Kueue image *temporarily* with these logs: https://github.com/kubernetes-sigs/kueue/pull/6593
Please take a look and let me know if I missed any areas that need logging coverage. Feel free to directly make a PR against https://github.com/amy/kueue/tree/v0.12.2-kueueteam with suggestions 🙏

We're hoping to catch the guarantee violation again in action & use these augmented logs to narrow down the code areas. @pajakd: I'm not entirely sure but is it possible that the [DRS calculation](https://github.com/kubernetes-sigs/kueue/blob/7aae396d525f000e907adfb521f34801fc3e2032/pkg/cache/fair_sharing.go#L49-L86) could be wrong in some cases?

In this line:
`ratio := b * 1000 / lr`
the ratio can be 0 (due to rounding) if some CQ is only borrowing 1 unit of quota and `lr > 1000` (`lr` is the total amount of resource in the cohort and `b` is the amount borrowed by the CQ). This means that in such a large cohort any CQ can borrow 1 GPU and get `DRS = 0`.

Now, when iterating over the candidates we [prune CQs with DRS = 0](https://github.com/kubernetes-sigs/kueue/blob/7aae396d525f000e907adfb521f34801fc3e2032/pkg/scheduler/preemption/fairsharing/ordering.go#L144) so (perhaps) this is why some valid candidates are not considered for preemption. 

If we can't find sufficiently many candidates for preemption that would allow to fit the workload we get `fits = false`.

I'm going to try to reproduce this scenario in a test and check my hypothesis.@tenzen-y: > I'm not entirely sure but is it possible that the [DRS calculation](https://github.com/kubernetes-sigs/kueue/blob/7aae396d525f000e907adfb521f34801fc3e2032/pkg/cache/fair_sharing.go#L49-L86) could be wrong in some cases?
> 
> In this line: `ratio := b * 1000 / lr` the ratio can be 0 (due to rounding) if some CQ is only borrowing 1 unit of quota and `lr > 1000` (`lr` is the total amount of resource in the cohort and `b` is the amount borrowed by the CQ). This means that in such a large cohort any CQ can borrow 1 GPU and get `DRS = 0`.
> 
> Now, when iterating over the candidates we [prune CQs with DRS = 0](https://github.com/kubernetes-sigs/kueue/blob/7aae396d525f000e907adfb521f34801fc3e2032/pkg/scheduler/preemption/fairsharing/ordering.go#L144) so (perhaps) this is why some valid candidates are not considered for preemption.
> 
> If we can't find sufficiently many candidates for preemption that would allow to fit the workload we get `fits = false`.
> 
> I'm going to try to reproduce this scenario in a test and check my hypothesis.

That's a good point. 
I think that weird fair sharing could be caused by (1 the unbalanced borrowing usage between resources (e.g., borrowing a lot of CPUs, but GPUs are accommodated within the nominal quota), OR (2 there is a CQ that has a large lendable resource (e.g., global-default) and others have a small lendable resource.

In the case of (1, the CPU dws is respected than the GPU dws, then the cq will be disadvantaged across the cohort based on CPU borrowing usage.
In the case of (2, the non-global-default cq dws could be rounded down to 0 (as @pajakd mentioned) due to subtree lendable resource (including global-default cq) being large, then the non-global-default cq dws could be pruned.

NOTE: When Kueue calculates the lendable resources, it does not consider the current usage. So, in the large cluster, the lendable resources could be technically large numbers due to the large nominal quota across all cq and cohort.
The proper borrowing limit could mitigate this large number of lendable resource since the borrowing limit is considered when Kueue calculates lendable resources.@amy: I've added this logging into fairsharing.go. Will use this logging image in production tomorrow:
```
			ratio := b * 1000 / lr

			// Log when ratio is 0 due to rounding (b < lr/1000)
			if ratio == 0 && b > 0 {
				logger.Info("DRS ratio calculation resulted in 0 due to rounding",
					"resource", rName,
					"borrowed", b,
					"lendable", lr,
					"threshold", lr/1000,
					"actualRatio", 1000.0*float64(b)/float64(lr))
			}
```

and 

```
dws := drs * 1000 / node.fairWeight().MilliValue()
	if dws == 0 && drs > 0 {
		logger.Info("DWS ratio calculation resulted in 0 due to rounding",
			"drs", drs,
			"fairsharingWeight", node.fairWeight().MilliValue(),
			"actualRatio", 1000.0*float64(drs)/float64(node.fairWeight().MilliValue()))
	}
```

For this latter log, even if your CQ is borrowing a lot of GPUs, if it has a high CQ weight, it can still result in a scenario where you're rounding down to `DRS=0`@pajakd: > if it has a high CQ weight, it can still result in a scenario where you're rounding down to `DRS=0`

Yes, large weights aggravate the rounding problem and this is the main reason of the problem. With weights it can happen that even a large amount of quota cannot be reclaimed.

I would expect this test to pass (can be added to `integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go`). But instead when I run it, the reclamation does not happen and I can observe familiar log lines `Workload requires preemption, but there are no candidate workloads allowed for preemption`.

```	
ginkgo.When("Preemption is enabled in fairsharing and there are large values of weight and quota", func() {
		var (
			cqA *kueue.ClusterQueue
			cqB *kueue.ClusterQueue
		)
		ginkgo.BeforeEach(func() {
			cqA = createQueue(testing.MakeClusterQueue("a").
				Cohort("all").FairWeight(resource.MustParse("300")).
				ResourceGroup(
					*testing.MakeFlavorQuotas("default").Resource(corev1.ResourceCPU, "600").Obj(),
				).Preemption(kueue.ClusterQueuePreemption{
				ReclaimWithinCohort: kueue.PreemptionPolicyAny,
				WithinClusterQueue:  kueue.PreemptionPolicyLowerPriority,
			}).Obj())

			cqB = createQueue(testing.MakeClusterQueue("b").
				Cohort("all").FairWeight(resource.MustParse("300")).
				ResourceGroup(
					*testing.MakeFlavorQuotas("default").Resource(corev1.ResourceCPU, "600").Obj(),
				).Preemption(kueue.ClusterQueuePreemption{
				ReclaimWithinCohort: kueue.PreemptionPolicyAny,
				WithinClusterQueue:  kueue.PreemptionPolicyLowerPriority,
			}).Obj())
		})

		ginkgo.It("Queue can reclaim its nominal quota", framework.SlowSpec, func() {
			ginkgo.By("Adding so many workloads in cqA that it borrows some quota from cqB")
			for range 10 {
				createWorkload("a", "100")
			}
			util.ExpectReservingActiveWorkloadsMetric(cqA, 10)
			ginkgo.By("Creating newer workload in cqB that needs only nominal quota")
			createWorkload("b", "500")
			ginkgo.By("Evict the some workloads in cqA and reclaim the nominal quota in cqB")
			util.FinishEvictionOfWorkloadsInCQ(ctx, k8sClient, cqA, 3)
			util.ExpectReservingActiveWorkloadsMetric(cqA, 7)
			util.ExpectReservingActiveWorkloadsMetric(cqB, 1)
		})
	})
```

EDIT: Changing the weights to resource.MustParse("1") makes the test pass.@amy: @pajakd nice work! And based off offline preliminary evidence. Theory seems promising. (We did find some incorrect DRS=0 values in tournament logs for candidate CRs that were clearly borrowing at the time. Need to wait for recurrence of bug to find how often this happens.)

Will update the thread if we catch the logs above during recurrence in prod. 

Solution patch:
diff --git a/pkg/cache/fair_sharing.go b/pkg/cache/fair_sharing.go
index 4a1716bb7..325ad5255 100644
--- a/pkg/cache/fair_sharing.go
+++ b/pkg/cache/fair_sharing.go
@@ -82,7 +82,7 @@ func dominantResourceShare(node dominantResourceShareNode, wlReq resources.Flavo
 	}
 
 	dws := drs * 1000 / node.fairWeight().MilliValue()
-	return int(dws), dRes
+	return max(1, int(dws)), dRes
 }
 
 // calculateLendable aggregates capacity for resources across all
diff --git a/site/content/en/_index.html b/site/content/en/_index.html
index 09476735d..e6facdbd9 100644
--- a/site/content/en/_index.html
+++ b/site/content/en/_index.html
@@ -6,11 +6,11 @@ description = "Kueue is a cloud-native job queueing system for batch, HPC, AI/ML
 
 {{< blocks/cover title="Welcome to Kueue" image_anchor="top" height="full" color="orange" >}}
 <div class="mx-auto">
-	<a class="btn btn-lg btn-primary mr-3 mb-4" href="/docs/">
-		Read the docs <i class="fas fa-arrow-alt-circle-right ml-2"></i>
+	<a class="btn btn-lg btn-primary me-3 mb-4" href="/docs/">
+		Read the docs <i class="fas fa-arrow-alt-circle-right ms-2"></i>
 	</a>
-	<a class="btn btn-lg btn-secondary mr-3 mb-4" href="https://github.com/kubernetes-sigs/kueue">
-		GitHub <i class="fab fa-github ml-2 "></i>
+	<a class="btn btn-lg btn-secondary me-3 mb-4" href="https://github.com/kubernetes-sigs/kueue">
+		GitHub <i class="fab fa-github ms-2 "></i>
 	</a>
 	<p class="lead mt-5">Kubernetes-native Job Queueing</p>
 	{{< blocks/link-down color="info" id="description" >}}
diff --git a/site/content/zh-CN/_index.html b/site/content/zh-CN/_index.html
index 9b84b9125..54443bd60 100644
--- a/site/content/zh-CN/_index.html
+++ b/site/content/zh-CN/_index.html
@@ -6,11 +6,11 @@ description = "Kueue 是一个云原生的作业队列系统，适用于 Kuberne
 
 {{< blocks/cover title="欢迎使用 Kueue" image_anchor="top" height="full" color="orange" >}}
 <div class="mx-auto">
-	<a class="btn btn-lg btn-primary mr-3 mb-4" href="/zh-cn/docs/">
-		阅读文档 <i class="fas fa-arrow-alt-circle-right ml-2"></i>
+	<a class="btn btn-lg btn-primary me-3 mb-4" href="/zh-cn/docs/">
+		阅读文档 <i class="fas fa-arrow-alt-circle-right ms-2"></i>
 	</a>
-	<a class="btn btn-lg btn-secondary mr-3 mb-4" href="https://github.com/kubernetes-sigs/kueue">
-		GitHub <i class="fab fa-github ml-2 "></i>
+	<a class="btn btn-lg btn-secondary me-3 mb-4" href="https://github.com/kubernetes-sigs/kueue">
+		GitHub <i class="fab fa-github ms-2 "></i>
 	</a>
 	<p class="lead mt-5">Kubernetes 原生作业队列</p>
 	{{< blocks/link-down color="info" id="description" >}}
diff --git a/site/hugo.toml b/site/hugo.toml
index 4596fe64f..582adebe2 100644
--- a/site/hugo.toml
+++ b/site/hugo.toml
@@ -27,7 +27,7 @@ ignoreFiles = []
   [[menu.main]]
     name = "GitHub"
     weight = 99
-    pre = "<i class='fab fa-github pr-2'></i>"
+    pre = "<i class='fab fa-github pe-2'></i>"
     url = "https://github.com/kubernetes-sigs/kueue"
 
 ###############################################################################
diff --git a/site/layouts/partials/footer.html b/site/layouts/partials/footer.html
index 7ec7dc688..60abc9bdb 100644
--- a/site/layouts/partials/footer.html
+++ b/site/layouts/partials/footer.html
@@ -18,7 +18,7 @@
       </div>
       <div class="col-12 col-sm-5 text-center py-2 order-sm-2">
         {{ with .Site.Params.copyright }}<small class="text-white">&copy; {{ now.Year}} {{ .}} {{ "| Documentation Distributed under CC BY 4.0" }}</small>{{ end }}
-        {{ with .Site.Params.privacy_policy }}<p><small class="ml-1"><a href="{{ . }}" target="_blank" rel="noopener">{{ T "footer_privacy_policy" }}</a></small></p>{{ end }}
+        {{ with .Site.Params.privacy_policy }}<p><small class="ms-1"><a href="{{ . }}" target="_blank" rel="noopener">{{ T "footer_privacy_policy" }}</a></small></p>{{ end }}
 	{{ if not .Site.Params.ui.footer_about_disable }}
 		{{ with .Site.GetPage "about" }}<p class="mt-2"><a href="{{ .RelPermalink }}">{{ .Title }}</a></p>{{ end }}
 	{{ end }}
diff --git a/site/layouts/partials/navbar.html b/site/layouts/partials/navbar.html
index d65dc0b54..b4ebd7cd0 100644
--- a/site/layouts/partials/navbar.html
+++ b/site/layouts/partials/navbar.html
@@ -6,11 +6,11 @@
 	<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#main_navbar" aria-controls="main_navbar" aria-expanded="false" aria-label="Toggle navigation">
 		<span class="navbar-toggler-icon"></span>
 	</button>
-	<div class="collapse navbar-collapse ml-md-auto" id="main_navbar">
-		<ul class="navbar-nav ml-auto pt-4 pt-md-0 my-2 my-md-1">
+	<div class="collapse navbar-collapse ms-md-auto" id="main_navbar">
+		<ul class="navbar-nav ms-auto pt-4 pt-md-0 my-2 my-md-1">
 			{{ $p := . }}
 			{{ range .Site.Menus.main }}
-			<li class="nav-item mr-2 mr-lg-4 mt-1 mt-lg-0">
+			<li class="nav-item me-2 mt-1 mt-lg-0">
 				{{ $active := or ($p.IsMenuCurrent "main" .) ($p.HasMenuCurrent "main" .) }}
 				{{ with .Page }}
 				{{ $active = or $active ( $.IsDescendant .)  }}
@@ -23,7 +23,7 @@
 			</li>
 			{{ end }}
 			{{ if  .Site.Params.versions }}
-			<li class="nav-item dropdown mt-1 mt-lg-0 mr-2">
+			<li class="nav-item dropdown mt-1 mt-lg-0 me-2">
 				{{ partial "navbar-version-selector.html" . }}
 			</li>
 			{{ end }}



Test patch:
diff --git a/pkg/cache/fair_sharing_test.go b/pkg/cache/fair_sharing_test.go
index b2f9b5728..4970cc43e 100644
--- a/pkg/cache/fair_sharing_test.go
+++ b/pkg/cache/fair_sharing_test.go
@@ -163,6 +163,88 @@ func TestDominantResourceShare(t *testing.T) {
 				},
 			},
 		},
+		"usage slightly above nominal in a cohort with large quotas": {
+			usage: resources.FlavorResourceQuantities{
+				{Flavor: "default", Resource: "example.com/gpu"}: 501,
+			},
+			clusterQueue: utiltesting.MakeClusterQueue("cq").
+				Cohort("test-cohort").
+				FairWeight(oneQuantity).
+				ResourceGroup(
+					*utiltesting.MakeFlavorQuotas("default").
+						ResourceQuotaWrapper("example.com/gpu").NominalQuota("500").Append().
+						Obj(),
+				).Obj(),
+			lendingClusterQueue: utiltesting.MakeClusterQueue("lending-cq").
+				Cohort("test-cohort").
+				FairWeight(resource.MustParse("300")).
+				ResourceGroup(
+					*utiltesting.MakeFlavorQuotas("default").
+						ResourceQuotaWrapper("example.com/gpu").NominalQuota("1000").Append().
+						Obj(),
+				).Obj(),
+			want: []fairSharingResult{
+				{
+					Name:     "cq",
+					NodeType: nodeTypeCq,
+					DrName:   "example.com/gpu",
+					DrValue:  1,
+				},
+				{
+					Name:     "lending-cq",
+					NodeType: nodeTypeCq,
+					DrName:   "",
+					DrValue:  0,
+				},
+				{
+					Name:     "test-cohort",
+					NodeType: nodeTypeCohort,
+					DrName:   "",
+					DrValue:  0,
+				},
+			},
+		},
+		"usage way above nominal in a cohort with large quotas and weights": {
+			usage: resources.FlavorResourceQuantities{
+				{Flavor: "default", Resource: "example.com/gpu"}: 800,
+			},
+			clusterQueue: utiltesting.MakeClusterQueue("cq").
+				Cohort("test-cohort").
+				FairWeight(resource.MustParse("300")).
+				ResourceGroup(
+					*utiltesting.MakeFlavorQuotas("default").
+						ResourceQuotaWrapper("example.com/gpu").NominalQuota("500").Append().
+						Obj(),
+				).Obj(),
+			lendingClusterQueue: utiltesting.MakeClusterQueue("lending-cq").
+				Cohort("test-cohort").
+				FairWeight(resource.MustParse("300")).
+				ResourceGroup(
+					*utiltesting.MakeFlavorQuotas("default").
+						ResourceQuotaWrapper("example.com/gpu").NominalQuota("1000").Append().
+						Obj(),
+				).Obj(),
+			want: []fairSharingResult{
+				{
+					Name:     "cq",
+					NodeType: nodeTypeCq,
+					DrName:   "example.com/gpu",
+					DrValue:  1,
+				},
+				{
+					Name:     "lending-cq",
+					NodeType: nodeTypeCq,
+					DrName:   "",
+					DrValue:  0,
+				},
+				{
+					Name:     "test-cohort",
+					NodeType: nodeTypeCohort,
+					DrName:   "",
+					DrValue:  0,
+				},
+			},
+		},
 		"one resource above nominal": {
 			usage: resources.FlavorResourceQuantities{
 				{Flavor: "default", Resource: corev1.ResourceCPU}: 3_000,
diff --git a/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go b/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
index 4202762fc..cceaa53e3 100644
--- a/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
+++ b/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
@@ -574,6 +574,46 @@ var _ = ginkgo.Describe("Scheduler", ginkgo.Ordered, ginkgo.ContinueOnFailure, f
 			util.ExpectWorkloadsToBeAdmitted(ctx, k8sClient, wlC)
 		})
 	})
+
+	ginkgo.When("Preemption is enabled in fairsharing and there are large values of quota and weights", func() {
+		var (
+			cqA *kueue.ClusterQueue
+			cqB *kueue.ClusterQueue
+		)
+		ginkgo.BeforeEach(func() {
+			cqA = createQueue(testing.MakeClusterQueue("a").
+				Cohort("all").FairWeight(resource.MustParse("300")).
+				ResourceGroup(
+					*testing.MakeFlavorQuotas("default").Resource(corev1.ResourceCPU, "600").Obj(),
+				).Preemption(kueue.ClusterQueuePreemption{
+				ReclaimWithinCohort: kueue.PreemptionPolicyAny,
+				WithinClusterQueue:  kueue.PreemptionPolicyLowerPriority,
+			}).Obj())
+
+			cqB = createQueue(testing.MakeClusterQueue("b").
+				Cohort("all").FairWeight(resource.MustParse("300")).
+				ResourceGroup(
+					*testing.MakeFlavorQuotas("default").Resource(corev1.ResourceCPU, "600").Obj(),
+				).Preemption(kueue.ClusterQueuePreemption{
+				ReclaimWithinCohort: kueue.PreemptionPolicyAny,
+				WithinClusterQueue:  kueue.PreemptionPolicyLowerPriority,
+			}).Obj())
+		})
+
+		ginkgo.It("Queue can reclaim its nominal quota", framework.SlowSpec, func() {
+			ginkgo.By("Adding so many workloads in cqA that it borrows some quota from cqB")
+			for range 10 {
+				createWorkload("a", "100")
+			}
+			util.ExpectReservingActiveWorkloadsMetric(cqA, 10)
+			ginkgo.By("Creating a newer workload in cqB that needs only nominal quota")
+			createWorkload("b", "500")
+			ginkgo.By("Evict the some workloads in cqA and reclaim the nominal quota in cqB")
+			util.FinishEvictionOfWorkloadsInCQ(ctx, k8sClient, cqA, 3)
+			util.ExpectReservingActiveWorkloadsMetric(cqA, 7)
+			util.ExpectReservingActiveWorkloadsMetric(cqB, 1)
+		})
+	})
 })
 
 func expectCohortWeightedShare(cohortName string, weightedShare int64) {