Repository: kubernetes-sigs/kueue

Issue:
[flaky FairSharing] Scheduler when Preemption is enabled in fairsharing and there are best effort and guaranteed workloads Guaranteed workloads cause preemption of a single best effort workload
**What would you like to be cleaned**:
Scheduler Fair Sharing Suite: [It] Scheduler when Preemption is enabled in fairsharing and there are best effort and guaranteed workloads Guaranteed workloads cause preemption of a single best effort workload

```
{Timed out after 10.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/util/util.go:441 with:
Not enough workloads are preempted
Expected
    <int>: 0
to equal
    <int>: 1 failed [FAILED] Timed out after 10.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/util/util.go:441 with:
Not enough workloads are preempted
Expected
    <int>: 0
to equal
    <int>: 1
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go:726 @ 09/25/25 08:37:00.544
}
```

See https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/6986/pull-kueue-test-integration-baseline-main/1971128077861785600

Discussion:
@mimowo: /kind bug
/kind flake@mimowo: /retitle [flaky FairSharing] Scheduler when Preemption is enabled in fairsharing and there are best effort and guaranteed workloads Guaranteed workloads cause preemption of a single best effort workload 

It makes very useful for deduplicating to add the bug and flake labels. I often use that fileter, then look for matching title

https://github.com/kubernetes-sigs/kueue/issues?q=is%3Aissue%20state%3Aopen%20label%3Akind%2Fflake@tenzen-y: /remove-kind cleanup@mimowo: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/7086/pull-kueue-test-integration-baseline-release-0-12/1973034863619477504
@mszadkow: /assign@mimowo: https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-kueue-test-integration-main/1986507375833518080@mimowo: https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-kueue-test-integration-release-0-14/1987911641877975040

https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-kueue-test-integration-main/1988500528350367744@mimowo: It seems also quite common recently, added to: https://github.com/kubernetes-sigs/kueue/issues/7471@mszadkow: > It seems also quite common recently, added to: [#7471](https://github.com/kubernetes-sigs/kueue/issues/7471)

Ack, I will treat it with priority@mimowo: @mszadkow is the problem because the code is choosing occasionally the workload from LQA instead of LQB as expected?@mimowo: It seems so, looking at the logs in https://storage.googleapis.com/kubernetes-ci-logs/logs/periodic-kueue-test-integration-main/1988500528350367744/build-log.txt

I can see `workload-vvvdg` is the preemption target
```
  2025-11-12T07:16:22.618871885Z	LEVEL(-3)	scheduler	preemption/preemption.go:186	Preemption ongoing	{"schedulingCycle": 585, "workload": {"name":"workload-z2x9k","namespace":"core-kpdzd"}, "clusterQueue": {"name":"guaranteed"}, "parentCohort": {"name":"all"}, "rootCohort": {"name":"all"}, "targetWorkload": {"name":"workload-vvvdg","namespace":"core-kpdzd"}, "preemptingWorkload": {"name":"workload-z2x9k","namespace":"core-kpdzd"}}
```
However, it comes from the LQA based on this log line:
```
 2025-11-12T07:16:22.03375673Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:815	Workload update event	{"workload": {"name":"workload-vvvdg","namespace":"core-kpdzd"}, "queue": "best-effort-a", "status": "admitted", "prevStatus": "pending", "clusterQueue": "best-effort-a"}
```
The question remains why this happens, maybe WaitForNextSecondAfterCreation is not enough.  @mimowo: Hm, looking at the timestamps of the Workload create event it seems like indeed they were created in the same second:
`2025-11-12T07:16:22.003431628Z` and `2025-11-12T07:16:22.287078622Z`. It remains unclear what were the exact creationTimestamps though.

```
2025-11-12T07:16:22.003431628Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:729	Workload create event	{"workload": {"name":"workload-vvvdg","namespace":"core-kpdzd"}, "queue": "best-effort-a", "status": "pending"}
...
2025-11-12T07:16:22.287078622Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:729	Workload create event	{"workload": {"name":"workload-2jf8g","namespace":"core-kpdzd"}, "queue": "best-effort-b", "status": "pending"}
```
EDIT: since the new workload was created quickly it seems that the first one had to have `2025-11-12T07:16:21` as the creation timestamp. @mimowo: Maybe in case of fair sharing we don't use creation timestamp for tie-breaking? Or we made the decision before the DRS got updated yet.@mimowo: Hm, in case of equal DRS we call `preemptioncommon.CandidatesOrdering`: https://github.com/kubernetes-sigs/kueue/blob/49141c9889b85093c1c519732187c5897cee29f5/pkg/scheduler/preemption/fairsharing/ordering.go#L155-L160

but this function is not using CreationTimestamp for tie-breaking indeed, but this function is using quotaReservation timestamps which are equal in this case `  2025-11-12T07:16:22`: https://github.com/kubernetes-sigs/kueue/blob/49141c9889b85093c1c519732187c5897cee29f5/pkg/scheduler/preemption/common/ordering.go#L73

@mimowo: So, the call to WaitForNextSecondAfterCreation is actually not directly relevant. It helps in most cases by separating the two workloads in different seconds for their CreationTimestamps, but it does not imply different QuotaReservation timestamps.
@mimowo: For the fix I could see two approaches:
1. introduce `util.WaitForNextSecondAfterReservation(wlBestEffortA)` and use it after WorkloadA has quota reservation to delay creatiion of Workload B, 
2. use priorities for tie-breaking

I would rather be leaning to (2.) as simpler, but in that case just verify that the test still pick up the changes done in this PR, so that we still capture the intention of the test: https://github.com/kubernetes-sigs/kueue/pull/6764/files#diff-c791c7f8f43735ff11274e29bfe6eb9320dfc913d4a5a0f14b3ba03038b076b4L144-R158

I think checking priorities would still capture the intention of the test, because prior to the PR we wouldn't have a special case for `case drs == highestCqDrs:` at all.@mimowo: cc @mszadkow @mbobrovskyi please check if this makes sense, if so I would try following (2.)@mimowo: Ok I could repro the issue fairly often with this snippet:

```golang
ginkgo.FIt("Guaranteed workloads cause preemption of a single best effort workload", func() {
	now := time.Now()
	nowTruncated := now.Truncate(time.Second)
	waitTime := nowTruncated.Add(time.Second - 5*time.Millisecond).Sub(now)
	time.Sleep(waitTime)
```
EDIT: it basically waits until 0.995 before the next second. This makes the creation timestamps likely to differ, but the quotaReservetion timestamps to be the same@mszadkow: The snippet works and helped to test the solution.
The solution 2., so priority was added to the workloads of best-effort cqs.

Solution patch:
diff --git a/hack/releasing/wait_for_images.sh b/hack/releasing/wait_for_images.sh
index b6f7f977d..2f9d1e67a 100755
--- a/hack/releasing/wait_for_images.sh
+++ b/hack/releasing/wait_for_images.sh
@@ -31,8 +31,8 @@ fi
 
 declare -r RELEASE_VERSION="$1"
 
-if [[ ! "$RELEASE_VERSION" =~ ^v[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
-  echo "!!! Invalid release version. It should be semantic version like v0.13.2"
+if [[ ! "$RELEASE_VERSION" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9]+\.[0-9]+)?$ ]]; then
+  echo "!!! Invalid release version. It should be semantic version like v0.13.2 or v0.13.2-rc.0"
   exit 1
 fi
 
diff --git a/keps/582-preempt-based-on-flavor-order/README.md b/keps/582-preempt-based-on-flavor-order/README.md
index eb094325a..6d811ccbb 100644
--- a/keps/582-preempt-based-on-flavor-order/README.md
+++ b/keps/582-preempt-based-on-flavor-order/README.md
@@ -165,52 +165,82 @@ proposal will be implemented, this is the place to discuss them.
 
 ### Cluster Queue API
 
-We extend the Cluster Queue API to introduce the new fields: flavorFungibility to opt-in and configure the new behavior.
+We extend the Cluster Queue API to introduce the new field `flavorFungibility` to opt-in and configure the behavior,
+and a `preference` field for choosing borrowing-first vs preemption-first when the scheduler is exploring multiple flavors.
 
-For each type of resource in each podSet, Kueue will traverse all resource groups and resource flavors to find a available flavor in present. When there are insufficient resources in the flavor, kueue will prioritize preemption or borrowing based on the configured policy. 
+For each type of resource in each podSet, Kueue will traverse all resource groups and resource flavors to find an available flavor in present.
+For each flavor evaluation we consider the tuple `(borrowingDistance, preemptionMode)`;
+the additional preference field indicates whether the comparison should minimize preemption first or avoid borrowing first when ordering feasible tuples.
+When there are insufficient resources in the flavor, Kueue will prioritize preemption or borrowing based on the configured policy and, when applicable, the preference.
 
 ```
 const (
-	Borrow        FlavorFungibilityPolicy = "Borrow"
-	Preempt       FlavorFungibilityPolicy = "Preempt"
-	MayStopSearch FlavorFungibilityPolicy = "MayStopSearch"
-	TryNextFlavor FlavorFungibilityPolicy = "TryNextFlavor"
+    Borrow        FlavorFungibilityPolicy = "Borrow"        // deprecated alias of MayStopSearch
+    Preempt       FlavorFungibilityPolicy = "Preempt"       // deprecated alias of MayStopSearch
+    MayStopSearch FlavorFungibilityPolicy = "MayStopSearch"
+    TryNextFlavor FlavorFungibilityPolicy = "TryNextFlavor"
+)
+
+type FlavorFungibilityPreference string
+
+const (
+    PreferenceBorrowingOverPreemption  FlavorFungibilityPreference = "BorrowingOverPreemption"
+    PreferencePreemptionOverBorrowing  FlavorFungibilityPreference = "PreemptionOverBorrowing"
 )
 
 type FlavorFungibility struct {
   // whenCanBorrow determines whether a workload should try the next flavor
-  // or stop the search. The possible values are:
+  // before borrowing in current flavor. The possible values are:
   //
-	// - `MayStopSearch` (default): stop the search for candidate flavors if workload
-	//   fits or requires borrowing to fit.
-	// - `TryNextFlavor`: try next flavor if workload requires borrowing to fit.
-	// - `Borrow` (deprecated): old name for `MayStopSearch`; please use new name.
+  // - `MayStopSearch` (default): stop the search for candidate flavors if workload
+  //   fits or requires borrowing to fit.
+  // - `TryNextFlavor`: try next flavor if workload requires borrowing to fit.
+  // - `Borrow` (deprecated): old name for `MayStopSearch`; please use new name.
   //
-	// +kubebuilder:validation:Enum={MayStopSearch,TryNextFlavor,Borrow}
-	// +kubebuilder:default="MayStopSearch"
-  WhenCanBorrow FlavorFungibilityPolicy  `json:"whenCanBorrow"`
+  // +kubebuilder:validation:Enum={MayStopSearch,TryNextFlavor,Borrow}
+  // +kubebuilder:default="MayStopSearch"
+  WhenCanBorrow FlavorFungibilityPolicy  `json:"whenCanBorrow,omitempty"`
+
   // whenCanPreempt determines whether a workload should try the next flavor
-  // or stop the search. The possible values are:
+  // before preempting in current flavor. The possible values are:
   //
-	// - `MayStopSearch`: stop the search for candidate flavors if workload fits or requires
-	//   preemption to fit.
-	// - `TryNextFlavor` (default): try next flavor if workload requires preemption
-	//   to fit in current flavor.
-	// - `Preempt` (deprecated): old name for `MayStopSearch`; please use new name.
+  // - `MayStopSearch`: stop the search for candidate flavors if workload fits or requires
+  //   preemption to fit.
+  // - `TryNextFlavor` (default): try next flavor if workload requires preemption
+  //   to fit in current flavor.
+  // - `Preempt` (deprecated): old name for `MayStopSearch`; please use new name.
   //
-	// +kubebuilder:validation:Enum={MayStopSearch,TryNextFlavor,Preempt}
-	// +kubebuilder:default="TryNextFlavor"
-  WhenCanPreempt FlavorFungibilityPolicy `json:"whenCanPreempt"`
+  // +kubebuilder:validation:Enum={MayStopSearch,TryNextFlavor,Preempt}
+  // +kubebuilder:default="TryNextFlavor"
+  WhenCanPreempt FlavorFungibilityPolicy `json:"whenCanPreempt,omitempty"`
+
+  // preference guides the choosing of the flavor for admission in case all candidate flavors
+  // require either preemption, borrowing, or both. The possible values are:
+  // - `BorrowingOverPreemption` (default): prefer to use borrowing rather than preemption
+  // when such a choice is possible. More technically it minimizes the borrowing distance
+  // in the cohort tree, and solves tie-breaks by preferring better preemption mode
+  // (reclaim over preemption within ClusterQueue).
+  // - `PreemptionOverBorrowing`: prefer to use preemption rather than borrowing
+  // when such a choice is possible.  More technically it optimizes the preemption mode
+  // (reclaim over preemption within ClusterQueue), and solves tie-breaks by minimizing
+  // the borrowing distance in the cohort tree.
+  //
+  // Validation:
+  // - Must be unset unless WhenCanBorrow=TryNextFlavor AND WhenCanPreempt=TryNextFlavor.
+  //
+  // +kubebuilder:validation:Enum={BorrowingOverPreemption,PreemptionOverBorrowing}
+  // +optional
+  Preference *FlavorFungibilityPreference `json:"preference,omitempty"`
 }
 
 // ClusterQueueSpec defines the desired state of ClusterQueue
 type ClusterQueueSpec struct {
-	...
-	FlavorFungibility FlavorFungibility `json:"flavorFungibility"`
+    ...
+    FlavorFungibility FlavorFungibility `json:"flavorFungibility,omitempty"`
 }
 ```
 
-If flavorFungibility is nil in configuration, we will set the `WhenCanBorrow` to `MayStopSearch` and set `WhenCanPreempt` to `TryNextFlavor` to maintain consistency with the current behavior.
+If `flavorFungibility` is nil in configuration, we will set `WhenCanBorrow` to `MayStopSearch` and `WhenCanPreempt` to `TryNextFlavor` to maintain consistency with the current behavior.
 
 ### Behavior Changes
 
@@ -227,13 +257,24 @@ We try to schedule a podset in succesive resource flavors in a loop and we decid
 | `Fit`, `Borrow` | `WhenCanBorrow = MayStopSearch` | break |
 | `Fit`, `NoBorrow`| any| break| 
 
-By `Borrow`/`NoBorrow` we mean whether borrowing is required or not to fit the considered podset in the sumulation result. After we complete the loop, either by trying all the flavors or by breaking out of it, we end up with a list of possible flavors containing all the considered flavors that yielded a result different than `NoFit`. We choose the flavor to assign based on the following default preference order: (`Fit`, `NoBorrow`), (`Fit`, `Borrow`), (`Preempt`, `NoBorrow`), (`Preempt`, `Borrow`).
+By `Borrow`/`NoBorrow` we mean whether borrowing is required or not to fit the considered podset in the simulation result. After we complete the loop, either by trying all the flavors or by breaking out of it, we end up with a list of possible flavors containing all the considered flavors that yielded a result different than `NoFit`. We choose the flavor to assign based on the following default preference order: (`Fit`, `NoBorrow`), (`Fit`, `Borrow`), (`Preempt`, `NoBorrow`), (`Preempt`, `Borrow`).
+
+When both `WhenCanBorrow` and `WhenCanPreempt` are `TryNextFlavor`, a ClusterQueue MAY set:
+`spec.flavorFungibility.preference: BorrowingOverPreemption | PreemptionOverBorrowing`.
+
+If preference is:
+- BorrowingOverPreemption (or unset): (`Fit`, `NoBorrow`) > (`Fit`, `Borrow`) > (`Preempt`, `NoBorrow`) > (`Preempt`, `Borrow`).
+- PreemptionOverBorrowing:            (`Fit`, `NoBorrow`) > (`Preempt`, `NoBorrow`) > (`Fit`, `Borrow`) > (`Preempt`, `Borrow`).
+
+The `PreemptionOverBorrowing` mode is introduced as a replacement for the `FlavorFungibilityImplicitPreferenceDefault` feature gate; enabling that gate applies the new ordering without requiring each ClusterQueue to opt in.
 
-An alternative order of preference can be set by enabling the feature gate `FlavorFungibilityImplicitPreferenceDefault`.
-The alternative order prioritizes assignments that don't borrow: (`Fit`, `NoBorrow`),(`Preempt`, `NoBorrow`), (`Fit`, `Borrow`), (`Preempt`, `Borrow`). It is used in two cases:
+In all other combinations (i.e., either policy is `MayStopSearch`), the search stops early and the preference field MUST be omitted; the implied order effectively prioritizes the stopping condition:
+- `WhenCanBorrow=TryNextFlavor`, `WhenCanPreempt=MayStopSearch` → preemption-first among viable options.
+- `WhenCanBorrow=MayStopSearch`, `WhenCanPreempt=TryNextFlavor` → borrowing-first among viable options.
 
-1. If `WhenCanPreempt = MayStopSearch` and `WhenCanBorrow = TryNextFlavor`
-2. If `WhenCanPreempt = TryNextFlavor` and `WhenCanBorrow = TryNextFlavor`
+Validation rules:
+- `spec.flavorFungibility.preference` MUST be unset unless both `WhenCanBorrow` and `WhenCanPreempt` are `TryNextFlavor`.
+- If set, `preference` MUST be one of: `BorrowingOverPreemption`, `PreemptionOverBorrowing`.
 
 We will store the scheduling context in workload info so that we can start from where we stop in previous scheduling attempts. This will be useful to avoid to waste time in one flavor all the time if we try to preempt in a flavor and failed. Scheduling context will contain the `LastTriedFlavorIdx`, `ClusterQueueGeneration` attached to the CQ and `CohortGeneration`. Any changes to these properties will lead to a scheduling from the first flavor.
 
@@ -410,10 +451,8 @@ milestones with these graduation criteria:
 [deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/
 -->
 
-The feature gate `FlavorFungibilityImplicitPreferenceDefault` is a temporary measure
-until a new shape of the API, that allows to explicitely define the preference
-is implemented. The feature gate will be removed once the API is updated
-(planned for kueue v0.14).
+The feature gate `FlavorFungibilityImplicitPreferenceDefault` is deprecated, but remains available through v0.15 to ease the transition to explicit preferences and will be removed in v0.16.
+Once the gate is removed, the preference must be configured per ClusterQueue via `spec.flavorFungibility.preference`, with `BorrowingOverPreemption` as the default.
 
 ## Implementation History
 
diff --git a/keps/582-preempt-based-on-flavor-order/kep.yaml b/keps/582-preempt-based-on-flavor-order/kep.yaml
index 4a9dd517f..59eeb2417 100644
--- a/keps/582-preempt-based-on-flavor-order/kep.yaml
+++ b/keps/582-preempt-based-on-flavor-order/kep.yaml
@@ -26,7 +26,10 @@ milestone:
 # List the feature gate name and the components for which it must be enabled
 feature-gates:
   - FlavorFungibility
-  - FlavorFungibilityImplicitPreferenceDefault
+  # FlavorFungibilityImplicitPreferenceDefault is a deprecated gate that keeps the
+  # implicit borrow-vs-preempt ordering until v0.16 when it will be removed in
+  # favor of configuring spec.flavorFungibility.preference per ClusterQueue.
+  # See https://github.com/kubernetes-sigs/kueue/issues/6184 and PR #6133.
 disable-supported: true
 
 # The following PRR answers are required at beta release



Test patch:
diff --git a/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go b/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
index 5dd9e20c1..8afef4443 100644
--- a/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
+++ b/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
@@ -1226,12 +1226,11 @@ var _ = ginkgo.Describe("Scheduler", ginkgo.Ordered, ginkgo.ContinueOnFailure, f
 
 		ginkgo.It("Guaranteed workloads cause preemption of a single best effort workload", func() {
 			ginkgo.By("Creating two best effort workloads in each best effort CQ")
-			wlBestEffortA := createWorkload("best-effort-a", "4")
-			util.WaitForNextSecondAfterCreation(wlBestEffortA)
+			wlBestEffortA := createWorkloadWithPriority("best-effort-a", "4", 2)
 			util.ExpectWorkloadsToBeAdmitted(ctx, k8sClient, wlBestEffortA)
 			util.ExpectAdmittedWorkloadsTotalMetric(bestEffortCQA, "", 1)
 			util.ExpectReservingActiveWorkloadsMetric(bestEffortCQA, 1)
-			wlBestEffortB := createWorkload("best-effort-b", "4")
+			wlBestEffortB := createWorkloadWithPriority("best-effort-b", "4", 1)
 			util.ExpectAdmittedWorkloadsTotalMetric(bestEffortCQB, "", 1)
 			util.ExpectReservingActiveWorkloadsMetric(bestEffortCQB, 1)
 
diff --git a/test/util/util.go b/test/util/util.go
index 0e3ecb1a9..97e913c3f 100644
--- a/test/util/util.go
+++ b/test/util/util.go
@@ -1204,6 +1204,7 @@ func FindDeploymentCondition(deployment *appsv1.Deployment, deploymentType appsv
 }
 
 func GetListOptsFromLabel(label string) *client.ListOptions {
+	ginkgo.GinkgoHelper()
 	selector, err := labels.Parse(label)
 	gomega.Expect(err).NotTo(gomega.HaveOccurred())
 	return &client.ListOptions{
@@ -1260,6 +1261,7 @@ func SetNodeCondition(ctx context.Context, k8sClient client.Client, node *corev1
 }
 
 func ExpectLocalQueueFairSharingUsageToBe(ctx context.Context, k8sClient client.Client, lqKey client.ObjectKey, comparator string, compareTo any) {
+	ginkgo.GinkgoHelper()
 	lq := &kueue.LocalQueue{}
 	gomega.Eventually(func(g gomega.Gomega) {
 		g.Expect(k8sClient.Get(ctx, lqKey, lq)).Should(gomega.Succeed())
@@ -1283,6 +1285,7 @@ func ExpectLocalQueueFairSharingUsageToBe(ctx context.Context, k8sClient client.
 //
 //	The slice of Workloads present in the namespace when the expectation is met.
 func ExpectWorkloadsInNamespace(ctx context.Context, k8sClient client.Client, namespace string, count int) []kueue.Workload {
+	ginkgo.GinkgoHelper()
 	list := &kueue.WorkloadList{}
 	gomega.Eventually(func(g gomega.Gomega) {
 		g.Expect(k8sClient.List(ctx, list, client.InNamespace(namespace))).To(gomega.Succeed())
@@ -1305,6 +1308,7 @@ func ExpectWorkloadsInNamespace(ctx context.Context, k8sClient client.Client, na
 //   - newWorkload: A pointer to the discovered replacement Workload. Guaranteed
 //     non-nil if the function succeeds; otherwise, the test fails before returning.
 func ExpectNewWorkloadSlice(ctx context.Context, k8sClient client.Client, oldWorkload *kueue.Workload) (newWorkload *kueue.Workload) {
+	ginkgo.GinkgoHelper()
 	gomega.Eventually(func(g gomega.Gomega) {
 		wlList := &kueue.WorkloadList{}
 		g.Expect(k8sClient.List(ctx, wlList, client.InNamespace(oldWorkload.Namespace))).To(gomega.Succeed())