Repository: kubernetes-sigs/kueue

Issue:
[flaky integration test] Preemption is enabled within ClusterQueue should preempt the low and mid priority workloads to fit the high-priority workload
/kind flake 

**What happened**:

failure on unrelated branch https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5394/pull-kueue-test-integration-baseline-main/1927649254336106496

**What you expected to happen**:
no failure
**How to reproduce it (as minimally and precisely as possible)**:
ci
**Anything else we need to know?**:

```
TopologyAwareScheduling Suite: [It] Topology Aware Scheduling when Single TAS Resource Flavor when Preemption is enabled within ClusterQueue should preempt the low and mid priority workloads to fit the high-priority workload expand_less	13s
{Timed out after 10.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/util/util.go:365 with:
Not enough workloads are admitted
Expected
    <int>: 1
to equal
    <int>: 2 failed [FAILED] Timed out after 10.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/util/util.go:365 with:
Not enough workloads are admitted
Expected
    <int>: 1
to equal
    <int>: 2
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/integration/singlecluster/tas/tas_test.go:1374 @ 05/28/25 09:06:47.971
}
```

Discussion:
@mimowo: cc @mbobrovskyi @mykysha ptal@tenzen-y: We observed the same failure on #5416, again.@mimowo: cc @mszadkow @mszadkow: /assign@mszadkow: Something very strange happened in that test run.
`wl1` got admitted and later it was verified, when `wl2` was evaluated for admission `wl1` disappeared without a trace...
What I mean is there was no `delete` or `preemption` recorded, maybe just this attempt to preempt `wl1` which meanwhile disappear.
 
```
2025-05-28T09:06:37.97242884Z	LEVEL(-2)	scheduler	scheduler/scheduler.go:226	Attempting to schedule workload	{"schedulingCycle": 38, "workload": {"name":"wl2","namespace":"tas-94dv5"}, "clusterQueue": {"name":"cluster-queue"}}
  2025-05-28T09:06:37.975342209Z	LEVEL(-2)	scheduler	scheduler/scheduler.go:572	Workload not admitted because it was deleted	{"schedulingCycle": 37, "workload": {"name":"wl1","namespace":"tas-7vrc8"}, "clusterQueue": {"name":"cluster-queue"}}
  2025-05-28T09:06:37.978364299Z	ERROR	scheduler	scheduler/scheduler.go:268	Failed to preempt workloads	{"schedulingCycle": 38, "workload": {"name":"wl2","namespace":"tas-94dv5"}, "clusterQueue": {"name":"cluster-queue"}, "error": "workloads.kueue.x-k8s.io \"wl1\" not found"}
  sigs.k8s.io/kueue/pkg/scheduler.(*Scheduler).schedule
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/scheduler/scheduler.go:268
  sigs.k8s.io/kueue/pkg/util/wait.untilWithBackoff.func1
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/util/wait/backoff.go:43
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233
  k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:255
  k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:256
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233
  sigs.k8s.io/kueue/pkg/util/wait.untilWithBackoff
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/util/wait/backoff.go:42
  sigs.k8s.io/kueue/pkg/util/wait.UntilWithBackoff
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/util/wait/backoff.go:34
  2025-05-28T09:06:37.978562602Z	LEVEL(-3)	scheduler	scheduler/logging.go:43	Workload evaluated for admission	{"schedulingCycle": 38, "workload": {"name":"wl2","namespace":"tas-94dv5"}, "clusterQueue": {"name":"cluster-queue"}, "status": "", "reason": "couldn't assign flavors to pod set worker: topology \"default\" doesn't allow to fit any of 1 pod(s)"}
```@mimowo: Could you try to repro locally by looping the test under some load? Maybe we could capture it locally.@mszadkow: I have tried already, nothing happens, works smoothly.
But I will keep trying, I just wanted to share this "weird" thing that `wl1` namespace somehow got changed.
From `tas-94dv5` to `tas-7vrc8`, which is from the previous test case...

```
  2025-05-28T09:06:37.710423748Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:747	Workload update event	{"workload": {"name":"wl1","namespace":"tas-94dv5"}, "queue": "local-queue", "status": "admitted", "prevStatus": "pending", "clusterQueue": "cluster-queue"}
...
2025-05-28T09:06:37.975342209Z	LEVEL(-2)	scheduler	scheduler/scheduler.go:572	Workload not admitted because it was deleted	{"schedulingCycle": 37, "workload": {"name":"wl1","namespace":"tas-7vrc8"}, "clusterQueue": {"name":"cluster-queue"}}
```
My suspect is that we share lq and cq as pointers between tests...
But I would like to get repro anyway@mimowo: Thank you for investigating that. Indeed, I don't exclude the possibility of cross-test interaction, this would be hard to capture. Even if you are out of ideas for now, please think if we could improve the logging (either in tests) or in code which could be helpful next time.@mimowo: another recent occurence: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5433/pull-kueue-test-integration-baseline-release-0-12/1929436649175912448@mszadkow: Yes, I believe the fix is ready to be merged - https://github.com/kubernetes-sigs/kueue/pull/5423@mbobrovskyi: Looks like we still have an issue https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5443/pull-kueue-test-integration-baseline-main/1929489390048907264.@mbobrovskyi: /reopen@k8s-ci-robot: @mbobrovskyi: Reopened this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/5395#issuecomment-2930073887):

>/reopen


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@mbobrovskyi: One more

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5225/pull-kueue-test-integration-baseline-main/1929477214840033280@mszadkow: ack, looking into logs@mimowo: same here: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5440/pull-kueue-test-integration-baseline-main/1929495861822230528@mszadkow: The next suspect is the `queue manager` as it seems that `wl1` from the previous test is considered for preemption.
While it should be `wl1` from the current test and `wl1` from previous test should be removed from the queue already.
Investigating there...although still it's not possible to reproduce locally.@mimowo: Hmm, this might be a bug related to this code: https://github.com/kubernetes-sigs/kueue/blob/e3bbc66fa7c95df49668a33934e9ae9c97d9b65a/pkg/queue/manager.go#L793-L795

It might be the goroutine is spawned to add the workload after 1s and it does not consider the context is already cancelled when test ended.

Another fix might be to do not enqueue if the CQ no longer exists. 

To test this hypothesis locally maybe you could extend 1s to 2s.@tenzen-y: We observed this again.

https://github.com/kubernetes-sigs/kueue/pull/5432
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5432/pull-kueue-test-integration-baseline-main/1930110922383167488@tenzen-y: Once more:

- https://github.com/kubernetes-sigs/kueue/pull/5491
- https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5491/pull-kueue-test-integration-baseline-release-0-12/1930119799250096128

Solution patch:
diff --git a/.github/workflows/sync-dependabot.yaml b/.github/workflows/sync-dependabot.yaml
index 41c3008e8..5969d11e7 100644
--- a/.github/workflows/sync-dependabot.yaml
+++ b/.github/workflows/sync-dependabot.yaml
@@ -36,7 +36,7 @@ jobs:
             - name: Sync controller-tools version
               if: ${{ contains(github.event.pull_request.title, 'sigs.k8s.io/controller-tools') }}
               run: |
-                make manifests
+                make manifests update-helm
                 if ! git diff --exit-code; then
                   git add .
                   git commit -m "Sync controller-tools version"
diff --git a/go.mod b/go.mod
index 217174620..3714a00df 100644
--- a/go.mod
+++ b/go.mod
@@ -5,7 +5,7 @@ go 1.24.0
 require (
 	github.com/cert-manager/cert-manager v1.17.2
 	github.com/fsnotify/fsnotify v1.9.0
-	github.com/go-logr/logr v1.4.2
+	github.com/go-logr/logr v1.4.3
 	github.com/google/go-cmp v0.7.0
 	github.com/json-iterator/go v1.1.12
 	github.com/kubeflow/mpi-operator v0.6.0
diff --git a/go.sum b/go.sum
index c9ab68560..a5b36fb15 100644
--- a/go.sum
+++ b/go.sum
@@ -54,8 +54,8 @@ github.com/fxamacker/cbor/v2 v2.7.0/go.mod h1:pxXPTn3joSm21Gbwsv0w9OSA2y1HFR9qXE
 github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
 github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
 github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
-github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
-github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
+github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
+github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
 github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
 github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
 github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ=
diff --git a/hack/internal/tools/go.mod b/hack/internal/tools/go.mod
index 5ff660770..3b0af3781 100644
--- a/hack/internal/tools/go.mod
+++ b/hack/internal/tools/go.mod
@@ -3,7 +3,7 @@ module sigs.k8s.io/kueue/internal/tools
 go 1.24.0
 
 require (
-	github.com/gohugoio/hugo v0.147.6
+	github.com/gohugoio/hugo v0.147.7
 	github.com/golangci/golangci-lint/v2 v2.1.6
 	github.com/google/go-cmp v0.7.0
 	github.com/mikefarah/yq/v4 v4.45.4
@@ -12,7 +12,7 @@ require (
 	gopkg.in/op/go-logging.v1 v1.0.0-20160211212156-b2cb9fa56473
 	gopkg.in/yaml.v3 v3.0.1
 	gotest.tools/gotestsum v1.12.2
-	helm.sh/helm/v3 v3.18.0
+	helm.sh/helm/v3 v3.18.1
 	k8s.io/code-generator v0.33.1
 	sigs.k8s.io/controller-runtime/tools/setup-envtest v0.0.0-20250407161039-6ad5c1dd4418
 	sigs.k8s.io/controller-tools v0.17.3
@@ -94,7 +94,7 @@ require (
 	github.com/russross/blackfriday/v2 v2.1.0 // indirect
 	github.com/shopspring/decimal v1.4.0 // indirect
 	github.com/spf13/afero v1.14.0 // indirect
-	github.com/spf13/cast v1.7.1 // indirect
+	github.com/spf13/cast v1.8.0 // indirect
 	github.com/spf13/cobra v1.9.1 // indirect
 	github.com/spf13/pflag v1.0.6 // indirect
 	github.com/x448/float16 v0.8.4 // indirect
diff --git a/hack/internal/tools/go.sum b/hack/internal/tools/go.sum
index aaf5d278a..04a5eaa74 100644
--- a/hack/internal/tools/go.sum
+++ b/hack/internal/tools/go.sum
@@ -88,8 +88,8 @@ github.com/goccy/go-yaml v1.17.1 h1:LI34wktB2xEE3ONG/2Ar54+/HJVBriAGJ55PHls4YuY=
 github.com/goccy/go-yaml v1.17.1/go.mod h1:XBurs7gK8ATbW4ZPGKgcbrY1Br56PdM69F7LkFRi1kA=
 github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
 github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
-github.com/gohugoio/hugo v0.147.6 h1:rL4rnus/5qzj4+FoA+JMzsVvFJ2YZdVIH6pbuCB2P84=
-github.com/gohugoio/hugo v0.147.6/go.mod h1:Sb2COQPDPYG+tRSpePtzKytiuVDqkBivEhgIew1QbNo=
+github.com/gohugoio/hugo v0.147.7 h1:7qQKI8wsPgF1ipYBcXgM8wFmqTyFpkmzqLEf3hpzpT8=
+github.com/gohugoio/hugo v0.147.7/go.mod h1:gBn9Oi4LomFk1XS9raAPHdxaPrhPoF8ZfRrEcZZFGpo=
 github.com/golangci/golangci-lint/v2 v2.1.6 h1:LXqShFfAGM5BDzEOWD2SL1IzJAgUOqES/HRBsfKjI+w=
 github.com/golangci/golangci-lint/v2 v2.1.6/go.mod h1:EPj+fgv4TeeBq3TcqaKZb3vkiV5dP4hHHKhXhEhzci8=
 github.com/gomarkdown/markdown v0.0.0-20240328165702-4d01890c35c0 h1:4gjrh/PN2MuWCCElk8/I4OCKRKWCCo2zEct3VKCbibU=
@@ -202,8 +202,8 @@ github.com/shopspring/decimal v1.4.0 h1:bxl37RwXBklmTi0C79JfXCEBD1cqqHt0bbgBAGFp
 github.com/shopspring/decimal v1.4.0/go.mod h1:gawqmDU56v4yIKSwfBSFip1HdCCXN8/+DMd9qYNcwME=
 github.com/spf13/afero v1.14.0 h1:9tH6MapGnn/j0eb0yIXiLjERO8RB6xIVZRDCX7PtqWA=
 github.com/spf13/afero v1.14.0/go.mod h1:acJQ8t0ohCGuMN3O+Pv0V0hgMxNYDlvdk+VTfyZmbYo=
-github.com/spf13/cast v1.7.1 h1:cuNEagBQEHWN1FnbGEjCXL2szYEXqfJPbP2HNUaca9Y=
-github.com/spf13/cast v1.7.1/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
+github.com/spf13/cast v1.8.0 h1:gEN9K4b8Xws4EX0+a0reLmhq8moKn7ntRlQYgjPeCDk=
+github.com/spf13/cast v1.8.0/go.mod h1:ancEpBxwJDODSW/UG4rDrAqiKolqNNh2DX3mk86cAdo=
 github.com/spf13/cobra v1.9.1 h1:CXSaggrXdbHK9CF+8ywj8Amf7PBRmPCOJugH954Nnlo=
 github.com/spf13/cobra v1.9.1/go.mod h1:nDyEzZ8ogv936Cinf6g1RU9MRY64Ir93oCnqb9wxYW0=
 github.com/spf13/pflag v1.0.6 h1:jFzHGLGAlb3ruxLB8MhbI6A8+AQX/2eW4qeyNZXNp2o=
@@ -313,8 +313,8 @@ gotest.tools/gotestsum v1.12.2 h1:eli4tu9Q2D/ogDsEGSr8XfQfl7mT0JsGOG6DFtUiZ/Q=
 gotest.tools/gotestsum v1.12.2/go.mod h1:kjRtCglPZVsSU0hFHX3M5VWBM6Y63emHuB14ER1/sow=
 gotest.tools/v3 v3.5.2 h1:7koQfIKdy+I8UTetycgUqXWSDwpgv193Ka+qRsmBY8Q=
 gotest.tools/v3 v3.5.2/go.mod h1:LtdLGcnqToBH83WByAAi/wiwSFCArdFIUV/xxN4pcjA=
-helm.sh/helm/v3 v3.18.0 h1:ItOAm3Quo0dus3NUHjs+lluqWWEIO7xrSW+zKWCrvlw=
-helm.sh/helm/v3 v3.18.0/go.mod h1:43QHS1W97RcoFJRk36ZBhHdTfykqBlJdsWp3yhzdq8w=
+helm.sh/helm/v3 v3.18.1 h1:qLhXmtqXOHQb0Xv9HJolOLlah8RWbgyzt50xrtTWAlg=
+helm.sh/helm/v3 v3.18.1/go.mod h1:43QHS1W97RcoFJRk36ZBhHdTfykqBlJdsWp3yhzdq8w=
 k8s.io/api v0.33.1 h1:tA6Cf3bHnLIrUK4IqEgb2v++/GYUtqiu9sRVk3iBXyw=
 k8s.io/api v0.33.1/go.mod h1:87esjTn9DRSRTD4fWMXamiXxJhpOIREjWOSjsW1kEHw=
 k8s.io/apiextensions-apiserver v0.33.0 h1:d2qpYL7Mngbsc1taA4IjJPRJ9ilnsXIrndH+r9IimOs=
diff --git a/netlify.toml b/netlify.toml
index 4cca16bc9..a30b007c6 100644
--- a/netlify.toml
+++ b/netlify.toml
@@ -4,7 +4,7 @@
   command = "npm install && hugo --gc --minify"
 
 [build.environment]
-  HUGO_VERSION = "0.147.6"
+  HUGO_VERSION = "0.147.7"
 
 [context.production.environment]
   HUGO_ENV = "production"
diff --git a/vendor/github.com/go-logr/logr/.golangci.yaml b/vendor/github.com/go-logr/logr/.golangci.yaml
index 0cffafa7b..0ed62c1a1 100644
--- a/vendor/github.com/go-logr/logr/.golangci.yaml
+++ b/vendor/github.com/go-logr/logr/.golangci.yaml
@@ -1,26 +1,28 @@
+version: "2"
+
 run:
   timeout: 1m
   tests: true
 
 linters:
-  disable-all: true
-  enable:
+  default: none
+  enable: # please keep this alphabetized
+    - asasalint
     - asciicheck
+    - copyloopvar
+    - dupl
     - errcheck
     - forcetypeassert
+    - goconst
     - gocritic
-    - gofmt
-    - goimports
-    - gosimple
     - govet
     - ineffassign
     - misspell
+    - musttag
     - revive
     - staticcheck
-    - typecheck
     - unused
 
 issues:
-  exclude-use-default: false
   max-issues-per-linter: 0
   max-same-issues: 10
diff --git a/vendor/github.com/go-logr/logr/funcr/funcr.go b/vendor/github.com/go-logr/logr/funcr/funcr.go
index 30568e768..b22c57d71 100644
--- a/vendor/github.com/go-logr/logr/funcr/funcr.go
+++ b/vendor/github.com/go-logr/logr/funcr/funcr.go
@@ -77,7 +77,7 @@ func newSink(fn func(prefix, args string), formatter Formatter) logr.LogSink {
 		write:     fn,
 	}
 	// For skipping fnlogger.Info and fnlogger.Error.
-	l.Formatter.AddCallDepth(1)
+	l.AddCallDepth(1) // via Formatter
 	return l
 }
 
@@ -164,17 +164,17 @@ type fnlogger struct {
 }
 
 func (l fnlogger) WithName(name string) logr.LogSink {
-	l.Formatter.AddName(name)
+	l.AddName(name) // via Formatter
 	return &l
 }
 
 func (l fnlogger) WithValues(kvList ...any) logr.LogSink {
-	l.Formatter.AddValues(kvList)
+	l.AddValues(kvList) // via Formatter
 	return &l
 }
 
 func (l fnlogger) WithCallDepth(depth int) logr.LogSink {
-	l.Formatter.AddCallDepth(depth)
+	l.AddCallDepth(depth) // via Formatter
 	return &l
 }
 
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 08a73b0ee..c8a83dd70 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -77,7 +77,7 @@ github.com/fxamacker/cbor/v2
 # github.com/go-errors/errors v1.4.2
 ## explicit; go 1.14
 github.com/go-errors/errors
-# github.com/go-logr/logr v1.4.2
+# github.com/go-logr/logr v1.4.3
 ## explicit; go 1.18
 github.com/go-logr/logr
 github.com/go-logr/logr/funcr



Test patch:
diff --git a/vendor/github.com/go-logr/logr/testr/testr.go b/vendor/github.com/go-logr/logr/testr/testr.go
index 5eabe2b6f..28168e0ae 100644
--- a/vendor/github.com/go-logr/logr/testr/testr.go
+++ b/vendor/github.com/go-logr/logr/testr/testr.go
@@ -130,12 +130,12 @@ type testloggerInterface struct {
 }
 
 func (l testloggerInterface) WithName(name string) logr.LogSink {
-	l.Formatter.AddName(name)
+	l.AddName(name) // via Formatter
 	return &l
 }
 
 func (l testloggerInterface) WithValues(kvList ...any) logr.LogSink {
-	l.Formatter.AddValues(kvList)
+	l.AddValues(kvList) // via Formatter
 	return &l
 }