Repository: kubernetes-sigs/kueue

Issue:
[flaky integration test] Preemption is enabled within ClusterQueue should preempt the low and mid priority workloads to fit the high-priority workload
/kind flake 

**What happened**:

failure on unrelated branch https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5394/pull-kueue-test-integration-baseline-main/1927649254336106496

**What you expected to happen**:
no failure
**How to reproduce it (as minimally and precisely as possible)**:
ci
**Anything else we need to know?**:

```
TopologyAwareScheduling Suite: [It] Topology Aware Scheduling when Single TAS Resource Flavor when Preemption is enabled within ClusterQueue should preempt the low and mid priority workloads to fit the high-priority workload expand_less	13s
{Timed out after 10.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/util/util.go:365 with:
Not enough workloads are admitted
Expected
    <int>: 1
to equal
    <int>: 2 failed [FAILED] Timed out after 10.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/util/util.go:365 with:
Not enough workloads are admitted
Expected
    <int>: 1
to equal
    <int>: 2
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/integration/singlecluster/tas/tas_test.go:1374 @ 05/28/25 09:06:47.971
}
```

Discussion:
@mimowo: cc @mbobrovskyi @mykysha ptal@tenzen-y: We observed the same failure on #5416, again.@mimowo: cc @mszadkow @mszadkow: /assign@mszadkow: Something very strange happened in that test run.
`wl1` got admitted and later it was verified, when `wl2` was evaluated for admission `wl1` disappeared without a trace...
What I mean is there was no `delete` or `preemption` recorded, maybe just this attempt to preempt `wl1` which meanwhile disappear.
 
```
2025-05-28T09:06:37.97242884Z	LEVEL(-2)	scheduler	scheduler/scheduler.go:226	Attempting to schedule workload	{"schedulingCycle": 38, "workload": {"name":"wl2","namespace":"tas-94dv5"}, "clusterQueue": {"name":"cluster-queue"}}
  2025-05-28T09:06:37.975342209Z	LEVEL(-2)	scheduler	scheduler/scheduler.go:572	Workload not admitted because it was deleted	{"schedulingCycle": 37, "workload": {"name":"wl1","namespace":"tas-7vrc8"}, "clusterQueue": {"name":"cluster-queue"}}
  2025-05-28T09:06:37.978364299Z	ERROR	scheduler	scheduler/scheduler.go:268	Failed to preempt workloads	{"schedulingCycle": 38, "workload": {"name":"wl2","namespace":"tas-94dv5"}, "clusterQueue": {"name":"cluster-queue"}, "error": "workloads.kueue.x-k8s.io \"wl1\" not found"}
  sigs.k8s.io/kueue/pkg/scheduler.(*Scheduler).schedule
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/scheduler/scheduler.go:268
  sigs.k8s.io/kueue/pkg/util/wait.untilWithBackoff.func1
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/util/wait/backoff.go:43
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233
  k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:255
  k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:256
  k8s.io/apimachinery/pkg/util/wait.BackoffUntil
  	/home/prow/go/src/sigs.k8s.io/kueue/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233
  sigs.k8s.io/kueue/pkg/util/wait.untilWithBackoff
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/util/wait/backoff.go:42
  sigs.k8s.io/kueue/pkg/util/wait.UntilWithBackoff
  	/home/prow/go/src/sigs.k8s.io/kueue/pkg/util/wait/backoff.go:34
  2025-05-28T09:06:37.978562602Z	LEVEL(-3)	scheduler	scheduler/logging.go:43	Workload evaluated for admission	{"schedulingCycle": 38, "workload": {"name":"wl2","namespace":"tas-94dv5"}, "clusterQueue": {"name":"cluster-queue"}, "status": "", "reason": "couldn't assign flavors to pod set worker: topology \"default\" doesn't allow to fit any of 1 pod(s)"}
```@mimowo: Could you try to repro locally by looping the test under some load? Maybe we could capture it locally.@mszadkow: I have tried already, nothing happens, works smoothly.
But I will keep trying, I just wanted to share this "weird" thing that `wl1` namespace somehow got changed.
From `tas-94dv5` to `tas-7vrc8`, which is from the previous test case...

```
  2025-05-28T09:06:37.710423748Z	LEVEL(-2)	workload-reconciler	core/workload_controller.go:747	Workload update event	{"workload": {"name":"wl1","namespace":"tas-94dv5"}, "queue": "local-queue", "status": "admitted", "prevStatus": "pending", "clusterQueue": "cluster-queue"}
...
2025-05-28T09:06:37.975342209Z	LEVEL(-2)	scheduler	scheduler/scheduler.go:572	Workload not admitted because it was deleted	{"schedulingCycle": 37, "workload": {"name":"wl1","namespace":"tas-7vrc8"}, "clusterQueue": {"name":"cluster-queue"}}
```
My suspect is that we share lq and cq as pointers between tests...
But I would like to get repro anyway@mimowo: Thank you for investigating that. Indeed, I don't exclude the possibility of cross-test interaction, this would be hard to capture. Even if you are out of ideas for now, please think if we could improve the logging (either in tests) or in code which could be helpful next time.@mimowo: another recent occurence: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5433/pull-kueue-test-integration-baseline-release-0-12/1929436649175912448@mszadkow: Yes, I believe the fix is ready to be merged - https://github.com/kubernetes-sigs/kueue/pull/5423@mbobrovskyi: Looks like we still have an issue https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5443/pull-kueue-test-integration-baseline-main/1929489390048907264.@mbobrovskyi: /reopen@k8s-ci-robot: @mbobrovskyi: Reopened this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/5395#issuecomment-2930073887):

>/reopen


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@mbobrovskyi: One more

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5225/pull-kueue-test-integration-baseline-main/1929477214840033280@mszadkow: ack, looking into logs@mimowo: same here: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5440/pull-kueue-test-integration-baseline-main/1929495861822230528@mszadkow: The next suspect is the `queue manager` as it seems that `wl1` from the previous test is considered for preemption.
While it should be `wl1` from the current test and `wl1` from previous test should be removed from the queue already.
Investigating there...although still it's not possible to reproduce locally.@mimowo: Hmm, this might be a bug related to this code: https://github.com/kubernetes-sigs/kueue/blob/e3bbc66fa7c95df49668a33934e9ae9c97d9b65a/pkg/queue/manager.go#L793-L795

It might be the goroutine is spawned to add the workload after 1s and it does not consider the context is already cancelled when test ended.

Another fix might be to do not enqueue if the CQ no longer exists. 

To test this hypothesis locally maybe you could extend 1s to 2s.@tenzen-y: We observed this again.

https://github.com/kubernetes-sigs/kueue/pull/5432
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5432/pull-kueue-test-integration-baseline-main/1930110922383167488@tenzen-y: Once more:

- https://github.com/kubernetes-sigs/kueue/pull/5491
- https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/5491/pull-kueue-test-integration-baseline-release-0-12/1930119799250096128

Solution patch:
diff --git a/apis/config/v1beta1/defaults.go b/apis/config/v1beta1/defaults.go
index 7feb5734b..9c2c961ed 100644
--- a/apis/config/v1beta1/defaults.go
+++ b/apis/config/v1beta1/defaults.go
@@ -35,6 +35,7 @@ const (
 	DefaultWebhookServiceName                           = "kueue-webhook-service"
 	DefaultWebhookSecretName                            = "kueue-webhook-server-cert"
 	DefaultWebhookPort                                  = 9443
+	DefaultWebhookCertDir                               = "/tmp/k8s-webhook-server/serving-certs"
 	DefaultHealthProbeBindAddress                       = ":8081"
 	DefaultMetricsBindAddress                           = ":8443"
 	DefaultLeaderElectionID                             = "c1f6bfd2.kueue.x-k8s.io"
@@ -74,6 +75,9 @@ func SetDefaults_Configuration(cfg *Configuration) {
 	if cfg.Webhook.Port == nil {
 		cfg.Webhook.Port = ptr.To(DefaultWebhookPort)
 	}
+	if cfg.Webhook.CertDir == "" {
+		cfg.Webhook.CertDir = DefaultWebhookCertDir
+	}
 	if len(cfg.Metrics.BindAddress) == 0 {
 		cfg.Metrics.BindAddress = DefaultMetricsBindAddress
 	}
diff --git a/pkg/util/cert/cert.go b/pkg/util/cert/cert.go
index 00c28051a..cf5986434 100644
--- a/pkg/util/cert/cert.go
+++ b/pkg/util/cert/cert.go
@@ -28,7 +28,6 @@ import (
 )
 
 const (
-	certDir        = "/tmp/k8s-webhook-server/serving-certs"
 	vwcName        = "kueue-validating-webhook-configuration"
 	mwcName        = "kueue-mutating-webhook-configuration"
 	caName         = "kueue-ca"
@@ -49,7 +48,7 @@ func ManageCerts(mgr ctrl.Manager, cfg config.Configuration, setupFinished chan
 			Namespace: *cfg.Namespace,
 			Name:      *cfg.InternalCertManagement.WebhookSecretName,
 		},
-		CertDir:        certDir,
+		CertDir:        cfg.Webhook.CertDir,
 		CAName:         caName,
 		CAOrganization: caOrganization,
 		DNSName:        dnsName,



Test patch:
diff --git a/apis/config/v1beta1/defaults_test.go b/apis/config/v1beta1/defaults_test.go
index ecf85c93b..d0288ab9f 100644
--- a/apis/config/v1beta1/defaults_test.go
+++ b/apis/config/v1beta1/defaults_test.go
@@ -30,6 +30,7 @@ import (
 const (
 	overwriteNamespace              = "kueue-tenant-a"
 	overwriteWebhookPort            = 9444
+	overwriteWebhookCertDir         = "/tmp/test"
 	overwriteMetricBindAddress      = ":38081"
 	overwriteHealthProbeBindAddress = ":38080"
 	overwriteLeaderElectionID       = "foo.kueue.x-k8s.io"
@@ -46,7 +47,8 @@ func TestSetDefaults_Configuration(t *testing.T) {
 			ResourceName:  "c1f6bfd2.kueue.x-k8s.io",
 		},
 		Webhook: ControllerWebhook{
-			Port: ptr.To(DefaultWebhookPort),
+			Port:    ptr.To(DefaultWebhookPort),
+			CertDir: DefaultWebhookCertDir,
 		},
 		Metrics: ControllerMetrics{
 			BindAddress: DefaultMetricsBindAddress,
@@ -139,7 +141,8 @@ func TestSetDefaults_Configuration(t *testing.T) {
 				Namespace: ptr.To(DefaultNamespace),
 				ControllerManager: ControllerManager{
 					Webhook: ControllerWebhook{
-						Port: ptr.To(DefaultWebhookPort),
+						Port:    ptr.To(DefaultWebhookPort),
+						CertDir: DefaultWebhookCertDir,
 					},
 					Metrics: ControllerMetrics{
 						BindAddress: DefaultMetricsBindAddress,
@@ -170,7 +173,8 @@ func TestSetDefaults_Configuration(t *testing.T) {
 			original: &Configuration{
 				ControllerManager: ControllerManager{
 					Webhook: ControllerWebhook{
-						Port: ptr.To(overwriteWebhookPort),
+						Port:    ptr.To(overwriteWebhookPort),
+						CertDir: overwriteWebhookCertDir,
 					},
 					Metrics: ControllerMetrics{
 						BindAddress: overwriteMetricBindAddress,
@@ -197,7 +201,8 @@ func TestSetDefaults_Configuration(t *testing.T) {
 				Namespace: ptr.To(DefaultNamespace),
 				ControllerManager: ControllerManager{
 					Webhook: ControllerWebhook{
-						Port: ptr.To(overwriteWebhookPort),
+						Port:    ptr.To(overwriteWebhookPort),
+						CertDir: overwriteWebhookCertDir,
 					},
 					Metrics: ControllerMetrics{
 						BindAddress: overwriteMetricBindAddress,
@@ -239,7 +244,8 @@ func TestSetDefaults_Configuration(t *testing.T) {
 				Namespace: ptr.To(DefaultNamespace),
 				ControllerManager: ControllerManager{
 					Webhook: ControllerWebhook{
-						Port: ptr.To(DefaultWebhookPort),
+						Port:    ptr.To(DefaultWebhookPort),
+						CertDir: DefaultWebhookCertDir,
 					},
 					Metrics: ControllerMetrics{
 						BindAddress: DefaultMetricsBindAddress,
diff --git a/pkg/config/config_test.go b/pkg/config/config_test.go
index 42ef53023..77cf9699a 100644
--- a/pkg/config/config_test.go
+++ b/pkg/config/config_test.go
@@ -354,7 +354,8 @@ objectRetentionPolicies:
 		RetryPeriod:                   ptr.To(configapi.DefaultLeaderElectionRetryPeriod),
 		WebhookServer: &webhook.DefaultServer{
 			Options: webhook.Options{
-				Port: configapi.DefaultWebhookPort,
+				Port:    configapi.DefaultWebhookPort,
+				CertDir: configapi.DefaultWebhookCertDir,
 			},
 		},
 	}
@@ -444,7 +445,8 @@ objectRetentionPolicies:
 				RetryPeriod:                   ptr.To(configapi.DefaultLeaderElectionRetryPeriod),
 				WebhookServer: &webhook.DefaultServer{
 					Options: webhook.Options{
-						Port: configapi.DefaultWebhookPort,
+						Port:    configapi.DefaultWebhookPort,
+						CertDir: configapi.DefaultWebhookCertDir,
 					},
 				},
 			},
@@ -518,7 +520,8 @@ objectRetentionPolicies:
 				RetryPeriod:                   ptr.To(configapi.DefaultLeaderElectionRetryPeriod),
 				WebhookServer: &webhook.DefaultServer{
 					Options: webhook.Options{
-						Port: 9444,
+						Port:    9444,
+						CertDir: configapi.DefaultWebhookCertDir,
 					},
 				},
 			},
@@ -598,7 +601,8 @@ objectRetentionPolicies:
 				LeaderElection:                false,
 				WebhookServer: &webhook.DefaultServer{
 					Options: webhook.Options{
-						Port: configapi.DefaultWebhookPort,
+						Port:    configapi.DefaultWebhookPort,
+						CertDir: configapi.DefaultWebhookCertDir,
 					},
 				},
 			},
@@ -646,7 +650,8 @@ objectRetentionPolicies:
 				RetryPeriod:                   ptr.To(configapi.DefaultLeaderElectionRetryPeriod),
 				WebhookServer: &webhook.DefaultServer{
 					Options: webhook.Options{
-						Port: configapi.DefaultWebhookPort,
+						Port:    configapi.DefaultWebhookPort,
+						CertDir: configapi.DefaultWebhookCertDir,
 					},
 				},
 			},
@@ -760,7 +765,8 @@ objectRetentionPolicies:
 				RetryPeriod:                   ptr.To(configapi.DefaultLeaderElectionRetryPeriod),
 				WebhookServer: &webhook.DefaultServer{
 					Options: webhook.Options{
-						Port: configapi.DefaultWebhookPort,
+						Port:    configapi.DefaultWebhookPort,
+						CertDir: configapi.DefaultWebhookCertDir,
 					},
 				},
 			},
@@ -801,7 +807,8 @@ objectRetentionPolicies:
 				RetryPeriod:                   ptr.To(configapi.DefaultLeaderElectionRetryPeriod),
 				WebhookServer: &webhook.DefaultServer{
 					Options: webhook.Options{
-						Port: configapi.DefaultWebhookPort,
+						Port:    configapi.DefaultWebhookPort,
+						CertDir: configapi.DefaultWebhookCertDir,
 					},
 				},
 			},
@@ -861,7 +868,8 @@ objectRetentionPolicies:
 				RetryPeriod:                   ptr.To(configapi.DefaultLeaderElectionRetryPeriod),
 				WebhookServer: &webhook.DefaultServer{
 					Options: webhook.Options{
-						Port: configapi.DefaultWebhookPort,
+						Port:    configapi.DefaultWebhookPort,
+						CertDir: configapi.DefaultWebhookCertDir,
 					},
 				},
 			},
@@ -1031,7 +1039,8 @@ func TestEncode(t *testing.T) {
 				"kind":       "Configuration",
 				"namespace":  configapi.DefaultNamespace,
 				"webhook": map[string]any{
-					"port": int64(configapi.DefaultWebhookPort),
+					"port":    int64(configapi.DefaultWebhookPort),
+					"certDir": configapi.DefaultWebhookCertDir,
 				},
 				"metrics": map[string]any{
 					"bindAddress": configapi.DefaultMetricsBindAddress,