Repository: kubernetes-sigs/kueue

Issue:
[flaky test] PyTorch integration when PyTorch created should admit group with leader only
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/7325/pull-kueue-test-e2e-main-1-31/1980248809606746112

**What you expected to happen**:
no failure
**How to reproduce it (as minimally and precisely as possible)**:
ci
**Anything else we need to know?**:

```
End To End Suite: kindest/node:v1.31.12: [It] PyTorch integration when PyTorch created should admit group with leader only expand_less	1m29s
{Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/pytorchjob_test.go:115 with:
Unexpected number of active Worker replicas
Expected
    <int32>: 1
to equal
    <int32>: 2 failed [FAILED] Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/pytorchjob_test.go:115 with:
Unexpected number of active Worker replicas
Expected
    <int32>: 1
to equal
    <int32>: 2
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/pytorchjob_test.go:124 @ 10/20/25 12:36:47.088
}
```

Discussion:
@mimowo: /kind flake@mimowo: Ok I analyzed logs based on  this build: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/7341/pull-kueue-test-e2e-main-1-31/1981043923610505216

take a look here: https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/7341/pull-kueue-test-e2e-main-1-31/1981043923610505216/artifacts/run-test-e2e-singlecluster-1.31.12/kind-worker/kubelet.log

We have:

```
\"alpine:3.10\""
Oct 22 17:17:42 kind-worker kubelet[277]: I1022 17:17:42.011896     277 provider.go:82] Docker config file not found: couldn't find valid .dockercfg after checking in [/var/lib/kubelet   /]
Oct 22 17:17:42 kind-worker kubelet[277]: I1022 17:17:42.011925     277 kuberuntime_image.go:51] "Pulling image without credentials" image="alpine:3.10"
Oct 22 17:17:42 kind-worker kubelet[277]: I1022 17:17:42.021229     277 event.go:389] "Event occurred" object="pytorch-e2e-fcjhp/pytorch-simple-worker-1" fieldPath="spec.initContainers{init-pytorch}" kind="Pod" apiVersion="v1" type="Normal" reason="Pulling" message="Pulling image \"alpine:3.10\""
```
So this looks like pulling external image alpine:3.10 from unit container.

And I found that PyTorch is adding that to all Jobs with workers, see links:
- https://github.com/kubeflow/trainer/blob/v1.9.3/pkg/controller.v1/pytorch/initcontainer.go#L57 
- https://github.com/kubeflow/trainer/blob/v1.9.3/pkg/config/config.go#L28C7-L28C50 

@mimowo: To solve the issue I consider:
1. don't create any Workers in the e2e tests
2. pre-pull the image
3. increase timeout

TBH I'm leaning to 1., because the mechanism from Kueue PoV looks the same. @mimowo: I could see this issue affecting a couple branches, seems like the most common flake currently.
Another example: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/7226/pull-kueue-test-e2e-main-1-31/1981042931980570624
@mimowo: Also likely related https://github.com/kubernetes-sigs/kueue/issues/7382

Solution patch:
diff --git a/go.mod b/go.mod
index 41a31750d..9b13cd842 100644
--- a/go.mod
+++ b/go.mod
@@ -3,7 +3,7 @@ module sigs.k8s.io/kueue
 go 1.25.0
 
 require (
-	github.com/cert-manager/cert-manager v1.19.0
+	github.com/cert-manager/cert-manager v1.19.1
 	github.com/fsnotify/fsnotify v1.9.0
 	github.com/go-logr/logr v1.4.3
 	github.com/google/go-cmp v0.7.0
diff --git a/go.sum b/go.sum
index a75b9686a..a3bd6af76 100644
--- a/go.sum
+++ b/go.sum
@@ -18,8 +18,8 @@ github.com/blang/semver/v4 v4.0.0 h1:1PFHFE6yCCTv8C1TeyNNarDzntLi7wMI5i/pzqYIsAM
 github.com/blang/semver/v4 v4.0.0/go.mod h1:IbckMUScFkM3pff0VJDNKRiT6TG/YpiHIM2yvyW5YoQ=
 github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
 github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
-github.com/cert-manager/cert-manager v1.19.0 h1:pPKtK06zPriliwq7B12xa7DGKbc4SvnfOjE0BXaAspk=
-github.com/cert-manager/cert-manager v1.19.0/go.mod h1:7VBp/ihp5Xgz/ZJuGy1fNIWbDouQKBonvEyWoJLuhmA=
+github.com/cert-manager/cert-manager v1.19.1 h1:Txh8L/nLWTDcb7ZnXuXbTe15BxQnLbLirXmbNk0fGgY=
+github.com/cert-manager/cert-manager v1.19.1/go.mod h1:8Ps1VXCQRGKT8zNvLQlhDK1gFKWmYKdIPQFmvTS2JeA=
 github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
 github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/chai2010/gettext-go v1.0.2 h1:1Lwwip6Q2QGsAdl/ZKPCwTe9fe0CjlUbqj5bFNSjIRk=
diff --git a/vendor/github.com/cert-manager/cert-manager/pkg/apis/meta/v1/types.go b/vendor/github.com/cert-manager/cert-manager/pkg/apis/meta/v1/types.go
index 6c36d40ec..2b294e1a9 100644
--- a/vendor/github.com/cert-manager/cert-manager/pkg/apis/meta/v1/types.go
+++ b/vendor/github.com/cert-manager/cert-manager/pkg/apis/meta/v1/types.go
@@ -53,12 +53,12 @@ type IssuerReference struct {
 	// Name of the issuer being referred to.
 	Name string `json:"name"`
 	// Kind of the issuer being referred to.
+	// Defaults to 'Issuer'.
 	// +optional
-	// +default="Issuer"
 	Kind string `json:"kind,omitempty"`
 	// Group of the issuer being referred to.
+	// Defaults to 'cert-manager.io'.
 	// +optional
-	// +default="cert-manager.io"
 	Group string `json:"group,omitempty"`
 }
 
diff --git a/vendor/modules.txt b/vendor/modules.txt
index 6b58d8331..3b555377a 100644
--- a/vendor/modules.txt
+++ b/vendor/modules.txt
@@ -26,7 +26,7 @@ github.com/blang/semver/v4
 # github.com/cenkalti/backoff/v5 v5.0.3
 ## explicit; go 1.23
 github.com/cenkalti/backoff/v5
-# github.com/cert-manager/cert-manager v1.19.0
+# github.com/cert-manager/cert-manager v1.19.1
 ## explicit; go 1.25.0
 github.com/cert-manager/cert-manager/pkg/apis/acme
 github.com/cert-manager/cert-manager/pkg/apis/acme/v1



Test patch:
diff --git a/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go b/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
index 2ccb030ba..4cef95808 100644
--- a/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
+++ b/test/integration/singlecluster/scheduler/fairsharing/fair_sharing_test.go
@@ -918,12 +918,13 @@ var _ = ginkgo.Describe("Scheduler", ginkgo.Ordered, ginkgo.ContinueOnFailure, f
 			ginkgo.By("Validate pending workloads")
 			util.ExpectPendingWorkloadsMetric(cq1, 1, 1)
 
-			gomega.Eventually(func(g gomega.Gomega) {
-				util.FinishEvictionsOfAnyWorkloadsInCq(ctx, k8sClient, cq2)
-				util.ExpectAdmittedWorkloadsTotalMetric(cq1, "", 1)
-				util.ExpectClusterQueueWeightedShareMetric(cq1, 0)
-				util.ExpectClusterQueueWeightedShareMetric(cq2, 0)
-			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			ginkgo.By("Complete preemption")
+			util.FinishEvictionOfWorkloadsInCQ(ctx, k8sClient, cq2, 2)
+
+			ginkgo.By("Expected Total Admitted Workloads and Weighted Share")
+			util.ExpectAdmittedWorkloadsTotalMetric(cq1, "", 1)
+			util.ExpectClusterQueueWeightedShareMetric(cq1, 0)
+			util.ExpectClusterQueueWeightedShareMetric(cq2, 0)
 		})
 
 		ginkgo.It("sticky workload deleted, next workload can admit", func() {
@@ -950,13 +951,13 @@ var _ = ginkgo.Describe("Scheduler", ginkgo.Ordered, ginkgo.ContinueOnFailure, f
 			ginkgo.By("Validate pending workloads")
 			util.ExpectPendingWorkloadsMetric(cq1, 1, 1)
 
+			ginkgo.By("Complete preemption")
+			util.FinishEvictionOfWorkloadsInCQ(ctx, k8sClient, cq2, 2)
+
 			ginkgo.By("Expected Total Admitted Workloads and Weighted Share")
-			gomega.Eventually(func(g gomega.Gomega) {
-				util.FinishEvictionsOfAnyWorkloadsInCq(ctx, k8sClient, cq2)
-				util.ExpectAdmittedWorkloadsTotalMetric(cq1, "", 1)
-				util.ExpectClusterQueueWeightedShareMetric(cq1, 0)
-				util.ExpectClusterQueueWeightedShareMetric(cq2, 0)
-			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+			util.ExpectAdmittedWorkloadsTotalMetric(cq1, "", 1)
+			util.ExpectClusterQueueWeightedShareMetric(cq1, 0)
+			util.ExpectClusterQueueWeightedShareMetric(cq2, 0)
 		})
 	})