Repository: kubernetes-sigs/kueue

Issue:
Flaky E2E Test: ManageJobsWithoutQueueName when manageJobsWithoutQueueName=true should suspend the pods created by a StatefulSet in the test namespace without queue-name label
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:

`End To End Custom Configs handling Suite: kindest/node:v1.34.0: [It] ManageJobsWithoutQueueName when manageJobsWithoutQueueName=true should suspend the pods created by a StatefulSet in the test namespace without queue-name label` failed in https://github.com/kubernetes-sigs/kueue/pull/6686

```
{Timed out after 45.000s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/customconfigs/managejobswithoutqueuename_test.go:648 with:
Expected success, but got an error:
    <*errors.StatusError | 0xc0008886e0>: 
    workloads.kueue.x-k8s.io "statefulset-sts-18b6e" not found
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "workloads.kueue.x-k8s.io \"statefulset-sts-18b6e\" not found",
            Reason: "NotFound",
            Details: {
                Name: "statefulset-sts-18b6e",
                Group: "kueue.x-k8s.io",
                Kind: "workloads",
                UID: "",
                Causes: nil,
                RetryAfterSeconds: 0,
            },
            Code: 404,
        },
    } failed [FAILED] Timed out after 45.000s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/customconfigs/managejobswithoutqueuename_test.go:648 with:
Expected success, but got an error:
    <*errors.StatusError | 0xc0008886e0>: 
    workloads.kueue.x-k8s.io "statefulset-sts-18b6e" not found
    {
        ErrStatus: {
            TypeMeta: {Kind: "", APIVersion: ""},
            ListMeta: {
                SelfLink: "",
                ResourceVersion: "",
                Continue: "",
                RemainingItemCount: nil,
            },
            Status: "Failure",
            Message: "workloads.kueue.x-k8s.io \"statefulset-sts-18b6e\" not found",
            Reason: "NotFound",
            Details: {
                Name: "statefulset-sts-18b6e",
                Group: "kueue.x-k8s.io",
                Kind: "workloads",
                UID: "",
                Causes: nil,
                RetryAfterSeconds: 0,
            },
            Code: 404,
        },
    }
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/customconfigs/managejobswithoutqueuename_test.go:650 @ 08/29/25 19:02:47.399
}
```

**What you expected to happen**:

No errors.

**How to reproduce it (as minimally and precisely as possible)**:

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/6686/pull-kueue-test-e2e-customconfigs-main/1961500167513313280

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`):
- Kueue version (use `git describe --tags --dirty --always`):
- Cloud provider or hardware configuration:
- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Install tools:
- Others:

Hints from issue comments:
@tenzen-y: /kind flake
@IrvingMg: /assign@mimowo: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/6765/pull-kueue-test-e2e-customconfigs-main/1971185466099109888@tenzen-y: This happened again.

https://github.com/kubernetes-sigs/kueue/pull/7356
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/7356/pull-kueue-test-e2e-customconfigs-main/1981316427394584576

/reopen@k8s-ci-robot: @tenzen-y: Reopened this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/6699#issuecomment-3436447942):

>This happened again.
>
>https://github.com/kubernetes-sigs/kueue/pull/7356
>https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/7356/pull-kueue-test-e2e-customconfigs-main/1981316427394584576
>
>/reopen


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@mimowo: It is the same test, but different assert, and I think a different root cause
```
{Timed out after 45.000s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/customconfigs/managejobswithoutqueuename_test.go:657 with:
Expected
    <int32>: 0
to equal
    <int32>: 3 failed [FAILED] Timed out after 45.000s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/customconfigs/managejobswithoutqueuename_test.go:657 with:
Expected
    <int32>: 0
to equal
    <int32>: 3
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/customconfigs/managejobswithoutqueuename_test.go:658 @ 10/23/25 11:18:28.239
}
```@tenzen-y: > It is the same test, but different assert, and I think a different root cause
> 
> ```
> {Timed out after 45.000s.
> The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/customconfigs/managejobswithoutqueuename_test.go:657 with:
> Expected
>     <int32>: 0
> to equal
>     <int32>: 3 failed [FAILED] Timed out after 45.000s.
> The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/customconfigs/managejobswithoutqueuename_test.go:657 with:
> Expected
>     <int32>: 0
> to equal
>     <int32>: 3
> In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/customconfigs/managejobswithoutqueuename_test.go:658 @ 10/23/25 11:18:28.239
> }
> ```

Good point, let me open another issue, thanks.@tenzen-y: /close
@k8s-ci-robot: @tenzen-y: Closing this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/6699#issuecomment-3436632531):

>/close
>


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@tenzen-y: Opened https://github.com/kubernetes-sigs/kueue/issues/7366

Solution patch:
diff --git a/Makefile-test.mk b/Makefile-test.mk
index 7c89fc764..173536019 100644
--- a/Makefile-test.mk
+++ b/Makefile-test.mk
@@ -51,6 +51,9 @@ E2E_USE_HELM ?= false
 # Default will delete default kind cluster
 KIND_CLUSTER_NAME ?= kind
 
+# For restricting to a specific directory
+GO_TEST_TARGET ?= .
+
 ##@ Tests
 
 # Periodic builds are tested with full ray image
@@ -62,7 +65,7 @@ endif
 
 .PHONY: test
 test: gotestsum ## Run tests.
-	$(GOTESTSUM) --junitfile $(ARTIFACTS)/junit.xml -- $(GOFLAGS) $(GO_TEST_FLAGS) $(shell $(GO_CMD) list ./... | grep -v '/test/') -coverpkg=./... -coverprofile $(ARTIFACTS)/cover.out
+	$(GOTESTSUM) --junitfile $(ARTIFACTS)/junit.xml -- $(GOFLAGS) $(GO_TEST_FLAGS) $(shell $(GO_CMD) list $(GO_TEST_TARGET)/... | grep -v '/test/') -coverpkg=$(GO_TEST_TARGET)/... -coverprofile $(ARTIFACTS)/cover.out
 
 .PHONY: test-integration
 test-integration: gomod-download envtest ginkgo dep-crds kueuectl ginkgo-top ## Run integration tests for all singlecluster suites.



Test patch:
diff --git a/pkg/controller/tas/node_failure_controller_test.go b/pkg/controller/tas/node_failure_controller_test.go
index 66bf2f629..d9385810b 100644
--- a/pkg/controller/tas/node_failure_controller_test.go
+++ b/pkg/controller/tas/node_failure_controller_test.go
@@ -28,7 +28,6 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/component-base/featuregate"
-	"k8s.io/utils/clock"
 	testingclock "k8s.io/utils/clock/testing"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
@@ -44,14 +43,6 @@ import (
 	testingpod "sigs.k8s.io/kueue/pkg/util/testingjobs/pod"
 )
 
-// newNodeTest is a helper to create a Node object for testing
-func newNodeTest(name string, readyStatus corev1.ConditionStatus, clock clock.Clock, transitionTimeOffset time.Duration) *corev1.Node {
-	return testingnode.MakeNode(name).StatusConditions(corev1.NodeCondition{
-		Type:               corev1.NodeReady,
-		Status:             readyStatus,
-		LastTransitionTime: metav1.NewTime(clock.Now().Add(-transitionTimeOffset))}).Obj()
-}
-
 func TestNodeFailureReconciler(t *testing.T) {
 	testStartTime := time.Now().Truncate(time.Second)
 	nodeName := "test-node-name"
@@ -108,6 +99,9 @@ func TestNodeFailureReconciler(t *testing.T) {
 		Admitted(true).
 		Obj()
 
+	now := metav1.NewTime(fakeClock.Now())
+	earlierTime := metav1.NewTime(now.Add(-NodeFailureDelay))
+
 	basePod := testingpod.MakePod("test-pod", nsName).
 		Annotation(kueuealpha.WorkloadAnnotation, wlName).
 		Label(kueuealpha.TASLabel, "true").
@@ -115,7 +109,7 @@ func TestNodeFailureReconciler(t *testing.T) {
 		Obj()
 
 	terminatingPod := basePod.DeepCopy()
-	terminatingPod.DeletionTimestamp = &metav1.Time{Time: fakeClock.Now()}
+	terminatingPod.DeletionTimestamp = &now
 	terminatingPod.Finalizers = []string{podcontroller.PodFinalizer}
 
 	failedPod := basePod.DeepCopy()
@@ -123,18 +117,20 @@ func TestNodeFailureReconciler(t *testing.T) {
 	failedPod.Status.ContainerStatuses = []corev1.ContainerStatus{
 		{
 			State: corev1.ContainerState{
-				Terminated: &corev1.ContainerStateTerminated{FinishedAt: metav1.NewTime(fakeClock.Now())},
+				Terminated: &corev1.ContainerStateTerminated{FinishedAt: now},
 			},
 		},
 	}
 	failedPod.Status.ContainerStatuses = []corev1.ContainerStatus{
 		{
 			State: corev1.ContainerState{
-				Terminated: &corev1.ContainerStateTerminated{FinishedAt: metav1.NewTime(fakeClock.Now())},
+				Terminated: &corev1.ContainerStateTerminated{FinishedAt: now},
 			},
 		},
 	}
 
+	baseNode := testingnode.MakeNode(nodeName)
+
 	tests := map[string]struct {
 		initObjs          []client.Object
 		reconcileRequests []reconcile.Request
@@ -146,7 +142,10 @@ func TestNodeFailureReconciler(t *testing.T) {
 	}{
 		"Node Found and Healthy - not marked as unavailable": {
 			initObjs: []client.Object{
-				newNodeTest(nodeName, corev1.ConditionTrue, fakeClock, time.Duration(0)),
+				baseNode.Clone().StatusConditions(corev1.NodeCondition{
+					Type:               corev1.NodeReady,
+					Status:             corev1.ConditionTrue,
+					LastTransitionTime: now}).Obj(),
 				baseWorkload.DeepCopy(),
 				basePod.DeepCopy(),
 			},
@@ -155,7 +154,10 @@ func TestNodeFailureReconciler(t *testing.T) {
 		},
 		"Node becomes healthy, annotation is removed": {
 			initObjs: []client.Object{
-				newNodeTest(nodeName, corev1.ConditionTrue, fakeClock, time.Duration(0)),
+				baseNode.Clone().StatusConditions(corev1.NodeCondition{
+					Type:               corev1.NodeReady,
+					Status:             corev1.ConditionTrue,
+					LastTransitionTime: now}).Obj(),
 				workloadWithAnnotation.DeepCopy(),
 				basePod.DeepCopy(),
 			},
@@ -165,7 +167,10 @@ func TestNodeFailureReconciler(t *testing.T) {
 
 		"Node Found and Unhealthy (NotReady), delay not passed - not marked as unavailable": {
 			initObjs: []client.Object{
-				newNodeTest(nodeName, corev1.ConditionFalse, fakeClock, time.Duration(0)),
+				baseNode.Clone().StatusConditions(corev1.NodeCondition{
+					Type:               corev1.NodeReady,
+					Status:             corev1.ConditionFalse,
+					LastTransitionTime: now}).Obj(),
 				baseWorkload.DeepCopy(),
 				basePod.DeepCopy(),
 			},
@@ -175,7 +180,10 @@ func TestNodeFailureReconciler(t *testing.T) {
 		},
 		"Node Found and Unhealthy (NotReady), delay passed - marked as unavailable": {
 			initObjs: []client.Object{
-				newNodeTest(nodeName, corev1.ConditionFalse, fakeClock, NodeFailureDelay),
+				baseNode.Clone().StatusConditions(corev1.NodeCondition{
+					Type:               corev1.NodeReady,
+					Status:             corev1.ConditionFalse,
+					LastTransitionTime: earlierTime}).Obj(),
 				baseWorkload.DeepCopy(),
 				basePod.DeepCopy(),
 			},
@@ -185,7 +193,10 @@ func TestNodeFailureReconciler(t *testing.T) {
 		"Node NotReady, pod terminating, marked as unavailable": {
 			featureGates: []featuregate.Feature{features.TASReplaceNodeOnPodTermination},
 			initObjs: []client.Object{
-				newNodeTest(nodeName, corev1.ConditionFalse, fakeClock, time.Duration(0)),
+				baseNode.Clone().StatusConditions(corev1.NodeCondition{
+					Type:               corev1.NodeReady,
+					Status:             corev1.ConditionFalse,
+					LastTransitionTime: earlierTime}).Obj(),
 				baseWorkload.DeepCopy(),
 				terminatingPod,
 			},
@@ -195,7 +206,10 @@ func TestNodeFailureReconciler(t *testing.T) {
 		"Node NotReady, pod failed, marked as unavailable": {
 			featureGates: []featuregate.Feature{features.TASReplaceNodeOnPodTermination},
 			initObjs: []client.Object{
-				newNodeTest(nodeName, corev1.ConditionFalse, fakeClock, time.Duration(0)),
+				baseNode.Clone().StatusConditions(corev1.NodeCondition{
+					Type:               corev1.NodeReady,
+					Status:             corev1.ConditionFalse,
+					LastTransitionTime: now}).Obj(),
 				baseWorkload.DeepCopy(),
 				failedPod,
 			},
@@ -204,7 +218,10 @@ func TestNodeFailureReconciler(t *testing.T) {
 		},
 		"Node NotReady, pod failed, ReplaceNodeOnPodTermination feature gate off, requeued": {
 			initObjs: []client.Object{
-				newNodeTest(nodeName, corev1.ConditionFalse, fakeClock, time.Duration(0)),
+				baseNode.Clone().StatusConditions(corev1.NodeCondition{
+					Type:               corev1.NodeReady,
+					Status:             corev1.ConditionFalse,
+					LastTransitionTime: now}).Obj(),
 				baseWorkload.DeepCopy(),
 				failedPod,
 			},
@@ -215,7 +232,10 @@ func TestNodeFailureReconciler(t *testing.T) {
 		"Node NotReady, pod running, not marked as unavailable, requeued": {
 			featureGates: []featuregate.Feature{features.TASReplaceNodeOnPodTermination},
 			initObjs: []client.Object{
-				newNodeTest(nodeName, corev1.ConditionFalse, fakeClock, time.Duration(0)),
+				baseNode.Clone().StatusConditions(corev1.NodeCondition{
+					Type:               corev1.NodeReady,
+					Status:             corev1.ConditionFalse,
+					LastTransitionTime: now}).Obj(),
 				baseWorkload.DeepCopy(),
 				basePod.DeepCopy(),
 			},
@@ -234,8 +254,14 @@ func TestNodeFailureReconciler(t *testing.T) {
 		"Two Nodes Unhealthy (NotReady), delay passed - workload evicted": {
 			featureGates: []featuregate.Feature{features.TASFailedNodeReplacement},
 			initObjs: []client.Object{
-				newNodeTest(nodeName, corev1.ConditionFalse, fakeClock, NodeFailureDelay),
-				newNodeTest(nodeName2, corev1.ConditionFalse, fakeClock, NodeFailureDelay),
+				baseNode.Clone().StatusConditions(corev1.NodeCondition{
+					Type:               corev1.NodeReady,
+					Status:             corev1.ConditionFalse,
+					LastTransitionTime: earlierTime}).Obj(),
+				baseNode.Clone().Name(nodeName2).StatusConditions(corev1.NodeCondition{
+					Type:               corev1.NodeReady,
+					Status:             corev1.ConditionFalse,
+					LastTransitionTime: earlierTime}).Obj(),
 				workloadWithTwoNodes.DeepCopy(),
 				testingpod.MakePod("pod1", nsName).Annotation(kueuealpha.WorkloadAnnotation, wlName).Label(kueuealpha.TASLabel, "true").NodeName(nodeName).Obj(),
 				testingpod.MakePod("pod2", nsName).Annotation(kueuealpha.WorkloadAnnotation, wlName).Label(kueuealpha.TASLabel, "true").NodeName(nodeName2).Obj(),
diff --git a/test/e2e/multikueue/e2e_test.go b/test/e2e/multikueue/e2e_test.go
index 98a2fd667..27ab2121a 100644
--- a/test/e2e/multikueue/e2e_test.go
+++ b/test/e2e/multikueue/e2e_test.go
@@ -237,7 +237,8 @@ var _ = ginkgo.Describe("MultiKueue", func() {
 				{
 					Type:   corev1.PodInitialized,
 					Status: corev1.ConditionTrue,
-					Reason: finishReason},
+					Reason: finishReason,
+				},
 				{
 					Type:   corev1.PodReady,
 					Status: corev1.ConditionFalse,
@@ -246,7 +247,8 @@ var _ = ginkgo.Describe("MultiKueue", func() {
 				{
 					Type:   corev1.ContainersReady,
 					Status: corev1.ConditionFalse,
-					Reason: finishReason},
+					Reason: finishReason,
+				},
 				{
 					Type:   corev1.PodScheduled,
 					Status: corev1.ConditionTrue,
@@ -258,7 +260,7 @@ var _ = ginkgo.Describe("MultiKueue", func() {
 					createdPod := corev1.Pod{}
 					g.Expect(k8sManagerClient.Get(ctx, client.ObjectKeyFromObject(pod), &createdPod)).To(gomega.Succeed())
 					g.Expect(createdPod.Status.Phase).To(gomega.Equal(corev1.PodSucceeded))
-					g.Expect(createdPod.Status.Conditions).To(gomega.BeComparableTo(finishPodConditions, cmpopts.IgnoreFields(corev1.PodCondition{}, "LastTransitionTime")))
+					g.Expect(createdPod.Status.Conditions).To(gomega.BeComparableTo(finishPodConditions, util.IgnorePodConditionTimestampsMessageAndObservedGeneration))
 				}, util.LongTimeout, util.Interval).Should(gomega.Succeed())
 			})
 		})
diff --git a/test/e2e/singlecluster/pod_test.go b/test/e2e/singlecluster/pod_test.go
index ddb43b866..6dc957cf8 100644
--- a/test/e2e/singlecluster/pod_test.go
+++ b/test/e2e/singlecluster/pod_test.go
@@ -19,7 +19,6 @@ package e2e
 import (
 	"fmt"
 
-	"github.com/google/go-cmp/cmp/cmpopts"
 	"github.com/onsi/ginkgo/v2"
 	"github.com/onsi/gomega"
 	corev1 "k8s.io/api/core/v1"
@@ -333,7 +332,7 @@ var _ = ginkgo.Describe("Pod groups", func() {
 						Type:   corev1.PodScheduled,
 						Status: corev1.ConditionFalse,
 						Reason: corev1.PodReasonUnschedulable,
-					}, cmpopts.IgnoreFields(corev1.PodCondition{}, "LastProbeTime", "LastTransitionTime", "Message"))))
+					}, util.IgnorePodConditionTimestampsMessageAndObservedGeneration)))
 				}, util.Timeout, util.Interval).Should(gomega.Succeed())
 			})
 
diff --git a/test/util/constants.go b/test/util/constants.go
index 4068d889c..32d9d6249 100644
--- a/test/util/constants.go
+++ b/test/util/constants.go
@@ -22,6 +22,7 @@ import (
 
 	"github.com/google/go-cmp/cmp/cmpopts"
 	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
@@ -47,11 +48,12 @@ const (
 )
 
 var (
-	IgnoreConditionTimestamps                      = cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime")
-	IgnoreConditionTimestampsAndObservedGeneration = cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "ObservedGeneration")
-	IgnoreConditionMessage                         = cmpopts.IgnoreFields(metav1.Condition{}, "Message")
-	IgnoreObjectMetaResourceVersion                = cmpopts.IgnoreFields(metav1.ObjectMeta{}, "ResourceVersion")
-	IgnoreDeploymentConditionTimestampsAndMessage  = cmpopts.IgnoreFields(appsv1.DeploymentCondition{}, "LastTransitionTime", "LastUpdateTime", "Message")
+	IgnoreConditionTimestamps                                = cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime")
+	IgnoreConditionTimestampsAndObservedGeneration           = cmpopts.IgnoreFields(metav1.Condition{}, "LastTransitionTime", "ObservedGeneration")
+	IgnoreConditionMessage                                   = cmpopts.IgnoreFields(metav1.Condition{}, "Message")
+	IgnoreObjectMetaResourceVersion                          = cmpopts.IgnoreFields(metav1.ObjectMeta{}, "ResourceVersion")
+	IgnoreDeploymentConditionTimestampsAndMessage            = cmpopts.IgnoreFields(appsv1.DeploymentCondition{}, "LastTransitionTime", "LastUpdateTime", "Message")
+	IgnorePodConditionTimestampsMessageAndObservedGeneration = cmpopts.IgnoreFields(corev1.PodCondition{}, "LastProbeTime", "LastTransitionTime", "Message", "ObservedGeneration")
 )
 
 var (