Repository: kubernetes-sigs/kueue

Issue:
Flaky Test: LeaderWorkerSet integration when LeaderWorkerSet created should allow to scale down LeaderReadyStartupPolicy
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:

`End To End Suite: kindest/node:v1.32.0: [It] LeaderWorkerSet integration when LeaderWorkerSet created should allow to scale down LeaderReadyStartupPolicy` accidentally failed in CI

```shell
{Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/leaderworkerset_test.go:409 with:
Expected
    <int32>: 2
to equal
    <int32>: 1 failed [FAILED] Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/leaderworkerset_test.go:409 with:
Expected
    <int32>: 2
to equal
    <int32>: 1
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/leaderworkerset_test.go:418 @ 03/16/25 04:32:08.556

There were additional failures detected after the initial failure. These are visible in the timeline
}
```

**What you expected to happen**:

no errors

**How to reproduce it (as minimally and precisely as possible)**:

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4625/pull-kueue-test-e2e-main-1-32/1901125782999142400

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`):
- Kueue version (use `git describe --tags --dirty --always`):
- Cloud provider or hardware configuration:
- OS (e.g: `cat /etc/os-release`):
- Kernel (e.g. `uname -a`):
- Install tools:
- Others:

Hints from issue comments:
@tenzen-y: /kind flake@mimowo: I looked at the failure- the scale down test failed because the number of replicas didn't go down from 2 to 1 within 45s.

The LWS in question is in the lws-e2e-thznm namespace.

The kubelet logs:
```
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.080604     202 kubelet.go:2474] "SyncLoop ADD" source="api" pods=["lws-e2e-thznm/lws-1"]
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.081394     202 kubelet_pods.go:1833] "Generating pod status" podIsTerminal=false pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.082083     202 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.105677     202 volume_manager.go:404] "Waiting for volumes to attach and mount for pod" pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.135020     202 status_manager.go:911] "Patch status for pod" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096" patch="{\"metadata\":{\"uid\":\"4ba4f666-f410-4d32-a8c5-f961e3332096\"},\"status\":{\"$setElementOrder/conditions\":[{\"type\":\"PodReadyToStartContainers\"},{\"type\":\"Initialized\"},{\"type\":\"Ready\"},{\"type\":\"ContainersReady\"},{\"type\":\"PodScheduled\"}],\"conditions\":[{\"lastProbeTime\":null,\"lastTransitionTime\":\"2025-03-16T04:31:21Z\",\"status\":\"False\",\"type\":\"PodReadyToStartContainers\"},{\"lastProbeTime\":null,\"lastTransitionTime\":\"2025-03-16T04:31:21Z\",\"status\":\"True\",\"type\":\"Initialized\"},{\"lastProbeTime\":null,\"lastTransitionTime\":\"2025-03-16T04:31:21Z\",\"message\":\"containers with unready status: [c]\",\"reason\":\"ContainersNotReady\",\"status\":\"False\",\"type\":\"Ready\"},{\"lastProbeTime\":null,\"lastTransitionTime\":\"2025-03-16T04:31:21Z\",\"message\":\"containers with unready status: [c]\",\"reason\":\"ContainersNotReady\",\"status\":\"False\",\"type\":\"ContainersReady\"}],\"containerStatuses\":[{\"image\":\"registry.k8s.io/e2e-test-images/agnhost:2.53@sha256:99c6b4bb4a1e1df3f0b3752168c89358794d02258ebebc26bf21c29399011a85\",\"imageID\":\"\",\"lastState\":{},\"name\":\"c\",\"ready\":false,\"restartCount\":0,\"started\":false,\"state\":{\"waiting\":{\"reason\":\"ContainerCreating\"}},\"volumeMounts\":[{\"mountPath\":\"/var/run/secrets/kubernetes.io/serviceaccount\",\"name\":\"kube-api-access-znkgb\",\"readOnly\":true,\"recursiveReadOnly\":\"Disabled\"}]}],\"hostIP\":\"172.18.0.4\",\"hostIPs\":[{\"ip\":\"172.18.0.4\"}],\"startTime\":\"2025-03-16T04:31:21Z\"}}"
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.135442     202 status_manager.go:920] "Status for pod updated successfully" pod="lws-e2e-thznm/lws-1" statusVersion=1 status={"phase":"Pending","conditions":[{"type":"PodReadyToStartContainers","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:21Z"},{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:21Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:21Z","reason":"ContainersNotReady","message":"containers with unready status: [c]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:21Z","reason":"ContainersNotReady","message":"containers with unready status: [c]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:21Z"}],"hostIP":"172.18.0.4","hostIPs":[{"ip":"172.18.0.4"}],"startTime":"2025-03-16T04:31:21Z","containerStatuses":[{"name":"c","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"registry.k8s.io/e2e-test-images/agnhost:2.53@sha256:99c6b4bb4a1e1df3f0b3752168c89358794d02258ebebc26bf21c29399011a85","imageID":"","started":false,"volumeMounts":[{"name":"kube-api-access-znkgb","mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","readOnly":true,"recursiveReadOnly":"Disabled"}]}],"qosClass":"Burstable"}
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.226531     202 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znkgb\" (UniqueName: \"kubernetes.io/projected/4ba4f666-f410-4d32-a8c5-f961e3332096-kube-api-access-znkgb\") pod \"lws-1\" (UID: \"4ba4f666-f410-4d32-a8c5-f961e3332096\") " pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.328148     202 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-znkgb\" (UniqueName: \"kubernetes.io/projected/4ba4f666-f410-4d32-a8c5-f961e3332096-kube-api-access-znkgb\") pod \"lws-1\" (UID: \"4ba4f666-f410-4d32-a8c5-f961e3332096\") " pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.374447     202 operation_generator.go:614] "MountVolume.SetUp succeeded for volume \"kube-api-access-znkgb\" (UniqueName: \"kubernetes.io/projected/4ba4f666-f410-4d32-a8c5-f961e3332096-kube-api-access-znkgb\") pod \"lws-1\" (UID: \"4ba4f666-f410-4d32-a8c5-f961e3332096\") " pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.406515     202 volume_manager.go:440] "All volumes are attached and mounted for pod" pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.406865     202 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.407005     202 kuberuntime_manager.go:1122] "computePodActions got for pod" podActions="KillPod: true, CreateSandbox: true, UpdatePodResources: false, Attempt: 0, InitContainersToStart: [], ContainersToStart: [0], EphemeralContainersToStart: [],ContainersToUpdate: map[], ContainersToKill: map[]" pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.907013     202 kubelet_pods.go:276] "Creating hosts mount for container" pod="lws-e2e-thznm/lws-1" containerName="c" podIPs=["10.244.1.46"] path=true
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.907975     202 event.go:389] "Event occurred" object="lws-e2e-thznm/lws-1" fieldPath="spec.containers{c}" kind="Pod" apiVersion="v1" type="Normal" reason="Pulled" message="Container image \"registry.k8s.io/e2e-test-images/agnhost:2.53@sha256:99c6b4bb4a1e1df3f0b3752168c89358794d02258ebebc26bf21c29399011a85\" already present on machine"
Mar 16 04:31:21 kind-worker kubelet[202]: I0316 04:31:21.967839     202 event.go:389] "Event occurred" object="lws-e2e-thznm/lws-1" fieldPath="spec.containers{c}" kind="Pod" apiVersion="v1" type="Normal" reason="Created" message="Created container: c"
Mar 16 04:31:22 kind-worker kubelet[202]: I0316 04:31:22.318990     202 event.go:389] "Event occurred" object="lws-e2e-thznm/lws-1" fieldPath="spec.containers{c}" kind="Pod" apiVersion="v1" type="Normal" reason="Started" message="Started container c"
Mar 16 04:31:22 kind-worker kubelet[202]: I0316 04:31:22.661747     202 kubelet.go:2506] "SyncLoop (PLEG): event for pod" pod="lws-e2e-thznm/lws-1" event={"ID":"4ba4f666-f410-4d32-a8c5-f961e3332096","Type":"ContainerStarted","Data":"0782dd0d297258a8f747621ef2b2e349c72bbede25c0d2740eb3eea125d0dbd7"}
Mar 16 04:31:22 kind-worker kubelet[202]: I0316 04:31:22.661920     202 kubelet.go:2506] "SyncLoop (PLEG): event for pod" pod="lws-e2e-thznm/lws-1" event={"ID":"4ba4f666-f410-4d32-a8c5-f961e3332096","Type":"ContainerStarted","Data":"f87c41877224033289de84a7c29c2d9d66f71ef7a88ec595780f882775ebc1d4"}
Mar 16 04:31:22 kind-worker kubelet[202]: I0316 04:31:22.662098     202 kubelet_pods.go:1833] "Generating pod status" podIsTerminal=false pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:22 kind-worker kubelet[202]: I0316 04:31:22.662657     202 volume_manager.go:404] "Waiting for volumes to attach and mount for pod" pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:22 kind-worker kubelet[202]: I0316 04:31:22.662871     202 volume_manager.go:440] "All volumes are attached and mounted for pod" pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:22 kind-worker kubelet[202]: I0316 04:31:22.663065     202 kuberuntime_manager.go:1122] "computePodActions got for pod" podActions="KillPod: false, CreateSandbox: false, UpdatePodResources: false, Attempt: 0, InitContainersToStart: [], ContainersToStart: [], EphemeralContainersToStart: [],ContainersToUpdate: map[], ContainersToKill: map[]" pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:22 kind-worker kubelet[202]: I0316 04:31:22.692124     202 status_manager.go:911] "Patch status for pod" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096" patch="{\"metadata\":{\"uid\":\"4ba4f666-f410-4d32-a8c5-f961e3332096\"},\"status\":{\"$setElementOrder/conditions\":[{\"type\":\"PodReadyToStartContainers\"},{\"type\":\"Initialized\"},{\"type\":\"Ready\"},{\"type\":\"ContainersReady\"},{\"type\":\"PodScheduled\"}],\"conditions\":[{\"lastTransitionTime\":\"2025-03-16T04:31:22Z\",\"status\":\"True\",\"type\":\"PodReadyToStartContainers\"},{\"lastTransitionTime\":\"2025-03-16T04:31:22Z\",\"message\":null,\"reason\":null,\"status\":\"True\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2025-03-16T04:31:22Z\",\"message\":null,\"reason\":null,\"status\":\"True\",\"type\":\"ContainersReady\"}],\"containerStatuses\":[{\"containerID\":\"containerd://0782dd0d297258a8f747621ef2b2e349c72bbede25c0d2740eb3eea125d0dbd7\",\"image\":\"registry.k8s.io/e2e-test-images/agnhost:2.53\",\"imageID\":\"docker.io/library/import-2025-03-16@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e\",\"lastState\":{},\"name\":\"c\",\"ready\":true,\"restartCount\":0,\"started\":true,\"state\":{\"running\":{\"startedAt\":\"2025-03-16T04:31:22Z\"}},\"volumeMounts\":[{\"mountPath\":\"/var/run/secrets/kubernetes.io/serviceaccount\",\"name\":\"kube-api-access-znkgb\",\"readOnly\":true,\"recursiveReadOnly\":\"Disabled\"}]}],\"phase\":\"Running\",\"podIP\":\"10.244.1.46\",\"podIPs\":[{\"ip\":\"10.244.1.46\"}]}}"
Mar 16 04:31:22 kind-worker kubelet[202]: I0316 04:31:22.692563     202 status_manager.go:920] "Status for pod updated successfully" pod="lws-e2e-thznm/lws-1" statusVersion=2 status={"phase":"Running","conditions":[{"type":"PodReadyToStartContainers","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:22Z"},{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:21Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:22Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:22Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:21Z"}],"hostIP":"172.18.0.4","hostIPs":[{"ip":"172.18.0.4"}],"podIP":"10.244.1.46","podIPs":[{"ip":"10.244.1.46"}],"startTime":"2025-03-16T04:31:21Z","containerStatuses":[{"name":"c","state":{"running":{"startedAt":"2025-03-16T04:31:22Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"registry.k8s.io/e2e-test-images/agnhost:2.53","imageID":"docker.io/library/import-2025-03-16@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e","containerID":"containerd://0782dd0d297258a8f747621ef2b2e349c72bbede25c0d2740eb3eea125d0dbd7","started":true,"volumeMounts":[{"name":"kube-api-access-znkgb","mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","readOnly":true,"recursiveReadOnly":"Disabled"}]}],"qosClass":"Burstable"}
Mar 16 04:31:22 kind-worker kubelet[202]: I0316 04:31:22.692818     202 pod_startup_latency_tracker.go:172] "Mark when the pod was running for the first time" pod="lws-e2e-thznm/lws-1" rv="3910"
Mar 16 04:31:22 kind-worker kubelet[202]: I0316 04:31:22.755080     202 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="lws-e2e-thznm/lws-1" podStartSLOduration=2.755046846 podStartE2EDuration="2.755046846s" podCreationTimestamp="2025-03-16 04:31:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2025-03-16 04:31:22.692926465 +0000 UTC m=+270.504844009" watchObservedRunningTime="2025-03-16 04:31:22.755046846 +0000 UTC m=+270.566964380"
Mar 16 04:31:23 kind-worker kubelet[202]: I0316 04:31:23.611176     202 kubelet.go:2490] "SyncLoop DELETE" source="api" pods=["lws-e2e-thznm/lws-1"]
Mar 16 04:31:23 kind-worker kubelet[202]: I0316 04:31:23.611610     202 pod_workers.go:970] "Cancelling current pod sync" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096" workType="terminating"
Mar 16 04:31:23 kind-worker kubelet[202]: I0316 04:31:23.671276     202 kubelet_pods.go:1833] "Generating pod status" podIsTerminal=false pod="lws-e2e-thznm/lws-1"
Mar 16 04:31:23 kind-worker kubelet[202]: I0316 04:31:23.671860     202 kuberuntime_container.go:788] "Killing container with a grace period override" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096" containerName="c" containerID="containerd://0782dd0d297258a8f747621ef2b2e349c72bbede25c0d2740eb3eea125d0dbd7" gracePeriod=30
Mar 16 04:31:23 kind-worker kubelet[202]: I0316 04:31:23.671891     202 kuberuntime_container.go:809] "Killing container with a grace period" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096" containerName="c" containerID="containerd://0782dd0d297258a8f747621ef2b2e349c72bbede25c0d2740eb3eea125d0dbd7" gracePeriod=30
Mar 16 04:31:23 kind-worker kubelet[202]: I0316 04:31:23.677289     202 event.go:389] "Event occurred" object="lws-e2e-thznm/lws-1" fieldPath="spec.containers{c}" kind="Pod" apiVersion="v1" type="Normal" reason="Killing" message="Stopping container c"
Mar 16 04:31:23 kind-worker kubelet[202]: I0316 04:31:23.731009     202 status_manager.go:911] "Patch status for pod" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096" patch="{\"metadata\":{\"uid\":\"4ba4f666-f410-4d32-a8c5-f961e3332096\"}}"
Mar 16 04:31:23 kind-worker kubelet[202]: I0316 04:31:23.731385     202 status_manager.go:918] "Status for pod is up-to-date" pod="lws-e2e-thznm/lws-1" statusVersion=3
Mar 16 04:31:23 kind-worker kubelet[202]: I0316 04:31:23.731422     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:31:32 kind-worker kubelet[202]: I0316 04:31:32.658309     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:31:42 kind-worker kubelet[202]: I0316 04:31:42.648802     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:31:52 kind-worker kubelet[202]: I0316 04:31:52.648242     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:32:02 kind-worker kubelet[202]: I0316 04:32:02.649381     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:32:03 kind-worker kubelet[202]: I0316 04:32:03.751653     202 kuberuntime_container.go:818] "Container exited normally" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096" containerName="c" containerID="containerd://0782dd0d297258a8f747621ef2b2e349c72bbede25c0d2740eb3eea125d0dbd7"
Mar 16 04:32:03 kind-worker kubelet[202]: I0316 04:32:03.920395     202 kubelet.go:2506] "SyncLoop (PLEG): event for pod" pod="lws-e2e-thznm/lws-1" event={"ID":"4ba4f666-f410-4d32-a8c5-f961e3332096","Type":"ContainerDied","Data":"0782dd0d297258a8f747621ef2b2e349c72bbede25c0d2740eb3eea125d0dbd7"}
Mar 16 04:32:12 kind-worker kubelet[202]: I0316 04:32:12.648865     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:32:22 kind-worker kubelet[202]: I0316 04:32:22.648565     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:32:32 kind-worker kubelet[202]: I0316 04:32:32.648532     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:32:42 kind-worker kubelet[202]: I0316 04:32:42.649315     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:32:52 kind-worker kubelet[202]: I0316 04:32:52.649298     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:33:02 kind-worker kubelet[202]: I0316 04:33:02.649017     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:33:12 kind-worker kubelet[202]: I0316 04:33:12.648718     202 status_manager.go:976] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:33:14 kind-worker kubelet[202]: I0316 04:33:14.647087     202 kubelet_pods.go:1833] "Generating pod status" podIsTerminal=true pod="lws-e2e-thznm/lws-1"
Mar 16 04:33:14 kind-worker kubelet[202]: I0316 04:33:14.647169     202 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="lws-e2e-thznm/lws-1"
Mar 16 04:33:14 kind-worker kubelet[202]: I0316 04:33:14.736045     202 status_manager.go:911] "Patch status for pod" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096" patch="{\"metadata\":{\"uid\":\"4ba4f666-f410-4d32-a8c5-f961e3332096\"},\"status\":{\"$setElementOrder/conditions\":[{\"type\":\"PodReadyToStartContainers\"},{\"type\":\"Initialized\"},{\"type\":\"Ready\"},{\"type\":\"ContainersReady\"},{\"type\":\"PodScheduled\"}],\"conditions\":[{\"lastTransitionTime\":\"2025-03-16T04:33:14Z\",\"status\":\"False\",\"type\":\"PodReadyToStartContainers\"},{\"reason\":\"PodCompleted\",\"type\":\"Initialized\"},{\"lastTransitionTime\":\"2025-03-16T04:33:14Z\",\"reason\":\"PodCompleted\",\"status\":\"False\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2025-03-16T04:33:14Z\",\"reason\":\"PodCompleted\",\"status\":\"False\",\"type\":\"ContainersReady\"}],\"containerStatuses\":[{\"containerID\":\"containerd://0782dd0d297258a8f747621ef2b2e349c72bbede25c0d2740eb3eea125d0dbd7\",\"image\":\"registry.k8s.io/e2e-test-images/agnhost:2.53\",\"imageID\":\"docker.io/library/import-2025-03-16@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e\",\"lastState\":{},\"name\":\"c\",\"ready\":false,\"restartCount\":0,\"started\":false,\"state\":{\"terminated\":{\"containerID\":\"containerd://0782dd0d297258a8f747621ef2b2e349c72bbede25c0d2740eb3eea125d0dbd7\",\"exitCode\":0,\"finishedAt\":\"2025-03-16T04:31:23Z\",\"reason\":\"Completed\",\"startedAt\":\"2025-03-16T04:31:22Z\"}},\"volumeMounts\":[{\"mountPath\":\"/var/run/secrets/kubernetes.io/serviceaccount\",\"name\":\"kube-api-access-znkgb\",\"readOnly\":true,\"recursiveReadOnly\":\"Disabled\"}]}],\"phase\":\"Succeeded\"}}"
Mar 16 04:33:14 kind-worker kubelet[202]: I0316 04:33:14.736128     202 status_manager.go:920] "Status for pod updated successfully" pod="lws-e2e-thznm/lws-1" statusVersion=4 status={"phase":"Succeeded","conditions":[{"type":"PodReadyToStartContainers","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:33:14Z"},{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:21Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:33:14Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:33:14Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-16T04:31:21Z"}],"hostIP":"172.18.0.4","hostIPs":[{"ip":"172.18.0.4"}],"podIP":"10.244.1.46","podIPs":[{"ip":"10.244.1.46"}],"startTime":"2025-03-16T04:31:21Z","containerStatuses":[{"name":"c","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2025-03-16T04:31:22Z","finishedAt":"2025-03-16T04:31:23Z","containerID":"containerd://0782dd0d297258a8f747621ef2b2e349c72bbede25c0d2740eb3eea125d0dbd7"}},"lastState":{},"ready":false,"restartCount":0,"image":"registry.k8s.io/e2e-test-images/agnhost:2.53","imageID":"docker.io/library/import-2025-03-16@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e","containerID":"containerd://0782dd0d297258a8f747621ef2b2e349c72bbede25c0d2740eb3eea125d0dbd7","started":false,"volumeMounts":[{"name":"kube-api-access-znkgb","mountPath":"/var/run/secrets/kubernetes.io/serviceaccount","readOnly":true,"recursiveReadOnly":"Disabled"}]}],"qosClass":"Burstable"}
Mar 16 04:33:15 kind-worker kubelet[202]: I0316 04:33:15.148257     202 kubelet.go:2506] "SyncLoop (PLEG): event for pod" pod="lws-e2e-thznm/lws-1" event={"ID":"4ba4f666-f410-4d32-a8c5-f961e3332096","Type":"ContainerDied","Data":"f87c41877224033289de84a7c29c2d9d66f71ef7a88ec595780f882775ebc1d4"}
Mar 16 04:33:15 kind-worker kubelet[202]: I0316 04:33:15.148290     202 kubelet_pods.go:1833] "Generating pod status" podIsTerminal=true pod="lws-e2e-thznm/lws-1"
Mar 16 04:33:15 kind-worker kubelet[202]: I0316 04:33:15.148478     202 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="lws-e2e-thznm/lws-1"
Mar 16 04:33:15 kind-worker kubelet[202]: I0316 04:33:15.148624     202 volume_manager.go:449] "Waiting for volumes to unmount for pod" pod="lws-e2e-thznm/lws-1"
Mar 16 04:33:15 kind-worker kubelet[202]: I0316 04:33:15.152425     202 status_manager.go:911] "Patch status for pod" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096" patch="{\"metadata\":{\"uid\":\"4ba4f666-f410-4d32-a8c5-f961e3332096\"}}"
Mar 16 04:33:15 kind-worker kubelet[202]: I0316 04:33:15.152479     202 status_manager.go:918] "Status for pod is up-to-date" pod="lws-e2e-thznm/lws-1" statusVersion=5
Mar 16 04:33:34 kind-worker kubelet[202]: I0316 04:33:34.049036     202 volume_manager.go:477] "All volumes are unmounted for pod" pod="lws-e2e-thznm/lws-1"
Mar 16 04:33:34 kind-worker kubelet[202]: I0316 04:33:34.049251     202 kubelet.go:2238] "Pod termination cleaned up volume paths" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:33:34 kind-worker kubelet[202]: I0316 04:33:34.094611     202 status_manager.go:911] "Patch status for pod" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096" patch="{\"metadata\":{\"uid\":\"4ba4f666-f410-4d32-a8c5-f961e3332096\"}}"
Mar 16 04:33:34 kind-worker kubelet[202]: I0316 04:33:34.094657     202 status_manager.go:918] "Status for pod is up-to-date" pod="lws-e2e-thznm/lws-1" statusVersion=6
Mar 16 04:33:34 kind-worker kubelet[202]: I0316 04:33:34.094680     202 status_manager.go:981] "The pod termination is finished as SyncTerminatedPod completes its execution" phase="Succeeded" localPhase="Succeeded" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:33:34 kind-worker kubelet[202]: I0316 04:33:34.126054     202 kubelet.go:2490] "SyncLoop DELETE" source="api" pods=["lws-e2e-thznm/lws-1"]
Mar 16 04:33:34 kind-worker kubelet[202]: I0316 04:33:34.142740     202 kubelet.go:2484] "SyncLoop REMOVE" source="api" pods=["lws-e2e-thznm/lws-1"]
Mar 16 04:33:34 kind-worker kubelet[202]: I0316 04:33:34.143928     202 kubelet.go:2322] "Pod has been deleted and must be killed" pod="lws-e2e-thznm/lws-1" podUID="4ba4f666-f410-4d32-a8c5-f961e3332096"
Mar 16 04:33:34 kind-worker kubelet[202]: I0316 04:33:34.143267     202 status_manager.go:949] "Pod fully terminated and removed from etcd" pod="lws-e2e-thznm/lws-1"
```
Key timeline: 
```
04:31:20.657 - LWS created / test starts
04:31:23.611 - scale down starts - DELETE request received by kubelet
04:32:08.556 - LongTimeout exceeded
04:33:14.736 - graceful termination completes as the Pod phase is Succeeded
```
Note that the failure is after we introduced waiting for deleting all pods in AfterEach: https://github.com/kubernetes-sigs/kueue/pull/4611, so it didn't happen due to pods from previous tests loading the cluster (which is great we did it).

The graceful termination took 1min 51s.

So, the remaining ideas I have:
1. increase the cpu for the problematic containers to 200m or even 500m if possible.
2. set gracePeriodTermination to 1s, as the tests don't assert how the Pod terminated
3. increase the timeout

cc @tenzen-y @mbobrovskyi @mszadkow @mimowo: Oh I see this increase of CPU is already merged: https://github.com/kubernetes-sigs/kueue/pull/4627. This is great, maybe we will see these failures less often then, but I feel like 200m might still not be enough given the termination took nearly 2min in this case, but we can wait and see if this repeats.@mimowo: Ah it still happens: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4642/pull-kueue-test-e2e-main-1-30/1901574873466015744@mimowo: > Ah it still happens: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4642/pull-kueue-test-e2e-main-1-30/1901574873466015744

All the pods were scheduled on kind-worker:
```
> cat scheduler.log | grep lws-e2e-bt54c | grep Success
2025-03-17T10:12:21.773520512Z stderr F I0317 10:12:21.773142       1 schedule_one.go:304] "Successfully bound pod to node" pod="lws-e2e-bt54c/lws-0" node="kind-worker" evaluatedNodes=3 feasibleNodes=1
2025-03-17T10:12:21.790983596Z stderr F I0317 10:12:21.790679       1 schedule_one.go:304] "Successfully bound pod to node" pod="lws-e2e-bt54c/lws-1" node="kind-worker" evaluatedNodes=3 feasibleNodes=1
2025-03-17T10:12:23.101886885Z stderr F I0317 10:12:23.101724       1 schedule_one.go:304] "Successfully bound pod to node" pod="lws-e2e-bt54c/lws-1-1" node="kind-worker" evaluatedNodes=3 feasibleNodes=1
2025-03-17T10:12:23.109690811Z stderr F I0317 10:12:23.109298       1 schedule_one.go:304] "Successfully bound pod to node" pod="lws-e2e-bt54c/lws-0-1" node="kind-worker" evaluatedNodes=3 feasibleNodes=1
2025-03-17T10:12:23.139494874Z stderr F I0317 10:12:23.139248       1 schedule_one.go:304] "Successfully bound pod to node" pod="lws-e2e-bt54c/lws-1-2" node="kind-worker" evaluatedNodes=3 feasibleNodes=1
2025-03-17T10:12:23.147068818Z stderr F I0317 10:12:23.146947       1 schedule_one.go:304] "Successfully bound pod to node" pod="lws-e2e-bt54c/lws-0-2" node="kind-worker" evaluatedNodes=3 feasibleNodes=1
```@mimowo: Actually, this assert is a bit different, because it requires all 3 pods to be terminated within 45s. So, I think it is expected kubelet is expected to work more than when terminating a single pod.

Looking at the termination of `lws-e2e-bt54c/lws-0-1` on kind-worker (but other Pods also took long-ish):
```
Mar 17 10:12:23 kind-worker kubelet[247]: I0317 10:12:23.133334     247 status_manager.go:883] "Status for pod updated successfully" pod="lws-e2e-bt54c/lws-0-1" statusVersion=1 status={"phase":"Pending","conditions":[{"type":"PodReadyToStartContainers","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z"},{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z","reason":"ContainersNotReady","message":"containers with unready status: [c]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z","reason":"ContainersNotReady","message":"containers with unready status: [c]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z"}],"hostIP":"172.18.0.2","hostIPs":[{"ip":"172.18.0.2"}],"startTime":"2025-03-17T10:12:23Z","containerStatuses":[{"name":"c","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"registry.k8s.io/e2e-test-images/agnhost:2.53@sha256:99c6b4bb4a1e1df3f0b3752168c89358794d02258ebebc26bf21c29399011a85","imageID":"","started":false}],"qosClass":"Burstable"}
Mar 17 10:12:25 kind-worker kubelet[247]: I0317 10:12:25.075254     247 status_manager.go:874] "Patch status for pod" pod="lws-e2e-bt54c/lws-0-2" podUID="d2bd8f17-5f5e-4eda-9654-2947e23cc6e7" patch="{\"metadata\":{\"uid\":\"d2bd8f17-5f5e-4eda-9654-2947e23cc6e7\"},\"status\":{\"$setElementOrder/conditions\":[{\"type\":\"PodReadyToStartContainers\"},{\"type\":\"Initialized\"},{\"type\":\"Ready\"},{\"type\":\"ContainersReady\"},{\"type\":\"PodScheduled\"}],\"conditions\":[{\"lastTransitionTime\":\"2025-03-17T10:12:25Z\",\"status\":\"True\",\"type\":\"PodReadyToStartContainers\"},{\"lastTransitionTime\":\"2025-03-17T10:12:25Z\",\"message\":null,\"reason\":null,\"status\":\"True\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2025-03-17T10:12:25Z\",\"message\":null,\"reason\":null,\"status\":\"True\",\"type\":\"ContainersReady\"}],\"containerStatuses\":[{\"containerID\":\"containerd://62fc0d94d0c7748b072c7b7208fb6faee1e22ec07a7497b88369ca3e2bdba27d\",\"image\":\"registry.k8s.io/e2e-test-images/agnhost:2.53\",\"imageID\":\"docker.io/library/import-2025-03-17@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e\",\"lastState\":{},\"name\":\"c\",\"ready\":true,\"restartCount\":0,\"started\":true,\"state\":{\"running\":{\"startedAt\":\"2025-03-17T10:12:24Z\"}}}],\"phase\":\"Running\",\"podIP\":\"10.244.2.46\",\"podIPs\":[{\"ip\":\"10.244.2.46\"}]}}"
Mar 17 10:12:25 kind-worker kubelet[247]: I0317 10:12:25.480887     247 status_manager.go:874] "Patch status for pod" pod="lws-e2e-bt54c/lws-0-1" podUID="36a632b4-2b0a-485f-a3e4-9a91803b411c" patch="{\"metadata\":{\"uid\":\"36a632b4-2b0a-485f-a3e4-9a91803b411c\"},\"status\":{\"$setElementOrder/conditions\":[{\"type\":\"PodReadyToStartContainers\"},{\"type\":\"Initialized\"},{\"type\":\"Ready\"},{\"type\":\"ContainersReady\"},{\"type\":\"PodScheduled\"}],\"conditions\":[{\"lastTransitionTime\":\"2025-03-17T10:12:25Z\",\"status\":\"True\",\"type\":\"PodReadyToStartContainers\"},{\"lastTransitionTime\":\"2025-03-17T10:12:25Z\",\"message\":null,\"reason\":null,\"status\":\"True\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2025-03-17T10:12:25Z\",\"message\":null,\"reason\":null,\"status\":\"True\",\"type\":\"ContainersReady\"}],\"containerStatuses\":[{\"containerID\":\"containerd://4750b1ee9a74e91a64dfb3b0a8417077a68878a76e10569569f1002c2df3f827\",\"image\":\"registry.k8s.io/e2e-test-images/agnhost:2.53\",\"imageID\":\"docker.io/library/import-2025-03-17@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e\",\"lastState\":{},\"name\":\"c\",\"ready\":true,\"restartCount\":0,\"started\":true,\"state\":{\"running\":{\"startedAt\":\"2025-03-17T10:12:24Z\"}}}],\"phase\":\"Running\",\"podIP\":\"10.244.2.44\",\"podIPs\":[{\"ip\":\"10.244.2.44\"}]}}"
Mar 17 10:12:25 kind-worker kubelet[247]: I0317 10:12:25.480983     247 status_manager.go:883] "Status for pod updated successfully" pod="lws-e2e-bt54c/lws-0-1" statusVersion=2 status={"phase":"Running","conditions":[{"type":"PodReadyToStartContainers","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:25Z"},{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:25Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:25Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z"}],"hostIP":"172.18.0.2","hostIPs":[{"ip":"172.18.0.2"}],"podIP":"10.244.2.44","podIPs":[{"ip":"10.244.2.44"}],"startTime":"2025-03-17T10:12:23Z","containerStatuses":[{"name":"c","state":{"running":{"startedAt":"2025-03-17T10:12:24Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"registry.k8s.io/e2e-test-images/agnhost:2.53","imageID":"docker.io/library/import-2025-03-17@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e","containerID":"containerd://4750b1ee9a74e91a64dfb3b0a8417077a68878a76e10569569f1002c2df3f827","started":true}],"qosClass":"Burstable"}
Mar 17 10:12:26 kind-worker kubelet[247]: I0317 10:12:26.027958     247 status_manager.go:693] "Ignoring same status for pod" pod="lws-e2e-bt54c/lws-0-1" status={"phase":"Running","conditions":[{"type":"PodReadyToStartContainers","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:25Z"},{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:25Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:25Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z"}],"hostIP":"172.18.0.2","hostIPs":[{"ip":"172.18.0.2"}],"podIP":"10.244.2.44","podIPs":[{"ip":"10.244.2.44"}],"startTime":"2025-03-17T10:12:23Z","containerStatuses":[{"name":"c","state":{"running":{"startedAt":"2025-03-17T10:12:24Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"registry.k8s.io/e2e-test-images/agnhost:2.53","imageID":"docker.io/library/import-2025-03-17@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e","containerID":"containerd://4750b1ee9a74e91a64dfb3b0a8417077a68878a76e10569569f1002c2df3f827","started":true}],"qosClass":"Burstable"}
Mar 17 10:13:18 kind-worker kubelet[247]: I0317 10:13:18.043798     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-1" podUID="36a632b4-2b0a-485f-a3e4-9a91803b411c"
Mar 17 10:13:18 kind-worker kubelet[247]: I0317 10:13:18.065597     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-2" podUID="d2bd8f17-5f5e-4eda-9654-2947e23cc6e7"
Mar 17 10:13:19 kind-worker kubelet[247]: I0317 10:13:19.878407     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-1" podUID="36a632b4-2b0a-485f-a3e4-9a91803b411c"
Mar 17 10:13:19 kind-worker kubelet[247]: I0317 10:13:19.878530     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-2" podUID="d2bd8f17-5f5e-4eda-9654-2947e23cc6e7"
Mar 17 10:13:29 kind-worker kubelet[247]: I0317 10:13:29.880225     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-1" podUID="36a632b4-2b0a-485f-a3e4-9a91803b411c"
Mar 17 10:13:29 kind-worker kubelet[247]: I0317 10:13:29.880300     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-2" podUID="d2bd8f17-5f5e-4eda-9654-2947e23cc6e7"
Mar 17 10:13:39 kind-worker kubelet[247]: I0317 10:13:39.873882     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-1" podUID="36a632b4-2b0a-485f-a3e4-9a91803b411c"
Mar 17 10:13:39 kind-worker kubelet[247]: I0317 10:13:39.874043     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-2" podUID="d2bd8f17-5f5e-4eda-9654-2947e23cc6e7"
Mar 17 10:13:49 kind-worker kubelet[247]: I0317 10:13:49.874491     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-1" podUID="36a632b4-2b0a-485f-a3e4-9a91803b411c"
Mar 17 10:13:49 kind-worker kubelet[247]: I0317 10:13:49.874609     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-2" podUID="d2bd8f17-5f5e-4eda-9654-2947e23cc6e7"
Mar 17 10:13:56 kind-worker kubelet[247]: I0317 10:13:56.128887     247 status_manager.go:874] "Patch status for pod" pod="lws-e2e-bt54c/lws-0-2" podUID="d2bd8f17-5f5e-4eda-9654-2947e23cc6e7" patch="{\"metadata\":{\"uid\":\"d2bd8f17-5f5e-4eda-9654-2947e23cc6e7\"},\"status\":{\"$setElementOrder/conditions\":[{\"type\":\"PodReadyToStartContainers\"},{\"type\":\"Initialized\"},{\"type\":\"Ready\"},{\"type\":\"ContainersReady\"},{\"type\":\"PodScheduled\"}],\"conditions\":[{\"lastTransitionTime\":\"2025-03-17T10:13:55Z\",\"status\":\"False\",\"type\":\"PodReadyToStartContainers\"},{\"reason\":\"PodCompleted\",\"type\":\"Initialized\"},{\"lastTransitionTime\":\"2025-03-17T10:13:55Z\",\"reason\":\"PodCompleted\",\"status\":\"False\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2025-03-17T10:13:55Z\",\"reason\":\"PodCompleted\",\"status\":\"False\",\"type\":\"ContainersReady\"}],\"containerStatuses\":[{\"containerID\":\"containerd://62fc0d94d0c7748b072c7b7208fb6faee1e22ec07a7497b88369ca3e2bdba27d\",\"image\":\"registry.k8s.io/e2e-test-images/agnhost:2.53\",\"imageID\":\"docker.io/library/import-2025-03-17@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e\",\"lastState\":{},\"name\":\"c\",\"ready\":false,\"restartCount\":0,\"started\":false,\"state\":{\"terminated\":{\"containerID\":\"containerd://62fc0d94d0c7748b072c7b7208fb6faee1e22ec07a7497b88369ca3e2bdba27d\",\"exitCode\":0,\"finishedAt\":\"2025-03-17T10:13:18Z\",\"reason\":\"Completed\",\"startedAt\":\"2025-03-17T10:12:24Z\"}}}],\"phase\":\"Succeeded\",\"podIP\":null,\"podIPs\":null}}"
Mar 17 10:13:56 kind-worker kubelet[247]: I0317 10:13:56.542204     247 status_manager.go:944] "The pod termination is finished as SyncTerminatedPod completes its execution" phase="Succeeded" localPhase="Succeeded" pod="lws-e2e-bt54c/lws-0-2" podUID="d2bd8f17-5f5e-4eda-9654-2947e23cc6e7"
Mar 17 10:13:59 kind-worker kubelet[247]: I0317 10:13:59.874385     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-1" podUID="36a632b4-2b0a-485f-a3e4-9a91803b411c"
Mar 17 10:14:09 kind-worker kubelet[247]: I0317 10:14:09.874874     247 status_manager.go:939] "Delaying pod deletion as the phase is non-terminal" phase="Running" localPhase="Running" pod="lws-e2e-bt54c/lws-0-1" podUID="36a632b4-2b0a-485f-a3e4-9a91803b411c"
Mar 17 10:14:14 kind-worker kubelet[247]: I0317 10:14:14.009565     247 status_manager.go:874] "Patch status for pod" pod="lws-e2e-bt54c/lws-0-1" podUID="36a632b4-2b0a-485f-a3e4-9a91803b411c" patch="{\"metadata\":{\"uid\":\"36a632b4-2b0a-485f-a3e4-9a91803b411c\"},\"status\":{\"$setElementOrder/conditions\":[{\"type\":\"PodReadyToStartContainers\"},{\"type\":\"Initialized\"},{\"type\":\"Ready\"},{\"type\":\"ContainersReady\"},{\"type\":\"PodScheduled\"}],\"conditions\":[{\"lastTransitionTime\":\"2025-03-17T10:14:13Z\",\"status\":\"False\",\"type\":\"PodReadyToStartContainers\"},{\"reason\":\"PodCompleted\",\"type\":\"Initialized\"},{\"lastTransitionTime\":\"2025-03-17T10:14:13Z\",\"reason\":\"PodCompleted\",\"status\":\"False\",\"type\":\"Ready\"},{\"lastTransitionTime\":\"2025-03-17T10:14:13Z\",\"reason\":\"PodCompleted\",\"status\":\"False\",\"type\":\"ContainersReady\"}],\"containerStatuses\":[{\"containerID\":\"containerd://4750b1ee9a74e91a64dfb3b0a8417077a68878a76e10569569f1002c2df3f827\",\"image\":\"registry.k8s.io/e2e-test-images/agnhost:2.53\",\"imageID\":\"docker.io/library/import-2025-03-17@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e\",\"lastState\":{},\"name\":\"c\",\"ready\":false,\"restartCount\":0,\"started\":false,\"state\":{\"terminated\":{\"containerID\":\"containerd://4750b1ee9a74e91a64dfb3b0a8417077a68878a76e10569569f1002c2df3f827\",\"exitCode\":0,\"finishedAt\":\"2025-03-17T10:13:18Z\",\"reason\":\"Completed\",\"startedAt\":\"2025-03-17T10:12:24Z\"}}}],\"phase\":\"Succeeded\",\"podIP\":null,\"podIPs\":null}}"
Mar 17 10:14:14 kind-worker kubelet[247]: I0317 10:14:14.009641     247 status_manager.go:883] "Status for pod updated successfully" pod="lws-e2e-bt54c/lws-0-1" statusVersion=4 status={"phase":"Succeeded","conditions":[{"type":"PodReadyToStartContainers","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:14:13Z"},{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:14:13Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:14:13Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z"}],"hostIP":"172.18.0.2","hostIPs":[{"ip":"172.18.0.2"}],"startTime":"2025-03-17T10:12:23Z","containerStatuses":[{"name":"c","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2025-03-17T10:12:24Z","finishedAt":"2025-03-17T10:13:18Z","containerID":"containerd://4750b1ee9a74e91a64dfb3b0a8417077a68878a76e10569569f1002c2df3f827"}},"lastState":{},"ready":false,"restartCount":0,"image":"registry.k8s.io/e2e-test-images/agnhost:2.53","imageID":"docker.io/library/import-2025-03-17@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e","containerID":"containerd://4750b1ee9a74e91a64dfb3b0a8417077a68878a76e10569569f1002c2df3f827","started":false}],"qosClass":"Burstable"}
Mar 17 10:14:14 kind-worker kubelet[247]: I0317 10:14:14.468765     247 status_manager.go:883] "Status for pod updated successfully" pod="lws-e2e-bt54c/lws-0-1" statusVersion=5 status={"phase":"Succeeded","conditions":[{"type":"PodReadyToStartContainers","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:14:13Z"},{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z","reason":"PodCompleted"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:14:13Z","reason":"PodCompleted"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:14:13Z","reason":"PodCompleted"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2025-03-17T10:12:23Z"}],"hostIP":"172.18.0.2","hostIPs":[{"ip":"172.18.0.2"}],"podIP":"10.244.2.44","podIPs":[{"ip":"10.244.2.44"}],"startTime":"2025-03-17T10:12:23Z","containerStatuses":[{"name":"c","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2025-03-17T10:12:24Z","finishedAt":"2025-03-17T10:13:18Z","containerID":"containerd://4750b1ee9a74e91a64dfb3b0a8417077a68878a76e10569569f1002c2df3f827"}},"lastState":{},"ready":false,"restartCount":0,"image":"registry.k8s.io/e2e-test-images/agnhost:2.53","imageID":"docker.io/library/import-2025-03-17@sha256:b834a6ce4fdd44c6a61b08aad29a66af701a413fe3e41c7147d0786e10b0215e","containerID":"containerd://4750b1ee9a74e91a64dfb3b0a8417077a68878a76e10569569f1002c2df3f827","started":false}],"qosClass":"Burstable"}
Mar 17 10:14:14 kind-worker kubelet[247]: I0317 10:14:14.472989     247 status_manager.go:944] "The pod termination is finished as SyncTerminatedPod completes its execution" phase="Succeeded" localPhase="Succeeded" pod="lws-e2e-bt54c/lws-0-1" podUID="36a632b4-2b0a-485f-a3e4-9a91803b411c"
```
So, DELETE was at `10:13:18`, and Succeeded phase was at `10:14:14`, i.e. 56s.

@tenzen-y: We specified only CPU in LWS E2E. So, what about adding memory requests and limits as well?@mimowo: yes, we only assigned CPU, we should also probably set the memory. However, I think it will not help here, because the Pods don't seem to be memory-heavy. I think about two approaches:
1. increase CPU yet to 300m by default, as 56s is already close to 45s - previously we were observing >2min terminations
2. use 3*LongTimeout for this specific assert, because it actually requires 3 pods, not just one to terminate

I'm leaning to do both, WDYT @mbobrovskyi @tenzen-y ? @tenzen-y: > yes, we only assigned CPU, we should also probably set the memory. However, I think it will not help here, because the Pods don't seem to be memory-heavy. I think about two approaches:
> 
> 1. increase CPU yet to 300m by default, as 56s is already close to 45s - previously we were observing >2min terminations
> 2. use 3*LongTimeout for this specific assert, because it actually requires 3 pods, not just one to terminate
> 
> I'm leaning to do both, WDYT [@mbobrovskyi](https://github.com/mbobrovskyi) [@tenzen-y](https://github.com/tenzen-y) ?

If we take 2. we want to use SUM(Longtimeout, GracefulTerminationSeconds) instead of 3 x LongTimeout to avoid increasing timeout everywhere.
Any thoughts?@mbobrovskyi: I think we should set terminationGracePeriodSeconds=1 everywhere to avoid waiting for this duration.@mimowo: I'm ok with that, we just need to be careful to check if the test doesn't assert on the end state of the Pods. Maybe some tests expect the pods to end successfully. If they don't, I'm fine with setting `terminationGracePeriodSeconds=1`.@mimowo: @mbobrovskyi can you please open the PR?@mbobrovskyi: /assign@mimowo: Actually, we cannot always just blindly use `terminationGracePeriodSeconds=1`. For example this test failed recently:
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4648/pull-kueue-test-e2e-main-1-32/1901605023159160832
```
{Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/jobset_test.go:107 with:
Expected object to be comparable, diff:   (*v1.Condition)(
- 	nil,
+ 	s"&Condition{Type:Finished,Status:True,ObservedGeneration:0,LastTransitionTime:0001-01-01 00:00:00 +0000 UTC,Reason:Succeeded,Message:jobset completed successfully,}",
  )
 failed [FAILED] Timed out after 45.001s.
The function passed to Eventually failed at /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/jobset_test.go:107 with:
Expected object to be comparable, diff:   (*v1.Condition)(
- 	nil,
+ 	s"&Condition{Type:Finished,Status:True,ObservedGeneration:0,LastTransitionTime:0001-01-01 00:00:00 +0000 UTC,Reason:Succeeded,Message:jobset completed successfully,}",
  )
In [It] at: /home/prow/go/src/sigs.k8s.io/kueue/test/e2e/singlecluster/jobset_test.go:113 @ 03/17/25 12:12:27.713
}
```
And the test assumes the JobSet completes successfully (See [code](https://github.com/kubernetes-sigs/kueue/blob/main/test/e2e/singlecluster/jobset_test.go#L104-L113)), so with `terminationGracePeriodSeconds=1` it would not. 

Unfortunately the test already is using 500m.  So, we would either need to relax the test, or we also need another approach for such cases. 

Maybe for such tests (assuming termination of multiple Pods we use VeryLongTimeout=5min, or DoubleLongTimeout=1min30s. Note that this test requires terminating 4 pods. WDYT @tenzen-y @mbobrovskyi ?
@mimowo: Opened dedicated issue: https://github.com/kubernetes-sigs/kueue/issues/4651@mbobrovskyi: > Actually, we cannot always just blindly use terminationGracePeriodSeconds=1. For example this test failed recently:
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_kueue/4648/pull-kueue-test-e2e-main-1-32/1901605023159160832

I think this is a different issue and not related to it. We need to set `terminationGracePeriodSeconds = 1` for `BehaviorWaitForDeletion` and `BehaviorWaitForDeletionFailOnExit`. But here, we are using `BehaviorExitFast`, which simply stops the pod after running.@mbobrovskyi: /assign @mszadkow@mbobrovskyi: /unassign@mimowo: > I think this is a different issue and not related to it. We need to set terminationGracePeriodSeconds = 1 for BehaviorWaitForDeletion and BehaviorWaitForDeletionFailOnExit

Agree

> But here, we are using BehaviorExitFast, which simply stops the pod after running.

Yeah, we can solve it separately. Still for `BehaviorExitFast` we probably need longer timeout for a small number of tests.


Solution patch:
diff --git a/apis/config/v1beta1/configuration_types.go b/apis/config/v1beta1/configuration_types.go
index d2484a6d4..6c29d467e 100644
--- a/apis/config/v1beta1/configuration_types.go
+++ b/apis/config/v1beta1/configuration_types.go
@@ -92,7 +92,7 @@ type Configuration struct {
 	// MultiKueue controls the behaviour of the MultiKueue AdmissionCheck Controller.
 	MultiKueue *MultiKueue `json:"multiKueue,omitempty"`
 
-	// FairSharing controls the fair sharing semantics across the cluster.
+	// FairSharing controls the Fair Sharing semantics across the cluster.
 	FairSharing *FairSharing `json:"fairSharing,omitempty"`
 
 	// Resources provides additional configuration options for handling the resources.
@@ -450,7 +450,7 @@ const (
 )
 
 type FairSharing struct {
-	// enable indicates whether to enable fair sharing for all cohorts.
+	// enable indicates whether to enable Fair Sharing for all cohorts.
 	// Defaults to false.
 	Enable bool `json:"enable"`
 
diff --git a/apis/kueue/v1beta1/clusterqueue_types.go b/apis/kueue/v1beta1/clusterqueue_types.go
index 3f7ba1b28..478b70737 100644
--- a/apis/kueue/v1beta1/clusterqueue_types.go
+++ b/apis/kueue/v1beta1/clusterqueue_types.go
@@ -431,11 +431,11 @@ type ClusterQueuePreemption struct {
 	//   Workloads in the cohort that have lower priority than the pending
 	//   Workload. **Fair Sharing** only preempt Workloads in the cohort that
 	//   have lower priority than the pending Workload and that satisfy the
-	//   fair sharing preemptionStategies.
+	//   Fair Sharing preemptionStategies.
 	// - `Any`: **Classic Preemption** if the pending Workload fits within
 	//    the nominal quota of its ClusterQueue, preempt any Workload in the
 	//    cohort, irrespective of priority. **Fair Sharing** preempt Workloads
-	//    in the cohort that satisfy the fair sharing preemptionStrategies.
+	//    in the cohort that satisfy the Fair Sharing preemptionStrategies.
 	//
 	// +kubebuilder:default=Never
 	// +kubebuilder:validation:Enum=Never;LowerPriority;Any
diff --git a/apis/kueue/v1beta1/fairsharing_types.go b/apis/kueue/v1beta1/fairsharing_types.go
index 9a5b7389b..bb175d975 100644
--- a/apis/kueue/v1beta1/fairsharing_types.go
+++ b/apis/kueue/v1beta1/fairsharing_types.go
@@ -39,7 +39,7 @@ type FairSharing struct {
 	Weight *resource.Quantity `json:"weight,omitempty"`
 }
 
-// fairSharing contains the information about the current status of fair sharing.
+// fairSharing contains the information about the current status of Fair Sharing.
 type FairSharingStatus struct {
 	// WeightedShare represent the maximum of the ratios of usage
 	// above nominal quota to the lendable resources in the
diff --git a/apis/kueue/v1beta1/workload_types.go b/apis/kueue/v1beta1/workload_types.go
index db8a7c2b1..6a717e827 100644
--- a/apis/kueue/v1beta1/workload_types.go
+++ b/apis/kueue/v1beta1/workload_types.go
@@ -524,7 +524,7 @@ const (
 	InCohortReclamationReason string = "InCohortReclamation"
 
 	// InCohortFairSharingReason indicates the Workload was preempted due to
-	// fair sharing within the cohort.
+	// Fair Sharing within the cohort.
 	InCohortFairSharingReason string = "InCohortFairSharing"
 
 	// InCohortReclaimWhileBorrowingReason indicates the Workload was preempted
diff --git a/charts/kueue/templates/crd/kueue.x-k8s.io_clusterqueues.yaml b/charts/kueue/templates/crd/kueue.x-k8s.io_clusterqueues.yaml
index 369792831..50271b502 100644
--- a/charts/kueue/templates/crd/kueue.x-k8s.io_clusterqueues.yaml
+++ b/charts/kueue/templates/crd/kueue.x-k8s.io_clusterqueues.yaml
@@ -307,11 +307,11 @@ spec:
                         Workloads in the cohort that have lower priority than the pending
                         Workload. **Fair Sharing** only preempt Workloads in the cohort that
                         have lower priority than the pending Workload and that satisfy the
-                        fair sharing preemptionStategies.
+                        Fair Sharing preemptionStategies.
                       - `Any`: **Classic Preemption** if the pending Workload fits within
                          the nominal quota of its ClusterQueue, preempt any Workload in the
                          cohort, irrespective of priority. **Fair Sharing** preempt Workloads
-                         in the cohort that satisfy the fair sharing preemptionStrategies.
+                         in the cohort that satisfy the Fair Sharing preemptionStrategies.
                     enum:
                     - Never
                     - LowerPriority
@@ -583,7 +583,7 @@ spec:
                 x-kubernetes-list-type: map
               fairSharing:
                 description: fairSharing contains the information about the current
-                  status of fair sharing.
+                  status of Fair Sharing.
                 properties:
                   weightedShare:
                     description: |-
diff --git a/charts/kueue/templates/crd/kueue.x-k8s.io_cohorts.yaml b/charts/kueue/templates/crd/kueue.x-k8s.io_cohorts.yaml
index 784b5c98c..3619a48f8 100644
--- a/charts/kueue/templates/crd/kueue.x-k8s.io_cohorts.yaml
+++ b/charts/kueue/templates/crd/kueue.x-k8s.io_cohorts.yaml
@@ -247,7 +247,7 @@ spec:
             properties:
               fairSharing:
                 description: fairSharing contains the information about the current
-                  status of fair sharing.
+                  status of Fair Sharing.
                 properties:
                   weightedShare:
                     description: |-
diff --git a/config/components/crd/bases/kueue.x-k8s.io_clusterqueues.yaml b/config/components/crd/bases/kueue.x-k8s.io_clusterqueues.yaml
index 337f4585a..962623045 100644
--- a/config/components/crd/bases/kueue.x-k8s.io_clusterqueues.yaml
+++ b/config/components/crd/bases/kueue.x-k8s.io_clusterqueues.yaml
@@ -292,11 +292,11 @@ spec:
                         Workloads in the cohort that have lower priority than the pending
                         Workload. **Fair Sharing** only preempt Workloads in the cohort that
                         have lower priority than the pending Workload and that satisfy the
-                        fair sharing preemptionStategies.
+                        Fair Sharing preemptionStategies.
                       - `Any`: **Classic Preemption** if the pending Workload fits within
                          the nominal quota of its ClusterQueue, preempt any Workload in the
                          cohort, irrespective of priority. **Fair Sharing** preempt Workloads
-                         in the cohort that satisfy the fair sharing preemptionStrategies.
+                         in the cohort that satisfy the Fair Sharing preemptionStrategies.
                     enum:
                     - Never
                     - LowerPriority
@@ -568,7 +568,7 @@ spec:
                 x-kubernetes-list-type: map
               fairSharing:
                 description: fairSharing contains the information about the current
-                  status of fair sharing.
+                  status of Fair Sharing.
                 properties:
                   weightedShare:
                     description: |-
diff --git a/config/components/crd/bases/kueue.x-k8s.io_cohorts.yaml b/config/components/crd/bases/kueue.x-k8s.io_cohorts.yaml
index 80ee76aa3..d56c73ac0 100644
--- a/config/components/crd/bases/kueue.x-k8s.io_cohorts.yaml
+++ b/config/components/crd/bases/kueue.x-k8s.io_cohorts.yaml
@@ -232,7 +232,7 @@ spec:
             properties:
               fairSharing:
                 description: fairSharing contains the information about the current
-                  status of fair sharing.
+                  status of Fair Sharing.
                 properties:
                   weightedShare:
                     description: |-
diff --git a/pkg/controller/core/clusterqueue_controller.go b/pkg/controller/core/clusterqueue_controller.go
index 40ac271ea..596dde967 100644
--- a/pkg/controller/core/clusterqueue_controller.go
+++ b/pkg/controller/core/clusterqueue_controller.go
@@ -35,11 +35,13 @@ import (
 	"k8s.io/utils/clock"
 	"k8s.io/utils/ptr"
 	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/builder"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller"
 	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 	"sigs.k8s.io/controller-runtime/pkg/event"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
+	"sigs.k8s.io/controller-runtime/pkg/predicate"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
 	"sigs.k8s.io/controller-runtime/pkg/source"
 
@@ -79,6 +81,9 @@ type ClusterQueueReconciler struct {
 	clock                                clock.Clock
 }
 
+var _ reconcile.Reconciler = (*ClusterQueueReconciler)(nil)
+var _ predicate.TypedPredicate[*kueue.ClusterQueue] = (*ClusterQueueReconciler)(nil)
+
 type ClusterQueueReconcilerOptions struct {
 	Watchers                             []ClusterQueueUpdateWatcher
 	ReportResourceMetrics                bool
@@ -311,87 +316,66 @@ func (r *ClusterQueueReconciler) NotifyAdmissionCheckUpdate(oldAc, newAc *kueue.
 // Event handlers return true to signal the controller to reconcile the
 // ClusterQueue associated with the event.
 
-func (r *ClusterQueueReconciler) Create(e event.CreateEvent) bool {
-	cq, match := e.Object.(*kueue.ClusterQueue)
-	if !match {
-		// No need to interact with the cache for other objects.
-		return true
-	}
-	defer r.notifyWatchers(nil, cq)
+func (r *ClusterQueueReconciler) Create(e event.TypedCreateEvent[*kueue.ClusterQueue]) bool {
+	defer r.notifyWatchers(nil, e.Object)
 
-	log := r.log.WithValues("clusterQueue", klog.KObj(cq))
+	log := r.log.WithValues("clusterQueue", klog.KObj(e.Object))
 	log.V(2).Info("ClusterQueue create event")
 	ctx := ctrl.LoggerInto(context.Background(), log)
-	if err := r.cache.AddClusterQueue(ctx, cq); err != nil {
+	if err := r.cache.AddClusterQueue(ctx, e.Object); err != nil {
 		log.Error(err, "Failed to add clusterQueue to cache")
 	}
 
-	if err := r.qManager.AddClusterQueue(ctx, cq); err != nil {
+	if err := r.qManager.AddClusterQueue(ctx, e.Object); err != nil {
 		log.Error(err, "Failed to add clusterQueue to queue manager")
 	}
 
 	if r.reportResourceMetrics {
-		recordResourceMetrics(cq)
+		recordResourceMetrics(e.Object)
 	}
 
 	return true
 }
 
-func (r *ClusterQueueReconciler) Delete(e event.DeleteEvent) bool {
-	cq, match := e.Object.(*kueue.ClusterQueue)
-	if !match {
-		// No need to interact with the cache for other objects.
-		return true
-	}
-	defer r.notifyWatchers(cq, nil)
+func (r *ClusterQueueReconciler) Delete(e event.TypedDeleteEvent[*kueue.ClusterQueue]) bool {
+	defer r.notifyWatchers(e.Object, nil)
 
-	r.log.V(2).Info("ClusterQueue delete event", "clusterQueue", klog.KObj(cq))
-	r.cache.DeleteClusterQueue(cq)
-	r.qManager.DeleteClusterQueue(cq)
-	r.qManager.DeleteSnapshot(cq)
+	r.log.V(2).Info("ClusterQueue delete event", "clusterQueue", klog.KObj(e.Object))
+	r.cache.DeleteClusterQueue(e.Object)
+	r.qManager.DeleteClusterQueue(e.Object)
+	r.qManager.DeleteSnapshot(e.Object)
 
-	metrics.ClearClusterQueueResourceMetrics(cq.Name)
-	r.log.V(2).Info("Cleared resource metrics for deleted ClusterQueue.", "clusterQueue", klog.KObj(cq))
+	metrics.ClearClusterQueueResourceMetrics(e.Object.Name)
+	r.log.V(2).Info("Cleared resource metrics for deleted ClusterQueue.", "clusterQueue", klog.KObj(e.Object))
 
 	return true
 }
 
-func (r *ClusterQueueReconciler) Update(e event.UpdateEvent) bool {
-	oldCq, match := e.ObjectOld.(*kueue.ClusterQueue)
-	if !match {
-		// No need to interact with the cache for other objects.
-		return true
-	}
-	newCq, match := e.ObjectNew.(*kueue.ClusterQueue)
-	if !match {
-		// No need to interact with the cache for other objects.
-		return true
-	}
-
-	log := r.log.WithValues("clusterQueue", klog.KObj(newCq))
+func (r *ClusterQueueReconciler) Update(e event.TypedUpdateEvent[*kueue.ClusterQueue]) bool {
+	log := r.log.WithValues("clusterQueue", klog.KObj(e.ObjectNew))
 	log.V(2).Info("ClusterQueue update event")
 
-	if newCq.DeletionTimestamp != nil {
+	if e.ObjectNew.DeletionTimestamp != nil {
 		return true
 	}
-	defer r.notifyWatchers(oldCq, newCq)
-	specUpdated := !equality.Semantic.DeepEqual(oldCq.Spec, newCq.Spec)
+	defer r.notifyWatchers(e.ObjectOld, e.ObjectNew)
+	specUpdated := !equality.Semantic.DeepEqual(e.ObjectOld.Spec, e.ObjectNew.Spec)
 
-	if err := r.cache.UpdateClusterQueue(newCq); err != nil {
+	if err := r.cache.UpdateClusterQueue(e.ObjectNew); err != nil {
 		log.Error(err, "Failed to update clusterQueue in cache")
 	}
-	if err := r.qManager.UpdateClusterQueue(context.Background(), newCq, specUpdated); err != nil {
+	if err := r.qManager.UpdateClusterQueue(context.Background(), e.ObjectNew, specUpdated); err != nil {
 		log.Error(err, "Failed to update clusterQueue in queue manager")
 	}
 
 	if r.reportResourceMetrics {
-		updateResourceMetrics(oldCq, newCq)
+		updateResourceMetrics(e.ObjectOld, e.ObjectNew)
 	}
 	return true
 }
 
-func (r *ClusterQueueReconciler) Generic(e event.GenericEvent) bool {
-	r.log.V(2).Info("Got generic event", "obj", klog.KObj(e.Object), "kind", e.Object.GetObjectKind().GroupVersionKind())
+func (r *ClusterQueueReconciler) Generic(e event.TypedGenericEvent[*kueue.ClusterQueue]) bool {
+	r.log.V(3).Info("Got ClusterQueue generic event", "clusterQueue", klog.KObj(e.Object))
 	return true
 }
 
@@ -595,13 +579,18 @@ func (r *ClusterQueueReconciler) SetupWithManager(mgr ctrl.Manager, cfg *config.
 	snapHandler := cqSnapshotHandler{
 		queueVisibilityUpdateInterval: r.queueVisibilityUpdateInterval,
 	}
-	return ctrl.NewControllerManagedBy(mgr).
-		For(&kueue.ClusterQueue{}).
+	return builder.TypedControllerManagedBy[reconcile.Request](mgr).
+		Named("clusterqueue_controller").
+		WatchesRawSource(source.TypedKind(
+			mgr.GetCache(),
+			&kueue.ClusterQueue{},
+			&handler.TypedEnqueueRequestForObject[*kueue.ClusterQueue]{},
+			r,
+		)).
 		WithOptions(controller.Options{NeedLeaderElection: ptr.To(false)}).
 		Watches(&corev1.Namespace{}, &nsHandler).
 		WatchesRawSource(source.Channel(r.snapUpdateCh, &snapHandler)).
 		WatchesRawSource(source.Channel(r.nonCQObjectUpdateCh, &nonCQObjectHandler{})).
-		WithEventFilter(r).
 		Complete(WithLeadingManager(mgr, r, &kueue.ClusterQueue{}, cfg))
 }
 
diff --git a/pkg/metrics/metrics.go b/pkg/metrics/metrics.go
index 3249155c1..0b8edf7a1 100644
--- a/pkg/metrics/metrics.go
+++ b/pkg/metrics/metrics.go
@@ -242,7 +242,7 @@ The label 'reason' can have the following values:
 The label 'reason' can have the following values:
 - "InClusterQueue" means that the workload was preempted by a workload in the same ClusterQueue.
 - "InCohortReclamation" means that the workload was preempted by a workload in the same cohort due to reclamation of nominal quota.
-- "InCohortFairSharing" means that the workload was preempted by a workload in the same cohort due to fair sharing.
+- "InCohortFairSharing" means that the workload was preempted by a workload in the same cohort Fair Sharing.
 - "InCohortReclaimWhileBorrowing" means that the workload was preempted by a workload in the same cohort due to reclamation of nominal quota while borrowing.`,
 		}, []string{"preempting_cluster_queue", "reason"},
 	)
diff --git a/pkg/scheduler/preemption/preemption.go b/pkg/scheduler/preemption/preemption.go
index 082c41c71..44f2593a3 100644
--- a/pkg/scheduler/preemption/preemption.go
+++ b/pkg/scheduler/preemption/preemption.go
@@ -206,7 +206,7 @@ func canBorrowWithinCohort(preemptionCtx *preemptionCtx) (bool, *int32) {
 var HumanReadablePreemptionReasons = map[string]string{
 	kueue.InClusterQueueReason:                "prioritization in the ClusterQueue",
 	kueue.InCohortReclamationReason:           "reclamation within the cohort",
-	kueue.InCohortFairSharingReason:           "fair sharing within the cohort",
+	kueue.InCohortFairSharingReason:           "Fair Sharing within the cohort",
 	kueue.InCohortReclaimWhileBorrowingReason: "reclamation within the cohort while borrowing",
 	"": "UNKNOWN",
 }
diff --git a/pkg/scheduler/scheduler.go b/pkg/scheduler/scheduler.go
index fe11884ae..378a2099a 100644
--- a/pkg/scheduler/scheduler.go
+++ b/pkg/scheduler/scheduler.go
@@ -606,7 +606,7 @@ func makeIterator(ctx context.Context, entries []entry, workloadOrdering workloa
 
 // classicalIterator returns entries ordered on:
 // 1. request under nominal quota before borrowing.
-// 2. fair sharing: lower DominantResourceShare first.
+// 2. Fair Sharing: lower DominantResourceShare first.
 // 3. higher priority first.
 // 4. FIFO on eviction or creation timestamp.
 type classicalIterator struct {
diff --git a/site/content/en/docs/reference/kueue-config.v1beta1.md b/site/content/en/docs/reference/kueue-config.v1beta1.md
index ec9c395b0..009c9242e 100644
--- a/site/content/en/docs/reference/kueue-config.v1beta1.md
+++ b/site/content/en/docs/reference/kueue-config.v1beta1.md
@@ -189,7 +189,7 @@ instead.</p>
 <a href="#FairSharing"><code>FairSharing</code></a>
 </td>
 <td>
-   <p>FairSharing controls the fair sharing semantics across the cluster.</p>
+   <p>FairSharing controls the Fair Sharing semantics across the cluster.</p>
 </td>
 </tr>
 <tr><td><code>resources</code> <B>[Required]</B><br/>
@@ -458,7 +458,7 @@ must be named tls.key and tls.crt, respectively.</p>
 <code>bool</code>
 </td>
 <td>
-   <p>enable indicates whether to enable fair sharing for all cohorts.
+   <p>enable indicates whether to enable Fair Sharing for all cohorts.
 Defaults to false.</p>
 </td>
 </tr>
diff --git a/site/content/en/docs/reference/kueue.v1beta1.md b/site/content/en/docs/reference/kueue.v1beta1.md
index 3256c1e0b..08f65f7bf 100644
--- a/site/content/en/docs/reference/kueue.v1beta1.md
+++ b/site/content/en/docs/reference/kueue.v1beta1.md
@@ -743,11 +743,11 @@ fits within the nominal quota of its ClusterQueue, only preempt
 Workloads in the cohort that have lower priority than the pending
 Workload. <strong>Fair Sharing</strong> only preempt Workloads in the cohort that
 have lower priority than the pending Workload and that satisfy the
-fair sharing preemptionStategies.</li>
+Fair Sharing preemptionStategies.</li>
 <li><code>Any</code>: <strong>Classic Preemption</strong> if the pending Workload fits within
 the nominal quota of its ClusterQueue, preempt any Workload in the
 cohort, irrespective of priority. <strong>Fair Sharing</strong> preempt Workloads
-in the cohort that satisfy the fair sharing preemptionStrategies.</li>
+in the cohort that satisfy the Fair Sharing preemptionStrategies.</li>
 </ul>
 </td>
 </tr>
@@ -1067,7 +1067,7 @@ disadvantage against other ClusterQueues and Cohorts.</p>
 - [ClusterQueueStatus](#kueue-x-k8s-io-v1beta1-ClusterQueueStatus)
 
 
-<p>fairSharing contains the information about the current status of fair sharing.</p>
+<p>fairSharing contains the information about the current status of Fair Sharing.</p>
 
 
 <table class="table">



Test patch:
diff --git a/pkg/scheduler/preemption/preemption_test.go b/pkg/scheduler/preemption/preemption_test.go
index 6e43944f1..8b49d0387 100644
--- a/pkg/scheduler/preemption/preemption_test.go
+++ b/pkg/scheduler/preemption/preemption_test.go
@@ -2165,7 +2165,7 @@ func TestFairPreemptions(t *testing.T) {
 			targetCQ: "a",
 		},
 		// preemption.borrowWithinCohort does not affect how
-		// we handle fair sharing preemptions. Lower priority
+		// we handle Fair Sharing preemptions. Lower priority
 		// workloads are not preempted unless
 		// DominantResourceShare value indicates that they
 		// should be preempted.
diff --git a/pkg/util/testingjobs/deployment/wrappers.go b/pkg/util/testingjobs/deployment/wrappers.go
index facd3e8b2..293233c4b 100644
--- a/pkg/util/testingjobs/deployment/wrappers.go
+++ b/pkg/util/testingjobs/deployment/wrappers.go
@@ -155,3 +155,8 @@ func (d *DeploymentWrapper) PodTemplateAnnotation(k, v string) *DeploymentWrappe
 func (d *DeploymentWrapper) PodTemplateSpecQueue(q string) *DeploymentWrapper {
 	return d.PodTemplateSpecLabel(constants.QueueLabel, q)
 }
+
+func (d *DeploymentWrapper) TerminationGracePeriod(seconds int64) *DeploymentWrapper {
+	d.Spec.Template.Spec.TerminationGracePeriodSeconds = &seconds
+	return d
+}
diff --git a/pkg/util/testingjobs/leaderworkerset/wrappers.go b/pkg/util/testingjobs/leaderworkerset/wrappers.go
index 45c22f14d..4d9efe332 100644
--- a/pkg/util/testingjobs/leaderworkerset/wrappers.go
+++ b/pkg/util/testingjobs/leaderworkerset/wrappers.go
@@ -202,3 +202,11 @@ func (w *LeaderWorkerSetWrapper) WorkerTemplate(worker corev1.PodTemplateSpec) *
 	w.Spec.LeaderWorkerTemplate.WorkerTemplate = worker
 	return w
 }
+
+func (w *LeaderWorkerSetWrapper) TerminationGracePeriod(seconds int64) *LeaderWorkerSetWrapper {
+	if w.Spec.LeaderWorkerTemplate.LeaderTemplate != nil {
+		w.Spec.LeaderWorkerTemplate.LeaderTemplate.Spec.TerminationGracePeriodSeconds = &seconds
+	}
+	w.Spec.LeaderWorkerTemplate.WorkerTemplate.Spec.TerminationGracePeriodSeconds = &seconds
+	return w
+}
diff --git a/pkg/util/testingjobs/mpijob/wrappers_mpijob.go b/pkg/util/testingjobs/mpijob/wrappers_mpijob.go
index afc85ab6a..c1fc6a826 100644
--- a/pkg/util/testingjobs/mpijob/wrappers_mpijob.go
+++ b/pkg/util/testingjobs/mpijob/wrappers_mpijob.go
@@ -254,3 +254,9 @@ func (j *MPIJobWrapper) ManagedBy(c string) *MPIJobWrapper {
 	j.Spec.RunPolicy.ManagedBy = &c
 	return j
 }
+
+func (j *MPIJobWrapper) TerminationGracePeriodSeconds(seconds int64) *MPIJobWrapper {
+	j.Spec.MPIReplicaSpecs[kfmpi.MPIReplicaTypeLauncher].Template.Spec.TerminationGracePeriodSeconds = ptr.To(seconds)
+	j.Spec.MPIReplicaSpecs[kfmpi.MPIReplicaTypeWorker].Template.Spec.TerminationGracePeriodSeconds = ptr.To(seconds)
+	return j
+}
diff --git a/pkg/util/testingjobs/pytorchjob/wrappers_pytorchjob.go b/pkg/util/testingjobs/pytorchjob/wrappers_pytorchjob.go
index 353edca10..2ec073784 100644
--- a/pkg/util/testingjobs/pytorchjob/wrappers_pytorchjob.go
+++ b/pkg/util/testingjobs/pytorchjob/wrappers_pytorchjob.go
@@ -243,3 +243,9 @@ func (j *PyTorchJobWrapper) ManagedBy(c string) *PyTorchJobWrapper {
 	j.Spec.RunPolicy.ManagedBy = &c
 	return j
 }
+
+func (j *PyTorchJobWrapper) TerminationGracePeriodSeconds(seconds int64) *PyTorchJobWrapper {
+	j.Spec.PyTorchReplicaSpecs[kftraining.PyTorchJobReplicaTypeMaster].Template.Spec.TerminationGracePeriodSeconds = ptr.To(seconds)
+	j.Spec.PyTorchReplicaSpecs[kftraining.PyTorchJobReplicaTypeWorker].Template.Spec.TerminationGracePeriodSeconds = ptr.To(seconds)
+	return j
+}
diff --git a/pkg/util/testingjobs/statefulset/wrappers.go b/pkg/util/testingjobs/statefulset/wrappers.go
index f28c168b9..96861b03f 100644
--- a/pkg/util/testingjobs/statefulset/wrappers.go
+++ b/pkg/util/testingjobs/statefulset/wrappers.go
@@ -215,3 +215,9 @@ func (ss *StatefulSetWrapper) Limit(r corev1.ResourceName, v string) *StatefulSe
 func (ss *StatefulSetWrapper) RequestAndLimit(r corev1.ResourceName, v string) *StatefulSetWrapper {
 	return ss.Request(r, v).Limit(r, v)
 }
+
+// TerminationGracePeriod sets terminationGracePeriodSeconds for the pod object
+func (ss *StatefulSetWrapper) TerminationGracePeriod(seconds int64) *StatefulSetWrapper {
+	ss.Spec.Template.Spec.TerminationGracePeriodSeconds = &seconds
+	return ss
+}
diff --git a/test/e2e/customconfigs/skipjobswithoutqueuename_test.go b/test/e2e/customconfigs/skipjobswithoutqueuename_test.go
index c4cc5fdad..c25072f35 100644
--- a/test/e2e/customconfigs/skipjobswithoutqueuename_test.go
+++ b/test/e2e/customconfigs/skipjobswithoutqueuename_test.go
@@ -264,6 +264,7 @@ var _ = ginkgo.Describe("ManageJobsWithoutQueueName", ginkgo.Ordered, func() {
 			ginkgo.By("creating a pod without a queue name", func() {
 				testPod = testingpod.MakePod("test-pod", "kube-system").
 					Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
+					TerminationGracePeriod(1).
 					RequestAndLimit(corev1.ResourceCPU, "1").
 					RequestAndLimit(corev1.ResourceMemory, "2Gi").
 					Obj()
@@ -321,6 +322,7 @@ var _ = ginkgo.Describe("ManageJobsWithoutQueueName", ginkgo.Ordered, func() {
 					Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
 					RequestAndLimit(corev1.ResourceCPU, "1").
 					RequestAndLimit(corev1.ResourceMemory, "2Gi").
+					TerminationGracePeriod(1).
 					Replicas(2).
 					Obj()
 				gomega.Expect(k8sClient.Create(ctx, testDeploy)).Should(gomega.Succeed())
diff --git a/test/e2e/multikueue/e2e_test.go b/test/e2e/multikueue/e2e_test.go
index ee2d83aea..ff8d842a2 100644
--- a/test/e2e/multikueue/e2e_test.go
+++ b/test/e2e/multikueue/e2e_test.go
@@ -398,6 +398,7 @@ var _ = ginkgo.Describe("MultiKueue", func() {
 				Queue(managerLq.Name).
 				RequestAndLimit("cpu", "1").
 				RequestAndLimit("memory", "2G").
+				TerminationGracePeriod(1).
 				// Give it the time to be observed Active in the live status update step.
 				Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
 				Obj()
@@ -443,7 +444,8 @@ var _ = ginkgo.Describe("MultiKueue", func() {
 			})
 
 			ginkgo.By("Finishing the job's pod", func() {
-				util.WaitForActivePodsAndTerminate(ctx, k8sWorker2Client, worker2RestClient, worker2Cfg, job.Namespace, 1, 0)
+				listOpts := util.GetListOptsFromLabel(fmt.Sprintf("batch.kubernetes.io/job-name=%s", job.Name))
+				util.WaitForActivePodsAndTerminate(ctx, k8sWorker2Client, worker2RestClient, worker2Cfg, job.Namespace, 1, 0, listOpts)
 			})
 
 			ginkgo.By("Waiting for the job to finish", func() {
@@ -537,7 +539,8 @@ var _ = ginkgo.Describe("MultiKueue", func() {
 			})
 
 			ginkgo.By("Finishing the jobset pods", func() {
-				util.WaitForActivePodsAndTerminate(ctx, k8sWorker1Client, worker1RestClient, worker1Cfg, jobSet.Namespace, 4, 0)
+				listOpts := util.GetListOptsFromLabel(fmt.Sprintf("jobset.sigs.k8s.io/jobset-name=%s", jobSet.Name))
+				util.WaitForActivePodsAndTerminate(ctx, k8sWorker1Client, worker1RestClient, worker1Cfg, jobSet.Namespace, 4, 0, listOpts)
 			})
 
 			ginkgo.By("Waiting for the jobSet to finish", func() {
@@ -579,14 +582,16 @@ var _ = ginkgo.Describe("MultiKueue", func() {
 
 		ginkgo.It("Should run an appwrapper containing a job on worker if admitted", func() {
 			// Since it requires 2 CPU in total, this appwrapper can only be admitted in worker 1.
+			jobName := "job-1"
 			aw := testingaw.MakeAppWrapper("aw", managerNs.Name).
 				Queue(managerLq.Name).
-				Component(testingjob.MakeJob("job-1", managerNs.Name).
+				Component(testingjob.MakeJob(jobName, managerNs.Name).
 					SetTypeMeta().
 					Suspend(false).
 					Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion). // Give it the time to be observed Active in the live status update step.
 					Parallelism(2).
 					RequestAndLimit(corev1.ResourceCPU, "1").
+					TerminationGracePeriod(1).
 					SetTypeMeta().Obj()).
 				Obj()
 
@@ -609,7 +614,8 @@ var _ = ginkgo.Describe("MultiKueue", func() {
 			})
 
 			ginkgo.By("Finishing the wrapped job's pods", func() {
-				util.WaitForActivePodsAndTerminate(ctx, k8sWorker1Client, worker1RestClient, worker1Cfg, aw.Namespace, 2, 0)
+				listOpts := util.GetListOptsFromLabel(fmt.Sprintf("batch.kubernetes.io/job-name=%s", jobName))
+				util.WaitForActivePodsAndTerminate(ctx, k8sWorker1Client, worker1RestClient, worker1Cfg, aw.Namespace, 2, 0, listOpts)
 			})
 
 			ginkgo.By("Waiting for the appwrapper to finish", func() {
diff --git a/test/e2e/singlecluster/deployment_test.go b/test/e2e/singlecluster/deployment_test.go
index c0056de7d..a69d48e3d 100644
--- a/test/e2e/singlecluster/deployment_test.go
+++ b/test/e2e/singlecluster/deployment_test.go
@@ -88,6 +88,7 @@ var _ = ginkgo.Describe("Deployment", func() {
 		deployment := deploymenttesting.MakeDeployment("deployment", ns.Name).
 			Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
 			RequestAndLimit(corev1.ResourceCPU, "200m").
+			TerminationGracePeriod(1).
 			Replicas(3).
 			Queue(lq.Name).
 			Obj()
@@ -137,6 +138,7 @@ var _ = ginkgo.Describe("Deployment", func() {
 		deployment := deploymenttesting.MakeDeployment("deployment", ns.Name).
 			Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
 			RequestAndLimit(corev1.ResourceCPU, "200m").
+			TerminationGracePeriod(1).
 			Replicas(3).
 			Queue("invalid-queue-name").
 			Obj()
diff --git a/test/e2e/singlecluster/leaderworkerset_test.go b/test/e2e/singlecluster/leaderworkerset_test.go
index c18e25aa5..2591b689f 100644
--- a/test/e2e/singlecluster/leaderworkerset_test.go
+++ b/test/e2e/singlecluster/leaderworkerset_test.go
@@ -89,6 +89,7 @@ var _ = ginkgo.Describe("LeaderWorkerSet integration", func() {
 				Size(1).
 				Replicas(1).
 				RequestAndLimit(corev1.ResourceCPU, "200m").
+				TerminationGracePeriod(1).
 				Queue(lq.Name).
 				Obj()
 
@@ -147,6 +148,7 @@ var _ = ginkgo.Describe("LeaderWorkerSet integration", func() {
 				Size(3).
 				Replicas(1).
 				RequestAndLimit(corev1.ResourceCPU, "200m").
+				TerminationGracePeriod(1).
 				Queue(lq.Name).
 				Obj()
 			ginkgo.By("Create a LeaderWorkerSet", func() {
@@ -201,6 +203,7 @@ var _ = ginkgo.Describe("LeaderWorkerSet integration", func() {
 				Size(3).
 				Replicas(2).
 				RequestAndLimit(corev1.ResourceCPU, "200m").
+				TerminationGracePeriod(1).
 				Queue(lq.Name).
 				Obj()
 
@@ -264,6 +267,7 @@ var _ = ginkgo.Describe("LeaderWorkerSet integration", func() {
 					Size(3).
 					Replicas(1).
 					RequestAndLimit(corev1.ResourceCPU, "200m").
+					TerminationGracePeriod(1).
 					Queue(lq.Name).
 					StartupPolicy(startupPolicyType).
 					Obj()
@@ -358,6 +362,7 @@ var _ = ginkgo.Describe("LeaderWorkerSet integration", func() {
 					Size(3).
 					Replicas(2).
 					RequestAndLimit(corev1.ResourceCPU, "200m").
+					TerminationGracePeriod(1).
 					Queue(lq.Name).
 					StartupPolicy(startupPolicyType).
 					Obj()
@@ -455,6 +460,7 @@ var _ = ginkgo.Describe("LeaderWorkerSet integration", func() {
 					Size(3).
 					Replicas(2).
 					RequestAndLimit(corev1.ResourceCPU, "200m").
+					TerminationGracePeriod(1).
 					Queue(lq.Name).
 					LeaderTemplate(corev1.PodTemplateSpec{
 						Spec: corev1.PodSpec{
@@ -474,6 +480,7 @@ var _ = ginkgo.Describe("LeaderWorkerSet integration", func() {
 						},
 					}).
 					StartupPolicy(startupPolicyType).
+					TerminationGracePeriod(1).
 					Obj()
 
 				ginkgo.By("Create a LeaderWorkerSet", func() {
@@ -556,6 +563,7 @@ var _ = ginkgo.Describe("LeaderWorkerSet integration", func() {
 						NodeSelector: map[string]string{},
 					},
 				}).
+				TerminationGracePeriod(1).
 				Obj()
 
 			ginkgo.By("Create a LeaderWorkerSet", func() {
diff --git a/test/e2e/singlecluster/metrics_test.go b/test/e2e/singlecluster/metrics_test.go
index 7bfa873b3..5b3244b02 100644
--- a/test/e2e/singlecluster/metrics_test.go
+++ b/test/e2e/singlecluster/metrics_test.go
@@ -84,6 +84,7 @@ var _ = ginkgo.Describe("Metrics", func() {
 		curlPod = testingjobspod.MakePod("curl-metrics", config.DefaultNamespace).
 			ServiceAccountName(serviceAccountName).
 			Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
+			TerminationGracePeriod(1).
 			Obj()
 		gomega.Expect(k8sClient.Create(ctx, curlPod)).Should(gomega.Succeed())
 
diff --git a/test/e2e/singlecluster/pod_test.go b/test/e2e/singlecluster/pod_test.go
index 547a93415..3f9b72340 100644
--- a/test/e2e/singlecluster/pod_test.go
+++ b/test/e2e/singlecluster/pod_test.go
@@ -17,13 +17,14 @@ limitations under the License.
 package e2e
 
 import (
+	"fmt"
+
 	"github.com/google/go-cmp/cmp/cmpopts"
 	"github.com/onsi/ginkgo/v2"
 	"github.com/onsi/gomega"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/sets"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -451,6 +452,7 @@ var _ = ginkgo.Describe("Pod groups", func() {
 				Queue(lq.Name).
 				PriorityClass("high").
 				RequestAndLimit(corev1.ResourceCPU, "1").
+				TerminationGracePeriod(1).
 				MakeGroup(2)
 			highGroupKey := client.ObjectKey{Namespace: ns.Name, Name: "high-priority-group"}
 
@@ -484,7 +486,7 @@ var _ = ginkgo.Describe("Pod groups", func() {
 					for _, origPod := range defaultPriorityGroup {
 						var p corev1.Pod
 						g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(origPod), &p)).To(gomega.Succeed())
-						g.Expect(p.Status.Phase).To(gomega.Equal(corev1.PodFailed))
+						g.Expect(p.Status.Phase).To(gomega.Equal(corev1.PodFailed), fmt.Sprintf("%#v", p.Status))
 					}
 				}, util.LongTimeout, util.Interval).Should(gomega.Succeed())
 			})
@@ -522,11 +524,7 @@ var _ = ginkgo.Describe("Pod groups", func() {
 			})
 
 			ginkgo.By("Call high priority group pods to complete", func() {
-				selector, err := labels.Parse("kueue.x-k8s.io/pod-group-name=high-priority-group")
-				gomega.Expect(err).NotTo(gomega.HaveOccurred())
-				listOpts := &client.ListOptions{
-					LabelSelector: selector,
-				}
+				listOpts := util.GetListOptsFromLabel("kueue.x-k8s.io/pod-group-name=high-priority-group")
 				util.WaitForActivePodsAndTerminate(ctx, k8sClient, restClient, cfg, ns.Name, 2, 0, listOpts)
 			})
 
diff --git a/test/e2e/singlecluster/statefulset_test.go b/test/e2e/singlecluster/statefulset_test.go
index 164b7880d..eda654dae 100644
--- a/test/e2e/singlecluster/statefulset_test.go
+++ b/test/e2e/singlecluster/statefulset_test.go
@@ -90,6 +90,7 @@ var _ = ginkgo.Describe("StatefulSet integration", func() {
 			statefulSet := statefulsettesting.MakeStatefulSet("sts", ns.Name).
 				Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
 				RequestAndLimit(corev1.ResourceCPU, "200m").
+				TerminationGracePeriod(1).
 				Replicas(3).
 				Queue(lq.Name).
 				Obj()
@@ -109,6 +110,7 @@ var _ = ginkgo.Describe("StatefulSet integration", func() {
 				conflictingStatefulSet := statefulsettesting.MakeStatefulSet("sts-conflict", ns.Name).
 					Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
 					RequestAndLimit(corev1.ResourceCPU, "200m").
+					TerminationGracePeriod(1).
 					Replicas(1).
 					Queue(lq.Name).
 					Obj()
@@ -146,6 +148,7 @@ var _ = ginkgo.Describe("StatefulSet integration", func() {
 			statefulSet := statefulsettesting.MakeStatefulSet("sts", ns.Name).
 				Image(util.E2eTestAgnHostImageOld, util.BehaviorWaitForDeletion).
 				RequestAndLimit(corev1.ResourceCPU, "200m").
+				TerminationGracePeriod(1).
 				Replicas(3).
 				Queue(lq.Name).
 				Obj()
@@ -196,6 +199,7 @@ var _ = ginkgo.Describe("StatefulSet integration", func() {
 			statefulSet := statefulsettesting.MakeStatefulSet("sts", ns.Name).
 				Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
 				RequestAndLimit(corev1.ResourceCPU, "200m").
+				TerminationGracePeriod(1).
 				Replicas(3).
 				Queue(lq.Name).
 				Obj()
@@ -280,6 +284,7 @@ var _ = ginkgo.Describe("StatefulSet integration", func() {
 			statefulSet := statefulsettesting.MakeStatefulSet("sts", ns.Name).
 				Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
 				RequestAndLimit(corev1.ResourceCPU, "200m").
+				TerminationGracePeriod(1).
 				Replicas(3).
 				Queue(lq.Name).
 				Obj()
@@ -386,6 +391,7 @@ var _ = ginkgo.Describe("StatefulSet integration", func() {
 			statefulSet := statefulsettesting.MakeStatefulSet("sts", ns.Name).
 				Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
 				RequestAndLimit(corev1.ResourceCPU, "200m").
+				TerminationGracePeriod(1).
 				Replicas(3).
 				Queue(localQueueName).
 				Obj()
diff --git a/test/e2e/tas/leaderworkerset_test.go b/test/e2e/tas/leaderworkerset_test.go
index f1a7c36de..5724aa256 100644
--- a/test/e2e/tas/leaderworkerset_test.go
+++ b/test/e2e/tas/leaderworkerset_test.go
@@ -114,6 +114,7 @@ var _ = ginkgo.Describe("TopologyAwareScheduling for LeaderWorkerSet", func() {
 						},
 					},
 				}).
+				TerminationGracePeriod(1).
 				Obj()
 			ginkgo.By("Creating a LeaderWorkerSet", func() {
 				gomega.Expect(k8sClient.Create(ctx, lws)).To(gomega.Succeed())
diff --git a/test/e2e/tas/statefulset_test.go b/test/e2e/tas/statefulset_test.go
index c248e3435..48833dae6 100644
--- a/test/e2e/tas/statefulset_test.go
+++ b/test/e2e/tas/statefulset_test.go
@@ -84,6 +84,7 @@ var _ = ginkgo.Describe("TopologyAwareScheduling for StatefulSet", func() {
 				Replicas(replicas).
 				Queue(localQueue.Name).
 				PodTemplateSpecAnnotation(kueuealpha.PodSetRequiredTopologyAnnotation, testing.DefaultBlockTopologyLevel).
+				TerminationGracePeriod(1).
 				Obj()
 			gomega.Expect(k8sClient.Create(ctx, sts)).Should(gomega.Succeed())
 
diff --git a/test/util/util.go b/test/util/util.go
index b10afa727..9ef8aa86c 100644
--- a/test/util/util.go
+++ b/test/util/util.go
@@ -1036,3 +1036,11 @@ func FindDeploymentCondition(deployment *appsv1.Deployment, deploymentType appsv
 	}
 	return nil
 }
+
+func GetListOptsFromLabel(label string) *client.ListOptions {
+	selector, err := labels.Parse(label)
+	gomega.Expect(err).NotTo(gomega.HaveOccurred())
+	return &client.ListOptions{
+		LabelSelector: selector,
+	}
+}