Repository: kubernetes-sigs/karpenter
Pull request #2173: test: Migrate the Termination suite test from the AWS Karpenter Provider
Author: login: engedaam
Labels: name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
Created: 2025-04-28T19:13:33Z | Merged: 2025-05-13T17:09:18Z

<!-- Please follow the guidelines at https://www.conventionalcommits.org/en/v1.0.0/ and use one of the following in your title:
feat:            <-- New features that require a MINOR version update
fix:             <-- Bug fixes that require at PATCH version update
chore:           <-- Smaller changes that impact behavior but aren't large enough to be features
perf:            <-- Code changes that improve performance but do not impact behavior
docs:            <-- Documentation changes that do not impact code
test:            <-- Test changes that do not impact behavior
ci:              <-- Changes that affect test or rollout automation
!${type}:        <-- Include ! if your change includes a backwards incompatible change.
-->

Fixes #N/A <!-- issue number -->

**Description**
##### Termination 

* High Level Description: Looks at forceful termination and termination grace period
* Source: https://github.com/aws/karpenter-provider-aws/tree/main/test/suites/termination
* Test Cases: 
    * Emptiness 
        * should not allow emptiness if the budget is fully blocking
        * should not allow emptiness if the budget is fully blocking during a scheduled time
        * should terminate an empty node
    * Termination Grace Period 
        * should delete pod with do-not-disrupt when it reaches its terminationGracePeriodSeconds
    * Termination 
        * should drain pods on a node in order


**How was this change tested?**

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.



Diff:
diff --git a/hack/kwok/stages/pod-delete.yaml b/hack/kwok/stages/pod-delete.yaml
new file mode 100644
index 0000000000..0c978fdb1a
--- /dev/null
+++ b/hack/kwok/stages/pod-delete.yaml
@@ -0,0 +1,21 @@
+apiVersion: kwok.x-k8s.io/v1alpha1
+kind: Stage
+metadata:
+  name: pod-delete
+spec:
+  delay:
+    durationFrom:
+      expressionFrom: .metadata.annotations["pod-delete.stage.kwok.x-k8s.io/delay"]
+  next:
+    delete: true
+    finalizers:
+      empty: true
+    statusSubresource: status
+  resourceRef:
+    apiGroup: v1
+    kind: Pod
+  selector:
+    matchExpressions:
+    - key: .metadata.deletionTimestamp
+      operator: Exists
+  weight: 100
\ No newline at end of file
diff --git a/pkg/test/pods.go b/pkg/test/pods.go
index 96195c1129..f846e0542f 100644
--- a/pkg/test/pods.go
+++ b/pkg/test/pods.go
@@ -74,7 +74,8 @@ type EphemeralVolumeTemplateOptions struct {
 }
 
 var (
-	DefaultImage = "public.ecr.aws/eks-distro/kubernetes/pause:3.2"
+	DefaultImage        = "public.ecr.aws/eks-distro/kubernetes/pause:3.2"
+	KWOKDelayAnnotation = "pod-delete.stage.kwok.x-k8s.io/delay"
 )
 
 // Pod creates a test pod with defaults that can be overridden by PodOptions.
@@ -156,13 +157,16 @@ func Pod(overrides ...PodOptions) *v1.Pod {
 	// Can't use v1.LifecycleHandler == v1.SleepAction as that's a feature gate in Alpha 1.29.
 	// https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-handler-implementations
 	if options.PreStopSleep != nil {
+		p.ObjectMeta.Annotations = lo.Assign(p.Annotations, map[string]string{
+			KWOKDelayAnnotation: fmt.Sprintf("%ds", lo.FromPtr(options.PreStopSleep)),
+		})
 		p.Spec.Containers[0].Lifecycle = &v1.Lifecycle{
 			PreStop: &v1.LifecycleHandler{
 				Exec: &v1.ExecAction{
 					Command: []string{
 						"/bin/sh",
 						"-c",
-						fmt.Sprintf("sleep %d", *options.PreStopSleep),
+						fmt.Sprintf("sleep %d", lo.FromPtr(options.PreStopSleep)),
 					},
 				},
 			},
diff --git a/test/pkg/environment/common/expectations.go b/test/pkg/environment/common/expectations.go
index 9961e5deda..4fb2f87fbc 100644
--- a/test/pkg/environment/common/expectations.go
+++ b/test/pkg/environment/common/expectations.go
@@ -236,6 +236,68 @@ func (env *Environment) ConsistentlyExpectDisruptionsUntilNoneLeft(nodesAtStart,
 	}).WithTimeout(timeout).WithPolling(5 * time.Second).Should(Succeed())
 }
 
+func (env *Environment) EventuallyExpectLaunchedNodeClaimCount(comparator string, count int) []*v1.NodeClaim {
+	GinkgoHelper()
+	By(fmt.Sprintf("waiting for nodes to be %s to %d", comparator, count))
+	nodeClaimList := &v1.NodeClaimList{}
+	Eventually(func(g Gomega) {
+		g.Expect(env.Client.List(env, nodeClaimList, client.HasLabels{test.DiscoveryLabel})).To(Succeed())
+		g.Expect(lo.CountBy(nodeClaimList.Items, func(nc v1.NodeClaim) bool { return nc.StatusConditions().IsTrue(v1.ConditionTypeLaunched) })).To(BeNumerically(comparator, count),
+			fmt.Sprintf("expected %d nodeclaims, had %d (%v)", count, len(nodeClaimList.Items), NodeClaimNames(lo.ToSlicePtr(nodeClaimList.Items))))
+	}).Should(Succeed())
+	return lo.ToSlicePtr(nodeClaimList.Items)
+}
+
+func (env *Environment) EventuallyExpectConsolidatable(nodeClaims ...*v1.NodeClaim) {
+	GinkgoHelper()
+	Eventually(func(g Gomega) {
+		for _, nc := range nodeClaims {
+			g.Expect(env.Client.Get(env, client.ObjectKeyFromObject(nc), nc)).To(Succeed())
+			g.Expect(nc.StatusConditions().Get(v1.ConditionTypeConsolidatable).IsTrue()).To(BeTrue())
+		}
+	}).Should(Succeed())
+}
+
+func (env *Environment) ConsistentlyExpectHealthyPods(duration time.Duration, pods ...*corev1.Pod) {
+	GinkgoHelper()
+	By(fmt.Sprintf("expecting %d pods to be ready for %s", len(pods), duration))
+	Consistently(func(g Gomega) {
+		for _, pod := range pods {
+			g.Expect(env.Client.Get(env, client.ObjectKeyFromObject(pod), pod)).To(Succeed())
+			g.Expect(pod.Status.Conditions).To(ContainElement(And(
+				HaveField("Type", Equal(corev1.PodReady)),
+				HaveField("Status", Equal(corev1.ConditionTrue)),
+			)))
+		}
+	}, duration.String()).Should(Succeed())
+}
+
+func (env *Environment) EventuallyExpectTerminating(pods ...*corev1.Pod) {
+	GinkgoHelper()
+	env.eventuallyExpectTerminatingWithTimeout(-1, pods...)
+}
+
+func (env *Environment) eventuallyExpectTerminatingWithTimeout(timeout time.Duration, pods ...*corev1.Pod) {
+	GinkgoHelper()
+	Eventually(func(g Gomega) {
+		for _, pod := range pods {
+			g.Expect(env.Client.Get(env, client.ObjectKeyFromObject(pod), pod)).To(Succeed())
+			g.Expect(pod.DeletionTimestamp.IsZero()).To(BeFalse())
+		}
+	}).WithTimeout(timeout).Should(Succeed())
+}
+
+func (env *Environment) ConsistentlyExpectActivePods(duration time.Duration, pods ...*corev1.Pod) {
+	GinkgoHelper()
+	By(fmt.Sprintf("expecting %d pods to be live for %s", len(pods), duration))
+	Consistently(func(g Gomega) {
+		for _, pod := range pods {
+			g.Expect(env.Client.Get(env, client.ObjectKeyFromObject(pod), pod)).To(Succeed())
+			g.Expect(pod.DeletionTimestamp.IsZero()).To(BeTrue())
+		}
+	}, duration.String()).Should(Succeed())
+}
+
 func (env *Environment) ExpectSettings() (res []corev1.EnvVar) {
 	GinkgoHelper()
 
diff --git a/test/suites/integration/termination_test.go b/test/suites/integration/termination_test.go
new file mode 100644
index 0000000000..1491dbb82c
--- /dev/null
+++ b/test/suites/integration/termination_test.go
@@ -0,0 +1,363 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package integration_test
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/samber/lo"
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
+
+	"sigs.k8s.io/karpenter/pkg/test"
+
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	karpv1 "sigs.k8s.io/karpenter/pkg/apis/v1"
+)
+
+var _ = Describe("Termination", func() {
+	Context("Emptiness", func() {
+		var dep *appsv1.Deployment
+		var selector labels.Selector
+		var numPods int
+		BeforeEach(func() {
+			nodePool.Spec.Disruption.ConsolidationPolicy = karpv1.ConsolidationPolicyWhenEmpty
+			nodePool.Spec.Disruption.ConsolidateAfter = karpv1.MustParseNillableDuration("0s")
+
+			numPods = 1
+			dep = test.Deployment(test.DeploymentOptions{
+				Replicas: int32(numPods),
+				PodOptions: test.PodOptions{
+					ObjectMeta: metav1.ObjectMeta{
+						Labels: map[string]string{"app": "large-app"},
+					},
+				},
+			})
+			selector = labels.SelectorFromSet(dep.Spec.Selector.MatchLabels)
+		})
+		Context("Budgets", func() {
+			It("should not allow emptiness if the budget is fully blocking", func() {
+				// We're going to define a budget that doesn't allow any emptiness disruption to happen
+				nodePool.Spec.Disruption.Budgets = []karpv1.Budget{{
+					Nodes: "0",
+				}}
+
+				env.ExpectCreated(nodeClass, nodePool, dep)
+
+				nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+				env.EventuallyExpectCreatedNodeCount("==", 1)
+				env.EventuallyExpectHealthyPodCount(selector, numPods)
+
+				// Delete the deployment so there is nothing running on the node
+				env.ExpectDeleted(dep)
+
+				env.EventuallyExpectConsolidatable(nodeClaim)
+				env.ConsistentlyExpectNoDisruptions(1, time.Minute)
+			})
+			It("should not allow emptiness if the budget is fully blocking during a scheduled time", func() {
+				// We're going to define a budget that doesn't allow any emptiness disruption to happen
+				// This is going to be on a schedule that only lasts 30 minutes, whose window starts 15 minutes before
+				// the current time and extends 15 minutes past the current time
+				// Times need to be in UTC since the karpenter containers were built in UTC time
+				windowStart := time.Now().Add(-time.Minute * 15).UTC()
+				nodePool.Spec.Disruption.Budgets = []karpv1.Budget{{
+					Nodes:    "0",
+					Schedule: lo.ToPtr(fmt.Sprintf("%d %d * * *", windowStart.Minute(), windowStart.Hour())),
+					Duration: &metav1.Duration{Duration: time.Minute * 30},
+				}}
+
+				env.ExpectCreated(nodeClass, nodePool, dep)
+
+				nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+				env.EventuallyExpectCreatedNodeCount("==", 1)
+				env.EventuallyExpectHealthyPodCount(selector, numPods)
+
+				// Delete the deployment so there is nothing running on the node
+				env.ExpectDeleted(dep)
+
+				env.EventuallyExpectConsolidatable(nodeClaim)
+				env.ConsistentlyExpectNoDisruptions(1, time.Minute)
+			})
+		})
+		It("should terminate an empty node", func() {
+			nodePool.Spec.Disruption.ConsolidateAfter = karpv1.MustParseNillableDuration("10s")
+
+			const numPods = 1
+			deployment := test.Deployment(test.DeploymentOptions{Replicas: numPods})
+
+			By("kicking off provisioning for a deployment")
+			env.ExpectCreated(nodeClass, nodePool, deployment)
+			nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+			node := env.EventuallyExpectCreatedNodeCount("==", 1)[0]
+			env.EventuallyExpectHealthyPodCount(labels.SelectorFromSet(deployment.Spec.Selector.MatchLabels), numPods)
+
+			By("making the nodeclaim empty")
+			persisted := deployment.DeepCopy()
+			deployment.Spec.Replicas = lo.ToPtr(int32(0))
+			Expect(env.Client.Patch(env, deployment, client.StrategicMergeFrom(persisted))).To(Succeed())
+
+			env.EventuallyExpectConsolidatable(nodeClaim)
+
+			By("waiting for the nodeclaim to deprovision when past its ConsolidateAfter timeout of 0")
+			nodePool.Spec.Disruption.ConsolidateAfter = karpv1.MustParseNillableDuration("0s")
+			env.ExpectUpdated(nodePool)
+
+			env.EventuallyExpectNotFound(nodeClaim, node)
+		})
+	})
+	Describe("TerminationGracePeriod", func() {
+		BeforeEach(func() {
+			nodePool.Spec.Template.Spec.TerminationGracePeriod = &metav1.Duration{Duration: time.Second * 60}
+		})
+		It("should delete pod with do-not-disrupt when it reaches its terminationGracePeriodSeconds", func() {
+			pod := test.UnschedulablePod(test.PodOptions{ObjectMeta: metav1.ObjectMeta{Annotations: map[string]string{
+				karpv1.DoNotDisruptAnnotationKey: "true",
+			}}, TerminationGracePeriodSeconds: lo.ToPtr(int64(30))})
+			env.ExpectCreated(nodeClass, nodePool, pod)
+
+			nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+			node := env.EventuallyExpectCreatedNodeCount("==", 1)[0]
+			env.EventuallyExpectHealthy(pod)
+
+			// Delete the nodeclaim to start the TerminationGracePeriod
+			env.ExpectDeleted(nodeClaim)
+
+			// Eventually the node will be tainted
+			Eventually(func(g Gomega) {
+				g.Expect(env.Client.Get(env.Context, client.ObjectKeyFromObject(node), node)).Should(Succeed())
+				_, ok := lo.Find(node.Spec.Taints, func(t corev1.Taint) bool {
+					return t.MatchTaint(&karpv1.DisruptedNoScheduleTaint)
+				})
+				g.Expect(ok).To(BeTrue())
+				//Reduced polling time from 100 to 50 to mitigate flakes
+				//TODO Investigate root cause of timing sensitivity and restructure test
+			}).WithTimeout(3 * time.Second).WithPolling(50 * time.Millisecond).Should(Succeed())
+
+			// Check that pod remains healthy until termination grace period
+			// subtracting 5s is close enough to say that we waited for the entire terminationGracePeriod
+			// and to stop us flaking from tricky timing bugs
+			env.ConsistentlyExpectHealthyPods(time.Duration(lo.FromPtr(pod.Spec.TerminationGracePeriodSeconds)-5)*time.Second, pod)
+
+			// Both nodeClaim and node should be gone once terminationGracePeriod is reached
+			env.EventuallyExpectNotFound(nodeClaim, node, pod)
+		})
+		It("should delete pod that has a pre-stop hook after termination grace period seconds", func() {
+			pod := test.UnschedulablePod(test.PodOptions{
+				PreStopSleep:                  lo.ToPtr(int64(300)),
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(30)),
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 30"},
+			})
+			env.ExpectCreated(nodeClass, nodePool, pod)
+
+			nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+			node := env.EventuallyExpectCreatedNodeCount("==", 1)[0]
+			env.EventuallyExpectHealthy(pod)
+
+			// Delete the nodeclaim to start the TerminationGracePeriod
+			env.ExpectDeleted(nodeClaim)
+
+			// Eventually the node will be tainted
+			Eventually(func(g Gomega) {
+				g.Expect(env.Client.Get(env.Context, client.ObjectKeyFromObject(node), node)).Should(Succeed())
+				_, ok := lo.Find(node.Spec.Taints, func(t corev1.Taint) bool {
+					return t.MatchTaint(&karpv1.DisruptedNoScheduleTaint)
+				})
+				g.Expect(ok).To(BeTrue())
+			}).WithTimeout(3 * time.Second).WithPolling(100 * time.Millisecond).Should(Succeed())
+
+			env.EventuallyExpectTerminating(pod)
+
+			// Check that pod remains healthy until termination grace period
+			// subtracting 5s is close enough to say that we waited for the entire terminationGracePeriod
+			// and to stop us flaking from tricky timing bugs
+			env.ConsistentlyExpectHealthyPods(time.Duration(lo.FromPtr(pod.Spec.TerminationGracePeriodSeconds)-5)*time.Second, pod)
+
+			// Both nodeClaim and node should be gone once terminationGracePeriod is reached
+			env.EventuallyExpectNotFound(nodeClaim, node, pod)
+		})
+	})
+	It("should terminate the node and the instance on deletion", func() {
+		pod := test.Pod()
+		env.ExpectCreated(nodeClass, nodePool, pod)
+		env.EventuallyExpectHealthy(pod)
+		env.ExpectCreatedNodeCount("==", 1)
+
+		nodes := env.Monitor.CreatedNodes()
+
+		// Pod is deleted so that we don't re-provision after node deletion
+		// NOTE: We have to do this right now to deal with a race condition in nodepool ownership
+		// This can be removed once this race is resolved with the NodePool
+		env.ExpectDeleted(pod)
+
+		// Node is deleted and now should be not found
+		env.ExpectDeleted(nodes[0])
+		env.EventuallyExpectNotFound(nodes[0])
+	})
+	// Pods from Karpenter nodes are expected to drain in the following order:
+	//   1. Non-Critical Non-Daemonset pods
+	//   2. Non-Critical Daemonset pods
+	//   3. Critical Non-Daemonset pods
+	//   4. Critical Daemonset pods
+	// Pods in one group are expected to be fully removed before the next group is executed
+	It("should drain pods on a node in order", func() {
+		daemonSet := test.DaemonSet(test.DaemonSetOptions{
+			Selector: map[string]string{"app": "non-critical-daemonset"},
+			PodOptions: test.PodOptions{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						"drain-test": "true",
+						"app":        "daemonset",
+					},
+				},
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(60)),
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 1000"},
+				PreStopSleep:                  lo.ToPtr(int64(60)),
+				ResourceRequirements:          corev1.ResourceRequirements{Limits: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("1Gi")}},
+			},
+		})
+		nodeCriticalDaemonSet := test.DaemonSet(test.DaemonSetOptions{
+			Selector: map[string]string{"app": "critical-daemonset"},
+			PodOptions: test.PodOptions{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						"drain-test": "true",
+						"app":        "node-critical-daemonset",
+					},
+				},
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(10)), // shorter terminationGracePeriod since it's the last pod
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 1000"},
+				PreStopSleep:                  lo.ToPtr(int64(10)), // shorter preStopSleep since it's the last pod
+				PriorityClassName:             "system-node-critical",
+				ResourceRequirements:          corev1.ResourceRequirements{Limits: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("1Gi")}},
+			},
+		})
+		clusterCriticalDaemonSet := test.DaemonSet(test.DaemonSetOptions{
+			Selector: map[string]string{"app": "critical-daemonset"},
+			PodOptions: test.PodOptions{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						"drain-test": "true",
+						"app":        "cluster-critical-daemonset",
+					},
+				},
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(10)), // shorter terminationGracePeriod since it's the last pod
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 1000"},
+				PreStopSleep:                  lo.ToPtr(int64(10)), // shorter preStopSleep since it's the last pod
+				PriorityClassName:             "system-cluster-critical",
+				ResourceRequirements:          corev1.ResourceRequirements{Limits: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("1Gi")}},
+			},
+		})
+		deployment := test.Deployment(test.DeploymentOptions{
+			Replicas: int32(1),
+			PodOptions: test.PodOptions{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						"drain-test": "true",
+						"app":        "deployment",
+					},
+				},
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(60)),
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 1000"},
+				PreStopSleep:                  lo.ToPtr(int64(60)),
+				ResourceRequirements:          corev1.ResourceRequirements{Limits: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("1Gi")}},
+			},
+		})
+		nodeCriticalDeployment := test.Deployment(test.DeploymentOptions{
+			Replicas: int32(1),
+			PodOptions: test.PodOptions{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						"drain-test": "true",
+						"app":        "node-critical-deployment",
+					},
+				},
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(60)),
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 1000"},
+				PreStopSleep:                  lo.ToPtr(int64(60)),
+				PriorityClassName:             "system-node-critical",
+				ResourceRequirements:          corev1.ResourceRequirements{Limits: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("1Gi")}},
+			},
+		})
+		clusterCriticalDeployment := test.Deployment(test.DeploymentOptions{
+			Replicas: int32(1),
+			PodOptions: test.PodOptions{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						"drain-test": "true",
+						"app":        "cluster-critical-deployment",
+					},
+				},
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(60)),
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 1000"},
+				PreStopSleep:                  lo.ToPtr(int64(60)),
+				PriorityClassName:             "system-cluster-critical",
+				ResourceRequirements:          corev1.ResourceRequirements{Limits: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("1Gi")}},
+			},
+		})
+		env.ExpectCreated(nodeClass, nodePool, daemonSet, nodeCriticalDaemonSet, clusterCriticalDaemonSet, deployment, nodeCriticalDeployment, clusterCriticalDeployment)
+
+		nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+		_ = env.EventuallyExpectCreatedNodeCount("==", 1)[0]
+		env.EventuallyExpectHealthyPodCount(labels.SelectorFromSet(map[string]string{"drain-test": "true"}), 6)
+
+		daemonsetPod := env.ExpectPodsMatchingSelector(labels.SelectorFromSet(map[string]string{"app": "daemonset"}))[0]
+		nodeCriticalDaemonsetPod := env.ExpectPodsMatchingSelector(labels.SelectorFromSet(map[string]string{"app": "node-critical-daemonset"}))[0]
+		clusterCriticalDaemonsetPod := env.ExpectPodsMatchingSelector(labels.SelectorFromSet(map[string]string{"app": "cluster-critical-daemonset"}))[0]
+		deploymentPod := env.ExpectPodsMatchingSelector(labels.SelectorFromSet(map[string]string{"app": "deployment"}))[0]
+		nodeCriticalDeploymentPod := env.ExpectPodsMatchingSelector(labels.SelectorFromSet(map[string]string{"app": "node-critical-deployment"}))[0]
+		clusterCriticalDeploymentPod := env.ExpectPodsMatchingSelector(labels.SelectorFromSet(map[string]string{"app": "cluster-critical-deployment"}))[0]
+
+		env.ExpectDeleted(nodeClaim)
+
+		// Wait for non-critical deployment pod to drain and delete
+		env.EventuallyExpectTerminating(deploymentPod)
+		// We check that other pods are live for 30s since pre-stop sleep and terminationGracePeriod are 60s
+		env.ConsistentlyExpectActivePods(time.Second*30, daemonsetPod, nodeCriticalDeploymentPod, nodeCriticalDaemonsetPod, clusterCriticalDeploymentPod, clusterCriticalDaemonsetPod)
+		env.EventuallyExpectNotFound(deploymentPod)
+
+		// Wait for non-critical daemonset pod to drain and delete
+		env.EventuallyExpectTerminating(daemonsetPod)
+		// We check that other pods are live for 30s since pre-stop sleep and terminationGracePeriod are 60s
+		env.ConsistentlyExpectActivePods(time.Second*30, nodeCriticalDeploymentPod, nodeCriticalDaemonsetPod, clusterCriticalDeploymentPod, clusterCriticalDaemonsetPod)
+		env.EventuallyExpectNotFound(daemonsetPod)
+
+		// Wait for critical deployment pod to drain and delete
+		env.EventuallyExpectTerminating(nodeCriticalDeploymentPod, clusterCriticalDeploymentPod)
+		// We check that other pods are live for 30s since pre-stop sleep and terminationGracePeriod are 60s
+		env.ConsistentlyExpectActivePods(time.Second*30, nodeCriticalDaemonsetPod, clusterCriticalDaemonsetPod)
+		env.EventuallyExpectNotFound(nodeCriticalDeploymentPod, clusterCriticalDeploymentPod)
+
+		// Wait for critical daemonset pod to drain and delete
+		env.EventuallyExpectTerminating(nodeCriticalDaemonsetPod, clusterCriticalDaemonsetPod)
+		env.EventuallyExpectNotFound(nodeCriticalDaemonsetPod, clusterCriticalDaemonsetPod)
+	})
+})


Reviews:
user: login: engedaam
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T18:54:32Z
user: login: engedaam
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-05T18:55:19Z
user: login: jonathan-innis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-06T16:10:05Z
user: login: engedaam
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-06T18:10:07Z
user: login: jonathan-innis
body: /lgtm
/approve
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-13T17:08:16Z

Review comments:
diff_hunk: @@ -0,0 +1,382 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package integration_test
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/samber/lo"
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
+
+	"sigs.k8s.io/karpenter/pkg/test"
+
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	karpv1 "sigs.k8s.io/karpenter/pkg/apis/v1"
+)
+
+var _ = Describe("Termination", func() {
+	Context("Emptiness", func() {
+		var dep *appsv1.Deployment
+		var selector labels.Selector
+		var numPods int
+		BeforeEach(func() {
+			nodePool.Spec.Disruption.ConsolidationPolicy = karpv1.ConsolidationPolicyWhenEmpty
+			nodePool.Spec.Disruption.ConsolidateAfter = karpv1.MustParseNillableDuration("0s")
+
+			numPods = 1
+			dep = test.Deployment(test.DeploymentOptions{
+				Replicas: int32(numPods),
+				PodOptions: test.PodOptions{
+					ObjectMeta: metav1.ObjectMeta{
+						Labels: map[string]string{"app": "large-app"},
+					},
+				},
+			})
+			selector = labels.SelectorFromSet(dep.Spec.Selector.MatchLabels)
+		})
+		Context("Budgets", func() {
+			It("should not allow emptiness if the budget is fully blocking", func() {
+				// We're going to define a budget that doesn't allow any emptiness disruption to happen
+				nodePool.Spec.Disruption.Budgets = []karpv1.Budget{{
+					Nodes: "0",
+				}}
+
+				env.ExpectCreated(nodeClass, nodePool, dep)
+
+				nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+				env.EventuallyExpectCreatedNodeCount("==", 1)
+				env.EventuallyExpectHealthyPodCount(selector, numPods)
+
+				// Delete the deployment so there is nothing running on the node
+				env.ExpectDeleted(dep)
+
+				env.EventuallyExpectConsolidatable(nodeClaim)
+				env.ConsistentlyExpectNoDisruptions(1, time.Minute)
+			})
+			It("should not allow emptiness if the budget is fully blocking during a scheduled time", func() {
+				// We're going to define a budget that doesn't allow any emptiness disruption to happen
+				// This is going to be on a schedule that only lasts 30 minutes, whose window starts 15 minutes before
+				// the current time and extends 15 minutes past the current time
+				// Times need to be in UTC since the karpenter containers were built in UTC time
+				windowStart := time.Now().Add(-time.Minute * 15).UTC()
+				nodePool.Spec.Disruption.Budgets = []karpv1.Budget{{
+					Nodes:    "0",
+					Schedule: lo.ToPtr(fmt.Sprintf("%d %d * * *", windowStart.Minute(), windowStart.Hour())),
+					Duration: &metav1.Duration{Duration: time.Minute * 30},
+				}}
+
+				env.ExpectCreated(nodeClass, nodePool, dep)
+
+				nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+				env.EventuallyExpectCreatedNodeCount("==", 1)
+				env.EventuallyExpectHealthyPodCount(selector, numPods)
+
+				// Delete the deployment so there is nothing running on the node
+				env.ExpectDeleted(dep)
+
+				env.EventuallyExpectConsolidatable(nodeClaim)
+				env.ConsistentlyExpectNoDisruptions(1, time.Minute)
+			})
+		})
+		It("should terminate an empty node", func() {
+			nodePool.Spec.Disruption.ConsolidateAfter = karpv1.MustParseNillableDuration("10s")
+
+			const numPods = 1
+			deployment := test.Deployment(test.DeploymentOptions{Replicas: numPods})
+
+			By("kicking off provisioning for a deployment")
+			env.ExpectCreated(nodeClass, nodePool, deployment)
+			nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+			node := env.EventuallyExpectCreatedNodeCount("==", 1)[0]
+			env.EventuallyExpectHealthyPodCount(labels.SelectorFromSet(deployment.Spec.Selector.MatchLabels), numPods)
+
+			By("making the nodeclaim empty")
+			persisted := deployment.DeepCopy()
+			deployment.Spec.Replicas = lo.ToPtr(int32(0))
+			Expect(env.Client.Patch(env, deployment, client.StrategicMergeFrom(persisted))).To(Succeed())
+
+			env.EventuallyExpectConsolidatable(nodeClaim)
+
+			By("waiting for the nodeclaim to deprovision when past its ConsolidateAfter timeout of 0")
+			nodePool.Spec.Disruption.ConsolidateAfter = karpv1.MustParseNillableDuration("0s")
+			env.ExpectUpdated(nodePool)
+
+			env.EventuallyExpectNotFound(nodeClaim, node)
+		})
+	})
+	Describe("TerminationGracePeriod", func() {
+		BeforeEach(func() {
+			nodePool.Spec.Template.Spec.TerminationGracePeriod = &metav1.Duration{Duration: time.Second * 60}
+		})
+		It("should delete pod with do-not-disrupt when it reaches its terminationGracePeriodSeconds", func() {
+			pod := test.UnschedulablePod(test.PodOptions{ObjectMeta: metav1.ObjectMeta{Annotations: map[string]string{
+				karpv1.DoNotDisruptAnnotationKey: "true",
+			}}, TerminationGracePeriodSeconds: lo.ToPtr(int64(30))})
+			env.ExpectCreated(nodeClass, nodePool, pod)
+
+			nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+			node := env.EventuallyExpectCreatedNodeCount("==", 1)[0]
+			env.EventuallyExpectHealthy(pod)
+
+			// Delete the nodeclaim to start the TerminationGracePeriod
+			env.ExpectDeleted(nodeClaim)
+
+			// Eventually the node will be tainted
+			Eventually(func(g Gomega) {
+				g.Expect(env.Client.Get(env.Context, client.ObjectKeyFromObject(node), node)).Should(Succeed())
+				_, ok := lo.Find(node.Spec.Taints, func(t corev1.Taint) bool {
+					return t.MatchTaint(&karpv1.DisruptedNoScheduleTaint)
+				})
+				g.Expect(ok).To(BeTrue())
+				//Reduced polling time from 100 to 50 to mitigate flakes
+				//TODO Investigate root cause of timing sensitivity and restructure test
+			}).WithTimeout(3 * time.Second).WithPolling(50 * time.Millisecond).Should(Succeed())
+
+			// Check that pod remains healthy until termination grace period
+			// subtracting 5s is close enough to say that we waited for the entire terminationGracePeriod
+			// and to stop us flaking from tricky timing bugs
+			env.ConsistentlyExpectHealthyPods(time.Duration(lo.FromPtr(pod.Spec.TerminationGracePeriodSeconds)-5)*time.Second, pod)
+
+			// Both nodeClaim and node should be gone once terminationGracePeriod is reached
+			env.EventuallyExpectNotFound(nodeClaim, node, pod)
+		})
+		It("should delete pod that has a pre-stop hook after termination grace period seconds", func() {
+			pod := test.UnschedulablePod(test.PodOptions{
+				PreStopSleep:                  lo.ToPtr(int64(300)),
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(30)),
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 30"},
+				ObjectMeta:                    metav1.ObjectMeta{Annotations: map[string]string{"pod-delete.stage.kwok.x-k8s.io/delay": "30s"}},
path: test/suites/integration/termination_test.go
user: login: engedaam
body: Kwok will be simulating the preStop hook behavior. Cloud providers will relay on the PreStopSleep as for all pods
created_at: 2025-05-05T18:54:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,21 @@
+apiVersion: kwok.x-k8s.io/v1alpha1
path: hack/kwok/stages/pod-delete.yaml
user: login: engedaam
body: This stage will be used to mock pod termination behavior from KWOK 
created_at: 2025-05-05T18:55:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,382 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package integration_test
+
+import (
+	"fmt"
+	"time"
+
+	"github.com/samber/lo"
+	appsv1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
+
+	"sigs.k8s.io/karpenter/pkg/test"
+
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	karpv1 "sigs.k8s.io/karpenter/pkg/apis/v1"
+)
+
+var _ = Describe("Termination", func() {
+	Context("Emptiness", func() {
+		var dep *appsv1.Deployment
+		var selector labels.Selector
+		var numPods int
+		BeforeEach(func() {
+			nodePool.Spec.Disruption.ConsolidationPolicy = karpv1.ConsolidationPolicyWhenEmpty
+			nodePool.Spec.Disruption.ConsolidateAfter = karpv1.MustParseNillableDuration("0s")
+
+			numPods = 1
+			dep = test.Deployment(test.DeploymentOptions{
+				Replicas: int32(numPods),
+				PodOptions: test.PodOptions{
+					ObjectMeta: metav1.ObjectMeta{
+						Labels: map[string]string{"app": "large-app"},
+					},
+				},
+			})
+			selector = labels.SelectorFromSet(dep.Spec.Selector.MatchLabels)
+		})
+		Context("Budgets", func() {
+			It("should not allow emptiness if the budget is fully blocking", func() {
+				// We're going to define a budget that doesn't allow any emptiness disruption to happen
+				nodePool.Spec.Disruption.Budgets = []karpv1.Budget{{
+					Nodes: "0",
+				}}
+
+				env.ExpectCreated(nodeClass, nodePool, dep)
+
+				nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+				env.EventuallyExpectCreatedNodeCount("==", 1)
+				env.EventuallyExpectHealthyPodCount(selector, numPods)
+
+				// Delete the deployment so there is nothing running on the node
+				env.ExpectDeleted(dep)
+
+				env.EventuallyExpectConsolidatable(nodeClaim)
+				env.ConsistentlyExpectNoDisruptions(1, time.Minute)
+			})
+			It("should not allow emptiness if the budget is fully blocking during a scheduled time", func() {
+				// We're going to define a budget that doesn't allow any emptiness disruption to happen
+				// This is going to be on a schedule that only lasts 30 minutes, whose window starts 15 minutes before
+				// the current time and extends 15 minutes past the current time
+				// Times need to be in UTC since the karpenter containers were built in UTC time
+				windowStart := time.Now().Add(-time.Minute * 15).UTC()
+				nodePool.Spec.Disruption.Budgets = []karpv1.Budget{{
+					Nodes:    "0",
+					Schedule: lo.ToPtr(fmt.Sprintf("%d %d * * *", windowStart.Minute(), windowStart.Hour())),
+					Duration: &metav1.Duration{Duration: time.Minute * 30},
+				}}
+
+				env.ExpectCreated(nodeClass, nodePool, dep)
+
+				nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+				env.EventuallyExpectCreatedNodeCount("==", 1)
+				env.EventuallyExpectHealthyPodCount(selector, numPods)
+
+				// Delete the deployment so there is nothing running on the node
+				env.ExpectDeleted(dep)
+
+				env.EventuallyExpectConsolidatable(nodeClaim)
+				env.ConsistentlyExpectNoDisruptions(1, time.Minute)
+			})
+		})
+		It("should terminate an empty node", func() {
+			nodePool.Spec.Disruption.ConsolidateAfter = karpv1.MustParseNillableDuration("10s")
+
+			const numPods = 1
+			deployment := test.Deployment(test.DeploymentOptions{Replicas: numPods})
+
+			By("kicking off provisioning for a deployment")
+			env.ExpectCreated(nodeClass, nodePool, deployment)
+			nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+			node := env.EventuallyExpectCreatedNodeCount("==", 1)[0]
+			env.EventuallyExpectHealthyPodCount(labels.SelectorFromSet(deployment.Spec.Selector.MatchLabels), numPods)
+
+			By("making the nodeclaim empty")
+			persisted := deployment.DeepCopy()
+			deployment.Spec.Replicas = lo.ToPtr(int32(0))
+			Expect(env.Client.Patch(env, deployment, client.StrategicMergeFrom(persisted))).To(Succeed())
+
+			env.EventuallyExpectConsolidatable(nodeClaim)
+
+			By("waiting for the nodeclaim to deprovision when past its ConsolidateAfter timeout of 0")
+			nodePool.Spec.Disruption.ConsolidateAfter = karpv1.MustParseNillableDuration("0s")
+			env.ExpectUpdated(nodePool)
+
+			env.EventuallyExpectNotFound(nodeClaim, node)
+		})
+	})
+	Describe("TerminationGracePeriod", func() {
+		BeforeEach(func() {
+			nodePool.Spec.Template.Spec.TerminationGracePeriod = &metav1.Duration{Duration: time.Second * 60}
+		})
+		It("should delete pod with do-not-disrupt when it reaches its terminationGracePeriodSeconds", func() {
+			pod := test.UnschedulablePod(test.PodOptions{ObjectMeta: metav1.ObjectMeta{Annotations: map[string]string{
+				karpv1.DoNotDisruptAnnotationKey: "true",
+			}}, TerminationGracePeriodSeconds: lo.ToPtr(int64(30))})
+			env.ExpectCreated(nodeClass, nodePool, pod)
+
+			nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+			node := env.EventuallyExpectCreatedNodeCount("==", 1)[0]
+			env.EventuallyExpectHealthy(pod)
+
+			// Delete the nodeclaim to start the TerminationGracePeriod
+			env.ExpectDeleted(nodeClaim)
+
+			// Eventually the node will be tainted
+			Eventually(func(g Gomega) {
+				g.Expect(env.Client.Get(env.Context, client.ObjectKeyFromObject(node), node)).Should(Succeed())
+				_, ok := lo.Find(node.Spec.Taints, func(t corev1.Taint) bool {
+					return t.MatchTaint(&karpv1.DisruptedNoScheduleTaint)
+				})
+				g.Expect(ok).To(BeTrue())
+				//Reduced polling time from 100 to 50 to mitigate flakes
+				//TODO Investigate root cause of timing sensitivity and restructure test
+			}).WithTimeout(3 * time.Second).WithPolling(50 * time.Millisecond).Should(Succeed())
+
+			// Check that pod remains healthy until termination grace period
+			// subtracting 5s is close enough to say that we waited for the entire terminationGracePeriod
+			// and to stop us flaking from tricky timing bugs
+			env.ConsistentlyExpectHealthyPods(time.Duration(lo.FromPtr(pod.Spec.TerminationGracePeriodSeconds)-5)*time.Second, pod)
+
+			// Both nodeClaim and node should be gone once terminationGracePeriod is reached
+			env.EventuallyExpectNotFound(nodeClaim, node, pod)
+		})
+		It("should delete pod that has a pre-stop hook after termination grace period seconds", func() {
+			pod := test.UnschedulablePod(test.PodOptions{
+				PreStopSleep:                  lo.ToPtr(int64(300)),
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(30)),
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 30"},
+				ObjectMeta:                    metav1.ObjectMeta{Annotations: map[string]string{"pod-delete.stage.kwok.x-k8s.io/delay": "30s"}},
+			})
+			env.ExpectCreated(nodeClass, nodePool, pod)
+
+			nodeClaim := env.EventuallyExpectCreatedNodeClaimCount("==", 1)[0]
+			node := env.EventuallyExpectCreatedNodeCount("==", 1)[0]
+			env.EventuallyExpectHealthy(pod)
+
+			// Delete the nodeclaim to start the TerminationGracePeriod
+			env.ExpectDeleted(nodeClaim)
+
+			// Eventually the node will be tainted
+			Eventually(func(g Gomega) {
+				g.Expect(env.Client.Get(env.Context, client.ObjectKeyFromObject(node), node)).Should(Succeed())
+				_, ok := lo.Find(node.Spec.Taints, func(t corev1.Taint) bool {
+					return t.MatchTaint(&karpv1.DisruptedNoScheduleTaint)
+				})
+				g.Expect(ok).To(BeTrue())
+			}).WithTimeout(3 * time.Second).WithPolling(100 * time.Millisecond).Should(Succeed())
+
+			env.EventuallyExpectTerminating(pod)
+
+			// Check that pod remains healthy until termination grace period
+			// subtracting 5s is close enough to say that we waited for the entire terminationGracePeriod
+			// and to stop us flaking from tricky timing bugs
+			env.ConsistentlyExpectHealthyPods(time.Duration(lo.FromPtr(pod.Spec.TerminationGracePeriodSeconds)-5)*time.Second, pod)
+
+			// Both nodeClaim and node should be gone once terminationGracePeriod is reached
+			env.EventuallyExpectNotFound(nodeClaim, node, pod)
+		})
+	})
+	It("should terminate the node and the instance on deletion", func() {
+		pod := test.Pod()
+		env.ExpectCreated(nodeClass, nodePool, pod)
+		env.EventuallyExpectHealthy(pod)
+		env.ExpectCreatedNodeCount("==", 1)
+
+		nodes := env.Monitor.CreatedNodes()
+
+		// Pod is deleted so that we don't re-provision after node deletion
+		// NOTE: We have to do this right now to deal with a race condition in nodepool ownership
+		// This can be removed once this race is resolved with the NodePool
+		env.ExpectDeleted(pod)
+
+		// Node is deleted and now should be not found
+		env.ExpectDeleted(nodes[0])
+		env.EventuallyExpectNotFound(nodes[0])
+	})
+	// Pods from Karpenter nodes are expected to drain in the following order:
+	//   1. Non-Critical Non-Daemonset pods
+	//   2. Non-Critical Daemonset pods
+	//   3. Critical Non-Daemonset pods
+	//   4. Critical Daemonset pods
+	// Pods in one group are expected to be fully removed before the next group is executed
+	It("should drain pods on a node in order", func() {
+		daemonSet := test.DaemonSet(test.DaemonSetOptions{
+			Selector: map[string]string{"app": "non-critical-daemonset"},
+			PodOptions: test.PodOptions{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						"drain-test": "true",
+						"app":        "daemonset",
+					},
+					Annotations: map[string]string{
+						"pod-delete.stage.kwok.x-k8s.io/delay": "60s",
+					},
+				},
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(60)),
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 1000"},
+				PreStopSleep:                  lo.ToPtr(int64(60)),
+				ResourceRequirements:          corev1.ResourceRequirements{Limits: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("1Gi")}},
+			},
+		})
+		nodeCriticalDaemonSet := test.DaemonSet(test.DaemonSetOptions{
+			Selector: map[string]string{"app": "critical-daemonset"},
+			PodOptions: test.PodOptions{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						"drain-test": "true",
+						"app":        "node-critical-daemonset",
+					},
+					Annotations: map[string]string{
+						"pod-delete.stage.kwok.x-k8s.io/delay": "10s",
+					},
+				},
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(10)), // shorter terminationGracePeriod since it's the last pod
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 1000"},
+				PreStopSleep:                  lo.ToPtr(int64(10)), // shorter preStopSleep since it's the last pod
+				PriorityClassName:             "system-node-critical",
+				ResourceRequirements:          corev1.ResourceRequirements{Limits: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("1Gi")}},
+			},
+		})
+		clusterCriticalDaemonSet := test.DaemonSet(test.DaemonSetOptions{
+			Selector: map[string]string{"app": "critical-daemonset"},
+			PodOptions: test.PodOptions{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						"drain-test": "true",
+						"app":        "cluster-critical-daemonset",
+					},
+					Annotations: map[string]string{
+						"pod-delete.stage.kwok.x-k8s.io/delay": "10s",
+					},
+				},
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(10)), // shorter terminationGracePeriod since it's the last pod
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 1000"},
+				PreStopSleep:                  lo.ToPtr(int64(10)), // shorter preStopSleep since it's the last pod
+				PriorityClassName:             "system-cluster-critical",
+				ResourceRequirements:          corev1.ResourceRequirements{Limits: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("1Gi")}},
+			},
+		})
+		deployment := test.Deployment(test.DeploymentOptions{
+			Replicas: int32(1),
+			PodOptions: test.PodOptions{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						"drain-test": "true",
+						"app":        "deployment",
+					},
+					Annotations: map[string]string{
+						"pod-delete.stage.kwok.x-k8s.io/delay": "60s",
+					},
+				},
+				TerminationGracePeriodSeconds: lo.ToPtr(int64(60)),
+				Image:                         "alpine:3.20.2",
+				Command:                       []string{"/bin/sh", "-c", "sleep 1000"},
+				PreStopSleep:                  lo.ToPtr(int64(60)),
+				ResourceRequirements:          corev1.ResourceRequirements{Limits: corev1.ResourceList{corev1.ResourceMemory: resource.MustParse("1Gi")}},
+			},
+		})
+		nodeCriticalDeployment := test.Deployment(test.DeploymentOptions{
+			Replicas: int32(1),
+			PodOptions: test.PodOptions{
+				ObjectMeta: metav1.ObjectMeta{
+					Labels: map[string]string{
+						"drain-test": "true",
+						"app":        "node-critical-deployment",
+					},
+					Annotations: map[string]string{
+						"pod-delete.stage.kwok.x-k8s.io/delay": "60s",
path: test/suites/integration/termination_test.go
user: login: jonathan-innis
body: Is it crazy to just inject this annotation onto the pod when you set a preStopSleep? Feels like it doesn't hurt anything to just do that
created_at: 2025-05-06T15:59:36Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,21 @@
+apiVersion: kwok.x-k8s.io/v1alpha1
path: hack/kwok/stages/pod-delete.yaml
user: login: jonathan-innis
body: Why put this in the hack and not make this part of the Kwok package? e.g. `kwok/stages` or even make it part of the chart directly? I don't really see this as a hack? Also, is this something that we built or something that's available upstream from kwok?
created_at: 2025-05-06T16:09:50Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,21 @@
+apiVersion: kwok.x-k8s.io/v1alpha1
path: hack/kwok/stages/pod-delete.yaml
user: login: engedaam
body: The chart is a better place for this CR, and the CR is something that we built wrote using docs/examples from upstream KWOK
created_at: 2025-05-06T18:10:07Z
author_association: CONTRIBUTOR

PR comments:
user: login: coveralls
created_at: 2025-04-28T19:26:45Z
body: ## Pull Request Test Coverage Report for [Build 14980281301](https://coveralls.io/builds/73630489)


### Details

* **0** of **4**   **(0.0%)**  changed or added relevant lines in **1** file are covered.
* **4** unchanged lines in **1** file lost coverage.
* Overall coverage increased (+**0.02%**) to **81.999%**

---

|  Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [pkg/test/pods.go](https://coveralls.io/builds/73630489/source?filename=pkg%2Ftest%2Fpods.go#L160) | 0 | 4 | 0.0%
<!-- | **Total:** | **0** | **4** | **0.0%** | -->

|  Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [pkg/controllers/disruption/consolidation.go](https://coveralls.io/builds/73630489/source?filename=pkg%2Fcontrollers%2Fdisruption%2Fconsolidation.go#L259) | 4 | 85.55% |
<!-- | **Total:** | **4** |  | -->

|  Totals | [![Coverage Status](https://coveralls.io/builds/73630489/badge)](https://coveralls.io/builds/73630489) |
| :-- | --: |
| Change from base [Build 14977025085](https://coveralls.io/builds/73627125): |  0.02% |
| Covered Lines: | 10099 |
| Relevant Lines: | 12316 |

---
##### 💛  - [Coveralls](https://coveralls.io)

author_association: NONE
user: login: k8s-ci-robot
created_at: 2025-05-13T17:08:26Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/karpenter/pull/2173#" title="Author self-approved">engedaam</a>*, *<a href="https://github.com/kubernetes-sigs/karpenter/pull/2173#pullrequestreview-2837543778" title="Approved">jonathan-innis</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkarpenter).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/karpenter/blob/main/OWNERS)~~ [jonathan-innis]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR