Repository: cloudnative-pg/cloudnative-pg

Issue:
[Feature]: Check for nodes cordoned by karpenter
### Is there an existing issue already for this feature request/idea?

- [X] I have searched for an existing issue, and could not find anything. I believe this is a new feature request to be evaluated.

### What problem is this feature going to solve? Why should it be added?

At the moment, when a primary runs on EKS nodes managed by karpenter, you will have to manually failover if a node is replaced or removed. This interrupts the automatic provisioning/deprovisioning of nodes.

Kubernetes uses the "node.kubernetes.io/unschedulable" taint and the "spec.unschedulable" field on a node to mark it as cordoned. The operator checks nodes for this information.
If a primary database is running on a node which was cordoned the operator will promote a secondary on another node.
This allows to evict the cordoned node without blocking the process due to the PDB of the primary.

Karpenter does not rely on "node.kubernetes.io/unschedulable" taint and/or the "spec.unschedulable" field.
Next to other things, the project basically needs to know that karpenter was the one that initiated the tainting.
https://github.com/kubernetes-sigs/karpenter/issues/1152#issuecomment-2032555856

### Describe the solution you'd like

Karpenter taints nodes with: "karpenter.sh/disruption:NoSchedule"
https://karpenter.sh/docs/concepts/disruption/#disruption-controller

It would be nice if the operator could also check for an additional/alternative taint like this.

### Describe alternatives you've considered

You can set "enablePDB: false" - but this is officially not recommended for production environments.

### Additional context

_No response_

### Backport?

N/A

### Are you willing to actively contribute to this feature?

No

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

Hints from issue comments:
@markwebsterUK: Ahhh, so _that's_ what's going on! Would be great to implement this as a custom field.@danielloader: I too found this after working out why my maximum age on karpenter issued nodes wasn't cycling the nodes out and instead blocked by postgres instances. Thanks OP for working this out.@c-thiel: Ran into the same problem, thanks for working it out!
Karpenter community is also arguing about this, but they seem to prefer to stick to their current taint and not leverage the k8s ones:
https://github.com/kubernetes-sigs/karpenter/issues/1152@mark-webster-catalyst: Any movement on this? It's making clusters managed by Karpenter unable to reliably consolidate or roll nodes without user interaction.

EDIT: If anyone gets to it before I have a chance to, this is how the AWS EBS CSI Driver handles it https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/7bacf2d36f397bd098b3388403e8759c480be7e5/cmd/hooks/prestop.go#L47@markwebsterUK: I am unreasonably excited that this has been fixed. Thank you!@quercus-carsten: Yep, i couldn't agree more :). This is going to be awesome - thank you!

Solution patch:
diff --git a/docs/src/operator_conf.md b/docs/src/operator_conf.md
index 7686a87a..b7df8ff0 100644
--- a/docs/src/operator_conf.md
+++ b/docs/src/operator_conf.md
@@ -53,6 +53,7 @@ Name | Description
 `POSTGRES_IMAGE_NAME` | The name of the PostgreSQL image used by default for new clusters. Defaults to the version specified in the operator.
 `PULL_SECRET_NAME` | Name of an additional pull secret to be defined in the operator's namespace and to be used to download images
 `STANDBY_TCP_USER_TIMEOUT` | Defines the [`TCP_USER_TIMEOUT` socket option](https://www.postgresql.org/docs/current/runtime-config-connection.html#GUC-TCP-USER-TIMEOUT) for replication connections from standby instances to the primary. Default is 0 (system's default).
+`DRAIN_TAINTS` | Specifies the taint keys that should be interpreted as indicators of node drain. By default, it includes the taints commonly applied by [kubectl](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/), [Cluster Autoscaler](https://github.com/kubernetes/autoscaler), and [Karpenter](https://github.com/aws/karpenter-provider-aws): `node.kubernetes.io/unschedulable`, `ToBeDeletedByClusterAutoscaler`, `karpenter.sh/disrupted`, `karpenter.sh/disruption`.
 
 Values in `INHERITED_ANNOTATIONS` and `INHERITED_LABELS` support path-like wildcards. For example, the value `example.com/*` will match
 both the value `example.com/one` and `example.com/two`.
diff --git a/internal/cmd/manager/controller/controller.go b/internal/cmd/manager/controller/controller.go
index 2c337d8a..4da98cff 100644
--- a/internal/cmd/manager/controller/controller.go
+++ b/internal/cmd/manager/controller/controller.go
@@ -226,6 +226,7 @@ func RunController(
 		mgr,
 		discoveryClient,
 		pluginRepository,
+		conf.DrainTaints,
 	).SetupWithManager(ctx, mgr, maxConcurrentReconciles); err != nil {
 		setupLog.Error(err, "unable to create controller", "controller", "Cluster")
 		return err
diff --git a/internal/configuration/configuration.go b/internal/configuration/configuration.go
index b9ffb538..28192b57 100644
--- a/internal/configuration/configuration.go
+++ b/internal/configuration/configuration.go
@@ -49,6 +49,27 @@ const (
 	DefaultKubernetesClusterDomain = "cluster.local"
 )
 
+// DefaultDrainTaints is the default list of taints the operator will watch and treat
+// as Unschedule
+var DefaultDrainTaints = []string{
+	// Kubernetes well-known unschedulable taint
+	// See: https://kubernetes.io/docs/reference/labels-annotations-taints/#node-kubernetes-io-unschedulable
+	"node.kubernetes.io/unschedulable",
+
+	// Used by the Kubernetes Cluster Autoscaler
+	// nolint: lll
+	// See: https://github.com/kubernetes/autoscaler/blob/aa1d413ea3bf319b56c7b2e65ade1a028e149439/cluster-autoscaler/cloudprovider/oci/nodepools/consts/annotations.go#L27
+	"ToBeDeletedByClusterAutoscaler",
+
+	// Used by Karpenter termination controller
+	// See: https://karpenter.sh/docs/concepts/disruption/#termination-controller
+	"karpenter.sh/disrupted",
+
+	// Used by Karpenter disruption controller
+	// See: https://karpenter.sh/v0.32/concepts/disruption/#disruption-controller
+	"karpenter.sh/disruption",
+}
+
 // DefaultPluginSocketDir is the default directory where the plugin sockets are located.
 const DefaultPluginSocketDir = "/plugins"
 
@@ -144,6 +165,9 @@ type Data struct {
 	// KubernetesClusterDomain defines the domain suffix for service FQDNs
 	// within the Kubernetes cluster. If left unset, it defaults to `cluster.local`.
 	KubernetesClusterDomain string `json:"kubernetesClusterDomain" env:"KUBERNETES_CLUSTER_DOMAIN"`
+
+	// DrainTaints is a list of taints the operator will watch and treat as Unschedule
+	DrainTaints []string `json:"drainTaints" env:"DRAIN_TAINTS"`
 }
 
 // Current is the configuration used by the operator
@@ -161,6 +185,7 @@ func newDefaultConfig() *Data {
 		ExpiringCheckThreshold:  ExpiringCheckThreshold,
 		StandbyTCPUserTimeout:   0,
 		KubernetesClusterDomain: DefaultKubernetesClusterDomain,
+		DrainTaints:             DefaultDrainTaints,
 	}
 }
 
diff --git a/internal/controller/cluster_controller.go b/internal/controller/cluster_controller.go
index 94683926..c5613d33 100644
--- a/internal/controller/cluster_controller.go
+++ b/internal/controller/cluster_controller.go
@@ -89,6 +89,7 @@ type ClusterReconciler struct {
 	InstanceClient  remote.InstanceClient
 	Plugins         repository.Interface
 
+	drainTaints    []string
 	rolloutManager *rolloutManager.Manager
 }
 
@@ -97,6 +98,7 @@ func NewClusterReconciler(
 	mgr manager.Manager,
 	discoveryClient *discovery.DiscoveryClient,
 	plugins repository.Interface,
+	drainTaints []string,
 ) *ClusterReconciler {
 	return &ClusterReconciler{
 		InstanceClient:  remote.NewClient().Instance(),
@@ -109,6 +111,7 @@ func NewClusterReconciler(
 			configuration.Current.GetClustersRolloutDelay(),
 			configuration.Current.GetInstancesRolloutDelay(),
 		),
+		drainTaints: drainTaints,
 	}
 }
 
@@ -1081,7 +1084,7 @@ func (r *ClusterReconciler) SetupWithManager(ctx context.Context, mgr ctrl.Manag
 		Watches(
 			&corev1.Node{},
 			handler.EnqueueRequestsFromMapFunc(r.mapNodeToClusters()),
-			builder.WithPredicates(nodesPredicate),
+			builder.WithPredicates(r.nodesPredicate()),
 		).
 		Watches(
 			&apiv1.ImageCatalog{},
@@ -1369,11 +1372,13 @@ func filterClustersUsingConfigMap(
 func (r *ClusterReconciler) mapNodeToClusters() handler.MapFunc {
 	return func(ctx context.Context, obj client.Object) []reconcile.Request {
 		node := obj.(*corev1.Node)
+
 		// exit if the node is schedulable (e.g. not cordoned)
 		// could be expanded here with other conditions (e.g. pressure or issues)
-		if !node.Spec.Unschedulable {
+		if !isNodeUnschedulableOrBeingDrained(node, r.drainTaints) {
 			return nil
 		}
+
 		var childPods corev1.PodList
 		// get all the pods handled by the operator on that node
 		err := r.List(ctx, &childPods,
diff --git a/internal/controller/cluster_predicates.go b/internal/controller/cluster_predicates.go
index 42490a2a..8647337f 100644
--- a/internal/controller/cluster_predicates.go
+++ b/internal/controller/cluster_predicates.go
@@ -20,6 +20,8 @@ SPDX-License-Identifier: Apache-2.0
 package controller
 
 import (
+	"slices"
+
 	corev1 "k8s.io/api/core/v1"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/event"
@@ -72,12 +74,42 @@ var (
 			return isUsefulClusterSecret(e.ObjectNew)
 		},
 	}
+)
 
-	nodesPredicate = predicate.Funcs{
+func (r *ClusterReconciler) nodesPredicate() predicate.Funcs {
+	return predicate.Funcs{
 		UpdateFunc: func(e event.UpdateEvent) bool {
 			oldNode, oldOk := e.ObjectOld.(*corev1.Node)
 			newNode, newOk := e.ObjectNew.(*corev1.Node)
-			return oldOk && newOk && oldNode.Spec.Unschedulable != newNode.Spec.Unschedulable
+			if !oldOk || !newOk {
+				return false
+			}
+
+			if oldNode.Spec.Unschedulable != newNode.Spec.Unschedulable {
+				return true
+			}
+
+			// check if any of the watched drain taints have changed.
+			for _, taint := range r.drainTaints {
+				oldTaintIndex := slices.IndexFunc(oldNode.Spec.Taints, func(t corev1.Taint) bool { return t.Key == taint })
+				newTaintIndex := slices.IndexFunc(newNode.Spec.Taints, func(t corev1.Taint) bool { return t.Key == taint })
+
+				switch {
+				case oldTaintIndex == -1 && newTaintIndex == -1:
+					continue
+				case oldTaintIndex == -1 || newTaintIndex == -1:
+					return true
+				}
+
+				// exists in both - check if value or effect is different
+				oldTaint := oldNode.Spec.Taints[oldTaintIndex]
+				newTaint := newNode.Spec.Taints[newTaintIndex]
+				if oldTaint.Value != newTaint.Value || oldTaint.Effect != newTaint.Effect {
+					return true
+				}
+			}
+
+			return false
 		},
 		CreateFunc: func(_ event.CreateEvent) bool {
 			return false
@@ -89,7 +121,7 @@ var (
 			return false
 		},
 	}
-)
+}
 
 func isOwnedByClusterOrSatisfiesPredicate(
 	object client.Object,
diff --git a/internal/controller/replicas.go b/internal/controller/replicas.go
index a4f02db0..5ab1ab6a 100644
--- a/internal/controller/replicas.go
+++ b/internal/controller/replicas.go
@@ -67,7 +67,7 @@ func (r *ClusterReconciler) reconcileTargetPrimaryFromPods(
 	if primary := status.Items[0]; (primary.IsPrimary || (cluster.IsReplica() && primary.IsPodReady)) &&
 		primary.Pod.Name == cluster.Status.CurrentPrimary &&
 		cluster.Status.TargetPrimary == cluster.Status.CurrentPrimary {
-		isPrimaryOnUnschedulableNode, err := r.isNodeUnschedulable(ctx, primary.Node)
+		isPrimaryOnUnschedulableNode, err := r.isNodeUnschedulableOrBeingDrained(ctx, primary.Node)
 		if err != nil {
 			contextLogger.Error(err, "while checking if current primary is on an unschedulable node")
 			// in case of error it's better to proceed with the normal target primary reconciliation
@@ -169,14 +169,31 @@ func (r *ClusterReconciler) reconcileTargetPrimaryForNonReplicaCluster(
 	return mostAdvancedInstance.Pod.Name, r.setPrimaryInstance(ctx, cluster, mostAdvancedInstance.Pod.Name)
 }
 
-// isNodeUnschedulable checks whether a node is set to unschedulable
-func (r *ClusterReconciler) isNodeUnschedulable(ctx context.Context, nodeName string) (bool, error) {
+// isNodeUnschedulableOrBeingDrained checks if a node is currently being drained.
+// nolint: lll
+// Copied from https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/7bacf2d36f397bd098b3388403e8759c480be7e5/cmd/hooks/prestop.go#L91
+func isNodeUnschedulableOrBeingDrained(node *corev1.Node, drainTaints []string) bool {
+	for _, taint := range node.Spec.Taints {
+		if slices.Contains(drainTaints, taint.Key) {
+			return true
+		}
+	}
+
+	return node.Spec.Unschedulable
+}
+
+// isNodeUnschedulableOrBeingDrained checks whether a node is set to unschedulable
+func (r *ClusterReconciler) isNodeUnschedulableOrBeingDrained(
+	ctx context.Context,
+	nodeName string,
+) (bool, error) {
 	var node corev1.Node
 	err := r.Get(ctx, client.ObjectKey{Name: nodeName}, &node)
 	if err != nil {
 		return false, err
 	}
-	return node.Spec.Unschedulable, nil
+
+	return isNodeUnschedulableOrBeingDrained(&node, r.drainTaints), nil
 }
 
 // Pick the next primary on a schedulable node, if the current is running on an unschedulable one,
@@ -220,7 +237,7 @@ func (r *ClusterReconciler) setPrimaryOnSchedulableNode(
 	// Start looking for the next primary among the pods
 	for _, candidate := range podsOnOtherNodes.Items {
 		// If candidate on an unschedulable node too, skip it
-		if unschedulable, _ := r.isNodeUnschedulable(ctx, candidate.Node); unschedulable {
+		if status, _ := r.isNodeUnschedulableOrBeingDrained(ctx, candidate.Node); status {
 			continue
 		}
 
diff --git a/internal/webhook/v1/cluster_webhook.go b/internal/webhook/v1/cluster_webhook.go
index 6ff16540..2e7f79aa 100644
--- a/internal/webhook/v1/cluster_webhook.go
+++ b/internal/webhook/v1/cluster_webhook.go
@@ -1969,7 +1969,7 @@ func (v *ClusterCustomValidator) validateTolerations(r *apiv1.Cluster) field.Err
 	return allErrors
 }
 
-// validateTaintEffect is used from validateTollerations and is a verbatim copy of the code
+// validateTaintEffect is used from validateToleration and is a verbatim copy of the code
 // at https://github.com/kubernetes/kubernetes/blob/4d38d21/pkg/apis/core/validation/validation.go#L3087
 func validateTaintEffect(effect *corev1.TaintEffect, allowEmpty bool, fldPath *field.Path) field.ErrorList {
 	if !allowEmpty && len(*effect) == 0 {



Test patch:
diff --git a/internal/controller/cluster_controller_test.go b/internal/controller/cluster_controller_test.go
index 4f517102..ec4c0214 100644
--- a/internal/controller/cluster_controller_test.go
+++ b/internal/controller/cluster_controller_test.go
@@ -29,6 +29,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
 
 	apiv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1"
+	"github.com/cloudnative-pg/cloudnative-pg/internal/configuration"
 	"github.com/cloudnative-pg/cloudnative-pg/pkg/postgres"
 	"github.com/cloudnative-pg/cloudnative-pg/pkg/reconciler/persistentvolumeclaim"
 	"github.com/cloudnative-pg/cloudnative-pg/pkg/specs"
@@ -273,3 +274,43 @@ var _ = Describe("Updating target primary", func() {
 		})
 	})
 })
+
+var _ = Describe("isNodeUnschedulableOrBeingDrained", func() {
+	node := &corev1.Node{}
+	nodeUnschedulable := &corev1.Node{
+		Spec: corev1.NodeSpec{
+			Unschedulable: true,
+		},
+	}
+	nodeTainted := &corev1.Node{
+		Spec: corev1.NodeSpec{
+			Taints: []corev1.Taint{
+				{
+					Key:    "karpenter.sh/disrupted",
+					Effect: corev1.TaintEffectNoSchedule,
+				},
+			},
+		},
+	}
+	nodeWithUnknownTaint := &corev1.Node{
+		Spec: corev1.NodeSpec{
+			Taints: []corev1.Taint{
+				{
+					Key:    "unknown.io/taint",
+					Effect: corev1.TaintEffectPreferNoSchedule,
+				},
+			},
+		},
+	}
+
+	DescribeTable(
+		"it detects nodes that are unschedulable or being drained",
+		func(node *corev1.Node, expected bool) {
+			Expect(isNodeUnschedulableOrBeingDrained(node, configuration.DefaultDrainTaints)).To(Equal(expected))
+		},
+		Entry("plain node", node, false),
+		Entry("node is unschedulable", nodeUnschedulable, true),
+		Entry("node is tainted", nodeTainted, true),
+		Entry("node has an unknown taint", nodeWithUnknownTaint, false),
+	)
+})
diff --git a/internal/controller/cluster_predicates_test.go b/internal/controller/cluster_predicates_test.go
new file mode 100644
index 00000000..6eb45536
--- /dev/null
+++ b/internal/controller/cluster_predicates_test.go
@@ -0,0 +1,149 @@
+/*
+Copyright © contributors to CloudNativePG, established as
+CloudNativePG a Series of LF Projects, LLC.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+SPDX-License-Identifier: Apache-2.0
+*/
+
+package controller
+
+import (
+	corev1 "k8s.io/api/core/v1"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/event"
+
+	"github.com/cloudnative-pg/cloudnative-pg/internal/configuration"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("nodesPredicate", func() {
+	fakeReconciler := &ClusterReconciler{
+		drainTaints: configuration.DefaultDrainTaints,
+	}
+	nodesPredicateFunctions := fakeReconciler.nodesPredicate()
+
+	pod := &corev1.Pod{}
+	nodeWithNoTaints := &corev1.Node{}
+	unschedulableNode := &corev1.Node{
+		Spec: corev1.NodeSpec{
+			Unschedulable: true,
+		},
+	}
+	nodeWithKarpenterNoSchedulableTaint := &corev1.Node{
+		Spec: corev1.NodeSpec{
+			Taints: []corev1.Taint{
+				{
+					Key:    "karpenter.sh/disrupted",
+					Effect: corev1.TaintEffectNoSchedule,
+				},
+			},
+		},
+	}
+	nodeWithKarpenterNoExecuteTaint := &corev1.Node{
+		Spec: corev1.NodeSpec{
+			Taints: []corev1.Taint{
+				{
+					Key:    "karpenter.sh/disrupted",
+					Effect: corev1.TaintEffectNoExecute,
+				},
+			},
+		},
+	}
+	nodeWithAutoscalerTaint := &corev1.Node{
+		Spec: corev1.NodeSpec{
+			Taints: []corev1.Taint{
+				{
+					Key: "ToBeDeletedByClusterAutoscaler",
+				},
+			},
+		},
+	}
+
+	DescribeTable(
+		"always skips node creation",
+		func(node client.Object, expectedResult bool) {
+			createEvent := event.CreateEvent{
+				Object: node,
+			}
+
+			result := nodesPredicateFunctions.Create(createEvent)
+			Expect(result).To(Equal(expectedResult))
+		},
+		Entry("with a node", nodeWithNoTaints, false),
+		Entry("with a pod", pod, false),
+	)
+
+	DescribeTable(
+		"always skips node delete",
+		func(node client.Object, expectedResult bool) {
+			deleteEvent := event.DeleteEvent{
+				Object: node,
+			}
+
+			result := nodesPredicateFunctions.Delete(deleteEvent)
+			Expect(result).To(Equal(expectedResult))
+		},
+		Entry("with a node", nodeWithNoTaints, false),
+		Entry("with a pod", pod, false),
+	)
+
+	DescribeTable(
+		"always skips generic events",
+		func(node client.Object, expectedResult bool) {
+			genericEvent := event.GenericEvent{
+				Object: node,
+			}
+
+			result := nodesPredicateFunctions.Generic(genericEvent)
+			Expect(result).To(Equal(expectedResult))
+		},
+		Entry("with a node", nodeWithNoTaints, false),
+		Entry("with a pod", pod, false),
+	)
+
+	DescribeTable(
+		"node updates",
+		func(objectOld, objectNew client.Object, expectedResult bool) {
+			updateEventOldToNew := event.UpdateEvent{
+				ObjectOld: objectOld,
+				ObjectNew: objectNew,
+			}
+			updateEventNewToOld := event.UpdateEvent{
+				ObjectOld: objectOld,
+				ObjectNew: objectNew,
+			}
+
+			result := nodesPredicateFunctions.Update(updateEventOldToNew)
+			Expect(result).To(Equal(expectedResult))
+
+			result = nodesPredicateFunctions.Update(updateEventNewToOld)
+			Expect(result).To(Equal(expectedResult))
+		},
+		Entry("with the same node",
+			nodeWithNoTaints, nodeWithNoTaints, false),
+		Entry("with the same tainted node",
+			nodeWithKarpenterNoSchedulableTaint, nodeWithKarpenterNoSchedulableTaint, false),
+		Entry("when a node becomes unschedulable",
+			nodeWithNoTaints, unschedulableNode, true),
+		Entry("when a node gets the karpenter disruption taint",
+			nodeWithNoTaints, nodeWithKarpenterNoSchedulableTaint, true),
+		Entry("when a node gets the karpenter disruption taint value changed",
+			nodeWithKarpenterNoSchedulableTaint, nodeWithKarpenterNoExecuteTaint, true),
+		Entry("when a node taints changed",
+			nodeWithKarpenterNoSchedulableTaint, nodeWithAutoscalerTaint, true),
+	)
+})
diff --git a/tests/e2e/drain_node_test.go b/tests/e2e/drain_node_test.go
index cc79f49c..4cd2ff25 100644
--- a/tests/e2e/drain_node_test.go
+++ b/tests/e2e/drain_node_test.go
@@ -84,6 +84,80 @@ var _ = Describe("E2E Drain Node", Serial, Label(tests.LabelDisruptive, tests.La
 		nodesWithLabels = nil
 	})
 
+	Context("Default maintenance and pvc", func() {
+		const sampleFile = fixturesDir + "/drain-node/cluster-drain-node-karpenter.yaml.template"
+		const clusterName = "cluster-drain-node-karpenter"
+
+		It("will remove the pod from a node tainted by karpenter", func() {
+			const namespacePrefix = "drain-node-e2e-karpeter-initiated"
+
+			var namespace string
+
+			By("creating the namespace and the cluster", func() {
+				var err error
+				namespace, err = env.CreateUniqueTestNamespace(env.Ctx, env.Client, namespacePrefix)
+				Expect(err).ToNot(HaveOccurred())
+				AssertCreateCluster(namespace, clusterName, sampleFile, env)
+			})
+
+			By("waiting for the jobs to be removed", func() {
+				timeout := 180
+				Eventually(func() (int, error) {
+					podList, err := pods.List(env.Ctx, env.Client, namespace)
+					if err != nil {
+						return 0, err
+					}
+					return len(podList.Items), err
+				}, timeout).Should(BeEquivalentTo(3))
+			})
+
+			tableLocator := TableLocator{
+				Namespace:    namespace,
+				ClusterName:  clusterName,
+				DatabaseName: postgres.AppDBName,
+				TableName:    "test",
+			}
+
+			By("loading test data", func() {
+				AssertCreateTestData(env, tableLocator)
+			})
+
+			oldPrimary, err := clusterutils.GetPrimary(env.Ctx, env.Client, namespace, clusterName)
+			Expect(err).ToNot(HaveOccurred())
+
+			By("adding a taint from karpenter to the node containing the primary", func() {
+				cmd := fmt.Sprintf("kubectl taint nodes %v karpenter.sh/disruption:NoSchedule", oldPrimary.Spec.NodeName)
+				_, _, err := run.Run(cmd)
+				Expect(err).ToNot(HaveOccurred())
+			})
+
+			By("verifying failover after drain", func() {
+				timeout := 180
+				Eventually(func() (string, error) {
+					pod, err := clusterutils.GetPrimary(env.Ctx, env.Client, namespace, clusterName)
+					if err != nil {
+						return "", err
+					}
+					return pod.Name, err
+				}, timeout).ShouldNot(BeEquivalentTo(oldPrimary.Name))
+			})
+
+			By("removing karpenter taint from node", func() {
+				cmd := fmt.Sprintf(
+					"kubectl taint nodes %v karpenter.sh/disruption=NoSchedule:NoSchedule-",
+					oldPrimary.Spec.NodeName,
+				)
+				_, _, err := run.Run(cmd)
+				Expect(err).ToNot(HaveOccurred())
+			})
+
+			By("data is present and standbys are streaming", func() {
+				AssertDataExpectedCount(env, tableLocator, 2)
+				AssertClusterStandbysAreStreaming(namespace, clusterName, 140)
+			})
+		})
+	})
+
 	Context("Maintenance on, reuse pvc on", func() {
 		// Initialize empty global namespace variable
 		var namespace string
diff --git a/tests/e2e/fixtures/drain-node/cluster-drain-node-karpenter.yaml.template b/tests/e2e/fixtures/drain-node/cluster-drain-node-karpenter.yaml.template
new file mode 100644
index 00000000..e2685308
--- /dev/null
+++ b/tests/e2e/fixtures/drain-node/cluster-drain-node-karpenter.yaml.template
@@ -0,0 +1,33 @@
+apiVersion: postgresql.cnpg.io/v1
+kind: Cluster
+metadata:
+  name: cluster-drain-node-karpenter
+spec:
+  instances: 3
+
+  affinity:
+      nodeSelector:
+        drain: 'drain'
+
+  postgresql:
+    parameters:
+      log_checkpoints: "on"
+      log_lock_waits: "on"
+      log_min_duration_statement: '1000'
+      log_statement: 'ddl'
+      log_temp_files: '1024'
+      log_autovacuum_min_duration: '1s'
+      log_replication_commands: 'on'
+      wal_receiver_timeout: '2s'
+
+  bootstrap:
+    initdb:
+      database: app
+      owner: appuser
+
+  storage:
+    size: 1Gi
+    storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
+  walStorage:
+    storageClass: ${E2E_DEFAULT_STORAGE_CLASS}
+    size: 1Gi