Repository: kubernetes-sigs/aws-ebs-csi-driver
Pull request #2492: Reliably remove driver startup taint
Author: login: torredil
Labels: name: kind/bug
description: Categorizes issue or PR as related to a bug.
name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: release-note
description: Denotes a PR that will be considered when it comes time to generate release notes.
name: size/XL
description: Denotes a PR that changes 500-999 lines, ignoring generated files.
Created: 2025-05-19T14:07:00Z | Merged: 2025-05-21T20:12:35Z

#### What type of PR is this?

Fixes https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/2447
<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
-->

/kind bug

#### What is this PR about? / Why do we need it?

The current implementation uses a 1 s sleep to look for the `ebs.csi.aws.com/agent‑not‑ready` taint.  If `eks:node‑manager` (or any other controller) applies the taint *after* that initial probe, the driver never retries and the node stays unschedulable until the pod is restarted.

This patch replaces that logic with a short‑lived, event‑driven Node informer filtered to the current node as a better mechanism for reliably removing the startup taint.

Note: Some minor refactoring was done in the existing unit tests because we no longer need to mock the call to retrieve the node object in `removeNotReadyTaint` -  now, the node object gets passed in directly into `removeNotReadyTaint` from the informer's event handling.

#### How was this change tested?

- `make verify && make test`.
- Manually, a scenario where where the taint is present before node pod starts, and a scenario where the taint is added after node pod starts.


Driver logs:
```
kubectl logs ebs-csi-node-hjq2g -n kube-system                                               ✔  torredil-Isengard@dev.us-east-1.eksctl.io ○  torredil@dev-dsk-torredil-1d-e1003bcc
Defaulted container "ebs-plugin" out of: ebs-plugin, node-driver-registrar, liveness-probe
I0519 13:43:52.611344       1 main.go:153] "Initializing metadata"
I0519 13:43:52.611404       1 metadata.go:54] "Attempting to retrieve instance metadata from IMDS"
I0519 13:43:52.615709       1 metadata.go:57] "Retrieved metadata from IMDS"
I0519 13:43:52.616754       1 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false
I0519 13:43:52.616768       1 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true
I0519 13:43:52.616772       1 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false
I0519 13:43:52.616776       1 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false
I0519 13:43:52.616780       1 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false
I0519 13:43:52.617071       1 driver.go:69] "Driver Information" Driver="ebs.csi.aws.com" Version="v1.43.0"
I0519 13:43:52.629161       1 reflector.go:430] "Caches populated" type="*v1.Node" reflector="/gomodcache/k8s.io/client-go@v0.33.0/tools/cache/reflector.go:285"
I0519 13:43:54.167774       1 node.go:1009] "CSINode Allocatable value is set" nodeName="ip-192-168-15-218.ec2.internal" count=26
I0519 13:43:54.184703       1 node.go:996] "Removed taint(s) from local node" node="ip-192-168-15-218.ec2.internal"
I0519 13:43:54.184719       1 node.go:877] "Successfully removed agent‑not‑ready taint; stopping watcher"
```

```
kubectl describe node ip-192-168-15-218.ec2.internal | grep "Taints"            
Taints:             <none>
```

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, enter your extended release note in the block below.
-->
```release-note
Reliably remove the `ebs.csi.aws.com/agent-not-ready` taint at start‑up using a short‑lived informer, eliminating a race condition that could leave nodes unschedulable.
```



Diff:
diff --git a/charts/aws-ebs-csi-driver/templates/clusterrole-csi-node.yaml b/charts/aws-ebs-csi-driver/templates/clusterrole-csi-node.yaml
index 2b7295aafd..c04310f606 100644
--- a/charts/aws-ebs-csi-driver/templates/clusterrole-csi-node.yaml
+++ b/charts/aws-ebs-csi-driver/templates/clusterrole-csi-node.yaml
@@ -8,7 +8,7 @@ metadata:
 rules:
   - apiGroups: [""]
     resources: ["nodes"]
-    verbs: ["get", "patch"]
+    verbs: ["get", "patch", "list", "watch"]
   - apiGroups: ["storage.k8s.io"]
     resources: ["volumeattachments"]
     verbs: ["get", "list", "watch"]
diff --git a/deploy/kubernetes/base/clusterrole-csi-node.yaml b/deploy/kubernetes/base/clusterrole-csi-node.yaml
index f4d08b63f7..77bac09e99 100644
--- a/deploy/kubernetes/base/clusterrole-csi-node.yaml
+++ b/deploy/kubernetes/base/clusterrole-csi-node.yaml
@@ -9,7 +9,7 @@ metadata:
 rules:
   - apiGroups: [""]
     resources: ["nodes"]
-    verbs: ["get", "patch"]
+    verbs: ["get", "patch", "list", "watch"]
   - apiGroups: ["storage.k8s.io"]
     resources: ["volumeattachments"]
     verbs: ["get", "list", "watch"]
diff --git a/pkg/driver/node.go b/pkg/driver/node.go
index f1a4a4f72a..c1fb653089 100644
--- a/pkg/driver/node.go
+++ b/pkg/driver/node.go
@@ -38,7 +38,9 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	k8stypes "k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/wait"
+	"k8s.io/client-go/informers"
 	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/tools/cache"
 	"k8s.io/klog/v2"
 )
 
@@ -66,15 +68,11 @@ var (
 		csi.NodeServiceCapability_RPC_EXPAND_VOLUME,
 		csi.NodeServiceCapability_RPC_GET_VOLUME_STATS,
 	}
+)
 
-	// taintRemovalInitialDelay is the initial delay for node taint removal.
-	taintRemovalInitialDelay = 1 * time.Second
-	// taintRemovalBackoff is the exponential backoff configuration for node taint removal.
-	taintRemovalBackoff = wait.Backoff{
-		Duration: 500 * time.Millisecond,
-		Factor:   2,
-		Steps:    10, // Max delay = 0.5 * 2^9 = ~4 minutes
-	}
+const (
+	// taintWatcherDuration is the maximum duration for the not-ready taint watcher to run.
+	taintWatcherDuration = 1 * time.Minute
 )
 
 // NodeService represents the node service of CSI driver.
@@ -89,11 +87,9 @@ type NodeService struct {
 // NewNodeService creates a new node service.
 func NewNodeService(o *Options, md metadata.MetadataService, m mounter.Mounter, k kubernetes.Interface) *NodeService {
 	if k != nil {
-		// Remove taint from node to indicate driver startup success
-		// This is done at the last possible moment to prevent race conditions or false positive removals
-		time.AfterFunc(taintRemovalInitialDelay, func() {
-			removeTaintInBackground(k, taintRemovalBackoff, removeNotReadyTaint)
-		})
+		// Watch for the agent‑not‑ready taint for up to one minute and remove it
+		// as soon as allocatable is available.
+		startNotReadyTaintWatcher(k, taintWatcherDuration)
 	}
 
 	return &NodeService{
@@ -848,45 +844,111 @@ func collectMountOptions(fsType string, mntFlags []string) []string {
 	return options
 }
 
-// Struct for JSON patch operations.
-type JSONPatch struct {
-	OP    string      `json:"op,omitempty"`
-	Path  string      `json:"path,omitempty"`
-	Value interface{} `json:"value"`
-}
+// startNotReadyTaintWatcher launches a short‑lived Node informer that removes the
+// ebs.csi.aws.com/agent‑not‑ready taint. The informer is stopped after maxWatchDuration.
+func startNotReadyTaintWatcher(clientset kubernetes.Interface, maxWatchDuration time.Duration) {
+	nodeName := os.Getenv("CSI_NODE_NAME")
+	if nodeName == "" {
+		klog.V(4).InfoS("CSI_NODE_NAME missing, skipping taint watcher")
+		return
+	}
+
+	factory := informers.NewSharedInformerFactoryWithOptions(
+		clientset,
+		0, // No resync
+		informers.WithTweakListOptions(func(lo *metav1.ListOptions) {
+			lo.FieldSelector = "metadata.name=" + nodeName
+		}),
+	)
+	informer := factory.Core().V1().Nodes().Informer()
+
+	attemptTaintRemoval := func(n *corev1.Node) {
+		if !hasNotReadyTaint(n) {
+			return
+		}
+
+		backoff := wait.Backoff{
+			Duration: 2 * time.Second,
+			Factor:   1.5,
+			Steps:    5,
+		}
+
+		ctx, cancel := context.WithTimeout(context.Background(), maxWatchDuration)
+		defer cancel()
+
+		err := wait.ExponentialBackoffWithContext(ctx, backoff, func(ctx context.Context) (bool, error) {
+			if err := removeNotReadyTaint(ctx, clientset, n); err != nil {
+				klog.ErrorS(err, "Failed to remove agent-not-ready taint, retrying", "node", n.Name)
+				return false, nil // Continue retrying
+			}
+			klog.V(2).InfoS("Successfully removed agent-not-ready taint", "node", n.Name)
+			return true, nil
+		})
 
-// removeTaintInBackground is a goroutine that retries removeNotReadyTaint with exponential backoff.
-func removeTaintInBackground(k8sClient kubernetes.Interface, backoff wait.Backoff, removalFunc func(kubernetes.Interface) error) {
-	backoffErr := wait.ExponentialBackoff(backoff, func() (bool, error) {
-		err := removalFunc(k8sClient)
 		if err != nil {
-			klog.ErrorS(err, "Unexpected failure when attempting to remove node taint(s)")
-			return false, nil
+			klog.ErrorS(err, "Timed out trying to remove agent-not-ready taint", "node", n.Name)
+		}
+	}
+
+	if _, err := informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
+		AddFunc: func(obj interface{}) {
+			if n, ok := obj.(*corev1.Node); ok {
+				attemptTaintRemoval(n)
+			}
+		},
+		UpdateFunc: func(_, newObj interface{}) {
+			if n, ok := newObj.(*corev1.Node); ok {
+				attemptTaintRemoval(n)
+			}
+		},
+	}); err != nil {
+		klog.ErrorS(err, "Taint‑watcher: failed to add event handler")
+		return
+	}
+
+	stopCh := make(chan struct{})
+
+	go func() {
+		factory.Start(stopCh)
+
+		if ok := cache.WaitForCacheSync(stopCh, informer.HasSynced); !ok {
+			klog.ErrorS(nil, "Taint-watcher: cache sync failed")
 		}
-		return true, nil
-	})
 
-	if backoffErr != nil {
-		klog.ErrorS(backoffErr, "Retries exhausted, giving up attempting to remove node taint(s)")
+		// Immediate scan in case the taint is already present and no event fires.
+		if obj, exists, err := informer.GetStore().GetByKey(nodeName); err == nil && exists {
+			if n, ok := obj.(*corev1.Node); ok {
+				attemptTaintRemoval(n)
+			}
+		}
+
+		<-time.After(maxWatchDuration)
+		klog.V(8).InfoS("Taint-watcher timeout reached; stopping")
+		close(stopCh)
+	}()
+}
+
+func hasNotReadyTaint(n *corev1.Node) bool {
+	for _, t := range n.Spec.Taints {
+		if t.Key == AgentNotReadyNodeTaintKey {
+			return true
+		}
 	}
+	return false
+}
+
+// Struct for JSON patch operations.
+type JSONPatch struct {
+	OP    string      `json:"op,omitempty"`
+	Path  string      `json:"path,omitempty"`
+	Value interface{} `json:"value"`
 }
 
 // removeNotReadyTaint removes the taint ebs.csi.aws.com/agent-not-ready from the local node
 // This taint can be optionally applied by users to prevent startup race conditions such as
 // https://github.com/kubernetes/kubernetes/issues/95911
-func removeNotReadyTaint(clientset kubernetes.Interface) error {
-	nodeName := os.Getenv("CSI_NODE_NAME")
-	if nodeName == "" {
-		klog.V(4).InfoS("CSI_NODE_NAME missing, skipping taint removal")
-		return nil
-	}
-
-	node, err := clientset.CoreV1().Nodes().Get(context.Background(), nodeName, metav1.GetOptions{})
-	if err != nil {
-		return err
-	}
-
-	err = checkAllocatable(clientset, nodeName)
+func removeNotReadyTaint(ctx context.Context, clientset kubernetes.Interface, node *corev1.Node) error {
+	err := checkAllocatable(ctx, clientset, node.Name)
 	if err != nil {
 		return err
 	}
@@ -923,16 +985,16 @@ func removeNotReadyTaint(clientset kubernetes.Interface) error {
 		return err
 	}
 
-	_, err = clientset.CoreV1().Nodes().Patch(context.Background(), nodeName, k8stypes.JSONPatchType, patch, metav1.PatchOptions{})
+	_, err = clientset.CoreV1().Nodes().Patch(ctx, node.Name, k8stypes.JSONPatchType, patch, metav1.PatchOptions{})
 	if err != nil {
 		return err
 	}
-	klog.InfoS("Removed taint(s) from local node", "node", nodeName)
+	klog.InfoS("Removed taint(s) from local node", "node", node.Name)
 	return nil
 }
 
-func checkAllocatable(clientset kubernetes.Interface, nodeName string) error {
-	csiNode, err := clientset.StorageV1().CSINodes().Get(context.Background(), nodeName, metav1.GetOptions{})
+func checkAllocatable(ctx context.Context, clientset kubernetes.Interface, nodeName string) error {
+	csiNode, err := clientset.StorageV1().CSINodes().Get(ctx, nodeName, metav1.GetOptions{})
 	if err != nil {
 		return fmt.Errorf("isAllocatableSet: failed to get CSINode for %s: %w", nodeName, err)
 	}
diff --git a/pkg/driver/node_test.go b/pkg/driver/node_test.go
index 1e29a3f323..62543d2efd 100644
--- a/pkg/driver/node_test.go
+++ b/pkg/driver/node_test.go
@@ -14,10 +14,10 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
-//nolint:forcetypeassert
 package driver
 
 import (
+	"context"
 	"errors"
 	"fmt"
 	"reflect"
@@ -31,14 +31,15 @@ import (
 	"github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/cloud/metadata"
 	"github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver/internal"
 	"github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/mounter"
-	"github.com/stretchr/testify/assert"
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/status"
 	corev1 "k8s.io/api/core/v1"
 	v1 "k8s.io/api/storage/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	k8stypes "k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/util/wait"
 	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/kubernetes/fake"
 )
 
 func TestNewNodeService(t *testing.T) {
@@ -2498,85 +2499,62 @@ func TestRemoveNotReadyTaint(t *testing.T) {
 	nodeName := "test-node-123"
 	testCases := []struct {
 		name      string
-		setup     func(t *testing.T, mockCtl *gomock.Controller) func() (kubernetes.Interface, error)
+		setup     func(t *testing.T, mockCtl *gomock.Controller) (kubernetes.Interface, *corev1.Node)
 		expResult error
 	}{
 		{
-			name: "failed to get node",
-			setup: func(t *testing.T, mockCtl *gomock.Controller) func() (kubernetes.Interface, error) {
+			name: "checkAllocatable returns error",
+			setup: func(t *testing.T, mockCtl *gomock.Controller) (kubernetes.Interface, *corev1.Node) {
 				t.Helper()
-				t.Setenv("CSI_NODE_NAME", nodeName)
-				getNodeMock, _ := getNodeMock(mockCtl, nodeName, nil, errors.New("Failed to get node!"))
 
-				return func() (kubernetes.Interface, error) {
-					return getNodeMock, nil
+				node := &corev1.Node{
+					ObjectMeta: metav1.ObjectMeta{
+						Name: nodeName,
+					},
 				}
-			},
-			expResult: errors.New("Failed to get node!"),
-		},
-		{
-			name: "no taints to remove",
-			setup: func(t *testing.T, mockCtl *gomock.Controller) func() (kubernetes.Interface, error) {
-				t.Helper()
-				t.Setenv("CSI_NODE_NAME", nodeName)
-				getNodeMock, _ := getNodeMock(mockCtl, nodeName, &corev1.Node{}, nil)
 
+				mockClient := NewMockKubernetesClient(mockCtl)
 				storageV1Mock := NewMockStorageV1Interface(mockCtl)
-				getNodeMock.(*MockKubernetesClient).EXPECT().StorageV1().Return(storageV1Mock).AnyTimes()
+				mockClient.EXPECT().StorageV1().Return(storageV1Mock).AnyTimes()
 
 				csiNodesMock := NewMockCSINodeInterface(mockCtl)
 				storageV1Mock.EXPECT().CSINodes().Return(csiNodesMock).Times(1)
 
-				count := int32(1)
-
-				mockCSINode := &v1.CSINode{
-					ObjectMeta: metav1.ObjectMeta{
-						Name: "test-node-123",
-					},
-					Spec: v1.CSINodeSpec{
-						Drivers: []v1.CSINodeDriver{
-							{
-								Name: DriverName,
-								Allocatable: &v1.VolumeNodeResources{
-									Count: &count,
-								},
-							},
-						},
-					},
-				}
-
 				csiNodesMock.EXPECT().
-					Get(gomock.Any(), gomock.Eq("test-node-123"), gomock.Any()).
-					Return(mockCSINode, nil).
+					Get(gomock.Any(), gomock.Eq(nodeName), gomock.Any()).
+					Return(nil, errors.New("failed to get CSINode")).
 					Times(1)
 
-				return func() (kubernetes.Interface, error) {
-					return getNodeMock, nil
-				}
+				return mockClient, node
 			},
-			expResult: nil,
+			expResult: fmt.Errorf("isAllocatableSet: failed to get CSINode for %s: failed to get CSINode", nodeName),
 		},
 		{
-			name: "failed to patch node",
-			setup: func(t *testing.T, mockCtl *gomock.Controller) func() (kubernetes.Interface, error) {
+			name: "no taints to remove",
+			setup: func(t *testing.T, mockCtl *gomock.Controller) (kubernetes.Interface, *corev1.Node) {
 				t.Helper()
-				t.Setenv("CSI_NODE_NAME", nodeName)
-				getNodeMock, mockNode := getNodeMock(mockCtl, nodeName, &corev1.Node{
+
+				node := &corev1.Node{
 					ObjectMeta: metav1.ObjectMeta{
 						Name: nodeName,
 					},
 					Spec: corev1.NodeSpec{
 						Taints: []corev1.Taint{
 							{
-								Key:    AgentNotReadyNodeTaintKey,
+								Key:    "some-other-taint",
+								Effect: corev1.TaintEffectNoSchedule,
+							},
+							{
+								Key:    "another-taint",
 								Effect: corev1.TaintEffectNoExecute,
 							},
 						},
 					},
-				}, nil)
+				}
 
+				mockClient := NewMockKubernetesClient(mockCtl)
 				storageV1Mock := NewMockStorageV1Interface(mockCtl)
-				getNodeMock.(*MockKubernetesClient).EXPECT().StorageV1().Return(storageV1Mock).AnyTimes()
+				mockClient.EXPECT().StorageV1().Return(storageV1Mock).AnyTimes()
 
 				csiNodesMock := NewMockCSINodeInterface(mockCtl)
 				storageV1Mock.EXPECT().CSINodes().Return(csiNodesMock).Times(1)
@@ -2589,8 +2567,7 @@ func TestRemoveNotReadyTaint(t *testing.T) {
 					Spec: v1.CSINodeSpec{
 						Drivers: []v1.CSINodeDriver{
 							{
-								Name:   DriverName,
-								NodeID: nodeName,
+								Name: DriverName,
 								Allocatable: &v1.VolumeNodeResources{
 									Count: &count,
 								},
@@ -2604,23 +2581,16 @@ func TestRemoveNotReadyTaint(t *testing.T) {
 					Return(mockCSINode, nil).
 					Times(1)
 
-				mockNode.EXPECT().
-					Patch(gomock.Any(), gomock.Eq(nodeName), gomock.Any(), gomock.Any(), gomock.Any()).
-					Return(nil, errors.New("Failed to patch node!")).
-					Times(1)
-
-				return func() (kubernetes.Interface, error) {
-					return getNodeMock, nil
-				}
+				return mockClient, node
 			},
-			expResult: errors.New("Failed to patch node!"),
+			expResult: nil,
 		},
 		{
-			name: "success",
-			setup: func(t *testing.T, mockCtl *gomock.Controller) func() (kubernetes.Interface, error) {
+			name: "successfully removes taint",
+			setup: func(t *testing.T, mockCtl *gomock.Controller) (kubernetes.Interface, *corev1.Node) {
 				t.Helper()
-				t.Setenv("CSI_NODE_NAME", nodeName)
-				getNodeMock, mockNode := getNodeMock(mockCtl, nodeName, &corev1.Node{
+
+				node := &corev1.Node{
 					ObjectMeta: metav1.ObjectMeta{
 						Name: nodeName,
 					},
@@ -2630,12 +2600,17 @@ func TestRemoveNotReadyTaint(t *testing.T) {
 								Key:    AgentNotReadyNodeTaintKey,
 								Effect: corev1.TaintEffectNoSchedule,
 							},
+							{
+								Key:    "some-other-taint",
+								Effect: corev1.TaintEffectNoExecute,
+							},
 						},
 					},
-				}, nil)
+				}
+				mockClient := NewMockKubernetesClient(mockCtl)
 
 				storageV1Mock := NewMockStorageV1Interface(mockCtl)
-				getNodeMock.(*MockKubernetesClient).EXPECT().StorageV1().Return(storageV1Mock).AnyTimes()
+				mockClient.EXPECT().StorageV1().Return(storageV1Mock).AnyTimes()
 
 				csiNodesMock := NewMockCSINodeInterface(mockCtl)
 				storageV1Mock.EXPECT().CSINodes().Return(csiNodesMock).Times(1)
@@ -2648,8 +2623,7 @@ func TestRemoveNotReadyTaint(t *testing.T) {
 					Spec: v1.CSINodeSpec{
 						Drivers: []v1.CSINodeDriver{
 							{
-								Name:   DriverName,
-								NodeID: nodeName,
+								Name: DriverName,
 								Allocatable: &v1.VolumeNodeResources{
 									Count: &count,
 								},
@@ -2663,208 +2637,145 @@ func TestRemoveNotReadyTaint(t *testing.T) {
 					Return(mockCSINode, nil).
 					Times(1)
 
-				mockNode.EXPECT().
-					Patch(gomock.Any(), gomock.Eq(nodeName), gomock.Any(), gomock.Any(), gomock.Any()).
-					Return(nil, nil).
+				coreV1Mock := NewMockCoreV1Interface(mockCtl)
+				mockClient.EXPECT().CoreV1().Return(coreV1Mock).AnyTimes()
+
+				nodesMock := NewMockNodeInterface(mockCtl)
+				coreV1Mock.EXPECT().Nodes().Return(nodesMock).Times(1)
+
+				nodesMock.EXPECT().
+					Patch(
+						gomock.Any(),
+						gomock.Eq(nodeName),
+						gomock.Eq(k8stypes.JSONPatchType),
+						gomock.Any(),
+						gomock.Any(),
+					).
+					Return(node, nil).
 					Times(1)
 
-				return func() (kubernetes.Interface, error) {
-					return getNodeMock, nil
-				}
+				return mockClient, node
 			},
 			expResult: nil,
 		},
-		{
-			name: "failed to get CSINode",
-			setup: func(t *testing.T, mockCtl *gomock.Controller) func() (kubernetes.Interface, error) {
-				t.Helper()
-				t.Setenv("CSI_NODE_NAME", nodeName)
-				getNodeMock, _ := getNodeMock(mockCtl, nodeName, &corev1.Node{
-					ObjectMeta: metav1.ObjectMeta{
-						Name: nodeName,
-					},
-					Spec: corev1.NodeSpec{
-						Taints: []corev1.Taint{
-							{
-								Key:    AgentNotReadyNodeTaintKey,
-								Effect: corev1.TaintEffectNoSchedule,
-							},
-						},
-					},
-				}, nil)
-
-				storageV1Mock := NewMockStorageV1Interface(mockCtl)
-				getNodeMock.(*MockKubernetesClient).EXPECT().StorageV1().Return(storageV1Mock).AnyTimes()
+	}
 
-				csiNodesMock := NewMockCSINodeInterface(mockCtl)
-				storageV1Mock.EXPECT().CSINodes().Return(csiNodesMock).Times(1)
+	for _, tc := range testCases {
+		t.Run(tc.name, func(t *testing.T) {
+			mockCtl := gomock.NewController(t)
+			defer mockCtl.Finish()
 
-				csiNodesMock.EXPECT().
-					Get(gomock.Any(), gomock.Eq(nodeName), gomock.Any()).
-					Return(nil, errors.New("Failed to get CSINode")).
-					Times(1)
+			client, node := tc.setup(t, mockCtl)
+			result := removeNotReadyTaint(t.Context(), client, node)
 
-				return func() (kubernetes.Interface, error) {
-					return getNodeMock, nil
+			if tc.expResult == nil {
+				if result != nil {
+					t.Fatalf("expected no error, got %v", result)
 				}
-			},
-			expResult: fmt.Errorf("isAllocatableSet: failed to get CSINode for %s: Failed to get CSINode", nodeName),
-		},
-		{
-			name: "allocatable value not set for driver on node",
-			setup: func(t *testing.T, mockCtl *gomock.Controller) func() (kubernetes.Interface, error) {
-				t.Helper()
-				t.Setenv("CSI_NODE_NAME", nodeName)
-				getNodeMock, _ := getNodeMock(mockCtl, nodeName, &corev1.Node{
-					ObjectMeta: metav1.ObjectMeta{
-						Name: nodeName,
-					},
-					Spec: corev1.NodeSpec{
-						Taints: []corev1.Taint{
-							{
-								Key:    AgentNotReadyNodeTaintKey,
-								Effect: corev1.TaintEffectNoSchedule,
-							},
-						},
-					},
-				}, nil)
-
-				storageV1Mock := NewMockStorageV1Interface(mockCtl)
-				getNodeMock.(*MockKubernetesClient).EXPECT().StorageV1().Return(storageV1Mock).AnyTimes()
-
-				csiNodesMock := NewMockCSINodeInterface(mockCtl)
-				storageV1Mock.EXPECT().CSINodes().Return(csiNodesMock).Times(1)
-
-				mockCSINode := &v1.CSINode{
-					ObjectMeta: metav1.ObjectMeta{
-						Name: nodeName,
-					},
-					Spec: v1.CSINodeSpec{
-						Drivers: []v1.CSINodeDriver{
-							{
-								Name:   DriverName,
-								NodeID: nodeName,
-							},
-						},
-					},
+			} else {
+				if result == nil {
+					t.Fatalf("expected error %v, got nil", tc.expResult)
+				}
+				if result.Error() != tc.expResult.Error() {
+					t.Fatalf("expected error %v, got %v", tc.expResult, result)
 				}
+			}
+		})
+	}
+}
 
-				csiNodesMock.EXPECT().
-					Get(gomock.Any(), gomock.Eq(nodeName), gomock.Any()).
-					Return(mockCSINode, nil).
-					Times(1)
+func TestStartNotReadyTaintWatcher(t *testing.T) {
+	const nodeName = "ip-10-0-0-1"
+	t.Setenv("CSI_NODE_NAME", nodeName)
 
-				return func() (kubernetes.Interface, error) {
-					return getNodeMock, nil
-				}
-			},
-			expResult: fmt.Errorf("isAllocatableSet: allocatable value not set for driver on node %s", nodeName),
+	tests := []struct {
+		name          string
+		seedWithTaint bool
+		addTaintLater bool
+		expectPatched bool
+	}{
+		{
+			name:          "taint-present-at-startup",
+			seedWithTaint: true,
+			expectPatched: true,
 		},
 		{
-			name: "driver not found on node",
-			setup: func(t *testing.T, mockCtl *gomock.Controller) func() (kubernetes.Interface, error) {
-				t.Helper()
-				t.Setenv("CSI_NODE_NAME", nodeName)
-				getNodeMock, _ := getNodeMock(mockCtl, nodeName, &corev1.Node{
-					ObjectMeta: metav1.ObjectMeta{
-						Name: nodeName,
-					},
-					Spec: corev1.NodeSpec{
-						Taints: []corev1.Taint{
-							{
-								Key:    AgentNotReadyNodeTaintKey,
-								Effect: corev1.TaintEffectNoSchedule,
-							},
-						},
-					},
-				}, nil)
-
-				storageV1Mock := NewMockStorageV1Interface(mockCtl)
-				getNodeMock.(*MockKubernetesClient).EXPECT().StorageV1().Return(storageV1Mock).AnyTimes()
+			name:          "taint-added-later",
+			addTaintLater: true,
+			expectPatched: true,
+		},
+	}
 
-				csiNodesMock := NewMockCSINodeInterface(mockCtl)
-				storageV1Mock.EXPECT().CSINodes().Return(csiNodesMock).Times(1)
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			node := &corev1.Node{ObjectMeta: metav1.ObjectMeta{Name: nodeName}}
+			if tc.seedWithTaint {
+				node.Spec.Taints = []corev1.Taint{{
+					Key:    AgentNotReadyNodeTaintKey,
+					Value:  "true",
+					Effect: corev1.TaintEffectNoSchedule,
+				}}
+			}
 
-				mockCSINode := &v1.CSINode{
-					ObjectMeta: metav1.ObjectMeta{
-						Name: nodeName,
+			count := int32(1)
+			csiNode := &v1.CSINode{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: nodeName,
+				},
+				Spec: v1.CSINodeSpec{
+					Drivers: []v1.CSINodeDriver{
+						{
+							Name: DriverName,
+							Allocatable: &v1.VolumeNodeResources{
+								Count: &count,
+							},
+						},
 					},
-					Spec: v1.CSINodeSpec{},
-				}
+				},
+			}
 
-				csiNodesMock.EXPECT().
-					Get(gomock.Any(), gomock.Eq(nodeName), gomock.Any()).
-					Return(mockCSINode, nil).
-					Times(1)
+			client := fake.NewSimpleClientset(node, csiNode)
 
-				return func() (kubernetes.Interface, error) {
-					return getNodeMock, nil
-				}
-			},
-			expResult: fmt.Errorf("isAllocatableSet: driver not found on node %s", nodeName),
-		},
-	}
-	for _, tc := range testCases {
-		t.Run(tc.name, func(t *testing.T) {
-			mockCtl := gomock.NewController(t)
-			defer mockCtl.Finish()
+			startNotReadyTaintWatcher(client, 1*time.Second)
 
-			k8sClientGetter := tc.setup(t, mockCtl)
-			client, err := k8sClientGetter()
-			if err != nil {
-				t.Fatalf("Unexpected error: %v", err)
+			if tc.addTaintLater {
+				time.AfterFunc(5*time.Millisecond, func() {
+					nodeCopy := node.DeepCopy()
+					nodeCopy.Spec.Taints = []corev1.Taint{{
+						Key:    AgentNotReadyNodeTaintKey,
+						Value:  "true",
+						Effect: corev1.TaintEffectNoSchedule,
+					}}
+					_, _ = client.CoreV1().Nodes().
+						Update(t.Context(), nodeCopy, metav1.UpdateOptions{})
+				})
 			}
-			result := removeNotReadyTaint(client)
 
-			if (result == nil) != (tc.expResult == nil) {
-				t.Fatalf("expected %v, got %v", tc.expResult, result)
+			patched := false
+			err := wait.PollUntilContextTimeout(
+				t.Context(),
+				5*time.Millisecond,
+				100*time.Millisecond,
+				true,
+				func(ctx context.Context) (bool, error) {
+					updated, _ := client.CoreV1().Nodes().
+						Get(ctx, nodeName, metav1.GetOptions{})
+					for _, taint := range updated.Spec.Taints {
+						if taint.Key == AgentNotReadyNodeTaintKey {
+							return false, nil // still tainted
+						}
+					}
+					patched = true
+					return true, nil
+				},
+			)
+
+			if tc.expectPatched && (err != nil || !patched) {
+				t.Fatalf("taint was not removed before timeout: %v", err)
 			}
-			if result != nil && tc.expResult != nil {
-				if result.Error() != tc.expResult.Error() {
-					t.Fatalf("Expected error message `%v`, got `%v`", tc.expResult.Error(), result.Error())
-				}
+			if !tc.expectPatched && patched {
+				t.Fatalf("taint unexpectedly removed")
 			}
 		})
 	}
 }
-
-func TestRemoveTaintInBackground(t *testing.T) {
-	t.Run("Successful taint removal", func(t *testing.T) {
-		mockRemovalCount := 0
-		mockRemovalFunc := func(_ kubernetes.Interface) error {
-			mockRemovalCount += 1
-			if mockRemovalCount == 3 {
-				return nil
-			} else {
-				return errors.New("Taint removal failed!")
-			}
-		}
-		removeTaintInBackground(nil, taintRemovalBackoff, mockRemovalFunc)
-		assert.Equal(t, 3, mockRemovalCount)
-	})
-
-	t.Run("Retries exhausted", func(t *testing.T) {
-		mockRemovalCount := 0
-		mockRemovalFunc := func(_ kubernetes.Interface) error {
-			mockRemovalCount += 1
-			return errors.New("Taint removal failed!")
-		}
-		removeTaintInBackground(nil, wait.Backoff{
-			Steps:    5,
-			Duration: 1 * time.Millisecond,
-		}, mockRemovalFunc)
-		assert.Equal(t, 5, mockRemovalCount)
-	})
-}
-
-func getNodeMock(mockCtl *gomock.Controller, nodeName string, returnNode *corev1.Node, returnError error) (kubernetes.Interface, *MockNodeInterface) {
-	mockClient := NewMockKubernetesClient(mockCtl)
-	mockCoreV1 := NewMockCoreV1Interface(mockCtl)
-	mockNode := NewMockNodeInterface(mockCtl)
-
-	mockClient.EXPECT().CoreV1().Return(mockCoreV1).MinTimes(1)
-	mockCoreV1.EXPECT().Nodes().Return(mockNode).MinTimes(1)
-	mockNode.EXPECT().Get(gomock.Any(), gomock.Eq(nodeName), gomock.Any()).Return(returnNode, returnError).MinTimes(1)
-
-	return mockClient, mockNode
-}


Reviews:
user: login: ConnorJC3
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T15:08:37Z
user: login: torredil
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T16:05:13Z
user: login: torredil
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T16:05:20Z
user: login: torredil
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T16:05:29Z
user: login: torredil
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T16:09:21Z
user: login: torredil
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T16:09:42Z
user: login: ConnorJC3
body: /lgtm

Manually tested by adding a taint to the node twice about 30 seconds after startup, and it was removed both times.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-21T18:16:08Z
user: login: ElijahQuinones
body: /lgtm 
/approve
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-21T19:31:36Z

Review comments:
diff_hunk: @@ -89,11 +82,9 @@ type NodeService struct {
 // NewNodeService creates a new node service.
 func NewNodeService(o *Options, md metadata.MetadataService, m mounter.Mounter, k kubernetes.Interface) *NodeService {
 	if k != nil {
-		// Remove taint from node to indicate driver startup success
-		// This is done at the last possible moment to prevent race conditions or false positive removals
-		time.AfterFunc(taintRemovalInitialDelay, func() {
-			removeTaintInBackground(k, taintRemovalBackoff, removeNotReadyTaint)
-		})
+		// Watch for the agent‑not‑ready taint for up to one minute and remove it
+		// as soon as allocatable is available.
+		startNotReadyTaintWatcher(k, 1*time.Minute)
path: pkg/driver/node.go
user: login: ConnorJC3
body: We should make `1*time.Minute` a constant similar to how `taintRemovalBackoff` and `taintRemovalInitialDelay` were in the old implementation
created_at: 2025-05-20T14:59:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -848,29 +839,104 @@ func collectMountOptions(fsType string, mntFlags []string) []string {
 	return options
 }
 
+// startNotReadyTaintWatcher launches a short‑lived Node informer that removes the
+// ebs.csi.aws.com/agent‑not‑ready taint as soon as both the taint is present and
+// the driver’s Allocatable value is set. The watcher stops automatically after
+// maxWatchDuration or immediately after successful removal.
+func startNotReadyTaintWatcher(clientset kubernetes.Interface, maxWatchDuration time.Duration) {
+	nodeName := os.Getenv("CSI_NODE_NAME")
+	if nodeName == "" {
+		klog.V(4).InfoS("CSI_NODE_NAME missing, skipping taint watcher")
+		return
+	}
+
+	stopCh := make(chan struct{})
+	var closeOnce sync.Once
+	closeWatch := func() { closeOnce.Do(func() { close(stopCh) }) }
+
+	factory := informers.NewSharedInformerFactoryWithOptions(
+		clientset,
+		0,
path: pkg/driver/node.go
user: login: ConnorJC3
body: ```suggestion
		0, // No resync
```

I had to look up what the `0` does here, maybe add a comment like this to indicate?
created_at: 2025-05-20T15:01:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -848,29 +839,104 @@ func collectMountOptions(fsType string, mntFlags []string) []string {
 	return options
 }
 
+// startNotReadyTaintWatcher launches a short‑lived Node informer that removes the
+// ebs.csi.aws.com/agent‑not‑ready taint as soon as both the taint is present and
+// the driver’s Allocatable value is set. The watcher stops automatically after
+// maxWatchDuration or immediately after successful removal.
+func startNotReadyTaintWatcher(clientset kubernetes.Interface, maxWatchDuration time.Duration) {
+	nodeName := os.Getenv("CSI_NODE_NAME")
+	if nodeName == "" {
+		klog.V(4).InfoS("CSI_NODE_NAME missing, skipping taint watcher")
+		return
+	}
+
+	stopCh := make(chan struct{})
+	var closeOnce sync.Once
+	closeWatch := func() { closeOnce.Do(func() { close(stopCh) }) }
+
+	factory := informers.NewSharedInformerFactoryWithOptions(
+		clientset,
+		0,
+		informers.WithTweakListOptions(func(lo *metav1.ListOptions) {
+			lo.FieldSelector = "metadata.name=" + nodeName
+		}),
+	)
+	informer := factory.Core().V1().Nodes().Informer()
+
+	attemptTaintRemoval := func(n *corev1.Node) {
+		if !hasNotReadyTaint(n) {
+			return
+		}
+		if err := checkAllocatable(clientset, nodeName); err != nil {
+			return
+		}
+		if err := removeNotReadyTaint(clientset); err != nil {
+			klog.ErrorS(err, "failed to remove agent‑not‑ready taint")
path: pkg/driver/node.go
user: login: ConnorJC3
body: np: start log message with a capital letter
created_at: 2025-05-20T15:01:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -848,29 +839,104 @@ func collectMountOptions(fsType string, mntFlags []string) []string {
 	return options
 }
 
+// startNotReadyTaintWatcher launches a short‑lived Node informer that removes the
+// ebs.csi.aws.com/agent‑not‑ready taint as soon as both the taint is present and
+// the driver’s Allocatable value is set. The watcher stops automatically after
+// maxWatchDuration or immediately after successful removal.
+func startNotReadyTaintWatcher(clientset kubernetes.Interface, maxWatchDuration time.Duration) {
+	nodeName := os.Getenv("CSI_NODE_NAME")
+	if nodeName == "" {
+		klog.V(4).InfoS("CSI_NODE_NAME missing, skipping taint watcher")
+		return
+	}
+
+	stopCh := make(chan struct{})
+	var closeOnce sync.Once
+	closeWatch := func() { closeOnce.Do(func() { close(stopCh) }) }
+
+	factory := informers.NewSharedInformerFactoryWithOptions(
+		clientset,
+		0,
+		informers.WithTweakListOptions(func(lo *metav1.ListOptions) {
+			lo.FieldSelector = "metadata.name=" + nodeName
+		}),
+	)
+	informer := factory.Core().V1().Nodes().Informer()
+
+	attemptTaintRemoval := func(n *corev1.Node) {
+		if !hasNotReadyTaint(n) {
+			return
+		}
+		if err := checkAllocatable(clientset, nodeName); err != nil {
+			return
+		}
+		if err := removeNotReadyTaint(clientset); err != nil {
+			klog.ErrorS(err, "failed to remove agent‑not‑ready taint")
+			return
+		}
+		klog.V(2).InfoS("Successfully removed agent‑not‑ready taint; stopping watcher")
+		closeWatch()
+	}
+
+	if _, err := informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
+		AddFunc: func(obj interface{}) {
+			if n, ok := obj.(*corev1.Node); ok {
+				attemptTaintRemoval(n)
+			}
+		},
+		UpdateFunc: func(_, newObj interface{}) {
+			if n, ok := newObj.(*corev1.Node); ok {
+				attemptTaintRemoval(n)
+			}
+		},
+	}); err != nil {
+		klog.ErrorS(err, "taint‑watcher: failed to add event handler")
path: pkg/driver/node.go
user: login: ConnorJC3
body: np: start log message with a capital letter (failed -> Failed)

Same np several times in this file later
created_at: 2025-05-20T15:02:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -848,29 +839,104 @@ func collectMountOptions(fsType string, mntFlags []string) []string {
 	return options
 }
 
+// startNotReadyTaintWatcher launches a short‑lived Node informer that removes the
+// ebs.csi.aws.com/agent‑not‑ready taint as soon as both the taint is present and
+// the driver’s Allocatable value is set. The watcher stops automatically after
+// maxWatchDuration or immediately after successful removal.
+func startNotReadyTaintWatcher(clientset kubernetes.Interface, maxWatchDuration time.Duration) {
+	nodeName := os.Getenv("CSI_NODE_NAME")
+	if nodeName == "" {
+		klog.V(4).InfoS("CSI_NODE_NAME missing, skipping taint watcher")
+		return
+	}
+
+	stopCh := make(chan struct{})
+	var closeOnce sync.Once
+	closeWatch := func() { closeOnce.Do(func() { close(stopCh) }) }
+
+	factory := informers.NewSharedInformerFactoryWithOptions(
+		clientset,
+		0,
+		informers.WithTweakListOptions(func(lo *metav1.ListOptions) {
+			lo.FieldSelector = "metadata.name=" + nodeName
+		}),
+	)
+	informer := factory.Core().V1().Nodes().Informer()
+
+	attemptTaintRemoval := func(n *corev1.Node) {
+		if !hasNotReadyTaint(n) {
+			return
+		}
+		if err := checkAllocatable(clientset, nodeName); err != nil {
+			return
+		}
+		if err := removeNotReadyTaint(clientset); err != nil {
+			klog.ErrorS(err, "failed to remove agent‑not‑ready taint")
+			return
+		}
+		klog.V(2).InfoS("Successfully removed agent‑not‑ready taint; stopping watcher")
+		closeWatch()
path: pkg/driver/node.go
user: login: ConnorJC3
body: Why are we stopping here? Isn't the point of this bugfix that the taint gets re-added and thus needs to be re-removed?
created_at: 2025-05-20T15:07:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -848,29 +839,104 @@ func collectMountOptions(fsType string, mntFlags []string) []string {
 	return options
 }
 
+// startNotReadyTaintWatcher launches a short‑lived Node informer that removes the
+// ebs.csi.aws.com/agent‑not‑ready taint as soon as both the taint is present and
+// the driver’s Allocatable value is set. The watcher stops automatically after
+// maxWatchDuration or immediately after successful removal.
+func startNotReadyTaintWatcher(clientset kubernetes.Interface, maxWatchDuration time.Duration) {
+	nodeName := os.Getenv("CSI_NODE_NAME")
+	if nodeName == "" {
+		klog.V(4).InfoS("CSI_NODE_NAME missing, skipping taint watcher")
+		return
+	}
+
+	stopCh := make(chan struct{})
+	var closeOnce sync.Once
+	closeWatch := func() { closeOnce.Do(func() { close(stopCh) }) }
path: pkg/driver/node.go
user: login: ConnorJC3
body: Given that we don't want to stop after removal (see my below comment), I think this channel can be removed entirely
created_at: 2025-05-20T15:08:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -848,29 +839,104 @@ func collectMountOptions(fsType string, mntFlags []string) []string {
 	return options
 }
 
+// startNotReadyTaintWatcher launches a short‑lived Node informer that removes the
+// ebs.csi.aws.com/agent‑not‑ready taint as soon as both the taint is present and
+// the driver’s Allocatable value is set. The watcher stops automatically after
+// maxWatchDuration or immediately after successful removal.
+func startNotReadyTaintWatcher(clientset kubernetes.Interface, maxWatchDuration time.Duration) {
+	nodeName := os.Getenv("CSI_NODE_NAME")
+	if nodeName == "" {
+		klog.V(4).InfoS("CSI_NODE_NAME missing, skipping taint watcher")
+		return
+	}
+
+	stopCh := make(chan struct{})
+	var closeOnce sync.Once
+	closeWatch := func() { closeOnce.Do(func() { close(stopCh) }) }
+
+	factory := informers.NewSharedInformerFactoryWithOptions(
+		clientset,
+		0,
path: pkg/driver/node.go
user: login: torredil
body: Done ✅ 
created_at: 2025-05-20T16:05:12Z
author_association: MEMBER
diff_hunk: @@ -848,29 +839,104 @@ func collectMountOptions(fsType string, mntFlags []string) []string {
 	return options
 }
 
+// startNotReadyTaintWatcher launches a short‑lived Node informer that removes the
+// ebs.csi.aws.com/agent‑not‑ready taint as soon as both the taint is present and
+// the driver’s Allocatable value is set. The watcher stops automatically after
+// maxWatchDuration or immediately after successful removal.
+func startNotReadyTaintWatcher(clientset kubernetes.Interface, maxWatchDuration time.Duration) {
+	nodeName := os.Getenv("CSI_NODE_NAME")
+	if nodeName == "" {
+		klog.V(4).InfoS("CSI_NODE_NAME missing, skipping taint watcher")
+		return
+	}
+
+	stopCh := make(chan struct{})
+	var closeOnce sync.Once
+	closeWatch := func() { closeOnce.Do(func() { close(stopCh) }) }
+
+	factory := informers.NewSharedInformerFactoryWithOptions(
+		clientset,
+		0,
+		informers.WithTweakListOptions(func(lo *metav1.ListOptions) {
+			lo.FieldSelector = "metadata.name=" + nodeName
+		}),
+	)
+	informer := factory.Core().V1().Nodes().Informer()
+
+	attemptTaintRemoval := func(n *corev1.Node) {
+		if !hasNotReadyTaint(n) {
+			return
+		}
+		if err := checkAllocatable(clientset, nodeName); err != nil {
+			return
+		}
+		if err := removeNotReadyTaint(clientset); err != nil {
+			klog.ErrorS(err, "failed to remove agent‑not‑ready taint")
path: pkg/driver/node.go
user: login: torredil
body: Done ✅ 
created_at: 2025-05-20T16:05:20Z
author_association: MEMBER
diff_hunk: @@ -89,11 +82,9 @@ type NodeService struct {
 // NewNodeService creates a new node service.
 func NewNodeService(o *Options, md metadata.MetadataService, m mounter.Mounter, k kubernetes.Interface) *NodeService {
 	if k != nil {
-		// Remove taint from node to indicate driver startup success
-		// This is done at the last possible moment to prevent race conditions or false positive removals
-		time.AfterFunc(taintRemovalInitialDelay, func() {
-			removeTaintInBackground(k, taintRemovalBackoff, removeNotReadyTaint)
-		})
+		// Watch for the agent‑not‑ready taint for up to one minute and remove it
+		// as soon as allocatable is available.
+		startNotReadyTaintWatcher(k, 1*time.Minute)
path: pkg/driver/node.go
user: login: torredil
body: Done ✅ 
created_at: 2025-05-20T16:05:29Z
author_association: MEMBER
diff_hunk: @@ -848,29 +839,104 @@ func collectMountOptions(fsType string, mntFlags []string) []string {
 	return options
 }
 
+// startNotReadyTaintWatcher launches a short‑lived Node informer that removes the
+// ebs.csi.aws.com/agent‑not‑ready taint as soon as both the taint is present and
+// the driver’s Allocatable value is set. The watcher stops automatically after
+// maxWatchDuration or immediately after successful removal.
+func startNotReadyTaintWatcher(clientset kubernetes.Interface, maxWatchDuration time.Duration) {
+	nodeName := os.Getenv("CSI_NODE_NAME")
+	if nodeName == "" {
+		klog.V(4).InfoS("CSI_NODE_NAME missing, skipping taint watcher")
+		return
+	}
+
+	stopCh := make(chan struct{})
+	var closeOnce sync.Once
+	closeWatch := func() { closeOnce.Do(func() { close(stopCh) }) }
+
+	factory := informers.NewSharedInformerFactoryWithOptions(
+		clientset,
+		0,
+		informers.WithTweakListOptions(func(lo *metav1.ListOptions) {
+			lo.FieldSelector = "metadata.name=" + nodeName
+		}),
+	)
+	informer := factory.Core().V1().Nodes().Informer()
+
+	attemptTaintRemoval := func(n *corev1.Node) {
+		if !hasNotReadyTaint(n) {
+			return
+		}
+		if err := checkAllocatable(clientset, nodeName); err != nil {
+			return
+		}
+		if err := removeNotReadyTaint(clientset); err != nil {
+			klog.ErrorS(err, "failed to remove agent‑not‑ready taint")
+			return
+		}
+		klog.V(2).InfoS("Successfully removed agent‑not‑ready taint; stopping watcher")
+		closeWatch()
path: pkg/driver/node.go
user: login: torredil
body: It's not that the taint gets re-added, its more that there are no taints to remove when the check is performed, so the taint removal is skipped, see logs in the original bug report:

```
...
I0418 20:03:42.343483       1 node.go:896] "No taints to remove on node, skipping taint removal"
```

I think not stopping the informer for the full `taintWatcherDuration` makes sense though to handle the scenario you mentioned (a controller re-applying the taint after it is removed), especially given its short lifespan. 
created_at: 2025-05-20T16:09:20Z
author_association: MEMBER
diff_hunk: @@ -848,29 +839,104 @@ func collectMountOptions(fsType string, mntFlags []string) []string {
 	return options
 }
 
+// startNotReadyTaintWatcher launches a short‑lived Node informer that removes the
+// ebs.csi.aws.com/agent‑not‑ready taint as soon as both the taint is present and
+// the driver’s Allocatable value is set. The watcher stops automatically after
+// maxWatchDuration or immediately after successful removal.
+func startNotReadyTaintWatcher(clientset kubernetes.Interface, maxWatchDuration time.Duration) {
+	nodeName := os.Getenv("CSI_NODE_NAME")
+	if nodeName == "" {
+		klog.V(4).InfoS("CSI_NODE_NAME missing, skipping taint watcher")
+		return
+	}
+
+	stopCh := make(chan struct{})
+	var closeOnce sync.Once
+	closeWatch := func() { closeOnce.Do(func() { close(stopCh) }) }
path: pkg/driver/node.go
user: login: torredil
body: Done ✅
created_at: 2025-05-20T16:09:42Z
author_association: MEMBER

PR comments:
user: login: github-actions[bot]
created_at: 2025-05-19T14:09:00Z
body: <!-- pr-coverage -->
## Code Coverage Diff
File | Old Coverage | New Coverage | Delta
---- |:------------:|:------------:|:-----:
github.com/kubernetes-sigs/aws-ebs-csi-driver/pkg/driver/node.go | 92.2% | 90.9% | -1.2
author_association: NONE
user: login: k8s-ci-robot
created_at: 2025-05-21T19:31:43Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/2492#pullrequestreview-2858901659" title="Approved">ElijahQuinones</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Faws-ebs-csi-driver).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/OWNERS)~~ [ElijahQuinones]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR