Repository: kubernetes-sigs/cluster-api-provider-aws

Issue:
capa-eks-e2e: "EKS cluster tests should create a cluster and add nodes" consistently failing
/kind bug
/area provider/eks
/area testing

**What steps did you take and what happened:**
[A clear and concise description of what the bug is.]

The EKS e2e tests are consistently failing on this test case:

`capa-eks-e2e: [It] [managed] [general] [ipv6] EKS cluster tests should create a cluster and add nodes`

**What did you expect to happen:**

e2e should continue passing.

**Anything else you would like to add:**

Last passing test: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/5498/pull-cluster-api-provider-aws-e2e-eks/1927445735872139264

History: https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/pr-logs/directory/pull-cluster-api-provider-aws-e2e-eks

**Environment:**

- Cluster-api-provider-aws version: main


Discussion:
@k8s-ci-robot: This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the `triage/accepted` label and provide further guidance.

The `triage/accepted` label can be added by org members by writing `/triage accepted` in a comment.


<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@richardcase: /assign
/lifecycle active@richardcase: Do we have some examples of this failing?
@richardcase: Seem to be looping on this:

```
I0609 11:12:30.068764       1 addons.go:75] "Executing addon procedure" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControlPlane" AWSManagedControlPlan │
│ I0609 11:12:30.068961       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:12:31.306109       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:12:33.379102       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:12:36.584944       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:12:42.420094       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:12:52.471935       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:13:08.014267       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:13:34.169904       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:13:57.171748       1 reflector.go:871] pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/reflector.go:243: Watch close - *v1beta1.Machine total 0 items received                                                      │
│ I0609 11:14:21.34
```

This might be a red-herring. Still looking.@richardcase: Also saw this error:

```
E0609 11:17:50.737080       1 controller.go:316] "Reconciler error" err="failed to reconcile machine pool for AWSManagedMachinePool eks-nodes-blnyr0/eks-nodes-0dja0i-pool-0: failed to wait for nodegroup to be active:  │
│ failed to wait for EKS nodegroup \"eks-nodes-blnyr0_eks-nodes-0dja0i-pool-0\": maximum wait time for waiter must be greater than zero" controller="awsmanagedmachinepool" controllerGroup="infrastructure.cluster.x-k8s.i │
│ o" controllerKind="AWSManagedMachinePool" AWSManagedMachinePool="eks-nodes-blnyr0/eks-nodes-0dja0i-pool-0" namespace="eks-nodes-blnyr0" name="eks-nodes-0dja0i-pool-0" reconcileID="b01f7221-9b82-4f14-a191-90a
```@nrb: Also seeing slightly different errors on other jobs:

```
   [FAILED] Timed out after 300.000s.
  should have been able to find the expected aws-node DaemonSet
  Expected success, but got an error:
      <*errors.errorString | 0xc000d7e140>: 
      unable to find the expected environment variables on the aws-node DaemonSet's container
      {
          s: "unable to find the expected environment variables on the aws-node DaemonSet's container",
      }
  In [It] at: /home/prow/go/src/sigs.k8s.io/cluster-api-provider-aws/test/e2e/suites/managed/aws_node_env.go:88 @ 06/14/25 15:34:28.487
```

from https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/5447/pull-cluster-api-provider-aws-e2e-eks/1933880375218540544

Unsure if this is a symptom or cause, though.@damdo: Attempting a fix here: https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/5554@richardcase: > Attempting a fix here: [#5554](https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/5554)

Excellent @damdo . I saw the wait time was zero but didn't get around to making a change, so thanks for doing this 🎉 @damdo: Turns out the PR that merged #5554 fixed the issue @richardcase was talking about here: https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/5514#issuecomment-2955499929 

but not the one causing https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/5514#issuecomment-2976699966

I think an issue with the addons is what is causing it, this is a snippet from the CAPA logs:
`
E0617 17:12:23.652552       1 controller.go:347] "Reconciler error" err="failed to reconcile control plane for AWSManagedControlPlane cluster-rfyyxd/cluster-7uau41-control-plane: failed reconciling eks addons: addon_create: creating eks addon vpc-cni: operation error EKS: CreateAddon, https response error StatusCode: 400, RequestID: 1d192232-9994-4a48-a98b-e447e5a73898, InvalidParameterException: Addon version specified is not supported" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControlPlane" AWSManagedControlPlane="cluster-rfyyxd/cluster-7uau41-control-plane" namespace="cluster-rfyyxd" name="cluster-7uau41-control-plane" reconcileID="815c15ab-f6d0-477a-a42a-67e5fdf76723"
`

IIRC there was something similar in the past regarding addons that was causing an aws-node DaemonSet issue.@damdo: ^ this might be unrelated to the consistent failing we have seen on main, and just be an issue related to the v1.32 bump we are doing. Added a comment here: https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/5447/files#r2155007805@damdo: Will re-close this issue.

Solution patch:
diff --git a/pkg/cloud/services/ec2/launchtemplate.go b/pkg/cloud/services/ec2/launchtemplate.go
index 60df3b75f..01e81fc01 100644
--- a/pkg/cloud/services/ec2/launchtemplate.go
+++ b/pkg/cloud/services/ec2/launchtemplate.go
@@ -802,6 +802,10 @@ func (s *Service) LaunchTemplateNeedsUpdate(scope scope.LaunchTemplateScope, inc
 		return true, nil
 	}
 
+	if !cmp.Equal(incoming.PrivateDNSName, existing.PrivateDNSName) {
+		return true, nil
+	}
+
 	if !cmp.Equal(incoming.SSHKeyName, existing.SSHKeyName) {
 		return true, nil
 	}



Test patch:
diff --git a/pkg/cloud/services/ec2/launchtemplate_test.go b/pkg/cloud/services/ec2/launchtemplate_test.go
index df5aac618..93fd040f2 100644
--- a/pkg/cloud/services/ec2/launchtemplate_test.go
+++ b/pkg/cloud/services/ec2/launchtemplate_test.go
@@ -799,6 +799,67 @@ func TestServiceLaunchTemplateNeedsUpdate(t *testing.T) {
 			want:    true,
 			wantErr: false,
 		},
+		{
+			name: "Should return true if incoming PrivateDNSName is different from existing PrivateDNSName",
+			incoming: &expinfrav1.AWSLaunchTemplate{
+				PrivateDNSName: &infrav1.PrivateDNSName{
+					EnableResourceNameDNSARecord:    aws.Bool(true),
+					EnableResourceNameDNSAAAARecord: aws.Bool(false),
+					HostnameType:                    aws.String("resource-name"),
+				},
+			},
+			existing: &expinfrav1.AWSLaunchTemplate{
+				AdditionalSecurityGroups: []infrav1.AWSResourceReference{
+					{ID: aws.String("sg-111")},
+					{ID: aws.String("sg-222")},
+				},
+				PrivateDNSName: &infrav1.PrivateDNSName{
+					EnableResourceNameDNSARecord:    aws.Bool(false),
+					EnableResourceNameDNSAAAARecord: aws.Bool(false),
+					HostnameType:                    aws.String("ip-name"),
+				},
+			},
+			want:    true,
+			wantErr: false,
+		},
+		{
+			name: "Should return true if incoming adds PrivateDNSName and existing has none",
+			incoming: &expinfrav1.AWSLaunchTemplate{
+				PrivateDNSName: &infrav1.PrivateDNSName{
+					EnableResourceNameDNSARecord:    aws.Bool(true),
+					EnableResourceNameDNSAAAARecord: aws.Bool(false),
+					HostnameType:                    aws.String("resource-name"),
+				},
+			},
+			existing: &expinfrav1.AWSLaunchTemplate{
+				AdditionalSecurityGroups: []infrav1.AWSResourceReference{
+					{ID: aws.String("sg-111")},
+					{ID: aws.String("sg-222")},
+				},
+				PrivateDNSName: nil,
+			},
+			want:    true,
+			wantErr: false,
+		},
+		{
+			name: "Should return true if incoming removes PrivateDNSName and existing has some",
+			incoming: &expinfrav1.AWSLaunchTemplate{
+				PrivateDNSName: nil,
+			},
+			existing: &expinfrav1.AWSLaunchTemplate{
+				AdditionalSecurityGroups: []infrav1.AWSResourceReference{
+					{ID: aws.String("sg-111")},
+					{ID: aws.String("sg-222")},
+				},
+				PrivateDNSName: &infrav1.PrivateDNSName{
+					EnableResourceNameDNSARecord:    aws.Bool(true),
+					EnableResourceNameDNSAAAARecord: aws.Bool(false),
+					HostnameType:                    aws.String("resource-name"),
+				},
+			},
+			want:    true,
+			wantErr: false,
+		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {