Repository: kubernetes-sigs/cluster-api-provider-aws
Pull request #3568 (2022-06-29 18:27:46)

Add deduplication handling and only update if there is a need for it
<!-- Thanks for this PR! If this is your first PR please read the [contributing guide](../CONTRIBUTING.md) -->
<!-- If this PR is still work-in-progress and is being open for visibility please prefix the title with `WIP:` -->

**What type of PR is this?**

/kind bug
**What this PR does / why we need it**:

Adds deduplication and only update the DaemonSet if there is a need for an update.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/3567

**Special notes for your reviewer**:

**Checklist**:
<!-- Put an "X" character inside the brackets of each completed task. Some may be optional depending on the PR in which case these can be deleted -->

- [x] squashed commits
- [ ] includes documentation
- [x] adds unit tests
- [x] adds or updates e2e tests


k8s-ci-robot: Hi @Skarlso. Thanks for your PR.

I'm waiting for a [kubernetes-sigs](https://github.com/orgs/kubernetes-sigs/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcluster-api-provider-aws).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by:
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **neolit123** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @neolit123` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcluster-api-provider-aws).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["neolit123"]} -->
Skarlso: Using this update the logs outputs change only once, when creating the cluster:

```
# second reconcile run doesn't have the `adding...` part of the log
I0629 18:16:17.271330 37 cni.go:68] "msg"="updating aws-node daemonset environment variables" "cluster-name"="ipv6-cluster" "cluster-namespace"="default"` `I0629 18:16:17.271405 37 reconcile.go:38] "msg"="Reconciling kube-proxy DaemonSet in cluster" "cluster-name"="ipv6-cluster" "cluster-namespace"="default"
```

Environment properties are as follows:

```yaml
Environment:
      ADDITIONAL_ENI_TAGS:                    {}
      AWS_VPC_CNI_NODE_PORT_SUPPORT:          true
      AWS_VPC_ENI_MTU:                        9001
      AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER:     false
      AWS_VPC_K8S_CNI_EXTERNALSNAT:           false
      AWS_VPC_K8S_CNI_LOGLEVEL:               DEBUG
      AWS_VPC_K8S_CNI_LOG_FILE:               /host/var/log/aws-routed-eni/ipamd.log
      AWS_VPC_K8S_CNI_RANDOMIZESNAT:          prng
      AWS_VPC_K8S_CNI_VETHPREFIX:             eni
      AWS_VPC_K8S_PLUGIN_LOG_FILE:            /var/log/aws-routed-eni/plugin.log
      AWS_VPC_K8S_PLUGIN_LOG_LEVEL:           DEBUG
      DISABLE_INTROSPECTION:                  false
      DISABLE_METRICS:                        false
      DISABLE_NETWORK_RESOURCE_PROVISIONING:  false
      ENABLE_IPv4:                            false
      ENABLE_IPv6:                            true
      ENABLE_POD_ENI:                         false
      ENABLE_PREFIX_DELEGATION:               true
      MY_NODE_NAME:                            (v1:spec.nodeName)
      WARM_ENI_TARGET:                        1
      WARM_PREFIX_TARGET:                     1
```
Skarlso: Hold off on merging, I'm adding some awesome unit tests. :) 
richardcase: /test ?

k8s-ci-robot: @richardcase: The following commands are available to trigger required jobs:
* `/test pull-cluster-api-provider-aws-build`
* `/test pull-cluster-api-provider-aws-test`
* `/test pull-cluster-api-provider-aws-verify`

The following commands are available to trigger optional jobs:
* `/test pull-cluster-api-provider-aws-apidiff-main`
* `/test pull-cluster-api-provider-aws-e2e`
* `/test pull-cluster-api-provider-aws-e2e-blocking`
* `/test pull-cluster-api-provider-aws-e2e-conformance`
* `/test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts`
* `/test pull-cluster-api-provider-aws-e2e-eks`

Use `/test all` to run the following jobs that were automatically triggered:
* `pull-cluster-api-provider-aws-apidiff-main`
* `pull-cluster-api-provider-aws-build`
* `pull-cluster-api-provider-aws-test`
* `pull-cluster-api-provider-aws-verify`



<details>

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/3568#issuecomment-1180161537):

>/test ?
>


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
richardcase: /test pull-cluster-api-provider-aws-e2e-eks
k8s-ci-robot: @Skarlso: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cluster-api-provider-aws-e2e-eks | 9890f48df2489ace4fe8ef0c745da4da1b0b7dd0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/3568/pull-cluster-api-provider-aws-e2e-eks/1547215231086759936) | false | `/test pull-cluster-api-provider-aws-e2e-eks`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api-provider-aws&pr=3568). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3ASkarlso). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
Skarlso: /test pull-cluster-api-provider-aws-e2e-eks
k8s-ci-robot: @Skarlso: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cluster-api-provider-aws-e2e-eks | 9890f48df2489ace4fe8ef0c745da4da1b0b7dd0 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/3568/pull-cluster-api-provider-aws-e2e-eks/1547241238334279680) | false | `/test pull-cluster-api-provider-aws-e2e-eks`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api-provider-aws&pr=3568). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3ASkarlso). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
Skarlso: /test pull-cluster-api-provider-aws-e2e-eks
richardcase: Looking at the logs for the failure:

'''

E0713 19:35:41.930286       1 controller.go:317] controller/awsmanagedcontrolplane "msg"="Reconciler error" "error"="failed to reconcile control plane for AWSManagedControlPlane eks-upgrade-onnuy2/eks-upgrade-sijk09-control-plane: checking owner of eks-upgrade-onnuy2_eks-upgrade-sijk09-control-plane is eks-upgrade-sijk09: %!w(<nil>)" "name"="eks-upgrade-sijk09-control-plane" "namespace"="eks-upgrade-onnuy2" "reconciler group"="controlplane.cluster.x-k8s.io" "reconciler kind"="AWSManagedControlPlane"
'''

This feels suspiciously related to a change that merged earlier today.
richardcase: The PR was #3573. Afraid I'm going away so can't take a look until I get back.
Skarlso: @richardcase Sorry Richard. It looks like that PR was actually NOT merged into this one. I updated the ref just before that pr was merged. :) And I thought I was on latest. :)
Skarlso: /test pull-cluster-api-provider-aws-e2e-eks
k8s-ci-robot: @Skarlso: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-cluster-api-provider-aws-e2e-eks | 3f7ec54a06b82f505af98ff040c7c1b7e9cdce7e | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/3568/pull-cluster-api-provider-aws-e2e-eks/1547445696439783424) | false | `/test pull-cluster-api-provider-aws-e2e-eks`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api-provider-aws&pr=3568). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3ASkarlso). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
Skarlso: Hmmmm............ Well this is annoying.
Skarlso: Okay, I think I have a fix. I'll run the test to see if I'm right:

/test pull-cluster-api-provider-aws-e2e-eks
Skarlso: https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/3597 is needed for this. Once that's in, I'll rebase to keep this PR clean of that fix.

Patch:
diff --git a/pkg/cloud/services/awsnode/cni.go b/pkg/cloud/services/awsnode/cni.go
index f4fbe19ca..ac48f1404 100644
--- a/pkg/cloud/services/awsnode/cni.go
+++ b/pkg/cloud/services/awsnode/cni.go
@@ -63,33 +63,21 @@ func (s *Service) ReconcileCNI(ctx context.Context) error {
 		return ErrCNIMissing
 	}
 
-	envVars := s.scope.VpcCni().Env
-	if len(envVars) > 0 {
+	var needsUpdate bool
+	if len(s.scope.VpcCni().Env) > 0 {
 		s.scope.Info("updating aws-node daemonset environment variables", "cluster-name", s.scope.Name(), "cluster-namespace", s.scope.Namespace())
 
 		for i := range ds.Spec.Template.Spec.Containers {
 			container := &ds.Spec.Template.Spec.Containers[i]
 			if container.Name == "aws-node" {
-				existingVars := s.filterEnv(container.Env)
-
-				for ei, e := range existingVars {
-					for ai, a := range envVars {
-						if e.Name == a.Name {
-							existingVars[ei].Value = a.Value
-
-							envVars = append(envVars[:ai], envVars[ai+1:]...)
-							break
-						}
-					}
-				}
-				container.Env = append(existingVars, envVars...)
-				break
+				container.Env = s.filterEnv(container.Env)
+				container.Env, needsUpdate = s.applyUserProvidedEnvironmentProperties(container.Env)
 			}
 		}
 	}
 
 	if s.scope.SecondaryCidrBlock() == nil {
-		if len(envVars) > 0 {
+		if needsUpdate {
 			if err = remoteClient.Update(ctx, &ds, &client.UpdateOptions{}); err != nil {
 				return err
 			}
@@ -217,6 +205,36 @@ func (s *Service) filterEnv(env []corev1.EnvVar) []corev1.EnvVar {
 	return env[:i]
 }
 
+// applyUserProvidedEnvironmentProperties takes a container environment and applies user provided values to it.
+func (s *Service) applyUserProvidedEnvironmentProperties(containerEnv []corev1.EnvVar) ([]corev1.EnvVar, bool) {
+	var (
+		envVars     = make(map[string]corev1.EnvVar)
+		needsUpdate = false
+	)
+	for _, e := range s.scope.VpcCni().Env {
+		envVars[e.Name] = e
+	}
+	// Handle the case where we overwrite an existing value if it's not already the desired value.
+	// This will prevent continuously updating the DaemonSet even though there are no changes.
+	for i, e := range containerEnv {
+		if v, ok := envVars[e.Name]; ok {
+			// Take care of comparing secret ref with Stringer.
+			if containerEnv[i].String() != v.String() {
+				needsUpdate = true
+				containerEnv[i] = v
+			}
+			delete(envVars, e.Name)
+		}
+	}
+	// Handle case when there are values that aren't in the list of environment properties
+	// of aws-node.
+	for _, v := range envVars {
+		needsUpdate = true
+		containerEnv = append(containerEnv, v)
+	}
+	return containerEnv, needsUpdate
+}
+
 func (s *Service) deleteCNI(ctx context.Context, remoteClient client.Client) error {
 	s.scope.Info("Ensuring aws-node DaemonSet in cluster is deleted", "cluster-name", s.scope.Name(), "cluster-namespace", s.scope.Namespace())
 


Test patch:
diff --git a/pkg/cloud/services/awsnode/cni_test.go b/pkg/cloud/services/awsnode/cni_test.go
new file mode 100644
index 000000000..5344badc2
--- /dev/null
+++ b/pkg/cloud/services/awsnode/cni_test.go
@@ -0,0 +1,244 @@
+package awsnode
+
+import (
+	"context"
+	"testing"
+
+	"github.com/golang/mock/gomock"
+	. "github.com/onsi/gomega"
+	v1 "k8s.io/api/apps/v1"
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	ekscontrolplanev1 "sigs.k8s.io/cluster-api-provider-aws/controlplane/eks/api/v1beta1"
+	"sigs.k8s.io/cluster-api-provider-aws/pkg/cloud/scope"
+)
+
+func TestReconcileCniVpcCniValues(t *testing.T) {
+	tests := []struct {
+		name       string
+		cniValues  ekscontrolplanev1.VpcCni
+		daemonSet  *v1.DaemonSet
+		consistsOf []corev1.EnvVar
+	}{
+		{
+			name: "users can set environment values",
+			cniValues: ekscontrolplanev1.VpcCni{
+				Env: []corev1.EnvVar{
+					{
+						Name:  "NAME1",
+						Value: "VALUE1",
+					},
+				},
+			},
+			daemonSet: &v1.DaemonSet{
+				TypeMeta: metav1.TypeMeta{
+					Kind: "DaemonSet",
+				},
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "aws-node",
+					Namespace: "kube-system",
+				},
+				Spec: v1.DaemonSetSpec{
+					Template: corev1.PodTemplateSpec{
+						ObjectMeta: metav1.ObjectMeta{},
+						Spec: corev1.PodSpec{
+							Containers: []corev1.Container{
+								{
+									Name: "aws-node",
+									Env:  []corev1.EnvVar{},
+								},
+							},
+						},
+					},
+				},
+			},
+			consistsOf: []corev1.EnvVar{
+				{
+					Name:  "NAME1",
+					Value: "VALUE1",
+				},
+			},
+		},
+		{
+			name: "users can set environment values without duplications",
+			cniValues: ekscontrolplanev1.VpcCni{
+				Env: []corev1.EnvVar{
+					{
+						Name:  "NAME1",
+						Value: "VALUE1",
+					},
+					{
+						Name:  "NAME1",
+						Value: "VALUE2",
+					},
+				},
+			},
+			daemonSet: &v1.DaemonSet{
+				TypeMeta: metav1.TypeMeta{
+					Kind: "DaemonSet",
+				},
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "aws-node",
+					Namespace: "kube-system",
+				},
+				Spec: v1.DaemonSetSpec{
+					Template: corev1.PodTemplateSpec{
+						ObjectMeta: metav1.ObjectMeta{},
+						Spec: corev1.PodSpec{
+							Containers: []corev1.Container{
+								{
+									Name: "aws-node",
+									Env:  []corev1.EnvVar{},
+								},
+							},
+						},
+					},
+				},
+			},
+			consistsOf: []corev1.EnvVar{
+				{
+					Name:  "NAME1",
+					Value: "VALUE2",
+				},
+			},
+		},
+		{
+			name: "users can set environment values overwriting existing values",
+			cniValues: ekscontrolplanev1.VpcCni{
+				Env: []corev1.EnvVar{
+					{
+						Name:  "NAME1",
+						Value: "VALUE1",
+					},
+					{
+						Name:  "NAME2",
+						Value: "VALUE2",
+					},
+				},
+			},
+			daemonSet: &v1.DaemonSet{
+				TypeMeta: metav1.TypeMeta{
+					Kind: "DaemonSet",
+				},
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "aws-node",
+					Namespace: "kube-system",
+				},
+				Spec: v1.DaemonSetSpec{
+					Template: corev1.PodTemplateSpec{
+						ObjectMeta: metav1.ObjectMeta{},
+						Spec: corev1.PodSpec{
+							Containers: []corev1.Container{
+								{
+									Name: "aws-node",
+									Env: []corev1.EnvVar{
+										{
+											Name:  "NAME1",
+											Value: "OVERWRITE",
+										},
+										{
+											Name:  "NAME3",
+											Value: "VALUE3",
+										},
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+			consistsOf: []corev1.EnvVar{
+				{
+					Name:  "NAME1",
+					Value: "VALUE1",
+				},
+				{
+					Name:  "NAME2",
+					Value: "VALUE2",
+				},
+				{
+					Name:  "NAME3",
+					Value: "VALUE3",
+				},
+			},
+		},
+	}
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			mockCtrl := gomock.NewController(t)
+			defer mockCtrl.Finish()
+			g := NewWithT(t)
+			mockClient := &cachingClient{
+				getValue: tc.daemonSet,
+			}
+			m := &mockScope{
+				client: mockClient,
+				cni:    tc.cniValues,
+			}
+			s := NewService(m)
+
+			err := s.ReconcileCNI(context.Background())
+			g.Expect(err).NotTo(HaveOccurred())
+			g.Expect(mockClient.updateChain).NotTo(BeEmpty())
+			ds, ok := mockClient.updateChain[0].(*v1.DaemonSet)
+			g.Expect(ok).To(BeTrue())
+			g.Expect(ds.Spec.Template.Spec.Containers).NotTo(BeEmpty())
+			g.Expect(ds.Spec.Template.Spec.Containers[0].Env).To(ConsistOf(tc.consistsOf))
+		})
+	}
+}
+
+type cachingClient struct {
+	client.Client
+	getValue    client.Object
+	updateChain []client.Object
+}
+
+func (c *cachingClient) Get(ctx context.Context, key client.ObjectKey, obj client.Object) error {
+	if _, ok := obj.(*v1.DaemonSet); ok {
+		daemonset, _ := obj.(*v1.DaemonSet)
+		*daemonset = *c.getValue.(*v1.DaemonSet)
+	}
+	return nil
+}
+
+func (c *cachingClient) Update(ctx context.Context, obj client.Object, opts ...client.UpdateOption) error {
+	c.updateChain = append(c.updateChain, obj)
+	return nil
+}
+
+type mockScope struct {
+	scope.AWSNodeScope
+	client client.Client
+	cni    ekscontrolplanev1.VpcCni
+}
+
+func (s *mockScope) RemoteClient() (client.Client, error) {
+	return s.client, nil
+}
+
+func (s *mockScope) VpcCni() ekscontrolplanev1.VpcCni {
+	return s.cni
+}
+
+func (s *mockScope) Info(msg string, keysAndValues ...interface{}) {
+
+}
+
+func (s *mockScope) Name() string {
+	return "mock-name"
+}
+
+func (s *mockScope) Namespace() string {
+	return "mock-namespace"
+}
+
+func (s *mockScope) DisableVPCCNI() bool {
+	return false
+}
+
+func (s *mockScope) SecondaryCidrBlock() *string {
+	return nil
+}