Repository: kubernetes/kops (go)

Issue:
AWS_VPC_K8S_CNI_RANDOMIZESNAT environment variable for `spec.networking.amazonvpc` resets to `prng` on every upgrade
**1. What `kops` version are you running? The command `kops version`, will display
 this information.**

1.19.1

**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**

1.18.12

**3. What cloud provider are you using?**

aws

**4. What commands did you run?  What is the simplest way to reproduce this issue?**

In `cluster.yaml`

```
spec:
  networking:
    amazonvpc:
      env:
        - name: AWS_VPC_K8S_CNI_EXTERNALSNAT
          value: "false"
        - name: AWS_VPC_K8S_CNI_RANDOMIZESNAT
          value: none
        - name: MAX_ENI
          value: none
        - name: WARM_IP_TARGET
          value: "2"
```

`kops update cluster && kops rolling-update cluster`

**5. What happened after the commands executed?**

From `kubectl get ds -n kube-system aws-node -oyaml`

```
spec:
  template:
    spec:
      containers:
      - env:
        - name: ADDITIONAL_ENI_TAGS
          value: '{}'
        - name: AWS_VPC_CNI_NODE_PORT_SUPPORT
          value: "true"
        - name: AWS_VPC_ENI_MTU
          value: "9001"
        - name: AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER
          value: "false"
        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
          value: "false"
        - name: AWS_VPC_K8S_CNI_EXTERNALSNAT
          value: "prng"
        - name: AWS_VPC_K8S_CNI_LOGLEVEL
          value: DEBUG
        - name: AWS_VPC_K8S_CNI_LOG_FILE
          value: /host/var/log/aws-routed-eni/ipamd.log
        - name: AWS_VPC_K8S_CNI_RANDOMIZESNAT
          value: none
        - name: AWS_VPC_K8S_CNI_VETHPREFIX
          value: eni
        - name: AWS_VPC_K8S_PLUGIN_LOG_FILE
          value: /var/log/aws-routed-eni/plugin.log
        - name: AWS_VPC_K8S_PLUGIN_LOG_LEVEL
          value: DEBUG
        - name: DISABLE_INTROSPECTION
          value: "false"
        - name: DISABLE_METRICS
          value: "false"
        - name: ENABLE_POD_ENI
          value: "false"
        - name: MY_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: WARM_ENI_TARGET
          value: "1"
        - name: CLUSTER_NAME
          value: [REDACTED]
        - name: MAX_ENI
          value: "1"
        - name: WARM_IP_TARGET
          value: "1"
```

**6. What did you expect to happen?**

`AWS_VPC_K8S_CNI_EXTERNALSNAT` should be set to `none`

**7. Please provide your cluster manifest. Execute
  `kops get --name my.example.com -o yaml` to display your cluster manifest.
  You may want to remove your cluster name and other sensitive information.**

```
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2018-10-31T22:42:45Z"
  name: [REDACTED]
spec:
  additionalPolicies:
    node: |
      [
        [REDACTED]
      ]
  api:
    loadBalancer:
      class: Classic
      type: Private
  authorization:
    rbac: {}
  channel: stable
  cloudLabels:
    environment: [REDACTED]
  cloudProvider: aws
  configBase: s3://[REDACTED]
  dnsZone: [REDACTED]
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-us-east-1a
      name: a
      volumeType: gp3
    - instanceGroup: master-us-east-1b
      name: b
      volumeType: gp3
    - instanceGroup: master-us-east-1c
      name: c
      volumeType: gp3
    name: main
  - etcdMembers:
    - instanceGroup: master-us-east-1a
      name: a
      volumeType: gp3
    - instanceGroup: master-us-east-1b
      name: b
      volumeType: gp3
    - instanceGroup: master-us-east-1c
      name: c
      volumeType: gp3
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeControllerManager:
    horizontalPodAutoscalerDownscaleStabilization: 22m0s
  kubeDNS:
    provider: CoreDNS
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.18.12
  masterInternalName: api.internal.[REDACTED]
  masterPublicName: api.[REDACTED]
  networkCIDR: 10.3.0.0/16
  networking:
    amazonvpc:
      env:
      - name: AWS_VPC_K8S_CNI_EXTERNALSNAT
        value: "false"
      - name: AWS_VPC_K8S_CNI_RANDOMIZESNAT
        value: none
      - name: MAX_ENI
        value: "1"
      - name: WARM_IP_TARGET
        value: "1"
      imageName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.6.3
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets: [REDACTED]
  topology:
    bastion:
      bastionPublicName: bastion.[REDACTED]
    dns:
      type: Public
    masters: private
    nodes: private
```

**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
  Paste the logs into this report, or in a gist and provide the gist link here.**

**9. Anything else do we need to know?**
`AWS_VPC_K8S_CNI_RANDOMIZESNAT` continues to reset to `"prng"` on every upgrade

Pull request:
Make aws-cni config more flexible and generalized
Fixes #11144 
/cc @hakman 

Patch:
diff --git a/k8s/crds/kops.k8s.io_clusters.yaml b/k8s/crds/kops.k8s.io_clusters.yaml
index c30d49dfbb..c3db3c5fe5 100644
--- a/k8s/crds/kops.k8s.io_clusters.yaml
+++ b/k8s/crds/kops.k8s.io_clusters.yaml
@@ -3168,10 +3168,11 @@ spec:
                           type: object
                         type: array
                       imageName:
-                        description: The container image name to use
+                        description: ImageName is the container image name to use.
                         type: string
                       initImageName:
-                        description: The init container image name to use
+                        description: InitImageName is the init container image name
+                          to use.
                         type: string
                     type: object
                   calico:
diff --git a/pkg/apis/kops/networking.go b/pkg/apis/kops/networking.go
index 7c88b01d75..f3ed2a80a3 100644
--- a/pkg/apis/kops/networking.go
+++ b/pkg/apis/kops/networking.go
@@ -248,9 +248,9 @@ type RomanaNetworkingSpec struct {
 
 // AmazonVPCNetworkingSpec declares that we want Amazon VPC CNI networking
 type AmazonVPCNetworkingSpec struct {
-	// The container image name to use
+	// ImageName is the container image name to use.
 	ImageName string `json:"imageName,omitempty"`
-	// The init container image name to use
+	// InitImageName is the init container image name to use.
 	InitImageName string `json:"initImageName,omitempty"`
 	// Env is a list of environment variables to set in the container.
 	Env []EnvVar `json:"env,omitempty"`
diff --git a/pkg/apis/kops/v1alpha2/networking.go b/pkg/apis/kops/v1alpha2/networking.go
index e15655b40f..cb44eae4ed 100644
--- a/pkg/apis/kops/v1alpha2/networking.go
+++ b/pkg/apis/kops/v1alpha2/networking.go
@@ -248,9 +248,9 @@ type RomanaNetworkingSpec struct {
 
 // AmazonVPCNetworkingSpec declares that we want Amazon VPC CNI networking
 type AmazonVPCNetworkingSpec struct {
-	// The container image name to use
+	// ImageName is the container image name to use.
 	ImageName string `json:"imageName,omitempty"`
-	// The init container image name to use
+	// InitImageName is the init container image name to use.
 	InitImageName string `json:"initImageName,omitempty"`
 	// Env is a list of environment variables to set in the container.
 	Env []EnvVar `json:"env,omitempty"`
diff --git a/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template b/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template
index eb77ce1a5a..e392be0f30 100644
--- a/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template
+++ b/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template
@@ -106,48 +106,50 @@
                 - "fargate"
       "containers":
       - "env":
-        - "name": "ADDITIONAL_ENI_TAGS"
-          "value": "{}"
-        - "name": "AWS_VPC_CNI_NODE_PORT_SUPPORT"
-          "value": "true"
-        - "name": "AWS_VPC_ENI_MTU"
-          "value": "9001"
-        - "name": "AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER"
-          "value": "false"
-        - "name": "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG"
-          "value": "false"
-        - "name": "AWS_VPC_K8S_CNI_EXTERNALSNAT"
-          "value": "false"
-        - "name": "AWS_VPC_K8S_CNI_LOGLEVEL"
-          "value": "DEBUG"
-        - "name": "AWS_VPC_K8S_CNI_LOG_FILE"
-          "value": "/host/var/log/aws-routed-eni/ipamd.log"
-        - "name": "AWS_VPC_K8S_CNI_RANDOMIZESNAT"
-          "value": "prng"
-        - "name": "AWS_VPC_K8S_CNI_VETHPREFIX"
-          "value": "eni"
-        - "name": "AWS_VPC_K8S_PLUGIN_LOG_FILE"
-          "value": "/var/log/aws-routed-eni/plugin.log"
-        - "name": "AWS_VPC_K8S_PLUGIN_LOG_LEVEL"
-          "value": "DEBUG"
-        - "name": "DISABLE_INTROSPECTION"
-          "value": "false"
-        - "name": "DISABLE_METRICS"
-          "value": "false"
-        - "name": "ENABLE_POD_ENI"
-          "value": "false"
+        {{- range $name, $value := AmazonVpcEnvVars }}
+        - "name": "{{ $name }}"
+          "value": "{{ $value }}"
+        {{- end }}
+        # The below envs are commented-out on purpose and replaced by the above range. 
+        # See https://github.com/kubernetes/kops/issues/11144 for more context.
+        # - "name": "ADDITIONAL_ENI_TAGS"
+        #   "value": "{}"
+        # - "name": "AWS_VPC_CNI_NODE_PORT_SUPPORT"
+        #   "value": "true"
+        # - "name": "AWS_VPC_ENI_MTU"
+        #   "value": "9001"
+        # - "name": "AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER"
+        #   "value": "false"
+        # - "name": "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG"
+        #   "value": "false"
+        # - "name": "AWS_VPC_K8S_CNI_EXTERNALSNAT"
+        #   "value": "false"
+        # - "name": "AWS_VPC_K8S_CNI_LOGLEVEL"
+        #   "value": "DEBUG"
+        # - "name": "AWS_VPC_K8S_CNI_LOG_FILE"
+        #   "value": "/host/var/log/aws-routed-eni/ipamd.log"
+        # - "name": "AWS_VPC_K8S_CNI_RANDOMIZESNAT"
+        #   "value": "prng"
+        # - "name": "AWS_VPC_K8S_CNI_VETHPREFIX"
+        #   "value": "eni"
+        # - "name": "AWS_VPC_K8S_PLUGIN_LOG_FILE"
+        #   "value": "/var/log/aws-routed-eni/plugin.log"
+        # - "name": "AWS_VPC_K8S_PLUGIN_LOG_LEVEL"
+        #   "value": "DEBUG"
+        # - "name": "DISABLE_INTROSPECTION"
+        #   "value": "false"
+        # - "name": "DISABLE_METRICS"
+        #   "value": "false"
+        # - "name": "ENABLE_POD_ENI"
+        #   "value": "false"
         - "name": "MY_NODE_NAME"
           "valueFrom":
             "fieldRef":
               "fieldPath": "spec.nodeName"
-        - "name": "WARM_ENI_TARGET"
-          "value": "1"
+        # - "name": "WARM_ENI_TARGET"
+        #   "value": "1"
         - "name": "CLUSTER_NAME"
           "value": "{{ ClusterName }}"
-        {{- range .Networking.AmazonVPC.Env }}
-        - "name": "{{ .Name }}"
-          "value": "{{ .Value }}"
-        {{- end }}
         "image": "{{- or .Networking.AmazonVPC.ImageName "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.8.0" }}"
         "imagePullPolicy": "Always"
         "livenessProbe":
diff --git a/upup/pkg/fi/cloudup/template_functions.go b/upup/pkg/fi/cloudup/template_functions.go
index 2adac8d858..81cc70a947 100644
--- a/upup/pkg/fi/cloudup/template_functions.go
+++ b/upup/pkg/fi/cloudup/template_functions.go
@@ -149,6 +149,19 @@ func (tf *TemplateFunctions) AddTo(dest template.FuncMap, secretStore fi.SecretS
 		}
 	}
 
+	if cluster.Spec.Networking != nil && cluster.Spec.Networking.AmazonVPC != nil {
+		c := cluster.Spec.Networking.AmazonVPC
+		dest["AmazonVpcEnvVars"] = func() map[string]string {
+			envVars := map[string]string{
+				"AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER": "false",
+			}
+			for _, e := range c.Env {
+				envVars[e.Name] = e.Value
+			}
+			return envVars
+		}
+	}
+
 	if cluster.Spec.Networking != nil && cluster.Spec.Networking.Calico != nil {
 		c := cluster.Spec.Networking.Calico
 		dest["CalicoIPv4PoolIPIPMode"] = func() string {


Test patch:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/manifest.yaml
index deb95692f3..b8060eb670 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/manifest.yaml
@@ -54,7 +54,7 @@ spec:
     version: 1.17.0
   - id: k8s-1.16
     manifest: networking.amazon-vpc-routed-eni/k8s-1.16.yaml
-    manifestHash: b89f35a6abb88822739e20ae2ad7776d8160e806
+    manifestHash: fea48420a1f8d5e929b59e08fb96612032d8e651
     name: networking.amazon-vpc-routed-eni
     needsRollingUpdate: all
     selector:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/networking.amazon-vpc-routed-eni-k8s-1.16.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
index e9989b3b2e..fb6afb45ef 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
@@ -132,48 +132,18 @@ spec:
                 - fargate
       containers:
       - env:
-        - name: ADDITIONAL_ENI_TAGS
-          value: '{}'
-        - name: AWS_VPC_CNI_NODE_PORT_SUPPORT
-          value: "true"
-        - name: AWS_VPC_ENI_MTU
-          value: "9001"
         - name: AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER
           value: "false"
-        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
-          value: "false"
-        - name: AWS_VPC_K8S_CNI_EXTERNALSNAT
-          value: "false"
         - name: AWS_VPC_K8S_CNI_LOGLEVEL
-          value: DEBUG
-        - name: AWS_VPC_K8S_CNI_LOG_FILE
-          value: /host/var/log/aws-routed-eni/ipamd.log
-        - name: AWS_VPC_K8S_CNI_RANDOMIZESNAT
-          value: prng
-        - name: AWS_VPC_K8S_CNI_VETHPREFIX
-          value: eni
-        - name: AWS_VPC_K8S_PLUGIN_LOG_FILE
-          value: /var/log/aws-routed-eni/plugin.log
-        - name: AWS_VPC_K8S_PLUGIN_LOG_LEVEL
-          value: DEBUG
-        - name: DISABLE_INTROSPECTION
-          value: "false"
-        - name: DISABLE_METRICS
-          value: "false"
-        - name: ENABLE_POD_ENI
-          value: "false"
+          value: debug
+        - name: WARM_IP_TARGET
+          value: "10"
         - name: MY_NODE_NAME
           valueFrom:
             fieldRef:
               fieldPath: spec.nodeName
-        - name: WARM_ENI_TARGET
-          value: "1"
         - name: CLUSTER_NAME
           value: minimal.example.com
-        - name: WARM_IP_TARGET
-          value: "10"
-        - name: AWS_VPC_K8S_CNI_LOGLEVEL
-          value: debug
         image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.8.0
         imagePullPolicy: Always
         livenessProbe:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml
index deb95692f3..b8060eb670 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml
@@ -54,7 +54,7 @@ spec:
     version: 1.17.0
   - id: k8s-1.16
     manifest: networking.amazon-vpc-routed-eni/k8s-1.16.yaml
-    manifestHash: b89f35a6abb88822739e20ae2ad7776d8160e806
+    manifestHash: fea48420a1f8d5e929b59e08fb96612032d8e651
     name: networking.amazon-vpc-routed-eni
     needsRollingUpdate: all
     selector:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/networking.amazon-vpc-routed-eni-k8s-1.16.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
index e9989b3b2e..fb6afb45ef 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
@@ -132,48 +132,18 @@ spec:
                 - fargate
       containers:
       - env:
-        - name: ADDITIONAL_ENI_TAGS
-          value: '{}'
-        - name: AWS_VPC_CNI_NODE_PORT_SUPPORT
-          value: "true"
-        - name: AWS_VPC_ENI_MTU
-          value: "9001"
         - name: AWS_VPC_K8S_CNI_CONFIGURE_RPFILTER
           value: "false"
-        - name: AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG
-          value: "false"
-        - name: AWS_VPC_K8S_CNI_EXTERNALSNAT
-          value: "false"
         - name: AWS_VPC_K8S_CNI_LOGLEVEL
-          value: DEBUG
-        - name: AWS_VPC_K8S_CNI_LOG_FILE
-          value: /host/var/log/aws-routed-eni/ipamd.log
-        - name: AWS_VPC_K8S_CNI_RANDOMIZESNAT
-          value: prng
-        - name: AWS_VPC_K8S_CNI_VETHPREFIX
-          value: eni
-        - name: AWS_VPC_K8S_PLUGIN_LOG_FILE
-          value: /var/log/aws-routed-eni/plugin.log
-        - name: AWS_VPC_K8S_PLUGIN_LOG_LEVEL
-          value: DEBUG
-        - name: DISABLE_INTROSPECTION
-          value: "false"
-        - name: DISABLE_METRICS
-          value: "false"
-        - name: ENABLE_POD_ENI
-          value: "false"
+          value: debug
+        - name: WARM_IP_TARGET
+          value: "10"
         - name: MY_NODE_NAME
           valueFrom:
             fieldRef:
               fieldPath: spec.nodeName
-        - name: WARM_ENI_TARGET
-          value: "1"
         - name: CLUSTER_NAME
           value: minimal.example.com
-        - name: WARM_IP_TARGET
-          value: "10"
-        - name: AWS_VPC_K8S_CNI_LOGLEVEL
-          value: debug
         image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.8.0
         imagePullPolicy: Always
         livenessProbe: