Repository: kubernetes/kops (go)

Issue:
Kops 1.29 with AWS CNI Networking not working in restricted AWS accounts
/kind bug

**1. What `kops` version are you running? The command `kops version`, will display
 this information.**

kops version
Client version: 1.29.0

**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**

kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.26.15

**3. What cloud provider are you using?**
AWS

**4. What commands did you run?  What is the simplest way to reproduce this issue?**
Upgrade a cluster running in an AWS account that is restricted to not use region `us-west-2`

**5. What happened after the commands executed?**
aws-node pods are unable to start because image `602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:v1.0.7` cannot be pulled

**6. What did you expect to happen?**
kops update should be successful

The images for aws cni are configurable in the networkingSpec for AWS with the update to aws-cni (https://github.com/kubernetes/kops/pull/16297/files#diff-9b8cbd7c318a268f12b9ddc6312ff0661cdfbddec45db2a4dc738470f0fbd93d) a dependency to image 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent was introduced without the option to customize the image.
This breaks the compatibility for clusters that has restricted image pull capabilities.
I would expect an option for customising the image like 
```yaml
  networking:
    amazonvpc:
      imageName: 602401143452.dkr.ecr.eu-central-1.amazonaws.com/amazon-k8s-cni:v1.18.1
      initImageName: 602401143452.dkr.ecr.eu-central-1.amazonaws.com/amazon-k8s-cni-init:v1.18.1
      networkPolicyAgentImageName: 602401143452.dkr.ecr.eu-central-1.amazonaws.com/amazon/aws-network-policy-agent
```
As the networt policy feature of the AWS CNI is optional it might be a good option to disable it at all. In our case we are using cilium in chained mode with the AWS CNI for enforcing network policies and don't need the AWS CNI feature at all.



Pull request:
Add new API field for VPC CNI's network policy agent image
Fixes #16679

For the new field's json struct tag in v1alpha2, note that I didn't use the `Name` suffix that `imageName` and `initImageName` have. I figure the new field can be consistent with the other v1alpha3 fields but if anyone objects I'm happy to change it to `networkPolicyAgentImageName`. 

Patch:
diff --git a/k8s/crds/kops.k8s.io_clusters.yaml b/k8s/crds/kops.k8s.io_clusters.yaml
index 073dac79ba..6eaa4d4f91 100644
--- a/k8s/crds/kops.k8s.io_clusters.yaml
+++ b/k8s/crds/kops.k8s.io_clusters.yaml
@@ -4856,6 +4856,10 @@ spec:
                         description: InitImageName is the init container image name
                           to use.
                         type: string
+                      networkPolicyAgentImage:
+                        description: NetworkPolicyAgentImage is the container image
+                          to use for the network policy agent
+                        type: string
                     type: object
                   calico:
                     description: CalicoNetworkingSpec declares that we want Calico
diff --git a/pkg/apis/kops/networking.go b/pkg/apis/kops/networking.go
index f95014b11d..be57ccdbe8 100644
--- a/pkg/apis/kops/networking.go
+++ b/pkg/apis/kops/networking.go
@@ -316,6 +316,8 @@ type AmazonVPCNetworkingSpec struct {
 	Image string `json:"image,omitempty"`
 	// InitImage is the init container image name to use.
 	InitImage string `json:"initImage,omitempty"`
+	// NetworkPolicyAgentImage is the container image to use for the network policy agent
+	NetworkPolicyAgentImage string `json:"networkPolicyAgentImage,omitempty"`
 	// Env is a list of environment variables to set in the container.
 	Env []EnvVar `json:"env,omitempty"`
 }
diff --git a/pkg/apis/kops/v1alpha2/networking.go b/pkg/apis/kops/v1alpha2/networking.go
index c1032a38bc..1edc2fbbcd 100644
--- a/pkg/apis/kops/v1alpha2/networking.go
+++ b/pkg/apis/kops/v1alpha2/networking.go
@@ -281,6 +281,8 @@ type AmazonVPCNetworkingSpec struct {
 	Image string `json:"imageName,omitempty"`
 	// InitImageName is the init container image name to use.
 	InitImage string `json:"initImageName,omitempty"`
+	// NetworkPolicyAgentImage is the container image to use for the network policy agent
+	NetworkPolicyAgentImage string `json:"networkPolicyAgentImage,omitempty"`
 	// Env is a list of environment variables to set in the container.
 	Env []EnvVar `json:"env,omitempty"`
 }
diff --git a/pkg/apis/kops/v1alpha2/zz_generated.conversion.go b/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
index ce8a7a6123..60dc3f7f64 100644
--- a/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
+++ b/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
@@ -1504,6 +1504,7 @@ func Convert_kops_AlwaysAllowAuthorizationSpec_To_v1alpha2_AlwaysAllowAuthorizat
 func autoConvert_v1alpha2_AmazonVPCNetworkingSpec_To_kops_AmazonVPCNetworkingSpec(in *AmazonVPCNetworkingSpec, out *kops.AmazonVPCNetworkingSpec, s conversion.Scope) error {
 	out.Image = in.Image
 	out.InitImage = in.InitImage
+	out.NetworkPolicyAgentImage = in.NetworkPolicyAgentImage
 	if in.Env != nil {
 		in, out := &in.Env, &out.Env
 		*out = make([]kops.EnvVar, len(*in))
@@ -1526,6 +1527,7 @@ func Convert_v1alpha2_AmazonVPCNetworkingSpec_To_kops_AmazonVPCNetworkingSpec(in
 func autoConvert_kops_AmazonVPCNetworkingSpec_To_v1alpha2_AmazonVPCNetworkingSpec(in *kops.AmazonVPCNetworkingSpec, out *AmazonVPCNetworkingSpec, s conversion.Scope) error {
 	out.Image = in.Image
 	out.InitImage = in.InitImage
+	out.NetworkPolicyAgentImage = in.NetworkPolicyAgentImage
 	if in.Env != nil {
 		in, out := &in.Env, &out.Env
 		*out = make([]EnvVar, len(*in))
diff --git a/pkg/apis/kops/v1alpha3/networking.go b/pkg/apis/kops/v1alpha3/networking.go
index 9fa5ec59a2..1f65a817cf 100644
--- a/pkg/apis/kops/v1alpha3/networking.go
+++ b/pkg/apis/kops/v1alpha3/networking.go
@@ -286,6 +286,8 @@ type AmazonVPCNetworkingSpec struct {
 	Image string `json:"image,omitempty"`
 	// InitImage is the init container image name to use.
 	InitImage string `json:"initImage,omitempty"`
+	// NetworkPolicyAgentImage is the container image to use for the network policy agent
+	NetworkPolicyAgentImage string `json:"networkPolicyAgentImage,omitempty"`
 	// Env is a list of environment variables to set in the container.
 	Env []EnvVar `json:"env,omitempty"`
 }
diff --git a/pkg/apis/kops/v1alpha3/zz_generated.conversion.go b/pkg/apis/kops/v1alpha3/zz_generated.conversion.go
index 35fd2b1dd6..054d95b16c 100644
--- a/pkg/apis/kops/v1alpha3/zz_generated.conversion.go
+++ b/pkg/apis/kops/v1alpha3/zz_generated.conversion.go
@@ -1664,6 +1664,7 @@ func Convert_kops_AlwaysAllowAuthorizationSpec_To_v1alpha3_AlwaysAllowAuthorizat
 func autoConvert_v1alpha3_AmazonVPCNetworkingSpec_To_kops_AmazonVPCNetworkingSpec(in *AmazonVPCNetworkingSpec, out *kops.AmazonVPCNetworkingSpec, s conversion.Scope) error {
 	out.Image = in.Image
 	out.InitImage = in.InitImage
+	out.NetworkPolicyAgentImage = in.NetworkPolicyAgentImage
 	if in.Env != nil {
 		in, out := &in.Env, &out.Env
 		*out = make([]kops.EnvVar, len(*in))
@@ -1686,6 +1687,7 @@ func Convert_v1alpha3_AmazonVPCNetworkingSpec_To_kops_AmazonVPCNetworkingSpec(in
 func autoConvert_kops_AmazonVPCNetworkingSpec_To_v1alpha3_AmazonVPCNetworkingSpec(in *kops.AmazonVPCNetworkingSpec, out *AmazonVPCNetworkingSpec, s conversion.Scope) error {
 	out.Image = in.Image
 	out.InitImage = in.InitImage
+	out.NetworkPolicyAgentImage = in.NetworkPolicyAgentImage
 	if in.Env != nil {
 		in, out := &in.Env, &out.Env
 		*out = make([]EnvVar, len(*in))
diff --git a/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template b/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template
index fa0c55b625..47c99ed52d 100644
--- a/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template
+++ b/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template
@@ -518,7 +518,7 @@ spec:
           - mountPath: /run/xtables.lock
             name: xtables-lock
         - name: aws-eks-nodeagent
-          image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:v1.1.1
+          image: "{{- or .Networking.AmazonVPC.NetworkPolicyAgentImage "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:v1.1.1" }}"
           env:
             - name: MY_NODE_NAME
               valueFrom:


Test patch:
diff --git a/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_cluster-completed.spec_content
index 45da2f61b4..142eb3e50c 100644
--- a/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_cluster-completed.spec_content
@@ -211,7 +211,10 @@ spec:
     enabled: true
   networkCIDR: 172.20.0.0/16
   networking:
-    amazonvpc: {}
+    amazonvpc:
+      imageName: image:123
+      initImageName: initimage:123
+      networkPolicyAgentImage: networkpolicyagentimage:123
   nodeTerminationHandler:
     cpuRequest: 50m
     deleteSQSMsgIfNodeNotFound: false
diff --git a/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_minimal.example.com-addons-bootstrap_content b/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
index 2178376f5a..7a4e2e6435 100644
--- a/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_minimal.example.com-addons-bootstrap_content
@@ -178,7 +178,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.amazon-vpc-routed-eni/k8s-1.16.yaml
-    manifestHash: 516820e30ab3bc0817c018c36ffd1841d5e6c53b553a0ddd8ae98d7d3779c0fc
+    manifestHash: 84642ad9b609d8e6ce59cbd1bd599e9410416c1619f4734112e1b338c4c4b469
     name: networking.amazon-vpc-routed-eni
     needsRollingUpdate: all
     selector:
diff --git a/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_minimal.example.com-addons-networking.amazon-vpc-routed-eni-k8s-1.16_content b/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_minimal.example.com-addons-networking.amazon-vpc-routed-eni-k8s-1.16_content
index e66a6852df..8a0a7e3a6d 100644
--- a/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_minimal.example.com-addons-networking.amazon-vpc-routed-eni-k8s-1.16_content
+++ b/tests/integration/update_cluster/many-addons-ccm/data/aws_s3_object_minimal.example.com-addons-networking.amazon-vpc-routed-eni-k8s-1.16_content
@@ -514,7 +514,7 @@ spec:
               fieldPath: metadata.name
         - name: CLUSTER_NAME
           value: minimal.example.com
-        image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.18.1
+        image: image:123
         livenessProbe:
           exec:
             command:
@@ -570,7 +570,7 @@ spec:
             fieldRef:
               apiVersion: v1
               fieldPath: spec.nodeName
-        image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:v1.1.1
+        image: networkpolicyagentimage:123
         name: aws-eks-nodeagent
         resources:
           requests:
@@ -596,7 +596,7 @@ spec:
           value: "false"
         - name: ENABLE_IPv6
           value: "false"
-        image: 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni-init:v1.18.1
+        image: initimage:123
         name: aws-vpc-cni-init
         resources:
           requests:
diff --git a/tests/integration/update_cluster/many-addons-ccm/in-v1alpha2.yaml b/tests/integration/update_cluster/many-addons-ccm/in-v1alpha2.yaml
index 36d227a2ba..a4a667110a 100644
--- a/tests/integration/update_cluster/many-addons-ccm/in-v1alpha2.yaml
+++ b/tests/integration/update_cluster/many-addons-ccm/in-v1alpha2.yaml
@@ -39,7 +39,10 @@ spec:
   masterPublicName: api.minimal.example.com
   networkCIDR: 172.20.0.0/16
   networking:
-    amazonvpc: {}
+    amazonvpc:
+      imageName: image:123
+      initImageName: initimage:123
+      networkPolicyAgentImage: networkpolicyagentimage:123
   nodeTerminationHandler:
     enabled: true
     enableRebalanceDraining: true