Repository: kubernetes/kops (go)

Issue:
[cilium - eni IPAM MODE]  problem with connection from POD to external resource in additional subnet after kops upgrade from 1.23 -> 1.25 
/kind bug

**1. What `kops` version are you running? The command `kops version`, will display
 this information.**

Version 1.23.2 (git-4125bbbe975ca104c609ceaa625a9a5ff3ac19f4) → Client version: 1.25.2 (git-v1.25.2)


**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**

1.23.8

Subnets:
  - cidr: 10.210.140.0/26
  - cidr: 10.210.140.64/26
  - cidr: 10.210.0.0/22


**3. What cloud provider are you using?**

AWS

**4. What commands did you run?  What is the simplest way to reproduce this issue?**

kops1.25 replace -f value.yaml
kops1.25 update cluster --name ${CLUSTER_NAME} --yes

**5. What happened after the commands executed?**

https://gist.github.com/glomikuz/5786db5989fa714cf80dd3642b068ee8
Cluster changes have been applied to the cloud.

**6. What did you expect to happen?**

Pods with IPs from my new subnet(  - cidr: 10.210.0.0/22 ) will able connect to external resources

**7. Please provide your cluster manifest. Execute
  `kops get --name my.example.com -o yaml` to display your cluster manifest.
  You may want to remove your cluster name and other sensitive information.**

```
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  name: k8s.tmp-test.test
spec:
  api:
    loadBalancer:
      type: Internal
      useForInternalApi: true
  authorization:
    rbac: {}
  cloudConfig:
    awsEBSCSIDriver:
      enabled: true
  cloudProvider: aws
  clusterAutoscaler:
    enabled: true
  configBase: s3://states/k8s.tmp-test.test
  containerRuntime: containerd
  docker:
    skipInstall: true
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-1a
      name: a
    manager:
      env:
      - name: ETCD_LISTEN_METRICS_URLS
        value: http://0.0.0.0:2379
      - name: ETCD_METRICS
        value: basic
      - name: ETCD_MANAGER_HOURLY_BACKUPS_RETENTION
        value: 1d
      - name: ETCD_MANAGER_DAILY_BACKUPS_RETENTION
        value: 1d
      - name: ETCD_MAX_REQUEST_BYTES
        value: "6291456"
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-1a
      name: a
    manager:
      env:
      - name: ETCD_MANAGER_HOURLY_BACKUPS_RETENTION
        value: 1d
      - name: ETCD_MANAGER_DAILY_BACKUPS_RETENTION
        value: 1d
      - name: ETCD_MAX_REQUEST_BYTES
        value: "6291456"
    memoryRequest: 100Mi
    name: events
  iam:
    legacy: false
  kubeAPIServer:
    featureGates:
      TTLAfterFinished: "true"
    oidcClientID: kubernetes
    oidcGroupsClaim: groups
    oidcIssuerURL: https://sso.test.io/auth/realms/test
  kubeControllerManager:
    featureGates:
      TTLAfterFinished: "true"
  kubeDNS:
    nodeLocalDNS:
      cpuRequest: 25m
      enabled: true
      memoryRequest: 5Mi
    provider: CoreDNS
  kubeProxy:
    enabled: false
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
    evictionHard: memory.available<7%,nodefs.available<10%,nodefs.inodesFree<5%,imagefs.available<10%,imagefs.inodesFree<5%
    evictionMaxPodGracePeriod: 30
    evictionSoft: memory.available<12%
    evictionSoftGracePeriod: memory.available=200s
  kubernetesApiAccess:
  - 10.0.0.0/8
  kubernetesVersion: 1.23.8
  masterInternalName: api-internal.k8s.tmp-test.test
  masterPublicName: api.k8s.tmp-test.test
  metricsServer:
    enabled: true
    insecure: true
  networkCIDR: 10.210.0.0/16
  networkID: vpc-ID
  networking:
    cilium:
      enableNodePort: true
      enablePrometheusMetrics: true
      ipam: eni
  nodeProblemDetector:
    enabled: false
  nodeTerminationHandler:
    enabled: true
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 10.0.0.0/8
  subnets:
  - cidr: 10.210.140.0/26
    name: kops-k8s-1a
    type: Private
    zone: eu-central-1a
  - cidr: 10.210.140.64/26
    name: kops-k8s-utility-1a
    type: Utility
    zone: eu-central-1a
  topology:
    dns:
      type: Private
    masters: private
    nodes: private
```
Instance groups
```
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: k8s.tmp-test
  name: master-1a
spec:
  image: ami-***
  machineType: t3.small
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-1a
  role: Master
  subnets:
  - kops-k8s-1a
 
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: k8s.tmp-test.test
  name: nodes
spec:
  additionalUserData:
  - content: |
      #!/bin/sh
      mkdir -p /var/lib/test-jvm-dump
    name: add-folder.sh
    type: text/x-shellscript
  image: ami-***
  machineType: t3.small
  maxSize: 2
  minSize: 2
  mixedInstancesPolicy:
    instances:
    - t3.small
    onDemandAboveBase: 0
    onDemandBase: 0
    spotAllocationStrategy: lowest-price
    spotInstancePools: 3
  nodeLabels:
    kops.k8s.io/instancegroup: nodes
  role: Node
  subnets:
  - kops-k8s-1a
  suspendProcesses:
  - AZRebalance
  sysctlParameters:
  - net.netfilter.nf_conntrack_max = 1048576
  - net.core.netdev_max_backlog = 30000
  - net.core.rmem_max = 134217728
  - net.core.wmem_max = 134217728
  - net.ipv4.tcp_wmem = 4096 87380 67108864
  - net.ipv4.tcp_rmem = 4096 87380 67108864
  - net.ipv4.tcp_mem = 187143 249527 1874286
  - net.ipv4.tcp_max_syn_backlog = 8192
  - net.ipv4.ip_local_port_range = 10240 65535
```

**9. Anything else do we need to know?**

Before upgrade all works fine
a get my additional subnet (10.210.0.0/22)  that Cilium using for allocation 

```
kubectl get po  -o wide
NAME                         READY   STATUS              RESTARTS   AGE    IP               NODE
tmp-shell-5b759b78d4-8hbqs   1/1     Running             0          20s   10.210.3.62   ip-10-210-140-26  
tmp-shell-5b759b78d4-dch6v   1/1     Running             0          20s   10.210.0.132  ip-10-210-140-38
tmp-shell-5b759b78d4-dpt7d   1/1     Running             0          20s   10.210.140.37 ip-10-210-140-38  

kubectl exec -it tmp-shell-5b759b78d4-8hbqs bash
ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=51 time=2.04 ms
```

all OK

```
kubectl exec -it tmp-shell-5b759b78d4-dpt7d bash
ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=49 ttl=51 time=1.01 ms
```

all OK

after upgrade to kops 1.25.2

```
kubectl get po  -o wide
NAME                         READY   STATUS              RESTARTS   AGE     IP             NODE
tmp-shell-5b759b78d4-mlxk4   1/1     Running             0          7m45s   10.210.140.46  ip-10-210-140-12
tmp-shell-5b759b78d4-sgf88   1/1     Running             0          7m45s   10.210.1.212   ip-10-210-140-33
tmp-shell-5b759b78d4-smsgq   1/1     Running             0          7m45s   10.210.3.63    ip-10-210-140-12

kubectl exec -it tmp-shell-5b759b78d4-mlxk4 bash
bash-5.2# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=51 time=2.29 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=51 time=1.30 ms
```

ALL OK 

```
kubectl exec -it tmp-shell-5b759b78d4-sgf88 bash
bash-5.2# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
22 packets transmitted, 0 received, 100% packet loss, time 21494ms
```

NOT OK - we have problem with connect to external host

after update kops created new manifest for cilium

```
ManagedFile/k8s.tmp-test.tmp-addons-networking.cilium.io-k8s-1.16
  	Location            	addons/networking.cilium.io/k8s-1.16-v1.11.yaml

- id: k8s-1.16
                    	+     manifest: networking.cilium.io/k8s-1.16-v1.11.yaml
                    	-     manifest: networking.cilium.io/k8s-1.16-v1.10.yaml
                    	+     manifestHash: 6919848c0feff5b38740fecdeaabe0abc2a7d71f148bfed6e701144fcc99a072
                    	-     manifestHash: cd46e38529278b2b4da5819c9d678c3ba71f7fae43acca2462d9fc5c9742b0bf
                    	      name: networking.cilium.io
                    	      needsRollingUpdate: all
```
and new version of cilium agent/operator
```
image: http://quay.io/cilium/cilium:v1.10.5   -> image: http://quay.io/cilium/cilium:v1.11.8@sha256:7f030cfb166a8aeabcb8208844d96abe5b00a3feb251e8ca47f21cf7b3a74f88
```
cilium manifest [kops1.23-cilium1.10.5](https://gist.github.com/glomikuz/54a6a00dc621f261b15be6709556074a) 
cilium manifest [kops1.25-cilium1.11.8](https://gist.github.com/glomikuz/c8c0cf25193d1f940548d005f0a88b4c)

If i downgrade cilium version -> it will works  

```
  networking:
    cilium:
      ****
      version: v1.10.5
```
Any ideas what i need add to Network configuration ?

PS: same issue when upgrades from 1.23 to 1.24

`Client version: 1.24.4 (git-v1.24.4)`

https://gist.github.com/glomikuz/012abde282948c1ff0037ec0ed1f0ec2

```
k get po -o wide     
NAME                         READY   STATUS    RESTARTS   AGE    IP              NODE                                           
tmp-shell-5b759b78d4-6cmdw   1/1     Running   0          5m1s   10.210.2.74     ip-10-210-140-43.eu-central-1.compute.internal   
tmp-shell-5b759b78d4-fw8sn   1/1     Running   0          5m1s   10.210.2.41     ip-10-210-140-53.eu-central-1.compute.internal   
tmp-shell-5b759b78d4-nvtxv   1/1     Running   0          5m1s   10.210.140.59   ip-10-210-140-53.eu-central-1.compute.internal  


k exec -it tmp-shell-5b759b78d4-fw8sn bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
bash-5.2# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6126ms
```
NOT OK

```
k exec -it tmp-shell-5b759b78d4-nvtxv bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
bash-5.2# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=110 time=2.47 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=110 time=1.76 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=110 time=2.05 ms
```
ALL OK with old subnet



Pull request description:
Fix Cilium ENI ipam
Fixes #14575

Patch:
diff --git a/pkg/apis/kops/validation/validation.go b/pkg/apis/kops/validation/validation.go
index 85493dd230..850e6d972e 100644
--- a/pkg/apis/kops/validation/validation.go
+++ b/pkg/apis/kops/validation/validation.go
@@ -1041,8 +1041,8 @@ func validateNetworkingCilium(cluster *kops.Cluster, v *kops.CiliumNetworkingSpe
 			if c.GetCloudProvider() != kops.CloudProviderAWS {
 				allErrs = append(allErrs, field.Forbidden(fldPath.Child("ipam"), "Cilum ENI IPAM is supported only in AWS"))
 			}
-			if v.Masquerade != nil && *v.Masquerade {
-				allErrs = append(allErrs, field.Forbidden(fldPath.Child("masquerade"), "Masquerade must be disabled when ENI IPAM is used"))
+			if v.Masquerade != nil && !*v.Masquerade {
+				allErrs = append(allErrs, field.Forbidden(fldPath.Child("masquerade"), "Masquerade must be enabled when ENI IPAM is used"))
 			}
 			if c.IsIPv6Only() {
 				allErrs = append(allErrs, field.Forbidden(fldPath.Child("ipam"), "Cilium ENI IPAM does not support IPv6"))
diff --git a/pkg/model/components/cilium.go b/pkg/model/components/cilium.go
index 520a448cab..6822f9732e 100644
--- a/pkg/model/components/cilium.go
+++ b/pkg/model/components/cilium.go
@@ -112,7 +112,7 @@ func (b *CiliumOptionsBuilder) BuildOptions(o interface{}) error {
 	}
 
 	if c.Masquerade == nil {
-		c.Masquerade = fi.PtrTo(!clusterSpec.IsIPv6Only() && c.IPAM != "eni")
+		c.Masquerade = fi.PtrTo(!clusterSpec.IsIPv6Only())
 	}
 
 	if c.Tunnel == "" {
@@ -128,7 +128,7 @@ func (b *CiliumOptionsBuilder) BuildOptions(o interface{}) error {
 	}
 
 	if c.EnableBPFMasquerade == nil {
-		c.EnableBPFMasquerade = fi.PtrTo(false)
+		c.EnableBPFMasquerade = fi.PtrTo(c.IPAM == "eni")
 	}
 
 	if c.EnableL7Proxy == nil {
diff --git a/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.11.yaml.template b/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.11.yaml.template
index b085df485c..d88889207d 100644
--- a/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.11.yaml.template
+++ b/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.11.yaml.template
@@ -231,7 +231,7 @@ data:
   # - none
   # - auto (automatically detect the container runtime)
   #
-  masquerade: "{{ .Masquerade }}"
+  enable-ipv4-masquerade: "{{ .Masquerade }}"
   enable-ipv6-masquerade: "false"
   install-iptables-rules: "{{ WithDefaultBool .InstallIptablesRules true }}"
   auto-direct-node-routes: "{{ .AutoDirectNodeRoutes }}"


Test patch:
diff --git a/pkg/apis/kops/validation/validation_test.go b/pkg/apis/kops/validation/validation_test.go
index 4b57224205..eabc5afcd0 100644
--- a/pkg/apis/kops/validation/validation_test.go
+++ b/pkg/apis/kops/validation/validation_test.go
@@ -843,7 +843,7 @@ func Test_Validate_Cilium(t *testing.T) {
 		},
 		{
 			Cilium: kops.CiliumNetworkingSpec{
-				Masquerade: fi.PtrTo(false),
+				Masquerade: fi.PtrTo(true),
 				IPAM:       "eni",
 			},
 			Spec: kops.ClusterSpec{
@@ -860,7 +860,7 @@ func Test_Validate_Cilium(t *testing.T) {
 		},
 		{
 			Cilium: kops.CiliumNetworkingSpec{
-				Masquerade: fi.PtrTo(true),
+				Masquerade: fi.PtrTo(false),
 				IPAM:       "eni",
 			},
 			Spec: kops.ClusterSpec{
diff --git a/tests/integration/update_cluster/minimal-ipv6-cilium/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content b/tests/integration/update_cluster/minimal-ipv6-cilium/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content
index 9c100028d3..ce5a952736 100644
--- a/tests/integration/update_cluster/minimal-ipv6-cilium/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/minimal-ipv6-cilium/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content
@@ -55,7 +55,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.11.yaml
-    manifestHash: 8dca7741f5f2c8cea1f5dd5e2b4fb5c6833816bf6a5968117406e4ab9737985a
+    manifestHash: 8e0768117104113c52ed1ff4bcc311914aa326187a3d10fe18ed63954f16ba0f
     name: networking.cilium.io
     needsRollingUpdate: all
     selector:
diff --git a/tests/integration/update_cluster/minimal-ipv6-cilium/data/aws_s3_object_minimal-ipv6.example.com-addons-networking.cilium.io-k8s-1.16_content b/tests/integration/update_cluster/minimal-ipv6-cilium/data/aws_s3_object_minimal-ipv6.example.com-addons-networking.cilium.io-k8s-1.16_content
index 7c57b917b6..03f703ea0b 100644
--- a/tests/integration/update_cluster/minimal-ipv6-cilium/data/aws_s3_object_minimal-ipv6.example.com-addons-networking.cilium.io-k8s-1.16_content
+++ b/tests/integration/update_cluster/minimal-ipv6-cilium/data/aws_s3_object_minimal-ipv6.example.com-addons-networking.cilium.io-k8s-1.16_content
@@ -44,6 +44,7 @@ data:
   enable-bpf-masquerade: "false"
   enable-endpoint-health-checking: "true"
   enable-ipv4: "false"
+  enable-ipv4-masquerade: "false"
   enable-ipv6: "true"
   enable-ipv6-masquerade: "false"
   enable-l7-proxy: "true"
@@ -55,7 +56,6 @@ data:
   install-iptables-rules: "true"
   ipam: kubernetes
   kube-proxy-replacement: partial
-  masquerade: "false"
   monitor-aggregation: medium
   nodes-gc-interval: 5m0s
   preallocate-bpf-maps: "false"
diff --git a/tests/integration/update_cluster/minimal-warmpool/data/aws_s3_object_minimal-warmpool.example.com-addons-bootstrap_content b/tests/integration/update_cluster/minimal-warmpool/data/aws_s3_object_minimal-warmpool.example.com-addons-bootstrap_content
index 6ede71a8ac..434d4bc582 100644
--- a/tests/integration/update_cluster/minimal-warmpool/data/aws_s3_object_minimal-warmpool.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/minimal-warmpool/data/aws_s3_object_minimal-warmpool.example.com-addons-bootstrap_content
@@ -48,7 +48,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.11.yaml
-    manifestHash: 925bfdc2b33c36273d5c2b1589b801bcf8d1d2b789ff5bd2bd80a840e278795a
+    manifestHash: c3ae71c91e47dbeda0c0a427f4262d3190ad5cb4efaf787033d793ed05c46f63
     name: networking.cilium.io
     needsRollingUpdate: all
     selector:
diff --git a/tests/integration/update_cluster/minimal-warmpool/data/aws_s3_object_minimal-warmpool.example.com-addons-networking.cilium.io-k8s-1.16_content b/tests/integration/update_cluster/minimal-warmpool/data/aws_s3_object_minimal-warmpool.example.com-addons-networking.cilium.io-k8s-1.16_content
index 8e6ea9c239..d26fdad685 100644
--- a/tests/integration/update_cluster/minimal-warmpool/data/aws_s3_object_minimal-warmpool.example.com-addons-networking.cilium.io-k8s-1.16_content
+++ b/tests/integration/update_cluster/minimal-warmpool/data/aws_s3_object_minimal-warmpool.example.com-addons-networking.cilium.io-k8s-1.16_content
@@ -44,6 +44,7 @@ data:
   enable-bpf-masquerade: "false"
   enable-endpoint-health-checking: "true"
   enable-ipv4: "true"
+  enable-ipv4-masquerade: "true"
   enable-ipv6: "false"
   enable-ipv6-masquerade: "false"
   enable-l7-proxy: "true"
@@ -55,7 +56,6 @@ data:
   install-iptables-rules: "true"
   ipam: kubernetes
   kube-proxy-replacement: partial
-  masquerade: "true"
   monitor-aggregation: medium
   nodes-gc-interval: 5m0s
   preallocate-bpf-maps: "false"
diff --git a/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_cluster-completed.spec_content
index 97d9fa875d..e7840005c4 100644
--- a/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_cluster-completed.spec_content
@@ -177,8 +177,8 @@ spec:
       clusterName: default
       cpuRequest: 25m
       disableCNPStatusUpdates: true
-      disableMasquerade: true
-      enableBPFMasquerade: false
+      disableMasquerade: false
+      enableBPFMasquerade: true
       enableEndpointHealthChecking: true
       enableL7Proxy: true
       enableRemoteNodeIdentity: true
diff --git a/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content b/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
index 41af00c11f..8f10e9747d 100644
--- a/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
@@ -48,7 +48,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.11.yaml
-    manifestHash: 8a5107386f0fea73a5d7e14cd94fc20219ac1672e6e35bb9aa529128b0d9bec9
+    manifestHash: e3eb2b6494c1a24704d9663423e8d388acf23a0aabb90651d178a675738f1462
     name: networking.cilium.io
     needsRollingUpdate: all
     selector:
diff --git a/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content b/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
index 76047e2a33..4b13a5c075 100644
--- a/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
+++ b/tests/integration/update_cluster/privatecilium-eni/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
@@ -43,10 +43,11 @@ data:
   debug: "false"
   disable-cnp-status-updates: "true"
   disable-endpoint-crd: "false"
-  enable-bpf-masquerade: "false"
+  enable-bpf-masquerade: "true"
   enable-endpoint-health-checking: "true"
   enable-endpoint-routes: "true"
   enable-ipv4: "true"
+  enable-ipv4-masquerade: "true"
   enable-ipv6: "false"
   enable-ipv6-masquerade: "false"
   enable-l7-proxy: "true"
@@ -58,7 +59,6 @@ data:
   install-iptables-rules: "true"
   ipam: eni
   kube-proxy-replacement: partial
-  masquerade: "false"
   monitor-aggregation: medium
   nodes-gc-interval: 5m0s
   preallocate-bpf-maps: "false"
diff --git a/tests/integration/update_cluster/privatecilium/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content b/tests/integration/update_cluster/privatecilium/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
index 0d13f20468..e468ca3a11 100644
--- a/tests/integration/update_cluster/privatecilium/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/privatecilium/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
@@ -48,7 +48,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.11.yaml
-    manifestHash: b8e6ace39f88ef81ca852eb08adc0f0fa294449c3387849a5a8b5808ad08207b
+    manifestHash: 26c6d43928b2338a73b52d857d7f7bf2676e6cbd6d5c57725f53b6cb45432929
     name: networking.cilium.io
     needsRollingUpdate: all
     selector:
diff --git a/tests/integration/update_cluster/privatecilium/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content b/tests/integration/update_cluster/privatecilium/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
index dcad8196bf..a62cefe919 100644
--- a/tests/integration/update_cluster/privatecilium/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
+++ b/tests/integration/update_cluster/privatecilium/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
@@ -44,6 +44,7 @@ data:
   enable-bpf-masquerade: "false"
   enable-endpoint-health-checking: "true"
   enable-ipv4: "true"
+  enable-ipv4-masquerade: "true"
   enable-ipv6: "false"
   enable-ipv6-masquerade: "false"
   enable-l7-proxy: "true"
@@ -55,7 +56,6 @@ data:
   install-iptables-rules: "true"
   ipam: kubernetes
   kube-proxy-replacement: partial
-  masquerade: "true"
   monitor-aggregation: medium
   nodes-gc-interval: 5m0s
   preallocate-bpf-maps: "false"
diff --git a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
index ed730fcf40..8148a1bc3b 100644
--- a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
@@ -61,7 +61,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.11.yaml
-    manifestHash: e2e5c9b1c0641e661bedfa24749597167be409d1e13e4511c62b68af88f09dc5
+    manifestHash: 83b60d444aea65103ec26335fe93bed3f428a2fcfabf6f5fabfa83521e85f19d
     name: networking.cilium.io
     needsPKI: true
     needsRollingUpdate: all
diff --git a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
index e1b4e23998..d16b4975a9 100644
--- a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
+++ b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
@@ -58,6 +58,7 @@ data:
   enable-endpoint-health-checking: "true"
   enable-hubble: "true"
   enable-ipv4: "true"
+  enable-ipv4-masquerade: "true"
   enable-ipv6: "false"
   enable-ipv6-masquerade: "false"
   enable-l7-proxy: "true"
@@ -75,7 +76,6 @@ data:
   install-iptables-rules: "true"
   ipam: kubernetes
   kube-proxy-replacement: partial
-  masquerade: "true"
   monitor-aggregation: medium
   nodes-gc-interval: 5m0s
   preallocate-bpf-maps: "false"
diff --git a/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_cluster-completed.spec_content
index 6303eaf92c..48776b675b 100644
--- a/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_cluster-completed.spec_content
@@ -185,8 +185,8 @@ spec:
       clusterName: default
       cpuRequest: 25m
       disableCNPStatusUpdates: true
-      disableMasquerade: true
-      enableBPFMasquerade: false
+      disableMasquerade: false
+      enableBPFMasquerade: true
       enableEndpointHealthChecking: true
       enableL7Proxy: true
       enableNodePort: true
diff --git a/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_privateciliumadvanced.example.com-addons-bootstrap_content b/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_privateciliumadvanced.example.com-addons-bootstrap_content
index c1f77a56f8..3fd02d9806 100644
--- a/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_privateciliumadvanced.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_privateciliumadvanced.example.com-addons-bootstrap_content
@@ -48,7 +48,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.11.yaml
-    manifestHash: e4a99245537437ec9596da5feea66740b45d873c840ae3b7dc917884cda582eb
+    manifestHash: b6dde3049975e0e183acfe020a65a5ea08202e02589a536184487c17bfb6b598
     name: networking.cilium.io
     needsRollingUpdate: all
     selector:
diff --git a/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_privateciliumadvanced.example.com-addons-networking.cilium.io-k8s-1.16_content b/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_privateciliumadvanced.example.com-addons-networking.cilium.io-k8s-1.16_content
index 7a454127e7..dcfb09da2c 100644
--- a/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_privateciliumadvanced.example.com-addons-networking.cilium.io-k8s-1.16_content
+++ b/tests/integration/update_cluster/privateciliumadvanced/data/aws_s3_object_privateciliumadvanced.example.com-addons-networking.cilium.io-k8s-1.16_content
@@ -43,10 +43,11 @@ data:
   debug: "false"
   disable-cnp-status-updates: "true"
   disable-endpoint-crd: "false"
-  enable-bpf-masquerade: "false"
+  enable-bpf-masquerade: "true"
   enable-endpoint-health-checking: "true"
   enable-endpoint-routes: "true"
   enable-ipv4: "true"
+  enable-ipv4-masquerade: "true"
   enable-ipv6: "false"
   enable-ipv6-masquerade: "false"
   enable-k8s-event-handover: "true"
@@ -69,7 +70,6 @@ data:
   kube-proxy-replacement: strict
   kvstore: etcd
   kvstore-opt: '{"etcd.config": "/var/lib/etcd-config/etcd.config"}'
-  masquerade: "false"
   monitor-aggregation: medium
   nodes-gc-interval: 5m0s
   preallocate-bpf-maps: "false"
diff --git a/tests/integration/update_cluster/privateciliumadvanced/in-v1alpha2.yaml b/tests/integration/update_cluster/privateciliumadvanced/in-v1alpha2.yaml
index 80227961a3..782ac81d12 100644
--- a/tests/integration/update_cluster/privateciliumadvanced/in-v1alpha2.yaml
+++ b/tests/integration/update_cluster/privateciliumadvanced/in-v1alpha2.yaml
@@ -34,7 +34,7 @@ spec:
     cilium:
       enableNodePort: true
       etcdManaged: true
-      disableMasquerade: true
+      disableMasquerade: false
       ipam: eni
   nonMasqueradeCIDR: 100.64.0.0/10
   sshAccess:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/manifest.yaml
index 0969c508fc..179052d355 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/manifest.yaml
@@ -48,7 +48,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.11.yaml
-    manifestHash: 678572068ce63e2afc9fafd712bdef61d715f2fec45f04a2c2de875271dd6e6d
+    manifestHash: 225f529de36a87bacd6d60df52f0b11c82b2f1b93b880adfd2d76cf625dea72a
     name: networking.cilium.io
     needsRollingUpdate: all
     selector:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/metrics-server/insecure-1.19/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/metrics-server/insecure-1.19/manifest.yaml
index 5445e8ac6f..b971bacf02 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/metrics-server/insecure-1.19/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/metrics-server/insecure-1.19/manifest.yaml
@@ -55,7 +55,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.11.yaml
-    manifestHash: 678572068ce63e2afc9fafd712bdef61d715f2fec45f04a2c2de875271dd6e6d
+    manifestHash: 225f529de36a87bacd6d60df52f0b11c82b2f1b93b880adfd2d76cf625dea72a
     name: networking.cilium.io
     needsRollingUpdate: all
     selector:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/metrics-server/secure-1.19/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/metrics-server/secure-1.19/manifest.yaml
index 53661789ee..f72e669b8e 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/metrics-server/secure-1.19/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/metrics-server/secure-1.19/manifest.yaml
@@ -62,7 +62,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.11.yaml
-    manifestHash: 678572068ce63e2afc9fafd712bdef61d715f2fec45f04a2c2de875271dd6e6d
+    manifestHash: 225f529de36a87bacd6d60df52f0b11c82b2f1b93b880adfd2d76cf625dea72a
     name: networking.cilium.io
     needsRollingUpdate: all
     selector: