Repository: kubernetes/kops (go)

Issue:
Calico IPv6 on Flatcar fails to validate cluster
/kind bug

**1. What `kops` version are you running? The command `kops version`, will display
 this information.**

Master branch

**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**

1.27 alpha

**3. What cloud provider are you using?**

AWS

**4. What commands did you run?  What is the simplest way to reproduce this issue?**

https://testgrid.k8s.io/kops-network-plugins#kops-aws-cni-calico-ipv6-flatcar

**5. What happened after the commands executed?**

Cluster fails to validate. One calico-node pod fails to become ready.

**6. What did you expect to happen?**

Cluster to validate

**7. Please provide your cluster manifest. Execute
  `kops get --name my.example.com -o yaml` to display your cluster manifest.
  You may want to remove your cluster name and other sensitive information.**

See test logs

**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
  Paste the logs into this report, or in a gist and provide the gist link here.**

See test logs

**9. Anything else do we need to know?**

This started happening sometime December 9.

Before the failures, calico-node pods would log from the upgrad-ipam container:

```
2022-12-07 19:59:23.613 [INFO][1] migrate.go 69: host-local IPAM data dir dir not found; no migration necessary, successfully exiting...
```

Now they are logging:

```
2022-12-10 04:05:31.847 [INFO][1] ipam_plugin.go 80: migrating from host-local to calico-ipam...
2022-12-10 04:05:31.868 [INFO][1] migrate.go 67: checking host-local IPAM data dir dir existence...
2022-12-10 04:05:31.870 [INFO][1] migrate.go 74: retrieving node for IPIP tunnel address
2022-12-10 04:05:31.901 [INFO][1] migrate.go 82: IPIP tunnel address not found, assigning...
...
2022-12-10 04:05:36.938 [INFO][1] ipam_plugin.go 110: migration from host-local to calico-ipam complete node="i-0488db8d1a5ce4ccf"
```

And one failing node logs:
```
2022-12-10 04:01:28.199 [INFO][1] ipam_plugin.go 80: migrating from host-local to calico-ipam...
2022-12-10 04:01:28.224 [INFO][1] migrate.go 67: checking host-local IPAM data dir dir existence...
2022-12-10 04:01:28.226 [INFO][1] migrate.go 74: retrieving node for IPIP tunnel address
2022-12-10 04:01:58.259 [ERROR][1] ipam_plugin.go 107: failed to migrate ipam, retrying... error=failed to get calico node resource: Get "https://[fd00:5e4f:ce::1]:443/api/v1/nodes/i-0acc88237700261ba": dial tcp [fd00:5e4f:ce::1]:443: i/o timeout node="i-0acc88237700261ba"
2022-12-10 04:01:59.259 [INFO][1] migrate.go 67: checking host-local IPAM data dir dir existence...
...
2022-12-10 04:02:10.355 [INFO][1] migrate.go 232: successfully released lock on host-local backend!
2022-12-10 04:02:10.355 [ERROR][1] ipam_plugin.go 107: failed to migrate ipam, retrying... error=failed to assign IP to calico backend: The provided IP address is not in a configured pool
 node="i-0acc88237700261ba"
...
```

I don't think it should be trying to migrate IPAM.


Pull request:
Calico: Don't try to upgrade IPAM in IPv6 clusters
Fixes #14766 


Patch:
diff --git a/upup/models/cloudup/resources/addons/networking.projectcalico.org/k8s-1.25.yaml.template b/upup/models/cloudup/resources/addons/networking.projectcalico.org/k8s-1.25.yaml.template
index 7596d380d4..4c341259a3 100644
--- a/upup/models/cloudup/resources/addons/networking.projectcalico.org/k8s-1.25.yaml.template
+++ b/upup/models/cloudup/resources/addons/networking.projectcalico.org/k8s-1.25.yaml.template
@@ -4452,6 +4452,7 @@ spec:
       terminationGracePeriodSeconds: 0
       priorityClassName: system-node-critical
       initContainers:
+        {{- if not IsIPv6Only }}
         # This container performs upgrade from host-local IPAM to calico-ipam.
         # It can be deleted if this is a fresh installation, or if you have already
         # upgraded to use calico-ipam.
@@ -4481,6 +4482,7 @@ spec:
               name: cni-bin-dir
           securityContext:
             privileged: true
+        {{- end }}
         # This container installs the CNI binaries
         # and CNI network config file on each node.
         - name: install-cni
@@ -4814,12 +4816,14 @@ spec:
         - name: cni-log-dir
           hostPath:
             path: /var/log/calico/cni
+        {{- if not IsIPv6Only }}
         # Mount in the directory for host-local IPAM allocations. This is
         # used when upgrading from host-local to calico-ipam, and can be removed
         # if not using the upgrade-ipam init container.
         - name: host-local-net-dir
           hostPath:
             path: /var/lib/cni/networks
+        {{- end }}
         # Used to create per-pod Unix Domain Sockets
         - name: policysync
           hostPath:


Test patch:
diff --git a/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content b/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content
index 20b9cc57b9..d2dfca73bf 100644
--- a/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-bootstrap_content
@@ -55,7 +55,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.25
     manifest: networking.projectcalico.org/k8s-1.25.yaml
-    manifestHash: 87046f7e66df6437ddf0a74ce7871dad02e0961a4a47c36a0fd6fff6434aac41
+    manifestHash: 219395986f66685fc3d5b0d070c875b3fc02470f28e4fb2fd6f2e30733ffc153
     name: networking.projectcalico.org
     prune:
       kinds:
diff --git a/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-networking.projectcalico.org-k8s-1.25_content b/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-networking.projectcalico.org-k8s-1.25_content
index c4b627b966..7042d1a69d 100644
--- a/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-networking.projectcalico.org-k8s-1.25_content
+++ b/tests/integration/update_cluster/minimal-ipv6-calico/data/aws_s3_object_minimal-ipv6.example.com-addons-networking.projectcalico.org-k8s-1.25_content
@@ -4631,33 +4631,6 @@ spec:
           readOnly: true
       hostNetwork: true
       initContainers:
-      - command:
-        - /opt/cni/bin/calico-ipam
-        - -upgrade
-        env:
-        - name: KUBERNETES_NODE_NAME
-          valueFrom:
-            fieldRef:
-              fieldPath: spec.nodeName
-        - name: CALICO_NETWORKING_BACKEND
-          valueFrom:
-            configMapKeyRef:
-              key: calico_backend
-              name: calico-config
-        envFrom:
-        - configMapRef:
-            name: kubernetes-services-endpoint
-            optional: true
-        image: docker.io/calico/cni:v3.24.5
-        imagePullPolicy: IfNotPresent
-        name: upgrade-ipam
-        securityContext:
-          privileged: true
-        volumeMounts:
-        - mountPath: /var/lib/cni/networks
-          name: host-local-net-dir
-        - mountPath: /host/opt/cni/bin
-          name: cni-bin-dir
       - command:
         - /opt/cni/bin/install
         env:
@@ -4758,9 +4731,6 @@ spec:
       - hostPath:
           path: /var/log/calico/cni
         name: cni-log-dir
-      - hostPath:
-          path: /var/lib/cni/networks
-        name: host-local-net-dir
       - hostPath:
           path: /var/run/nodeagent
           type: DirectoryOrCreate