Repository: kubernetes/kops (go)

Issue:
When migrating a pre-existing cluster to containerd, the AWS VPC CNI DaemonSet still mounts dockershim
**1. What `kops` version are you running? The command `kops version`, will display
 this information.**
1.20.1

**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**
1.19.10

**3. What cloud provider are you using?**
AWS

**4. What commands did you run?  What is the simplest way to reproduce this issue?**
Take a long-running cluster that uses the AWS VPC CNI define the cluster's `spec.containerRuntime` as `containerd`, and begin rolling the nodes.

**5. What happened after the commands executed?**
Cluster refuses to validate because of `aws-node` pods in `CrashLoopBackoff`.
Upon examination, we see in the describe output (partial output supplied, for clarity):
```text
Containers:                                                                                                                                              
  aws-node:                                                                                                                                              
    Container ID:   containerd://5c55c39d43ba87a3e968c524e54dc3b6675e222d9289a6fe61e117a08d7140c0                                                        
    Image:          602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.7.10                                                                  
    State:          Running                                                                                                                              
    Ready:          False                                                                                                                                
    Mounts:                                                                                                                                              
      /host/etc/cni/net.d from cni-net-dir (rw)                                                                                                          
      /host/opt/cni/bin from cni-bin-dir (rw)                                                                                                            
      /host/var/log/aws-routed-eni from log-dir (rw)                                                                                                     
      /run/xtables.lock from xtables-lock (rw)                                                                                                           
      /var/run/aws-node from run-dir (rw)                                                                                                                
      /var/run/dockershim.sock from dockershim (rw)                                                                                                      
      /var/run/secrets/kubernetes.io/serviceaccount from aws-node-token-98jgt (ro)                                                                       
Conditions:                                              
  Type              Status                               
  Initialized       True                                 
  Ready             False                                
  ContainersReady   False                                
  PodScheduled      True                                 
Volumes:                                                 
  cni-bin-dir:                                           
    Type:          HostPath (bare host directory volume)                    
    Path:          /opt/cni/bin                          
    HostPathType:                                        
  cni-net-dir:                                           
    Type:          HostPath (bare host directory volume)                    
    Path:          /etc/cni/net.d                        
    HostPathType:                                        
  dockershim:                                            
    Type:          HostPath (bare host directory volume)                    
    Path:          /var/run/dockershim.sock              
    HostPathType:                                        
  xtables-lock:                                          
    Type:          HostPath (bare host directory volume)                    
    Path:          /run/xtables.lock                     
    HostPathType:                                        
  log-dir:                                               
    Type:          HostPath (bare host directory volume)                    
    Path:          /var/log/aws-routed-eni               
    HostPathType:  DirectoryOrCreate                     
  run-dir:                                               
    Type:          HostPath (bare host directory volume)                    
    Path:          /var/run/aws-node                     
    HostPathType:  DirectoryOrCreate                     
  aws-node-token-98jgt:                                  
    Type:        Secret (a volume populated by a Secret)                    
    SecretName:  aws-node-token-98jgt                    
    Optional:    false                                   
```

**6. What did you expect to happen?**
https://github.com/kubernetes/kops/pull/10502 changed the mount to containerd.sock, which I presume is mandatory when the underlying node has been upgraded to `containerd`.

This is a tricky upgrade on a production cluster as we don't want to update all of the pods on the DaemonSet before the underlying nodes have been rolled. Ideally, there would be two DaemonSets, one that selects onto `dockershim` nodes and one that selects onto `containerd` nodes, with the former DaemonSet cleaned up after the upgrade, or some other way of ensuring that the DaemonSet has access to the relevant socket as the nodes are upgraded.

Pull request:
Use the OnDelete updateStrategy for AWS VPC CNI DaemonSet
Fixes #11589 

Patch:
diff --git a/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template b/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template
index 20e2084a3a..bbcd774c36 100644
--- a/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template
+++ b/upup/models/cloudup/resources/addons/networking.amazon-vpc-routed-eni/k8s-1.16.yaml.template
@@ -226,9 +226,7 @@
           "type": "DirectoryOrCreate"
         "name": "run-dir"
   "updateStrategy":
-    "rollingUpdate":
-      "maxUnavailable": "10%"
-    "type": "RollingUpdate"
+    "type": "OnDelete"
 ---
 "apiVersion": "v1"
 "kind": "ServiceAccount"
diff --git a/upup/pkg/fi/cloudup/bootstrapchannelbuilder/bootstrapchannelbuilder.go b/upup/pkg/fi/cloudup/bootstrapchannelbuilder/bootstrapchannelbuilder.go
index ebe9d88a5d..9697780618 100644
--- a/upup/pkg/fi/cloudup/bootstrapchannelbuilder/bootstrapchannelbuilder.go
+++ b/upup/pkg/fi/cloudup/bootstrapchannelbuilder/bootstrapchannelbuilder.go
@@ -816,11 +816,12 @@ func (b *BootstrapChannelBuilder) buildAddons(c *fi.ModelBuilderContext) (*chann
 			location := key + "/" + id + ".yaml"
 
 			addons.Spec.Addons = append(addons.Spec.Addons, &channelsapi.AddonSpec{
-				Name:     fi.String(key),
-				Version:  fi.String(versions[id]),
-				Selector: networkingSelector(),
-				Manifest: fi.String(location),
-				Id:       id,
+				Name:               fi.String(key),
+				Version:            fi.String(versions[id]),
+				Selector:           networkingSelector(),
+				Manifest:           fi.String(location),
+				Id:                 id,
+				NeedsRollingUpdate: "all",
 			})
 		}
 	}


Test patch:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/manifest.yaml
index 3cd7cf4aea..f16b09442a 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/manifest.yaml
@@ -54,8 +54,9 @@ spec:
     version: 1.17.0
   - id: k8s-1.16
     manifest: networking.amazon-vpc-routed-eni/k8s-1.16.yaml
-    manifestHash: ba5c98ae679d9feb8a84a063075acee4956cc126
+    manifestHash: 997781194bcdfa7602dfb69e2a1f42cb5106fc81
     name: networking.amazon-vpc-routed-eni
+    needsRollingUpdate: all
     selector:
       role.kubernetes.io/networking: "1"
     version: 1.7.8-kops.1
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/networking.amazon-vpc-routed-eni-k8s-1.16.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
index 08dc4b2e47..18a85b78e2 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc-containerd/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
@@ -252,9 +252,7 @@ spec:
           type: DirectoryOrCreate
         name: run-dir
   updateStrategy:
-    rollingUpdate:
-      maxUnavailable: 10%
-    type: RollingUpdate
+    type: OnDelete
 
 ---
 
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml
index 3cd7cf4aea..f16b09442a 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml
@@ -54,8 +54,9 @@ spec:
     version: 1.17.0
   - id: k8s-1.16
     manifest: networking.amazon-vpc-routed-eni/k8s-1.16.yaml
-    manifestHash: ba5c98ae679d9feb8a84a063075acee4956cc126
+    manifestHash: 997781194bcdfa7602dfb69e2a1f42cb5106fc81
     name: networking.amazon-vpc-routed-eni
+    needsRollingUpdate: all
     selector:
       role.kubernetes.io/networking: "1"
     version: 1.7.8-kops.1
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/networking.amazon-vpc-routed-eni-k8s-1.16.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
index 08dc4b2e47..18a85b78e2 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/networking.amazon-vpc-routed-eni-k8s-1.16.yaml
@@ -252,9 +252,7 @@ spec:
           type: DirectoryOrCreate
         name: run-dir
   updateStrategy:
-    rollingUpdate:
-      maxUnavailable: 10%
-    type: RollingUpdate
+    type: OnDelete
 
 ---