Repository: kubernetes/kops (go)

Issue:
not possible to rolling update cluster
**1. What `kops` version are you running? The command `kops version`, will display
 this information.**

1.19 alpha 4

**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**

1.19.1

**3. What cloud provider are you using?**

openstack / aws

**4. What commands did you run?  What is the simplest way to reproduce this issue?**

I am updating clusters from 1.17 / 1.18 -> 1.19.1

kops update cluster --yes && kops rolling-update --yes

**5. What happened after the commands executed?**

Bastion is rotated fine. However, after the bastion is rotated something is executing new manifests for kops-controller. This will lead to following situation:

```
kops-controller-68fc4                                        1/1     Running             2          20d
kops-controller-9hq4s                                        1/1     Running             0          15d
kops-controller-b7jrx                                        0/1     ContainerCreating   0          20m
```

```
% kubectl describe pod kops-controller-b7jrx

  Warning  FailedMount  14s (x18 over 20m)   kubelet, master-zone-1-1-1-jannem-k8s-local  MountVolume.SetUp failed for volume "kops-controller-pki" : hostPath type check failed: /etc/kubernetes/kops-controller/ is not a directory
```

So the new kops-controller manifest is NOT backwards compatible and this means that only way to update kops cluster masters currently is roll ALL masters at once (or skip cluster validation or modify kops controller manifest and be fast). The folder does not exist (yet) in old masters installed by older kops version. This folder is available only in newest kops version. So it should not update the manifest before the folder really exists.

**6. What did you expect to happen?**

I expect that rolling update should work usually



Pull request description:
Add label to prevent kops-controller from running on old nodes
Adds a label and nodeSelector to prevent kops-controller from running on nodes that were created with an older version of kops (and thus don't have the necessary PKI files provisioned by nodeup).

Fixes #9953 

Upgrading a 3-node control plane from kops 1.18.1 appears to be working.

Patch:
diff --git a/docs/getting_started/openstack.md b/docs/getting_started/openstack.md
index 4e0c2d0ea8..663dfe2d9b 100644
--- a/docs/getting_started/openstack.md
+++ b/docs/getting_started/openstack.md
@@ -5,7 +5,7 @@ OpenStack support on kops is currently **beta**, which means that OpenStack supp
 
 ## OpenStack requirements
 
-In order to deploy a kops-managed cluster on OpenStack, you need the following OpenStack services:
+In order to deploy a kops-managed clustr on OpenStack, you need the following OpenStack servces:
 
 * Nova (compute)
 * Neutron (networking)
@@ -219,4 +219,4 @@ spec:
 
 # Next steps
 
-Now that you have a working _kops_ cluster, read through the [recommendations for production setups guide](production.md) to learn more about how to configure _kops_ for production workloads.
+Now that you have a working _kops_ cluster, read through the [recommendations for production setups guide](production.md) to learn more about how to configure _kops_ for production workloads.
\ No newline at end of file
diff --git a/protokube/pkg/protokube/labeler.go b/protokube/pkg/protokube/labeler.go
index e2d4a9c7e2..89a35eb863 100644
--- a/protokube/pkg/protokube/labeler.go
+++ b/protokube/pkg/protokube/labeler.go
@@ -40,7 +40,8 @@ func bootstrapMasterNodeLabels(ctx context.Context, kubeContext *KubernetesConte
 	}
 
 	labels := map[string]string{
-		"node-role.kubernetes.io/master": "",
+		"node-role.kubernetes.io/master":  "",
+		"kops.k8s.io/kops-controller-pki": "",
 	}
 
 	shouldPatch := false
diff --git a/upup/models/bindata.go b/upup/models/bindata.go
index 0ef4bb23f7..157e830089 100644
--- a/upup/models/bindata.go
+++ b/upup/models/bindata.go
@@ -2751,6 +2751,7 @@ spec:
         operator: Exists
       nodeSelector:
         node-role.kubernetes.io/master: ""
+        kops.k8s.io/kops-controller-pki: ""
       dnsPolicy: Default  # Don't use cluster DNS (we are likely running before kube-dns)
       hostNetwork: true
       serviceAccount: kops-controller
diff --git a/upup/models/cloudup/resources/addons/kops-controller.addons.k8s.io/k8s-1.16.yaml.template b/upup/models/cloudup/resources/addons/kops-controller.addons.k8s.io/k8s-1.16.yaml.template
index 382bcca253..ae7b8bb4e0 100644
--- a/upup/models/cloudup/resources/addons/kops-controller.addons.k8s.io/k8s-1.16.yaml.template
+++ b/upup/models/cloudup/resources/addons/kops-controller.addons.k8s.io/k8s-1.16.yaml.template
@@ -41,6 +41,7 @@ spec:
         operator: Exists
       nodeSelector:
         node-role.kubernetes.io/master: ""
+        kops.k8s.io/kops-controller-pki: ""
       dnsPolicy: Default  # Don't use cluster DNS (we are likely running before kube-dns)
       hostNetwork: true
       serviceAccount: kops-controller


Test patch:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml
index f81c1c6bcd..46d0f0b5f4 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/amazonvpc/manifest.yaml
@@ -7,7 +7,7 @@ spec:
   - id: k8s-1.16
     kubernetesVersion: '>=1.16.0-alpha.0'
     manifest: kops-controller.addons.k8s.io/k8s-1.16.yaml
-    manifestHash: b047de23df3b0caff3784aaa89ae0c967e866c95
+    manifestHash: 00457a1b369f839820f5fc094c5610e2065da388
     name: kops-controller.addons.k8s.io
     selector:
       k8s-addon: kops-controller.addons.k8s.io
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/manifest.yaml
index 3b4ec1fa88..15d60137cf 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/manifest.yaml
@@ -7,7 +7,7 @@ spec:
   - id: k8s-1.16
     kubernetesVersion: '>=1.16.0-alpha.0'
     manifest: kops-controller.addons.k8s.io/k8s-1.16.yaml
-    manifestHash: 3c73d81e8c87bf4f409323ca904d22ee9c375284
+    manifestHash: db684c93db61d505f87af71577409945bc290fc1
     name: kops-controller.addons.k8s.io
     selector:
       k8s-addon: kops-controller.addons.k8s.io
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/kops-controller.addons.k8s.io-k8s-1.16.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/kops-controller.addons.k8s.io-k8s-1.16.yaml
index 146da3823d..afcf5c93d0 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/kops-controller.addons.k8s.io-k8s-1.16.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/kops-controller.addons.k8s.io-k8s-1.16.yaml
@@ -52,6 +52,7 @@ spec:
       dnsPolicy: Default
       hostNetwork: true
       nodeSelector:
+        kops.k8s.io/kops-controller-pki: ""
         node-role.kubernetes.io/master: ""
       priorityClassName: system-node-critical
       serviceAccount: kops-controller
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/manifest.yaml
index 174e8126b7..56febff913 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/cilium/manifest.yaml
@@ -7,7 +7,7 @@ spec:
   - id: k8s-1.16
     kubernetesVersion: '>=1.16.0-alpha.0'
     manifest: kops-controller.addons.k8s.io/k8s-1.16.yaml
-    manifestHash: b047de23df3b0caff3784aaa89ae0c967e866c95
+    manifestHash: 00457a1b369f839820f5fc094c5610e2065da388
     name: kops-controller.addons.k8s.io
     selector:
       k8s-addon: kops-controller.addons.k8s.io
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/kops-controller.addons.k8s.io-k8s-1.16.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/kops-controller.addons.k8s.io-k8s-1.16.yaml
index 146da3823d..afcf5c93d0 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/kops-controller.addons.k8s.io-k8s-1.16.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/kops-controller.addons.k8s.io-k8s-1.16.yaml
@@ -52,6 +52,7 @@ spec:
       dnsPolicy: Default
       hostNetwork: true
       nodeSelector:
+        kops.k8s.io/kops-controller-pki: ""
         node-role.kubernetes.io/master: ""
       priorityClassName: system-node-critical
       serviceAccount: kops-controller
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/manifest.yaml
index 4cf22d55ee..062b7fb806 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/manifest.yaml
@@ -7,7 +7,7 @@ spec:
   - id: k8s-1.16
     kubernetesVersion: '>=1.16.0-alpha.0'
     manifest: kops-controller.addons.k8s.io/k8s-1.16.yaml
-    manifestHash: b047de23df3b0caff3784aaa89ae0c967e866c95
+    manifestHash: 00457a1b369f839820f5fc094c5610e2065da388
     name: kops-controller.addons.k8s.io
     selector:
       k8s-addon: kops-controller.addons.k8s.io
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/kops-controller.addons.k8s.io-k8s-1.16.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/kops-controller.addons.k8s.io-k8s-1.16.yaml
index dfa28e93b0..5d3bc662a9 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/kops-controller.addons.k8s.io-k8s-1.16.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/kops-controller.addons.k8s.io-k8s-1.16.yaml
@@ -52,6 +52,7 @@ spec:
       dnsPolicy: Default
       hostNetwork: true
       nodeSelector:
+        kops.k8s.io/kops-controller-pki: ""
         node-role.kubernetes.io/master: ""
       priorityClassName: system-node-critical
       serviceAccount: kops-controller
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/manifest.yaml
index 719dd81a07..8de8dbe589 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/manifest.yaml
@@ -7,7 +7,7 @@ spec:
   - id: k8s-1.16
     kubernetesVersion: '>=1.16.0-alpha.0'
     manifest: kops-controller.addons.k8s.io/k8s-1.16.yaml
-    manifestHash: 3c73d81e8c87bf4f409323ca904d22ee9c375284
+    manifestHash: db684c93db61d505f87af71577409945bc290fc1
     name: kops-controller.addons.k8s.io
     selector:
       k8s-addon: kops-controller.addons.k8s.io
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/weave/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/weave/manifest.yaml
index 13f4219699..a01cbba5a2 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/weave/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/weave/manifest.yaml
@@ -7,7 +7,7 @@ spec:
   - id: k8s-1.16
     kubernetesVersion: '>=1.16.0-alpha.0'
     manifest: kops-controller.addons.k8s.io/k8s-1.16.yaml
-    manifestHash: b047de23df3b0caff3784aaa89ae0c967e866c95
+    manifestHash: 00457a1b369f839820f5fc094c5610e2065da388
     name: kops-controller.addons.k8s.io
     selector:
       k8s-addon: kops-controller.addons.k8s.io