Repository: kubernetes/kops (go)

Issue:
Inaccurate instructions for enabling Cilium
**1. What `kops` version are you running? The command `kops version`, will display
 this information.**
master

**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**
1.19.1

**3. What cloud provider are you using?**
AWS

**4. What commands did you run?  What is the simplest way to reproduce this issue?**

```
kops create cluster --name foo.k8s.local --node-count 1 --master-count 1 --zones us-west-2a --cloud aws
kops edit cluster --name foo.k8s.local # see details below
```
**5. What happened after the commands executed?**
`edit cluster` failed
**6. What did you expect to happen?**
`edit cluster` to succeed
**7. Please provide your cluster manifest. Execute
  `kops get --name my.example.com -o yaml` to display your cluster manifest.
  You may want to remove your cluster name and other sensitive information.**

```yaml
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2020-09-17T20:04:28Z"
  generation: 2
  name: foo.k8s.local
spec:
  api:
    dns: {}
  authentication:
    aws:
      backendMode: CRD
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://tune-kops-state-us-west-2/foo.k8s.local
  containerRuntime: docker
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - instanceGroup: master-us-west-2a
      name: a
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - instanceGroup: master-us-west-2a
      name: a
    memoryRequest: 100Mi
    name: events
  iam:
    allowContainerRegistry: true
    legacy: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 67.4.208.203/32
  kubernetesVersion: 1.19.0-rc.1
  masterInternalName: api.internal.foo.k8s.local
  masterPublicName: api.foo.k8s.local
  networkCIDR: 172.20.0.0/16
  networking:
    cilium:
      hubble: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 67.4.208.203/32
  subnets:
  - cidr: 172.20.32.0/19
    name: us-west-2a
    type: Public
    zone: us-west-2a
  topology:
    dns:
      type: Public
    masters: public
    nodes: public
```

**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
  Paste the logs into this report, or in a gist and provide the gist link here.**

**9. Anything else do we need to know?**

The [docs](https://kops.sigs.k8s.io/networking/cilium/#installing-cilium-on-a-new-cluster) mention that enabling cilium just requires:

```yaml
spec:
  networking:
    cilium: {}
```

but the following error is reported when using this cluster spec:
```
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
# Found fields that are not recognized
# ...
#     networkCIDR: 172.20.0.0/16
#     networking:
# -     cilium:
# +     cilium: {}
# -       hubble: {}
#     nonMasqueradeCIDR: 100.64.0.0/10
#     sshAccess:
# ...
```

It seems like we should either make the hubble field optional by making it a pointer, or update the docs to include `hubble: {}`


Pull request:
Cilium hubble pointer
Fixes #9966 

/cc @hakman @rifelpet 

Patch:
diff --git a/channels/alpha b/channels/alpha
index 9b458b544f..ae93f4946c 100644
--- a/channels/alpha
+++ b/channels/alpha
@@ -44,7 +44,7 @@ spec:
     - name: kope.io/k8s-1.17-debian-stretch-amd64-hvm-ebs-2020-07-20
       providerID: aws
       kubernetesVersion: ">=1.17.0 <1.18.0"
-    - name: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200907
+    - name: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200817
       providerID: aws
       kubernetesVersion: ">=1.18.0"
     - providerID: gce
@@ -59,10 +59,10 @@ spec:
       kubenet: {}
   kubernetesVersions:
   - range: ">=1.18.0"
-    recommendedVersion: 1.18.9
+    recommendedVersion: 1.18.8
     requiredVersion: 1.18.0
   - range: ">=1.17.0"
-    recommendedVersion: 1.17.12
+    recommendedVersion: 1.17.11
     requiredVersion: 1.17.0
   - range: ">=1.16.0"
     recommendedVersion: 1.16.15
@@ -89,15 +89,15 @@ spec:
   - range: ">=1.19.0-alpha.1"
     #recommendedVersion: "1.19.0-alpha.1"
     #requiredVersion: 1.19.0
-    kubernetesVersion: 1.19.2
+    kubernetesVersion: 1.19.0-rc.1
   - range: ">=1.18.0-alpha.1"
     recommendedVersion: "1.18.0"
     #requiredVersion: 1.18.0
-    kubernetesVersion: 1.18.9
+    kubernetesVersion: 1.18.8
   - range: ">=1.17.0-alpha.1"
     recommendedVersion: "1.17.1"
     #requiredVersion: 1.17.0
-    kubernetesVersion: 1.17.12
+    kubernetesVersion: 1.17.11
   - range: ">=1.16.0-alpha.1"
     recommendedVersion: "1.16.4"
     #requiredVersion: 1.16.0
diff --git a/channels/stable b/channels/stable
index ae93f4946c..949fea0f91 100644
--- a/channels/stable
+++ b/channels/stable
@@ -44,7 +44,7 @@ spec:
     - name: kope.io/k8s-1.17-debian-stretch-amd64-hvm-ebs-2020-07-20
       providerID: aws
       kubernetesVersion: ">=1.17.0 <1.18.0"
-    - name: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200817
+    - name: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200716
       providerID: aws
       kubernetesVersion: ">=1.18.0"
     - providerID: gce
@@ -65,7 +65,7 @@ spec:
     recommendedVersion: 1.17.11
     requiredVersion: 1.17.0
   - range: ">=1.16.0"
-    recommendedVersion: 1.16.15
+    recommendedVersion: 1.16.14
     requiredVersion: 1.16.0
   - range: ">=1.15.0"
     recommendedVersion: 1.15.12
@@ -101,7 +101,7 @@ spec:
   - range: ">=1.16.0-alpha.1"
     recommendedVersion: "1.16.4"
     #requiredVersion: 1.16.0
-    kubernetesVersion: 1.16.15
+    kubernetesVersion: 1.16.14
   - range: ">=1.15.0-alpha.1"
     recommendedVersion: "1.15.3"
     #requiredVersion: 1.15.0
diff --git a/pkg/apis/kops/networking.go b/pkg/apis/kops/networking.go
index 638ca02136..fd97620b50 100644
--- a/pkg/apis/kops/networking.go
+++ b/pkg/apis/kops/networking.go
@@ -441,7 +441,7 @@ type CiliumNetworkingSpec struct {
 	// Default: false
 	EnableRemoteNodeIdentity *bool `json:"enableRemoteNodeIdentity,omitempty"`
 	// Hubble configures the Hubble service on the Cilium agent.
-	Hubble HubbleSpec `json:"hubble,omitempty"`
+	Hubble *HubbleSpec `json:"hubble,omitempty"`
 
 	// RemoveCbrBridge is not implemented and may be removed in the future.
 	// Setting this has no effect.
diff --git a/pkg/apis/kops/v1alpha2/networking.go b/pkg/apis/kops/v1alpha2/networking.go
index a868f1f584..f1e9a94eec 100644
--- a/pkg/apis/kops/v1alpha2/networking.go
+++ b/pkg/apis/kops/v1alpha2/networking.go
@@ -439,7 +439,7 @@ type CiliumNetworkingSpec struct {
 	// Default: false
 	EnableRemoteNodeIdentity *bool `json:"enableRemoteNodeIdentity,omitempty"`
 	// Hubble configures the Hubble service on the Cilium agent.
-	Hubble HubbleSpec `json:"hubble,omitempty"`
+	Hubble *HubbleSpec `json:"hubble,omitempty"`
 
 	// RemoveCbrBridge is not implemented and may be removed in the future.
 	// Setting this has no effect.
diff --git a/pkg/apis/kops/v1alpha2/zz_generated.conversion.go b/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
index 1ab4b3dae4..4fdaaa0a43 100644
--- a/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
+++ b/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
@@ -1474,8 +1474,14 @@ func autoConvert_v1alpha2_CiliumNetworkingSpec_To_kops_CiliumNetworkingSpec(in *
 	out.EnableNodePort = in.EnableNodePort
 	out.EtcdManaged = in.EtcdManaged
 	out.EnableRemoteNodeIdentity = in.EnableRemoteNodeIdentity
-	if err := Convert_v1alpha2_HubbleSpec_To_kops_HubbleSpec(&in.Hubble, &out.Hubble, s); err != nil {
-		return err
+	if in.Hubble != nil {
+		in, out := &in.Hubble, &out.Hubble
+		*out = new(kops.HubbleSpec)
+		if err := Convert_v1alpha2_HubbleSpec_To_kops_HubbleSpec(*in, *out, s); err != nil {
+			return err
+		}
+	} else {
+		out.Hubble = nil
 	}
 	out.RemoveCbrBridge = in.RemoveCbrBridge
 	out.RestartPods = in.RestartPods
@@ -1560,8 +1566,14 @@ func autoConvert_kops_CiliumNetworkingSpec_To_v1alpha2_CiliumNetworkingSpec(in *
 	out.EnableNodePort = in.EnableNodePort
 	out.EtcdManaged = in.EtcdManaged
 	out.EnableRemoteNodeIdentity = in.EnableRemoteNodeIdentity
-	if err := Convert_kops_HubbleSpec_To_v1alpha2_HubbleSpec(&in.Hubble, &out.Hubble, s); err != nil {
-		return err
+	if in.Hubble != nil {
+		in, out := &in.Hubble, &out.Hubble
+		*out = new(HubbleSpec)
+		if err := Convert_kops_HubbleSpec_To_v1alpha2_HubbleSpec(*in, *out, s); err != nil {
+			return err
+		}
+	} else {
+		out.Hubble = nil
 	}
 	out.RemoveCbrBridge = in.RemoveCbrBridge
 	out.RestartPods = in.RestartPods
diff --git a/pkg/apis/kops/v1alpha2/zz_generated.deepcopy.go b/pkg/apis/kops/v1alpha2/zz_generated.deepcopy.go
index 2f2f3291b6..d95b474ee8 100644
--- a/pkg/apis/kops/v1alpha2/zz_generated.deepcopy.go
+++ b/pkg/apis/kops/v1alpha2/zz_generated.deepcopy.go
@@ -386,7 +386,11 @@ func (in *CiliumNetworkingSpec) DeepCopyInto(out *CiliumNetworkingSpec) {
 		*out = new(bool)
 		**out = **in
 	}
-	in.Hubble.DeepCopyInto(&out.Hubble)
+	if in.Hubble != nil {
+		in, out := &in.Hubble, &out.Hubble
+		*out = new(HubbleSpec)
+		(*in).DeepCopyInto(*out)
+	}
 	return
 }
 
diff --git a/pkg/apis/kops/validation/validation.go b/pkg/apis/kops/validation/validation.go
index 20ce13adad..ccd6992580 100644
--- a/pkg/apis/kops/validation/validation.go
+++ b/pkg/apis/kops/validation/validation.go
@@ -705,7 +705,7 @@ func validateNetworkingCilium(cluster *kops.Cluster, v *kops.CiliumNetworkingSpe
 			allErrs = append(allErrs, field.Forbidden(versionFld, "Version 1.8 requires kubernetesVersion 1.12 or newer"))
 		}
 
-		if fi.BoolValue(v.Hubble.Enabled) {
+		if v.Hubble != nil && fi.BoolValue(v.Hubble.Enabled) {
 			if version.Minor < 8 {
 				allErrs = append(allErrs, field.Forbidden(fldPath.Root().Child("hubble", "enabled"), "Hubble requires Cilium 1.8 or newer"))
 			}
diff --git a/pkg/apis/kops/zz_generated.deepcopy.go b/pkg/apis/kops/zz_generated.deepcopy.go
index 22abfb9a8e..0ccaeda0d8 100644
--- a/pkg/apis/kops/zz_generated.deepcopy.go
+++ b/pkg/apis/kops/zz_generated.deepcopy.go
@@ -486,7 +486,11 @@ func (in *CiliumNetworkingSpec) DeepCopyInto(out *CiliumNetworkingSpec) {
 		*out = new(bool)
 		**out = **in
 	}
-	in.Hubble.DeepCopyInto(&out.Hubble)
+	if in.Hubble != nil {
+		in, out := &in.Hubble, &out.Hubble
+		*out = new(HubbleSpec)
+		(*in).DeepCopyInto(*out)
+	}
 	return
 }
 
diff --git a/pkg/model/components/cilium.go b/pkg/model/components/cilium.go
index 2f923dc666..2fb8f0e1bd 100644
--- a/pkg/model/components/cilium.go
+++ b/pkg/model/components/cilium.go
@@ -96,6 +96,17 @@ func (b *CiliumOptionsBuilder) BuildOptions(o interface{}) error {
 		c.EnableRemoteNodeIdentity = fi.Bool(true)
 	}
 
+	hubble := c.Hubble
+	if hubble != nil {
+		if hubble.Enabled == nil {
+			hubble.Enabled = fi.Bool(true)
+		}
+	} else {
+		c.Hubble = &kops.HubbleSpec{
+			Enabled: fi.Bool(false),
+		}
+	}
+
 	return nil
 
 }
diff --git a/upup/models/bindata.go b/upup/models/bindata.go
index 3113db4ee5..a54a250083 100644
--- a/upup/models/bindata.go
+++ b/upup/models/bindata.go
@@ -4551,7 +4551,7 @@ data:
   {{ end }}
   {{ end }}
 
-  {{ if .Hubble.Enabled }}
+  {{ if WithDefaultBool .Hubble.Enabled false }}
   # Enable Hubble gRPC service.
   enable-hubble: "true"
   # UNIX domain socket for Hubble server to listen to.
@@ -5212,7 +5212,7 @@ spec:
       {{ end }}
 {{ end }}
 
-{{ if .Networking.Cilium.Hubble.Enabled }}
+{{ if WithDefaultBool .Networking.Cilium.Hubble.Enabled false }}
 ---
 # Source: cilium/charts/hubble-relay/templates/service.yaml
 kind: Service
diff --git a/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.12-v1.8.yaml.template b/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.12-v1.8.yaml.template
index a1d3f73dac..cae7dea195 100644
--- a/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.12-v1.8.yaml.template
+++ b/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.12-v1.8.yaml.template
@@ -166,7 +166,7 @@ data:
   {{ end }}
   {{ end }}
 
-  {{ if .Hubble.Enabled }}
+  {{ if WithDefaultBool .Hubble.Enabled false }}
   # Enable Hubble gRPC service.
   enable-hubble: "true"
   # UNIX domain socket for Hubble server to listen to.
@@ -827,7 +827,7 @@ spec:
       {{ end }}
 {{ end }}
 
-{{ if .Networking.Cilium.Hubble.Enabled }}
+{{ if WithDefaultBool .Networking.Cilium.Hubble.Enabled false }}
 ---
 # Source: cilium/charts/hubble-relay/templates/service.yaml
 kind: Service
diff --git a/upup/pkg/fi/cloudup/awsup/aws_cloud.go b/upup/pkg/fi/cloudup/awsup/aws_cloud.go
index 6afd2fa046..dd7fdc508b 100644
--- a/upup/pkg/fi/cloudup/awsup/aws_cloud.go
+++ b/upup/pkg/fi/cloudup/awsup/aws_cloud.go
@@ -847,11 +847,9 @@ func addCloudInstanceData(cm *cloudinstances.CloudInstance, instance *ec2.Instan
 }
 
 func findInstances(c AWSCloud, ig *kops.InstanceGroup) (map[string]*ec2.Instance, error) {
-	clusterName := c.Tags()[TagClusterName]
 	req := &ec2.DescribeInstancesInput{
 		Filters: []*ec2.Filter{
 			NewEC2Filter("tag:"+identity_aws.CloudTagInstanceGroupName, ig.ObjectMeta.Name),
-			NewEC2Filter("tag:"+TagClusterName, clusterName),
 			NewEC2Filter("instance-state-name", "pending", "running", "stopping", "stopped"),
 		},
 	}
@@ -873,11 +871,9 @@ func findInstances(c AWSCloud, ig *kops.InstanceGroup) (map[string]*ec2.Instance
 }
 
 func findDetachedInstances(c AWSCloud, g *autoscaling.Group) ([]*string, error) {
-	clusterName := c.Tags()[TagClusterName]
 	req := &ec2.DescribeInstancesInput{
 		Filters: []*ec2.Filter{
 			NewEC2Filter("tag:"+tagNameDetachedInstance, aws.StringValue(g.AutoScalingGroupName)),
-			NewEC2Filter("tag:"+TagClusterName, clusterName),
 			NewEC2Filter("instance-state-name", "pending", "running", "stopping", "stopped"),
 		},
 	}


Test patch:
diff --git a/pkg/apis/kops/validation/validation_test.go b/pkg/apis/kops/validation/validation_test.go
index 1df10af153..fa73bca92e 100644
--- a/pkg/apis/kops/validation/validation_test.go
+++ b/pkg/apis/kops/validation/validation_test.go
@@ -630,7 +630,7 @@ func Test_Validate_Cilium(t *testing.T) {
 		{
 			Cilium: kops.CiliumNetworkingSpec{
 				Version: "v1.7.0",
-				Hubble: kops.HubbleSpec{
+				Hubble: &kops.HubbleSpec{
 					Enabled: fi.Bool(true),
 				},
 			},
@@ -639,7 +639,7 @@ func Test_Validate_Cilium(t *testing.T) {
 		{
 			Cilium: kops.CiliumNetworkingSpec{
 				Version: "v1.8.0",
-				Hubble: kops.HubbleSpec{
+				Hubble: &kops.HubbleSpec{
 					Enabled: fi.Bool(true),
 				},
 			},
diff --git a/upup/pkg/fi/cloudup/bootstrapchannelbuilder_test.go b/upup/pkg/fi/cloudup/bootstrapchannelbuilder_test.go
index eaf282bf1e..6e5877cd37 100644
--- a/upup/pkg/fi/cloudup/bootstrapchannelbuilder_test.go
+++ b/upup/pkg/fi/cloudup/bootstrapchannelbuilder_test.go
@@ -106,39 +106,21 @@ func runChannelBuilderTest(t *testing.T, key string, addonManifests []string) {
 	if err != nil {
 		t.Error(err)
 	}
-	role := "arn:aws:iam::1234567890108:instance-profile/kops-custom-node-role"
-	kopsModel := model.KopsModelContext{
-		IAMModelContext: iam.IAMModelContext{
-			Cluster:      cluster,
-			AWSAccountID: "123456789012",
-		},
-		Region: "us-east-1",
-		InstanceGroups: []*kopsapi.InstanceGroup{
-			{
-				Spec: kopsapi.InstanceGroupSpec{
-					IAM: &kopsapi.IAMProfileSpec{
-						Profile: &role,
-					},
-					Role: kopsapi.InstanceGroupRoleNode,
-				},
-			},
-			{
-				Spec: kopsapi.InstanceGroupSpec{
-					Role: kopsapi.InstanceGroupRoleNode,
-				},
-			},
-		},
-	}
 
 	tf := &TemplateFunctions{
-		KopsModelContext: kopsModel,
+		KopsModelContext: model.KopsModelContext{
+			IAMModelContext: iam.IAMModelContext{Cluster: cluster},
+			Region:          "us-east-1",
+		},
 	}
 	tf.AddTo(templates.TemplateFunctions, secretStore)
 
 	bcb := BootstrapChannelBuilder{
-		KopsModelContext: &kopsModel,
-		templates:        templates,
-		assetBuilder:     assets.NewAssetBuilder(cluster, ""),
+		KopsModelContext: &model.KopsModelContext{
+			IAMModelContext: iam.IAMModelContext{Cluster: cluster},
+		},
+		templates:    templates,
+		assetBuilder: assets.NewAssetBuilder(cluster, ""),
 	}
 
 	context := &fi.ModelBuilderContext{
diff --git a/upup/pkg/fi/cloudup/new_cluster_test.go b/upup/pkg/fi/cloudup/new_cluster_test.go
index df796956d5..379db86a5c 100644
--- a/upup/pkg/fi/cloudup/new_cluster_test.go
+++ b/upup/pkg/fi/cloudup/new_cluster_test.go
@@ -276,7 +276,6 @@ func TestSetupNetworking(t *testing.T) {
 					Networking: &api.NetworkingSpec{
 						Cilium: &api.CiliumNetworkingSpec{
 							EnableNodePort: true,
-							Hubble:         api.HubbleSpec{},
 						},
 					},
 				},
@@ -295,7 +294,6 @@ func TestSetupNetworking(t *testing.T) {
 						Cilium: &api.CiliumNetworkingSpec{
 							EnableNodePort: true,
 							EtcdManaged:    true,
-							Hubble:         api.HubbleSpec{},
 						},
 					},
 				},
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/cluster.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/cluster.yaml
index c70f2c6dfb..6d9e9dd295 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/cluster.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/cluster.yaml
@@ -25,7 +25,7 @@ spec:
       name: master-us-test-1a
     name: events
   iam: {}
-  kubernetesVersion: v1.19.2
+  kubernetesVersion: v1.14.6
   masterInternalName: api.internal.minimal.example.com
   masterPublicName: api.minimal.example.com
   additionalSans:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/manifest.yaml
index 3b4ec1fa88..c1f7a48872 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/awsiamauthenticator/manifest.yaml
@@ -7,7 +7,7 @@ spec:
   - id: k8s-1.16
     kubernetesVersion: '>=1.16.0-alpha.0'
     manifest: kops-controller.addons.k8s.io/k8s-1.16.yaml
-    manifestHash: 3c73d81e8c87bf4f409323ca904d22ee9c375284
+    manifestHash: b047de23df3b0caff3784aaa89ae0c967e866c95
     name: kops-controller.addons.k8s.io
     selector:
       k8s-addon: kops-controller.addons.k8s.io
@@ -34,6 +34,13 @@ spec:
     selector:
       k8s-addon: kube-dns.addons.k8s.io
     version: 1.15.13-kops.3
+  - id: k8s-1.8
+    manifest: rbac.addons.k8s.io/k8s-1.8.yaml
+    manifestHash: 5d53ce7b920cd1e8d65d2306d80a041420711914
+    name: rbac.addons.k8s.io
+    selector:
+      k8s-addon: rbac.addons.k8s.io
+    version: 1.8.0
   - id: k8s-1.9
     manifest: kubelet-api.rbac.addons.k8s.io/k8s-1.9.yaml
     manifestHash: e1508d77cb4e527d7a2939babe36dc350dd83745
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/dns-controller.addons.k8s.io-k8s-1.12.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/dns-controller.addons.k8s.io-k8s-1.12.yaml
index f8ddbf272f..c8c56255fd 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/dns-controller.addons.k8s.io-k8s-1.12.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/dns-controller.addons.k8s.io-k8s-1.12.yaml
@@ -35,7 +35,7 @@ spec:
         - name: KUBERNETES_SERVICE_HOST
           value: 127.0.0.1
         - name: AWS_ROLE_ARN
-          value: arn:aws:iam::123456789012:role/dns-controller.kube-system.sa.minimal.example.com
+          value: arn:aws:iam:::role/dns-controller.kube-system.sa.minimal.example.com
         - name: AWS_WEB_IDENTITY_TOKEN_FILE
           value: /var/run/secrets/amazonaws.com/token
         image: k8s.gcr.io/kops/dns-controller:1.19.0-alpha.4
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/manifest.yaml
index 90282862b7..e9cfac355c 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/public-jwks/manifest.yaml
@@ -73,7 +73,7 @@ spec:
   - id: k8s-1.12
     kubernetesVersion: '>=1.12.0'
     manifest: dns-controller.addons.k8s.io/k8s-1.12.yaml
-    manifestHash: ec9465805a830d1029d3fb7a181202a1f9266b8e
+    manifestHash: 767e92b277dad99d830ddfa0dfc37f15731f4b8e
     name: dns-controller.addons.k8s.io
     selector:
       k8s-addon: dns-controller.addons.k8s.io
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/cluster.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/cluster.yaml
index 36a0df3c89..91d03333db 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/cluster.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/cluster.yaml
@@ -21,7 +21,7 @@ spec:
       name: master-us-test-1a
     name: events
   iam: {}
-  kubernetesVersion: v1.19.2
+  kubernetesVersion: v1.14.6
   masterInternalName: api.internal.minimal.example.com
   masterPublicName: api.minimal.example.com
   additionalSans:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/kops-controller.addons.k8s.io-k8s-1.16.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/kops-controller.addons.k8s.io-k8s-1.16.yaml
index dfa28e93b0..146da3823d 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/kops-controller.addons.k8s.io-k8s-1.16.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/kops-controller.addons.k8s.io-k8s-1.16.yaml
@@ -1,7 +1,7 @@
 apiVersion: v1
 data:
   config.yaml: |
-    {"cloud":"aws","configBase":"memfs://clusters.example.com/minimal.example.com","server":{"Listen":":3992","provider":{"aws":{"nodesRoles":["kops-custom-node-role","nodes.minimal.example.com"],"Region":"us-east-1"}},"serverKeyPath":"/etc/kubernetes/kops-controller/pki/kops-controller.key","serverCertificatePath":"/etc/kubernetes/kops-controller/pki/kops-controller.crt","caBasePath":"/etc/kubernetes/kops-controller/pki","signingCAs":["ca"],"certNames":["kubelet","kube-proxy"]}}
+    {"cloud":"aws","configBase":"memfs://clusters.example.com/minimal.example.com"}
 kind: ConfigMap
 metadata:
   labels:
diff --git a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/manifest.yaml b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/manifest.yaml
index 719dd81a07..d8a48b62a9 100644
--- a/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/manifest.yaml
+++ b/upup/pkg/fi/cloudup/tests/bootstrapchannelbuilder/simple/manifest.yaml
@@ -7,7 +7,7 @@ spec:
   - id: k8s-1.16
     kubernetesVersion: '>=1.16.0-alpha.0'
     manifest: kops-controller.addons.k8s.io/k8s-1.16.yaml
-    manifestHash: 3c73d81e8c87bf4f409323ca904d22ee9c375284
+    manifestHash: b047de23df3b0caff3784aaa89ae0c967e866c95
     name: kops-controller.addons.k8s.io
     selector:
       k8s-addon: kops-controller.addons.k8s.io
@@ -34,6 +34,13 @@ spec:
     selector:
       k8s-addon: kube-dns.addons.k8s.io
     version: 1.15.13-kops.3
+  - id: k8s-1.8
+    manifest: rbac.addons.k8s.io/k8s-1.8.yaml
+    manifestHash: 5d53ce7b920cd1e8d65d2306d80a041420711914
+    name: rbac.addons.k8s.io
+    selector:
+      k8s-addon: rbac.addons.k8s.io
+    version: 1.8.0
   - id: k8s-1.9
     manifest: kubelet-api.rbac.addons.k8s.io/k8s-1.9.yaml
     manifestHash: e1508d77cb4e527d7a2939babe36dc350dd83745