Repository: kubernetes/kops (go)

Issue:
ClusterID missing in Cilium
/kind bug

**1. What `kops` version are you running? The command `kops version`, will display
this information.**

`Client version: 1.26.4 (git-v1.26.4)`

**2. What Kubernetes version are you running? `kubectl version` will print the
version if a cluster is running or provide the Kubernetes version specified as
a `kops` flag.**

```
{
  "clientVersion": {
    "major": "1",
    "minor": "27",
    "gitVersion": "v1.27.3",
    "gitCommit": "25b4e43193bcda6c7328a6d147b1fb73a33f1598",
    "gitTreeState": "archive",
    "buildDate": "2023-06-15T16:18:27Z",
    "goVersion": "go1.20.5",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "kustomizeVersion": "v5.0.1",
  "serverVersion": {
    "major": "1",
    "minor": "27",
    "gitVersion": "v1.27.3",
    "gitCommit": "25b4e43193bcda6c7328a6d147b1fb73a33f1598",
    "gitTreeState": "clean",
    "buildDate": "2023-06-14T09:47:40Z",
    "goVersion": "go1.20.5",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}
```

**3. What cloud provider are you using?**

`openstack`

**4. What commands did you run? What is the simplest way to reproduce this issue?**

After creating a cluster with cilium CNI and adding the `spec.networking.cilium.clusterName`, try to activate the cluster mesh:

```bash
cilium clustermesh enable --service-type LoadBalancer
```

**5. What happened after the commands executed?**

```bash
$ cilium clustermesh enable --service-type LoadBalancer

⚠️  Cluster not configured for clustermesh, use '--cluster-id' and '--cluster-name' with 'cilium install'. External workloads may still be configured.
```

**6. What did you expect to happen?**

I expected to have the `spec.networking.cilium.clusterID` setting in kops.

```yaml
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  name: mycluster
spec:
  networking:
    cilium:
      clusterName: mycluster
      clusterID: 1 # expected to have this
 ```

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
  labels:
    addon.kops.k8s.io/name: networking.cilium.io
    app.kubernetes.io/managed-by: kops
    role.kubernetes.io/networking: '1'
data:
  cluster-id: '1'  # expected this
  cluster-name: mycluster
```

**7. Please provide your cluster manifest.**

Execute
`kops get --name my.example.com -o yaml` to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.\*\*

Here is the relevant part

```yaml
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  name: mycluster
spec:
  networking:
    cilium:
      clusterName: mycluster
      clusterID: 1 # expected to have this
 ```

**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
Paste the logs into this report, or in a gist and provide the gist link here.**

**9. Anything else do we need to know?**

I fixed this by adding the `cluster-id` field to the config and restarting all cilium related workloads.

```bash
kubectl patch configmap cilium-config -n kube-system -p '{"data": {"cluster-id": "1"}}'
```


Pull request:
feature: Add cluster-id for Cilium
Fixes #15633

Question regarding testing: How can I test a case where the clusterID is set to `253` ? I tried adding it [here](https://github.com/kubernetes/kops/blob/master/tests/integration/update_cluster/minimal-ipv6-cilium/in-v1alpha2.yaml#L37) but it did not work.

Patch:
diff --git a/k8s/crds/kops.k8s.io_clusters.yaml b/k8s/crds/kops.k8s.io_clusters.yaml
index 10eea53c29..816f7cd2db 100644
--- a/k8s/crds/kops.k8s.io_clusters.yaml
+++ b/k8s/crds/kops.k8s.io_clusters.yaml
@@ -4830,6 +4830,11 @@ spec:
                           to provide L3/L4 network visibility, policy enforcement
                           and other advanced features. Default: none'
                         type: string
+                      clusterID:
+                        description: ClusterID is the ID of the cluster. It is only
+                          relevant when building a mesh of clusters. Must be a number
+                          between 1 and 255.
+                        type: integer
                       clusterName:
                         description: ClusterName is the name of the cluster. It is
                           only relevant when building a mesh of clusters.
diff --git a/pkg/apis/kops/networking.go b/pkg/apis/kops/networking.go
index 8abb5e6b4a..c2a7e31100 100644
--- a/pkg/apis/kops/networking.go
+++ b/pkg/apis/kops/networking.go
@@ -438,6 +438,9 @@ type CiliumNetworkingSpec struct {
 	SidecarIstioProxyImage string `json:"sidecarIstioProxyImage,omitempty"`
 	// ClusterName is the name of the cluster. It is only relevant when building a mesh of clusters.
 	ClusterName string `json:"clusterName,omitempty"`
+	// ClusterID is the ID of the cluster. It is only relevant when building a mesh of clusters.
+	// Must be a number between 1 and 255.
+	ClusterID uint8 `json:"clusterID,omitempty"`
 	// ToFQDNsDNSRejectResponseCode sets the DNS response code for rejecting DNS requests.
 	// Possible values are "nameError" or "refused".
 	// Default: refused
diff --git a/pkg/apis/kops/v1alpha2/networking.go b/pkg/apis/kops/v1alpha2/networking.go
index 1b7d7a9016..3ff1c4fe67 100644
--- a/pkg/apis/kops/v1alpha2/networking.go
+++ b/pkg/apis/kops/v1alpha2/networking.go
@@ -541,6 +541,9 @@ type CiliumNetworkingSpec struct {
 	SidecarIstioProxyImage string `json:"sidecarIstioProxyImage,omitempty"`
 	// ClusterName is the name of the cluster. It is only relevant when building a mesh of clusters.
 	ClusterName string `json:"clusterName,omitempty"`
+	// ClusterID is the ID of the cluster. It is only relevant when building a mesh of clusters.
+	// Must be a number between 1 and 255.
+	ClusterID uint8 `json:"clusterID,omitempty"`
 	// ToFQDNsDNSRejectResponseCode sets the DNS response code for rejecting DNS requests.
 	// Possible values are "nameError" or "refused".
 	// Default: refused
diff --git a/pkg/apis/kops/v1alpha2/zz_generated.conversion.go b/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
index ff7dcd32c4..a79b1203bd 100644
--- a/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
+++ b/pkg/apis/kops/v1alpha2/zz_generated.conversion.go
@@ -1988,6 +1988,7 @@ func autoConvert_v1alpha2_CiliumNetworkingSpec_To_kops_CiliumNetworkingSpec(in *
 	out.PreallocateBPFMaps = in.PreallocateBPFMaps
 	out.SidecarIstioProxyImage = in.SidecarIstioProxyImage
 	out.ClusterName = in.ClusterName
+	out.ClusterID = in.ClusterID
 	out.ToFQDNsDNSRejectResponseCode = in.ToFQDNsDNSRejectResponseCode
 	out.ToFQDNsEnablePoller = in.ToFQDNsEnablePoller
 	// INFO: in.ContainerRuntimeLabels opted out of conversion generation
@@ -2053,6 +2054,7 @@ func autoConvert_kops_CiliumNetworkingSpec_To_v1alpha2_CiliumNetworkingSpec(in *
 	out.PreallocateBPFMaps = in.PreallocateBPFMaps
 	out.SidecarIstioProxyImage = in.SidecarIstioProxyImage
 	out.ClusterName = in.ClusterName
+	out.ClusterID = in.ClusterID
 	out.ToFQDNsDNSRejectResponseCode = in.ToFQDNsDNSRejectResponseCode
 	out.ToFQDNsEnablePoller = in.ToFQDNsEnablePoller
 	out.IPAM = in.IPAM
diff --git a/pkg/apis/kops/v1alpha3/networking.go b/pkg/apis/kops/v1alpha3/networking.go
index 2d4b5d5e05..519a6afa86 100644
--- a/pkg/apis/kops/v1alpha3/networking.go
+++ b/pkg/apis/kops/v1alpha3/networking.go
@@ -401,6 +401,9 @@ type CiliumNetworkingSpec struct {
 	SidecarIstioProxyImage string `json:"sidecarIstioProxyImage,omitempty"`
 	// ClusterName is the name of the cluster. It is only relevant when building a mesh of clusters.
 	ClusterName string `json:"clusterName,omitempty"`
+	// ClusterID is the ID of the cluster. It is only relevant when building a mesh of clusters.
+	// Must be a number between 1 and 255.
+	ClusterID uint8 `json:"clusterID,omitempty"`
 	// ToFQDNsDNSRejectResponseCode sets the DNS response code for rejecting DNS requests.
 	// Possible values are "nameError" or "refused".
 	// Default: refused
diff --git a/pkg/apis/kops/v1alpha3/zz_generated.conversion.go b/pkg/apis/kops/v1alpha3/zz_generated.conversion.go
index e2f0edc638..c3cf2aa49b 100644
--- a/pkg/apis/kops/v1alpha3/zz_generated.conversion.go
+++ b/pkg/apis/kops/v1alpha3/zz_generated.conversion.go
@@ -2124,6 +2124,7 @@ func autoConvert_v1alpha3_CiliumNetworkingSpec_To_kops_CiliumNetworkingSpec(in *
 	out.PreallocateBPFMaps = in.PreallocateBPFMaps
 	out.SidecarIstioProxyImage = in.SidecarIstioProxyImage
 	out.ClusterName = in.ClusterName
+	out.ClusterID = in.ClusterID
 	out.ToFQDNsDNSRejectResponseCode = in.ToFQDNsDNSRejectResponseCode
 	out.ToFQDNsEnablePoller = in.ToFQDNsEnablePoller
 	out.IPAM = in.IPAM
@@ -2188,6 +2189,7 @@ func autoConvert_kops_CiliumNetworkingSpec_To_v1alpha3_CiliumNetworkingSpec(in *
 	out.PreallocateBPFMaps = in.PreallocateBPFMaps
 	out.SidecarIstioProxyImage = in.SidecarIstioProxyImage
 	out.ClusterName = in.ClusterName
+	out.ClusterID = in.ClusterID
 	out.ToFQDNsDNSRejectResponseCode = in.ToFQDNsDNSRejectResponseCode
 	out.ToFQDNsEnablePoller = in.ToFQDNsEnablePoller
 	out.IPAM = in.IPAM
diff --git a/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.13.yaml.template b/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.13.yaml.template
index 049d2490e7..93e966be68 100644
--- a/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.13.yaml.template
+++ b/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.13.yaml.template
@@ -193,6 +193,11 @@ data:
 
   # Name of the cluster. Only relevant when building a mesh of clusters.
   cluster-name: "{{ .ClusterName }}"
+  # ID of the cluster. Only relevant when building a mesh of clusters.
+  # Must be a number between 1 and 255.
+  {{ if .ClusterID }}
+  cluster-id: "{{ .ClusterID }}"
+  {{ end }}
 
   # DNS response code for rejecting DNS requests,
   # available options are "nameError" and "refused"


Test patch:
diff --git a/pkg/apis/kops/validation/validation_test.go b/pkg/apis/kops/validation/validation_test.go
index c9f2e232a2..cf1dc28142 100644
--- a/pkg/apis/kops/validation/validation_test.go
+++ b/pkg/apis/kops/validation/validation_test.go
@@ -895,6 +895,11 @@ func Test_Validate_Cilium(t *testing.T) {
 				},
 			},
 		},
+		{
+			Cilium: kops.CiliumNetworkingSpec{
+				ClusterID: 253,
+			},
+		},
 		{
 			Cilium: kops.CiliumNetworkingSpec{
 				Masquerade: fi.PtrTo(true),
diff --git a/tests/e2e/kubetest2-kops/deployer/up.go b/tests/e2e/kubetest2-kops/deployer/up.go
index 2accf227ae..c19534984e 100644
--- a/tests/e2e/kubetest2-kops/deployer/up.go
+++ b/tests/e2e/kubetest2-kops/deployer/up.go
@@ -134,21 +134,7 @@ func (d *deployer) createCluster(zones []string, adminAccess string, yes bool) e
 		args = append(args, createArgs...)
 	}
 	args = appendIfUnset(args, "--admin-access", adminAccess)
-
-	// Dont set --master-count if either --control-plane-count or --master-count
-	// has been provided in --create-args
-	foundCPCount := false
-	for _, existingArg := range args {
-		existingKey := strings.Split(existingArg, "=")
-		if existingKey[0] == "--control-plane-count" || existingKey[0] == "--master-count" {
-			foundCPCount = true
-			break
-		}
-	}
-	if !foundCPCount {
-		args = appendIfUnset(args, "--master-count", fmt.Sprintf("%d", d.ControlPlaneCount))
-	}
-
+	args = appendIfUnset(args, "--master-count", fmt.Sprintf("%d", d.ControlPlaneCount))
 	args = appendIfUnset(args, "--master-volume-size", "48")
 	args = appendIfUnset(args, "--node-count", "4")
 	args = appendIfUnset(args, "--node-volume-size", "48")
diff --git a/tests/e2e/scenarios/aws-lb-controller/run-test.sh b/tests/e2e/scenarios/aws-lb-controller/run-test.sh
index 8e9022df1b..d40ef0dfff 100755
--- a/tests/e2e/scenarios/aws-lb-controller/run-test.sh
+++ b/tests/e2e/scenarios/aws-lb-controller/run-test.sh
@@ -26,7 +26,6 @@ NETWORKING="amazonvpc"
 OVERRIDES="${OVERRIDES-} --set=cluster.spec.cloudProvider.aws.loadBalancerController.enabled=true"
 OVERRIDES="${OVERRIDES} --set=cluster.spec.certManager.enabled=true"
 OVERRIDES="${OVERRIDES} --master-size=t4g.medium --node-size=t4g.medium"
-OVERRIDES="${OVERRIDES} --image=${INSTANCE_IMAGE:-ssm:/aws/service/canonical/ubuntu/server/20.04/stable/current/amd64/hvm/ebs-gp2/ami-id}"
 
 # shellcheck disable=SC2034
 ZONES="eu-west-1a,eu-west-1b,eu-west-1c"
diff --git a/tests/e2e/scenarios/podidentitywebhook/cluster.yaml.tmpl b/tests/e2e/scenarios/podidentitywebhook/cluster.yaml.tmpl
index bac59c1fa0..1ed334e524 100644
--- a/tests/e2e/scenarios/podidentitywebhook/cluster.yaml.tmpl
+++ b/tests/e2e/scenarios/podidentitywebhook/cluster.yaml.tmpl
@@ -71,7 +71,7 @@ metadata:
     kops.k8s.io/cluster: {{.clusterName}}
 spec:
   associatePublicIp: true
-  image: ssm:/aws/service/canonical/ubuntu/server/20.04/stable/current/amd64/hvm/ebs-gp2/ami-id
+  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-20221018
   machineType: t4g.medium
   maxSize: 4
   minSize: 4
@@ -89,7 +89,7 @@ metadata:
     kops.k8s.io/cluster: {{.clusterName}}
 spec:
   associatePublicIp: true
-  image: ssm:/aws/service/canonical/ubuntu/server/20.04/stable/current/amd64/hvm/ebs-gp2/ami-id
+  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-20221018
   machineType: c6g.large
   maxSize: 1
   minSize: 1
diff --git a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_cluster-completed.spec_content
index 695950d3d8..81daf221c9 100644
--- a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_cluster-completed.spec_content
@@ -206,6 +206,7 @@ spec:
       bpfNATGlobalMax: 524288
       bpfNeighGlobalMax: 524288
       bpfPolicyMapMax: 16384
+      clusterID: 253
       clusterName: default
       cpuRequest: 25m
       disableCNPStatusUpdates: true
diff --git a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
index d8014d3bb6..61eda3d4aa 100644
--- a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
@@ -162,7 +162,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.13.yaml
-    manifestHash: 94282761f698163d4f9aa873a28fc28992db00d8f335c9c9a27d2a91cb8802fd
+    manifestHash: d2ad53421a11b1a5fca6315c724813682a49d4c2e66b0213d1e6faf6efb1cb50
     name: networking.cilium.io
     needsPKI: true
     needsRollingUpdate: all
diff --git a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
index 4465aeeb68..823ad8dd7a 100644
--- a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
+++ b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
@@ -51,6 +51,7 @@ data:
   bpf-neigh-global-max: "524288"
   bpf-policy-map-max: "16384"
   cgroup-root: /run/cilium/cgroupv2
+  cluster-id: "253"
   cluster-name: default
   debug: "false"
   disable-cnp-status-updates: "true"
diff --git a/tests/integration/update_cluster/privatecilium2/in-v1alpha2.yaml b/tests/integration/update_cluster/privatecilium2/in-v1alpha2.yaml
index 71280f14cf..9cf91f3a0c 100644
--- a/tests/integration/update_cluster/privatecilium2/in-v1alpha2.yaml
+++ b/tests/integration/update_cluster/privatecilium2/in-v1alpha2.yaml
@@ -28,6 +28,7 @@ spec:
   networkCIDR: 172.20.0.0/16
   networking:
     cilium:
+      clusterID: 253
       hubble:
         enabled: true
   nonMasqueradeCIDR: 100.64.0.0/10