Repository: kubernetes/kops (go)

Issue:
The hubble-metrics port of the cilium DaemonSet is not in sync with the hubble-metrics service
/kind bug

When setting up hubble-metrics via cluster.yaml:

```yaml
[...]
networking:
    cilium:
      [...]
      hubble:
        enabled: true
        metrics:
        - dns:query;ignoreAAAA
        - drop
        - tcp
        - flow
        - icmp
        - http
[...]
``` 
A `hubble-metrics` service is created and the cilium pods expose the `hubble-metrics` port. Unfortunately, the port of the service and the port of the pod are not the same. The service uses the [port](https://github.com/kubernetes/kops/blob/master/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.15.yaml.template#L937) `9965` (which is the port cilium uses). The pod uses the [port](https://github.com/kubernetes/kops/blob/master/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.15.yaml.template#L1135) `9091` (which cilium used a long time ago, now they use port `9965`). Since the service is a [headless services](https://kubernetes.io/docs/concepts/services-networking/service/#headless-services) it must use the same port as the port exposed by the pod. For more explanation see this: https://github.com/kubernetes/kubernetes/issues/20488#issuecomment-178682638

**1. What `kops` version are you running? The command `kops version`, will display
 this information.**

```
kops version
Client version: 1.30.3 (git-v1.30.3
```

**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**

```
kubectl version
Client Version: v1.30.8
```

**3. What cloud provider are you using?**

`AWS`

**4. What commands did you run?  What is the simplest way to reproduce this issue?**



**5. What happened after the commands executed?**

**6. What did you expect to happen?**

**7. Please provide your cluster manifest. Execute
  `kops get --name my.example.com -o yaml` to display your cluster manifest.
  You may want to remove your cluster name and other sensitive information.**

```yaml
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: null
  generation: 1
  name: [REDACTED]
spec:
  api:
    loadBalancer:
      class: Network
      type: Internal
  authorization:
    rbac: {}
  certManager:
    defaultIssuer: letsencrypt-live
    enabled: true
    hostedZoneIDs:
      - [REDACTED]
    managed: true
  channel: stable
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
  cloudProvider: aws
  clusterAutoscaler:
    balanceSimilarNodeGroups: true
    cpuRequest: 100m
    enabled: true
    memoryRequest: 800Mi
    scaleDownUtilizationThreshold: "0.8"
    skipNodesWithLocalStorage: false
  configBase: s3://[REDACTED]
  containerRuntime: containerd
  etcdClusters:
    - cpuRequest: 200m
      etcdMembers:
        - encryptedVolume: true
          instanceGroup: master-eu-west-1a
          name: a
        - encryptedVolume: true
          instanceGroup: master-eu-west-1b
          name: b
        - encryptedVolume: true
          instanceGroup: master-eu-west-1c
          name: c
      memoryRequest: 100Mi
      name: main
    - cpuRequest: 100m
      etcdMembers:
        - encryptedVolume: true
          instanceGroup: master-eu-west-1a
          name: a
        - encryptedVolume: true
          instanceGroup: master-eu-west-1b
          name: b
        - encryptedVolume: true
          instanceGroup: master-eu-west-1c
          name: c
      memoryRequest: 100Mi
      name: events
  externalPolicies:
    node:
      - arn:aws:iam::[REDACTED]
  iam:
    allowContainerRegistry: true
    legacy: false
    serviceAccountExternalPermissions:
      - aws:
          policyARNs:
            - arn:aws:iam::[REDACTED]
        name: [REDACTED]
        namespace: default
    useServiceAccountExternalPermissions: true
  kubeDNS:
    nodeLocalDNS:
      enabled: false
    provider: CoreDNS
  kubeProxy:
    metricsBindAddress: 0.0.0.0
  kubelet:
    anonymousAuth: false
    authenticationTokenWebhook: true
    authorizationMode: Webhook
    cpuCFSQuota: false
  kubernetesApiAccess:
    - [REDACTED]
  kubernetesVersion: 1.30.8
  masterPublicName: api.[REDACTED]
  networkCIDR: [REDACTED]
  networkID: [REDACTED]
  networking:
    cilium:
      hubble:
        enabled: true
        metrics:
        - dns:query;ignoreAAAA
        - drop
        - tcp
        - flow
        - icmp
        - http
  nodeTerminationHandler:
    enableSQSTerminationDraining: true
    enabled: true
    managedASGTag: kubernetes.io/cluster/[REDACTED]
  nonMasqueradeCIDR: 100.64.0.0/10
  podIdentityWebhook:
    enabled: true
  serviceAccountIssuerDiscovery:
    discoveryStore: s3://[REDACTED]
    enableAWSOIDCProvider: true
  sshAccess:
    - [REDACTED]
  subnets:
    - cidr: [REDACTED]
      egress: [REDACTED]
      id: [REDACTED]
      name: eu-west-1a
      type: Private
      zone: eu-west-1a
    - cidr: [REDACTED]
      egress: [REDACTED]
      id: [REDACTED]
      name: eu-west-1b
      type: Private
      zone: eu-west-1b
    - cidr: [REDACTED]
      egress: [REDACTED]
      id: [REDACTED]
      name: eu-west-1c
      type: Private
      zone: eu-west-1c
    - cidr: [REDACTED]
      id: [REDACTED]
      name: utility-eu-west-1a
      type: Utility
      zone: eu-west-1a
    - cidr: [REDACTED]
      id: [REDACTED]
      name: utility-eu-west-1b
      type: Utility
      zone: [REDACTED]
    - cidr: [REDACTED]
      id: [REDACTED]
      name: utility-eu-west-1c
      type: Utility
      zone: eu-west-1c
  topology:
    dns:
      type: Public
  updatePolicy: external

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:41Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: master-eu-west-1a
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.2xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-eu-west-1a
    type: control-plane
  role: Master
  subnets:
    - eu-west-1a

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:41Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: master-eu-west-1b
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.2xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-eu-west-1b
    type: control-plane
  role: Master
  subnets:
    - eu-west-1b

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:41Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: master-eu-west-1c
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.2xlarge
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-eu-west-1c
    type: control-plane
  role: Master
  subnets:
    - eu-west-1c

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:41Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: nodes-eu-west-1a
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/staging: ""
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.xlarge
  maxSize: 18
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes-eu-west-1a
    type: node
  role: Node
  subnets:
    - eu-west-1a

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:42Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: nodes-eu-west-1b
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/staging: ""
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.xlarge
  maxSize: 18
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes-eu-west-1b
    type: node
  role: Node
  subnets:
    - eu-west-1b

---
apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2024-11-27T12:26:42Z"
  labels:
    kops.k8s.io/cluster: [REDACTED]
  name: nodes-eu-west-1c
spec:
  additionalSecurityGroups:
    - [REDACTED]
  cloudLabels:
    Environment: [REDACTED]
    Owner: [REDACTED]
    k8s.io/cluster-autoscaler/enabled: ""
    k8s.io/cluster-autoscaler/staging: ""
  image: 075585003325/Flatcar-stable-4081.2.0-arm64-hvm
  machineType: m6g.xlarge
  maxSize: 18
  minSize: 2
  nodeLabels:
    kops.k8s.io/instancegroup: nodes-eu-west-1c
    type: node
  role: Node
  subnets:
    - eu-west-1c
```

**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
  Paste the logs into this report, or in a gist and provide the gist link here.**

n/a

**9. Anything else do we need to know?**

I'm happy to provide a PR to fix it. Just let me know.

Pull request:
Use the same port for hubble-metrics that is used by cilium
Cilium uses port `9965` for hubble-metrics. See https://docs.cilium.io/en/stable/observability/metrics/#hubble-metrics

Fixes #17174

Patch:
diff --git a/docs/contributing/ports.md b/docs/contributing/ports.md
index 23a0cadc49..dc3b1a415b 100644
--- a/docs/contributing/ports.md
+++ b/docs/contributing/ports.md
@@ -5,9 +5,8 @@ so we can avoid port collisions.
 
 See also pkg/wellknownports/wellknownports.go
 
-
 | Port | Description                              |
-|------|------------------------------------------|
+| ---- | ---------------------------------------- |
 | 22   | SSH                                      |
 | 443  | Kubernetes API                           |
 | 179  | Calico                                   |
@@ -31,4 +30,4 @@ See also pkg/wellknownports/wellknownports.go
 | 4789 | VXLAN                                    |
 | 6942 | Cilium operator prometheus port          |
 | 9090 | Cilium prometheus port                   |
-| 9091 | Cilium hubble prometheus port            |
+| 9965 | Cilium hubble prometheus port            |
diff --git a/pkg/wellknownports/wellknownports.go b/pkg/wellknownports/wellknownports.go
index 5d43a0038e..2228021fe9 100644
--- a/pkg/wellknownports/wellknownports.go
+++ b/pkg/wellknownports/wellknownports.go
@@ -71,7 +71,7 @@ const (
 	CiliumPrometheusPort = 9090
 
 	// CiliumHubblePrometheusPort is the default port where Hubble exposes metrics
-	CiliumHubblePrometheusPort = 9091
+	CiliumHubblePrometheusPort = 9965
 
 	// VxlanUDP is the port used by VXLAN tunneling over UDP
 	VxlanUDP = 8472
diff --git a/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.15.yaml.template b/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.15.yaml.template
index 0fff996820..cc8dfac6d5 100644
--- a/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.15.yaml.template
+++ b/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.15.yaml.template
@@ -334,7 +334,7 @@ data:
   hubble-tls-key-file: /var/lib/cilium/tls/hubble/server.key
   hubble-tls-client-ca-files: /var/lib/cilium/tls/hubble/client-ca.crt
   {{ if .Hubble.Metrics }}
-  hubble-metrics-server: ":9091"
+  hubble-metrics-server: ":9965"
   hubble-metrics:
   {{- range .Hubble.Metrics }}
     {{ . }}
@@ -1132,8 +1132,8 @@ spec:
           hostPort: 4244
           protocol: TCP
         {{- if .Hubble.Metrics }}
-        - containerPort: 9091
-          hostPort: 9091
+        - containerPort: 9965
+          hostPort: 9965
           name: hubble-metrics
           protocol: TCP
         {{- end }}


Test patch:
diff --git a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
index 394a7a9a30..d3784a50cb 100644
--- a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
@@ -155,7 +155,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.15.yaml
-    manifestHash: a634b1417a8cda30b27d6167bad863baccffcd5944662aa73f2afee23c3f6d0d
+    manifestHash: d517037904591938b5a8f391024ed9c561e54f7adc5bf4bfaf243d2a10a4967a
     name: networking.cilium.io
     needsPKI: true
     needsRollingUpdate: all
diff --git a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
index e091f89583..1214364275 100644
--- a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
+++ b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
@@ -79,7 +79,7 @@ data:
   hubble-disable-tls: "false"
   hubble-listen-address: :4244
   hubble-metrics: drop
-  hubble-metrics-server: :9091
+  hubble-metrics-server: :9965
   hubble-socket-path: /var/run/cilium/hubble.sock
   hubble-tls-cert-file: /var/lib/cilium/tls/hubble/server.crt
   hubble-tls-client-ca-files: /var/lib/cilium/tls/hubble/client-ca.crt
@@ -863,8 +863,8 @@ spec:
           hostPort: 4244
           name: peer-service
           protocol: TCP
-        - containerPort: 9091
-          hostPort: 9091
+        - containerPort: 9965
+          hostPort: 9965
           name: hubble-metrics
           protocol: TCP
         readinessProbe: