Repository: kubernetes/kops (go)

Issue:
Enabling hubble-metrics should create a headless service with annotations for prometheus
/kind bug

**1. What `kops` version are you running? The command `kops version`, will display
 this information.**
1.27.0

**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**

1.27.4

**3. What cloud provider are you using?**
AWS

**4. What commands did you run?  What is the simplest way to reproduce this issue?**
Create a cluster with cilium and hubble metrics enabled:

```yaml
  networking:
    cilium:
      enablePrometheusMetrics: true
      hubble:
        enabled: true
        metrics:
        - dns
        - drop
        - tcp
        - flow
        - icmp
        - http
```

**5. What happened after the commands executed?**

cilium agents serves cilium agent metrics on port 9090  and hubble metrics on port 9091. Prometheus only scrapes port 9090 since the pods have these annotations:

```yaml
metadata:
  annotations:
    prometheus.io/port: "9090"
    prometheus.io/scrape: "true"
```
As there is no official way to add extra ports using extra annotations on the pods, the cilium helm chart will create a headless service named "hubble-metrics" that includes the annotations to allow prometheus to discover the endpoint correctly

**6. What did you expect to happen?**
I expected kops to also create the same headless service as the cilium helm chart. Adding the following service by hand caused the metrics to be correctly scraped by prometheus:

```yaml
kind: Service
apiVersion: v1
metadata:
  namespace: kube-system
  name: hubble-metrics
  annotations:
    prometheus.io/scrape: 'true'
    prometheus.io/port: '9091'
spec:
  clusterIP: "None"
  selector:
    k8s-app: cilium
  ports:
    - protocol: TCP
      port: 9091
      targetPort: 9091
      name: hubble-metrics
```
**7. Please provide your cluster manifest. Execute
  `kops get --name my.example.com -o yaml` to display your cluster manifest.
  You may want to remove your cluster name and other sensitive information.**

```yaml
apiVersion: kops.k8s.io/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: "2023-05-30T13:43:11Z"
  generation: 21
  name: XXXXXXXX
spec:
  api:
    loadBalancer:
      class: Network
      type: Public
  authorization:
    rbac: {}
  certManager:
    enabled: true
  channel: stable
  cloudProvider: aws
  clusterAutoscaler:
    enabled: true
    skipNodesWithLocalStorage: true
    skipNodesWithSystemPods: true
  configBase: s3:/XXXXXXXX/XXXXXXXX
  etcdClusters:
  - cpuRequest: 200m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: XXXXXXXX
      name: b
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: main
  - cpuRequest: 100m
    etcdMembers:
    - encryptedVolume: true
      instanceGroup: XXXXXXXX
      name: b
    manager:
      backupRetentionDays: 90
    memoryRequest: 100Mi
    name: events
  externalPolicies:
    master:
    - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
    - arn:aws:iam::XXXXXXXX:policy/ThanosS3Access
    node:
    - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
    - arn:aws:iam::XXXXXXXX:policy/ThanosS3Access
  iam:
    allowContainerRegistry: true
    legacy: false
  kubeProxy:
    enabled: false
  kubelet:
    anonymousAuth: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  - ::/0
  kubernetesVersion: 1.27.4
  masterPublicName: XXXXXXXX
  metricsServer:
    enabled: true
    insecure: false
  networkCIDR: XXXXXXXX
  networkID: XXXXXXXX
  networking:
    cilium:
      enableNodePort: true
      enablePrometheusMetrics: true
      hubble:
        enabled: true
        metrics:
        - dns
        - drop
        - tcp
        - flow
        - icmp
        - http
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  - ::/0
  subnets:
  - cidr: XXXXXXXX
    egress: nat-XXXXXXXX
    id: XXXXXXXX
    name: XXXXXXXX
    type: Private
    zone: eu-central-1b
  - cidr: XXXXXXXX
    id: XXXXXXXX
    name: XXXXXXXX
    type: Utility
    zone: eu-central-1b
  topology:
    dns:
      type: Public
    masters: private
    nodes: private

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2023-05-30T13:43:11Z"
  generation: 8
  labels:
    kops.k8s.io/cluster: XXXXXXXX
  name: XXXXXXXX
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20230711
  instanceMetadata:
    httpPutResponseHopLimit: 2
  machineType: t3a.large
  maxSize: 1
  minSize: 1
  role: Master
  subnets:
  - XXXXXXXX

---

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: "2023-05-30T13:43:11Z"
  generation: 13
  labels:
    kops.k8s.io/cluster: XXXXXXXX
  name: XXXXXXXX
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20230711
  instanceMetadata:
    httpPutResponseHopLimit: 2
  machineType: t3a.large
  maxSize: 9
  minSize: 3
  role: Node
  subnets:
  - XXXXXXXX
```

**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
  Paste the logs into this report, or in a gist and provide the gist link here.**

**9. Anything else do we need to know?**


Pull request:
Add the hubble-metrics service for cilium
Copied from https://github.com/cilium/cilium/blob/main/install/kubernetes/cilium/templates/hubble/metrics-service.yaml

Fixes #16032

Patch:
diff --git a/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.15.yaml.template b/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.15.yaml.template
index 6832d6b9d2..08dd28f961 100644
--- a/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.15.yaml.template
+++ b/upup/models/cloudup/resources/addons/networking.cilium.io/k8s-1.16-v1.15.yaml.template
@@ -886,6 +886,32 @@ subsets:
 {{ end }}
 {{ end }}
 {{ if WithDefaultBool .Hubble.Enabled false }}
+{{ if .Hubble.Metrics }}
+---
+# Source: cilium/templates/hubble/metrics-service.yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: hubble-metrics
+  namespace: kube-system
+  labels:
+    k8s-app: hubble
+    app.kubernetes.io/name: hubble
+    app.kubernetes.io/part-of: cilium
+  annotations:
+    prometheus.io/scrape: "true"
+    prometheus.io/port: "9965"
+spec:
+  clusterIP: None
+  type: ClusterIP
+  ports:
+  - name: hubble-metrics
+    port: 9965
+    protocol: TCP
+    targetPort: hubble-metrics
+  selector:
+    k8s-app: cilium
+{{ end }}
 ---
 # Source: cilium/templates/hubble-relay-service.yaml
 kind: Service


Test patch:
diff --git a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_cluster-completed.spec_content b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_cluster-completed.spec_content
index 71c684ffa6..9ba20b8856 100644
--- a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_cluster-completed.spec_content
+++ b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_cluster-completed.spec_content
@@ -208,6 +208,8 @@ spec:
       enableUnreachableRoutes: false
       hubble:
         enabled: true
+        metrics:
+        - drop
       identityAllocationMode: crd
       identityChangeGracePeriod: 5s
       ingress:
diff --git a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
index 83d048ca9a..491e420859 100644
--- a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
+++ b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-bootstrap_content
@@ -155,7 +155,7 @@ spec:
     version: 9.99.0
   - id: k8s-1.16
     manifest: networking.cilium.io/k8s-1.16-v1.15.yaml
-    manifestHash: 45ecd4b0c81a42b7fe61b223bc4fc869f89c9aae2a1863235782134e346c7151
+    manifestHash: 3fdb869ea26ce50ae6db32e1b997749f18cbb30ebf31468f2c5da2c692681a54
     name: networking.cilium.io
     needsPKI: true
     needsRollingUpdate: all
diff --git a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
index 6e6538203a..aae25ca1e1 100644
--- a/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
+++ b/tests/integration/update_cluster/privatecilium2/data/aws_s3_object_privatecilium.example.com-addons-networking.cilium.io-k8s-1.16_content
@@ -77,6 +77,8 @@ data:
   external-envoy-proxy: "false"
   hubble-disable-tls: "false"
   hubble-listen-address: :4244
+  hubble-metrics: drop
+  hubble-metrics-server: :9091
   hubble-socket-path: /var/run/cilium/hubble.sock
   hubble-tls-cert-file: /var/lib/cilium/tls/hubble/tls.crt
   hubble-tls-client-ca-files: /var/lib/cilium/tls/hubble/ca.crt
@@ -678,6 +680,35 @@ spec:
 
 ---
 
+apiVersion: v1
+kind: Service
+metadata:
+  annotations:
+    prometheus.io/port: "9965"
+    prometheus.io/scrape: "true"
+  creationTimestamp: null
+  labels:
+    addon.kops.k8s.io/name: networking.cilium.io
+    app.kubernetes.io/managed-by: kops
+    app.kubernetes.io/name: hubble
+    app.kubernetes.io/part-of: cilium
+    k8s-app: hubble
+    role.kubernetes.io/networking: "1"
+  name: hubble-metrics
+  namespace: kube-system
+spec:
+  clusterIP: None
+  ports:
+  - name: hubble-metrics
+    port: 9965
+    protocol: TCP
+    targetPort: hubble-metrics
+  selector:
+    k8s-app: cilium
+  type: ClusterIP
+
+---
+
 apiVersion: v1
 kind: Service
 metadata:
@@ -800,6 +831,10 @@ spec:
           hostPort: 4244
           name: peer-service
           protocol: TCP
+        - containerPort: 9091
+          hostPort: 9091
+          name: hubble-metrics
+          protocol: TCP
         readinessProbe:
           failureThreshold: 3
           httpGet:
diff --git a/tests/integration/update_cluster/privatecilium2/in-v1alpha2.yaml b/tests/integration/update_cluster/privatecilium2/in-v1alpha2.yaml
index a14d153953..e8328babaa 100644
--- a/tests/integration/update_cluster/privatecilium2/in-v1alpha2.yaml
+++ b/tests/integration/update_cluster/privatecilium2/in-v1alpha2.yaml
@@ -39,6 +39,8 @@ spec:
         sharedLoadBalancerServiceName: private-ingress
       hubble:
         enabled: true
+        metrics:
+        - drop
   nonMasqueradeCIDR: 100.64.0.0/10
   sshAccess:
   - 0.0.0.0/0