Repository: unikorn-cloud/helm-cluster-api
Pull request #29: Easier metrics access
Author: login: Coalmine-Canary
Created: 2025-03-25T10:35:48Z | Merged: 2025-03-26T09:03:04Z

fix: set insecure diagnostics to true to allow easier scraping of metrics on pods within vclusters.

- Adds a way of overriding the default values in values.yaml generated from the kustomize environment variables. 
- Changes ports to 8080 to reflect the fact they are HTTP (a necessary change unfortunately, changing CAPI to not use auth also changes it to use HTTP instead of HTTPS)
- Turns on insecure diagnostics (CAPI/CAPO), and removed RBAC that makes ORC require auth (as documented [here](https://github.com/k-orc/openstack-resource-controller/blob/874c2174b097d4cefc092a5deac6b14d4e50ff3a/config/rbac/kustomization.yaml#L11-L16)) 


Diff:
diff --git a/charts/cluster-api-bootstrap-kubeadm/templates/deployment-0.yaml b/charts/cluster-api-bootstrap-kubeadm/templates/deployment-0.yaml
index 1e3f627..1ae5844 100644
--- a/charts/cluster-api-bootstrap-kubeadm/templates/deployment-0.yaml
+++ b/charts/cluster-api-bootstrap-kubeadm/templates/deployment-0.yaml
@@ -55,7 +55,7 @@ spec:
         - containerPort: 9440
           name: healthz
           protocol: TCP
-        - containerPort: 8443
+        - containerPort: 8080
           name: metrics
           protocol: TCP
         readinessProbe:
diff --git a/charts/cluster-api-bootstrap-kubeadm/values.yaml b/charts/cluster-api-bootstrap-kubeadm/values.yaml
index 7a99d79..62aa5e0 100644
--- a/charts/cluster-api-bootstrap-kubeadm/values.yaml
+++ b/charts/cluster-api-bootstrap-kubeadm/values.yaml
@@ -1,5 +1,5 @@
-capi_diagnostics_address: :8443
-capi_insecure_diagnostics: false
+capi_diagnostics_address: :8080
+capi_insecure_diagnostics: true
 exp_kubeadm_bootstrap_format_ignition: false
 exp_machine_pool: true
 image: registry.k8s.io/cluster-api/kubeadm-bootstrap-controller:v1.9.4
diff --git a/charts/cluster-api-control-plane-kubeadm/values.yaml b/charts/cluster-api-control-plane-kubeadm/values.yaml
index 3f2f0a8..801f35c 100644
--- a/charts/cluster-api-control-plane-kubeadm/values.yaml
+++ b/charts/cluster-api-control-plane-kubeadm/values.yaml
@@ -1,5 +1,5 @@
-capi_diagnostics_address: :8443
-capi_insecure_diagnostics: false
+capi_diagnostics_address: :8080
+capi_insecure_diagnostics: true
 capi_use_deprecated_infra_machine_naming: false
 cluster_topology: false
 exp_kubeadm_bootstrap_format_ignition: false
diff --git a/charts/cluster-api-core/templates/deployment-0.yaml b/charts/cluster-api-core/templates/deployment-0.yaml
index cf78042..99fa548 100644
--- a/charts/cluster-api-core/templates/deployment-0.yaml
+++ b/charts/cluster-api-core/templates/deployment-0.yaml
@@ -55,7 +55,7 @@ spec:
         - containerPort: 9440
           name: healthz
           protocol: TCP
-        - containerPort: 8443
+        - containerPort: 8080
           name: metrics
           protocol: TCP
         readinessProbe:
diff --git a/charts/cluster-api-core/values.yaml b/charts/cluster-api-core/values.yaml
index d5ea76b..10968ae 100644
--- a/charts/cluster-api-core/values.yaml
+++ b/charts/cluster-api-core/values.yaml
@@ -1,5 +1,5 @@
-capi_diagnostics_address: :8443
-capi_insecure_diagnostics: false
+capi_diagnostics_address: :8080
+capi_insecure_diagnostics: true
 capi_use_deprecated_infra_machine_naming: false
 cluster_topology: false
 exp_cluster_resource_set: true
diff --git a/charts/cluster-api/values.yaml b/charts/cluster-api/values.yaml
index 77fa2d0..2d48fcf 100644
--- a/charts/cluster-api/values.yaml
+++ b/charts/cluster-api/values.yaml
@@ -7,8 +7,8 @@ openstack:
   enabled: true
 
 cluster-api-core:
-  capi_diagnostics_address: :8443
-  capi_insecure_diagnostics: false
+  capi_diagnostics_address: :8080
+  capi_insecure_diagnostics: true
   capi_use_deprecated_infra_machine_naming: false
   cluster_topology: false
   exp_cluster_resource_set: true
@@ -19,16 +19,16 @@ cluster-api-core:
   image: registry.k8s.io/cluster-api/cluster-api-controller:v1.9.4
 
 cluster-api-bootstrap-kubeadm:
-  capi_diagnostics_address: :8443
-  capi_insecure_diagnostics: false
+  capi_diagnostics_address: :8080
+  capi_insecure_diagnostics: true
   exp_kubeadm_bootstrap_format_ignition: false
   exp_machine_pool: true
   image: registry.k8s.io/cluster-api/kubeadm-bootstrap-controller:v1.9.4
   kubeadm_bootstrap_token_ttl: 15m
 
 cluster-api-control-plane-kubeadm:
-  capi_diagnostics_address: :8443
-  capi_insecure_diagnostics: false
+  capi_diagnostics_address: :8080
+  capi_insecure_diagnostics: true
   capi_use_deprecated_infra_machine_naming: false
   cluster_topology: false
   exp_kubeadm_bootstrap_format_ignition: false
diff --git a/charts/openstack-resource-controller/templates/clusterrole-3.yaml b/charts/openstack-resource-controller/templates/clusterrole-3.yaml
deleted file mode 100644
index 1fc0765..0000000
--- a/charts/openstack-resource-controller/templates/clusterrole-3.yaml
+++ /dev/null
@@ -1,17 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: orc-metrics-auth-role
-rules:
-- apiGroups:
-  - authentication.k8s.io
-  resources:
-  - tokenreviews
-  verbs:
-  - create
-- apiGroups:
-  - authorization.k8s.io
-  resources:
-  - subjectaccessreviews
-  verbs:
-  - create
diff --git a/charts/openstack-resource-controller/templates/clusterrole-4.yaml b/charts/openstack-resource-controller/templates/clusterrole-4.yaml
deleted file mode 100644
index 6f709c6..0000000
--- a/charts/openstack-resource-controller/templates/clusterrole-4.yaml
+++ /dev/null
@@ -1,9 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
-metadata:
-  name: orc-metrics-reader
-rules:
-- nonResourceURLs:
-  - /metrics
-  verbs:
-  - get
diff --git a/charts/openstack-resource-controller/templates/clusterrolebinding-1.yaml b/charts/openstack-resource-controller/templates/clusterrolebinding-1.yaml
deleted file mode 100644
index b45c6d3..0000000
--- a/charts/openstack-resource-controller/templates/clusterrolebinding-1.yaml
+++ /dev/null
@@ -1,12 +0,0 @@
-apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRoleBinding
-metadata:
-  name: orc-metrics-auth-rolebinding
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: orc-metrics-auth-role
-subjects:
-- kind: ServiceAccount
-  name: orc-controller-manager
-  namespace: orc-system
diff --git a/charts/openstack-resource-controller/templates/deployment-0.yaml b/charts/openstack-resource-controller/templates/deployment-0.yaml
index 78ae3cb..0564f23 100644
--- a/charts/openstack-resource-controller/templates/deployment-0.yaml
+++ b/charts/openstack-resource-controller/templates/deployment-0.yaml
@@ -21,7 +21,7 @@ spec:
     spec:
       containers:
       - args:
-        - --metrics-bind-address=:8443
+        - --metrics-bind-address=:8080
         - --leader-elect
         - --health-probe-bind-address=:8081
         command:
@@ -36,7 +36,7 @@ spec:
           periodSeconds: 20
         name: manager
         ports:
-        - containerPort: 8443
+        - containerPort: 8080
           name: metrics
           protocol: TCP
         readinessProbe:
diff --git a/charts/openstack-resource-controller/templates/service-0.yaml b/charts/openstack-resource-controller/templates/service-0.yaml
index e7f2cd5..24f1210 100644
--- a/charts/openstack-resource-controller/templates/service-0.yaml
+++ b/charts/openstack-resource-controller/templates/service-0.yaml
@@ -9,9 +9,9 @@ metadata:
   namespace: orc-system
 spec:
   ports:
-  - name: https
-    port: 8443
+  - name: metrics
+    port: 8080
     protocol: TCP
-    targetPort: 8443
+    targetPort: 8080
   selector:
     control-plane: controller-manager
diff --git a/generate.py b/generate.py
index 531e126..02b3804 100755
--- a/generate.py
+++ b/generate.py
@@ -10,6 +10,18 @@
 import textwrap
 import yaml
 
+# Overrides for values.yaml (for all charts)
+VALUES = {
+  # Allow access to metrics endpoints without authentication
+  "capi_diagnostics_address": ":8080",
+  "capi_insecure_diagnostics":  "true"
+}
+
+STRING_CONVERSIONS = {
+  "true": True,
+  "false": False,
+}
+
 def main():
     parser = argparse.ArgumentParser()
     parser.add_argument('--chart', required=True, help='Chart name')
@@ -77,6 +89,12 @@ def main():
     for o in objects:
         kind = o['kind']
 
+        # Skip resources
+        if kind in ['ClusterRole', 'ClusterRoleBinding']:
+          # Remove auth requirement to access /metrics endpoints
+          if o['metadata']['name'] in ['orc-metrics-reader', 'orc-metrics-auth-rolebinding', 'orc-metrics-auth-role']:
+            continue
+
         # Remove erroneous fields added by upstream.
         if 'creationTimestamp' in o['metadata']:
             del o['metadata']['creationTimestamp']
@@ -98,17 +116,16 @@ def main():
             # https://regex101.com/r/8r9GZU/1
             fields = re.match(r'\$\{([A-Z0-9_]+)(?::=(.*))?\}', m)
 
-            value = fields.group(1).lower()
-            default = fields.group(2)
+            variable = fields.group(1).lower()
+
+            value = VALUES[variable] if variable in VALUES else fields.group(2)
 
-            if default == 'true':
-                default = True
-            elif default == 'false':
-                default = False
+            if value in STRING_CONVERSIONS:
+                value = STRING_CONVERSIONS[value]
 
-            values[value] = default
+            values[variable] = value
 
-            resource = resource.replace(m, '{{ .Values.' + value + ' }}')
+            resource = resource.replace(m, '{{ .Values.' + variable + ' }}')
 
         count = counts[kind]
         counts[kind] += 1
diff --git a/patches.yaml b/patches.yaml
index e560d7c..deb7374 100644
--- a/patches.yaml
+++ b/patches.yaml
@@ -1,4 +1,4 @@
-# Add metrics port to capo-controller-manager
+# Add metrics port to deployment templates
 - target: 
     kind: Deployment
     name: capo-controller-manager
@@ -10,7 +10,26 @@
         containerPort: 8080
         name: metrics
         protocol: TCP
-# Add metrics port to openstack resource controller
+- target:
+    kind: Deployment
+    name: capi-kubeadm-bootstrap-controller-manager
+  patch: |-
+    - op: replace
+      path: '/spec/template/spec/containers/0/ports/2'
+      value:
+        containerPort: 8080
+        name: metrics
+        protocol: TCP
+- target:
+    kind: Deployment
+    name: capi-controller-manager
+  patch: |-
+    - op: replace
+      path: '/spec/template/spec/containers/0/ports/2'
+      value:
+        containerPort: 8080
+        name: metrics
+        protocol: TCP
 - target:
     kind: Deployment
     name: orc-controller-manager
@@ -18,9 +37,29 @@
     - op: add
       path: '/spec/template/spec/containers/0/ports'
       value:
-        - containerPort: 8443
+        - containerPort: 8080
           name: metrics
           protocol: TCP
+# Add metrics port to openstack-resource-controller args
+- target:
+    kind: Deployment
+    name: orc-controller-manager
+  patch: |-
+    - op: replace
+      path: /spec/template/spec/containers/0/args/0
+      value: --metrics-bind-address=:8080
+# Change port of openstack-resource-controller metrics service
+- target:
+    kind: Service
+    name: orc-controller-manager-metrics-service
+  patch: |-
+    - op: replace
+      path: '/spec/ports'
+      value:
+        - name: metrics
+          port: 8080
+          protocol: TCP
+          targetPort: 8080
 # Changes to all deployments
 - target:
     kind: Deployment


Reviews:
user: login: spjmurray
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T11:04:02Z
user: login: spjmurray
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T11:08:02Z
user: login: spjmurray
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T11:11:09Z
user: login: Coalmine-Canary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T11:20:45Z
user: login: Coalmine-Canary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T11:24:04Z
user: login: spjmurray
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T13:11:07Z
user: login: spjmurray
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T13:14:36Z
user: login: spjmurray
body: Open for debate as always, I just like minimal code 😸 
state: CHANGES_REQUESTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T15:01:47Z
user: login: Coalmine-Canary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T15:18:04Z
user: login: Coalmine-Canary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T15:21:59Z
user: login: spjmurray
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T15:50:25Z
user: login: Coalmine-Canary
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-25T16:22:48Z
user: login: spjmurray
body: It's a yes from me
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T09:01:38Z

Review comments:
diff_hunk: @@ -10,6 +10,13 @@
 import textwrap
 import yaml
 
+# Overrides for values.yaml (for all charts)
path: generate.py
user: login: spjmurray
body: Oh no!  Why u no kustomize?!  I thought we were onto a beautiful thing!
created_at: 2025-03-25T11:04:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -77,6 +84,12 @@ def main():
     for o in objects:
         kind = o['kind']
 
+        # Skip resources
path: generate.py
user: login: spjmurray
body: To be fair, leaving them in doesn't do any harm does it?  Just makes the code less of a pain to grok.
created_at: 2025-03-25T11:08:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -10,6 +10,13 @@
 import textwrap
 import yaml
 
+# Overrides for values.yaml (for all charts)
path: generate.py
user: login: spjmurray
body: Actually, it's a positional parameter isn't it 🤔 
created_at: 2025-03-25T11:11:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -10,6 +10,13 @@
 import textwrap
 import yaml
 
+# Overrides for values.yaml (for all charts)
path: generate.py
user: login: Coalmine-Canary
body: Yeah can't do this with Kustomize unfortunately :( 
created_at: 2025-03-25T11:20:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -77,6 +84,12 @@ def main():
     for o in objects:
         kind = o['kind']
 
+        # Skip resources
path: generate.py
user: login: Coalmine-Canary
body: Nah according to the docs (well, a comment...) having the rbac resources there is what enables auth in orc, I honestly haven't been able to test that works yet, it's awkward to test in nks-dev so trying to set up a local cluster and install unikorn on it to test now

Here's the comment: https://github.com/k-orc/openstack-resource-controller/blob/874c2174b097d4cefc092a5deac6b14d4e50ff3a/config/rbac/kustomization.yaml#L11-L16
created_at: 2025-03-25T11:24:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -77,6 +84,12 @@ def main():
     for o in objects:
         kind = o['kind']
 
+        # Skip resources
path: generate.py
user: login: spjmurray
body: oy vey!
created_at: 2025-03-25T13:11:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -10,6 +10,13 @@
 import textwrap
 import yaml
 
+# Overrides for values.yaml (for all charts)
path: generate.py
user: login: spjmurray
body: Yeah, was looking at strategic merge patches, thinking they were miraculous, but alas no!
created_at: 2025-03-25T13:14:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,17 +111,21 @@ def main():
             # https://regex101.com/r/8r9GZU/1
             fields = re.match(r'\$\{([A-Z0-9_]+)(?::=(.*))?\}', m)
 
-            value = fields.group(1).lower()
-            default = fields.group(2)
+            variable = fields.group(1)
+
+            if variable in VALUES.keys():
path: generate.py
user: login: spjmurray
body: ```python
variable = VALUES[variable] if variable in VALUES else fields.group(2)
```

Nice and terse...
created_at: 2025-03-25T14:55:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -10,6 +10,13 @@
 import textwrap
 import yaml
 
+# Overrides for values.yaml (for all charts)
+VALUES = {
+  # Allow access to metrics endpoints without authentication
+  "CAPI_DIAGNOSTICS_ADDRESS": ":8080",
+  "CAPI_INSECURE_DIAGNOSTICS":  "true"
path: generate.py
user: login: spjmurray
body: I would personally just have:
```python
{
    ...
    "capi_insecure_diagnostics": True,
}
```

Then see below...
created_at: 2025-03-25T14:56:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,17 +111,21 @@ def main():
             # https://regex101.com/r/8r9GZU/1
             fields = re.match(r'\$\{([A-Z0-9_]+)(?::=(.*))?\}', m)
 
-            value = fields.group(1).lower()
-            default = fields.group(2)
+            variable = fields.group(1)
path: generate.py
user: login: spjmurray
body: ... with the changes to `VALUES` we can `lower()` this straight away and save replication...
created_at: 2025-03-25T14:57:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,17 +111,21 @@ def main():
             # https://regex101.com/r/8r9GZU/1
             fields = re.match(r'\$\{([A-Z0-9_]+)(?::=(.*))?\}', m)
 
-            value = fields.group(1).lower()
-            default = fields.group(2)
+            variable = fields.group(1)
+
+            if variable in VALUES.keys():
+              value = VALUES[variable]
+            else:
+              value = fields.group(2)
 
-            if default == 'true':
-                default = True
-            elif default == 'false':
-                default = False
+            if value == 'true':
path: generate.py
user: login: spjmurray
body: and finally, I like the translation table idea, perhaps...

```python
STRING_CONVERSIONS = {
  "true": True,
  "false": False,
}
```

then

```python
if value in STRING_CONVERSIONS:
    value = STRING_CONVERSIONS[value]
```
created_at: 2025-03-25T15:00:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,17 +111,21 @@ def main():
             # https://regex101.com/r/8r9GZU/1
             fields = re.match(r'\$\{([A-Z0-9_]+)(?::=(.*))?\}', m)
 
-            value = fields.group(1).lower()
-            default = fields.group(2)
+            variable = fields.group(1)
path: generate.py
user: login: Coalmine-Canary
body: Yeah that makes sense, done :) 
created_at: 2025-03-25T15:18:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,17 +111,21 @@ def main():
             # https://regex101.com/r/8r9GZU/1
             fields = re.match(r'\$\{([A-Z0-9_]+)(?::=(.*))?\}', m)
 
-            value = fields.group(1).lower()
-            default = fields.group(2)
+            variable = fields.group(1)
+
+            if variable in VALUES.keys():
+              value = VALUES[variable]
+            else:
+              value = fields.group(2)
 
-            if default == 'true':
-                default = True
-            elif default == 'false':
-                default = False
+            if value == 'true':
path: generate.py
user: login: Coalmine-Canary
body: I think the original is nice and simple, but happy to make that change. Could also go with this... a bit odd but it works and is more concise than the original :D 
```
if value in ['true', 'false']:
    value = value == 'true'
```
created_at: 2025-03-25T15:21:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,17 +111,21 @@ def main():
             # https://regex101.com/r/8r9GZU/1
             fields = re.match(r'\$\{([A-Z0-9_]+)(?::=(.*))?\}', m)
 
-            value = fields.group(1).lower()
-            default = fields.group(2)
+            variable = fields.group(1)
+
+            if variable in VALUES.keys():
+              value = VALUES[variable]
+            else:
+              value = fields.group(2)
 
-            if default == 'true':
-                default = True
-            elif default == 'false':
-                default = False
+            if value == 'true':
path: generate.py
user: login: spjmurray
body: Does kinda limit it to two values!  I like your perversion though!
created_at: 2025-03-25T15:50:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -98,17 +111,21 @@ def main():
             # https://regex101.com/r/8r9GZU/1
             fields = re.match(r'\$\{([A-Z0-9_]+)(?::=(.*))?\}', m)
 
-            value = fields.group(1).lower()
-            default = fields.group(2)
+            variable = fields.group(1)
+
+            if variable in VALUES.keys():
+              value = VALUES[variable]
+            else:
+              value = fields.group(2)
 
-            if default == 'true':
-                default = True
-            elif default == 'false':
-                default = False
+            if value == 'true':
path: generate.py
user: login: Coalmine-Canary
body: Changed to your suggestion :) 
created_at: 2025-03-25T16:22:48Z
author_association: CONTRIBUTOR

PR comments:
user: login: Coalmine-Canary
created_at: 2025-03-25T16:32:44Z
body: Just realised there's still a few deployments with metrics on 8443 instead of 8080, fixing 
author_association: CONTRIBUTOR
user: login: Coalmine-Canary
created_at: 2025-03-25T16:51:48Z
body: > Just realised there's still a few deployments with metrics on 8443 instead of 8080, fixing

Fixed, unfortunately had to add quite a few patches - there's no way to group them together (the targeting conditions are strictly AND'ed, there's no way to OR them) 
author_association: CONTRIBUTOR