Repository: kubernetes-sigs/jobset
Pull request #863: Enable TLS metrics
Author: login: atiratree
Labels: name: kind/feature
description: Categorizes issue or PR as related to a new feature.
name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
Created: 2025-04-03T08:54:12Z | Merged: 2025-04-24T05:46:31Z

<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

/kind feature

#### What this PR does / why we need it:
Add an option `PROMETHEUS WITH CERTMANAGER` to use a certificate generated by the cert manager in metrics serving instead of a localhost self-signed certificate. Optionally another certifiicate can be configured in `metrics.certDir` when `PROMETHEUS` is used. ServiceMonitor can then be secured with this certificate CA and  `insecureSkipVerify: false` can now be used when collecting metrics.


#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #793 

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".
-->
```release-note
Add an option `PROMETHEUS WITH CERTMANAGER` to use a certificate generated by the cert manager in metrics serving instead of a localhost self-signed certificate. Optionally another certifiicate can be configured in `metrics.certDir` when `PROMETHEUS` is used. ServiceMonitor can then be secured with this certificate CA and  `insecureSkipVerify: false` can now be used when collecting metrics. Prometheus support in helm can now also be enabled.
```

Linked issue #793: Secure metrics with Cert manager
<!-- Please only use this template for submitting enhancement requests -->

**What would you like to be added**:

Right now, our service monitor template assumes no tls options.

We want to use cert manager and we would have to use tls no verify.

Kubebuilder has a series of examples on how to use cert manager to secure your metrics endpoint and require TLS. As of now, we at least have https on our metrics endpoint but it is insecure.

I think one needs to expose a few metric options in our config.

**Why is this needed**:

Secure metrics and allows one to use certificates to manage metrics.


Linked issue comments: user: login: kannon92
created_at: 2025-02-27T14:38:54Z
body: I drafted a KEP for Kueue to show how one can do this.

https://github.com/kubernetes-sigs/kueue/pull/4404
author_association: CONTRIBUTOR
user: login: kannon92
created_at: 2025-02-27T14:39:20Z
body: /kind feature

author_association: CONTRIBUTOR
user: login: kannon92
created_at: 2025-04-04T18:49:51Z
body: https://github.com/kubernetes-sigs/jobset/pull/863 resolves this.
author_association: CONTRIBUTOR
user: login: kannon92
created_at: 2025-07-31T17:44:12Z
body: /milestone v0.9
author_association: CONTRIBUTOR

Diff:
diff --git a/api/config/v1alpha1/configuration_types.go b/api/config/v1alpha1/configuration_types.go
index 23d51a7dd..ff0fb7576 100644
--- a/api/config/v1alpha1/configuration_types.go
+++ b/api/config/v1alpha1/configuration_types.go
@@ -83,6 +83,12 @@ type ControllerMetrics struct {
 	// It can be set to "0" to disable the metrics serving.
 	// +optional
 	BindAddress string `json:"bindAddress,omitempty"`
+
+	// CertDir is the directory that contains the server key and certificate.
+	// If not set, a self-signed certificate will be used. The server key and
+	// certificate must be named tls.key and tls.crt, respectively.
+	// +optional
+	CertDir string `json:"certDir,omitempty"`
 }
 
 // ControllerHealth defines the health configs.
diff --git a/charts/jobset/README.md b/charts/jobset/README.md
index 72e26439d..b4ad30570 100644
--- a/charts/jobset/README.md
+++ b/charts/jobset/README.md
@@ -91,6 +91,8 @@ See [helm uninstall](https://helm.sh/docs/helm/helm_uninstall) for command docum
 | certManager.enable | bool | `false` | Whether to use cert-manager to generate certificates for the jobset webhook. |
 | certManager.issuerRef | object | `{}` | The reference to the issuer. If empty, self-signed issuer will be created and used. |
 | prometheus.enable | bool | `false` | Whether to enable Prometheus metrics exporting. |
+| prometheus.prometheusNamespace | string | `"monitoring"` |  |
+| prometheus.prometheusClientCertSecretName | string | `"jobset-metrics-server-cert"` |  |
 
 ## Maintainers
 
diff --git a/charts/jobset/templates/NOTES.txt b/charts/jobset/templates/NOTES.txt
index e156b4a4e..ef5a540b6 100644
--- a/charts/jobset/templates/NOTES.txt
+++ b/charts/jobset/templates/NOTES.txt
@@ -16,7 +16,5 @@ limitations under the License.
 
 Thank you for installing {{ .Chart.Name }}. The JobSet controller is now deployed in your cluster. To verify that the controller is running, you can run:
   kubectl get deployment -n {{ .Release.Namespace }} {{ include "jobset.controller.deployment.name" . }}
-{{- if .Values.prometheus.enable }}
 To access the metrics endpoint:
-  kubectl port-forward svc/{{ include "jobset.metrics.service.name" . }} 443
-{{- end }}
+  kubectl port-forward svc/{{ include "jobset.metrics.service.name" . }} 8443
diff --git a/charts/jobset/templates/certmanager/_helpers.tpl b/charts/jobset/templates/certmanager/_helpers.tpl
index b3e48c166..982e56ab1 100644
--- a/charts/jobset/templates/certmanager/_helpers.tpl
+++ b/charts/jobset/templates/certmanager/_helpers.tpl
@@ -28,3 +28,11 @@ Create the name of the jobset webhook certificate.
 {{- define "jobset.certManager.certificate.name" -}}
 {{ include "jobset.name" . }}-cert
 {{- end -}}
+
+
+{{/*
+Create the name of the jobset metrics certificate.
+*/}}
+{{- define "jobset.certManager.metricsCertificate.name" -}}
+{{ include "jobset.name" . }}-metrics-cert
+{{- end -}}
diff --git a/charts/jobset/templates/certmanager/certificate.yaml b/charts/jobset/templates/certmanager/certificate.yaml
index d73fce7f3..94efbac61 100644
--- a/charts/jobset/templates/certmanager/certificate.yaml
+++ b/charts/jobset/templates/certmanager/certificate.yaml
@@ -32,7 +32,7 @@ spec:
   {{- else }}
     {{- toYaml .Values.certManager.issuerRef | nindent 4 }}
   {{- end }}
-  commonName: {{ include "jobset.webhook.service.name" . }}.{{ .Release.Namespace }}.svc
+  commonName: {{ include "jobset.name" . }}-webhook
   dnsNames:
   - {{ include "jobset.webhook.service.name" . }}.{{ .Release.Namespace }}.svc
   - {{ include "jobset.webhook.service.name" . }}.{{ .Release.Namespace }}.svc.cluster.local
diff --git a/charts/jobset/templates/certmanager/certificate_metrics.yaml b/charts/jobset/templates/certmanager/certificate_metrics.yaml
new file mode 100644
index 000000000..13b43d3e2
--- /dev/null
+++ b/charts/jobset/templates/certmanager/certificate_metrics.yaml
@@ -0,0 +1,44 @@
+{{- /*
+Copyright 2025 The Kubernetes authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/ -}}
+
+{{- if .Values.certManager.enable }}
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  name: {{ include "jobset.certManager.metricsCertificate.name" . }}
+  namespace: {{ .Release.Namespace}}
+  labels:
+    {{- include "jobset.labels" . | nindent 4 }}
+spec:
+  secretName: {{ include "jobset.metrics.secret.name" . }}
+  issuerRef:
+  {{- with .Values.certManager.issuerRef }}
+    {{ toYaml . | nindent 4 }}
+  {{- else }}
+    group: cert-manager.io
+    kind: Issuer
+    name: {{ include "jobset.certManager.issuer.name" . }}
+  {{- end }}
+  commonName: {{ include "jobset.name" . }}-metrics
+  dnsNames:
+  - {{ include "jobset.metrics.service.name" . }}.{{ .Release.Namespace }}.svc
+  - {{ include "jobset.metrics.service.name" . }}.{{ .Release.Namespace }}.svc.cluster.local
+  duration: 8760h
+  renewBefore: 720h
+  usages:
+  - server auth
+  - client auth
+{{- end }}
diff --git a/charts/jobset/templates/controller/deployment.yaml b/charts/jobset/templates/controller/deployment.yaml
index 080254e10..f9694d256 100644
--- a/charts/jobset/templates/controller/deployment.yaml
+++ b/charts/jobset/templates/controller/deployment.yaml
@@ -62,6 +62,11 @@ spec:
         - name: webhook-cert
           mountPath: /tmp/k8s-webhook-server/serving-certs
           readOnly: true
+        {{- if .Values.certManager.enable }}
+        - mountPath: /tmp/k8s-metrics-server/serving-certs
+          name: metrics-certs
+          readOnly: true
+        {{- end }}
         ports:
         - name: health-probe
           containerPort: 8081
@@ -69,11 +74,9 @@ spec:
         - name: webhook
           containerPort: 9443
           protocol: TCP
-        {{- if .Values.prometheus.enable }}
         - name: metrics
           containerPort: 8443
           protocol: TCP
-        {{- end }}
         livenessProbe:
           httpGet:
             port: 8081
@@ -111,6 +114,12 @@ spec:
         secret:
           secretName: {{ include "jobset.webhook.secret.name" . }}
           defaultMode: 420
+      {{- if .Values.certManager.enable }}
+      - name: metrics-certs
+        secret:
+          defaultMode: 420
+          secretName: {{ include "jobset.metrics.secret.name" . }}
+      {{- end }}
       {{- with .Values.controller.nodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 8 }}
diff --git a/charts/jobset/templates/prometheus/_helpers.tpl b/charts/jobset/templates/prometheus/_helpers.tpl
index 7fe11e97c..aa278f31c 100644
--- a/charts/jobset/templates/prometheus/_helpers.tpl
+++ b/charts/jobset/templates/prometheus/_helpers.tpl
@@ -34,3 +34,10 @@ Create the name of the jobset Prometheus service monitor.
 {{- define "jobset.metrics.serviceMonitor.name" -}}
 {{ include "jobset.metrics.name" . }}-service-monitor
 {{- end -}}
+
+{{/*
+Create the name of the jobset serving metrics TLS secret.
+*/}}
+{{- define "jobset.metrics.secret.name" -}}
+{{ include "jobset.fullname" . }}-metrics-server-cert
+{{- end -}}
diff --git a/charts/jobset/templates/prometheus/role.yaml b/charts/jobset/templates/prometheus/role.yaml
new file mode 100644
index 000000000..e194c8822
--- /dev/null
+++ b/charts/jobset/templates/prometheus/role.yaml
@@ -0,0 +1,36 @@
+{{- if .Values.prometheus.enable }}
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: {{ include "jobset.fullname" . }}-prometheus-k8s
+  namespace: {{ .Release.Namespace }}
+  labels:
+  {{- include "jobset.controller.labels" . | nindent 4 }}
+rules:
+- apiGroups:
+  - ""
+  resources:
+  - services
+  - endpoints
+  - pods
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - extensions
+  resources:
+  - ingresses
+  verbs:
+  - get
+  - list
+  - watch
+- apiGroups:
+  - networking.k8s.io
+  resources:
+  - ingresses
+  verbs:
+  - get
+  - list
+  - watch
+{{- end }}
diff --git a/charts/jobset/templates/prometheus/role_binding.yaml b/charts/jobset/templates/prometheus/role_binding.yaml
new file mode 100644
index 000000000..58fb7ee66
--- /dev/null
+++ b/charts/jobset/templates/prometheus/role_binding.yaml
@@ -0,0 +1,20 @@
+{{- if .Values.prometheus.enable }}
+apiVersion: rbac.authorization.k8s.io/v1
+kind: RoleBinding
+metadata:
+  name: {{ include "jobset.fullname" . }}-prometheus-k8s
+  namespace: {{ .Release.Namespace }}
+  labels:
+  {{- include "jobset.controller.labels" . | nindent 4 }}
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: Role
+  name: {{ include "jobset.fullname" . }}-prometheus-k8s
+subjects:
+- kind: ServiceAccount
+  name: prometheus-k8s
+  namespace: '{{ .Values.prometheus.prometheusNamespace }}'
+- kind: ServiceAccount
+  name: prometheus-operator
+  namespace: '{{ .Values.prometheus.prometheusNamespace }}'
+{{- end }}
diff --git a/charts/jobset/templates/prometheus/service.yaml b/charts/jobset/templates/prometheus/service.yaml
index ee07de277..7e4242356 100644
--- a/charts/jobset/templates/prometheus/service.yaml
+++ b/charts/jobset/templates/prometheus/service.yaml
@@ -13,22 +13,19 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 */ -}}
-
-{{- if .Values.prometheus.enable }}
 apiVersion: v1
 kind: Service
 metadata:
   name: {{ include "jobset.metrics.service.name" . }}
   namespace: {{ .Release.Namespace}}
   labels:
-    {{- include "jobset.labels" . | nindent 4 }}
+    {{- include "jobset.controller.selectorLabels" . | nindent 4 }}
 spec:
   type: ClusterIP
   selector:
     {{- include "jobset.controller.selectorLabels" . | nindent 4 }}
   ports:
-  - name: metrics
+  - name: https
     port: 8443
-    targetPort: metrics
     protocol: TCP
-{{- end }}
+    targetPort: 8443
diff --git a/charts/jobset/templates/prometheus/service_monitor.yaml b/charts/jobset/templates/prometheus/service_monitor.yaml
index 129e1aaaa..a0c9ca554 100644
--- a/charts/jobset/templates/prometheus/service_monitor.yaml
+++ b/charts/jobset/templates/prometheus/service_monitor.yaml
@@ -30,5 +30,27 @@ spec:
     matchLabels:
       {{- include "jobset.controller.selectorLabels" . | nindent 6 }}
   endpoints:
-  - port: metrics
+  - port: https
+    path: /metrics
+    scheme: https
+    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+{{- if .Values.certManager.enable }}
+    tlsConfig:
+      ca:
+        secret:
+          key: ca.crt
+          name: {{ include "jobset.metrics.secret.name" . }}
+      cert:
+        secret:
+          key: tls.crt
+          name: {{ .Values.prometheus.prometheusClientCertSecretName }}
+      insecureSkipVerify: false
+      keySecret:
+        key: tls.key
+        name: {{ .Values.prometheus.prometheusClientCertSecretName }}
+      serverName: {{ include "jobset.metrics.service.name" . }}.{{ .Release.Namespace }}.svc
+{{- else }}
+    tlsConfig:
+      insecureSkipVerify: true
+{{- end }}
 {{- end }}
diff --git a/charts/jobset/tests/prometheus/service_test.yaml b/charts/jobset/tests/prometheus/service_test.yaml
index 74219fe32..aa8171bec 100644
--- a/charts/jobset/tests/prometheus/service_test.yaml
+++ b/charts/jobset/tests/prometheus/service_test.yaml
@@ -24,12 +24,7 @@ release:
   namespace: jobset-system
 
 tests:
-  - it: Should not create Prometheus metrics service by default
-    asserts:
-      - hasDocuments:
-          count: 0
-
-  - it: Should create Prometheus Service if `prometheus.enable` is `true`
+  - it: Should create Prometheus Service by default
     set:
       prometheus:
         enable: true
diff --git a/charts/jobset/values.yaml b/charts/jobset/values.yaml
index e8bdff84c..a79069680 100644
--- a/charts/jobset/values.yaml
+++ b/charts/jobset/values.yaml
@@ -86,3 +86,11 @@ certManager:
 prometheus:
   # -- Whether to enable Prometheus metrics exporting.
   enable: false
+  # SHOULD BE CONFIGURED
+  prometheusNamespace: monitoring
+  # SHOULD BE CONFIGURED
+  # The same certificate for the server ("jobset-metrics-server-cert") can potentially be also used for the client.
+  # It is preferable to use prometheus client certificate if available:
+  # Either by creating a secret in the jobset controller namespace
+  # or by using prometheus mounted secret which can be configured via certFile and keyFile in the service_monitor.yaml
+  prometheusClientCertSecretName: "jobset-metrics-server-cert"
diff --git a/config/components/certmanager/certificate.yaml b/config/components/certmanager/certificate.yaml
index 294bbc270..4626ae04a 100644
--- a/config/components/certmanager/certificate.yaml
+++ b/config/components/certmanager/certificate.yaml
@@ -33,6 +33,12 @@ spec:
   dnsNames:
   - $(SERVICE_NAME).$(SERVICE_NAMESPACE).svc
   - $(SERVICE_NAME).$(SERVICE_NAMESPACE).svc.cluster.local
+  commonName: jobset-webhook
+  duration: 8760h
+  renewBefore: 720h
+  usages:
+    - server auth
+    - client auth
   issuerRef:
     kind: Issuer
     name: selfsigned-issuer
diff --git a/config/components/certmanager/certificate_metrics.yaml b/config/components/certmanager/certificate_metrics.yaml
new file mode 100644
index 000000000..30735fb59
--- /dev/null
+++ b/config/components/certmanager/certificate_metrics.yaml
@@ -0,0 +1,29 @@
+# The following manifests contain a certificate CR.
+# More document can be found at https://docs.cert-manager.io
+apiVersion: cert-manager.io/v1
+kind: Certificate
+metadata:
+  labels:
+    app.kubernetes.io/name: certificate
+    app.kubernetes.io/instance: metrics-cert
+    app.kubernetes.io/component: certificate
+    app.kubernetes.io/created-by: jobset
+    app.kubernetes.io/part-of: jobset
+    app.kubernetes.io/managed-by: kustomize
+  name: metrics-cert  # this name should match the one appeared in kustomizeconfig.yaml
+  namespace: system
+spec:
+  # METRICS_SERVICE_NAME and METRICS_SERVICE_NAMESPACE will be substituted by kustomize
+  dnsNames:
+    - $(METRICS_SERVICE_NAME).$(METRICS_SERVICE_NAMESPACE).svc
+    - $(METRICS_SERVICE_NAME).$(METRICS_SERVICE_NAMESPACE).svc.cluster.local
+  commonName: jobset-metrics
+  duration: 8760h
+  renewBefore: 720h
+  usages:
+    - server auth
+    - client auth
+  issuerRef:
+    kind: Issuer
+    name: selfsigned-issuer
+  secretName: metrics-server-cert # this secret will not be prefixed, since it's not managed by kustomize
diff --git a/config/components/certmanager/kustomization.yaml b/config/components/certmanager/kustomization.yaml
index bebea5a59..664197a2e 100644
--- a/config/components/certmanager/kustomization.yaml
+++ b/config/components/certmanager/kustomization.yaml
@@ -1,5 +1,6 @@
 resources:
 - certificate.yaml
+- certificate_metrics.yaml
 
 configurations:
 - kustomizeconfig.yaml
diff --git a/config/components/certmanager/kustomizeconfig.yaml b/config/components/certmanager/kustomizeconfig.yaml
index e631f7773..ebc9836b1 100644
--- a/config/components/certmanager/kustomizeconfig.yaml
+++ b/config/components/certmanager/kustomizeconfig.yaml
@@ -8,9 +8,6 @@ nameReference:
     path: spec/issuerRef/name
 
 varReference:
-- kind: Certificate
-  group: cert-manager.io
-  path: spec/commonName
 - kind: Certificate
   group: cert-manager.io
   path: spec/dnsNames
diff --git a/config/components/manager/controller_manager_config.yaml b/config/components/manager/controller_manager_config.yaml
index 95e0fda55..c6b29ca29 100644
--- a/config/components/manager/controller_manager_config.yaml
+++ b/config/components/manager/controller_manager_config.yaml
@@ -3,3 +3,5 @@ apiVersion: config.jobset.x-k8s.io/v1alpha1
 kind: Configuration
 leaderElection:
   leaderElect: true
+internalCertManagement:
+  enable: true
diff --git a/config/components/prometheus/kustomization.yaml b/config/components/prometheus/kustomization.yaml
index 0a21ea0fd..812802e66 100644
--- a/config/components/prometheus/kustomization.yaml
+++ b/config/components/prometheus/kustomization.yaml
@@ -1,3 +1,19 @@
 resources:
 - monitor.yaml
-- role.yaml
\ No newline at end of file
+- role.yaml
+
+# [PROMETHEUS WITH CERTMANAGER] The following patch configures the ServiceMonitor in ../prometheus
+# to securely reference certificates created and managed by cert-manager.
+# Ensure correct reference are set in the monitor_tls_patch.yaml file:
+# METRICS_SERVICE_NAME (e.g. jobset-controller-manager-metrics-service.jobset-system.svc),
+# and ${METRICS_PROMETHEUS_CLIENT_CERT_SECRET_NAME} (e.g. metrics-server-cert).
+# Additionally
+# - ensure that you uncomment the [METRICS WITH CERTMANAGER] patch under config/default/kustomization.yaml
+# to mount the "metrics-server-cert" secret in the Manager Deployment.
+# - ensure that you uncomment the [METRICS WITH CERTMANAGER] patch under config/certmanager/kustomization.yaml
+# to enable certmanager for metrics.
+# - ensure prometheus RoleBinding in ./role.yaml references a correct namespace
+#patches:
+#  - path: monitor_tls_patch.yaml
+#    target:
+#      kind: ServiceMonitor
diff --git a/config/components/prometheus/monitor_tls_patch.yaml b/config/components/prometheus/monitor_tls_patch.yaml
new file mode 100644
index 000000000..b675a03c7
--- /dev/null
+++ b/config/components/prometheus/monitor_tls_patch.yaml
@@ -0,0 +1,34 @@
+# Patch for Prometheus ServiceMonitor to enable secure TLS configuration
+# using certificates managed by cert-manager
+#
+# [PROMETHEUS WITH CERTMANAGER]
+# ${METRICS_PROMETHEUS_CLIENT_CERT_SECRET_NAME} should be replaced with a prometheus client secret.
+# The same certificate for the server ("metrics-server-cert") can potentially be also used for the client.
+# It is preferable to use prometheus client certificate if available:
+# Either by creating a secret in the jobset controller namespace
+# or by using prometheus mounted secret which can be configured via certFile and keyFile in the service_monitor.yaml
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: controller-manager-metrics-monitor
+  namespace: system
+spec:
+  endpoints:
+    - path: /metrics
+      port: https
+      scheme: https
+      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+      tlsConfig:
+        insecureSkipVerify: false
+        serverName: ${METRICS_SERVICE_NAME}.${METRICS_SERVICE_NAMESPACE}.svc
+        ca:
+          secret:
+            name: metrics-server-cert
+            key: ca.crt
+        cert:
+          secret:
+            name: ${METRICS_PROMETHEUS_CLIENT_CERT_SECRET_NAME}
+            key: tls.crt
+        keySecret:
+          name: ${METRICS_PROMETHEUS_CLIENT_CERT_SECRET_NAME}
+          key: tls.key
diff --git a/config/default/certmanager_metrics_manager_patch.yaml b/config/default/certmanager_metrics_manager_patch.yaml
new file mode 100644
index 000000000..92a78e45a
--- /dev/null
+++ b/config/default/certmanager_metrics_manager_patch.yaml
@@ -0,0 +1,19 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: controller-manager
+  namespace: system
+spec:
+  template:
+    spec:
+      containers:
+        - name: manager
+          volumeMounts:
+            - mountPath: /tmp/k8s-metrics-server/serving-certs
+              name: metrics-certs
+              readOnly: true
+      volumes:
+        - name: metrics-certs
+          secret:
+            defaultMode: 420
+            secretName: metrics-server-cert
diff --git a/config/default/kustomization.yaml b/config/default/kustomization.yaml
index 330514ec0..fad1834ae 100644
--- a/config/default/kustomization.yaml
+++ b/config/default/kustomization.yaml
@@ -20,6 +20,7 @@ resources:
 # crd/kustomization.yaml
 - ../components/webhook
 # [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER'. 'WEBHOOK' components are required.
+# - ../components/certmanager
 - ../components/internalcert
 # [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
 #- ../prometheus
@@ -42,6 +43,13 @@ patchesStrategicMerge:
 # 'CERTMANAGER' needs to be enabled to use ca injection
 #- webhookcainjection_patch.yaml
 
+#  Uncomment the patches line if you enable Metrics and CertManager
+# [METRICS WITH CERTMANGER] To enable metrics protected with certmanager, uncomment the following line.
+# This patch will protect the metrics with certmanager self-signed certs.
+#- certmanager_metrics_manager_patch.yaml
+
+# [PROMETHEUS WITH CERTMANAGER] to enable prometheus with certmanager, ../prometheus dir should be configured
+
 # the following config is for teaching kustomize how to do var substitution
 #vars:
 # [CERTMANAGER] To enable cert-manager, uncomment all sections with 'CERTMANAGER' prefix.
@@ -71,3 +79,15 @@ patchesStrategicMerge:
 #    kind: Service
 #    version: v1
 #    name: webhook-service
+#- name: METRICS_SERVICE_NAMESPACE # namespace of the service
+#  objref:
+#    kind: Service
+#    version: v1
+#    name: controller-manager-metrics-service
+#  fieldref:
+#    fieldpath: metadata.namespace
+#- name: METRICS_SERVICE_NAME
+#  objref:
+#    kind: Service
+#    version: v1
+#    name: controller-manager-metrics-service
diff --git a/main.go b/main.go
index c009721bd..ce3a4d79c 100644
--- a/main.go
+++ b/main.go
@@ -166,6 +166,9 @@ func main() {
 		SecureServing:  true,
 		FilterProvider: filters.WithAuthenticationAndAuthorization,
 		TLSOpts:        []func(*tls.Config){disableHTTP2},
+		CertDir:        options.Metrics.CertDir,
+		CertName:       options.Metrics.CertName,
+		KeyName:        options.Metrics.KeyName,
 	}
 	options.Metrics = metricsServerOptions
 	mgr, err := ctrl.NewManager(kubeConfig, options)
diff --git a/pkg/config/config.go b/pkg/config/config.go
index 22c082395..e5b6e459d 100644
--- a/pkg/config/config.go
+++ b/pkg/config/config.go
@@ -9,6 +9,7 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/runtime/serializer"
+	"k8s.io/utils/ptr"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/webhook"
 
@@ -37,6 +38,16 @@ func addTo(o *ctrl.Options, cfg *configapi.Configuration) {
 		o.Metrics.BindAddress = cfg.Metrics.BindAddress
 	}
 
+	internalCertManagementEnabled := ptr.Deref(cfg.InternalCertManagement.Enable, false)
+	if o.Metrics.CertDir == "" && (cfg.Metrics.CertDir != "" || !internalCertManagementEnabled) {
+		o.Metrics.CertDir = cfg.Metrics.CertDir
+		if o.Metrics.CertDir == "" {
+			o.Metrics.CertDir = "/tmp/k8s-metrics-server/serving-certs"
+		}
+		o.Metrics.CertName = "tls.crt"
+		o.Metrics.KeyName = "tls.key"
+	}
+
 	if o.HealthProbeBindAddress == "" && cfg.Health.HealthProbeBindAddress != "" {
 		o.HealthProbeBindAddress = cfg.Health.HealthProbeBindAddress
 	}
diff --git a/pkg/config/config_test.go b/pkg/config/config_test.go
index 3c6f80bc6..1f1b0de8e 100644
--- a/pkg/config/config_test.go
+++ b/pkg/config/config_test.go
@@ -141,6 +141,48 @@ clientConnection:
 		t.Fatal(err)
 	}
 
+	metricsCertConfig := filepath.Join(tmpDir, "metrics-cert.yaml")
+	if err := os.WriteFile(metricsCertConfig, []byte(`
+apiVersion: config.jobset.x-k8s.io/v1alpha1
+kind: Configuration
+health:
+  healthProbeBindAddress: :8081
+metrics:
+  bindAddress: :8443
+  certDir: /tmp/my-metrics-server/my-serving-certs
+leaderElection:
+  leaderElect: true
+  resourceName: 6d4f6a47.jobset.x-k8s.io
+webhook:
+  port: 9443
+internalCertManagement:
+  enable: true
+  webhookServiceName: jobset-tenant-a-webhook-service
+  webhookSecretName: jobset-tenant-a-webhook-server-cert
+`), os.FileMode(0600)); err != nil {
+		t.Fatal(err)
+	}
+
+	metricsCertWithExternalCertsConfig := filepath.Join(tmpDir, "metrics-cert-with-external-certs.yaml")
+	if err := os.WriteFile(metricsCertWithExternalCertsConfig, []byte(`
+apiVersion: config.jobset.x-k8s.io/v1alpha1
+kind: Configuration
+health:
+  healthProbeBindAddress: :8081
+metrics:
+  bindAddress: :8443
+  certDir: /tmp/my-metrics-server/my-serving-certs
+leaderElection:
+  leaderElect: true
+  resourceName: 6d4f6a47.jobset.x-k8s.io
+webhook:
+  port: 9443
+internalCertManagement:
+  enable: false
+`), os.FileMode(0600)); err != nil {
+		t.Fatal(err)
+	}
+
 	invalidConfig := filepath.Join(tmpDir, "invalid-config.yaml")
 	if err := os.WriteFile(invalidConfig, []byte(`
 apiVersion: config.jobset.x-k8s.io/v1alpha1
@@ -302,7 +344,26 @@ webhook:
 				},
 				ClientConnection: defaultClientConnection,
 			},
-			wantOptions: defaultControlOptions,
+			wantOptions: ctrl.Options{
+				HealthProbeBindAddress: configapi.DefaultHealthProbeBindAddress,
+				Metrics: metricsserver.Options{
+					BindAddress: configapi.DefaultMetricsBindAddress,
+					CertDir:     "/tmp/k8s-metrics-server/serving-certs",
+					CertName:    "tls.crt",
+					KeyName:     "tls.key",
+				},
+				LeaderElection:             true,
+				LeaderElectionID:           configapi.DefaultLeaderElectionID,
+				LeaderElectionResourceLock: resourcelock.LeasesResourceLock,
+				LeaseDuration:              ptr.To(configapi.DefaultLeaderElectionLeaseDuration),
+				RenewDeadline:              ptr.To(configapi.DefaultLeaderElectionRenewDeadline),
+				RetryPeriod:                ptr.To(configapi.DefaultLeaderElectionRetryPeriod),
+				WebhookServer: &webhook.DefaultServer{
+					Options: webhook.Options{
+						Port: configapi.DefaultWebhookPort,
+					},
+				},
+			},
 		},
 		{
 			name:       "leaderElection disabled config",
@@ -349,6 +410,77 @@ webhook:
 			},
 			wantOptions: defaultControlOptions,
 		},
+		{
+			name:       "metrics cert config",
+			configFile: metricsCertConfig,
+			wantConfiguration: configapi.Configuration{
+				TypeMeta: metav1.TypeMeta{
+					APIVersion: configapi.GroupVersion.String(),
+					Kind:       "Configuration",
+				},
+				InternalCertManagement: &configapi.InternalCertManagement{
+					Enable:             ptr.To(true),
+					WebhookServiceName: ptr.To("jobset-tenant-a-webhook-service"),
+					WebhookSecretName:  ptr.To("jobset-tenant-a-webhook-server-cert"),
+				},
+				ClientConnection: defaultClientConnection,
+			},
+			wantOptions: ctrl.Options{
+				HealthProbeBindAddress: configapi.DefaultHealthProbeBindAddress,
+				Metrics: metricsserver.Options{
+					BindAddress: ":8443",
+					CertDir:     "/tmp/my-metrics-server/my-serving-certs",
+					CertName:    "tls.crt",
+					KeyName:     "tls.key",
+				},
+				LeaderElection:             true,
+				LeaderElectionID:           configapi.DefaultLeaderElectionID,
+				LeaderElectionResourceLock: resourcelock.LeasesResourceLock,
+				LeaseDuration:              ptr.To(configapi.DefaultLeaderElectionLeaseDuration),
+				RenewDeadline:              ptr.To(configapi.DefaultLeaderElectionRenewDeadline),
+				RetryPeriod:                ptr.To(configapi.DefaultLeaderElectionRetryPeriod),
+				WebhookServer: &webhook.DefaultServer{
+					Options: webhook.Options{
+						Port: configapi.DefaultWebhookPort,
+					},
+				},
+			},
+		},
+
+		{
+			name:       "metrics cert config with external certs",
+			configFile: metricsCertWithExternalCertsConfig,
+			wantConfiguration: configapi.Configuration{
+				TypeMeta: metav1.TypeMeta{
+					APIVersion: configapi.GroupVersion.String(),
+					Kind:       "Configuration",
+				},
+				InternalCertManagement: &configapi.InternalCertManagement{
+					Enable: ptr.To(false),
+				},
+				ClientConnection: defaultClientConnection,
+			},
+			wantOptions: ctrl.Options{
+				HealthProbeBindAddress: configapi.DefaultHealthProbeBindAddress,
+				Metrics: metricsserver.Options{
+					BindAddress: ":8443",
+					CertDir:     "/tmp/my-metrics-server/my-serving-certs",
+					CertName:    "tls.crt",
+					KeyName:     "tls.key",
+				},
+				LeaderElection:             true,
+				LeaderElectionID:           configapi.DefaultLeaderElectionID,
+				LeaderElectionResourceLock: resourcelock.LeasesResourceLock,
+				LeaseDuration:              ptr.To(configapi.DefaultLeaderElectionLeaseDuration),
+				RenewDeadline:              ptr.To(configapi.DefaultLeaderElectionRenewDeadline),
+				RetryPeriod:                ptr.To(configapi.DefaultLeaderElectionRetryPeriod),
+				WebhookServer: &webhook.DefaultServer{
+					Options: webhook.Options{
+						Port: configapi.DefaultWebhookPort,
+					},
+				},
+			},
+		},
 		{
 			name:       "invalid config",
 			configFile: invalidConfig,


Reviews:
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T08:57:11Z
user: login: ardaguclu
body: I think, you need to configure helm charts?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T13:47:21Z
user: login: kannon92
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T21:27:03Z
user: login: ardaguclu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T05:46:13Z
user: login: kannon92
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-04T18:46:38Z
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T22:08:07Z
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T22:13:43Z
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T22:15:54Z
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T22:18:24Z
user: login: kannon92
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T00:08:30Z
user: login: kannon92
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T00:09:44Z
user: login: kannon92
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T00:11:14Z
user: login: kannon92
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T00:12:17Z
user: login: kannon92
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T00:13:12Z
user: login: ardaguclu
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T13:38:55Z
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T13:55:28Z
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T13:55:40Z
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T13:56:21Z
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T13:56:34Z
user: login: kannon92
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T17:10:24Z
user: login: kannon92
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T17:10:54Z
user: login: ardaguclu
body: Overall looks good to me. Dropped a few comments.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T06:21:00Z
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T22:10:46Z
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T22:16:38Z
user: login: atiratree
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T22:21:19Z

Review comments:
diff_hunk: @@ -42,6 +42,11 @@ patchesStrategicMerge:
 # 'CERTMANAGER' needs to be enabled to use ca injection
 #- webhookcainjection_patch.yaml
 
+#  Uncomment the patches line if you enable Metrics and CertManager
path: config/default/kustomization.yaml
user: login: atiratree
body: Bunch of deprecated fields like patchesStrategicMerge and vars are currently used by the kustomize and should be later updated. 
created_at: 2025-04-03T08:57:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,19 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: controller-manager
+  namespace: system
+spec:
+  template:
+    spec:
+      containers:
+        - name: manager
+          volumeMounts:
+            - mountPath: /tmp/k8s-metrics-server/metrics-certs
+              name: metrics-certs
+              readOnly: true
+      volumes:
+        - name: metrics-certs
path: config/default/certmanager_metrics_manager_patch.yaml
user: login: ardaguclu
body: `metrics-certs` name in Certificate will be prefixed to `jobset-metrics-certs`. Does the prefixing mechanism will prepend `jobset` to here as well?
created_at: 2025-04-03T13:41:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -42,6 +42,11 @@ patchesStrategicMerge:
 # 'CERTMANAGER' needs to be enabled to use ca injection
 #- webhookcainjection_patch.yaml
 
+#  Uncomment the patches line if you enable Metrics and CertManager
path: config/default/kustomization.yaml
user: login: ardaguclu
body: yeah, agree with you.
created_at: 2025-04-03T13:41:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,27 @@
+# Patch for Prometheus ServiceMonitor to enable secure TLS configuration
+# using certificates managed by cert-manager
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: controller-manager-metrics-monitor
+  namespace: system
+spec:
+  endpoints:
+    - path: /metrics
+      port: https
+      scheme: https
+      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+      tlsConfig:
+        insecureSkipVerify: false
+        serverName: $(METRICS_SERVICE_NAME).$(METRICS_SERVICE_NAMESPACE).svc
+        ca:
+          secret:
+            name: metrics-server-cert
+            key: ca.crt
+        cert:
+          secret:
+            name: $(CLIENT_CERT_NAME)
path: config/components/prometheus/monitor_tls_patch.yaml
user: login: ardaguclu
body: Where is the default value defined?
created_at: 2025-04-03T13:43:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +38,17 @@ func addTo(o *ctrl.Options, cfg *configapi.Configuration) {
 		o.Metrics.BindAddress = cfg.Metrics.BindAddress
 	}
 
+	internalCertManagementEnabled := ptr.Deref(cfg.InternalCertManagement.Enable, false)
+	if o.Metrics.CertDir == "" && (cfg.Metrics.CertDir != "" || !internalCertManagementEnabled) {
path: pkg/config/config.go
user: login: ardaguclu
body: internal cert management is enabled by default and nobody creates or injects these certificates (tls.crt, tls.key). I'd expect that CI should fail by throwing `certificates not found`, etc. But it didn't fail. How?
created_at: 2025-04-03T13:46:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -83,6 +83,12 @@ type ControllerMetrics struct {
 	// It can be set to "0" to disable the metrics serving.
 	// +optional
 	BindAddress string `json:"bindAddress,omitempty"`
+
+	// CertDir is the directory that contains the server key and certificate.
path: api/config/v1alpha1/configuration_types.go
user: login: kannon92
body: For kueue, we weren't really sure if there was any need to have this Cert configurable.

Any reason why we can't just default to a hardcoded location?
created_at: 2025-04-03T21:27:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -83,6 +83,12 @@ type ControllerMetrics struct {
 	// It can be set to "0" to disable the metrics serving.
 	// +optional
 	BindAddress string `json:"bindAddress,omitempty"`
+
+	// CertDir is the directory that contains the server key and certificate.
path: api/config/v1alpha1/configuration_types.go
user: login: ardaguclu
body: If we mount the default hardcoded locations `/tmp/k8s-metrics-server/serving-certs`, `tls.crt`, `tls.key`, there won't be any code changes needed in main.go. 
created_at: 2025-04-04T05:46:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -83,6 +83,12 @@ type ControllerMetrics struct {
 	// It can be set to "0" to disable the metrics serving.
 	// +optional
 	BindAddress string `json:"bindAddress,omitempty"`
+
+	// CertDir is the directory that contains the server key and certificate.
path: api/config/v1alpha1/configuration_types.go
user: login: kannon92
body: I think there would be one minor change in setting the right tlsOpts for metrics server. No having this CertDir does simply the code.

But if there is a reason for it I think leaving it in is fine.
created_at: 2025-04-04T18:46:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,19 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: controller-manager
+  namespace: system
+spec:
+  template:
+    spec:
+      containers:
+        - name: manager
+          volumeMounts:
+            - mountPath: /tmp/k8s-metrics-server/metrics-certs
+              name: metrics-certs
+              readOnly: true
+      volumes:
+        - name: metrics-certs
path: config/default/certmanager_metrics_manager_patch.yaml
user: login: atiratree
body: nope, no prefix here
created_at: 2025-04-07T22:08:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,27 @@
+# Patch for Prometheus ServiceMonitor to enable secure TLS configuration
+# using certificates managed by cert-manager
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: controller-manager-metrics-monitor
+  namespace: system
+spec:
+  endpoints:
+    - path: /metrics
+      port: https
+      scheme: https
+      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+      tlsConfig:
+        insecureSkipVerify: false
+        serverName: $(METRICS_SERVICE_NAME).$(METRICS_SERVICE_NAMESPACE).svc
+        ca:
+          secret:
+            name: metrics-server-cert
+            key: ca.crt
+        cert:
+          secret:
+            name: $(CLIENT_CERT_NAME)
path: config/components/prometheus/monitor_tls_patch.yaml
user: login: atiratree
body: I have added documentation to this file and to kustomize.yaml on what to set. There are many manual steps already and I believe it is better for the user to decide how the prometheus authenticates.

Similar approach is done in the helm charts.
created_at: 2025-04-07T22:13:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +38,17 @@ func addTo(o *ctrl.Options, cfg *configapi.Configuration) {
 		o.Metrics.BindAddress = cfg.Metrics.BindAddress
 	}
 
+	internalCertManagementEnabled := ptr.Deref(cfg.InternalCertManagement.Enable, false)
+	if o.Metrics.CertDir == "" && (cfg.Metrics.CertDir != "" || !internalCertManagementEnabled) {
path: pkg/config/config.go
user: login: atiratree
body: self signed certificate is generated by the metrics server instead
created_at: 2025-04-07T22:15:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -83,6 +83,12 @@ type ControllerMetrics struct {
 	// It can be set to "0" to disable the metrics serving.
 	// +optional
 	BindAddress string `json:"bindAddress,omitempty"`
+
+	// CertDir is the directory that contains the server key and certificate.
path: api/config/v1alpha1/configuration_types.go
user: login: atiratree
body: In kustomize and helm we offer prometheus support without cert manager. This gives the users a way to inject their certificates for the metrics server to the deployment.

We also offer the same thing for the webhook.
created_at: 2025-04-07T22:18:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +38,17 @@ func addTo(o *ctrl.Options, cfg *configapi.Configuration) {
 		o.Metrics.BindAddress = cfg.Metrics.BindAddress
 	}
 
+	internalCertManagementEnabled := ptr.Deref(cfg.InternalCertManagement.Enable, false)
+	if o.Metrics.CertDir == "" && (cfg.Metrics.CertDir != "" || !internalCertManagementEnabled) {
path: pkg/config/config.go
user: login: kannon92
body: Could we default to /tmp/k8s-metrics-server if internalCertManagement is false?

That way we can remove some of this setting and checking for nil?
created_at: 2025-04-11T00:08:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +38,17 @@ func addTo(o *ctrl.Options, cfg *configapi.Configuration) {
 		o.Metrics.BindAddress = cfg.Metrics.BindAddress
 	}
 
+	internalCertManagementEnabled := ptr.Deref(cfg.InternalCertManagement.Enable, false)
+	if o.Metrics.CertDir == "" && (cfg.Metrics.CertDir != "" || !internalCertManagementEnabled) {
path: pkg/config/config.go
user: login: kannon92
body: forget this. I thought I was looking elsewhere.
created_at: 2025-04-11T00:09:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -141,6 +141,28 @@ clientConnection:
 		t.Fatal(err)
 	}
 
+	metricsCertConfig := filepath.Join(tmpDir, "metrics-cert.yaml")
path: pkg/config/config_test.go
user: login: kannon92
body: Can we have a test where internalCertManagement is false and you specify a different value for certDir?
created_at: 2025-04-11T00:11:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -28,3 +28,11 @@ Create the name of the jobset webhook certificate.
 {{- define "jobset.certManager.certificate.name" -}}
 {{ include "jobset.name" . }}-cert
 {{- end -}}
+
+
+{{/*
+Create the name of the jobset metrics certificate.
+*/}}
+{{- define "jobset.certManager.metricscertificate.name" -}}
path: charts/jobset/templates/certmanager/_helpers.tpl
user: login: kannon92
body: ```suggestion
{{- define "jobset.certManager.metricsCertificate.name" -}}
```
created_at: 2025-04-11T00:12:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -62,18 +62,21 @@ spec:
         - name: webhook-cert
           mountPath: /tmp/k8s-webhook-server/serving-certs
           readOnly: true
+        {{- if .Values.certManager.enable }}
+        - mountPath: /tmp/k8s-metrics-server/metrics-certs
path: charts/jobset/templates/controller/deployment.yaml
user: login: kannon92
body: Would this work if someone specifies a different location than this in CertDir?
created_at: 2025-04-11T00:13:12Z
author_association: CONTRIBUTOR
diff_hunk: @@ -30,5 +30,24 @@ spec:
     matchLabels:
       {{- include "jobset.controller.selectorLabels" . | nindent 6 }}
   endpoints:
-  - port: metrics
+  - port: https
+{{- if .Values.certManager.enable }}
path: charts/jobset/templates/prometheus/service_monitor.yaml
user: login: ardaguclu
body: I think, there should be `else` if cert manager is not enabled and it should be `insecureSkipVerify: true`
created_at: 2025-04-15T13:38:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +38,17 @@ func addTo(o *ctrl.Options, cfg *configapi.Configuration) {
 		o.Metrics.BindAddress = cfg.Metrics.BindAddress
 	}
 
+	internalCertManagementEnabled := ptr.Deref(cfg.InternalCertManagement.Enable, false)
+	if o.Metrics.CertDir == "" && (cfg.Metrics.CertDir != "" || !internalCertManagementEnabled) {
path: pkg/config/config.go
user: login: atiratree
body: btw, I had an incorrect reference there, changed to `/tmp/k8s-metrics-server/serving-certs`
created_at: 2025-04-17T13:55:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -141,6 +141,28 @@ clientConnection:
 		t.Fatal(err)
 	}
 
+	metricsCertConfig := filepath.Join(tmpDir, "metrics-cert.yaml")
path: pkg/config/config_test.go
user: login: atiratree
body: +1 added a `metrics cert config with external certs` test
created_at: 2025-04-17T13:55:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -62,18 +62,21 @@ spec:
         - name: webhook-cert
           mountPath: /tmp/k8s-webhook-server/serving-certs
           readOnly: true
+        {{- if .Values.certManager.enable }}
+        - mountPath: /tmp/k8s-metrics-server/metrics-certs
path: charts/jobset/templates/controller/deployment.yaml
user: login: atiratree
body: no it would not, they would have to modify the deployment
created_at: 2025-04-17T13:56:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -30,5 +30,24 @@ spec:
     matchLabels:
       {{- include "jobset.controller.selectorLabels" . | nindent 6 }}
   endpoints:
-  - port: metrics
+  - port: https
+{{- if .Values.certManager.enable }}
path: charts/jobset/templates/prometheus/service_monitor.yaml
user: login: atiratree
body: +1, falling back to insecure
created_at: 2025-04-17T13:56:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -141,6 +141,28 @@ clientConnection:
 		t.Fatal(err)
 	}
 
+	metricsCertConfig := filepath.Join(tmpDir, "metrics-cert.yaml")
path: pkg/config/config_test.go
user: login: kannon92
body: looks good.
created_at: 2025-04-17T17:10:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -62,18 +62,21 @@ spec:
         - name: webhook-cert
           mountPath: /tmp/k8s-webhook-server/serving-certs
           readOnly: true
+        {{- if .Values.certManager.enable }}
+        - mountPath: /tmp/k8s-metrics-server/metrics-certs
path: charts/jobset/templates/controller/deployment.yaml
user: login: kannon92
body: So we should maybe mention this in our documentation.

Otherwise I worry this will be a support issue.
created_at: 2025-04-17T17:10:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +38,17 @@ func addTo(o *ctrl.Options, cfg *configapi.Configuration) {
 		o.Metrics.BindAddress = cfg.Metrics.BindAddress
 	}
 
+	internalCertManagementEnabled := ptr.Deref(cfg.InternalCertManagement.Enable, false)
+	if o.Metrics.CertDir == "" && (cfg.Metrics.CertDir != "" || !internalCertManagementEnabled) {
path: pkg/config/config.go
user: login: ardaguclu
body: I'm not sure about the logic in here. `o.Metrics.CertDir == ""` must be true, but we are also checking `cfg.Metrics.CertDir != ""` (this is always true already?)

Isn't this simpler?
```go
if !internalCertManagementEnabled {
	o.Metrics.CertName = "tls.crt"
	o.Metrics.KeyName = "tls.key"
        o.Metrics.CertDir = "/tmp/k8s-metrics-server/serving-certs"
        if cfg.Metrics.CertDir != "" {
           o.Metrics.CertDir = cfg.Metrics.CertDir
        }
}
```
created_at: 2025-04-18T06:08:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,34 @@
+# Patch for Prometheus ServiceMonitor to enable secure TLS configuration
+# using certificates managed by cert-manager
+#
+# [PROMETHEUS WITH CERTMANAGER]
+# ${METRICS_PROMETHEUS_CLIENT_CERT_SECRET_NAME} should be replaced with a prometheus client secret.
+# The same certificate for the server ("metrics-server-cert") can potentially be also used for the client.
+# It is preferable to use prometheus client certificate if available:
+# Either by creating a secret in the jobset controller namespace
+# or by using prometheus mounted secret which can be configured via certFile and keyFile in the service_monitor.yaml
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: controller-manager-metrics-monitor
+  namespace: system
+spec:
+  endpoints:
+    - path: /metrics
+      port: https
+      scheme: https
+      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+      tlsConfig:
+        insecureSkipVerify: false
path: config/components/prometheus/monitor_tls_patch.yaml
user: login: ardaguclu
body: I think, this should be `insecureSkipVerify: true` and there will be a "op: replace" command to replace with these values when cert-manager is enabled, in order to be compatible with https://github.com/kubernetes-sigs/jobset/pull/863/files#diff-89204119394a9cad711f1fdf90f9dd71be4f7ae8ef5bca4cd545661002a419bdR37-R55

created_at: 2025-04-18T06:20:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -37,6 +38,17 @@ func addTo(o *ctrl.Options, cfg *configapi.Configuration) {
 		o.Metrics.BindAddress = cfg.Metrics.BindAddress
 	}
 
+	internalCertManagementEnabled := ptr.Deref(cfg.InternalCertManagement.Enable, false)
+	if o.Metrics.CertDir == "" && (cfg.Metrics.CertDir != "" || !internalCertManagementEnabled) {
path: pkg/config/config.go
user: login: atiratree
body: - `o.Metrics.CertDir == ""` this is just a defense against overwriting the value in case it is already defined
-  `cfg.Metrics.CertDir` we can still pass empty certDir to have the self signed certs be generated by the jobset controller unless you disable the cert management

I have tried to simplify the branching a bit.
created_at: 2025-04-23T22:10:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -62,18 +62,21 @@ spec:
         - name: webhook-cert
           mountPath: /tmp/k8s-webhook-server/serving-certs
           readOnly: true
+        {{- if .Values.certManager.enable }}
+        - mountPath: /tmp/k8s-metrics-server/metrics-certs
path: charts/jobset/templates/controller/deployment.yaml
user: login: atiratree
body: This is the same approach we use with the webhook cert dir. Do we have to support such changes (i.e. custom deployments)? And we have documentation in the config API.
created_at: 2025-04-23T22:16:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,34 @@
+# Patch for Prometheus ServiceMonitor to enable secure TLS configuration
+# using certificates managed by cert-manager
+#
+# [PROMETHEUS WITH CERTMANAGER]
+# ${METRICS_PROMETHEUS_CLIENT_CERT_SECRET_NAME} should be replaced with a prometheus client secret.
+# The same certificate for the server ("metrics-server-cert") can potentially be also used for the client.
+# It is preferable to use prometheus client certificate if available:
+# Either by creating a secret in the jobset controller namespace
+# or by using prometheus mounted secret which can be configured via certFile and keyFile in the service_monitor.yaml
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: controller-manager-metrics-monitor
+  namespace: system
+spec:
+  endpoints:
+    - path: /metrics
+      port: https
+      scheme: https
+      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
+      tlsConfig:
+        insecureSkipVerify: false
path: config/components/prometheus/monitor_tls_patch.yaml
user: login: atiratree
body: This is applied only when used in `PROMETHEUS WITH CERTMANAGER` configuration. 

We do use `insecureSkipVerify: true` otherwise in https://github.com/kubernetes-sigs/jobset/blob/789481a770af28fd94bf7724c54bd11d85759aea/config/components/prometheus/monitor.yaml#L23
created_at: 2025-04-23T22:21:19Z
author_association: CONTRIBUTOR

PR comments:
user: login: netlify[bot]
created_at: 2025-04-03T08:54:33Z
body: ### <span aria-hidden="true">✅</span> Deploy Preview for *kubernetes-sigs-jobset* canceled.


|  Name | Link |
|:-:|------------------------|
|<span aria-hidden="true">🔨</span> Latest commit | aa390c8e04207a0019ce6cbc506528301c7895a5 |
|<span aria-hidden="true">🔍</span> Latest deploy log | https://app.netlify.com/sites/kubernetes-sigs-jobset/deploys/6809642737c60b000822c08c |
author_association: NONE
user: login: kannon92
created_at: 2025-04-07T20:46:38Z
body: Looks like metrics were never functionally with helm.. :(

Can you still look into enabling them as you are doing this?
author_association: CONTRIBUTOR
user: login: atiratree
created_at: 2025-04-07T22:21:24Z
body: @kannon92 @ardaguclu Yes, I was missing helm and there were so many things broken that it took me a fw days to get it working. 

Now the metrics collection works - tested with cert manager and prometheus - with both helm and kustomize.

The commits are all over the place, but a bit interwoven, so not sure if it is a good idea to split the the PR into multiple ones. Please let me know if you have any thoughts on this.
author_association: CONTRIBUTOR
user: login: kannon92
created_at: 2025-04-10T15:07:55Z
body: Looks like our prometheus tests caught something!
author_association: CONTRIBUTOR
user: login: atiratree
created_at: 2025-04-17T13:58:14Z
body: > Looks like our prometheus tests caught something!

@kannon92 I think it is useful to have the service present even without the prometheus. I have updated the `service_test.yaml` to reflect that.
author_association: CONTRIBUTOR
user: login: kannon92
created_at: 2025-04-17T17:11:18Z
body: just one item.

/approve

I leave final lgtm for @ardaguclu 
author_association: CONTRIBUTOR
user: login: k8s-ci-robot
created_at: 2025-04-17T17:11:25Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/jobset/pull/863#" title="Author self-approved">atiratree</a>*, *<a href="https://github.com/kubernetes-sigs/jobset/pull/863#issuecomment-2813583622" title="Approved">kannon92</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fjobset).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/jobset/blob/main/OWNERS)~~ [kannon92]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: ardaguclu
created_at: 2025-04-24T05:44:41Z
body: /lgtm
author_association: CONTRIBUTOR