Repository: kubernetes-sigs/kueue

Issue:
Support the use of Cert Manager
For my workflow, we want to rely on Cert Manager for deploying Kueue.

We found a few problems with using the upstream manifests.

I opened up https://github.com/kubernetes-sigs/kueue/pull/4082 but there is some more work needed.

- [x] https://github.com/kubernetes-sigs/kueue/pull/4082 
   -  Fix cert manager manifests
- [x] make sure the cert-manager supports running webhooks for all Job CRDs (like JobSet, KubeRay, KubeflowJobs etc),
- [x] use for metrics endpoint. 
  As discussed also in the https://github.com/kubernetes-sigs/kueue/issues/3259#issuecomment-2578652710 and https://github.com/kubernetes-sigs/kueue/pull/4082#issuecomment-2623625040

- [x] Verify step for cert manager manifests
- [x] e2e test



Verify step can be as simple as confirming that kustomize build works with Cert Manager configs

E2E test should be a simple test to verify kueue deployment works and we can follow [4112](https://github.com/kubernetes-sigs/kueue/pull/4112) as a template for how to test custom configs.

Hints from issue comments:
@tenzen-y: /kind feature@sohankunkerkar: /assign sohankunkerkar@mimowo: /retitle Support the use of Cert Manager

Proposal, I think "coverage" could be mistakenly understood as just tests.

I would also like to add to the requirements:
- use for metrics endpoint. As discussed also in the [comments](https://github.com/kubernetes-sigs/kueue/issues/3259#issuecomment-2578652710) and [here](https://github.com/kubernetes-sigs/kueue/pull/4082#issuecomment-2623625040)
- make sure the cert-manager supports running webhooks for all Job CRDs (like JobSet, KubeRay, KubeflowJobs etc), I'm not sure it was done as part of the work [here](https://github.com/kubernetes-sigs/kueue/pull/4082), @kannon92 ? @kannon92: I created https://github.com/kubernetes-sigs/kueue/issues/4377 for metrics endpoint. Right now one cannot use cert manager with prometheus as their are no options provided to configure it.@sohankunkerkar: >make sure the cert-manager supports running webhooks for all Job CRDs (like JobSet, KubeRay, KubeflowJobs etc),

I'm able to verify this by installing cert-manager using this [manifest](https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml) and running an e2e test suite for the single-cluster setup, which looks fine. Is there anything else I need to confirm? I will be writing an e2e test for this to further strengthen things.@sohankunkerkar: Is the idea here to enable cert-manager by default or make it opt-in?




@kannon92: opt-in.@kannon92: I think the idea is to make sure we have some kind of CI protection that Cert Manager integration still works.

We can inject a custom config and run e2e tests just to make sure everything works as intended.

Correct @mimowo?
@mimowo: Yes, automated E2e tests for the cert manager would be awesome. I think you can try to use the customconfigs CI job and the suite for the purpose.@mimowo: /reopen 
to complete the testing of the support@k8s-ci-robot: @mimowo: Reopened this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/4164#issuecomment-2734092653):

>/reopen 
>to complete the testing of the support


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@kannon92: For posterity, https://github.com/kubernetes-sigs/kueue/pull/4579#issuecomment-2722262381.@mimowo: @sohankunkerkar I think ideally, we also have an e2e test (part of the new certmanager suite), but for metrics endpoint.

We already have some testing for metrics, but we use -k, as the certs are self-signed and not trusted, see [here](https://github.com/kubernetes-sigs/kueue/blob/016692ab8908263063e1ef252edcf232bfafd738/test/e2e/singlecluster/metrics_test.go#L577). wdyt?@sohankunkerkar: I think that's a good suggestion. I'll work on adding that.@kannon92: @sohankunkerkar @mimowo anything outstanding on this?@mimowo: not from me, I'm good to close, along with https://github.com/kubernetes-sigs/kueue/issues/3259

@kannon92 and @sohankunkerkar thank you for all the effort!@tenzen-y: thank you everyone. I'm fine with closing this one as well@kannon92: /close@k8s-ci-robot: @kannon92: Closing this issue.

<details>

In response to [this](https://github.com/kubernetes-sigs/kueue/issues/4164#issuecomment-2828528009):

>/close


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>

Solution patch:
diff --git a/charts/kueue/templates/certmanager/certificatemetrics.yaml b/charts/kueue/templates/certmanager/certificatemetrics.yaml
index 4c5cf15fb..75c932d82 100644
--- a/charts/kueue/templates/certmanager/certificatemetrics.yaml
+++ b/charts/kueue/templates/certmanager/certificatemetrics.yaml
@@ -7,6 +7,7 @@ metadata:
   labels:
   {{- include "kueue.labels" . | nindent 4 }}
 spec:
+  commonName: {{ include "kueue.fullname" . }}-metrics
   dnsNames:
   - '{{ include "kueue.fullname" . }}-controller-manager-metrics-service.{{ .Release.Namespace }}.svc'
   - '{{ include "kueue.fullname" . }}-controller-manager-metrics-service.{{ .Release.Namespace }}.svc.{{ .Values.kubernetesClusterDomain }}'
diff --git a/config/components/certmanager/certificate-metrics.yaml b/config/components/certmanager/certificate-metrics.yaml
index 958cb7c16..2fab4da27 100644
--- a/config/components/certmanager/certificate-metrics.yaml
+++ b/config/components/certmanager/certificate-metrics.yaml
@@ -6,11 +6,12 @@ metadata:
   name: metrics-certs  # this name should match the one appeared in kustomizeconfig.yaml
   namespace: system
 spec:
+  commonName: kueue-metrics
   dnsNames:
-  # SERVICE_NAME and SERVICE_NAMESPACE will be substituted by kustomize
+  # METRICS_SERVICE_NAME and METRICS_SERVICE_NAMESPACE will be substituted by kustomize
   # replacements in the config/default/kustomization.yaml file.
-  - SERVICE_NAME.SERVICE_NAMESPACE.svc
-  - SERVICE_NAME.SERVICE_NAMESPACE.svc.cluster.local
+  - $(METRICS_SERVICE_NAME).$(METRICS_SERVICE_NAMESPACE).svc
+  - $(METRICS_SERVICE_NAME).$(METRICS_SERVICE_NAMESPACE).svc.cluster.local
   issuerRef:
     kind: Issuer
     name: selfsigned-issuer
diff --git a/config/default/kustomization.yaml b/config/default/kustomization.yaml
index 18a0dc483..f27795131 100644
--- a/config/default/kustomization.yaml
+++ b/config/default/kustomization.yaml
@@ -95,3 +95,15 @@ patches:
 #     kind: Service
 #     version: v1
 #     name: webhook-service
+# - name: METRICS_SERVICE_NAME
+#   objref:
+#     kind: Service
+#     version: v1
+#     name: controller-manager-metrics-service
+# - name: METRICS_SERVICE_NAMESPACE
+#   objref:
+#     kind: Service
+#     version: v1
+#     name: controller-manager-metrics-service
+#   fieldref:
+#     fieldpath: metadata.namespace
diff --git a/hack/internal/tools/go.mod b/hack/internal/tools/go.mod
index 7ccabc0ce..f1bd4cefc 100644
--- a/hack/internal/tools/go.mod
+++ b/hack/internal/tools/go.mod
@@ -9,7 +9,7 @@ require (
 	github.com/onsi/ginkgo/v2 v2.23.4
 	gopkg.in/yaml.v3 v3.0.1
 	gotest.tools/gotestsum v1.12.1
-	helm.sh/helm/v3 v3.17.2
+	helm.sh/helm/v3 v3.17.3
 	k8s.io/code-generator v0.32.3
 	sigs.k8s.io/controller-runtime/tools/setup-envtest v0.0.0-20240813183042-b901db121e1f
 	sigs.k8s.io/controller-tools v0.17.3
diff --git a/hack/internal/tools/go.sum b/hack/internal/tools/go.sum
index abfe714eb..21874f0b1 100644
--- a/hack/internal/tools/go.sum
+++ b/hack/internal/tools/go.sum
@@ -323,8 +323,8 @@ gotest.tools/gotestsum v1.12.1 h1:dvcxFBTFR1QsQmrCQa4k/vDXow9altdYz4CjdW+XeBE=
 gotest.tools/gotestsum v1.12.1/go.mod h1:mwDmLbx9DIvr09dnAoGgQPLaSXszNpXpWo2bsQge5BE=
 gotest.tools/v3 v3.5.1 h1:EENdUnS3pdur5nybKYIh2Vfgc8IUNBjxDPSjtiJcOzU=
 gotest.tools/v3 v3.5.1/go.mod h1:isy3WKz7GK6uNw/sbHzfKBLvlvXwUyV06n6brMxxopU=
-helm.sh/helm/v3 v3.17.2 h1:agYQ5ew2jq5vdx2K7q5W44KyKQrnSubUMCQsjkiv3/o=
-helm.sh/helm/v3 v3.17.2/go.mod h1:+uJKMH/UiMzZQOALR3XUf3BLIoczI2RKKD6bMhPh4G8=
+helm.sh/helm/v3 v3.17.3 h1:3n5rW3D0ArjFl0p4/oWO8IbY/HKaNNwJtOQFdH2AZHg=
+helm.sh/helm/v3 v3.17.3/go.mod h1:+uJKMH/UiMzZQOALR3XUf3BLIoczI2RKKD6bMhPh4G8=
 k8s.io/api v0.32.3 h1:Hw7KqxRusq+6QSplE3NYG4MBxZw1BZnq4aP4cJVINls=
 k8s.io/api v0.32.3/go.mod h1:2wEDTXADtm/HA7CCMD8D8bK4yuBUptzaRhYcYEEYA3k=
 k8s.io/apiextensions-apiserver v0.32.2 h1:2YMk285jWMk2188V2AERy5yDwBYrjgWYggscghPCvV4=
diff --git a/site/content/en/docs/talks_and_presentations/_index.md b/site/content/en/docs/talks_and_presentations/_index.md
index 5ee5d9885..ea9ee25a3 100644
--- a/site/content/en/docs/talks_and_presentations/_index.md
+++ b/site/content/en/docs/talks_and_presentations/_index.md
@@ -12,38 +12,45 @@ description: >
 # 2024
 
 * KubeCon NA, November, Salt Lake City<br>
-  [Keynote: Multicluster Batch Jobs Dispatching with Kueue at CERN](https://youtu.be/xMmskWIlktA)<br>
-  Ricardo Rocha, CERN & Marcin Wielgus, Google
+  Keynote: Multicluster Batch Jobs Dispatching with Kueue at CERN<br>
+  Ricardo Rocha, CERN & Marcin Wielgus, Google<br>
+  [talk details](https://sched.co/1iCOV) | [video](https://youtu.be/xMmskWIlktA)<br>
 
 * KubeCon NA, November, Salt Lake City<br>
-  [Multitenancy and Fairness at Scale with Kueue: A Case Study](https://youtu.be/GYiuTQCvTx8)<br>
-  Aldo Culquicondor, Google & Rajat Phull, Apple
+  Multitenancy and Fairness at Scale with Kueue: A Case Study<br>
+  Aldo Culquicondor, Google & Rajat Phull, Apple<br>
+  [talk details](https://sched.co/1izqO) | [video](https://youtu.be/GYiuTQCvTx8)
 
 * KubeCon EU, March, Paris<br>
-  [Advanced Resource Management for Running AI/ML Workloads with Kueue](https://youtu.be/6k_8Go3u8Qk)<br>
-  Michał Woźniak, Google & Yuki Iwai, CyberAgent, Inc.
+  Advanced Resource Management for Running AI/ML Workloads with Kueue<br>
+  Michał Woźniak, Google & Yuki Iwai, CyberAgent, Inc.<br>
+  [talk details](https://kccnceu2024.sched.com/event/1YeLj) | [video](https://youtu.be/6k_8Go3u8Qk)
 
 * KubeCon EU, March, Paris<br>
-  [Scale Your Batch / Big Data / AI Workloads Beyond the Kubernetes Scheduler](https://youtu.be/Ij5EAnuF-jk)<br>
+  Scale Your Batch / Big Data / AI Workloads Beyond the Kubernetes Scheduler<br>
   Antonin Stefanutti & Anish Asthana, Red Hat<br>
-  Supplementary materials: [astefanutti/kube-schedulers](https://github.com/astefanutti/kube-schedulers)
+  [talk details](https://sched.co/1YFjv) | [video](https://youtu.be/Ij5EAnuF-jk) | [supplementary materials](https://github.com/astefanutti/kube-schedulers)
 
 # 2023
 
 * KubeCon NA, November, Chicago<br>
-  [Batch Systems in Production with Kueue: Multi-Tenancy and Fungibility](https://youtu.be/cEnor-oW9_s)<br>
-  Yuki Iwai, CyberAgent, Inc. & Aldo Culquicondor, Google
+  Batch Systems in Production with Kueue: Multi-Tenancy and Fungibility<br>
+  Yuki Iwai, CyberAgent, Inc. & Aldo Culquicondor, Google<br>
+  [talk details](https://sched.co/1Rj23) | [video](https://youtu.be/cEnor-oW9_s)
 
 * KubeCon China, September, Shanghai<br>
-  [Sailing Ray Workloads with KubeRay and Kueue in Kubernetes](https://youtu.be/Q-sQLDMeJ8M)<br>
-  Jason Hu, Volcano Engine & Kante Yin, DaoCloud
+  Sailing Ray Workloads with KubeRay and Kueue in Kubernetes<br>
+  Jason Hu, Volcano Engine & Kante Yin, DaoCloud<br>
+  [talk details](https://sched.co/1PTGw) | [video](https://youtu.be/Q-sQLDMeJ8M)<br>
 
 * KubeCon EU, April, Amsterdam<br>
-  [Building a Batch System for the Cloud with Kueue](https://youtu.be/5qasif08vnM)<br>
-  Aldo Culquicondor, Google & Kante Yin, DaoCloud
+  Building a Batch System for the Cloud with Kueue<br>
+  Aldo Culquicondor, Google & Kante Yin, DaoCloud<br>
+  [talk details](https://sched.co/1Jo8r) | [video](https://youtu.be/5qasif08vnM)
 
 # 2022
 
 * KubeCon EU, March, Valencia<br>
-  [Kueue: A Kubernetes-native Job Queueing](https://youtu.be/YwSZUdU3iRY)<br>
-  Abdullah Gharaibeh, Google
+  Kueue: A Kubernetes-native Job Queueing<br>
+  Abdullah Gharaibeh, Google<br>
+  [talk details](https://sched.co/10F0V) | [video](https://youtu.be/YwSZUdU3iRY)<br>
diff --git a/site/static/examples/multikueue/create-multikueue-kubeconfig.sh b/site/static/examples/multikueue/create-multikueue-kubeconfig.sh
index 341b69d06..a8974b8e1 100644
--- a/site/static/examples/multikueue/create-multikueue-kubeconfig.sh
+++ b/site/static/examples/multikueue/create-multikueue-kubeconfig.sh
@@ -207,6 +207,22 @@ rules:
   - rayclusters/status
   verbs:
   - get
+- apiGroups:
+  - workload.codeflare.dev
+  resources:
+  - appwrappers
+  verbs:
+  - create
+  - delete
+  - get
+  - list
+  - watch
+- apiGroups:
+  - workload.codeflare.dev
+  resources:
+  - appwrappers/status
+  verbs:
+  - get
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: ClusterRoleBinding



Test patch:
diff --git a/test/e2e/certmanager/certmanager_test.go b/test/e2e/certmanager/certmanager_test.go
index d7bc4dafd..47bd44d2f 100644
--- a/test/e2e/certmanager/certmanager_test.go
+++ b/test/e2e/certmanager/certmanager_test.go
@@ -17,16 +17,12 @@ limitations under the License.
 package certmanager
 
 import (
-	"time"
-
 	"github.com/onsi/ginkgo/v2"
 	"github.com/onsi/gomega"
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
-	"k8s.io/utils/ptr"
 
-	"sigs.k8s.io/kueue/apis/config/v1beta1"
 	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta1"
 	workloadjob "sigs.k8s.io/kueue/pkg/controller/jobs/job"
 	"sigs.k8s.io/kueue/pkg/util/testing"
@@ -42,18 +38,6 @@ var _ = ginkgo.Describe("CertManager", ginkgo.Ordered, func() {
 		clusterQueue *kueue.ClusterQueue
 	)
 
-	ginkgo.BeforeAll(func() {
-		configurationUpdate := time.Now()
-		config := defaultKueueCfg.DeepCopy()
-		config.InternalCertManagement = &v1beta1.InternalCertManagement{
-			Enable: ptr.To(false),
-		}
-		util.ApplyKueueConfiguration(ctx, k8sClient, config)
-		util.RestartKueueController(ctx, k8sClient)
-		util.WaitForKueueAvailability(ctx, k8sClient)
-		ginkgo.GinkgoLogr.Info("Kueue configuration updated", "took", time.Since(configurationUpdate))
-	})
-
 	ginkgo.BeforeEach(func() {
 		ns = &corev1.Namespace{
 			ObjectMeta: metav1.ObjectMeta{GenerateName: "e2e-cert-manager-"},
diff --git a/test/e2e/certmanager/metrics_test.go b/test/e2e/certmanager/metrics_test.go
new file mode 100644
index 000000000..67ef82d53
--- /dev/null
+++ b/test/e2e/certmanager/metrics_test.go
@@ -0,0 +1,210 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package certmanager
+
+import (
+	"fmt"
+
+	"github.com/onsi/ginkgo/v2"
+	"github.com/onsi/gomega"
+	corev1 "k8s.io/api/core/v1"
+	rbacv1 "k8s.io/api/rbac/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	config "sigs.k8s.io/kueue/apis/config/v1beta1"
+	"sigs.k8s.io/kueue/apis/kueue/v1beta1"
+	utiltesting "sigs.k8s.io/kueue/pkg/util/testing"
+	testingjobspod "sigs.k8s.io/kueue/pkg/util/testingjobs/pod"
+	"sigs.k8s.io/kueue/test/util"
+)
+
+const (
+	serviceAccountName           = "kueue-controller-manager"
+	metricsReaderClusterRoleName = "kueue-metrics-reader"
+	metricsServiceName           = "kueue-controller-manager-metrics-service"
+	certSecretName               = "metrics-server-cert"
+	certMountPath                = "/etc/kueue/metrics/certs"
+)
+
+var _ = ginkgo.Describe("Metrics", ginkgo.Ordered, func() {
+	var (
+		ns             *corev1.Namespace
+		resourceFlavor *v1beta1.ResourceFlavor
+
+		metricsReaderClusterRoleBinding *rbacv1.ClusterRoleBinding
+
+		curlContainerName string
+		curlPod           *corev1.Pod
+	)
+
+	ginkgo.BeforeEach(func() {
+		ns = utiltesting.MakeNamespaceWithGenerateName("e2e-metrics-")
+		gomega.Expect(k8sClient.Create(ctx, ns)).To(gomega.Succeed())
+		resourceFlavor = utiltesting.MakeResourceFlavor("test-flavor").Obj()
+		gomega.Expect(k8sClient.Create(ctx, resourceFlavor)).To(gomega.Succeed())
+
+		metricsReaderClusterRoleBinding = &rbacv1.ClusterRoleBinding{
+			ObjectMeta: metav1.ObjectMeta{Name: "metrics-reader-rolebinding"},
+			Subjects: []rbacv1.Subject{
+				{
+					Kind:      "ServiceAccount",
+					Name:      serviceAccountName,
+					Namespace: config.DefaultNamespace,
+				},
+			},
+			RoleRef: rbacv1.RoleRef{
+				APIGroup: rbacv1.GroupName,
+				Kind:     "ClusterRole",
+				Name:     metricsReaderClusterRoleName,
+			},
+		}
+		gomega.Expect(k8sClient.Create(ctx, metricsReaderClusterRoleBinding)).Should(gomega.Succeed())
+
+		curlPod = testingjobspod.MakePod("curl-metrics", config.DefaultNamespace).
+			ServiceAccountName(serviceAccountName).
+			Image(util.E2eTestAgnHostImage, util.BehaviorWaitForDeletion).
+			TerminationGracePeriod(1).
+			Obj()
+		curlPod.Spec.Volumes = []corev1.Volume{
+			corev1.Volume{
+				Name: "metrics-certs",
+				VolumeSource: corev1.VolumeSource{
+					Secret: &corev1.SecretVolumeSource{
+						SecretName: certSecretName,
+						Items: []corev1.KeyToPath{
+							{Key: "ca.crt", Path: "ca.crt"},
+						},
+					},
+				},
+			},
+		}
+		curlPod.Spec.Containers[0].VolumeMounts = []corev1.VolumeMount{
+			{
+				Name:      "metrics-certs",
+				MountPath: certMountPath,
+				ReadOnly:  true,
+			},
+		}
+		gomega.Expect(k8sClient.Create(ctx, curlPod)).To(gomega.Succeed())
+
+		ginkgo.By("Waiting for metrics-server-cert secret", func() {
+			gomega.Eventually(func(g gomega.Gomega) {
+				secret := &corev1.Secret{}
+				g.Expect(k8sClient.Get(ctx, client.ObjectKey{
+					Namespace: config.DefaultNamespace,
+					Name:      certSecretName,
+				}, secret)).To(gomega.Succeed())
+			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+		})
+
+		ginkgo.By("Waiting for the curl-metrics pod to run", func() {
+			gomega.Eventually(func(g gomega.Gomega) {
+				createdPod := &corev1.Pod{}
+				g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(curlPod), createdPod)).To(gomega.Succeed())
+				g.Expect(createdPod.Status.Phase).To(gomega.Equal(corev1.PodRunning))
+				curlContainerName = createdPod.Spec.Containers[0].Name
+			}, util.Timeout, util.Interval).Should(gomega.Succeed())
+		})
+	})
+
+	ginkgo.AfterEach(func() {
+		gomega.Expect(util.DeleteNamespace(ctx, k8sClient, ns)).To(gomega.Succeed())
+		util.ExpectObjectToBeDeleted(ctx, k8sClient, resourceFlavor, true)
+		util.ExpectObjectToBeDeleted(ctx, k8sClient, metricsReaderClusterRoleBinding, true)
+		util.ExpectObjectToBeDeletedWithTimeout(ctx, k8sClient, curlPod, true, util.LongTimeout)
+		util.ExpectAllPodsInNamespaceDeleted(ctx, k8sClient, ns)
+	})
+
+	ginkgo.When("workload is admitted", func() {
+		var (
+			clusterQueue *v1beta1.ClusterQueue
+			localQueue   *v1beta1.LocalQueue
+			workload     *v1beta1.Workload
+		)
+
+		ginkgo.BeforeEach(func() {
+			clusterQueue = utiltesting.MakeClusterQueue("").
+				GeneratedName("test-cq-").
+				ResourceGroup(
+					*utiltesting.MakeFlavorQuotas(resourceFlavor.Name).
+						Resource(corev1.ResourceCPU, "1").
+						Resource(corev1.ResourceMemory, "1Gi").
+						Obj(),
+				).
+				Obj()
+			gomega.Expect(k8sClient.Create(ctx, clusterQueue)).To(gomega.Succeed())
+
+			localQueue = utiltesting.MakeLocalQueue("", ns.Name).
+				GeneratedName("test-lq-").
+				ClusterQueue(clusterQueue.Name).
+				Obj()
+			gomega.Expect(k8sClient.Create(ctx, localQueue)).To(gomega.Succeed())
+
+			workload = utiltesting.MakeWorkload("test-workload", ns.Name).
+				Queue(localQueue.Name).
+				PodSets(
+					*utiltesting.MakePodSet("ps1", 1).Obj(),
+				).
+				RequestAndLimit(corev1.ResourceCPU, "1").
+				Obj()
+			gomega.Expect(k8sClient.Create(ctx, workload)).To(gomega.Succeed())
+		})
+
+		ginkgo.AfterEach(func() {
+			util.ExpectObjectToBeDeleted(ctx, k8sClient, workload, true)
+			util.ExpectObjectToBeDeleted(ctx, k8sClient, localQueue, true)
+			util.ExpectObjectToBeDeleted(ctx, k8sClient, clusterQueue, true)
+		})
+
+		ginkgo.It("should expose quota reserved workload metric", func() {
+			util.ExpectWorkloadsToBeAdmitted(ctx, k8sClient, workload)
+
+			ginkgo.By("checking that the quota reserved workload metric is present", func() {
+				expectedMetric := []string{
+					"kueue_quota_reserved_workloads_total",
+					clusterQueue.Name,
+				}
+				expectMetricsToBeAvailable(curlPod.Name, curlContainerName, [][]string{expectedMetric})
+			})
+		})
+	})
+})
+
+func getKueueMetricsSecure(curlPodName, curlContainerName string) ([]byte, error) {
+	metricsOutput, _, err := util.KExecute(ctx, cfg, restClient, config.DefaultNamespace, curlPodName, curlContainerName,
+		[]string{
+			"/bin/sh", "-c",
+			fmt.Sprintf(
+				"curl -s --cacert %s/ca.crt -H \"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)\" https://%s.%s.svc.cluster.local:8443/metrics",
+				certMountPath,
+				metricsServiceName,
+				config.DefaultNamespace,
+			),
+		})
+
+	return metricsOutput, err
+}
+
+func expectMetricsToBeAvailable(curlPodName, curlContainerName string, metrics [][]string) {
+	gomega.Eventually(func(g gomega.Gomega) {
+		metricsOutput, err := getKueueMetricsSecure(curlPodName, curlContainerName)
+		g.Expect(err).NotTo(gomega.HaveOccurred())
+
+		g.Expect(string(metricsOutput)).Should(utiltesting.ContainMetrics(metrics))
+	}, util.Timeout).Should(gomega.Succeed())
+}
diff --git a/test/e2e/certmanager/suite_test.go b/test/e2e/certmanager/suite_test.go
index 87b023ccc..4f8a5fc4e 100644
--- a/test/e2e/certmanager/suite_test.go
+++ b/test/e2e/certmanager/suite_test.go
@@ -25,6 +25,7 @@ import (
 
 	"github.com/onsi/ginkgo/v2"
 	"github.com/onsi/gomega"
+	"k8s.io/client-go/rest"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
 	"sigs.k8s.io/kueue/apis/config/v1beta1"
@@ -35,6 +36,8 @@ var (
 	k8sClient       client.WithWatch
 	ctx             context.Context
 	defaultKueueCfg *v1beta1.Configuration
+	cfg             *rest.Config
+	restClient      *rest.RESTClient
 )
 
 func TestAPIs(t *testing.T) {
@@ -51,7 +54,8 @@ func TestAPIs(t *testing.T) {
 var _ = ginkgo.BeforeSuite(func() {
 	util.SetupLogger()
 
-	k8sClient, _ = util.CreateClientUsingCluster("")
+	k8sClient, cfg = util.CreateClientUsingCluster("")
+	restClient = util.CreateRestClient(cfg)
 	ctx = ginkgo.GinkgoT().Context()
 
 	waitForAvailableStart := time.Now()
diff --git a/test/e2e/config/certmanager/cert_metrics_manager_patch.yaml b/test/e2e/config/certmanager/cert_metrics_manager_patch.yaml
index b43585fc5..16cfdd7e6 100644
--- a/test/e2e/config/certmanager/cert_metrics_manager_patch.yaml
+++ b/test/e2e/config/certmanager/cert_metrics_manager_patch.yaml
@@ -1,14 +1,35 @@
-- op: add
-  path: /spec/template/spec/containers/0/volumeMounts/-
+- op: replace
+  path: /spec/template/spec/containers/0/volumeMounts
   value:
-    name: metrics-certs
-    mountPath: /etc/kueue/metrics/certs
-    readOnly: true
+    - name: cert
+      mountPath: /tmp/k8s-webhook-server/serving-certs
+      readOnly: true
+    - name: metrics-certs
+      mountPath: /etc/kueue/metrics/certs
+      readOnly: true
+    - name: config-volume
+      mountPath: /controller_manager_config.yaml
+      subPath: controller_manager_config.yaml
+      readOnly: true
 
-- op: add
-  path: /spec/template/spec/volumes/-
+- op: replace
+  path: /spec/template/spec/volumes
   value:
-    name: metrics-certs
-    secret:
-      secretName: metrics-server-cert
-      optional: false
+    - name: cert
+      secret:
+        secretName: webhook-server-cert
+        optional: false
+    - name: metrics-certs
+      secret:
+        secretName: metrics-server-cert
+        optional: false
+    - name: config-volume
+      configMap:
+        name: kueue-manager-config
+        items:
+          - key: controller_manager_config.yaml
+            path: controller_manager_config.yaml
+
+- op: replace
+  path: /spec/template/spec/containers/0/imagePullPolicy
+  value: IfNotPresent
diff --git a/test/e2e/config/certmanager/controller_manager_config.yaml b/test/e2e/config/certmanager/controller_manager_config.yaml
new file mode 100644
index 000000000..6334a94a2
--- /dev/null
+++ b/test/e2e/config/certmanager/controller_manager_config.yaml
@@ -0,0 +1,37 @@
+apiVersion: config.kueue.x-k8s.io/v1beta1
+kind: Configuration
+metrics:
+  enableClusterQueueResources: true
+leaderElection:
+  leaderElect: true
+controller:
+  groupKindConcurrency:
+    Job.batch: 5
+    Pod: 5
+    Workload.kueue.x-k8s.io: 5
+    LocalQueue.kueue.x-k8s.io: 1
+    ClusterQueue.kueue.x-k8s.io: 1
+    ResourceFlavor.kueue.x-k8s.io: 1
+fairSharing:
+  enable: true
+internalCertManagement:
+  enable: false
+clientConnection:
+  qps: 50
+  burst: 100
+integrations:
+  frameworks:
+  - "batch/job"
+  - "kubeflow.org/mpijob"
+  - "ray.io/rayjob"
+  - "ray.io/raycluster"
+  - "jobset.x-k8s.io/jobset"
+  - "kubeflow.org/paddlejob"
+  - "kubeflow.org/pytorchjob"
+  - "kubeflow.org/tfjob"
+  - "kubeflow.org/xgboostjob"
+  - "workload.codeflare.dev/appwrapper"
+  - "pod"
+  - "deployment"
+  - "statefulset"
+  - "leaderworkerset.x-k8s.io/leaderworkerset"
diff --git a/test/e2e/config/certmanager/kustomization.yaml b/test/e2e/config/certmanager/kustomization.yaml
index eab576812..c8764f5c5 100644
--- a/test/e2e/config/certmanager/kustomization.yaml
+++ b/test/e2e/config/certmanager/kustomization.yaml
@@ -4,8 +4,18 @@ kind: Kustomization
 namespace: kueue-system
 
 resources:
-  - ../default
   - ../../../../config/components/certmanager
+  - ../../../../config/dev
+
+# Delete the secret created by internalCertManagement workflow.
+patchesStrategicMerge:
+  - |-
+    apiVersion: v1
+    kind: Secret
+    metadata:
+      name: kueue-webhook-server-cert
+      namespace: kueue-system
+    $patch: delete
 
 transformers:
   - |
@@ -32,10 +42,32 @@ patches:
       name: validating-webhook-configuration
   - path: cert_metrics_manager_patch.yaml
     target:
+      group: apps
+      version: v1
       kind: Deployment
+      name: kueue-controller-manager
+      namespace: kueue-system
 
+configMapGenerator:
+- behavior: merge
+  files:
+  - controller_manager_config.yaml
+  name: kueue-manager-config
+  namespace: kueue-system
 
 vars:
+- name: METRICS_SERVICE_NAME
+  objref:
+    kind: Service
+    version: v1
+    name: controller-manager-metrics-service
+- name: METRICS_SERVICE_NAMESPACE
+  objref:
+    kind: Service
+    version: v1
+    name: controller-manager-metrics-service
+  fieldref:
+    fieldpath: metadata.namespace
 - name: SERVICE_NAMESPACE
   objref:
     kind: Service
@@ -48,3 +80,17 @@ vars:
     kind: Service
     version: v1
     name: webhook-service
+- name: CERTIFICATE_NAMESPACE
+  objref:
+    kind: Certificate
+    group: cert-manager.io
+    version: v1
+    name: serving-cert
+  fieldref:
+    fieldpath: metadata.namespace
+- name: CERTIFICATE_NAME
+  objref:
+    kind: Certificate
+    group: cert-manager.io
+    version: v1
+    name: serving-cert
diff --git a/test/e2e/config/certmanager/mutating_webhookcainjection_patch.yaml b/test/e2e/config/certmanager/mutating_webhookcainjection_patch.yaml
index cec283228..28b1826d4 100644
--- a/test/e2e/config/certmanager/mutating_webhookcainjection_patch.yaml
+++ b/test/e2e/config/certmanager/mutating_webhookcainjection_patch.yaml
@@ -5,4 +5,4 @@ kind: MutatingWebhookConfiguration
 metadata:
   name: mutating-webhook-configuration
   annotations:
-    cert-manager.io/inject-ca-from: $(namespace)/serving-cert
+    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
diff --git a/test/e2e/config/certmanager/validating_webhookcainjection_patch.yaml b/test/e2e/config/certmanager/validating_webhookcainjection_patch.yaml
index d29b5202c..47ef1d13c 100644
--- a/test/e2e/config/certmanager/validating_webhookcainjection_patch.yaml
+++ b/test/e2e/config/certmanager/validating_webhookcainjection_patch.yaml
@@ -5,4 +5,4 @@ kind: ValidatingWebhookConfiguration
 metadata:
   name: validating-webhook-configuration
   annotations:
-    cert-manager.io/inject-ca-from: $(namespace)/serving-cert
+    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)