Repository: kubernetes-sigs/kueue

Issue:
v1beta2: Kueue does not start after upgrade to 0.15.0-rc.0
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!

If the matter is security related, please disclose it privately via https://kubernetes.io/security/
-->


**What happened**:
When I deploy Kueue using the helm deployment at head 

```
$ git clone git@github.com:kubernetes-sigs/kueue.git
$ cd kueue/charts
$ helm install kueue kueue/ --create-namespace --namespace kueue-system
```

And deploy a localQueue, I get the following error

```
$ k get localqueue
Error from server: conversion webhook for kueue.x-k8s.io/v1beta1, Kind=LocalQueue failed: Post "https://kueue-webhook-service.kueue-system.svc:443/convert?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority
```

This doesn't happen if using the released version of the helm chart.

**What you expected to happen**:
For the following command to return without an error
```
$ k get localqueue
```

**How to reproduce it (as minimally and precisely as possible)**:
Follow the steps above.

**Anything else we need to know?**:

**Environment**:
- Kubernetes version (use `kubectl version`): 1.33
- Kueue version (use `git describe --tags --dirty --always`): f2aff8485
- Cloud provider or hardware configuration:
- OS (e.g: `cat /etc/os-release`): 
- Kernel (e.g. `uname -a`):
- Install tools:
- Others:

Discussion:
@mimowo: Hmm, seeing these errors make me wonder maybe it is fixed in this branch, can you try: https://github.com/kubernetes-sigs/kueue/pull/7318?@Edwinhr716: Tested it with that branch, still hit the same error 
```
 k get localqueue
Error from server: conversion webhook for kueue.x-k8s.io/v1beta1, Kind=LocalQueue failed: Post "https://kueue-webhook-service.kueue-system.svc:443/convert?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority
```@mimowo: I know what is going on. The templates in the branch are referencing main image by default, however for the conversion to work you need to deploy Kueue image built by this branch.

We do this when using the E2E_USE_HELM flag in Makefile (I have tested that locally and the Helm tests passed: https://github.com/kubernetes-sigs/kueue/pull/7318#discussion_r2452775544).  Or it will be easier to use once the PR is merged.@mimowo: As the PR https://github.com/kubernetes-sigs/kueue/pull/7318 is merged I believe helms should work  fine, can you test maybe again?@Edwinhr716: Now getting this error when trying to install using helm chart at head 
```
{"caller":"runtime/runtime.go:221", "error":"failed to list apiextensions.k8s.io/v1, Kind=CustomResourceDefinition: customresourcedefinitions.apiextensions.k8s.io is forbidden: User "system:serviceaccount:kueue-system:kueue-controller-manager" cannot list resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope", "level":"error", "logger":"controller-runtime.cache.UnhandledError", "msg":"Failed to watch", "reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114", "stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError
	/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:221
k8s.io/apimachinery/pkg/util/runtime.handleError
	/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:212
k8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext
	/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:198
k8s.io/client-go/tools/cache.DefaultWatchErrorHandler
	/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:205
k8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1
	/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:361
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:255
k8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:256
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233
k8s.io/client-go/tools/cache.(*Reflector).RunWithContext
	/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:359
k8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:63
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
	/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72", "ts":"2025-10-24T15:49:07.227199806Z", "type":"apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"}
```@mimowo: Weird, that PR introduce the required permissions here: https://github.com/kubernetes-sigs/kueue/pull/7318/files#diff-551fa6e9f4f56c712cbde8e7e176be653d8b14821f0d6d7d2e7fd86dedcd0fb7R75-R83
@Edwinhr716: That's odd, I've been running into different issues ever since I rebased into main. I now have to disable Multikueue in all versions of Kueue that I deploy because I keep running into 
```
could not setup multikueue indexer: setting index on workloads admission checks: indexer conflict: map[field:status.admissionChecks:{}]"
```
@mimowo: Ok, I tried today and succeeded using exactly this command, `helm install kueue kueue/ --create-namespace --namespace kueue-system`, but I had to also apply these changes: https://github.com/kubernetes-sigs/kueue/pull/7410. Please test again from that branch or main if merged.@izturn: something wrong:
## 1.  Install 0.14.0: 
```
helm install kueue ./kueue-0.14.0.tgz   --namespace kueue-system   --create-namespace   --wait --timeout 300s
```
 
 
 **Result:  Running**

## 2.  Create the CRs: 
```
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: lq
  namespace: kueue-system
spec:
  clusterQueue: cq
  stopPolicy: None

---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
  annotations:
    baize.io/description: ''
  generation: 1
  labels:
  name: cq
spec:
  flavorFungibility:
    whenCanBorrow: Borrow
    whenCanPreempt: TryNextFlavor
  namespaceSelector: {}
  preemption:
    borrowWithinCohort:
      policy: Never
    reclaimWithinCohort: Never
    withinClusterQueue: Never
  queueingStrategy: BestEffortFIFO
  resourceGroups:
    - coveredResources:
        - cpu
        - memory
      flavors:
        - name: flv
          resources:
            - name: cpu
              nominalQuota: '200'
            - name: memory
              nominalQuota: '214748364800'
  stopPolicy: None

---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
  name: flv
spec:
  nodeLabels:
  tolerations:
    - effect: NoExecute
      key: node.kubernetes.io/not-ready
      operator: Exists
      tolerationSeconds: 300
    - effect: NoExecute
      key: node.kubernetes.io/unreachable
      operator: Exists
      tolerationSeconds: 300
    - effect: NoSchedule
      key: kwok.x-k8s.io/node
      operator: Exists

---
apiVersion: kueue.x-k8s.io/v1beta1
kind: Topology
metadata:
  name: top
spec:
  levels:
    - nodeLabel: rdma-zone
    - nodeLabel: supernode

---
```
## 3. Upgrade to 0.15.0-rc.0
```
 helm upgrade --install kueue ./kueue-0.15.0-rc.0.tgz   --namespace kueue-system   --create-namespace   --wait   --timeout 300s
``` 
 **Result:  Failed**

### The log: 
```
{"level":"info","ts":"2025-11-18T10:06:58.865373665Z","logger":"setup","caller":"kueue/main.go:507","msg":"Successfully loaded configuration","config":"apiVersion: config.kueue.x-k8s.io/v1beta2\nclientConnection:\n  burst: 100\n  qps: 50\ncontroller:\n  groupKindConcurrency:\n    ClusterQueue.kueue.x-k8s.io: 1\n    Job.batch: 5\n    LocalQueue.kueue.x-k8s.io: 1\n    Pod: 5\n    ResourceFlavor.kueue.x-k8s.io: 1\n    Workload.kueue.x-k8s.io: 5\nhealth:\n  healthProbeBindAddress: :8081\nintegrations:\n  frameworks:\n  - batch/job\n  - kubeflow.org/mpijob\n  - ray.io/rayjob\n  - ray.io/raycluster\n  - jobset.x-k8s.io/jobset\n  - trainer.kubeflow.org/trainjob\n  - kubeflow.org/paddlejob\n  - kubeflow.org/pytorchjob\n  - kubeflow.org/tfjob\n  - kubeflow.org/xgboostjob\n  - kubeflow.org/jaxjob\n  - workload.codeflare.dev/appwrapper\ninternalCertManagement:\n  enable: true\n  webhookSecretName: kueue-webhook-server-cert\n  webhookServiceName: kueue-webhook-service\nkind: Configuration\nleaderElection:\n  leaderElect: true\n  leaseDuration: 15s\n  renewDeadline: 10s\n  resourceLock: leases\n  resourceName: c1f6bfd2.kueue.x-k8s.io\n  resourceNamespace: \"\"\n  retryPeriod: 2s\nmanageJobsWithoutQueueName: false\nmanagedJobsNamespaceSelector:\n  matchExpressions:\n  - key: kubernetes.io/metadata.name\n    operator: NotIn\n    values:\n    - kube-system\n    - kueue-system\nmetrics:\n  bindAddress: :8443\nmultiKueue:\n  dispatcherName: kueue.x-k8s.io/multikueue-dispatcher-all-at-once\n  gcInterval: 1m0s\n  origin: multikueue\n  workerLostTimeout: 15m0s\nnamespace: kueue-system\nwaitForPodsReady: {}\nwebhook:\n  certDir: /tmp/k8s-webhook-server/serving-certs\n  port: 9443\n"}
{"level":"info","ts":"2025-11-18T10:06:58.865846417Z","logger":"setup","caller":"kueue/main.go:153","msg":"Initializing","gitVersion":"v0.15.0-rc.0","gitCommit":"f6435a6d32fab03de5d05e2521473f5c60a7993d","buildDate":"2025-11-13T06:57:40Z"}
{"level":"Level(-2)","ts":"2025-11-18T10:06:58.866078755Z","logger":"setup","caller":"features/kube_features.go:344","msg":"Loaded feature gates","featureGates":{"AdmissionFairSharing":true,"ConfigurableResourceTransformations":true,"DynamicResourceAllocation":false,"ElasticJobsViaWorkloadSlices":false,"FlavorFungibility":true,"FlavorFungibilityImplicitPreferenceDefault":false,"HierarchicalCohorts":true,"LendingLimit":true,"LocalQueueDefaulting":true,"LocalQueueMetrics":false,"ManagedJobsNamespaceSelectorAlwaysRespected":true,"MultiKueue":true,"MultiKueueAdaptersForCustomJobs":false,"MultiKueueAllowInsecureKubeconfigs":false,"MultiKueueBatchJobWithManagedBy":true,"ObjectRetentionPolicies":true,"PartialAdmission":true,"PrioritySortingWithinCohort":true,"ReclaimablePods":true,"SanitizePodSets":true,"TASFailedNodeReplacement":true,"TASFailedNodeReplacementFailFast":true,"TASProfileLeastFreeCapacity":false,"TASProfileMixed":true,"TASReplaceNodeOnPodTermination":true,"TopologyAwareScheduling":true,"VisibilityOnDemand":true,"WorkloadRequestUseMergePatch":false}}
{"level":"Level(-2)","ts":"2025-11-18T10:06:58.866693241Z","logger":"setup","caller":"kueue/main.go:202","msg":"K8S Client","qps":50,"burst":100}
{"level":"error","ts":"2025-11-18T10:06:58.907713982Z","logger":"setup","caller":"kueue/main.go:306","msg":"Skipping admission check controller setup: Provisioning Requests not supported (Possible cause: missing or unsupported cluster-autoscaler)","error":"no matches for kind \"ProvisioningRequest\" in version \"autoscaling.x-k8s.io/v1\"","stacktrace":"main.setupIndexes\n\t/workspace/cmd/kueue/main.go:306\nmain.main\n\t/workspace/cmd/kueue/main.go:247\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:285"}
{"level":"info","ts":"2025-11-18T10:06:58.915568922Z","logger":"setup","caller":"kueue/main.go:448","msg":"Probe endpoints are configured on healthz and readyz"}
{"level":"info","ts":"2025-11-18T10:06:58.915708883Z","logger":"setup","caller":"kueue/main.go:291","msg":"Starting manager"}
{"level":"info","ts":"2025-11-18T10:06:58.91575849Z","logger":"setup","caller":"cert/cert.go:84","msg":"Waiting for certificate generation to complete"}
{"level":"info","ts":"2025-11-18T10:06:58.915969545Z","caller":"manager/server.go:83","msg":"starting server","name":"health probe","addr":"[::]:8081"}
{"level":"info","ts":"2025-11-18T10:06:58.915882143Z","logger":"controller-runtime.metrics","caller":"server/server.go:208","msg":"Starting metrics server"}
{"level":"Level(-2)","ts":"2025-11-18T10:06:58.919970473Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1.LimitRange","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"}
{"level":"Level(-2)","ts":"2025-11-18T10:06:58.920129657Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1.Job","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"}
{"level":"Level(-2)","ts":"2025-11-18T10:06:58.922045321Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1beta2.MultiKueueConfig","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"}
{"level":"Level(-2)","ts":"2025-11-18T10:06:58.922265888Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1beta2.AdmissionCheck","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"}
{"level":"Level(-2)","ts":"2025-11-18T10:06:58.923737257Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1.Node","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"}
{"level":"Level(-2)","ts":"2025-11-18T10:06:58.92453172Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1.Pod","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"}
{"level":"Level(-2)","ts":"2025-11-18T10:06:58.924691011Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1beta2.MultiKueueCluster","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"}
{"level":"Level(-2)","ts":"2025-11-18T10:06:58.925247552Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1beta2.ResourceFlavor","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"}
{"level":"Level(-2)","ts":"2025-11-18T10:06:58.927882486Z","logger":"controller-runtime.cache","caller":"cache/reflector.go:436","msg":"Caches populated","type":"*v1beta2.Workload","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114"}
{"level":"error","ts":"2025-11-18T10:06:58.935678135Z","logger":"controller-runtime.cache.UnhandledError","caller":"runtime/runtime.go:221","msg":"Failed to watch","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114","type":"*v1beta2.LocalQueue","error":"failed to list *v1beta2.LocalQueue: conversion webhook for kueue.x-k8s.io/v1beta1, Kind=LocalQueue failed: Post \"https://kueue-webhook-service.kueue-system.svc:443/convert?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:205\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:361\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:359\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72"}
{"level":"info","ts":"2025-11-18T10:06:59.025708118Z","logger":"controller-runtime.metrics","caller":"server/server.go:247","msg":"Serving metrics server","bindAddress":":8443","secure":true}
I1118 10:06:59.193884       1 serving.go:380] Generated self-signed cert (/visibility/apiserver.crt, /visibility/apiserver.key)
I1118 10:06:59.682015       1 plugins.go:157] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionPolicy.
I1118 10:06:59.689500       1 handler.go:285] Adding GroupVersion visibility.kueue.x-k8s.io v1beta2 to ResourceManager
I1118 10:06:59.689954       1 handler.go:285] Adding GroupVersion visibility.kueue.x-k8s.io v1beta1 to ResourceManager
I1118 10:06:59.703155       1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController
I1118 10:06:59.703249       1 shared_informer.go:349] "Waiting for caches to sync" controller="RequestHeaderAuthRequestController"
I1118 10:06:59.703340       1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1118 10:06:59.703357       1 shared_informer.go:349] "Waiting for caches to sync" controller="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1118 10:06:59.703383       1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1118 10:06:59.703395       1 shared_informer.go:349] "Waiting for caches to sync" controller="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1118 10:06:59.703974       1 secure_serving.go:211] Serving securely on [::]:8082
I1118 10:06:59.704086       1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/visibility/apiserver.crt::/visibility/apiserver.key"
I1118 10:06:59.704349       1 tlsconfig.go:243] "Starting DynamicServingCertificateController"
I1118 10:06:59.704483       1 apf_controller.go:377] Starting API Priority and Fairness config controller
{"level":"error","ts":"2025-11-18T10:06:59.77640903Z","logger":"controller-runtime.cache.UnhandledError","caller":"runtime/runtime.go:221","msg":"Failed to watch","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114","type":"*v1beta2.LocalQueue","error":"failed to list *v1beta2.LocalQueue: conversion webhook for kueue.x-k8s.io/v1beta1, Kind=LocalQueue failed: Post \"https://kueue-webhook-service.kueue-system.svc:443/convert?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:205\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:361\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:359\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72"}
I1118 10:06:59.803504       1 shared_informer.go:356] "Caches are synced" controller="RequestHeaderAuthRequestController"
I1118 10:06:59.803618       1 shared_informer.go:356] "Caches are synced" controller="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1118 10:06:59.803660       1 shared_informer.go:356] "Caches are synced" controller="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1118 10:06:59.806031       1 apf_controller.go:382] Running API Priority and Fairness config worker
I1118 10:06:59.806075       1 apf_controller.go:385] Running API Priority and Fairness periodic rebalancing process
{"level":"error","ts":"2025-11-18T10:07:02.324646054Z","logger":"controller-runtime.cache.UnhandledError","caller":"runtime/runtime.go:221","msg":"Failed to watch","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114","type":"*v1beta2.LocalQueue","error":"failed to list *v1beta2.LocalQueue: conversion webhook for kueue.x-k8s.io/v1beta1, Kind=LocalQueue failed: Post \"https://kueue-webhook-service.kueue-system.svc:443/convert?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:205\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:361\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:359\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72"}
{"level":"debug","ts":"2025-11-18T10:07:05.044039429Z","logger":"controller-runtime.healthz","caller":"healthz/healthz.go:60","msg":"healthz check failed","checker":"readyz","error":"certificates are not ready"}
{"level":"info","ts":"2025-11-18T10:07:05.044301784Z","logger":"controller-runtime.healthz","caller":"healthz/healthz.go:128","msg":"healthz check failed","statuses":[{}]}
{"level":"error","ts":"2025-11-18T10:07:07.468805324Z","logger":"controller-runtime.cache.UnhandledError","caller":"runtime/runtime.go:221","msg":"Failed to watch","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114","type":"*v1beta2.LocalQueue","error":"failed to list *v1beta2.LocalQueue: conversion webhook for kueue.x-k8s.io/v1beta1, Kind=LocalQueue failed: Post \"https://kueue-webhook-service.kueue-system.svc:443/convert?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:205\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:361\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:359\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72"}
{"level":"debug","ts":"2025-11-18T10:07:10.04631679Z","logger":"controller-runtime.healthz","caller":"healthz/healthz.go:60","msg":"healthz check failed","checker":"readyz","error":"certificates are not ready"}
{"level":"info","ts":"2025-11-18T10:07:10.046422247Z","logger":"controller-runtime.healthz","caller":"healthz/healthz.go:128","msg":"healthz check failed","statuses":[{}]}
{"level":"debug","ts":"2025-11-18T10:07:15.044177661Z","logger":"controller-runtime.healthz","caller":"healthz/healthz.go:60","msg":"healthz check failed","checker":"readyz","error":"certificates are not ready"}
{"level":"info","ts":"2025-11-18T10:07:15.0443267Z","logger":"controller-runtime.healthz","caller":"healthz/healthz.go:128","msg":"healthz check failed","statuses":[{}]}
{"level":"error","ts":"2025-11-18T10:07:19.314142527Z","logger":"controller-runtime.cache.UnhandledError","caller":"runtime/runtime.go:221","msg":"Failed to watch","reflector":"sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114","type":"*v1beta2.LocalQueue","error":"failed to list *v1beta2.LocalQueue: conversion webhook for kueue.x-k8s.io/v1beta1, Kind=LocalQueue failed: Post \"https://kueue-webhook-service.kueue-system.svc:443/convert?timeout=30s\": tls: failed to verify certificate: x509: certificate signed by unknown authority","stacktrace":"k8s.io/apimachinery/pkg/util/runtime.logError\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:221\nk8s.io/apimachinery/pkg/util/runtime.handleError\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:212\nk8s.io/apimachinery/pkg/util/runtime.HandleErrorWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:198\nk8s.io/client-go/tools/cache.DefaultWatchErrorHandler\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:205\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext.func1\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:361\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:255\nk8s.io/apimachinery/pkg/util/wait.BackoffUntilWithContext\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:256\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:233\nk8s.io/client-go/tools/cache.(*Reflector).RunWithContext\n\t/workspace/vendor/k8s.io/client-go/tools/cache/reflector.go:359\nk8s.io/client-go/tools/cache.(*controller).RunWithContext.(*Group).StartWithContext.func3\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:63\nk8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1\n\t/workspace/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:72"}

...............
```

@mimowo pls help me to fix it @mimowo: can you describe the `kubectl describe deploy -nkueue-system`@izturn: ```
Name:                   kueue-controller-manager
Namespace:              kueue-system
CreationTimestamp:      Tue, 18 Nov 2025 17:55:49 +0800
Labels:                 app.kubernetes.io/component=controller
                        app.kubernetes.io/instance=kueue
                        app.kubernetes.io/managed-by=Helm
                        app.kubernetes.io/name=kueue
                        app.kubernetes.io/version=v0.15.0-rc.0
                        control-plane=controller-manager
                        helm.sh/chart=kueue-0.15.0-rc.0
                        k8slens-edit-resource-version=v1
Annotations:            deployment.kubernetes.io/revision: 4
                        meta.helm.sh/release-name: kueue
                        meta.helm.sh/release-namespace: kueue-system
Selector:               app.kubernetes.io/instance=kueue,app.kubernetes.io/name=kueue,control-plane=controller-manager     
Replicas:               1 desired | 1 updated | 2 total | 1 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app.kubernetes.io/instance=kueue
                    app.kubernetes.io/name=kueue
                    control-plane=controller-manager
  Annotations:      charts.kueue.x-k8s.io/config-checksum: 95acca0b796d0bfd293b271cea2ee7ca622e18b683a05d13258541a6421fc67e
                    kubectl.kubernetes.io/default-container: manager
  Service Account:  kueue-controller-manager
  Containers:
   manager:
    Image:       us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue:v0.15.0-rc.0
    Ports:       8082/TCP, 9443/TCP, 8443/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Command:
      /manager
    Args:
      --config=/controller_manager_config.yaml
      --zap-log-level=2
    Limits:
      cpu:     2
      memory:  512Mi
    Requests:
      cpu:        500m
      memory:     512Mi
    Liveness:     http-get http://:8081/healthz delay=15s timeout=1s period=15s #success=1 #failure=3
    Readiness:    http-get http://:8081/readyz delay=5s timeout=1s period=5s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /controller_manager_config.yaml from manager-config (rw,path="controller_manager_config.yaml")
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /visibility from visibility (rw)
  Volumes:
   cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kueue-webhook-server-cert
    Optional:    false
   manager-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kueue-manager-config
    Optional:  false
   visibility:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  kueue-controller-manager-7bd58d49f7 (1/1 replicas created)
NewReplicaSet:   kueue-controller-manager-6b7d8b7d86 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  42m   deployment-controller  Scaled up replica set kueue-controller-manager-8569448d9c from 0 to 1
  Normal  ScalingReplicaSet  41m   deployment-controller  Scaled up replica set kueue-controller-manager-59b769dc5c from 0 to 1
  Normal  ScalingReplicaSet  39m   deployment-controller  Scaled down replica set kueue-controller-manager-8569448d9c from 1 to 0
  Normal  ScalingReplicaSet  39m   deployment-controller  Scaled up replica set kueue-controller-manager-7bd58d49f7 from 0 to 1
  Normal  ScalingReplicaSet  39m   deployment-controller  Scaled down replica set kueue-controller-manager-59b769dc5c from 1 to 0
  Normal  ScalingReplicaSet  31m   deployment-controller  Scaled up replica set kueue-controller-manager-6b7d8b7d86 from 0 to 1
```@mimowo: hm, it seems you have one Pod that is unavailable, maybe do:

`kubectl describe pods -nkueue-system`@izturn: ```
Name:             kueue-controller-manager-556f858896-r6k55
Namespace:        kueue-system
Priority:         0
Service Account:  kueue-controller-manager
Node:             bms4-control-plane/172.18.0.3
Start Time:       Wed, 19 Nov 2025 09:46:50 +0800
Labels:           app.kubernetes.io/instance=kueue
                  app.kubernetes.io/name=kueue
                  control-plane=controller-manager
                  pod-template-hash=556f858896
Annotations:      charts.kueue.x-k8s.io/config-checksum: 95acca0b796d0bfd293b271cea2ee7ca622e18b683a05d13258541a6421fc67e
                  kubectl.kubernetes.io/default-container: manager
                  kubectl.kubernetes.io/restartedAt: 2025-11-19T01:46:49.845Z
Status:           Running
IP:               10.244.0.30
IPs:
  IP:           10.244.0.30
Controlled By:  ReplicaSet/kueue-controller-manager-556f858896
Containers:
  manager:
    Container ID:  containerd://a0889ea2d2b1a79d2612137cfaadacc5297a21ed94db32d3ad20c127aedfcfad
    Image:         us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue:v0.15.0-rc.0
    Image ID:      docker.io/library/import-2025-11-18@sha256:ab5155823807904394e936e62f0ac6c1df09a69a47350b67cb1140e0bc80a1f4
    Ports:         8082/TCP, 9443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      /manager
    Args:
      --config=/controller_manager_config.yaml
      --zap-log-level=2
    State:          Running
      Started:      Wed, 19 Nov 2025 09:46:51 +0800
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  512Mi
    Requests:
      cpu:        500m
      memory:     512Mi
    Liveness:     http-get http://:8081/healthz delay=15s timeout=1s period=15s #success=1 #failure=3
    Readiness:    http-get http://:8081/readyz delay=5s timeout=1s period=5s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /controller_manager_config.yaml from manager-config (rw,path="controller_manager_config.yaml")
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-47sjj (ro)
      /visibility from visibility (rw)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kueue-webhook-server-cert
    Optional:    false
  manager-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kueue-manager-config
    Optional:  false
  visibility:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-47sjj:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  5m49s                 default-scheduler  Successfully assigned kueue-system/kueue-controller-manager-556f858896-r6k55 to bms4-control-plane
  Normal   Pulled     5m49s                 kubelet            Container image "us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue:v0.15.0-rc.0" already present on machine
  Normal   Created    5m49s                 kubelet            Created container: manager
  Normal   Started    5m48s                 kubelet            Started container manager
  Warning  Unhealthy  41s (x63 over 5m42s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500


Name:             kueue-controller-manager-7bd58d49f7-92wqh
Namespace:        kueue-system
Priority:         0
Service Account:  kueue-controller-manager
Node:             bms4-control-plane/172.18.0.3
Start Time:       Wed, 19 Nov 2025 09:47:49 +0800
Labels:           app.kubernetes.io/instance=kueue
                  app.kubernetes.io/name=kueue
                  control-plane=controller-manager
                  pod-template-hash=7bd58d49f7
Annotations:      charts.kueue.x-k8s.io/config-checksum: fb8e3eb0c9ff5f1b55564110b5d8387361f3e4b8bc2a7b32de4637a1f66ac92d
                  kubectl.kubernetes.io/default-container: manager
Status:           Running
IP:               10.244.0.31
IPs:
  IP:           10.244.0.31
Controlled By:  ReplicaSet/kueue-controller-manager-7bd58d49f7
Containers:
  manager:
    Container ID:  containerd://026a5c76b0c9294ca8cf0bfa0b18f3b4f1f5f9e09296c763157d0e568034e051
    Image:         us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue:v0.14.0
    Image ID:      docker.io/library/import-2025-11-18@sha256:ef9077ccda662210f93820b96c8ce5bdf2bd30ab879d6d86f987389f786eefb9
    Ports:         8082/TCP, 9443/TCP, 8443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Command:
      /manager
    Args:
      --config=/controller_manager_config.yaml
      --zap-log-level=2
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Wed, 19 Nov 2025 09:50:46 +0800
      Finished:     Wed, 19 Nov 2025 09:50:46 +0800
    Ready:          False
    Restart Count:  5
    Limits:
      cpu:     2
      memory:  512Mi
    Requests:
      cpu:        500m
      memory:     512Mi
    Liveness:     http-get http://:8081/healthz delay=15s timeout=1s period=15s #success=1 #failure=3
    Readiness:    http-get http://:8081/readyz delay=5s timeout=1s period=5s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /controller_manager_config.yaml from manager-config (rw,path="controller_manager_config.yaml")
      /tmp/k8s-webhook-server/serving-certs from cert (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2mb4m (ro)
      /visibility from visibility (rw)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  kueue-webhook-server-cert
    Optional:    false
  manager-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kueue-manager-config
    Optional:  false
  visibility:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-2mb4m:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  4m50s                 default-scheduler  Successfully assigned kueue-system/kueue-controller-manager-7bd58d49f7-92wqh to bms4-control-plane
  Normal   Pulled     114s (x6 over 4m49s)  kubelet            Container image "us-central1-docker.pkg.dev/k8s-staging-images/kueue/kueue:v0.14.0" already present on machine
  Normal   Created    114s (x6 over 4m49s)  kubelet            Created container: manager
  Normal   Started    113s (x6 over 4m49s)  kubelet            Started container manager
  Warning  BackOff    69s (x25 over 4m47s)  kubelet            Back-off restarting failed container manager in pod kueue-controller-manager-7bd58d49f7-92wqh_kueue-system(22f3aad9-ed17-47d4-bdfa-146d7178344d)
```@izturn: I retried this on a different cluster and encountered the same error.
@mimowo, any suggestions?@mimowo: I will be looking into this today, maybe there is a bug.@mimowo: I tested locally by installing 0.14.4 first, and upgrading to 0.15.0-rc.0, and it worked fine, I executed:

Install old Kueue:
```sh
❯ helm install kueue oci://registry.k8s.io/kueue/charts/kueue \
  --version=0.14.4 \
  --namespace  kueue-system \
  --create-namespace \
  --wait --timeout 300s
```
download the RC:
```
curl -Lv https://github.com/kubernetes-sigs/kueue/releases/tag/v0.15.0-rc.0.tar.gz -ohelm.tar.gz
```

upgrade
```sh
❯ helm upgrade --install kueue ./helm.tar.gz \
  --namespace kueue-system \
  --create-namespace \
  --wait --timeout 300s
```
I can create resources:
```sh
❯ k create -f cluster.yaml
resourceflavor.kueue.x-k8s.io/default-flavor created
clusterqueue.kueue.x-k8s.io/cluster-queue created
localqueue.kueue.x-k8s.io/user-queue created
```
```sh
❯ k create -f job.yaml
job.batch/sample-jobbgzkq created
```
That was tested on Kind. I will later check on GKE. can you maybe also check on Kind? What is the cluster host you are testing it on?
@sohankunkerkar: I suspect what’s happening here is that the upgrade only works in @mimowo's workflow because the conversion webhooks are not called until the new certs settle. When LocalQueues already exist before the upgrade, the RC rotates kueue-webhook-server-cert but never updates the CRD conversion caBundle. That leaves the API server unable to reach the conversion webhook. The controller cache then loops forever and the RC pod never becomes Ready, which causes Helm’s wait to time out.@mimowo: Indeed, I replicated the issue on Kind by just creating an instance of LQ before upgrade. Let's think how to solve, and research how other projects solve this, because it shouldn't be a problem unique to Kueue.@sohankunkerkar: I will assign this issue to me if that's ok with you.@sohankunkerkar: /assign @sohankunkerkar @izturn: btw, This issue doesn’t occur only with `helm upgrade`; it also happens when using `kubectl apply`. @sohankunkerkar @mimowo: Yes, let me retitle
/retitle v1beta2: Kueue does not start after upgrade to 0.15.0-rc.0

@mimowo: @sohankunkerkar have you been able to confirm the hypothesis from https://github.com/kubernetes-sigs/kueue/issues/7344#issuecomment-3554091666. I'm wondering what exactly is the difference why upgrade works if LQ is not created vs. if it is created. Maybe we just need to make sure something is cleaned on startup as if the system was "clean". 

Solution patch:
diff --git a/Makefile-test.mk b/Makefile-test.mk
index d991bde62..28e7d7a8a 100644
--- a/Makefile-test.mk
+++ b/Makefile-test.mk
@@ -46,6 +46,7 @@ E2E_K8S_FULL_VERSION := $(or $(E2E_K8S_FULL_VERSION),$(E2E_K8S_VERSION).0)
 E2E_KIND_VERSION ?= kindest/node:v$(E2E_K8S_FULL_VERSION)
 E2E_RUN_ONLY_ENV ?= false
 E2E_USE_HELM ?= false
+KUEUE_UPGRADE_FROM_VERSION ?= v0.14.4
 
 # For local testing, we should allow user to use different kind cluster name
 # Default will delete default kind cluster
@@ -129,6 +130,12 @@ test-e2e-customconfigs-helm: test-e2e-customconfigs
 .PHONY: test-e2e-certmanager
 test-e2e-certmanager: setup-e2e-env run-test-e2e-certmanager-$(E2E_KIND_VERSION:kindest/node:v%=%)
 
+.PHONY: test-e2e-upgrade
+test-e2e-upgrade: setup-e2e-env run-test-e2e-upgrade-$(E2E_KIND_VERSION:kindest/node:v%=%)
+
+.PHONY: test-e2e-certmanager-upgrade
+test-e2e-certmanager-upgrade: setup-e2e-env run-test-e2e-certmanager-upgrade-$(E2E_KIND_VERSION:kindest/node:v%=%)
+
 run-test-e2e-singlecluster-%: K8S_VERSION = $(@:run-test-e2e-singlecluster-%=%)
 run-test-e2e-singlecluster-%:
 	@echo Running e2e for k8s ${K8S_VERSION}
@@ -202,6 +209,29 @@ run-test-e2e-certmanager-%:
 		E2E_USE_HELM=$(E2E_USE_HELM) \
 		./hack/e2e-test.sh
 
+run-test-e2e-upgrade-%: K8S_VERSION = $(@:run-test-e2e-upgrade-%=%)
+run-test-e2e-upgrade-%:
+	@echo Running upgrade e2e for k8s ${K8S_VERSION}
+	E2E_KIND_VERSION="kindest/node:v$(K8S_VERSION)" KIND_CLUSTER_NAME=$(KIND_CLUSTER_NAME) CREATE_KIND_CLUSTER=$(CREATE_KIND_CLUSTER) \
+		ARTIFACTS="$(ARTIFACTS)/$@" IMAGE_TAG=$(IMAGE_TAG) GINKGO_ARGS="$(GINKGO_ARGS)" \
+		KIND_CLUSTER_FILE="kind-cluster.yaml" E2E_TARGET_FOLDER="upgrade" \
+		KUEUE_UPGRADE_FROM_VERSION=$(KUEUE_UPGRADE_FROM_VERSION) \
+		TEST_LOG_LEVEL=$(TEST_LOG_LEVEL) \
+		E2E_RUN_ONLY_ENV=$(E2E_RUN_ONLY_ENV) \
+		./hack/e2e-test.sh
+
+run-test-e2e-certmanager-upgrade-%: K8S_VERSION = $(@:run-test-e2e-certmanager-upgrade-%=%)
+run-test-e2e-certmanager-upgrade-%:
+	@echo Running upgrade e2e for k8s ${K8S_VERSION}
+	E2E_KIND_VERSION="kindest/node:v$(K8S_VERSION)" KIND_CLUSTER_NAME=$(KIND_CLUSTER_NAME) CREATE_KIND_CLUSTER=$(CREATE_KIND_CLUSTER) \
+		ARTIFACTS="$(ARTIFACTS)/$@" IMAGE_TAG=$(IMAGE_TAG) GINKGO_ARGS="$(GINKGO_ARGS)" \
+		KIND_CLUSTER_FILE="kind-cluster.yaml" E2E_TARGET_FOLDER="upgrade" \
+		KUEUE_UPGRADE_FROM_VERSION=$(KUEUE_UPGRADE_FROM_VERSION) \
+		CERTMANAGER_VERSION=$(CERTMANAGER_VERSION) \
+		TEST_LOG_LEVEL=$(TEST_LOG_LEVEL) \
+		E2E_RUN_ONLY_ENV=$(E2E_RUN_ONLY_ENV) \
+		./hack/e2e-test.sh
+
 SCALABILITY_RUNNER := $(BIN_DIR)/performance-scheduler-runner
 .PHONY: performance-scheduler-runner
 performance-scheduler-runner:
diff --git a/cmd/kueue/main.go b/cmd/kueue/main.go
index 224d845c7..b9964289a 100644
--- a/cmd/kueue/main.go
+++ b/cmd/kueue/main.go
@@ -62,6 +62,7 @@ import (
 	"sigs.k8s.io/kueue/pkg/controller/admissionchecks/provisioning"
 	"sigs.k8s.io/kueue/pkg/controller/core"
 	"sigs.k8s.io/kueue/pkg/controller/core/indexer"
+	"sigs.k8s.io/kueue/pkg/controller/failurerecovery"
 	"sigs.k8s.io/kueue/pkg/controller/jobframework"
 	"sigs.k8s.io/kueue/pkg/controller/tas"
 	tasindexer "sigs.k8s.io/kueue/pkg/controller/tas/indexer"
@@ -204,6 +205,16 @@ func main() {
 		kubeConfig.RateLimiter = flowcontrol.NewTokenBucketRateLimiter(*cfg.ClientConnection.QPS, int(*cfg.ClientConnection.Burst))
 	}
 	setupLog.V(2).Info("K8S Client", "qps", *cfg.ClientConnection.QPS, "burst", *cfg.ClientConnection.Burst)
+
+	// Bootstrap certificates before creating the main manager
+	// This ensures certs are ready and CA bundles are injected into conversion CRDs
+	if cfg.InternalCertManagement != nil && *cfg.InternalCertManagement.Enable {
+		if err := cert.BootstrapCerts(kubeConfig, cfg); err != nil {
+			setupLog.Error(err, "Unable to bootstrap certificates")
+			os.Exit(1)
+		}
+	}
+
 	mgr, err := ctrl.NewManager(kubeConfig, options)
 	if err != nil {
 		setupLog.Error(err, "Unable to start manager")
@@ -265,15 +276,15 @@ func main() {
 		os.Exit(1)
 	}
 
-	// Cert won't be ready until manager starts, so start a goroutine here which
-	// will block until the cert is ready before setting up the controllers.
-	// Controllers who register after manager starts will start directly.
-	go func() {
-		if err := setupControllers(ctx, mgr, cCache, queues, certsReady, &cfg, serverVersionFetcher); err != nil {
-			setupLog.Error(err, "Unable to setup controllers")
-			os.Exit(1)
-		}
-	}()
+	if err := setupControllers(ctx, mgr, cCache, queues, &cfg, serverVersionFetcher); err != nil {
+		setupLog.Error(err, "Unable to setup controllers")
+		os.Exit(1)
+	}
+
+	if failedWebhook, err := webhooks.Setup(mgr); err != nil {
+		setupLog.Error(err, "Unable to create webhook", "webhook", failedWebhook)
+		os.Exit(1)
+	}
 
 	go queues.CleanUpOnContext(ctx)
 	go cCache.CleanUpOnContext(ctx)
@@ -330,14 +341,15 @@ func setupIndexes(ctx context.Context, mgr ctrl.Manager, cfg *configapi.Configur
 	return jobframework.SetupIndexes(ctx, mgr.GetFieldIndexer(), opts...)
 }
 
-func setupControllers(ctx context.Context, mgr ctrl.Manager, cCache *schdcache.Cache, queues *qcache.Manager, certsReady chan struct{}, cfg *configapi.Configuration, serverVersionFetcher *kubeversion.ServerVersionFetcher) error {
-	// The controllers won't work until the webhooks are operating, and the webhook won't work until the
-	// certs are all in place.
-	cert.WaitForCertsReady(setupLog, certsReady)
-
+func setupControllers(ctx context.Context, mgr ctrl.Manager, cCache *schdcache.Cache, queues *qcache.Manager, cfg *configapi.Configuration, serverVersionFetcher *kubeversion.ServerVersionFetcher) error {
 	if failedCtrl, err := core.SetupControllers(mgr, queues, cCache, cfg); err != nil {
 		return fmt.Errorf("unable to create controller %s: %w", failedCtrl, err)
 	}
+	if features.Enabled(features.FailureRecoveryPolicy) {
+		if failedCtrlName, err := failurerecovery.SetupControllers(mgr, cfg); err != nil {
+			return fmt.Errorf("could not setup FailureRecovery controller %s: %w", failedCtrlName, err)
+		}
+	}
 
 	// setup provision admission check controller
 	if err := provisioning.ServerSupportsProvisioningRequest(mgr); err != nil {
@@ -395,10 +407,6 @@ func setupControllers(ctx context.Context, mgr ctrl.Manager, cCache *schdcache.C
 		}
 	}
 
-	if failedWebhook, err := webhooks.Setup(mgr); err != nil {
-		return fmt.Errorf("unable to create webhook %s: %w", failedWebhook, err)
-	}
-
 	opts := []jobframework.Option{
 		jobframework.WithManageJobsWithoutQueueName(cfg.ManageJobsWithoutQueueName),
 		jobframework.WithWaitForPodsReady(cfg.WaitForPodsReady),
diff --git a/config/components/crd/patches/cainjection_in_localqueues.yaml b/config/components/crd/patches/cainjection_in_localqueues.yaml
new file mode 100644
index 000000000..a0a85c03e
--- /dev/null
+++ b/config/components/crd/patches/cainjection_in_localqueues.yaml
@@ -0,0 +1,7 @@
+# The following patch adds a directive for certmanager to inject CA into the CRD
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  annotations:
+    cert-manager.io/inject-ca-from: $(CERTIFICATE_NAMESPACE)/$(CERTIFICATE_NAME)
+  name: localqueues.kueue.x-k8s.io
diff --git a/hack/e2e-common.sh b/hack/e2e-common.sh
index 3cc8706e9..0028219fc 100644
--- a/hack/e2e-common.sh
+++ b/hack/e2e-common.sh
@@ -72,6 +72,10 @@ if [[ -n "${CERTMANAGER_VERSION:-}" ]]; then
     export CERTMANAGER_MANIFEST="https://github.com/cert-manager/cert-manager/releases/download/${CERTMANAGER_VERSION}/cert-manager.yaml"
 fi
 
+if [[ -n "${KUEUE_UPGRADE_FROM_VERSION:-}" ]]; then
+    export KUEUE_OLD_VERSION_MANIFEST="https://github.com/kubernetes-sigs/kueue/releases/download/${KUEUE_UPGRADE_FROM_VERSION}/manifests.yaml"
+fi
+
 # agnhost image to use for testing.
 E2E_TEST_AGNHOST_IMAGE_OLD_WITH_SHA=registry.k8s.io/e2e-test-images/agnhost:2.52@sha256:b173c7d0ffe3d805d49f4dfe48375169b7b8d2e1feb81783efd61eb9d08042e6
 export E2E_TEST_AGNHOST_IMAGE_OLD=${E2E_TEST_AGNHOST_IMAGE_OLD_WITH_SHA%%@*}
@@ -141,6 +145,10 @@ function prepare_docker_images {
     if [[ -n ${LEADERWORKERSET_VERSION:-} ]]; then
         docker pull "${LEADERWORKERSET_IMAGE}"
     fi
+    if [[ -n ${KUEUE_UPGRADE_FROM_VERSION:-} ]]; then
+        local current_image="${IMAGE_TAG%:*}:${KUEUE_UPGRADE_FROM_VERSION}"
+        docker pull "${current_image}"
+    fi
 }
 
 # $1 cluster
@@ -148,6 +156,10 @@ function cluster_kind_load {
     cluster_kind_load_image "$1" "${E2E_TEST_AGNHOST_IMAGE_OLD}"
     cluster_kind_load_image "$1" "${E2E_TEST_AGNHOST_IMAGE}"
     cluster_kind_load_image "$1" "$IMAGE_TAG"
+    if [[ -n ${KUEUE_UPGRADE_FROM_VERSION:-} ]]; then
+        local old_image="${IMAGE_TAG%:*}:${KUEUE_UPGRADE_FROM_VERSION}"
+        cluster_kind_load_image "$1" "${old_image}"
+    fi
 }
 
 # $1 cluster
@@ -218,6 +230,7 @@ function deploy_with_certmanager() {
         cd "${ROOT_DIR}/config/components/crd" || exit
         $KUSTOMIZE edit add patch --path "patches/cainjection_in_clusterqueues.yaml"
         $KUSTOMIZE edit add patch --path "patches/cainjection_in_cohorts.yaml"
+        $KUSTOMIZE edit add patch --path "patches/cainjection_in_localqueues.yaml"
         $KUSTOMIZE edit add patch --path "patches/cainjection_in_resourceflavors.yaml"
         $KUSTOMIZE edit add patch --path "patches/cainjection_in_workloads.yaml"
     )
@@ -240,6 +253,12 @@ function deploy_with_certmanager() {
 
 # $1 kubeconfig
 function cluster_kueue_deploy {
+    # Handle upgrade test mode
+    if [[ -n ${KUEUE_UPGRADE_FROM_VERSION:-} ]]; then
+        upgrade_test_flow "$1"
+        return
+    fi
+    # Normal deployment flows
     if [[ -n ${CERTMANAGER_VERSION:-} ]]; then
         kubectl -n cert-manager wait --for condition=ready pod \
             -l app.kubernetes.io/instance=cert-manager \
@@ -274,8 +293,7 @@ function helm_install {
 function build_and_apply_kueue_manifests {
     local build_output
     build_output=$($KUSTOMIZE build "$2")
-    # shellcheck disable=SC2001 # bash parameter substitution does not work on macOS
-    build_output=$(echo "$build_output" | sed "s/kueue-system/$KUEUE_NAMESPACE/g")
+    build_output=${build_output//kueue-system/$KUEUE_NAMESPACE}
     echo "$build_output" | kubectl apply --kubeconfig="$1" --server-side -f -
 }
 
@@ -405,3 +423,79 @@ EOF
         --user="$kind_name"
     fi
 }
+
+# Upgrade test flow: install old version, create resources, upgrade to current
+# $1 kubeconfig
+function upgrade_test_flow {
+    local old_version="${KUEUE_UPGRADE_FROM_VERSION}"
+    local old_image="${IMAGE_TAG%:*}:${old_version}"
+
+    echo "Upgrade Test: $old_version -> current"
+    echo "Old image: $old_image"
+    echo "New image: $IMAGE_TAG"
+    
+    # Step 1: Install old version
+    echo "Installing $old_version..."
+    echo "  Manifest URL: ${KUEUE_OLD_VERSION_MANIFEST}"
+    echo "  Downloading and modifying manifests..."
+    
+    # Download and modify manifests inline
+    curl -sL "${KUEUE_OLD_VERSION_MANIFEST}" | \
+      sed "s|registry.k8s.io/kueue/kueue:${old_version}|${old_image}|g" | \
+      sed 's|imagePullPolicy: Always|imagePullPolicy: IfNotPresent|g' | \
+      kubectl apply --server-side -f -
+
+    kubectl wait --for=condition=available --timeout=180s deployment/kueue-controller-manager -n kueue-system
+    echo "✓ $old_version ready"
+    
+    # Step 2: Create test resources
+    echo "Creating test resources..."
+    kubectl apply --kubeconfig="$1" -f - <<EOF
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: ResourceFlavor
+metadata:
+  name: upgrade-test-flavor
+---
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: ClusterQueue
+metadata:
+  name: upgrade-test-cq
+spec:
+  namespaceSelector: {}
+  resourceGroups:
+  - coveredResources: ["cpu", "memory"]
+    flavors:
+    - name: upgrade-test-flavor
+      resources:
+      - name: "cpu"
+        nominalQuota: 10
+      - name: "memory"
+        nominalQuota: 10Gi
+---
+apiVersion: kueue.x-k8s.io/v1beta1
+kind: LocalQueue
+metadata:
+  name: upgrade-test-lq
+  namespace: default
+spec:
+  clusterQueue: upgrade-test-cq
+EOF
+    echo "✓ Resources created"
+    
+    # Step 3: Upgrade to current (rolling update)
+    echo "Upgrading to current..."
+    
+    # Apply upgrade - rolling update will replace pods
+    (
+        set_managers_image
+        trap restore_managers_image EXIT
+        
+        local build_output
+        build_output=$($KUSTOMIZE build "${ROOT_DIR}/test/e2e/config/default")
+        build_output=${build_output//kueue-system/$KUEUE_NAMESPACE}
+        echo "$build_output" | kubectl apply --kubeconfig="$1" --server-side --force-conflicts -f -
+    )
+    
+    echo "✓ Upgrade complete (rolling update in progress)"
+    echo "========================================="
+}
diff --git a/pkg/controller/core/core.go b/pkg/controller/core/core.go
index 4569d62bd..ee0598cd4 100644
--- a/pkg/controller/core/core.go
+++ b/pkg/controller/core/core.go
@@ -25,7 +25,6 @@ import (
 	qcache "sigs.k8s.io/kueue/pkg/cache/queue"
 	schdcache "sigs.k8s.io/kueue/pkg/cache/scheduler"
 	"sigs.k8s.io/kueue/pkg/constants"
-	"sigs.k8s.io/kueue/pkg/controller/failurerecovery"
 	"sigs.k8s.io/kueue/pkg/features"
 	"sigs.k8s.io/kueue/pkg/scheduler/preemption/fairsharing"
 	"sigs.k8s.io/kueue/pkg/util/waitforpodsready"
@@ -63,20 +62,6 @@ func SetupControllers(mgr ctrl.Manager, qManager *qcache.Manager, cc *schdcache.
 		watchers = append(watchers, cohortRec)
 	}
 
-	if features.Enabled(features.FailureRecoveryPolicy) {
-		tpRec, err := failurerecovery.NewTerminatingPodReconciler(
-			mgr.GetClient(),
-			mgr.GetEventRecorderFor(constants.PodTerminationControllerName),
-		)
-		if err != nil {
-			return "FailureRecoveryPolicy", err
-		}
-
-		if err := tpRec.SetupWithManager(mgr); err != nil {
-			return "FailureRecoveryPolicy", err
-		}
-	}
-
 	cqRec := NewClusterQueueReconciler(
 		mgr.GetClient(),
 		qManager,
diff --git a/pkg/controller/failurerecovery/failurerecovery.go b/pkg/controller/failurerecovery/failurerecovery.go
new file mode 100644
index 000000000..75670295d
--- /dev/null
+++ b/pkg/controller/failurerecovery/failurerecovery.go
@@ -0,0 +1,32 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package failurerecovery
+
+import (
+	"sigs.k8s.io/controller-runtime/pkg/manager"
+
+	configapi "sigs.k8s.io/kueue/apis/config/v1beta2"
+	"sigs.k8s.io/kueue/pkg/constants"
+)
+
+func SetupControllers(mgr manager.Manager, cfg *configapi.Configuration) (string, error) {
+	tpRec := NewTerminatingPodReconciler(
+		mgr.GetClient(),
+		mgr.GetEventRecorderFor(constants.PodTerminationControllerName),
+	)
+	return tpRec.SetupWithManager(mgr, cfg)
+}
diff --git a/pkg/controller/failurerecovery/pod_termination_controller.go b/pkg/controller/failurerecovery/pod_termination_controller.go
index 8ece90b43..467221aad 100644
--- a/pkg/controller/failurerecovery/pod_termination_controller.go
+++ b/pkg/controller/failurerecovery/pod_termination_controller.go
@@ -28,11 +28,15 @@ import (
 	"k8s.io/utils/ptr"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/controller"
 	"sigs.k8s.io/controller-runtime/pkg/event"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
 	"sigs.k8s.io/controller-runtime/pkg/source"
 
+	configapi "sigs.k8s.io/kueue/apis/config/v1beta2"
+	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2"
 	"sigs.k8s.io/kueue/pkg/controller/constants"
+	"sigs.k8s.io/kueue/pkg/controller/core"
 	utilclient "sigs.k8s.io/kueue/pkg/util/client"
 	utilpod "sigs.k8s.io/kueue/pkg/util/pod"
 	utiltaints "sigs.k8s.io/kueue/pkg/util/taints"
@@ -84,7 +88,7 @@ func NewTerminatingPodReconciler(
 	client client.Client,
 	recorder record.EventRecorder,
 	opts ...TerminatingPodReconcilerOption,
-) (*TerminatingPodReconciler, error) {
+) *TerminatingPodReconciler {
 	options := defaultOptions
 	for _, opt := range opts {
 		opt(&options)
@@ -95,7 +99,7 @@ func NewTerminatingPodReconciler(
 		clock:                          options.clock,
 		forcefulTerminationGracePeriod: options.forcefulTerminationGracePeriod,
 		recorder:                       recorder,
-	}, nil
+	}
 }
 
 func (r *TerminatingPodReconciler) Generic(event.TypedGenericEvent[*corev1.Pod]) bool {
@@ -197,8 +201,8 @@ func podEligibleForTermination(p *corev1.Pod) bool {
 	return true
 }
 
-func (r *TerminatingPodReconciler) SetupWithManager(mgr ctrl.Manager) error {
-	return ctrl.NewControllerManagedBy(mgr).
+func (r *TerminatingPodReconciler) SetupWithManager(mgr ctrl.Manager, cfg *configapi.Configuration) (string, error) {
+	return "failure-recovery-pod-termination-controller", ctrl.NewControllerManagedBy(mgr).
 		Named("pod_termination_controller").
 		WatchesRawSource(source.TypedKind(
 			mgr.GetCache(),
@@ -206,5 +210,9 @@ func (r *TerminatingPodReconciler) SetupWithManager(mgr ctrl.Manager) error {
 			&handler.TypedEnqueueRequestForObject[*corev1.Pod]{},
 			r,
 		)).
-		Complete(r)
+		WithOptions(controller.Options{
+			NeedLeaderElection:      ptr.To(false),
+			MaxConcurrentReconciles: mgr.GetControllerOptions().GroupKindConcurrency[kueue.GroupVersion.WithKind("Pod").GroupKind().String()],
+		}).
+		Complete(core.WithLeadingManager(mgr, r, &corev1.Pod{}, cfg))
 }
diff --git a/pkg/util/cert/cert.go b/pkg/util/cert/cert.go
index fca3b4703..813be4c34 100644
--- a/pkg/util/cert/cert.go
+++ b/pkg/util/cert/cert.go
@@ -17,13 +17,16 @@ limitations under the License.
 package cert
 
 import (
+	"context"
 	"fmt"
 	"strings"
 
-	"github.com/go-logr/logr"
 	cert "github.com/open-policy-agent/cert-controller/pkg/rotator"
 	"k8s.io/apimachinery/pkg/types"
+	"k8s.io/client-go/rest"
 	ctrl "sigs.k8s.io/controller-runtime"
+	"sigs.k8s.io/controller-runtime/pkg/metrics/filters"
+	metricsserver "sigs.k8s.io/controller-runtime/pkg/metrics/server"
 
 	config "sigs.k8s.io/kueue/apis/config/v1beta2"
 )
@@ -38,17 +41,92 @@ const (
 // +kubebuilder:rbac:groups="admissionregistration.k8s.io",resources=validatingwebhookconfigurations,verbs=get;list;watch;update
 // +kubebuilder:rbac:groups="apiextensions.k8s.io",resources=customresourcedefinitions,verbs=get;list;watch;update
 
-// ManageCerts creates all certs for webhooks. This function is called from main.go.
+// BootstrapCerts creates a minimal manager to generate certificates and inject CA bundles.
+// This function blocks until certificates are ready and CA bundles are injected into CRDs.
+func BootstrapCerts(kubeConfig *rest.Config, cfg config.Configuration) error {
+	log := ctrl.Log.WithName("cert-bootstrap")
+
+	// Create a minimal bootstrap manager with leader election.
+	log.Info("Creating bootstrap manager for certificate generation")
+	bootstrapMgr, err := ctrl.NewManager(kubeConfig, ctrl.Options{
+		Metrics: metricsserver.Options{
+			BindAddress:    "0",
+			FilterProvider: filters.WithAuthenticationAndAuthorization,
+		},
+	})
+	if err != nil {
+		return fmt.Errorf("unable to create bootstrap manager: %w", err)
+	}
+
+	certsReady := make(chan struct{})
+
+	// Add cert rotator to bootstrap manager using shared config.
+	rotatorConfig := buildCertRotatorConfig(cfg, "cert-rotator-bootstrap", certsReady)
+	err = cert.AddRotator(bootstrapMgr, rotatorConfig)
+	if err != nil {
+		return fmt.Errorf("unable to add cert rotator to bootstrap manager: %w", err)
+	}
+
+	bootstrapCtx, bootstrapCancel := context.WithCancel(context.Background())
+	defer bootstrapCancel()
+
+	managerStopped := make(chan struct{})
+	go func() {
+		log.Info("Starting bootstrap manager")
+		if err := bootstrapMgr.Start(bootstrapCtx); err != nil {
+			log.Error(err, "Bootstrap manager failed")
+		}
+		close(managerStopped)
+	}()
+
+	// Wait for cert-rotator to complete cert generation and CA injection
+	log.Info("Waiting for certificate generation and CA injection to complete")
+	<-certsReady
+	log.Info("Certificates ready and CA bundles injected")
+
+	log.Info("Stopping bootstrap manager")
+	bootstrapCancel()
+
+	log.Info("Waiting for the bootstrap manager to stop")
+	<-managerStopped
+
+	log.Info("Certificate bootstrap complete")
+	return nil
+}
+
+// ManageCerts adds the cert rotator to the main manager for ongoing certificate rotation.
 func ManageCerts(mgr ctrl.Manager, cfg config.Configuration, setupFinished chan struct{}) error {
-	// DNSName is <service name>.<namespace>.svc
-	var dnsName = fmt.Sprintf("%s.%s.svc", *cfg.InternalCertManagement.WebhookServiceName, *cfg.Namespace)
+	// Certs are already ready from BootstrapCerts, so we signal immediately
+	// The rotator runs in background for ongoing rotation.
+	rotatorReady := make(chan struct{})
+	rotatorConfig := buildCertRotatorConfig(cfg, "cert-rotator", rotatorReady)
+
+	if err := cert.AddRotator(mgr, rotatorConfig); err != nil {
+		return fmt.Errorf("unable to add cert rotator to manager: %w", err)
+	}
+
+	// Signal that certs are ready (they were set up by BootstrapCerts).
+	close(setupFinished)
+	return nil
+}
+
+// deriveWebhookBaseName extracts the base name from a webhook service name
+func deriveWebhookBaseName(webhookServiceName string) string {
+	return strings.TrimSuffix(webhookServiceName, webhookServiceSuffix)
+}
 
-	// Derive webhook configuration names from the webhook service name
+// buildWebhookConfigurationName constructs a webhook configuration name
+// from a base name and webhook type suffix.
+func buildWebhookConfigurationName(baseName, webhookType string) string {
+	return fmt.Sprintf("%s-%s-webhook-configuration", baseName, webhookType)
+}
+
+// buildCertRotatorConfig creates common CertRotator configuration.
+func buildCertRotatorConfig(cfg config.Configuration, controllerName string, certsReady chan struct{}) *cert.CertRotator {
+	dnsName := fmt.Sprintf("%s.%s.svc", *cfg.InternalCertManagement.WebhookServiceName, *cfg.Namespace)
 	webhookBaseName := deriveWebhookBaseName(*cfg.InternalCertManagement.WebhookServiceName)
-	mutatingWebhookName := buildWebhookConfigurationName(webhookBaseName, "mutating")
-	validatingWebhookName := buildWebhookConfigurationName(webhookBaseName, "validating")
 
-	return cert.AddRotator(mgr, &cert.CertRotator{
+	return &cert.CertRotator{
 		SecretKey: types.NamespacedName{
 			Namespace: *cfg.Namespace,
 			Name:      *cfg.InternalCertManagement.WebhookSecretName,
@@ -57,13 +135,14 @@ func ManageCerts(mgr ctrl.Manager, cfg config.Configuration, setupFinished chan
 		CAName:         caName,
 		CAOrganization: caOrganization,
 		DNSName:        dnsName,
-		IsReady:        setupFinished,
+		IsReady:        certsReady,
+		ControllerName: controllerName,
 		Webhooks: []cert.WebhookInfo{{
 			Type: cert.Validating,
-			Name: validatingWebhookName,
+			Name: buildWebhookConfigurationName(webhookBaseName, "validating"),
 		}, {
 			Type: cert.Mutating,
-			Name: mutatingWebhookName,
+			Name: buildWebhookConfigurationName(webhookBaseName, "mutating"),
 		}, {
 			Type: cert.CRDConversion,
 			Name: "localqueues.kueue.x-k8s.io",
@@ -80,25 +159,6 @@ func ManageCerts(mgr ctrl.Manager, cfg config.Configuration, setupFinished chan
 			Type: cert.CRDConversion,
 			Name: "multikueueclusters.kueue.x-k8s.io",
 		}},
-		// When kueue is running in the leader election mode,
-		// we expect webhook server will run in primary and secondary instance
 		RequireLeaderElection: false,
-	})
-}
-
-func WaitForCertsReady(log logr.Logger, certsReady chan struct{}) {
-	log.Info("Waiting for certificate generation to complete")
-	<-certsReady
-	log.Info("Certs ready")
-}
-
-// deriveWebhookBaseName extracts the base name from a webhook service name
-func deriveWebhookBaseName(webhookServiceName string) string {
-	return strings.TrimSuffix(webhookServiceName, webhookServiceSuffix)
-}
-
-// buildWebhookConfigurationName constructs a webhook configuration name
-// from a base name and webhook type suffix.
-func buildWebhookConfigurationName(baseName, webhookType string) string {
-	return fmt.Sprintf("%s-%s-webhook-configuration", baseName, webhookType)
+	}
 }



Test patch:
diff --git a/pkg/controller/failurerecovery/pod_termination_controller_test.go b/pkg/controller/failurerecovery/pod_termination_controller_test.go
index b5c92957e..d873239d2 100644
--- a/pkg/controller/failurerecovery/pod_termination_controller_test.go
+++ b/pkg/controller/failurerecovery/pod_termination_controller_test.go
@@ -52,10 +52,7 @@ func TestCreateEventFilter(t *testing.T) {
 	fakeClock := testingclock.NewFakeClock(now)
 	cl := utiltesting.NewFakeClient()
 	recorder := &utiltesting.EventRecorder{}
-	reconciler, err := NewTerminatingPodReconciler(cl, recorder, WithClock(fakeClock))
-	if err != nil {
-		t.Fatalf("could not create reconciler: %v", err)
-	}
+	reconciler := NewTerminatingPodReconciler(cl, recorder, WithClock(fakeClock))
 
 	podToReconcile := testingpod.MakePod("pod", "ns").
 		StatusPhase(corev1.PodRunning).
@@ -103,10 +100,7 @@ func TestUpdateEventFilter(t *testing.T) {
 	fakeClock := testingclock.NewFakeClock(now)
 	cl := utiltesting.NewFakeClient()
 	recorder := &utiltesting.EventRecorder{}
-	reconciler, err := NewTerminatingPodReconciler(cl, recorder, WithClock(fakeClock))
-	if err != nil {
-		t.Fatalf("could not create reconciler: %v", err)
-	}
+	reconciler := NewTerminatingPodReconciler(cl, recorder, WithClock(fakeClock))
 
 	oldPod := testingpod.MakePod("pod", "ns").
 		StatusPhase(corev1.PodRunning).
@@ -261,10 +255,7 @@ func TestReconciler(t *testing.T) {
 			clientBuilder := utiltesting.NewClientBuilder().WithObjects(objs...)
 			cl := clientBuilder.Build()
 			recorder := &utiltesting.EventRecorder{}
-			reconciler, err := NewTerminatingPodReconciler(cl, recorder, WithClock(fakeClock))
-			if err != nil {
-				t.Fatalf("could not create reconciler: %v", err)
-			}
+			reconciler := NewTerminatingPodReconciler(cl, recorder, WithClock(fakeClock))
 
 			ctxWithLogger, _ := utiltesting.ContextWithLog(t)
 			ctx, ctxCancel := context.WithCancel(ctxWithLogger)
diff --git a/test/e2e/upgrade/suite_test.go b/test/e2e/upgrade/suite_test.go
new file mode 100644
index 000000000..f67a3d61a
--- /dev/null
+++ b/test/e2e/upgrade/suite_test.go
@@ -0,0 +1,79 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package upgrade
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/onsi/ginkgo/v2"
+	"github.com/onsi/gomega"
+	"k8s.io/client-go/rest"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	"sigs.k8s.io/kueue/test/util"
+)
+
+var (
+	k8sClient   client.WithWatch
+	cfg         *rest.Config
+	ctx         context.Context
+	kueueNS     = util.GetKueueNamespace()
+	upgradeFrom = os.Getenv("KUEUE_UPGRADE_FROM_VERSION")
+)
+
+func TestUpgrade(t *testing.T) {
+	suiteName := "Upgrade Test Suite"
+	if upgradeFrom != "" {
+		suiteName = fmt.Sprintf("%s: %s -> current", suiteName, upgradeFrom)
+	}
+	if ver, found := os.LookupEnv("E2E_KIND_VERSION"); found {
+		suiteName = fmt.Sprintf("%s (K8s %s)", suiteName, ver)
+	}
+	gomega.RegisterFailHandler(ginkgo.Fail)
+	ginkgo.RunSpecs(t, suiteName)
+}
+
+var _ = ginkgo.BeforeSuite(func() {
+	util.SetupLogger()
+
+	if upgradeFrom == "" {
+		ginkgo.Fail("KUEUE_UPGRADE_FROM_VERSION environment variable must be set")
+	}
+
+	ginkgo.GinkgoLogr.Info("Starting upgrade test suite",
+		"upgradeFrom", upgradeFrom,
+		"namespace", kueueNS)
+
+	var err error
+
+	k8sClient, cfg, err = util.CreateClientUsingCluster("")
+	gomega.Expect(err).NotTo(gomega.HaveOccurred())
+	ctx = ginkgo.GinkgoT().Context()
+
+	ginkgo.GinkgoLogr.Info("Waiting for Kueue to be available", "version", upgradeFrom)
+	waitForAvailableStart := time.Now()
+	util.WaitForKueueAvailabilityNoRestartCountCheck(ctx, k8sClient)
+	ginkgo.GinkgoLogr.Info(
+		"Kueue is available",
+		"version", upgradeFrom,
+		"waitingTime", time.Since(waitForAvailableStart),
+	)
+})
diff --git a/test/e2e/upgrade/upgrade_validation_test.go b/test/e2e/upgrade/upgrade_validation_test.go
new file mode 100644
index 000000000..dc773c023
--- /dev/null
+++ b/test/e2e/upgrade/upgrade_validation_test.go
@@ -0,0 +1,162 @@
+/*
+Copyright The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package upgrade
+
+import (
+	"strings"
+
+	"github.com/onsi/ginkgo/v2"
+	"github.com/onsi/gomega"
+	admissionregistrationv1 "k8s.io/api/admissionregistration/v1"
+	corev1 "k8s.io/api/core/v1"
+	apiextensionsv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
+	"k8s.io/apimachinery/pkg/types"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+
+	kueue "sigs.k8s.io/kueue/apis/kueue/v1beta2"
+	workloadjob "sigs.k8s.io/kueue/pkg/controller/jobs/job"
+	testingjob "sigs.k8s.io/kueue/pkg/util/testingjobs/job"
+	"sigs.k8s.io/kueue/test/util"
+)
+
+var _ = ginkgo.Describe("Upgrade Validation", ginkgo.Ordered, func() {
+	ginkgo.It("should have completed upgrade successfully", func() {
+		ginkgo.By("Waiting for conversion webhook to be ready to list LocalQueues")
+		gomega.Eventually(func(g gomega.Gomega) {
+			lqList := &kueue.LocalQueueList{}
+			g.Expect(k8sClient.List(ctx, lqList)).To(gomega.Succeed(), "Should be able to list LocalQueues (conversion webhook should be ready)")
+			g.Expect(lqList.Items).NotTo(gomega.BeEmpty(), "Should have at least one LocalQueue")
+		}, util.StartUpTimeout, util.Interval).Should(gomega.Succeed())
+
+		ginkgo.By("Waiting for mutating webhook to be ready")
+		gomega.Eventually(func(g gomega.Gomega) {
+			mwc := &admissionregistrationv1.MutatingWebhookConfiguration{}
+			g.Expect(k8sClient.Get(ctx, client.ObjectKey{Name: "kueue-mutating-webhook-configuration"}, mwc)).To(gomega.Succeed())
+
+			for _, webhook := range mwc.Webhooks {
+				g.Expect(webhook.ClientConfig.CABundle).ToNot(gomega.BeEmpty())
+			}
+		}, util.StartUpTimeout, util.Interval).Should(gomega.Succeed())
+	})
+
+	ginkgo.It("should have CA bundles injected in all CRD conversion webhooks", func() {
+		crdList := &apiextensionsv1.CustomResourceDefinitionList{}
+		gomega.Expect(k8sClient.List(ctx, crdList)).To(gomega.Succeed())
+
+		for _, crd := range crdList.Items {
+			if !strings.HasPrefix(crd.Name, "kueue.x-k8s.io") {
+				continue
+			}
+			if crd.Spec.Conversion == nil || crd.Spec.Conversion.Strategy != apiextensionsv1.WebhookConverter {
+				continue
+			}
+			if crd.Spec.Conversion.Webhook == nil || crd.Spec.Conversion.Webhook.ClientConfig == nil {
+				continue
+			}
+
+			caBundle := crd.Spec.Conversion.Webhook.ClientConfig.CABundle
+			gomega.Expect(caBundle).NotTo(gomega.BeEmpty(), "CA bundle should not be empty for CRD %s", crd.Name)
+			gomega.Expect(string(caBundle)).To(gomega.ContainSubstring("-----BEGIN CERTIFICATE-----"),
+				"CA bundle should contain a valid PEM certificate for CRD %s", crd.Name)
+		}
+	})
+
+	ginkgo.It("should read pre-existing LocalQueue without x509 errors", func() {
+		lq := &kueue.LocalQueue{}
+		lqKey := types.NamespacedName{Name: "upgrade-test-lq", Namespace: "default"}
+		gomega.Expect(k8sClient.Get(ctx, lqKey, lq)).To(gomega.Succeed())
+	})
+
+	ginkgo.It("should read pre-existing ClusterQueue without x509 errors", func() {
+		cq := &kueue.ClusterQueue{}
+		cqKey := types.NamespacedName{Name: "upgrade-test-cq"}
+		gomega.Expect(k8sClient.Get(ctx, cqKey, cq)).To(gomega.Succeed())
+	})
+
+	ginkgo.It("should have webhook secret with valid certificates", func() {
+		secret := &corev1.Secret{}
+		secretKey := types.NamespacedName{Name: "kueue-webhook-server-cert", Namespace: kueueNS}
+		gomega.Expect(k8sClient.Get(ctx, secretKey, secret)).To(gomega.Succeed())
+
+		gomega.Expect(secret.Data).To(gomega.HaveKey("tls.crt"))
+		gomega.Expect(secret.Data).To(gomega.HaveKey("tls.key"))
+		gomega.Expect(secret.Data["tls.crt"]).NotTo(gomega.BeEmpty())
+		gomega.Expect(secret.Data["tls.key"]).NotTo(gomega.BeEmpty())
+	})
+
+	ginkgo.It("should have healthy controller manager pods", func() {
+		podList := &corev1.PodList{}
+		listOpts := []client.ListOption{
+			client.InNamespace(kueueNS),
+			client.MatchingLabels{"control-plane": "controller-manager"},
+		}
+		gomega.Expect(k8sClient.List(ctx, podList, listOpts...)).To(gomega.Succeed())
+		gomega.Expect(podList.Items).NotTo(gomega.BeEmpty())
+
+		for _, pod := range podList.Items {
+			gomega.Expect(pod.Status.ContainerStatuses).NotTo(gomega.BeEmpty())
+			for _, containerStatus := range pod.Status.ContainerStatuses {
+				if containerStatus.Name == "manager" {
+					gomega.Expect(containerStatus.Ready).To(gomega.BeTrue(),
+						"Container %s in pod %s should be ready", containerStatus.Name, pod.Name)
+					ginkgo.GinkgoLogr.Info("Pod health check",
+						"pod", pod.Name,
+						"container", containerStatus.Name,
+						"restarts", containerStatus.RestartCount,
+						"ready", containerStatus.Ready)
+				}
+			}
+		}
+	})
+
+	ginkgo.It("should admit jobs using pre-existing LocalQueue", func() {
+		const jobNamespace = "default"
+		const queueName = "upgrade-test-lq"
+
+		testJob := testingjob.MakeJob("upgrade-validation-job", jobNamespace).
+			Queue(kueue.LocalQueueName(queueName)).
+			Image(util.GetAgnHostImage(), util.BehaviorExitFast).
+			RequestAndLimit(corev1.ResourceCPU, "100m").
+			RequestAndLimit(corev1.ResourceMemory, "50Mi").
+			Obj()
+
+		ginkgo.By("Creating test job")
+		util.MustCreate(ctx, k8sClient, testJob)
+		ginkgo.DeferCleanup(func() {
+			gomega.Expect(util.DeleteObject(ctx, k8sClient, testJob)).To(gomega.Succeed())
+		})
+
+		ginkgo.GinkgoLogr.Info("Created test job", "job", testJob.Name, "queue", queueName)
+
+		wlLookupKey := types.NamespacedName{
+			Name:      workloadjob.GetWorkloadNameForJob(testJob.Name, testJob.UID),
+			Namespace: jobNamespace,
+		}
+
+		ginkgo.By("Waiting for workload to be created")
+		createdWorkload := &kueue.Workload{}
+		gomega.Eventually(func(g gomega.Gomega) {
+			g.Expect(k8sClient.Get(ctx, wlLookupKey, createdWorkload)).To(gomega.Succeed())
+		}, util.Timeout, util.Interval).Should(gomega.Succeed())
+
+		ginkgo.By("Waiting for job to be admitted")
+		util.ExpectJobUnsuspendedWithNodeSelectors(ctx, k8sClient, client.ObjectKeyFromObject(testJob), nil)
+
+		ginkgo.By("Verifying workload is admitted")
+		util.ExpectWorkloadsToBeAdmitted(ctx, k8sClient, createdWorkload)
+	})
+})
diff --git a/test/integration/singlecluster/controller/failurerecovery/suite_test.go b/test/integration/singlecluster/controller/failurerecovery/suite_test.go
index d9d2ebe06..bc53cfa4f 100644
--- a/test/integration/singlecluster/controller/failurerecovery/suite_test.go
+++ b/test/integration/singlecluster/controller/failurerecovery/suite_test.go
@@ -27,6 +27,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/manager"
 
+	configapi "sigs.k8s.io/kueue/apis/config/v1beta2"
 	"sigs.k8s.io/kueue/pkg/constants"
 	"sigs.k8s.io/kueue/pkg/controller/failurerecovery"
 	"sigs.k8s.io/kueue/test/integration/framework"
@@ -58,14 +59,13 @@ var _ = ginkgo.AfterSuite(func() {
 	fwk.Teardown()
 })
 
-func managerSetup(ctx context.Context, mgr manager.Manager) {
-	terminatingPodReconciler, err := failurerecovery.NewTerminatingPodReconciler(
+func managerSetup(_ context.Context, mgr manager.Manager) {
+	terminatingPodReconciler := failurerecovery.NewTerminatingPodReconciler(
 		mgr.GetClient(),
 		mgr.GetEventRecorderFor(constants.PodTerminationControllerName),
 		failurerecovery.WithForcefulTerminationGracePeriod(time.Millisecond),
 	)
-	gomega.Expect(err).NotTo(gomega.HaveOccurred())
 
-	err = terminatingPodReconciler.SetupWithManager(mgr)
+	_, err := terminatingPodReconciler.SetupWithManager(mgr, &configapi.Configuration{})
 	gomega.Expect(err).NotTo(gomega.HaveOccurred())
 }