Repository: minio/operator

Issue:
Operator is not resiliant to a short API-Server downtime
Similar to #2450 (but this time, **both pods** are stopped)

I found the same problem as last time. Following a small unavailability of the Kubernetes API-Server: the operator can't update its lease.
This behavior leads to a **total stop (without the pod restarting)** in addition to an abnormal consumption of resources.

Please note that the two pods are on 2 differents nodes. This micro-disruption in API-Server availability only affected one pod at a time (a first replica several days ago on node-1, the other replica recently on node-2). Since you can't set liveness, pods never restart and are stuck in this state

## Expected Behavior

I expected there to be a retry until the pod could succeed in its request (or better, could properly stop itself).

## Current Behavior

Currently, the operator's two replicas are stopped and no longer respond to anything (creating/deleting a CRD Tenant does nothing). On top of that, the pods will consume **1000m CPU**, as if there were a loop in the code that was never exited.

## Possible Solution

Add a retry (or fix the problem where the pod can't stop if the API-Server is unavailable).


## Steps to Reproduce (for bugs)

1. Deploy a cluster with a single controlplane
2. Install Minio Operator (ensure both pods are not on the cp)
3. Restart the controlplane to create a downtime of api-server
4. Operator is stuck

## Your Environment

- Version used (minio-operator): quay.io/minio/operator:v7.0.0
- Environment name and version (e.g. kubernetes v1.17.2): v1.32.2
- Server type and version: Talos 1.9.3 (VM)
- Link to your deployment file: https://gist.github.com/qjoly/b96a1509d130d3902ef4957e8dba8d85



Discussion:
@jiuker: Could you share the full log? @qjoly: > Could you share the full log?

Of course, here are the logs (they are quite similar to my first issue)

```log
I0528 06:47:58.013441       1 controller.go:81] Starting MinIO Operator
I0528 06:47:58.014560       1 main-controller.go:293] Setting up event handlers
I0528 06:47:58.114707       1 main-controller.go:514] Using Kubernetes CSR Version: v1
I0528 06:47:58.114767       1 main-controller.go:534] Waiting for STS API to start
I0528 06:47:58.114804       1 leaderelection.go:254] attempting to acquire leader lease minio/minio-operator-lock...
I0528 06:47:58.114889       1 main-controller.go:399] Starting STS API server
I0528 06:47:58.145916       1 leaderelection.go:268] successfully acquired lease minio/minio-operator-lock
I0528 06:47:58.145993       1 main-controller.go:577] minio-operator-9d9788f54-58tph: I am the leader, applying leader labels on myself
I0528 06:47:58.146084       1 main-controller.go:428] Waiting for Upgrade Server to start
I0528 06:47:58.146116       1 main-controller.go:432] Starting Tenant controller
I0528 06:47:58.146124       1 main-controller.go:435] Waiting for informer caches to sync
I0528 06:47:58.146156       1 main-controller.go:456] STS Autocert is enabled, starting API certificate setup.
I0528 06:47:58.146190       1 main-controller.go:388] Starting HTTP Upgrade Tenant Image server
I0528 06:48:03.124640       1 minio-services.go:156] Headless Services don't match: service ports don't match
I0528 06:48:03.142675       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio", Name:"lucca", UID:"5b5e542e-21a8-4a67-82a5-dc11c3c2a8cd", APIVersion:"minio.min.io/v2", ResourceVersion:"104157495", FieldPath:""}): typ
E0528 16:44:14.245747       1 leaderelection.go:429] Failed to update lock optimitically: Put "https://10.2.0.1:443/apis/coordination.k8s.io/v1/namespaces/minio/leases/minio-operator-lock": context deadline exceeded, falling back to slow p
E0528 16:44:14.245814       1 leaderelection.go:436] error retrieving resource lock minio/minio-operator-lock: client rate limiter Wait returned an error: context deadline exceeded
I0528 16:44:14.245822       1 leaderelection.go:297] failed to renew lease minio/minio-operator-lock: timed out waiting for the condition
E0528 16:44:29.245979       1 leaderelection.go:322] Failed to release lock: Put "https://10.2.0.1:443/apis/coordination.k8s.io/v1/namespaces/minio/leases/minio-operator-lock": context deadline exceeded
I0528 16:44:29.246021       1 main-controller.go:559] leader lost, removing any leader labels that I 'minio-operator-9d9788f54-58tph' might have
W0528 16:44:39.239583       1 reflector.go:484] k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: watch of *v2.Tenant ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost"
W0528 16:44:39.239583       1 reflector.go:484] k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost"
W0528 16:44:39.239582       1 reflector.go:484] k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: watch of *v1.Deployment ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection l
W0528 16:44:39.239639       1 reflector.go:484] k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost
W0528 16:44:39.239647       1 reflector.go:484] k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: watch of *v1beta1.PolicyBinding ended with: an error on the server ("unable to decode an event from the watch stream: http2: client conn
I0528 16:44:39.239717       1 main-controller.go:617] Stopping the minio controller webservers
W0528 16:44:39.239710       1 reflector.go:484] k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: watch of *v1.Pod ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") h
W0528 16:44:39.239729       1 reflector.go:484] k8s.io/client-go@v0.31.1/tools/cache/reflector.go:243: watch of *v1.StatefulSet ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection
I0528 16:44:39.239959       1 main-controller.go:624] Stopping the minio controller
I0530 06:47:58.245720       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0530 06:48:00.324927       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0530 06:52:59.773806       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0530 06:57:59.544509       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0530 07:02:59.873807       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0530 07:07:59.782127       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0530 07:12:59.824196       1 status.go:89] Hit conflict issue, getting latest version of tenant
I0530 07:17:59.661505       1 status.go:89] Hit conflict issue, getting latest version of tenant
```@jiuker: Is it stuck all the time? Don't refresh the logs? @qjoly @qjoly: > Is it stuck all the time? Don't refresh the logs? [@qjoly](https://github.com/qjoly)

Yes, the error appeared on May 28, and I got some logs this morning (but I think it's because I restarted one of the two replicas to fix it).
Currently, the still broken pod continues to consume 1000m (my loop theory in the code) and does nothing more.

![Image](https://github.com/user-attachments/assets/bac91666-73e8-4e3b-8390-2e8ee2486637)


@JoelRuizRojas: It is not very common for Kubernetes API server to go down and this can be fixed whenever API server recovers by restarting the Operator. This is not a critical issue for Operator. Closing it@qjoly: > It is not very common for Kubernetes API server to go down and this can be fixed whenever API server recovers by restarting the Operator. This is not a critical issue for Operator. Closing it

Knowing that there's no liveness / readiness on the operator (and according to the developers on slack: it's not possible), it's fun to have to restart pods by hand every time. Also, note that we're talking about very short network outages. It's not normal not to be resilient to these events. @zwindler: > It is not very common for Kubernetes API server to go down and this can be fixed whenever API server recovers by restarting the Operator. This is not a critical issue for Operator. Closing it

I'm sorry but have you ever run any kind of production???? I'm having trouble to believe I'm reading this.

Servers go down. Maintenances happen. 

ALL. THE. TIME.

Kubernetes / cloud native whole point is to fix those issues by self healing and reconciling.@tmmorin: > It is not very common for Kubernetes API server to go down and this can be fixed whenever API server recovers by restarting the Operator. This is not a critical issue for Operator. 

In the past years there has been an increased adoption of Cluster API, and in that context nodes it is a routine even to see nodes being drained to be deleted and replaced by other nodes.  On control plane nodes, this of course implies that API server will go down.


@kivikakk: Not to crowd this out with "me too"s, but I encountered this almost immediately with Operator 7.1.1. As with the OP here, the operator pods both gave up when they encountered API server unavailability, and reported they were stopping. Both continued to use 100% of a core on each node they ran on.

`kubectl --namespace minio-operator rollout restart deploy/minio-operator` resolves the issue, once.

Logs look much the same, but included in case of interest.

<details>

<summary>Logs from one of the <code>minio-operator</code> pods</summary>

```
I0722 05:37:21.168602       1 controller.go:81] Starting MinIO Operator
I0722 05:37:21.171594       1 main-controller.go:293] Setting up event handlers
I0722 05:37:21.210104       1 main-controller.go:514] Using Kubernetes CSR Version: v1
I0722 05:37:21.210148       1 main-controller.go:534] Waiting for STS API to start
I0722 05:37:21.210186       1 leaderelection.go:257] attempting to acquire leader lease minio-operator/minio-operator-lock...
I0722 05:37:21.210488       1 main-controller.go:399] Starting STS API server
I0722 05:37:21.216001       1 main-controller.go:595] minio-operator-7697f69655-2v5fz: is the leader, removing any leader labels that I 'minio-operator-7697f69655-mfts7' mig
ht have
I0722 05:37:21.216856       1 tls.go:54] Waiting for the sts certificates secret to be issued
I0722 09:02:49.359352       1 leaderelection.go:271] successfully acquired lease minio-operator/minio-operator-lock
I0722 09:02:49.359515       1 main-controller.go:577] minio-operator-7697f69655-mfts7: I am the leader, applying leader labels on myself
I0722 09:02:49.359523       1 main-controller.go:428] Waiting for Upgrade Server to start
I0722 09:02:49.359804       1 main-controller.go:432] Starting Tenant controller
I0722 09:02:49.359814       1 main-controller.go:435] Waiting for informer caches to sync
I0722 09:02:49.359906       1 main-controller.go:456] STS Autocert is enabled, starting API certificate setup.
I0722 09:02:49.359979       1 main-controller.go:388] Starting HTTP Upgrade Tenant Image server
E0722 09:09:05.026735       1 leaderelection.go:429] Failed to update lock optimistically: Put "https://10.43.0.1:443/apis/coordination.k8s.io/v1/namespaces/minio-operator/leases/minio-operator-lock": context deadline exceeded, falling back to slow path
E0722 09:09:05.027357       1 leaderelection.go:436] error retrieving resource lock minio-operator/minio-operator-lock: client rate limiter Wait returned an error: context deadline exceeded
I0722 09:09:05.027427       1 leaderelection.go:297] failed to renew lease minio-operator/minio-operator-lock: context deadline exceeded
I0722 09:09:05.034598       1 main-controller.go:559] leader lost, removing any leader labels that I 'minio-operator-7697f69655-mfts7' might have
I0722 09:09:05.048356       1 main-controller.go:617] Stopping the minio controller webservers
I0722 09:09:05.048685       1 main-controller.go:624] Stopping the minio controller
```

</details>

<details>

<summary>Logs from the other</summary>

```
I0722 05:37:21.165923       1 controller.go:81] Starting MinIO Operator
I0722 05:37:21.168236       1 main-controller.go:293] Setting up event handlers
I0722 05:37:21.197195       1 main-controller.go:514] Using Kubernetes CSR Version: v1
I0722 05:37:21.197324       1 main-controller.go:534] Waiting for STS API to start
I0722 05:37:21.197367       1 leaderelection.go:257] attempting to acquire leader lease minio-operator/minio-operator-lock...
I0722 05:37:21.197438       1 main-controller.go:399] Starting STS API server
I0722 05:37:21.204905       1 tls.go:54] Waiting for the sts certificates secret to be issued
I0722 05:37:21.214686       1 leaderelection.go:271] successfully acquired lease minio-operator/minio-operator-lock
I0722 05:37:21.214768       1 main-controller.go:577] minio-operator-7697f69655-2v5fz: I am the leader, applying leader labels on myself
I0722 05:37:21.214852       1 main-controller.go:428] Waiting for Upgrade Server to start
I0722 05:37:21.214890       1 main-controller.go:432] Starting Tenant controller
I0722 05:37:21.214895       1 main-controller.go:435] Waiting for informer caches to sync
I0722 05:37:21.214932       1 main-controller.go:388] Starting HTTP Upgrade Tenant Image server
I0722 05:37:21.315204       1 main-controller.go:456] STS Autocert is enabled, starting API certificate setup.
I0722 05:37:21.326170       1 tls.go:127] sts-tls TLS secret not found: secrets "sts-tls" not found
I0722 05:37:21.350124       1 csr.go:183] Start polling for certificate of csr/sts-minio-operator-csr, every 5s, timeout after 20m0s
I0722 05:37:26.355407       1 csr.go:209] Certificate successfully fetched, creating secret with Private key and Certificate
I0722 05:37:26.363588       1 tls.go:130] Waiting for the sts certificates to be issued waiting for sts cert
W0722 09:02:34.282971       1 reflector.go:492] k8s.io/client-go@v0.32.3/tools/cache/reflector.go:251: watch of *v1.Pod ended with: etcdserver: leader changed
E0722 09:02:46.839705       1 leaderelection.go:429] Failed to update lock optimistically: Put "https://10.43.0.1:443/apis/coordination.k8s.io/v1/namespaces/minio-operator/leases/minio-operator-lock": context deadline exceeded, falling back to slow path
E0722 09:02:46.840743       1 leaderelection.go:436] error retrieving resource lock minio-operator/minio-operator-lock: client rate limiter Wait returned an error: context deadline exceeded
I0722 09:02:46.840889       1 leaderelection.go:297] failed to renew lease minio-operator/minio-operator-lock: context deadline exceeded
I0722 09:02:46.855714       1 main-controller.go:559] leader lost, removing any leader labels that I 'minio-operator-7697f69655-2v5fz' might have
I0722 09:02:46.870170       1 main-controller.go:617] Stopping the minio controller webservers
I0722 09:02:46.870738       1 main-controller.go:624] Stopping the minio controller
```

</details>@mx-sylveon: > It is not very common for Kubernetes API server to go down and this can be fixed whenever API server recovers by restarting the Operator. This is not a critical issue for Operator. Closing it

On the contrary, this issue is literally a case study in what a critical issue for a k8s operator could be. I'm astounded this was originally closed citing reasoning that was so far off the mark.

An operator is supposed to manage resources on the cluster. If it silently stops running reconciliation for those resources with no outward indication that something is wrong, it is completely failing to serve its entire purpose in life.

And fwiw when dealing with k8s or anything else, requiring a manual restart for a service is not how you build resilient systems.@ramondeklein: @mx-sylveon Can you retest with the current `master` if that fixes your issue?@mx-sylveon: @ramondeklein I've been busy with my own projects, but I finally got around to testing this, and it looks like the issue still exists even with the operator built from the latest `master` branch.

# Testing steps:
## Control test with latest released operator
- I confirmed the bug by simultaneously restarting the k3s service on my all nodes in 3x master node k3s cluster where minio-operator is already running.
- Whichever operator pod (I have 2 replicas) was the leader gets stuck in the 1000mCPU resource utilization state after logging "Stopping the minio controller", as others have seen above.

## Test with latest `master` branch
- I checked out the latest as of this commit: https://github.com/minio/operator/commit/34dc3980190390f4ed2f4b4355dbcb936f667a10
- Per the `DEVELOPMENT.md` instructions, I ran `make build TAG="my-private-registry.local/minio/operator:sha-34dc3980190390f4ed2f4b4355dbcb936f667a10"`
- I pushed that image to my private registry with `docker push my-private-registry.local/minio/operator:sha-34dc3980190390f4ed2f4b4355dbcb936f667a10`
- I updated my deployed operator to that image by setting the helm chart values appropriately:

```
operator:
  image:
    repository: my-private-registry.local/minio/operator
    tag: sha-34dc3980190390f4ed2f4b4355dbcb936f667a10
```

- I confirmed after the operator deployed that the image of the running pods was indeed `image: my-private-registry.local/minio/operator:sha-34dc3980190390f4ed2f4b4355dbcb936f667a10`
- Repeated the test, restarting k3s on all nodes in my k3s cluster.
- Observed the leader minio-operator pod immediately getting stuck at 1000mCPU, with the same log as before: "Stopping the minio controller".

Last logs for reference:
```
leaderelection.go:297] failed to renew lease minio-operator/minio-operator-lock: context deadline exceeded
main-controller.go:559] leader lost, removing any leader labels that I 'minio-operator-6cb977df95-fx6nt' might have
main-controller.go:617] Stopping the minio controller webservers
main-controller.go:624] Stopping the minio controller
```

~~So it looks like this issue definitely needs to be reopened.~~ EDIT: Latest `master` commit wasn't actually checked out -> https://github.com/minio/operator/issues/2458#issuecomment-3558962470@jiuker: Sorry,  you are not using the master branch. `main-controller.go:624` is not print that line, please rebuild and retest @mx-sylveon @jiuker: Should be `main-controller.go:607]`@mx-sylveon: > Should be `main-controller.go:607]`

Good catch, I made 100% sure I pulled the latest changes, but ended up not having them checked out. 😅 

I repeated the test, and now once k3s services are restarted and the k8s api is available again I can see that the minio-operator leader actually stopped & restarted as expected instead of getting stuck.

```
leaderelection.go:297] failed to renew lease minio-operator/minio-operator-lock: context deadline exceeded
leaderelection.go:322] Failed to release lock: apiserver not ready
main-controller.go:543] leader lost, removing any leader labels that I 'minio-operator-668c89b5f7-vqp2h' might have
monitoring.go:48] recurrent pod status monitor closed
controller.go:185] Shutting down the MinIO Operator
main-controller.go:600] Stopping the minio controller webservers
main-controller.go:607] Stopping the minio controller
```

<img width="678" height="94" alt="Image" src="https://github.com/user-attachments/assets/39a7250c-46a3-418d-bd0f-97a114eb0c21" />

Issue doesn't need to be reopened then, though I'm curious when the next release will happen so I don't need a custom operator image deployed, given it has been some months since the last release.

Solution patch:
diff --git a/README.md b/README.md
index 65ef1b33..479df45f 100644
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
 # MinIO Operator
 
-![build](https://github.com/minio/operator/workflows/Go/badge.svg) ![license](https://img.shields.io/badge/license-AGPL%20V3-blue)
+[![build](https://img.shields.io/badge/build-passing-green.svg)](https://github.com/minio/operator/actions) [![license](https://img.shields.io/badge/license-AGPL%20V3-blue)](https://github.com/minio/operator/blob/master/LICENSE)
 
 [![MinIO](https://raw.githubusercontent.com/minio/minio/master/.github/logo.svg?sanitize=true)](https://min.io)
 
diff --git a/helm/tenant/templates/tenant.yaml b/helm/tenant/templates/tenant.yaml
index a80fa45d..e1e9ee80 100644
--- a/helm/tenant/templates/tenant.yaml
+++ b/helm/tenant/templates/tenant.yaml
@@ -29,6 +29,9 @@ spec:
   {{- with (dig "initContainers" (list) .) }}
   initContainers: {{- toYaml . | nindent 4 }}
   {{- end }}
+  {{- with (dig "sideCars" (dict) .) }}
+  sideCars: {{- toYaml . | nindent 4 }}
+  {{- end }}
   ## Secret with default environment variable configurations
   configuration:
     name: {{ .configSecret.name }}
@@ -181,22 +184,24 @@ spec:
     {{- end }}
     ## Key name to be created on the KMS, default is "my-minio-key"
     keyName: {{ .kes.keyName | quote }}
-    {{- with (dig "resources" (dict) .) }}
-    resources: {{- toYaml . | nindent 4 }}
+    {{- with (dig "kes" "resources" (dict) .) | default (dig "resources" (dict) .)}}
+    resources: {{- toYaml . | nindent 6 }}
     {{- end }}
-    {{- with (dig "nodeSelector" (dict) .) }}
-    nodeSelector: {{- toYaml . | nindent 4 }}
+    {{- with (dig "kes" "nodeSelector" (dict) .) | default (dig "nodeSelector" (dict) .) }}
+    nodeSelector: {{- toYaml . | nindent 6 }}
     {{- end }}
     affinity:
-      nodeAffinity: {}
-      podAffinity: {}
-      podAntiAffinity: {}
-    tolerations: []
-    {{- with (dig "annotations" (dict) .) }}
-    annotations: {{- toYaml . | nindent 4 }}
+      nodeAffinity: {{- (dig "kes" "affinity" "nodeAffinity" (dict) .) | toYaml | nindent 8 }}
+      podAffinity: {{- (dig "kes" "affinity" "podAffinity" (dict) .) | toYaml | nindent 8 }}
+      podAntiAffinity: {{- (dig "kes" "affinity" "podAntiAffinity" (dict) .) | toYaml | nindent 8 }}
+    {{- with (dig "kes" "tolerations" (list) .) }}
+    tolerations: {{- toYaml . | nindent 4 }}
+    {{- end }}
+    {{- with (dig "kes" "annotations" (dict) .) | default (dig "annotations" (dict) .) }}
+    annotations: {{- toYaml . | nindent 6 }}
     {{- end }}
-    {{- with (dig "labels" (dict) .) }}
-    labels: {{- toYaml . | nindent 4 }}
+    {{- with (dig "kes" "labels" (dict) .) | default (dig "labels" (dict) .) }}
+    labels: {{- toYaml . | nindent 6 }}
     {{- end }}
     serviceAccountName: {{ .kes.serviceAccountName | quote }}
     {{- if hasKey .kes "securityContext" }}
@@ -212,4 +217,7 @@ spec:
       {{- end }}
       {{- end }}
   {{- end }}
+  {{- with (dig "sideCars" (dict) .) }}
+  sideCars: {{- toYaml . | nindent 4 }}
+  {{- end }}
 {{- end }}
diff --git a/helm/tenant/values.yaml b/helm/tenant/values.yaml
index 02971d51..ccb5e2b1 100644
--- a/helm/tenant/values.yaml
+++ b/helm/tenant/values.yaml
@@ -62,6 +62,12 @@ tenant:
   #
   initContainers: [ ]
   ###
+  #
+  # Specify `sideCars <https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/>`__.
+  # See https://github.com/minio/operator/blob/master/docs/tenant_crd.adoc#sidecars
+  #
+  sideCars: { }
+  ###
   # The Kubernetes `Scheduler <https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/>`__ to use for dispatching Tenant pods.
   #
   # Specify an empty dictionary ``{}`` to dispatch pods with the default scheduler.
@@ -475,6 +481,11 @@ tenant:
   #    seccompProfile:
   #      type: RuntimeDefault
 
+  ###
+  # Customize sideCars, as defined in https://min.io/docs/minio/kubernetes/upstream/reference/operator-crd.html#sidecars
+  #sideCars:
+  #  resources: { }
+
 ###
 # Configures `Ingress <https://kubernetes.io/docs/concepts/services-networking/ingress/>`__ for the Tenant S3 API and Console.
 #
diff --git a/pkg/apis/minio.min.io/v2/names.go b/pkg/apis/minio.min.io/v2/names.go
index b0761f91..b65a667b 100644
--- a/pkg/apis/minio.min.io/v2/names.go
+++ b/pkg/apis/minio.min.io/v2/names.go
@@ -207,7 +207,7 @@ func (t *Tenant) PrometheusConfigJobName() string {
 // PrometheusOperatorAddlConfigJobName returns the name of the prometheus job
 // when prometheus operator is enabled
 func (t *Tenant) PrometheusOperatorAddlConfigJobName() string {
-	return fmt.Sprintf("%s-minio-job", t.Name)
+	return fmt.Sprintf("%s-%s-minio-job", t.Name, t.Namespace)
 }
 
 // PrometheusConfigMapName returns name of the config map for Prometheus.
diff --git a/pkg/controller/controller.go b/pkg/controller/controller.go
index 2b60fa7e..0dda2213 100644
--- a/pkg/controller/controller.go
+++ b/pkg/controller/controller.go
@@ -15,6 +15,7 @@
 package controller
 
 import (
+	"context"
 	"flag"
 	"fmt"
 	"os"
@@ -79,8 +80,10 @@ func StartOperator(kubeconfig string) {
 	_ = stsv1beta1.AddToScheme(scheme.Scheme)
 	_ = stsv1alpha1.AddToScheme(scheme.Scheme)
 	klog.Info("Starting MinIO Operator")
+
 	// set up signals, so we handle the first shutdown signal gracefully
-	stopCh := setupSignalHandler()
+	ctx, cancel := setupSignalHandler(context.Background())
+	defer cancel()
 
 	flag.Parse()
 
@@ -170,15 +173,15 @@ func StartOperator(kubeconfig string) {
 		kubeInformerFactoryInOperatorNamespace,
 	)
 
-	go kubeInformerFactory.Start(stopCh)
-	go minioInformerFactory.Start(stopCh)
-	go kubeInformerFactoryInOperatorNamespace.Start(stopCh)
+	go kubeInformerFactory.Start(ctx.Done())
+	go minioInformerFactory.Start(ctx.Done())
+	go kubeInformerFactoryInOperatorNamespace.Start(ctx.Done())
 
-	if err = mainController.Start(2, stopCh); err != nil {
+	if err = mainController.Start(ctx, cancel, 2); err != nil {
 		klog.Fatalf("Error running mainController: %s", err.Error())
 	}
 
-	<-stopCh
+	<-ctx.Done()
 	klog.Info("Shutting down the MinIO Operator")
 	mainController.Stop()
 }
@@ -186,22 +189,11 @@ func StartOperator(kubeconfig string) {
 // setupSignalHandler registered for SIGTERM and SIGINT. A stop channel is returned
 // which is closed on one of these signals. If a second signal is caught, the program
 // is terminated with exit code 1.
-func setupSignalHandler() (stopCh <-chan struct{}) {
+func setupSignalHandler(ctx context.Context) (context.Context, context.CancelFunc) {
 	// panics when called twice
 	close(onlyOneSignalHandler)
 
-	stop := make(chan struct{})
-	c := make(chan os.Signal, 2)
-	signal.Notify(c, shutdownSignals...)
-	go func() {
-		<-c
-		close(stop)
-		<-c
-		// second signal. Exit directly.
-		os.Exit(1)
-	}()
-
-	return stop
+	return signal.NotifyContext(ctx, shutdownSignals...)
 }
 
 // Result contains the result of a sync invocation.
diff --git a/pkg/controller/csr.go b/pkg/controller/csr.go
index e627a187..c6763fc3 100644
--- a/pkg/controller/csr.go
+++ b/pkg/controller/csr.go
@@ -24,7 +24,6 @@ import (
 	"errors"
 	"fmt"
 	"io"
-	"os"
 	"os/signal"
 	"syscall"
 	"time"
@@ -188,15 +187,14 @@ func (c *Controller) fetchCertificate(ctx context.Context, csrName string) ([]by
 
 	timeout := time.NewTimer(miniov2.DefaultQueryTimeout)
 
-	ch := make(chan os.Signal, 1) // should be always un-buffered SA1017
-	signal.Notify(ch, syscall.SIGINT, syscall.SIGTERM)
-	defer signal.Stop(ch)
+	ctx, cancel := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM)
+	defer cancel()
 
 	for {
 		select {
-		case s := <-ch:
-			klog.Infof("Signal %s received, exiting ...", s.String())
-			return nil, fmt.Errorf("%s", s.String())
+		case <-ctx.Done():
+			klog.Infof("Termination signal received, exiting.")
+			return nil, errors.New("Termination signal received, exiting.")
 
 		case <-tick.C:
 			if certificates.GetCertificatesAPIVersion(c.kubeClientSet) == certificates.CSRV1 {
diff --git a/pkg/controller/main-controller.go b/pkg/controller/main-controller.go
index 4f8b70f6..c7f1758e 100644
--- a/pkg/controller/main-controller.go
+++ b/pkg/controller/main-controller.go
@@ -21,10 +21,7 @@ import (
 	"fmt"
 	"maps"
 	"net/http"
-	"os"
-	"os/signal"
 	"strings"
-	"syscall"
 	"time"
 
 	kubeinformers "k8s.io/client-go/informers"
@@ -304,6 +301,9 @@ func NewController(
 			}
 			controller.enqueueTenant(newObj)
 		},
+		// Enqueue tenant to perform some delete handling actions
+		// during reconciliation
+		DeleteFunc: controller.enqueueTenant,
 	})
 
 	// Set up an event handler for when StatefulSet resources change. This
@@ -376,8 +376,8 @@ func NewController(
 }
 
 // StartPodInformer runs PodInformer
-func (c *Controller) StartPodInformer(stopCh <-chan struct{}) {
-	c.podInformer.Run(stopCh)
+func (c *Controller) StartPodInformer(ctx context.Context) {
+	c.podInformer.Run(ctx.Done())
 }
 
 // startUpgradeServer Starts the Upgrade tenant API server and notifies the start and stop via notificationChannel returned
@@ -421,7 +421,7 @@ func (c *Controller) startSTSAPIServer(ctx context.Context, notificationChannel
 
 // leaderRun start the Controller and the API's
 // When a new leader is elected this function is ran in the pod
-func leaderRun(ctx context.Context, c *Controller, threadiness int, stopCh <-chan struct{}, notificationChannel chan *EventNotification) {
+func leaderRun(ctx context.Context, c *Controller, threadiness int, notificationChannel chan *EventNotification) {
 	// we declate the channel to communicate on servers errors
 	var upgradeServerChannel <-chan error
 
@@ -433,21 +433,21 @@ func leaderRun(ctx context.Context, c *Controller, threadiness int, stopCh <-cha
 
 	// Wait for the caches to be synced before starting workers
 	klog.Info("Waiting for informer caches to sync")
-	if ok := cache.WaitForCacheSync(stopCh, c.statefulSetListerSynced, c.deploymentListerSynced, c.tenantsSynced, c.policyBindingListerSynced, c.secretListerSynced); !ok {
+	if ok := cache.WaitForCacheSync(ctx.Done(), c.statefulSetListerSynced, c.deploymentListerSynced, c.tenantsSynced, c.policyBindingListerSynced, c.secretListerSynced); !ok {
 		panic("failed to wait for caches to sync")
 	}
 
 	// Launch two workers to process Job resources
 	for i := 0; i < threadiness; i++ {
-		go wait.Until(c.runWorker, time.Second, stopCh)
+		go wait.Until(c.runWorker, time.Second, ctx.Done())
 	}
 
 	// Launch a single worker for Health Check reacting to Pod Changes
-	go wait.Until(c.runHealthCheckWorker, time.Second, stopCh)
+	go wait.Until(c.runHealthCheckWorker, time.Second, ctx.Done())
 
 	// Launch a goroutine to monitor all Tenants
-	go c.recurrentTenantStatusMonitor(stopCh)
-	go c.StartPodInformer(stopCh)
+	go c.recurrentTenantStatusMonitor(ctx)
+	go c.StartPodInformer(ctx)
 
 	// 2) we need to make sure we have STS API certificates (if enabled)
 	if IsSTSEnabled() {
@@ -475,7 +475,7 @@ func leaderRun(ctx context.Context, c *Controller, threadiness int, stopCh <-cha
 			}
 			// webserver was instructed to stop, do not attempt to restart
 			continue
-		case <-stopCh:
+		case <-ctx.Done():
 			return
 		}
 	}
@@ -485,23 +485,7 @@ func leaderRun(ctx context.Context, c *Controller, threadiness int, stopCh <-cha
 // as syncing informer caches and starting workers. It will block until stopCh
 // is closed, at which point it will shutdown the workqueue and wait for
 // workers to finish processing their current work items.
-func (c *Controller) Start(threadiness int, stopCh <-chan struct{}) error {
-	// use a Go context so we can tell the leaderelection code when we
-	// want to step down
-	ctx, cancel := context.WithCancel(context.Background())
-	defer cancel()
-
-	// listen for interrupts or the Linux SIGTERM signal and cancel
-	// our context, which the leader election code will observe and
-	// step down
-	ch := make(chan os.Signal, 1)
-	signal.Notify(ch, os.Interrupt, syscall.SIGTERM)
-	go func() {
-		<-ch
-		klog.Info("Received termination, signaling shutdown")
-		cancel()
-	}()
-
+func (c *Controller) Start(ctx context.Context, cancel context.CancelFunc, threadiness int) error {
 	leaseLockName := "minio-operator-lock"
 	leaseLockNamespace := miniov2.GetNSFromFile()
 
@@ -553,7 +537,7 @@ func (c *Controller) Start(threadiness int, stopCh <-chan struct{}) error {
 		Callbacks: leaderelection.LeaderCallbacks{
 			OnStartedLeading: func(ctx context.Context) {
 				// start the controller + API code
-				leaderRun(ctx, c, threadiness, stopCh, notificationChannel)
+				leaderRun(ctx, c, threadiness, notificationChannel)
 			},
 			OnStoppedLeading: func() {
 				klog.Infof("leader lost, removing any leader labels that I '%s' might have", c.podName)
@@ -568,7 +552,6 @@ func (c *Controller) Start(threadiness int, stopCh <-chan struct{}) error {
 				} else {
 					c.kubeClientSet.CoreV1().Pods(leaseLockNamespace).Patch(ctx, c.podName, types.JSONPatchType, payloadBytes, metav1.PatchOptions{})
 				}
-				c.Stop()
 				cancel()
 			},
 			OnNewLeader: func(identity string) {
diff --git a/pkg/controller/monitoring.go b/pkg/controller/monitoring.go
index 884c3eb1..abf3fe35 100644
--- a/pkg/controller/monitoring.go
+++ b/pkg/controller/monitoring.go
@@ -40,7 +40,7 @@ const (
 )
 
 // recurrentTenantStatusMonitor loop that checks every N minutes for tenants health
-func (c *Controller) recurrentTenantStatusMonitor(stopCh <-chan struct{}) {
+func (c *Controller) recurrentTenantStatusMonitor(ctx context.Context) {
 	// How often will this function run
 	interval := miniov2.GetMonitoringInterval()
 	ticker := time.NewTicker(time.Duration(interval) * time.Minute)
@@ -53,7 +53,7 @@ func (c *Controller) recurrentTenantStatusMonitor(stopCh <-chan struct{}) {
 			if err := c.tenantsHealthMonitor(); err != nil {
 				klog.Infof("%v", err)
 			}
-		case <-stopCh:
+		case <-ctx.Done():
 			ticker.Stop()
 			return
 		}
diff --git a/pkg/controller/prometheus.go b/pkg/controller/prometheus.go
index 7af822c7..249d2356 100644
--- a/pkg/controller/prometheus.go
+++ b/pkg/controller/prometheus.go
@@ -140,11 +140,13 @@ func (c *Controller) checkAndCreatePrometheusAddlConfig(ctx context.Context, ten
 	}
 
 	promCfg := configmaps.GetPrometheusConfig(tenant, accessKey, secretKey)
+	minIOScrapeConfigs := make([]configmaps.ScrapeConfig, len(promCfg.ScrapeConfigs))
+	copy(minIOScrapeConfigs, promCfg.ScrapeConfigs)
 
 	// If the secret is not found, create the secret
 	if k8serrors.IsNotFound(err) {
 		klog.Infof("Adding MinIO tenant %s/%s Prometheus scrape config", ns, tenant.Name)
-		scrapeCfgYaml, err := yaml.Marshal(&promCfg.ScrapeConfigs)
+		scrapeCfgYaml, err := yaml.Marshal(&minIOScrapeConfigs)
 		if err != nil {
 			return err
 		}
@@ -168,33 +170,47 @@ func (c *Controller) checkAndCreatePrometheusAddlConfig(ctx context.Context, ten
 		if err != nil {
 			return err
 		}
-		// get other scrape configs
+		// Get all non-MinIO scrape configs, as well as MinIO scrape configs for other tenants
 		for _, sc := range scrapeConfigs {
 			if !strings.HasPrefix(sc.JobName, tenant.PrometheusOperatorAddlConfigJobName()) {
 				expectedScrapeConfigs = append(expectedScrapeConfigs, sc)
 			}
 		}
-		ignoreScrapeConfigsIndex := len(expectedScrapeConfigs)
-		expectedScrapeConfigs = append(expectedScrapeConfigs, promCfg.ScrapeConfigs...)
+
+		// Add the expected MinIO scrape configs for the current tenant
+		expectedScrapeConfigs = append(expectedScrapeConfigs, minIOScrapeConfigs...)
+
+		// Verify that all expected scrape configs are present in the installed configs
 		updateScrapeConfig := false
 		if len(scrapeConfigs) != len(expectedScrapeConfigs) {
 			updateScrapeConfig = true
 		} else {
-			for i := range scrapeConfigs {
-				// can't compare that is generated by operator
-				if i < ignoreScrapeConfigsIndex {
-					continue
-				}
-				if scrapeConfigs[i].JobName != expectedScrapeConfigs[i].JobName ||
-					scrapeConfigs[i].MetricsPath != expectedScrapeConfigs[i].MetricsPath ||
-					scrapeConfigs[i].Scheme != expectedScrapeConfigs[i].Scheme ||
-					!reflect.DeepEqual(scrapeConfigs[i].TLSConfig, expectedScrapeConfigs[i].TLSConfig) ||
-					!reflect.DeepEqual(scrapeConfigs[i].StaticConfigs, expectedScrapeConfigs[i].StaticConfigs) {
-					updateScrapeConfig = true
-					break
+			// Verify that all MinIO scrape configs for the current tenant
+			// are included in the installed configs
+			for _, minIOConfig := range minIOScrapeConfigs {
+				configMatched := false
+				for _, config := range scrapeConfigs {
+					if minIOConfig.JobName == config.JobName {
+						if config.MetricsPath != minIOConfig.MetricsPath ||
+							config.Scheme != minIOConfig.Scheme ||
+							!reflect.DeepEqual(config.TLSConfig, minIOConfig.TLSConfig) ||
+							!reflect.DeepEqual(config.StaticConfigs, minIOConfig.StaticConfigs) {
+							updateScrapeConfig = true
+							break
+						}
+						accKey, err := miniov2.GetAccessKeyFromBearerToken(config.BearerToken, secretKey)
+						if err != nil || accKey != accessKey {
+							updateScrapeConfig = true
+							break
+						}
+
+						configMatched = true
+						break
+					}
 				}
-				accKey, _ := miniov2.GetAccessKeyFromBearerToken(scrapeConfigs[i].BearerToken, secretKey)
-				if accKey != accessKey {
+				// Update the installed scrape configs if any MinIO configs for
+				// the current tenant are missing
+				if !configMatched {
 					updateScrapeConfig = true
 					break
 				}
@@ -255,6 +271,7 @@ func (c *Controller) deletePrometheusAddlConfig(ctx context.Context, tenant *min
 		return err
 	}
 	for _, sc := range scrapeConfigs {
+		// Get all non-MinIO scrape configs, as well as MinIO scrape configs for other tenants
 		if !strings.HasPrefix(sc.JobName, tenant.PrometheusOperatorAddlConfigJobName()) {
 			exceptedScrapeConfigs = append(exceptedScrapeConfigs, sc)
 		}
diff --git a/resources/base/cluster-role.yaml b/resources/base/cluster-role.yaml
index 84bc6a06..dce73c50 100644
--- a/resources/base/cluster-role.yaml
+++ b/resources/base/cluster-role.yaml
@@ -154,8 +154,9 @@ rules:
       - monitoring.coreos.com
     resources:
       - prometheuses
+      - prometheusagents
     verbs:
-      - '*'
+      - "*"
   - apiGroups:
       - "coordination.k8s.io"
     resources:



Test patch:
diff --git a/pkg/controller/prometheus_test.go b/pkg/controller/prometheus_test.go
index c934e90d..d327adef 100644
--- a/pkg/controller/prometheus_test.go
+++ b/pkg/controller/prometheus_test.go
@@ -16,6 +16,7 @@ package controller
 
 import (
 	"context"
+	"fmt"
 	"testing"
 	"time"
 
@@ -38,26 +39,38 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 		changedTokenNumber int
 	}
 	type arg struct {
-		accessKey     string
-		secretKey     string
-		beforeScrapes []configmaps.ScrapeConfig
-		paths         []string
-		expects       expect
+		accessKey       string
+		secretKey       string
+		tenantName      string
+		tenantNamespace string
+		beforeScrapes   []configmaps.ScrapeConfig
+		paths           []string
+		expects         expect
 	}
 	type args struct {
 		name string
 		arg  arg
 	}
+
+	tenantAName := "tenantA"
+	tenantANamespace := "tenantA-ns"
+	tenantBName := "tenantB"
+	tenantBNamespace := "tenantB-ns"
+
+	// Next is to emulate a tenant with same name as tenantA (different namespace)
+	tenantCName := tenantAName
+	tenantCNamespace := "tenantC-ns"
+
 	testRunCheckAndCreatePrometheusAddlConfig := func(t *testing.T, arg arg) {
 		beforeScrapesBytes, err := yaml.Marshal(arg.beforeScrapes)
 		if err != nil {
 			t.Fatal(err)
 		}
-		testNS := "default"
+		testNS := arg.tenantNamespace
 		prometheus := &v1.Prometheus{
 			ObjectMeta: metav1.ObjectMeta{
 				Name:      "testPrometheus",
-				Namespace: testNS,
+				Namespace: "default",
 			},
 			Spec: v1.PrometheusSpec{
 				CommonPrometheusFields: v1.CommonPrometheusFields{
@@ -73,7 +86,7 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 		beforeSecret := &v3.Secret{
 			ObjectMeta: metav1.ObjectMeta{
 				Name:      miniov2.PrometheusAddlScrapeConfigSecret,
-				Namespace: testNS,
+				Namespace: "default",
 			},
 			Data: map[string][]byte{
 				miniov2.PrometheusAddlScrapeConfigKey: beforeScrapesBytes,
@@ -86,7 +99,7 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 		}
 		tenant := &miniov2.Tenant{
 			ObjectMeta: metav1.ObjectMeta{
-				Name:      "testTenant",
+				Name:      arg.tenantName,
 				Namespace: testNS,
 			},
 			Spec: miniov2.TenantSpec{
@@ -99,7 +112,7 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 		if err != nil {
 			t.Fatal(err)
 		}
-		afterSecret, err := kubeclient.CoreV1().Secrets(testNS).Get(context.Background(), miniov2.PrometheusAddlScrapeConfigSecret, metav1.GetOptions{})
+		afterSecret, err := kubeclient.CoreV1().Secrets("default").Get(context.Background(), miniov2.PrometheusAddlScrapeConfigSecret, metav1.GetOptions{})
 		if err != nil {
 			t.Fatal(err)
 		}
@@ -138,13 +151,17 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 		{
 			name: "testDefault",
 			arg: arg{
-				accessKey:     "accessKey",
-				secretKey:     "secretKey",
-				beforeScrapes: []configmaps.ScrapeConfig{},
+				accessKey:       "accessKey",
+				secretKey:       "secretKey",
+				beforeScrapes:   []configmaps.ScrapeConfig{},
+				tenantName:      tenantAName,
+				tenantNamespace: tenantANamespace,
 				expects: expect{
-					afterScrapeNumber:  1,
-					afterScrapePath:    []string{"/minio/v2/metrics/cluster"},
-					afterScrapeName:    []string{"testTenant-minio-job-0"},
+					afterScrapeNumber: 1,
+					afterScrapePath:   []string{"/minio/v2/metrics/cluster"},
+					afterScrapeName: []string{
+						fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
+					},
 					changedTokenNumber: 0,
 				},
 			},
@@ -152,14 +169,19 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 		{
 			name: "testFirst",
 			arg: arg{
-				accessKey:     "accessKey",
-				secretKey:     "secretKey",
-				paths:         []string{"/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
-				beforeScrapes: []configmaps.ScrapeConfig{},
+				accessKey:       "accessKey",
+				secretKey:       "secretKey",
+				paths:           []string{"/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
+				beforeScrapes:   []configmaps.ScrapeConfig{},
+				tenantName:      tenantAName,
+				tenantNamespace: tenantANamespace,
 				expects: expect{
-					afterScrapeNumber:  2,
-					afterScrapePath:    []string{"/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
-					afterScrapeName:    []string{"testTenant-minio-job-0", "testTenant-minio-job-1"},
+					afterScrapeNumber: 2,
+					afterScrapePath:   []string{"/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
+					afterScrapeName: []string{
+						fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
+						fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
+					},
 					changedTokenNumber: 0,
 				},
 			},
@@ -180,10 +202,16 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 						},
 					},
 				},
+				tenantName:      tenantAName,
+				tenantNamespace: tenantANamespace,
 				expects: expect{
-					afterScrapeNumber:  3,
-					afterScrapePath:    []string{"", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
-					afterScrapeName:    []string{"testTarget", "testTenant-minio-job-0", "testTenant-minio-job-1"},
+					afterScrapeNumber: 3,
+					afterScrapePath:   []string{"", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
+					afterScrapeName: []string{
+						"testTarget",
+						fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
+						fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
+					},
 					changedTokenNumber: 0,
 				},
 			},
@@ -204,7 +232,7 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 						},
 					},
 					{
-						JobName:     "testTenant-minio-job-0",
+						JobName:     fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
 						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
 						MetricsPath: "/minio/v2/metrics/cluster",
 						Scheme:      "https",
@@ -213,12 +241,12 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 						},
 						StaticConfigs: []configmaps.StaticConfig{
 							{
-								Targets: []string{"minio.default.svc.cluster.local:443"},
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantANamespace)},
 							},
 						},
 					},
 					{
-						JobName:     "testTenant-minio-job-1",
+						JobName:     fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
 						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
 						MetricsPath: "/minio/metrics/v3/api",
 						Scheme:      "https",
@@ -227,15 +255,21 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 						},
 						StaticConfigs: []configmaps.StaticConfig{
 							{
-								Targets: []string{"minio.default.svc.cluster.local:443"},
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantANamespace)},
 							},
 						},
 					},
 				},
+				tenantName:      tenantAName,
+				tenantNamespace: tenantANamespace,
 				expects: expect{
-					afterScrapeNumber:  3,
-					afterScrapePath:    []string{"", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
-					afterScrapeName:    []string{"testTarget", "testTenant-minio-job-0", "testTenant-minio-job-1"},
+					afterScrapeNumber: 3,
+					afterScrapePath:   []string{"", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
+					afterScrapeName: []string{
+						"testTarget",
+						fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
+						fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
+					},
 					changedTokenNumber: 0,
 				},
 			},
@@ -256,7 +290,7 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 						},
 					},
 					{
-						JobName:     "testTenant-minio-job-0",
+						JobName:     fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
 						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
 						MetricsPath: "/minio/v2/metrics/cluster",
 						Scheme:      "https",
@@ -265,12 +299,12 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 						},
 						StaticConfigs: []configmaps.StaticConfig{
 							{
-								Targets: []string{"minio.default.svc.cluster.local:443"},
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantANamespace)},
 							},
 						},
 					},
 					{
-						JobName:     "testTenant-minio-job-1",
+						JobName:     fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
 						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
 						MetricsPath: "/minio/metrics/v3/api",
 						Scheme:      "https",
@@ -279,19 +313,233 @@ func Test_checkAndCreatePrometheusAddlConfig(t *testing.T) {
 						},
 						StaticConfigs: []configmaps.StaticConfig{
 							{
-								Targets: []string{"minio.default.svc.cluster.local:443"},
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantANamespace)},
 							},
 						},
 					},
 				},
+				tenantName:      tenantAName,
+				tenantNamespace: tenantANamespace,
 				expects: expect{
-					afterScrapeNumber:  3,
-					afterScrapePath:    []string{"", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
-					afterScrapeName:    []string{"testTarget", "testTenant-minio-job-0", "testTenant-minio-job-1"},
+					afterScrapeNumber: 3,
+					afterScrapePath:   []string{"", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
+					afterScrapeName: []string{
+						"testTarget",
+						fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
+						fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
+					},
 					changedTokenNumber: 2,
 				},
 			},
 		},
+		{
+			name: "testMultiTenantDifferentName",
+			arg: arg{
+				accessKey: "accessKey",
+				secretKey: "secretKey",
+				paths:     []string{"/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
+				beforeScrapes: []configmaps.ScrapeConfig{
+					{
+						JobName: "testTarget",
+						StaticConfigs: []configmaps.StaticConfig{
+							{
+								Targets: []string{"testTarget"},
+							},
+						},
+					},
+					{
+						JobName:     fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
+						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
+						MetricsPath: "/minio/v2/metrics/cluster",
+						Scheme:      "https",
+						TLSConfig: configmaps.TLSConfig{
+							CAFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
+						},
+						StaticConfigs: []configmaps.StaticConfig{
+							{
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantANamespace)},
+							},
+						},
+					},
+					{
+						JobName:     fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
+						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
+						MetricsPath: "/minio/metrics/v3/api",
+						Scheme:      "https",
+						TLSConfig: configmaps.TLSConfig{
+							CAFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
+						},
+						StaticConfigs: []configmaps.StaticConfig{
+							{
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantANamespace)},
+							},
+						},
+					},
+				},
+				tenantName:      tenantBName,
+				tenantNamespace: tenantBNamespace,
+				expects: expect{
+					afterScrapeNumber: 5,
+					afterScrapePath:   []string{"", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
+					afterScrapeName: []string{
+						"testTarget",
+						fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
+						fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
+						fmt.Sprintf("%s-%s-minio-job-0", tenantBName, tenantBNamespace),
+						fmt.Sprintf("%s-%s-minio-job-1", tenantBName, tenantBNamespace),
+					},
+					changedTokenNumber: 0,
+				},
+			},
+		},
+		{
+			name: "testMultiTenantSameName",
+			arg: arg{
+				accessKey: "accessKey",
+				secretKey: "secretKey",
+				paths:     []string{"/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
+				beforeScrapes: []configmaps.ScrapeConfig{
+					{
+						JobName: "testTarget",
+						StaticConfigs: []configmaps.StaticConfig{
+							{
+								Targets: []string{"testTarget"},
+							},
+						},
+					},
+					{
+						JobName:     fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
+						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
+						MetricsPath: "/minio/v2/metrics/cluster",
+						Scheme:      "https",
+						TLSConfig: configmaps.TLSConfig{
+							CAFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
+						},
+						StaticConfigs: []configmaps.StaticConfig{
+							{
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantANamespace)},
+							},
+						},
+					},
+					{
+						JobName:     fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
+						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
+						MetricsPath: "/minio/metrics/v3/api",
+						Scheme:      "https",
+						TLSConfig: configmaps.TLSConfig{
+							CAFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
+						},
+						StaticConfigs: []configmaps.StaticConfig{
+							{
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantANamespace)},
+							},
+						},
+					},
+				},
+				tenantName:      tenantCName,
+				tenantNamespace: tenantCNamespace,
+				expects: expect{
+					afterScrapeNumber: 5,
+					afterScrapePath:   []string{"", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
+					afterScrapeName: []string{
+						"testTarget",
+						fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
+						fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
+						fmt.Sprintf("%s-%s-minio-job-0", tenantCName, tenantCNamespace),
+						fmt.Sprintf("%s-%s-minio-job-1", tenantCName, tenantCNamespace),
+					},
+					changedTokenNumber: 0,
+				},
+			},
+		},
+		{
+			name: "testMultiTenantSameNameNoChange",
+			arg: arg{
+				accessKey: "accessKey",
+				secretKey: "secretKey",
+				paths:     []string{"/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
+				beforeScrapes: []configmaps.ScrapeConfig{
+					{
+						JobName: "testTarget",
+						StaticConfigs: []configmaps.StaticConfig{
+							{
+								Targets: []string{"testTarget"},
+							},
+						},
+					},
+					{
+						JobName:     fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
+						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
+						MetricsPath: "/minio/v2/metrics/cluster",
+						Scheme:      "https",
+						TLSConfig: configmaps.TLSConfig{
+							CAFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
+						},
+						StaticConfigs: []configmaps.StaticConfig{
+							{
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantANamespace)},
+							},
+						},
+					},
+					{
+						JobName:     fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
+						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
+						MetricsPath: "/minio/metrics/v3/api",
+						Scheme:      "https",
+						TLSConfig: configmaps.TLSConfig{
+							CAFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
+						},
+						StaticConfigs: []configmaps.StaticConfig{
+							{
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantANamespace)},
+							},
+						},
+					},
+					{
+						JobName:     fmt.Sprintf("%s-%s-minio-job-0", tenantCName, tenantCNamespace),
+						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
+						MetricsPath: "/minio/v2/metrics/cluster",
+						Scheme:      "https",
+						TLSConfig: configmaps.TLSConfig{
+							CAFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
+						},
+						StaticConfigs: []configmaps.StaticConfig{
+							{
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantCNamespace)},
+							},
+						},
+					},
+					{
+						JobName:     fmt.Sprintf("%s-%s-minio-job-1", tenantCName, tenantCNamespace),
+						BearerToken: (&miniov2.Tenant{}).GenBearerToken("accessKey", "secretKey"),
+						MetricsPath: "/minio/metrics/v3/api",
+						Scheme:      "https",
+						TLSConfig: configmaps.TLSConfig{
+							CAFile: "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
+						},
+						StaticConfigs: []configmaps.StaticConfig{
+							{
+								Targets: []string{fmt.Sprintf("minio.%s.svc.cluster.local:443", tenantCNamespace)},
+							},
+						},
+					},
+				},
+				tenantName:      tenantAName,
+				tenantNamespace: tenantANamespace,
+				expects: expect{
+					afterScrapeNumber: 5,
+					afterScrapePath:   []string{"", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api", "/minio/v2/metrics/cluster", "/minio/metrics/v3/api"},
+					afterScrapeName: []string{
+						"testTarget",
+						fmt.Sprintf("%s-%s-minio-job-0", tenantAName, tenantANamespace),
+						fmt.Sprintf("%s-%s-minio-job-1", tenantAName, tenantANamespace),
+						fmt.Sprintf("%s-%s-minio-job-0", tenantCName, tenantCNamespace),
+						fmt.Sprintf("%s-%s-minio-job-1", tenantCName, tenantCNamespace),
+					},
+					changedTokenNumber: 0,
+				},
+			},
+		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {