Repository: rook/rook
Pull request #11895 (2023-03-13 16:07:16)

docs: an update to disaster recovery with new scenario
<!-- Please take a look at our [Contributing](https://rook.io/docs/rook/latest/Contributing/development-flow/)
documentation before submitting a Pull Request!
Thank you for contributing to Rook! -->

**Description of your changes:**

add recovering steps when rook crs and ceph daemons are all deleted.
here is the context: https://github.com/rook/rook/discussions/11821

**Which issue is resolved by this Pull Request:**
Resolves #

**Checklist:**

- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/latest/Contributing/development-flow/#commit-structure)).
- [x] **Skip Tests for Docs**: If this is only a documentation change, add the label `skip-ci` on the PR.
- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/latest/Contributing/development-flow/#submitting-a-pull-request)
- [ ] [Pending release notes](https://github.com/rook/rook/blob/master/PendingReleaseNotes.md) updated with breaking and/or notable changes for the next minor release.
- [ ] Documentation has been updated, if necessary.
- [ ] Unit tests have been added, if necessary.
- [ ] Integration tests have been added, if necessary.


gaord: > @gaord That's great to hear you were able to fully recover your cluster even after the entire rook namespace was deleted. We have wanted to document this scenario for a long time as seen in #7154 and [rook/kubectl-rook-ceph#74](https://github.com/rook/kubectl-rook-ceph/issues/74). It's great to see the steps were not even too complicated.
> 
> @gauravsitlani Could you repro this recover to get more validation since you have been investigating it?
> 
> I will have some suggestions on the formatting of the document, but will wait first to see if it's possible for someone else to verify the steps.
> 
> Thanks for the contribution!

Glad to be able to contribute to ceph/rook community. any questions over the course of repro?  
gaord: > @travisn @gauravsitlani How should we proceed with this PR/ document?
> 
> The instructions look sane enough that after fixing up the formatting points, I think would be okay to merge and improve upon in future PRs. What do you think?
> 
> @gaord Are you available to update the formatting comments in the whole document?

@galexrt Done.
gaord: > @gaord can you share a demo / test steps if you can try reproducing and demonstrating here ?

The steps should be complete in general for restoring. Is there anything not working in the lab?
parth-gr: @travisn @subhamkrai can we have a krew command for this?
gaord: > Looks good, I just have some wording suggestions, thanks!

wording is polished as wish. @travisn 
travisn: > > Looks good, I just have some wording suggestions, thanks!
> 
> wording is polished as wish. @travisn

Thanks! Looks good to me, and agreed with @galexrt 's final suggestions.
gaord: > Some last nits from my side after that I'm good with merging it

nice formatting. done. @galexrt 

Patch:
diff --git a/.github/workflows/auto-assign.yaml b/.github/workflows/auto-assign.yaml
index 3da215ecb..5942656b3 100644
--- a/.github/workflows/auto-assign.yaml
+++ b/.github/workflows/auto-assign.yaml
@@ -10,7 +10,8 @@ jobs:
     steps:
       - name: take the issue
         uses: bdougie/take-action@main
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
         with:
           message: Thanks for taking this issue! Let us know if you have any questions!
           trigger: /assign
-          token: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index dd99a0259..34d0d7623 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -25,7 +25,7 @@ jobs:
         with:
           fetch-depth: 0
 
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v3
         with:
           go-version: 1.19
 
@@ -92,7 +92,7 @@ jobs:
           fetch-depth: 0
 
       - name: setup golang ${{ matrix.go-version }}
-        uses: actions/setup-go@v4
+        uses: actions/setup-go@v3
         with:
           go-version: ${{ matrix.go-version }}
 
diff --git a/.github/workflows/codegen.yml b/.github/workflows/codegen.yml
index d44f6f50d..794da24f1 100644
--- a/.github/workflows/codegen.yml
+++ b/.github/workflows/codegen.yml
@@ -29,7 +29,7 @@ jobs:
     runs-on: ubuntu-20.04
     if: "!contains(github.event.pull_request.labels.*.name, 'skip-ci')"
     steps:
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v3
         with:
           go-version: 1.19
 
diff --git a/.github/workflows/crds-gen.yml b/.github/workflows/crds-gen.yml
index c5fa464f6..5db14f3ec 100644
--- a/.github/workflows/crds-gen.yml
+++ b/.github/workflows/crds-gen.yml
@@ -28,7 +28,7 @@ jobs:
   crds-gen:
     runs-on: ubuntu-20.04
     steps:
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v3
         with:
           go-version: 1.19
 
diff --git a/.github/workflows/daily-nightly-jobs.yml b/.github/workflows/daily-nightly-jobs.yml
index fd1cac8e7..376b8274c 100644
--- a/.github/workflows/daily-nightly-jobs.yml
+++ b/.github/workflows/daily-nightly-jobs.yml
@@ -22,7 +22,7 @@ jobs:
           fetch-depth: 0
 
       - name: setup golang
-        uses: actions/setup-go@v4
+        uses: actions/setup-go@v3
         with:
           go-version: 1.19
 
diff --git a/.github/workflows/docs-check.yml b/.github/workflows/docs-check.yml
index bd6a7c03c..946efc0e1 100644
--- a/.github/workflows/docs-check.yml
+++ b/.github/workflows/docs-check.yml
@@ -25,7 +25,7 @@ jobs:
         with:
           fetch-depth: 0
 
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v3
         with:
           go-version: 1.19
 
diff --git a/.github/workflows/golangci-lint.yaml b/.github/workflows/golangci-lint.yaml
index 3626b2ff7..0d670a944 100644
--- a/.github/workflows/golangci-lint.yaml
+++ b/.github/workflows/golangci-lint.yaml
@@ -24,7 +24,7 @@ jobs:
     name: golangci-lint
     runs-on: ubuntu-20.04
     steps:
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v3
         with:
           go-version: 1.19
       - uses: actions/checkout@v3
@@ -34,13 +34,13 @@ jobs:
         uses: golangci/golangci-lint-action@v3
         with:
           # Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
-          version: v1.51.2
+          version: v1.48.0
 
           # Optional: working directory, useful for monorepos
           # working-directory: somedir
 
           # Optional: golangci-lint command line arguments.
-          args: -E gosec --timeout=10m
+          args: -E gosec --timeout=6m
 
           # Optional: show only new issues if it's a pull request. The default value is `false`.
           # only-new-issues: true
diff --git a/.github/workflows/helm-lint.yaml b/.github/workflows/helm-lint.yaml
index 71d98eb81..c56915e8a 100644
--- a/.github/workflows/helm-lint.yaml
+++ b/.github/workflows/helm-lint.yaml
@@ -40,7 +40,7 @@ jobs:
           python-version: 3.9
 
       - name: Set up chart-testing
-        uses: helm/chart-testing-action@v2.4.0
+        uses: helm/chart-testing-action@v2.3.1
 
       - name: Run chart-testing (lint)
         run: ct lint --charts=./deploy/charts/rook-ceph --validate-yaml=false --validate-maintainers=false
diff --git a/.github/workflows/mod-check.yml b/.github/workflows/mod-check.yml
index 1e2fa35ac..ebd1b0915 100644
--- a/.github/workflows/mod-check.yml
+++ b/.github/workflows/mod-check.yml
@@ -33,7 +33,7 @@ jobs:
         with:
           fetch-depth: 0
 
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v3
         with:
           go-version: 1.19
 
diff --git a/.github/workflows/push-build.yaml b/.github/workflows/push-build.yaml
index 051286f3a..9d0388977 100644
--- a/.github/workflows/push-build.yaml
+++ b/.github/workflows/push-build.yaml
@@ -25,7 +25,7 @@ jobs:
         with:
           fetch-depth: 0
 
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v3
         with:
           go-version: 1.19
 
@@ -42,7 +42,7 @@ jobs:
           password: ${{ secrets.DOCKER_PASSWORD }}
 
       - name: Configure AWS Credentials
-        uses: aws-actions/configure-aws-credentials@v2
+        uses: aws-actions/configure-aws-credentials@v1
         with:
           aws-access-key-id: ${{ secrets.AWS_USR }}
           aws-secret-access-key: ${{ secrets.AWS_PSW }}
diff --git a/.github/workflows/rbac-gen.yaml b/.github/workflows/rbac-gen.yaml
index bc2023ad3..30bc74eff 100644
--- a/.github/workflows/rbac-gen.yaml
+++ b/.github/workflows/rbac-gen.yaml
@@ -28,7 +28,7 @@ jobs:
   gen-rbac:
     runs-on: ubuntu-20.04
     steps:
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v3
         with:
           go-version: 1.19
 
diff --git a/.github/workflows/stale.yaml b/.github/workflows/stale.yaml
index d94d9e0c3..337ac1b2a 100644
--- a/.github/workflows/stale.yaml
+++ b/.github/workflows/stale.yaml
@@ -15,7 +15,7 @@ jobs:
     runs-on: ubuntu-20.04
     if: github.repository == 'rook/rook'
     steps:
-      - uses: actions/stale@v8
+      - uses: actions/stale@v7
         with:
           repo-token: ${{ secrets.GITHUB_TOKEN }}
           days-before-issue-stale: 60
diff --git a/.vscode/extensions.json b/.vscode/extensions.json
index 572161f64..9ab6622d1 100644
--- a/.vscode/extensions.json
+++ b/.vscode/extensions.json
@@ -6,8 +6,7 @@
   "recommendations": [
     "golang.go",
     "redhat.vscode-yaml",
-    "yzhang.markdown-all-in-one",
-    "timonwong.shellcheck"
+    "yzhang.markdown-all-in-one"
   ],
   // List of extensions recommended by VS Code that should not be recommended for users of this workspace.
   "unwantedRecommendations": []
diff --git a/Documentation/CRDs/Cluster/ceph-cluster-crd.md b/Documentation/CRDs/Cluster/ceph-cluster-crd.md
index 02b0fd530..acebd3f8d 100755
--- a/Documentation/CRDs/Cluster/ceph-cluster-crd.md
+++ b/Documentation/CRDs/Cluster/ceph-cluster-crd.md
@@ -45,24 +45,21 @@ If this value is empty, each pod will get an ephemeral directory to store their
     * `port`: Allows to change the default port where the dashboard is served
     * `ssl`: Whether to serve the dashboard via SSL, ignored on Ceph versions older than `13.2.2`
 * `monitoring`: Settings for monitoring Ceph using Prometheus. To enable monitoring on your cluster see the [monitoring guide](../../Storage-Configuration/Monitoring/ceph-monitoring.md#prometheus-alerts).
-    * `enabled`: Whether to enable prometheus based monitoring for external or internal cluster
+    * `enabled`: Whether to enable prometheus based monitoring for this cluster
     * `externalMgrEndpoints`: external cluster manager endpoints
     * `externalMgrPrometheusPort`: external prometheus manager module port. See [external cluster configuration](#external-cluster) for more details.
     * `rulesNamespace`: Namespace to deploy prometheusRule. If empty, namespace of the cluster will be used.
       Recommended:
         * If you have a single Rook cluster, set the `rulesNamespace` to the same namespace as the cluster or keep it empty.
         * If you have multiple Rook clusters in the same Kubernetes cluster, choose the same namespace to set `rulesNamespace` for all the clusters (ideally, namespace with prometheus deployed). Otherwise, you will get duplicate alerts with duplicate alert definitions.
-    * `port`: The internal prometheus manager module port where the prometheus mgr module listens. The port may need to be configured when host networking is enabled.
-    * `interval`: The interval for the prometheus module to to scrape targets.
 * `network`: For the network settings for the cluster, refer to the [network configuration settings](#network-configuration-settings)
 * `mon`: contains mon related options [mon settings](#mon-settings)
 For more details on the mons and when to choose a number other than `3`, see the [mon health doc](../../Storage-Configuration/Advanced/ceph-mon-health.md).
 * `mgr`: manager top level section
     * `count`: set number of ceph managers between `1` to `2`. The default value is 2.
-    If there are two managers, it is important for all mgr services point to the active mgr and not the standby mgr. Rook automatically
-	updates the label `mgr_role` on the mgr pods to be either `active` or `standby`. Therefore, services need just to add the label
-    `mgr_role=active` to their selector to point to the active mgr. This applies to all services that rely on the ceph mgr such as
-	the dashboard or the prometheus metrics collector.
+    If there are two managers, it is important for all mgr services point to the active mgr and not the passive mgr. Therefore, Rook will
+    automatically update all services (in the cluster namespace) that have a label `app=rook-ceph-mgr` with a selector pointing to the
+    active mgr. This commonly applies to services for the dashboard or the prometheus metrics collector.
     * `modules`: is the list of Ceph manager modules to enable
 * `crashCollector`: The settings for crash collector daemon(s).
     * `disable`: is set to `true`, the crash collector will not run on any node where a Ceph daemon runs
diff --git a/Documentation/CRDs/ceph-nfs-crd.md b/Documentation/CRDs/ceph-nfs-crd.md
index c1c4add25..ea7ec4a65 100644
--- a/Documentation/CRDs/ceph-nfs-crd.md
+++ b/Documentation/CRDs/ceph-nfs-crd.md
@@ -101,8 +101,6 @@ The `server` spec sets configuration for Rook-created NFS-Ganesha server pods.
 * `logLevel`: The log level that NFS-Ganesha servers should output.</br>
   Default value: `NIV_INFO`</br>
   Supported values: `NIV_NULL | NIV_FATAL | NIV_MAJ | NIV_CRIT | NIV_WARN | NIV_EVENT | NIV_INFO | NIV_DEBUG | NIV_MID_DEBUG | NIV_FULL_DEBUG | NB_LOG_LEVEL`
-* `hostNetwork`: Whether host networking is enabled for the NFS server pod(s). If not set, the network
-  settings from the CephCluster CR will be applied.
 
 ### Security
 
diff --git a/Documentation/Contributing/development-flow.md b/Documentation/Contributing/development-flow.md
index cb015c814..bb8588676 100644
--- a/Documentation/Contributing/development-flow.md
+++ b/Documentation/Contributing/development-flow.md
@@ -332,43 +332,3 @@ The flow for getting a fix into a release branch is:
 3. After your PR is merged to master, the mergify bot will automatically open a PR with your commits backported to the release branch
 4. If there are any conflicts you will need to resolve them by pulling the branch, resolving the conflicts and force push back the branch
 5. After the CI is green, the bot will automatically merge the backport PR.
-
-## Testing changes and debugging issues in Ceph manager modules
-
-The Ceph manager modules are written in Python and can be individually and dynamically loaded from the manager. We can take advantage of this feature in order to test changes and to debug issues in the modules.
-This is just a hack to test/debug quickly and easily any modification in the manager modules.
-
-The ceph dashboard and the rook orchestrator modules are probably the two modules more prone to have modifications affecting the rook cluster, and therefore it is interesting to know how to debug and test changes easily in these modules.
-
-The way to proceed is to make the modification directly in the manager module and reload it:
-
-1. Update the cluster so only a single mgr pod is running. Set the `mgr.count: 1` in the CephCluster CR if it is not already.
-
-Now shell into the manager container:
-```console
-kubectl exec -n rook-ceph --stdin --tty $(kubectl get pod -n rook-ceph -l ceph_daemon_type=mgr,instance=a  -o jsonpath='{.items[0].metadata.name}') -c mgr  -- /bin/bash
-```
-
-2. Make the modifications needed in the required manager module.
-
-The source code of all the manager modules is in:
-```console
-/usr/share/ceph/mgr/
-```
-
-Enter the folder of the manager module to make your changes.
-
-!!! Note
-    If the manager pod is restarted for whatever reason, all the modifications made in the mgr container will be lost.
-
-3. Restart the manager module modified to test/debug the modifications:
-
-Example for restarting the rook manager module with the [krew plugin](https://github.com/rook/kubectl-rook-ceph):
-```console
-kubectl rook-ceph ceph mgr module disable rook
-kubectl rook-ceph ceph mgr module enable rook
-```
-
-Once the module is restarted the modifications will be running in the active manager, and you will be able to test/debug them.  
-Is a good practice to have the manager container log always visible to realize immediately any kind of problem during the restart of the module (maybe a syntax error?), and to debug your changes.
-
diff --git a/Documentation/Helm-Charts/operator-chart.md b/Documentation/Helm-Charts/operator-chart.md
index 7a2e6e2d2..f72511e2f 100644
--- a/Documentation/Helm-Charts/operator-chart.md
+++ b/Documentation/Helm-Charts/operator-chart.md
@@ -54,7 +54,6 @@ The following table lists the configurable parameters of the rook-operator chart
 | `crds.enabled` | Whether the helm chart should create and update the CRDs. If false, the CRDs must be managed independently with deploy/examples/crds.yaml. **WARNING** Only set during first deployment. If later disabled the cluster may be DESTROYED. If the CRDs are deleted in this case, see [the disaster recovery guide](https://rook.io/docs/rook/latest/Troubleshooting/disaster-recovery/#restoring-crds-after-deletion) to restore them. | `true` |
 | `csi.allowUnsupportedVersion` | Allow starting an unsupported ceph-csi image | `false` |
 | `csi.attacher.image` | Kubernetes CSI Attacher image | `registry.k8s.io/sig-storage/csi-attacher:v4.1.0` |
-| `csi.cephFSAttachRequired` | Whether to skip any attach operation altogether for CephFS PVCs. See more details [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object). If cephFSAttachRequired is set to false it skips the volume attachments and makes the creation of pods using the CephFS PVC fast. **WARNING** It's highly discouraged to use this for CephFS RWO volumes. Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details. | `true` |
 | `csi.cephFSFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted. supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html | `"File"` |
 | `csi.cephFSKernelMountOptions` | Set CephFS Kernel mount options to use https://docs.ceph.com/en/latest/man/8/mount.ceph/#options. Set to "ms_mode=secure" when connections.encrypted is enabled in CephCluster CR | `nil` |
 | `csi.cephFSPluginUpdateStrategy` | CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate | `RollingUpdate` |
@@ -94,7 +93,6 @@ The following table lists the configurable parameters of the rook-operator chart
 | `csi.kubeletDirPath` | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag) | `/var/lib/kubelet` |
 | `csi.logLevel` | Set logging level for cephCSI containers maintained by the cephCSI. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. | `0` |
 | `csi.nfs.enabled` | Enable the nfs csi driver | `false` |
-| `csi.nfsAttachRequired` | Whether to skip any attach operation altogether for NFS PVCs. See more details [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object). If cephFSAttachRequired is set to false it skips the volume attachments and makes the creation of pods using the NFS PVC fast. **WARNING** It's highly discouraged to use this for NFS RWO volumes. Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details. | `true` |
 | `csi.nfsFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the NFS PVC is being mounted. supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html | `"File"` |
 | `csi.nfsPluginUpdateStrategy` | CSI NFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate | `RollingUpdate` |
 | `csi.nfsPodLabels` | Labels to add to the CSI NFS Deployments and DaemonSets Pods | `nil` |
@@ -106,7 +104,6 @@ The following table lists the configurable parameters of the rook-operator chart
 | `csi.provisionerPriorityClassName` | PriorityClassName to be set on csi driver provisioner pods | `"system-cluster-critical"` |
 | `csi.provisionerReplicas` | Set replicas for csi provisioner deployment | `2` |
 | `csi.provisionerTolerations` | Array of tolerations in YAML format which will be added to CSI provisioner deployment | `nil` |
-| `csi.rbdAttachRequired` | Whether to skip any attach operation altogether for RBD PVCs. See more details [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object). If set to false it skips the volume attachments and makes the creation of pods using the RBD PVC fast. **WARNING** It's highly discouraged to use this for RWO volumes as it can cause data corruption. csi-addons operations like Reclaimspace and PVC Keyrotation will also not be supported if set to false since we'll have no VolumeAttachments to determine which node the PVC is mounted on. Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details. | `true` |
 | `csi.rbdFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted. supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html | `"File"` |
 | `csi.rbdGrpcMetricsPort` | Ceph CSI RBD driver GRPC metrics port | `9090` |
 | `csi.rbdLivenessMetricsPort` | Ceph CSI RBD driver metrics port | `8080` |
diff --git a/Documentation/Troubleshooting/disaster-recovery.md b/Documentation/Troubleshooting/disaster-recovery.md
index 5d094aa75..3b488b682 100644
--- a/Documentation/Troubleshooting/disaster-recovery.md
+++ b/Documentation/Troubleshooting/disaster-recovery.md
@@ -276,3 +276,94 @@ and deleting the other deployments. An example command to do this is `k -n rook-
 1. Copy the endpoints configmap from the old cluster: `rook-ceph-mon-endpoints`
 1. Scale the rook operator up again : `kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas 1`
 1. Wait until the reconciliation is over.
+
+## Restoring the Rook cluster after the Rook namespace is deleted
+
+When the rook-ceph namespace is accidentally deleted, the good news is that the cluster can be restored. With the content in the directory `dataDirHostPath` and the original OSD disks, the ceph cluster could be restored with this guide.
+
+You need to manually create a ConfigMap and a Secret to make it work. The information required for the ConfigMap and Secret can be found in the `dataDirHostPath` directory.
+
+The first resource is the secret named `rook-ceph-mon` as seen in this example below:
+
+```yaml
+apiVersion: v1
+data:
+  ceph-secret: QVFCZ0h6VmorcVNhSGhBQXVtVktNcjcrczNOWW9Oa2psYkErS0E9PQ==
+  ceph-username: Y2xpZW50LmFkbWlu
+  fsid: M2YyNzE4NDEtNjE4OC00N2MxLWIzZmQtOTBmZDRmOTc4Yzc2
+  mon-secret: QVFCZ0h6VmorcVNhSGhBQXVtVktNcjcrczNOWW9Oa2psYkErS0E9PQ==
+kind: Secret
+metadata:
+  finalizers:
+  - ceph.rook.io/disaster-protection
+  name: rook-ceph-mon
+  namespace: rook-ceph
+  ownerReferences: null
+type: kubernetes.io/rook
+
+```
+
+The values for the secret can be found in `$dataDirHostPath/rook-ceph/client.admin.keyring` and `$dataDirHostPath/rook-ceph/rook-ceph.config`.
+- `ceph-secret` and `mon-secret` are to be filled with the `client.admin`'s keyring contents.
+- `ceph-username`: set to the string `client.admin`
+- `fsid`: set to the original ceph cluster id.
+
+All the fields in data section need to be encoded in base64. Coding could be done like this:
+```console
+echo -n "string to code" | base64 -i -
+```
+Now save the secret as `rook-ceph-mon.yaml`, to be created later in the restore.
+
+The second resource is the configmap named rook-ceph-mon-endpoints as seen in this example below:
+
+```yaml
+apiVersion: v1
+data:
+  csi-cluster-config-json: '[{"clusterID":"rook-ceph","monitors":["169.169.241.153:6789","169.169.82.57:6789","169.169.7.81:6789"],"namespace":""}]'
+  data: k=169.169.241.153:6789,m=169.169.82.57:6789,o=169.169.7.81:6789
+  mapping: '{"node":{"k":{"Name":"10.138.55.111","Hostname":"10.138.55.111","Address":"10.138.55.111"},"m":{"Name":"10.138.55.120","Hostname":"10.138.55.120","Address":"10.138.55.120"},"o":{"Name":"10.138.55.112","Hostname":"10.138.55.112","Address":"10.138.55.112"}}}'
+  maxMonId: "15"
+kind: ConfigMap
+metadata:
+  finalizers:
+  - ceph.rook.io/disaster-protection
+  name: rook-ceph-mon-endpoints
+  namespace: rook-ceph
+  ownerReferences: null
+```
+
+The Monitor's service IPs are kept in the monitor data store and you need to create them by original ones. After you create this configmap with the original service IPs, the rook operator will create the correct services for you with IPs matching in the monitor data store. Along with monitor ids, their service IPs and mapping relationship of them can be found in dataDirHostPath/rook-ceph/rook-ceph.config, for example:
+
+```console
+[global]
+fsid                = 3f271841-6188-47c1-b3fd-90fd4f978c76
+mon initial members = m o k
+mon host            = [v2:169.169.82.57:3300,v1:169.169.82.57:6789],[v2:169.169.7.81:3300,v1:169.169.7.81:6789],[v2:169.169.241.153:3300,v1:169.169.241.153:6789]
+```
+
+`mon initial members` and `mon host` are holding sequences of monitors' id and IP respectively; the sequence are going in the same order among monitors as a result you can tell which monitors have which service IP addresses. Modify your `rook-ceph-mon-endpoints.yaml` on fields `csi-cluster-config-json` and `data` based on the understanding of `rook-ceph.config` above.
+The field `mapping` tells rook where to schedule monitor's pods. you could search in `dataDirHostPath` in all Ceph cluster hosts for `mon-m,mon-o,mon-k`. If you find `mon-m` in host `10.138.55.120`, you should fill `10.138.55.120` in field `mapping` for `m`. Others are the same.
+Update the `maxMonId` to be the max numeric ID of the highest monitor ID. For example, 15 is the 0-based ID for mon `o`.
+Now save this configmap in the file rook-ceph-mon-endpoints.yaml, to be created later in the restore.
+
+Now that you have the info for the secret and the configmap, you are ready to restore the running cluster.
+
+Deploy Rook Ceph using the YAML files or Helm, with the same settings you had previously.
+
+```console
+kubectl create -f crds.yaml -f common.yaml -f operator.yaml
+```
+
+After the operator is running, create the configmap and secret you have just crafted:
+
+```console
+kubectl create -f rook-ceph-mon.yaml -f rook-ceph-mon-endpoints.yaml
+```
+
+Create your Ceph cluster CR (if possible, with the same settings as existed previously):
+
+```console
+kubectl create -f cluster.yaml
+```
+
+Now your Rook Ceph cluster should be running again.
diff --git a/build/csv/csv-gen.sh b/build/csv/csv-gen.sh
index a75c67ea2..e2e4efeb8 100755
--- a/build/csv/csv-gen.sh
+++ b/build/csv/csv-gen.sh
@@ -54,6 +54,4 @@ function generate_csv() {
     rm -rf "../../build/csv/ceph/$PLATFORM"
 }
 
-if [ "$PLATFORM" == "amd64" ]; then
-    generate_csv
-fi
+generate_csv
diff --git a/build/makelib/common.mk b/build/makelib/common.mk
index a59e7e75c..99244430c 100644
--- a/build/makelib/common.mk
+++ b/build/makelib/common.mk
@@ -24,13 +24,6 @@ else
 $(error "please install 'shasum' or 'sha256sum'")
 endif
 
-ifeq ($(origin DOCKERCMD),undefined)
-DOCKERCMD?=$(shell docker version >/dev/null 2>&1 && echo docker)
-ifeq ($(DOCKERCMD),)
-DOCKERCMD=$(shell podman version >/dev/null 2>&1 && echo podman)
-endif
-endif
-
 ifeq ($(origin PLATFORM), undefined)
 ifeq ($(origin GOOS), undefined)
 GOOS := $(shell go env GOOS)
diff --git a/build/release/Makefile b/build/release/Makefile
index a771a5d86..64b8196cd 100644
--- a/build/release/Makefile
+++ b/build/release/Makefile
@@ -31,7 +31,7 @@ ifneq ($(TAG_WITH_SUFFIX),true)
 override VERSION := $(shell echo "$(VERSION)" | sed -e 's/-alpha.0//' -e 's/-beta.0//' -e 's/-rc.0//')
 endif
 
-DOCS_PREFIX ?= docs/rook
+DOCS_PREFIX := docs/rook
 
 ifeq ($(shell echo $(BRANCH_NAME)),master)
 DOCS_VERSION := latest
@@ -45,7 +45,15 @@ else
 DOCS_VERSION_ALIAS :=
 endif
 
-DOCS_GIT_REPO ?= git@github.com:rook/rook.github.io.git
+DOCS_DIR ?= $(ROOT_DIR)/Documentation
+DOCS_WORK_DIR := $(WORK_DIR)/rook.github.io
+DOCS_VERSION_DIR := $(DOCS_WORK_DIR)/docs/rook/$(DOCS_VERSION)
+
+ifdef GIT_API_TOKEN
+DOCS_GIT_REPO := https://$(GIT_API_TOKEN)@github.com/rook/rook.github.io.git
+else
+DOCS_GIT_REPO := git@github.com:rook/rook.github.io.git
+endif
 
 ifeq ($(origin BRANCH_NAME), undefined)
 BRANCH_NAME := $(shell git branch | grep \* | cut -d ' ' -f2)
@@ -168,22 +176,22 @@ promote.output:
 # 1: registry 2: image, 3: arch
 define repo.targets
 build.image.$(1).$(2).$(3):
-	$(DOCKERCMD) tag $(BUILD_REGISTRY)/$(2)-$(3) $(1)/$(2)-$(3):$(VERSION)
+	docker tag $(BUILD_REGISTRY)/$(2)-$(3) $(1)/$(2)-$(3):$(VERSION)
 	@# Save image as _output/images/linux_<arch>/<image>.tar.gz (no builds for darwin or windows)
 	mkdir -p $(OUTPUT_DIR)/images/linux_$(3)
-	$(DOCKERCMD) save $(BUILD_REGISTRY)/$(2)-$(3) | gzip -c > $(OUTPUT_DIR)/images/linux_$(3)/$(2).tar.gz
+	docker save $(BUILD_REGISTRY)/$(2)-$(3) | gzip -c > $(OUTPUT_DIR)/images/linux_$(3)/$(2).tar.gz
 build.all.images: build.image.$(1).$(2).$(3)
-publish.image.$(1).$(2).$(3): ; @$(DOCKERCMD) push $(1)/$(2)-$(3):$(VERSION)
+publish.image.$(1).$(2).$(3): ; @docker push $(1)/$(2)-$(3):$(VERSION)
 publish.all.images: publish.image.$(1).$(2).$(3)
 # tag the master image, but do not tag the release image with a generic channel tag
 promote.image.$(1).$(2).$(3):
-	$(DOCKERCMD) pull $(1)/$(2)-$(3):$(VERSION)
-	$(DOCKERCMD) tag $(1)/$(2)-$(3):$(VERSION) $(1)/$(2)-$(3):$(CHANNEL)
-	[ "$(CHANNEL)" = "release" ] || $(DOCKERCMD) push $(1)/$(2)-$(3):$(CHANNEL)
+	docker pull $(1)/$(2)-$(3):$(VERSION)
+	docker tag $(1)/$(2)-$(3):$(VERSION) $(1)/$(2)-$(3):$(CHANNEL)
+	[ "$(CHANNEL)" = "release" ] || docker push $(1)/$(2)-$(3):$(CHANNEL)
 promote.all.images: promote.image.$(1).$(2).$(3)
 clean.image.$(1).$(2).$(3):
-	[ -z "$$$$($(DOCKERCMD) images -q $(1)/$(2)-$(3):$(VERSION))" ] || $(DOCKERCMD) rmi $(1)/$(2)-$(3):$(VERSION)
-	[ -z "$$$$($(DOCKERCMD) images -q $(1)/$(2)-$(3):$(CHANNEL))" ] || $(DOCKERCMD) rmi $(1)/$(2)-$(3):$(CHANNEL)
+	[ -z "$$$$(docker images -q $(1)/$(2)-$(3):$(VERSION))" ] || docker rmi $(1)/$(2)-$(3):$(VERSION)
+	[ -z "$$$$(docker images -q $(1)/$(2)-$(3):$(CHANNEL))" ] || docker rmi $(1)/$(2)-$(3):$(CHANNEL)
 clean.all.images: clean.image.$(1).$(2).$(3)
 endef
 $(foreach r,$(REGISTRIES), $(foreach i,$(IMAGES), $(foreach a,$(IMAGE_ARCHS),$(eval $(call repo.targets,$(r),$(i),$(a))))))
diff --git a/cmd/rook/ceph/mgr.go b/cmd/rook/ceph/mgr.go
index 334c2ccd9..99e774af2 100644
--- a/cmd/rook/ceph/mgr.go
+++ b/cmd/rook/ceph/mgr.go
@@ -96,13 +96,12 @@ func runMgrSidecar(cmd *cobra.Command, args []string) error {
 	clusterInfo.CephVersion = *version
 
 	m := mgr.New(context, &clusterInfo, clusterSpec, "")
-	prevActiveMgr := "unknown"
 	for {
-		prevActiveMgr, err = m.UpdateActiveMgrLabel(daemonName, prevActiveMgr)
+		err := m.ReconcileActiveMgrServices(daemonName)
 		if err != nil {
 			logger.Errorf("failed to reconcile services. %v", err)
 		} else {
-			logger.Infof("successfully checked mgr_role label. checking again in %ds", (int)(interval.Seconds()))
+			logger.Infof("successfully reconciled services. checking again in %ds", (int)(interval.Seconds()))
 		}
 		time.Sleep(interval)
 	}
diff --git a/deploy/charts/library/templates/_cluster-role.tpl b/deploy/charts/library/templates/_cluster-role.tpl
index 9006e2335..13de8de58 100644
--- a/deploy/charts/library/templates/_cluster-role.tpl
+++ b/deploy/charts/library/templates/_cluster-role.tpl
@@ -99,12 +99,9 @@ rules:
     resources:
       - deployments/scale
       - deployments
-      - pods
     verbs:
       - patch
       - delete
-      - get
-      - update
   - apiGroups:
       - ''
     resources:
diff --git a/deploy/charts/rook-ceph/templates/clusterrole.yaml b/deploy/charts/rook-ceph/templates/clusterrole.yaml
index 830798983..f2af4fb7c 100644
--- a/deploy/charts/rook-ceph/templates/clusterrole.yaml
+++ b/deploy/charts/rook-ceph/templates/clusterrole.yaml
@@ -441,12 +441,6 @@ rules:
   - apiGroups: [""]
     resources: ["persistentvolumeclaims/status"]
     verbs: ["patch"]
-  - apiGroups: ["storage.k8s.io"]
-    resources: ["volumeattachments"]
-    verbs: ["get", "list", "watch", "patch"]
-  - apiGroups: ["storage.k8s.io"]
-    resources: ["volumeattachments/status"]
-    verbs: ["patch"]
 ---
 # TODO: remove this, once https://github.com/rook/rook/issues/10141
 # is resolved.
@@ -598,7 +592,7 @@ rules:
     verbs: ["create"]
   - apiGroups: [""]
     resources: ["nodes"]
-    verbs: ["get", "list", "watch"]
+    verbs: ["get", "list", watch"]
   - apiGroups: ["storage.k8s.io"]
     resources: ["csinodes"]
     verbs: ["get", "list", "watch"]
diff --git a/deploy/charts/rook-ceph/templates/configmap.yaml b/deploy/charts/rook-ceph/templates/configmap.yaml
index cc312f55a..aada7be63 100644
--- a/deploy/charts/rook-ceph/templates/configmap.yaml
+++ b/deploy/charts/rook-ceph/templates/configmap.yaml
@@ -232,7 +232,4 @@ data:
 {{- if .Values.csi.csiCephFSPluginVolumeMount }}
   CSI_CEPHFS_PLUGIN_VOLUME_MOUNT: {{ toYaml .Values.csi.csiCephFSPluginVolumeMount | quote }}
 {{- end }}
-  CSI_CEPHFS_ATTACH_REQUIRED: {{ .Values.csi.cephFSAttachRequired | quote }}
-  CSI_RBD_ATTACH_REQUIRED: {{ .Values.csi.rbdAttachRequired | quote }}
-  CSI_NFS_ATTACH_REQUIRED: {{ .Values.csi.nfsAttachRequired | quote }}
 {{- end }}
diff --git a/deploy/charts/rook-ceph/templates/resources.yaml b/deploy/charts/rook-ceph/templates/resources.yaml
index 0dd90f80d..72dd27a02 100644
--- a/deploy/charts/rook-ceph/templates/resources.yaml
+++ b/deploy/charts/rook-ceph/templates/resources.yaml
@@ -1867,14 +1867,6 @@ spec:
                       maximum: 65535
                       minimum: 0
                       type: integer
-                    interval:
-                      description: Interval determines prometheus scrape interval
-                      type: string
-                    port:
-                      description: Port is the prometheus server port
-                      maximum: 65535
-                      minimum: 0
-                      type: integer
                   type: object
                 network:
                   description: Network related configuration
@@ -2569,6 +2561,7 @@ spec:
                           description: Enabled represents whether the key rotation is enabled.
                           type: boolean
                         schedule:
+                          default: '@weekly'
                           description: Schedule represents the cron schedule for key rotation.
                           type: string
                       type: object
@@ -10988,10 +10981,6 @@ spec:
                       nullable: true
                       type: object
                       x-kubernetes-preserve-unknown-fields: true
-                    hostNetwork:
-                      description: Whether host networking is enabled for the Ganesha server. If not set, the network settings from the cluster CR will be applied.
-                      nullable: true
-                      type: boolean
                     labels:
                       additionalProperties:
                         type: string
@@ -11955,11 +11944,6 @@ spec:
                       description: The name of the secret that stores custom ca-bundle with root and intermediate certificates.
                       nullable: true
                       type: string
-                    dashboardEnabled:
-                      description: Whether rgw dashboard is enabled for the rgw daemon. If not set, the rgw dashboard will be enabled.
-                      nullable: true
-                      type: boolean
-                      x-kubernetes-preserve-unknown-fields: true
                     externalRgwEndpoints:
                       description: ExternalRgwEndpoints points to external RGW endpoint(s). Multiple endpoints can be given, but for stability of ObjectBucketClaims, we highly recommend that users give only a single external RGW endpoint that is a load balancer that sends requests to the multiple RGWs.
                       items:
@@ -13028,6 +13012,7 @@ spec:
                           description: Enabled represents whether the key rotation is enabled.
                           type: boolean
                         schedule:
+                          default: '@weekly'
                           description: Schedule represents the cron schedule for key rotation.
                           type: string
                       type: object
diff --git a/deploy/charts/rook-ceph/values.yaml b/deploy/charts/rook-ceph/values.yaml
index cfc940eef..624adac06 100644
--- a/deploy/charts/rook-ceph/values.yaml
+++ b/deploy/charts/rook-ceph/values.yaml
@@ -378,14 +378,6 @@ csi:
         limits:
           memory: 1Gi
           cpu: 500m
-    - name : csi-attacher
-      resource:
-        requests:
-          memory: 512Mi
-          cpu: 250m
-        limits:
-          memory: 1Gi
-          cpu: 500m
 
   # -- CEPH CSI NFS plugin resource requirement list
   # @default -- see values.yaml
@@ -537,27 +529,6 @@ csi:
     # @default -- labels listed [here](../CRDs/Cluster/ceph-cluster-crd.md#osd-topology)
     crushLocationLabels:
 
-  # -- Whether to skip any attach operation altogether for CephFS PVCs. See more details
-  # [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object).
-  # If cephFSAttachRequired is set to false it skips the volume attachments and makes the creation
-  # of pods using the CephFS PVC fast. **WARNING** It's highly discouraged to use this for
-  # CephFS RWO volumes. Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details.
-  cephFSAttachRequired: true
-  # -- Whether to skip any attach operation altogether for RBD PVCs. See more details
-  # [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object).
-  # If set to false it skips the volume attachments and makes the creation of pods using the RBD PVC fast.
-  # **WARNING** It's highly discouraged to use this for RWO volumes as it can cause data corruption.
-  # csi-addons operations like Reclaimspace and PVC Keyrotation will also not be supported if set
-  # to false since we'll have no VolumeAttachments to determine which node the PVC is mounted on.
-  # Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details.
-  rbdAttachRequired: true
-  # -- Whether to skip any attach operation altogether for NFS PVCs. See more details
-  # [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object).
-  # If cephFSAttachRequired is set to false it skips the volume attachments and makes the creation
-  # of pods using the NFS PVC fast. **WARNING** It's highly discouraged to use this for
-  # NFS RWO volumes. Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details.
-  nfsAttachRequired: true
-
 # -- Enable discovery daemon
 enableDiscoveryDaemon: false
 
diff --git a/deploy/examples/common.yaml b/deploy/examples/common.yaml
index fea047050..811a49066 100644
--- a/deploy/examples/common.yaml
+++ b/deploy/examples/common.yaml
@@ -156,7 +156,7 @@ rules:
     verbs: ["create"]
   - apiGroups: [""]
     resources: ["nodes"]
-    verbs: ["get", "list", "watch"]
+    verbs: ["get", "list", watch"]
   - apiGroups: ["storage.k8s.io"]
     resources: ["csinodes"]
     verbs: ["get", "list", "watch"]
@@ -793,12 +793,9 @@ rules:
     resources:
       - deployments/scale
       - deployments
-      - pods
     verbs:
       - patch
       - delete
-      - get
-      - update
   - apiGroups:
       - ''
     resources:
diff --git a/deploy/examples/crds.yaml b/deploy/examples/crds.yaml
index ea2eea9ab..84837caf6 100644
--- a/deploy/examples/crds.yaml
+++ b/deploy/examples/crds.yaml
@@ -1865,14 +1865,6 @@ spec:
                       maximum: 65535
                       minimum: 0
                       type: integer
-                    interval:
-                      description: Interval determines prometheus scrape interval
-                      type: string
-                    port:
-                      description: Port is the prometheus server port
-                      maximum: 65535
-                      minimum: 0
-                      type: integer
                   type: object
                 network:
                   description: Network related configuration
@@ -2567,6 +2559,7 @@ spec:
                           description: Enabled represents whether the key rotation is enabled.
                           type: boolean
                         schedule:
+                          default: '@weekly'
                           description: Schedule represents the cron schedule for key rotation.
                           type: string
                       type: object
@@ -10982,10 +10975,6 @@ spec:
                       nullable: true
                       type: object
                       x-kubernetes-preserve-unknown-fields: true
-                    hostNetwork:
-                      description: Whether host networking is enabled for the Ganesha server. If not set, the network settings from the cluster CR will be applied.
-                      nullable: true
-                      type: boolean
                     labels:
                       additionalProperties:
                         type: string
@@ -11947,11 +11936,6 @@ spec:
                       description: The name of the secret that stores custom ca-bundle with root and intermediate certificates.
                       nullable: true
                       type: string
-                    dashboardEnabled:
-                      description: Whether rgw dashboard is enabled for the rgw daemon. If not set, the rgw dashboard will be enabled.
-                      nullable: true
-                      type: boolean
-                      x-kubernetes-preserve-unknown-fields: true
                     externalRgwEndpoints:
                       description: ExternalRgwEndpoints points to external RGW endpoint(s). Multiple endpoints can be given, but for stability of ObjectBucketClaims, we highly recommend that users give only a single external RGW endpoint that is a load balancer that sends requests to the multiple RGWs.
                       items:
@@ -13020,6 +13004,7 @@ spec:
                           description: Enabled represents whether the key rotation is enabled.
                           type: boolean
                         schedule:
+                          default: '@weekly'
                           description: Schedule represents the cron schedule for key rotation.
                           type: string
                       type: object
diff --git a/deploy/examples/create-external-cluster-resources.py b/deploy/examples/create-external-cluster-resources.py
index 39472cd8f..51d53e6fa 100644
--- a/deploy/examples/create-external-cluster-resources.py
+++ b/deploy/examples/create-external-cluster-resources.py
@@ -38,13 +38,13 @@ ModuleNotFoundError = ImportError
 try:
     import rados
 except ModuleNotFoundError as noModErr:
-    print(f"Error: {noModErr}\nExiting the script...")
+    print("Error: %s\nExiting the script..." % noModErr)
     sys.exit(1)
 
 try:
     import rbd
 except ModuleNotFoundError as noModErr:
-    print(f"Error: {noModErr}\nExiting the script...")
+    print("Error: %s\nExiting the script..." % noModErr)
     sys.exit(1)
 
 try:
@@ -190,7 +190,11 @@ class DummyRados(object):
         json_cmd = json.loads(cmd)
         json_cmd_str = json.dumps(json_cmd, sort_keys=True)
         cmd_output = self.cmd_output_map[json_cmd_str]
-        return self.return_val, cmd_output, str(self.err_message.encode("utf-8"))
+        return (
+            self.return_val,
+            cmd_output,
+            "{}".format(self.err_message).encode("utf-8"),
+        )
 
     def _convert_hostname_to_ip(self, host_name):
         ip_reg_x = re.compile(r"\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}")
@@ -201,7 +205,9 @@ class DummyRados(object):
 
         host_ip = self.dummy_host_ip_map.get(host_name, "")
         if not host_ip:
-            host_ip = f"172.9.{random.randint(0, 254)}.{random.randint(0, 254)}"
+            host_ip = "172.9.{}.{}".format(
+                random.randint(0, 254), random.randint(0, 254)
+            )
             self.dummy_host_ip_map[host_name] = host_ip
         del random
         return host_ip
@@ -230,7 +236,7 @@ class S3Auth(AuthBase):
         signature = self.get_signature(r)
         if py3k:
             signature = signature.decode("utf-8")
-        r.headers["Authorization"] = f"AWS {self.access_key}:{signature}"
+        r.headers["Authorization"] = "AWS %s:%s" % (self.access_key, signature)
         return r
 
     def get_signature(self, r):
@@ -273,20 +279,20 @@ class S3Auth(AuthBase):
             if "x-amz-date" in interesting_headers:
                 interesting_headers["date"] = ""
 
-        buf = f"{method}\n"
+        buf = "%s\n" % method
         for key in sorted(interesting_headers.keys()):
             val = interesting_headers[key]
             if key.startswith("x-amz-"):
-                buf += f"{key}:{val}\n"
+                buf += "%s:%s\n" % (key, val)
             else:
-                buf += f"{val}\n"
+                buf += "%s\n" % val
 
         # append the bucket if it exists
         if bucket != "":
-            buf += f"/{bucket}"
+            buf += "/%s" % bucket
 
         # add the objectkey. even if it doesn't exist, add the slash
-        buf += f"/{objectkey}"
+        buf += "/%s" % objectkey
 
         return buf
 
@@ -453,7 +459,7 @@ class RadosJSON:
             args_to_parse = sys.argv[1:]
         return argP.parse_args(args_to_parse)
 
-    def validate_rbd_metadata_ec_pool_name(self):
+    def validate_rgw_metadata_ec_pool_name(self):
         if self._arg_parser.rbd_metadata_ec_pool_name:
             rbd_metadata_ec_pool_name = self._arg_parser.rbd_metadata_ec_pool_name
             rbd_pool_name = self._arg_parser.rbd_data_pool_name
@@ -472,8 +478,11 @@ class RadosJSON:
             ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
             if ret_val != 0 or len(json_out) == 0:
                 raise ExecutionFailureException(
-                    f"{cmd_json['prefix']} command failed.\n"
-                    f"Error: {err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST}"
+                    "{}".format(cmd_json["prefix"])
+                    + " command failed.\n"
+                    + "Error: {}".format(
+                        err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST
+                    )
                 )
             metadata_pool_exist, pool_exist = False, False
 
@@ -490,12 +499,15 @@ class RadosJSON:
 
             if not metadata_pool_exist:
                 raise ExecutionFailureException(
-                    "Provided rbd_ec_metadata_pool name,"
-                    f" {rbd_metadata_ec_pool_name}, does not exist"
+                    "Provided rbd_ec_metadata_pool name, {}, does not exist".format(
+                        rbd_metadata_ec_pool_name
+                    )
                 )
             if not pool_exist:
                 raise ExecutionFailureException(
-                    f"Provided rbd_data_pool name, {rbd_pool_name}, does not exist"
+                    "Provided rbd_data_pool name, {}, does not exist".format(
+                        rbd_pool_name
+                    )
                 )
             return rbd_metadata_ec_pool_name
 
@@ -520,36 +532,40 @@ class RadosJSON:
             ipv4, port = endpoint_str.split(":")
         except ValueError:
             raise ExecutionFailureException(
-                f"Not a proper endpoint: {endpoint_str}, <IPv4>:<PORT>, format is expected"
+                "Not a proper endpoint: {}, <IPv4>:<PORT>, format is expected".format(
+                    endpoint_str
+                )
             )
         ipParts = ipv4.split(".")
         if len(ipParts) != 4:
-            raise ExecutionFailureException(f"Not a valid IP address: {ipv4}")
+            raise ExecutionFailureException("Not a valid IP address: {}".format(ipv4))
         for eachPart in ipParts:
             if not eachPart.isdigit():
                 raise ExecutionFailureException(
-                    f"IP address parts should be numbers: {ipv4}"
+                    "IP address parts should be numbers: {}".format(ipv4)
                 )
             intPart = int(eachPart)
             if intPart < 0 or intPart > 254:
-                raise ExecutionFailureException(f"Out of range IP addresses: {ipv4}")
+                raise ExecutionFailureException(
+                    "Out of range IP addresses: {}".format(ipv4)
+                )
         if not port.isdigit():
-            raise ExecutionFailureException(f"Port not valid: {port}")
+            raise ExecutionFailureException("Port not valid: {}".format(port))
         intPort = int(port)
         if intPort < 1 or intPort > 2**16 - 1:
-            raise ExecutionFailureException(f"Out of range port number: {port}")
+            raise ExecutionFailureException("Out of range port number: {}".format(port))
         return False
 
     def endpoint_dial(self, endpoint_str, timeout=3, cert=None):
         # if the 'cluster' instance is a dummy one,
         # don't try to reach out to the endpoint
         if isinstance(self.cluster, DummyRados):
-            return "", "", ""
+            return
         protocols = ["http", "https"]
         response_error = None
         for prefix in protocols:
             try:
-                ep = f"{prefix}://{endpoint_str}"
+                ep = "{}://{}".format(prefix, endpoint_str)
                 verify = None
                 # If verify is set to a path to a directory,
                 # the directory must have been processed using the c_rehash utility supplied with OpenSSL.
@@ -562,17 +578,14 @@ class RadosJSON:
                 else:
                     r = requests.head(ep, timeout=timeout)
                 if r.status_code == 200:
-                    return prefix, verify, ""
+                    return prefix, verify
             except Exception as err:
                 response_error = err
                 continue
-        sys.stderr.write(
-            f"unable to connect to endpoint: {endpoint_str}, failed error: {response_error}"
-        )
-        return (
-            "",
-            "",
-            ("-1"),
+        raise ExecutionFailureException(
+            "unable to connect to endpoint: {}, failed error: {}".format(
+                endpoint_str, response_error
+            )
         )
 
     def __init__(self, arg_list=None):
@@ -617,10 +630,11 @@ class RadosJSON:
         cmd = json.dumps(cmd_json, sort_keys=True)
         ret_val, cmd_out, err_msg = self.cluster.mon_command(cmd, b"")
         if self._arg_parser.verbose:
-            print(f"Command Input: {cmd}")
+            print("Command Input: {}".format(cmd))
             print(
-                f"Return Val: {ret_val}\nCommand Output: {cmd_out}\n"
-                f"Error Message: {err_msg}\n----------\n"
+                "Return Val: {}\nCommand Output: {}\nError Message: {}\n----------\n".format(
+                    ret_val, cmd_out, err_msg
+                )
             )
         json_out = {}
         # if there is no error (i.e; ret_val is ZERO) and 'cmd_out' is not empty
@@ -638,7 +652,9 @@ class RadosJSON:
         if ret_val != 0 or len(json_out) == 0:
             raise ExecutionFailureException(
                 "'quorum_status' command failed.\n"
-                f"Error: {err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST}"
+                + "Error: {}".format(
+                    err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST
+                )
             )
         q_leader_name = json_out["quorum_leader_name"]
         q_leader_details = {}
@@ -659,15 +675,15 @@ class RadosJSON:
                 "Only 'v2' address type is enabled, user should also enable 'v1' type as well"
             )
         ip_port = str(q_leader_details["public_addr"].split("/")[0])
-        return f"{str(q_leader_name)}={ip_port}"
+        return "{}={}".format(str(q_leader_name), ip_port)
 
     def _join_host_port(self, endpoint, port):
-        port = str(port)
+        port = "{}".format(port)
         # regex to check the given endpoint is enclosed in square brackets
         ipv6_regex = re.compile(r"^\[[^]]*\]$")
         # endpoint has ':' in it and if not (already) enclosed in square brackets
         if endpoint.count(":") and not ipv6_regex.match(endpoint):
-            endpoint = f"[{endpoint}]"
+            endpoint = "[{}]".format(endpoint)
         if not port:
             return endpoint
         return ":".join([endpoint, port])
@@ -698,14 +714,16 @@ class RadosJSON:
             if ret_val != 0 or len(json_out) == 0:
                 raise ExecutionFailureException(
                     "'mgr services' command failed.\n"
-                    f"Error: {err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST}"
+                    + "Error: {}".format(
+                        err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST
+                    )
                 )
             monitoring_endpoint = (
                 json_out.get("mgrmap", {}).get("services", {}).get("prometheus", "")
             )
             if not monitoring_endpoint:
                 raise ExecutionFailureException(
-                    "'prometheus' service not found, is the exporter enabled?.\n"
+                    "'prometheus' service not found, is the exporter enabled?'.\n"
                 )
             # now check the stand-by mgr-s
             standby_arr = json_out.get("mgrmap", {}).get("standbys", [])
@@ -716,11 +734,11 @@ class RadosJSON:
                 parsed_endpoint = urlparse(monitoring_endpoint)
             except ValueError:
                 raise ExecutionFailureException(
-                    f"invalid endpoint: {monitoring_endpoint}"
+                    "invalid endpoint: {}".format(monitoring_endpoint)
                 )
             monitoring_endpoint_ip_list = parsed_endpoint.hostname
             if not monitoring_endpoint_port:
-                monitoring_endpoint_port = str(parsed_endpoint.port)
+                monitoring_endpoint_port = "{}".format(parsed_endpoint.port)
 
         # if monitoring endpoint port is not set, put a default mon port
         if not monitoring_endpoint_port:
@@ -749,23 +767,24 @@ class RadosJSON:
                 mgr_ips.append(self._convert_hostname_to_ip(each_standby_mgr))
         except:
             raise ExecutionFailureException(
-                f"Conversion of host: {failed_ip} to IP failed. "
-                "Please enter the IP addresses of all the ceph-mgrs with the '--monitoring-endpoint' flag"
+                "Conversion of host: {} to IP failed. "
+                "Please enter the IP addresses of all the ceph-mgrs with the '--monitoring-endpoint' flag".format(
+                    failed_ip
+                )
             )
         monitoring_endpoint = self._join_host_port(
             monitoring_endpoint_ip, monitoring_endpoint_port
         )
         self._invalid_endpoint(monitoring_endpoint)
-        _, _, err = self.endpoint_dial(monitoring_endpoint)
-        if err == "-1":
-            raise ExecutionFailureException(err)
+        self.endpoint_dial(monitoring_endpoint)
+
         # add the validated active mgr IP into the first index
         mgr_ips.insert(0, monitoring_endpoint_ip)
         all_mgr_ips_str = ",".join(mgr_ips)
         return all_mgr_ips_str, monitoring_endpoint_port
 
     def check_user_exist(self, user):
-        cmd_json = {"prefix": "auth get", "entity": f"{user}", "format": "json"}
+        cmd_json = {"prefix": "auth get", "entity": "{}".format(user), "format": "json"}
         ret_val, json_out, _ = self._common_cmd_json_gen(cmd_json)
         if ret_val != 0 or len(json_out) == 0:
             return ""
@@ -786,10 +805,12 @@ class RadosJSON:
                 )
             cephfs_filesystem = self._arg_parser.cephfs_filesystem_name
             if cephfs_filesystem == "":
-                entity = f"{entity}-{cluster_name}"
+                entity = "{}-{}".format(entity, cluster_name)
             else:
-                entity = f"{entity}-{cluster_name}-{cephfs_filesystem}"
-                caps["osd"] = f"allow rw tag cephfs metadata={cephfs_filesystem}"
+                entity = "{}-{}-{}".format(entity, cluster_name, cephfs_filesystem)
+                caps["osd"] = "allow rw tag cephfs metadata={}".format(
+                    cephfs_filesystem
+                )
 
         return caps, entity
 
@@ -809,10 +830,10 @@ class RadosJSON:
                 )
             cephfs_filesystem = self._arg_parser.cephfs_filesystem_name
             if cephfs_filesystem == "":
-                entity = f"{entity}-{cluster_name}"
+                entity = "{}-{}".format(entity, cluster_name)
             else:
-                entity = f"{entity}-{cluster_name}-{cephfs_filesystem}"
-                caps["osd"] = f"allow rw tag cephfs *={cephfs_filesystem}"
+                entity = "{}-{}-{}".format(entity, cluster_name, cephfs_filesystem)
+                caps["osd"] = "allow rw tag cephfs *={}".format(cephfs_filesystem)
 
         return caps, entity
 
@@ -830,8 +851,8 @@ class RadosJSON:
                 raise ExecutionFailureException(
                     "mandatory flags not found, please set the '--rbd-data-pool-name', '--cluster-name' flags"
                 )
-            entity = f"{entity}-{cluster_name}-{rbd_pool_name}"
-            caps["osd"] = f"profile rbd pool={rbd_pool_name}"
+            entity = "{}-{}-{}".format(entity, cluster_name, rbd_pool_name)
+            caps["osd"] = "profile rbd pool={}".format(rbd_pool_name)
 
         return caps, entity
 
@@ -848,8 +869,8 @@ class RadosJSON:
                 raise ExecutionFailureException(
                     "mandatory flags not found, please set the '--rbd-data-pool-name', '--cluster-name' flags"
                 )
-            entity = f"{entity}-{cluster_name}-{rbd_pool_name}"
-            caps["osd"] = f"profile rbd pool={rbd_pool_name}"
+            entity = "{}-{}-{}".format(entity, cluster_name, rbd_pool_name)
+            caps["osd"] = "profile rbd pool={}".format(rbd_pool_name)
 
         return caps, entity
 
@@ -872,8 +893,8 @@ class RadosJSON:
             return self.get_rbd_node_caps_and_entity()
 
         raise ExecutionFailureException(
-            f"no user found with user_name: {user_name}, "
-            "get_caps_and_entity command failed.\n"
+            "no user found with user_name: {} ,".format(user_name)
+            + "get_caps_and_entity command failed.\n"
         )
 
     def create_cephCSIKeyring_user(self, user):
@@ -903,18 +924,22 @@ class RadosJSON:
         # check if user already exist
         user_key = self.check_user_exist(entity)
         if user_key != "":
-            return user_key, f"{entity.split('.', 1)[1]}"
-            # entity.split('.',1)[1] to rename entity(client.csi-rbd-node) as csi-rbd-node
+            return user_key, "{}".format(
+                entity.split(".", 1)[1]
+            )  # entity.split('.',1)[1] to rename entity(client.csi-rbd-node) as csi-rbd-node
 
         ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
         # if there is an unsuccessful attempt,
         if ret_val != 0 or len(json_out) == 0:
             raise ExecutionFailureException(
-                f"'auth get-or-create {user}' command failed.\n"
-                f"Error: {err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST}"
+                "'auth get-or-create {}' command failed.\n".format(user)
+                + "Error: {}".format(
+                    err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST
+                )
             )
-        return str(json_out[0]["key"]), f"{entity.split('.', 1)[1]}"
-        # entity.split('.',1)[1] to rename entity(client.csi-rbd-node) as csi-rbd-node
+        return str(json_out[0]["key"]), "{}".format(
+            entity.split(".", 1)[1]
+        )  # entity.split('.',1)[1] to rename entity(client.csi-rbd-node) as csi-rbd-node
 
     def get_cephfs_data_pool_details(self):
         cmd_json = {"prefix": "fs ls", "format": "json"}
@@ -933,7 +958,7 @@ class RadosJSON:
             # '--cephfs-filesystem-name' or '--cephfs-data-pool-name' arguments,
             # raise an exception as we are unable to verify the args
             raise ExecutionFailureException(
-                f"'fs ls' ceph call failed with error: {err_msg}"
+                "'fs ls' ceph call failed with error: {}".format(err_msg)
             )
 
         matching_json_out = {}
@@ -949,8 +974,13 @@ class RadosJSON:
             # unable to find a matching fs-name, raise an error
             if len(matching_json_out_list) == 0:
                 raise ExecutionFailureException(
-                    f"Filesystem provided, '{self._arg_parser.cephfs_filesystem_name}', "
-                    f"is not found in the fs-list: {[str(x['name']) for x in json_out]}"
+                    (
+                        "Filesystem provided, '{}', "
+                        + "is not found in the fs-list: '{}'"
+                    ).format(
+                        self._arg_parser.cephfs_filesystem_name,
+                        [str(x["name"]) for x in json_out],
+                    )
                 )
             matching_json_out = matching_json_out_list[0]
         # if cephfs filesystem name is not provided,
@@ -969,8 +999,9 @@ class RadosJSON:
                 # if there is no matching fs exists, that means provided data_pool name is invalid
                 if not matching_json_out:
                     raise ExecutionFailureException(
-                        f"Provided data_pool name, {self._arg_parser.cephfs_data_pool_name},"
-                        " does not exists"
+                        "Provided data_pool name, {}, does not exists".format(
+                            self._arg_parser.cephfs_data_pool_name
+                        )
                     )
             # c. if nothing is set and couldn't find a default,
             else:
@@ -993,9 +1024,12 @@ class RadosJSON:
                     not in matching_json_out["data_pools"]
                 ):
                     raise ExecutionFailureException(
-                        f"Provided data-pool-name: '{self._arg_parser.cephfs_data_pool_name}', "
-                        "doesn't match from the data-pools list: "
-                        f"{[str(x) for x in matching_json_out['data_pools']]}"
+                        "{}: '{}', {}: {}".format(
+                            "Provided data-pool-name",
+                            self._arg_parser.cephfs_data_pool_name,
+                            "doesn't match from the data-pools' list",
+                            [str(x) for x in matching_json_out["data_pools"]],
+                        )
                     )
             # if data_pool name is not provided,
             # then try to find a default data pool name
@@ -1010,9 +1044,12 @@ class RadosJSON:
             # then warn the user that we are using the selected name
             if len(matching_json_out["data_pools"]) > 1:
                 print(
-                    "WARNING: Multiple data pools detected: "
-                    f"{[str(x) for x in matching_json_out['data_pools']]}\n"
-                    f"Using the data-pool: '{self._arg_parser.cephfs_data_pool_name}'\n"
+                    "{}: {}\n{}: '{}'\n".format(
+                        "WARNING: Multiple data pools detected",
+                        [str(x) for x in matching_json_out["data_pools"]],
+                        "Using the data-pool",
+                        self._arg_parser.cephfs_data_pool_name,
+                    )
                 )
 
     def create_checkerKey(self):
@@ -1044,8 +1081,10 @@ class RadosJSON:
         # if there is an unsuccessful attempt,
         if ret_val != 0 or len(json_out) == 0:
             raise ExecutionFailureException(
-                f"'auth get-or-create {self.run_as_user}' command failed\n"
-                f"Error: {err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST}"
+                "'auth get-or-create {}' command failed\n".format(self.run_as_user)
+                + "Error: {}".format(
+                    err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST
+                )
             )
         return str(json_out[0]["key"])
 
@@ -1093,18 +1132,16 @@ class RadosJSON:
                     output = subprocess.check_output(cmd, stderr=subprocess.PIPE)
                 except subprocess.CalledProcessError as execErr:
                     err_msg = (
-                        f"failed to execute command {cmd}. Output: {execErr.output}. "
-                        f"Code: {execErr.returncode}. Error: {execErr.stderr}"
+                        "failed to execute command %s. Output: %s. Code: %s. Error: %s"
+                        % (cmd, execErr.output, execErr.returncode, execErr.stderr)
                     )
-                    sys.stderr.write(err_msg)
-                    return None, None, False, "-1"
+                    raise Exception(err_msg)
             else:
                 err_msg = (
-                    f"failed to execute command {cmd}. Output: {execErr.output}. "
-                    f"Code: {execErr.returncode}. Error: {execErr.stderr}"
+                    "failed to execute command %s. Output: %s. Code: %s. Error: %s"
+                    % (cmd, execErr.output, execErr.returncode, execErr.stderr)
                 )
-                sys.stderr.write(err_msg)
-                return None, None, False, "-1"
+                raise Exception(err_msg)
 
         # if it is python2, don't check for ceph version for adding `info=read` cap(rgw_validation)
         if sys.version_info.major < 3:
@@ -1113,7 +1150,6 @@ class RadosJSON:
                 jsonoutput["keys"][0]["access_key"],
                 jsonoutput["keys"][0]["secret_key"],
                 False,
-                "",
             )
 
         # separately add info=read caps for rgw-endpoint ip validation
@@ -1139,18 +1175,16 @@ class RadosJSON:
                 info_cap_supported = False
             else:
                 err_msg = (
-                    f"failed to execute command {cmd}. Output: {execErr.output}. "
-                    f"Code: {execErr.returncode}. Error: {execErr.stderr}"
+                    "failed to execute command %s. Output: %s. Code: %s. Error: %s"
+                    % (cmd, execErr.output, execErr.returncode, execErr.stderr)
                 )
-                sys.stderr.write(err_msg)
-                return None, None, False, "-1"
+                raise Exception(err_msg)
 
         jsonoutput = json.loads(output)
         return (
             jsonoutput["keys"][0]["access_key"],
             jsonoutput["keys"][0]["secret_key"],
             info_cap_supported,
-            "",
         )
 
     def convert_fqdn_rgw_endpoint_to_ip(self, fqdn_rgw_endpoint):
@@ -1158,19 +1192,43 @@ class RadosJSON:
             fqdn, port = fqdn_rgw_endpoint.split(":")
         except ValueError:
             raise ExecutionFailureException(
-                f"Not a proper endpoint: {fqdn_rgw_endpoint}, "
-                "<FQDN>:<PORT>, format is expected"
+                "Not a proper endpoint: {}, <FQDN>:<PORT>, format is expected".format(
+                    fqdn_rgw_endpoint
+                )
             )
         rgw_endpoint_ip = self._convert_hostname_to_ip(fqdn)
         rgw_endpoint_port = port
         rgw_endpoint = self._join_host_port(rgw_endpoint_ip, rgw_endpoint_port)
         return rgw_endpoint
 
-    def validate_rbd_pool(self):
-        if not self.cluster.pool_exists(self._arg_parser.rbd_data_pool_name):
-            raise ExecutionFailureException(
-                f"The provided pool, '{self._arg_parser.rbd_data_pool_name}', does not exist"
-            )
+    def validate_pool(self):
+        pools_to_validate = [self._arg_parser.rbd_data_pool_name]
+        # if rgw_endpoint is provided, validate it
+        if self._arg_parser.rgw_endpoint:
+            rgw_endpoint = self._arg_parser.rgw_endpoint
+            self._invalid_endpoint(rgw_endpoint)
+            cert = None
+            if (
+                not self._arg_parser.rgw_skip_tls
+                and self.validate_rgw_endpoint_tls_cert()
+            ):
+                cert = self._arg_parser.rgw_tls_cert_path
+            self.endpoint_dial(rgw_endpoint, cert=cert)
+            # only validate if rgw_pool_prefix is passed else it will take default value and we don't create these default pools
+            if self._arg_parser.rgw_pool_prefix != "default":
+                rgw_pool_to_validate = [
+                    "{0}.rgw.meta".format(self._arg_parser.rgw_pool_prefix),
+                    ".rgw.root",
+                    "{0}.rgw.control".format(self._arg_parser.rgw_pool_prefix),
+                    "{0}.rgw.log".format(self._arg_parser.rgw_pool_prefix),
+                ]
+                pools_to_validate.extend(rgw_pool_to_validate)
+
+        for pool in pools_to_validate:
+            if not self.cluster.pool_exists(pool):
+                raise ExecutionFailureException(
+                    "The provided pool, '{}', does not exist".format(pool)
+                )
 
     def validate_rados_namespace(self):
         rbd_pool_name = self._arg_parser.rbd_data_pool_name
@@ -1181,8 +1239,9 @@ class RadosJSON:
         ioctx = self.cluster.open_ioctx(rbd_pool_name)
         if rbd_inst.namespace_exists(ioctx, rados_namespace) is False:
             raise ExecutionFailureException(
-                f"The provided rados Namespace, '{rados_namespace}', "
-                f"is not found in the pool '{rbd_pool_name}'"
+                (
+                    "The provided rados Namespace, '{}', is not found in the pool '{}'"
+                ).format(rados_namespace, rbd_pool_name)
             )
 
     def validate_subvolume_group(self):
@@ -1207,13 +1266,17 @@ class RadosJSON:
             _ = subprocess.check_output(cmd, stderr=subprocess.PIPE)
         except subprocess.CalledProcessError as execErr:
             raise ExecutionFailureException(
-                f"subvolume group {subvolume_group} passed doesn't exist"
+                "subvolume group {} passed doesn't exist".format(subvolume_group)
             )
 
-    def get_rgw_fsid(self, base_url, verify):
+    def get_rgw_fsid(self):
         access_key = self.out_map["RGW_ADMIN_OPS_USER_ACCESS_KEY"]
         secret_key = self.out_map["RGW_ADMIN_OPS_USER_SECRET_KEY"]
         rgw_endpoint = self._arg_parser.rgw_endpoint
+        cert = None
+        if not self._arg_parser.rgw_skip_tls and self.validate_rgw_endpoint_tls_cert():
+            cert = self._arg_parser.rgw_tls_cert_path
+        base_url, verify = self.endpoint_dial(rgw_endpoint, cert=cert)
         base_url = base_url + "://" + rgw_endpoint + "/admin/info?"
         params = {"format": "json"}
         request_url = base_url + urllib.parse.urlencode(params)
@@ -1225,64 +1288,30 @@ class RadosJSON:
                 verify=verify,
             )
         except requests.exceptions.Timeout:
-            sys.stderr.write(
-                f"invalid endpoint:, not able to call admin-ops api{rgw_endpoint}"
+            raise ExecutionFailureException(
+                "invalid endpoint:, not able to call admin-ops api{}".format(
+                    rgw_endpoint
+                )
             )
-            return "", "-1"
         r1 = r.json()
         if r1 is None or r1.get("info") is None:
-            sys.stderr.write(
-                f"The provided rgw Endpoint, '{self._arg_parser.rgw_endpoint}', is invalid."
-            )
-            return (
-                "",
-                "-1",
-            )
+            return ""  # Invalid rgw-endpoint exception will returned by validate_rgw_endpoint()
 
-        return r1["info"]["storage_backends"][0]["cluster_id"], ""
+        return r1["info"]["storage_backends"][0]["cluster_id"]
 
     def validate_rgw_endpoint(self):
         # if the 'cluster' instance is a dummy one,
         # don't try to reach out to the endpoint
         if isinstance(self.cluster, DummyRados):
             return
-
-        # check if the rgw endpoint is reachable
-        rgw_endpoint = self._arg_parser.rgw_endpoint
-        cert = None
-        if not self._arg_parser.rgw_skip_tls and self.validate_rgw_endpoint_tls_cert():
-            cert = self._arg_parser.rgw_tls_cert_path
-        base_url, verify, err = self.endpoint_dial(rgw_endpoint, cert=cert)
-        if err != "":
-            return "-1"
-
-        # check if the rgw endpoint is valid and belongs to the same cluster
         fsid = self.get_fsid()
-        rgw_fsid, err = self.get_rgw_fsid(base_url, verify)
-        if err == "-1":
-            return "-1"
+        rgw_fsid = self.get_rgw_fsid()
         if fsid != rgw_fsid:
-            sys.stderr.write(
-                f"The provided rgw Endpoint, '{self._arg_parser.rgw_endpoint}', is invalid. We are validating by calling the adminops api through rgw-endpoint and validating the cluster_id '{rgw_fsid}' is equal to the ceph cluster fsid '{fsid}'"
+            raise ExecutionFailureException(
+                (
+                    "The provided rgw Endpoint, '{}', is invalid. We are validating by calling the adminops api through rgw-endpoint and validating the cluster_id '{}' is equal to the ceph cluster fsid '{}'"
+                ).format(self._arg_parser.rgw_endpoint, rgw_fsid, fsid)
             )
-            return "-1"
-
-        # check if the rgw endpoint exist
-        # only validate if rgw_pool_prefix is passed else it will take default value and we don't create these default pools
-        if self._arg_parser.rgw_pool_prefix != "default":
-            rgw_pool_to_validate = [
-                f"{self._arg_parser.rgw_pool_prefix}.rgw.meta",
-                ".rgw.root",
-                f"{self._arg_parser.rgw_pool_prefix}.rgw.control",
-                f"{self._arg_parser.rgw_pool_prefix}.rgw.log",
-            ]
-            if not self.cluster.pool_exists(self._arg_parser.rgw_pool_to_validate):
-                sys.stderr.write(
-                    f"The provided pool, '{self._arg_parser.rgw_pool_to_validate}', does not exist"
-                )
-                return "-1"
-
-        return ""
 
     def validate_rgw_realm_name(self):
         if self._arg_parser.rgw_realm_name != "":
@@ -1294,11 +1323,11 @@ class RadosJSON:
                 self._arg_parser.rgw_realm_name,
             ]
             try:
-                _ = subprocess.check_output(cmd, stderr=subprocess.PIPE)
+                output = subprocess.check_output(cmd, stderr=subprocess.PIPE)
             except subprocess.CalledProcessError as execErr:
                 err_msg = (
-                    f"failed to execute command {cmd}. Output: {execErr.output}. "
-                    f"Code: {execErr.returncode}. Error: {execErr.stderr}"
+                    "failed to execute command %s. Output: %s. Code: %s. Error: %s"
+                    % (cmd, execErr.output, execErr.returncode, execErr.stderr)
                 )
                 raise Exception(err_msg)
 
@@ -1312,7 +1341,7 @@ class RadosJSON:
             self._arg_parser.rgw_endpoint = self.convert_fqdn_rgw_endpoint_to_ip(
                 self._arg_parser.rgw_endpoint
             )
-        self.validate_rbd_pool()
+        self.validate_pool()
         self.validate_rados_namespace()
         self.validate_subvolume_group()
         self.validate_rgw_realm_name()
@@ -1355,6 +1384,7 @@ class RadosJSON:
                 self.out_map["CSI_CEPHFS_PROVISIONER_SECRET"],
                 self.out_map["CSI_CEPHFS_PROVISIONER_SECRET_NAME"],
             ) = self.create_cephCSIKeyring_user("client.csi-cephfs-provisioner")
+        self.out_map["RGW_ENDPOINT"] = self._arg_parser.rgw_endpoint
         self.out_map["RGW_TLS_CERT"] = ""
         (
             self.out_map["MONITORING_ENDPOINT"],
@@ -1363,10 +1393,9 @@ class RadosJSON:
         self.out_map["RBD_POOL_NAME"] = self._arg_parser.rbd_data_pool_name
         self.out_map[
             "RBD_METADATA_EC_POOL_NAME"
-        ] = self.validate_rbd_metadata_ec_pool_name()
+        ] = self.validate_rgw_metadata_ec_pool_name()
         self.out_map["RGW_REALM_NAME"] = self._arg_parser.rgw_realm_name
         self.out_map["RGW_POOL_PREFIX"] = self._arg_parser.rgw_pool_prefix
-        self.out_map["RGW_ENDPOINT"] = ""
         if self._arg_parser.rgw_endpoint:
             if self._arg_parser.dry_run:
                 self.create_rgw_admin_ops_user()
@@ -1375,22 +1404,18 @@ class RadosJSON:
                     self.out_map["RGW_ADMIN_OPS_USER_ACCESS_KEY"],
                     self.out_map["RGW_ADMIN_OPS_USER_SECRET_KEY"],
                     info_cap_supported,
-                    err,
                 ) = self.create_rgw_admin_ops_user()
                 if info_cap_supported:
-                    err = self.validate_rgw_endpoint()
-                if self._arg_parser.rgw_tls_cert_path:
-                    self.out_map["RGW_TLS_CERT"] = self.validate_rgw_endpoint_tls_cert()
-                # if there is no error, set the RGW_ENDPOINT
-                if err != "-1":
-                    self.out_map["RGW_ENDPOINT"] = self._arg_parser.rgw_endpoint
+                    self.validate_rgw_endpoint()
+            if self._arg_parser.rgw_tls_cert_path:
+                self.out_map["RGW_TLS_CERT"] = self.validate_rgw_endpoint_tls_cert()
 
     def gen_shell_out(self):
         self._gen_output_map()
         shOutIO = StringIO()
         for k, v in self.out_map.items():
             if v and k not in self._excluded_keys:
-                shOutIO.write(f"export {k}={v}{LINESEP}")
+                shOutIO.write("export {}={}{}".format(k, v, LINESEP))
         shOut = shOutIO.getvalue()
         shOutIO.close()
         return shOut
@@ -1443,7 +1468,7 @@ class RadosJSON:
         ):
             json_out.append(
                 {
-                    "name": f"rook-{self.out_map['CSI_RBD_NODE_SECRET_NAME']}",
+                    "name": "rook-{}".format(self.out_map["CSI_RBD_NODE_SECRET_NAME"]),
                     "kind": "Secret",
                     "data": {
                         "userID": self.out_map["CSI_RBD_NODE_SECRET_NAME"],
@@ -1458,7 +1483,9 @@ class RadosJSON:
         ):
             json_out.append(
                 {
-                    "name": f"rook-{self.out_map['CSI_RBD_PROVISIONER_SECRET_NAME']}",
+                    "name": "rook-{}".format(
+                        self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"]
+                    ),
                     "kind": "Secret",
                     "data": {
                         "userID": self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"],
@@ -1473,7 +1500,9 @@ class RadosJSON:
         ):
             json_out.append(
                 {
-                    "name": f"rook-{self.out_map['CSI_CEPHFS_PROVISIONER_SECRET_NAME']}",
+                    "name": "rook-{}".format(
+                        self.out_map["CSI_CEPHFS_PROVISIONER_SECRET_NAME"]
+                    ),
                     "kind": "Secret",
                     "data": {
                         "adminID": self.out_map["CSI_CEPHFS_PROVISIONER_SECRET_NAME"],
@@ -1488,7 +1517,9 @@ class RadosJSON:
         ):
             json_out.append(
                 {
-                    "name": f"rook-{self.out_map['CSI_CEPHFS_NODE_SECRET_NAME']}",
+                    "name": "rook-{}".format(
+                        self.out_map["CSI_CEPHFS_NODE_SECRET_NAME"]
+                    ),
                     "kind": "Secret",
                     "data": {
                         "adminID": self.out_map["CSI_CEPHFS_NODE_SECRET_NAME"],
@@ -1514,11 +1545,17 @@ class RadosJSON:
                     "name": "ceph-rbd",
                     "kind": "StorageClass",
                     "data": {
-                        "dataPool": self.out_map["RBD_POOL_NAME"],
-                        "pool": self.out_map["RBD_METADATA_EC_POOL_NAME"],
-                        "csi.storage.k8s.io/provisioner-secret-name": f"rook-{self.out_map['CSI_RBD_PROVISIONER_SECRET_NAME']}",
-                        "csi.storage.k8s.io/controller-expand-secret-name": f"rook-{self.out_map['CSI_RBD_PROVISIONER_SECRET_NAME']}",
-                        "csi.storage.k8s.io/node-stage-secret-name": f"rook-{self.out_map['CSI_RBD_NODE_SECRET_NAME']}",
+                        "dataPool": self.out_map["RBD_METADATA_EC_POOL_NAME"],
+                        "pool": self.out_map["RBD_POOL_NAME"],
+                        "csi.storage.k8s.io/provisioner-secret-name": "rook-{}".format(
+                            self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"]
+                        ),
+                        "csi.storage.k8s.io/controller-expand-secret-name": "rook-{}".format(
+                            self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"]
+                        ),
+                        "csi.storage.k8s.io/node-stage-secret-name": "rook-{}".format(
+                            self.out_map["CSI_RBD_NODE_SECRET_NAME"]
+                        ),
                     },
                 }
             )
@@ -1529,9 +1566,15 @@ class RadosJSON:
                     "kind": "StorageClass",
                     "data": {
                         "pool": self.out_map["RBD_POOL_NAME"],
-                        "csi.storage.k8s.io/provisioner-secret-name": f"rook-{self.out_map['CSI_RBD_PROVISIONER_SECRET_NAME']}",
-                        "csi.storage.k8s.io/controller-expand-secret-name": f"rook-{self.out_map['CSI_RBD_PROVISIONER_SECRET_NAME']}",
-                        "csi.storage.k8s.io/node-stage-secret-name": f"rook-{self.out_map['CSI_RBD_NODE_SECRET_NAME']}",
+                        "csi.storage.k8s.io/provisioner-secret-name": "rook-{}".format(
+                            self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"]
+                        ),
+                        "csi.storage.k8s.io/controller-expand-secret-name": "rook-{}".format(
+                            self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"]
+                        ),
+                        "csi.storage.k8s.io/node-stage-secret-name": "rook-{}".format(
+                            self.out_map["CSI_RBD_NODE_SECRET_NAME"]
+                        ),
                     },
                 }
             )
@@ -1544,9 +1587,15 @@ class RadosJSON:
                     "data": {
                         "fsName": self.out_map["CEPHFS_FS_NAME"],
                         "pool": self.out_map["CEPHFS_POOL_NAME"],
-                        "csi.storage.k8s.io/provisioner-secret-name": f"rook-{self.out_map['CSI_CEPHFS_PROVISIONER_SECRET_NAME']}",
-                        "csi.storage.k8s.io/controller-expand-secret-name": f"rook-{self.out_map['CSI_CEPHFS_PROVISIONER_SECRET_NAME']}",
-                        "csi.storage.k8s.io/node-stage-secret-name": f"rook-{self.out_map['CSI_CEPHFS_NODE_SECRET_NAME']}",
+                        "csi.storage.k8s.io/provisioner-secret-name": "rook-{}".format(
+                            self.out_map["CSI_CEPHFS_PROVISIONER_SECRET_NAME"]
+                        ),
+                        "csi.storage.k8s.io/controller-expand-secret-name": "rook-{}".format(
+                            self.out_map["CSI_CEPHFS_PROVISIONER_SECRET_NAME"]
+                        ),
+                        "csi.storage.k8s.io/node-stage-secret-name": "rook-{}".format(
+                            self.out_map["CSI_CEPHFS_NODE_SECRET_NAME"]
+                        ),
                     },
                 }
             )
@@ -1600,10 +1649,10 @@ class RadosJSON:
 
     def upgrade_user_permissions(self, user):
         # check whether the given user exists or not
-        cmd_json = {"prefix": "auth get", "entity": f"{user}", "format": "json"}
+        cmd_json = {"prefix": "auth get", "entity": "{}".format(user), "format": "json"}
         ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
         if ret_val != 0 or len(json_out) == 0:
-            print(f"user {user} not found for upgrading.")
+            print("user {} not found for upgrading.".format(user))
             return
         existing_caps = json_out[0]["caps"]
         new_cap, _ = self.get_caps_and_entity(user)
@@ -1637,9 +1686,10 @@ class RadosJSON:
         ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
         if ret_val != 0:
             raise ExecutionFailureException(
-                f"'auth caps {user}' command failed.\n Error: {err_msg}"
+                "'auth caps {}' command failed.\n".format(user)
+                + "Error: {}".format(err_msg)
             )
-        print(f"Updated user {user} successfully.")
+        print("Updated user, {}, successfully.".format(user))
 
     def main(self):
         generated_output = ""
@@ -1651,9 +1701,9 @@ class RadosJSON:
             generated_output = self.gen_shell_out()
         else:
             raise ExecutionFailureException(
-                f"Unsupported format: {self._arg_parser.format}"
+                "Unsupported format: {}".format(self._arg_parser.format)
             )
-        print(generated_output)
+        print("{}".format(generated_output))
         if self.output_file and generated_output:
             fOut = open(self.output_file, mode="w", encoding="UTF-8")
             fOut.write(generated_output)
@@ -1668,11 +1718,11 @@ if __name__ == "__main__":
     try:
         rjObj.main()
     except ExecutionFailureException as err:
-        print(f"Execution Failed: {err}")
+        print("Execution Failed: {}".format(err))
         raise err
     except KeyError as kErr:
-        print(f"KeyError: {kErr}")
+        print("KeyError: %s", kErr)
     except OSError as osErr:
-        print(f"Error while trying to output the data: {osErr}")
+        print("Error while trying to output the data: {}".format(osErr))
     finally:
         rjObj.shutdown()
diff --git a/deploy/examples/csi/nfs/rbac.yaml b/deploy/examples/csi/nfs/rbac.yaml
index 029ec4458..5acf98a13 100644
--- a/deploy/examples/csi/nfs/rbac.yaml
+++ b/deploy/examples/csi/nfs/rbac.yaml
@@ -61,12 +61,6 @@ rules:
   - apiGroups: [""]
     resources: ["persistentvolumeclaims/status"]
     verbs: ["patch"]
-  - apiGroups: ["storage.k8s.io"]
-    resources: ["volumeattachments"]
-    verbs: ["get", "list", "watch", "patch"]
-  - apiGroups: ["storage.k8s.io"]
-    resources: ["volumeattachments/status"]
-    verbs: ["patch"]
 ---
 # TODO: remove this, once https://github.com/rook/rook/issues/10141
 # is resolved.
diff --git a/deploy/examples/dashboard-external-http.yaml b/deploy/examples/dashboard-external-http.yaml
index 82338367f..27fc59866 100644
--- a/deploy/examples/dashboard-external-http.yaml
+++ b/deploy/examples/dashboard-external-http.yaml
@@ -14,7 +14,6 @@ spec:
       targetPort: 7000
   selector:
     app: rook-ceph-mgr
-    mgr_role: active
     rook_cluster: rook-ceph
   sessionAffinity: None
   type: NodePort
diff --git a/deploy/examples/dashboard-external-https.yaml b/deploy/examples/dashboard-external-https.yaml
index 6a18dd30c..9d30fadca 100644
--- a/deploy/examples/dashboard-external-https.yaml
+++ b/deploy/examples/dashboard-external-https.yaml
@@ -14,7 +14,6 @@ spec:
       targetPort: 8443
   selector:
     app: rook-ceph-mgr
-    mgr_role: active
     rook_cluster: rook-ceph
   sessionAffinity: None
   type: NodePort
diff --git a/deploy/examples/dashboard-loadbalancer.yaml b/deploy/examples/dashboard-loadbalancer.yaml
index 19fe8eb5c..98c0b9075 100644
--- a/deploy/examples/dashboard-loadbalancer.yaml
+++ b/deploy/examples/dashboard-loadbalancer.yaml
@@ -14,7 +14,6 @@ spec:
       targetPort: 8443
   selector:
     app: rook-ceph-mgr
-    mgr_role: active
     rook_cluster: rook-ceph
   sessionAffinity: None
   type: LoadBalancer
diff --git a/deploy/examples/import-external-cluster.sh b/deploy/examples/import-external-cluster.sh
index c988a50a6..15f3479dc 100644
--- a/deploy/examples/import-external-cluster.sh
+++ b/deploy/examples/import-external-cluster.sh
@@ -20,8 +20,8 @@ ROOK_EXTERNAL_MAPPING={}
 RBD_STORAGE_CLASS_NAME=ceph-rbd
 CEPHFS_STORAGE_CLASS_NAME=cephfs
 ROOK_EXTERNAL_MONITOR_SECRET=mon-secret
-OPERATOR_NAMESPACE=rook-ceph                                 # default set to rook-ceph
-RBD_PROVISIONER=$OPERATOR_NAMESPACE".rbd.csi.ceph.com"       # driver:namespace:operator
+OPERATOR_NAMESPACE=rook-ceph # default set to rook-ceph
+RBD_PROVISIONER=$OPERATOR_NAMESPACE".rbd.csi.ceph.com" # driver:namespace:operator
 CEPHFS_PROVISIONER=$OPERATOR_NAMESPACE".cephfs.csi.ceph.com" # driver:namespace:operator
 CLUSTER_ID_RBD=$NAMESPACE
 CLUSTER_ID_CEPHFS=$NAMESPACE
@@ -41,6 +41,26 @@ function checkEnvVars() {
     echo "For a kernel earlier than 5.4 use a value of 'layering'; for 5.4 or later"
     echo "use 'layering,fast-diff,object-map,deep-flatten,exclusive-lock'"
     exit 1
+  fi
+   if [ -z "$RBD_POOL_NAME" ]; then
+    echo "Please populate the environment variable RBD_POOL_NAME"
+    exit 1
+  fi
+  if [ -z "$CSI_RBD_NODE_SECRET_NAME" ]; then
+    echo "Please populate the environment variable CSI_RBD_NODE_SECRET_NAME"
+    exit 1
+  fi
+  if [ -z "$CSI_RBD_PROVISIONER_SECRET_NAME" ]; then
+    echo "Please populate the environment variable CSI_RBD_PROVISIONER_SECRET_NAME"
+    exit 1
+  fi
+  if [ -z "$CSI_CEPHFS_NODE_SECRET_NAME" ]; then
+    echo "Please populate the environment variable CSI_CEPHFS_NODE_SECRET_NAME"
+    exit 1
+  fi
+  if [ -z "$CSI_CEPHFS_PROVISIONER_SECRET_NAME" ]; then
+    echo "Please populate the environment variable CSI_CEPHFS_PROVISIONER_SECRET_NAME"
+    exit 1
   fi
   if [ -z "$ROOK_EXTERNAL_FSID" ]; then
     echo "Please populate the environment variable ROOK_EXTERNAL_FSID"
@@ -59,8 +79,24 @@ function checkEnvVars() {
       echo "Please populate the environment variable ROOK_EXTERNAL_USERNAME"
       exit 1
     fi
+    if [ -z "$CSI_RBD_NODE_SECRET" ]; then
+      echo "Please populate the environment variable CSI_RBD_NODE_SECRET"
+      exit 1
+    fi
+    if [ -z "$CSI_RBD_PROVISIONER_SECRET" ]; then
+      echo "Please populate the environment variable CSI_RBD_PROVISIONER_SECRET"
+      exit 1
+    fi
+    if [ -z "$CSI_CEPHFS_NODE_SECRET" ]; then
+      echo "Please populate the environment variable CSI_CEPHFS_NODE_SECRET"
+      exit 1
+    fi
+    if [ -z "$CSI_CEPHFS_PROVISIONER_SECRET" ]; then
+      echo "Please populate the environment variable CSI_CEPHFS_PROVISIONER_SECRET"
+      exit 1
+    fi
   fi
-  if [[ "$ROOK_EXTERNAL_ADMIN_SECRET" != "admin-secret" ]] && [ -n "$ROOK_EXTERNAL_USER_SECRET" ]; then
+  if [[ "$ROOK_EXTERNAL_ADMIN_SECRET" != "admin-secret" ]] && [ -n "$ROOK_EXTERNAL_USER_SECRET" ] ; then
     echo "Providing both ROOK_EXTERNAL_ADMIN_SECRET and ROOK_EXTERNAL_USER_SECRET is not supported, choose one only."
     exit 1
   fi
@@ -77,86 +113,86 @@ function importClusterID() {
 
 function importSecret() {
   kubectl -n "$NAMESPACE" \
-    create \
-    secret \
-    generic \
-    --type="kubernetes.io/rook" \
-    "$MON_SECRET_NAME" \
-    --from-literal="$MON_SECRET_CLUSTER_NAME_KEYNAME"="$ROOK_EXTERNAL_CLUSTER_NAME" \
-    --from-literal="$MON_SECRET_FSID_KEYNAME"="$ROOK_EXTERNAL_FSID" \
-    --from-literal="$MON_SECRET_ADMIN_KEYRING_KEYNAME"="$ROOK_EXTERNAL_ADMIN_SECRET" \
-    --from-literal="$MON_SECRET_MON_KEYRING_KEYNAME"="$ROOK_EXTERNAL_MONITOR_SECRET" \
-    --from-literal="$MON_SECRET_CEPH_USERNAME_KEYNAME"="$ROOK_EXTERNAL_USERNAME" \
-    --from-literal="$MON_SECRET_CEPH_SECRET_KEYNAME"="$ROOK_EXTERNAL_USER_SECRET"
+  create \
+  secret \
+  generic \
+  --type="kubernetes.io/rook" \
+  "$MON_SECRET_NAME" \
+  --from-literal="$MON_SECRET_CLUSTER_NAME_KEYNAME"="$ROOK_EXTERNAL_CLUSTER_NAME" \
+  --from-literal="$MON_SECRET_FSID_KEYNAME"="$ROOK_EXTERNAL_FSID" \
+  --from-literal="$MON_SECRET_ADMIN_KEYRING_KEYNAME"="$ROOK_EXTERNAL_ADMIN_SECRET" \
+  --from-literal="$MON_SECRET_MON_KEYRING_KEYNAME"="$ROOK_EXTERNAL_MONITOR_SECRET" \
+  --from-literal="$MON_SECRET_CEPH_USERNAME_KEYNAME"="$ROOK_EXTERNAL_USERNAME" \
+  --from-literal="$MON_SECRET_CEPH_SECRET_KEYNAME"="$ROOK_EXTERNAL_USER_SECRET"
 }
 
 function importConfigMap() {
   kubectl -n "$NAMESPACE" \
-    create \
-    configmap \
-    "$MON_ENDPOINT_CONFIGMAP_NAME" \
-    --from-literal=data="$ROOK_EXTERNAL_CEPH_MON_DATA" \
-    --from-literal=mapping="$ROOK_EXTERNAL_MAPPING" \
-    --from-literal=maxMonId="$ROOK_EXTERNAL_MAX_MON_ID"
+  create \
+  configmap \
+  "$MON_ENDPOINT_CONFIGMAP_NAME" \
+  --from-literal=data="$ROOK_EXTERNAL_CEPH_MON_DATA" \
+  --from-literal=mapping="$ROOK_EXTERNAL_MAPPING" \
+  --from-literal=maxMonId="$ROOK_EXTERNAL_MAX_MON_ID"
 }
 
 function importCsiRBDNodeSecret() {
   kubectl -n "$NAMESPACE" \
-    create \
-    secret \
-    generic \
-    --type="kubernetes.io/rook" \
-    "rook-""$CSI_RBD_NODE_SECRET_NAME" \
-    --from-literal=userID="$CSI_RBD_NODE_SECRET_NAME" \
-    --from-literal=userKey="$CSI_RBD_NODE_SECRET"
+  create \
+  secret \
+  generic \
+  --type="kubernetes.io/rook" \
+  "rook-""$CSI_RBD_NODE_SECRET_NAME" \
+  --from-literal=userID="$CSI_RBD_NODE_SECRET_NAME" \
+  --from-literal=userKey="$CSI_RBD_NODE_SECRET"
 }
 
 function importCsiRBDProvisionerSecret() {
   kubectl -n "$NAMESPACE" \
-    create \
-    secret \
-    generic \
-    --type="kubernetes.io/rook" \
-    "rook-""$CSI_RBD_PROVISIONER_SECRET_NAME" \
-    --from-literal=userID="$CSI_RBD_PROVISIONER_SECRET_NAME" \
-    --from-literal=userKey="$CSI_RBD_PROVISIONER_SECRET"
+  create \
+  secret \
+  generic \
+  --type="kubernetes.io/rook" \
+  "rook-""$CSI_RBD_PROVISIONER_SECRET_NAME" \
+  --from-literal=userID="$CSI_RBD_PROVISIONER_SECRET_NAME" \
+  --from-literal=userKey="$CSI_RBD_PROVISIONER_SECRET"
 }
 
 function importCsiCephFSNodeSecret() {
   kubectl -n "$NAMESPACE" \
-    create \
-    secret \
-    generic \
-    --type="kubernetes.io/rook" \
-    "rook-""$CSI_CEPHFS_NODE_SECRET_NAME" \
-    --from-literal=adminID="$CSI_CEPHFS_NODE_SECRET_NAME" \
-    --from-literal=adminKey="$CSI_CEPHFS_NODE_SECRET"
+  create \
+  secret \
+  generic \
+  --type="kubernetes.io/rook" \
+  "rook-""$CSI_CEPHFS_NODE_SECRET_NAME" \
+  --from-literal=adminID="$CSI_CEPHFS_NODE_SECRET_NAME" \
+  --from-literal=adminKey="$CSI_CEPHFS_NODE_SECRET"
 }
 
 function importCsiCephFSProvisionerSecret() {
   kubectl -n "$NAMESPACE" \
-    create \
-    secret \
-    generic \
-    --type="kubernetes.io/rook" \
-    "rook-""$CSI_CEPHFS_PROVISIONER_SECRET_NAME" \
-    --from-literal=adminID="$CSI_CEPHFS_PROVISIONER_SECRET_NAME" \
-    --from-literal=adminKey="$CSI_CEPHFS_PROVISIONER_SECRET"
+  create \
+  secret \
+  generic \
+  --type="kubernetes.io/rook" \
+  "rook-""$CSI_CEPHFS_PROVISIONER_SECRET_NAME" \
+  --from-literal=adminID="$CSI_CEPHFS_PROVISIONER_SECRET_NAME" \
+  --from-literal=adminKey="$CSI_CEPHFS_PROVISIONER_SECRET"
 }
 
 function importRGWAdminOpsUser() {
   kubectl -n "$NAMESPACE" \
-    create \
-    secret \
-    generic \
-    --type="kubernetes.io/rook" \
-    "$RGW_ADMIN_OPS_USER_SECRET_NAME" \
-    --from-literal=accessKey="$RGW_ADMIN_OPS_USER_ACCESS_KEY" \
-    --from-literal=secretKey="$RGW_ADMIN_OPS_USER_SECRET_KEY"
+  create \
+  secret \
+  generic \
+  --type="kubernetes.io/rook" \
+  "$RGW_ADMIN_OPS_USER_SECRET_NAME" \
+  --from-literal=accessKey="$RGW_ADMIN_OPS_USER_ACCESS_KEY" \
+  --from-literal=secretKey="$RGW_ADMIN_OPS_USER_SECRET_KEY"
 }
 
 function createECRBDStorageClass() {
-  cat <<eof | kubectl create -f -
+cat <<eof | kubectl create -f -
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
@@ -164,8 +200,8 @@ metadata:
 provisioner: $RBD_PROVISIONER
 parameters:
   clusterID: $CLUSTER_ID_RBD
-  pool: $RBD_METADATA_EC_POOL_NAME
-  dataPool: $RBD_POOL_NAME
+  pool: $RBD_POOL_NAME
+  dataPool: $RBD_METADATA_EC_POOL_NAME
   imageFormat: "2"
   imageFeatures: $ROOK_RBD_FEATURES
   csi.storage.k8s.io/provisioner-secret-name: "rook-$CSI_RBD_PROVISIONER_SECRET_NAME"
@@ -181,7 +217,7 @@ eof
 }
 
 function createRBDStorageClass() {
-  cat <<eof | kubectl create -f -
+cat <<eof | kubectl create -f -
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
@@ -205,7 +241,7 @@ eof
 }
 
 function createCephFSStorageClass() {
-  cat <<eof | kubectl create -f -
+cat <<eof | kubectl create -f -
 apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
@@ -233,27 +269,15 @@ checkEnvVars
 importClusterID
 importSecret
 importConfigMap
-if [ -n "$CSI_RBD_NODE_SECRET_NAME" ] && [ -n "$CSI_RBD_NODE_SECRET" ]; then
-  importCsiRBDNodeSecret
-fi
-if [ -n "$CSI_RBD_PROVISIONER_SECRET_NAME" ] && [ -n "$CSI_RBD_PROVISIONER_SECRET" ]; then
-  importCsiRBDProvisionerSecret
-fi
-if [ -n "$RGW_ADMIN_OPS_USER_ACCESS_KEY" ] && [ -n "$RGW_ADMIN_OPS_USER_SECRET_KEY" ]; then
-  importRGWAdminOpsUser
-fi
-if [ -n "$CSI_CEPHFS_NODE_SECRET_NAME" ] && [ -n "$CSI_CEPHFS_NODE_SECRET" ]; then
-  importCsiCephFSNodeSecret
-fi
-if [ -n "$CSI_CEPHFS_PROVISIONER_SECRET_NAME" ] && [ -n "$CSI_CEPHFS_PROVISIONER_SECRET" ]; then
-  importCsiCephFSProvisionerSecret
-fi
-if [ -n "$RBD_POOL_NAME" ]; then
-  if [ -n "$RBD_METADATA_EC_POOL_NAME" ]; then
-    createECRBDStorageClass
-  else
-    createRBDStorageClass
-  fi
+importCsiRBDNodeSecret
+importCsiRBDProvisionerSecret
+importCsiCephFSNodeSecret
+importCsiCephFSProvisionerSecret
+importRGWAdminOpsUser
+if [ -n "$RBD_METADATA_EC_POOL_NAME" ]; then
+  createECRBDStorageClass
+else
+  createRBDStorageClass
 fi
 if [ -n "$CEPHFS_FS_NAME" ] && [ -n "$CEPHFS_POOL_NAME" ]; then
   createCephFSStorageClass
diff --git a/deploy/examples/monitoring/service-monitor.yaml b/deploy/examples/monitoring/service-monitor.yaml
index 44a7a4f7b..77e82c959 100644
--- a/deploy/examples/monitoring/service-monitor.yaml
+++ b/deploy/examples/monitoring/service-monitor.yaml
@@ -11,9 +11,9 @@ spec:
       - rook-ceph
   selector:
     matchLabels:
-      mgr_role: active
       app: rook-ceph-mgr
       rook_cluster: rook-ceph
+      ceph_daemon_id: a
   endpoints:
   - port: http-metrics
     path: /metrics
diff --git a/deploy/examples/operator-openshift.yaml b/deploy/examples/operator-openshift.yaml
index e3af8bffc..f525850e5 100644
--- a/deploy/examples/operator-openshift.yaml
+++ b/deploy/examples/operator-openshift.yaml
@@ -517,14 +517,6 @@ data:
   #      limits:
   #        memory: 1Gi
   #        cpu: 500m
-  #  - name : csi-attacher
-  #    resource:
-  #      requests:
-  #        memory: 128Mi
-  #        cpu: 100m
-  #      limits:
-  #        memory: 256Mi
-  #        cpu: 200m
   # (Optional) CEPH CSI NFS plugin resource requirement list, Put here list of resource
   # requests and limits you want to apply for plugin pod
   # CSI_NFS_PLUGIN_RESOURCE: |
@@ -592,17 +584,6 @@ data:
   # Defaults to all the labels mentioned in
   # https://rook.io/docs/rook/latest/CRDs/Cluster/ceph-cluster-crd/#osd-topology
   # CSI_CRUSH_LOCATION_LABELS: "kubernetes.io/hostname,topology.kubernetes.io/zone,topology.rook.io/rack"
-
-  # Whether to skip any attach operation altogether for CephCSI PVCs.
-  # See more details [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object).
-  # If set to false it skips the volume attachments and makes the creation of pods using the CephCSI PVC fast.
-  # **WARNING** It's highly discouraged to use this for RWO volumes. for RBD PVC it can cause data corruption,
-  # csi-addons operations like Reclaimspace and PVC Keyrotation will also not be supported if set to false
-  # since we'll have no VolumeAttachments to determine which node the PVC is mounted on.
-  # Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details.
-  CSI_CEPHFS_ATTACH_REQUIRED: "true"
-  CSI_RBD_ATTACH_REQUIRED: "true"
-  CSI_NFS_ATTACH_REQUIRED: "true"
 ---
 # The deployment for the rook operator
 # OLM: BEGIN OPERATOR DEPLOYMENT
diff --git a/deploy/examples/operator.yaml b/deploy/examples/operator.yaml
index 2b38033d5..357eb7055 100644
--- a/deploy/examples/operator.yaml
+++ b/deploy/examples/operator.yaml
@@ -445,14 +445,6 @@ data:
   #      limits:
   #        memory: 1Gi
   #        cpu: 500m
-  #  - name : csi-attacher
-  #    resource:
-  #      requests:
-  #        memory: 128Mi
-  #        cpu: 100m
-  #      limits:
-  #        memory: 256Mi
-  #        cpu: 200m
   # (Optional) CEPH CSI NFS plugin resource requirement list, Put here list of resource
   # requests and limits you want to apply for plugin pod
   # CSI_NFS_PLUGIN_RESOURCE: |
@@ -520,17 +512,6 @@ data:
   # Defaults to all the labels mentioned in
   # https://rook.io/docs/rook/latest/CRDs/Cluster/ceph-cluster-crd/#osd-topology
   # CSI_CRUSH_LOCATION_LABELS: "kubernetes.io/hostname,topology.kubernetes.io/zone,topology.rook.io/rack"
-
-  # Whether to skip any attach operation altogether for CephCSI PVCs.
-  # See more details [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object).
-  # If set to false it skips the volume attachments and makes the creation of pods using the CephCSI PVC fast.
-  # **WARNING** It's highly discouraged to use this for RWO volumes. for RBD PVC it can cause data corruption,
-  # csi-addons operations like Reclaimspace and PVC Keyrotation will also not be supported if set to false
-  # since we'll have no VolumeAttachments to determine which node the PVC is mounted on.
-  # Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details.
-  CSI_CEPHFS_ATTACH_REQUIRED: "true"
-  CSI_RBD_ATTACH_REQUIRED: "true"
-  CSI_NFS_ATTACH_REQUIRED: "true"
 ---
 # OLM: BEGIN OPERATOR DEPLOYMENT
 apiVersion: apps/v1
diff --git a/design/ceph/object/store.md b/design/ceph/object/store.md
index 5c8fd6201..4ec577a10 100644
--- a/design/ceph/object/store.md
+++ b/design/ceph/object/store.md
@@ -119,60 +119,3 @@ Specifying this section also ensures that the pool section in the ceph-object-zo
   zone:
     name: "name"
 ```
-
-### LDAP
-
-The Ceph Object Gateway supports integrating with LDAP for authenticating and creating users, please refer [here](https://docs.ceph.com/docs/master/radosgw/ldap-auth/). This means that the `rgw backend user` is also required to be part of groups in the LDAP server otherwise, authentication will fail. The `rgw backend user` can be generated from `CephObjectStoreUser` or `ObjectBucketClaim` CRDs. For the both resources credentials are saved in Kubernetes Secrets which may not be valid with `LDAP Server`, user need to follow the steps mentioned [here](https://docs.ceph.com/en/latest/radosgw/ldap-auth/#generating-an-access-token-for-ldap-authentication).The following settings need to be configured in the RGW server:
-```
-rgw ldap binddn =
-rgw ldap secret = /etc/ceph/ldap/bindpass.secret
-rgw ldap uri =
-rgw ldap searchdn =
-rgw ldap dnattr =
-rgw ldap searchfilter =
-rgw s3 auth use ldap = true
-```
-So the CRD for the Ceph Object Store will be modified to include the above changes:
-
-```yaml
-spec:
-  security
-    ldap:
-      config:
-        uri: ldaps://ldap-server:636
-        binddn: "uid=ceph,cn=users,cn=accounts,dc=example,dc=com"
-        searchdn: "cn=users,cn=accounts,dc=example,dc=com"
-        dnattr: "uid"
-        searchfilter: "memberof=cn=s3,cn=groups,cn=accounts,dc=example,dc=com"
-      credential:
-        volumeSource:
-          secret:
-            secretName: object-my-store-ldap-creds
-            defaultMode: 0600 #required
-
-```
-
-The `config` section includes options used for RGW wrt LDAP server. These options are strongly typed rather than string map approach since very less chance to modify in future.
-
-* `uri`: It specifies the address of LDAP server to use.* `binddn`: The bind domain for the service account used by RGW server.
-* `searchdn`: The search domain where can it look for the user details.
-* `dnattr`: The attribute being used in the constructed search filter to match a username, this can either be `uid` or `cn`.
-* `searchfilter`: A generic search filter. If `dnattr` is set, this filter is `&()`'d together with the automatically constructed filter.
-
-The `credential` defines where the password for accessing ldap server should be sourced from
-
-* `volumeSource`: this is a standard Kubernetes VolumeSource for the Kerberos keytab file like
-      what is normally used to configure Volumes for a Pod. For example, a Secret or HostPath.
-      There are two requirements for the source's content:
-    1. The config file must be mountable via `subPath: password`. For example, in a Secret, the
-        data item must be named `password`, or `items` must be defined to select the key and
-        give it path `password`. A HostPath directory must have the `password` file.
-    2. The volume or config file must have mode 0600.
-
-The CA bundle for ldap can be added to the `caBundleRef` option in `Gateway` settings:
-
-```yaml
-spec:
-  gateway:
-    caBundleRef: #ldaps-cabundle
-```
diff --git a/go.mod b/go.mod
index 65fb631f3..b6fa8e3fb 100644
--- a/go.mod
+++ b/go.mod
@@ -4,16 +4,16 @@ go 1.19
 
 require (
 	github.com/IBM/keyprotect-go-client v0.9.2
-	github.com/aws/aws-sdk-go v1.44.229
+	github.com/aws/aws-sdk-go v1.44.214
 	github.com/banzaicloud/k8s-objectmatcher v1.8.0
-	github.com/ceph/go-ceph v0.20.0
+	github.com/ceph/go-ceph v0.18.0
 	github.com/coreos/pkg v0.0.0-20220810130054-c7d1c02cb6cf
-	github.com/gemalto/kmip-go v0.0.9
+	github.com/gemalto/kmip-go v0.0.8
 	github.com/go-ini/ini v1.67.0
 	github.com/google/go-cmp v0.5.9
 	github.com/google/uuid v1.3.0
-	github.com/hashicorp/vault v1.13.1
-	github.com/hashicorp/vault-plugin-secrets-kv v0.14.3
+	github.com/hashicorp/vault v1.13.0
+	github.com/hashicorp/vault-plugin-secrets-kv v0.14.2
 	github.com/hashicorp/vault/api v1.9.0
 	github.com/hashicorp/vault/sdk v0.8.1
 	github.com/jetstack/cert-manager v1.7.3
@@ -22,8 +22,8 @@ require (
 	github.com/libopenstorage/secrets v0.0.0-20220823020833-2ecadaf59d8a
 	github.com/openshift/api v0.0.0-20210105115604-44119421ec6b
 	github.com/pkg/errors v0.9.1
-	github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.64.0
-	github.com/prometheus-operator/prometheus-operator/pkg/client v0.64.0
+	github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.63.0
+	github.com/prometheus-operator/prometheus-operator/pkg/client v0.63.0
 	github.com/spf13/cobra v1.6.1
 	github.com/spf13/pflag v1.0.5
 	github.com/stretchr/testify v1.8.2
@@ -31,11 +31,11 @@ require (
 	gopkg.in/ini.v1 v1.67.0
 	gopkg.in/yaml.v2 v2.4.0
 	k8s.io/api v0.26.1
-	k8s.io/apimachinery v0.26.3
+	k8s.io/apimachinery v0.26.1
 	k8s.io/client-go v0.26.1
 	k8s.io/cloud-provider v0.26.1
 	k8s.io/utils v0.0.0-20230202215443-34013725500c
-	sigs.k8s.io/controller-runtime v0.14.5
+	sigs.k8s.io/controller-runtime v0.14.4
 	sigs.k8s.io/mcs-api v0.1.0
 	sigs.k8s.io/yaml v1.3.0
 )
@@ -116,7 +116,7 @@ require (
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang-jwt/jwt/v4 v4.4.2 // indirect
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
-	github.com/golang/protobuf v1.5.3 // indirect
+	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/golang/snappy v0.0.4 // indirect
 	github.com/google/gnostic v0.6.9 // indirect
 	github.com/google/go-metrics-stackdriver v0.2.0 // indirect
@@ -130,7 +130,7 @@ require (
 	github.com/hashicorp/eventlogger v0.1.1 // indirect
 	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
 	github.com/hashicorp/go-discover v0.0.0-20210818145131-c573d69da192 // indirect
-	github.com/hashicorp/go-hclog v1.5.0 // indirect
+	github.com/hashicorp/go-hclog v1.4.0 // indirect
 	github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
 	github.com/hashicorp/go-kms-wrapping/entropy/v2 v2.0.0 // indirect
 	github.com/hashicorp/go-kms-wrapping/v2 v2.0.8 // indirect
@@ -238,18 +238,18 @@ require (
 	go.uber.org/multierr v1.8.0 // indirect
 	go.uber.org/zap v1.24.0 // indirect
 	golang.org/x/crypto v0.6.0 // indirect
-	golang.org/x/net v0.8.0 // indirect
+	golang.org/x/net v0.7.0 // indirect
 	golang.org/x/oauth2 v0.4.0 // indirect
-	golang.org/x/sys v0.6.0 // indirect
-	golang.org/x/term v0.6.0 // indirect
-	golang.org/x/text v0.8.0 // indirect
+	golang.org/x/sys v0.5.0 // indirect
+	golang.org/x/term v0.5.0 // indirect
+	golang.org/x/text v0.7.0 // indirect
 	golang.org/x/time v0.3.0 // indirect
 	gomodules.xyz/jsonpatch/v2 v2.2.0 // indirect
 	google.golang.org/api v0.109.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
 	google.golang.org/genproto v0.0.0-20221227171554-f9683d7f8bef // indirect
 	google.golang.org/grpc v1.51.0 // indirect
-	google.golang.org/protobuf v1.30.0 // indirect
+	google.golang.org/protobuf v1.28.1 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/resty.v1 v1.12.0 // indirect
 	gopkg.in/square/go-jose.v2 v2.6.0 // indirect
diff --git a/go.sum b/go.sum
index 854bd6617..4d9b0f9c2 100644
--- a/go.sum
+++ b/go.sum
@@ -224,8 +224,8 @@ github.com/aws/aws-sdk-go v1.25.37/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpi
 github.com/aws/aws-sdk-go v1.25.41/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
 github.com/aws/aws-sdk-go v1.30.27/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0=
 github.com/aws/aws-sdk-go v1.34.28/go.mod h1:H7NKnBqNVzoTJpGfLrQkkD+ytBA93eiDYi/+8rV9s48=
-github.com/aws/aws-sdk-go v1.44.229 h1:lku0ZSHRzj/qtFVM//QE8VjV6kvJ6CFijDZSsjNaD9A=
-github.com/aws/aws-sdk-go v1.44.229/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI=
+github.com/aws/aws-sdk-go v1.44.214 h1:YzDuC+9UtrAOUkItlK7l3BvKI9o6qAog9X8i289HORc=
+github.com/aws/aws-sdk-go v1.44.214/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI=
 github.com/aws/aws-sdk-go-v2 v1.8.0 h1:HcN6yDnHV9S7D69E7To0aUppJhiJNEzQSNcUxc7r3qo=
 github.com/aws/aws-sdk-go-v2/credentials v1.3.2 h1:Uud/fZzm0lqqhE8kvXYJFAJ3PGnagKoUcvHq1hXfBZw=
 github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.4.0 h1:Iqp2aHeRF3kaaNuDS82bHBzER285NM6lLPAgsxHCR2A=
@@ -268,8 +268,8 @@ github.com/cenkalti/backoff/v4 v4.1.3/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInq
 github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
 github.com/centrify/cloud-golang-sdk v0.0.0-20190214225812-119110094d0f/go.mod h1:C0rtzmGXgN78pYR0tGJFhtHgkbAs0lIbHwkB81VxDQE=
 github.com/centrify/cloud-golang-sdk v0.0.0-20210923165758-a8c48d049166 h1:jQ93fKqb/wRmK/KiHpa7Tk9rmHeKXhp4j+5Sg/tENiY=
-github.com/ceph/go-ceph v0.20.0 h1:weYE7zoVrIBk9bIVq8jS9z3RukGlnZwFGCth7Tu1ssU=
-github.com/ceph/go-ceph v0.20.0/go.mod h1:/4iZyaN5wTpIFMEKO6GU3xX3vHnyvd2Fcdf05n9VuD0=
+github.com/ceph/go-ceph v0.18.0 h1:4WM6yAq/iqBDaeeADDiPKLqKiP0iZ4fffdgCr1lnOL4=
+github.com/ceph/go-ceph v0.18.0/go.mod h1:cflETVTBNAQM6jdr7hpNHHFHKYiJiWWcAeRDrRx/1ng=
 github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
 github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE=
@@ -432,8 +432,8 @@ github.com/gammazero/workerpool v0.0.0-20190406235159-88d534f22b56 h1:VzbudKn/nv
 github.com/gammazero/workerpool v0.0.0-20190406235159-88d534f22b56/go.mod h1:w9RqFVO2BM3xwWEcAB8Fwp0OviTBBEiRmSBDfbXnd3w=
 github.com/gemalto/flume v0.13.0 h1:EEeQvAxyFys3BH8IxEU7ZpM6Kr1sYn20HuZq6dgyMR8=
 github.com/gemalto/flume v0.13.0/go.mod h1:3iOEZiK/HD8SnFTqHCQoOHQKaHlBY0b6z55P8SLaOzk=
-github.com/gemalto/kmip-go v0.0.9 h1:PMm0j3k5l7H6BIp13+V6aq0mvp0O3YlKVHkDweLkzik=
-github.com/gemalto/kmip-go v0.0.9/go.mod h1:YBUgKtkYpGCL+xA1oClQrMYTWDvHN/A2wSh2LmT5+JY=
+github.com/gemalto/kmip-go v0.0.8 h1:RvKWTd2ACxOs7OF1f6SvPYebjmQbN0myfDHVQmX/k8g=
+github.com/gemalto/kmip-go v0.0.8/go.mod h1:7bAnjuzri8yGoJMwngnAd0HdXMRDQU+l1Zaiz12Tr68=
 github.com/getkin/kin-openapi v0.76.0/go.mod h1:660oXbgy5JFMKreazJaQTw7o+X00qeSyhcnluiMv+Xg=
 github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
@@ -597,6 +597,7 @@ github.com/go-test/deep v1.0.1/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3a
 github.com/go-test/deep v1.0.2-0.20181118220953-042da051cf31/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3aSFNA=
 github.com/go-test/deep v1.0.2/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3aSFNA=
 github.com/go-test/deep v1.0.4/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3aSFNA=
+github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
 github.com/go-test/deep v1.1.0 h1:WOcxcdHcvdgThNXjw0t76K42FXTU7HpNQWHpA2HHNlg=
 github.com/go-test/deep v1.1.0/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
 github.com/go-yaml/yaml v2.1.0+incompatible/go.mod h1:w2MrLa16VYP0jy6N7M5kHaCkaLENm+P+Tv+MfurjSw0=
@@ -682,9 +683,8 @@ github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw
 github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
 github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
 github.com/golang/protobuf v1.5.1/go.mod h1:DopwsBzvsk0Fs44TXzsVbJyPhcCPeIwnvohx4u74HPM=
+github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
 github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
-github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
-github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
 github.com/golang/snappy v0.0.0-20170215233205-553a64147049/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
 github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
 github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
@@ -816,8 +816,9 @@ github.com/hashicorp/go-hclog v0.10.1/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39
 github.com/hashicorp/go-hclog v0.12.0/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ=
 github.com/hashicorp/go-hclog v0.14.1/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ=
 github.com/hashicorp/go-hclog v0.16.2/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ=
-github.com/hashicorp/go-hclog v1.5.0 h1:bI2ocEMgcVlz55Oj1xZNBsVi900c7II+fWDyV9o+13c=
-github.com/hashicorp/go-hclog v1.5.0/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M=
+github.com/hashicorp/go-hclog v1.3.1/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M=
+github.com/hashicorp/go-hclog v1.4.0 h1:ctuWFGrhFha8BnnzxqeRGidlEcQkDyL5u8J8t5eA11I=
+github.com/hashicorp/go-hclog v1.4.0/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M=
 github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
 github.com/hashicorp/go-immutable-radix v1.1.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
 github.com/hashicorp/go-immutable-radix v1.3.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
@@ -828,6 +829,7 @@ github.com/hashicorp/go-kms-wrapping v0.5.1/go.mod h1:cGIibZmMx9qlxS1pZTUrEgGqA+
 github.com/hashicorp/go-kms-wrapping/entropy v0.1.0/go.mod h1:d1g9WGtAunDNpek8jUIEJnBlbgKS1N2Q61QkHiZyR1g=
 github.com/hashicorp/go-kms-wrapping/entropy/v2 v2.0.0 h1:pSjQfW3vPtrOTcasTUKgCTQT7OGPPTTMVRrOfU6FJD8=
 github.com/hashicorp/go-kms-wrapping/entropy/v2 v2.0.0/go.mod h1:xvb32K2keAc+R8DSFG2IwDcydK9DBQE+fGA5fsw6hSk=
+github.com/hashicorp/go-kms-wrapping/v2 v2.0.7/go.mod h1:sDQAfwJGv25uGPZA04x87ERglCG6avnRcBT9wYoMII8=
 github.com/hashicorp/go-kms-wrapping/v2 v2.0.8 h1:9Q2lu1YbbmiAgvYZ7Pr31RdlVonUpX+mmDL7Z7qTA2U=
 github.com/hashicorp/go-kms-wrapping/v2 v2.0.8/go.mod h1:qTCjxGig/kjuj3hk1z8pOUrzbse/GxB1tGfbrq8tGJg=
 github.com/hashicorp/go-kms-wrapping/wrappers/aead/v2 v2.0.7-1 h1:ZV26VJYcITBom0QqYSUOIj4HOHCVPEFjLqjxyXV/AbA=
@@ -857,6 +859,7 @@ github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9
 github.com/hashicorp/go-plugin v1.0.0/go.mod h1:++UyYGoz3o5w9ZzAdZxtQKrWWP+iqPBn3cQptSMzBuY=
 github.com/hashicorp/go-plugin v1.0.1/go.mod h1:++UyYGoz3o5w9ZzAdZxtQKrWWP+iqPBn3cQptSMzBuY=
 github.com/hashicorp/go-plugin v1.4.3/go.mod h1:5fGEH17QVwTTcR0zV7yhDPLLmFX9YSZ38b18Udy6vYQ=
+github.com/hashicorp/go-plugin v1.4.5/go.mod h1:viDMjcLJuDui6pXb8U4HVfb8AamCWhHGUjr2IrTF67s=
 github.com/hashicorp/go-plugin v1.4.8 h1:CHGwpxYDOttQOY7HOWgETU9dyVjOXzniXDqJcYJE1zM=
 github.com/hashicorp/go-plugin v1.4.8/go.mod h1:viDMjcLJuDui6pXb8U4HVfb8AamCWhHGUjr2IrTF67s=
 github.com/hashicorp/go-raftchunking v0.6.3-0.20191002164813-7e9e8525653a h1:FmnBDwGwlTgugDGbVxwV8UavqSMACbGrUpfc98yFLR4=
@@ -952,8 +955,8 @@ github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/J
 github.com/hashicorp/serf v0.8.3/go.mod h1:UpNcs7fFbpKIyZaUuSW6EPiH+eZC7OuyFD+wc1oal+k=
 github.com/hashicorp/serf v0.10.1 h1:Z1H2J60yRKvfDYAOZLd2MU0ND4AH/WDz7xYHDWQsIPY=
 github.com/hashicorp/vault v1.4.2/go.mod h1:500fLOj7p92Ys4X265LizqF78MzmHJUf1jV1zNJt060=
-github.com/hashicorp/vault v1.13.1 h1:4Q31hCWCwD2XRCwGL+9gsrCwxn6+ngToBgjT0FCTG+M=
-github.com/hashicorp/vault v1.13.1/go.mod h1:aD5a/VHOADe8hiMHx4rYJwv6W6+1WYlco/hf8MWbYaw=
+github.com/hashicorp/vault v1.13.0 h1:/J/ldGnqRHKDlaSznOQAXy4STah76tCYy2fsUSsvYwk=
+github.com/hashicorp/vault v1.13.0/go.mod h1:W1VOUMJ1klrGAHFnTJYDw2N2RlBh+kdwmCyYZTSGap0=
 github.com/hashicorp/vault-plugin-auth-alicloud v0.5.5/go.mod h1:sQ+VNwPQlemgXHXikYH6onfH9gPwDZ1GUVRLz0ZvHx8=
 github.com/hashicorp/vault-plugin-auth-alicloud v0.14.0 h1:O6tNk0s/arubLUbLeCyaRs5xGo9VwmbQazISY/BfPK4=
 github.com/hashicorp/vault-plugin-auth-azure v0.5.6-0.20200422235613-1b5c70f9ef68/go.mod h1:RCVBsf8AJndh4c6iGZtvVZFui9SG0Bj9fnF0SodNIkw=
@@ -975,7 +978,7 @@ github.com/hashicorp/vault-plugin-auth-oci v0.5.4/go.mod h1:j05O2b9fw2Q82NxDPhHM
 github.com/hashicorp/vault-plugin-auth-oci v0.13.1 h1:xThaZC9jzZoqqccfxTk11hfwgqwN3yEZ3kYOnY2v2Fs=
 github.com/hashicorp/vault-plugin-database-couchbase v0.9.0 h1:hJOHJ9yZ9kt1/DuRaU5Sa339j3/QcPL4esT9JLQonYA=
 github.com/hashicorp/vault-plugin-database-elasticsearch v0.5.4/go.mod h1:QjGrrxcRXv/4XkEZAlM0VMZEa3uxKAICFqDj27FP/48=
-github.com/hashicorp/vault-plugin-database-elasticsearch v0.13.1 h1:nVO6F8V69E2fAQklh/Ds+EypVMutN4iIlt3sat9qW9M=
+github.com/hashicorp/vault-plugin-database-elasticsearch v0.13.0 h1:NwcbzQB529WtB/m7tZKxKiB6pQc0IyD3L80tk3mtBl8=
 github.com/hashicorp/vault-plugin-database-mongodbatlas v0.1.2-0.20200520204052-f840e9d4895c/go.mod h1:MP3kfr0N+7miOTZFwKv952b9VkXM4S2Q6YtQCiNKWq8=
 github.com/hashicorp/vault-plugin-database-mongodbatlas v0.9.0 h1:wlWrg1z5Pyx+FTUCOzA9yh0FTI+pfA9tMrsFPFBcjjA=
 github.com/hashicorp/vault-plugin-database-redis v0.2.0 h1:Fg1inevnDhj58+/y5SY1CihLftytG1D+3QqbUJbHYUM=
@@ -994,12 +997,12 @@ github.com/hashicorp/vault-plugin-secrets-gcpkms v0.5.5/go.mod h1:b6RwFD1bny1zbf
 github.com/hashicorp/vault-plugin-secrets-gcpkms v0.14.0 h1:eUC5ltK+1bkc+SVMzAUq4tBeNrsDXyCuITH8jeajXcM=
 github.com/hashicorp/vault-plugin-secrets-kubernetes v0.3.0 h1:Joz9SBwjpEOGu+Ynv60JC3fAA4UuLJzu7NcrKm6wMMs=
 github.com/hashicorp/vault-plugin-secrets-kv v0.5.5/go.mod h1:oNyUoMMQq6uNTwyYPnkldiedaknYbPfQIdKoyKQdy2g=
-github.com/hashicorp/vault-plugin-secrets-kv v0.14.3 h1:6hV/p/yXff50v540Kc6RexOP4z3My398NRmSI5xaX90=
-github.com/hashicorp/vault-plugin-secrets-kv v0.14.3/go.mod h1:tQ87Y0wm1JhliWuiXReCvh6uM0lpFXyrEkuWsFJUR6o=
+github.com/hashicorp/vault-plugin-secrets-kv v0.14.2 h1:13p50RIltQM/JH32uWZe9sAp16Uaj0zCLmVGPvS09qo=
+github.com/hashicorp/vault-plugin-secrets-kv v0.14.2/go.mod h1:cAxt2o3BjRT5CbNLtgXuxTReaejvrgN/qk+no+DnwJ8=
 github.com/hashicorp/vault-plugin-secrets-mongodbatlas v0.1.2/go.mod h1:YRW9zn9NZNitRlPYNAWRp/YEdKCF/X8aOg8IYSxFT5Y=
 github.com/hashicorp/vault-plugin-secrets-mongodbatlas v0.9.1 h1:WkW8fyHxEdz1wGSTxCnSCrzXvgLXqXr8Iqp7upa/s4E=
 github.com/hashicorp/vault-plugin-secrets-openldap v0.1.3-0.20200518214608-746aba5fead6/go.mod h1:9Cy4Jp779BjuIOhYLjEfH3M3QCUxZgPnvJ3tAOOmof4=
-github.com/hashicorp/vault-plugin-secrets-openldap v0.10.1 h1:EN3/iEjPPmcpX9yihybQNHvewc+YoJw7aoKsio1WK5s=
+github.com/hashicorp/vault-plugin-secrets-openldap v0.10.0 h1:Q3nKBbHQ6E/kOa3amKvcbhYTbkz4U25BBTwH66LnF+0=
 github.com/hashicorp/vault-plugin-secrets-terraform v0.7.0 h1:jgJpVKhV0Eh6EjpUEIf7VYH2D6D0xW2Lry9/3PI8hy0=
 github.com/hashicorp/vault/api v1.0.1/go.mod h1:AV/+M5VPDpB90arloVX0rVDUIHkONiwz5Uza9HRtpUE=
 github.com/hashicorp/vault/api v1.0.5-0.20190730042357-746c0b111519/go.mod h1:i9PKqwFko/s/aihU1uuHGh/FaQS+Xcgvd9dvnfAvQb0=
@@ -1277,6 +1280,7 @@ github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826/go.mod h1:TaXosZuwd
 github.com/mongodb-forks/digest v1.0.3 h1:ZUK1vyZnBiRMvET0O1SzmnBmv935CkcOTjhfR4zIQ2s=
 github.com/mongodb/go-client-mongodb-atlas v0.1.2/go.mod h1:LS8O0YLkA+sbtOb3fZLF10yY3tJM+1xATXMJ3oU35LU=
 github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc=
+github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
 github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
@@ -1371,6 +1375,7 @@ github.com/pierrec/lz4/v4 v4.1.8 h1:ieHkV+i2BRzngO4Wd/3HGowuZStgq6QkPsD1eolNAO4=
 github.com/pires/go-proxyproto v0.6.1 h1:EBupykFmo22SDjv4fQVQd2J9NOoLPmyZA/15ldOGkPw=
 github.com/pires/go-proxyproto v0.6.1/go.mod h1:Odh9VFOZJCf9G8cLW5o435Xf1J95Jw9Gw5rnCjcwzAY=
 github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 h1:KoWmjvw+nsYOo29YJK9vDA65RGE3NrOnUtO7a+RF9HU=
+github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
@@ -1395,11 +1400,11 @@ github.com/pquerna/otp v1.2.1-0.20191009055518-468c2dd2b58d h1:PinQItctnaL2LtkaS
 github.com/pquerna/otp v1.2.1-0.20191009055518-468c2dd2b58d/go.mod h1:dkJfzwRKNiegxyNb54X/3fLwhCynbMspSyWKnvi1AEg=
 github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.44.1/go.mod h1:3WYi4xqXxGGXWDdQIITnLNmuDzO5n6wYva9spVhR4fg=
 github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.46.0/go.mod h1:3WYi4xqXxGGXWDdQIITnLNmuDzO5n6wYva9spVhR4fg=
-github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.64.0 h1:bqFOzWYCuSZEcuFx/ez8DFW+fqGiUEATrgezynCjpP4=
-github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.64.0/go.mod h1:cfNgxpCPGyIydmt3HcwDqKDt0nYdlGRhzftl+DZH7WA=
+github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.63.0 h1:efsW3CfymG5bZUpeIsYfdihB33YItCn7uHBOEbnHQG8=
+github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.63.0/go.mod h1:/UtstAaWVaS3Z9GK9jo8+4SN9T+RMSq7VlOcQMmiEsc=
 github.com/prometheus-operator/prometheus-operator/pkg/client v0.46.0/go.mod h1:k4BrWlVQQsvBiTcDnKEMgyh/euRxyxgrHdur/ZX/sdA=
-github.com/prometheus-operator/prometheus-operator/pkg/client v0.64.0 h1:bMvSvfmfL9NAuWnUyZoFSyb0jzzcTI3LCMs5vVnBfZ4=
-github.com/prometheus-operator/prometheus-operator/pkg/client v0.64.0/go.mod h1:J51EiS4fKYZkSHLWo1t8Ad0iEw8RhhC6LLvYarM/+30=
+github.com/prometheus-operator/prometheus-operator/pkg/client v0.63.0 h1:8Au9x/807lcDRCmypzfWvr9haO5eF/eX7hiMQCy1k4A=
+github.com/prometheus-operator/prometheus-operator/pkg/client v0.63.0/go.mod h1:sjCnCbwVFVTO90iChFOK0eUat70C4Q11EVik0Rk3EeU=
 github.com/prometheus/client_golang v0.9.0/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
 github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
 github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4xf/QclQDMrYNZzcM=
@@ -1460,6 +1465,7 @@ github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFR
 github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
 github.com/rogpeppe/go-internal v1.6.2/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
 github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
+github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
 github.com/rs/zerolog v1.4.0/go.mod h1:YbFCdg8HfsridGWAh22vktObvhZbQsZXe4/zB0OKkWU=
 github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
 github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@@ -1687,7 +1693,9 @@ golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5y
 golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
+golang.org/x/crypto v0.0.0-20220313003712-b769efc7c000/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
 golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
+golang.org/x/crypto v0.5.0/go.mod h1:NK/OQwhpMQP3MwtdjgLlYHnH9ebylxKWv3e0fK+mkQU=
 golang.org/x/crypto v0.6.0 h1:qfktjS5LUO+fFKeJXZ+ikTRijMmljikvG68fpMMruSc=
 golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
@@ -1791,8 +1799,9 @@ golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su
 golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco=
-golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ=
-golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
+golang.org/x/net v0.5.0/go.mod h1:DivGGAXEgPSlEBzxGzZI+ZLohi+xUj054jfeKui00ws=
+golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g=
+golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190130055435-99b60b757ec1/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -1926,15 +1935,17 @@ golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=
-golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU=
+golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
-golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw=
-golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
+golang.org/x/term v0.4.0/go.mod h1:9P2UbLfCdcvo3p/nzKvsmas4TnlujnuoV9hGgYzW1lQ=
+golang.org/x/term v0.5.0 h1:n2a8QNdAb0sZNpU9R1ALUXBbY+w51fCQDN+7EdxNBsY=
+golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
 golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -1947,8 +1958,9 @@ golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
-golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68=
-golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
+golang.org/x/text v0.6.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
+golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo=
+golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@@ -2175,8 +2187,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba
 google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
 google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
 google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
-google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng=
-google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
+google.golang.org/protobuf v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w=
+google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
 gopkg.in/airbrake/gobrake.v2 v2.0.9/go.mod h1:/h5ZAUhDkGaJfjzjKLSjv6zCL6O0LLBxU4K+aSYdM/U=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d/go.mod h1:cuepJuh7vyXfUyUwEgHQXw849cJrilpS5NeIjOWESAw=
@@ -2278,8 +2290,8 @@ k8s.io/apimachinery v0.20.0/go.mod h1:WlLqWAHZGg07AeltaI0MV5uk1Omp8xaN0JGLY6gkRp
 k8s.io/apimachinery v0.20.1/go.mod h1:WlLqWAHZGg07AeltaI0MV5uk1Omp8xaN0JGLY6gkRpU=
 k8s.io/apimachinery v0.20.4/go.mod h1:WlLqWAHZGg07AeltaI0MV5uk1Omp8xaN0JGLY6gkRpU=
 k8s.io/apimachinery v0.23.5/go.mod h1:BEuFMMBaIbcOqVIJqNZJXGFTP4W6AycEpb5+m/97hrM=
-k8s.io/apimachinery v0.26.3 h1:dQx6PNETJ7nODU3XPtrwkfuubs6w7sX0M8n61zHIV/k=
-k8s.io/apimachinery v0.26.3/go.mod h1:ats7nN1LExKHvJ9TmwootT00Yz05MuYqPXEXaVeOy5I=
+k8s.io/apimachinery v0.26.1 h1:8EZ/eGJL+hY/MYCNwhmDzVqq2lPl3N3Bo8rvweJwXUQ=
+k8s.io/apimachinery v0.26.1/go.mod h1:tnPmbONNJ7ByJNz9+n9kMjNP8ON+1qoAIIC70lztu74=
 k8s.io/apiserver v0.18.2/go.mod h1:Xbh066NqrZO8cbsoenCwyDJ1OSi8Ag8I2lezeHxzwzw=
 k8s.io/apiserver v0.18.3/go.mod h1:tHQRmthRPLUtwqsOnJJMoI8SW3lnoReZeE861lH8vUw=
 k8s.io/apiserver v0.18.4/go.mod h1:q+zoFct5ABNnYkGIaGQ3bcbUNdmPyOCoEBcg51LChY8=
@@ -2355,8 +2367,8 @@ sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.7/go.mod h1:PHgbrJT
 sigs.k8s.io/apiserver-network-proxy/konnectivity-client v0.0.14/go.mod h1:LEScyzhFmoF5pso/YSeBstl57mOzx9xlU9n85RGrDQg=
 sigs.k8s.io/controller-runtime v0.2.2/go.mod h1:9dyohw3ZtoXQuV1e766PHUn+cmrRCIcBh6XIMFNMZ+I=
 sigs.k8s.io/controller-runtime v0.6.1/go.mod h1:XRYBPdbf5XJu9kpS84VJiZ7h/u1hF3gEORz0efEja7A=
-sigs.k8s.io/controller-runtime v0.14.5 h1:6xaWFqzT5KuAQ9ufgUaj1G/+C4Y1GRkhrxl+BJ9i+5s=
-sigs.k8s.io/controller-runtime v0.14.5/go.mod h1:WqIdsAY6JBsjfc/CqO0CORmNtoCtE4S6qbPc9s68h+0=
+sigs.k8s.io/controller-runtime v0.14.4 h1:Kd/Qgx5pd2XUL08eOV2vwIq3L9GhIbJ5Nxengbd4/0M=
+sigs.k8s.io/controller-runtime v0.14.4/go.mod h1:WqIdsAY6JBsjfc/CqO0CORmNtoCtE4S6qbPc9s68h+0=
 sigs.k8s.io/controller-tools v0.3.0/go.mod h1:enhtKGfxZD1GFEoMgP8Fdbu+uKQ/cq1/WGJhdVChfvI=
 sigs.k8s.io/json v0.0.0-20211020170558-c049b76a60c6/go.mod h1:p4QtZmO4uMYipTQNzagwnNoseA6OxSUutVw05NhYDRs=
 sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd h1:EDPBXCAspyGV4jQlpZSudPeMmr1bNJefnuqLsRAsHZo=
diff --git a/images/image.mk b/images/image.mk
index 2aee0c427..9014ee2c3 100644
--- a/images/image.mk
+++ b/images/image.mk
@@ -17,6 +17,13 @@
 
 override GOOS=linux
 
+ifeq ($(origin DOCKERCMD),undefined)
+DOCKERCMD?=$(shell docker version >/dev/null 2>&1 && echo docker)
+ifeq ($(DOCKERCMD),)
+DOCKERCMD=$(shell podman version >/dev/null 2>&1 && echo podman)
+endif
+endif
+
 # include the common make file
 SELF_DIR := $(dir $(lastword $(MAKEFILE_LIST)))
 include $(SELF_DIR)/../build/makelib/common.mk
diff --git a/pkg/apis/ceph.rook.io/v1/nfs.go b/pkg/apis/ceph.rook.io/v1/nfs.go
index 1325c748c..e43cf1b1e 100644
--- a/pkg/apis/ceph.rook.io/v1/nfs.go
+++ b/pkg/apis/ceph.rook.io/v1/nfs.go
@@ -43,13 +43,6 @@ func (k *KerberosSpec) GetPrincipalName() string {
 	return k.PrincipalName
 }
 
-func (n *CephNFS) IsHostNetwork(c *ClusterSpec) bool {
-	if n.Spec.Server.HostNetwork != nil {
-		return *n.Spec.Server.HostNetwork
-	}
-	return c.Network.IsHost()
-}
-
 func (n *CephNFS) ValidateCreate() error {
 	return n.Spec.Security.Validate()
 }
diff --git a/pkg/apis/ceph.rook.io/v1/object.go b/pkg/apis/ceph.rook.io/v1/object.go
index a6ae0081d..23d787147 100644
--- a/pkg/apis/ceph.rook.io/v1/object.go
+++ b/pkg/apis/ceph.rook.io/v1/object.go
@@ -41,10 +41,6 @@ func (s *ObjectStoreSpec) IsTLSEnabled() bool {
 	return s.Gateway.SecurePort != 0 && (s.Gateway.SSLCertificateRef != "" || s.GetServiceServingCert() != "")
 }
 
-func (s *ObjectStoreSpec) IsRGWDashboardEnabled() bool {
-	return s.Gateway.DashboardEnabled == nil || *s.Gateway.DashboardEnabled
-}
-
 func (s *ObjectStoreSpec) GetPort() (int32, error) {
 	if s.IsTLSEnabled() {
 		return s.Gateway.SecurePort, nil
@@ -85,15 +81,8 @@ func ValidateObjectSpec(gs *CephObjectStore) error {
 	if gs.Namespace == "" {
 		return errors.New("missing namespace")
 	}
-
-	// validate the object store name only if it is not an external cluster
-	// as external cluster won't create the rgw daemon and it's other resources
-	// and there is some legacy external cluster which has more length of objectstore
-	// so to run them successfully we are not validating the objectstore name
-	if !gs.Spec.IsExternal() {
-		if len(gs.Name) > objectStoreNameMaxLen {
-			return errors.New("object store name cannot be longer than 38 characters")
-		}
+	if len(gs.Name) > objectStoreNameMaxLen {
+		return errors.New("object store name cannot be longer than 38 characters")
 	}
 	securePort := gs.Spec.Gateway.SecurePort
 	if securePort < 0 || securePort > 65535 {
diff --git a/pkg/apis/ceph.rook.io/v1/types.go b/pkg/apis/ceph.rook.io/v1/types.go
index 72ec1675a..3790106b7 100755
--- a/pkg/apis/ceph.rook.io/v1/types.go
+++ b/pkg/apis/ceph.rook.io/v1/types.go
@@ -281,6 +281,7 @@ type KeyRotationSpec struct {
 	Enabled bool `json:"enabled,omitempty"`
 	// Schedule represents the cron schedule for key rotation.
 	// +optional
+	// +kubebuilder:default="@weekly"
 	Schedule string `json:"schedule,omitempty"`
 }
 
@@ -337,16 +338,6 @@ type MonitoringSpec struct {
 	// +kubebuilder:validation:Maximum=65535
 	// +optional
 	ExternalMgrPrometheusPort uint16 `json:"externalMgrPrometheusPort,omitempty"`
-
-	// Port is the prometheus server port
-	// +kubebuilder:validation:Minimum=0
-	// +kubebuilder:validation:Maximum=65535
-	// +optional
-	Port int `json:"port,omitempty"`
-
-	// Interval determines prometheus scrape interval
-	// +optional
-	Interval *metav1.Duration `json:"interval,omitempty"`
 }
 
 // ClusterStatus represents the status of a Ceph cluster
@@ -1467,12 +1458,6 @@ type GatewaySpec struct {
 	// +nullable
 	// +optional
 	HostNetwork *bool `json:"hostNetwork,omitempty"`
-
-	// Whether rgw dashboard is enabled for the rgw daemon. If not set, the rgw dashboard will be enabled.
-	// +kubebuilder:pruning:PreserveUnknownFields
-	// +nullable
-	// +optional
-	DashboardEnabled *bool `json:"dashboardEnabled,omitempty"`
 }
 
 // EndpointAddress is a tuple that describes a single IP address or host name. This is a subset of
@@ -2014,11 +1999,6 @@ type GaneshaServerSpec struct {
 	// LogLevel set logging level
 	// +optional
 	LogLevel string `json:"logLevel,omitempty"`
-
-	// Whether host networking is enabled for the Ganesha server. If not set, the network settings from the cluster CR will be applied.
-	// +nullable
-	// +optional
-	HostNetwork *bool `json:"hostNetwork,omitempty"`
 }
 
 // NFSSecuritySpec represents security configurations for an NFS server pod
diff --git a/pkg/daemon/ceph/client/config.go b/pkg/daemon/ceph/client/config.go
index f2cc6119b..8d4148b0d 100644
--- a/pkg/daemon/ceph/client/config.go
+++ b/pkg/daemon/ceph/client/config.go
@@ -261,8 +261,7 @@ func PopulateMonHostMembers(clusterInfo *ClusterInfo) ([]string, []string) {
 		currentMonPort := cephutil.GetPortFromEndpoint(monitor.Endpoint)
 
 		if currentMonPort == Msgr2port {
-			msgr2Endpoint := net.JoinHostPort(monIP, strconv.Itoa(int(Msgr2port)))
-			monHosts = append(monHosts, "[v2:"+msgr2Endpoint+"]")
+			monHosts = append(monHosts, fmt.Sprintf("[v2:%s:%d]", monIP, Msgr2port))
 		} else {
 			msgr2Endpoint := net.JoinHostPort(monIP, strconv.Itoa(int(Msgr2port)))
 			msgr1Endpoint := net.JoinHostPort(monIP, strconv.Itoa(int(currentMonPort)))
diff --git a/pkg/daemon/ceph/osd/daemon.go b/pkg/daemon/ceph/osd/daemon.go
index 99a85b744..8bf087203 100644
--- a/pkg/daemon/ceph/osd/daemon.go
+++ b/pkg/daemon/ceph/osd/daemon.go
@@ -531,7 +531,19 @@ func getAvailableDevices(context *clusterd.Context, agent *OsdAgent) (*DeviceOsd
 
 				if matched {
 					matchedDevice = desiredDevice
-					matchedDevice.UpdateDeviceClass(agent, device)
+					if matchedDevice.DeviceClass == "" {
+						classNotSet := true
+						if agent.pvcBacked {
+							crushDeviceClass := os.Getenv(oposd.CrushDeviceClassVarName)
+							if crushDeviceClass != "" {
+								matchedDevice.DeviceClass = crushDeviceClass
+								classNotSet = false
+							}
+						}
+						if classNotSet {
+							matchedDevice.DeviceClass = sys.GetDiskDeviceClass(device)
+						}
+					}
 					break
 				}
 			}
diff --git a/pkg/daemon/ceph/osd/device.go b/pkg/daemon/ceph/osd/device.go
index 3c4676762..152666502 100644
--- a/pkg/daemon/ceph/osd/device.go
+++ b/pkg/daemon/ceph/osd/device.go
@@ -18,9 +18,7 @@ package osd
 
 import (
 	"encoding/json"
-	"os"
 
-	oposd "github.com/rook/rook/pkg/operator/ceph/cluster/osd"
 	"github.com/rook/rook/pkg/util/sys"
 )
 
@@ -69,29 +67,3 @@ func (m *DeviceOsdMapping) String() string {
 	b, _ := json.Marshal(m)
 	return string(b)
 }
-
-func (d *DesiredDevice) UpdateDeviceClass(agent *OsdAgent, device *sys.LocalDisk) {
-	// Rook sets the storage class of a device with the following priority.
-	//
-	// 1. the device-level configuration
-	// 2. the global or node-local configuration
-	// 3. the default value estimated from sysfs.
-	if d.DeviceClass != "" {
-		return
-	}
-
-	if agent.pvcBacked {
-		crushDeviceClass := os.Getenv(oposd.CrushDeviceClassVarName)
-		if crushDeviceClass != "" {
-			d.DeviceClass = crushDeviceClass
-			return
-		}
-	} else {
-		if agent.storeConfig.DeviceClass != "" {
-			d.DeviceClass = agent.storeConfig.DeviceClass
-			return
-		}
-	}
-
-	d.DeviceClass = sys.GetDiskDeviceClass(device)
-}
diff --git a/pkg/daemon/ceph/osd/kms/vault.go b/pkg/daemon/ceph/osd/kms/vault.go
index 97df49059..40e9cd052 100644
--- a/pkg/daemon/ceph/osd/kms/vault.go
+++ b/pkg/daemon/ceph/osd/kms/vault.go
@@ -147,7 +147,7 @@ func configTLS(ctx context.Context, clusterdContext *clusterd.Context, namespace
 			}
 
 			// Write into a file
-			err = os.WriteFile(file.Name(), secret.Data[tlsSecretKeyToCheck(tlsOption)], 0400)
+			err = os.WriteFile(file.Name(), secret.Data[tlsSecretKeyToCheck(tlsOption)], 0444)
 			if err != nil {
 				return nil, removeCertFiles, errors.Wrapf(err, "failed to write k8s secret %q content to a file", tlsSecretName)
 			}
diff --git a/pkg/daemon/ceph/util/util.go b/pkg/daemon/ceph/util/util.go
index 8722357dd..eeeeac227 100644
--- a/pkg/daemon/ceph/util/util.go
+++ b/pkg/daemon/ceph/util/util.go
@@ -53,7 +53,5 @@ func GetPortFromEndpoint(endpoint string) int32 {
 			logger.Errorf("failed to convert %q to integer. %v", portString, err)
 		}
 	}
-	//#nosec G109 -- This is just a lower bit to bigger bit conversion
-	portInt32 := int32(port)
-	return portInt32
+	return int32(port)
 }
diff --git a/pkg/operator/ceph/cluster/cluster.go b/pkg/operator/ceph/cluster/cluster.go
index 5df5caaf3..db772b7e7 100755
--- a/pkg/operator/ceph/cluster/cluster.go
+++ b/pkg/operator/ceph/cluster/cluster.go
@@ -634,7 +634,7 @@ func (c *cluster) configureMsgr2() error {
 
 		// set default rbd map options to enable msgr2 in the kernel if it's
 		// required even with encryption disabled
-		if c.Spec.RequireMsgr2() {
+		if c.Spec.Network.Connections != nil && c.Spec.Network.Connections.RequireMsgr2 {
 			if err := monStore.SetAll("global", map[string]string{rbdMapOptions: "ms_mode=prefer-crc"}); err != nil {
 				return err
 			}
diff --git a/pkg/operator/ceph/cluster/cluster_external.go b/pkg/operator/ceph/cluster/cluster_external.go
index c21e7f191..e5b5c2a93 100644
--- a/pkg/operator/ceph/cluster/cluster_external.go
+++ b/pkg/operator/ceph/cluster/cluster_external.go
@@ -211,7 +211,7 @@ func (c *ClusterController) configureExternalClusterMonitoring(context *clusterd
 	)
 
 	// Create external monitoring Service
-	service, err := manager.MakeMetricsService(opcontroller.ExternalMgrAppName, opcontroller.ServiceExternalMetricName)
+	service, err := manager.MakeMetricsService(opcontroller.ExternalMgrAppName, "", opcontroller.ServiceExternalMetricName)
 	if err != nil {
 		return err
 	}
@@ -231,7 +231,7 @@ func (c *ClusterController) configureExternalClusterMonitoring(context *clusterd
 	// Deploy external ServiceMonitor
 	logger.Info("creating external service monitor")
 	// servicemonitor takes some metadata from the service for easy mapping
-	err = manager.EnableServiceMonitor()
+	err = manager.EnableServiceMonitor("")
 	if err != nil {
 		logger.Errorf("failed to enable external service monitor. %v", err)
 	} else {
diff --git a/pkg/operator/ceph/cluster/mgr/dashboard.go b/pkg/operator/ceph/cluster/mgr/dashboard.go
index c849e51d7..7abf6c25a 100644
--- a/pkg/operator/ceph/cluster/mgr/dashboard.go
+++ b/pkg/operator/ceph/cluster/mgr/dashboard.go
@@ -54,8 +54,8 @@ var (
 	dashboardInitWaitTime = 5 * time.Second
 )
 
-func (c *Cluster) configureDashboardService() error {
-	dashboardService, err := c.makeDashboardService(AppName)
+func (c *Cluster) configureDashboardService(activeDaemon string) error {
+	dashboardService, err := c.makeDashboardService(AppName, activeDaemon)
 	if err != nil {
 		return err
 	}
@@ -88,12 +88,11 @@ func (c *Cluster) configureDashboardModules() error {
 		return nil
 	}
 
-	err := c.initializeSecureDashboard()
+	hasChanged, err := c.initializeSecureDashboard()
 	if err != nil {
 		return errors.Wrap(err, "failed to initialize dashboard")
 	}
 
-	var hasChanged bool
 	for _, daemonID := range c.getDaemonIDs() {
 		changed, err := c.configureDashboardModuleSettings(daemonID)
 		if err != nil {
@@ -105,7 +104,7 @@ func (c *Cluster) configureDashboardModules() error {
 	}
 	if hasChanged {
 		logger.Info("dashboard config has changed. restarting the dashboard module")
-		return c.restartMgrModule(dashboardModuleName)
+		return c.restartDashboard()
 	}
 	return nil
 }
@@ -149,33 +148,30 @@ func (c *Cluster) configureDashboardModuleSettings(daemonID string) (bool, error
 	return hasChanged, nil
 }
 
-func (c *Cluster) initializeSecureDashboard() error {
+func (c *Cluster) initializeSecureDashboard() (bool, error) {
 	// we need to wait a short period after enabling the module before we can call the `ceph dashboard` commands.
 	time.Sleep(dashboardInitWaitTime)
 
 	password, err := c.getOrGenerateDashboardPassword()
 	if err != nil {
-		return errors.Wrap(err, "failed to generate a password for the ceph dashboard")
+		return false, errors.Wrap(err, "failed to generate a password for the ceph dashboard")
 	}
 
 	if c.spec.Dashboard.SSL {
 		alreadyCreated, err := c.createSelfSignedCert()
 		if err != nil {
-			return errors.Wrap(err, "failed to create a self signed cert for the ceph dashboard")
+			return false, errors.Wrap(err, "failed to create a self signed cert for the ceph dashboard")
 		}
 		if alreadyCreated {
-			return nil
-		}
-		if err := c.restartMgrModule(dashboardModuleName); err != nil {
-			logger.Warningf("failed to restart dashboard after generating ssl cert. %v", err)
+			return false, nil
 		}
 	}
 
 	if err := c.setLoginCredentials(password); err != nil {
-		return errors.Wrap(err, "failed to set login credentials for the ceph dashboard")
+		return false, errors.Wrap(err, "failed to set login credentials for the ceph dashboard")
 	}
 
-	return nil
+	return false, nil
 }
 
 func (c *Cluster) createSelfSignedCert() (bool, error) {
@@ -308,3 +304,14 @@ func decodeSecret(secret *v1.Secret) (string, error) {
 	}
 	return string(password), nil
 }
+
+func (c *Cluster) restartDashboard() error {
+	logger.Info("restarting the mgr module")
+	if err := client.MgrDisableModule(c.context, c.clusterInfo, dashboardModuleName); err != nil {
+		return errors.Wrapf(err, "failed to disable mgr module %q.", dashboardModuleName)
+	}
+	if err := client.MgrEnableModule(c.context, c.clusterInfo, dashboardModuleName, true); err != nil {
+		return errors.Wrapf(err, "failed to enable mgr module %q.", dashboardModuleName)
+	}
+	return nil
+}
diff --git a/pkg/operator/ceph/cluster/mgr/mgr.go b/pkg/operator/ceph/cluster/mgr/mgr.go
index 1efb2f005..efd6d2a60 100644
--- a/pkg/operator/ceph/cluster/mgr/mgr.go
+++ b/pkg/operator/ceph/cluster/mgr/mgr.go
@@ -20,7 +20,6 @@ package mgr
 import (
 	"fmt"
 	"path"
-	"strconv"
 
 	"github.com/banzaicloud/k8s-objectmatcher/patch"
 	"github.com/coreos/pkg/capnslog"
@@ -51,7 +50,6 @@ const (
 	PgautoscalerModuleName = "pg_autoscaler"
 	balancerModuleName     = "balancer"
 	balancerModuleMode     = "upmap"
-	mgrRoleLabelName       = "mgr_role"
 	monitoringPath         = "/etc/ceph-monitoring/"
 	serviceMonitorFile     = "service-monitor.yaml"
 	// minimum amount of memory in MB to run the pod
@@ -179,7 +177,18 @@ func (c *Cluster) Start() error {
 	// check if any extra mgrs need to be removed
 	c.removeExtraMgrs(daemonIDs)
 
+	activeMgr := daemonIDs[0]
 	if len(daemonIDs) > 1 {
+		// When multiple mgrs are running, the mgr sidecar for the active mgr
+		// will create the services. However, the sidecar will only reconcile all
+		// the services when the active mgr changes. Here as part of the regular reconcile
+		// we trigger reconciling all the services to ensure they are current.
+		activeMgr, err = c.getActiveMgr()
+		if err != nil || activeMgr == "" {
+			activeMgr = ""
+			logger.Infof("cannot reconcile mgr services, no active mgr found. err=%v", err)
+		}
+
 		// reconcile mgr PDB
 		if err := c.reconcileMgrPDB(); err != nil {
 			return errors.Wrap(err, "failed to reconcile mgr PDB")
@@ -188,9 +197,10 @@ func (c *Cluster) Start() error {
 		// delete MGR PDB as the count is less than 2
 		c.deleteMgrPDB()
 	}
-
-	if err := c.reconcileServices(); err != nil {
-		return errors.Wrap(err, "failed to enable mgr services")
+	if activeMgr != "" {
+		if err := c.reconcileServices(activeMgr); err != nil {
+			return errors.Wrap(err, "failed to enable mgr services")
+		}
 	}
 
 	// configure the mgr modules
@@ -211,52 +221,35 @@ func (c *Cluster) removeExtraMgrs(daemonIDs []string) {
 	}
 }
 
-// UpdateActiveMgrLabel updates the mgr_role label value to either
-// active or standby depending on the status of the ceph mgr running
-// in the pod
-func (c *Cluster) UpdateActiveMgrLabel(daemonNameToUpdate string, prevActiveMgr string) (string, error) {
-
-	logger.Infof("Checking mgr_role label value of daemon %s (prev active mgr was %s)", daemonNameToUpdate, prevActiveMgr)
-	currActiveMgr, err := c.getActiveMgr()
-	if err != nil || currActiveMgr == "" {
-		logger.Infof("cannot update active mgr, no active mgr found. err=%v", err)
-		return "", err
-	} else if prevActiveMgr == currActiveMgr {
-		logger.Infof("active mgr is still the same (%s). No need to update mgr_role label on daemon %s.", currActiveMgr, daemonNameToUpdate)
-		return currActiveMgr, err
+// ReconcileActiveMgrServices reconciles the services if the active mgr is the one running
+// in the sidecar
+func (c *Cluster) ReconcileActiveMgrServices(daemonNameToUpdate string) error {
+	// If the services are already set to this daemon, no need to attempt to update
+	svc, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Get(c.clusterInfo.Context, AppName, metav1.GetOptions{})
+	if err != nil {
+		logger.Errorf("failed to check current mgr service, proceeding to update. %v", err)
+	} else {
+		currentDaemon := svc.Spec.Selector[controller.DaemonIDLabel]
+		if currentDaemon == daemonNameToUpdate {
+			logger.Infof("mgr services already set to daemon %q, no need to update", daemonNameToUpdate)
+			return nil
+		}
+		logger.Infof("mgr service currently set to %q, checking if need to update to %q", currentDaemon, daemonNameToUpdate)
 	}
 
-	pods, err := c.context.Clientset.CoreV1().Pods(c.clusterInfo.Namespace).List(c.clusterInfo.Context, metav1.ListOptions{
-		LabelSelector: fmt.Sprintf("mgr=%s", daemonNameToUpdate),
-	})
+	activeName, err := c.getActiveMgr()
 	if err != nil {
-		logger.Infof("cannot get pod for mgr daemon %s", daemonNameToUpdate)
-		return "", err // force mrg_role update in the next call
+		return err
 	}
-
-	for i, pod := range pods.Items {
-
-		labels := pod.GetLabels()
-		cephDaemonId := labels[controller.DaemonIDLabel]
-		newMgrRole := "standby"
-		if currActiveMgr == cephDaemonId {
-			newMgrRole = "active"
-		}
-
-		currMgrRole, mgrHasLabel := labels[mgrRoleLabelName]
-		if !mgrHasLabel || currMgrRole != newMgrRole {
-
-			logger.Infof("updating mgr_role label value of daemon %s to '%s'. New active mgr is %s.", daemonNameToUpdate, newMgrRole, currActiveMgr)
-			labels[mgrRoleLabelName] = newMgrRole
-			pod.SetLabels(labels)
-			_, err = c.context.Clientset.CoreV1().Pods(c.clusterInfo.Namespace).Update(c.clusterInfo.Context, &pods.Items[i], metav1.UpdateOptions{})
-			if err != nil {
-				logger.Infof("cannot update the active mgr pod %q. err=%v", pods.Items[i].Name, err)
-			}
-		}
+	if activeName == "" {
+		return errors.New("active mgr not found")
+	}
+	if daemonNameToUpdate != activeName {
+		logger.Infof("no need for the mgr update since the active mgr is %q, rather than the local mgr %q", activeName, daemonNameToUpdate)
+		return nil
 	}
 
-	return currActiveMgr, c.reconcileServices()
+	return c.reconcileServices(activeName)
 }
 
 func (c *Cluster) getActiveMgr() (string, error) {
@@ -267,15 +260,16 @@ func (c *Cluster) getActiveMgr() (string, error) {
 	return mgrStat.ActiveName, nil
 }
 
-// reconcile the services,
-func (c *Cluster) reconcileServices() error {
+// reconcile the services, if the active mgr is not detected, use the default mgr
+func (c *Cluster) reconcileServices(activeDaemon string) error {
+	logger.Infof("setting services to point to mgr %q", activeDaemon)
 
-	if err := c.configureDashboardService(); err != nil {
+	if err := c.configureDashboardService(activeDaemon); err != nil {
 		return errors.Wrap(err, "failed to configure dashboard svc")
 	}
 
 	// create the metrics service
-	service, err := c.MakeMetricsService(AppName, serviceMetricName)
+	service, err := c.MakeMetricsService(AppName, activeDaemon, serviceMetricName)
 	if err != nil {
 		return err
 	}
@@ -285,61 +279,46 @@ func (c *Cluster) reconcileServices() error {
 
 	// enable monitoring if `monitoring: enabled: true`
 	if c.spec.Monitoring.Enabled {
-		if err := c.EnableServiceMonitor(); err != nil {
+		if err := c.EnableServiceMonitor(activeDaemon); err != nil {
 			return errors.Wrap(err, "failed to enable service monitor")
 		}
 	}
 
-	c.updateServiceSelectors()
-	return nil
+	return c.updateServiceSelectors(activeDaemon)
 }
 
-// For the upgrade scenario: we remove any selector DaemonIDLabel from the all
-// the services since new mgr HA doesn't rely on this label anymore and we add
-// the new "mgr_role=active" instead.
-func (c *Cluster) updateServiceSelectors() {
+// Make a best effort to update the services that have been labeled for being updated
+// when the mgr has changed. They might be services for node ports, ingress, etc
+func (c *Cluster) updateServiceSelectors(activeDaemon string) error {
 	selector := metav1.ListOptions{LabelSelector: "app=rook-ceph-mgr"}
 	services, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).List(c.clusterInfo.Context, selector)
 	if err != nil {
-		logger.Errorf("failed to query mgr services to update labels: %v", err)
-		return
+		return errors.Wrap(err, "failed to query mgr services to update")
 	}
 	for i, service := range services.Items {
 		if service.Spec.Selector == nil {
-			continue
-		}
-
-		updateService := false
-
-		// Check if the service has a DaemonIDLabel (legacy mgr HA implementation) and remove it
-		_, hasDaemonLabel := service.Spec.Selector[controller.DaemonIDLabel]
-		if hasDaemonLabel {
-			logger.Infof("removing %s selector label on mgr service %q", controller.DaemonIDLabel, service.Name)
-			delete(service.Spec.Selector, controller.DaemonIDLabel)
-			updateService = true
+			service.Spec.Selector = map[string]string{}
 		}
-
-		// In case the service doesn't have the new mgr_role DaemonIDLabel we add it
-		_, hasMgrRoleLabel := service.Spec.Selector[mgrRoleLabelName]
-		if !hasMgrRoleLabel {
-			logger.Infof("adding %s selector label on mgr service %q", mgrRoleLabelName, service.Name)
-			service.Spec.Selector[mgrRoleLabelName] = "active"
-			updateService = true
+		// Update the selector on the service to point to the active mgr
+		if service.Spec.Selector[controller.DaemonIDLabel] == activeDaemon {
+			logger.Infof("no need to update service %q", service.Name)
+			continue
 		}
-
-		if updateService {
-			if _, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Update(c.clusterInfo.Context, &services.Items[i], metav1.UpdateOptions{}); err != nil {
-				logger.Errorf("failed to update service %q. %v", service.Name, err)
-			} else {
-				logger.Infof("service %q successfully updated", service.Name)
-			}
+		// Update the service to point to the new active mgr
+		service.Spec.Selector[controller.DaemonIDLabel] = activeDaemon
+		logger.Infof("updating selector on mgr service %q to active mgr %q", service.Name, activeDaemon)
+		if _, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Update(c.clusterInfo.Context, &services.Items[i], metav1.UpdateOptions{}); err != nil {
+			logger.Errorf("failed to update service %q. %v", service.Name, err)
+		} else {
+			logger.Infof("service %q successfully updated to active mgr %q", service.Name, activeDaemon)
 		}
 	}
+	return nil
 }
 
 func (c *Cluster) configureModules(daemonIDs []string) {
 	// Configure the modules asynchronously so we can complete all the configuration much sooner.
-	startModuleConfiguration("prometheus", c.configurePrometheusModule)
+	startModuleConfiguration("prometheus", c.enablePrometheusModule)
 	startModuleConfiguration("dashboard", c.configureDashboardModules)
 
 	// It is a bit confusing but modules that are in the "always_on_modules" list
@@ -360,58 +339,9 @@ func startModuleConfiguration(description string, configureModules func() error)
 }
 
 // Ceph docs about the prometheus module: http://docs.ceph.com/docs/master/mgr/prometheus/
-func (c *Cluster) configurePrometheusModule() error {
-	if c.spec.Monitoring.Enabled {
-		if err := cephclient.MgrEnableModule(c.context, c.clusterInfo, PrometheusModuleName, true); err != nil {
-			return errors.Wrap(err, "failed to enable mgr prometheus module")
-		}
-	} else {
-		if err := cephclient.MgrDisableModule(c.context, c.clusterInfo, PrometheusModuleName); err != nil {
-			logger.Errorf("failed to disable mgr prometheus module. %v", err)
-		}
-		return nil
-	}
-
-	var (
-		err                error
-		portHasChanged     bool
-		intervalHasChanged bool
-		daemonID           = "mgr"
-	)
-	monStore := config.GetMonStore(c.context, c.clusterInfo)
-	// port
-	if c.spec.Monitoring.Port != 0 {
-		port := strconv.Itoa(c.spec.Monitoring.Port)
-		portHasChanged, err = monStore.SetIfChanged(daemonID, "mgr/prometheus/server_port", port)
-		if err != nil {
-			return err
-		}
-		logger.Infof("prometheus config will change, port: %s", port)
-	}
-	// scrape interval
-	if c.spec.Monitoring.Interval != nil {
-		interval := c.spec.Monitoring.Interval.Duration.Seconds()
-		intervalHasChanged, err = monStore.SetIfChanged(daemonID, "mgr/prometheus/scrape_interval", fmt.Sprintf("%v", interval))
-		if err != nil {
-			return err
-		}
-		logger.Infof("prometheus config will change, interval: %v", interval)
-	}
-
-	if portHasChanged || intervalHasChanged {
-		logger.Info("prometheus config has changed. restarting the prometheus module")
-		return c.restartMgrModule(PrometheusModuleName)
-	}
-	return nil
-}
-
-func (c *Cluster) restartMgrModule(name string) error {
-	logger.Infof("restarting the mgr module: %s", name)
-	if err := cephclient.MgrDisableModule(c.context, c.clusterInfo, name); err != nil {
-		return errors.Wrapf(err, "failed to disable mgr module %q.", name)
-	}
-	if err := cephclient.MgrEnableModule(c.context, c.clusterInfo, name, true); err != nil {
-		return errors.Wrapf(err, "failed to enable mgr module %q.", name)
+func (c *Cluster) enablePrometheusModule() error {
+	if err := cephclient.MgrEnableModule(c.context, c.clusterInfo, PrometheusModuleName, true); err != nil {
+		return errors.Wrap(err, "failed to enable mgr prometheus module")
 	}
 	return nil
 }
@@ -500,7 +430,7 @@ func wellKnownModule(name string) bool {
 }
 
 // EnableServiceMonitor add a servicemonitor that allows prometheus to scrape from the monitoring endpoint of the cluster
-func (c *Cluster) EnableServiceMonitor() error {
+func (c *Cluster) EnableServiceMonitor(activeDaemon string) error {
 	serviceMonitor, err := k8sutil.GetServiceMonitor(path.Join(monitoringPath, serviceMonitorFile))
 	if err != nil {
 		return errors.Wrap(err, "service monitor could not be enabled")
@@ -517,6 +447,7 @@ func (c *Cluster) EnableServiceMonitor() error {
 		return errors.Wrapf(err, "failed to set owner reference to service monitor %q", serviceMonitor.Name)
 	}
 	serviceMonitor.Spec.NamespaceSelector.MatchNames = []string{c.clusterInfo.Namespace}
+	serviceMonitor.Spec.Selector.MatchLabels = c.selectorLabels(activeDaemon)
 
 	applyMonitoringLabels(c, serviceMonitor)
 
diff --git a/pkg/operator/ceph/cluster/mgr/spec.go b/pkg/operator/ceph/cluster/mgr/spec.go
index de575c3f3..a73997c52 100644
--- a/pkg/operator/ceph/cluster/mgr/spec.go
+++ b/pkg/operator/ceph/cluster/mgr/spec.go
@@ -255,10 +255,8 @@ func (c *Cluster) makeCmdProxySidecarContainer(mgrConfig *mgrConfig) v1.Containe
 }
 
 // MakeMetricsService generates the Kubernetes service object for the monitoring service
-func (c *Cluster) MakeMetricsService(name, servicePortMetricName string) (*v1.Service, error) {
-
-	labels := controller.AppLabels(AppName, c.clusterInfo.Namespace)
-	selectorLabels := c.buildSelectorLabels(labels)
+func (c *Cluster) MakeMetricsService(name, activeDaemon, servicePortMetricName string) (*v1.Service, error) {
+	labels := c.selectorLabels(activeDaemon)
 
 	svc := &v1.Service{
 		ObjectMeta: metav1.ObjectMeta{
@@ -280,7 +278,7 @@ func (c *Cluster) MakeMetricsService(name, servicePortMetricName string) (*v1.Se
 
 	// If the cluster is external we don't need to add the selector
 	if name != controller.ExternalMgrAppName {
-		svc.Spec.Selector = selectorLabels
+		svc.Spec.Selector = labels
 	}
 
 	err := c.clusterInfo.OwnerInfo.SetControllerReference(svc)
@@ -290,10 +288,8 @@ func (c *Cluster) MakeMetricsService(name, servicePortMetricName string) (*v1.Se
 	return svc, nil
 }
 
-func (c *Cluster) makeDashboardService(name string) (*v1.Service, error) {
-
-	labels := controller.AppLabels(AppName, c.clusterInfo.Namespace)
-	selectorLabels := c.buildSelectorLabels(labels)
+func (c *Cluster) makeDashboardService(name, activeDaemon string) (*v1.Service, error) {
+	labels := c.selectorLabels(activeDaemon)
 
 	portName := "https-dashboard"
 	if !c.spec.Dashboard.SSL {
@@ -306,7 +302,7 @@ func (c *Cluster) makeDashboardService(name string) (*v1.Service, error) {
 			Labels:    labels,
 		},
 		Spec: v1.ServiceSpec{
-			Selector: selectorLabels,
+			Selector: labels,
 			Type:     v1.ServiceTypeClusterIP,
 			Ports: []v1.ServicePort{
 				{
@@ -356,11 +352,10 @@ func (c *Cluster) cephMgrOrchestratorModuleEnvs() []v1.EnvVar {
 	return envVars
 }
 
-func (c *Cluster) buildSelectorLabels(labels map[string]string) map[string]string {
-	selectorLabels := make(map[string]string)
-	for k, v := range labels {
-		selectorLabels[k] = v
+func (c *Cluster) selectorLabels(activeDaemon string) map[string]string {
+	labels := controller.AppLabels(AppName, c.clusterInfo.Namespace)
+	if activeDaemon != "" {
+		labels[controller.DaemonIDLabel] = activeDaemon
 	}
-	selectorLabels["mgr_role"] = "active"
-	return selectorLabels
+	return labels
 }
diff --git a/pkg/operator/ceph/cluster/mon/spec.go b/pkg/operator/ceph/cluster/mon/spec.go
index 7e7cb91f9..60c9c869f 100644
--- a/pkg/operator/ceph/cluster/mon/spec.go
+++ b/pkg/operator/ceph/cluster/mon/spec.go
@@ -271,6 +271,15 @@ func (c *Cluster) makeMonFSInitContainer(monConfig *monConfig) corev1.Container
 
 func (c *Cluster) makeMonDaemonContainer(monConfig *monConfig) corev1.Container {
 	podIPEnvVar := "ROOK_POD_IP"
+	publicAddr := monConfig.PublicIP
+
+	// Handle the non-default port for host networking. If host networking is not being used,
+	// the service created elsewhere will handle the non-default port redirection to the default port inside the container.
+	if monConfig.UseHostNetwork && monConfig.Port != DefaultMsgr1Port {
+		logger.Warningf("Starting mon %s with host networking on a non-default port %d. The mon must be failed over before enabling msgr2.",
+			monConfig.DaemonName, monConfig.Port)
+		publicAddr = fmt.Sprintf("%s:%d", publicAddr, monConfig.Port)
+	}
 
 	container := corev1.Container{
 		Name: "mon",
@@ -282,7 +291,7 @@ func (c *Cluster) makeMonDaemonContainer(monConfig *monConfig) corev1.Container
 			"--foreground",
 			// If the mon is already in the monmap, when the port is left off of --public-addr,
 			// it will still advertise on the previous port b/c monmap is saved to mon database.
-			config.NewFlag("public-addr", monConfig.PublicIP),
+			config.NewFlag("public-addr", publicAddr),
 			// Set '--setuser-match-path' so that existing directory owned by root won't affect the daemon startup.
 			// For existing data store owned by root, the daemon will continue to run as root
 			//
diff --git a/pkg/operator/ceph/cluster/nodedaemon/exporter.go b/pkg/operator/ceph/cluster/nodedaemon/exporter.go
index c039dc6d7..0b281042c 100644
--- a/pkg/operator/ceph/cluster/nodedaemon/exporter.go
+++ b/pkg/operator/ceph/cluster/nodedaemon/exporter.go
@@ -71,8 +71,12 @@ func (r *ReconcileNode) createOrUpdateCephExporter(node corev1.Node, tolerations
 		return controllerutil.OperationResultNone, errors.Errorf("failed to set owner reference of ceph-exporter deployment %q", deploy.Name)
 	}
 
+	configDir := path.Join(cephCluster.Spec.DataDirHostPath, cephCluster.Namespace)
+	configHostPathType := v1.HostPathDirectory
+	src := v1.VolumeSource{HostPath: &v1.HostPathVolumeSource{Path: configDir, Type: &configHostPathType}}
 	volumes := append(
 		controller.DaemonVolumesBase(config.NewDatalessDaemonDataPathMap(cephCluster.GetNamespace(), cephCluster.Spec.DataDirHostPath), "", cephCluster.Spec.DataDirHostPath),
+		v1.Volume{Name: "ceph-conf-dir", VolumeSource: src},
 		keyring.Volume().Admin())
 
 	mutateFunc := func() error {
@@ -138,7 +142,8 @@ func (r *ReconcileNode) createOrUpdateCephExporter(node corev1.Node, tolerations
 
 func getCephExporterChownInitContainer(cephCluster cephv1.CephCluster) corev1.Container {
 	dataPathMap := config.NewDatalessDaemonDataPathMap(cephCluster.GetNamespace(), cephCluster.Spec.DataDirHostPath)
-	mounts := controller.DaemonVolumeMounts(dataPathMap, "", cephCluster.Spec.DataDirHostPath)
+	configDir := path.Join(cephCluster.Spec.DataDirHostPath, cephCluster.Namespace)
+	mounts := append(controller.DaemonVolumeMounts(dataPathMap, "", cephCluster.Spec.DataDirHostPath), v1.VolumeMount{Name: "ceph-conf-dir", MountPath: configDir})
 
 	return controller.ChownCephDataDirsInitContainer(
 		*dataPathMap,
diff --git a/pkg/operator/ceph/cluster/osd/key_rotation.go b/pkg/operator/ceph/cluster/osd/key_rotation.go
index f5c2e4f31..b32edd82d 100644
--- a/pkg/operator/ceph/cluster/osd/key_rotation.go
+++ b/pkg/operator/ceph/cluster/osd/key_rotation.go
@@ -193,11 +193,6 @@ func (c *Cluster) makeKeyRotationCronJob(pvcName string, osd OSDInfo, osdProps o
 		return nil, err
 	}
 	c.applyResourcesToAllContainers(&podSpec.Spec, cephv1.GetOSDResources(c.spec.Resources, osd.DeviceClass))
-	schedule := c.spec.Security.KeyRotation.Schedule
-	if schedule == "" {
-		// default to rotate keyrings weekly (default is in code since default in crds causes issues)
-		schedule = "@weekly"
-	}
 	cronJob := &batch.CronJob{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:        keyRotationCronJobName(osd.ID),
@@ -207,7 +202,7 @@ func (c *Cluster) makeKeyRotationCronJob(pvcName string, osd OSDInfo, osdProps o
 		},
 		Spec: batch.CronJobSpec{
 			ConcurrencyPolicy: batch.ForbidConcurrent,
-			Schedule:          schedule,
+			Schedule:          c.spec.Security.KeyRotation.Schedule,
 			JobTemplate: batch.JobTemplateSpec{
 				Spec: batch.JobSpec{
 					Template: *podSpec,
diff --git a/pkg/operator/ceph/cluster/osd/topology/topology.go b/pkg/operator/ceph/cluster/osd/topology/topology.go
index ab25bc173..c1128b951 100644
--- a/pkg/operator/ceph/cluster/osd/topology/topology.go
+++ b/pkg/operator/ceph/cluster/osd/topology/topology.go
@@ -43,7 +43,6 @@ var (
 
 const (
 	topologyLabelPrefix = "topology.rook.io/"
-	labelHostname       = "kubernetes.io/hostname"
 )
 
 // ExtractTopologyFromLabels extracts rook topology from labels and returns a map from topology type to value
@@ -57,33 +56,6 @@ func ExtractOSDTopologyFromLabels(labels map[string]string) (map[string]string,
 	return topology, topologyAffinity
 }
 
-func rookTopologyLabelsOrdered() []string {
-	topologyLabelsOrdered := []string{}
-	for i := len(CRUSHTopologyLabels) - 1; i >= 0; i-- {
-		label := CRUSHTopologyLabels[i]
-		topologyLabelsOrdered = append(topologyLabelsOrdered, topologyLabelPrefix+label)
-	}
-	return topologyLabelsOrdered
-}
-
-func allKubernetesTopologyLabelsOrdered() []string {
-	return append(
-		append([]string{corev1.LabelTopologyRegion,
-			corev1.LabelTopologyZone},
-			rookTopologyLabelsOrdered()...),
-		labelHostname, //  host is the lowest level in the crush map hierarchy
-	)
-}
-
-func kubernetesTopologyLabelToCRUSHLabel(label string) string {
-	crushLabel := strings.Split(label, "/")
-	if crushLabel[len(crushLabel)-1] == "hostname" {
-		// kubernetes uses "kubernetes.io/hostname" whereas CRUSH uses "host"
-		return "host"
-	}
-	return crushLabel[len(crushLabel)-1]
-}
-
 // ExtractTopologyFromLabels extracts rook topology from labels and returns a map from topology type to value
 func extractTopologyFromLabels(labels map[string]string) (map[string]string, string) {
 	topology := make(map[string]string)
@@ -91,34 +63,59 @@ func extractTopologyFromLabels(labels map[string]string) (map[string]string, str
 	// The topology affinity for the osd is the lowest topology label found in the hierarchy,
 	// not including the host name
 	var topologyAffinity string
-	allKubernetesTopologyLabels := allKubernetesTopologyLabelsOrdered()
+
+	// check for the region k8s topology label that was deprecated in 1.17
+	const regionLabel = "region"
+	region, ok := labels[corev1.LabelZoneRegion]
+	if ok {
+		topology[regionLabel] = region
+		topologyAffinity = formatTopologyAffinity(corev1.LabelZoneRegion, region)
+	}
+
+	// check for the region k8s topology label that is GA in 1.17.
+	region, ok = labels[corev1.LabelZoneRegionStable]
+	if ok {
+		topology[regionLabel] = region
+		topologyAffinity = formatTopologyAffinity(corev1.LabelZoneRegionStable, region)
+	}
+
+	// check for the zone k8s topology label that was deprecated in 1.17
+	const zoneLabel = "zone"
+	zone, ok := labels[corev1.LabelZoneFailureDomain]
+	if ok {
+		topology[zoneLabel] = zone
+		topologyAffinity = formatTopologyAffinity(corev1.LabelZoneFailureDomain, zone)
+	}
+
+	// check for the zone k8s topology label that is GA in 1.17.
+	zone, ok = labels[corev1.LabelZoneFailureDomainStable]
+	if ok {
+		topology[zoneLabel] = zone
+		topologyAffinity = formatTopologyAffinity(corev1.LabelZoneFailureDomainStable, zone)
+	}
+
+	// ignore the region k8s topology label when it has the same value as the k8s zone label
+	if topology[regionLabel] == topology[zoneLabel] {
+		delete(topology, regionLabel)
+	}
+
+	// get host
+	host, ok := labels[corev1.LabelHostname]
+	if ok {
+		topology["host"] = host
+	}
 
 	// get the labels for the CRUSH map hierarchy
-	// iterate in a way so the last topology found will be the lowest level in the hierarchy
+	// iterate in reverse order so that the last topology found will be the lowest level in the hierarchy
 	// for the topology affinity
-	for _, label := range allKubernetesTopologyLabels {
-		topologyID := kubernetesTopologyLabelToCRUSHLabel(label)
+	for i := len(CRUSHTopologyLabels) - 1; i >= 0; i-- {
+		topologyID := CRUSHTopologyLabels[i]
+		label := topologyLabelPrefix + topologyID
 		if value, ok := labels[label]; ok {
 			topology[topologyID] = value
-			if topologyID != "host" {
-				topologyAffinity = formatTopologyAffinity(label, value)
-			}
+			topologyAffinity = formatTopologyAffinity(label, value)
 		}
 	}
-	// iterate in lowest to highest order as the lowest level should be sustained and higher level duplicate
-	// should be removed
-	duplicateTopology := make(map[string]int)
-	for i := len(allKubernetesTopologyLabels) - 1; i >= 0; i-- {
-		topologyLabel := allKubernetesTopologyLabels[i]
-		if value, ok := labels[topologyLabel]; ok {
-			if _, ok := duplicateTopology[value]; ok {
-				delete(topology, kubernetesTopologyLabelToCRUSHLabel(topologyLabel))
-			} else {
-				duplicateTopology[value] = 1
-			}
-		}
-	}
-
 	return topology, topologyAffinity
 }
 
diff --git a/pkg/operator/ceph/config/monstore.go b/pkg/operator/ceph/config/monstore.go
index 2eaf5caa3..086551e69 100644
--- a/pkg/operator/ceph/config/monstore.go
+++ b/pkg/operator/ceph/config/monstore.go
@@ -236,7 +236,7 @@ func (m *MonStore) SetAll(clientName string, settings map[string]string) error {
 
 	out, err := cephCmd.RunWithTimeout(exec.CephCommandsTimeout)
 	if err != nil {
-		logger.Errorf("failed to run command ceph %s: %v", args, err)
+		logger.Errorf("failed to run command ceph %s", args)
 
 		fileContent, err := os.ReadFile(assimilateConfPath.Name() + ".out")
 		if err != nil {
diff --git a/pkg/operator/ceph/csi/betav1csidriver.go b/pkg/operator/ceph/csi/betav1csidriver.go
index 8ffd38539..cf7b487b8 100644
--- a/pkg/operator/ceph/csi/betav1csidriver.go
+++ b/pkg/operator/ceph/csi/betav1csidriver.go
@@ -73,9 +73,9 @@ func (d beta1CsiDriver) createCSIDriverInfo(ctx context.Context, clientset kuber
 		return err
 	}
 
-	// As FSGroupPolicy and AttachRequired fields are immutable, should be set only during create time.
-	// if the request is to change the FSGroupPolicy or AttachRequired, we are deleting the CSIDriver object and creating it.
-	if (driver.Spec.FSGroupPolicy != nil && csiDriver.Spec.FSGroupPolicy != nil && *driver.Spec.FSGroupPolicy != *csiDriver.Spec.FSGroupPolicy) || *driver.Spec.AttachRequired != *csiDriver.Spec.AttachRequired {
+	// As FSGroupPolicy field is immutable, should be set only during create time.
+	// if the request is to change the FSGroupPolicy, we are deleting the CSIDriver object and creating it.
+	if driver.Spec.FSGroupPolicy != nil && csiDriver.Spec.FSGroupPolicy != nil && *driver.Spec.FSGroupPolicy != *csiDriver.Spec.FSGroupPolicy {
 		d.csiClient = csidrivers
 		d.csiDriver = csiDriver
 		return d.reCreateCSIDriverInfo(ctx)
diff --git a/pkg/operator/ceph/csi/cluster_config.go b/pkg/operator/ceph/csi/cluster_config.go
index a9d62aa0e..043b094e6 100644
--- a/pkg/operator/ceph/csi/cluster_config.go
+++ b/pkg/operator/ceph/csi/cluster_config.go
@@ -108,10 +108,13 @@ func MonEndpoints(mons map[string]*cephclient.MonInfo, requireMsgr2 bool) []stri
 		endpoint := m.Endpoint
 		if requireMsgr2 {
 			logger.Debugf("evaluating mon %q for msgr1 on endpoint %q", m.Name, m.Endpoint)
-			msgr1Suffix := fmt.Sprintf(":%d", client.Msgr1port)
-			if strings.HasSuffix(m.Endpoint, msgr1Suffix) {
-				address := m.Endpoint[0:strings.LastIndex(m.Endpoint, msgr1Suffix)]
-				endpoint = fmt.Sprintf("%s:%d", address, client.Msgr2port)
+			if strings.HasSuffix(m.Endpoint, fmt.Sprintf(":%d", client.Msgr1port)) {
+				parts := strings.Split(m.Endpoint, ":")
+				if len(parts) != 2 {
+					logger.Errorf("endpoint %q does not contain two parts to extract the port", m.Endpoint)
+					continue
+				}
+				endpoint = fmt.Sprintf("%s:%d", parts[0], client.Msgr2port)
 				logger.Debugf("mon %q will use the msgrv2 port: %q", m.Name, endpoint)
 			}
 		}
diff --git a/pkg/operator/ceph/csi/csi.go b/pkg/operator/ceph/csi/csi.go
index 0ced8b3f4..2e441037c 100644
--- a/pkg/operator/ceph/csi/csi.go
+++ b/pkg/operator/ceph/csi/csi.go
@@ -283,17 +283,5 @@ func (r *ReconcileCSI) setParams(ver *version.Info) error {
 	CSIParam.ImagePullPolicy = k8sutil.GetValue(r.opConfig.Parameters, "ROOK_CSI_IMAGE_PULL_POLICY", DefaultCSIImagePullPolicy)
 	CSIParam.CephFSKernelMountOptions = k8sutil.GetValue(r.opConfig.Parameters, "CSI_CEPHFS_KERNEL_MOUNT_OPTIONS", "")
 
-	CSIParam.CephFSAttachRequired = true
-	if strings.EqualFold(k8sutil.GetValue(r.opConfig.Parameters, "CSI_CEPHFS_ATTACH_REQUIRED", "true"), "false") {
-		CSIParam.CephFSAttachRequired = false
-	}
-	CSIParam.RBDAttachRequired = true
-	if strings.EqualFold(k8sutil.GetValue(r.opConfig.Parameters, "CSI_RBD_ATTACH_REQUIRED", "true"), "false") {
-		CSIParam.RBDAttachRequired = false
-	}
-	CSIParam.NFSAttachRequired = true
-	if strings.EqualFold(k8sutil.GetValue(r.opConfig.Parameters, "CSI_NFS_ATTACH_REQUIRED", "true"), "false") {
-		CSIParam.NFSAttachRequired = false
-	}
 	return nil
 }
diff --git a/pkg/operator/ceph/csi/csidriver.go b/pkg/operator/ceph/csi/csidriver.go
index 1159fd098..19d47408d 100644
--- a/pkg/operator/ceph/csi/csidriver.go
+++ b/pkg/operator/ceph/csi/csidriver.go
@@ -68,9 +68,9 @@ func (d v1CsiDriver) createCSIDriverInfo(ctx context.Context, clientset kubernet
 		return err
 	}
 
-	// As FSGroupPolicy and AttachRequired fields are immutable, should be set only during create time.
-	// if the request is to change the FSGroupPolicy or AttachRequired, we are deleting the CSIDriver object and creating it.
-	if (driver.Spec.FSGroupPolicy != nil && csiDriver.Spec.FSGroupPolicy != nil && *driver.Spec.FSGroupPolicy != *csiDriver.Spec.FSGroupPolicy) || *driver.Spec.AttachRequired != *csiDriver.Spec.AttachRequired {
+	// As FSGroupPolicy field is immutable, should be set only during create time.
+	// if the request is to change the FSGroupPolicy, we are deleting the CSIDriver object and creating it.
+	if driver.Spec.FSGroupPolicy != nil && csiDriver.Spec.FSGroupPolicy != nil && *driver.Spec.FSGroupPolicy != *csiDriver.Spec.FSGroupPolicy {
 		d.csiClient = csidrivers
 		d.csiDriver = csiDriver
 		return d.reCreateCSIDriverInfo(ctx)
diff --git a/pkg/operator/ceph/csi/spec.go b/pkg/operator/ceph/csi/spec.go
index 1886373fb..cbe26cecd 100644
--- a/pkg/operator/ceph/csi/spec.go
+++ b/pkg/operator/ceph/csi/spec.go
@@ -78,9 +78,6 @@ type Param struct {
 	EnableCSITopology                     bool
 	EnableLiveness                        bool
 	EnableReadAffinity                    bool
-	CephFSAttachRequired                  bool
-	RBDAttachRequired                     bool
-	NFSAttachRequired                     bool
 	LogLevel                              uint8
 	SidecarLogLevel                       uint8
 	CephFSGRPCMetricsPort                 uint16
@@ -647,19 +644,19 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 	}
 
 	if EnableRBD {
-		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, RBDDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_RBD_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), tp.Param.RBDAttachRequired)
+		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, RBDDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_RBD_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), true)
 		if err != nil {
 			return errors.Wrapf(err, "failed to create CSI driver object for %q", RBDDriverName)
 		}
 	}
 	if EnableCephFS {
-		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, CephFSDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_CEPHFS_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), tp.Param.CephFSAttachRequired)
+		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, CephFSDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_CEPHFS_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), true)
 		if err != nil {
 			return errors.Wrapf(err, "failed to create CSI driver object for %q", CephFSDriverName)
 		}
 	}
 	if EnableNFS {
-		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, NFSDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_NFS_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), tp.Param.NFSAttachRequired)
+		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, NFSDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_NFS_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), false)
 		if err != nil {
 			return errors.Wrapf(err, "failed to create CSI driver object for %q", NFSDriverName)
 		}
diff --git a/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml b/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml
index f2ba703df..f118b5797 100644
--- a/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml
+++ b/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml
@@ -22,7 +22,6 @@ spec:
       priorityClassName: {{ .ProvisionerPriorityClassName }}
       {{ end }}
       containers:
-        {{ if .CephFSAttachRequired }}
         - name: csi-attacher
           image: {{ .AttacherImage }}
           args:
@@ -34,6 +33,7 @@ spec:
             - "--leader-election-lease-duration=137s"
             - "--leader-election-renew-deadline=107s"
             - "--leader-election-retry-period=26s"
+            - "--default-fstype=ext4"
           env:
             - name: ADDRESS
               value: /csi/csi-provisioner.sock
@@ -41,7 +41,6 @@ spec:
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
-        {{ end }}
         {{ if .EnableCephFSSnapshotter }}
         - name: csi-snapshotter
           image:  {{ .SnapshotterImage }}
diff --git a/pkg/operator/ceph/csi/template/nfs/csi-nfsplugin-provisioner-dep.yaml b/pkg/operator/ceph/csi/template/nfs/csi-nfsplugin-provisioner-dep.yaml
index b8aeebda9..7cf5961b8 100644
--- a/pkg/operator/ceph/csi/template/nfs/csi-nfsplugin-provisioner-dep.yaml
+++ b/pkg/operator/ceph/csi/template/nfs/csi-nfsplugin-provisioner-dep.yaml
@@ -21,26 +21,6 @@ spec:
       priorityClassName: {{ .ProvisionerPriorityClassName }}
       {{ end }}
       containers:
-        {{ if .NFSAttachRequired }}
-        - name: csi-attacher
-          image: {{ .AttacherImage }}
-          args:
-            - "--v={{ .SidecarLogLevel }}"
-            - "--csi-address=$(ADDRESS)"
-            - "--leader-election=true"
-            - "--timeout={{ .GRPCTimeout }}"
-            - "--leader-election-namespace={{ .Namespace }}"
-            - "--leader-election-lease-duration=137s"
-            - "--leader-election-renew-deadline=107s"
-            - "--leader-election-retry-period=26s"
-          env:
-            - name: ADDRESS
-              value: /csi/csi-provisioner.sock
-          imagePullPolicy: {{ .ImagePullPolicy }}
-          volumeMounts:
-            - name: socket-dir
-              mountPath: /csi
-        {{ end }}
         {{ if .EnableNFSSnapshotter }}
         - name: csi-snapshotter
           image:  {{ .SnapshotterImage }}
diff --git a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
index 3d453c7e1..17fc28ba2 100644
--- a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
+++ b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
@@ -65,7 +65,6 @@ spec:
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
-        {{ if .RBDAttachRequired }}
         - name: csi-attacher
           image: {{ .AttacherImage }}
           args:
@@ -85,7 +84,6 @@ spec:
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
-        {{ end }}
         {{ if .EnableRBDSnapshotter }}
         - name: csi-snapshotter
           image:  {{ .SnapshotterImage }}
diff --git a/pkg/operator/ceph/nfs/spec.go b/pkg/operator/ceph/nfs/spec.go
index cf5d37f84..3ae741899 100644
--- a/pkg/operator/ceph/nfs/spec.go
+++ b/pkg/operator/ceph/nfs/spec.go
@@ -35,11 +35,10 @@ import (
 
 const (
 	// AppName is the name of the app
-	AppName               = "rook-ceph-nfs"
-	ganeshaConfigVolume   = "ganesha-config"
-	nfsPort               = 2049
-	ganeshaPid            = "/var/run/ganesha/ganesha.pid"
-	nfsGaneshaMetricsPort = 9587
+	AppName             = "rook-ceph-nfs"
+	ganeshaConfigVolume = "ganesha-config"
+	nfsPort             = 2049
+	ganeshaPid          = "/var/run/ganesha/ganesha.pid"
 )
 
 func (r *ReconcileCephNFS) generateCephNFSService(nfs *cephv1.CephNFS, cfg daemonConfig) *v1.Service {
@@ -60,18 +59,11 @@ func (r *ReconcileCephNFS) generateCephNFSService(nfs *cephv1.CephNFS, cfg daemo
 					TargetPort: intstr.FromInt(int(nfsPort)),
 					Protocol:   v1.ProtocolTCP,
 				},
-				{
-					Name:       "nfs-metrics",
-					Port:       nfsGaneshaMetricsPort,
-					TargetPort: intstr.FromInt(int(nfsGaneshaMetricsPort)),
-					Protocol:   v1.ProtocolTCP,
-				},
 			},
 		},
 	}
 
-	hostNetwork := nfs.IsHostNetwork(r.cephClusterSpec)
-	if hostNetwork {
+	if r.cephClusterSpec.Network.IsHost() {
 		svc.Spec.ClusterIP = v1.ClusterIPNone
 	}
 
@@ -109,11 +101,6 @@ func (r *ReconcileCephNFS) makeDeployment(nfs *cephv1.CephNFS, cfg daemonConfig)
 			Labels:    getLabels(nfs, cfg.ID, true),
 		},
 	}
-
-	// If host network is defined on Spec.Server.HostNetwork, use it.
-	// elsedefault to whatever the cluster has defined
-	hostNetwork := nfs.IsHostNetwork(r.cephClusterSpec)
-
 	k8sutil.AddRookVersionLabelToDeployment(deployment)
 	controller.AddCephVersionLabelToDeployment(r.clusterInfo.CephVersion, deployment)
 	nfs.Spec.Server.Annotations.ApplyToObjectMeta(&deployment.ObjectMeta)
@@ -141,7 +128,7 @@ func (r *ReconcileCephNFS) makeDeployment(nfs *cephv1.CephNFS, cfg daemonConfig)
 			nfsConfigVol,
 			dbusVol,
 		},
-		HostNetwork:       hostNetwork,
+		HostNetwork:       r.cephClusterSpec.Network.IsHost(),
 		PriorityClassName: nfs.Spec.Server.PriorityClassName,
 		// for kerberos, nfs-ganesha uses the hostname via getaddrinfo() and uses that when
 		// connecting to the krb server. give all ganesha servers the same hostname so they can all
@@ -156,7 +143,7 @@ func (r *ReconcileCephNFS) makeDeployment(nfs *cephv1.CephNFS, cfg daemonConfig)
 	// Replace default unreachable node toleration
 	k8sutil.AddUnreachableNodeToleration(&podSpec)
 
-	if hostNetwork {
+	if r.cephClusterSpec.Network.IsHost() {
 		podSpec.DNSPolicy = v1.DNSClusterFirstWithHostNet
 	}
 	nfs.Spec.Server.Placement.ApplyToPodSpec(&podSpec)
@@ -179,7 +166,7 @@ func (r *ReconcileCephNFS) makeDeployment(nfs *cephv1.CephNFS, cfg daemonConfig)
 		Spec: podSpec,
 	}
 
-	if hostNetwork {
+	if r.cephClusterSpec.Network.IsHost() {
 		podSpec.DNSPolicy = v1.DNSClusterFirstWithHostNet
 	} else if r.cephClusterSpec.Network.IsMultus() {
 		if err := k8sutil.ApplyMultus(r.cephClusterSpec.Network, &podTemplateSpec.ObjectMeta); err != nil {
diff --git a/pkg/operator/ceph/object/notification/obc_label_controller.go b/pkg/operator/ceph/object/notification/obc_label_controller.go
index 4aa07ca33..1a5f0d8cc 100644
--- a/pkg/operator/ceph/object/notification/obc_label_controller.go
+++ b/pkg/operator/ceph/object/notification/obc_label_controller.go
@@ -44,9 +44,7 @@ import (
 )
 
 const (
-	notificationLabelPrefix   = "bucket-notification-"
-	bucketProvisionerLabelKey = "bucket-provisioner"
-	bucketProvisionerLabelVal = "ceph.rook.io-bucket"
+	notificationLabelPrefix = "bucket-notification-"
 )
 
 // ReconcileOBCLabels reconciles a ObjectBucketClaim labels
@@ -121,18 +119,11 @@ func (r *ReconcileOBCLabels) reconcile(request reconcile.Request) (reconcile.Res
 	if err := r.client.Get(r.opManagerContext, bucketName, &ob); err != nil {
 		return reconcile.Result{}, errors.Wrapf(err, "failed to retrieve ObjectBucket %q", bucketName)
 	}
-
-	// validate if the bucket is provisioned by the ceph provisioner
-	if !strings.Contains(ob.Labels[bucketProvisionerLabelKey], bucketProvisionerLabelVal) {
-		logger.Debugf("ObjectBucket %q was not provisioned by the ceph object store provisioner and tagged with provisioner %q. ignoring",
-			bucketName, ob.Labels[bucketProvisionerLabelKey])
-		return reconcile.Result{}, nil
-	}
-	// validate object store name
 	objectStoreName, err := getCephObjectStoreName(ob)
 	if err != nil {
 		return reconcile.Result{}, errors.Wrapf(err, "failed to get object store from ObjectBucket %q", bucketName)
 	}
+
 	// Populate clusterInfo during each reconcile
 	clusterInfo, clusterSpec, err := getReadyCluster(r.client, r.opManagerContext, *r.context, objectStoreName.Namespace)
 	if err != nil {
diff --git a/pkg/operator/ceph/object/rgw.go b/pkg/operator/ceph/object/rgw.go
index 889b7494a..b396c5b9c 100644
--- a/pkg/operator/ceph/object/rgw.go
+++ b/pkg/operator/ceph/object/rgw.go
@@ -82,12 +82,8 @@ func (c *clusterConfig) createOrUpdateStore(realmName, zoneGroupName, zoneName s
 		return nil
 	}
 
-	if c.clusterSpec.Dashboard.Enabled {
-		if !c.store.Spec.IsRGWDashboardEnabled() {
-			disableRGWDashboard(objContext)
-		} else if err = enableRGWDashboard(objContext); err != nil {
-			logger.Warningf("failed to enable dashboard for rgw. %v", err)
-		}
+	if err = enableRGWDashboard(objContext); err != nil {
+		logger.Warningf("failed to enable dashboard for rgw. %v", err)
 	}
 
 	logger.Infof("created object store %q in namespace %q", c.store.Name, c.store.Namespace)
diff --git a/pkg/operator/discover/discover.go b/pkg/operator/discover/discover.go
index 26e68f06d..c5331012f 100644
--- a/pkg/operator/discover/discover.go
+++ b/pkg/operator/discover/discover.go
@@ -357,7 +357,9 @@ func ListDevicesInUse(ctx context.Context, clusterdContext *clusterd.Context, na
 			logger.Warningf("failed to unmarshal %s", deviceJson)
 			continue
 		}
-		devices = append(devices, d...)
+		for i := range d {
+			devices = append(devices, d[i])
+		}
 	}
 	logger.Debugf("devices in use %+v", devices)
 	return devices, nil
diff --git a/pkg/util/file.go b/pkg/util/file.go
index e43c02674..e5dc0f385 100644
--- a/pkg/util/file.go
+++ b/pkg/util/file.go
@@ -70,7 +70,7 @@ func CreateTempFile(content string) (*os.File, error) {
 	}
 
 	// Write content into file
-	err = os.WriteFile(file.Name(), []byte(content), 0400)
+	err = os.WriteFile(file.Name(), []byte(content), 0440)
 	if err != nil {
 		return nil, errors.Wrap(err, "failed to write content into file")
 	}


Test patch:
diff --git a/.github/workflows/canary-integration-test.yml b/.github/workflows/canary-integration-test.yml
index d99631c84..2910d9491 100644
--- a/.github/workflows/canary-integration-test.yml
+++ b/.github/workflows/canary-integration-test.yml
@@ -53,16 +53,12 @@ jobs:
       - name: wait for ceph to be ready
         run: tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready all 2
 
-      - name: wait for ceph mgr to be ready
+      - name: test external script create-external-cluster-resources.py
         run: |
           toolbox=$(kubectl get pod -l app=rook-ceph-tools -n rook-ceph -o jsonpath='{.items[*].metadata.name}')
           timeout 15 sh -c "until kubectl -n rook-ceph exec $toolbox -- ceph mgr dump -f json|jq --raw-output .active_addr|grep -Eosq \"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\" ; do sleep 1 && echo 'waiting for the manager IP to be available'; done"
           mgr_raw=$(kubectl -n rook-ceph exec $toolbox -- ceph mgr dump -f json|jq --raw-output .active_addr)
           timeout 60 sh -c "until kubectl -n rook-ceph exec $toolbox -- curl --silent --show-error ${mgr_raw%%:*}:9283; do echo 'waiting for mgr prometheus exporter to be ready' && sleep 1; done"
-
-      - name: test external script create-external-cluster-resources.py
-        run: |
-          toolbox=$(kubectl get pod -l app=rook-ceph-tools -n rook-ceph -o jsonpath='{.items[*].metadata.name}')
           kubectl -n rook-ceph exec $toolbox -- mkdir -p /etc/ceph/test-data
           kubectl -n rook-ceph cp tests/ceph-status-out $toolbox:/etc/ceph/test-data/
           kubectl -n rook-ceph cp deploy/examples/create-external-cluster-resources.py $toolbox:/etc/ceph
@@ -171,7 +167,8 @@ jobs:
           timeout 15 sh -c "until kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rgw-endpoint $rgw_endpoint; do sleep 1 && echo 'waiting for the rgw endpoint to be validated'; done"
           # pass the invalid rgw-endpoint of different ceph cluster
           if output=$(timeout 15 sh -c "until kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rgw-endpoint 10.108.96.128:80; do sleep 1 && echo 'waiting for the rgw endpoint to be validated'; done"); then
-            echo "script run completed with stderr error after passing the wrong rgw-endpoint: $output"
+            echo "unexpectedly succeeded after passing the wrong rgw-endpoint: $output"
+            exit 1
           else
             echo "validation failed because wrong endpoint was provided"
           fi
@@ -903,6 +900,9 @@ jobs:
           timeout 120 bash -c 'while kubectl -n rook-ceph get cephobjectstore my-store; do echo "waiting for objectstore my-store to delete"; sleep 5; done'
           kubectl create -f tests/manifests/test-object.yaml
           tests/scripts/validate_cluster.sh rgw
+          # the test is already waiting for the rgw daemon to start, but this is just a hack until we find
+          # what else we need to wait for
+          sleep 30
           tests/scripts/deploy-validate-vault.sh validate_rgw
 
       - name: collect common logs
@@ -1182,6 +1182,7 @@ jobs:
           cp deploy/examples/filesystem-test.yaml "$PRIMARY_YAML"
           yq merge --inplace --arrays append "$PRIMARY_YAML" tests/manifests/test-fs-mirror-spec.yaml
           kubectl create -f "$PRIMARY_YAML"
+          timeout 200 sh -c 'until [ "$(kubectl -n rook-ceph get cephfilesystem myfs -o jsonpath='{.status.phase}'|grep -c "Ready")" -eq 1 ]; do echo "waiting for filesystem to be created" && sleep 1; done'
 
       - name: create replicated mirrored filesystem on cluster 2
         run: |
@@ -1189,14 +1190,7 @@ jobs:
           yq w -i filesystem-test.yaml metadata.namespace rook-ceph-secondary
           yq w -i filesystem-test.yaml spec.mirroring.enabled true
           kubectl create -f filesystem-test.yaml
-
-      - name: wait for filesystem on cluster 1
-        run: |
-          timeout 300 sh -c 'until [ "$(kubectl -n rook-ceph get cephfilesystem myfs -o jsonpath='{.status.phase}'|grep -c "Ready")" -eq 1 ]; do echo "waiting for filesystem to be created" && sleep 1; done'
-
-      - name: wait for filesystem on cluster 2
-        run: |
-          timeout 300 sh -c 'until [ "$(kubectl -n rook-ceph-secondary get cephfilesystem myfs -o jsonpath='{.status.phase}'|grep -c "Ready")" -eq 1 ]; do echo "waiting for filesystem to be created" && sleep 1; done'
+          timeout 120 sh -c 'until [ "$(kubectl -n rook-ceph-secondary get cephfilesystem myfs -o jsonpath='{.status.phase}'|grep -c "Ready")" -eq 1 ]; do echo "waiting for filesystem to be created" && sleep 1; done'
 
       - name: copy filesystem mirror peer secret from the secondary cluster to the primary one
         run: |
@@ -1232,12 +1226,12 @@ jobs:
 
       - name: Deploy Direct Tools pod on primary cluster
         run: |
-          tests/scripts/github-action-helper.sh deploy_manifest_with_local_build deploy/examples/direct-mount.yaml
+          kubectl -n rook-ceph create -f deploy/examples/direct-mount.yaml
 
       - name: Deploy Direct Tools pod on secondary cluster
         run: |
           sed -i "s/rook-ceph # namespace/rook-ceph-secondary # namespace/" deploy/examples/direct-mount.yaml
-          tests/scripts/github-action-helper.sh deploy_manifest_with_local_build deploy/examples/direct-mount.yaml
+          kubectl -n rook-ceph-secondary create -f deploy/examples/direct-mount.yaml
 
       - name: Configure a directory for snapshot mirroring on primary cluster
         run: |
@@ -1266,17 +1260,17 @@ jobs:
           EOF
           clusterfsid=$($exec_fs_mirror ceph --admin-daemon /var/run/ceph/$mirror_daemon fs mirror status myfs@1 |jq -r '.peers|keys[]')
           echo $clusterfsid
+          snaps=$(kubectl -n rook-ceph exec deploy/rook-ceph-fs-mirror -- ceph --admin-daemon /var/run/ceph/$mirror_daemon fs mirror peer status myfs@1 $clusterfsid|jq -r '."/volumes/_nogroup/testsubvolume"."snaps_synced"')
+          echo $snaps
           kubectl -n rook-ceph-secondary wait pod -l app=rook-direct-mount --for condition=Ready --timeout=300s
           kubectl -n rook-ceph-secondary exec deploy/rook-direct-mount -- mkdir /tmp/registry
           mon_endpoints=$(kubectl -n rook-ceph-secondary exec deploy/rook-direct-mount -- grep mon_host /etc/ceph/ceph.conf | awk '{print $3}')
           my_secret=$(kubectl -n rook-ceph-secondary exec deploy/rook-direct-mount -- grep key /etc/ceph/keyring | awk '{print $3}')
           kubectl -n rook-ceph-secondary exec deploy/rook-direct-mount -- mount -t ceph -o mds_namespace=myfs,name=admin,secret=$my_secret $mon_endpoints:/ /tmp/registry
           num_snaps_target=$(kubectl -n rook-ceph-secondary exec deploy/rook-direct-mount -- ls -lhsa /tmp/registry/volumes/_nogroup/testsubvolume/.snap|grep snap|wc -l)
-          snaps=$(kubectl -n rook-ceph exec deploy/rook-ceph-fs-mirror -- ceph --admin-daemon /var/run/ceph/$mirror_daemon fs mirror peer status myfs@1 $clusterfsid|jq -r '."/volumes/_nogroup/testsubvolume"."snaps_synced"')
-          echo "snapshots: $snaps"
           if [ $num_snaps_target = $snaps ]
           then echo "Snaphots have synced."
-          else echo "Snaps have not synced. NEEDS INVESTIGATION"
+          else echo "Snaps have not synced" ; exit 1
           fi
 
       - name: collect common logs
@@ -1363,7 +1357,7 @@ jobs:
           fetch-depth: 0
 
       - name: setup golang
-        uses: actions/setup-go@v4
+        uses: actions/setup-go@v3
         with:
           go-version: 1.19
 
diff --git a/.github/workflows/unit-test.yml b/.github/workflows/unit-test.yml
index 10ad89e3c..bad06cd75 100644
--- a/.github/workflows/unit-test.yml
+++ b/.github/workflows/unit-test.yml
@@ -34,7 +34,7 @@ jobs:
         with:
           fetch-depth: 0
 
-      - uses: actions/setup-go@v4
+      - uses: actions/setup-go@v3
         with:
           go-version: 1.19
 
diff --git a/deploy/examples/cluster-test.yaml b/deploy/examples/cluster-test.yaml
index 5bfa62515..70a367027 100644
--- a/deploy/examples/cluster-test.yaml
+++ b/deploy/examples/cluster-test.yaml
@@ -45,8 +45,6 @@ spec:
     useAllNodes: true
     useAllDevices: true
     #deviceFilter:
-  monitoring:
-    enabled: false
   healthCheck:
     daemonHealth:
       mon:
diff --git a/deploy/examples/create-external-cluster-resources-tests.py b/deploy/examples/create-external-cluster-resources-tests.py
index aaf01d1c5..f1e3a18e0 100644
--- a/deploy/examples/create-external-cluster-resources-tests.py
+++ b/deploy/examples/create-external-cluster-resources-tests.py
@@ -62,24 +62,32 @@ class TestRadosJSON(unittest.TestCase):
             self.rjObj.main()
             self.fail("Function should have thrown an Exception")
         except ext.ExecutionFailureException as err:
-            print(f"Exception thrown successfully: {err}")
+            print("Exception thrown successfully: {}".format(err))
 
     def test_method_create_cephCSIKeyring_cephFSProvisioner(self):
         csiKeyring = self.rjObj.create_cephCSIKeyring_user(
             "client.csi-cephfs-provisioner"
         )
-        print(f"cephCSIKeyring without restricting it to a metadata pool. {csiKeyring}")
+        print(
+            "cephCSIKeyring without restricting it to a metadata pool. {}".format(
+                csiKeyring
+            )
+        )
         self.rjObj._arg_parser.restricted_auth_permission = True
         self.rjObj._arg_parser.cluster_name = "openshift-storage"
         csiKeyring = self.rjObj.create_cephCSIKeyring_user(
             "client.csi-cephfs-provisioner"
         )
-        print(f"cephCSIKeyring for a specific cluster. {csiKeyring}")
+        print("cephCSIKeyring for a specific cluster. {}".format(csiKeyring))
         self.rjObj._arg_parser.cephfs_filesystem_name = "myfs"
         csiKeyring = self.rjObj.create_cephCSIKeyring_user(
             "client.csi-cephfs-provisioner"
         )
-        print(f"cephCSIKeyring for a specific metadata pool and cluster. {csiKeyring}")
+        print(
+            "cephCSIKeyring for a specific metadata pool and cluster. {}".format(
+                csiKeyring
+            )
+        )
 
     def test_non_zero_return_and_error(self):
         self.rjObj.cluster.return_val = 1
@@ -88,7 +96,7 @@ class TestRadosJSON(unittest.TestCase):
             self.rjObj.create_checkerKey()
             self.fail("Failed to raise an exception, 'ext.ExecutionFailureException'")
         except ext.ExecutionFailureException as err:
-            print(f"Successfully thrown error.\nError: {err}")
+            print("Successfully thrown error.\nError: {}".format(err))
 
     def test_multi_filesystem_scenario(self):
         cmd_key = self.rjObj.cluster.cmd_names["fs ls"]
@@ -105,28 +113,30 @@ class TestRadosJSON(unittest.TestCase):
             print("As we are returning silently, no error thrown as expected")
         except ext.ExecutionFailureException as err:
             self.fail(
-                f"Supposed to get returned silently, but instead error thrown: {err}"
+                "Supposed to get returned silently, but instead error thrown: {}".format(
+                    err
+                )
             )
         # pass an existing filesystem name
         try:
             self.rjObj._arg_parser.cephfs_filesystem_name = second_fs_details["name"]
             self.rjObj.get_cephfs_data_pool_details()
         except ext.ExecutionFailureException as err:
-            self.fail(f"Should not have thrown error: {err}")
+            self.fail("Should not have thrown error: {}".format(err))
         # pass a non-existing filesystem name
         try:
             self.rjObj._arg_parser.cephfs_filesystem_name += "-non-existing-fs-name"
             self.rjObj.get_cephfs_data_pool_details()
             self.fail("An Exception was expected to be thrown")
         except ext.ExecutionFailureException as err:
-            print(f"Successfully thrown error: {err}")
+            print("Successfully thrown error: {}".format(err))
         # empty file-system array
         try:
             self.rjObj.cluster.cmd_output_map[cmd_key] = json.dumps([])
             self.rjObj.get_cephfs_data_pool_details()
             self.fail("An Exception was expected to be thrown")
         except ext.ExecutionFailureException as err:
-            print(f"Successfully thrown error: {err}")
+            print("Successfully thrown error: {}".format(err))
 
     def test_multi_data_pool_scenario(self):
         cmd_key = self.rjObj.cluster.cmd_names["fs ls"]
@@ -135,7 +145,7 @@ class TestRadosJSON(unittest.TestCase):
         first_fs_details = cmd_json_out[0]
         new_data_pool_name = "myfs-data1"
         first_fs_details["data_pools"].append(new_data_pool_name)
-        print(f"Modified JSON Cmd Out: {cmd_json_out}")
+        print("Modified JSON Cmd Out: {}".format(cmd_json_out))
         self.rjObj._arg_parser.cephfs_data_pool_name = new_data_pool_name
         self.rjObj.cluster.cmd_output_map[cmd_key] = json.dumps(cmd_json_out)
         self.rjObj.get_cephfs_data_pool_details()
@@ -146,7 +156,7 @@ class TestRadosJSON(unittest.TestCase):
             self.rjObj.get_cephfs_data_pool_details()
             self.fail("An Exception was expected to be thrown")
         except ext.ExecutionFailureException as err:
-            print(f"Successfully thrown error: {err}")
+            print("Successfully thrown error: {}".format(err))
         # empty data-pool scenario
         first_fs_details["data_pools"] = []
         self.rjObj.cluster.cmd_output_map[cmd_key] = json.dumps(cmd_json_out)
@@ -154,7 +164,7 @@ class TestRadosJSON(unittest.TestCase):
             self.rjObj.get_cephfs_data_pool_details()
             self.fail("An Exception was expected to be thrown")
         except ext.ExecutionFailureException as err:
-            print(f"Successfully thrown error: {err}")
+            print("Successfully thrown error: {}".format(err))
 
     def test_valid_rgw_endpoint(self):
         self.rjObj._invalid_endpoint("10.10.212.133:8000")
@@ -163,38 +173,42 @@ class TestRadosJSON(unittest.TestCase):
             self.rjObj._invalid_endpoint("10.10.212.133:238000")
             self.fail("An Exception was expected to be thrown")
         except ext.ExecutionFailureException as err:
-            print(f"Successfully thrown error: {err}")
+            print("Successfully thrown error: {}".format(err))
         # out of range IP
         try:
             self.rjObj._invalid_endpoint("10.1033.212.133:8000")
             self.fail("An Exception was expected to be thrown")
         except ext.ExecutionFailureException as err:
-            print(f"Successfully thrown error: {err}")
+            print("Successfully thrown error: {}".format(err))
         # malformatted IP
         try:
             self.rjObj._invalid_endpoint("10.103..212.133:8000")
             self.fail("An Exception was expected to be thrown")
         except ext.ExecutionFailureException as err:
-            print(f"Successfully thrown error: {err}")
+            print("Successfully thrown error: {}".format(err))
         try:
             self.rjObj._invalid_endpoint("10.103.212.133::8000")
             self.fail("An Exception was expected to be thrown")
         except ext.ExecutionFailureException as err:
-            print(f"Successfully thrown error: {err}")
+            print("Successfully thrown error: {}".format(err))
         try:
             self.rjObj._invalid_endpoint("10.10.103.212.133:8000")
             self.fail("An Exception was expected to be thrown")
         except ext.ExecutionFailureException as err:
-            print(f"Successfully thrown error: {err}")
+            print("Successfully thrown error: {}".format(err))
 
     def test_convert_fqdn_rgw_endpoint_to_ip(self):
         try:
             rgw_endpoint_ip = self.rjObj.convert_fqdn_rgw_endpoint_to_ip(
                 "www.redhat.com:80"
             )
-            print(f"Successfully Converted www.redhat.com to it's IP {rgw_endpoint_ip}")
+            print(
+                "Successfully Converted www.redhat.com to it's IP {}".format(
+                    rgw_endpoint_ip
+                )
+            )
         except ext.ExecutionFailureException as err:
-            print(f"Successfully thrown error: {err}")
+            print("Successfully thrown error: {}".format(err))
 
     def test_upgrade_user_permissions(self):
         self.rjObj = ext.RadosJSON(
@@ -236,10 +250,16 @@ class TestRadosJSON(unittest.TestCase):
             mon_ips, mon_port = self.rjObj.get_active_and_standby_mgrs()
             mon_ip = mon_ips.split(",")[0]
             if check_ip_val and check_ip_val != mon_ip:
-                self.fail(f"Expected IP: {check_ip_val}, Returned IP: {mon_ip}")
+                self.fail(
+                    "Expected IP: {}, Returned IP: {}".format(check_ip_val, mon_ip)
+                )
             if check_port_val and check_port_val != mon_port:
-                self.fail(f"Expected Port: {check_port_val}, Returned Port: {mon_port}")
-            print(f"MonIP: {mon_ip}, MonPort: {mon_port}")
+                self.fail(
+                    "Expected Port: '{}', Returned Port: '{}'".format(
+                        check_port_val, mon_port
+                    )
+                )
+            print("MonIP: {}, MonPort: {}".format(mon_ip, mon_port))
 
         invalid_ip_ports = [
             ("10.22.31.131.43", "5334"),
@@ -259,7 +279,7 @@ class TestRadosJSON(unittest.TestCase):
                 self.rjObj._arg_parser.monitoring_endpoint_port = new_mon_port
             try:
                 mon_ip, mon_port = self.rjObj.get_active_and_standby_mgrs()
-                print(f"[Wrong] MonIP: {mon_ip}, MonPort: {mon_port}")
+                print("[Wrong] MonIP: {}, MonPort: {}".format(mon_ip, mon_port))
                 self.fail("An exception was expected")
             except ext.ExecutionFailureException as err:
-                print(f"Exception thrown successfully: {err}")
+                print("Exception thrown successfully: {}".format(err))
diff --git a/pkg/daemon/ceph/osd/device_test.go b/pkg/daemon/ceph/osd/device_test.go
index bf445904d..312d7245e 100644
--- a/pkg/daemon/ceph/osd/device_test.go
+++ b/pkg/daemon/ceph/osd/device_test.go
@@ -23,9 +23,7 @@ import (
 
 	"github.com/rook/rook/pkg/clusterd"
 	"github.com/rook/rook/pkg/daemon/ceph/client"
-	"github.com/rook/rook/pkg/operator/ceph/cluster/osd"
 	exectest "github.com/rook/rook/pkg/util/exec/test"
-	"github.com/rook/rook/pkg/util/sys"
 	"github.com/stretchr/testify/assert"
 )
 
@@ -49,36 +47,3 @@ func TestOSDBootstrap(t *testing.T) {
 	assert.NotEqual(t, -1, strings.Index(string(contents), "key = mysecurekey"))
 	assert.NotEqual(t, -1, strings.Index(string(contents), "caps mon = \"allow profile bootstrap-osd\""))
 }
-
-func TestUpdateDeviceClass(t *testing.T) {
-	d := &DesiredDevice{}
-	agent := &OsdAgent{}
-	disk := &sys.LocalDisk{}
-
-	d.DeviceClass = "test"
-	d.UpdateDeviceClass(agent, disk)
-	assert.Equal(t, "test", d.DeviceClass)
-
-	d.DeviceClass = ""
-	agent.pvcBacked = true
-	t.Setenv(osd.CrushDeviceClassVarName, "test")
-	d.UpdateDeviceClass(agent, disk)
-	assert.Equal(t, "test", d.DeviceClass)
-
-	d.DeviceClass = ""
-	t.Setenv(osd.CrushDeviceClassVarName, "")
-	d.UpdateDeviceClass(agent, disk)
-	t.Log(d)
-	t.Log(disk)
-	assert.Equal(t, "ssd", d.DeviceClass)
-
-	d.DeviceClass = ""
-	agent.pvcBacked = false
-	d.UpdateDeviceClass(agent, disk)
-	assert.Equal(t, "ssd", d.DeviceClass)
-
-	d.DeviceClass = ""
-	agent.storeConfig.DeviceClass = "test"
-	d.UpdateDeviceClass(agent, disk)
-	assert.Equal(t, "test", d.DeviceClass)
-}
diff --git a/pkg/operator/ceph/cluster/cluster_test.go b/pkg/operator/ceph/cluster/cluster_test.go
index 5f6069234..3cc94e585 100644
--- a/pkg/operator/ceph/cluster/cluster_test.go
+++ b/pkg/operator/ceph/cluster/cluster_test.go
@@ -18,9 +18,6 @@ limitations under the License.
 package cluster
 
 import (
-	"flag"
-	"fmt"
-	"strings"
 	"testing"
 	"time"
 
@@ -34,8 +31,6 @@ import (
 	testop "github.com/rook/rook/pkg/operator/test"
 	exectest "github.com/rook/rook/pkg/util/exec/test"
 	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-	"gopkg.in/ini.v1"
 )
 
 func TestPreClusterStartValidation(t *testing.T) {
@@ -158,137 +153,60 @@ func TestPreMonChecks(t *testing.T) {
 
 func TestConfigureMsgr2(t *testing.T) {
 	type fields struct {
-		expectedGlobalConfigSettings map[string]string
-		cephVersion                  cephver.CephVersion
-		Spec                         *cephv1.ClusterSpec
+		encryptionExpected  bool
+		compressionExpected bool
+		cephVersion         cephver.CephVersion
+		Spec                *cephv1.ClusterSpec
 	}
 
 	tests := []struct {
 		name   string
 		fields fields
 	}{
-		{
-			name: "default settings",
-			fields: fields{
-				expectedGlobalConfigSettings: nil,
-				cephVersion:                  cephver.CephVersion{Major: 15},
-				Spec:                         &cephv1.ClusterSpec{},
-			},
-		},
-		{
-			name: "encryption enabled",
-			fields: fields{
-				expectedGlobalConfigSettings: map[string]string{
-					"ms_cluster_mode":         "secure",
-					"ms_service_mode":         "secure",
-					"ms_client_mode":          "secure",
-					"rbd_default_map_options": "ms_mode=secure",
-				},
-				cephVersion: cephver.CephVersion{Major: 16},
-				Spec: &cephv1.ClusterSpec{
-					Network: cephv1.NetworkSpec{
-						Connections: &cephv1.ConnectionsSpec{
-							Encryption: &cephv1.EncryptionSpec{
-								Enabled: true,
-							},
-						},
-					},
-				},
-			},
-		},
-		{
-			name: "compression enabled old version",
-			fields: fields{
-				expectedGlobalConfigSettings: map[string]string{
-					"rbd_default_map_options": "ms_mode=prefer-crc",
-				},
-				cephVersion: cephver.CephVersion{Major: 16},
-				Spec: &cephv1.ClusterSpec{
-					Network: cephv1.NetworkSpec{
-						Connections: &cephv1.ConnectionsSpec{
-							Compression: &cephv1.CompressionSpec{
-								Enabled: true,
-							},
-						},
-					},
-				},
-			},
-		},
-		{
-			name: "compression enabled good version",
-			fields: fields{
-				expectedGlobalConfigSettings: map[string]string{
-					"rbd_default_map_options": "ms_mode=prefer-crc",
-				},
-				cephVersion: cephver.CephVersion{Major: 17},
-				Spec: &cephv1.ClusterSpec{
-					Network: cephv1.NetworkSpec{
-						Connections: &cephv1.ConnectionsSpec{
-							Compression: &cephv1.CompressionSpec{
-								Enabled: true,
-							},
-						},
-					},
-				},
-			},
-		},
+		{"default settings", fields{false, false, cephver.CephVersion{Major: 15}, &cephv1.ClusterSpec{}}},
+		{"encryption enabled", fields{true, false, cephver.CephVersion{Major: 16}, &cephv1.ClusterSpec{Network: cephv1.NetworkSpec{Connections: &cephv1.ConnectionsSpec{Encryption: &cephv1.EncryptionSpec{Enabled: true}}}}}},
+		{"compression enabled old version", fields{false, false, cephver.CephVersion{Major: 16}, &cephv1.ClusterSpec{Network: cephv1.NetworkSpec{Connections: &cephv1.ConnectionsSpec{Compression: &cephv1.CompressionSpec{Enabled: true}}}}}},
+		{"compression enabled good version", fields{false, true, cephver.CephVersion{Major: 17}, &cephv1.ClusterSpec{Network: cephv1.NetworkSpec{Connections: &cephv1.ConnectionsSpec{Compression: &cephv1.CompressionSpec{Enabled: true}}}}}},
 	}
 
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			var configFile *ini.File
+			enabledCompression := false
+			enabledEncryption := false
+			executor := &exectest.MockExecutor{
+				MockExecuteCommandWithTimeout: func(timeout time.Duration, command string, args ...string) (string, error) {
+					if args[0] == "config" && args[1] == "assimilate-conf" && args[2] == "-i" && args[4] == "-o" {
+						if tt.fields.Spec.Network.Connections.Encryption != nil && tt.fields.Spec.Network.Connections.Compression == nil {
+							enabledEncryption = true
+						} else if tt.fields.Spec.Network.Connections.Encryption == nil && tt.fields.Spec.Network.Connections.Compression != nil {
+							enabledCompression = true
+						}
+						return "", nil
+					}
+					if args[0] == "config" && args[1] == "rm" {
+						return "", nil
+					}
 
+					return "", errors.Errorf("unexpected ceph command %q", args)
+				},
+			}
 			clusterInfo := cephclient.AdminTestClusterInfo("rook-ceph")
 			clusterInfo.CephVersion = tt.fields.cephVersion
-
+			context := &clusterd.Context{
+				Clientset: testop.New(t, 3),
+				Executor:  executor,
+			}
 			c := &cluster{
 				ClusterInfo: clusterInfo,
-				Namespace:   "rook-ceph",
 				Spec:        tt.fields.Spec,
-				context: &clusterd.Context{
-					Clientset: testop.New(t, 3),
-					Executor: &exectest.MockExecutor{
-						MockExecuteCommandWithTimeout: func(timeout time.Duration, command string, args ...string) (string, error) {
-							joinedArgs := strings.Join(args, " ")
-							switch {
-							case strings.HasPrefix(joinedArgs, "config assimilate-conf"):
-								fs := flag.NewFlagSet("", flag.ContinueOnError)
-								inputFile := fs.String("i", "", "")
-
-								if err := fs.Parse(args[2:4]); err != nil {
-									return "", fmt.Errorf("parse flags: %w", err)
-								}
-
-								f, err := ini.Load(*inputFile)
-								if err != nil {
-									return "", fmt.Errorf("load ini file: %w", err)
-								}
-								configFile = f
-
-								fallthrough
-							case
-								strings.HasPrefix(joinedArgs, "config rm"),
-								strings.HasPrefix(joinedArgs, "config get global rbd_default_map_options"):
-								return "", nil
-							}
-							return "", errors.Errorf("unexpected ceph command %q", args)
-						},
-					},
-				},
+				context:     context,
+				Namespace:   "rook-ceph",
 			}
 
 			err := c.configureMsgr2()
-			require.NoError(t, err)
-
-			if assert.Equal(t, tt.fields.expectedGlobalConfigSettings == nil, configFile == nil) {
-				return
-			}
-
-			section := configFile.Section("global")
-			for k, v := range tt.fields.expectedGlobalConfigSettings {
-				assert.Equal(t, v, section.Key(k).String(), k)
-			}
-			assert.Len(t, section.Keys(), len(tt.fields.expectedGlobalConfigSettings))
+			assert.NoError(t, err)
+			assert.Equal(t, tt.fields.compressionExpected, enabledCompression)
+			assert.Equal(t, tt.fields.encryptionExpected, enabledEncryption)
 		})
 	}
 }
diff --git a/pkg/operator/ceph/cluster/mgr/dashboard_test.go b/pkg/operator/ceph/cluster/mgr/dashboard_test.go
index 8b7f14a02..965be9a6e 100644
--- a/pkg/operator/ceph/cluster/mgr/dashboard_test.go
+++ b/pkg/operator/ceph/cluster/mgr/dashboard_test.go
@@ -131,13 +131,13 @@ func TestStartSecureDashboard(t *testing.T) {
 	}
 
 	dashboardInitWaitTime = 0
-	err := c.configureDashboardService()
+	err := c.configureDashboardService("a")
 	assert.NoError(t, err)
 	err = c.configureDashboardModules()
 	assert.NoError(t, err)
 	// the dashboard is enabled once with the new dashboard and modules
-	assert.Equal(t, 3, enables)
-	assert.Equal(t, 2, disables)
+	assert.Equal(t, 2, enables)
+	assert.Equal(t, 1, disables)
 	assert.Equal(t, 2, moduleRetries)
 
 	svc, err := c.context.Clientset.CoreV1().Services(clusterInfo.Namespace).Get(ctx, "rook-ceph-mgr-dashboard", metav1.GetOptions{})
@@ -148,12 +148,12 @@ func TestStartSecureDashboard(t *testing.T) {
 
 	// disable the dashboard
 	c.spec.Dashboard.Enabled = false
-	err = c.configureDashboardService()
+	err = c.configureDashboardService("a")
 	assert.Nil(t, err)
 	err = c.configureDashboardModules()
 	assert.NoError(t, err)
-	assert.Equal(t, 3, enables)
-	assert.Equal(t, 3, disables)
+	assert.Equal(t, 2, enables)
+	assert.Equal(t, 2, disables)
 
 	svc, err = c.context.Clientset.CoreV1().Services(clusterInfo.Namespace).Get(ctx, "rook-ceph-mgr-dashboard", metav1.GetOptions{})
 	assert.NotNil(t, err)
@@ -163,7 +163,7 @@ func TestStartSecureDashboard(t *testing.T) {
 	// Set the port to something over 1024 and confirm the port and targetPort are the same
 	c.spec.Dashboard.Enabled = true
 	c.spec.Dashboard.Port = 1025
-	err = c.configureDashboardService()
+	err = c.configureDashboardService("a")
 	assert.Nil(t, err)
 
 	svc, err = c.context.Clientset.CoreV1().Services(clusterInfo.Namespace).Get(ctx, "rook-ceph-mgr-dashboard", metav1.GetOptions{})
@@ -175,7 +175,7 @@ func TestStartSecureDashboard(t *testing.T) {
 	// Fall back to the default port
 	c.spec.Dashboard.Enabled = true
 	c.spec.Dashboard.Port = 0
-	err = c.configureDashboardService()
+	err = c.configureDashboardService("a")
 	assert.Nil(t, err)
 
 	svc, err = c.context.Clientset.CoreV1().Services(clusterInfo.Namespace).Get(ctx, "rook-ceph-mgr-dashboard", metav1.GetOptions{})
diff --git a/pkg/operator/ceph/cluster/mgr/mgr_test.go b/pkg/operator/ceph/cluster/mgr/mgr_test.go
index 51e1e0c63..ad050cf41 100644
--- a/pkg/operator/ceph/cluster/mgr/mgr_test.go
+++ b/pkg/operator/ceph/cluster/mgr/mgr_test.go
@@ -31,7 +31,6 @@ import (
 	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
 	"github.com/rook/rook/pkg/operator/ceph/controller"
 	cephver "github.com/rook/rook/pkg/operator/ceph/version"
-	"github.com/rook/rook/pkg/operator/k8sutil"
 	testopk8s "github.com/rook/rook/pkg/operator/k8sutil/test"
 	testop "github.com/rook/rook/pkg/operator/test"
 	exectest "github.com/rook/rook/pkg/util/exec/test"
@@ -46,7 +45,10 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
 )
 
-func createNewCluster(t *testing.T) *Cluster {
+func TestStartMgr(t *testing.T) {
+	var deploymentsUpdated *[]*apps.Deployment
+	updateDeploymentAndWait, deploymentsUpdated = testopk8s.UpdateDeploymentAndWaitStub()
+
 	executor := &exectest.MockExecutor{
 		MockExecuteCommandWithOutput: func(command string, args ...string) (string, error) {
 			logger.Infof("Execute: %s %v", command, args)
@@ -90,17 +92,8 @@ func createNewCluster(t *testing.T) *Cluster {
 	c := New(ctx, clusterInfo, clusterSpec, "myversion")
 	defer os.RemoveAll(c.spec.DataDirHostPath)
 
-	return c
-}
-
-func TestStartMgr(t *testing.T) {
-	var deploymentsUpdated *[]*apps.Deployment
-	updateDeploymentAndWait, deploymentsUpdated = testopk8s.UpdateDeploymentAndWaitStub()
-
-	c := createNewCluster(t)
-
 	// start a basic service
-	err := c.Start()
+	err = c.Start()
 	assert.NoError(t, err)
 	validateStart(t, c)
 	assert.ElementsMatch(t, []string{}, testopk8s.DeploymentNamesUpdated(deploymentsUpdated))
@@ -183,109 +176,128 @@ func validateServices(t *testing.T, c *Cluster) {
 	}
 }
 
-func TestActiveMgrLabels(t *testing.T) {
-	var deploymentsUpdated *[]*apps.Deployment
-	updateDeploymentAndWait, deploymentsUpdated = testopk8s.UpdateDeploymentAndWaitStub()
-
-	c := createNewCluster(t)
-	err := c.Start()
-	assert.NoError(t, err)
-	validateStart(t, c)
+func TestUpdateServiceSelectors(t *testing.T) {
+	clientset := testop.New(t, 3)
+	ctx := &clusterd.Context{Clientset: clientset}
+	clusterInfo := cephclient.AdminTestClusterInfo("mycluster")
+	spec := cephv1.ClusterSpec{
+		Dashboard: cephv1.DashboardSpec{
+			Enabled: true,
+			Port:    7000,
+		},
+	}
+	c := &Cluster{spec: spec, context: ctx, clusterInfo: clusterInfo}
 
-	mgrNames := []string{"a", "b"}
-	for i := 0; i < c.spec.Mgr.Count; i++ {
-		// Simulate the Mgr pod having been created
-		daemonName := mgrNames[i]
-		mgrPod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{
-			Name: fmt.Sprintf("rook-ceph-mgr-%s", daemonName),
-			Labels: map[string]string{k8sutil.AppAttr: "rook-ceph-mgr",
-				"instance":       daemonName,
-				"ceph_daemon_id": daemonName,
-				"mgr":            daemonName,
-				"rook_cluster":   "ns"}}}
-		_, err = c.context.Clientset.CoreV1().Pods(c.clusterInfo.Namespace).Create(c.clusterInfo.Context, mgrPod, metav1.CreateOptions{})
+	t.Run("initial active daemon", func(t *testing.T) {
+		activeDaemon := "a"
+		err := c.reconcileServices(activeDaemon)
 		assert.NoError(t, err)
-	}
+		validateServiceActiveDaemon(t, c, activeDaemon, 2, 0)
+	})
 
-	for i := 0; i < c.spec.Mgr.Count; i++ {
-		daemonName := mgrNames[i]
-		_, err := c.context.Clientset.AppsV1().Deployments(c.clusterInfo.Namespace).Get(context.TODO(), fmt.Sprintf("rook-ceph-mgr-%s", daemonName), metav1.GetOptions{})
+	t.Run("update active daemon", func(t *testing.T) {
+		activeDaemon := "b"
+		err := c.updateServiceSelectors(activeDaemon)
 		assert.NoError(t, err)
-		_, err = c.UpdateActiveMgrLabel(daemonName, "")
+		validateServiceActiveDaemon(t, c, activeDaemon, 2, 0)
+	})
+
+	t.Run("skip non-mgr services", func(t *testing.T) {
+		svc := corev1.Service{ObjectMeta: metav1.ObjectMeta{Name: "mysvc"}}
+		_, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Create(clusterInfo.Context, &svc, metav1.CreateOptions{})
 		assert.NoError(t, err)
-		pods, err := c.context.Clientset.CoreV1().Pods(c.clusterInfo.Namespace).List(c.clusterInfo.Context, metav1.ListOptions{LabelSelector: "app=rook-ceph-mgr"})
+
+		activeDaemon := "c"
+		err = c.updateServiceSelectors(activeDaemon)
 		assert.NoError(t, err)
-		assert.Contains(t, pods.Items[i].Labels, "mgr_role")
-		if daemonName == "a" {
-			assert.Equal(t, pods.Items[i].Labels["mgr_role"], "active")
-		} else if daemonName == "b" {
-			assert.Equal(t, pods.Items[i].Labels["mgr_role"], "standby")
+		validateServiceActiveDaemon(t, c, activeDaemon, 2, 1)
+	})
+}
+
+func validateServiceActiveDaemon(t *testing.T, c *Cluster, activeDaemon string, expectedUpdated, expectedSkipped int) {
+	services, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).List(c.clusterInfo.Context, metav1.ListOptions{})
+	assert.NoError(t, err)
+	skipped := 0
+	updated := 0
+	for _, service := range services.Items {
+		if service.Labels["app"] == "rook-ceph-mgr" {
+			updated++
+			assert.Equal(t, activeDaemon, service.Spec.Selector[controller.DaemonIDLabel])
+		} else {
+			skipped++
+			assert.Equal(t, "", service.Spec.Selector[controller.DaemonIDLabel])
 		}
 	}
-
-	testopk8s.ClearDeploymentsUpdated(deploymentsUpdated)
+	assert.Equal(t, expectedUpdated, updated)
+	assert.Equal(t, expectedSkipped, skipped)
 }
 
-func TestUpdateServiceSelectors(t *testing.T) {
-	clientset := testop.New(t, 3)
-	ctx := &clusterd.Context{Clientset: clientset}
-	clusterInfo := cephclient.AdminTestClusterInfo("mycluster")
+func TestMgrSidecarReconcile(t *testing.T) {
+	activeMgr := "a"
+	calledMgrStat := false
 	spec := cephv1.ClusterSpec{
+		Mgr: cephv1.MgrSpec{Count: 1},
 		Dashboard: cephv1.DashboardSpec{
 			Enabled: true,
 			Port:    7000,
 		},
 	}
+	executor := &exectest.MockExecutor{
+		MockExecuteCommandWithOutput: func(command string, args ...string) (string, error) {
+			logger.Infof("Command: %s %v", command, args)
+			if args[1] == "stat" {
+				calledMgrStat = true
+			}
+			return fmt.Sprintf(`{"active_name":"%s"}`, activeMgr), nil
+		},
+	}
+	clientset := testop.New(t, 3)
+	configDir := t.TempDir()
+	ctx := &clusterd.Context{
+		Executor:  executor,
+		ConfigDir: configDir,
+		Clientset: clientset,
+	}
+	clusterInfo := cephclient.AdminTestClusterInfo("mycluster")
+	clusterInfo.SetName("test")
 	c := &Cluster{spec: spec, context: ctx, clusterInfo: clusterInfo}
 
-	// Make sure we remove the daemon_id label from the selector
-	// of all services with a label "app=rook-ceph-mgr"
-	t.Run("remove daemon_id from mgr services", func(t *testing.T) {
-		svc1 := corev1.Service{
-			ObjectMeta: metav1.ObjectMeta{
-				Name: "svc1",
-				Labels: map[string]string{
-					"app":                    "rook-ceph-mgr",
-					"svc":                    "rook-ceph-mgr",
-					controller.DaemonIDLabel: "a",
-				},
-			},
-			Spec: corev1.ServiceSpec{
-				Selector: map[string]string{
-					"app":                    "rook-ceph-mgr",
-					controller.DaemonIDLabel: "a",
-				}},
-		}
-		svc2 := corev1.Service{
-			ObjectMeta: metav1.ObjectMeta{Name: "svc2"},
-			Spec: corev1.ServiceSpec{
-				Selector: map[string]string{
-					"app":                    "rook-ceph-mgr",
-					controller.DaemonIDLabel: "a",
-				}},
-		}
-		_, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Create(clusterInfo.Context, &svc1, metav1.CreateOptions{})
-		assert.NoError(t, err)
-		_, err2 := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Create(clusterInfo.Context, &svc2, metav1.CreateOptions{})
-		assert.NoError(t, err2)
+	// Update services according to the active mgr
+	calledMgrStat = false
+	clusterInfo.CephVersion = cephver.CephVersion{Major: 17, Minor: 2, Build: 0}
+	err := c.ReconcileActiveMgrServices(activeMgr)
+	assert.NoError(t, err)
+	assert.True(t, calledMgrStat)
+	validateServices(t, c)
+	validateServiceMatches(t, c, "a")
 
-		// Check the the label has been only removed from svc1
-		c.updateServiceSelectors()
+	// nothing is created or updated when the requested mgr is not the active mgr
+	clusterInfo.CephVersion = cephver.CephVersion{Major: 16, Minor: 2, Build: 5}
+	err = c.ReconcileActiveMgrServices("b")
+	assert.NoError(t, err)
+	assert.True(t, calledMgrStat)
+	_, err = c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Get(context.TODO(), "rook-ceph-mgr", metav1.GetOptions{})
+	assert.True(t, kerrors.IsNotFound(err))
 
-		// Make sure we remove controller.DaemonIDLabel label from all services tagged as 'app: rook-ceph-mgr'
-		// and we add the selector label "mgr_role" set to "active"
-		updatedService1, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Get(clusterInfo.Context, "svc1", metav1.GetOptions{})
-		assert.NoError(t, err)
-		assert.NotContains(t, updatedService1.Spec.Selector, controller.DaemonIDLabel)
-		assert.Contains(t, updatedService1.Spec.Selector, "mgr_role")
-		assert.Equal(t, updatedService1.Spec.Selector["mgr_role"], "active")
+	// nothing is updated when the requested mgr is not the active mgr
+	activeMgr = "b"
+	err = c.ReconcileActiveMgrServices("b")
+	assert.NoError(t, err)
+	validateServices(t, c)
+	validateServiceMatches(t, c, "b")
+}
 
-		// Make sure services not tagged as 'app: rook-ceph-mgr' are not modified
-		updatedService2, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Get(clusterInfo.Context, "svc2", metav1.GetOptions{})
-		assert.NoError(t, err)
-		assert.Contains(t, updatedService2.Spec.Selector, controller.DaemonIDLabel)
-		assert.NotContains(t, updatedService2.Spec.Selector, "mgr_role")
-	})
+func validateServiceMatches(t *testing.T, c *Cluster, expectedActive string) {
+	// The service labels should match the active mgr
+	svc, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Get(context.TODO(), "rook-ceph-mgr", metav1.GetOptions{})
+	assert.NoError(t, err)
+	matchDaemon, ok := svc.Spec.Selector["ceph_daemon_id"]
+	assert.True(t, ok)
+	assert.Equal(t, expectedActive, matchDaemon)
+
+	// clean up the service for the next test
+	err = c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Delete(context.TODO(), "rook-ceph-mgr", metav1.DeleteOptions{})
+	assert.NoError(t, err)
 }
 
 func TestConfigureModules(t *testing.T) {
@@ -446,100 +458,3 @@ func TestCluster_enableBalancerModule(t *testing.T) {
 		assert.NoError(t, err)
 	})
 }
-
-func TestCluster_configurePrometheusModule(t *testing.T) {
-	modulesEnabled := 0
-	modulesDisabled := 0
-	configSettings := map[string]string{}
-	lastModuleConfigured := ""
-	modulesGetOutput := ""
-	executor := &exectest.MockExecutor{
-		MockExecuteCommandWithOutput: func(command string, args ...string) (string, error) {
-			logger.Infof("Command: %s %v", command, args)
-			if command == "ceph" && len(args) > 3 {
-				if args[0] == "mgr" && args[1] == "module" {
-					if args[2] == "enable" {
-						modulesEnabled++
-					}
-					if args[2] == "disable" {
-						modulesDisabled++
-					}
-					lastModuleConfigured = args[3]
-				}
-			}
-			return "", nil //return "{\"key\":\"mysecurekey\"}", nil
-		},
-		MockExecuteCommandWithTimeout: func(timeout time.Duration, command string, args ...string) (string, error) {
-			if args[0] == "config" && args[1] == "set" && args[2] == "mgr" {
-				configSettings[args[3]] = args[4]
-			}
-			if args[0] == "config" && args[1] == "get" && args[2] == "mgr" {
-				return modulesGetOutput, nil
-			}
-			return "", nil
-		},
-	}
-
-	c := &Cluster{
-		context:     &clusterd.Context{Executor: executor, Clientset: testop.New(t, 3)},
-		clusterInfo: cephclient.AdminTestClusterInfo("mycluster"),
-		spec: cephv1.ClusterSpec{
-			Monitoring: cephv1.MonitoringSpec{
-				Enabled: false,
-			},
-		},
-	}
-	// Disable prometheus module
-	err := c.configurePrometheusModule()
-	assert.NoError(t, err)
-	assert.Equal(t, 0, modulesEnabled)
-	assert.Equal(t, 1, modulesDisabled)
-	assert.Equal(t, PrometheusModuleName, lastModuleConfigured)
-
-	// Enable prometheus module, no changed
-	modulesDisabled = 0
-	c.spec.Monitoring.Enabled = true
-	err = c.configurePrometheusModule()
-	assert.NoError(t, err)
-	assert.Equal(t, 1, modulesEnabled)
-	assert.Equal(t, 0, modulesDisabled)
-
-	// Enable prometheus module, port changed
-	modulesEnabled = 0
-	c.spec.Monitoring.Port = 30001
-	err = c.configurePrometheusModule()
-	assert.NoError(t, err)
-	assert.Equal(t, 2, modulesEnabled)
-	assert.Equal(t, 1, modulesDisabled)
-	assert.Equal(t, "30001", configSettings["mgr/prometheus/server_port"])
-	assert.Equal(t, 1, len(configSettings))
-
-	modulesEnabled = 0
-	modulesDisabled = 0
-	c.spec.Monitoring.Interval = &metav1.Duration{
-		Duration: 30 * time.Second,
-	}
-	modulesGetOutput = "30"
-	configSettings = make(map[string]string)
-	err = c.configurePrometheusModule()
-	assert.NoError(t, err)
-	assert.Equal(t, 2, modulesEnabled)
-	assert.Equal(t, 1, modulesDisabled)
-	assert.Equal(t, "30001", configSettings["mgr/prometheus/server_port"])
-	assert.Equal(t, 1, len(configSettings))
-
-	// Enable prometheus module, port and interval changed
-	modulesEnabled = 0
-	modulesDisabled = 0
-	c.spec.Monitoring.Port = 30002
-	c.spec.Monitoring.Interval = &metav1.Duration{
-		Duration: 60 * time.Second,
-	}
-	err = c.configurePrometheusModule()
-	assert.NoError(t, err)
-	assert.Equal(t, 2, modulesEnabled)
-	assert.Equal(t, 1, modulesDisabled)
-	assert.Equal(t, "30002", configSettings["mgr/prometheus/server_port"])
-	assert.Equal(t, "60", configSettings["mgr/prometheus/scrape_interval"])
-
-}
diff --git a/pkg/operator/ceph/cluster/mgr/spec_test.go b/pkg/operator/ceph/cluster/mgr/spec_test.go
index 84e5ab6a7..612e7f639 100644
--- a/pkg/operator/ceph/cluster/mgr/spec_test.go
+++ b/pkg/operator/ceph/cluster/mgr/spec_test.go
@@ -23,6 +23,7 @@ import (
 	"github.com/rook/rook/pkg/clusterd"
 	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
 	"github.com/rook/rook/pkg/operator/ceph/config"
+	"github.com/rook/rook/pkg/operator/ceph/controller"
 	"github.com/rook/rook/pkg/operator/ceph/test"
 	optest "github.com/rook/rook/pkg/operator/test"
 	"github.com/stretchr/testify/assert"
@@ -112,17 +113,14 @@ func TestServiceSpec(t *testing.T) {
 	clusterSpec := cephv1.ClusterSpec{}
 	c := New(&clusterd.Context{Clientset: clientset}, clusterInfo, clusterSpec, "myversion")
 
-	s, err := c.MakeMetricsService("rook-mgr", serviceMetricName)
+	s, err := c.MakeMetricsService("rook-mgr", "foo", serviceMetricName)
 	assert.NoError(t, err)
 	assert.NotNil(t, s)
 	assert.Equal(t, "rook-mgr", s.Name)
 	assert.Equal(t, 1, len(s.Spec.Ports))
-	assert.Equal(t, 2, len(s.Labels))
-	// Check service selector
+	assert.Equal(t, 3, len(s.Labels))
 	assert.Equal(t, 3, len(s.Spec.Selector))
-	assert.Equal(t, s.Spec.Selector["app"], "rook-ceph-mgr")
-	assert.Equal(t, s.Spec.Selector["mgr_role"], "active")
-	assert.Equal(t, s.Spec.Selector["rook_cluster"], "ns")
+	assert.Equal(t, "foo", s.Spec.Selector[controller.DaemonIDLabel])
 }
 
 func TestHostNetwork(t *testing.T) {
diff --git a/pkg/operator/ceph/cluster/mon/node_test.go b/pkg/operator/ceph/cluster/mon/node_test.go
index 67549859d..352aba712 100644
--- a/pkg/operator/ceph/cluster/mon/node_test.go
+++ b/pkg/operator/ceph/cluster/mon/node_test.go
@@ -185,6 +185,13 @@ func TestHostNetwork(t *testing.T) {
 	assert.Equal(t, "", val)
 	assert.Equal(t, "arg not found: --public-bind-addr", message)
 
+	monConfig.Port = 6790
+	pod, err = c.makeMonPod(monConfig, false)
+	assert.NoError(t, err)
+	val, message = extractArgValue(pod.Spec.Containers[0].Args, "--public-addr")
+	assert.Equal(t, "2.4.6.3:6790", val, message)
+	assert.NotNil(t, pod)
+
 	// Host network setting of mons should be maintained even if the cluster spec hostnetwork is different
 	// from the mons to not be using host networking
 	monConfig.UseHostNetwork = false
diff --git a/pkg/operator/ceph/cluster/osd/envs_test.go b/pkg/operator/ceph/cluster/osd/envs_test.go
index f64383796..704cf8eab 100644
--- a/pkg/operator/ceph/cluster/osd/envs_test.go
+++ b/pkg/operator/ceph/cluster/osd/envs_test.go
@@ -86,7 +86,7 @@ func TestGetTcmallocMaxTotalThreadCacheBytes(t *testing.T) {
 	assert.Equal(t, "67108864", v.Value)
 
 	// Read the file now
-	err = os.WriteFile(file.Name(), sysconfig, 0400)
+	err = os.WriteFile(file.Name(), sysconfig, 0444)
 	assert.NoError(t, err)
 	v = getTcmallocMaxTotalThreadCacheBytes("")
 	assert.Equal(t, "134217728", v.Value)
diff --git a/pkg/operator/ceph/cluster/osd/osd_test.go b/pkg/operator/ceph/cluster/osd/osd_test.go
index 2e8bedf16..bcaa024cf 100644
--- a/pkg/operator/ceph/cluster/osd/osd_test.go
+++ b/pkg/operator/ceph/cluster/osd/osd_test.go
@@ -570,18 +570,18 @@ func TestDetectCrushLocation(t *testing.T) {
 
 	// update the location with valid topology labels
 	nodeLabels = map[string]string{
-		"topology.kubernetes.io/region": "region1",
-		"topology.kubernetes.io/zone":   "zone",
-		"topology.rook.io/rack":         "rack1",
-		"topology.rook.io/row":          "row1",
+		"failure-domain.beta.kubernetes.io/region": "region1",
+		"failure-domain.beta.kubernetes.io/zone":   "zone1",
+		"topology.rook.io/rack":                    "rack1",
+		"topology.rook.io/row":                     "row1",
 	}
-	// sorted in alphabetical order
+
 	expected := []string{
 		"host=foo",
 		"rack=rack1",
 		"region=region1",
 		"row=row1",
-		"zone=zone",
+		"zone=zone1",
 	}
 	updateLocationWithNodeLabels(&location, nodeLabels)
 
diff --git a/pkg/operator/ceph/cluster/osd/topology/topology_test.go b/pkg/operator/ceph/cluster/osd/topology/topology_test.go
index 33a5c4cdd..720414f15 100644
--- a/pkg/operator/ceph/cluster/osd/topology/topology_test.go
+++ b/pkg/operator/ceph/cluster/osd/topology/topology_test.go
@@ -42,26 +42,22 @@ func TestOrderedCRUSHLabels(t *testing.T) {
 func TestCleanTopologyLabels(t *testing.T) {
 	// load all the expected labels
 	nodeLabels := map[string]string{
-		corev1.LabelZoneRegionStable:  "r.region",
-		"kubernetes.io/hostname":      "host.name",
-		"topology.rook.io/rack":       "r.rack",
-		"topology.rook.io/row":        "r.row",
-		"topology.rook.io/datacenter": "d.datacenter",
-		"topology.rook.io/room":       "test",
-		"topology.rook.io/chassis":    "test",
-		"topology.rook.io/pod":        "test"}
+		corev1.LabelZoneRegionStable:        "r.region",
+		corev1.LabelZoneFailureDomainStable: "z.zone",
+		"kubernetes.io/hostname":            "host.name",
+		"topology.rook.io/rack":             "r.rack",
+		"topology.rook.io/row":              "r.row",
+		"topology.rook.io/datacenter":       "d.datacenter",
+	}
 	topology, affinity := ExtractOSDTopologyFromLabels(nodeLabels)
 	assert.Equal(t, 6, len(topology))
 	assert.Equal(t, "r-region", topology["region"])
+	assert.Equal(t, "z-zone", topology["zone"])
 	assert.Equal(t, "host-name", topology["host"])
 	assert.Equal(t, "r-rack", topology["rack"])
 	assert.Equal(t, "r-row", topology["row"])
 	assert.Equal(t, "d-datacenter", topology["datacenter"])
-	assert.Equal(t, "topology.rook.io/chassis=test", affinity)
-	assert.Equal(t, "test", topology["chassis"])
-	assert.Equal(t, "", topology["pod"])
-	assert.Equal(t, "", topology["room"])
-
+	assert.Equal(t, "topology.rook.io/rack=r.rack", affinity)
 }
 
 func TestTopologyLabels(t *testing.T) {
@@ -90,21 +86,47 @@ func TestTopologyLabels(t *testing.T) {
 
 	// load all the expected labels
 	nodeLabels = map[string]string{
-		corev1.LabelZoneRegionStable:  "r1",
-		"kubernetes.io/hostname":      "myhost",
-		"topology.rook.io/rack":       "rack1",
-		"topology.rook.io/row":        "row1",
-		"topology.rook.io/datacenter": "d1",
+		corev1.LabelZoneRegionStable:        "r1",
+		corev1.LabelZoneFailureDomainStable: "z1",
+		"kubernetes.io/hostname":            "myhost",
+		"topology.rook.io/rack":             "rack1",
+		"topology.rook.io/row":              "row1",
+		"topology.rook.io/datacenter":       "d1",
 	}
 	topology, affinity = extractTopologyFromLabels(nodeLabels)
-	assert.Equal(t, 5, len(topology))
+	assert.Equal(t, 6, len(topology))
 	assert.Equal(t, "r1", topology["region"])
+	assert.Equal(t, "z1", topology["zone"])
 	assert.Equal(t, "myhost", topology["host"])
 	assert.Equal(t, "rack1", topology["rack"])
 	assert.Equal(t, "row1", topology["row"])
 	assert.Equal(t, "d1", topology["datacenter"])
 	assert.Equal(t, "topology.rook.io/rack=rack1", affinity)
 
+	// ensure deprecated k8s labels are loaded
+	nodeLabels = map[string]string{
+		corev1.LabelZoneRegion:        "r1",
+		corev1.LabelZoneFailureDomain: "z1",
+	}
+	topology, affinity = extractTopologyFromLabels(nodeLabels)
+	assert.Equal(t, 2, len(topology))
+	assert.Equal(t, "r1", topology["region"])
+	assert.Equal(t, "z1", topology["zone"])
+	assert.Equal(t, "failure-domain.beta.kubernetes.io/zone=z1", affinity)
+
+	// ensure deprecated k8s labels are overridden
+	nodeLabels = map[string]string{
+		corev1.LabelZoneRegionStable:        "r1",
+		corev1.LabelZoneFailureDomainStable: "z1",
+		corev1.LabelZoneRegion:              "oldregion",
+		corev1.LabelZoneFailureDomain:       "oldzone",
+	}
+	topology, affinity = extractTopologyFromLabels(nodeLabels)
+	assert.Equal(t, 2, len(topology))
+	assert.Equal(t, "r1", topology["region"])
+	assert.Equal(t, "z1", topology["zone"])
+	assert.Equal(t, "topology.kubernetes.io/zone=z1", affinity)
+
 	// invalid labels under topology.rook.io return an error
 	nodeLabels = map[string]string{
 		"topology.rook.io/row/bad": "r1",
@@ -112,6 +134,16 @@ func TestTopologyLabels(t *testing.T) {
 	topology, affinity = extractTopologyFromLabels(nodeLabels)
 	assert.Equal(t, 0, len(topology))
 	assert.Equal(t, "", affinity)
+
+	// ignore the region k8s topology label when it has the same value as the k8s zone label
+	nodeLabels = map[string]string{
+		corev1.LabelZoneRegionStable:        "zone",
+		corev1.LabelZoneFailureDomainStable: "zone",
+	}
+	topology, affinity = extractTopologyFromLabels(nodeLabels)
+	assert.Equal(t, 1, len(topology))
+	assert.Equal(t, "zone", topology["zone"])
+	assert.Equal(t, "topology.kubernetes.io/zone=zone", affinity)
 }
 
 func TestGetDefaultTopologyLabels(t *testing.T) {
diff --git a/pkg/operator/ceph/csi/cluster_config_test.go b/pkg/operator/ceph/csi/cluster_config_test.go
index 5f778a16d..dc64ed60f 100644
--- a/pkg/operator/ceph/csi/cluster_config_test.go
+++ b/pkg/operator/ceph/csi/cluster_config_test.go
@@ -369,20 +369,15 @@ func TestMonEndpoints(t *testing.T) {
 		verifyEndpointPort(t, endpoints, "3300")
 	})
 
-	t.Run("ipv6 endpoint conversion", func(t *testing.T) {
+	t.Run("bad endpoint", func(t *testing.T) {
 		monInfo := map[string]*cephclient.MonInfo{
-			"a": {Name: "a", Endpoint: "[fd07:aaaa:bbbb:cccc::11]:6789"},
-			"b": {Name: "a", Endpoint: "[1234:6789:bbbb:cccc::11]:6789"},
+			"a": {Name: "a", Endpoint: "1.2.3.4:5:6789"},
+			"b": {Name: "b", Endpoint: "1.2.3.5:6789"},
+			"c": {Name: "c", Endpoint: "1.2.3.6:6789"},
 		}
 		endpoints := MonEndpoints(monInfo, true)
 		assert.Equal(t, 2, len(endpoints))
 		verifyEndpointPort(t, endpoints, "3300")
-		for _, endpoint := range endpoints {
-			// Verify that the v1 port inside the ipv6 address will not be replaced
-			if strings.HasPrefix(endpoint, "[1234") {
-				assert.True(t, strings.HasPrefix(endpoint, "[1234:6789"))
-			}
-		}
 	})
 }
 
diff --git a/pkg/operator/ceph/object/controller_test.go b/pkg/operator/ceph/object/controller_test.go
index e0217adce..d12a64c81 100644
--- a/pkg/operator/ceph/object/controller_test.go
+++ b/pkg/operator/ceph/object/controller_test.go
@@ -355,7 +355,9 @@ func TestCephObjectStoreController(t *testing.T) {
 			objectStore,
 		}
 
-		objects = append(objects, additionalObjects...)
+		for i := range additionalObjects {
+			objects = append(objects, additionalObjects[i])
+		}
 
 		executor := &exectest.MockExecutor{
 			MockExecuteCommandWithOutput: func(command string, args ...string) (string, error) {
diff --git a/pkg/operator/ceph/object/notification/controller_test.go b/pkg/operator/ceph/object/notification/controller_test.go
index 0b720f45e..e31ea8abc 100644
--- a/pkg/operator/ceph/object/notification/controller_test.go
+++ b/pkg/operator/ceph/object/notification/controller_test.go
@@ -57,7 +57,6 @@ var (
 	multipleCreateBucketName = "multi-create"
 	multipleDeleteBucketName = "multi-delete"
 	multipleBothBucketName   = "multi-both"
-	otherClusterBucketName   = "other-cluster"
 	startEvent               = string(cephv1.ReconcileStarted)
 	finishedEvent            = string(cephv1.ReconcileSucceeded)
 	failedEvent              = string(cephv1.ReconcileFailed)
diff --git a/pkg/operator/ceph/object/notification/obc_label_controller_test.go b/pkg/operator/ceph/object/notification/obc_label_controller_test.go
index aa421a933..6a2425258 100644
--- a/pkg/operator/ceph/object/notification/obc_label_controller_test.go
+++ b/pkg/operator/ceph/object/notification/obc_label_controller_test.go
@@ -30,16 +30,11 @@ import (
 	"k8s.io/apimachinery/pkg/runtime"
 )
 
-const (
-	testProvisionerOtherCluster = "other-provisioner.io-bucket"
-)
-
-func createOBResources(name, provisioner string) (*bktv1alpha1.ObjectBucketClaim, *bktv1alpha1.ObjectBucket) {
+func createOBResources(name string) (*bktv1alpha1.ObjectBucketClaim, *bktv1alpha1.ObjectBucket) {
 	return &bktv1alpha1.ObjectBucketClaim{
 			ObjectMeta: metav1.ObjectMeta{
 				Name:      name,
 				Namespace: testNamespace,
-				Labels:    map[string]string{bucketProvisionerLabelKey: provisioner},
 			},
 			TypeMeta: metav1.TypeMeta{
 				Kind: "ObjectBucketClaim",
@@ -55,7 +50,6 @@ func createOBResources(name, provisioner string) (*bktv1alpha1.ObjectBucketClaim
 			ObjectMeta: metav1.ObjectMeta{
 				Name:      name,
 				Namespace: testNamespace,
-				Labels:    map[string]string{bucketProvisionerLabelKey: provisioner},
 			},
 			TypeMeta: metav1.TypeMeta{
 				Kind: "ObjectBucket",
@@ -136,7 +130,7 @@ func TestCephBucketNotificationOBCLabelController(t *testing.T) {
 		},
 	}
 
-	obc, ob := createOBResources(testBucketName, bucketProvisionerLabelVal)
+	obc, ob := createOBResources(testBucketName)
 	obc.Labels = setNotificationLabels([]string{testNotificationName})
 
 	t.Run("provision OBC with notification label with no ob", func(t *testing.T) {
@@ -232,7 +226,7 @@ func TestCephBucketNotificationOBCLabelController(t *testing.T) {
 
 	t.Run("reconcile with already existing label for the obc", func(t *testing.T) {
 		resetValues()
-		noChangeOBC, noChangeOB := createOBResources(noChangeBucketName, bucketProvisionerLabelVal)
+		noChangeOBC, noChangeOB := createOBResources(noChangeBucketName)
 		noChangeOBC.Labels = setNotificationLabels([]string{testNotificationName})
 		noChangeOBC.Spec.ObjectBucketName = noChangeBucketName
 		noChangeOBC.Status.Phase = bktv1alpha1.ObjectBucketClaimStatusPhaseBound
@@ -255,7 +249,7 @@ func TestCephBucketNotificationOBCLabelController(t *testing.T) {
 
 	t.Run("delete notification from the obc", func(t *testing.T) {
 		resetValues()
-		deleteOBC, deleteOB := createOBResources(deleteBucketName, bucketProvisionerLabelVal)
+		deleteOBC, deleteOB := createOBResources(deleteBucketName)
 		deleteOBC.Spec.GenerateBucketName = deleteBucketName
 		deleteOBC.Spec.ObjectBucketName = deleteBucketName
 		deleteOBC.Status.Phase = bktv1alpha1.ObjectBucketClaimStatusPhaseBound
@@ -277,7 +271,7 @@ func TestCephBucketNotificationOBCLabelController(t *testing.T) {
 
 	t.Run("provision OBC with multiple notification labels", func(t *testing.T) {
 		resetValues()
-		multipleCreateOBC, multipleCreateOB := createOBResources(multipleCreateBucketName, bucketProvisionerLabelVal)
+		multipleCreateOBC, multipleCreateOB := createOBResources(multipleCreateBucketName)
 		multipleCreateOBC.Labels = setNotificationLabels([]string{testNotificationName + "-1", testNotificationName + "-2"})
 		multipleCreateOBC.Spec.ObjectBucketName = multipleCreateBucketName
 		multipleCreateOBC.Status.Phase = bktv1alpha1.ObjectBucketClaimStatusPhaseBound
@@ -304,7 +298,7 @@ func TestCephBucketNotificationOBCLabelController(t *testing.T) {
 
 	t.Run("delete multiple notifications from the obc", func(t *testing.T) {
 		resetValues()
-		multipleDeleteOBC, multipleDeleteOB := createOBResources(multipleDeleteBucketName, bucketProvisionerLabelVal)
+		multipleDeleteOBC, multipleDeleteOB := createOBResources(multipleDeleteBucketName)
 		multipleDeleteOBC.Spec.GenerateBucketName = multipleDeleteBucketName
 		multipleDeleteOBC.Spec.ObjectBucketName = multipleDeleteBucketName
 		multipleDeleteOBC.Status.Phase = bktv1alpha1.ObjectBucketClaimStatusPhaseBound
@@ -325,7 +319,7 @@ func TestCephBucketNotificationOBCLabelController(t *testing.T) {
 	})
 	t.Run("provision OBC with multiple delete and create of notifications", func(t *testing.T) {
 		resetValues()
-		multipleBothOBC, multipleBothOB := createOBResources(multipleBothBucketName, bucketProvisionerLabelVal)
+		multipleBothOBC, multipleBothOB := createOBResources(multipleBothBucketName)
 		multipleBothOBC.Labels = setNotificationLabels([]string{testNotificationName + "-1", testNotificationName + "-2"})
 		multipleBothOBC.Spec.ObjectBucketName = multipleBothBucketName
 		multipleBothOBC.Status.Phase = bktv1alpha1.ObjectBucketClaimStatusPhaseBound
@@ -352,25 +346,4 @@ func TestCephBucketNotificationOBCLabelController(t *testing.T) {
 			[]string{multipleDeleteBucketName + testNotificationName + "-1", multipleDeleteBucketName + testNotificationName + "-2"})
 		verifyEvents(t, []string{startEvent, finishedEvent})
 	})
-
-	t.Run("OBC does not belong to the ceph cluster", func(t *testing.T) {
-		resetValues()
-		otherClusterOBC, otherClusterOB := createOBResources(otherClusterBucketName, testProvisionerOtherCluster)
-		otherClusterOBC.Labels = setNotificationLabels([]string{testNotificationName})
-		otherClusterOBC.Spec.ObjectBucketName = otherClusterBucketName
-		otherClusterOBC.Status.Phase = bktv1alpha1.ObjectBucketClaimStatusPhaseBound
-		objects := []runtime.Object{
-			cephCluster,
-			otherClusterOBC,
-			otherClusterOB,
-		}
-
-		res, err := testOBCLabelReconciler(objects, otherClusterBucketName)
-		assert.NoError(t, err)
-		assert.False(t, res.Requeue)
-		assert.False(t, getWasInvoked)
-		assert.Equal(t, 0, len(createdNotifications))
-		assert.Equal(t, 0, len(deletedNotifications))
-		verifyEvents(t, []string{startEvent, finishedEvent})
-	})
 }
diff --git a/pkg/operator/test/client.go b/pkg/operator/test/client.go
index 115144d4d..07d3e950e 100644
--- a/pkg/operator/test/client.go
+++ b/pkg/operator/test/client.go
@@ -141,9 +141,6 @@ func NewComplexClientset(t *testing.T) *fake.Clientset {
 		}
 		obj := createAction.GetObject()
 		objMeta, err := meta.Accessor(obj)
-		if err != nil {
-			panic(fmt.Errorf("failed to objMeta"))
-		}
 		resource := action.GetResource().Resource
 		name := objMeta.GetName()
 
diff --git a/pkg/util/sys/device_test.go b/pkg/util/sys/device_test.go
index 01023c0ff..10342f857 100644
--- a/pkg/util/sys/device_test.go
+++ b/pkg/util/sys/device_test.go
@@ -196,13 +196,3 @@ func TestListDevicesChildListDevicesChild(t *testing.T) {
 	assert.NoError(t, err)
 	assert.Equal(t, 3, len(child))
 }
-
-func TestGetDiskDeviceClass(t *testing.T) {
-	d := &LocalDisk{}
-	assert.Equal(t, "ssd", GetDiskDeviceClass(d))
-	d.Rotational = true
-	assert.Equal(t, "hdd", GetDiskDeviceClass(d))
-	d.Rotational = false
-	d.RealPath = "nvme"
-	assert.Equal(t, "nvme", GetDiskDeviceClass(d))
-}
diff --git a/tests/scripts/build-release.sh b/tests/scripts/build-release.sh
index cabbfcf80..2672a91ed 100755
--- a/tests/scripts/build-release.sh
+++ b/tests/scripts/build-release.sh
@@ -31,21 +31,6 @@ function promote() {
 # MAIN      #
 #############
 
-# Load dot env file if available
-if [ -f .env ]; then
-    # shellcheck disable=SC2046
-    export $(grep -v '^#' .env | xargs -d '\n')
-fi
-
-# Use Git access token for accessing the docs repo if set
-# shellcheck disable=SC2034
-export DOCS_GIT_REPO="${DOCS_GIT_REPO:-git@github.com:rook/rook.github.io.git}"
-if [ -n "${GIT_API_TOKEN}" ]; then
-    DOCS_GIT_REPO="${DOCS_GIT_REPO//git@/}"
-    DOCS_GIT_REPO="${DOCS_GIT_REPO//:/\/}"
-    export DOCS_GIT_REPO="https://${GIT_API_TOKEN}@${DOCS_GIT_REPO}"
-fi
-
 SHOULD_PROMOTE=true
 if [[ ${GITHUB_REF} =~ master ]]; then
     echo "Publishing from master"
diff --git a/tests/scripts/github-action-helper.sh b/tests/scripts/github-action-helper.sh
index d40206d19..5f428381b 100755
--- a/tests/scripts/github-action-helper.sh
+++ b/tests/scripts/github-action-helper.sh
@@ -252,12 +252,6 @@ function deploy_cluster() {
     echo "invalid argument: $*" >&2
     exit 1
   fi
-  # enable monitoring
-  yq w -i -d1 cluster-test.yaml spec.monitoring.enabled true
-  kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/v0.40.0/bundle.yaml
-  kubectl create -f monitoring/rbac.yaml
-
-  # create the cluster resources
   kubectl create -f cluster-test.yaml
   kubectl create -f object-test.yaml
   kubectl create -f pool-test.yaml
diff --git a/tests/scripts/multus-test.sh b/tests/scripts/multus-test.sh
deleted file mode 100755
index 6405a7fd4..000000000
--- a/tests/scripts/multus-test.sh
+++ /dev/null
@@ -1,139 +0,0 @@
-#!/usr/bin/env bash
-set -eEuo pipefail
-
-##
-## This script will install a configurable number of multus-networked daemonsets to test that your
-## multus configuration and network hardware can provide enough IP addresses for Ceph.
-## Configuration options are commented below.
-##
-## TODO: in the future, this test should also check network stability under a reasonable amount of
-##       load to/from daemons.
-##
-
-## A comma-delimited list of multus networks to use for test pods
-: "${MULTUS_NETWORKS:="public-net,cluster-net"}"
-
-## The number of daemons to run per node as a test. This should reflect the maximum number of
-## daemons that could run on a single node for your Ceph install. The default value assumes that one
-## daemon type will run on one node, but that there will be 3 OSDs on said node. As an easy
-## guideline, add one for each additional OSD beyond 3 that might run simultaneously on any node.
-: "${NUM_DAEMONS_PER_NODE:=20}"
-
-## The namespace to install multus test daemons into.
-: "${VALIDATION_NAMESPACE:="rook-ceph"}"
-
-## If you use a command other than 'kubectl' for getting/creating cluster resources, change this.
-: "${KUBECTL:="kubectl"}"
-
-## If you use a command other than 'jq' for json querying, change this.
-: "${JQ:="jq"}"
-
-ALL_DAEMONSETS=() # global for cleanup function
-function main() {
-  trap cleanup EXIT
-
-  ALL_DAEMONSETS=()
-  for i in $(seq "$NUM_DAEMONS_PER_NODE"); do
-    daemonset_name="multus-validator-${i}"
-    ALL_DAEMONSETS+=("${daemonset_name}")
-
-    render_daemonset "${daemonset_name}" | $KUBECTL create -f -
-  done
-
-  echo "Waiting for ${#ALL_DAEMONSETS[@]} daemonsets to have pods scheduled"
-  num_expected_pods=0
-  for ds in "${ALL_DAEMONSETS[@]}"; do
-    echo " └─ waiting for daemonset '${ds}' to have pods scheduled"
-    while true; do
-      num_desired="$($KUBECTL --namespace $VALIDATION_NAMESPACE get daemonset "${ds}" \
-        --output jsonpath='{.status.desiredNumberScheduled}' || true)"
-      if [[ -n "$num_desired" ]] && [[ "$num_desired" -gt 0 ]]; then
-        num_expected_pods=$((num_expected_pods + num_desired))
-        break
-      fi
-      sleep 1
-    done
-  done
-
-  echo "Waiting for ${num_expected_pods} pods from ${#ALL_DAEMONSETS[@]} daemonsets"
-  num_ready_with_multus=0
-  until [[ $num_ready_with_multus -eq $num_expected_pods ]]; do
-    sleep 2
-    pods="$(get_validator_pods_status_and_multus)"
-    num_ready_with_multus="$(echo "${pods}" | grep 'Running' | grep 'multus=true' --count)"
-    echo " └─ ${num_ready_with_multus} of ${num_expected_pods} pods have multus networks"
-  done
-
-  echo "Test successful!"
-}
-
-function cleanup() {
-  echo "Cleaning up daemonsets"
-  set +eE # don't fail out out on errors on cleanup
-  for ds in "${ALL_DAEMONSETS[@]}"; do
-    $KUBECTL --namespace "$VALIDATION_NAMESPACE" delete daemonset "${ds}" &
-  done
-  until [[ "$(number_of_validation_pods)" -eq 0 ]]; do
-    sleep 2
-    echo -n '.'
-  done
-}
-
-function number_of_validation_pods() {
-  $KUBECTL --namespace "$VALIDATION_NAMESPACE" get pod \
-    --selector app=multus-validation --no-headers | wc -l
-}
-
-# Output:
-#    <name> <phase> multus=true  - if multus k8s.v1.cni.cncf.io/network-status annotation is present
-#    <name> <phase> multus=false - if multus k8s.v1.cni.cncf.io/network-status annotation is NOT present
-#
-# Annotation presence is important because if no multus NADs are specified, pod can still be Ready.
-function get_validator_pods_status_and_multus() {
-  $KUBECTL --namespace "$VALIDATION_NAMESPACE" get pods --selector app=multus-validation --output json |
-    jq -r '.items[]
-      | [
-          .metadata.name,
-          .status.phase,
-          (if .metadata.annotations["k8s.v1.cni.cncf.io/network-status"] then "multus=true" else "multus=false" end)
-        ]
-      | @tsv'
-  # @tsv makes the output tab-delimited
-}
-
-function render_daemonset() {
-  local daemonset_name="$1"
-
-  cat <<EOF
-apiVersion: apps/v1
-kind: DaemonSet
-metadata:
-  name: "${daemonset_name}"
-  namespace: "$VALIDATION_NAMESPACE"
-  labels:
-    app: multus-validation
-spec:
-  selector:
-    matchLabels:
-      name: "${daemonset_name}"
-      app: multus-validation
-  template:
-    metadata:
-      labels:
-        name: "${daemonset_name}"
-        app: multus-validation
-      annotations:
-        k8s.v1.cni.cncf.io/networks: $MULTUS_NETWORKS
-    spec:
-      # TODO: node selectors, tolerations, etc.
-      containers:
-        - name: multus-validator
-          image: quay.io/fedora/fedora:latest
-          command:
-            - sleep
-            - infinity
-          resources: {}
-EOF
-}
-
-main # run main
diff --git a/tests/scripts/validate_cluster.sh b/tests/scripts/validate_cluster.sh
index fa2b3a749..702c1d689 100755
--- a/tests/scripts/validate_cluster.sh
+++ b/tests/scripts/validate_cluster.sh
@@ -62,12 +62,8 @@ function test_demo_osd {
 }
 
 function test_demo_rgw {
-    timeout 360 bash -x <<-'EOF'
-    until [[ "$(kubectl -n rook-ceph get pods -l app=rook-ceph-rgw -o 'jsonpath={..status.conditions[?(@.type=="Ready")].status}')" == "True" ]]; do
-      echo "waiting for rgw pods to be ready"
-      sleep 5
-    done
-EOF
+  # shellcheck disable=SC2046
+  return $(wait_for_daemon "$EXEC_COMMAND -s | grep -sq 'rgw:'")
 }
 
 function test_demo_mds {