Repository: rook/rook
Pull request #11947 (2023-03-21 14:18:42)

external: make rgw call separate from cephfs and rbd
if the rgw endpoint is not reachable don't block the creation of rbd and cephfs sc, just print the error instead of raising an exception

<!-- Please take a look at our [Contributing](https://rook.io/docs/rook/latest/Contributing/development-flow/)
documentation before submitting a Pull Request!
Thank you for contributing to Rook! -->

**Description of your changes:**

**Which issue is resolved by this Pull Request:**
Resolves #

**Checklist:**

- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/latest/Contributing/development-flow/#commit-structure)).
- [ ] **Skip Tests for Docs**: If this is only a documentation change, add the label `skip-ci` on the PR.
- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/latest/Contributing/development-flow/#submitting-a-pull-request)
- [ ] [Pending release notes](https://github.com/rook/rook/blob/master/PendingReleaseNotes.md) updated with breaking and/or notable changes for the next minor release.
- [ ] Documentation has been updated, if necessary.
- [ ] Unit tests have been added, if necessary.
- [ ] Integration tests have been added, if necessary.


parth-gr: Currently, the script is tightly coupled with cephfs and rbd, and rgw
if  `propagate the errors` to the main function makes it more weird and complicated.

Talking about the original problem it needs to have some fix in the ocs-operator too.

Some concerns are:
1) For making it work we should make the main call to 2 separate functions one for rgw second cephfs and rbd(which require a lot of refactoring).
2) If we print the errors with the output we get output that is not directly molded to a json file.
travisn: > Some concerns are:
> 
> 1. For making it work we should make the main call to 2 separate functions one for rgw second cephfs and rbd(which require a lot of refactoring).

Can you expand on why a refactor would be needed? As long as we can surface all the errors, maybe we don't need a refactor?

> 2. If we print the errors with the output we get output that is not directly molded to a json file.

Usually when the output is collected for a json, it would be collected from stdout. As long as the errors are written to stderr, it wouldn't be included in the json. 
parth-gr: This is how the output is looking if we print errors,

```
[ceph: root@rhcs-2-node-1 /]# python3 a.py --rbd-data-pool-name rbd --rgw-endpoint 10.1.115.118:8080
failed to execute command ['radosgw-admin', 'user', 'create', '--uid', 'rgw-admin-ops-user', '--display-name', 'Rook RGW Admin Ops user', '--caps', 'buckets=*;users=*;usage=read;metadata=read;zone=read', '--rgw-realm', '']. Output: b''. Code: 5. Error: b"2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 rgw_init_ioctx ERROR: librados::Rados::pool_create returned (34) Numerical result out of range (this can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd exceeded)\n2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 ERROR: failed reading data (obj=zone.rgw.log:bucket.sync-source-hints.), r=-34\n2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 ERROR: failed to update sources index for bucket=:[]) r=-34\n2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 ERROR: failed to initialize bucket sync policy handler: get_bucket_sync_hints() on bucket=-- returned r=-34\n2023-03-28T14:51:07.223+0000 7f7ddedd1500 -1 ERROR: could not initialize zone policy handler for zone=zone\n2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 ERROR: failed to start notify service ((34) Numerical result out of range\n2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 ERROR: failed to init services (ret=(34) Numerical result out of range)\ncouldn't init storage provider\n"
[{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "rhcs-2-node-1=10.1.115.118:6789", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "15b49e52-5bf5-11ed-a2e7-0050568fd833", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "client.healthchecker", "userKey": "AQD+SpFj/XXhGBAA3Ne52aWEFmDp33ngDyDNIA=="}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "10.1.115.118", "MonitoringPort": "9283"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "AQDlT+JjhGNxIxAAriypBwv7UCMauS4S99+5lA=="}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "AQDlT+JjOVrWIxAAyVIl5R3AHoXtpg0v87RuRg=="}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "AQDlT+JjkGQVJRAAHRQ/5Pim9D4j4IbUIUg5tQ=="}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "AQDlT+JjA5lKJBAAccZ1BF2+OVW/yse598pFBg=="}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "https://10.1.115.118:8443/"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "rbd", "csi.storage.k8s.io/provisioner-secret-name": "rook-csi-rbd-provisioner", "csi.storage.k8s.io/controller-expand-secret-name": "rook-csi-rbd-provisioner", "csi.storage.k8s.io/node-stage-secret-name": "rook-csi-rbd-node"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs.cephfs.data", "csi.storage.k8s.io/provisioner-secret-name": "rook-csi-cephfs-provisioner", "csi.storage.k8s.io/controller-expand-secret-name": "rook-csi-cephfs-provisioner", "csi.storage.k8s.io/node-stage-secret-name": "rook-csi-cephfs-node"}}]

```
travisn: That output looks acceptable as long as the error is going to stderr. This allows them to pipe the results to different files. For example, this should pipe the json to its own file, and the error to a separate file.
```
$ python3 a.py --rbd-data-pool-name rbd --rgw-endpoint 10.1.115.118:8080 > output.json 2> error.txt
```
parth-gr: > This is how the output is looking if we print errors,
> 
> ```
> [ceph: root@rhcs-2-node-1 /]# python3 a.py --rbd-data-pool-name rbd --rgw-endpoint 10.1.115.118:8080
> failed to execute command ['radosgw-admin', 'user', 'create', '--uid', 'rgw-admin-ops-user', '--display-name', 'Rook RGW Admin Ops user', '--caps', 'buckets=*;users=*;usage=read;metadata=read;zone=read', '--rgw-realm', '']. Output: b''. Code: 5. Error: b"2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 rgw_init_ioctx ERROR: librados::Rados::pool_create returned (34) Numerical result out of range (this can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd exceeded)\n2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 ERROR: failed reading data (obj=zone.rgw.log:bucket.sync-source-hints.), r=-34\n2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 ERROR: failed to update sources index for bucket=:[]) r=-34\n2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 ERROR: failed to initialize bucket sync policy handler: get_bucket_sync_hints() on bucket=-- returned r=-34\n2023-03-28T14:51:07.223+0000 7f7ddedd1500 -1 ERROR: could not initialize zone policy handler for zone=zone\n2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 ERROR: failed to start notify service ((34) Numerical result out of range\n2023-03-28T14:51:07.223+0000 7f7ddedd1500  0 ERROR: failed to init services (ret=(34) Numerical result out of range)\ncouldn't init storage provider\n"
> [{"name": "rook-ceph-mon-endpoints", "kind": "ConfigMap", "data": {"data": "rhcs-2-node-1=10.1.115.118:6789", "maxMonId": "0", "mapping": "{}"}}, {"name": "rook-ceph-mon", "kind": "Secret", "data": {"admin-secret": "admin-secret", "fsid": "15b49e52-5bf5-11ed-a2e7-0050568fd833", "mon-secret": "mon-secret"}}, {"name": "rook-ceph-operator-creds", "kind": "Secret", "data": {"userID": "client.healthchecker", "userKey": "AQD+SpFj/XXhGBAA3Ne52aWEFmDp33ngDyDNIA=="}}, {"name": "monitoring-endpoint", "kind": "CephCluster", "data": {"MonitoringEndpoint": "10.1.115.118", "MonitoringPort": "9283"}}, {"name": "rook-csi-rbd-node", "kind": "Secret", "data": {"userID": "csi-rbd-node", "userKey": "AQDlT+JjhGNxIxAAriypBwv7UCMauS4S99+5lA=="}}, {"name": "rook-csi-rbd-provisioner", "kind": "Secret", "data": {"userID": "csi-rbd-provisioner", "userKey": "AQDlT+JjOVrWIxAAyVIl5R3AHoXtpg0v87RuRg=="}}, {"name": "rook-csi-cephfs-provisioner", "kind": "Secret", "data": {"adminID": "csi-cephfs-provisioner", "adminKey": "AQDlT+JjkGQVJRAAHRQ/5Pim9D4j4IbUIUg5tQ=="}}, {"name": "rook-csi-cephfs-node", "kind": "Secret", "data": {"adminID": "csi-cephfs-node", "adminKey": "AQDlT+JjA5lKJBAAccZ1BF2+OVW/yse598pFBg=="}}, {"name": "rook-ceph-dashboard-link", "kind": "Secret", "data": {"userID": "ceph-dashboard-link", "userKey": "https://10.1.115.118:8443/"}}, {"name": "ceph-rbd", "kind": "StorageClass", "data": {"pool": "rbd", "csi.storage.k8s.io/provisioner-secret-name": "rook-csi-rbd-provisioner", "csi.storage.k8s.io/controller-expand-secret-name": "rook-csi-rbd-provisioner", "csi.storage.k8s.io/node-stage-secret-name": "rook-csi-rbd-node"}}, {"name": "cephfs", "kind": "StorageClass", "data": {"fsName": "cephfs", "pool": "cephfs.cephfs.data", "csi.storage.k8s.io/provisioner-secret-name": "rook-csi-cephfs-provisioner", "csi.storage.k8s.io/controller-expand-secret-name": "rook-csi-cephfs-provisioner", "csi.storage.k8s.io/node-stage-secret-name": "rook-csi-cephfs-node"}}]
> ```



> > output.json 2> error.txt

Yes you are right,

But it is not stder, its just printing of the error.
Which will be a common output
parth-gr: I found a solution python has 
`sys.stderr.write` so instead of `print` I will use this,

And it will separate the output

Patch:
diff --git a/Documentation/CRDs/Cluster/ceph-cluster-crd.md b/Documentation/CRDs/Cluster/ceph-cluster-crd.md
index ce20cef9b..acebd3f8d 100755
--- a/Documentation/CRDs/Cluster/ceph-cluster-crd.md
+++ b/Documentation/CRDs/Cluster/ceph-cluster-crd.md
@@ -57,10 +57,9 @@ If this value is empty, each pod will get an ephemeral directory to store their
 For more details on the mons and when to choose a number other than `3`, see the [mon health doc](../../Storage-Configuration/Advanced/ceph-mon-health.md).
 * `mgr`: manager top level section
     * `count`: set number of ceph managers between `1` to `2`. The default value is 2.
-    If there are two managers, it is important for all mgr services point to the active mgr and not the standby mgr. Rook automatically
-	updates the label `mgr_role` on the mgr pods to be either `active` or `standby`. Therefore, services need just to add the label
-    `mgr_role=active` to their selector to point to the active mgr. This applies to all services that rely on the ceph mgr such as
-	the dashboard or the prometheus metrics collector.
+    If there are two managers, it is important for all mgr services point to the active mgr and not the passive mgr. Therefore, Rook will
+    automatically update all services (in the cluster namespace) that have a label `app=rook-ceph-mgr` with a selector pointing to the
+    active mgr. This commonly applies to services for the dashboard or the prometheus metrics collector.
     * `modules`: is the list of Ceph manager modules to enable
 * `crashCollector`: The settings for crash collector daemon(s).
     * `disable`: is set to `true`, the crash collector will not run on any node where a Ceph daemon runs
diff --git a/Documentation/Helm-Charts/operator-chart.md b/Documentation/Helm-Charts/operator-chart.md
index 7a2e6e2d2..f72511e2f 100644
--- a/Documentation/Helm-Charts/operator-chart.md
+++ b/Documentation/Helm-Charts/operator-chart.md
@@ -54,7 +54,6 @@ The following table lists the configurable parameters of the rook-operator chart
 | `crds.enabled` | Whether the helm chart should create and update the CRDs. If false, the CRDs must be managed independently with deploy/examples/crds.yaml. **WARNING** Only set during first deployment. If later disabled the cluster may be DESTROYED. If the CRDs are deleted in this case, see [the disaster recovery guide](https://rook.io/docs/rook/latest/Troubleshooting/disaster-recovery/#restoring-crds-after-deletion) to restore them. | `true` |
 | `csi.allowUnsupportedVersion` | Allow starting an unsupported ceph-csi image | `false` |
 | `csi.attacher.image` | Kubernetes CSI Attacher image | `registry.k8s.io/sig-storage/csi-attacher:v4.1.0` |
-| `csi.cephFSAttachRequired` | Whether to skip any attach operation altogether for CephFS PVCs. See more details [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object). If cephFSAttachRequired is set to false it skips the volume attachments and makes the creation of pods using the CephFS PVC fast. **WARNING** It's highly discouraged to use this for CephFS RWO volumes. Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details. | `true` |
 | `csi.cephFSFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted. supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html | `"File"` |
 | `csi.cephFSKernelMountOptions` | Set CephFS Kernel mount options to use https://docs.ceph.com/en/latest/man/8/mount.ceph/#options. Set to "ms_mode=secure" when connections.encrypted is enabled in CephCluster CR | `nil` |
 | `csi.cephFSPluginUpdateStrategy` | CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate | `RollingUpdate` |
@@ -94,7 +93,6 @@ The following table lists the configurable parameters of the rook-operator chart
 | `csi.kubeletDirPath` | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag) | `/var/lib/kubelet` |
 | `csi.logLevel` | Set logging level for cephCSI containers maintained by the cephCSI. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. | `0` |
 | `csi.nfs.enabled` | Enable the nfs csi driver | `false` |
-| `csi.nfsAttachRequired` | Whether to skip any attach operation altogether for NFS PVCs. See more details [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object). If cephFSAttachRequired is set to false it skips the volume attachments and makes the creation of pods using the NFS PVC fast. **WARNING** It's highly discouraged to use this for NFS RWO volumes. Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details. | `true` |
 | `csi.nfsFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the NFS PVC is being mounted. supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html | `"File"` |
 | `csi.nfsPluginUpdateStrategy` | CSI NFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate | `RollingUpdate` |
 | `csi.nfsPodLabels` | Labels to add to the CSI NFS Deployments and DaemonSets Pods | `nil` |
@@ -106,7 +104,6 @@ The following table lists the configurable parameters of the rook-operator chart
 | `csi.provisionerPriorityClassName` | PriorityClassName to be set on csi driver provisioner pods | `"system-cluster-critical"` |
 | `csi.provisionerReplicas` | Set replicas for csi provisioner deployment | `2` |
 | `csi.provisionerTolerations` | Array of tolerations in YAML format which will be added to CSI provisioner deployment | `nil` |
-| `csi.rbdAttachRequired` | Whether to skip any attach operation altogether for RBD PVCs. See more details [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object). If set to false it skips the volume attachments and makes the creation of pods using the RBD PVC fast. **WARNING** It's highly discouraged to use this for RWO volumes as it can cause data corruption. csi-addons operations like Reclaimspace and PVC Keyrotation will also not be supported if set to false since we'll have no VolumeAttachments to determine which node the PVC is mounted on. Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details. | `true` |
 | `csi.rbdFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted. supported values are documented at https://kubernetes-csi.github.io/docs/support-fsgroup.html | `"File"` |
 | `csi.rbdGrpcMetricsPort` | Ceph CSI RBD driver GRPC metrics port | `9090` |
 | `csi.rbdLivenessMetricsPort` | Ceph CSI RBD driver metrics port | `8080` |
diff --git a/cmd/rook/ceph/mgr.go b/cmd/rook/ceph/mgr.go
index 334c2ccd9..99e774af2 100644
--- a/cmd/rook/ceph/mgr.go
+++ b/cmd/rook/ceph/mgr.go
@@ -96,13 +96,12 @@ func runMgrSidecar(cmd *cobra.Command, args []string) error {
 	clusterInfo.CephVersion = *version
 
 	m := mgr.New(context, &clusterInfo, clusterSpec, "")
-	prevActiveMgr := "unknown"
 	for {
-		prevActiveMgr, err = m.UpdateActiveMgrLabel(daemonName, prevActiveMgr)
+		err := m.ReconcileActiveMgrServices(daemonName)
 		if err != nil {
 			logger.Errorf("failed to reconcile services. %v", err)
 		} else {
-			logger.Infof("successfully checked mgr_role label. checking again in %ds", (int)(interval.Seconds()))
+			logger.Infof("successfully reconciled services. checking again in %ds", (int)(interval.Seconds()))
 		}
 		time.Sleep(interval)
 	}
diff --git a/deploy/charts/library/templates/_cluster-role.tpl b/deploy/charts/library/templates/_cluster-role.tpl
index 9006e2335..13de8de58 100644
--- a/deploy/charts/library/templates/_cluster-role.tpl
+++ b/deploy/charts/library/templates/_cluster-role.tpl
@@ -99,12 +99,9 @@ rules:
     resources:
       - deployments/scale
       - deployments
-      - pods
     verbs:
       - patch
       - delete
-      - get
-      - update
   - apiGroups:
       - ''
     resources:
diff --git a/deploy/charts/rook-ceph/templates/clusterrole.yaml b/deploy/charts/rook-ceph/templates/clusterrole.yaml
index 830798983..a766e44e5 100644
--- a/deploy/charts/rook-ceph/templates/clusterrole.yaml
+++ b/deploy/charts/rook-ceph/templates/clusterrole.yaml
@@ -441,12 +441,6 @@ rules:
   - apiGroups: [""]
     resources: ["persistentvolumeclaims/status"]
     verbs: ["patch"]
-  - apiGroups: ["storage.k8s.io"]
-    resources: ["volumeattachments"]
-    verbs: ["get", "list", "watch", "patch"]
-  - apiGroups: ["storage.k8s.io"]
-    resources: ["volumeattachments/status"]
-    verbs: ["patch"]
 ---
 # TODO: remove this, once https://github.com/rook/rook/issues/10141
 # is resolved.
diff --git a/deploy/charts/rook-ceph/templates/configmap.yaml b/deploy/charts/rook-ceph/templates/configmap.yaml
index cc312f55a..aada7be63 100644
--- a/deploy/charts/rook-ceph/templates/configmap.yaml
+++ b/deploy/charts/rook-ceph/templates/configmap.yaml
@@ -232,7 +232,4 @@ data:
 {{- if .Values.csi.csiCephFSPluginVolumeMount }}
   CSI_CEPHFS_PLUGIN_VOLUME_MOUNT: {{ toYaml .Values.csi.csiCephFSPluginVolumeMount | quote }}
 {{- end }}
-  CSI_CEPHFS_ATTACH_REQUIRED: {{ .Values.csi.cephFSAttachRequired | quote }}
-  CSI_RBD_ATTACH_REQUIRED: {{ .Values.csi.rbdAttachRequired | quote }}
-  CSI_NFS_ATTACH_REQUIRED: {{ .Values.csi.nfsAttachRequired | quote }}
 {{- end }}
diff --git a/deploy/charts/rook-ceph/templates/resources.yaml b/deploy/charts/rook-ceph/templates/resources.yaml
index 2b0ceb967..6973a53b3 100644
--- a/deploy/charts/rook-ceph/templates/resources.yaml
+++ b/deploy/charts/rook-ceph/templates/resources.yaml
@@ -2561,6 +2561,7 @@ spec:
                           description: Enabled represents whether the key rotation is enabled.
                           type: boolean
                         schedule:
+                          default: '@weekly'
                           description: Schedule represents the cron schedule for key rotation.
                           type: string
                       type: object
@@ -13020,6 +13021,7 @@ spec:
                           description: Enabled represents whether the key rotation is enabled.
                           type: boolean
                         schedule:
+                          default: '@weekly'
                           description: Schedule represents the cron schedule for key rotation.
                           type: string
                       type: object
diff --git a/deploy/charts/rook-ceph/values.yaml b/deploy/charts/rook-ceph/values.yaml
index cfc940eef..624adac06 100644
--- a/deploy/charts/rook-ceph/values.yaml
+++ b/deploy/charts/rook-ceph/values.yaml
@@ -378,14 +378,6 @@ csi:
         limits:
           memory: 1Gi
           cpu: 500m
-    - name : csi-attacher
-      resource:
-        requests:
-          memory: 512Mi
-          cpu: 250m
-        limits:
-          memory: 1Gi
-          cpu: 500m
 
   # -- CEPH CSI NFS plugin resource requirement list
   # @default -- see values.yaml
@@ -537,27 +529,6 @@ csi:
     # @default -- labels listed [here](../CRDs/Cluster/ceph-cluster-crd.md#osd-topology)
     crushLocationLabels:
 
-  # -- Whether to skip any attach operation altogether for CephFS PVCs. See more details
-  # [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object).
-  # If cephFSAttachRequired is set to false it skips the volume attachments and makes the creation
-  # of pods using the CephFS PVC fast. **WARNING** It's highly discouraged to use this for
-  # CephFS RWO volumes. Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details.
-  cephFSAttachRequired: true
-  # -- Whether to skip any attach operation altogether for RBD PVCs. See more details
-  # [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object).
-  # If set to false it skips the volume attachments and makes the creation of pods using the RBD PVC fast.
-  # **WARNING** It's highly discouraged to use this for RWO volumes as it can cause data corruption.
-  # csi-addons operations like Reclaimspace and PVC Keyrotation will also not be supported if set
-  # to false since we'll have no VolumeAttachments to determine which node the PVC is mounted on.
-  # Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details.
-  rbdAttachRequired: true
-  # -- Whether to skip any attach operation altogether for NFS PVCs. See more details
-  # [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object).
-  # If cephFSAttachRequired is set to false it skips the volume attachments and makes the creation
-  # of pods using the NFS PVC fast. **WARNING** It's highly discouraged to use this for
-  # NFS RWO volumes. Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details.
-  nfsAttachRequired: true
-
 # -- Enable discovery daemon
 enableDiscoveryDaemon: false
 
diff --git a/deploy/examples/common.yaml b/deploy/examples/common.yaml
index fea047050..6254dfc7b 100644
--- a/deploy/examples/common.yaml
+++ b/deploy/examples/common.yaml
@@ -793,12 +793,9 @@ rules:
     resources:
       - deployments/scale
       - deployments
-      - pods
     verbs:
       - patch
       - delete
-      - get
-      - update
   - apiGroups:
       - ''
     resources:
diff --git a/deploy/examples/crds.yaml b/deploy/examples/crds.yaml
index 213dde29b..40f17ac20 100644
--- a/deploy/examples/crds.yaml
+++ b/deploy/examples/crds.yaml
@@ -2559,6 +2559,7 @@ spec:
                           description: Enabled represents whether the key rotation is enabled.
                           type: boolean
                         schedule:
+                          default: '@weekly'
                           description: Schedule represents the cron schedule for key rotation.
                           type: string
                       type: object
@@ -13012,6 +13013,7 @@ spec:
                           description: Enabled represents whether the key rotation is enabled.
                           type: boolean
                         schedule:
+                          default: '@weekly'
                           description: Schedule represents the cron schedule for key rotation.
                           type: string
                       type: object
diff --git a/deploy/examples/create-external-cluster-resources.py b/deploy/examples/create-external-cluster-resources.py
index a9ac2afda..518620c6f 100644
--- a/deploy/examples/create-external-cluster-resources.py
+++ b/deploy/examples/create-external-cluster-resources.py
@@ -544,7 +544,7 @@ class RadosJSON:
         # if the 'cluster' instance is a dummy one,
         # don't try to reach out to the endpoint
         if isinstance(self.cluster, DummyRados):
-            return
+            return "", "", ""
         protocols = ["http", "https"]
         response_error = None
         for prefix in protocols:
@@ -562,13 +562,18 @@ class RadosJSON:
                 else:
                     r = requests.head(ep, timeout=timeout)
                 if r.status_code == 200:
-                    return prefix, verify
+                    return prefix, verify, ""
             except Exception as err:
                 response_error = err
                 continue
-        raise ExecutionFailureException(
+        sys.stderr.write(
             f"unable to connect to endpoint: {endpoint_str}, failed error: {response_error}"
         )
+        return (
+            "",
+            "",
+            ("-1"),
+        )
 
     def __init__(self, arg_list=None):
         self.out_map = {}
@@ -751,8 +756,9 @@ class RadosJSON:
             monitoring_endpoint_ip, monitoring_endpoint_port
         )
         self._invalid_endpoint(monitoring_endpoint)
-        self.endpoint_dial(monitoring_endpoint)
-
+        _, _, err = self.endpoint_dial(monitoring_endpoint)
+        if err == "-1":
+            raise ExecutionFailureException(err)
         # add the validated active mgr IP into the first index
         mgr_ips.insert(0, monitoring_endpoint_ip)
         all_mgr_ips_str = ",".join(mgr_ips)
@@ -1090,14 +1096,15 @@ class RadosJSON:
                         f"failed to execute command {cmd}. Output: {execErr.output}. "
                         f"Code: {execErr.returncode}. Error: {execErr.stderr}"
                     )
-                    raise Exception(err_msg)
+                    sys.stderr.write(err_msg)
+                    return None, None, False, "-1"
             else:
                 err_msg = (
                     f"failed to execute command {cmd}. Output: {execErr.output}. "
                     f"Code: {execErr.returncode}. Error: {execErr.stderr}"
                 )
-
-                raise Exception(err_msg)
+                sys.stderr.write(err_msg)
+                return None, None, False, "-1"
 
         # if it is python2, don't check for ceph version for adding `info=read` cap(rgw_validation)
         if sys.version_info.major < 3:
@@ -1106,6 +1113,7 @@ class RadosJSON:
                 jsonoutput["keys"][0]["access_key"],
                 jsonoutput["keys"][0]["secret_key"],
                 False,
+                "",
             )
 
         # separately add info=read caps for rgw-endpoint ip validation
@@ -1134,13 +1142,15 @@ class RadosJSON:
                     f"failed to execute command {cmd}. Output: {execErr.output}. "
                     f"Code: {execErr.returncode}. Error: {execErr.stderr}"
                 )
-                raise Exception(err_msg)
+                sys.stderr.write(err_msg)
+                return None, None, False, "-1"
 
         jsonoutput = json.loads(output)
         return (
             jsonoutput["keys"][0]["access_key"],
             jsonoutput["keys"][0]["secret_key"],
             info_cap_supported,
+            "",
         )
 
     def convert_fqdn_rgw_endpoint_to_ip(self, fqdn_rgw_endpoint):
@@ -1156,34 +1166,11 @@ class RadosJSON:
         rgw_endpoint = self._join_host_port(rgw_endpoint_ip, rgw_endpoint_port)
         return rgw_endpoint
 
-    def validate_pool(self):
-        pools_to_validate = [self._arg_parser.rbd_data_pool_name]
-        # if rgw_endpoint is provided, validate it
-        if self._arg_parser.rgw_endpoint:
-            rgw_endpoint = self._arg_parser.rgw_endpoint
-            self._invalid_endpoint(rgw_endpoint)
-            cert = None
-            if (
-                not self._arg_parser.rgw_skip_tls
-                and self.validate_rgw_endpoint_tls_cert()
-            ):
-                cert = self._arg_parser.rgw_tls_cert_path
-            self.endpoint_dial(rgw_endpoint, cert=cert)
-            # only validate if rgw_pool_prefix is passed else it will take default value and we don't create these default pools
-            if self._arg_parser.rgw_pool_prefix != "default":
-                rgw_pool_to_validate = [
-                    f"{self._arg_parser.rgw_pool_prefix}.rgw.meta",
-                    ".rgw.root",
-                    f"{self._arg_parser.rgw_pool_prefix}.rgw.control",
-                    f"{self._arg_parser.rgw_pool_prefix}.rgw.log",
-                ]
-                pools_to_validate.extend(rgw_pool_to_validate)
-
-        for pool in pools_to_validate:
-            if not self.cluster.pool_exists(pool):
-                raise ExecutionFailureException(
-                    f"The provided pool, '{pool}', does not exist"
-                )
+    def validate_rbd_pool(self):
+        if not self.cluster.pool_exists(self._arg_parser.rbd_data_pool_name):
+            raise ExecutionFailureException(
+                f"The provided pool, '{self._arg_parser.rbd_data_pool_name}', does not exist"
+            )
 
     def validate_rados_namespace(self):
         rbd_pool_name = self._arg_parser.rbd_data_pool_name
@@ -1223,14 +1210,10 @@ class RadosJSON:
                 f"subvolume group {subvolume_group} passed doesn't exist"
             )
 
-    def get_rgw_fsid(self):
+    def get_rgw_fsid(self, base_url, verify):
         access_key = self.out_map["RGW_ADMIN_OPS_USER_ACCESS_KEY"]
         secret_key = self.out_map["RGW_ADMIN_OPS_USER_SECRET_KEY"]
         rgw_endpoint = self._arg_parser.rgw_endpoint
-        cert = None
-        if not self._arg_parser.rgw_skip_tls and self.validate_rgw_endpoint_tls_cert():
-            cert = self._arg_parser.rgw_tls_cert_path
-        base_url, verify = self.endpoint_dial(rgw_endpoint, cert=cert)
         base_url = base_url + "://" + rgw_endpoint + "/admin/info?"
         params = {"format": "json"}
         request_url = base_url + urllib.parse.urlencode(params)
@@ -1242,28 +1225,64 @@ class RadosJSON:
                 verify=verify,
             )
         except requests.exceptions.Timeout:
-            raise ExecutionFailureException(
+            sys.stderr.write(
                 f"invalid endpoint:, not able to call admin-ops api{rgw_endpoint}"
             )
+            return "", "-1"
         r1 = r.json()
         if r1 is None or r1.get("info") is None:
-            return ""  # Invalid rgw-endpoint exception will returned by validate_rgw_endpoint()
+            sys.stderr.write(
+                f"The provided rgw Endpoint, '{self._arg_parser.rgw_endpoint}', is invalid."
+            )
+            return (
+                "",
+                "-1",
+            )
 
-        return r1["info"]["storage_backends"][0]["cluster_id"]
+        return r1["info"]["storage_backends"][0]["cluster_id"], ""
 
     def validate_rgw_endpoint(self):
         # if the 'cluster' instance is a dummy one,
         # don't try to reach out to the endpoint
         if isinstance(self.cluster, DummyRados):
             return
+
+        # check if the rgw endpoint is reachable
+        rgw_endpoint = self._arg_parser.rgw_endpoint
+        cert = None
+        if not self._arg_parser.rgw_skip_tls and self.validate_rgw_endpoint_tls_cert():
+            cert = self._arg_parser.rgw_tls_cert_path
+        base_url, verify, err = self.endpoint_dial(rgw_endpoint, cert=cert)
+        if err != "":
+            return "-1"
+
+        # check if the rgw endpoint is valid and belongs to the same cluster
         fsid = self.get_fsid()
-        rgw_fsid = self.get_rgw_fsid()
+        rgw_fsid, err = self.get_rgw_fsid(base_url, verify)
+        if err == "-1":
+            return "-1"
         if fsid != rgw_fsid:
-            raise ExecutionFailureException(
-                (
-                    f"The provided rgw Endpoint, '{self._arg_parser.rgw_endpoint}', is invalid. We are validating by calling the adminops api through rgw-endpoint and validating the cluster_id '{rgw_fsid}' is equal to the ceph cluster fsid '{fsid}'"
-                )
+            sys.stderr.write(
+                f"The provided rgw Endpoint, '{self._arg_parser.rgw_endpoint}', is invalid. We are validating by calling the adminops api through rgw-endpoint and validating the cluster_id '{rgw_fsid}' is equal to the ceph cluster fsid '{fsid}'"
             )
+            return "-1"
+
+        # check if the rgw endpoint exist
+        # only validate if rgw_pool_prefix is passed else it will take default value and we don't create these default pools
+        if self._arg_parser.rgw_pool_prefix != "default":
+            rgw_pool_to_validate = [
+                f"{self._arg_parser.rgw_pool_prefix}.rgw.meta",
+                ".rgw.root",
+                f"{self._arg_parser.rgw_pool_prefix}.rgw.control",
+                f"{self._arg_parser.rgw_pool_prefix}.rgw.log",
+            ]
+            if not self.cluster.pool_exists(self._arg_parser.rgw_pool_to_validate):
+                sys.stderr.write(
+                    f"The provided pool, '{self._arg_parser.rgw_pool_to_validate}', does not exist"
+                )
+                return "-1"
+
+        return ""
 
     def validate_rgw_realm_name(self):
         if self._arg_parser.rgw_realm_name != "":
@@ -1275,7 +1294,7 @@ class RadosJSON:
                 self._arg_parser.rgw_realm_name,
             ]
             try:
-                output = subprocess.check_output(cmd, stderr=subprocess.PIPE)
+                _ = subprocess.check_output(cmd, stderr=subprocess.PIPE)
             except subprocess.CalledProcessError as execErr:
                 err_msg = (
                     f"failed to execute command {cmd}. Output: {execErr.output}. "
@@ -1293,7 +1312,7 @@ class RadosJSON:
             self._arg_parser.rgw_endpoint = self.convert_fqdn_rgw_endpoint_to_ip(
                 self._arg_parser.rgw_endpoint
             )
-        self.validate_pool()
+        self.validate_rbd_pool()
         self.validate_rados_namespace()
         self.validate_subvolume_group()
         self.validate_rgw_realm_name()
@@ -1336,7 +1355,6 @@ class RadosJSON:
                 self.out_map["CSI_CEPHFS_PROVISIONER_SECRET"],
                 self.out_map["CSI_CEPHFS_PROVISIONER_SECRET_NAME"],
             ) = self.create_cephCSIKeyring_user("client.csi-cephfs-provisioner")
-        self.out_map["RGW_ENDPOINT"] = self._arg_parser.rgw_endpoint
         self.out_map["RGW_TLS_CERT"] = ""
         (
             self.out_map["MONITORING_ENDPOINT"],
@@ -1348,6 +1366,7 @@ class RadosJSON:
         ] = self.validate_rgw_metadata_ec_pool_name()
         self.out_map["RGW_REALM_NAME"] = self._arg_parser.rgw_realm_name
         self.out_map["RGW_POOL_PREFIX"] = self._arg_parser.rgw_pool_prefix
+        self.out_map["RGW_ENDPOINT"] = ""
         if self._arg_parser.rgw_endpoint:
             if self._arg_parser.dry_run:
                 self.create_rgw_admin_ops_user()
@@ -1356,11 +1375,15 @@ class RadosJSON:
                     self.out_map["RGW_ADMIN_OPS_USER_ACCESS_KEY"],
                     self.out_map["RGW_ADMIN_OPS_USER_SECRET_KEY"],
                     info_cap_supported,
+                    err,
                 ) = self.create_rgw_admin_ops_user()
                 if info_cap_supported:
-                    self.validate_rgw_endpoint()
-            if self._arg_parser.rgw_tls_cert_path:
-                self.out_map["RGW_TLS_CERT"] = self.validate_rgw_endpoint_tls_cert()
+                    err = self.validate_rgw_endpoint()
+                if self._arg_parser.rgw_tls_cert_path:
+                    self.out_map["RGW_TLS_CERT"] = self.validate_rgw_endpoint_tls_cert()
+                # if there is no error, set the RGW_ENDPOINT
+                if err != "-1":
+                    self.out_map["RGW_ENDPOINT"] = self._arg_parser.rgw_endpoint
 
     def gen_shell_out(self):
         self._gen_output_map()
diff --git a/deploy/examples/csi/nfs/rbac.yaml b/deploy/examples/csi/nfs/rbac.yaml
index 029ec4458..5acf98a13 100644
--- a/deploy/examples/csi/nfs/rbac.yaml
+++ b/deploy/examples/csi/nfs/rbac.yaml
@@ -61,12 +61,6 @@ rules:
   - apiGroups: [""]
     resources: ["persistentvolumeclaims/status"]
     verbs: ["patch"]
-  - apiGroups: ["storage.k8s.io"]
-    resources: ["volumeattachments"]
-    verbs: ["get", "list", "watch", "patch"]
-  - apiGroups: ["storage.k8s.io"]
-    resources: ["volumeattachments/status"]
-    verbs: ["patch"]
 ---
 # TODO: remove this, once https://github.com/rook/rook/issues/10141
 # is resolved.
diff --git a/deploy/examples/dashboard-external-http.yaml b/deploy/examples/dashboard-external-http.yaml
index 82338367f..27fc59866 100644
--- a/deploy/examples/dashboard-external-http.yaml
+++ b/deploy/examples/dashboard-external-http.yaml
@@ -14,7 +14,6 @@ spec:
       targetPort: 7000
   selector:
     app: rook-ceph-mgr
-    mgr_role: active
     rook_cluster: rook-ceph
   sessionAffinity: None
   type: NodePort
diff --git a/deploy/examples/dashboard-external-https.yaml b/deploy/examples/dashboard-external-https.yaml
index 6a18dd30c..9d30fadca 100644
--- a/deploy/examples/dashboard-external-https.yaml
+++ b/deploy/examples/dashboard-external-https.yaml
@@ -14,7 +14,6 @@ spec:
       targetPort: 8443
   selector:
     app: rook-ceph-mgr
-    mgr_role: active
     rook_cluster: rook-ceph
   sessionAffinity: None
   type: NodePort
diff --git a/deploy/examples/dashboard-loadbalancer.yaml b/deploy/examples/dashboard-loadbalancer.yaml
index 19fe8eb5c..98c0b9075 100644
--- a/deploy/examples/dashboard-loadbalancer.yaml
+++ b/deploy/examples/dashboard-loadbalancer.yaml
@@ -14,7 +14,6 @@ spec:
       targetPort: 8443
   selector:
     app: rook-ceph-mgr
-    mgr_role: active
     rook_cluster: rook-ceph
   sessionAffinity: None
   type: LoadBalancer
diff --git a/deploy/examples/monitoring/service-monitor.yaml b/deploy/examples/monitoring/service-monitor.yaml
index 44a7a4f7b..77e82c959 100644
--- a/deploy/examples/monitoring/service-monitor.yaml
+++ b/deploy/examples/monitoring/service-monitor.yaml
@@ -11,9 +11,9 @@ spec:
       - rook-ceph
   selector:
     matchLabels:
-      mgr_role: active
       app: rook-ceph-mgr
       rook_cluster: rook-ceph
+      ceph_daemon_id: a
   endpoints:
   - port: http-metrics
     path: /metrics
diff --git a/deploy/examples/operator-openshift.yaml b/deploy/examples/operator-openshift.yaml
index e3af8bffc..f525850e5 100644
--- a/deploy/examples/operator-openshift.yaml
+++ b/deploy/examples/operator-openshift.yaml
@@ -517,14 +517,6 @@ data:
   #      limits:
   #        memory: 1Gi
   #        cpu: 500m
-  #  - name : csi-attacher
-  #    resource:
-  #      requests:
-  #        memory: 128Mi
-  #        cpu: 100m
-  #      limits:
-  #        memory: 256Mi
-  #        cpu: 200m
   # (Optional) CEPH CSI NFS plugin resource requirement list, Put here list of resource
   # requests and limits you want to apply for plugin pod
   # CSI_NFS_PLUGIN_RESOURCE: |
@@ -592,17 +584,6 @@ data:
   # Defaults to all the labels mentioned in
   # https://rook.io/docs/rook/latest/CRDs/Cluster/ceph-cluster-crd/#osd-topology
   # CSI_CRUSH_LOCATION_LABELS: "kubernetes.io/hostname,topology.kubernetes.io/zone,topology.rook.io/rack"
-
-  # Whether to skip any attach operation altogether for CephCSI PVCs.
-  # See more details [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object).
-  # If set to false it skips the volume attachments and makes the creation of pods using the CephCSI PVC fast.
-  # **WARNING** It's highly discouraged to use this for RWO volumes. for RBD PVC it can cause data corruption,
-  # csi-addons operations like Reclaimspace and PVC Keyrotation will also not be supported if set to false
-  # since we'll have no VolumeAttachments to determine which node the PVC is mounted on.
-  # Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details.
-  CSI_CEPHFS_ATTACH_REQUIRED: "true"
-  CSI_RBD_ATTACH_REQUIRED: "true"
-  CSI_NFS_ATTACH_REQUIRED: "true"
 ---
 # The deployment for the rook operator
 # OLM: BEGIN OPERATOR DEPLOYMENT
diff --git a/deploy/examples/operator.yaml b/deploy/examples/operator.yaml
index 2b38033d5..357eb7055 100644
--- a/deploy/examples/operator.yaml
+++ b/deploy/examples/operator.yaml
@@ -445,14 +445,6 @@ data:
   #      limits:
   #        memory: 1Gi
   #        cpu: 500m
-  #  - name : csi-attacher
-  #    resource:
-  #      requests:
-  #        memory: 128Mi
-  #        cpu: 100m
-  #      limits:
-  #        memory: 256Mi
-  #        cpu: 200m
   # (Optional) CEPH CSI NFS plugin resource requirement list, Put here list of resource
   # requests and limits you want to apply for plugin pod
   # CSI_NFS_PLUGIN_RESOURCE: |
@@ -520,17 +512,6 @@ data:
   # Defaults to all the labels mentioned in
   # https://rook.io/docs/rook/latest/CRDs/Cluster/ceph-cluster-crd/#osd-topology
   # CSI_CRUSH_LOCATION_LABELS: "kubernetes.io/hostname,topology.kubernetes.io/zone,topology.rook.io/rack"
-
-  # Whether to skip any attach operation altogether for CephCSI PVCs.
-  # See more details [here](https://kubernetes-csi.github.io/docs/skip-attach.html#skip-attach-with-csi-driver-object).
-  # If set to false it skips the volume attachments and makes the creation of pods using the CephCSI PVC fast.
-  # **WARNING** It's highly discouraged to use this for RWO volumes. for RBD PVC it can cause data corruption,
-  # csi-addons operations like Reclaimspace and PVC Keyrotation will also not be supported if set to false
-  # since we'll have no VolumeAttachments to determine which node the PVC is mounted on.
-  # Refer to this [issue](https://github.com/kubernetes/kubernetes/issues/103305) for more details.
-  CSI_CEPHFS_ATTACH_REQUIRED: "true"
-  CSI_RBD_ATTACH_REQUIRED: "true"
-  CSI_NFS_ATTACH_REQUIRED: "true"
 ---
 # OLM: BEGIN OPERATOR DEPLOYMENT
 apiVersion: apps/v1
diff --git a/design/ceph/object/store.md b/design/ceph/object/store.md
index 5c8fd6201..4ec577a10 100644
--- a/design/ceph/object/store.md
+++ b/design/ceph/object/store.md
@@ -119,60 +119,3 @@ Specifying this section also ensures that the pool section in the ceph-object-zo
   zone:
     name: "name"
 ```
-
-### LDAP
-
-The Ceph Object Gateway supports integrating with LDAP for authenticating and creating users, please refer [here](https://docs.ceph.com/docs/master/radosgw/ldap-auth/). This means that the `rgw backend user` is also required to be part of groups in the LDAP server otherwise, authentication will fail. The `rgw backend user` can be generated from `CephObjectStoreUser` or `ObjectBucketClaim` CRDs. For the both resources credentials are saved in Kubernetes Secrets which may not be valid with `LDAP Server`, user need to follow the steps mentioned [here](https://docs.ceph.com/en/latest/radosgw/ldap-auth/#generating-an-access-token-for-ldap-authentication).The following settings need to be configured in the RGW server:
-```
-rgw ldap binddn =
-rgw ldap secret = /etc/ceph/ldap/bindpass.secret
-rgw ldap uri =
-rgw ldap searchdn =
-rgw ldap dnattr =
-rgw ldap searchfilter =
-rgw s3 auth use ldap = true
-```
-So the CRD for the Ceph Object Store will be modified to include the above changes:
-
-```yaml
-spec:
-  security
-    ldap:
-      config:
-        uri: ldaps://ldap-server:636
-        binddn: "uid=ceph,cn=users,cn=accounts,dc=example,dc=com"
-        searchdn: "cn=users,cn=accounts,dc=example,dc=com"
-        dnattr: "uid"
-        searchfilter: "memberof=cn=s3,cn=groups,cn=accounts,dc=example,dc=com"
-      credential:
-        volumeSource:
-          secret:
-            secretName: object-my-store-ldap-creds
-            defaultMode: 0600 #required
-
-```
-
-The `config` section includes options used for RGW wrt LDAP server. These options are strongly typed rather than string map approach since very less chance to modify in future.
-
-* `uri`: It specifies the address of LDAP server to use.* `binddn`: The bind domain for the service account used by RGW server.
-* `searchdn`: The search domain where can it look for the user details.
-* `dnattr`: The attribute being used in the constructed search filter to match a username, this can either be `uid` or `cn`.
-* `searchfilter`: A generic search filter. If `dnattr` is set, this filter is `&()`'d together with the automatically constructed filter.
-
-The `credential` defines where the password for accessing ldap server should be sourced from
-
-* `volumeSource`: this is a standard Kubernetes VolumeSource for the Kerberos keytab file like
-      what is normally used to configure Volumes for a Pod. For example, a Secret or HostPath.
-      There are two requirements for the source's content:
-    1. The config file must be mountable via `subPath: password`. For example, in a Secret, the
-        data item must be named `password`, or `items` must be defined to select the key and
-        give it path `password`. A HostPath directory must have the `password` file.
-    2. The volume or config file must have mode 0600.
-
-The CA bundle for ldap can be added to the `caBundleRef` option in `Gateway` settings:
-
-```yaml
-spec:
-  gateway:
-    caBundleRef: #ldaps-cabundle
-```
diff --git a/go.mod b/go.mod
index caa2a25fa..0f06a53de 100644
--- a/go.mod
+++ b/go.mod
@@ -4,7 +4,7 @@ go 1.19
 
 require (
 	github.com/IBM/keyprotect-go-client v0.9.2
-	github.com/aws/aws-sdk-go v1.44.229
+	github.com/aws/aws-sdk-go v1.44.224
 	github.com/banzaicloud/k8s-objectmatcher v1.8.0
 	github.com/ceph/go-ceph v0.20.0
 	github.com/coreos/pkg v0.0.0-20220810130054-c7d1c02cb6cf
@@ -13,7 +13,7 @@ require (
 	github.com/google/go-cmp v0.5.9
 	github.com/google/uuid v1.3.0
 	github.com/hashicorp/vault v1.13.0
-	github.com/hashicorp/vault-plugin-secrets-kv v0.14.3
+	github.com/hashicorp/vault-plugin-secrets-kv v0.14.2
 	github.com/hashicorp/vault/api v1.9.0
 	github.com/hashicorp/vault/sdk v0.8.1
 	github.com/jetstack/cert-manager v1.7.3
@@ -116,7 +116,7 @@ require (
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang-jwt/jwt/v4 v4.4.2 // indirect
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
-	github.com/golang/protobuf v1.5.3 // indirect
+	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/golang/snappy v0.0.4 // indirect
 	github.com/google/gnostic v0.6.9 // indirect
 	github.com/google/go-metrics-stackdriver v0.2.0 // indirect
@@ -130,7 +130,7 @@ require (
 	github.com/hashicorp/eventlogger v0.1.1 // indirect
 	github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
 	github.com/hashicorp/go-discover v0.0.0-20210818145131-c573d69da192 // indirect
-	github.com/hashicorp/go-hclog v1.5.0 // indirect
+	github.com/hashicorp/go-hclog v1.4.0 // indirect
 	github.com/hashicorp/go-immutable-radix v1.3.1 // indirect
 	github.com/hashicorp/go-kms-wrapping/entropy/v2 v2.0.0 // indirect
 	github.com/hashicorp/go-kms-wrapping/v2 v2.0.8 // indirect
@@ -238,18 +238,18 @@ require (
 	go.uber.org/multierr v1.8.0 // indirect
 	go.uber.org/zap v1.24.0 // indirect
 	golang.org/x/crypto v0.6.0 // indirect
-	golang.org/x/net v0.8.0 // indirect
+	golang.org/x/net v0.7.0 // indirect
 	golang.org/x/oauth2 v0.4.0 // indirect
-	golang.org/x/sys v0.6.0 // indirect
-	golang.org/x/term v0.6.0 // indirect
-	golang.org/x/text v0.8.0 // indirect
+	golang.org/x/sys v0.5.0 // indirect
+	golang.org/x/term v0.5.0 // indirect
+	golang.org/x/text v0.7.0 // indirect
 	golang.org/x/time v0.3.0 // indirect
 	gomodules.xyz/jsonpatch/v2 v2.2.0 // indirect
 	google.golang.org/api v0.109.0 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
 	google.golang.org/genproto v0.0.0-20221227171554-f9683d7f8bef // indirect
 	google.golang.org/grpc v1.51.0 // indirect
-	google.golang.org/protobuf v1.30.0 // indirect
+	google.golang.org/protobuf v1.28.1 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/resty.v1 v1.12.0 // indirect
 	gopkg.in/square/go-jose.v2 v2.6.0 // indirect
diff --git a/go.sum b/go.sum
index e927c3738..6296d6cdc 100644
--- a/go.sum
+++ b/go.sum
@@ -224,8 +224,8 @@ github.com/aws/aws-sdk-go v1.25.37/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpi
 github.com/aws/aws-sdk-go v1.25.41/go.mod h1:KmX6BPdI08NWTb3/sm4ZGu5ShLoqVDhKgpiN924inxo=
 github.com/aws/aws-sdk-go v1.30.27/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0=
 github.com/aws/aws-sdk-go v1.34.28/go.mod h1:H7NKnBqNVzoTJpGfLrQkkD+ytBA93eiDYi/+8rV9s48=
-github.com/aws/aws-sdk-go v1.44.229 h1:lku0ZSHRzj/qtFVM//QE8VjV6kvJ6CFijDZSsjNaD9A=
-github.com/aws/aws-sdk-go v1.44.229/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI=
+github.com/aws/aws-sdk-go v1.44.224 h1:09CiaaF35nRmxrzWZ2uRq5v6Ghg/d2RiPjZnSgtt+RQ=
+github.com/aws/aws-sdk-go v1.44.224/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI=
 github.com/aws/aws-sdk-go-v2 v1.8.0 h1:HcN6yDnHV9S7D69E7To0aUppJhiJNEzQSNcUxc7r3qo=
 github.com/aws/aws-sdk-go-v2/credentials v1.3.2 h1:Uud/fZzm0lqqhE8kvXYJFAJ3PGnagKoUcvHq1hXfBZw=
 github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.4.0 h1:Iqp2aHeRF3kaaNuDS82bHBzER285NM6lLPAgsxHCR2A=
@@ -597,6 +597,7 @@ github.com/go-test/deep v1.0.1/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3a
 github.com/go-test/deep v1.0.2-0.20181118220953-042da051cf31/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3aSFNA=
 github.com/go-test/deep v1.0.2/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3aSFNA=
 github.com/go-test/deep v1.0.4/go.mod h1:wGDj63lr65AM2AQyKZd/NYHGb0R+1RLqB8NKt3aSFNA=
+github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
 github.com/go-test/deep v1.1.0 h1:WOcxcdHcvdgThNXjw0t76K42FXTU7HpNQWHpA2HHNlg=
 github.com/go-test/deep v1.1.0/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
 github.com/go-yaml/yaml v2.1.0+incompatible/go.mod h1:w2MrLa16VYP0jy6N7M5kHaCkaLENm+P+Tv+MfurjSw0=
@@ -682,9 +683,8 @@ github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw
 github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
 github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
 github.com/golang/protobuf v1.5.1/go.mod h1:DopwsBzvsk0Fs44TXzsVbJyPhcCPeIwnvohx4u74HPM=
+github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
 github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
-github.com/golang/protobuf v1.5.3 h1:KhyjKVUg7Usr/dYsdSqoFveMYd5ko72D+zANwlG1mmg=
-github.com/golang/protobuf v1.5.3/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
 github.com/golang/snappy v0.0.0-20170215233205-553a64147049/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
 github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
 github.com/golang/snappy v0.0.1/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
@@ -816,8 +816,9 @@ github.com/hashicorp/go-hclog v0.10.1/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39
 github.com/hashicorp/go-hclog v0.12.0/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ=
 github.com/hashicorp/go-hclog v0.14.1/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ=
 github.com/hashicorp/go-hclog v0.16.2/go.mod h1:whpDNt7SSdeAju8AWKIWsul05p54N/39EeqMAyrmvFQ=
-github.com/hashicorp/go-hclog v1.5.0 h1:bI2ocEMgcVlz55Oj1xZNBsVi900c7II+fWDyV9o+13c=
-github.com/hashicorp/go-hclog v1.5.0/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M=
+github.com/hashicorp/go-hclog v1.3.1/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M=
+github.com/hashicorp/go-hclog v1.4.0 h1:ctuWFGrhFha8BnnzxqeRGidlEcQkDyL5u8J8t5eA11I=
+github.com/hashicorp/go-hclog v1.4.0/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M=
 github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
 github.com/hashicorp/go-immutable-radix v1.1.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
 github.com/hashicorp/go-immutable-radix v1.3.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
@@ -828,6 +829,7 @@ github.com/hashicorp/go-kms-wrapping v0.5.1/go.mod h1:cGIibZmMx9qlxS1pZTUrEgGqA+
 github.com/hashicorp/go-kms-wrapping/entropy v0.1.0/go.mod h1:d1g9WGtAunDNpek8jUIEJnBlbgKS1N2Q61QkHiZyR1g=
 github.com/hashicorp/go-kms-wrapping/entropy/v2 v2.0.0 h1:pSjQfW3vPtrOTcasTUKgCTQT7OGPPTTMVRrOfU6FJD8=
 github.com/hashicorp/go-kms-wrapping/entropy/v2 v2.0.0/go.mod h1:xvb32K2keAc+R8DSFG2IwDcydK9DBQE+fGA5fsw6hSk=
+github.com/hashicorp/go-kms-wrapping/v2 v2.0.7/go.mod h1:sDQAfwJGv25uGPZA04x87ERglCG6avnRcBT9wYoMII8=
 github.com/hashicorp/go-kms-wrapping/v2 v2.0.8 h1:9Q2lu1YbbmiAgvYZ7Pr31RdlVonUpX+mmDL7Z7qTA2U=
 github.com/hashicorp/go-kms-wrapping/v2 v2.0.8/go.mod h1:qTCjxGig/kjuj3hk1z8pOUrzbse/GxB1tGfbrq8tGJg=
 github.com/hashicorp/go-kms-wrapping/wrappers/aead/v2 v2.0.7-1 h1:ZV26VJYcITBom0QqYSUOIj4HOHCVPEFjLqjxyXV/AbA=
@@ -857,6 +859,7 @@ github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9
 github.com/hashicorp/go-plugin v1.0.0/go.mod h1:++UyYGoz3o5w9ZzAdZxtQKrWWP+iqPBn3cQptSMzBuY=
 github.com/hashicorp/go-plugin v1.0.1/go.mod h1:++UyYGoz3o5w9ZzAdZxtQKrWWP+iqPBn3cQptSMzBuY=
 github.com/hashicorp/go-plugin v1.4.3/go.mod h1:5fGEH17QVwTTcR0zV7yhDPLLmFX9YSZ38b18Udy6vYQ=
+github.com/hashicorp/go-plugin v1.4.5/go.mod h1:viDMjcLJuDui6pXb8U4HVfb8AamCWhHGUjr2IrTF67s=
 github.com/hashicorp/go-plugin v1.4.8 h1:CHGwpxYDOttQOY7HOWgETU9dyVjOXzniXDqJcYJE1zM=
 github.com/hashicorp/go-plugin v1.4.8/go.mod h1:viDMjcLJuDui6pXb8U4HVfb8AamCWhHGUjr2IrTF67s=
 github.com/hashicorp/go-raftchunking v0.6.3-0.20191002164813-7e9e8525653a h1:FmnBDwGwlTgugDGbVxwV8UavqSMACbGrUpfc98yFLR4=
@@ -994,8 +997,8 @@ github.com/hashicorp/vault-plugin-secrets-gcpkms v0.5.5/go.mod h1:b6RwFD1bny1zbf
 github.com/hashicorp/vault-plugin-secrets-gcpkms v0.14.0 h1:eUC5ltK+1bkc+SVMzAUq4tBeNrsDXyCuITH8jeajXcM=
 github.com/hashicorp/vault-plugin-secrets-kubernetes v0.3.0 h1:Joz9SBwjpEOGu+Ynv60JC3fAA4UuLJzu7NcrKm6wMMs=
 github.com/hashicorp/vault-plugin-secrets-kv v0.5.5/go.mod h1:oNyUoMMQq6uNTwyYPnkldiedaknYbPfQIdKoyKQdy2g=
-github.com/hashicorp/vault-plugin-secrets-kv v0.14.3 h1:6hV/p/yXff50v540Kc6RexOP4z3My398NRmSI5xaX90=
-github.com/hashicorp/vault-plugin-secrets-kv v0.14.3/go.mod h1:tQ87Y0wm1JhliWuiXReCvh6uM0lpFXyrEkuWsFJUR6o=
+github.com/hashicorp/vault-plugin-secrets-kv v0.14.2 h1:13p50RIltQM/JH32uWZe9sAp16Uaj0zCLmVGPvS09qo=
+github.com/hashicorp/vault-plugin-secrets-kv v0.14.2/go.mod h1:cAxt2o3BjRT5CbNLtgXuxTReaejvrgN/qk+no+DnwJ8=
 github.com/hashicorp/vault-plugin-secrets-mongodbatlas v0.1.2/go.mod h1:YRW9zn9NZNitRlPYNAWRp/YEdKCF/X8aOg8IYSxFT5Y=
 github.com/hashicorp/vault-plugin-secrets-mongodbatlas v0.9.1 h1:WkW8fyHxEdz1wGSTxCnSCrzXvgLXqXr8Iqp7upa/s4E=
 github.com/hashicorp/vault-plugin-secrets-openldap v0.1.3-0.20200518214608-746aba5fead6/go.mod h1:9Cy4Jp779BjuIOhYLjEfH3M3QCUxZgPnvJ3tAOOmof4=
@@ -1277,6 +1280,7 @@ github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826/go.mod h1:TaXosZuwd
 github.com/mongodb-forks/digest v1.0.3 h1:ZUK1vyZnBiRMvET0O1SzmnBmv935CkcOTjhfR4zIQ2s=
 github.com/mongodb/go-client-mongodb-atlas v0.1.2/go.mod h1:LS8O0YLkA+sbtOb3fZLF10yY3tJM+1xATXMJ3oU35LU=
 github.com/montanaflynn/stats v0.0.0-20171201202039-1bf9dbcd8cbe/go.mod h1:wL8QJuTMNUDYhXwkmfOly8iTdp5TEcJFWZD2D7SIkUc=
+github.com/mr-tron/base58 v1.2.0/go.mod h1:BinMc/sQntlIE1frQmRFPUoPA1Zkr8VRgBdjWI2mNwc=
 github.com/munnerz/goautoneg v0.0.0-20120707110453-a547fc61f48d/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
@@ -1371,6 +1375,7 @@ github.com/pierrec/lz4/v4 v4.1.8 h1:ieHkV+i2BRzngO4Wd/3HGowuZStgq6QkPsD1eolNAO4=
 github.com/pires/go-proxyproto v0.6.1 h1:EBupykFmo22SDjv4fQVQd2J9NOoLPmyZA/15ldOGkPw=
 github.com/pires/go-proxyproto v0.6.1/go.mod h1:Odh9VFOZJCf9G8cLW5o435Xf1J95Jw9Gw5rnCjcwzAY=
 github.com/pkg/browser v0.0.0-20210911075715-681adbf594b8 h1:KoWmjvw+nsYOo29YJK9vDA65RGE3NrOnUtO7a+RF9HU=
+github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
 github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
@@ -1460,6 +1465,7 @@ github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFR
 github.com/rogpeppe/go-internal v1.6.1/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
 github.com/rogpeppe/go-internal v1.6.2/go.mod h1:xXDCJY+GAPziupqXw64V24skbSoqbTEfhy4qGm1nDQc=
 github.com/rogpeppe/go-internal v1.9.0 h1:73kH8U+JUqXU8lRuOHeVHaa/SZPifC7BkcraZVejAe8=
+github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
 github.com/rs/zerolog v1.4.0/go.mod h1:YbFCdg8HfsridGWAh22vktObvhZbQsZXe4/zB0OKkWU=
 github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
 github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@@ -1687,7 +1693,9 @@ golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5y
 golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
+golang.org/x/crypto v0.0.0-20220313003712-b769efc7c000/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
 golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
+golang.org/x/crypto v0.5.0/go.mod h1:NK/OQwhpMQP3MwtdjgLlYHnH9ebylxKWv3e0fK+mkQU=
 golang.org/x/crypto v0.6.0 h1:qfktjS5LUO+fFKeJXZ+ikTRijMmljikvG68fpMMruSc=
 golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
@@ -1791,8 +1799,9 @@ golang.org/x/net v0.0.0-20220127200216-cd36cc0744dd/go.mod h1:CfG3xpIq0wQ8r1q4Su
 golang.org/x/net v0.0.0-20220225172249-27dd8689420f/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco=
-golang.org/x/net v0.8.0 h1:Zrh2ngAOFYneWTAIAPethzeaQLuHwhuBkuV6ZiRnUaQ=
-golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
+golang.org/x/net v0.5.0/go.mod h1:DivGGAXEgPSlEBzxGzZI+ZLohi+xUj054jfeKui00ws=
+golang.org/x/net v0.7.0 h1:rJrUqqhjsgNp7KqAIc25s9pZnjU7TUcSY7HcVZjdn1g=
+golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20181203162652-d668ce993890/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190130055435-99b60b757ec1/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -1926,15 +1935,17 @@ golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.6.0 h1:MVltZSvRTcU2ljQOhs94SXPftV6DCNnZViHeQps87pQ=
-golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.4.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.5.0 h1:MUK/U/4lj1t1oPg0HfuXDN/Z1wv31ZJ/YcPiGccS4DU=
+golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210615171337-6886f2dfbf5b/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
-golang.org/x/term v0.6.0 h1:clScbb1cHjoCkyRbWwBEUZ5H/tIFu5TAXIqaZD0Gcjw=
-golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
+golang.org/x/term v0.4.0/go.mod h1:9P2UbLfCdcvo3p/nzKvsmas4TnlujnuoV9hGgYzW1lQ=
+golang.org/x/term v0.5.0 h1:n2a8QNdAb0sZNpU9R1ALUXBbY+w51fCQDN+7EdxNBsY=
+golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
 golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -1947,8 +1958,9 @@ golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
-golang.org/x/text v0.8.0 h1:57P1ETyNKtuIjB4SRd15iJxuhj8Gc416Y78H3qgMh68=
-golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
+golang.org/x/text v0.6.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
+golang.org/x/text v0.7.0 h1:4BRB4x83lYWy72KwLD/qYDuTu7q9PjSagHvijDw7cLo=
+golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
@@ -2175,8 +2187,8 @@ google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlba
 google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
 google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
 google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
-google.golang.org/protobuf v1.30.0 h1:kPPoIgf3TsEvrm0PFe15JQ+570QVxYzEvvHqChK+cng=
-google.golang.org/protobuf v1.30.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
+google.golang.org/protobuf v1.28.1 h1:d0NfwRgPtno5B1Wa6L2DAG+KivqkdutMf1UhdNx175w=
+google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
 gopkg.in/airbrake/gobrake.v2 v2.0.9/go.mod h1:/h5ZAUhDkGaJfjzjKLSjv6zCL6O0LLBxU4K+aSYdM/U=
 gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
 gopkg.in/asn1-ber.v1 v1.0.0-20181015200546-f715ec2f112d/go.mod h1:cuepJuh7vyXfUyUwEgHQXw849cJrilpS5NeIjOWESAw=
diff --git a/pkg/apis/ceph.rook.io/v1/types.go b/pkg/apis/ceph.rook.io/v1/types.go
index 633f3f54c..52762d9aa 100755
--- a/pkg/apis/ceph.rook.io/v1/types.go
+++ b/pkg/apis/ceph.rook.io/v1/types.go
@@ -281,6 +281,7 @@ type KeyRotationSpec struct {
 	Enabled bool `json:"enabled,omitempty"`
 	// Schedule represents the cron schedule for key rotation.
 	// +optional
+	// +kubebuilder:default="@weekly"
 	Schedule string `json:"schedule,omitempty"`
 }
 
diff --git a/pkg/daemon/ceph/client/config.go b/pkg/daemon/ceph/client/config.go
index f2cc6119b..8d4148b0d 100644
--- a/pkg/daemon/ceph/client/config.go
+++ b/pkg/daemon/ceph/client/config.go
@@ -261,8 +261,7 @@ func PopulateMonHostMembers(clusterInfo *ClusterInfo) ([]string, []string) {
 		currentMonPort := cephutil.GetPortFromEndpoint(monitor.Endpoint)
 
 		if currentMonPort == Msgr2port {
-			msgr2Endpoint := net.JoinHostPort(monIP, strconv.Itoa(int(Msgr2port)))
-			monHosts = append(monHosts, "[v2:"+msgr2Endpoint+"]")
+			monHosts = append(monHosts, fmt.Sprintf("[v2:%s:%d]", monIP, Msgr2port))
 		} else {
 			msgr2Endpoint := net.JoinHostPort(monIP, strconv.Itoa(int(Msgr2port)))
 			msgr1Endpoint := net.JoinHostPort(monIP, strconv.Itoa(int(currentMonPort)))
diff --git a/pkg/daemon/ceph/osd/daemon.go b/pkg/daemon/ceph/osd/daemon.go
index 99a85b744..8bf087203 100644
--- a/pkg/daemon/ceph/osd/daemon.go
+++ b/pkg/daemon/ceph/osd/daemon.go
@@ -531,7 +531,19 @@ func getAvailableDevices(context *clusterd.Context, agent *OsdAgent) (*DeviceOsd
 
 				if matched {
 					matchedDevice = desiredDevice
-					matchedDevice.UpdateDeviceClass(agent, device)
+					if matchedDevice.DeviceClass == "" {
+						classNotSet := true
+						if agent.pvcBacked {
+							crushDeviceClass := os.Getenv(oposd.CrushDeviceClassVarName)
+							if crushDeviceClass != "" {
+								matchedDevice.DeviceClass = crushDeviceClass
+								classNotSet = false
+							}
+						}
+						if classNotSet {
+							matchedDevice.DeviceClass = sys.GetDiskDeviceClass(device)
+						}
+					}
 					break
 				}
 			}
diff --git a/pkg/daemon/ceph/osd/device.go b/pkg/daemon/ceph/osd/device.go
index 3c4676762..152666502 100644
--- a/pkg/daemon/ceph/osd/device.go
+++ b/pkg/daemon/ceph/osd/device.go
@@ -18,9 +18,7 @@ package osd
 
 import (
 	"encoding/json"
-	"os"
 
-	oposd "github.com/rook/rook/pkg/operator/ceph/cluster/osd"
 	"github.com/rook/rook/pkg/util/sys"
 )
 
@@ -69,29 +67,3 @@ func (m *DeviceOsdMapping) String() string {
 	b, _ := json.Marshal(m)
 	return string(b)
 }
-
-func (d *DesiredDevice) UpdateDeviceClass(agent *OsdAgent, device *sys.LocalDisk) {
-	// Rook sets the storage class of a device with the following priority.
-	//
-	// 1. the device-level configuration
-	// 2. the global or node-local configuration
-	// 3. the default value estimated from sysfs.
-	if d.DeviceClass != "" {
-		return
-	}
-
-	if agent.pvcBacked {
-		crushDeviceClass := os.Getenv(oposd.CrushDeviceClassVarName)
-		if crushDeviceClass != "" {
-			d.DeviceClass = crushDeviceClass
-			return
-		}
-	} else {
-		if agent.storeConfig.DeviceClass != "" {
-			d.DeviceClass = agent.storeConfig.DeviceClass
-			return
-		}
-	}
-
-	d.DeviceClass = sys.GetDiskDeviceClass(device)
-}
diff --git a/pkg/operator/ceph/cluster/cluster_external.go b/pkg/operator/ceph/cluster/cluster_external.go
index c21e7f191..e5b5c2a93 100644
--- a/pkg/operator/ceph/cluster/cluster_external.go
+++ b/pkg/operator/ceph/cluster/cluster_external.go
@@ -211,7 +211,7 @@ func (c *ClusterController) configureExternalClusterMonitoring(context *clusterd
 	)
 
 	// Create external monitoring Service
-	service, err := manager.MakeMetricsService(opcontroller.ExternalMgrAppName, opcontroller.ServiceExternalMetricName)
+	service, err := manager.MakeMetricsService(opcontroller.ExternalMgrAppName, "", opcontroller.ServiceExternalMetricName)
 	if err != nil {
 		return err
 	}
@@ -231,7 +231,7 @@ func (c *ClusterController) configureExternalClusterMonitoring(context *clusterd
 	// Deploy external ServiceMonitor
 	logger.Info("creating external service monitor")
 	// servicemonitor takes some metadata from the service for easy mapping
-	err = manager.EnableServiceMonitor()
+	err = manager.EnableServiceMonitor("")
 	if err != nil {
 		logger.Errorf("failed to enable external service monitor. %v", err)
 	} else {
diff --git a/pkg/operator/ceph/cluster/mgr/dashboard.go b/pkg/operator/ceph/cluster/mgr/dashboard.go
index 8d2497a7a..7abf6c25a 100644
--- a/pkg/operator/ceph/cluster/mgr/dashboard.go
+++ b/pkg/operator/ceph/cluster/mgr/dashboard.go
@@ -54,8 +54,8 @@ var (
 	dashboardInitWaitTime = 5 * time.Second
 )
 
-func (c *Cluster) configureDashboardService() error {
-	dashboardService, err := c.makeDashboardService(AppName)
+func (c *Cluster) configureDashboardService(activeDaemon string) error {
+	dashboardService, err := c.makeDashboardService(AppName, activeDaemon)
 	if err != nil {
 		return err
 	}
diff --git a/pkg/operator/ceph/cluster/mgr/mgr.go b/pkg/operator/ceph/cluster/mgr/mgr.go
index 580e09438..efd6d2a60 100644
--- a/pkg/operator/ceph/cluster/mgr/mgr.go
+++ b/pkg/operator/ceph/cluster/mgr/mgr.go
@@ -50,7 +50,6 @@ const (
 	PgautoscalerModuleName = "pg_autoscaler"
 	balancerModuleName     = "balancer"
 	balancerModuleMode     = "upmap"
-	mgrRoleLabelName       = "mgr_role"
 	monitoringPath         = "/etc/ceph-monitoring/"
 	serviceMonitorFile     = "service-monitor.yaml"
 	// minimum amount of memory in MB to run the pod
@@ -178,7 +177,18 @@ func (c *Cluster) Start() error {
 	// check if any extra mgrs need to be removed
 	c.removeExtraMgrs(daemonIDs)
 
+	activeMgr := daemonIDs[0]
 	if len(daemonIDs) > 1 {
+		// When multiple mgrs are running, the mgr sidecar for the active mgr
+		// will create the services. However, the sidecar will only reconcile all
+		// the services when the active mgr changes. Here as part of the regular reconcile
+		// we trigger reconciling all the services to ensure they are current.
+		activeMgr, err = c.getActiveMgr()
+		if err != nil || activeMgr == "" {
+			activeMgr = ""
+			logger.Infof("cannot reconcile mgr services, no active mgr found. err=%v", err)
+		}
+
 		// reconcile mgr PDB
 		if err := c.reconcileMgrPDB(); err != nil {
 			return errors.Wrap(err, "failed to reconcile mgr PDB")
@@ -187,9 +197,10 @@ func (c *Cluster) Start() error {
 		// delete MGR PDB as the count is less than 2
 		c.deleteMgrPDB()
 	}
-
-	if err := c.reconcileServices(); err != nil {
-		return errors.Wrap(err, "failed to enable mgr services")
+	if activeMgr != "" {
+		if err := c.reconcileServices(activeMgr); err != nil {
+			return errors.Wrap(err, "failed to enable mgr services")
+		}
 	}
 
 	// configure the mgr modules
@@ -210,52 +221,35 @@ func (c *Cluster) removeExtraMgrs(daemonIDs []string) {
 	}
 }
 
-// UpdateActiveMgrLabel updates the mgr_role label value to either
-// active or standby depending on the status of the ceph mgr running
-// in the pod
-func (c *Cluster) UpdateActiveMgrLabel(daemonNameToUpdate string, prevActiveMgr string) (string, error) {
-
-	logger.Infof("Checking mgr_role label value of daemon %s (prev active mgr was %s)", daemonNameToUpdate, prevActiveMgr)
-	currActiveMgr, err := c.getActiveMgr()
-	if err != nil || currActiveMgr == "" {
-		logger.Infof("cannot update active mgr, no active mgr found. err=%v", err)
-		return "", err
-	} else if prevActiveMgr == currActiveMgr {
-		logger.Infof("active mgr is still the same (%s). No need to update mgr_role label on daemon %s.", currActiveMgr, daemonNameToUpdate)
-		return currActiveMgr, err
+// ReconcileActiveMgrServices reconciles the services if the active mgr is the one running
+// in the sidecar
+func (c *Cluster) ReconcileActiveMgrServices(daemonNameToUpdate string) error {
+	// If the services are already set to this daemon, no need to attempt to update
+	svc, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Get(c.clusterInfo.Context, AppName, metav1.GetOptions{})
+	if err != nil {
+		logger.Errorf("failed to check current mgr service, proceeding to update. %v", err)
+	} else {
+		currentDaemon := svc.Spec.Selector[controller.DaemonIDLabel]
+		if currentDaemon == daemonNameToUpdate {
+			logger.Infof("mgr services already set to daemon %q, no need to update", daemonNameToUpdate)
+			return nil
+		}
+		logger.Infof("mgr service currently set to %q, checking if need to update to %q", currentDaemon, daemonNameToUpdate)
 	}
 
-	pods, err := c.context.Clientset.CoreV1().Pods(c.clusterInfo.Namespace).List(c.clusterInfo.Context, metav1.ListOptions{
-		LabelSelector: fmt.Sprintf("mgr=%s", daemonNameToUpdate),
-	})
+	activeName, err := c.getActiveMgr()
 	if err != nil {
-		logger.Infof("cannot get pod for mgr daemon %s", daemonNameToUpdate)
-		return "", err // force mrg_role update in the next call
+		return err
 	}
-
-	for i, pod := range pods.Items {
-
-		labels := pod.GetLabels()
-		cephDaemonId := labels[controller.DaemonIDLabel]
-		newMgrRole := "standby"
-		if currActiveMgr == cephDaemonId {
-			newMgrRole = "active"
-		}
-
-		currMgrRole, mgrHasLabel := labels[mgrRoleLabelName]
-		if !mgrHasLabel || currMgrRole != newMgrRole {
-
-			logger.Infof("updating mgr_role label value of daemon %s to '%s'. New active mgr is %s.", daemonNameToUpdate, newMgrRole, currActiveMgr)
-			labels[mgrRoleLabelName] = newMgrRole
-			pod.SetLabels(labels)
-			_, err = c.context.Clientset.CoreV1().Pods(c.clusterInfo.Namespace).Update(c.clusterInfo.Context, &pods.Items[i], metav1.UpdateOptions{})
-			if err != nil {
-				logger.Infof("cannot update the active mgr pod %q. err=%v", pods.Items[i].Name, err)
-			}
-		}
+	if activeName == "" {
+		return errors.New("active mgr not found")
+	}
+	if daemonNameToUpdate != activeName {
+		logger.Infof("no need for the mgr update since the active mgr is %q, rather than the local mgr %q", activeName, daemonNameToUpdate)
+		return nil
 	}
 
-	return currActiveMgr, c.reconcileServices()
+	return c.reconcileServices(activeName)
 }
 
 func (c *Cluster) getActiveMgr() (string, error) {
@@ -266,15 +260,16 @@ func (c *Cluster) getActiveMgr() (string, error) {
 	return mgrStat.ActiveName, nil
 }
 
-// reconcile the services,
-func (c *Cluster) reconcileServices() error {
+// reconcile the services, if the active mgr is not detected, use the default mgr
+func (c *Cluster) reconcileServices(activeDaemon string) error {
+	logger.Infof("setting services to point to mgr %q", activeDaemon)
 
-	if err := c.configureDashboardService(); err != nil {
+	if err := c.configureDashboardService(activeDaemon); err != nil {
 		return errors.Wrap(err, "failed to configure dashboard svc")
 	}
 
 	// create the metrics service
-	service, err := c.MakeMetricsService(AppName, serviceMetricName)
+	service, err := c.MakeMetricsService(AppName, activeDaemon, serviceMetricName)
 	if err != nil {
 		return err
 	}
@@ -284,56 +279,41 @@ func (c *Cluster) reconcileServices() error {
 
 	// enable monitoring if `monitoring: enabled: true`
 	if c.spec.Monitoring.Enabled {
-		if err := c.EnableServiceMonitor(); err != nil {
+		if err := c.EnableServiceMonitor(activeDaemon); err != nil {
 			return errors.Wrap(err, "failed to enable service monitor")
 		}
 	}
 
-	c.updateServiceSelectors()
-	return nil
+	return c.updateServiceSelectors(activeDaemon)
 }
 
-// For the upgrade scenario: we remove any selector DaemonIDLabel from the all
-// the services since new mgr HA doesn't rely on this label anymore and we add
-// the new "mgr_role=active" instead.
-func (c *Cluster) updateServiceSelectors() {
+// Make a best effort to update the services that have been labeled for being updated
+// when the mgr has changed. They might be services for node ports, ingress, etc
+func (c *Cluster) updateServiceSelectors(activeDaemon string) error {
 	selector := metav1.ListOptions{LabelSelector: "app=rook-ceph-mgr"}
 	services, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).List(c.clusterInfo.Context, selector)
 	if err != nil {
-		logger.Errorf("failed to query mgr services to update labels: %v", err)
-		return
+		return errors.Wrap(err, "failed to query mgr services to update")
 	}
 	for i, service := range services.Items {
 		if service.Spec.Selector == nil {
-			continue
-		}
-
-		updateService := false
-
-		// Check if the service has a DaemonIDLabel (legacy mgr HA implementation) and remove it
-		_, hasDaemonLabel := service.Spec.Selector[controller.DaemonIDLabel]
-		if hasDaemonLabel {
-			logger.Infof("removing %s selector label on mgr service %q", controller.DaemonIDLabel, service.Name)
-			delete(service.Spec.Selector, controller.DaemonIDLabel)
-			updateService = true
+			service.Spec.Selector = map[string]string{}
 		}
-
-		// In case the service doesn't have the new mgr_role DaemonIDLabel we add it
-		_, hasMgrRoleLabel := service.Spec.Selector[mgrRoleLabelName]
-		if !hasMgrRoleLabel {
-			logger.Infof("adding %s selector label on mgr service %q", mgrRoleLabelName, service.Name)
-			service.Spec.Selector[mgrRoleLabelName] = "active"
-			updateService = true
+		// Update the selector on the service to point to the active mgr
+		if service.Spec.Selector[controller.DaemonIDLabel] == activeDaemon {
+			logger.Infof("no need to update service %q", service.Name)
+			continue
 		}
-
-		if updateService {
-			if _, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Update(c.clusterInfo.Context, &services.Items[i], metav1.UpdateOptions{}); err != nil {
-				logger.Errorf("failed to update service %q. %v", service.Name, err)
-			} else {
-				logger.Infof("service %q successfully updated", service.Name)
-			}
+		// Update the service to point to the new active mgr
+		service.Spec.Selector[controller.DaemonIDLabel] = activeDaemon
+		logger.Infof("updating selector on mgr service %q to active mgr %q", service.Name, activeDaemon)
+		if _, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Update(c.clusterInfo.Context, &services.Items[i], metav1.UpdateOptions{}); err != nil {
+			logger.Errorf("failed to update service %q. %v", service.Name, err)
+		} else {
+			logger.Infof("service %q successfully updated to active mgr %q", service.Name, activeDaemon)
 		}
 	}
+	return nil
 }
 
 func (c *Cluster) configureModules(daemonIDs []string) {
@@ -450,7 +430,7 @@ func wellKnownModule(name string) bool {
 }
 
 // EnableServiceMonitor add a servicemonitor that allows prometheus to scrape from the monitoring endpoint of the cluster
-func (c *Cluster) EnableServiceMonitor() error {
+func (c *Cluster) EnableServiceMonitor(activeDaemon string) error {
 	serviceMonitor, err := k8sutil.GetServiceMonitor(path.Join(monitoringPath, serviceMonitorFile))
 	if err != nil {
 		return errors.Wrap(err, "service monitor could not be enabled")
@@ -467,6 +447,7 @@ func (c *Cluster) EnableServiceMonitor() error {
 		return errors.Wrapf(err, "failed to set owner reference to service monitor %q", serviceMonitor.Name)
 	}
 	serviceMonitor.Spec.NamespaceSelector.MatchNames = []string{c.clusterInfo.Namespace}
+	serviceMonitor.Spec.Selector.MatchLabels = c.selectorLabels(activeDaemon)
 
 	applyMonitoringLabels(c, serviceMonitor)
 
diff --git a/pkg/operator/ceph/cluster/mgr/spec.go b/pkg/operator/ceph/cluster/mgr/spec.go
index de575c3f3..a73997c52 100644
--- a/pkg/operator/ceph/cluster/mgr/spec.go
+++ b/pkg/operator/ceph/cluster/mgr/spec.go
@@ -255,10 +255,8 @@ func (c *Cluster) makeCmdProxySidecarContainer(mgrConfig *mgrConfig) v1.Containe
 }
 
 // MakeMetricsService generates the Kubernetes service object for the monitoring service
-func (c *Cluster) MakeMetricsService(name, servicePortMetricName string) (*v1.Service, error) {
-
-	labels := controller.AppLabels(AppName, c.clusterInfo.Namespace)
-	selectorLabels := c.buildSelectorLabels(labels)
+func (c *Cluster) MakeMetricsService(name, activeDaemon, servicePortMetricName string) (*v1.Service, error) {
+	labels := c.selectorLabels(activeDaemon)
 
 	svc := &v1.Service{
 		ObjectMeta: metav1.ObjectMeta{
@@ -280,7 +278,7 @@ func (c *Cluster) MakeMetricsService(name, servicePortMetricName string) (*v1.Se
 
 	// If the cluster is external we don't need to add the selector
 	if name != controller.ExternalMgrAppName {
-		svc.Spec.Selector = selectorLabels
+		svc.Spec.Selector = labels
 	}
 
 	err := c.clusterInfo.OwnerInfo.SetControllerReference(svc)
@@ -290,10 +288,8 @@ func (c *Cluster) MakeMetricsService(name, servicePortMetricName string) (*v1.Se
 	return svc, nil
 }
 
-func (c *Cluster) makeDashboardService(name string) (*v1.Service, error) {
-
-	labels := controller.AppLabels(AppName, c.clusterInfo.Namespace)
-	selectorLabels := c.buildSelectorLabels(labels)
+func (c *Cluster) makeDashboardService(name, activeDaemon string) (*v1.Service, error) {
+	labels := c.selectorLabels(activeDaemon)
 
 	portName := "https-dashboard"
 	if !c.spec.Dashboard.SSL {
@@ -306,7 +302,7 @@ func (c *Cluster) makeDashboardService(name string) (*v1.Service, error) {
 			Labels:    labels,
 		},
 		Spec: v1.ServiceSpec{
-			Selector: selectorLabels,
+			Selector: labels,
 			Type:     v1.ServiceTypeClusterIP,
 			Ports: []v1.ServicePort{
 				{
@@ -356,11 +352,10 @@ func (c *Cluster) cephMgrOrchestratorModuleEnvs() []v1.EnvVar {
 	return envVars
 }
 
-func (c *Cluster) buildSelectorLabels(labels map[string]string) map[string]string {
-	selectorLabels := make(map[string]string)
-	for k, v := range labels {
-		selectorLabels[k] = v
+func (c *Cluster) selectorLabels(activeDaemon string) map[string]string {
+	labels := controller.AppLabels(AppName, c.clusterInfo.Namespace)
+	if activeDaemon != "" {
+		labels[controller.DaemonIDLabel] = activeDaemon
 	}
-	selectorLabels["mgr_role"] = "active"
-	return selectorLabels
+	return labels
 }
diff --git a/pkg/operator/ceph/cluster/mon/spec.go b/pkg/operator/ceph/cluster/mon/spec.go
index 7e7cb91f9..60c9c869f 100644
--- a/pkg/operator/ceph/cluster/mon/spec.go
+++ b/pkg/operator/ceph/cluster/mon/spec.go
@@ -271,6 +271,15 @@ func (c *Cluster) makeMonFSInitContainer(monConfig *monConfig) corev1.Container
 
 func (c *Cluster) makeMonDaemonContainer(monConfig *monConfig) corev1.Container {
 	podIPEnvVar := "ROOK_POD_IP"
+	publicAddr := monConfig.PublicIP
+
+	// Handle the non-default port for host networking. If host networking is not being used,
+	// the service created elsewhere will handle the non-default port redirection to the default port inside the container.
+	if monConfig.UseHostNetwork && monConfig.Port != DefaultMsgr1Port {
+		logger.Warningf("Starting mon %s with host networking on a non-default port %d. The mon must be failed over before enabling msgr2.",
+			monConfig.DaemonName, monConfig.Port)
+		publicAddr = fmt.Sprintf("%s:%d", publicAddr, monConfig.Port)
+	}
 
 	container := corev1.Container{
 		Name: "mon",
@@ -282,7 +291,7 @@ func (c *Cluster) makeMonDaemonContainer(monConfig *monConfig) corev1.Container
 			"--foreground",
 			// If the mon is already in the monmap, when the port is left off of --public-addr,
 			// it will still advertise on the previous port b/c monmap is saved to mon database.
-			config.NewFlag("public-addr", monConfig.PublicIP),
+			config.NewFlag("public-addr", publicAddr),
 			// Set '--setuser-match-path' so that existing directory owned by root won't affect the daemon startup.
 			// For existing data store owned by root, the daemon will continue to run as root
 			//
diff --git a/pkg/operator/ceph/cluster/nodedaemon/exporter.go b/pkg/operator/ceph/cluster/nodedaemon/exporter.go
index c039dc6d7..0b281042c 100644
--- a/pkg/operator/ceph/cluster/nodedaemon/exporter.go
+++ b/pkg/operator/ceph/cluster/nodedaemon/exporter.go
@@ -71,8 +71,12 @@ func (r *ReconcileNode) createOrUpdateCephExporter(node corev1.Node, tolerations
 		return controllerutil.OperationResultNone, errors.Errorf("failed to set owner reference of ceph-exporter deployment %q", deploy.Name)
 	}
 
+	configDir := path.Join(cephCluster.Spec.DataDirHostPath, cephCluster.Namespace)
+	configHostPathType := v1.HostPathDirectory
+	src := v1.VolumeSource{HostPath: &v1.HostPathVolumeSource{Path: configDir, Type: &configHostPathType}}
 	volumes := append(
 		controller.DaemonVolumesBase(config.NewDatalessDaemonDataPathMap(cephCluster.GetNamespace(), cephCluster.Spec.DataDirHostPath), "", cephCluster.Spec.DataDirHostPath),
+		v1.Volume{Name: "ceph-conf-dir", VolumeSource: src},
 		keyring.Volume().Admin())
 
 	mutateFunc := func() error {
@@ -138,7 +142,8 @@ func (r *ReconcileNode) createOrUpdateCephExporter(node corev1.Node, tolerations
 
 func getCephExporterChownInitContainer(cephCluster cephv1.CephCluster) corev1.Container {
 	dataPathMap := config.NewDatalessDaemonDataPathMap(cephCluster.GetNamespace(), cephCluster.Spec.DataDirHostPath)
-	mounts := controller.DaemonVolumeMounts(dataPathMap, "", cephCluster.Spec.DataDirHostPath)
+	configDir := path.Join(cephCluster.Spec.DataDirHostPath, cephCluster.Namespace)
+	mounts := append(controller.DaemonVolumeMounts(dataPathMap, "", cephCluster.Spec.DataDirHostPath), v1.VolumeMount{Name: "ceph-conf-dir", MountPath: configDir})
 
 	return controller.ChownCephDataDirsInitContainer(
 		*dataPathMap,
diff --git a/pkg/operator/ceph/cluster/osd/key_rotation.go b/pkg/operator/ceph/cluster/osd/key_rotation.go
index f5c2e4f31..b32edd82d 100644
--- a/pkg/operator/ceph/cluster/osd/key_rotation.go
+++ b/pkg/operator/ceph/cluster/osd/key_rotation.go
@@ -193,11 +193,6 @@ func (c *Cluster) makeKeyRotationCronJob(pvcName string, osd OSDInfo, osdProps o
 		return nil, err
 	}
 	c.applyResourcesToAllContainers(&podSpec.Spec, cephv1.GetOSDResources(c.spec.Resources, osd.DeviceClass))
-	schedule := c.spec.Security.KeyRotation.Schedule
-	if schedule == "" {
-		// default to rotate keyrings weekly (default is in code since default in crds causes issues)
-		schedule = "@weekly"
-	}
 	cronJob := &batch.CronJob{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:        keyRotationCronJobName(osd.ID),
@@ -207,7 +202,7 @@ func (c *Cluster) makeKeyRotationCronJob(pvcName string, osd OSDInfo, osdProps o
 		},
 		Spec: batch.CronJobSpec{
 			ConcurrencyPolicy: batch.ForbidConcurrent,
-			Schedule:          schedule,
+			Schedule:          c.spec.Security.KeyRotation.Schedule,
 			JobTemplate: batch.JobTemplateSpec{
 				Spec: batch.JobSpec{
 					Template: *podSpec,
diff --git a/pkg/operator/ceph/csi/betav1csidriver.go b/pkg/operator/ceph/csi/betav1csidriver.go
index 8ffd38539..cf7b487b8 100644
--- a/pkg/operator/ceph/csi/betav1csidriver.go
+++ b/pkg/operator/ceph/csi/betav1csidriver.go
@@ -73,9 +73,9 @@ func (d beta1CsiDriver) createCSIDriverInfo(ctx context.Context, clientset kuber
 		return err
 	}
 
-	// As FSGroupPolicy and AttachRequired fields are immutable, should be set only during create time.
-	// if the request is to change the FSGroupPolicy or AttachRequired, we are deleting the CSIDriver object and creating it.
-	if (driver.Spec.FSGroupPolicy != nil && csiDriver.Spec.FSGroupPolicy != nil && *driver.Spec.FSGroupPolicy != *csiDriver.Spec.FSGroupPolicy) || *driver.Spec.AttachRequired != *csiDriver.Spec.AttachRequired {
+	// As FSGroupPolicy field is immutable, should be set only during create time.
+	// if the request is to change the FSGroupPolicy, we are deleting the CSIDriver object and creating it.
+	if driver.Spec.FSGroupPolicy != nil && csiDriver.Spec.FSGroupPolicy != nil && *driver.Spec.FSGroupPolicy != *csiDriver.Spec.FSGroupPolicy {
 		d.csiClient = csidrivers
 		d.csiDriver = csiDriver
 		return d.reCreateCSIDriverInfo(ctx)
diff --git a/pkg/operator/ceph/csi/cluster_config.go b/pkg/operator/ceph/csi/cluster_config.go
index a9d62aa0e..043b094e6 100644
--- a/pkg/operator/ceph/csi/cluster_config.go
+++ b/pkg/operator/ceph/csi/cluster_config.go
@@ -108,10 +108,13 @@ func MonEndpoints(mons map[string]*cephclient.MonInfo, requireMsgr2 bool) []stri
 		endpoint := m.Endpoint
 		if requireMsgr2 {
 			logger.Debugf("evaluating mon %q for msgr1 on endpoint %q", m.Name, m.Endpoint)
-			msgr1Suffix := fmt.Sprintf(":%d", client.Msgr1port)
-			if strings.HasSuffix(m.Endpoint, msgr1Suffix) {
-				address := m.Endpoint[0:strings.LastIndex(m.Endpoint, msgr1Suffix)]
-				endpoint = fmt.Sprintf("%s:%d", address, client.Msgr2port)
+			if strings.HasSuffix(m.Endpoint, fmt.Sprintf(":%d", client.Msgr1port)) {
+				parts := strings.Split(m.Endpoint, ":")
+				if len(parts) != 2 {
+					logger.Errorf("endpoint %q does not contain two parts to extract the port", m.Endpoint)
+					continue
+				}
+				endpoint = fmt.Sprintf("%s:%d", parts[0], client.Msgr2port)
 				logger.Debugf("mon %q will use the msgrv2 port: %q", m.Name, endpoint)
 			}
 		}
diff --git a/pkg/operator/ceph/csi/csi.go b/pkg/operator/ceph/csi/csi.go
index 0ced8b3f4..2e441037c 100644
--- a/pkg/operator/ceph/csi/csi.go
+++ b/pkg/operator/ceph/csi/csi.go
@@ -283,17 +283,5 @@ func (r *ReconcileCSI) setParams(ver *version.Info) error {
 	CSIParam.ImagePullPolicy = k8sutil.GetValue(r.opConfig.Parameters, "ROOK_CSI_IMAGE_PULL_POLICY", DefaultCSIImagePullPolicy)
 	CSIParam.CephFSKernelMountOptions = k8sutil.GetValue(r.opConfig.Parameters, "CSI_CEPHFS_KERNEL_MOUNT_OPTIONS", "")
 
-	CSIParam.CephFSAttachRequired = true
-	if strings.EqualFold(k8sutil.GetValue(r.opConfig.Parameters, "CSI_CEPHFS_ATTACH_REQUIRED", "true"), "false") {
-		CSIParam.CephFSAttachRequired = false
-	}
-	CSIParam.RBDAttachRequired = true
-	if strings.EqualFold(k8sutil.GetValue(r.opConfig.Parameters, "CSI_RBD_ATTACH_REQUIRED", "true"), "false") {
-		CSIParam.RBDAttachRequired = false
-	}
-	CSIParam.NFSAttachRequired = true
-	if strings.EqualFold(k8sutil.GetValue(r.opConfig.Parameters, "CSI_NFS_ATTACH_REQUIRED", "true"), "false") {
-		CSIParam.NFSAttachRequired = false
-	}
 	return nil
 }
diff --git a/pkg/operator/ceph/csi/csidriver.go b/pkg/operator/ceph/csi/csidriver.go
index 1159fd098..19d47408d 100644
--- a/pkg/operator/ceph/csi/csidriver.go
+++ b/pkg/operator/ceph/csi/csidriver.go
@@ -68,9 +68,9 @@ func (d v1CsiDriver) createCSIDriverInfo(ctx context.Context, clientset kubernet
 		return err
 	}
 
-	// As FSGroupPolicy and AttachRequired fields are immutable, should be set only during create time.
-	// if the request is to change the FSGroupPolicy or AttachRequired, we are deleting the CSIDriver object and creating it.
-	if (driver.Spec.FSGroupPolicy != nil && csiDriver.Spec.FSGroupPolicy != nil && *driver.Spec.FSGroupPolicy != *csiDriver.Spec.FSGroupPolicy) || *driver.Spec.AttachRequired != *csiDriver.Spec.AttachRequired {
+	// As FSGroupPolicy field is immutable, should be set only during create time.
+	// if the request is to change the FSGroupPolicy, we are deleting the CSIDriver object and creating it.
+	if driver.Spec.FSGroupPolicy != nil && csiDriver.Spec.FSGroupPolicy != nil && *driver.Spec.FSGroupPolicy != *csiDriver.Spec.FSGroupPolicy {
 		d.csiClient = csidrivers
 		d.csiDriver = csiDriver
 		return d.reCreateCSIDriverInfo(ctx)
diff --git a/pkg/operator/ceph/csi/spec.go b/pkg/operator/ceph/csi/spec.go
index 1886373fb..cbe26cecd 100644
--- a/pkg/operator/ceph/csi/spec.go
+++ b/pkg/operator/ceph/csi/spec.go
@@ -78,9 +78,6 @@ type Param struct {
 	EnableCSITopology                     bool
 	EnableLiveness                        bool
 	EnableReadAffinity                    bool
-	CephFSAttachRequired                  bool
-	RBDAttachRequired                     bool
-	NFSAttachRequired                     bool
 	LogLevel                              uint8
 	SidecarLogLevel                       uint8
 	CephFSGRPCMetricsPort                 uint16
@@ -647,19 +644,19 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 	}
 
 	if EnableRBD {
-		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, RBDDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_RBD_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), tp.Param.RBDAttachRequired)
+		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, RBDDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_RBD_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), true)
 		if err != nil {
 			return errors.Wrapf(err, "failed to create CSI driver object for %q", RBDDriverName)
 		}
 	}
 	if EnableCephFS {
-		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, CephFSDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_CEPHFS_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), tp.Param.CephFSAttachRequired)
+		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, CephFSDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_CEPHFS_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), true)
 		if err != nil {
 			return errors.Wrapf(err, "failed to create CSI driver object for %q", CephFSDriverName)
 		}
 	}
 	if EnableNFS {
-		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, NFSDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_NFS_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), tp.Param.NFSAttachRequired)
+		err = csiDriverobj.createCSIDriverInfo(r.opManagerContext, r.context.Clientset, NFSDriverName, k8sutil.GetValue(r.opConfig.Parameters, "CSI_NFS_FSGROUPPOLICY", string(k8scsi.FileFSGroupPolicy)), false)
 		if err != nil {
 			return errors.Wrapf(err, "failed to create CSI driver object for %q", NFSDriverName)
 		}
diff --git a/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml b/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml
index f2ba703df..f118b5797 100644
--- a/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml
+++ b/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml
@@ -22,7 +22,6 @@ spec:
       priorityClassName: {{ .ProvisionerPriorityClassName }}
       {{ end }}
       containers:
-        {{ if .CephFSAttachRequired }}
         - name: csi-attacher
           image: {{ .AttacherImage }}
           args:
@@ -34,6 +33,7 @@ spec:
             - "--leader-election-lease-duration=137s"
             - "--leader-election-renew-deadline=107s"
             - "--leader-election-retry-period=26s"
+            - "--default-fstype=ext4"
           env:
             - name: ADDRESS
               value: /csi/csi-provisioner.sock
@@ -41,7 +41,6 @@ spec:
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
-        {{ end }}
         {{ if .EnableCephFSSnapshotter }}
         - name: csi-snapshotter
           image:  {{ .SnapshotterImage }}
diff --git a/pkg/operator/ceph/csi/template/nfs/csi-nfsplugin-provisioner-dep.yaml b/pkg/operator/ceph/csi/template/nfs/csi-nfsplugin-provisioner-dep.yaml
index b8aeebda9..7cf5961b8 100644
--- a/pkg/operator/ceph/csi/template/nfs/csi-nfsplugin-provisioner-dep.yaml
+++ b/pkg/operator/ceph/csi/template/nfs/csi-nfsplugin-provisioner-dep.yaml
@@ -21,26 +21,6 @@ spec:
       priorityClassName: {{ .ProvisionerPriorityClassName }}
       {{ end }}
       containers:
-        {{ if .NFSAttachRequired }}
-        - name: csi-attacher
-          image: {{ .AttacherImage }}
-          args:
-            - "--v={{ .SidecarLogLevel }}"
-            - "--csi-address=$(ADDRESS)"
-            - "--leader-election=true"
-            - "--timeout={{ .GRPCTimeout }}"
-            - "--leader-election-namespace={{ .Namespace }}"
-            - "--leader-election-lease-duration=137s"
-            - "--leader-election-renew-deadline=107s"
-            - "--leader-election-retry-period=26s"
-          env:
-            - name: ADDRESS
-              value: /csi/csi-provisioner.sock
-          imagePullPolicy: {{ .ImagePullPolicy }}
-          volumeMounts:
-            - name: socket-dir
-              mountPath: /csi
-        {{ end }}
         {{ if .EnableNFSSnapshotter }}
         - name: csi-snapshotter
           image:  {{ .SnapshotterImage }}
diff --git a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
index 3d453c7e1..17fc28ba2 100644
--- a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
+++ b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
@@ -65,7 +65,6 @@ spec:
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
-        {{ if .RBDAttachRequired }}
         - name: csi-attacher
           image: {{ .AttacherImage }}
           args:
@@ -85,7 +84,6 @@ spec:
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
-        {{ end }}
         {{ if .EnableRBDSnapshotter }}
         - name: csi-snapshotter
           image:  {{ .SnapshotterImage }}


Test patch:
diff --git a/.github/workflows/canary-integration-test.yml b/.github/workflows/canary-integration-test.yml
index c5b7da216..3fddb13cd 100644
--- a/.github/workflows/canary-integration-test.yml
+++ b/.github/workflows/canary-integration-test.yml
@@ -167,8 +167,7 @@ jobs:
           timeout 15 sh -c "until kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rgw-endpoint $rgw_endpoint; do sleep 1 && echo 'waiting for the rgw endpoint to be validated'; done"
           # pass the invalid rgw-endpoint of different ceph cluster
           if output=$(timeout 15 sh -c "until kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rgw-endpoint 10.108.96.128:80; do sleep 1 && echo 'waiting for the rgw endpoint to be validated'; done"); then
-            echo "unexpectedly succeeded after passing the wrong rgw-endpoint: $output"
-            exit 1
+            echo "script run completed with stderr error after passing the wrong rgw-endpoint: $output"
           else
             echo "validation failed because wrong endpoint was provided"
           fi
@@ -1182,6 +1181,7 @@ jobs:
           cp deploy/examples/filesystem-test.yaml "$PRIMARY_YAML"
           yq merge --inplace --arrays append "$PRIMARY_YAML" tests/manifests/test-fs-mirror-spec.yaml
           kubectl create -f "$PRIMARY_YAML"
+          timeout 200 sh -c 'until [ "$(kubectl -n rook-ceph get cephfilesystem myfs -o jsonpath='{.status.phase}'|grep -c "Ready")" -eq 1 ]; do echo "waiting for filesystem to be created" && sleep 1; done'
 
       - name: create replicated mirrored filesystem on cluster 2
         run: |
@@ -1189,14 +1189,7 @@ jobs:
           yq w -i filesystem-test.yaml metadata.namespace rook-ceph-secondary
           yq w -i filesystem-test.yaml spec.mirroring.enabled true
           kubectl create -f filesystem-test.yaml
-
-      - name: wait for filesystem on cluster 1
-        run: |
-          timeout 300 sh -c 'until [ "$(kubectl -n rook-ceph get cephfilesystem myfs -o jsonpath='{.status.phase}'|grep -c "Ready")" -eq 1 ]; do echo "waiting for filesystem to be created" && sleep 1; done'
-
-      - name: wait for filesystem on cluster 2
-        run: |
-          timeout 300 sh -c 'until [ "$(kubectl -n rook-ceph-secondary get cephfilesystem myfs -o jsonpath='{.status.phase}'|grep -c "Ready")" -eq 1 ]; do echo "waiting for filesystem to be created" && sleep 1; done'
+          timeout 120 sh -c 'until [ "$(kubectl -n rook-ceph-secondary get cephfilesystem myfs -o jsonpath='{.status.phase}'|grep -c "Ready")" -eq 1 ]; do echo "waiting for filesystem to be created" && sleep 1; done'
 
       - name: copy filesystem mirror peer secret from the secondary cluster to the primary one
         run: |
@@ -1232,12 +1225,12 @@ jobs:
 
       - name: Deploy Direct Tools pod on primary cluster
         run: |
-          tests/scripts/github-action-helper.sh deploy_manifest_with_local_build deploy/examples/direct-mount.yaml
+          kubectl -n rook-ceph create -f deploy/examples/direct-mount.yaml
 
       - name: Deploy Direct Tools pod on secondary cluster
         run: |
           sed -i "s/rook-ceph # namespace/rook-ceph-secondary # namespace/" deploy/examples/direct-mount.yaml
-          tests/scripts/github-action-helper.sh deploy_manifest_with_local_build deploy/examples/direct-mount.yaml
+          kubectl -n rook-ceph-secondary create -f deploy/examples/direct-mount.yaml
 
       - name: Configure a directory for snapshot mirroring on primary cluster
         run: |
@@ -1266,17 +1259,17 @@ jobs:
           EOF
           clusterfsid=$($exec_fs_mirror ceph --admin-daemon /var/run/ceph/$mirror_daemon fs mirror status myfs@1 |jq -r '.peers|keys[]')
           echo $clusterfsid
+          snaps=$(kubectl -n rook-ceph exec deploy/rook-ceph-fs-mirror -- ceph --admin-daemon /var/run/ceph/$mirror_daemon fs mirror peer status myfs@1 $clusterfsid|jq -r '."/volumes/_nogroup/testsubvolume"."snaps_synced"')
+          echo $snaps
           kubectl -n rook-ceph-secondary wait pod -l app=rook-direct-mount --for condition=Ready --timeout=300s
           kubectl -n rook-ceph-secondary exec deploy/rook-direct-mount -- mkdir /tmp/registry
           mon_endpoints=$(kubectl -n rook-ceph-secondary exec deploy/rook-direct-mount -- grep mon_host /etc/ceph/ceph.conf | awk '{print $3}')
           my_secret=$(kubectl -n rook-ceph-secondary exec deploy/rook-direct-mount -- grep key /etc/ceph/keyring | awk '{print $3}')
           kubectl -n rook-ceph-secondary exec deploy/rook-direct-mount -- mount -t ceph -o mds_namespace=myfs,name=admin,secret=$my_secret $mon_endpoints:/ /tmp/registry
           num_snaps_target=$(kubectl -n rook-ceph-secondary exec deploy/rook-direct-mount -- ls -lhsa /tmp/registry/volumes/_nogroup/testsubvolume/.snap|grep snap|wc -l)
-          snaps=$(kubectl -n rook-ceph exec deploy/rook-ceph-fs-mirror -- ceph --admin-daemon /var/run/ceph/$mirror_daemon fs mirror peer status myfs@1 $clusterfsid|jq -r '."/volumes/_nogroup/testsubvolume"."snaps_synced"')
-          echo "snapshots: $snaps"
           if [ $num_snaps_target = $snaps ]
           then echo "Snaphots have synced."
-          else echo "Snaps have not synced. NEEDS INVESTIGATION"
+          else echo "Snaps have not synced" ; exit 1
           fi
 
       - name: collect common logs
diff --git a/pkg/daemon/ceph/osd/device_test.go b/pkg/daemon/ceph/osd/device_test.go
index bf445904d..312d7245e 100644
--- a/pkg/daemon/ceph/osd/device_test.go
+++ b/pkg/daemon/ceph/osd/device_test.go
@@ -23,9 +23,7 @@ import (
 
 	"github.com/rook/rook/pkg/clusterd"
 	"github.com/rook/rook/pkg/daemon/ceph/client"
-	"github.com/rook/rook/pkg/operator/ceph/cluster/osd"
 	exectest "github.com/rook/rook/pkg/util/exec/test"
-	"github.com/rook/rook/pkg/util/sys"
 	"github.com/stretchr/testify/assert"
 )
 
@@ -49,36 +47,3 @@ func TestOSDBootstrap(t *testing.T) {
 	assert.NotEqual(t, -1, strings.Index(string(contents), "key = mysecurekey"))
 	assert.NotEqual(t, -1, strings.Index(string(contents), "caps mon = \"allow profile bootstrap-osd\""))
 }
-
-func TestUpdateDeviceClass(t *testing.T) {
-	d := &DesiredDevice{}
-	agent := &OsdAgent{}
-	disk := &sys.LocalDisk{}
-
-	d.DeviceClass = "test"
-	d.UpdateDeviceClass(agent, disk)
-	assert.Equal(t, "test", d.DeviceClass)
-
-	d.DeviceClass = ""
-	agent.pvcBacked = true
-	t.Setenv(osd.CrushDeviceClassVarName, "test")
-	d.UpdateDeviceClass(agent, disk)
-	assert.Equal(t, "test", d.DeviceClass)
-
-	d.DeviceClass = ""
-	t.Setenv(osd.CrushDeviceClassVarName, "")
-	d.UpdateDeviceClass(agent, disk)
-	t.Log(d)
-	t.Log(disk)
-	assert.Equal(t, "ssd", d.DeviceClass)
-
-	d.DeviceClass = ""
-	agent.pvcBacked = false
-	d.UpdateDeviceClass(agent, disk)
-	assert.Equal(t, "ssd", d.DeviceClass)
-
-	d.DeviceClass = ""
-	agent.storeConfig.DeviceClass = "test"
-	d.UpdateDeviceClass(agent, disk)
-	assert.Equal(t, "test", d.DeviceClass)
-}
diff --git a/pkg/operator/ceph/cluster/mgr/dashboard_test.go b/pkg/operator/ceph/cluster/mgr/dashboard_test.go
index 8a568758f..965be9a6e 100644
--- a/pkg/operator/ceph/cluster/mgr/dashboard_test.go
+++ b/pkg/operator/ceph/cluster/mgr/dashboard_test.go
@@ -131,7 +131,7 @@ func TestStartSecureDashboard(t *testing.T) {
 	}
 
 	dashboardInitWaitTime = 0
-	err := c.configureDashboardService()
+	err := c.configureDashboardService("a")
 	assert.NoError(t, err)
 	err = c.configureDashboardModules()
 	assert.NoError(t, err)
@@ -148,7 +148,7 @@ func TestStartSecureDashboard(t *testing.T) {
 
 	// disable the dashboard
 	c.spec.Dashboard.Enabled = false
-	err = c.configureDashboardService()
+	err = c.configureDashboardService("a")
 	assert.Nil(t, err)
 	err = c.configureDashboardModules()
 	assert.NoError(t, err)
@@ -163,7 +163,7 @@ func TestStartSecureDashboard(t *testing.T) {
 	// Set the port to something over 1024 and confirm the port and targetPort are the same
 	c.spec.Dashboard.Enabled = true
 	c.spec.Dashboard.Port = 1025
-	err = c.configureDashboardService()
+	err = c.configureDashboardService("a")
 	assert.Nil(t, err)
 
 	svc, err = c.context.Clientset.CoreV1().Services(clusterInfo.Namespace).Get(ctx, "rook-ceph-mgr-dashboard", metav1.GetOptions{})
@@ -175,7 +175,7 @@ func TestStartSecureDashboard(t *testing.T) {
 	// Fall back to the default port
 	c.spec.Dashboard.Enabled = true
 	c.spec.Dashboard.Port = 0
-	err = c.configureDashboardService()
+	err = c.configureDashboardService("a")
 	assert.Nil(t, err)
 
 	svc, err = c.context.Clientset.CoreV1().Services(clusterInfo.Namespace).Get(ctx, "rook-ceph-mgr-dashboard", metav1.GetOptions{})
diff --git a/pkg/operator/ceph/cluster/mgr/mgr_test.go b/pkg/operator/ceph/cluster/mgr/mgr_test.go
index 2e5401c3c..ad050cf41 100644
--- a/pkg/operator/ceph/cluster/mgr/mgr_test.go
+++ b/pkg/operator/ceph/cluster/mgr/mgr_test.go
@@ -31,7 +31,6 @@ import (
 	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
 	"github.com/rook/rook/pkg/operator/ceph/controller"
 	cephver "github.com/rook/rook/pkg/operator/ceph/version"
-	"github.com/rook/rook/pkg/operator/k8sutil"
 	testopk8s "github.com/rook/rook/pkg/operator/k8sutil/test"
 	testop "github.com/rook/rook/pkg/operator/test"
 	exectest "github.com/rook/rook/pkg/util/exec/test"
@@ -46,7 +45,10 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
 )
 
-func createNewCluster(t *testing.T) *Cluster {
+func TestStartMgr(t *testing.T) {
+	var deploymentsUpdated *[]*apps.Deployment
+	updateDeploymentAndWait, deploymentsUpdated = testopk8s.UpdateDeploymentAndWaitStub()
+
 	executor := &exectest.MockExecutor{
 		MockExecuteCommandWithOutput: func(command string, args ...string) (string, error) {
 			logger.Infof("Execute: %s %v", command, args)
@@ -90,17 +92,8 @@ func createNewCluster(t *testing.T) *Cluster {
 	c := New(ctx, clusterInfo, clusterSpec, "myversion")
 	defer os.RemoveAll(c.spec.DataDirHostPath)
 
-	return c
-}
-
-func TestStartMgr(t *testing.T) {
-	var deploymentsUpdated *[]*apps.Deployment
-	updateDeploymentAndWait, deploymentsUpdated = testopk8s.UpdateDeploymentAndWaitStub()
-
-	c := createNewCluster(t)
-
 	// start a basic service
-	err := c.Start()
+	err = c.Start()
 	assert.NoError(t, err)
 	validateStart(t, c)
 	assert.ElementsMatch(t, []string{}, testopk8s.DeploymentNamesUpdated(deploymentsUpdated))
@@ -183,109 +176,128 @@ func validateServices(t *testing.T, c *Cluster) {
 	}
 }
 
-func TestActiveMgrLabels(t *testing.T) {
-	var deploymentsUpdated *[]*apps.Deployment
-	updateDeploymentAndWait, deploymentsUpdated = testopk8s.UpdateDeploymentAndWaitStub()
-
-	c := createNewCluster(t)
-	err := c.Start()
-	assert.NoError(t, err)
-	validateStart(t, c)
+func TestUpdateServiceSelectors(t *testing.T) {
+	clientset := testop.New(t, 3)
+	ctx := &clusterd.Context{Clientset: clientset}
+	clusterInfo := cephclient.AdminTestClusterInfo("mycluster")
+	spec := cephv1.ClusterSpec{
+		Dashboard: cephv1.DashboardSpec{
+			Enabled: true,
+			Port:    7000,
+		},
+	}
+	c := &Cluster{spec: spec, context: ctx, clusterInfo: clusterInfo}
 
-	mgrNames := []string{"a", "b"}
-	for i := 0; i < c.spec.Mgr.Count; i++ {
-		// Simulate the Mgr pod having been created
-		daemonName := mgrNames[i]
-		mgrPod := &corev1.Pod{ObjectMeta: metav1.ObjectMeta{
-			Name: fmt.Sprintf("rook-ceph-mgr-%s", daemonName),
-			Labels: map[string]string{k8sutil.AppAttr: "rook-ceph-mgr",
-				"instance":       daemonName,
-				"ceph_daemon_id": daemonName,
-				"mgr":            daemonName,
-				"rook_cluster":   "ns"}}}
-		_, err = c.context.Clientset.CoreV1().Pods(c.clusterInfo.Namespace).Create(c.clusterInfo.Context, mgrPod, metav1.CreateOptions{})
+	t.Run("initial active daemon", func(t *testing.T) {
+		activeDaemon := "a"
+		err := c.reconcileServices(activeDaemon)
 		assert.NoError(t, err)
-	}
+		validateServiceActiveDaemon(t, c, activeDaemon, 2, 0)
+	})
 
-	for i := 0; i < c.spec.Mgr.Count; i++ {
-		daemonName := mgrNames[i]
-		_, err := c.context.Clientset.AppsV1().Deployments(c.clusterInfo.Namespace).Get(context.TODO(), fmt.Sprintf("rook-ceph-mgr-%s", daemonName), metav1.GetOptions{})
+	t.Run("update active daemon", func(t *testing.T) {
+		activeDaemon := "b"
+		err := c.updateServiceSelectors(activeDaemon)
 		assert.NoError(t, err)
-		_, err = c.UpdateActiveMgrLabel(daemonName, "")
+		validateServiceActiveDaemon(t, c, activeDaemon, 2, 0)
+	})
+
+	t.Run("skip non-mgr services", func(t *testing.T) {
+		svc := corev1.Service{ObjectMeta: metav1.ObjectMeta{Name: "mysvc"}}
+		_, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Create(clusterInfo.Context, &svc, metav1.CreateOptions{})
 		assert.NoError(t, err)
-		pods, err := c.context.Clientset.CoreV1().Pods(c.clusterInfo.Namespace).List(c.clusterInfo.Context, metav1.ListOptions{LabelSelector: "app=rook-ceph-mgr"})
+
+		activeDaemon := "c"
+		err = c.updateServiceSelectors(activeDaemon)
 		assert.NoError(t, err)
-		assert.Contains(t, pods.Items[i].Labels, "mgr_role")
-		if daemonName == "a" {
-			assert.Equal(t, pods.Items[i].Labels["mgr_role"], "active")
-		} else if daemonName == "b" {
-			assert.Equal(t, pods.Items[i].Labels["mgr_role"], "standby")
+		validateServiceActiveDaemon(t, c, activeDaemon, 2, 1)
+	})
+}
+
+func validateServiceActiveDaemon(t *testing.T, c *Cluster, activeDaemon string, expectedUpdated, expectedSkipped int) {
+	services, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).List(c.clusterInfo.Context, metav1.ListOptions{})
+	assert.NoError(t, err)
+	skipped := 0
+	updated := 0
+	for _, service := range services.Items {
+		if service.Labels["app"] == "rook-ceph-mgr" {
+			updated++
+			assert.Equal(t, activeDaemon, service.Spec.Selector[controller.DaemonIDLabel])
+		} else {
+			skipped++
+			assert.Equal(t, "", service.Spec.Selector[controller.DaemonIDLabel])
 		}
 	}
-
-	testopk8s.ClearDeploymentsUpdated(deploymentsUpdated)
+	assert.Equal(t, expectedUpdated, updated)
+	assert.Equal(t, expectedSkipped, skipped)
 }
 
-func TestUpdateServiceSelectors(t *testing.T) {
-	clientset := testop.New(t, 3)
-	ctx := &clusterd.Context{Clientset: clientset}
-	clusterInfo := cephclient.AdminTestClusterInfo("mycluster")
+func TestMgrSidecarReconcile(t *testing.T) {
+	activeMgr := "a"
+	calledMgrStat := false
 	spec := cephv1.ClusterSpec{
+		Mgr: cephv1.MgrSpec{Count: 1},
 		Dashboard: cephv1.DashboardSpec{
 			Enabled: true,
 			Port:    7000,
 		},
 	}
+	executor := &exectest.MockExecutor{
+		MockExecuteCommandWithOutput: func(command string, args ...string) (string, error) {
+			logger.Infof("Command: %s %v", command, args)
+			if args[1] == "stat" {
+				calledMgrStat = true
+			}
+			return fmt.Sprintf(`{"active_name":"%s"}`, activeMgr), nil
+		},
+	}
+	clientset := testop.New(t, 3)
+	configDir := t.TempDir()
+	ctx := &clusterd.Context{
+		Executor:  executor,
+		ConfigDir: configDir,
+		Clientset: clientset,
+	}
+	clusterInfo := cephclient.AdminTestClusterInfo("mycluster")
+	clusterInfo.SetName("test")
 	c := &Cluster{spec: spec, context: ctx, clusterInfo: clusterInfo}
 
-	// Make sure we remove the daemon_id label from the selector
-	// of all services with a label "app=rook-ceph-mgr"
-	t.Run("remove daemon_id from mgr services", func(t *testing.T) {
-		svc1 := corev1.Service{
-			ObjectMeta: metav1.ObjectMeta{
-				Name: "svc1",
-				Labels: map[string]string{
-					"app":                    "rook-ceph-mgr",
-					"svc":                    "rook-ceph-mgr",
-					controller.DaemonIDLabel: "a",
-				},
-			},
-			Spec: corev1.ServiceSpec{
-				Selector: map[string]string{
-					"app":                    "rook-ceph-mgr",
-					controller.DaemonIDLabel: "a",
-				}},
-		}
-		svc2 := corev1.Service{
-			ObjectMeta: metav1.ObjectMeta{Name: "svc2"},
-			Spec: corev1.ServiceSpec{
-				Selector: map[string]string{
-					"app":                    "rook-ceph-mgr",
-					controller.DaemonIDLabel: "a",
-				}},
-		}
-		_, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Create(clusterInfo.Context, &svc1, metav1.CreateOptions{})
-		assert.NoError(t, err)
-		_, err2 := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Create(clusterInfo.Context, &svc2, metav1.CreateOptions{})
-		assert.NoError(t, err2)
+	// Update services according to the active mgr
+	calledMgrStat = false
+	clusterInfo.CephVersion = cephver.CephVersion{Major: 17, Minor: 2, Build: 0}
+	err := c.ReconcileActiveMgrServices(activeMgr)
+	assert.NoError(t, err)
+	assert.True(t, calledMgrStat)
+	validateServices(t, c)
+	validateServiceMatches(t, c, "a")
 
-		// Check the the label has been only removed from svc1
-		c.updateServiceSelectors()
+	// nothing is created or updated when the requested mgr is not the active mgr
+	clusterInfo.CephVersion = cephver.CephVersion{Major: 16, Minor: 2, Build: 5}
+	err = c.ReconcileActiveMgrServices("b")
+	assert.NoError(t, err)
+	assert.True(t, calledMgrStat)
+	_, err = c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Get(context.TODO(), "rook-ceph-mgr", metav1.GetOptions{})
+	assert.True(t, kerrors.IsNotFound(err))
 
-		// Make sure we remove controller.DaemonIDLabel label from all services tagged as 'app: rook-ceph-mgr'
-		// and we add the selector label "mgr_role" set to "active"
-		updatedService1, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Get(clusterInfo.Context, "svc1", metav1.GetOptions{})
-		assert.NoError(t, err)
-		assert.NotContains(t, updatedService1.Spec.Selector, controller.DaemonIDLabel)
-		assert.Contains(t, updatedService1.Spec.Selector, "mgr_role")
-		assert.Equal(t, updatedService1.Spec.Selector["mgr_role"], "active")
+	// nothing is updated when the requested mgr is not the active mgr
+	activeMgr = "b"
+	err = c.ReconcileActiveMgrServices("b")
+	assert.NoError(t, err)
+	validateServices(t, c)
+	validateServiceMatches(t, c, "b")
+}
 
-		// Make sure services not tagged as 'app: rook-ceph-mgr' are not modified
-		updatedService2, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Get(clusterInfo.Context, "svc2", metav1.GetOptions{})
-		assert.NoError(t, err)
-		assert.Contains(t, updatedService2.Spec.Selector, controller.DaemonIDLabel)
-		assert.NotContains(t, updatedService2.Spec.Selector, "mgr_role")
-	})
+func validateServiceMatches(t *testing.T, c *Cluster, expectedActive string) {
+	// The service labels should match the active mgr
+	svc, err := c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Get(context.TODO(), "rook-ceph-mgr", metav1.GetOptions{})
+	assert.NoError(t, err)
+	matchDaemon, ok := svc.Spec.Selector["ceph_daemon_id"]
+	assert.True(t, ok)
+	assert.Equal(t, expectedActive, matchDaemon)
+
+	// clean up the service for the next test
+	err = c.context.Clientset.CoreV1().Services(c.clusterInfo.Namespace).Delete(context.TODO(), "rook-ceph-mgr", metav1.DeleteOptions{})
+	assert.NoError(t, err)
 }
 
 func TestConfigureModules(t *testing.T) {
diff --git a/pkg/operator/ceph/cluster/mgr/spec_test.go b/pkg/operator/ceph/cluster/mgr/spec_test.go
index 84e5ab6a7..612e7f639 100644
--- a/pkg/operator/ceph/cluster/mgr/spec_test.go
+++ b/pkg/operator/ceph/cluster/mgr/spec_test.go
@@ -23,6 +23,7 @@ import (
 	"github.com/rook/rook/pkg/clusterd"
 	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
 	"github.com/rook/rook/pkg/operator/ceph/config"
+	"github.com/rook/rook/pkg/operator/ceph/controller"
 	"github.com/rook/rook/pkg/operator/ceph/test"
 	optest "github.com/rook/rook/pkg/operator/test"
 	"github.com/stretchr/testify/assert"
@@ -112,17 +113,14 @@ func TestServiceSpec(t *testing.T) {
 	clusterSpec := cephv1.ClusterSpec{}
 	c := New(&clusterd.Context{Clientset: clientset}, clusterInfo, clusterSpec, "myversion")
 
-	s, err := c.MakeMetricsService("rook-mgr", serviceMetricName)
+	s, err := c.MakeMetricsService("rook-mgr", "foo", serviceMetricName)
 	assert.NoError(t, err)
 	assert.NotNil(t, s)
 	assert.Equal(t, "rook-mgr", s.Name)
 	assert.Equal(t, 1, len(s.Spec.Ports))
-	assert.Equal(t, 2, len(s.Labels))
-	// Check service selector
+	assert.Equal(t, 3, len(s.Labels))
 	assert.Equal(t, 3, len(s.Spec.Selector))
-	assert.Equal(t, s.Spec.Selector["app"], "rook-ceph-mgr")
-	assert.Equal(t, s.Spec.Selector["mgr_role"], "active")
-	assert.Equal(t, s.Spec.Selector["rook_cluster"], "ns")
+	assert.Equal(t, "foo", s.Spec.Selector[controller.DaemonIDLabel])
 }
 
 func TestHostNetwork(t *testing.T) {
diff --git a/pkg/operator/ceph/cluster/mon/node_test.go b/pkg/operator/ceph/cluster/mon/node_test.go
index 67549859d..352aba712 100644
--- a/pkg/operator/ceph/cluster/mon/node_test.go
+++ b/pkg/operator/ceph/cluster/mon/node_test.go
@@ -185,6 +185,13 @@ func TestHostNetwork(t *testing.T) {
 	assert.Equal(t, "", val)
 	assert.Equal(t, "arg not found: --public-bind-addr", message)
 
+	monConfig.Port = 6790
+	pod, err = c.makeMonPod(monConfig, false)
+	assert.NoError(t, err)
+	val, message = extractArgValue(pod.Spec.Containers[0].Args, "--public-addr")
+	assert.Equal(t, "2.4.6.3:6790", val, message)
+	assert.NotNil(t, pod)
+
 	// Host network setting of mons should be maintained even if the cluster spec hostnetwork is different
 	// from the mons to not be using host networking
 	monConfig.UseHostNetwork = false
diff --git a/pkg/operator/ceph/csi/cluster_config_test.go b/pkg/operator/ceph/csi/cluster_config_test.go
index 5f778a16d..dc64ed60f 100644
--- a/pkg/operator/ceph/csi/cluster_config_test.go
+++ b/pkg/operator/ceph/csi/cluster_config_test.go
@@ -369,20 +369,15 @@ func TestMonEndpoints(t *testing.T) {
 		verifyEndpointPort(t, endpoints, "3300")
 	})
 
-	t.Run("ipv6 endpoint conversion", func(t *testing.T) {
+	t.Run("bad endpoint", func(t *testing.T) {
 		monInfo := map[string]*cephclient.MonInfo{
-			"a": {Name: "a", Endpoint: "[fd07:aaaa:bbbb:cccc::11]:6789"},
-			"b": {Name: "a", Endpoint: "[1234:6789:bbbb:cccc::11]:6789"},
+			"a": {Name: "a", Endpoint: "1.2.3.4:5:6789"},
+			"b": {Name: "b", Endpoint: "1.2.3.5:6789"},
+			"c": {Name: "c", Endpoint: "1.2.3.6:6789"},
 		}
 		endpoints := MonEndpoints(monInfo, true)
 		assert.Equal(t, 2, len(endpoints))
 		verifyEndpointPort(t, endpoints, "3300")
-		for _, endpoint := range endpoints {
-			// Verify that the v1 port inside the ipv6 address will not be replaced
-			if strings.HasPrefix(endpoint, "[1234") {
-				assert.True(t, strings.HasPrefix(endpoint, "[1234:6789"))
-			}
-		}
 	})
 }
 
diff --git a/pkg/util/sys/device_test.go b/pkg/util/sys/device_test.go
index 01023c0ff..10342f857 100644
--- a/pkg/util/sys/device_test.go
+++ b/pkg/util/sys/device_test.go
@@ -196,13 +196,3 @@ func TestListDevicesChildListDevicesChild(t *testing.T) {
 	assert.NoError(t, err)
 	assert.Equal(t, 3, len(child))
 }
-
-func TestGetDiskDeviceClass(t *testing.T) {
-	d := &LocalDisk{}
-	assert.Equal(t, "ssd", GetDiskDeviceClass(d))
-	d.Rotational = true
-	assert.Equal(t, "hdd", GetDiskDeviceClass(d))
-	d.Rotational = false
-	d.RealPath = "nvme"
-	assert.Equal(t, "nvme", GetDiskDeviceClass(d))
-}