Repository: rook/rook
Pull request #10405 (2022-06-07 09:37:43)

mon: update node labels as per new key value
From kube v1.24 onwards, Kubeadm applies a "node-role" label to its
control plane Nodes.Currently this label key is node-role.kubernetes.io/master
and it will be renamed to node-role.kubernetes.io/control-plane. Kubeadm
also uses the same "node-role" as key for a taint it applies on control
plane Nodes. This taint key should also be renamed to
"node-role.kubernetes.io/control-plane".

Signed-off-by: Humble Chirammal <hchiramm@redhat.com>

<!-- Please take a look at our [Contributing](https://rook.io/docs/rook/latest/Contributing/development-flow/)
documentation before submitting a Pull Request!
Thank you for contributing to Rook! -->

**Description of your changes:**

**Which issue is resolved by this Pull Request:**
Resolves #https://github.com/rook/rook/issues/10398

**Checklist:**

- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/latest/Contributing/development-flow/#commit-structure)).
- [ ] **Skip Tests for Docs**: If this is only a documentation change, add the label `skip-ci` on the PR.
- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/latest/Contributing/development-flow/#submitting-a-pull-request)
- [ ] [Pending release notes](https://github.com/rook/rook/blob/master/PendingReleaseNotes.md) updated with breaking and/or notable changes for the next minor release.
- [ ] Documentation has been updated, if necessary.
- [ ] Unit tests have been added, if necessary.
- [ ] Integration tests have been added, if necessary.


Madhu-1: @humblec as this is not related to CSI please change the commit tag as per https://github.com/rook/rook/blob/master/.commitlintrc.json
humblec: > @humblec as this is not related to CSI please change the commit tag as per https://github.com/rook/rook/blob/master/.commitlintrc.json

used `mon` , hopefully it is close.

Patch:
diff --git a/.docs/macros/includes/main.py b/.docs/macros/includes/main.py
index 332abd794..07f6e6e1c 100644
--- a/.docs/macros/includes/main.py
+++ b/.docs/macros/includes/main.py
@@ -10,15 +10,13 @@ GitHub branch/tag URL replacer
 regex = r"(github\.com/.+/rook/.+)/master/"
 subst = "\\1/%s/"
 
-
 def define_env(env):
 
-    repo = Repository(".")
+    repo = Repository('.')
     if repo is not None:
         target = repo.head.shorthand
 
-    env.variables["current_branch"] = target
-
+    env.variables['current_branch'] = target
 
 def on_post_page_macros(env):
     """
@@ -26,8 +24,8 @@ def on_post_page_macros(env):
     with the correct one that is currently active.
     """
 
-    target = env.variables["current_branch"]
-    if target == "master":
+    target = env.variables['current_branch']
+    if target == 'master':
         return
 
     env.raw_markdown = re.sub(regex, subst % target, env.raw_markdown, 0)
diff --git a/.github/workflows/daily-nightly-jobs.yml b/.github/workflows/daily-nightly-jobs.yml
index 7629e991b..33dbe11fe 100644
--- a/.github/workflows/daily-nightly-jobs.yml
+++ b/.github/workflows/daily-nightly-jobs.yml
@@ -98,7 +98,7 @@ jobs:
           fetch-depth: 0
 
       - name: setup cluster resources
-        uses: ./.github/workflows/integration-test-config
+        uses: ./.github/workflows/integration-test-setup
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           kubernetes-version: "1.23.0"
@@ -132,7 +132,7 @@ jobs:
           fetch-depth: 0
 
       - name: setup cluster resources
-        uses: ./.github/workflows/integration-test-config
+        uses: ./.github/workflows/integration-test-setup
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           kubernetes-version: "1.23.0"
@@ -166,7 +166,7 @@ jobs:
           fetch-depth: 0
 
       - name: setup cluster resources
-        uses: ./.github/workflows/integration-test-config
+        uses: ./.github/workflows/integration-test-setup
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           kubernetes-version: "1.23.0"
@@ -200,7 +200,7 @@ jobs:
           fetch-depth: 0
 
       - name: setup cluster resources
-        uses: ./.github/workflows/integration-test-config
+        uses: ./.github/workflows/integration-test-setup
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           kubernetes-version: "1.23.0"
@@ -234,7 +234,7 @@ jobs:
           fetch-depth: 0
 
       - name: setup cluster resources
-        uses: ./.github/workflows/integration-test-config
+        uses: ./.github/workflows/integration-test-setup
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           kubernetes-version: "1.23.0"
@@ -268,7 +268,7 @@ jobs:
           fetch-depth: 0
 
       - name: setup cluster resources
-        uses: ./.github/workflows/integration-test-config
+        uses: ./.github/workflows/integration-test-setup
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           kubernetes-version: "1.23.0"
@@ -302,7 +302,7 @@ jobs:
           fetch-depth: 0
 
       - name: setup cluster resources
-        uses: ./.github/workflows/integration-test-config
+        uses: ./.github/workflows/integration-test-setup
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           kubernetes-version: "1.23.0"
diff --git a/.github/workflows/encryption-pvc-kms-ibm-kp/action.yml b/.github/workflows/encryption-pvc-kms-ibm-kp/action.yml
index 6d397a4a1..ee8fdc6db 100644
--- a/.github/workflows/encryption-pvc-kms-ibm-kp/action.yml
+++ b/.github/workflows/encryption-pvc-kms-ibm-kp/action.yml
@@ -23,7 +23,7 @@ runs:
       run: echo "IBM_KP_SERVICE_INSTANCE_ID and IBM_KP_SERVICE_API_KEY must be set in the environment" && exit 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ inputs.github-token }}
 
diff --git a/.github/workflows/canary-test-config/action.yaml b/.github/workflows/setup-cluster-resources/action.yaml
similarity index 100%
rename from .github/workflows/canary-test-config/action.yaml
rename to .github/workflows/setup-cluster-resources/action.yaml
diff --git a/Documentation/CRDs/ceph-cluster-crd.md b/Documentation/CRDs/ceph-cluster-crd.md
index a31e1c10e..a762d17fc 100755
--- a/Documentation/CRDs/ceph-cluster-crd.md
+++ b/Documentation/CRDs/ceph-cluster-crd.md
@@ -29,7 +29,7 @@ metadata:
 spec:
   cephVersion:
     # see the "Cluster Settings" section below for more details on which image of ceph to run
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -55,7 +55,7 @@ metadata:
 spec:
   cephVersion:
     # see the "Cluster Settings" section below for more details on which image of ceph to run
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -124,7 +124,7 @@ spec:
       - name: c
   cephVersion:
     # Stretch cluster is supported in Ceph Pacific or newer.
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
     allowUnsupported: true
   # Either storageClassDeviceSets or the storage section can be specified for creating OSDs.
   # This example uses all devices for simplicity.
@@ -162,7 +162,7 @@ Settings can be specified at the global level to apply to the cluster as a whole
 * `external`:
   * `enable`: if `true`, the cluster will not be managed by Rook but via an external entity. This mode is intended to connect to an existing cluster. In this case, Rook will only consume the external cluster. However, Rook will be able to deploy various daemons in Kubernetes such as object gateways, mds and nfs if an image is provided and will refuse otherwise. If this setting is enabled **all** the other options will be ignored except `cephVersion.image` and `dataDirHostPath`. See [external cluster configuration](#external-cluster). If `cephVersion.image` is left blank, Rook will refuse the creation of extra CRs like object, file and nfs.
 * `cephVersion`: The version information for launching the ceph daemons.
-  * `image`: The image used for running the ceph daemons. For example, `quay.io/ceph/ceph:v15.2.12` or `v17.2.0`. For more details read the [container images section](#ceph-container-images).
+  * `image`: The image used for running the ceph daemons. For example, `quay.io/ceph/ceph:v15.2.12` or `v16.2.7`. For more details read the [container images section](#ceph-container-images).
   For the latest ceph images, see the [Ceph DockerHub](https://hub.docker.com/r/ceph/ceph/tags/).
   To ensure a consistent version of the image is running across all nodes in the cluster, it is recommended to use a very specific image version.
   Tags also exist that would give the latest version, but they are only recommended for test environments. For example, the tag `v15` will be updated each time a new Octopus build is released.
@@ -733,7 +733,7 @@ kubectl -n rook-ceph get CephCluster -o yaml
       deviceClasses:
       - name: hdd
     version:
-      image: quay.io/ceph/ceph:v17.2.0
+      image: quay.io/ceph/ceph:v16.2.7
       version: 16.2.6-0
     conditions:
     - lastHeartbeatTime: "2021-03-02T21:22:11Z"
@@ -797,7 +797,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -829,7 +829,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -869,7 +869,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -917,7 +917,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -1019,7 +1019,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -1065,7 +1065,7 @@ spec:
           requests:
             storage: 10Gi
   cephVersion:
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
     allowUnsupported: false
   dashboard:
     enabled: true
@@ -1367,16 +1367,16 @@ export RGW_POOL_PREFIX=default
 
 1. Deploy Rook-Ceph, create [common.yaml](/deploy/examples/common.yaml), [crds.yaml](/deploy/examples/crds.yaml) and [operator.yaml](/deploy/examples/operator.yaml) manifests.
 
-2. Create [common-external.yaml](/deploy/examples/common-external.yaml) and [cluster-external.yaml](/deploy/examples/cluster-external.yaml)
+2. Paste the above output from `create-external-cluster-resources.py` into your current shell to allow importing the source data.
 
-3. Paste the above output from `create-external-cluster-resources.py` into your current shell to allow importing the source data.
-
-4. Run the [import](/deploy/examples/import-external-cluster.sh) script.
+3. Run the [import](/deploy/examples/import-external-cluster.sh) script.
 
     ```console
     . import-external-cluster.sh
     ```
 
+4. Create [common-external.yaml](/deploy/examples/common-external.yaml) and [cluster-external.yaml](/deploy/examples/cluster-external.yaml)
+
 5. Verify the consumer cluster is connected to the source ceph cluster:
 
     ```console
@@ -1385,15 +1385,54 @@ export RGW_POOL_PREFIX=default
     rook-ceph-external   /var/lib/rook                162m   Connected   HEALTH_OK
     ```
 
-6. StorageClass will also be created, verify its creation.
+### Create StorageClass
+
+Create a StorageClass based on the pool that was already created in the source Ceph cluster.
+In this example, the pool `replicated_2g` exists in the source cluster.
+
 ```console
-kubectl -n rook-ceph-external get sc
+cat << EOF | kubectl apply -f -
 ```
-`ceph-rbd` and `cephfs` StorageClass would be respective name for RBD and CephFS StorageClass.
 
-**NOTE:** For CephFS StorageClass you also need to export `CEPHFS_FS_NAME`, `CEPHFS_POOL_NAME` or you can pass these parameters with a CLI flag (--cephfs-data-pool-name,--cephfs-filesystem-name) while running the python script. For creating ECRBDStorageClass you need to pass --rbd-metadata-ec-pool-name CLI flag or export `RBD_METADATA_EC_POOL_NAME`.
+```yaml
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+   name: rook-ceph-block-ext
+# Change "rook-ceph" provisioner prefix to match the operator namespace if needed
+provisioner: rook-ceph.rbd.csi.ceph.com
+parameters:
+    # clusterID is the namespace where the rook cluster is running
+    clusterID: rook-ceph-external
+    # Ceph pool into which the RBD image shall be created
+    pool: replicated_2g
+
+    # RBD image format. Defaults to "2".
+    imageFormat: "2"
+
+    # RBD image features. Available for imageFormat: "2". CSI RBD currently supports only `layering` feature.
+    imageFeatures: layering
+
+    # The secrets contain Ceph admin credentials.
+    csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
+    csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph-external
+    csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
+    csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph-external
+    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
+    csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph-external
+
+    # Specify the filesystem type of the volume. If not specified, csi-provisioner
+    # will set default as `ext4`. Note that `xfs` is not recommended due to potential deadlock
+    # in hyperconverged settings where the volume is mounted on the same node as the osds.
+    csi.storage.k8s.io/fstype: ext4
+
+# Delete the rbd volume when a PVC is deleted
+reclaimPolicy: Delete
+allowVolumeExpansion: true
+EOF
+```
 
-7. Then you can now create a [persistent volume](/deploy/examples/csi) based on these StorageClass.
+You can now create a persistent volume based on this StorageClass.
 
 ### CephCluster example (management)
 
@@ -1414,7 +1453,7 @@ spec:
     enable: true
   dataDirHostPath: /var/lib/rook
   cephVersion:
-    image: quay.io/ceph/ceph:v17.2.0 # Should match external cluster version
+    image: quay.io/ceph/ceph:v16.2.7 # Should match external cluster version
 ```
 
 ## Deleting a CephCluster
diff --git a/Documentation/Getting-Started/Prerequisites/prerequisites.md b/Documentation/Getting-Started/Prerequisites/prerequisites.md
index 89df3ce62..c750d6fb5 100644
--- a/Documentation/Getting-Started/Prerequisites/prerequisites.md
+++ b/Documentation/Getting-Started/Prerequisites/prerequisites.md
@@ -50,7 +50,7 @@ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7
 Ceph OSDs have a dependency on LVM in the following scenarios:
 
 * OSDs are created on raw devices or partitions
-* If encryption is enabled (`encryptedDevice: "true"` in the cluster CR)
+* If encryption is enabled (`encryptedDevice: true` in the cluster CR)
 * A `metadata` device is specified
 
 LVM is not required for OSDs in these scenarios:
diff --git a/Documentation/Helm-Charts/operator-chart.md b/Documentation/Helm-Charts/operator-chart.md
index 97d72d4c3..490ac1fc4 100644
--- a/Documentation/Helm-Charts/operator-chart.md
+++ b/Documentation/Helm-Charts/operator-chart.md
@@ -109,15 +109,15 @@ The following tables lists the configurable parameters of the rook-operator char
 | `csi.rbdLivenessMetricsPort`        | Ceph CSI RBD driver metrics port.                                                                                                                | `8080`                                                    |
 | `csi.forceCephFSKernelClient`       | Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs.                                                                     | `true`                                                    |
 | `csi.kubeletDirPath`                | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag)                                                     | `/var/lib/kubelet`                                        |
-| `csi.cephcsi.image`                 | Ceph CSI image.                                                                                                                                  | `quay.io/cephcsi/cephcsi:v3.6.2`                          |
+| `csi.cephcsi.image`                 | Ceph CSI image.                                                                                                                                  | `quay.io/cephcsi/cephcsi:v3.6.1`                          |
 | `csi.rbdPluginUpdateStrategy`       | CSI Rbd plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.                                                       | `RollingUpdate`                                           |
 | `csi.cephFSPluginUpdateStrategy`    | CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.                                                    | `RollingUpdate`                                           |
 | `csi.nfsPluginUpdateStrategy`       | CSI NFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.                                                       | `RollingUpdate`                                           |
-| `csi.registrar.image`               | Kubernetes CSI registrar image.                                                                                                                  | `registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1` |
-| `csi.resizer.image`                 | Kubernetes CSI resizer image.                                                                                                                    | `registry.k8s.io/sig-storage/csi-resizer:v1.4.0`               |
-| `csi.provisioner.image`             | Kubernetes CSI provisioner image.                                                                                                                | `registry.k8s.io/sig-storage/csi-provisioner:v3.1.0`           |
-| `csi.snapshotter.image`             | Kubernetes CSI snapshotter image.                                                                                                                | `registry.k8s.io/sig-storage/csi-snapshotter:v6.0.1`           |
-| `csi.attacher.image`                | Kubernetes CSI Attacher image.                                                                                                                   | `registry.k8s.io/sig-storage/csi-attacher:v3.4.0`              |
+| `csi.registrar.image`               | Kubernetes CSI registrar image.                                                                                                                  | `k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0` |
+| `csi.resizer.image`                 | Kubernetes CSI resizer image.                                                                                                                    | `k8s.gcr.io/sig-storage/csi-resizer:v1.4.0`               |
+| `csi.provisioner.image`             | Kubernetes CSI provisioner image.                                                                                                                | `k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0`           |
+| `csi.snapshotter.image`             | Kubernetes CSI snapshotter image.                                                                                                                | `k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1`           |
+| `csi.attacher.image`                | Kubernetes CSI Attacher image.                                                                                                                   | `k8s.gcr.io/sig-storage/csi-attacher:v3.4.0`              |
 | `csi.cephfsPodLabels`               | Labels to add to the CSI CephFS Pods.                                                                                                            | `<none>`                                                  |
 | `csi.rbdPodLabels`                  | Labels to add to the CSI RBD Pods.                                                                                                               | `<none>`                                                  |
 | `csi.volumeReplication.enabled`     | Enable Volume Replication.                                                                                                                       | `false`                                                   |
@@ -125,7 +125,7 @@ The following tables lists the configurable parameters of the rook-operator char
 | `csi.csiAddons.enabled`             | Enable CSIAddons                                                                                                                                 | `false`                                                   |
 | `csi.csiAddons.image`               | CSIAddons Sidecar image.                                                                                                                         | `quay.io/csiaddons/k8s-sidecar:v0.2.1`                    |
 | `csi.nfs.enabled`                   | Enable nfs driver.                                                                                                                               | `false`                                                   |
-| `csi.nfs.image`                     | NFS nodeplugin image.                                                                                                                            | `registry.k8s.io/sig-storage/nfsplugin:v4.0.0`                 |
+| `csi.nfs.image`                     | NFS nodeplugin image.                                                                                                                            | `k8s.gcr.io/sig-storage/nfsplugin:v3.1.0`                 |
 | `admissionController.tolerations`   | Array of tolerations in YAML format which will be added to admission controller deployment.                                                      | `<none>`                                                  |
 | `admissionController.nodeAffinity`  | The node labels for affinity of the admission controller deployment [^1]                                                                         | `<none>`                                                  |
 | `monitoring.enabled`                | Create necessary RBAC rules for Rook to integrate with Prometheus monitoring in the operator namespace. Requires Prometheus to be pre-installed. | `false`                                                   |
diff --git a/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md b/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md
index cf3c21be6..c66dd52e6 100644
--- a/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md
+++ b/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-snapshot.md
@@ -4,14 +4,12 @@ title: Snapshots
 
 ## Prerequisites
 
-- Rook officially supports v1 snapshots for Kubernetes v1.20+.
+- Rook officially supports v1/v1beta1 snapshots for kubernetes v1.17+.
 
-- Install the snapshot controller and snapshot v1 CRD as required. More info can be found [here](https://github.com/kubernetes-csi/external-snapshotter/tree/v6.0.1#usage).
+- Install the snapshot controller and snapshot v1/v1beta1 CRD as required. More info can be found [here](https://github.com/kubernetes-csi/external-snapshotter/tree/v4.0.0#usage).
 
 !!! note
-    If only Alpha snapshots are available, enable snapshotter in `rook-ceph-operator-config` or helm chart `values.yaml`, change the external-snapshotter image to `registry.k8s.io/sig-storage/csi-snapshotter:v1.2.2` and refer to the [alpha snapshots documentation](https://github.com/rook/rook/blob/release-1.3/Documentation/ceph-csi-drivers.md#rbd-snapshots)
-
-    VolumeSnapshot betav1 is deprecated in Kubernetes 1.20+ and removed in 1.24.0. If you still require betav1 snapshots, change the external-snapshotter image to `registry.k8s.io/sig-storage/csi-snapshotter:v5.0.1` and refer to the [betav1 snapshots documentation](https://rook.github.io/docs/rook/v1.8/ceph-csi-snapshot.html#rbd-snapshots)
+    If only Alpha snapshots are available, enable snapshotter in `rook-ceph-operator-config` or helm chart `values.yaml`, change the external-snapshotter image to `k8s.gcr.io/sig-storage/csi-snapshotter:v1.2.2` and refer to the [alpha snapshots documentation](https://github.com/rook/rook/blob/release-1.3/Documentation/ceph-csi-drivers.md#rbd-snapshots)
 
 - We also need a `VolumeSnapshotClass` for volume snapshot to work. The purpose of a `VolumeSnapshotClass` is
 defined in [the kubernetes
diff --git a/Documentation/Storage-Configuration/Ceph-CSI/custom-images.md b/Documentation/Storage-Configuration/Ceph-CSI/custom-images.md
index 8f1d88b01..f63f77a7c 100644
--- a/Documentation/Storage-Configuration/Ceph-CSI/custom-images.md
+++ b/Documentation/Storage-Configuration/Ceph-CSI/custom-images.md
@@ -18,8 +18,8 @@ kubectl -n $ROOK_OPERATOR_NAMESPACE edit configmap rook-ceph-operator-config
 The default upstream images are included below, which you can change to your desired images.
 
 ```yaml
-ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.6.2"
-ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.1"
+ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.6.1"
+ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0"
 ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0"
 ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.4.0"
 ROOK_CSI_RESIZER_IMAGE: "k8s.gcr.io/sig-storage/csi-resizer:v1.4.0"
diff --git a/Documentation/Upgrade/ceph-upgrade.md b/Documentation/Upgrade/ceph-upgrade.md
index c9577ed56..5600a1f63 100644
--- a/Documentation/Upgrade/ceph-upgrade.md
+++ b/Documentation/Upgrade/ceph-upgrade.md
@@ -64,15 +64,15 @@ Official Ceph container images can be found on [Quay](https://quay.io/repository
 
 These images are tagged in a few ways:
 
-* The most explicit form of tags are full-ceph-version-and-build tags (e.g., `v17.2.0-20220420`).
+* The most explicit form of tags are full-ceph-version-and-build tags (e.g., `v16.2.7-20220216`).
   These tags are recommended for production clusters, as there is no possibility for the cluster to
   be heterogeneous with respect to the version of Ceph running in containers.
-* Ceph major version tags (e.g., `v17`) are useful for development and test clusters so that the
+* Ceph major version tags (e.g., `v16`) are useful for development and test clusters so that the
   latest version of Ceph is always available.
 
 **Ceph containers other than the official images from the registry above will not be supported.**
 
-### Example Upgrade to Ceph Quincy
+### Example Upgrade to Ceph Pacific
 
 #### **1. Update the Ceph daemons**
 
@@ -81,7 +81,7 @@ in the cluster CRD (`spec.cephVersion.image`).
 
 ```console
 ROOK_CLUSTER_NAMESPACE=rook-ceph
-NEW_CEPH_IMAGE='quay.io/ceph/ceph:v17.2.0-20220420'
+NEW_CEPH_IMAGE='quay.io/ceph/ceph:v16.2.7-20220216'
 kubectl -n $ROOK_CLUSTER_NAMESPACE patch CephCluster $ROOK_CLUSTER_NAMESPACE --type=merge -p "{\"spec\": {\"cephVersion\": {\"image\": \"$NEW_CEPH_IMAGE\"}}}"
 ```
 
@@ -100,9 +100,9 @@ Confirm the upgrade is completed when the versions are all on the desired Ceph v
 kubectl -n $ROOK_CLUSTER_NAMESPACE get deployment -l rook_cluster=$ROOK_CLUSTER_NAMESPACE -o jsonpath='{range .items[*]}{"ceph-version="}{.metadata.labels.ceph-version}{"\n"}{end}' | sort | uniq
 This cluster is not yet finished:
     ceph-version=15.2.13-0
-    ceph-version=v17.2.0-0
+    ceph-version=16.2.6-0
 This cluster is finished:
-    ceph-version=v17.2.0-0
+    ceph-version=16.2.6-0
 ```
 
 #### **3. Verify cluster health**
diff --git a/deploy/charts/rook-ceph-cluster/values.yaml b/deploy/charts/rook-ceph-cluster/values.yaml
index 137ebf9b5..314199b48 100644
--- a/deploy/charts/rook-ceph-cluster/values.yaml
+++ b/deploy/charts/rook-ceph-cluster/values.yaml
@@ -65,7 +65,7 @@ cephClusterSpec:
     # versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
     # If you want to be more precise, you can always use a timestamp tag such quay.io/ceph/ceph:v15.2.11-20200419
     # This tag might not contain a new Ceph version, just security fixes from the underlying operating system, which will reduce vulnerabilities
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
     # Whether to allow unsupported versions of Ceph. Currently `octopus` and `pacific` are supported.
     # Future versions such as `pacific` would require this to be set to `true`.
     # Do not set to true in production.
diff --git a/deploy/charts/rook-ceph/values.yaml b/deploy/charts/rook-ceph/values.yaml
index b15cfa93c..88b9da6ec 100644
--- a/deploy/charts/rook-ceph/values.yaml
+++ b/deploy/charts/rook-ceph/values.yaml
@@ -347,17 +347,17 @@ csi:
   #rbdLivenessMetricsPort: 9080
   #kubeletDirPath: /var/lib/kubelet
   #cephcsi:
-    #image: quay.io/cephcsi/cephcsi:v3.6.2
+    #image: quay.io/cephcsi/cephcsi:v3.6.1
   #registrar:
-    #image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1
+    #image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0
   #provisioner:
-    #image: registry.k8s.io/sig-storage/csi-provisioner:v3.1.0
+    #image: k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
   #snapshotter:
-    #image: registry.k8s.io/sig-storage/csi-snapshotter:v6.0.1
+    #image: k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1
   #attacher:
-    #image: registry.k8s.io/sig-storage/csi-attacher:v3.4.0
+    #image: k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
   #resizer:
-    #image: registry.k8s.io/sig-storage/csi-resizer:v1.4.0
+    #image: k8s.gcr.io/sig-storage/csi-resizer:v1.4.0
   # Labels to add to the CSI CephFS Deployments and DaemonSets Pods.
   #cephfsPodLabels: "key1=value1,key2=value2"
   # Labels to add to the CSI NFS Deployments and DaemonSets Pods.
@@ -377,7 +377,7 @@ csi:
   # Enable the nfs csi driver.
   nfs:
     enabled: false
-    #image: "registry.k8s.io/sig-storage/nfsplugin:v4.0.0"
+    #image: "k8s.gcr.io/sig-storage/nfsplugin:v3.1.0"
 enableDiscoveryDaemon: false
 cephCommandsTimeoutSeconds: "15"
 
diff --git a/deploy/examples/cluster-external-management.yaml b/deploy/examples/cluster-external-management.yaml
index fd7953152..341ad9eff 100644
--- a/deploy/examples/cluster-external-management.yaml
+++ b/deploy/examples/cluster-external-management.yaml
@@ -19,4 +19,4 @@ spec:
   dataDirHostPath: /var/lib/rook
   # providing an image is required, if you want to create other CRs (rgw, mds, nfs)
   cephVersion:
-    image: quay.io/ceph/ceph:v17.2.0 # Should match external cluster version
+    image: quay.io/ceph/ceph:v16.2.7 # Should match external cluster version
diff --git a/deploy/examples/cluster-on-local-pvc.yaml b/deploy/examples/cluster-on-local-pvc.yaml
index 4eb171596..39aaf3fd4 100644
--- a/deploy/examples/cluster-on-local-pvc.yaml
+++ b/deploy/examples/cluster-on-local-pvc.yaml
@@ -173,7 +173,7 @@ spec:
           requests:
             storage: 10Gi
   cephVersion:
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
     allowUnsupported: false
   skipUpgradeChecks: false
   continueUpgradeAfterChecksEvenIfNotHealthy: false
diff --git a/deploy/examples/cluster-on-pvc.yaml b/deploy/examples/cluster-on-pvc.yaml
index 2fe0370d6..37d2020d5 100644
--- a/deploy/examples/cluster-on-pvc.yaml
+++ b/deploy/examples/cluster-on-pvc.yaml
@@ -33,7 +33,7 @@ spec:
           requests:
             storage: 10Gi
   cephVersion:
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
     allowUnsupported: false
   skipUpgradeChecks: false
   continueUpgradeAfterChecksEvenIfNotHealthy: false
diff --git a/deploy/examples/cluster-stretched-aws.yaml b/deploy/examples/cluster-stretched-aws.yaml
index 55462a541..d646a14af 100644
--- a/deploy/examples/cluster-stretched-aws.yaml
+++ b/deploy/examples/cluster-stretched-aws.yaml
@@ -45,7 +45,7 @@ spec:
     count: 2
   cephVersion:
     # Stretch cluster support upstream is only available starting in Ceph Pacific
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
     allowUnsupported: true
   skipUpgradeChecks: false
   continueUpgradeAfterChecksEvenIfNotHealthy: false
@@ -126,10 +126,13 @@ spec:
   placement:
     # The arbiter mon can have its own placement settings that will be different from the mons.
     # If the arbiter section is not included in the placement, the arbiter will use the same placement
-    # settings as other mons. In this example, the arbiter has a toleration to run on a master node.
+    # settings as other mons. In this example, the arbiter has a toleration to run on a control-plane node.
     arbiter:
       tolerations:
-        - key: node-role.kubernetes.io/master
+        # kubernetes v1.24 clusters would need the taint `node-role.kubernetes.io/control-plane`
+        # configuration. For earlier versions you may use `node-role.kubernetes.io/master` if
+        # available in your cluster.
+        - key: node-role.kubernetes.io/control-plane
           operator: Exists
           effect: NoSchedule
   priorityClassNames:
diff --git a/deploy/examples/cluster-stretched.yaml b/deploy/examples/cluster-stretched.yaml
index 1e95b6bde..a96d06326 100644
--- a/deploy/examples/cluster-stretched.yaml
+++ b/deploy/examples/cluster-stretched.yaml
@@ -39,7 +39,7 @@ spec:
     count: 2
   cephVersion:
     # Stretch cluster support upstream is only available starting in Ceph Pacific
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
     allowUnsupported: true
   skipUpgradeChecks: false
   continueUpgradeAfterChecksEvenIfNotHealthy: false
@@ -54,10 +54,13 @@ spec:
   placement:
     # The arbiter mon can have its own placement settings that will be different from the mons.
     # If the arbiter section is not included in the placement, the arbiter will use the same placement
-    # settings as other mons. In this example, the arbiter has a toleration to run on a master node.
+    # settings as other mons. In this example, the arbiter has a toleration to run on a control-plane node.
     arbiter:
       tolerations:
-        - key: node-role.kubernetes.io/master
+        # kubernetes v1.24 clusters would need the taint `node-role.kubernetes.io/control-plane`
+        # configuration. For earlier versions you may use `node-role.kubernetes.io/master` if
+        # available in your cluster.
+        - key: node-role.kubernetes.io/control-plane
           operator: Exists
           effect: NoSchedule
     osd:
diff --git a/deploy/examples/cluster.yaml b/deploy/examples/cluster.yaml
index 43b4822a1..d3ef458a9 100644
--- a/deploy/examples/cluster.yaml
+++ b/deploy/examples/cluster.yaml
@@ -16,12 +16,12 @@ metadata:
 spec:
   cephVersion:
     # The container image used to launch the Ceph daemon pods (mon, mgr, osd, mds, rgw).
-    # v15 is octopus, v16 is pacific, and v17 is quincy.
-    # RECOMMENDATION: In production, use a specific version tag instead of the general v17 flag, which pulls the latest release and could result in different
+    # v15 is octopus, and v16 is pacific.
+    # RECOMMENDATION: In production, use a specific version tag instead of the general v14 flag, which pulls the latest release and could result in different
     # versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
-    # If you want to be more precise, you can always use a timestamp tag such quay.io/ceph/ceph:v17.2.0-20220420
+    # If you want to be more precise, you can always use a timestamp tag such quay.io/ceph/ceph:v16.2.7-20220216
     # This tag might not contain a new Ceph version, just security fixes from the underlying operating system, which will reduce vulnerabilities
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
     # Whether to allow unsupported versions of Ceph. Currently `octopus` and `pacific` are supported.
     # Future versions such as `pacific` would require this to be set to `true`.
     # Do not set to true in production.
diff --git a/deploy/examples/create-external-cluster-resources.py b/deploy/examples/create-external-cluster-resources.py
index 4ea0621c2..302376c3a 100644
--- a/deploy/examples/create-external-cluster-resources.py
+++ b/deploy/examples/create-external-cluster-resources.py
@@ -1,4 +1,4 @@
-"""
+'''
 Copyright 2020 The Rook Authors. All rights reserved.
 
 Licensed under the Apache License, Version 2.0 (the "License");
@@ -12,7 +12,7 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-"""
+'''
 
 import errno
 import sys
@@ -67,11 +67,9 @@ except:
     from urllib.parse import urlparse, unquote
     from base64 import encodebytes as encodestring
 
-
 class ExecutionFailureException(Exception):
     pass
 
-
 ################################################
 ################## DummyRados ##################
 ################################################
@@ -81,100 +79,54 @@ class ExecutionFailureException(Exception):
 class DummyRados(object):
     def __init__(self):
         self.return_val = 0
-        self.err_message = ""
-        self.state = "connected"
+        self.err_message = ''
+        self.state = 'connected'
         self.cmd_output_map = {}
         self.cmd_names = {}
         self._init_cmd_output_map()
         self.dummy_host_ip_map = {}
 
     def _init_cmd_output_map(self):
-        json_file_name = "test-data/ceph-status-out"
+        json_file_name = 'test-data/ceph-status-out'
         script_dir = path.abspath(path.dirname(__file__))
         ceph_status_str = ""
-        with open(path.join(script_dir, json_file_name), "r") as json_file:
+        with open(path.join(script_dir, json_file_name), 'r') as json_file:
             ceph_status_str = json_file.read()
-        self.cmd_names["fs ls"] = """{"format": "json", "prefix": "fs ls"}"""
-        self.cmd_names[
-            "quorum_status"
-        ] = """{"format": "json", "prefix": "quorum_status"}"""
-        self.cmd_names[
-            "mgr services"
-        ] = """{"format": "json", "prefix": "mgr services"}"""
+        self.cmd_names['fs ls'] = '''{"format": "json", "prefix": "fs ls"}'''
+        self.cmd_names['quorum_status'] = '''{"format": "json", "prefix": "quorum_status"}'''
+        self.cmd_names['mgr services'] = '''{"format": "json", "prefix": "mgr services"}'''
         # all the commands and their output
-        self.cmd_output_map[
-            self.cmd_names["fs ls"]
-        ] = """[{"name":"myfs","metadata_pool":"myfs-metadata","metadata_pool_id":2,"data_pool_ids":[3],"data_pools":["myfs-replicated"]}]"""
-        self.cmd_output_map[
-            self.cmd_names["quorum_status"]
-        ] = """{"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_leader_name":"a","quorum_age":14385,"features":{"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"monmap":{"epoch":1,"fsid":"af4e1673-0b72-402d-990a-22d2919d0f1c","modified":"2020-05-07T03:36:39.918035Z","created":"2020-05-07T03:36:39.918035Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"10.110.205.174:3300","nonce":0},{"type":"v1","addr":"10.110.205.174:6789","nonce":0}]},"addr":"10.110.205.174:6789/0","public_addr":"10.110.205.174:6789/0","priority":0,"weight":0}]}}"""
-        self.cmd_output_map[
-            self.cmd_names["mgr services"]
-        ] = """{"dashboard":"https://ceph-dashboard:8443/","prometheus":"http://ceph-dashboard-db:9283/"}"""
-        self.cmd_output_map[
-            """{"caps": ["mon", "allow r, allow command quorum_status", "osd", "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow x pool=default.rgw.buckets.index"], "entity": "client.healthchecker", "format": "json", "prefix": "auth get-or-create"}"""
-        ] = """[{"entity":"client.healthchecker","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon":"allow r, allow command quorum_status","osd":"allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow x pool=default.rgw.buckets.index"}}]"""
-        self.cmd_output_map[
-            """{"caps": ["mon", "profile rbd, allow command 'osd blocklist'", "osd", "profile rbd"], "entity": "client.csi-rbd-node", "format": "json", "prefix": "auth get-or-create"}"""
-        ] = """[{"entity":"client.csi-rbd-node","key":"AQBOgrNeHbK1AxAAubYBeV8S1U/GPzq5SVeq6g==","caps":{"mon":"profile rbd, allow command 'osd blocklist'","osd":"profile rbd"}}]"""
-        self.cmd_output_map[
-            """{"caps": ["mon", "profile rbd, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "profile rbd"], "entity": "client.csi-rbd-provisioner", "format": "json", "prefix": "auth get-or-create"}"""
-        ] = """[{"entity":"client.csi-rbd-provisioner","key":"AQBNgrNe1geyKxAA8ekViRdE+hss5OweYBkwNg==","caps":{"mgr":"allow rw","mon":"profile rbd, allow command 'osd blocklist'","osd":"profile rbd"}}]"""
-        self.cmd_output_map[
-            """{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs *=*", "mds", "allow rw"], "entity": "client.csi-cephfs-node", "format": "json", "prefix": "auth get-or-create"}"""
-        ] = """[{"entity":"client.csi-cephfs-node","key":"AQBOgrNeENunKxAAPCmgE7R6G8DcXnaJ1F32qg==","caps":{"mds":"allow rw","mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs *=*"}}]"""
-        self.cmd_output_map[
-            """{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=*"], "entity": "client.csi-cephfs-provisioner", "format": "json", "prefix": "auth get-or-create"}"""
-        ] = """[{"entity":"client.csi-cephfs-provisioner","key":"AQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs metadata=*"}}]"""
-        self.cmd_output_map[
-            """{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=*"], "entity": "client.csi-cephfs-provisioner-openshift-storage", "format": "json", "prefix": "auth get-or-create"}"""
-        ] = """[{"entity":"client.csi-cephfs-provisioner-openshift-storage","key":"BQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs metadata=*"}}]"""
-        self.cmd_output_map[
-            """{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=myfs"], "entity": "client.csi-cephfs-provisioner-openshift-storage-myfs", "format": "json", "prefix": "auth get-or-create"}"""
-        ] = """[{"entity":"client.csi-cephfs-provisioner-openshift-storage-myfs","key":"CQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs metadata=myfs"}}]"""
-        self.cmd_output_map[
-            """{"caps": ["mon", "allow r, allow command quorum_status, allow command version", "mgr", "allow command config", "osd", "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index"], "entity": "client.healthchecker", "format": "json", "prefix": "auth get-or-create"}"""
-        ] = """[{"entity":"client.healthchecker","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon": "allow r, allow command quorum_status, allow command version", "mgr": "allow command config", "osd": "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index"}}]"""
-        self.cmd_output_map[
-            """{"format": "json", "prefix": "mgr services"}"""
-        ] = """{"dashboard": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:7000/", "prometheus": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:9283/"}"""
-        self.cmd_output_map[
-            """{"entity": "client.healthchecker", "format": "json", "prefix": "auth get"}"""
-        ] = """{"dashboard": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:7000/", "prometheus": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:9283/"}"""
-        self.cmd_output_map[
-            """{"entity": "client.healthchecker", "format": "json", "prefix": "auth get"}"""
-        ] = """[{"entity":"client.healthchecker","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon": "allow r, allow command quorum_status, allow command version", "mgr": "allow command config", "osd": "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index"}}]"""
-        self.cmd_output_map[
-            """{"entity": "client.csi-cephfs-node", "format": "json", "prefix": "auth get"}"""
-        ] = """[]"""
-        self.cmd_output_map[
-            """{"entity": "client.csi-rbd-node", "format": "json", "prefix": "auth get"}"""
-        ] = """[]"""
-        self.cmd_output_map[
-            """{"entity": "client.csi-rbd-provisioner", "format": "json", "prefix": "auth get"}"""
-        ] = """[]"""
-        self.cmd_output_map[
-            """{"entity": "client.csi-cephfs-provisioner", "format": "json", "prefix": "auth get"}"""
-        ] = """[]"""
-        self.cmd_output_map[
-            """{"entity": "client.csi-cephfs-provisioner-openshift-storage", "format": "json", "prefix": "auth get"}"""
-        ] = """[]"""
-        self.cmd_output_map[
-            """{"entity": "client.csi-cephfs-provisioner-openshift-storage-myfs", "format": "json", "prefix": "auth get"}"""
-        ] = """[]"""
-        self.cmd_output_map[
-            """{"entity": "client.csi-cephfs-provisioner", "format": "json", "prefix": "auth get"}"""
-        ] = """[{"entity":"client.csi-cephfs-provisioner","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon":"allow r", "mgr":"allow rw", "osd":"allow rw tag cephfs metadata=*"}}]"""
-        self.cmd_output_map[
-            """{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=*"], "entity": "client.csi-cephfs-provisioner", "format": "json", "prefix": "auth caps"}"""
-        ] = """[{"entity":"client.csi-cephfs-provisioner","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon":"allow r,  allow command 'osd blocklist'", "mgr":"allow rw", "osd":"allow rw tag cephfs metadata=*"}}]"""
+        self.cmd_output_map[self.cmd_names['fs ls']
+                            ] = '''[{"name":"myfs","metadata_pool":"myfs-metadata","metadata_pool_id":2,"data_pool_ids":[3],"data_pools":["myfs-replicated"]}]'''
+        self.cmd_output_map[self.cmd_names['quorum_status']] = '''{"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_leader_name":"a","quorum_age":14385,"features":{"quorum_con":"4540138292836696063","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"]},"monmap":{"epoch":1,"fsid":"af4e1673-0b72-402d-990a-22d2919d0f1c","modified":"2020-05-07T03:36:39.918035Z","created":"2020-05-07T03:36:39.918035Z","min_mon_release":15,"min_mon_release_name":"octopus","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"10.110.205.174:3300","nonce":0},{"type":"v1","addr":"10.110.205.174:6789","nonce":0}]},"addr":"10.110.205.174:6789/0","public_addr":"10.110.205.174:6789/0","priority":0,"weight":0}]}}'''
+        self.cmd_output_map[self.cmd_names['mgr services']
+                            ] = '''{"dashboard":"https://ceph-dashboard:8443/","prometheus":"http://ceph-dashboard-db:9283/"}'''
+        self.cmd_output_map['''{"caps": ["mon", "allow r, allow command quorum_status", "osd", "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow x pool=default.rgw.buckets.index"], "entity": "client.healthchecker", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.healthchecker","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon":"allow r, allow command quorum_status","osd":"allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow x pool=default.rgw.buckets.index"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "profile rbd, allow command 'osd blocklist'", "osd", "profile rbd"], "entity": "client.csi-rbd-node", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-rbd-node","key":"AQBOgrNeHbK1AxAAubYBeV8S1U/GPzq5SVeq6g==","caps":{"mon":"profile rbd, allow command 'osd blocklist'","osd":"profile rbd"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "profile rbd, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "profile rbd"], "entity": "client.csi-rbd-provisioner", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-rbd-provisioner","key":"AQBNgrNe1geyKxAA8ekViRdE+hss5OweYBkwNg==","caps":{"mgr":"allow rw","mon":"profile rbd, allow command 'osd blocklist'","osd":"profile rbd"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs *=*", "mds", "allow rw"], "entity": "client.csi-cephfs-node", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-node","key":"AQBOgrNeENunKxAAPCmgE7R6G8DcXnaJ1F32qg==","caps":{"mds":"allow rw","mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs *=*"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=*"], "entity": "client.csi-cephfs-provisioner", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-provisioner","key":"AQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs metadata=*"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=*"], "entity": "client.csi-cephfs-provisioner-openshift-storage", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-provisioner-openshift-storage","key":"BQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs metadata=*"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=myfs"], "entity": "client.csi-cephfs-provisioner-openshift-storage-myfs", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-provisioner-openshift-storage-myfs","key":"CQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs metadata=myfs"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "allow r, allow command quorum_status, allow command version", "mgr", "allow command config", "osd", "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index"], "entity": "client.healthchecker", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.healthchecker","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon": "allow r, allow command quorum_status, allow command version", "mgr": "allow command config", "osd": "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index"}}]'''
+        self.cmd_output_map['''{"format": "json", "prefix": "mgr services"}'''] = '''{"dashboard": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:7000/", "prometheus": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:9283/"}'''
+        self.cmd_output_map['''{"entity": "client.healthchecker", "format": "json", "prefix": "auth get"}'''] = '''{"dashboard": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:7000/", "prometheus": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:9283/"}'''
+        self.cmd_output_map['''{"entity": "client.healthchecker", "format": "json", "prefix": "auth get"}'''] = '''[{"entity":"client.healthchecker","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon": "allow r, allow command quorum_status, allow command version", "mgr": "allow command config", "osd": "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index"}}]'''
+        self.cmd_output_map['''{"entity": "client.csi-cephfs-node", "format": "json", "prefix": "auth get"}'''] = '''[]'''
+        self.cmd_output_map['''{"entity": "client.csi-rbd-node", "format": "json", "prefix": "auth get"}'''] = '''[]'''
+        self.cmd_output_map['''{"entity": "client.csi-rbd-provisioner", "format": "json", "prefix": "auth get"}'''] = '''[]'''
+        self.cmd_output_map['''{"entity": "client.csi-cephfs-provisioner", "format": "json", "prefix": "auth get"}'''] = '''[]'''
+        self.cmd_output_map['''{"entity": "client.csi-cephfs-provisioner-openshift-storage", "format": "json", "prefix": "auth get"}'''] = '''[]'''
+        self.cmd_output_map['''{"entity": "client.csi-cephfs-provisioner-openshift-storage-myfs", "format": "json", "prefix": "auth get"}'''] = '''[]'''
+        self.cmd_output_map['''{"entity": "client.csi-cephfs-provisioner", "format": "json", "prefix": "auth get"}'''] = '''[{"entity":"client.csi-cephfs-provisioner","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon":"allow r", "mgr":"allow rw", "osd":"allow rw tag cephfs metadata=*"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=*"], "entity": "client.csi-cephfs-provisioner", "format": "json", "prefix": "auth caps"}'''] = '''[{"entity":"client.csi-cephfs-provisioner","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon":"allow r,  allow command 'osd blocklist'", "mgr":"allow rw", "osd":"allow rw tag cephfs metadata=*"}}]'''
         self.cmd_output_map['{"format": "json", "prefix": "status"}'] = ceph_status_str
 
     def shutdown(self):
         pass
 
     def get_fsid(self):
-        return "af4e1673-0b72-402d-990a-22d2919d0f1c"
+        return 'af4e1673-0b72-402d-990a-22d2919d0f1c'
 
     def conf_read_file(self):
         pass
@@ -189,24 +141,20 @@ class DummyRados(object):
         json_cmd = json.loads(cmd)
         json_cmd_str = json.dumps(json_cmd, sort_keys=True)
         cmd_output = self.cmd_output_map[json_cmd_str]
-        return (
-            self.return_val,
-            cmd_output,
-            "{}".format(self.err_message).encode("utf-8"),
-        )
+        return self.return_val, \
+            cmd_output, \
+            "{}".format(self.err_message).encode('utf-8')
 
     def _convert_hostname_to_ip(self, host_name):
-        ip_reg_x = re.compile(r"\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}")
+        ip_reg_x = re.compile(r'\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}')
         # if provided host is directly an IP address, return the same
         if ip_reg_x.match(host_name):
             return host_name
         import random
-
         host_ip = self.dummy_host_ip_map.get(host_name, "")
         if not host_ip:
             host_ip = "172.9.{}.{}".format(
-                random.randint(0, 254), random.randint(0, 254)
-            )
+                random.randint(0, 254), random.randint(0, 254))
             self.dummy_host_ip_map[host_name] = host_ip
         del random
         return host_ip
@@ -215,12 +163,10 @@ class DummyRados(object):
     def Rados(conffile=None):
         return DummyRados()
 
-
 class S3Auth(AuthBase):
 
     """Attaches AWS Authentication to the given Request object."""
-
-    service_base_url = "s3.amazonaws.com"
+    service_base_url = 's3.amazonaws.com'
 
     def __init__(self, access_key, secret_key, service_url=None):
         if service_url:
@@ -230,19 +176,23 @@ class S3Auth(AuthBase):
 
     def __call__(self, r):
         # Create date header if it is not created yet.
-        if "date" not in r.headers and "x-amz-date" not in r.headers:
-            r.headers["date"] = formatdate(timeval=None, localtime=False, usegmt=True)
+        if 'date' not in r.headers and 'x-amz-date' not in r.headers:
+            r.headers['date'] = formatdate(
+                timeval=None,
+                localtime=False,
+                usegmt=True)
         signature = self.get_signature(r)
         if py3k:
-            signature = signature.decode("utf-8")
-        r.headers["Authorization"] = "AWS %s:%s" % (self.access_key, signature)
+            signature = signature.decode('utf-8')
+        r.headers['Authorization'] = 'AWS %s:%s' % (self.access_key, signature)
         return r
 
     def get_signature(self, r):
-        canonical_string = self.get_canonical_string(r.url, r.headers, r.method)
+        canonical_string = self.get_canonical_string(
+            r.url, r.headers, r.method)
         if py3k:
-            key = self.secret_key.encode("utf-8")
-            msg = canonical_string.encode("utf-8")
+            key = self.secret_key.encode('utf-8')
+            msg = canonical_string.encode('utf-8')
         else:
             key = self.secret_key
             msg = canonical_string
@@ -253,49 +203,50 @@ class S3Auth(AuthBase):
         parsedurl = urlparse(url)
         objectkey = parsedurl.path[1:]
 
-        bucket = parsedurl.netloc[: -len(self.service_base_url)]
+        bucket = parsedurl.netloc[:-len(self.service_base_url)]
         if len(bucket) > 1:
             # remove last dot
             bucket = bucket[:-1]
 
-        interesting_headers = {"content-md5": "", "content-type": "", "date": ""}
+        interesting_headers = {
+            'content-md5': '',
+            'content-type': '',
+            'date': ''}
         for key in headers:
             lk = key.lower()
             try:
-                lk = lk.decode("utf-8")
+                lk = lk.decode('utf-8')
             except:
                 pass
-            if headers[key] and (
-                lk in interesting_headers.keys() or lk.startswith("x-amz-")
-            ):
+            if headers[key] and (lk in interesting_headers.keys()
+                                 or lk.startswith('x-amz-')):
                 interesting_headers[lk] = headers[key].strip()
 
         # If x-amz-date is used it supersedes the date header.
         if not py3k:
-            if "x-amz-date" in interesting_headers:
-                interesting_headers["date"] = ""
+            if 'x-amz-date' in interesting_headers:
+                interesting_headers['date'] = ''
         else:
-            if "x-amz-date" in interesting_headers:
-                interesting_headers["date"] = ""
+            if 'x-amz-date' in interesting_headers:
+                interesting_headers['date'] = ''
 
-        buf = "%s\n" % method
+        buf = '%s\n' % method
         for key in sorted(interesting_headers.keys()):
             val = interesting_headers[key]
-            if key.startswith("x-amz-"):
-                buf += "%s:%s\n" % (key, val)
+            if key.startswith('x-amz-'):
+                buf += '%s:%s\n' % (key, val)
             else:
-                buf += "%s\n" % val
+                buf += '%s\n' % val
 
         # append the bucket if it exists
-        if bucket != "":
-            buf += "/%s" % bucket
+        if bucket != '':
+            buf += '/%s' % bucket
 
         # add the objectkey. even if it doesn't exist, add the slash
-        buf += "/%s" % objectkey
+        buf += '/%s' % objectkey
 
         return buf
 
-
 class RadosJSON:
     EXTERNAL_USER_NAME = "client.healthchecker"
     EXTERNAL_RGW_ADMIN_OPS_USER_NAME = "rgw-admin-ops-user"
@@ -307,140 +258,70 @@ class RadosJSON:
     def gen_arg_parser(cls, args_to_parse=None):
         argP = argparse.ArgumentParser()
 
-        common_group = argP.add_argument_group("common")
-        common_group.add_argument("--verbose", "-v", action="store_true", default=False)
-        common_group.add_argument(
-            "--ceph-conf", "-c", help="Provide a ceph conf file.", type=str
-        )
-        common_group.add_argument(
-            "--run-as-user",
-            "-u",
-            default="",
-            type=str,
-            help="Provides a user name to check the cluster's health status, must be prefixed by 'client.'",
-        )
-        common_group.add_argument(
-            "--cluster-name", default="", help="Ceph cluster name"
-        )
-        common_group.add_argument(
-            "--namespace", default="", help="Namespace where CephCluster is running"
-        )
-        common_group.add_argument(
-            "--rgw-pool-prefix", default="", help="RGW Pool prefix"
-        )
-        common_group.add_argument(
-            "--restricted-auth-permission",
-            default=False,
-            help="Restricted cephCSIKeyrings auth permissions to specific pools, cluster."
-            + "Mandatory flags that need to be set are --rbd-data-pool-name, and --cluster-name."
-            + "--cephfs-filesystem-name flag can also be passed in case of cephfs user restriction, so it can restrict user to particular cephfs filesystem"
-            + "sample run: `python3 /etc/ceph/create-external-cluster-resources.py --cephfs-filesystem-name myfs --rbd-data-pool-name replicapool --cluster-name rookStorage --restricted-auth-permission true`"
-            + "Note: Restricting the users per pool, and per cluster will require to create new users and new secrets for that users.",
-        )
-
-        output_group = argP.add_argument_group("output")
-        output_group.add_argument(
-            "--format",
-            "-t",
-            choices=["json", "bash"],
-            default="json",
-            help="Provides the output format (json | bash)",
-        )
-        output_group.add_argument(
-            "--output",
-            "-o",
-            default="",
-            help="Output will be stored into the provided file",
-        )
-        output_group.add_argument(
-            "--cephfs-filesystem-name",
-            default="",
-            help="Provides the name of the Ceph filesystem",
-        )
-        output_group.add_argument(
-            "--cephfs-metadata-pool-name",
-            default="",
-            help="Provides the name of the cephfs metadata pool",
-        )
-        output_group.add_argument(
-            "--cephfs-data-pool-name",
-            default="",
-            help="Provides the name of the cephfs data pool",
-        )
-        output_group.add_argument(
-            "--rbd-data-pool-name",
-            default="",
-            required=False,
-            help="Provides the name of the RBD datapool",
-        )
-        output_group.add_argument(
-            "--rgw-endpoint",
-            default="",
-            required=False,
-            help="RADOS Gateway endpoint (in <IP>:<PORT> format). Note: FQDN is also supported(in <FQDN>:<PORT> format)",
-        )
-        output_group.add_argument(
-            "--rgw-tls-cert-path",
-            default="",
-            required=False,
-            help="RADOS Gateway endpoint TLS certificate",
-        )
-        output_group.add_argument(
-            "--rgw-skip-tls",
-            required=False,
-            default=False,
-            help="Ignore TLS certification validation when a self-signed certificate is provided (NOT RECOMMENDED",
-        )
-        output_group.add_argument(
-            "--monitoring-endpoint",
-            default="",
-            required=False,
-            help="Ceph Manager prometheus exporter endpoints (comma separated list of <IP> entries of active and standby mgrs)",
-        )
-        output_group.add_argument(
-            "--monitoring-endpoint-port",
-            default="",
-            required=False,
-            help="Ceph Manager prometheus exporter port",
-        )
-        output_group.add_argument(
-            "--rbd-metadata-ec-pool-name",
-            default="",
-            required=False,
-            help="Provides the name of erasure coded RBD metadata pool",
-        )
-        output_group.add_argument(
-            "--dry-run",
-            default=False,
-            action="store_true",
-            help="Dry run prints the executed commands without running them",
-        )
-        output_group.add_argument(
-            "--rados-namespace",
-            default="",
-            required=False,
-            help="divides a pool into separate logical namespaces",
-        )
-
-        upgrade_group = argP.add_argument_group("upgrade")
-        upgrade_group.add_argument(
-            "--upgrade",
-            action="store_true",
-            default=False,
-            help="Upgrades the cephCSIKeyrings(For example: client.csi-cephfs-provisioner) with new permissions needed for the new cluster version and older permission will still be applied."
-            + "Sample run: `python3 /etc/ceph/create-external-cluster-resources.py --upgrade`, this will upgrade all the default csi users(non-restricted)"
-            + "For restricted users(For example: client.csi-cephfs-provisioner-openshift-storage-myfs), users created using --restricted-auth-permission flag need to pass mandatory flags"
-            + "mandatory flags: '--rbd-data-pool-name, --cluster-name and --run-as-user' flags while upgrading"
-            + "in case of cephfs users if you have passed --cephfs-filesystem-name flag while creating user then while upgrading it will be mandatory too"
-            + "Sample run: `python3 /etc/ceph/create-external-cluster-resources.py --upgrade --rbd-data-pool-name replicapool --cluster-name rookStorage  --run-as-user client.csi-rbd-node-rookStorage-replicapool`"
-            + "PS: An existing non-restricted user cannot be downgraded to a restricted user by upgrading. Admin need to create a new restricted user for this by re-running the script."
-            + "Upgrade flag should only be used to append new permissions to users, it shouldn't be used for changing user already applied permission, for example you shouldn't change in which pool user has access",
-        )
+        common_group = argP.add_argument_group('common')
+        common_group.add_argument("--verbose", "-v",
+                                  action='store_true', default=False)
+        common_group.add_argument("--ceph-conf", "-c",
+                                  help="Provide a ceph conf file.", type=str)
+        common_group.add_argument("--run-as-user", "-u", default="", type=str,
+                                  help="Provides a user name to check the cluster's health status, must be prefixed by 'client.'")
+        common_group.add_argument("--cluster-name", default="",
+                                  help="Ceph cluster name")
+        common_group.add_argument("--namespace", default="",
+                                  help="Namespace where CephCluster is running")
+        common_group.add_argument("--rgw-pool-prefix", default="",
+                                  help="RGW Pool prefix")
+        common_group.add_argument("--restricted-auth-permission", default=False,
+                                  help="Restricted cephCSIKeyrings auth permissions to specific pools, cluster and pool namespaces." +
+                                  "Mandatory flags that need to be set are --rbd-data-pool-name, --rados-namespace and --cluster-name." +
+                                  "--cephfs-filesystem-name flag can also be passed in case of cephfs user restriction, so it can restrict user to particular cephfs filesystem" +
+                                  "sample run: `python3 /etc/ceph/create-external-cluster-resources.py --cephfs-filesystem-name myfs --rbd-data-pool-name replicapool --rados-namespace radosNamespace --cluster-name rookStorage --restricted-auth-permission true`" +
+                                  "Note: Restricting the users per pool, per cluster and per pool namespace will require to create new users and new secrets for that users.")
+
+        output_group = argP.add_argument_group('output')
+        output_group.add_argument("--format", "-t", choices=["json", "bash"],
+                                  default='json', help="Provides the output format (json | bash)")
+        output_group.add_argument("--output", "-o", default="",
+                                  help="Output will be stored into the provided file")
+        output_group.add_argument("--cephfs-filesystem-name", default="",
+                                  help="Provides the name of the Ceph filesystem")
+        output_group.add_argument("--cephfs-metadata-pool-name", default="",
+                                  help="Provides the name of the cephfs metadata pool")
+        output_group.add_argument("--cephfs-data-pool-name", default="",
+                                  help="Provides the name of the cephfs data pool")
+        output_group.add_argument("--rbd-data-pool-name", default="", required=False,
+                                  help="Provides the name of the RBD datapool")
+        output_group.add_argument("--rgw-endpoint", default="", required=False,
+                                  help="RADOS Gateway endpoint (in <IP>:<PORT> format). Note: FQDN is also supported(in <FQDN>:<PORT> format)")
+        output_group.add_argument("--rgw-tls-cert-path", default="", required=False,
+                                  help="RADOS Gateway endpoint TLS certificate")
+        output_group.add_argument("--rgw-skip-tls", required=False, default=False,
+                                  help="Ignore TLS certification validation when a self-signed certificate is provided (NOT RECOMMENDED")
+        output_group.add_argument("--monitoring-endpoint", default="", required=False,
+                                  help="Ceph Manager prometheus exporter endpoints (comma separated list of <IP> entries of active and standby mgrs)")
+        output_group.add_argument("--monitoring-endpoint-port", default="", required=False,
+                                  help="Ceph Manager prometheus exporter port")
+        output_group.add_argument("--rbd-metadata-ec-pool-name", default="", required=False,
+                                  help="Provides the name of erasure coded RBD metadata pool")
+        output_group.add_argument("--dry-run", default=False, action='store_true',
+                                  help="Dry run prints the executed commands without running them")
+        output_group.add_argument("--rados-namespace", default="", required=False,
+                                  help="divides a pool into separate logical namespaces")
+
+        upgrade_group = argP.add_argument_group('upgrade')
+        upgrade_group.add_argument("--upgrade", action='store_true', default=False,
+                                   help="Upgrades the cephCSIKeyrings(For example: client.csi-cephfs-provisioner) with new permissions needed for the new cluster version and older permission will still be applied." +
+                                   "Sample run: `python3 /etc/ceph/create-external-cluster-resources.py --upgrade`, this will upgrade all the default csi users(non-restricted)" +
+                                   "For restricted users(For example: client.csi-cephfs-provisioner-openshift-storage-myfs), users created using --restricted-auth-permission flag need to pass mandatory flags" +
+                                   "mandatory flags: '--rbd-data-pool-name, --rados-namespace, --cluster-name and --run-as-user' flags while upgrading" +
+                                   "in case of cephfs users if you have passed --cephfs-filesystem-name flag while creating user then while upgrading it will be mandatory too" +
+                                   "Sample run: `python3 /etc/ceph/create-external-cluster-resources.py --upgrade --rbd-data-pool-name replicapool --rados-namespace radosNamespace --cluster-name rookStorage  --run-as-user client.csi-rbd-node-rookStorage-replicapool-radosNamespace`" +
+                                   "PS: An existing non-restricted user cannot be downgraded to a restricted user by upgrading. Admin need to create a new restricted user for this by re-running the script." +
+                                   "Upgrade flag should only be used to append new permissions to users, it shouldn't be used for changing user already applied permission, for example you shouldn't change in which pool user has access")
 
         if args_to_parse:
-            assert (
-                type(args_to_parse) == list
-            ), "Argument to 'gen_arg_parser' should be a list"
+            assert type(args_to_parse) == list, \
+                "Argument to 'gen_arg_parser' should be a list"
         else:
             args_to_parse = sys.argv[1:]
         return argP.parse_args(args_to_parse)
@@ -460,41 +341,32 @@ class RadosJSON:
                     "Flag '--rbd-metadata-ec-pool-name' should not be empty"
                 )
 
-            cmd_json = {"prefix": "osd dump", "format": "json"}
+            cmd_json = {
+                "prefix": "osd dump", "format": "json"
+            }
             ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
             if ret_val != 0 or len(json_out) == 0:
                 raise ExecutionFailureException(
-                    "{}".format(cmd_json["prefix"])
-                    + " command failed.\n"
-                    + "Error: {}".format(
-                        err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST
-                    )
+                    "{}".format(cmd_json['prefix']) + " command failed.\n" +
+                    "Error: {}".format(err_msg if ret_val !=
+                                       0 else self.EMPTY_OUTPUT_LIST)
                 )
             metadata_pool_exist, pool_exist = False, False
 
-            for key in json_out["pools"]:
+            for key in json_out['pools']:
                 # if erasure_code_profile is empty and pool name exists then it replica pool
-                if (
-                    key["erasure_code_profile"] == ""
-                    and key["pool_name"] == rbd_metadata_ec_pool_name
-                ):
+                if key['erasure_code_profile'] == "" and key['pool_name'] == rbd_metadata_ec_pool_name:
                     metadata_pool_exist = True
                 # if erasure_code_profile is not empty and pool name exists then it is ec pool
-                if key["erasure_code_profile"] and key["pool_name"] == rbd_pool_name:
+                if key['erasure_code_profile'] and key['pool_name'] == rbd_pool_name:
                     pool_exist = True
 
             if not metadata_pool_exist:
                 raise ExecutionFailureException(
-                    "Provided rbd_ec_metadata_pool name, {}, does not exist".format(
-                        rbd_metadata_ec_pool_name
-                    )
-                )
+                    "Provided rbd_ec_metadata_pool name, {}, does not exist".format(rbd_metadata_ec_pool_name))
             if not pool_exist:
                 raise ExecutionFailureException(
-                    "Provided rbd_data_pool name, {}, does not exist".format(
-                        rbd_pool_name
-                    )
-                )
+                    "Provided rbd_data_pool name, {}, does not exist".format(rbd_pool_name))
             return rbd_metadata_ec_pool_name
 
     def dry_run(self, msg):
@@ -503,43 +375,39 @@ class RadosJSON:
 
     def validate_rgw_endpoint_tls_cert(self):
         if self._arg_parser.rgw_tls_cert_path:
-            with open(self._arg_parser.rgw_tls_cert_path, encoding="utf8") as f:
+            with open(self._arg_parser.rgw_tls_cert_path, encoding='utf8') as f:
                 contents = f.read()
                 return contents.rstrip()
 
     def _check_conflicting_options(self):
         if not self._arg_parser.upgrade and not self._arg_parser.rbd_data_pool_name:
             raise ExecutionFailureException(
-                "Either '--upgrade' or '--rbd-data-pool-name <pool_name>' should be specified"
-            )
+                "Either '--upgrade' or '--rbd-data-pool-name <pool_name>' should be specified")
 
     def _invalid_endpoint(self, endpoint_str):
         try:
-            ipv4, port = endpoint_str.split(":")
+            ipv4, port = endpoint_str.split(':')
         except ValueError:
             raise ExecutionFailureException(
-                "Not a proper endpoint: {}, <IPv4>:<PORT>, format is expected".format(
-                    endpoint_str
-                )
-            )
-        ipParts = ipv4.split(".")
+                "Not a proper endpoint: {}, <IPv4>:<PORT>, format is expected".format(endpoint_str))
+        ipParts = ipv4.split('.')
         if len(ipParts) != 4:
-            raise ExecutionFailureException("Not a valid IP address: {}".format(ipv4))
+            raise ExecutionFailureException(
+                "Not a valid IP address: {}".format(ipv4))
         for eachPart in ipParts:
             if not eachPart.isdigit():
                 raise ExecutionFailureException(
-                    "IP address parts should be numbers: {}".format(ipv4)
-                )
+                    "IP address parts should be numbers: {}".format(ipv4))
             intPart = int(eachPart)
             if intPart < 0 or intPart > 254:
                 raise ExecutionFailureException(
-                    "Out of range IP addresses: {}".format(ipv4)
-                )
+                    "Out of range IP addresses: {}".format(ipv4))
         if not port.isdigit():
             raise ExecutionFailureException("Port not valid: {}".format(port))
         intPort = int(port)
-        if intPort < 1 or intPort > 2**16 - 1:
-            raise ExecutionFailureException("Out of range port number: {}".format(port))
+        if intPort < 1 or intPort > 2**16-1:
+            raise ExecutionFailureException(
+                "Out of range port number: {}".format(port))
         return False
 
     def endpoint_dial(self, endpoint_str, timeout=3, cert=None):
@@ -564,8 +432,7 @@ class RadosJSON:
             except:
                 continue
         raise ExecutionFailureException(
-            "unable to connect to endpoint: {}".format(endpoint_str)
-        )
+            "unable to connect to endpoint: {}".format(endpoint_str))
 
     def __init__(self, arg_list=None):
         self.out_map = {}
@@ -576,13 +443,13 @@ class RadosJSON:
         self.output_file = self._arg_parser.output
         self.ceph_conf = self._arg_parser.ceph_conf
         self.MIN_USER_CAP_PERMISSIONS = {
-            "mgr": "allow command config",
-            "mon": "allow r, allow command quorum_status, allow command version",
-            "osd": "allow rwx pool={0}.rgw.meta, "
-            + "allow r pool=.rgw.root, "
-            + "allow rw pool={0}.rgw.control, "
-            + "allow rx pool={0}.rgw.log, "
-            + "allow x pool={0}.rgw.buckets.index",
+            'mgr': 'allow command config',
+            'mon': 'allow r, allow command quorum_status, allow command version',
+            'osd': "allow rwx pool={0}.rgw.meta, " +
+            "allow r pool=.rgw.root, " +
+            "allow rw pool={0}.rgw.control, " +
+            "allow rx pool={0}.rgw.log, " +
+            "allow x pool={0}.rgw.buckets.index"
         }
         # if user not provided, give a default user
         if not self.run_as_user and not self._arg_parser.upgrade:
@@ -607,14 +474,11 @@ class RadosJSON:
 
     def _common_cmd_json_gen(self, cmd_json):
         cmd = json.dumps(cmd_json, sort_keys=True)
-        ret_val, cmd_out, err_msg = self.cluster.mon_command(cmd, b"")
+        ret_val, cmd_out, err_msg = self.cluster.mon_command(cmd, b'')
         if self._arg_parser.verbose:
             print("Command Input: {}".format(cmd))
-            print(
-                "Return Val: {}\nCommand Output: {}\nError Message: {}\n----------\n".format(
-                    ret_val, cmd_out, err_msg
-                )
-            )
+            print("Return Val: {}\nCommand Output: {}\nError Message: {}\n----------\n".format(
+                  ret_val, cmd_out, err_msg))
         json_out = {}
         # if there is no error (i.e; ret_val is ZERO) and 'cmd_out' is not empty
         # then convert 'cmd_out' to a json output
@@ -625,47 +489,43 @@ class RadosJSON:
     def get_ceph_external_mon_data(self):
         cmd_json = {"prefix": "quorum_status", "format": "json"}
         if self._arg_parser.dry_run:
-            return self.dry_run("ceph " + cmd_json["prefix"])
+            return self.dry_run("ceph " + cmd_json['prefix'])
         ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
         # if there is an unsuccessful attempt,
         if ret_val != 0 or len(json_out) == 0:
             raise ExecutionFailureException(
-                "'quorum_status' command failed.\n"
-                + "Error: {}".format(
-                    err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST
-                )
-            )
-        q_leader_name = json_out["quorum_leader_name"]
+                "'quorum_status' command failed.\n" +
+                "Error: {}".format(err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST))
+        q_leader_name = json_out['quorum_leader_name']
         q_leader_details = {}
-        q_leader_matching_list = [
-            l for l in json_out["monmap"]["mons"] if l["name"] == q_leader_name
-        ]
+        q_leader_matching_list = [l for l in json_out['monmap']['mons']
+                                  if l['name'] == q_leader_name]
         if len(q_leader_matching_list) == 0:
             raise ExecutionFailureException("No matching 'mon' details found")
         q_leader_details = q_leader_matching_list[0]
         # get the address vector of the quorum-leader
-        q_leader_addrvec = q_leader_details.get("public_addrs", {}).get("addrvec", [])
+        q_leader_addrvec = q_leader_details.get(
+            'public_addrs', {}).get('addrvec', [])
         # if the quorum-leader has only one address in the address-vector
         # and it is of type 'v2' (ie; with <IP>:3300),
         # raise an exception to make user aware that
         # they have to enable 'v1' (ie; with <IP>:6789) type as well
-        if len(q_leader_addrvec) == 1 and q_leader_addrvec[0]["type"] == "v2":
+        if len(q_leader_addrvec) == 1 and q_leader_addrvec[0]['type'] == 'v2':
             raise ExecutionFailureException(
-                "Only 'v2' address type is enabled, user should also enable 'v1' type as well"
-            )
-        ip_port = str(q_leader_details["public_addr"].split("/")[0])
+                "Only 'v2' address type is enabled, user should also enable 'v1' type as well")
+        ip_port = str(q_leader_details['public_addr'].split('/')[0])
         return "{}={}".format(str(q_leader_name), ip_port)
 
     def _join_host_port(self, endpoint, port):
         port = "{}".format(port)
         # regex to check the given endpoint is enclosed in square brackets
-        ipv6_regx = re.compile(r"^\[[^]]*\]$")
+        ipv6_regx = re.compile(r'^\[[^]]*\]$')
         # endpoint has ':' in it and if not (already) enclosed in square brackets
-        if endpoint.count(":") and not ipv6_regx.match(endpoint):
-            endpoint = "[{}]".format(endpoint)
+        if endpoint.count(':') and not ipv6_regx.match(endpoint):
+            endpoint = '[{}]'.format(endpoint)
         if not port:
             return endpoint
-        return ":".join([endpoint, port])
+        return ':'.join([endpoint, port])
 
     def _convert_hostname_to_ip(self, host_name):
         # if 'cluster' instance is a dummy type,
@@ -675,7 +535,6 @@ class RadosJSON:
         if isinstance(self.cluster, DummyRados):
             return self.cluster._convert_hostname_to_ip(host_name)
         import socket
-
         ip = socket.gethostbyname(host_name)
         del socket
         return ip
@@ -692,29 +551,23 @@ class RadosJSON:
             # if there is an unsuccessful attempt,
             if ret_val != 0 or len(json_out) == 0:
                 raise ExecutionFailureException(
-                    "'mgr services' command failed.\n"
-                    + "Error: {}".format(
-                        err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST
-                    )
-                )
-            monitoring_endpoint = (
-                json_out.get("mgrmap", {}).get("services", {}).get("prometheus", "")
-            )
+                    "'mgr services' command failed.\n" +
+                    "Error: {}".format(err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST))
+            monitoring_endpoint = json_out.get('mgrmap', {}).get(
+                'services', {}).get('prometheus', '')
             if not monitoring_endpoint:
                 raise ExecutionFailureException(
-                    "'prometheus' service not found, is the exporter enabled?'.\n"
-                )
+                    "'prometheus' service not found, is the exporter enabled?'.\n")
             # now check the stand-by mgr-s
-            standby_arr = json_out.get("mgrmap", {}).get("standbys", [])
+            standby_arr = json_out.get('mgrmap', {}).get('standbys', [])
             for each_standby in standby_arr:
-                if "name" in each_standby.keys():
-                    standby_mgrs.append(each_standby["name"])
+                if 'name' in each_standby.keys():
+                    standby_mgrs.append(each_standby['name'])
             try:
                 parsed_endpoint = urlparse(monitoring_endpoint)
             except ValueError:
                 raise ExecutionFailureException(
-                    "invalid endpoint: {}".format(monitoring_endpoint)
-                )
+                    "invalid endpoint: {}".format(monitoring_endpoint))
             monitoring_endpoint_ip_list = parsed_endpoint.hostname
             if not monitoring_endpoint_port:
                 monitoring_endpoint_port = "{}".format(parsed_endpoint.port)
@@ -724,7 +577,8 @@ class RadosJSON:
             monitoring_endpoint_port = self.DEFAULT_MONITORING_ENDPOINT_PORT
 
         # user could give comma and space separated inputs (like --monitoring-endpoint="<ip1>, <ip2>")
-        monitoring_endpoint_ip_list = monitoring_endpoint_ip_list.replace(",", " ")
+        monitoring_endpoint_ip_list = monitoring_endpoint_ip_list.replace(
+            ",", " ")
         monitoring_endpoint_ip_list_split = monitoring_endpoint_ip_list.split()
         # if monitoring-endpoint could not be found, raise an error
         if len(monitoring_endpoint_ip_list_split) == 0:
@@ -737,23 +591,19 @@ class RadosJSON:
 
         try:
             monitoring_endpoint_ip = self._convert_hostname_to_ip(
-                monitoring_endpoint_ip
-            )
+                monitoring_endpoint_ip)
             # collect all the 'stand-by' mgr ips
             mgr_ips = []
             for each_standby_mgr in standby_mgrs:
                 failed_ip = each_standby_mgr
-                mgr_ips.append(self._convert_hostname_to_ip(each_standby_mgr))
+                mgr_ips.append(
+                    self._convert_hostname_to_ip(each_standby_mgr))
         except:
             raise ExecutionFailureException(
                 "Conversion of host: {} to IP failed. "
-                "Please enter the IP addresses of all the ceph-mgrs with the '--monitoring-endpoint' flag".format(
-                    failed_ip
-                )
-            )
+                "Please enter the IP addresses of all the ceph-mgrs with the '--monitoring-endpoint' flag".format(failed_ip))
         monitoring_endpoint = self._join_host_port(
-            monitoring_endpoint_ip, monitoring_endpoint_port
-        )
+            monitoring_endpoint_ip, monitoring_endpoint_port)
         self._invalid_endpoint(monitoring_endpoint)
         self.endpoint_dial(monitoring_endpoint)
 
@@ -763,96 +613,87 @@ class RadosJSON:
         return all_mgr_ips_str, monitoring_endpoint_port
 
     def check_user_exist(self, user):
-        cmd_json = {"prefix": "auth get", "entity": "{}".format(user), "format": "json"}
+        cmd_json = {"prefix": "auth get", "entity": "{}".format(
+            user), "format": "json"}
         ret_val, json_out, _ = self._common_cmd_json_gen(cmd_json)
         if ret_val != 0 or len(json_out) == 0:
             return ""
-        return str(json_out[0]["key"])
+        return str(json_out[0]['key'])
 
     def get_cephfs_provisioner_caps_and_entity(self):
         entity = "client.csi-cephfs-provisioner"
-        caps = {
-            "mon": "allow r, allow command 'osd blocklist'",
-            "mgr": "allow rw",
-            "osd": "allow rw tag cephfs metadata=*",
-        }
+        caps = {"mon": "allow r, allow command 'osd blocklist'",
+                "mgr": "allow rw",
+                "osd": "allow rw tag cephfs metadata=*"}
         if self._arg_parser.restricted_auth_permission:
             cluster_name = self._arg_parser.cluster_name
             if cluster_name == "":
                 raise ExecutionFailureException(
-                    "cluster_name not found, please set the '--cluster-name' flag"
-                )
+                    "cluster_name not found, please set the '--cluster-name' flag")
             cephfs_filesystem = self._arg_parser.cephfs_filesystem_name
             if cephfs_filesystem == "":
                 entity = "{}-{}".format(entity, cluster_name)
             else:
-                entity = "{}-{}-{}".format(entity, cluster_name, cephfs_filesystem)
+                entity = "{}-{}-{}".format(entity,
+                                           cluster_name, cephfs_filesystem)
                 caps["osd"] = "allow rw tag cephfs metadata={}".format(
-                    cephfs_filesystem
-                )
+                    cephfs_filesystem)
 
         return caps, entity
 
     def get_cephfs_node_caps_and_entity(self):
         entity = "client.csi-cephfs-node"
-        caps = {
-            "mon": "allow r, allow command 'osd blocklist'",
-            "mgr": "allow rw",
-            "osd": "allow rw tag cephfs *=*",
-            "mds": "allow rw",
-        }
+        caps = {"mon": "allow r, allow command 'osd blocklist'",
+                "mgr": "allow rw",
+                "osd": "allow rw tag cephfs *=*",
+                "mds": "allow rw"}
         if self._arg_parser.restricted_auth_permission:
             cluster_name = self._arg_parser.cluster_name
             if cluster_name == "":
                 raise ExecutionFailureException(
-                    "cluster_name not found, please set the '--cluster-name' flag"
-                )
+                    "cluster_name not found, please set the '--cluster-name' flag")
             cephfs_filesystem = self._arg_parser.cephfs_filesystem_name
             if cephfs_filesystem == "":
                 entity = "{}-{}".format(entity, cluster_name)
             else:
-                entity = "{}-{}-{}".format(entity, cluster_name, cephfs_filesystem)
-                caps["osd"] = "allow rw tag cephfs *={}".format(cephfs_filesystem)
+                entity = "{}-{}-{}".format(entity,
+                                           cluster_name, cephfs_filesystem)
+                caps["osd"] = "allow rw tag cephfs *={}".format(
+                    cephfs_filesystem)
 
         return caps, entity
 
     def get_rbd_provisioner_caps_and_entity(self):
         entity = "client.csi-rbd-provisioner"
-        caps = {
-            "mon": "profile rbd, allow command 'osd blocklist'",
-            "mgr": "allow rw",
-            "osd": "profile rbd",
-        }
+        caps = {"mon": "profile rbd, allow command 'osd blocklist'",
+                "mgr": "allow rw",
+                "osd": "profile rbd"}
         if self._arg_parser.restricted_auth_permission:
             rbd_pool_name = self._arg_parser.rbd_data_pool_name
             cluster_name = self._arg_parser.cluster_name
-            if rbd_pool_name == "" or cluster_name == "":
+            rados_namespace = self._arg_parser.rados_namespace
+            if rbd_pool_name == "" or cluster_name == "" or rados_namespace == "":
                 raise ExecutionFailureException(
-                    "mandatory flags not found, please set the '--rbd-data-pool-name', '--cluster-name' flags"
-                )
-            entity = "{}-{}-{}".format(
-                entity, cluster_name, rbd_pool_name
-            )
+                    "mandatory flags not found, please set the '--rbd-data-pool-name', '--cluster-name' and --rados-namespace flags")
+            entity = "{}-{}-{}-{}".format(entity, cluster_name,
+                                          rbd_pool_name, rados_namespace)
             caps["osd"] = "profile rbd pool={}".format(rbd_pool_name)
 
         return caps, entity
 
     def get_rbd_node_caps_and_entity(self):
         entity = "client.csi-rbd-node"
-        caps = {
-            "mon": "profile rbd, allow command 'osd blocklist'",
-            "osd": "profile rbd",
-        }
+        caps = {"mon": "profile rbd, allow command 'osd blocklist'",
+                "osd": "profile rbd"}
         if self._arg_parser.restricted_auth_permission:
             rbd_pool_name = self._arg_parser.rbd_data_pool_name
             cluster_name = self._arg_parser.cluster_name
-            if rbd_pool_name == "" or cluster_name == "":
+            rados_namespace = self._arg_parser.rados_namespace
+            if rbd_pool_name == "" or cluster_name == "" or rados_namespace == "":
                 raise ExecutionFailureException(
-                    "mandatory flags not found, please set the '--rbd-data-pool-name', '--cluster-name' flags"
-                )
-            entity = "{}-{}-{}".format(
-                entity, cluster_name, rbd_pool_name
-            )
+                    "mandatory flags not found, please set the '--rbd-data-pool-name', '--cluster-name' and --rados-namespace flags")
+            entity = "{}-{}-{}-{}".format(entity, cluster_name,
+                                          rbd_pool_name, rados_namespace)
             caps["osd"] = "profile rbd pool={}".format(rbd_pool_name)
 
         return caps, entity
@@ -877,87 +718,63 @@ class RadosJSON:
 
         raise ExecutionFailureException(
             "no user found with user_name: {} ,".format(user_name)
-            + "get_caps_and_entity command failed.\n"
-        )
+            + "get_caps_and_entity command failed.\n")
 
     def create_cephCSIKeyring_user(self, user):
-        """
+        '''
         command: ceph auth get-or-create client.csi-cephfs-provisioner mon 'allow r' mgr 'allow rw' osd 'allow rw tag cephfs metadata=*'
-        """
+        '''
         caps, entity = self.get_caps_and_entity(user)
-        cmd_json = {
-            "prefix": "auth get-or-create",
-            "entity": entity,
-            "caps": [cap for cap_list in list(caps.items()) for cap in cap_list],
-            "format": "json",
-        }
+        cmd_json = {"prefix": "auth get-or-create",
+                    "entity": entity,
+                    "caps": [cap for cap_list in list(caps.items()) for cap in cap_list],
+                    "format": "json"}
 
         if self._arg_parser.dry_run:
-            return self.dry_run(
-                "ceph "
-                + cmd_json["prefix"]
-                + " "
-                + cmd_json["entity"]
-                + " "
-                + " ".join(cmd_json["caps"])
-            ), ""
+            return self.dry_run("ceph " + cmd_json['prefix'] + " " + cmd_json['entity'] + " " + " ".join(cmd_json['caps']))
         # check if user already exist
         user_key = self.check_user_exist(entity)
         if user_key != "":
-            return user_key, "{}".format(entity.split('.',1)[1])  # entity.split('.',1)[1] to rename entity(client.csi-rbd-node) as csi-rbd-node
+            return user_key
 
         ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
         # if there is an unsuccessful attempt,
         if ret_val != 0 or len(json_out) == 0:
             raise ExecutionFailureException(
-                "'auth get-or-create {}' command failed.\n".format(user)
-                + "Error: {}".format(
-                    err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST
-                )
-            )
-        return str(json_out[0]["key"]), "{}".format(entity.split('.',1)[1])  # entity.split('.',1)[1] to rename entity(client.csi-rbd-node) as csi-rbd-node
+                "'auth get-or-create {}' command failed.\n".format(user) +
+                "Error: {}".format(err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST))
+        return str(json_out[0]['key'])
 
     def get_cephfs_data_pool_details(self):
         cmd_json = {"prefix": "fs ls", "format": "json"}
         if self._arg_parser.dry_run:
-            return self.dry_run("ceph " + cmd_json["prefix"])
+            return self.dry_run("ceph " + cmd_json['prefix'])
         ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
         # if there is an unsuccessful attempt, report an error
         if ret_val != 0:
             # if fs and data_pool arguments are not set, silently return
-            if (
-                self._arg_parser.cephfs_filesystem_name == ""
-                and self._arg_parser.cephfs_data_pool_name == ""
-            ):
+            if self._arg_parser.cephfs_filesystem_name == "" and self._arg_parser.cephfs_data_pool_name == "":
                 return
             # if user has provided any of the
             # '--cephfs-filesystem-name' or '--cephfs-data-pool-name' arguments,
             # raise an exception as we are unable to verify the args
             raise ExecutionFailureException(
-                "'fs ls' ceph call failed with error: {}".format(err_msg)
-            )
+                "'fs ls' ceph call failed with error: {}".format(err_msg))
 
         matching_json_out = {}
         # if '--cephfs-filesystem-name' argument is provided,
         # check whether the provided filesystem-name exists or not
         if self._arg_parser.cephfs_filesystem_name:
             # get the matching list
-            matching_json_out_list = [
-                matched
-                for matched in json_out
-                if str(matched["name"]) == self._arg_parser.cephfs_filesystem_name
-            ]
+            matching_json_out_list = [matched for matched in json_out
+                                      if str(matched['name']) == self._arg_parser.cephfs_filesystem_name]
             # unable to find a matching fs-name, raise an error
             if len(matching_json_out_list) == 0:
                 raise ExecutionFailureException(
-                    (
-                        "Filesystem provided, '{}', "
-                        + "is not found in the fs-list: '{}'"
-                    ).format(
+                    ("Filesystem provided, '{}', " +
+                     "is not found in the fs-list: '{}'").format(
                         self._arg_parser.cephfs_filesystem_name,
-                        [str(x["name"]) for x in json_out],
-                    )
-                )
+                        [str(x['name']) for x in json_out]))
             matching_json_out = matching_json_out_list[0]
         # if cephfs filesystem name is not provided,
         # try to get a default fs name by doing the following
@@ -969,201 +786,139 @@ class RadosJSON:
             elif self._arg_parser.cephfs_data_pool_name:
                 # and if present, check whether there exists a fs which has the data_pool
                 for eachJ in json_out:
-                    if self._arg_parser.cephfs_data_pool_name in eachJ["data_pools"]:
+                    if self._arg_parser.cephfs_data_pool_name in eachJ['data_pools']:
                         matching_json_out = eachJ
                         break
                 # if there is no matching fs exists, that means provided data_pool name is invalid
                 if not matching_json_out:
                     raise ExecutionFailureException(
                         "Provided data_pool name, {}, does not exists".format(
-                            self._arg_parser.cephfs_data_pool_name
-                        )
-                    )
+                            self._arg_parser.cephfs_data_pool_name))
             # c. if nothing is set and couldn't find a default,
             else:
                 # just return silently
                 return
 
         if matching_json_out:
-            self._arg_parser.cephfs_filesystem_name = str(matching_json_out["name"])
+            self._arg_parser.cephfs_filesystem_name = str(
+                matching_json_out['name'])
             self._arg_parser.cephfs_metadata_pool_name = str(
-                matching_json_out["metadata_pool"]
-            )
+                matching_json_out['metadata_pool'])
 
-        if type(matching_json_out["data_pools"]) == list:
+        if type(matching_json_out['data_pools']) == list:
             # if the user has already provided data-pool-name,
             # through --cephfs-data-pool-name
             if self._arg_parser.cephfs_data_pool_name:
                 # if the provided name is not matching with the one in the list
-                if (
-                    self._arg_parser.cephfs_data_pool_name
-                    not in matching_json_out["data_pools"]
-                ):
+                if self._arg_parser.cephfs_data_pool_name not in matching_json_out['data_pools']:
                     raise ExecutionFailureException(
                         "{}: '{}', {}: {}".format(
                             "Provided data-pool-name",
                             self._arg_parser.cephfs_data_pool_name,
                             "doesn't match from the data-pools' list",
-                            [str(x) for x in matching_json_out["data_pools"]],
-                        )
-                    )
+                            [str(x) for x in matching_json_out['data_pools']]))
             # if data_pool name is not provided,
             # then try to find a default data pool name
             else:
                 # if no data_pools exist, silently return
-                if len(matching_json_out["data_pools"]) == 0:
+                if len(matching_json_out['data_pools']) == 0:
                     return
                 self._arg_parser.cephfs_data_pool_name = str(
-                    matching_json_out["data_pools"][0]
-                )
+                    matching_json_out['data_pools'][0])
             # if there are more than one 'data_pools' exist,
             # then warn the user that we are using the selected name
-            if len(matching_json_out["data_pools"]) > 1:
-                print(
-                    "{}: {}\n{}: '{}'\n".format(
-                        "WARNING: Multiple data pools detected",
-                        [str(x) for x in matching_json_out["data_pools"]],
-                        "Using the data-pool",
-                        self._arg_parser.cephfs_data_pool_name,
-                    )
-                )
+            if len(matching_json_out['data_pools']) > 1:
+                print("{}: {}\n{}: '{}'\n".format(
+                    "WARNING: Multiple data pools detected",
+                    [str(x) for x in matching_json_out['data_pools']],
+                    "Using the data-pool",
+                    self._arg_parser.cephfs_data_pool_name))
 
     def create_checkerKey(self):
-        cmd_json = {
-            "prefix": "auth get-or-create",
-            "entity": self.run_as_user,
-            "caps": [
-                "mon",
-                self.MIN_USER_CAP_PERMISSIONS["mon"],
-                "mgr",
-                self.MIN_USER_CAP_PERMISSIONS["mgr"],
-                "osd",
-                self.MIN_USER_CAP_PERMISSIONS["osd"].format(
-                    self._arg_parser.rgw_pool_prefix
-                ),
-            ],
-            "format": "json",
-        }
+        cmd_json = {"prefix": "auth get-or-create",
+                    "entity": self.run_as_user,
+                    "caps": ["mon", self.MIN_USER_CAP_PERMISSIONS['mon'],
+                             "mgr", self.MIN_USER_CAP_PERMISSIONS['mgr'],
+                             "osd", self.MIN_USER_CAP_PERMISSIONS['osd'].format(self._arg_parser.rgw_pool_prefix)],
+                    "format": "json"}
         if self._arg_parser.dry_run:
-            return self.dry_run(
-                "ceph "
-                + cmd_json["prefix"]
-                + " "
-                + cmd_json["entity"]
-                + " "
-                + " ".join(cmd_json["caps"])
-            )
+            return self.dry_run("ceph " + cmd_json['prefix'] + " " + cmd_json['entity'] + " " + " ".join(cmd_json['caps']))
         ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
         # if there is an unsuccessful attempt,
         if ret_val != 0 or len(json_out) == 0:
             raise ExecutionFailureException(
-                "'auth get-or-create {}' command failed\n".format(self.run_as_user)
-                + "Error: {}".format(
-                    err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST
-                )
-            )
-        return str(json_out[0]["key"])
+                "'auth get-or-create {}' command failed\n".format(self.run_as_user) +
+                "Error: {}".format(err_msg if ret_val != 0 else self.EMPTY_OUTPUT_LIST))
+        return str(json_out[0]['key'])
 
     def get_ceph_dashboard_link(self):
         cmd_json = {"prefix": "mgr services", "format": "json"}
         if self._arg_parser.dry_run:
-            return self.dry_run("ceph " + cmd_json["prefix"])
+            return self.dry_run("ceph " + cmd_json['prefix'])
         ret_val, json_out, _ = self._common_cmd_json_gen(cmd_json)
         # if there is an unsuccessful attempt,
         if ret_val != 0 or len(json_out) == 0:
             return None
-        if not "dashboard" in json_out:
+        if not 'dashboard' in json_out:
             return None
-        return json_out["dashboard"]
+        return json_out['dashboard']
 
     def create_rgw_admin_ops_user(self):
-        cmd = [
-            "radosgw-admin",
-            "user",
-            "create",
-            "--uid",
-            self.EXTERNAL_RGW_ADMIN_OPS_USER_NAME,
-            "--display-name",
-            "Rook RGW Admin Ops user",
-            "--caps",
-            "info=read;buckets=*;users=*;usage=read;metadata=read;zone=read",
-        ]
+        cmd = ['radosgw-admin', 'user', 'create', '--uid', self.EXTERNAL_RGW_ADMIN_OPS_USER_NAME, '--display-name',
+               'Rook RGW Admin Ops user', '--caps', 'info=read;buckets=*;users=*;usage=read;metadata=read;zone=read']
         if self._arg_parser.dry_run:
             return self.dry_run("ceph " + " ".join(cmd))
         try:
-            output = subprocess.check_output(cmd, stderr=subprocess.PIPE)
+            output = subprocess.check_output(cmd,
+                                             stderr=subprocess.PIPE)
         except subprocess.CalledProcessError as execErr:
             # if the user already exists, we just query it
             if execErr.returncode == errno.EEXIST:
-                cmd = [
-                    "radosgw-admin",
-                    "user",
-                    "info",
-                    "--uid",
-                    self.EXTERNAL_RGW_ADMIN_OPS_USER_NAME,
-                ]
+                cmd = ['radosgw-admin', 'user', 'info',
+                       '--uid', self.EXTERNAL_RGW_ADMIN_OPS_USER_NAME
+                       ]
                 try:
-                    output = subprocess.check_output(cmd, stderr=subprocess.PIPE)
+                    output = subprocess.check_output(cmd,
+                                                     stderr=subprocess.PIPE)
                 except subprocess.CalledProcessError as execErr:
-                    err_msg = (
-                        "failed to execute command %s. Output: %s. Code: %s. Error: %s"
-                        % (cmd, execErr.output, execErr.returncode, execErr.stderr)
-                    )
+                    err_msg = "failed to execute command %s. Output: %s. Code: %s. Error: %s" % (
+                        cmd, execErr.output, execErr.returncode, execErr.stderr)
                     raise Exception(err_msg)
             else:
-                err_msg = (
-                    "failed to execute command %s. Output: %s. Code: %s. Error: %s"
-                    % (cmd, execErr.output, execErr.returncode, execErr.stderr)
-                )
+                err_msg = "failed to execute command %s. Output: %s. Code: %s. Error: %s" % (
+                    cmd, execErr.output, execErr.returncode, execErr.stderr)
                 raise Exception(err_msg)
 
+
         # separately add info=read caps, because sometimes users already exited and the cap doesn't update
         info_cap_supported = True
-        cmd = [
-            "radosgw-admin",
-            "caps",
-            "add",
-            "--uid",
-            self.EXTERNAL_RGW_ADMIN_OPS_USER_NAME,
-            "--caps",
-            "info=read",
-        ]
+        cmd = ['radosgw-admin', 'caps', 'add', '--uid', self.EXTERNAL_RGW_ADMIN_OPS_USER_NAME,
+            '--caps', 'info=read']
         try:
-            output = subprocess.check_output(cmd, stderr=subprocess.PIPE)
+            output = subprocess.check_output(cmd,
+                                            stderr=subprocess.PIPE)
         except subprocess.CalledProcessError as execErr:
             # if the ceph version not supported for adding `info=read` cap(rgw_validation)
-            if (
-                "could not add caps: unable to add caps: info=read\n"
-                in execErr.stderr.decode("utf-8")
-                and execErr.returncode == 244
-            ):
+            if 'could not add caps: unable to add caps: info=read\n' in execErr.stderr.decode("utf-8") and execErr.returncode == 244:
                 info_cap_supported = False
             else:
-                err_msg = (
-                    "failed to execute command %s. Output: %s. Code: %s. Error: %s"
-                    % (cmd, execErr.output, execErr.returncode, execErr.stderr)
-                )
+                err_msg = "failed to execute command %s. Output: %s. Code: %s. Error: %s" % (
+                        cmd, execErr.output, execErr.returncode, execErr.stderr)
                 raise Exception(err_msg)
 
         jsonoutput = json.loads(output)
-        return (
-            jsonoutput["keys"][0]["access_key"],
-            jsonoutput["keys"][0]["secret_key"],
-            info_cap_supported,
-        )
+        return jsonoutput["keys"][0]['access_key'], jsonoutput["keys"][0]['secret_key'], info_cap_supported
 
-    def convert_fqdn_rgw_endpoint_to_ip(self, fqdn_rgw_endpoint):
+    def convert_fqdn_rgw_endpoint_to_ip(self,fqdn_rgw_endpoint):
         try:
-            fqdn, port = fqdn_rgw_endpoint.split(":")
+            fqdn, port = fqdn_rgw_endpoint.split(':')
         except ValueError:
             raise ExecutionFailureException(
-                "Not a proper endpoint: {}, <FQDN>:<PORT>, format is expected".format(
-                    fqdn_rgw_endpoint
-                )
-            )
+            "Not a proper endpoint: {}, <FQDN>:<PORT>, format is expected".format(fqdn_rgw_endpoint))
         rgw_endpoint_ip = self._convert_hostname_to_ip(fqdn)
         rgw_endpoint_port = port
-        rgw_endpoint = self._join_host_port(rgw_endpoint_ip, rgw_endpoint_port)
+        rgw_endpoint = self._join_host_port(
+        rgw_endpoint_ip, rgw_endpoint_port)
         return rgw_endpoint
 
     def validate_pool(self):
@@ -1172,22 +927,20 @@ class RadosJSON:
         if self._arg_parser.rgw_endpoint:
             rgw_endpoint = self._arg_parser.rgw_endpoint
             self._invalid_endpoint(rgw_endpoint)
-            self.endpoint_dial(rgw_endpoint, cert=self.validate_rgw_endpoint_tls_cert())
+            self.endpoint_dial(rgw_endpoint,
+                               cert=self.validate_rgw_endpoint_tls_cert())
             # only validate if rgw_pool_prefix is passed else it will take default value and we don't create these default pools
             if self._arg_parser.rgw_pool_prefix != "default":
-                rgw_pool_to_validate = [
-                    "{0}.rgw.meta".format(self._arg_parser.rgw_pool_prefix),
-                    ".rgw.root",
-                    "{0}.rgw.control".format(self._arg_parser.rgw_pool_prefix),
-                    "{0}.rgw.log".format(self._arg_parser.rgw_pool_prefix),
-                ]
+                rgw_pool_to_validate = ["{0}.rgw.meta".format(self._arg_parser.rgw_pool_prefix),
+                                        ".rgw.root",
+                                        "{0}.rgw.control".format(self._arg_parser.rgw_pool_prefix),
+                                        "{0}.rgw.log".format(self._arg_parser.rgw_pool_prefix)]
                 pools_to_validate.extend(rgw_pool_to_validate)
 
         for pool in pools_to_validate:
             if not self.cluster.pool_exists(pool):
                 raise ExecutionFailureException(
-                    "The provided pool, '{}', does not exist".format(pool)
-                )
+                    "The provided pool, '{}', does not exist".format(pool))
 
     def validate_rados_namespace(self):
         rbd_pool_name = self._arg_parser.rbd_data_pool_name
@@ -1198,14 +951,12 @@ class RadosJSON:
         ioctx = self.cluster.open_ioctx(rbd_pool_name)
         if rbd_inst.namespace_exists(ioctx, rados_namespace) == False:
             raise ExecutionFailureException(
-                (
-                    "The provided rados Namespace, '{}', is not found in the pool '{}'"
-                ).format(rados_namespace, rbd_pool_name)
-            )
+                ("The provided rados Namespace, '{}', is not found in the pool '{}'").format(
+                    rados_namespace, rbd_pool_name))
 
     def get_rgw_fsid(self):
-        access_key = self.out_map["RGW_ADMIN_OPS_USER_ACCESS_KEY"]
-        secret_key = self.out_map["RGW_ADMIN_OPS_USER_SECRET_KEY"]
+        access_key = self.out_map['RGW_ADMIN_OPS_USER_ACCESS_KEY']
+        secret_key = self.out_map['RGW_ADMIN_OPS_USER_SECRET_KEY']
         rgw_endpoint = self._arg_parser.rgw_endpoint
         cert = None
         verify = None
@@ -1214,29 +965,21 @@ class RadosJSON:
             verify = True
         if self._arg_parser.rgw_skip_tls:
             verify = False
-        base_url = self.endpoint_dial(rgw_endpoint, cert=cert) + "://"
+        base_url =  self.endpoint_dial(rgw_endpoint,cert=cert) + "://"
         base_url = base_url + rgw_endpoint + "/admin/info?"
-        params = {"format": "json"}
+        params = {'format': 'json'}
         request_url = base_url + urllib.parse.urlencode(params)
 
         try:
-            r = requests.get(
-                request_url,
-                auth=S3Auth(access_key, secret_key, rgw_endpoint),
-                cert=cert,
-                verify=verify,
-            )
+            r = requests.get(request_url, auth=S3Auth(access_key, secret_key,rgw_endpoint), cert=cert, verify=verify)
         except requests.exceptions.Timeout:
             raise ExecutionFailureException(
-                "invalid endpoint:, not able to call admin-ops api{}".format(
-                    rgw_endpoint
-                )
-            )
+                    "invalid endpoint:, not able to call admin-ops api{}".format(rgw_endpoint))
         r1 = r.json()
-        if r1 is None or r1.get("info") is None:
+        if r1 is None or r1.get('info') is None:
             return ""  # Invalid rgw-endpoint exception will returned by validate_rgw_endpoint()
 
-        return r1["info"]["storage_backends"][0]["cluster_id"]
+        return r1['info']['storage_backends'][0]['cluster_id']
 
     def validate_rgw_endpoint(self):
         # if the 'cluster' instance is a dummy one,
@@ -1247,84 +990,65 @@ class RadosJSON:
         rgw_fsid = self.get_rgw_fsid()
         if fsid != rgw_fsid:
             raise ExecutionFailureException(
-                (
-                    "The provided rgw Endpoint, '{}', is invalid. We are validating by calling the adminops api through rgw-endpoint and validating the cluster_id '{}' is equal to the ceph cluster fsid '{}'"
-                ).format(self._arg_parser.rgw_endpoint, rgw_fsid, fsid)
-            )
+                ("The provided rgw Endpoint, '{}', is invalid. We are validating by calling the adminops api through rgw-endpoint and validating the cluster_id '{}' is equal to the ceph cluster fsid '{}'").format(
+                    self._arg_parser.rgw_endpoint,rgw_fsid,fsid))
 
     def _gen_output_map(self):
         if self.out_map:
             return
-        self._arg_parser.cluster_name = self._arg_parser.cluster_name.lower() # always convert cluster name to lowercase characters
         if self._arg_parser.rgw_endpoint:
-            self._arg_parser.rgw_endpoint = self.convert_fqdn_rgw_endpoint_to_ip(
-                self._arg_parser.rgw_endpoint
-            )
+            self._arg_parser.rgw_endpoint = self.convert_fqdn_rgw_endpoint_to_ip(self._arg_parser.rgw_endpoint)
         self.validate_pool()
         self.validate_rados_namespace()
-        self._excluded_keys.add("CLUSTER_NAME")
+        self._excluded_keys.add('CLUSTER_NAME')
         self.get_cephfs_data_pool_details()
-        self.out_map["NAMESPACE"] = self._arg_parser.namespace
-        self.out_map["CLUSTER_NAME"] = self._arg_parser.cluster_name
-        self.out_map["ROOK_EXTERNAL_FSID"] = self.get_fsid()
-        self.out_map["ROOK_EXTERNAL_USERNAME"] = self.run_as_user
-        self.out_map["ROOK_EXTERNAL_CEPH_MON_DATA"] = self.get_ceph_external_mon_data()
-        self.out_map["ROOK_EXTERNAL_USER_SECRET"] = self.create_checkerKey()
-        self.out_map["ROOK_EXTERNAL_DASHBOARD_LINK"] = self.get_ceph_dashboard_link()
-        self.out_map["CSI_RBD_NODE_SECRET"], self.out_map['CSI_RBD_NODE_SECRET_NAME'] = self.create_cephCSIKeyring_user(
-            "client.csi-rbd-node"
-        )
-        self.out_map["CSI_RBD_PROVISIONER_SECRET"], self.out_map['CSI_RBD_PROVISIONER_SECRET_NAME'] = self.create_cephCSIKeyring_user(
-            "client.csi-rbd-provisioner"
-        )
-        self.out_map["CEPHFS_POOL_NAME"] = self._arg_parser.cephfs_data_pool_name
-        self.out_map[
-            "CEPHFS_METADATA_POOL_NAME"
-        ] = self._arg_parser.cephfs_metadata_pool_name
-        self.out_map["CEPHFS_FS_NAME"] = self._arg_parser.cephfs_filesystem_name
-        self.out_map[
-            "RESTRICTED_AUTH_PERMISSION"
-        ] = self._arg_parser.restricted_auth_permission
-        self.out_map["RADOS_NAMESPACE"] = self._arg_parser.rados_namespace
-        self.out_map["CSI_CEPHFS_NODE_SECRET"] = ""
-        self.out_map["CSI_CEPHFS_PROVISIONER_SECRET"] = ""
+        self.out_map['NAMESPACE'] = self._arg_parser.namespace
+        self.out_map['CLUSTER_NAME'] = self._arg_parser.cluster_name
+        self.out_map['ROOK_EXTERNAL_FSID'] = self.get_fsid()
+        self.out_map['ROOK_EXTERNAL_USERNAME'] = self.run_as_user
+        self.out_map['ROOK_EXTERNAL_CEPH_MON_DATA'] = self.get_ceph_external_mon_data()
+        self.out_map['ROOK_EXTERNAL_USER_SECRET'] = self.create_checkerKey()
+        self.out_map['ROOK_EXTERNAL_DASHBOARD_LINK'] = self.get_ceph_dashboard_link()
+        self.out_map['CSI_RBD_NODE_SECRET'] = self.create_cephCSIKeyring_user(
+            "client.csi-rbd-node")
+        self.out_map['CSI_RBD_PROVISIONER_SECRET'] = self.create_cephCSIKeyring_user(
+            "client.csi-rbd-provisioner")
+        self.out_map['CEPHFS_POOL_NAME'] = self._arg_parser.cephfs_data_pool_name
+        self.out_map['CEPHFS_METADATA_POOL_NAME'] = self._arg_parser.cephfs_metadata_pool_name
+        self.out_map['CEPHFS_FS_NAME'] = self._arg_parser.cephfs_filesystem_name
+        self.out_map['RESTRICTED_AUTH_PERMISSION'] = self._arg_parser.restricted_auth_permission
+        self.out_map['RADOS_NAMESPACE'] = self._arg_parser.rados_namespace
+        self.out_map['CSI_CEPHFS_NODE_SECRET'] = ''
+        self.out_map['CSI_CEPHFS_PROVISIONER_SECRET'] = ''
         # create CephFS node and provisioner keyring only when MDS exists
-        if self.out_map["CEPHFS_FS_NAME"] and self.out_map["CEPHFS_POOL_NAME"]:
-            self.out_map["CSI_CEPHFS_NODE_SECRET"], self.out_map['CSI_CEPHFS_NODE_SECRET_NAME'] = self.create_cephCSIKeyring_user(
-                "client.csi-cephfs-node"
-            )
-            self.out_map["CSI_CEPHFS_PROVISIONER_SECRET"], self.out_map['CSI_CEPHFS_PROVISIONER_SECRET_NAME'] = self.create_cephCSIKeyring_user("client.csi-cephfs-provisioner")
-        self.out_map["RGW_ENDPOINT"] = self._arg_parser.rgw_endpoint
-        self.out_map["RGW_TLS_CERT"] = ""
-        (
-            self.out_map["MONITORING_ENDPOINT"],
-            self.out_map["MONITORING_ENDPOINT_PORT"],
-        ) = self.get_active_and_standby_mgrs()
-        self.out_map["RBD_POOL_NAME"] = self._arg_parser.rbd_data_pool_name
-        self.out_map[
-            "RBD_METADATA_EC_POOL_NAME"
-        ] = self.validate_rgw_metadata_ec_pool_name()
-        self.out_map["RGW_POOL_PREFIX"] = self._arg_parser.rgw_pool_prefix
+        if self.out_map['CEPHFS_FS_NAME'] and self.out_map['CEPHFS_POOL_NAME']:
+            self.out_map['CSI_CEPHFS_NODE_SECRET'] = self.create_cephCSIKeyring_user(
+                "client.csi-cephfs-node")
+            self.out_map['CSI_CEPHFS_PROVISIONER_SECRET'] = self.create_cephCSIKeyring_user(
+                "client.csi-cephfs-provisioner")
+        self.out_map['RGW_ENDPOINT'] = self._arg_parser.rgw_endpoint
+        self.out_map['RGW_TLS_CERT'] = ''
+        self.out_map['MONITORING_ENDPOINT'], \
+            self.out_map['MONITORING_ENDPOINT_PORT'] = self.get_active_and_standby_mgrs()
+        self.out_map['RBD_POOL_NAME'] = self._arg_parser.rbd_data_pool_name
+        self.out_map['RBD_METADATA_EC_POOL_NAME'] = self.validate_rgw_metadata_ec_pool_name()
+        self.out_map['RGW_POOL_PREFIX'] = self._arg_parser.rgw_pool_prefix
         if self._arg_parser.rgw_endpoint:
             if self._arg_parser.dry_run:
                 self.create_rgw_admin_ops_user()
             else:
-                (
-                    self.out_map["RGW_ADMIN_OPS_USER_ACCESS_KEY"],
-                    self.out_map["RGW_ADMIN_OPS_USER_SECRET_KEY"],
-                    info_cap_supported,
-                ) = self.create_rgw_admin_ops_user()
+                self.out_map['RGW_ADMIN_OPS_USER_ACCESS_KEY'], self.out_map['RGW_ADMIN_OPS_USER_SECRET_KEY'], info_cap_supported = self.create_rgw_admin_ops_user()
                 if info_cap_supported:
                     self.validate_rgw_endpoint()
             if self._arg_parser.rgw_tls_cert_path:
-                self.out_map["RGW_TLS_CERT"] = self.validate_rgw_endpoint_tls_cert()
+                self.out_map['RGW_TLS_CERT'] = self.validate_rgw_endpoint_tls_cert()
 
     def gen_shell_out(self):
         self._gen_output_map()
         shOutIO = StringIO()
         for k, v in self.out_map.items():
             if v and k not in self._excluded_keys:
-                shOutIO.write("export {}={}{}".format(k, v, LINESEP))
+                shOutIO.write('export {}={}{}'.format(k, v, LINESEP))
         shOut = shOutIO.getvalue()
         shOutIO.close()
         return shOut
@@ -1338,112 +1062,165 @@ class RadosJSON:
                 "name": "rook-ceph-mon-endpoints",
                 "kind": "ConfigMap",
                 "data": {
-                    "data": self.out_map["ROOK_EXTERNAL_CEPH_MON_DATA"],
+                    "data": self.out_map['ROOK_EXTERNAL_CEPH_MON_DATA'],
                     "maxMonId": "0",
-                    "mapping": "{}",
-                },
+                    "mapping": "{}"
+                }
             },
             {
                 "name": "rook-ceph-mon",
                 "kind": "Secret",
                 "data": {
                     "admin-secret": "admin-secret",
-                    "fsid": self.out_map["ROOK_EXTERNAL_FSID"],
-                    "mon-secret": "mon-secret",
+                    "fsid": self.out_map['ROOK_EXTERNAL_FSID'],
+                    "mon-secret": "mon-secret"
                 },
             },
             {
                 "name": "rook-ceph-operator-creds",
                 "kind": "Secret",
                 "data": {
-                    "userID": self.out_map["ROOK_EXTERNAL_USERNAME"],
-                    "userKey": self.out_map["ROOK_EXTERNAL_USER_SECRET"],
-                },
+                    "userID": self.out_map['ROOK_EXTERNAL_USERNAME'],
+                    "userKey": self.out_map['ROOK_EXTERNAL_USER_SECRET']
+                }
             },
             {
                 "name": "monitoring-endpoint",
                 "kind": "CephCluster",
                 "data": {
-                    "MonitoringEndpoint": self.out_map["MONITORING_ENDPOINT"],
-                    "MonitoringPort": self.out_map["MONITORING_ENDPOINT_PORT"],
-                },
-            },
+                    "MonitoringEndpoint": self.out_map['MONITORING_ENDPOINT'],
+                    "MonitoringPort": self.out_map['MONITORING_ENDPOINT_PORT']
+                }
+            }
         ]
 
-        # if 'CSI_RBD_NODE_SECRET' exists, then only add 'rook-csi-rbd-provisioner' Secret
-        if self.out_map['CSI_RBD_NODE_SECRET'] and self.out_map["CSI_RBD_NODE_SECRET_NAME"]:
-            json_out.append({
-                "name": "rook-{}".format(self.out_map["CSI_RBD_NODE_SECRET_NAME"]),
-                "kind": "Secret",
-                "data": {
-                    "userID": self.out_map["CSI_RBD_NODE_SECRET_NAME"],
-                    "userKey": self.out_map['CSI_RBD_NODE_SECRET']
-                }
-            })
-        # if 'CSI_RBD_PROVISIONER_SECRET' exists, then only add 'rook-csi-rbd-provisioner' Secret
-        if self.out_map['CSI_RBD_PROVISIONER_SECRET'] and self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"]:
+        if self.out_map['RBD_METADATA_EC_POOL_NAME']:
             json_out.append({
-                "name": "rook-{}".format(self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"]),
-                "kind": "Secret",
+                "name": "ceph-rbd",
+                "kind": "StorageClass",
                 "data": {
-                    "userID": self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"],
-                    "userKey": self.out_map['CSI_RBD_PROVISIONER_SECRET']
+                    "dataPool": self.out_map['RBD_POOL_NAME'],
+                    "pool": self.out_map['RBD_METADATA_EC_POOL_NAME']
                 },
             })
-        # if 'CSI_CEPHFS_PROVISIONER_SECRET' exists, then only add 'rook-csi-cephfs-provisioner' Secret
-        if self.out_map['CSI_CEPHFS_PROVISIONER_SECRET'] and self.out_map['CSI_CEPHFS_PROVISIONER_SECRET_NAME']:
+        else:
             json_out.append({
-                "name": "rook-{}".format(self.out_map["CSI_CEPHFS_PROVISIONER_SECRET_NAME"]),
-                "kind": "Secret",
+                "name": "ceph-rbd",
+                "kind": "StorageClass",
                 "data": {
-                    "adminID": self.out_map["CSI_CEPHFS_PROVISIONER_SECRET_NAME"],
-                    "adminKey": self.out_map['CSI_CEPHFS_PROVISIONER_SECRET']
+                    "pool": self.out_map['RBD_POOL_NAME']
                 },
             })
-        # if 'CSI_CEPHFS_NODE_SECRET' exists, then only add 'rook-csi-cephfs-node' Secret
-        if self.out_map['CSI_CEPHFS_NODE_SECRET'] and self.out_map['CSI_CEPHFS_NODE_SECRET_NAME']:
+
+        if self._arg_parser.restricted_auth_permission:
+            cluster_name = self._arg_parser.cluster_name
+            rbd_pool_name = self._arg_parser.rbd_data_pool_name
+            cephfs_filesystem = self._arg_parser.cephfs_filesystem_name
+            rados_namespace = self._arg_parser.rados_namespace
             json_out.append({
-                "name": "rook-{}".format(self.out_map["CSI_CEPHFS_NODE_SECRET_NAME"]),
+                "name": "rook-csi-rbd-node-{}-{}-{}".format(cluster_name, rbd_pool_name, rados_namespace),
                 "kind": "Secret",
                 "data": {
-                    "adminID": self.out_map["CSI_CEPHFS_NODE_SECRET_NAME"],
-                    "adminKey": self.out_map['CSI_CEPHFS_NODE_SECRET']
+                    "userID": 'csi-rbd-node-{}-{}-{}'.format(cluster_name, rbd_pool_name, rados_namespace),
+                    "userKey": self.out_map['CSI_RBD_NODE_SECRET']
                 }
             })
-        # if 'ROOK_EXTERNAL_DASHBOARD_LINK' exists, then only add 'rook-ceph-dashboard-link' Secret
-        if self.out_map["ROOK_EXTERNAL_DASHBOARD_LINK"]:
-            json_out.append(
-                {
-                    "name": "rook-ceph-dashboard-link",
+            # if 'CSI_RBD_PROVISIONER_SECRET' exists, then only add 'rook-csi-rbd-provisioner' Secret
+            if self.out_map['CSI_RBD_PROVISIONER_SECRET']:
+                json_out.append({
+                    "name": "rook-csi-rbd-provisioner-{}-{}-{}".format(cluster_name, rbd_pool_name, rados_namespace),
                     "kind": "Secret",
                     "data": {
-                        "userID": "ceph-dashboard-link",
-                        "userKey": self.out_map["ROOK_EXTERNAL_DASHBOARD_LINK"],
+                        "userID": 'csi-rbd-provisioner-{}-{}-{}'.format(cluster_name, rbd_pool_name, rados_namespace),
+                        "userKey": self.out_map['CSI_RBD_PROVISIONER_SECRET']
                     },
-                }
-            )
-        if self.out_map['RBD_METADATA_EC_POOL_NAME']:
+                })
+            # if 'CSI_CEPHFS_PROVISIONER_SECRET' exists, then only add 'rook-csi-cephfs-provisioner' Secret
+            if self.out_map['CSI_CEPHFS_PROVISIONER_SECRET'] and cephfs_filesystem != "":
+                json_out.append({
+                    "name": "rook-csi-cephfs-provisioner-{}-{}".format(cluster_name, cephfs_filesystem),
+                    "kind": "Secret",
+                    "data": {
+                        "adminID": 'csi-cephfs-provisioner-{}-{}'.format(cluster_name, cephfs_filesystem),
+                        "adminKey": self.out_map['CSI_CEPHFS_PROVISIONER_SECRET']
+                    },
+                })
+            if self.out_map['CSI_CEPHFS_PROVISIONER_SECRET'] and cephfs_filesystem == "":
+                json_out.append({
+                    "name": "rook-csi-cephfs-provisioner-{}".format(cluster_name),
+                    "kind": "Secret",
+                    "data": {
+                        "adminID": 'csi-cephfs-provisioner-{}'.format(cluster_name),
+                        "adminKey": self.out_map['CSI_CEPHFS_PROVISIONER_SECRET']
+                    },
+                })
+            # if 'CSI_CEPHFS_NODE_SECRET' exists, then only add 'rook-csi-cephfs-node' Secret
+            if self.out_map['CSI_CEPHFS_NODE_SECRET'] and cephfs_filesystem != "":
+                json_out.append({
+                    "name": "rook-csi-cephfs-node-{}-{}".format(cluster_name, cephfs_filesystem),
+                    "kind": "Secret",
+                    "data": {
+                        "adminID": 'csi-cephfs-node-{}-{}'.format(cluster_name, cephfs_filesystem),
+                        "adminKey": self.out_map['CSI_CEPHFS_NODE_SECRET']
+                    }
+                })
+            if self.out_map['CSI_CEPHFS_NODE_SECRET'] and cephfs_filesystem == "":
+                json_out.append({
+                    "name": "rook-csi-cephfs-node-{}".format(cluster_name),
+                    "kind": "Secret",
+                    "data": {
+                        "adminID": 'csi-cephfs-node-{}'.format(cluster_name),
+                        "adminKey": self.out_map['CSI_CEPHFS_NODE_SECRET']
+                    }
+                })
+        else:
             json_out.append({
-                "name": "ceph-rbd",
-                "kind": "StorageClass",
+                "name": "rook-csi-rbd-node",
+                "kind": "Secret",
                 "data": {
-                    "dataPool": self.out_map['RBD_METADATA_EC_POOL_NAME'],
-                    "pool": self.out_map['RBD_POOL_NAME'],
-                    "csi.storage.k8s.io/provisioner-secret-name": "rook-{}".format(self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"]),
-                    "csi.storage.k8s.io/controller-expand-secret-name": "rook-{}".format(self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"]),
-                    "csi.storage.k8s.io/node-stage-secret-name": "rook-{}".format(self.out_map["CSI_RBD_NODE_SECRET_NAME"])
-                },
+                    "userID": 'csi-rbd-node',
+                    "userKey": self.out_map['CSI_RBD_NODE_SECRET']
+                }
             })
-        else:
+            # if 'CSI_RBD_PROVISIONER_SECRET' exists, then only add 'rook-csi-rbd-provisioner' Secret
+            if self.out_map['CSI_RBD_PROVISIONER_SECRET']:
+                json_out.append({
+                    "name": "rook-csi-rbd-provisioner",
+                    "kind": "Secret",
+                    "data": {
+                        "userID": 'csi-rbd-provisioner',
+                        "userKey": self.out_map['CSI_RBD_PROVISIONER_SECRET']
+                    },
+                })
+            # if 'CSI_CEPHFS_PROVISIONER_SECRET' exists, then only add 'rook-csi-cephfs-provisioner' Secret
+            if self.out_map['CSI_CEPHFS_PROVISIONER_SECRET']:
+                json_out.append({
+                    "name": "rook-csi-cephfs-provisioner",
+                    "kind": "Secret",
+                    "data": {
+                        "adminID": 'csi-cephfs-provisioner',
+                        "adminKey": self.out_map['CSI_CEPHFS_PROVISIONER_SECRET']
+                    },
+                })
+            # if 'CSI_CEPHFS_NODE_SECRET' exists, then only add 'rook-csi-cephfs-node' Secret
+            if self.out_map['CSI_CEPHFS_NODE_SECRET']:
+                json_out.append({
+                    "name": "rook-csi-cephfs-node",
+                    "kind": "Secret",
+                    "data": {
+                        "adminID": 'csi-cephfs-node',
+                        "adminKey": self.out_map['CSI_CEPHFS_NODE_SECRET']
+                    }
+                })
+        # if 'ROOK_EXTERNAL_DASHBOARD_LINK' exists, then only add 'rook-ceph-dashboard-link' Secret
+        if self.out_map['ROOK_EXTERNAL_DASHBOARD_LINK']:
             json_out.append({
-                "name": "ceph-rbd",
-                "kind": "StorageClass",
+                "name": "rook-ceph-dashboard-link",
+                "kind": "Secret",
                 "data": {
-                    "pool": self.out_map['RBD_POOL_NAME'],
-                    "csi.storage.k8s.io/provisioner-secret-name": "rook-{}".format(self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"]),
-                    "csi.storage.k8s.io/controller-expand-secret-name": "rook-{}".format(self.out_map["CSI_RBD_PROVISIONER_SECRET_NAME"]),
-                    "csi.storage.k8s.io/node-stage-secret-name": "rook-{}".format(self.out_map["CSI_RBD_NODE_SECRET_NAME"])
-                },
+                    "userID": 'ceph-dashboard-link',
+                    "userKey": self.out_map['ROOK_EXTERNAL_DASHBOARD_LINK']
+                }
             })
         # if 'CEPHFS_FS_NAME' exists, then only add 'cephfs' StorageClass
         if self.out_map['CEPHFS_FS_NAME']:
@@ -1452,55 +1229,43 @@ class RadosJSON:
                 "kind": "StorageClass",
                 "data": {
                     "fsName": self.out_map['CEPHFS_FS_NAME'],
-                    "pool": self.out_map['CEPHFS_POOL_NAME'],
-                    "csi.storage.k8s.io/provisioner-secret-name": "rook-{}".format(self.out_map["CSI_CEPHFS_PROVISIONER_SECRET_NAME"]),
-                    "csi.storage.k8s.io/controller-expand-secret-name": "rook-{}".format(self.out_map["CSI_CEPHFS_PROVISIONER_SECRET_NAME"]),
-                    "csi.storage.k8s.io/node-stage-secret-name": "rook-{}".format(self.out_map["CSI_CEPHFS_NODE_SECRET_NAME"])
+                    "pool": self.out_map['CEPHFS_POOL_NAME']
                 }
             })
         # if 'RGW_ENDPOINT' exists, then only add 'ceph-rgw' StorageClass
-        if self.out_map["RGW_ENDPOINT"]:
-            json_out.append(
-                {
-                    "name": "ceph-rgw",
-                    "kind": "StorageClass",
-                    "data": {
-                        "endpoint": self.out_map["RGW_ENDPOINT"],
-                        "poolPrefix": self.out_map["RGW_POOL_PREFIX"],
-                    },
+        if self.out_map['RGW_ENDPOINT']:
+            json_out.append({
+                "name": "ceph-rgw",
+                "kind": "StorageClass",
+                "data": {
+                    "endpoint": self.out_map['RGW_ENDPOINT'],
+                    "poolPrefix": self.out_map['RGW_POOL_PREFIX']
                 }
-            )
+            })
             json_out.append(
                 {
                     "name": "rgw-admin-ops-user",
                     "kind": "Secret",
                     "data": {
-                        "accessKey": self.out_map["RGW_ADMIN_OPS_USER_ACCESS_KEY"],
-                        "secretKey": self.out_map["RGW_ADMIN_OPS_USER_SECRET_KEY"],
-                    },
-                }
-            )
+                        "accessKey": self.out_map['RGW_ADMIN_OPS_USER_ACCESS_KEY'],
+                        "secretKey": self.out_map['RGW_ADMIN_OPS_USER_SECRET_KEY']
+                    }
+                })
         # if 'RGW_TLS_CERT' exists, then only add the "ceph-rgw-tls-cert" secret
-        if self.out_map["RGW_TLS_CERT"]:
-            json_out.append(
-                {
-                    "name": "ceph-rgw-tls-cert",
-                    "kind": "Secret",
-                    "data": {
-                        "cert": self.out_map["RGW_TLS_CERT"],
-                    },
+        if self.out_map['RGW_TLS_CERT']:
+            json_out.append({
+                "name": "ceph-rgw-tls-cert",
+                "kind": "Secret",
+                "data": {
+                    "cert": self.out_map['RGW_TLS_CERT'],
                 }
-            )
+            })
 
-        return json.dumps(json_out) + LINESEP
+        return json.dumps(json_out)+LINESEP
 
     def upgrade_users_permissions(self):
-        users = [
-            "client.csi-cephfs-node",
-            "client.csi-cephfs-provisioner",
-            "client.csi-rbd-node",
-            "client.csi-rbd-provisioner",
-        ]
+        users = ["client.csi-cephfs-node", "client.csi-cephfs-provisioner",
+                 "client.csi-rbd-node", "client.csi-rbd-provisioner"]
         if self.run_as_user != "" and self.run_as_user not in users:
             users.append(self.run_as_user)
         for user in users:
@@ -1508,63 +1273,58 @@ class RadosJSON:
 
     def upgrade_user_permissions(self, user):
         # check whether the given user exists or not
-        cmd_json = {"prefix": "auth get", "entity": "{}".format(user), "format": "json"}
+        cmd_json = {"prefix": "auth get", "entity": "{}".format(
+            user), "format": "json"}
         ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
         if ret_val != 0 or len(json_out) == 0:
             print("user {} not found for upgrading.".format(user))
             return
-        existing_caps = json_out[0]["caps"]
+        existing_caps = json_out[0]['caps']
         new_cap, _ = self.get_caps_and_entity(user)
         cap_keys = ["mon", "mgr", "osd", "mds"]
         caps = []
         for eachCap in cap_keys:
-            cur_cap_values = existing_caps.get(eachCap, "")
-            new_cap_values = new_cap.get(eachCap, "")
-            cur_cap_perm_list = [
-                x.strip() for x in cur_cap_values.split(",") if x.strip()
-            ]
-            new_cap_perm_list = [
-                x.strip() for x in new_cap_values.split(",") if x.strip()
-            ]
+            cur_cap_values = existing_caps.get(eachCap, '')
+            new_cap_values = new_cap.get(eachCap, '')
+            cur_cap_perm_list = [x.strip()
+                                 for x in cur_cap_values.split(',') if x.strip()]
+            new_cap_perm_list = [x.strip()
+                                 for x in new_cap_values.split(',') if x.strip()]
             # append new_cap_list to cur_cap_list to maintain the order of caps
             cur_cap_perm_list.extend(new_cap_perm_list)
             # eliminate duplicates without using 'set'
             # set re-orders items in the list and we have to keep the order
             new_cap_list = []
-            [new_cap_list.append(x) for x in cur_cap_perm_list if x not in new_cap_list]
+            [new_cap_list.append(
+                x) for x in cur_cap_perm_list if x not in new_cap_list]
             existing_caps[eachCap] = ", ".join(new_cap_list)
             if existing_caps[eachCap]:
                 caps.append(eachCap)
                 caps.append(existing_caps[eachCap])
-        cmd_json = {
-            "prefix": "auth caps",
-            "entity": user,
-            "caps": caps,
-            "format": "json",
-        }
+        cmd_json = {"prefix": "auth caps",
+                    "entity": user,
+                    "caps": caps,
+                    "format": "json"}
         ret_val, json_out, err_msg = self._common_cmd_json_gen(cmd_json)
         if ret_val != 0:
-            raise ExecutionFailureException(
-                "'auth caps {}' command failed.\n".format(user)
-                + "Error: {}".format(err_msg)
-            )
+            raise ExecutionFailureException("'auth caps {}' command failed.\n".format(user) +
+                                            "Error: {}".format(err_msg))
         print("Updated user, {}, successfully.".format(user))
 
     def main(self):
-        generated_output = ""
+        generated_output = ''
         if self._arg_parser.upgrade:
             self.upgrade_users_permissions()
-        elif self._arg_parser.format == "json":
+        elif self._arg_parser.format == 'json':
             generated_output = self.gen_json_out()
-        elif self._arg_parser.format == "bash":
+        elif self._arg_parser.format == 'bash':
             generated_output = self.gen_shell_out()
         else:
-            raise ExecutionFailureException(
-                "Unsupported format: {}".format(self._arg_parser.format)
-            )
-        print("{}".format(generated_output))
+            raise ExecutionFailureException("Unsupported format: {}".format(
+                self._arg_parser.format))
+        print('{}'.format(generated_output))
         if self.output_file and generated_output:
-            fOut = open(self.output_file, "w")
+            fOut = open(self.output_file, 'w')
             fOut.write(generated_output)
             fOut.close()
 
@@ -1572,7 +1332,7 @@ class RadosJSON:
 ################################################
 ##################### MAIN #####################
 ################################################
-if __name__ == "__main__":
+if __name__ == '__main__':
     rjObj = RadosJSON()
     try:
         rjObj.main()
diff --git a/deploy/examples/images.txt b/deploy/examples/images.txt
index 5d92e272b..ef68c6f80 100644
--- a/deploy/examples/images.txt
+++ b/deploy/examples/images.txt
@@ -1,11 +1,11 @@
- quay.io/ceph/ceph:v17.2.0
- quay.io/cephcsi/cephcsi:v3.6.2
+ k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
+ k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0
+ k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
+ k8s.gcr.io/sig-storage/csi-resizer:v1.4.0
+ k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1
+ k8s.gcr.io/sig-storage/nfsplugin:v3.1.0
+ quay.io/ceph/ceph:v16.2.7
+ quay.io/cephcsi/cephcsi:v3.6.1
  quay.io/csiaddons/k8s-sidecar:v0.2.1
  quay.io/csiaddons/volumereplication-operator:v0.3.0
- registry.k8s.io/sig-storage/csi-attacher:v3.4.0
- registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1
- registry.k8s.io/sig-storage/csi-provisioner:v3.1.0
- registry.k8s.io/sig-storage/csi-resizer:v1.4.0
- registry.k8s.io/sig-storage/csi-snapshotter:v6.0.1
- registry.k8s.io/sig-storage/nfsplugin:v4.0.0
  rook/ceph:master
diff --git a/deploy/examples/import-external-cluster.sh b/deploy/examples/import-external-cluster.sh
index bccfabc2f..ea0f438be 100644
--- a/deploy/examples/import-external-cluster.sh
+++ b/deploy/examples/import-external-cluster.sh
@@ -5,6 +5,10 @@ set -e
 # VARIABLES #
 #############
 MON_SECRET_NAME=rook-ceph-mon
+CSI_RBD_NODE_SECRET_NAME=rook-csi-rbd-node
+CSI_RBD_PROVISIONER_SECRET_NAME=rook-csi-rbd-provisioner
+CSI_CEPHFS_NODE_SECRET_NAME=rook-csi-cephfs-node
+CSI_CEPHFS_PROVISIONER_SECRET_NAME=rook-csi-cephfs-provisioner
 RGW_ADMIN_OPS_USER_SECRET_NAME=rgw-admin-ops-user
 MON_SECRET_CLUSTER_NAME_KEYNAME=cluster-name
 MON_SECRET_FSID_KEYNAME=fsid
@@ -16,12 +20,7 @@ MON_ENDPOINT_CONFIGMAP_NAME=rook-ceph-mon-endpoints
 ROOK_EXTERNAL_CLUSTER_NAME=$NAMESPACE
 ROOK_EXTERNAL_MAX_MON_ID=2
 ROOK_EXTERNAL_MAPPING={}
-RBD_STORAGE_CLASS_NAME=ceph-rbd
-CEPHFS_STORAGE_CLASS_NAME=cephfs
 ROOK_EXTERNAL_MONITOR_SECRET=mon-secret
-OPERATOR_NAMESPACE=rook-ceph # default set to rook-ceph
-RBD_PROVISIONER=$OPERATOR_NAMESPACE".rbd.csi.ceph.com" # driver:namespace:operator
-CEPHFS_PROVISIONER=$OPERATOR_NAMESPACE".cephfs.csi.ceph.com" # driver:namespace:operator
 : "${ROOK_EXTERNAL_ADMIN_SECRET:=admin-secret}"
 
 #############
@@ -32,26 +31,6 @@ function checkEnvVars() {
   if [ -z "$NAMESPACE" ]; then
     echo "Please populate the environment variable NAMESPACE"
     exit 1
-  fi
-   if [ -z "$RBD_POOL_NAME" ]; then
-    echo "Please populate the environment variable RBD_POOL_NAME"
-    exit 1
-  fi
-  if [ -z "$CSI_RBD_NODE_SECRET_NAME" ]; then
-    echo "Please populate the environment variable CSI_RBD_NODE_SECRET_NAME"
-    exit 1
-  fi
-  if [ -z "$CSI_RBD_PROVISIONER_SECRET_NAME" ]; then
-    echo "Please populate the environment variable CSI_RBD_PROVISIONER_SECRET_NAME"
-    exit 1
-  fi
-  if [ -z "$CSI_CEPHFS_NODE_SECRET_NAME" ]; then
-    echo "Please populate the environment variable CSI_CEPHFS_NODE_SECRET_NAME"
-    exit 1
-  fi
-  if [ -z "$CSI_CEPHFS_PROVISIONER_SECRET_NAME" ]; then
-    echo "Please populate the environment variable CSI_CEPHFS_PROVISIONER_SECRET_NAME"
-    exit 1
   fi
   if [ -z "$ROOK_EXTERNAL_FSID" ]; then
     echo "Please populate the environment variable ROOK_EXTERNAL_FSID"
@@ -124,8 +103,8 @@ function importCsiRBDNodeSecret() {
   secret \
   generic \
   --type="kubernetes.io/rook" \
-  "rook-""$CSI_RBD_NODE_SECRET_NAME" \
-  --from-literal=userID="$CSI_RBD_NODE_SECRET_NAME" \
+  "$CSI_RBD_NODE_SECRET_NAME" \
+  --from-literal=userID=csi-rbd-node \
   --from-literal=userKey="$CSI_RBD_NODE_SECRET"
 }
 
@@ -135,8 +114,8 @@ function importCsiRBDProvisionerSecret() {
   secret \
   generic \
   --type="kubernetes.io/rook" \
-  "rook-""$CSI_RBD_PROVISIONER_SECRET_NAME" \
-  --from-literal=userID="$CSI_RBD_PROVISIONER_SECRET_NAME" \
+  "$CSI_RBD_PROVISIONER_SECRET_NAME" \
+  --from-literal=userID=csi-rbd-provisioner \
   --from-literal=userKey="$CSI_RBD_PROVISIONER_SECRET"
 }
 
@@ -146,8 +125,8 @@ function importCsiCephFSNodeSecret() {
   secret \
   generic \
   --type="kubernetes.io/rook" \
-  "rook-""$CSI_CEPHFS_NODE_SECRET_NAME" \
-  --from-literal=adminID="$CSI_CEPHFS_NODE_SECRET_NAME" \
+  "$CSI_CEPHFS_NODE_SECRET_NAME" \
+  --from-literal=adminID=csi-cephfs-node \
   --from-literal=adminKey="$CSI_CEPHFS_NODE_SECRET"
 }
 
@@ -157,8 +136,8 @@ function importCsiCephFSProvisionerSecret() {
   secret \
   generic \
   --type="kubernetes.io/rook" \
-  "rook-""$CSI_CEPHFS_PROVISIONER_SECRET_NAME" \
-  --from-literal=adminID="$CSI_CEPHFS_PROVISIONER_SECRET_NAME" \
+  "$CSI_CEPHFS_PROVISIONER_SECRET_NAME" \
+  --from-literal=adminID=csi-cephfs-provisioner \
   --from-literal=adminKey="$CSI_CEPHFS_PROVISIONER_SECRET"
 }
 
@@ -173,77 +152,6 @@ function importRGWAdminOpsUser() {
   --from-literal=secretKey="$RGW_ADMIN_OPS_USER_SECRET_KEY"
 }
 
-function createECRBDStorageClass() {
-cat <<eof | kubectl create -f -
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata:
-  name: $RBD_STORAGE_CLASS_NAME
-provisioner: $RBD_PROVISIONER
-parameters:
-  clusterID: $NAMESPACE
-  pool: $RBD_POOL_NAME
-  dataPool: $RBD_METADATA_EC_POOL_NAME
-  imageFormat: "2"
-  imageFeatures: layering
-  csi.storage.k8s.io/provisioner-secret-name: $CSI_RBD_PROVISIONER_SECRET_NAME
-  csi.storage.k8s.io/provisioner-secret-namespace: $NAMESPACE
-  csi.storage.k8s.io/controller-expand-secret-name:  $CSI_RBD_PROVISIONER_SECRET_NAME
-  csi.storage.k8s.io/controller-expand-secret-namespace: $NAMESPACE
-  csi.storage.k8s.io/node-stage-secret-name: $CSI_RBD_NODE_SECRET_NAME
-  csi.storage.k8s.io/node-stage-secret-namespace: $NAMESPACE
-  csi.storage.k8s.io/fstype: ext4
-allowVolumeExpansion: true
-reclaimPolicy: Delete
-eof
-}
-
-function createRBDStorageClass() {
-cat <<eof | kubectl create -f -
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata:
-  name: $RBD_STORAGE_CLASS_NAME
-provisioner: $RBD_PROVISIONER
-parameters:
-  clusterID: $NAMESPACE
-  pool: $RBD_POOL_NAME
-  imageFormat: "2"
-  imageFeatures: layering
-  csi.storage.k8s.io/provisioner-secret-name: $CSI_RBD_PROVISIONER_SECRET_NAME
-  csi.storage.k8s.io/provisioner-secret-namespace: $NAMESPACE
-  csi.storage.k8s.io/controller-expand-secret-name:  $CSI_RBD_PROVISIONER_SECRET_NAME
-  csi.storage.k8s.io/controller-expand-secret-namespace: $NAMESPACE
-  csi.storage.k8s.io/node-stage-secret-name: $CSI_RBD_NODE_SECRET_NAME
-  csi.storage.k8s.io/node-stage-secret-namespace: $NAMESPACE
-  csi.storage.k8s.io/fstype: ext4
-allowVolumeExpansion: true
-reclaimPolicy: Delete
-eof
-}
-
-function createCephFSStorageClass() {
-cat <<eof | kubectl create -f -
-apiVersion: storage.k8s.io/v1
-kind: StorageClass
-metadata:
-  name: $CEPHFS_STORAGE_CLASS_NAME
-provisioner: $CEPHFS_PROVISIONER
-parameters:
-  clusterID: $NAMESPACE
-  fsName: $CEPHFS_FS_NAME
-  pool: $CEPHFS_POOL_NAME
-  csi.storage.k8s.io/provisioner-secret-name: $CSI_CEPHFS_PROVISIONER_SECRET_NAME
-  csi.storage.k8s.io/provisioner-secret-namespace: $NAMESPACE
-  csi.storage.k8s.io/controller-expand-secret-name: $CSI_CEPHFS_PROVISIONER_SECRET_NAME
-  csi.storage.k8s.io/controller-expand-secret-namespace: $NAMESPACE
-  csi.storage.k8s.io/node-stage-secret-name: $CSI_CEPHFS_NODE_SECRET_NAME
-  csi.storage.k8s.io/node-stage-secret-namespace: $NAMESPACE
-allowVolumeExpansion: true
-reclaimPolicy: Delete
-eof
-}
-
 ########
 # MAIN #
 ########
@@ -255,11 +163,3 @@ importCsiRBDProvisionerSecret
 importCsiCephFSNodeSecret
 importCsiCephFSProvisionerSecret
 importRGWAdminOpsUser
-if [ -n "$RBD_METADATA_EC_POOL_NAME" ]; then
-  createECRBDStorageClass
-else
-  createRBDStorageClass
-fi
-if [ -n "$CEPHFS_FS_NAME" ] && [ -n "$CEPHFS_POOL_NAME" ]; then
-  createCephFSStorageClass
-fi
diff --git a/deploy/examples/operator-openshift.yaml b/deploy/examples/operator-openshift.yaml
index 31abb727d..ab03a3f95 100644
--- a/deploy/examples/operator-openshift.yaml
+++ b/deploy/examples/operator-openshift.yaml
@@ -173,13 +173,13 @@ data:
   # The default version of CSI supported by Rook will be started. To change the version
   # of the CSI driver to something other than what is officially supported, change
   # these images to the desired release of the CSI driver.
-  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.6.2"
-  # ROOK_CSI_REGISTRAR_IMAGE: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1"
-  # ROOK_CSI_RESIZER_IMAGE: "registry.k8s.io/sig-storage/csi-resizer:v1.4.0"
-  # ROOK_CSI_PROVISIONER_IMAGE: "registry.k8s.io/sig-storage/csi-provisioner:v3.1.0"
-  # ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.k8s.io/sig-storage/csi-snapshotter:v6.0.1"
-  # ROOK_CSI_ATTACHER_IMAGE: "registry.k8s.io/sig-storage/csi-attacher:v3.4.0"
-  # ROOK_CSI_NFS_IMAGE: "registry.k8s.io/sig-storage/nfsplugin:v4.0.0"
+  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.6.1"
+  # ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0"
+  # ROOK_CSI_RESIZER_IMAGE: "k8s.gcr.io/sig-storage/csi-resizer:v1.4.0"
+  # ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0"
+  # ROOK_CSI_SNAPSHOTTER_IMAGE: "k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1"
+  # ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.4.0"
+  # ROOK_CSI_NFS_IMAGE: "k8s.gcr.io/sig-storage/nfsplugin:v3.1.0"
 
   # (Optional) set user created priorityclassName for csi plugin pods.
   CSI_PLUGIN_PRIORITY_CLASSNAME: "system-node-critical"
diff --git a/deploy/examples/operator.yaml b/deploy/examples/operator.yaml
index 06b744d97..cabbb83f9 100644
--- a/deploy/examples/operator.yaml
+++ b/deploy/examples/operator.yaml
@@ -88,13 +88,13 @@ data:
   # The default version of CSI supported by Rook will be started. To change the version
   # of the CSI driver to something other than what is officially supported, change
   # these images to the desired release of the CSI driver.
-  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.6.2"
-  # ROOK_CSI_REGISTRAR_IMAGE: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1"
-  # ROOK_CSI_RESIZER_IMAGE: "registry.k8s.io/sig-storage/csi-resizer:v1.4.0"
-  # ROOK_CSI_PROVISIONER_IMAGE: "registry.k8s.io/sig-storage/csi-provisioner:v3.1.0"
-  # ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.k8s.io/sig-storage/csi-snapshotter:v6.0.1"
-  # ROOK_CSI_ATTACHER_IMAGE: "registry.k8s.io/sig-storage/csi-attacher:v3.4.0"
-  # ROOK_CSI_NFS_IMAGE: "registry.k8s.io/sig-storage/nfsplugin:v4.0.0"
+  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.6.1"
+  # ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0"
+  # ROOK_CSI_RESIZER_IMAGE: "k8s.gcr.io/sig-storage/csi-resizer:v1.4.0"
+  # ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0"
+  # ROOK_CSI_SNAPSHOTTER_IMAGE: "k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1"
+  # ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.4.0"
+  # ROOK_CSI_NFS_IMAGE: "k8s.gcr.io/sig-storage/nfsplugin:v3.1.0"
 
   # (Optional) set user created priorityclassName for csi plugin pods.
   CSI_PLUGIN_PRIORITY_CLASSNAME: "system-node-critical"
diff --git a/deploy/olm/assemble/metadata-common.yaml b/deploy/olm/assemble/metadata-common.yaml
index 697af3e87..8c67cc19b 100644
--- a/deploy/olm/assemble/metadata-common.yaml
+++ b/deploy/olm/assemble/metadata-common.yaml
@@ -250,7 +250,7 @@ metadata:
           },
           "spec": {
             "cephVersion": {
-              "image": "quay.io/ceph/ceph:v17.2.0"
+              "image": "quay.io/ceph/ceph:v16.2.7"
             },
             "dataDirHostPath": "/var/lib/rook",
             "mon": {
diff --git a/design/ceph/ceph-cluster-cleanup.md b/design/ceph/ceph-cluster-cleanup.md
index f69865a9a..b7b132979 100644
--- a/design/ceph/ceph-cluster-cleanup.md
+++ b/design/ceph/ceph-cluster-cleanup.md
@@ -34,7 +34,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v17.2.0
+    image: quay.io/ceph/ceph:v16.2.7
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
diff --git a/images/ceph/Makefile b/images/ceph/Makefile
index c801226fe..48e7ed150 100755
--- a/images/ceph/Makefile
+++ b/images/ceph/Makefile
@@ -18,9 +18,9 @@ include ../image.mk
 # Image Build Options
 
 ifeq ($(GOARCH),amd64)
-CEPH_VERSION ?= v17.2.0-20220420
+CEPH_VERSION ?= v16.2.7-20220216
 else
-CEPH_VERSION ?= v17.2.0-20220420
+CEPH_VERSION ?= v16.2.7-20220216
 endif
 REGISTRY_NAME = quay.io
 BASEIMAGE = $(REGISTRY_NAME)/ceph/ceph-$(GOARCH):$(CEPH_VERSION)
@@ -147,7 +147,7 @@ list-image: ## Create a list of images for offline installation
 	rm -f $(IMAGE_TMP)
 	awk '/image:/ {print $2}' $(MANIFESTS_DIR)/operator.yaml $(MANIFESTS_DIR)/cluster.yaml | \
 		cut -d: -f2- | tee $(IMAGE_TMP)
-	awk '/quay.io/ || /registry.k8s.io/ {print $3}' ../../pkg/operator/ceph/csi/spec.go | \
+	awk '/quay.io/ || /k8s.gcr.io/ {print $3}' ../../pkg/operator/ceph/csi/spec.go | \
 		cut -d= -f2- | tr -d '"' | tee -a $(IMAGE_TMP)
 	rm -f $(MANIFESTS_DIR)/images.txt
 	cat $(IMAGE_TMP) | sort -h | uniq | tee $(MANIFESTS_DIR)/images.txt
diff --git a/pkg/operator/ceph/cluster/cluster.go b/pkg/operator/ceph/cluster/cluster.go
index d0ba40816..8fbb0f28f 100755
--- a/pkg/operator/ceph/cluster/cluster.go
+++ b/pkg/operator/ceph/cluster/cluster.go
@@ -22,8 +22,6 @@ import (
 	"fmt"
 	"os/exec"
 	"path"
-	"strconv"
-	"sync"
 	"syscall"
 
 	"github.com/pkg/errors"
@@ -42,7 +40,6 @@ import (
 	"github.com/rook/rook/pkg/operator/ceph/csi"
 	cephver "github.com/rook/rook/pkg/operator/ceph/version"
 	"github.com/rook/rook/pkg/operator/k8sutil"
-	rookversion "github.com/rook/rook/pkg/version"
 	v1 "k8s.io/api/core/v1"
 	kerrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -53,8 +50,6 @@ const (
 	detectVersionName = "rook-ceph-detect-version"
 )
 
-var telemetryMutex sync.Mutex
-
 type cluster struct {
 	ClusterInfo        *client.ClusterInfo
 	context            *clusterd.Context
@@ -222,10 +217,6 @@ func (c *ClusterController) initializeCluster(cluster *cluster) error {
 
 	// Start the monitoring if not already started
 	c.configureCephMonitoring(cluster, cluster.ClusterInfo)
-
-	// Asynchronously report the telemetry to allow another reconcile to proceed if needed
-	go cluster.reportTelemetry()
-
 	return nil
 }
 
@@ -495,6 +486,9 @@ func (c *cluster) skipMDSSanityChecks(skip bool) error {
 // It gets executed right after the main mon Start() method
 // Basically, it is executed between the monitors and the manager sequence
 func (c *cluster) postMonStartupActions() error {
+	// Identify this as a rook cluster for Ceph telemetry by setting the Rook version.
+	telemetry.SetRookVersion(c.context, c.ClusterInfo)
+
 	// Create CSI Kubernetes Secrets
 	err := csi.CreateCSISecrets(c.context, c.ClusterInfo)
 	if err != nil {
@@ -539,62 +533,6 @@ func (c *cluster) postMonStartupActions() error {
 	return nil
 }
 
-func (c *cluster) reportTelemetry() {
-	// In the corner case that reconciles are started in quick succession and the telemetry
-	// hasn't had a chance to complete yet from a previous reconcile, simply allow
-	// a single goroutine to report telemetry at a time.
-	telemetryMutex.Lock()
-	defer telemetryMutex.Unlock()
-
-	// Identify this as a rook cluster for Ceph telemetry by setting the Rook version.
-	logger.Info("reporting cluster telemetry")
-	telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.RookVersionKey, rookversion.Version)
-
-	// Report the K8s version
-	serverVersion, err := c.context.Clientset.Discovery().ServerVersion()
-	if err != nil {
-		logger.Warningf("failed to report the K8s server version. %v", err)
-	} else {
-		telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.K8sVersionKey, serverVersion.String())
-	}
-
-	// Report the CSI version if it has been detected
-	if telemetry.CSIVersion != "" {
-		telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.CSIVersionKey, telemetry.CSIVersion)
-	}
-
-	// Report the max mon id
-	telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.MonMaxIDKey, strconv.Itoa(c.mons.MaxMonID()))
-
-	// Report the telemetry for mon settings
-	telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.MonCountKey, strconv.Itoa(c.Spec.Mon.Count))
-	telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.MonAllowMultiplePerNodeKey, strconv.FormatBool(c.Spec.Mon.AllowMultiplePerNode))
-	telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.MonPVCEnabledKey, strconv.FormatBool(c.Spec.Mon.VolumeClaimTemplate != nil))
-	telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.MonStretchEnabledKey, strconv.FormatBool(c.Spec.IsStretchCluster()))
-
-	// Set the telemetry for device sets
-	deviceSets := 0
-	portableDeviceSets := 0
-	nonportableDeviceSets := 0
-	for _, deviceSet := range c.Spec.Storage.StorageClassDeviceSets {
-		deviceSets++
-		if deviceSet.Portable {
-			portableDeviceSets++
-		} else {
-			nonportableDeviceSets++
-		}
-	}
-	telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.DeviceSetTotalKey, strconv.Itoa(deviceSets))
-	telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.DeviceSetPortableKey, strconv.Itoa(portableDeviceSets))
-	telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.DeviceSetNonPortableKey, strconv.Itoa(nonportableDeviceSets))
-
-	// Set the telemetry for network settings
-	telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.NetworkProviderKey, c.Spec.Network.Provider)
-
-	// Set the telemetry for external cluster settings
-	telemetry.ReportKeyValue(c.context, c.ClusterInfo, telemetry.ExternalModeEnabledKey, strconv.FormatBool(c.Spec.External.Enable))
-}
-
 func (c *cluster) configureMsgr2() error {
 	if c.Spec.Network.Connections == nil {
 		return nil
diff --git a/pkg/operator/ceph/cluster/mon/mon.go b/pkg/operator/ceph/cluster/mon/mon.go
index 6fe6c5097..72a1bbf2d 100644
--- a/pkg/operator/ceph/cluster/mon/mon.go
+++ b/pkg/operator/ceph/cluster/mon/mon.go
@@ -156,10 +156,6 @@ func New(ctx context.Context, clusterdContext *clusterd.Context, namespace strin
 	}
 }
 
-func (c *Cluster) MaxMonID() int {
-	return c.maxMonID
-}
-
 // Start begins the process of running a cluster of Ceph mons.
 func (c *Cluster) Start(clusterInfo *cephclient.ClusterInfo, rookVersion string, cephVersion cephver.CephVersion, spec cephv1.ClusterSpec) (*cephclient.ClusterInfo, error) {
 	// Only one goroutine can orchestrate the mons at a time
diff --git a/pkg/operator/ceph/cluster/osd/spec.go b/pkg/operator/ceph/cluster/osd/spec.go
index b70d8e13e..4be1124df 100644
--- a/pkg/operator/ceph/cluster/osd/spec.go
+++ b/pkg/operator/ceph/cluster/osd/spec.go
@@ -64,7 +64,6 @@ const (
 	bluestoreBlockName    = "block"
 	bluestoreMetadataName = "block.db"
 	bluestoreWalName      = "block.wal"
-	tempEtcCephDir        = "/etc/temp-ceph"
 )
 
 const (
@@ -75,36 +74,12 @@ set -o nounset # fail if variables are unset
 set -o xtrace
 
 OSD_ID="$ROOK_OSD_ID"
-CEPH_FSID=%s
 OSD_UUID=%s
 OSD_STORE_FLAG="%s"
 OSD_DATA_DIR=/var/lib/ceph/osd/ceph-"$OSD_ID"
 CV_MODE=%s
 DEVICE="$%s"
 
-# "ceph.conf" must have the "fsid" global configuration to activate encrypted OSDs
-# after the following Ceph's PR is merged.
-# https://github.com/ceph/ceph/commit/25655e5a8829e001adf467511a6bde8142b0a575
-# This limitation will be removed later. After that, we can remove this
-# fsid injection code. Probably a good time is when to remove Quincy support.
-# https://github.com/rook/rook/pull/10333#discussion_r892817877
-cp --no-preserve=mode /etc/temp-ceph/ceph.conf /etc/ceph/ceph.conf
-python3 -c "
-import configparser
-
-config = configparser.ConfigParser()
-config.read('/etc/ceph/ceph.conf')
-
-if not config.has_section('global'):
-    config['global'] = {}
-
-if not config.has_option('global','fsid'):
-    config['global']['fsid'] = '$CEPH_FSID'
-
-with open('/etc/ceph/ceph.conf', 'w') as configfile:
-    config.write(configfile)
-"
-
 # create new keyring
 ceph -n client.admin auth get-or-create osd."$OSD_ID" mon 'allow profile osd' mgr 'allow profile osd' osd 'allow *' -k /etc/ceph/admin-keyring-store/keyring
 
@@ -775,7 +750,7 @@ func (c *Cluster) getActivateOSDInitContainer(configDir, namespace, osdID string
 	volMounts := []v1.VolumeMount{
 		{Name: activateOSDVolumeName, MountPath: activateOSDMountPathID},
 		{Name: "devices", MountPath: "/dev"},
-		{Name: k8sutil.ConfigOverrideName, ReadOnly: true, MountPath: tempEtcCephDir},
+		{Name: k8sutil.ConfigOverrideName, ReadOnly: true, MountPath: opconfig.EtcCephDir},
 	}
 	volMounts = append(volMounts, adminKeyringVolMount)
 
@@ -787,7 +762,7 @@ func (c *Cluster) getActivateOSDInitContainer(configDir, namespace, osdID string
 		Command: []string{
 			"/bin/bash",
 			"-c",
-			fmt.Sprintf(activateOSDOnNodeCode, c.clusterInfo.FSID, osdInfo.UUID, osdStore, osdInfo.CVMode, blockPathVarName),
+			fmt.Sprintf(activateOSDOnNodeCode, osdInfo.UUID, osdStore, osdInfo.CVMode, blockPathVarName),
 		},
 		Name:            "activate",
 		Image:           c.spec.CephVersion.Image,
diff --git a/pkg/operator/ceph/cluster/telemetry/telemetry.go b/pkg/operator/ceph/cluster/telemetry/telemetry.go
index 4528ddf34..c8f6c1672 100644
--- a/pkg/operator/ceph/cluster/telemetry/telemetry.go
+++ b/pkg/operator/ceph/cluster/telemetry/telemetry.go
@@ -23,33 +23,22 @@ import (
 	"github.com/rook/rook/pkg/clusterd"
 	"github.com/rook/rook/pkg/daemon/ceph/client"
 	"github.com/rook/rook/pkg/operator/ceph/config"
+	rookversion "github.com/rook/rook/pkg/version"
 )
 
 var logger = capnslog.NewPackageLogger("github.com/rook/rook", "telemetry")
 
 const (
-	RookVersionKey             = "rook/version"
-	K8sVersionKey              = "rook/kubernetes/version"
-	CSIVersionKey              = "rook/csi/version"
-	MonMaxIDKey                = "rook/cluster/mon/max-id"
-	MonCountKey                = "rook/cluster/mon/count"
-	MonAllowMultiplePerNodeKey = "rook/cluster/mon/allow-multiple-per-node"
-	MonPVCEnabledKey           = "rook/cluster/mon/pvc/enabled"
-	MonStretchEnabledKey       = "rook/cluster/mon/stretch/enabled"
-	DeviceSetTotalKey          = "rook/cluster/storage/device-set/count/total"
-	DeviceSetPortableKey       = "rook/cluster/storage/device-set/count/portable"
-	DeviceSetNonPortableKey    = "rook/cluster/storage/device-set/count/non-portable"
-	NetworkProviderKey         = "rook/cluster/network/provider"
-	ExternalModeEnabledKey     = "rook/cluster/external-mode"
+	RookVersionKey = `rook/version`
 )
 
-var CSIVersion string
-
-func ReportKeyValue(context *clusterd.Context, clusterInfo *client.ClusterInfo, key, value string) {
+// SetRookVersion sets the Rook version in Ceph's config-key store to allow Rook clusters that
+// enable Ceph telemetry to be identified easily as Rook clusters while additionally providing
+// useful information about the version of Rook that is managing the cluster.
+// e.g., rook/version=v1.8.7
+func SetRookVersion(context *clusterd.Context, clusterInfo *client.ClusterInfo) {
 	ms := config.GetMonStore(context, clusterInfo)
-	if err := ms.SetKeyValue(key, value); err != nil {
-		logger.Warningf("failed to set telemetry key %q. %v", key, err)
-		return
+	if err := ms.SetKeyValue(RookVersionKey, rookversion.Version); err != nil {
+		logger.Warningf("failed to set telemetry key; this cluster may not be identifiable by ceph telemetry as a Rook cluster. %v", err)
 	}
-	logger.Debugf("set telemetry key: %s=%s", key, value)
 }
diff --git a/pkg/operator/ceph/csi/controller.go b/pkg/operator/ceph/csi/controller.go
index e7d26aa3a..48eb2ba86 100644
--- a/pkg/operator/ceph/csi/controller.go
+++ b/pkg/operator/ceph/csi/controller.go
@@ -19,7 +19,6 @@ package csi
 import (
 	"context"
 	"os"
-	"strconv"
 
 	cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
 	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
@@ -47,11 +46,11 @@ const (
 
 // ReconcileCSI reconciles a ceph-csi driver
 type ReconcileCSI struct {
-	client             client.Client
-	context            *clusterd.Context
-	opManagerContext   context.Context
-	opConfig           opcontroller.OperatorConfig
-	clustersWithHolder []ClusterDetail
+	client           client.Client
+	context          *clusterd.Context
+	opManagerContext context.Context
+	opConfig         opcontroller.OperatorConfig
+	multusClusters   []ClusterDetail
 }
 
 // ClusterDetail is a struct that holds the information of a cluster, it knows its internals (like
@@ -70,11 +69,11 @@ func Add(mgr manager.Manager, context *clusterd.Context, opManagerContext contex
 // newReconciler returns a new reconcile.Reconciler
 func newReconciler(mgr manager.Manager, context *clusterd.Context, opManagerContext context.Context, opConfig opcontroller.OperatorConfig) reconcile.Reconciler {
 	return &ReconcileCSI{
-		client:             mgr.GetClient(),
-		context:            context,
-		opConfig:           opConfig,
-		opManagerContext:   opManagerContext,
-		clustersWithHolder: []ClusterDetail{},
+		client:           mgr.GetClient(),
+		context:          context,
+		opConfig:         opConfig,
+		opManagerContext: opManagerContext,
+		multusClusters:   []ClusterDetail{},
 	}
 }
 
@@ -154,29 +153,6 @@ func (r *ReconcileCSI) reconcile(request reconcile.Request) (reconcile.Result, e
 
 		return reconcile.Result{}, nil
 	}
-	// Fetch the operator's configmap. We force the NamespaceName to the operator since the request
-	// could be a CephCluster. If so the NamespaceName will be the one from the cluster and thus the
-	// CM won't be found
-	opNamespaceName := types.NamespacedName{Name: opcontroller.OperatorSettingConfigMapName, Namespace: r.opConfig.OperatorNamespace}
-	opConfig := &v1.ConfigMap{}
-	err = r.client.Get(r.opManagerContext, opNamespaceName, opConfig)
-	if err != nil {
-		if kerrors.IsNotFound(err) {
-			logger.Debug("operator's configmap resource not found. will use default value or env var.")
-			r.opConfig.Parameters = make(map[string]string)
-		} else {
-			// Error reading the object - requeue the request.
-			return opcontroller.ImmediateRetryResult, errors.Wrap(err, "failed to get operator's configmap")
-		}
-	} else {
-		// Populate the operator's config
-		r.opConfig.Parameters = opConfig.Data
-	}
-
-	csiHostNetworkEnabled, err := strconv.ParseBool(k8sutil.GetValue(r.opConfig.Parameters, "CSI_ENABLE_HOST_NETWORK", "true"))
-	if err != nil {
-		return reconcile.Result{}, errors.Wrap(err, "failed to parse value for 'CSI_ENABLE_HOST_NETWORK'")
-	}
 
 	for i, cluster := range cephClusters.Items {
 		if !cluster.DeletionTimestamp.IsZero() {
@@ -188,10 +164,10 @@ func (r *ReconcileCSI) reconcile(request reconcile.Request) (reconcile.Result, e
 			logger.Debugf("ceph cluster %q has cleanup policy, the cluster will soon go away, no need to reconcile the csi driver", cluster.Name)
 			return reconcile.Result{}, nil
 		}
-		holderEnabled := !csiHostNetworkEnabled || cluster.Spec.Network.IsMultus()
-		// Do we have a multus cluster or csi host network disabled?
+
+		// Do we have a multus cluster?
 		// If so deploy the plugin holder with the fsid attached
-		if holderEnabled {
+		if cluster.Spec.Network.IsMultus() {
 			// Load cluster info for later use in updating the ceph-csi configmap
 			clusterInfo, _, _, err := opcontroller.LoadClusterInfo(r.context, r.opManagerContext, cluster.Namespace)
 			if err != nil {
@@ -205,13 +181,30 @@ func (r *ReconcileCSI) reconcile(request reconcile.Request) (reconcile.Result, e
 				return opcontroller.ImmediateRetryResult, errors.Wrapf(err, "failed to load cluster info for cluster %q", cluster.Name)
 			}
 
-			logger.Debugf("cluster %q is running on multus or CSI_ENABLE_HOST_NETWORK is false, deploying the ceph-csi plugin holder", cluster.Name)
-
-			r.clustersWithHolder = append(r.clustersWithHolder, ClusterDetail{cluster: &cephClusters.Items[i], clusterInfo: clusterInfo})
+			logger.Debugf("cluster %q is running on multus, deploying the ceph-csi plugin holder", cluster.Name)
+			r.multusClusters = append(r.multusClusters, ClusterDetail{cluster: &cephClusters.Items[i], clusterInfo: clusterInfo})
+		} else {
+			logger.Debugf("not a multus cluster %q, not deploying the ceph-csi plugin holder", request.NamespacedName)
+		}
+	}
 
+	// Fetch the operator's configmap. We force the NamespaceName to the operator since the request
+	// could be a CephCluster. If so the NamespaceName will be the one from the cluster and thus the
+	// CM won't be found
+	opNamespaceName := types.NamespacedName{Name: opcontroller.OperatorSettingConfigMapName, Namespace: r.opConfig.OperatorNamespace}
+	opConfig := &v1.ConfigMap{}
+	err = r.client.Get(r.opManagerContext, opNamespaceName, opConfig)
+	if err != nil {
+		if kerrors.IsNotFound(err) {
+			logger.Debug("operator's configmap resource not found. will use default value or env var.")
+			r.opConfig.Parameters = make(map[string]string)
 		} else {
-			logger.Debugf("not a multus cluster %q or CSI_ENABLE_HOST_NETWORK is true, not deploying the ceph-csi plugin holder", request.NamespacedName)
+			// Error reading the object - requeue the request.
+			return opcontroller.ImmediateRetryResult, errors.Wrap(err, "failed to get operator's configmap")
 		}
+	} else {
+		// Populate the operator's config
+		r.opConfig.Parameters = opConfig.Data
 	}
 
 	ownerRef, err := k8sutil.GetDeploymentOwnerReference(r.opManagerContext, r.context.Clientset, os.Getenv(k8sutil.PodNameEnvVar), r.opConfig.OperatorNamespace)
diff --git a/pkg/operator/ceph/csi/spec.go b/pkg/operator/ceph/csi/spec.go
index 2a845feb5..8ce509571 100644
--- a/pkg/operator/ceph/csi/spec.go
+++ b/pkg/operator/ceph/csi/spec.go
@@ -22,7 +22,6 @@ import (
 	"fmt"
 	"time"
 
-	"github.com/rook/rook/pkg/operator/ceph/cluster/telemetry"
 	opcontroller "github.com/rook/rook/pkg/operator/ceph/controller"
 	"github.com/rook/rook/pkg/operator/k8sutil"
 	"github.com/rook/rook/pkg/operator/k8sutil/cmdreporter"
@@ -123,13 +122,13 @@ var (
 // manually challenging.
 var (
 	// image names
-	DefaultCSIPluginImage         = "quay.io/cephcsi/cephcsi:v3.6.2"
-	DefaultNFSPluginImage         = "registry.k8s.io/sig-storage/nfsplugin:v4.0.0"
-	DefaultRegistrarImage         = "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.5.1"
-	DefaultProvisionerImage       = "registry.k8s.io/sig-storage/csi-provisioner:v3.1.0"
-	DefaultAttacherImage          = "registry.k8s.io/sig-storage/csi-attacher:v3.4.0"
-	DefaultSnapshotterImage       = "registry.k8s.io/sig-storage/csi-snapshotter:v6.0.1"
-	DefaultResizerImage           = "registry.k8s.io/sig-storage/csi-resizer:v1.4.0"
+	DefaultCSIPluginImage         = "quay.io/cephcsi/cephcsi:v3.6.1"
+	DefaultNFSPluginImage         = "k8s.gcr.io/sig-storage/nfsplugin:v3.1.0"
+	DefaultRegistrarImage         = "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.5.0"
+	DefaultProvisionerImage       = "k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0"
+	DefaultAttacherImage          = "k8s.gcr.io/sig-storage/csi-attacher:v3.4.0"
+	DefaultSnapshotterImage       = "k8s.gcr.io/sig-storage/csi-snapshotter:v5.0.1"
+	DefaultResizerImage           = "k8s.gcr.io/sig-storage/csi-resizer:v1.4.0"
 	DefaultVolumeReplicationImage = "quay.io/csiaddons/volumereplication-operator:v0.3.0"
 	DefaultCSIAddonsImage         = "quay.io/csiaddons/k8s-sidecar:v0.2.1"
 
@@ -371,15 +370,8 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 			return errors.Wrap(err, "failed to load nfs provisioner deployment template")
 		}
 	}
+	multusApplied := len(r.multusClusters) > 0
 
-	holderEnabled := !CSIParam.EnableCSIHostNetwork
-
-	for i := range r.clustersWithHolder {
-		if r.clustersWithHolder[i].cluster.Spec.Network.IsMultus() {
-			holderEnabled = true
-			break
-		}
-	}
 	// get common provisioner tolerations and node affinity
 	provisionerTolerations := getToleration(r.opConfig.Parameters, provisionerTolerationsEnv, []corev1.Toleration{})
 	provisionerNodeAffinity := getNodeAffinity(r.opConfig.Parameters, provisionerNodeAffinityEnv, &corev1.NodeAffinity{})
@@ -388,12 +380,11 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 	pluginTolerations := getToleration(r.opConfig.Parameters, pluginTolerationsEnv, []corev1.Toleration{})
 	pluginNodeAffinity := getNodeAffinity(r.opConfig.Parameters, pluginNodeAffinityEnv, &corev1.NodeAffinity{})
 
-	if holderEnabled && !v.SupportsNsenter() {
-		return errors.Errorf("multus/csi pod networking is applied but the csi version %q does not support it, need at least %q", v.String(), nsenterSupportedVersion.String())
+	if multusApplied && !v.SupportsMultus() {
+		return errors.Errorf("multus is applied but the csi version %q does not support it, need at least %q", v.String(), multusSupportedVersion.String())
 	}
 
-	// Deploy the CSI Holder DaemonSet if Multus is enabled or
-	// EnableCSIHostNetwork is disabled.
+	// Deploy the CSI Holder DaemonSet if Multus is enabled
 	err = r.configureHolders(enabledDrivers, tp, pluginTolerations, pluginNodeAffinity)
 	if err != nil {
 		return errors.Wrap(err, "failed to configure holder")
@@ -411,11 +402,11 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 		if err != nil {
 			return errors.Wrapf(err, "failed to set owner reference to rbd plugin daemonset %q", rbdPlugin.Name)
 		}
-		err = r.applyCephClusterNetworkConfig(r.opManagerContext, &rbdPlugin.Spec.Template.ObjectMeta)
+		_, err := r.applyCephClusterNetworkConfig(r.opManagerContext, &rbdPlugin.Spec.Template.ObjectMeta)
 		if err != nil {
 			return errors.Wrapf(err, "failed to apply network config to rbd plugin daemonset %q", rbdPlugin.Name)
 		}
-		if holderEnabled {
+		if multusApplied {
 			rbdPlugin.Spec.Template.Spec.HostNetwork = false
 		}
 		err = k8sutil.CreateDaemonSet(r.opManagerContext, r.opConfig.OperatorNamespace, r.context.Clientset, rbdPlugin)
@@ -443,7 +434,7 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 			Type: apps.RecreateDeploymentStrategyType,
 		}
 
-		err = r.applyCephClusterNetworkConfig(r.opManagerContext, &rbdProvisionerDeployment.Spec.Template.ObjectMeta)
+		_, err = r.applyCephClusterNetworkConfig(r.opManagerContext, &rbdProvisionerDeployment.Spec.Template.ObjectMeta)
 		if err != nil {
 			return errors.Wrapf(err, "failed to apply network config to rbd plugin provisioner deployment %q", rbdProvisionerDeployment.Name)
 		}
@@ -479,11 +470,11 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 		if err != nil {
 			return errors.Wrapf(err, "failed to set owner reference to cephfs plugin daemonset %q", cephfsPlugin.Name)
 		}
-		err = r.applyCephClusterNetworkConfig(r.opManagerContext, &cephfsPlugin.Spec.Template.ObjectMeta)
+		multusApplied, err := r.applyCephClusterNetworkConfig(r.opManagerContext, &cephfsPlugin.Spec.Template.ObjectMeta)
 		if err != nil {
 			return errors.Wrapf(err, "failed to apply network config to cephfs plugin daemonset %q", cephfsPlugin.Name)
 		}
-		if holderEnabled {
+		if multusApplied {
 			cephfsPlugin.Spec.Template.Spec.HostNetwork = false
 			// HostPID is used to communicate with the network namespace
 			cephfsPlugin.Spec.Template.Spec.HostPID = true
@@ -515,7 +506,7 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 			Type: apps.RecreateDeploymentStrategyType,
 		}
 
-		err = r.applyCephClusterNetworkConfig(r.opManagerContext, &cephfsProvisionerDeployment.Spec.Template.ObjectMeta)
+		_, err = r.applyCephClusterNetworkConfig(r.opManagerContext, &cephfsProvisionerDeployment.Spec.Template.ObjectMeta)
 		if err != nil {
 			return errors.Wrapf(err, "failed to apply network config to cephfs plugin provisioner deployment %q", cephfsProvisionerDeployment.Name)
 		}
@@ -550,11 +541,11 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 		if err != nil {
 			return errors.Wrapf(err, "failed to set owner reference to nfs plugin daemonset %q", nfsPlugin.Name)
 		}
-		err = r.applyCephClusterNetworkConfig(r.opManagerContext, &nfsPlugin.Spec.Template.ObjectMeta)
+		multusApplied, err := r.applyCephClusterNetworkConfig(r.opManagerContext, &nfsPlugin.Spec.Template.ObjectMeta)
 		if err != nil {
 			return errors.Wrapf(err, "failed to apply network config to nfs plugin daemonset %q", nfsPlugin.Name)
 		}
-		if holderEnabled {
+		if multusApplied {
 			nfsPlugin.Spec.Template.Spec.HostNetwork = false
 		}
 		err = k8sutil.CreateDaemonSet(r.opManagerContext, r.opConfig.OperatorNamespace, r.context.Clientset, nfsPlugin)
@@ -583,7 +574,7 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 			Type: apps.RecreateDeploymentStrategyType,
 		}
 
-		err = r.applyCephClusterNetworkConfig(r.opManagerContext, &nfsProvisionerDeployment.Spec.Template.ObjectMeta)
+		_, err = r.applyCephClusterNetworkConfig(r.opManagerContext, &nfsProvisionerDeployment.Spec.Template.ObjectMeta)
 		if err != nil {
 			return errors.Wrapf(err, "failed to apply network config to nfs provisioner deployment %q", nfsProvisionerDeployment.Name)
 		}
@@ -679,21 +670,23 @@ func (r *ReconcileCSI) deleteCSIDriverResources(ver *version.Info, daemonset, de
 	return nil
 }
 
-func (r *ReconcileCSI) applyCephClusterNetworkConfig(ctx context.Context, objectMeta *metav1.ObjectMeta) error {
+func (r *ReconcileCSI) applyCephClusterNetworkConfig(ctx context.Context, objectMeta *metav1.ObjectMeta) (bool, error) {
+	var isMultusApplied bool
 	cephClusters, err := r.context.RookClientset.CephV1().CephClusters(objectMeta.Namespace).List(ctx, metav1.ListOptions{})
 	if err != nil {
-		return errors.Wrap(err, "failed to find CephClusters")
+		return false, errors.Wrap(err, "failed to find CephClusters")
 	}
 	for _, cephCluster := range cephClusters.Items {
 		if cephCluster.Spec.Network.IsMultus() {
 			err = k8sutil.ApplyMultus(cephCluster.Spec.Network, objectMeta)
 			if err != nil {
-				return errors.Wrapf(err, "failed to apply multus configuration to CephCluster %q", cephCluster.Name)
+				return false, errors.Wrapf(err, "failed to apply multus configuration to CephCluster %q", cephCluster.Name)
 			}
+			isMultusApplied = true
 		}
 	}
 
-	return nil
+	return isMultusApplied, nil
 }
 
 // ValidateCSIVersion checks if the configured ceph-csi image is supported
@@ -741,10 +734,6 @@ func (r *ReconcileCSI) validateCSIVersion(ownerInfo *k8sutil.OwnerInfo) (*CephCS
 	}
 	logger.Infof("Detected ceph CSI image version: %q", version)
 
-	// Set the CSI version on a package variable so that each cluster reporting the
-	// telemetry can report the same CSI version
-	telemetry.CSIVersion = version.String()
-
 	if !version.Supported() {
 		return nil, errors.Errorf("ceph CSI image needs to be at least version %q", minimum.String())
 	}
@@ -752,11 +741,11 @@ func (r *ReconcileCSI) validateCSIVersion(ownerInfo *k8sutil.OwnerInfo) (*CephCS
 }
 
 func (r *ReconcileCSI) configureHolders(enabledDrivers []driverDetails, tp templateParam, pluginTolerations []corev1.Toleration, pluginNodeAffinity *corev1.NodeAffinity) error {
-	for _, cluster := range r.clustersWithHolder {
+	for _, multusCluster := range r.multusClusters {
 		for _, driver := range enabledDrivers {
-			err := r.configureHolder(driver, cluster, tp, pluginTolerations, pluginNodeAffinity)
+			err := r.configureHolder(driver, multusCluster, tp, pluginTolerations, pluginNodeAffinity)
 			if err != nil {
-				return errors.Wrapf(err, "failed to configure holder %q for %q/%q", driver.name, cluster.cluster.Name, cluster.cluster.Namespace)
+				return errors.Wrapf(err, "failed to configure holder %q for %q/%q", driver.name, multusCluster.cluster.Name, multusCluster.cluster.Namespace)
 			}
 		}
 	}
@@ -764,7 +753,7 @@ func (r *ReconcileCSI) configureHolders(enabledDrivers []driverDetails, tp templ
 	return nil
 }
 
-func (r *ReconcileCSI) configureHolder(driver driverDetails, c ClusterDetail, tp templateParam, pluginTolerations []corev1.Toleration, pluginNodeAffinity *corev1.NodeAffinity) error {
+func (r *ReconcileCSI) configureHolder(driver driverDetails, multusCluster ClusterDetail, tp templateParam, pluginTolerations []corev1.Toleration, pluginNodeAffinity *corev1.NodeAffinity) error {
 	cephPluginHolder, err := templateToDaemonSet("cephpluginholder", driver.holderTemplate, tp)
 	if err != nil {
 		return errors.Wrapf(err, "failed to load ceph %q plugin holder template", driver.fullName)
@@ -786,7 +775,7 @@ func (r *ReconcileCSI) configureHolder(driver driverDetails, c ClusterDetail, tp
 		cephPluginHolder.Spec.Template.Spec.Containers[0].Env,
 		corev1.EnvVar{
 			Name:  "CEPH_CLUSTER_NAMESPACE",
-			Value: c.cluster.Namespace,
+			Value: multusCluster.cluster.Namespace,
 		},
 	)
 
@@ -801,37 +790,35 @@ func (r *ReconcileCSI) configureHolder(driver driverDetails, c ClusterDetail, tp
 	)
 
 	// Make the DS name unique per Ceph cluster
-	cephPluginHolder.Name = fmt.Sprintf("%s-%s", cephPluginHolder.Name, c.cluster.Name)
-	cephPluginHolder.Spec.Template.Name = fmt.Sprintf("%s-%s", cephPluginHolder.Spec.Template.Name, c.cluster.Name)
-	cephPluginHolder.Spec.Template.Spec.Containers[0].Name = fmt.Sprintf("%s-%s", cephPluginHolder.Spec.Template.Spec.Containers[0].Name, c.cluster.Name)
+	cephPluginHolder.Name = fmt.Sprintf("%s-%s", cephPluginHolder.Name, multusCluster.cluster.Name)
+	cephPluginHolder.Spec.Template.Name = fmt.Sprintf("%s-%s", cephPluginHolder.Spec.Template.Name, multusCluster.cluster.Name)
+	cephPluginHolder.Spec.Template.Spec.Containers[0].Name = fmt.Sprintf("%s-%s", cephPluginHolder.Spec.Template.Spec.Containers[0].Name, multusCluster.cluster.Name)
 
 	// Add default labels
 	k8sutil.AddRookVersionLabelToDaemonSet(cephPluginHolder)
 
-	// If multus is enabled, add the multus plugin label
-	if c.cluster.Spec.Network.IsMultus() {
-		// Apply Multus annotations to daemonset spec
-		err = k8sutil.ApplyMultus(c.cluster.Spec.Network, &cephPluginHolder.Spec.Template.ObjectMeta)
-		if err != nil {
-			return errors.Wrapf(err, "failed to apply multus configuration for holder %q in cluster %q", cephPluginHolder.Name, c.cluster.Namespace)
-		}
+	// Apply Multus annotations to daemonset spec
+	err = k8sutil.ApplyMultus(multusCluster.cluster.Spec.Network, &cephPluginHolder.Spec.Template.ObjectMeta)
+	if err != nil {
+		return errors.Wrapf(err, "failed to apply multus configuration for holder %q in cluster %q", cephPluginHolder.Name, multusCluster.cluster.Namespace)
 	}
+
 	// Finally create the DaemonSet
 	_, err = r.context.Clientset.AppsV1().DaemonSets(r.opConfig.OperatorNamespace).Create(r.opManagerContext, cephPluginHolder, metav1.CreateOptions{})
 	if err != nil {
 		if kerrors.IsAlreadyExists(err) {
-			logger.Debugf("holder %q already exists for cluster %q, it should never be updated", cephPluginHolder.Name, c.cluster.Namespace)
+			logger.Debugf("holder %q already exists for cluster %q, it should never be updated", cephPluginHolder.Name, multusCluster.cluster.Namespace)
 		} else {
 			return errors.Wrapf(err, "failed to start ceph plugin holder daemonset %q", cephPluginHolder.Name)
 		}
 	}
 
 	clusterConfigEntry := &CsiClusterConfigEntry{
-		Monitors: MonEndpoints(c.clusterInfo.Monitors),
+		Monitors: MonEndpoints(multusCluster.clusterInfo.Monitors),
 		RBD:      &CsiRBDSpec{},
 		CephFS:   &CsiCephFSSpec{},
 	}
-	netNamespaceFilePath := generateNetNamespaceFilePath(CSIParam.KubeletDirPath, driver.fullName, c.cluster.Namespace)
+	netNamespaceFilePath := generateNetNamespaceFilePath(CSIParam.KubeletDirPath, driver.fullName, multusCluster.cluster.Namespace)
 	if driver.name == RBDDriverShortName {
 		clusterConfigEntry.RBD.NetNamespaceFilePath = netNamespaceFilePath
 	}
@@ -839,7 +826,7 @@ func (r *ReconcileCSI) configureHolder(driver driverDetails, c ClusterDetail, tp
 		clusterConfigEntry.CephFS.NetNamespaceFilePath = netNamespaceFilePath
 	}
 	// Save the path of the network namespace file for ceph-csi to use
-	err = SaveClusterConfig(r.context.Clientset, c.cluster.Namespace, c.clusterInfo, clusterConfigEntry)
+	err = SaveClusterConfig(r.context.Clientset, multusCluster.cluster.Namespace, multusCluster.clusterInfo, clusterConfigEntry)
 	if err != nil {
 		return errors.Wrapf(err, "failed to save cluster config for csi holder %q", driver.fullName)
 	}
diff --git a/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-holder.yaml b/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-holder.yaml
index e7d447ee9..686f6d1e5 100644
--- a/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-holder.yaml
+++ b/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-holder.yaml
@@ -1,9 +1,9 @@
-# This DaemonSet is only active when the cluster is deployed with Multus enabled or EnableCSIHostNetwork
-# is set to false. Hopefully in the future it will be part of every Ceph-CSI deployment. Its role is to
-# "hold" the Ceph Filesystem mounts that are created by the CSI driver. Essentially, it exposes its
-# network namespace into the Kubelet plugin directly and Ceph-CSI picks it up. When mounting, the Ceph-CSI
-# driver will use the network namespace of the DaemonSet instead of its own (the plugin pod). This pod
-# is not expected to be updated nor restarted unless the node reboots.
+# This DaemonSet is only active when the cluster is deployed with Multus enabled. Hopefully in the
+# future it will be part of every Ceph-CSI deployment. Its role is to "hold" the Ceph Filesystem
+# mounts that are created by the CSI driver. Essentially, it exposes its network namespace into the
+# Kubelet plugin directly and Ceph-CSI picks it up. When mounting, the Ceph-CSI driver will use the
+# network namespace of the DaemonSet instead of its own (the plugin pod). This pod is not expected
+# to be updated nor restarted unless the node reboots.
 kind: DaemonSet
 apiVersion: apps/v1
 metadata:
diff --git a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-holder.yaml b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-holder.yaml
index c64250673..6bf9b080f 100644
--- a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-holder.yaml
+++ b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-holder.yaml
@@ -1,9 +1,9 @@
-# This DaemonSet is only active when the cluster is deployed with Multus enabled or EnableCSIHostNetwork
-# is set to false. Hopefully in the future it will be part of every Ceph-CSI deployment. Its role is
-# to "hold" the network RBD block mapping that are created by the CSI driver. Essentially, it exposes
-# its network namespace into the Kubelet plugin directly and Ceph-CSI picks it up. When mapping an RBD
-# image, the Ceph-CSI driver will use the network namespace of the DaemonSet instead of its own (the plugin pod).
-# This pod is not expected to be updated nor restarted unless the node reboots.
+# This DaemonSet is only active when the cluster is deployed with Multus enabled. Hopefully in the
+# future it will be part of every Ceph-CSI deployment. Its role is to "hold" the network RBD block
+# mapping that are created by the CSI driver. Essentially, it exposes its network namespace into the
+# Kubelet plugin directly and Ceph-CSI picks it up. When mapping an RBD image, the Ceph-CSI driver
+# will use the network namespace of the DaemonSet instead of its own (the plugin pod). This pod is
+# not expected to be updated nor restarted unless the node reboots.
 kind: DaemonSet
 apiVersion: apps/v1
 metadata:
diff --git a/pkg/operator/ceph/csi/version.go b/pkg/operator/ceph/csi/version.go
index 5c27fbf5f..1a7f21b9e 100644
--- a/pkg/operator/ceph/csi/version.go
+++ b/pkg/operator/ceph/csi/version.go
@@ -40,8 +40,8 @@ var (
 	// custom ceph.conf is supported in v3.5.0+
 	cephConfSupportedVersion = CephCSIVersion{3, 5, 0}
 
-	// pod networking fix with the extra holder pod is supported with at least v3.6.1
-	nsenterSupportedVersion = CephCSIVersion{3, 6, 1}
+	// multus fix with the extra holder pod is supported with at least v3.6.1
+	multusSupportedVersion = CephCSIVersion{3, 6, 1}
 
 	// for parsing the output of `cephcsi`
 	versionCSIPattern = regexp.MustCompile(`v(\d+)\.(\d+)\.(\d+)`)
@@ -131,13 +131,13 @@ func (v *CephCSIVersion) SupportsCustomCephConf() bool {
 	return v.isAtLeast(&cephConfSupportedVersion)
 }
 
-// SupportsNsenter checks if the csi image has support for calling "nsenter" while executing
+// SupportsMultus checks if the csi image has support for calling "nsenter" while executing
 // mount/map commands. This is needed for Multus scenarios.
-func (v *CephCSIVersion) SupportsNsenter() bool {
+func (v *CephCSIVersion) SupportsMultus() bool {
 	// if AllowUnsupported is set also a csi-image greater than the supported ones are allowed
 	if AllowUnsupported {
 		return true
 	}
 
-	return v.isAtLeast(&nsenterSupportedVersion)
+	return v.isAtLeast(&multusSupportedVersion)
 }


Test patch:
diff --git a/.github/workflows/canary-integration-test.yml b/.github/workflows/canary-integration-test.yml
index cc4ad864e..052b071c1 100644
--- a/.github/workflows/canary-integration-test.yml
+++ b/.github/workflows/canary-integration-test.yml
@@ -32,7 +32,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -104,12 +104,26 @@ jobs:
     - name: test external script with restricted_auth_permission flag and without having cephfs_filesystem flag
       run: |
         toolbox=$(kubectl get pod -l app=rook-ceph-tools -n rook-ceph -o jsonpath='{.items[*].metadata.name}')
-        kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --cluster-name rookstorage --restricted-auth-permission true
+        # create `radosNamespace1` rados-namespace for `replicapool` rbd data-pool
+        kubectl -n rook-ceph exec $toolbox -- rbd namespace create replicapool/radosNamespace1
+        kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rados-namespace radosNamespace1 --cluster-name rookStorage --restricted-auth-permission true
+        # test the CRD created rados namespace `namespace-a` using external script
+        kubectl create -f deploy/examples/radosnamespace.yaml
+        timeout 60 sh -c "until kubectl -n rook-ceph exec $toolbox -- rbd namespace ls replicapool --format=json|jq .[0].name|grep -q "namespace-a"; do sleep 1 && echo 'waiting for the rados namespace to be created'; done"
+        kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rados-namespace namespace-a --cluster-name rookStorage --restricted-auth-permission true
+        # test the rados namespace which not exit for replicapool(false testing)
+        if output=$(kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rados-namespace false-test-namespace --cluster-name rookStorage --restricted-auth-permission true); then
+          echo $output
+        else
+          echo "script failed because wrong rados namespace was passed"
+        fi  
 
     - name: test external script with restricted_auth_permission flag
       run: |
         toolbox=$(kubectl get pod -l app=rook-ceph-tools -n rook-ceph -o jsonpath='{.items[*].metadata.name}')
-        kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --cephfs-filesystem-name myfs --rbd-data-pool-name replicapool --cluster-name rookstorage --restricted-auth-permission true
+        # create `radosNamespace` rados-namespace for `replicapool` rbd data-pool
+        kubectl -n rook-ceph exec $toolbox -- rbd namespace create replicapool/radosNamespace
+        kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --cephfs-filesystem-name myfs --rbd-data-pool-name replicapool --rados-namespace radosNamespace --cluster-name rookStorage --restricted-auth-permission true
 
     - name: test the upgrade flag
       run: |
@@ -125,12 +139,12 @@ jobs:
       run: |
         toolbox=$(kubectl get pod -l app=rook-ceph-tools -n rook-ceph -o jsonpath='{.items[*].metadata.name}')
         # print existing client auth
-        kubectl -n rook-ceph exec $toolbox -- ceph auth get client.csi-rbd-node-rookstorage-replicapool
-        # restricted auth user need to provide --rbd-data-pool-name,
+        kubectl -n rook-ceph exec $toolbox -- ceph auth get client.csi-rbd-node-rookStorage-replicapool-radosNamespace
+        # restricted auth user need to provide --rbd-data-pool-name, --rados-namespace,
         #  --cluster-name and --run-as-user flag while upgrading
-        kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --upgrade --rbd-data-pool-name replicapool --cluster-name rookstorage  --run-as-user client.csi-rbd-node-rookstorage-replicapool
+        kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --upgrade --rbd-data-pool-name replicapool --rados-namespace radosNamespace --cluster-name rookStorage  --run-as-user client.csi-rbd-node-rookStorage-replicapool-radosNamespace
         # print ugraded client auth
-        kubectl -n rook-ceph exec $toolbox -- ceph auth get client.csi-rbd-node-rookstorage-replicapool
+        kubectl -n rook-ceph exec $toolbox -- ceph auth get client.csi-rbd-node-rookStorage-replicapool-radosNamespace
 
     - name: check-ownerreferences
       run: tests/scripts/github-action-helper.sh check_ownerreferences
@@ -170,9 +184,9 @@ jobs:
       uses: actions/checkout@v2
       with:
         fetch-depth: 0
-
+    
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
       env:
@@ -193,10 +207,10 @@ jobs:
       run: tests/scripts/github-action-helper.sh wait_for_prepare_pod
 
     - name: wait for ceph to be ready
-      run: tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready all 2
+      run: tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready all 2 
 
     - name: validate-rgw-endpoint
-      run: |
+      run: | 
         rgw_endpoint=$(kubectl get service -n rook-ceph |  awk '/rgw/ {print $3":80"}')
         toolbox=$(kubectl get pod -l app=rook-ceph-tools -n rook-ceph -o jsonpath='{.items[*].metadata.name}')
         timeout 15 sh -c "until kubectl -n rook-ceph exec $toolbox -- ceph mgr dump -f json|jq --raw-output .active_addr|grep -Eosq \"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\" ; do sleep 1 && echo 'waiting for the manager IP to be available'; done"
@@ -204,13 +218,13 @@ jobs:
         timeout 60 sh -c "until kubectl -n rook-ceph exec $toolbox -- curl --silent --show-error ${mgr_raw%%:*}:9283; do echo 'waiting for mgr prometheus exporter to be ready' && sleep 1; done"
         kubectl -n rook-ceph cp deploy/examples/create-external-cluster-resources.py $toolbox:/etc/ceph
         # pass the valid rgw-endpoint of same ceph cluster
-        timeout 15 sh -c "until kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rgw-endpoint $rgw_endpoint; do sleep 1 && echo 'waiting for the rgw endpoint to be validated'; done"
+        timeout 15 sh -c "until kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rgw-endpoint $rgw_endpoint; do sleep 1 && echo 'waiting for the rgw endpoint to be validated'; done"        
         # pass the invalid rgw-endpoint of different ceph cluster
         if output=$(timeout 15 sh -c "until kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rgw-endpoint 10.108.96.128:80; do sleep 1 && echo 'waiting for the rgw endpoint to be validated'; done"); then
           echo $output
         else
           echo "validation failed because wrong endpoint was provided"
-        fi
+        fi    
         # pass the valid rgw-endpoint of same ceph cluster with --rgw-tls-cert-path
         timeout 15 sh -c "until kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rgw-endpoint $rgw_endpoint --rgw-tls-cert-path my-cert; do sleep 1 && echo 'waiting for the rgw endpoint to be validated'; done"
         # pass the valid rgw-endpoint of same ceph cluster with --rgw-skip-tls
@@ -237,7 +251,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -283,7 +297,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -329,7 +343,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -338,6 +352,7 @@ jobs:
 
     - name: use local disk as OSD
       run: |
+
         BLOCK=$(sudo lsblk --paths|awk '/14G/ {print $1}'| head -1)
         tests/scripts/github-action-helper.sh use_local_disk
         tests/scripts/create-bluestore-partitions.sh --disk "$BLOCK" --wipe-only
@@ -371,52 +386,6 @@ jobs:
       uses: mxschmitt/action-tmate@v3
       timeout-minutes: 60
 
-  encryption:
-    runs-on: ubuntu-18.04
-    if: "!contains(github.event.pull_request.labels.*.name, 'skip-ci')"
-    steps:
-    - name: checkout
-      uses: actions/checkout@v2
-      with:
-        fetch-depth: 0
-
-    - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
-      with:
-        github-token: ${{ secrets.GITHUB_TOKEN }}
-
-    - name: validate-yaml
-      run: tests/scripts/github-action-helper.sh validate_yaml
-
-    - name: use local disk as OSD
-      run: |
-        BLOCK=$(sudo lsblk --paths|awk '/14G/ {print $1}'| head -1)
-        tests/scripts/github-action-helper.sh use_local_disk
-        tests/scripts/create-bluestore-partitions.sh --disk "$BLOCK" --wipe-only
-
-    - name: deploy cluster
-      run: tests/scripts/github-action-helper.sh deploy_cluster encryptiton
-
-    - name: wait for prepare pod
-      run: tests/scripts/github-action-helper.sh wait_for_prepare_pod
-
-    - name: wait for ceph to be ready
-      run: tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready osd 1
-
-    - name: check-ownerreferences
-      run: tests/scripts/github-action-helper.sh check_ownerreferences
-
-    - name: collect common logs
-      if: always()
-      uses: ./.github/workflows/collect-logs
-      with:
-        name: canary
-
-    - name: setup tmate session for debugging when event is PR
-      if: failure() && github.event_name == 'pull_request'
-      uses: mxschmitt/action-tmate@v3
-      timeout-minutes: 60
-
   pvc:
     runs-on: ubuntu-18.04
     if: "!contains(github.event.pull_request.labels.*.name, 'skip-ci')"
@@ -427,7 +396,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -492,7 +461,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -540,7 +509,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -591,7 +560,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -659,7 +628,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -709,7 +678,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -760,7 +729,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -838,7 +807,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -899,7 +868,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -952,7 +921,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -1200,7 +1169,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
 
@@ -1222,7 +1191,7 @@ jobs:
       run: tests/scripts/github-action-helper.sh wait_for_prepare_pod
 
     - name: wait for ceph to be ready
-      run: IS_POD_NETWORK=true IS_MULTUS=true tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready osd_multus 2
+      run: IS_MULTUS=true tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready osd_multus 2
 
     - name: wait for ceph-csi configmap to be updated with network namespace
       run: tests/scripts/github-action-helper.sh wait_for_ceph_csi_configmap_to_be_updated
@@ -1243,52 +1212,3 @@ jobs:
       if: failure() && github.event_name == 'pull_request'
       uses: mxschmitt/action-tmate@v3
       timeout-minutes: 60
-
-  csi-hostnetwork-disabled:
-    runs-on: ubuntu-18.04
-    if: "!contains(github.event.pull_request.labels.*.name, 'skip-ci')"
-    steps:
-    - name: checkout
-      uses: actions/checkout@v2
-      with:
-        fetch-depth: 0
-
-    - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
-      with:
-        github-token: ${{ secrets.GITHUB_TOKEN }}
-
-
-    - name: use local disk and create partitions for osds
-      run: |
-        tests/scripts/github-action-helper.sh use_local_disk
-        tests/scripts/github-action-helper.sh create_partitions_for_osds
-
-    - name: deploy CSI hostNetworking disabled cluster
-      run: tests/scripts/github-action-helper.sh deploy_csi_hostnetwork_disabled_cluster
-
-    - name: wait for prepare pod
-      run: tests/scripts/github-action-helper.sh wait_for_prepare_pod
-
-    - name: wait for ceph to be ready
-      run: IS_POD_NETWORK=true tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready osd 2
-
-    - name: wait for ceph-csi configmap to be updated with network namespace
-      run: tests/scripts/github-action-helper.sh wait_for_ceph_csi_configmap_to_be_updated
-
-    - name: test ceph-csi-rbd plugin restart
-      run: tests/scripts/github-action-helper.sh test_csi_rbd_workload
-
-    - name: test ceph-csi-cephfs plugin restart
-      run: tests/scripts/github-action-helper.sh test_csi_cephfs_workload
-
-    - name: collect common logs
-      if: always()
-      uses: ./.github/workflows/collect-logs
-      with:
-        name: csi-hostnetwork-disabled
-
-    - name: setup tmate session for debugging when event is PR
-      if: failure() && github.event_name == 'pull_request'
-      uses: mxschmitt/action-tmate@v3
-      timeout-minutes: 60
diff --git a/.github/workflows/integration-test-helm-suite.yaml b/.github/workflows/integration-test-helm-suite.yaml
index 88877aa9c..0a95c663e 100644
--- a/.github/workflows/integration-test-helm-suite.yaml
+++ b/.github/workflows/integration-test-helm-suite.yaml
@@ -30,7 +30,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
diff --git a/.github/workflows/integration-test-mgr-suite.yaml b/.github/workflows/integration-test-mgr-suite.yaml
index 08c8fdc6a..ebdb704ec 100644
--- a/.github/workflows/integration-test-mgr-suite.yaml
+++ b/.github/workflows/integration-test-mgr-suite.yaml
@@ -29,7 +29,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
diff --git a/.github/workflows/integration-test-multi-cluster-suite.yaml b/.github/workflows/integration-test-multi-cluster-suite.yaml
index 782090580..80325e16f 100644
--- a/.github/workflows/integration-test-multi-cluster-suite.yaml
+++ b/.github/workflows/integration-test-multi-cluster-suite.yaml
@@ -30,7 +30,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
diff --git a/.github/workflows/integration-test-object-suite.yaml b/.github/workflows/integration-test-object-suite.yaml
index 4ecc57300..3e2cc7f91 100644
--- a/.github/workflows/integration-test-object-suite.yaml
+++ b/.github/workflows/integration-test-object-suite.yaml
@@ -30,7 +30,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
diff --git a/.github/workflows/integration-test-config/action.yaml b/.github/workflows/integration-test-setup/action.yaml
similarity index 93%
rename from .github/workflows/integration-test-config/action.yaml
rename to .github/workflows/integration-test-setup/action.yaml
index e0c93ac19..7f64f885d 100644
--- a/.github/workflows/integration-test-config/action.yaml
+++ b/.github/workflows/integration-test-setup/action.yaml
@@ -28,7 +28,6 @@ runs:
     shell: bash --noprofile --norc -eo pipefail -x {0}
     run: tests/scripts/github-action-helper.sh print_k8s_cluster_status
 
-  # This step is required for all the integration tests except the canary tests
   - name: use local disk
     shell: bash --noprofile --norc -eo pipefail -x {0}
     run: tests/scripts/github-action-helper.sh use_local_disk_for_integration_test
diff --git a/.github/workflows/integration-test-smoke-suite.yaml b/.github/workflows/integration-test-smoke-suite.yaml
index 875b3c99c..c6ed08a13 100644
--- a/.github/workflows/integration-test-smoke-suite.yaml
+++ b/.github/workflows/integration-test-smoke-suite.yaml
@@ -30,7 +30,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
diff --git a/.github/workflows/integration-test-upgrade-suite.yaml b/.github/workflows/integration-test-upgrade-suite.yaml
index f8c8f77f6..e75f5621a 100644
--- a/.github/workflows/integration-test-upgrade-suite.yaml
+++ b/.github/workflows/integration-test-upgrade-suite.yaml
@@ -30,7 +30,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
@@ -75,7 +75,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
diff --git a/.github/workflows/integration-tests-on-release.yaml b/.github/workflows/integration-tests-on-release.yaml
index 8f12772e0..0f8c747b9 100644
--- a/.github/workflows/integration-tests-on-release.yaml
+++ b/.github/workflows/integration-tests-on-release.yaml
@@ -26,7 +26,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
@@ -67,7 +67,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
@@ -107,7 +107,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
@@ -146,7 +146,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
@@ -185,7 +185,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
@@ -228,7 +228,7 @@ jobs:
         fetch-depth: 0
 
     - name: setup cluster resources
-      uses: ./.github/workflows/integration-test-config
+      uses: ./.github/workflows/integration-test-setup
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         kubernetes-version: ${{ matrix.kubernetes-versions }}
diff --git a/.github/workflows/rgw-multisite-test/action.yml b/.github/workflows/rgw-multisite-test/action.yml
index b1697ebc1..ca37c24f7 100644
--- a/.github/workflows/rgw-multisite-test/action.yml
+++ b/.github/workflows/rgw-multisite-test/action.yml
@@ -5,14 +5,14 @@ inputs:
     description: GITHUB_TOKEN from the calling workflow
     required: true
   ceph-image:
-    description: Ceph image to use for the workflow (e.g., quay.io/ceph/ceph:v17.2.0)
+    description: Ceph image to use for the workflow (e.g., quay.io/ceph/ceph:v16.2.7)
     required: false
 
 runs:
   using: "composite"
   steps:
     - name: setup cluster resources
-      uses: ./.github/workflows/canary-test-config
+      uses: ./.github/workflows/setup-cluster-resources
       with:
         github-token: ${{ inputs.github-token }}
 
diff --git a/deploy/examples/create-external-cluster-resources-tests.py b/deploy/examples/create-external-cluster-resources-tests.py
index f1e3a18e0..06c6b8769 100644
--- a/deploy/examples/create-external-cluster-resources-tests.py
+++ b/deploy/examples/create-external-cluster-resources-tests.py
@@ -1,4 +1,4 @@
-"""
+'''
 Copyright 2022 The Rook Authors. All rights reserved.
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -9,10 +9,9 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-"""
+'''
 
 import unittest
-
 # This package is available since python 3.x and this is the unit test so backward compatibility will not be an issue.
 import importlib
 import sys
@@ -35,13 +34,7 @@ ext = importlib.import_module("create-external-cluster-resources")
 class TestRadosJSON(unittest.TestCase):
     def setUp(self):
         print("\nI am in setup")
-        self.rjObj = ext.RadosJSON(
-            [
-                "--rbd-data-pool-name=abc",
-                "--rgw-endpoint=10.10.212.122:9000",
-                "--format=json",
-            ]
-        )
+        self.rjObj = ext.RadosJSON(['--rbd-data-pool-name=abc', '--rgw-endpoint=10.10.212.122:9000', '--format=json'])
         # for testing, we are using 'DummyRados' object
         self.rjObj.cluster = ext.DummyRados.Rados()
 
@@ -58,7 +51,7 @@ class TestRadosJSON(unittest.TestCase):
         self.rjObj.main()
         print("\n\nNon compatible output (--abcd)")
         try:
-            self.rjObj._arg_parser.format = "abcd"
+            self.rjObj._arg_parser.format = 'abcd'
             self.rjObj.main()
             self.fail("Function should have thrown an Exception")
         except ext.ExecutionFailureException as err:
@@ -66,44 +59,36 @@ class TestRadosJSON(unittest.TestCase):
 
     def test_method_create_cephCSIKeyring_cephFSProvisioner(self):
         csiKeyring = self.rjObj.create_cephCSIKeyring_user(
-            "client.csi-cephfs-provisioner"
-        )
-        print(
-            "cephCSIKeyring without restricting it to a metadata pool. {}".format(
-                csiKeyring
-            )
-        )
+            "client.csi-cephfs-provisioner")
+        print("cephCSIKeyring without restricting it to a metadata pool. {}".format(
+            csiKeyring))
         self.rjObj._arg_parser.restricted_auth_permission = True
         self.rjObj._arg_parser.cluster_name = "openshift-storage"
         csiKeyring = self.rjObj.create_cephCSIKeyring_user(
-            "client.csi-cephfs-provisioner"
-        )
+            "client.csi-cephfs-provisioner")
         print("cephCSIKeyring for a specific cluster. {}".format(csiKeyring))
         self.rjObj._arg_parser.cephfs_filesystem_name = "myfs"
         csiKeyring = self.rjObj.create_cephCSIKeyring_user(
-            "client.csi-cephfs-provisioner"
-        )
-        print(
-            "cephCSIKeyring for a specific metadata pool and cluster. {}".format(
-                csiKeyring
-            )
-        )
+            "client.csi-cephfs-provisioner")
+        print("cephCSIKeyring for a specific metadata pool and cluster. {}".format(
+            csiKeyring))
 
     def test_non_zero_return_and_error(self):
         self.rjObj.cluster.return_val = 1
         self.rjObj.cluster.err_message = "Dummy Error"
         try:
             self.rjObj.create_checkerKey()
-            self.fail("Failed to raise an exception, 'ext.ExecutionFailureException'")
+            self.fail(
+                "Failed to raise an exception, 'ext.ExecutionFailureException'")
         except ext.ExecutionFailureException as err:
             print("Successfully thrown error.\nError: {}".format(err))
 
     def test_multi_filesystem_scenario(self):
-        cmd_key = self.rjObj.cluster.cmd_names["fs ls"]
+        cmd_key = self.rjObj.cluster.cmd_names['fs ls']
         cmd_out = self.rjObj.cluster.cmd_output_map[cmd_key]
         cmd_json_out = json.loads(cmd_out)
         second_fs_details = dict(cmd_json_out[0])
-        second_fs_details["name"] += "-2"
+        second_fs_details['name'] += '-2'
         cmd_json_out.append(second_fs_details)
         self.rjObj.cluster.cmd_output_map[cmd_key] = json.dumps(cmd_json_out)
         # multiple filesystem present,
@@ -113,13 +98,10 @@ class TestRadosJSON(unittest.TestCase):
             print("As we are returning silently, no error thrown as expected")
         except ext.ExecutionFailureException as err:
             self.fail(
-                "Supposed to get returned silently, but instead error thrown: {}".format(
-                    err
-                )
-            )
+                "Supposed to get returned silently, but instead error thrown: {}".format(err))
         # pass an existing filesystem name
         try:
-            self.rjObj._arg_parser.cephfs_filesystem_name = second_fs_details["name"]
+            self.rjObj._arg_parser.cephfs_filesystem_name = second_fs_details['name']
             self.rjObj.get_cephfs_data_pool_details()
         except ext.ExecutionFailureException as err:
             self.fail("Should not have thrown error: {}".format(err))
@@ -139,18 +121,18 @@ class TestRadosJSON(unittest.TestCase):
             print("Successfully thrown error: {}".format(err))
 
     def test_multi_data_pool_scenario(self):
-        cmd_key = self.rjObj.cluster.cmd_names["fs ls"]
+        cmd_key = self.rjObj.cluster.cmd_names['fs ls']
         cmd_out = self.rjObj.cluster.cmd_output_map[cmd_key]
         cmd_json_out = json.loads(cmd_out)
         first_fs_details = cmd_json_out[0]
-        new_data_pool_name = "myfs-data1"
-        first_fs_details["data_pools"].append(new_data_pool_name)
+        new_data_pool_name = 'myfs-data1'
+        first_fs_details['data_pools'].append(new_data_pool_name)
         print("Modified JSON Cmd Out: {}".format(cmd_json_out))
         self.rjObj._arg_parser.cephfs_data_pool_name = new_data_pool_name
         self.rjObj.cluster.cmd_output_map[cmd_key] = json.dumps(cmd_json_out)
         self.rjObj.get_cephfs_data_pool_details()
         # use a non-existing data-pool-name
-        bad_data_pool_name = "myfs-data3"
+        bad_data_pool_name = 'myfs-data3'
         self.rjObj._arg_parser.cephfs_data_pool_name = bad_data_pool_name
         try:
             self.rjObj.get_cephfs_data_pool_details()
@@ -158,7 +140,7 @@ class TestRadosJSON(unittest.TestCase):
         except ext.ExecutionFailureException as err:
             print("Successfully thrown error: {}".format(err))
         # empty data-pool scenario
-        first_fs_details["data_pools"] = []
+        first_fs_details['data_pools'] = []
         self.rjObj.cluster.cmd_output_map[cmd_key] = json.dumps(cmd_json_out)
         try:
             self.rjObj.get_cephfs_data_pool_details()
@@ -196,50 +178,35 @@ class TestRadosJSON(unittest.TestCase):
             self.fail("An Exception was expected to be thrown")
         except ext.ExecutionFailureException as err:
             print("Successfully thrown error: {}".format(err))
-
+            
     def test_convert_fqdn_rgw_endpoint_to_ip(self):
         try:
-            rgw_endpoint_ip = self.rjObj.convert_fqdn_rgw_endpoint_to_ip(
-                "www.redhat.com:80"
-            )
-            print(
-                "Successfully Converted www.redhat.com to it's IP {}".format(
-                    rgw_endpoint_ip
-                )
-            )
-        except ext.ExecutionFailureException as err:
-            print("Successfully thrown error: {}".format(err))
-
+            rgw_endpoint_ip = self.rjObj.convert_fqdn_rgw_endpoint_to_ip("www.redhat.com:80")
+            print("Successfully Converted www.redhat.com to it's IP {}".format(rgw_endpoint_ip))
+        except  ext.ExecutionFailureException as err:
+            print("Successfully thrown error: {}".format(err))   
+        
     def test_upgrade_user_permissions(self):
         self.rjObj = ext.RadosJSON(
-            [
-                "--upgrade",
-                "--run-as-user=client.csi-cephfs-provisioner",
-                "--format=json",
-            ]
-        )
+            ['--upgrade', '--run-as-user=client.csi-cephfs-provisioner', '--format=json'])
         # for testing, we are using 'DummyRados' object
         self.rjObj.cluster = ext.DummyRados.Rados()
         self.rjObj.main()
 
     def test_monitoring_endpoint_validation(self):
-        self.rjObj = ext.RadosJSON(["--rbd-data-pool-name=abc", "--format=json"])
+        self.rjObj = ext.RadosJSON(
+            ['--rbd-data-pool-name=abc', '--format=json'])
         self.rjObj.cluster = ext.DummyRados.Rados()
 
-        valid_ip_ports = [
-            ("10.22.31.131", "3534"),
-            ("10.177.3.81", ""),
-            ("", ""),
-            ("", "9092"),
-        ]
+        valid_ip_ports = [("10.22.31.131", "3534"),
+                          ("10.177.3.81", ""), ("", ""), ("", "9092")]
         for each_ip_port_pair in valid_ip_ports:
             # reset monitoring ip and port
-            self.rjObj._arg_parser.monitoring_endpoint = ""
-            self.rjObj._arg_parser.monitoring_endpoint_port = ""
+            self.rjObj._arg_parser.monitoring_endpoint = ''
+            self.rjObj._arg_parser.monitoring_endpoint_port = ''
             new_mon_ip, new_mon_port = each_ip_port_pair
             check_ip_val = self.rjObj.cluster.dummy_host_ip_map.get(
-                new_mon_ip, new_mon_ip
-            )
+                new_mon_ip, new_mon_ip)
             check_port_val = ext.RadosJSON.DEFAULT_MONITORING_ENDPOINT_PORT
             if new_mon_ip:
                 self.rjObj._arg_parser.monitoring_endpoint = new_mon_ip
@@ -250,28 +217,19 @@ class TestRadosJSON(unittest.TestCase):
             mon_ips, mon_port = self.rjObj.get_active_and_standby_mgrs()
             mon_ip = mon_ips.split(",")[0]
             if check_ip_val and check_ip_val != mon_ip:
-                self.fail(
-                    "Expected IP: {}, Returned IP: {}".format(check_ip_val, mon_ip)
-                )
+                self.fail("Expected IP: {}, Returned IP: {}".format(
+                    check_ip_val, mon_ip))
             if check_port_val and check_port_val != mon_port:
-                self.fail(
-                    "Expected Port: '{}', Returned Port: '{}'".format(
-                        check_port_val, mon_port
-                    )
-                )
+                self.fail("Expected Port: '{}', Returned Port: '{}'".format(
+                    check_port_val, mon_port))
             print("MonIP: {}, MonPort: {}".format(mon_ip, mon_port))
 
-        invalid_ip_ports = [
-            ("10.22.31.131.43", "5334"),
-            ("", "91943"),
-            ("10.177.3.81", "90320"),
-            ("", "73422"),
-            ("10.232.12.8", "90922"),
-        ]
+        invalid_ip_ports = [("10.22.31.131.43", "5334"), ("", "91943"),
+                            ("10.177.3.81", "90320"), ("", "73422"), ("10.232.12.8", "90922")]
         for each_ip_port_pair in invalid_ip_ports:
             # reset the command-line monitoring args
-            self.rjObj._arg_parser.monitoring_endpoint = ""
-            self.rjObj._arg_parser.monitoring_endpoint_port = ""
+            self.rjObj._arg_parser.monitoring_endpoint = ''
+            self.rjObj._arg_parser.monitoring_endpoint_port = ''
             new_mon_ip, new_mon_port = each_ip_port_pair
             if new_mon_ip:
                 self.rjObj._arg_parser.monitoring_endpoint = new_mon_ip
diff --git a/pkg/operator/ceph/cluster/cluster_test.go b/pkg/operator/ceph/cluster/cluster_test.go
index a53a29014..c0512bad2 100644
--- a/pkg/operator/ceph/cluster/cluster_test.go
+++ b/pkg/operator/ceph/cluster/cluster_test.go
@@ -25,8 +25,6 @@ import (
 	cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
 	"github.com/rook/rook/pkg/clusterd"
 	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
-	"github.com/rook/rook/pkg/operator/ceph/cluster/mon"
-	"github.com/rook/rook/pkg/operator/ceph/cluster/telemetry"
 	cephver "github.com/rook/rook/pkg/operator/ceph/version"
 	testop "github.com/rook/rook/pkg/operator/test"
 	exectest "github.com/rook/rook/pkg/util/exec/test"
@@ -218,77 +216,3 @@ func TestConfigureMsgr2(t *testing.T) {
 		})
 	}
 }
-
-func TestTelemetry(t *testing.T) {
-	var expectedSettings map[string]string
-	context := &clusterd.Context{Clientset: testop.New(t, 3)}
-	context.Executor = &exectest.MockExecutor{
-		MockExecuteCommandWithTimeout: func(timeout time.Duration, command string, args ...string) (string, error) {
-			logger.Infof("Command: %s %v", command, args)
-			if args[0] == "config-key" && args[1] == "set" {
-				key := args[2]
-				if key == telemetry.RookVersionKey {
-					// the rook version will vary depending on the build, so it just shouldn't be blank
-					assert.NotEqual(t, "", args[3])
-				} else {
-					assert.Equal(t, expectedSettings[key], args[3], key)
-				}
-				return "", nil
-			}
-			return "", errors.New("mock error to simulate failure of mon store config")
-		},
-	}
-	c := cluster{
-		context:     context,
-		ClusterInfo: cephclient.AdminTestClusterInfo("cluster"),
-		mons:        &mon.Cluster{},
-	}
-
-	t.Run("normal cluster", func(t *testing.T) {
-		c.Spec = &cephv1.ClusterSpec{
-			Mon: cephv1.MonSpec{Count: 3, AllowMultiplePerNode: true},
-			Storage: cephv1.StorageScopeSpec{
-				StorageClassDeviceSets: []cephv1.StorageClassDeviceSet{
-					{Name: "one", Count: 3, Portable: true},
-					{Name: "two", Count: 3, Portable: true},
-					{Name: "three", Count: 3, Portable: false},
-				},
-			},
-			Network: cephv1.NetworkSpec{Provider: "host"},
-		}
-		expectedSettings = map[string]string{
-			telemetry.K8sVersionKey:              "v0.0.0-master+$Format:%H$",
-			telemetry.MonMaxIDKey:                "0",
-			telemetry.MonCountKey:                "3",
-			telemetry.MonAllowMultiplePerNodeKey: "true",
-			telemetry.MonPVCEnabledKey:           "false",
-			telemetry.MonStretchEnabledKey:       "false",
-			telemetry.DeviceSetTotalKey:          "3",
-			telemetry.DeviceSetPortableKey:       "2",
-			telemetry.DeviceSetNonPortableKey:    "1",
-			telemetry.NetworkProviderKey:         "host",
-			telemetry.ExternalModeEnabledKey:     "false",
-		}
-		c.reportTelemetry()
-	})
-
-	t.Run("external cluster", func(t *testing.T) {
-		c.Spec = &cephv1.ClusterSpec{
-			External: cephv1.ExternalSpec{Enable: true},
-		}
-		expectedSettings = map[string]string{
-			telemetry.K8sVersionKey:              "v0.0.0-master+$Format:%H$",
-			telemetry.MonMaxIDKey:                "0",
-			telemetry.MonCountKey:                "0",
-			telemetry.MonAllowMultiplePerNodeKey: "false",
-			telemetry.MonPVCEnabledKey:           "false",
-			telemetry.MonStretchEnabledKey:       "false",
-			telemetry.DeviceSetTotalKey:          "0",
-			telemetry.DeviceSetPortableKey:       "0",
-			telemetry.DeviceSetNonPortableKey:    "0",
-			telemetry.NetworkProviderKey:         "",
-			telemetry.ExternalModeEnabledKey:     "true",
-		}
-		c.reportTelemetry()
-	})
-}
diff --git a/pkg/operator/ceph/csi/spec_test.go b/pkg/operator/ceph/csi/spec_test.go
index bc8c74534..d530e25e1 100644
--- a/pkg/operator/ceph/csi/spec_test.go
+++ b/pkg/operator/ceph/csi/spec_test.go
@@ -42,9 +42,9 @@ func TestReconcileCSI_configureHolders(t *testing.T) {
 			Clientset:     testop.New(t, 1),
 			RookClientset: rookclient.NewSimpleClientset(),
 		},
-		opManagerContext:   context.TODO(),
-		opConfig:           opcontroller.OperatorConfig{},
-		clustersWithHolder: []ClusterDetail{},
+		opManagerContext: context.TODO(),
+		opConfig:         opcontroller.OperatorConfig{},
+		multusClusters:   []ClusterDetail{},
 	}
 
 	t.Run("no clusters, noop", func(t *testing.T) {
@@ -68,7 +68,7 @@ func TestReconcileCSI_configureHolders(t *testing.T) {
 			Namespace: r.opConfig.OperatorNamespace,
 		}
 
-		r.clustersWithHolder = []ClusterDetail{
+		r.multusClusters = []ClusterDetail{
 			{
 				cluster: &cephv1.CephCluster{
 					ObjectMeta: metav1.ObjectMeta{Namespace: "rook-ceph", Name: "my-cluster"},
diff --git a/pkg/operator/ceph/csi/version_test.go b/pkg/operator/ceph/csi/version_test.go
index 6ee5b8455..c1ef2bbcd 100644
--- a/pkg/operator/ceph/csi/version_test.go
+++ b/pkg/operator/ceph/csi/version_test.go
@@ -114,22 +114,22 @@ func TestSupportsCustomCephConf(t *testing.T) {
 	assert.True(t, ret)
 }
 
-func TestSupportsNsenter(t *testing.T) {
+func TestSupportsMultus(t *testing.T) {
 	t.Run("AllowUnsupported=true regardless of the version", func(t *testing.T) {
 		AllowUnsupported = true
-		ret := testMinVersion.SupportsNsenter()
+		ret := testMinVersion.SupportsMultus()
 		assert.True(t, ret)
 	})
 
 	t.Run("AllowUnsupported=false and version 3.5 is too old", func(t *testing.T) {
 		AllowUnsupported = false
-		ret := testMinVersion.SupportsNsenter()
+		ret := testMinVersion.SupportsMultus()
 		assert.False(t, ret)
 	})
 
 	t.Run("AllowUnsupported=false and version 3.6.1 is fine", func(t *testing.T) {
 		AllowUnsupported = false
-		ret := testReleaseV361.SupportsNsenter()
+		ret := testReleaseV361.SupportsMultus()
 		assert.True(t, ret)
 	})
 }
diff --git a/tests/framework/installer/ceph_manifests.go b/tests/framework/installer/ceph_manifests.go
index 448bd32a0..35f174f56 100644
--- a/tests/framework/installer/ceph_manifests.go
+++ b/tests/framework/installer/ceph_manifests.go
@@ -244,7 +244,7 @@ func (m *CephManifestsMaster) getClusterPriorityClasses() string {
 func (m *CephManifestsMaster) GetBlockSnapshotClass(snapshotClassName, reclaimPolicy string) string {
 	// Create a CSI driver snapshotclass
 	return `
-apiVersion: snapshot.storage.k8s.io/v1
+apiVersion: snapshot.storage.k8s.io/v1beta1
 kind: VolumeSnapshotClass
 metadata:
   name: ` + snapshotClassName + `
@@ -260,7 +260,7 @@ parameters:
 func (m *CephManifestsMaster) GetFileStorageSnapshotClass(snapshotClassName, reclaimPolicy string) string {
 	// Create a CSI driver snapshotclass
 	return `
-apiVersion: snapshot.storage.k8s.io/v1
+apiVersion: snapshot.storage.k8s.io/v1beta1
 kind: VolumeSnapshotClass
 metadata:
   name: ` + snapshotClassName + `
diff --git a/tests/framework/installer/general_manifests.go b/tests/framework/installer/general_manifests.go
index 37f548aa7..e544cbf78 100644
--- a/tests/framework/installer/general_manifests.go
+++ b/tests/framework/installer/general_manifests.go
@@ -98,7 +98,7 @@ spec:
 }
 
 func GetSnapshot(snapshotName, claimName, snapshotClassName, namespace string) string {
-	return `apiVersion: snapshot.storage.k8s.io/v1
+	return `apiVersion: snapshot.storage.k8s.io/v1beta1
 kind: VolumeSnapshot
 metadata:
   name: ` + snapshotName + `
diff --git a/tests/integration/ceph_smoke_test.go b/tests/integration/ceph_smoke_test.go
index 679442b8d..e6fee02e0 100644
--- a/tests/integration/ceph_smoke_test.go
+++ b/tests/integration/ceph_smoke_test.go
@@ -23,7 +23,6 @@ import (
 	"testing"
 	"time"
 
-	cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
 	"github.com/rook/rook/pkg/daemon/ceph/client"
 	opcontroller "github.com/rook/rook/pkg/operator/ceph/controller"
 	"github.com/rook/rook/tests/framework/clients"
@@ -34,7 +33,6 @@ import (
 	"github.com/stretchr/testify/suite"
 	appsv1 "k8s.io/api/apps/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/types"
 )
 
 // ************************************************
@@ -343,104 +341,3 @@ func (s *SmokeSuite) getNonCanaryMonDeployments() ([]appsv1.Deployment, error) {
 	}
 	return nonCanaryMonDeployments, nil
 }
-
-// Smoke Test for Webhooks
-func (s *SmokeSuite) TestWebhooks() {
-	logger.Info("Create webhook Smoke Test")
-	ctx := context.TODO()
-	t := s.T()
-
-	t.Run("During cephCluster update, hostPath cannot be changed", func(t *testing.T) {
-		cephCRList, err := s.k8sh.RookClientset.CephV1().CephClusters(s.settings.Namespace).List(ctx, metav1.ListOptions{})
-		require.NoError(s.T(), err)
-		cephCR := cephCRList.Items[0]
-		cephCR.Spec.DataDirHostPath = "/var"
-
-		_, err = s.k8sh.RookClientset.CephV1().CephClusters(s.settings.Namespace).Update(ctx, &cephCR, metav1.UpdateOptions{})
-		require.Error(s.T(), err)
-	})
-
-	t.Run("During blockPool creation, both erasurecoded and replicated fields cannot be set at the same time", func(t *testing.T) {
-		pool := &cephv1.CephBlockPool{
-			ObjectMeta: metav1.ObjectMeta{
-				Name:      "webhook-pool",
-				Namespace: s.settings.Namespace,
-				UID:       types.UID("c47cac40-9bee-4d52-823b-ccd803ba5bfe"),
-			},
-			Spec: cephv1.NamedBlockPoolSpec{
-				PoolSpec: cephv1.PoolSpec{
-					Replicated: cephv1.ReplicatedSpec{
-						Size: uint(1),
-					},
-					ErasureCoded: cephv1.ErasureCodedSpec{
-						CodingChunks: 1,
-						DataChunks:   1,
-					},
-				},
-			},
-		}
-		_, err := s.k8sh.RookClientset.CephV1().CephBlockPools(s.settings.Namespace).Create(ctx, pool, metav1.CreateOptions{})
-		require.Error(s.T(), err)
-	})
-
-	t.Run("During objectStore creation, either of port or securePort fields should be not be zero ", func(t *testing.T) {
-		objectStore := &cephv1.CephObjectStore{
-			ObjectMeta: metav1.ObjectMeta{
-				Name:      "webhook-store",
-				Namespace: s.settings.Namespace,
-			},
-			TypeMeta: metav1.TypeMeta{
-				Kind: "CephObjectStore",
-			},
-			Spec: cephv1.ObjectStoreSpec{
-				Gateway: cephv1.GatewaySpec{
-					Port:       0,
-					SecurePort: 0,
-				},
-			},
-		}
-		_, err := s.k8sh.RookClientset.CephV1().CephObjectStores(s.settings.Namespace).Create(ctx, objectStore, metav1.CreateOptions{})
-		require.Error(s.T(), err)
-	})
-
-	t.Run("During filesystemSubVolumeGroup update filesystem name cannot be changed", func(t *testing.T) {
-		subVolumeGroup := &cephv1.CephFilesystemSubVolumeGroup{
-			ObjectMeta: metav1.ObjectMeta{
-				Name: "subvol-name",
-			},
-			Spec: cephv1.CephFilesystemSubVolumeGroupSpec{
-				FilesystemName: "filesystem-name",
-			},
-		}
-		subVolumeGroup, err := s.k8sh.RookClientset.CephV1().CephFilesystemSubVolumeGroups(s.settings.Namespace).Create(ctx, subVolumeGroup, metav1.CreateOptions{})
-		require.NoError(s.T(), err)
-		subVolumeGroup.Spec.FilesystemName = "new-fs"
-		_, err = s.k8sh.RookClientset.CephV1().CephFilesystemSubVolumeGroups(s.settings.Namespace).Update(ctx, subVolumeGroup, metav1.UpdateOptions{})
-		require.Error(s.T(), err)
-
-		// Delete the subVolumeGroup for cleanup
-		err = s.k8sh.RookClientset.CephV1().CephFilesystemSubVolumeGroups(s.settings.Namespace).Delete(ctx, subVolumeGroup.Name, metav1.DeleteOptions{})
-		require.NoError(s.T(), err)
-	})
-
-	t.Run("During blockPoolRadosNamespace update blockpool name cannot be changed", func(t *testing.T) {
-		rn := &cephv1.CephBlockPoolRadosNamespace{
-			ObjectMeta: metav1.ObjectMeta{
-				Name: "rados-name",
-			},
-			Spec: cephv1.CephBlockPoolRadosNamespaceSpec{
-				BlockPoolName: "pool-name",
-			},
-		}
-
-		rn, err := s.k8sh.RookClientset.CephV1().CephBlockPoolRadosNamespaces(s.settings.Namespace).Create(ctx, rn, metav1.CreateOptions{})
-		require.NoError(s.T(), err)
-		rn.Spec.BlockPoolName = "new-pool"
-		_, err = s.k8sh.RookClientset.CephV1().CephBlockPoolRadosNamespaces(s.settings.Namespace).Update(ctx, rn, metav1.UpdateOptions{})
-		require.Error(s.T(), err)
-
-		// Delete the radosNamespace for cleanup
-		err = s.k8sh.RookClientset.CephV1().CephBlockPoolRadosNamespaces(s.settings.Namespace).Delete(ctx, rn.Name, metav1.DeleteOptions{})
-		require.NoError(s.T(), err)
-	})
-}
diff --git a/tests/scripts/github-action-helper.sh b/tests/scripts/github-action-helper.sh
index dfc3f401b..8962e2409 100755
--- a/tests/scripts/github-action-helper.sh
+++ b/tests/scripts/github-action-helper.sh
@@ -214,8 +214,6 @@ function deploy_cluster() {
     sed -i "s|#deviceFilter:|deviceFilter: ${BLOCK/\/dev\/}\n    config:\n      osdsPerDevice: \"2\"|g" cluster-test.yaml
   elif [ "$1" = "osd_with_metadata_device" ] ; then
     sed -i "s|#deviceFilter:|deviceFilter: ${BLOCK/\/dev\/}\n    config:\n      metadataDevice: /dev/test-rook-vg/test-rook-lv|g" cluster-test.yaml
-  elif [ "$1" = "encryption" ] ; then
-    sed -i "s|#deviceFilter:|deviceFilter: ${BLOCK/\/dev\/}\n    config:\n      encryptedDevice: \"true\"|g" cluster-test.yaml
   fi
   kubectl create -f cluster-test.yaml
   kubectl create -f object-test.yaml
@@ -230,23 +228,6 @@ function deploy_cluster() {
   deploy_manifest_with_local_build toolbox.yaml
 }
 
-function deploy_csi_hostnetwork_disabled_cluster() {
-  create_cluster_prerequisites
-  cd deploy/examples
-  sed -i 's/.*CSI_ENABLE_HOST_NETWORK:.*/  CSI_ENABLE_HOST_NETWORK: \"false\"/g' operator.yaml
-  deploy_manifest_with_local_build operator.yaml
-  if [ $# == 0 ]; then
-    sed -i "s|#deviceFilter:|deviceFilter: ${BLOCK/\/dev\/}|g" cluster-test.yaml
-  elif [ "$1" = "two_osds_in_device" ] ; then
-    sed -i "s|#deviceFilter:|deviceFilter: ${BLOCK/\/dev\/}\n    config:\n      osdsPerDevice: \"2\"|g" cluster-test.yaml
-  elif [ "$1" = "osd_with_metadata_device" ] ; then
-    sed -i "s|#deviceFilter:|deviceFilter: ${BLOCK/\/dev\/}\n    config:\n      metadataDevice: /dev/test-rook-vg/test-rook-lv|g" cluster-test.yaml
-  fi
-  kubectl create -f cluster-test.yaml
-  kubectl create -f filesystem-test.yaml
-  deploy_manifest_with_local_build toolbox.yaml
-}
-
 function wait_for_prepare_pod() {
   get_pod_cmd=(kubectl --namespace rook-ceph get pod --no-headers)
   timeout=450
diff --git a/tests/scripts/pythonwebserver/server.py b/tests/scripts/pythonwebserver/server.py
index a9c359001..d2b1459e3 100644
--- a/tests/scripts/pythonwebserver/server.py
+++ b/tests/scripts/pythonwebserver/server.py
@@ -1,4 +1,4 @@
-"""
+'''
 Copyright 2022 The Rook Authors. All rights reserved.
 
 Licensed under the Apache License, Version 2.0 (the "License");
@@ -12,7 +12,7 @@ distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-"""
+'''
 #!/usr/bin/env python3
 """
 Very simple HTTP server in python for logging requests
@@ -22,35 +22,30 @@ Usage::
 from http.server import BaseHTTPRequestHandler, HTTPServer
 import logging
 
-
 class S(BaseHTTPRequestHandler):
     def _set_response(self):
         self.send_response(200)
-        self.send_header("Content-type", "text/html")
+        self.send_header('Content-type', 'text/html')
         self.end_headers()
 
     def do_POST(self):
-        content_length = int(
-            self.headers["Content-Length"]
-        )  # <--- Gets the size of data
-        post_data = self.rfile.read(content_length)  # <--- Gets the data itself
-        logging.info("POST request\nBody:\n%s\n", post_data.decode("utf-8"))
-
+        content_length = int(self.headers['Content-Length']) # <--- Gets the size of data
+        post_data = self.rfile.read(content_length) # <--- Gets the data itself
+        logging.info("POST request\nBody:\n%s\n", post_data.decode('utf-8'))
 
 def run(server_class=HTTPServer, handler_class=S, port=8080):
     logging.basicConfig(level=logging.INFO)
-    server_address = ("", port)
+    server_address = ('', port)
     httpd = server_class(server_address, handler_class)
-    logging.info("Starting httpd...\n")
+    logging.info('Starting httpd...\n')
     try:
         httpd.serve_forever()
     except KeyboardInterrupt:
         pass
     httpd.server_close()
-    logging.info("Stopping httpd...\n")
-
+    logging.info('Stopping httpd...\n')
 
-if __name__ == "__main__":
+if __name__ == '__main__':
     from sys import argv
 
     if len(argv) == 2:
diff --git a/tests/scripts/validate_cluster.sh b/tests/scripts/validate_cluster.sh
index 6ff47c635..ac170130d 100755
--- a/tests/scripts/validate_cluster.sh
+++ b/tests/scripts/validate_cluster.sh
@@ -89,20 +89,17 @@ function test_demo_pool {
 
 function test_csi {
   timeout 360 bash -x <<-'EOF'
-    echo $IS_POD_NETWORK
     echo $IS_MULTUS
-    if [ -z "$IS_POD_NETWORK" ]; then
+    if [ -z $IS_MULTUS ]; then
       until [[ "$(kubectl -n rook-ceph get pods --field-selector=status.phase=Running|grep -c ^csi-)" -eq 4 ]]; do
         echo "waiting for csi pods to be ready"
         sleep 5
       done
     else
       until [[ "$(kubectl -n rook-ceph get pods --field-selector=status.phase=Running|grep -c ^csi-)" -eq 6 ]]; do
-        echo "waiting for csi pods to be ready with multus or pod networking"
+        echo "waiting for csi pods to be ready with multus"
         sleep 5
       done
-    fi
-    if [ -n "$IS_MULTUS" ]; then
       echo "verifying csi holder interfaces (multus ones must be present)"
       kubectl -n rook-ceph exec -t ds/csi-rbdplugin-holder-my-cluster -- grep net /proc/net/dev
       kubectl -n rook-ceph exec -t ds/csi-cephfsplugin-holder-my-cluster -- grep net /proc/net/dev