Repository: rook/rook
Pull request #8774 (2021-09-21 14:17:12)

ceph: prometheus rules format changes
Due to the latest jsonnet and jsonnetfmt, v0.17.0, we have the
`description` and `message` lines format changed.

Signed-off-by: Arun Kumar Mohan <amohan@redhat.com>

<!-- Please take a look at our [Contributing](https://rook.io/docs/rook/master/development-flow.html)
documentation before submitting a Pull Request!
Thank you for contributing to Rook! -->

**Description of your changes:**

**Which issue is resolved by this Pull Request:**
Resolves #

**Checklist:**

- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/master/development-flow.html#commit-structure).
- [ ] **Skip Tests for Docs**: Add the flag for skipping the build if this is only a documentation change. See [here](https://github.com/rook/rook/blob/master/INSTALL.md#skip-ci) for the flag.
- [ ] **Skip Unrelated Tests**: Add a flag to run tests for a specific storage provider. See [test options](https://github.com/rook/rook/blob/master/INSTALL.md#test-storage-provider).
- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/master/development-flow.html#submitting-a-pull-request)
- [ ] Documentation has been updated, if necessary.
- [ ] Unit tests have been added, if necessary.
- [ ] Integration tests have been added, if necessary.
- [ ] Pending release notes updated with breaking and/or notable changes, if necessary.
- [ ] Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
- [ ] Code generation (`make codegen`) has been run to update object specifications, if necessary.



Patch:
diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md
index 3125b2b96..b82428d22 100644
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@@ -45,4 +45,4 @@ Read [Github documentation if you need help](https://help.github.com/en/articles
 * Storage backend version (e.g. for ceph do `ceph -v`):
 * Kubernetes version (use `kubectl version`):
 * Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
-* Storage backend status (e.g. for Ceph use `ceph health` in the [Rook Ceph toolbox](https://rook.io/docs/rook/latest/ceph-toolbox.html)):
+* Storage backend status (e.g. for Ceph use `ceph health` in the [Rook Ceph toolbox](https://rook.io/docs/rook/master/ceph-toolbox.html)):
diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
index c367c28ea..86899613b 100644
--- a/.github/ISSUE_TEMPLATE/bug_report.md
+++ b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -36,4 +36,4 @@ Read [Github documentation if you need help](https://help.github.com/en/articles
 * Storage backend version (e.g. for ceph do `ceph -v`):
 * Kubernetes version (use `kubectl version`):
 * Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
-* Storage backend status (e.g. for Ceph use `ceph health` in the [Rook Ceph toolbox](https://rook.io/docs/rook/latest/ceph-toolbox.html)):
+* Storage backend status (e.g. for Ceph use `ceph health` in the [Rook Ceph toolbox](https://rook.io/docs/rook/master/ceph-toolbox.html)):
diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md
index aa2f8882b..0fe9c72df 100644
--- a/.github/PULL_REQUEST_TEMPLATE.md
+++ b/.github/PULL_REQUEST_TEMPLATE.md
@@ -1,4 +1,4 @@
-<!-- Please take a look at our [Contributing](https://rook.io/docs/rook/latest/development-flow.html)
+<!-- Please take a look at our [Contributing](https://rook.io/docs/rook/master/development-flow.html)
 documentation before submitting a Pull Request!
 Thank you for contributing to Rook! -->
 
@@ -9,10 +9,10 @@ Resolves #
 
 **Checklist:**
 
-- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/latest/development-flow.html#commit-structure).
+- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/master/development-flow.html#commit-structure).
 - [ ] **Skip Tests for Docs**: Add the flag for skipping the build if this is only a documentation change. See [here](https://github.com/rook/rook/blob/master/INSTALL.md#skip-ci) for the flag.
 - [ ] **Skip Unrelated Tests**: Add a flag to run tests for a specific storage provider. See [test options](https://github.com/rook/rook/blob/master/INSTALL.md#test-storage-provider).
-- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/latest/development-flow.html#submitting-a-pull-request)
+- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/master/development-flow.html#submitting-a-pull-request)
 - [ ] Documentation has been updated, if necessary.
 - [ ] Unit tests have been added, if necessary.
 - [ ] Integration tests have been added, if necessary.
diff --git a/.github/workflows/commitlint.yml b/.github/workflows/commitlint.yml
index 5624d95aa..23921b5d5 100644
--- a/.github/workflows/commitlint.yml
+++ b/.github/workflows/commitlint.yml
@@ -23,4 +23,4 @@ jobs:
       - uses: wagoid/commitlint-github-action@v2.0.3
         with:
           configFile: './.commitlintrc.json'
-          helpURL: https://rook.io/docs/rook/latest/development-flow.html#commit-structure
+          helpURL: https://rook.io/docs/rook/master/development-flow.html#commit-structure
diff --git a/.mergify.yml b/.mergify.yml
index ba5e50ea4..3e3979226 100644
--- a/.mergify.yml
+++ b/.mergify.yml
@@ -15,7 +15,7 @@ pull_request_rules:
       - conflict
     actions:
       comment:
-        message: This pull request has merge conflicts that must be resolved before it can be merged. @{{author}} please rebase it. https://rook.io/docs/rook/latest/development-flow.html#updating-your-fork
+        message: This pull request has merge conflicts that must be resolved before it can be merged. @{{author}} please rebase it. https://rook.io/docs/rook/master/development-flow.html#updating-your-fork
 
   # automerge on master only under certain strict conditions
   - name: automerge merge master with specific label and approvals with code change
diff --git a/Documentation/ceph-advanced-configuration.md b/Documentation/ceph-advanced-configuration.md
index 329f741b1..f2913f0d2 100644
--- a/Documentation/ceph-advanced-configuration.md
+++ b/Documentation/ceph-advanced-configuration.md
@@ -22,7 +22,7 @@ storage cluster.
 * [OSD Dedicated Network](#osd-dedicated-network)
 * [Phantom OSD Removal](#phantom-osd-removal)
 * [Change Failure Domain](#change-failure-domain)
-* [Auto Expansion of OSDs](#auto-expansion-of-osds)
+* [Auto Expansion of OSDs](#auto-expansion-of-OSDs)
 
 ## Prerequisites
 
@@ -467,7 +467,7 @@ Two changes are necessary to the configuration to enable this capability:
 
 ### Use hostNetwork in the rook ceph cluster configuration
 
-Enable the `hostNetwork` setting in the [Ceph Cluster CRD configuration](ceph-cluster-crd.md#samples).
+Enable the `hostNetwork` setting in the [Ceph Cluster CRD configuration](https://rook.io/docs/rook/master/ceph-cluster-crd.html#samples).
 For example,
 
 ```yaml
@@ -606,13 +606,13 @@ Exactly the same approach can be used to change from `host` back to `osd`.
 
 Run the following script to auto-grow the size of OSDs on a PVC-based Rook-Ceph cluster whenever the OSDs have reached the storage near-full threshold.
 ```console
-tests/scripts/auto-grow-storage.sh size  --max maxSize --growth-rate percent
+tests/scripts/auto-grow-storage.sh size  --max maxSize --growth-rate percent 
 ```
 >growth-rate percentage represents the percent increase you want in the OSD capacity and maxSize represent the maximum disk size.
 
 For example, if you need to increase the size of OSD by 30% and max disk size is 1Ti
 ```console
-./auto-grow-storage.sh size  --max 1Ti --growth-rate 30
+./auto-grow-storage.sh size  --max 1Ti --growth-rate 30 
 ```
 
 ### To scale OSDs Horizontally
diff --git a/Documentation/ceph-cluster-crd.md b/Documentation/ceph-cluster-crd.md
index 5f25e4432..04e3979dd 100755
--- a/Documentation/ceph-cluster-crd.md
+++ b/Documentation/ceph-cluster-crd.md
@@ -32,7 +32,7 @@ metadata:
 spec:
   cephVersion:
     # see the "Cluster Settings" section below for more details on which image of ceph to run
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -60,7 +60,7 @@ metadata:
 spec:
   cephVersion:
     # see the "Cluster Settings" section below for more details on which image of ceph to run
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -129,7 +129,7 @@ spec:
       - name: c
   cephVersion:
     # Stretch cluster is supported in Ceph Pacific or newer.
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
     allowUnsupported: true
   # Either storageClassDeviceSets or the storage section can be specified for creating OSDs.
   # This example uses all devices for simplicity.
@@ -167,7 +167,7 @@ Settings can be specified at the global level to apply to the cluster as a whole
 * `external`:
   * `enable`: if `true`, the cluster will not be managed by Rook but via an external entity. This mode is intended to connect to an existing cluster. In this case, Rook will only consume the external cluster. However, Rook will be able to deploy various daemons in Kubernetes such as object gateways, mds and nfs if an image is provided and will refuse otherwise. If this setting is enabled **all** the other options will be ignored except `cephVersion.image` and `dataDirHostPath`. See [external cluster configuration](#external-cluster). If `cephVersion.image` is left blank, Rook will refuse the creation of extra CRs like object, file and nfs.
 * `cephVersion`: The version information for launching the ceph daemons.
-  * `image`: The image used for running the ceph daemons. For example, `quay.io/ceph/ceph:v15.2.12` or `v16.2.6`. For more details read the [container images section](#ceph-container-images).
+  * `image`: The image used for running the ceph daemons. For example, `quay.io/ceph/ceph:v15.2.12` or `v16.2.5`. For more details read the [container images section](#ceph-container-images).
   For the latest ceph images, see the [Ceph DockerHub](https://hub.docker.com/r/ceph/ceph/tags/).
   To ensure a consistent version of the image is running across all nodes in the cluster, it is recommended to use a very specific image version.
   Tags also exist that would give the latest version, but they are only recommended for test environments. For example, the tag `v14` will be updated each time a new nautilus build is released.
@@ -685,8 +685,8 @@ kubectl -n rook-ceph get CephCluster -o yaml
       deviceClasses:
       - name: hdd
     version:
-      image: quay.io/ceph/ceph:v16.2.6
-      version: 16.2.6-0
+      image: quay.io/ceph/ceph:v16.2.5
+      version: 16.2.5-0
     conditions:
     - lastHeartbeatTime: "2021-03-02T21:22:11Z"
       lastTransitionTime: "2021-03-02T21:21:09Z"
@@ -747,7 +747,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -779,7 +779,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -819,7 +819,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -866,7 +866,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -972,7 +972,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
@@ -1018,7 +1018,7 @@ spec:
           requests:
             storage: 10Gi
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
     allowUnsupported: false
   dashboard:
     enabled: true
@@ -1476,7 +1476,7 @@ spec:
     enable: true
   dataDirHostPath: /var/lib/rook
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6 # Should match external cluster version
+    image: quay.io/ceph/ceph:v16.2.5 # Should match external cluster version
 ```
 
 ### Security
diff --git a/Documentation/ceph-csi-drivers.md b/Documentation/ceph-csi-drivers.md
index 9c216d00e..ccc3ba244 100644
--- a/Documentation/ceph-csi-drivers.md
+++ b/Documentation/ceph-csi-drivers.md
@@ -19,10 +19,6 @@ For documentation on consuming the storage:
 * RBD: See the [Block Storage](ceph-block.md) topic
 * CephFS: See the [Shared Filesystem](ceph-filesystem.md) topic
 
-## Supported Versions
-The supported Ceph CSI version is 3.3.0 or greater with Rook. Refer to ceph csi [releases](https://github.com/ceph/ceph-csi/releases)
-for more information.
-
 ## Static Provisioning
 
 Both drivers also support the creation of static PV and static PVC from existing RBD image/CephFS volume. Refer to [static PVC](https://github.com/ceph/ceph-csi/blob/devel/docs/static-pvc.md) for more information.
diff --git a/Documentation/ceph-dashboard.md b/Documentation/ceph-dashboard.md
index 426582175..8f7490654 100755
--- a/Documentation/ceph-dashboard.md
+++ b/Documentation/ceph-dashboard.md
@@ -48,7 +48,7 @@ in this example at `https://10.110.113.240:8443`.
 ### Login Credentials
 
 After you connect to the dashboard you will need to login for secure access. Rook creates a default user named
-`admin` and generates a secret called `rook-ceph-dashboard-password` in the namespace where the Rook Ceph cluster is running.
+`admin` and generates a secret called `rook-ceph-dashboard-admin-password` in the namespace where the Rook Ceph cluster is running.
 To retrieve the generated password, you can run the following:
 
 ```console
diff --git a/Documentation/ceph-upgrade.md b/Documentation/ceph-upgrade.md
index f69d1315d..1191f54b1 100644
--- a/Documentation/ceph-upgrade.md
+++ b/Documentation/ceph-upgrade.md
@@ -430,7 +430,7 @@ Prior to August 2021, official images were on docker.io. While those images will
 
 These images are tagged in a few ways:
 
-* The most explicit form of tags are full-ceph-version-and-build tags (e.g., `v16.2.6-20210916`).
+* The most explicit form of tags are full-ceph-version-and-build tags (e.g., `v16.2.5-20210708`).
   These tags are recommended for production clusters, as there is no possibility for the cluster to
   be heterogeneous with respect to the version of Ceph running in containers.
 * Ceph major version tags (e.g., `v16`) are useful for development and test clusters so that the
@@ -446,7 +446,7 @@ The majority of the upgrade will be handled by the Rook operator. Begin the upgr
 Ceph image field in the cluster CRD (`spec.cephVersion.image`).
 
 ```sh
-NEW_CEPH_IMAGE='quay.io/ceph/ceph:v16.2.6-20210916'
+NEW_CEPH_IMAGE='quay.io/ceph/ceph:v16.2.5-20210708'
 CLUSTER_NAME="$ROOK_CLUSTER_NAMESPACE"  # change if your cluster name is not the Rook namespace
 kubectl -n $ROOK_CLUSTER_NAMESPACE patch CephCluster $CLUSTER_NAME --type=merge -p "{\"spec\": {\"cephVersion\": {\"image\": \"$NEW_CEPH_IMAGE\"}}}"
 ```
@@ -466,9 +466,9 @@ Determining when the Ceph has fully updated is rather simple.
 kubectl -n $ROOK_CLUSTER_NAMESPACE get deployment -l rook_cluster=$ROOK_CLUSTER_NAMESPACE -o jsonpath='{range .items[*]}{"ceph-version="}{.metadata.labels.ceph-version}{"\n"}{end}' | sort | uniq
 This cluster is not yet finished:
     ceph-version=15.2.13-0
-    ceph-version=16.2.6-0
+    ceph-version=16.2.5-0
 This cluster is finished:
-    ceph-version=16.2.6-0
+    ceph-version=16.2.5-0
 ```
 
 #### **3. Verify the updated cluster**
diff --git a/Documentation/development-environment.md b/Documentation/development-environment.md
index 91468b617..31e2c6efd 100644
--- a/Documentation/development-environment.md
+++ b/Documentation/development-environment.md
@@ -174,7 +174,7 @@ and you can use that token to log into the UI at http://localhost:8001/ui.
 
 Everything should happen on the host, your development environment will reside on the host machine NOT inside the virtual machines running the Kubernetes cluster.
 
-Now, please refer to the [development flow](development-flow.md) to setup your development environment (go, git etc).
+Now, please refer to [https://rook.io/docs/rook/master/development-flow.html](https://rook.io/docs/rook/master/development-flow.html) to setup your development environment (go, git etc).
 
 At this stage, Rook should be cloned on your host.
 
diff --git a/Documentation/helm-operator.md b/Documentation/helm-operator.md
index 9d245be20..3c5ba0324 100644
--- a/Documentation/helm-operator.md
+++ b/Documentation/helm-operator.md
@@ -108,7 +108,6 @@ The following tables lists the configurable parameters of the rook-operator char
 | `csi.rbdFSGroupPolicy`              | Policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted                                  | ReadWriteOnceWithFSType                                   |
 | `csi.cephFSFSGroupPolicy`           | Policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted                               | `None`                                                    |
 | `csi.logLevel`                      | Set logging level for csi containers. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. | `0`                                                       |
-| `csi.provisionerReplicas`           | Set replicas for csi provisioner deployment.                                                                                | `2`                                                       |
 | `csi.enableGrpcMetrics`             | Enable Ceph CSI GRPC Metrics.                                                                                               | `false`                                                   |
 | `csi.enableCSIHostNetwork`          | Enable Host Networking for Ceph CSI nodeplugins.                                                                            | `false`                                                   |
 | `csi.provisionerTolerations`        | Array of tolerations in YAML format which will be added to CSI provisioner deployment.                                      | <none>                                                    |
diff --git a/Documentation/pre-reqs.md b/Documentation/pre-reqs.md
index 5cdef8849..a991a3452 100644
--- a/Documentation/pre-reqs.md
+++ b/Documentation/pre-reqs.md
@@ -83,7 +83,7 @@ Ceph requires a Linux kernel built with the RBD module. Many Linux distributions
 For example, the GKE Container-Optimised OS (COS) does not have RBD.
 
 You can test your Kubernetes nodes by running `modprobe rbd`.
-If it says 'not found', you may have to [rebuild your kernel](common-issues.md#rook-agent-rbd-module-missing-error)
+If it says 'not found', you may have to [rebuild your kernel](https://rook.io/docs/rook/master/common-issues.html#rook-agent-rbd-module-missing-error)
 or choose a different Linux distribution.
 
 ### CephFS
diff --git a/Documentation/storage-providers.md b/Documentation/storage-providers.md
index 560f41278..67a5a7ada 100644
--- a/Documentation/storage-providers.md
+++ b/Documentation/storage-providers.md
@@ -53,7 +53,7 @@ Rook provides the following framework to assist storage providers in building an
   each storage provider has its own repo.
 * Helm charts are published to [charts.rook.io](https://charts.rook.io/release)
 * Documentation for the storage provider is to be written by the storage provider
-  members. The build process will publish the documentation to the [Rook website](https://rook.github.io/docs/rook/latest/).
+  members. The build process will publish the documentation to the [Rook website](https://rook.github.io/docs/rook/master/).
 * All storage providers are added to the [Rook.io website](https://rook.io/)
 * A great Slack community is available where you can communicate amongst developers and users
 
diff --git a/INSTALL.md b/INSTALL.md
index 032d5c26d..9bed0b8b2 100644
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -46,7 +46,7 @@ On every commit to PR and master the CI will build, run unit tests, and run inte
 If the build is for master or a release, the build will also be published to
 [dockerhub.com](https://cloud.docker.com/u/rook/repository/list).
 
-> Note that if the pull request title follows Rook's [contribution guidelines](https://rook.io/docs/rook/latest/development-flow.html#commit-structure), the CI will automatically run the appropriate test scenario. For example if a pull request title is "ceph: add a feature", then the tests for the Ceph storage provider will run. Similarly, tests will only run for a single provider with the "cassandra:" and "nfs:" prefixes.
+> Note that if the pull request title follows Rook's [contribution guidelines](https://rook.io/docs/rook/master/development-flow.html#commit-structure), the CI will automatically run the appropriate test scenario. For example if a pull request title is "ceph: add a feature", then the tests for the Ceph storage provider will run. Similarly, tests will only run for a single provider with the "cassandra:" and "nfs:" prefixes.
 
 ## Building for other platforms
 
diff --git a/README.md b/README.md
index a59a48a01..2d5d3161a 100644
--- a/README.md
+++ b/README.md
@@ -24,7 +24,7 @@ Rook is hosted by the [Cloud Native Computing Foundation](https://cncf.io) (CNCF
 
 ## Getting Started and Documentation
 
-For installation, deployment, and administration, see our [Documentation](https://rook.github.io/docs/rook/latest).
+For installation, deployment, and administration, see our [Documentation](https://rook.github.io/docs/rook/master).
 
 ## Contributing
 
@@ -80,7 +80,7 @@ More details about API versioning and status in Kubernetes can be found on the K
 | ---- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------- | ------ |
 | Ceph | [Ceph](https://ceph.com/) is a distributed storage system that provides file, block and object storage and is deployed in large scale production clusters. | ceph.rook.io/v1 | Stable |
 
-This repo is for the Ceph storage provider. The [Cassandra](https://github.com/rook/cassandra) and [NFS](https://github.com/rook/nfs) storage providers moved to a separate repo to allow for each [storage provider](https://rook.github.io/docs/rook/latest/storage-providers.html) to have an independent development and release schedule.
+This repo is for the Ceph storage provider. The [Cassandra](https://github.com/rook/cassandra) and [NFS](https://github.com/rook/nfs) storage providers moved to a separate repo to allow for each [storage provider](https://rook.github.io/docs/rook/master/storage-providers.html) to have an independent development and release schedule.
 
 ### Official Releases
 
diff --git a/cluster/charts/rook-ceph-cluster/templates/NOTES.txt b/cluster/charts/rook-ceph-cluster/templates/NOTES.txt
index 74f78811a..83659c9a2 100644
--- a/cluster/charts/rook-ceph-cluster/templates/NOTES.txt
+++ b/cluster/charts/rook-ceph-cluster/templates/NOTES.txt
@@ -1,7 +1,7 @@
 The Ceph Cluster has been installed. Check its status by running:
   kubectl --namespace {{ .Release.Namespace }} get cephcluster
 
-Visit https://rook.github.io/docs/rook/latest/ceph-cluster-crd.html for more information about the Ceph CRD.
+Visit https://rook.github.io/docs/rook/master/ceph-cluster-crd.html for more information about the Ceph CRD.
 
 Important Notes:
 - You can only deploy a single cluster per namespace
diff --git a/cluster/charts/rook-ceph-cluster/templates/cephfilesystem.yaml b/cluster/charts/rook-ceph-cluster/templates/cephfilesystem.yaml
index c2eeb5032..73be71f60 100644
--- a/cluster/charts/rook-ceph-cluster/templates/cephfilesystem.yaml
+++ b/cluster/charts/rook-ceph-cluster/templates/cephfilesystem.yaml
@@ -13,8 +13,6 @@ apiVersion: storage.k8s.io/v1
 kind: StorageClass
 metadata:
   name: {{ $filesystem.storageClass.name }}
-  annotations:
-    storageclass.kubernetes.io/is-default-class: "{{ if default false $filesystem.storageClass.isDefault }}true{{ else }}false{{ end }}"
 provisioner: {{ $root.Values.operatorNamespace }}.cephfs.csi.ceph.com
 parameters:
   fsName: {{ $filesystem.name }}
diff --git a/cluster/charts/rook-ceph-cluster/values.yaml b/cluster/charts/rook-ceph-cluster/values.yaml
index b785b9ae4..8cace5e27 100644
--- a/cluster/charts/rook-ceph-cluster/values.yaml
+++ b/cluster/charts/rook-ceph-cluster/values.yaml
@@ -42,7 +42,7 @@ cephClusterSpec:
     # versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
     # If you want to be more precise, you can always use a timestamp tag such quay.io/ceph/ceph:v15.2.11-20200419
     # This tag might not contain a new Ceph version, just security fixes from the underlying operating system, which will reduce vulnerabilities
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
     # Whether to allow unsupported versions of Ceph. Currently `nautilus` and `octopus` are supported.
     # Future versions such as `pacific` would require this to be set to `true`.
     # Do not set to true in production.
@@ -56,7 +56,7 @@ cephClusterSpec:
   # Whether or not upgrade should continue even if a check fails
   # This means Ceph's status could be degraded and we don't recommend upgrading but you might decide otherwise
   # Use at your OWN risk
-  # To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/latest/ceph-upgrade.html#ceph-version-upgrades
+  # To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/master/ceph-upgrade.html#ceph-version-upgrades
   skipUpgradeChecks: false
 
   # Whether or not continue if PGs are not clean during an upgrade
@@ -369,7 +369,6 @@ cephFileSystems:
         activeStandby: true
     storageClass:
       enabled: true
-      isDefault: false
       name: ceph-filesystem
       reclaimPolicy: Delete
       # see https://github.com/rook/rook/blob/master/Documentation/ceph-filesystem.md#provision-storage for available configuration
diff --git a/cluster/charts/rook-ceph/templates/NOTES.txt b/cluster/charts/rook-ceph/templates/NOTES.txt
index 60a3dbabd..92791bf62 100644
--- a/cluster/charts/rook-ceph/templates/NOTES.txt
+++ b/cluster/charts/rook-ceph/templates/NOTES.txt
@@ -1,7 +1,7 @@
 The Rook Operator has been installed. Check its status by running:
   kubectl --namespace {{ .Release.Namespace }} get pods -l "app=rook-ceph-operator"
 
-Visit https://rook.io/docs/rook/latest for instructions on how to create and configure Rook clusters
+Visit https://rook.io/docs/rook/master for instructions on how to create and configure Rook clusters
 
 Important Notes:
 - You must customize the 'CephCluster' resource in the sample manifests for your cluster.
diff --git a/cluster/charts/rook-ceph/templates/deployment.yaml b/cluster/charts/rook-ceph/templates/deployment.yaml
index 083265e3f..7efb599eb 100644
--- a/cluster/charts/rook-ceph/templates/deployment.yaml
+++ b/cluster/charts/rook-ceph/templates/deployment.yaml
@@ -275,10 +275,6 @@ spec:
         - name: CSI_LOG_LEVEL
           value: {{ .Values.csi.logLevel | quote }}
 {{- end }}
-{{- if .Values.csi.provisionerReplicas }}
-        - name: CSI_PROVISIONER_REPLICAS
-          value: {{ .Values.csi.provisionerReplicas | quote }}
-{{- end }}
 {{- if .Values.csi.csiRBDProvisionerResource }}
         - name: CSI_RBD_PROVISIONER_RESOURCE
           value: {{ .Values.csi.csiRBDProvisionerResource | quote }}
diff --git a/cluster/charts/rook-ceph/values.yaml b/cluster/charts/rook-ceph/values.yaml
index f7c377f60..9d7962a43 100644
--- a/cluster/charts/rook-ceph/values.yaml
+++ b/cluster/charts/rook-ceph/values.yaml
@@ -13,7 +13,7 @@ crds:
   # managed independently with cluster/examples/kubernetes/ceph/crds.yaml.
   # **WARNING** Only set during first deployment. If later disabled the cluster may be DESTROYED.
   # If the CRDs are deleted in this case, see the disaster recovery guide to restore them.
-  # https://rook.github.io/docs/rook/latest/ceph-disaster-recovery.html#restoring-crds-after-deletion
+  # https://rook.github.io/docs/rook/master/ceph-disaster-recovery.html#restoring-crds-after-deletion
   enabled: true
 
 resources:
@@ -87,9 +87,6 @@ csi:
   # sidecar with CSI provisioner pod, to enable set it to true.
   enableOMAPGenerator: false
 
-  # Set replicas for csi provisioner deployment.
-  provisionerReplicas: 2
-
   # Set logging level for csi containers.
   # Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity.
   #logLevel: 0
@@ -321,7 +318,7 @@ allowMultipleFilesystems: false
 #     effect: NoSchedule
 #   nodeAffinity: key1=value1,value2; key2=value3
 #   mountSecurityMode: Any
-## For information on FlexVolume path, please refer to https://rook.io/docs/rook/latest/flexvolume.html
+## For information on FlexVolume path, please refer to https://rook.io/docs/rook/master/flexvolume.html
 #   flexVolumeDirPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
 #   libModulesDirPath: /lib/modules
 #   mounts: mount1=/host/path:/container/path,/host/path2:/container/path2
diff --git a/cluster/examples/kubernetes/ceph/cluster-external-management.yaml b/cluster/examples/kubernetes/ceph/cluster-external-management.yaml
index 8d50dcfd6..c8cd5f90b 100644
--- a/cluster/examples/kubernetes/ceph/cluster-external-management.yaml
+++ b/cluster/examples/kubernetes/ceph/cluster-external-management.yaml
@@ -19,4 +19,4 @@ spec:
   dataDirHostPath: /var/lib/rook
   # providing an image is required, if you want to create other CRs (rgw, mds, nfs)
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6 # Should match external cluster version
+    image: quay.io/ceph/ceph:v16.2.5 # Should match external cluster version
diff --git a/cluster/examples/kubernetes/ceph/cluster-on-local-pvc.yaml b/cluster/examples/kubernetes/ceph/cluster-on-local-pvc.yaml
index a8ba26851..e8f814c1e 100644
--- a/cluster/examples/kubernetes/ceph/cluster-on-local-pvc.yaml
+++ b/cluster/examples/kubernetes/ceph/cluster-on-local-pvc.yaml
@@ -171,7 +171,7 @@ spec:
           requests:
             storage: 10Gi
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
     allowUnsupported: false
   skipUpgradeChecks: false
   continueUpgradeAfterChecksEvenIfNotHealthy: false
diff --git a/cluster/examples/kubernetes/ceph/cluster-on-pvc.yaml b/cluster/examples/kubernetes/ceph/cluster-on-pvc.yaml
index 2c612596b..561334960 100644
--- a/cluster/examples/kubernetes/ceph/cluster-on-pvc.yaml
+++ b/cluster/examples/kubernetes/ceph/cluster-on-pvc.yaml
@@ -33,7 +33,7 @@ spec:
           requests:
             storage: 10Gi
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
     allowUnsupported: false
   skipUpgradeChecks: false
   continueUpgradeAfterChecksEvenIfNotHealthy: false
diff --git a/cluster/examples/kubernetes/ceph/cluster-stretched-aws.yaml b/cluster/examples/kubernetes/ceph/cluster-stretched-aws.yaml
index dd7cf2aae..34ae42e70 100644
--- a/cluster/examples/kubernetes/ceph/cluster-stretched-aws.yaml
+++ b/cluster/examples/kubernetes/ceph/cluster-stretched-aws.yaml
@@ -22,7 +22,7 @@ spec:
     stretchCluster:
       # The ceph failure domain will be extracted from the label, which by default is the zone. The nodes running OSDs must have
       # this label in order for the OSDs to be configured in the correct topology. For topology labels, see
-      # https://rook.github.io/docs/rook/latest/ceph-cluster-crd.html#osd-topology.
+      # https://rook.github.io/docs/rook/master/ceph-cluster-crd.html#osd-topology.
       failureDomainLabel: topology.kubernetes.io/zone
       # The sub failure domain is the secondary level at which the data will be placed to maintain data durability and availability.
       # The default is "host", which means that each OSD must be on a different node and you would need at least two nodes per zone.
diff --git a/cluster/examples/kubernetes/ceph/cluster-stretched.yaml b/cluster/examples/kubernetes/ceph/cluster-stretched.yaml
index dd25805dc..d26ca53b1 100644
--- a/cluster/examples/kubernetes/ceph/cluster-stretched.yaml
+++ b/cluster/examples/kubernetes/ceph/cluster-stretched.yaml
@@ -22,7 +22,7 @@ spec:
     stretchCluster:
       # The ceph failure domain will be extracted from the label, which by default is the zone. The nodes running OSDs must have
       # this label in order for the OSDs to be configured in the correct topology. For topology labels, see
-      # https://rook.github.io/docs/rook/latest/ceph-cluster-crd.html#osd-topology.
+      # https://rook.github.io/docs/rook/master/ceph-cluster-crd.html#osd-topology.
       failureDomainLabel: topology.kubernetes.io/zone
       # The sub failure domain is the secondary level at which the data will be placed to maintain data durability and availability.
       # The default is "host", which means that each OSD must be on a different node and you would need at least two nodes per zone.
@@ -39,7 +39,7 @@ spec:
     count: 2
   cephVersion:
     # Stretch cluster support upstream is only available starting in Ceph Pacific
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
     allowUnsupported: true
   skipUpgradeChecks: false
   continueUpgradeAfterChecksEvenIfNotHealthy: false
diff --git a/cluster/examples/kubernetes/ceph/cluster.yaml b/cluster/examples/kubernetes/ceph/cluster.yaml
index 46ee53de4..75860fec3 100644
--- a/cluster/examples/kubernetes/ceph/cluster.yaml
+++ b/cluster/examples/kubernetes/ceph/cluster.yaml
@@ -19,9 +19,9 @@ spec:
     # v14 is nautilus, v15 is octopus, and v16 is pacific.
     # RECOMMENDATION: In production, use a specific version tag instead of the general v16 flag, which pulls the latest release and could result in different
     # versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
-    # If you want to be more precise, you can always use a timestamp tag such quay.io/ceph/ceph:v16.2.6-20210916
+    # If you want to be more precise, you can always use a timestamp tag such quay.io/ceph/ceph:v16.2.5-20210708
     # This tag might not contain a new Ceph version, just security fixes from the underlying operating system, which will reduce vulnerabilities
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
     # Whether to allow unsupported versions of Ceph. Currently `nautilus`, `octopus`, and `pacific` are supported.
     # Future versions such as `pacific` would require this to be set to `true`.
     # Do not set to true in production.
@@ -33,7 +33,7 @@ spec:
   # Whether or not upgrade should continue even if a check fails
   # This means Ceph's status could be degraded and we don't recommend upgrading but you might decide otherwise
   # Use at your OWN risk
-  # To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/latest/ceph-upgrade.html#ceph-version-upgrades
+  # To understand Rook's upgrade process of Ceph, read https://rook.io/docs/rook/master/ceph-upgrade.html#ceph-version-upgrades
   skipUpgradeChecks: false
   # Whether or not continue if PGs are not clean during an upgrade
   continueUpgradeAfterChecksEvenIfNotHealthy: false
diff --git a/cluster/examples/kubernetes/ceph/images.txt b/cluster/examples/kubernetes/ceph/images.txt
index 4095a4e36..9323b7c75 100644
--- a/cluster/examples/kubernetes/ceph/images.txt
+++ b/cluster/examples/kubernetes/ceph/images.txt
@@ -1,5 +1,5 @@
  rook/ceph:master
- quay.io/ceph/ceph:v16.2.6
+ quay.io/ceph/ceph:v16.2.5
  quay.io/cephcsi/cephcsi:v3.4.0
  k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0
  k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2
diff --git a/cluster/examples/kubernetes/ceph/monitoring/prometheus-ceph-v14-rules-external.yaml b/cluster/examples/kubernetes/ceph/monitoring/prometheus-ceph-v14-rules-external.yaml
index cf8488efb..5c1e5df3e 100644
--- a/cluster/examples/kubernetes/ceph/monitoring/prometheus-ceph-v14-rules-external.yaml
+++ b/cluster/examples/kubernetes/ceph/monitoring/prometheus-ceph-v14-rules-external.yaml
@@ -12,10 +12,8 @@ spec:
     rules:
     - alert: PersistentVolumeUsageNearFull
       annotations:
-        description: PVC {{ $labels.persistentvolumeclaim }} utilization has crossed
-          75%. Free up some space.
-        message: PVC {{ $labels.persistentvolumeclaim }} is nearing full. Data deletion
-          is required.
+        description: PVC {{ $labels.persistentvolumeclaim }} utilization has crossed 75%. Free up some space or expand the PVC.
+        message: PVC {{ $labels.persistentvolumeclaim }} is nearing full. Data deletion or PVC expansion is required.
         severity_level: warning
         storage_type: ceph
       expr: |
@@ -25,10 +23,8 @@ spec:
         severity: warning
     - alert: PersistentVolumeUsageCritical
       annotations:
-        description: PVC {{ $labels.persistentvolumeclaim }} utilization has crossed
-          85%. Free up some space immediately.
-        message: PVC {{ $labels.persistentvolumeclaim }} is critically full. Data
-          deletion is required.
+        description: PVC {{ $labels.persistentvolumeclaim }} utilization has crossed 85%. Free up some space or expand the PVC immediately.
+        message: PVC {{ $labels.persistentvolumeclaim }} is critically full. Data deletion or PVC expansion is required.
         severity_level: error
         storage_type: ceph
       expr: |
@@ -36,3 +32,4 @@ spec:
       for: 5s
       labels:
         severity: critical
+
diff --git a/cluster/examples/kubernetes/ceph/monitoring/prometheus-ceph-v14-rules.yaml b/cluster/examples/kubernetes/ceph/monitoring/prometheus-ceph-v14-rules.yaml
index 932deb56b..e75b4e371 100644
--- a/cluster/examples/kubernetes/ceph/monitoring/prometheus-ceph-v14-rules.yaml
+++ b/cluster/examples/kubernetes/ceph/monitoring/prometheus-ceph-v14-rules.yaml
@@ -61,8 +61,7 @@ spec:
     rules:
     - alert: CephMdsMissingReplicas
       annotations:
-        description: Minimum required replicas for storage metadata service not available.
-          Might affect the working of storage cluster.
+        description: Minimum required replicas for storage metadata service not available. Might affect the working of storage cluster.
         message: Insufficient replicas for storage metadata service.
         severity_level: warning
         storage_type: ceph
@@ -86,8 +85,7 @@ spec:
         severity: critical
     - alert: CephMonHighNumberOfLeaderChanges
       annotations:
-        description: Ceph Monitor {{ $labels.ceph_daemon }} on host {{ $labels.hostname
-          }} has seen {{ $value | printf "%.2f" }} leader changes per minute recently.
+        description: Ceph Monitor {{ $labels.ceph_daemon }} on host {{ $labels.hostname }} has seen {{ $value | printf "%.2f" }} leader changes per minute recently.
         message: Storage Cluster has seen many leader changes recently.
         severity_level: warning
         storage_type: ceph
@@ -100,8 +98,7 @@ spec:
     rules:
     - alert: CephNodeDown
       annotations:
-        description: Storage node {{ $labels.node }} went down. Please check the node
-          immediately.
+        description: Storage node {{ $labels.node }} went down. Please check the node immediately.
         message: Storage node {{ $labels.node }} went down
         severity_level: error
         storage_type: ceph
@@ -114,9 +111,7 @@ spec:
     rules:
     - alert: CephOSDCriticallyFull
       annotations:
-        description: Utilization of storage device {{ $labels.ceph_daemon }} of device_class
-          type {{$labels.device_class}} has crossed 80% on host {{ $labels.hostname
-          }}. Immediately free up some space or add capacity of type {{$labels.device_class}}.
+        description: Utilization of storage device {{ $labels.ceph_daemon }} of device_class type {{$labels.device_class}} has crossed 80% on host {{ $labels.hostname }}. Immediately free up some space or add capacity of type {{$labels.device_class}}.
         message: Back-end storage device is critically full.
         severity_level: error
         storage_type: ceph
@@ -127,9 +122,7 @@ spec:
         severity: critical
     - alert: CephOSDFlapping
       annotations:
-        description: Storage daemon {{ $labels.ceph_daemon }} has restarted 5 times
-          in last 5 minutes. Please check the pod events or ceph status to find out
-          the cause.
+        description: Storage daemon {{ $labels.ceph_daemon }} has restarted 5 times in last 5 minutes. Please check the pod events or ceph status to find out the cause.
         message: Ceph storage osd flapping.
         severity_level: error
         storage_type: ceph
@@ -140,9 +133,7 @@ spec:
         severity: critical
     - alert: CephOSDNearFull
       annotations:
-        description: Utilization of storage device {{ $labels.ceph_daemon }} of device_class
-          type {{$labels.device_class}} has crossed 75% on host {{ $labels.hostname
-          }}. Immediately free up some space or add capacity of type {{$labels.device_class}}.
+        description: Utilization of storage device {{ $labels.ceph_daemon }} of device_class type {{$labels.device_class}} has crossed 75% on host {{ $labels.hostname }}. Immediately free up some space or add capacity of type {{$labels.device_class}}.
         message: Back-end storage device is nearing full.
         severity_level: warning
         storage_type: ceph
@@ -153,8 +144,7 @@ spec:
         severity: warning
     - alert: CephOSDDiskNotResponding
       annotations:
-        description: Disk device {{ $labels.device }} not responding, on host {{ $labels.host
-          }}.
+        description: Disk device {{ $labels.device }} not responding, on host {{ $labels.host }}.
         message: Disk not responding
         severity_level: error
         storage_type: ceph
@@ -165,8 +155,7 @@ spec:
         severity: critical
     - alert: CephOSDDiskUnavailable
       annotations:
-        description: Disk device {{ $labels.device }} not accessible on host {{ $labels.host
-          }}.
+        description: Disk device {{ $labels.device }} not accessible on host {{ $labels.host }}.
         message: Disk not accessible
         severity_level: error
         storage_type: ceph
@@ -177,8 +166,7 @@ spec:
         severity: critical
     - alert: CephOSDSlowOps
       annotations:
-        description: '{{ $value }} Ceph OSD requests are taking too long to process.
-          Please check ceph status to find out the cause.'
+        description: '{{ $value }} Ceph OSD requests are taking too long to process. Please check ceph status to find out the cause.'
         message: OSD requests are taking too long to process.
         severity_level: warning
         storage_type: ceph
@@ -213,10 +201,8 @@ spec:
     rules:
     - alert: PersistentVolumeUsageNearFull
       annotations:
-        description: PVC {{ $labels.persistentvolumeclaim }} utilization has crossed
-          75%. Free up some space or expand the PVC.
-        message: PVC {{ $labels.persistentvolumeclaim }} is nearing full. Data deletion
-          or PVC expansion is required.
+        description: PVC {{ $labels.persistentvolumeclaim }} utilization has crossed 75%. Free up some space or expand the PVC.
+        message: PVC {{ $labels.persistentvolumeclaim }} is nearing full. Data deletion or PVC expansion is required.
         severity_level: warning
         storage_type: ceph
       expr: |
@@ -226,10 +212,8 @@ spec:
         severity: warning
     - alert: PersistentVolumeUsageCritical
       annotations:
-        description: PVC {{ $labels.persistentvolumeclaim }} utilization has crossed
-          85%. Free up some space or expand the PVC immediately.
-        message: PVC {{ $labels.persistentvolumeclaim }} is critically full. Data
-          deletion or PVC expansion is required.
+        description: PVC {{ $labels.persistentvolumeclaim }} utilization has crossed 85%. Free up some space or expand the PVC immediately.
+        message: PVC {{ $labels.persistentvolumeclaim }} is critically full. Data deletion or PVC expansion is required.
         severity_level: error
         storage_type: ceph
       expr: |
@@ -263,8 +247,7 @@ spec:
         severity: warning
     - alert: CephOSDVersionMismatch
       annotations:
-        description: There are {{ $value }} different versions of Ceph OSD components
-          running.
+        description: There are {{ $value }} different versions of Ceph OSD components running.
         message: There are multiple versions of storage services running.
         severity_level: warning
         storage_type: ceph
@@ -275,8 +258,7 @@ spec:
         severity: warning
     - alert: CephMonVersionMismatch
       annotations:
-        description: There are {{ $value }} different versions of Ceph Mon components
-          running.
+        description: There are {{ $value }} different versions of Ceph Mon components running.
         message: There are multiple versions of storage services running.
         severity_level: warning
         storage_type: ceph
@@ -289,10 +271,8 @@ spec:
     rules:
     - alert: CephClusterNearFull
       annotations:
-        description: Storage cluster utilization has crossed 75% and will become read-only
-          at 85%. Free up some space or expand the storage cluster.
-        message: Storage cluster is nearing full. Data deletion or cluster expansion
-          is required.
+        description: Storage cluster utilization has crossed 75% and will become read-only at 85%. Free up some space or expand the storage cluster.
+        message: Storage cluster is nearing full. Data deletion or cluster expansion is required.
         severity_level: warning
         storage_type: ceph
       expr: |
@@ -302,10 +282,8 @@ spec:
         severity: warning
     - alert: CephClusterCriticallyFull
       annotations:
-        description: Storage cluster utilization has crossed 80% and will become read-only
-          at 85%. Free up some space or expand the storage cluster immediately.
-        message: Storage cluster is critically full and needs immediate data deletion
-          or cluster expansion.
+        description: Storage cluster utilization has crossed 80% and will become read-only at 85%. Free up some space or expand the storage cluster immediately.
+        message: Storage cluster is critically full and needs immediate data deletion or cluster expansion.
         severity_level: error
         storage_type: ceph
       expr: |
@@ -315,10 +293,8 @@ spec:
         severity: critical
     - alert: CephClusterReadOnly
       annotations:
-        description: Storage cluster utilization has crossed 85% and will become read-only
-          now. Free up some space or expand the storage cluster immediately.
-        message: Storage cluster is read-only now and needs immediate data deletion
-          or cluster expansion.
+        description: Storage cluster utilization has crossed 85% and will become read-only now. Free up some space or expand the storage cluster immediately.
+        message: Storage cluster is read-only now and needs immediate data deletion or cluster expansion.
         severity_level: error
         storage_type: ceph
       expr: |
diff --git a/cluster/examples/kubernetes/ceph/operator-openshift.yaml b/cluster/examples/kubernetes/ceph/operator-openshift.yaml
index 58ddca5d1..c4ad20e4d 100644
--- a/cluster/examples/kubernetes/ceph/operator-openshift.yaml
+++ b/cluster/examples/kubernetes/ceph/operator-openshift.yaml
@@ -120,9 +120,6 @@ data:
   # Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity.
   # CSI_LOG_LEVEL: "0"
 
-  # Set replicas for csi provisioner deployment.
-  CSI_PROVISIONER_REPLICAS: "2"
-
   # OMAP generator generates the omap mapping between the PV name and the RBD image
   # which helps CSI to identify the rbd images for CSI operations.
   # CSI_ENABLE_OMAP_GENERATOR need to be enabled when we are using rbd mirroring feature.
@@ -138,7 +135,7 @@ data:
 
   # Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs
   # If you disable the kernel client, your application may be disrupted during upgrade.
-  # See the upgrade guide: https://rook.io/docs/rook/latest/ceph-upgrade.html
+  # See the upgrade guide: https://rook.io/docs/rook/master/ceph-upgrade.html
   CSI_FORCE_CEPHFS_KERNEL_CLIENT: "true"
 
   # (Optional) policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted.
diff --git a/cluster/examples/kubernetes/ceph/operator.yaml b/cluster/examples/kubernetes/ceph/operator.yaml
index 70ac6b47c..0a3d70594 100644
--- a/cluster/examples/kubernetes/ceph/operator.yaml
+++ b/cluster/examples/kubernetes/ceph/operator.yaml
@@ -41,9 +41,6 @@ data:
   # Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity.
   # CSI_LOG_LEVEL: "0"
 
-  # Set replicas for csi provisioner deployment.
-  CSI_PROVISIONER_REPLICAS: "2"
-
   # OMAP generator will generate the omap mapping between the PV name and the RBD image.
   # CSI_ENABLE_OMAP_GENERATOR need to be enabled when we are using rbd mirroring feature.
   # By default OMAP generator sidecar is deployed with CSI provisioner pod, to disable
@@ -58,7 +55,7 @@ data:
 
   # Enable cephfs kernel driver instead of ceph-fuse.
   # If you disable the kernel client, your application may be disrupted during upgrade.
-  # See the upgrade guide: https://rook.io/docs/rook/latest/ceph-upgrade.html
+  # See the upgrade guide: https://rook.io/docs/rook/master/ceph-upgrade.html
   # NOTE! cephfs quota is not supported in kernel version < 4.17
   CSI_FORCE_CEPHFS_KERNEL_CLIENT: "true"
 
diff --git a/cluster/olm/ceph/assemble/metadata-common.yaml b/cluster/olm/ceph/assemble/metadata-common.yaml
index 34e7c405d..48b4734e7 100644
--- a/cluster/olm/ceph/assemble/metadata-common.yaml
+++ b/cluster/olm/ceph/assemble/metadata-common.yaml
@@ -230,7 +230,7 @@ metadata:
           },
           "spec": {
             "cephVersion": {
-              "image": "quay.io/ceph/ceph:v16.2.6"
+              "image": "quay.io/ceph/ceph:v16.2.5"
             },
             "dataDirHostPath": "/var/lib/rook",
             "mon": {
diff --git a/design/README.md b/design/README.md
index ea943a642..f443ff8c2 100644
--- a/design/README.md
+++ b/design/README.md
@@ -1,3 +1,3 @@
 # Rook Feature Proposal
 
-Please follow the [design section guideline](https://rook.io/docs/rook/latest/development-flow.html#design-document).
+Please follow the [design section guideline](https://rook.io/docs/rook/master/development-flow.html#design-document).
diff --git a/design/ceph/ceph-cluster-cleanup.md b/design/ceph/ceph-cluster-cleanup.md
index 4cc116dda..251373a12 100644
--- a/design/ceph/ceph-cluster-cleanup.md
+++ b/design/ceph/ceph-cluster-cleanup.md
@@ -34,7 +34,7 @@ metadata:
   namespace: rook-ceph
 spec:
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
   dataDirHostPath: /var/lib/rook
   mon:
     count: 3
diff --git a/design/ceph/ceph-stretch-cluster.md b/design/ceph/ceph-stretch-cluster.md
index f522ee054..a605613ce 100644
--- a/design/ceph/ceph-stretch-cluster.md
+++ b/design/ceph/ceph-stretch-cluster.md
@@ -54,7 +54,7 @@ The topology of the K8s cluster is to be determined by the admin, outside the sc
 Rook will simply detect the topology labels that have been added to the nodes.
 
 If the desired failure domain is a "zone", the `topology.kubernetes.io/zone` label should
-be added to the nodes. Any of the [topology labels](https://rook.io/docs/rook/latest/ceph-cluster-crd.html#osd-topology)
+be added to the nodes. Any of the [topology labels](https://rook.io/docs/rook/master/ceph-cluster-crd.html#osd-topology)
 supported by OSDs can be used.
 
 In the minimum configuration, two nodes in each data zone would be labeled, while one
@@ -89,7 +89,7 @@ three zones must be listed and the arbiter zone is identified.
     stretchCluster:
       # The cluster is most commonly stretched over zones, but could also be stretched over
       # another failure domain such as datacenter or region. Must be one of the labels
-      # used by OSDs as documented at https://rook.io/docs/rook/latest/ceph-cluster-crd.html#osd-topology.
+      # used by OSDs as documented at https://rook.io/docs/rook/master/ceph-cluster-crd.html#osd-topology.
       failureDomainLabel: topology.kubernetes.io/zone
       zones:
       # There must be exactly three zones in this list, with one of them being the arbiter
diff --git a/images/ceph/Makefile b/images/ceph/Makefile
index a8f38c131..19379f654 100755
--- a/images/ceph/Makefile
+++ b/images/ceph/Makefile
@@ -18,9 +18,9 @@ include ../image.mk
 # Image Build Options
 
 ifeq ($(GOARCH),amd64)
-CEPH_VERSION = v16.2.6-20210916
+CEPH_VERSION = v16.2.5-20210708
 else
-CEPH_VERSION = v16.2.6-20210916
+CEPH_VERSION = v16.2.5-20210708
 endif
 REGISTRY_NAME = quay.io
 BASEIMAGE = $(REGISTRY_NAME)/ceph/ceph-$(GOARCH):$(CEPH_VERSION)
diff --git a/pkg/operator/ceph/cluster/cluster.go b/pkg/operator/ceph/cluster/cluster.go
index c14751d3c..de46e96ad 100755
--- a/pkg/operator/ceph/cluster/cluster.go
+++ b/pkg/operator/ceph/cluster/cluster.go
@@ -224,9 +224,8 @@ func (c *ClusterController) configureLocalCephCluster(cluster *cluster) error {
 	cluster.isUpgrade = isUpgrade
 
 	if cluster.Spec.IsStretchCluster() {
-		stretchVersion := cephver.CephVersion{Major: 16, Minor: 2, Build: 5}
-		if !cephVersion.IsAtLeast(stretchVersion) {
-			return errors.Errorf("stretch clusters minimum ceph version is %q, but is running %s", stretchVersion.String(), cephVersion.String())
+		if !cephVersion.IsAtLeast(cephver.CephVersion{Major: 16, Minor: 2, Build: 5}) {
+			return errors.Errorf("stretch clusters minimum ceph version is v16.2.5, but is running %s", cephVersion.String())
 		}
 	}
 
diff --git a/pkg/operator/ceph/csi/spec.go b/pkg/operator/ceph/csi/spec.go
index 7f8782349..05979b846 100644
--- a/pkg/operator/ceph/csi/spec.go
+++ b/pkg/operator/ceph/csi/spec.go
@@ -61,7 +61,7 @@ type Param struct {
 	CephFSLivenessMetricsPort      uint16
 	RBDGRPCMetricsPort             uint16
 	RBDLivenessMetricsPort         uint16
-	ProvisionerReplicas            int32
+	ProvisionerReplicas            uint8
 	CSICephFSPodLabels             map[string]string
 	CSIRBDPodLabels                map[string]string
 }
@@ -169,9 +169,6 @@ const (
 	// default log level for csi containers
 	defaultLogLevel uint8 = 0
 
-	// default provisioner replicas
-	defaultProvisionerReplicas int32 = 2
-
 	// update strategy
 	rollingUpdate = "RollingUpdate"
 	onDelete      = "OnDelete"
@@ -284,6 +281,13 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 	// critical pods in cluster but less priority than plugin pods
 	tp.ProvisionerPriorityClassName = k8sutil.GetValue(r.opConfig.Parameters, "CSI_PROVISIONER_PRIORITY_CLASSNAME", "")
 
+	// OMAP generator will be enabled by default
+	// If AllowUnsupported is set to false and if CSI version is less than
+	// <3.2.0 disable OMAP generator sidecar
+	if !v.SupportsOMAPController() {
+		tp.EnableOMAPGenerator = false
+	}
+
 	if strings.EqualFold(k8sutil.GetValue(r.opConfig.Parameters, "CSI_ENABLE_OMAP_GENERATOR", "false"), "true") {
 		tp.EnableOMAPGenerator = true
 	}
@@ -341,22 +345,14 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 		}
 	}
 
-	tp.ProvisionerReplicas = defaultProvisionerReplicas
+	tp.ProvisionerReplicas = 2
 	nodes, err := r.context.Clientset.CoreV1().Nodes().List(r.opManagerContext, metav1.ListOptions{})
 	if err == nil {
 		if len(nodes.Items) == 1 {
 			tp.ProvisionerReplicas = 1
-		} else {
-			replicas := k8sutil.GetValue(r.opConfig.Parameters, "CSI_PROVISIONER_REPLICAS", "2")
-			r, err := strconv.ParseInt(replicas, 10, 32)
-			if err != nil {
-				logger.Errorf("failed to parse CSI_PROVISIONER_REPLICAS. Defaulting to %d. %v", defaultProvisionerReplicas, err)
-			} else {
-				tp.ProvisionerReplicas = int32(r)
-			}
 		}
 	} else {
-		logger.Errorf("failed to get nodes. Defaulting the number of replicas of provisioner pods to %d. %v", tp.ProvisionerReplicas, err)
+		logger.Errorf("failed to get nodes. Defaulting the number of replicas of provisioner pods to 2. %v", err)
 	}
 
 	if EnableRBD {
diff --git a/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml b/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml
index c8d7c2a37..91a2521cd 100644
--- a/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml
+++ b/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-provisioner-dep.yaml
@@ -34,6 +34,11 @@ spec:
             - name: ADDRESS
               value: /csi/csi-provisioner.sock
           imagePullPolicy: "IfNotPresent"
+          securityContext:
+            privileged: true
+            capabilities:
+              add: ["SYS_ADMIN"]
+            allowPrivilegeEscalation: true
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
@@ -50,6 +55,11 @@ spec:
             - name: ADDRESS
               value: unix:///csi/csi-provisioner.sock
           imagePullPolicy: "IfNotPresent"
+          securityContext:
+            privileged: true
+            capabilities:
+              add: ["SYS_ADMIN"]
+            allowPrivilegeEscalation: true
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
@@ -67,6 +77,11 @@ spec:
             - name: ADDRESS
               value: unix:///csi/csi-provisioner.sock
           imagePullPolicy: "IfNotPresent"
+          securityContext:
+            privileged: true
+            capabilities:
+              add: ["SYS_ADMIN"]
+            allowPrivilegeEscalation: true
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
@@ -83,6 +98,11 @@ spec:
             - name: ADDRESS
               value: unix:///csi/csi-provisioner.sock
           imagePullPolicy: "IfNotPresent"
+          securityContext:
+            privileged: true
+            capabilities:
+              add: ["SYS_ADMIN"]
+            allowPrivilegeEscalation: true
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
@@ -116,6 +136,11 @@ spec:
             - name: CSI_ENDPOINT
               value: unix:///csi/csi-provisioner.sock
           imagePullPolicy: "IfNotPresent"
+          securityContext:
+            privileged: true
+            capabilities:
+              add: ["SYS_ADMIN"]
+            allowPrivilegeEscalation: true
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
@@ -150,6 +175,11 @@ spec:
             - name: socket-dir
               mountPath: /csi
           imagePullPolicy: "IfNotPresent"
+          securityContext:
+            privileged: true
+            capabilities:
+              add: ["SYS_ADMIN"]
+            allowPrivilegeEscalation: true
       volumes:
         - name: socket-dir
           emptyDir: {
diff --git a/pkg/operator/ceph/csi/version.go b/pkg/operator/ceph/csi/version.go
index bcc8166b3..685a488f2 100644
--- a/pkg/operator/ceph/csi/version.go
+++ b/pkg/operator/ceph/csi/version.go
@@ -25,16 +25,22 @@ import (
 )
 
 var (
-	//minimum supported version is 3.3.0
-	minimum = CephCSIVersion{3, 3, 0}
+	//minimum supported version is 3.0.0
+	minimum = CephCSIVersion{3, 0, 0}
 	//supportedCSIVersions are versions that rook supports
+	releasev310          = CephCSIVersion{3, 1, 0}
+	releasev320          = CephCSIVersion{3, 2, 0}
 	releasev330          = CephCSIVersion{3, 3, 0}
 	releasev340          = CephCSIVersion{3, 4, 0}
 	supportedCSIVersions = []CephCSIVersion{
 		minimum,
+		releasev310,
+		releasev320,
 		releasev330,
 		releasev340,
 	}
+	// omap generator is supported in v3.2.0+
+	omapSupportedVersions = releasev320
 	// for parsing the output of `cephcsi`
 	versionCSIPattern = regexp.MustCompile(`v(\d+)\.(\d+)\.(\d+)`)
 )
@@ -51,6 +57,33 @@ func (v *CephCSIVersion) String() string {
 		v.Major, v.Minor, v.Bugfix)
 }
 
+// SupportsOMAPController checks if the detected version supports OMAP generator
+func (v *CephCSIVersion) SupportsOMAPController() bool {
+
+	// if AllowUnsupported is set also a csi-image greater than the supported ones are allowed
+	if AllowUnsupported {
+		return true
+	}
+
+	if !v.isAtLeast(&minimum) {
+		return false
+	}
+
+	if v.Major > omapSupportedVersions.Major {
+		return true
+	}
+	if v.Major == omapSupportedVersions.Major {
+		if v.Minor > omapSupportedVersions.Minor {
+			return true
+		}
+		if v.Minor == omapSupportedVersions.Minor {
+			return v.Bugfix >= omapSupportedVersions.Bugfix
+		}
+	}
+
+	return false
+}
+
 // Supported checks if the detected version is part of the known supported CSI versions
 func (v *CephCSIVersion) Supported() bool {
 	if !v.isAtLeast(&minimum) {
diff --git a/pkg/operator/ceph/file/filesystem.go b/pkg/operator/ceph/file/filesystem.go
index 2430c9b08..c6cba903b 100644
--- a/pkg/operator/ceph/file/filesystem.go
+++ b/pkg/operator/ceph/file/filesystem.go
@@ -18,8 +18,10 @@ package file
 
 import (
 	"fmt"
+	"syscall"
 
 	"github.com/rook/rook/pkg/operator/k8sutil"
+	"github.com/rook/rook/pkg/util/exec"
 
 	"github.com/pkg/errors"
 	cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
@@ -49,31 +51,37 @@ func createFilesystem(
 	ownerInfo *k8sutil.OwnerInfo,
 	dataDirHostPath string,
 ) error {
-	logger.Infof("start running mdses for filesystem %q", fs.Name)
-	c := mds.NewCluster(clusterInfo, context, clusterSpec, fs, ownerInfo, dataDirHostPath)
-	if err := c.Start(); err != nil {
-		return err
-	}
-
 	if len(fs.Spec.DataPools) != 0 {
 		f := newFS(fs.Name, fs.Namespace)
 		if err := f.doFilesystemCreate(context, clusterInfo, clusterSpec, fs.Spec); err != nil {
 			return errors.Wrapf(err, "failed to create filesystem %q", fs.Name)
 		}
 	}
+
+	filesystem, err := cephclient.GetFilesystem(context, clusterInfo, fs.Name)
+	if err != nil {
+		return errors.Wrapf(err, "failed to get filesystem %q", fs.Name)
+	}
+
 	if fs.Spec.MetadataServer.ActiveStandby {
-		if err := cephclient.AllowStandbyReplay(context, clusterInfo, fs.Name, fs.Spec.MetadataServer.ActiveStandby); err != nil {
+		if err = cephclient.AllowStandbyReplay(context, clusterInfo, fs.Name, fs.Spec.MetadataServer.ActiveStandby); err != nil {
 			return errors.Wrapf(err, "failed to set allow_standby_replay to filesystem %q", fs.Name)
 		}
 	}
 
 	// set the number of active mds instances
 	if fs.Spec.MetadataServer.ActiveCount > 1 {
-		if err := cephclient.SetNumMDSRanks(context, clusterInfo, fs.Name, fs.Spec.MetadataServer.ActiveCount); err != nil {
+		if err = cephclient.SetNumMDSRanks(context, clusterInfo, fs.Name, fs.Spec.MetadataServer.ActiveCount); err != nil {
 			logger.Warningf("failed setting active mds count to %d. %v", fs.Spec.MetadataServer.ActiveCount, err)
 		}
 	}
 
+	logger.Infof("start running mdses for filesystem %q", fs.Name)
+	c := mds.NewCluster(clusterInfo, context, clusterSpec, fs, filesystem, ownerInfo, dataDirHostPath)
+	if err := c.Start(); err != nil {
+		return err
+	}
+
 	return nil
 }
 
@@ -86,7 +94,15 @@ func deleteFilesystem(
 	ownerInfo *k8sutil.OwnerInfo,
 	dataDirHostPath string,
 ) error {
-	c := mds.NewCluster(clusterInfo, context, clusterSpec, fs, ownerInfo, dataDirHostPath)
+	filesystem, err := cephclient.GetFilesystem(context, clusterInfo, fs.Name)
+	if err != nil {
+		if code, ok := exec.ExitStatus(err); ok && code == int(syscall.ENOENT) {
+			// If we're deleting the filesystem anyway, ignore the error that the filesystem doesn't exist
+			return nil
+		}
+		return errors.Wrapf(err, "failed to get filesystem %q", fs.Name)
+	}
+	c := mds.NewCluster(clusterInfo, context, clusterSpec, fs, filesystem, ownerInfo, dataDirHostPath)
 
 	// Delete mds CephX keys and configuration in centralized mon database
 	replicas := fs.Spec.MetadataServer.ActiveCount * 2
@@ -94,7 +110,7 @@ func deleteFilesystem(
 		daemonLetterID := k8sutil.IndexToName(i)
 		daemonName := fmt.Sprintf("%s-%s", fs.Name, daemonLetterID)
 
-		err := c.DeleteMdsCephObjects(daemonName)
+		err = c.DeleteMdsCephObjects(daemonName)
 		if err != nil {
 			return errors.Wrapf(err, "failed to delete mds ceph objects for filesystem %q", fs.Name)
 		}
diff --git a/pkg/operator/ceph/file/mds/mds.go b/pkg/operator/ceph/file/mds/mds.go
index 368562507..d43d144b2 100644
--- a/pkg/operator/ceph/file/mds/mds.go
+++ b/pkg/operator/ceph/file/mds/mds.go
@@ -20,6 +20,7 @@ package mds
 import (
 	"context"
 	"fmt"
+	"strconv"
 	"strings"
 	"syscall"
 	"time"
@@ -57,6 +58,7 @@ type Cluster struct {
 	context         *clusterd.Context
 	clusterSpec     *cephv1.ClusterSpec
 	fs              cephv1.CephFilesystem
+	fsID            string
 	ownerInfo       *k8sutil.OwnerInfo
 	dataDirHostPath string
 }
@@ -73,6 +75,7 @@ func NewCluster(
 	context *clusterd.Context,
 	clusterSpec *cephv1.ClusterSpec,
 	fs cephv1.CephFilesystem,
+	fsdetails *cephclient.CephFilesystemDetails,
 	ownerInfo *k8sutil.OwnerInfo,
 	dataDirHostPath string,
 ) *Cluster {
@@ -81,6 +84,7 @@ func NewCluster(
 		context:         context,
 		clusterSpec:     clusterSpec,
 		fs:              fs,
+		fsID:            strconv.Itoa(fsdetails.ID),
 		ownerInfo:       ownerInfo,
 		dataDirHostPath: dataDirHostPath,
 	}
@@ -228,7 +232,7 @@ func (c *Cluster) isCephUpgrade() (bool, error) {
 			return false, err
 		}
 		if cephver.IsSuperior(c.clusterInfo.CephVersion, *currentVersion) {
-			logger.Debugf("ceph version for MDS %q is %q and target version is %q", key, currentVersion.String(), c.clusterInfo.CephVersion.String())
+			logger.Debugf("ceph version for MDS %q is %q and target version is %q", key, currentVersion, c.clusterInfo.CephVersion)
 			return true, err
 		}
 	}
@@ -245,8 +249,7 @@ func (c *Cluster) upgradeMDS() error {
 		return errors.Wrap(err, "failed to setting allow_standby_replay to false")
 	}
 
-	// In Pacific, standby-replay daemons are stopped automatically. Older versions of Ceph require
-	// us to stop these daemons manually.
+	// In Pacific, standby-replay daemons are stopped automatically. Older versions of Ceph require us to stop these daemons manually.
 	if err := cephclient.FailAllStandbyReplayMDS(c.context, c.clusterInfo, c.fs.Name); err != nil {
 		return errors.Wrap(err, "failed to fail mds agent in up:standby-replay state")
 	}
diff --git a/pkg/operator/ceph/object/admin.go b/pkg/operator/ceph/object/admin.go
index 1c60da838..250c10f11 100644
--- a/pkg/operator/ceph/object/admin.go
+++ b/pkg/operator/ceph/object/admin.go
@@ -323,19 +323,7 @@ func GetAdminOPSUserCredentials(objContext *Context, spec *cephv1.ObjectStoreSpe
 		AdminOpsUser: true,
 	}
 	logger.Debugf("creating s3 user object %q for object store %q", userConfig.UserID, ns)
-
-	forceUserCreation := false
-	// If the cluster where we are running the rgw user create for the admin ops user is configured
-	// as a secondary cluster the gateway will error out with:
-	// 		Please run the command on master zone. Performing this operation on non-master zone leads to
-	// 		inconsistent metadata between zones
-	// It is safe to force it since the creation will return that the user already exists since it
-	// has been created by the primary cluster. In this case, we simply read the user details.
-	if spec.IsMultisite() {
-		forceUserCreation = true
-	}
-
-	user, rgwerr, err := CreateUser(objContext, userConfig, forceUserCreation)
+	user, rgwerr, err := CreateUser(objContext, userConfig)
 	if err != nil {
 		if rgwerr == ErrorCodeFileExists {
 			user, _, err = GetUser(objContext, userConfig.UserID)
diff --git a/pkg/operator/ceph/object/bucket/provisioner.go b/pkg/operator/ceph/object/bucket/provisioner.go
index 7743f3218..31770ede8 100644
--- a/pkg/operator/ceph/object/bucket/provisioner.go
+++ b/pkg/operator/ceph/object/bucket/provisioner.go
@@ -81,7 +81,7 @@ func (p Provisioner) Provision(options *apibkt.BucketOptions) (*bktv1alpha1.Obje
 		return nil, errors.Wrap(err, "Provision: can't create ceph user")
 	}
 
-	s3svc, err := cephObject.NewS3Agent(p.accessKeyID, p.secretAccessKey, p.getObjectStoreEndpoint(), p.region, logger.LevelAt(capnslog.DEBUG), p.tlsCert)
+	s3svc, err := cephObject.NewS3Agent(p.accessKeyID, p.secretAccessKey, p.getObjectStoreEndpoint(), logger.LevelAt(capnslog.DEBUG), p.tlsCert)
 	if err != nil {
 		p.deleteOBCResourceLogError("")
 		return nil, err
@@ -158,7 +158,7 @@ func (p Provisioner) Grant(options *apibkt.BucketOptions) (*bktv1alpha1.ObjectBu
 		return nil, errors.Wrapf(err, "failed to get user %q", stats.Owner)
 	}
 
-	s3svc, err := cephObject.NewS3Agent(objectUser.Keys[0].AccessKey, objectUser.Keys[0].SecretKey, p.getObjectStoreEndpoint(), p.region, logger.LevelAt(capnslog.DEBUG), p.tlsCert)
+	s3svc, err := cephObject.NewS3Agent(objectUser.Keys[0].AccessKey, objectUser.Keys[0].SecretKey, p.getObjectStoreEndpoint(), logger.LevelAt(capnslog.DEBUG), p.tlsCert)
 	if err != nil {
 		p.deleteOBCResourceLogError("")
 		return nil, err
@@ -254,7 +254,7 @@ func (p Provisioner) Revoke(ob *bktv1alpha1.ObjectBucket) error {
 			return err
 		}
 
-		s3svc, err := cephObject.NewS3Agent(user.Keys[0].AccessKey, user.Keys[0].SecretKey, p.getObjectStoreEndpoint(), p.region, logger.LevelAt(capnslog.DEBUG), p.tlsCert)
+		s3svc, err := cephObject.NewS3Agent(user.Keys[0].AccessKey, user.Keys[0].SecretKey, p.getObjectStoreEndpoint(), logger.LevelAt(capnslog.DEBUG), p.tlsCert)
 		if err != nil {
 			return err
 		}
diff --git a/pkg/operator/ceph/object/health.go b/pkg/operator/ceph/object/health.go
index c2963902a..b3b051a7d 100644
--- a/pkg/operator/ceph/object/health.go
+++ b/pkg/operator/ceph/object/health.go
@@ -166,7 +166,7 @@ func (c *bucketChecker) checkObjectStoreHealth() error {
 
 	// Initiate s3 agent
 	logger.Debugf("initializing s3 connection for object store %q", c.namespacedName.Name)
-	s3client, err := NewS3Agent(s3AccessKey, s3SecretKey, s3endpoint, "", false, tlsCert)
+	s3client, err := NewS3Agent(s3AccessKey, s3SecretKey, s3endpoint, false, tlsCert)
 	if err != nil {
 		return errors.Wrap(err, "failed to initialize s3 connection")
 	}
diff --git a/pkg/operator/ceph/object/objectstore.go b/pkg/operator/ceph/object/objectstore.go
index 1a4c9c087..007f36f07 100644
--- a/pkg/operator/ceph/object/objectstore.go
+++ b/pkg/operator/ceph/object/objectstore.go
@@ -861,10 +861,7 @@ func enableRGWDashboard(context *Context) error {
 	}
 	// TODO:
 	// Use admin ops user instead!
-	// It's safe to create the user with the force flag regardless if the cluster's dashboard is
-	// configured as a secondary rgw site. The creation will return the user already exists and we
-	// will just fetch it (it has been created by the primary cluster)
-	u, errCode, err := CreateUser(context, user, true)
+	u, errCode, err := CreateUser(context, user)
 	if err != nil || errCode != 0 {
 		return errors.Wrapf(err, "failed to create user %q", DashboardUser)
 	}
diff --git a/pkg/operator/ceph/object/s3-handlers.go b/pkg/operator/ceph/object/s3-handlers.go
index 74b8b76c1..98701f734 100644
--- a/pkg/operator/ceph/object/s3-handlers.go
+++ b/pkg/operator/ceph/object/s3-handlers.go
@@ -36,19 +36,16 @@ type S3Agent struct {
 	Client *s3.S3
 }
 
-func NewS3Agent(accessKey, secretKey, endpoint, region string, debug bool, tlsCert []byte) (*S3Agent, error) {
-	return newS3Agent(accessKey, secretKey, endpoint, region, debug, tlsCert, false)
+func NewS3Agent(accessKey, secretKey, endpoint string, debug bool, tlsCert []byte) (*S3Agent, error) {
+	return newS3Agent(accessKey, secretKey, endpoint, debug, tlsCert, false)
 }
 
-func NewTestOnlyS3Agent(accessKey, secretKey, endpoint, region string, debug bool) (*S3Agent, error) {
-	return newS3Agent(accessKey, secretKey, endpoint, region, debug, nil, true)
+func NewTestOnlyS3Agent(accessKey, secretKey, endpoint string, debug bool) (*S3Agent, error) {
+	return newS3Agent(accessKey, secretKey, endpoint, debug, nil, true)
 }
 
-func newS3Agent(accessKey, secretKey, endpoint, region string, debug bool, tlsCert []byte, insecure bool) (*S3Agent, error) {
-	var cephRegion = "us-east-1"
-	if region != "" {
-		cephRegion = region
-	}
+func newS3Agent(accessKey, secretKey, endpoint string, debug bool, tlsCert []byte, insecure bool) (*S3Agent, error) {
+	const cephRegion = "us-east-1"
 
 	logLevel := aws.LogOff
 	if debug {
diff --git a/pkg/operator/ceph/object/status.go b/pkg/operator/ceph/object/status.go
index ddf4c5ad4..a369a981b 100644
--- a/pkg/operator/ceph/object/status.go
+++ b/pkg/operator/ceph/object/status.go
@@ -89,12 +89,12 @@ func updateStatusBucket(client client.Client, name types.NamespacedName, status
 		}
 		objectStore.Status.BucketStatus = toCustomResourceStatus(objectStore.Status.BucketStatus, details, status)
 
-		// do not transition to other statuses once deletion begins
 		if objectStore.Status.Phase != cephv1.ConditionDeleting {
+			// do not transition to to other statuses once deletion begins
+			logger.Debugf("object store %q status not updated to %q because it is deleting", name.String(), status)
 			objectStore.Status.Phase = status
 		}
 
-		// but we still need to update the health checker status
 		if err := reporting.UpdateStatus(client, objectStore); err != nil {
 			return errors.Wrapf(err, "failed to set object store %q status to %v", name.String(), status)
 		}
diff --git a/pkg/operator/ceph/object/user.go b/pkg/operator/ceph/object/user.go
index cae2d055d..9eda431c8 100644
--- a/pkg/operator/ceph/object/user.go
+++ b/pkg/operator/ceph/object/user.go
@@ -104,7 +104,7 @@ func GetUser(c *Context, id string) (*ObjectUser, int, error) {
 // CreateUser creates a new user with the information given.
 // The function is used **ONCE** only to provision so the RGW Admin Ops User
 // Subsequent interaction with the API will be done with the created user
-func CreateUser(c *Context, user ObjectUser, force bool) (*ObjectUser, int, error) {
+func CreateUser(c *Context, user ObjectUser) (*ObjectUser, int, error) {
 	logger.Debugf("creating s3 user %q", user.UserID)
 
 	if strings.TrimSpace(user.UserID) == "" {
@@ -134,16 +134,8 @@ func CreateUser(c *Context, user ObjectUser, force bool) (*ObjectUser, int, erro
 		args = append(args, "--caps", rgwAdminOpsUserCaps)
 	}
 
-	if force {
-		args = append(args, "--yes-i-really-mean-it")
-	}
-
 	result, err := runAdminCommand(c, true, args...)
 	if err != nil {
-		if code, err := exec.ExtractExitCode(err); err == nil && code == int(syscall.EEXIST) {
-			return nil, ErrorCodeFileExists, errors.New("s3 user already exists")
-		}
-
 		if strings.Contains(result, "could not create user: unable to create user, user: ") {
 			return nil, ErrorCodeFileExists, errors.New("s3 user already exists")
 		}


Test patch:
diff --git a/.github/workflows/canary-integration-test.yml b/.github/workflows/canary-integration-test.yml
index 93cb408bc..032fd06bb 100644
--- a/.github/workflows/canary-integration-test.yml
+++ b/.github/workflows/canary-integration-test.yml
@@ -860,7 +860,7 @@ jobs:
 
     - name: verify fs mirroring is working
       run: |
-        timeout 45 sh -c 'until [ "$(kubectl -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ls -1 /var/run/ceph/|grep -c asok)" -lt 3 ]; do echo "waiting for connection to peer" && sleep 1; done'
+        timeout 45 sh -c 'until [ "$(kubectl -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ls -1 /var/run/ceph/|grep -c asok)" -eq 3 ]; do echo "waiting for connection to peer" && sleep 1; done'
         sockets=$(kubectl -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ls -1 /var/run/ceph/)
         status=$(for socket in $sockets; do minikube kubectl -- -n rook-ceph exec -t deploy/rook-ceph-fs-mirror -- ceph --admin-daemon /var/run/ceph/$socket help|awk -F ":" '/get filesystem mirror status/ {print $1}'; done)
         if [ "${#status}" -lt 1 ]; then echo "peer addition failed" && exit 1; fi
diff --git a/cluster/examples/kubernetes/ceph/cluster-test.yaml b/cluster/examples/kubernetes/ceph/cluster-test.yaml
index 0855f95bd..9900c734b 100644
--- a/cluster/examples/kubernetes/ceph/cluster-test.yaml
+++ b/cluster/examples/kubernetes/ceph/cluster-test.yaml
@@ -28,7 +28,7 @@ metadata:
 spec:
   dataDirHostPath: /var/lib/rook
   cephVersion:
-    image: quay.io/ceph/ceph:v16.2.6
+    image: quay.io/ceph/ceph:v16.2.5
     allowUnsupported: true
   mon:
     count: 1
diff --git a/pkg/operator/ceph/csi/version_test.go b/pkg/operator/ceph/csi/version_test.go
index c69ae0c4c..f09dacf1a 100644
--- a/pkg/operator/ceph/csi/version_test.go
+++ b/pkg/operator/ceph/csi/version_test.go
@@ -23,7 +23,11 @@ import (
 )
 
 var (
-	testMinVersion         = CephCSIVersion{3, 3, 0}
+	testMinVersion         = CephCSIVersion{2, 0, 0}
+	testReleaseV210        = CephCSIVersion{2, 1, 0}
+	testReleaseV300        = CephCSIVersion{3, 0, 0}
+	testReleaseV320        = CephCSIVersion{3, 2, 0}
+	testReleaseV321        = CephCSIVersion{3, 2, 1}
 	testReleaseV330        = CephCSIVersion{3, 3, 0}
 	testReleaseV340        = CephCSIVersion{3, 4, 0}
 	testVersionUnsupported = CephCSIVersion{4, 0, 0}
@@ -39,8 +43,44 @@ func TestIsAtLeast(t *testing.T) {
 	ret = testMinVersion.isAtLeast(&testMinVersion)
 	assert.Equal(t, true, ret)
 
+	// Test version which is greater (minor)
+	version = CephCSIVersion{2, 1, 0}
+	ret = testMinVersion.isAtLeast(&version)
+	assert.Equal(t, false, ret)
+
+	// Test version which is greater (bugfix)
+	version = CephCSIVersion{2, 2, 0}
+	ret = testMinVersion.isAtLeast(&version)
+	assert.Equal(t, false, ret)
+
+	// Test for v2.1.0
+	// Test version which is greater (bugfix)
+	version = CephCSIVersion{2, 0, 1}
+	ret = testReleaseV210.isAtLeast(&version)
+	assert.Equal(t, true, ret)
+
 	// Test version which is equal
-	ret = testReleaseV330.isAtLeast(&testReleaseV330)
+	ret = testReleaseV210.isAtLeast(&testReleaseV210)
+	assert.Equal(t, true, ret)
+
+	// Test version which is greater (minor)
+	version = CephCSIVersion{2, 1, 1}
+	ret = testReleaseV210.isAtLeast(&version)
+	assert.Equal(t, false, ret)
+
+	// Test version which is greater (bugfix)
+	version = CephCSIVersion{2, 2, 0}
+	ret = testReleaseV210.isAtLeast(&version)
+	assert.Equal(t, false, ret)
+
+	// Test for 3.0.0
+	// Test version which is equal
+	ret = testReleaseV300.isAtLeast(&testReleaseV300)
+	assert.Equal(t, true, ret)
+
+	// Test for 3.3.0
+	// Test version which is lesser
+	ret = testReleaseV330.isAtLeast(&testReleaseV300)
 	assert.Equal(t, true, ret)
 
 	// Test for 3.4.0
@@ -48,23 +88,50 @@ func TestIsAtLeast(t *testing.T) {
 	ret = testReleaseV340.isAtLeast(&testReleaseV330)
 	assert.Equal(t, true, ret)
 
+	// Test version which is greater (minor)
+	version = CephCSIVersion{3, 1, 1}
+	ret = testReleaseV300.isAtLeast(&version)
+	assert.Equal(t, false, ret)
+
+	// Test version which is greater (bugfix)
+	version = CephCSIVersion{3, 2, 0}
+	ret = testReleaseV300.isAtLeast(&version)
+	assert.Equal(t, false, ret)
 }
 
 func TestSupported(t *testing.T) {
 	AllowUnsupported = false
 	ret := testMinVersion.Supported()
-	assert.Equal(t, true, ret)
+	assert.Equal(t, false, ret)
 
 	ret = testVersionUnsupported.Supported()
 	assert.Equal(t, false, ret)
 
-	ret = testReleaseV330.Supported()
-	assert.Equal(t, true, ret)
-
 	ret = testReleaseV340.Supported()
 	assert.Equal(t, true, ret)
 }
 
+func TestSupportOMAPController(t *testing.T) {
+	AllowUnsupported = true
+	ret := testMinVersion.SupportsOMAPController()
+	assert.True(t, ret)
+
+	AllowUnsupported = false
+	ret = testMinVersion.SupportsOMAPController()
+	assert.False(t, ret)
+
+	ret = testReleaseV300.SupportsOMAPController()
+	assert.False(t, ret)
+
+	ret = testReleaseV320.SupportsOMAPController()
+	assert.True(t, ret)
+
+	ret = testReleaseV321.SupportsOMAPController()
+	assert.True(t, ret)
+
+	ret = testReleaseV330.SupportsOMAPController()
+	assert.True(t, ret)
+}
 func Test_extractCephCSIVersion(t *testing.T) {
 	expectedVersion := CephCSIVersion{3, 0, 0}
 	csiString := []byte(`Cephcsi Version: v3.0.0
diff --git a/pkg/operator/ceph/file/filesystem_test.go b/pkg/operator/ceph/file/filesystem_test.go
index b3a488455..263781038 100644
--- a/pkg/operator/ceph/file/filesystem_test.go
+++ b/pkg/operator/ceph/file/filesystem_test.go
@@ -95,7 +95,7 @@ func isBasePoolOperation(fsName, command string, args []string) bool {
 	return false
 }
 
-func fsExecutor(t *testing.T, fsName, configDir string, multiFS bool, createDataOnePoolCount, addDataOnePoolCount *int) *exectest.MockExecutor {
+func fsExecutor(t *testing.T, fsName, configDir string, multiFS bool) *exectest.MockExecutor {
 	mdsmap := cephclient.CephFilesystemDetails{
 		ID: 0,
 		MDSMap: cephclient.MDSMap{
@@ -160,16 +160,6 @@ func fsExecutor(t *testing.T, fsName, configDir string, multiFS bool, createData
 					return "", nil
 				} else if contains(args, "flag") && contains(args, "enable_multiple") {
 					return "", nil
-				} else if reflect.DeepEqual(args[0:5], []string{"osd", "crush", "rule", "create-replicated", fsName + "-data1"}) {
-					return "", nil
-				} else if reflect.DeepEqual(args[0:4], []string{"osd", "pool", "create", fsName + "-data1"}) {
-					*createDataOnePoolCount++
-					return "", nil
-				} else if reflect.DeepEqual(args[0:6], []string{"osd", "pool", "set", fsName + "-data1", "size", "1"}) {
-					return "", nil
-				} else if reflect.DeepEqual(args[0:4], []string{"fs", "add_data_pool", fsName, fsName + "-data1"}) {
-					*addDataOnePoolCount++
-					return "", nil
 				} else if contains(args, "versions") {
 					versionStr, _ := json.Marshal(
 						map[string]map[string]int{
@@ -223,16 +213,6 @@ func fsExecutor(t *testing.T, fsName, configDir string, multiFS bool, createData
 				return "", nil
 			} else if contains(args, "config") && contains(args, "get") {
 				return "{}", nil
-			} else if reflect.DeepEqual(args[0:5], []string{"osd", "crush", "rule", "create-replicated", fsName + "-data1"}) {
-				return "", nil
-			} else if reflect.DeepEqual(args[0:4], []string{"osd", "pool", "create", fsName + "-data1"}) {
-				*createDataOnePoolCount++
-				return "", nil
-			} else if reflect.DeepEqual(args[0:6], []string{"osd", "pool", "set", fsName + "-data1", "size", "1"}) {
-				return "", nil
-			} else if reflect.DeepEqual(args[0:4], []string{"fs", "add_data_pool", fsName, fsName + "-data1"}) {
-				*addDataOnePoolCount++
-				return "", nil
 			} else if contains(args, "versions") {
 				versionStr, _ := json.Marshal(
 					map[string]map[string]int{
@@ -277,10 +257,9 @@ func TestCreateFilesystem(t *testing.T) {
 	var deploymentsUpdated *[]*apps.Deployment
 	mds.UpdateDeploymentAndWait, deploymentsUpdated = testopk8s.UpdateDeploymentAndWaitStub()
 	configDir, _ := ioutil.TempDir("", "")
+
 	fsName := "myfs"
-	addDataOnePoolCount := 0
-	createDataOnePoolCount := 0
-	executor := fsExecutor(t, fsName, configDir, false, &createDataOnePoolCount, &addDataOnePoolCount)
+	executor := fsExecutor(t, fsName, configDir, false)
 	defer os.RemoveAll(configDir)
 	clientset := testop.New(t, 1)
 	context := &clusterd.Context{
@@ -289,58 +268,117 @@ func TestCreateFilesystem(t *testing.T) {
 		Clientset: clientset}
 	fs := fsTest(fsName)
 	clusterInfo := &cephclient.ClusterInfo{FSID: "myfsid", CephVersion: version.Octopus, Context: ctx}
+
+	// start a basic cluster
 	ownerInfo := cephclient.NewMinimumOwnerInfoWithOwnerRef()
+	err := createFilesystem(context, clusterInfo, fs, &cephv1.ClusterSpec{}, ownerInfo, "/var/lib/rook/")
+	assert.Nil(t, err)
+	validateStart(ctx, t, context, fs)
+	assert.ElementsMatch(t, []string{}, testopk8s.DeploymentNamesUpdated(deploymentsUpdated))
+	testopk8s.ClearDeploymentsUpdated(deploymentsUpdated)
 
-	t.Run("start basic filesystem", func(t *testing.T) {
-		// start a basic cluster
-		err := createFilesystem(context, clusterInfo, fs, &cephv1.ClusterSpec{}, ownerInfo, "/var/lib/rook/")
-		assert.Nil(t, err)
-		validateStart(ctx, t, context, fs)
-		assert.ElementsMatch(t, []string{}, testopk8s.DeploymentNamesUpdated(deploymentsUpdated))
-		testopk8s.ClearDeploymentsUpdated(deploymentsUpdated)
-	})
-
-	t.Run("start again should no-op", func(t *testing.T) {
-		err := createFilesystem(context, clusterInfo, fs, &cephv1.ClusterSpec{}, ownerInfo, "/var/lib/rook/")
-		assert.Nil(t, err)
-		validateStart(ctx, t, context, fs)
-		assert.ElementsMatch(t, []string{fmt.Sprintf("rook-ceph-mds-%s-a", fsName), fmt.Sprintf("rook-ceph-mds-%s-b", fsName)}, testopk8s.DeploymentNamesUpdated(deploymentsUpdated))
-		testopk8s.ClearDeploymentsUpdated(deploymentsUpdated)
-	})
-
-	t.Run("increasing the number of data pools should be successful.", func(t *testing.T) {
-		context = &clusterd.Context{
-			Executor:  executor,
-			ConfigDir: configDir,
-			Clientset: clientset}
-		fs.Spec.DataPools = append(fs.Spec.DataPools, cephv1.PoolSpec{Replicated: cephv1.ReplicatedSpec{Size: 1, RequireSafeReplicaSize: false}})
-		err := createFilesystem(context, clusterInfo, fs, &cephv1.ClusterSpec{}, ownerInfo, "/var/lib/rook/")
-		assert.Nil(t, err)
-		validateStart(ctx, t, context, fs)
-		assert.ElementsMatch(t, []string{fmt.Sprintf("rook-ceph-mds-%s-a", fsName), fmt.Sprintf("rook-ceph-mds-%s-b", fsName)}, testopk8s.DeploymentNamesUpdated(deploymentsUpdated))
-		assert.Equal(t, 1, createDataOnePoolCount)
-		assert.Equal(t, 1, addDataOnePoolCount)
-		testopk8s.ClearDeploymentsUpdated(deploymentsUpdated)
-	})
-
-	t.Run("multiple filesystem creation", func(t *testing.T) {
-		context = &clusterd.Context{
-			Executor:  fsExecutor(t, fsName, configDir, true, &createDataOnePoolCount, &addDataOnePoolCount),
-			ConfigDir: configDir,
-			Clientset: clientset,
-		}
+	// starting again should be a no-op
+	err = createFilesystem(context, clusterInfo, fs, &cephv1.ClusterSpec{}, ownerInfo, "/var/lib/rook/")
+	assert.Nil(t, err)
+	validateStart(ctx, t, context, fs)
+	assert.ElementsMatch(t, []string{fmt.Sprintf("rook-ceph-mds-%s-a", fsName), fmt.Sprintf("rook-ceph-mds-%s-b", fsName)}, testopk8s.DeploymentNamesUpdated(deploymentsUpdated))
+	testopk8s.ClearDeploymentsUpdated(deploymentsUpdated)
 
-		// Create another filesystem which should fail
-		err := createFilesystem(context, clusterInfo, fs, &cephv1.ClusterSpec{}, &k8sutil.OwnerInfo{}, "/var/lib/rook/")
-		assert.Error(t, err)
-		assert.Equal(t, fmt.Sprintf("failed to create filesystem %q: multiple filesystems are only supported as of ceph pacific", fsName), err.Error())
-	})
-
-	t.Run("multi filesystem creation now works since ceph version is pacific", func(t *testing.T) {
-		clusterInfo.CephVersion = version.Pacific
-		err := createFilesystem(context, clusterInfo, fs, &cephv1.ClusterSpec{}, ownerInfo, "/var/lib/rook/")
-		assert.NoError(t, err)
-	})
+	// Increasing the number of data pools should be successful.
+	createDataOnePoolCount := 0
+	addDataOnePoolCount := 0
+	createdFsResponse := fmt.Sprintf(`{"fs_name": "%s", "metadata_pool": 2, "data_pools":[3]}`, fsName)
+	executor = &exectest.MockExecutor{
+		MockExecuteCommandWithOutput: func(command string, args ...string) (string, error) {
+			if contains(args, "fs") && contains(args, "get") {
+				return createdFsResponse, nil
+			} else if isBasePoolOperation(fsName, command, args) {
+				return "", nil
+			} else if reflect.DeepEqual(args[0:4], []string{"osd", "pool", "create", fsName + "-data1"}) {
+				createDataOnePoolCount++
+				return "", nil
+			} else if reflect.DeepEqual(args[0:4], []string{"fs", "add_data_pool", fsName, fsName + "-data1"}) {
+				addDataOnePoolCount++
+				return "", nil
+			} else if contains(args, "set") && contains(args, "max_mds") {
+				return "", nil
+			} else if contains(args, "auth") && contains(args, "get-or-create-key") {
+				return "{\"key\":\"mysecurekey\"}", nil
+			} else if reflect.DeepEqual(args[0:5], []string{"osd", "crush", "rule", "create-replicated", fsName + "-data1"}) {
+				return "", nil
+			} else if reflect.DeepEqual(args[0:6], []string{"osd", "pool", "set", fsName + "-data1", "size", "1"}) {
+				return "", nil
+			} else if args[0] == "config" && args[1] == "set" {
+				return "", nil
+			} else if contains(args, "versions") {
+				versionStr, _ := json.Marshal(
+					map[string]map[string]int{
+						"mds": {
+							"ceph version 16.0.0-4-g2f728b9 (2f728b952cf293dd7f809ad8a0f5b5d040c43010) pacific (stable)": 2,
+						},
+					})
+				return string(versionStr), nil
+			}
+			assert.Fail(t, fmt.Sprintf("Unexpected command: %v", args))
+			return "", nil
+		},
+	}
+	context = &clusterd.Context{
+		Executor:  executor,
+		ConfigDir: configDir,
+		Clientset: clientset}
+	fs.Spec.DataPools = append(fs.Spec.DataPools, cephv1.PoolSpec{Replicated: cephv1.ReplicatedSpec{Size: 1, RequireSafeReplicaSize: false}})
+
+	err = createFilesystem(context, clusterInfo, fs, &cephv1.ClusterSpec{}, ownerInfo, "/var/lib/rook/")
+	assert.Nil(t, err)
+	validateStart(ctx, t, context, fs)
+	assert.ElementsMatch(t, []string{fmt.Sprintf("rook-ceph-mds-%s-a", fsName), fmt.Sprintf("rook-ceph-mds-%s-b", fsName)}, testopk8s.DeploymentNamesUpdated(deploymentsUpdated))
+	assert.Equal(t, 1, createDataOnePoolCount)
+	assert.Equal(t, 1, addDataOnePoolCount)
+	testopk8s.ClearDeploymentsUpdated(deploymentsUpdated)
+
+	// Test multiple filesystem creation
+	// Output to check multiple filesystem creation
+	fses := `[{"name":"myfs","metadata_pool":"myfs-metadata","metadata_pool_id":4,"data_pool_ids":[5],"data_pools":["myfs-data0"]},{"name":"myfs2","metadata_pool":"myfs2-metadata","metadata_pool_id":6,"data_pool_ids":[7],"data_pools":["myfs2-data0"]},{"name":"leseb","metadata_pool":"cephfs.leseb.meta","metadata_pool_id":8,"data_pool_ids":[9],"data_pools":["cephfs.leseb.data"]}]`
+	executorMultiFS := &exectest.MockExecutor{
+		MockExecuteCommandWithOutput: func(command string, args ...string) (string, error) {
+			if contains(args, "ls") {
+				return fses, nil
+			} else if contains(args, "versions") {
+				versionStr, _ := json.Marshal(
+					map[string]map[string]int{
+						"mds": {
+							"ceph version 16.0.0-4-g2f728b9 (2f728b952cf293dd7f809ad8a0f5b5d040c43010) pacific (stable)": 2,
+						},
+					})
+				return string(versionStr), nil
+			}
+			return "{\"key\":\"mysecurekey\"}", errors.New("multiple fs")
+		},
+	}
+	context = &clusterd.Context{
+		Executor:  executorMultiFS,
+		ConfigDir: configDir,
+		Clientset: clientset,
+	}
+
+	// Create another filesystem which should fail
+	err = createFilesystem(context, clusterInfo, fs, &cephv1.ClusterSpec{}, &k8sutil.OwnerInfo{}, "/var/lib/rook/")
+	assert.Error(t, err)
+	assert.Equal(t, fmt.Sprintf("failed to create filesystem %q: multiple filesystems are only supported as of ceph pacific", fsName), err.Error())
+
+	// It works since the Ceph version is Pacific
+	fsName = "myfs3"
+	fs = fsTest(fsName)
+	executor = fsExecutor(t, fsName, configDir, true)
+	clusterInfo.CephVersion = version.Pacific
+	context = &clusterd.Context{
+		Executor:  executor,
+		ConfigDir: configDir,
+		Clientset: clientset,
+	}
+	err = createFilesystem(context, clusterInfo, fs, &cephv1.ClusterSpec{}, ownerInfo, "/var/lib/rook/")
+	assert.NoError(t, err)
 }
 
 func TestUpgradeFilesystem(t *testing.T) {
@@ -350,9 +388,7 @@ func TestUpgradeFilesystem(t *testing.T) {
 	configDir, _ := ioutil.TempDir("", "")
 
 	fsName := "myfs"
-	addDataOnePoolCount := 0
-	createDataOnePoolCount := 0
-	executor := fsExecutor(t, fsName, configDir, false, &createDataOnePoolCount, &addDataOnePoolCount)
+	executor := fsExecutor(t, fsName, configDir, false)
 	defer os.RemoveAll(configDir)
 	clientset := testop.New(t, 1)
 	context := &clusterd.Context{
diff --git a/pkg/operator/ceph/file/mds/spec_test.go b/pkg/operator/ceph/file/mds/spec_test.go
index 864145a92..d6c9d53e4 100644
--- a/pkg/operator/ceph/file/mds/spec_test.go
+++ b/pkg/operator/ceph/file/mds/spec_test.go
@@ -72,6 +72,7 @@ func testDeploymentObject(t *testing.T, network cephv1.NetworkSpec) (*apps.Deplo
 			Network:     network,
 		},
 		fs,
+		&cephclient.CephFilesystemDetails{ID: 15},
 		&k8sutil.OwnerInfo{},
 		"/var/lib/rook/",
 	)
diff --git a/tests/framework/installer/ceph_helm_installer.go b/tests/framework/installer/ceph_helm_installer.go
index ec0fa60c4..234c225ea 100644
--- a/tests/framework/installer/ceph_helm_installer.go
+++ b/tests/framework/installer/ceph_helm_installer.go
@@ -77,7 +77,7 @@ func (h *CephInstaller) CreateRookCephClusterViaHelm(values map[string]interface
 	values["configOverride"] = clusterCustomSettings
 	values["toolbox"] = map[string]interface{}{
 		"enabled": true,
-		"image":   "rook/ceph:" + LocalBuildTag,
+		"image":   "rook/ceph:master",
 	}
 	values["cephClusterSpec"] = clusterCRD["spec"]
 
diff --git a/tests/framework/installer/ceph_installer.go b/tests/framework/installer/ceph_installer.go
index 6fef6d02f..3615b61ac 100644
--- a/tests/framework/installer/ceph_installer.go
+++ b/tests/framework/installer/ceph_installer.go
@@ -50,7 +50,7 @@ const (
 	// test with the latest octopus build
 	octopusTestImage = "quay.io/ceph/ceph:v15"
 	// test with the latest pacific build
-	pacificTestImage = "quay.io/ceph/ceph:v16.2.6"
+	pacificTestImage = "quay.io/ceph/ceph:v16.2.5"
 	// test with the latest master image
 	masterTestImage    = "ceph/daemon-base:latest-master-devel"
 	cephOperatorLabel  = "app=rook-ceph-operator"
@@ -448,7 +448,7 @@ func (h *CephInstaller) installRookOperator() (bool, error) {
 		startDiscovery = true
 		err := h.CreateRookOperatorViaHelm(map[string]interface{}{
 			"enableDiscoveryDaemon": true,
-			"image":                 map[string]interface{}{"tag": LocalBuildTag},
+			"image":                 map[string]interface{}{"tag": "master"},
 		})
 		if err != nil {
 			return false, errors.Wrap(err, "failed to configure helm")
@@ -479,7 +479,7 @@ func (h *CephInstaller) installRookOperator() (bool, error) {
 }
 
 func (h *CephInstaller) InstallRook() (bool, error) {
-	if h.settings.RookVersion != LocalBuildTag {
+	if h.settings.RookVersion != VersionMaster {
 		// make sure we have the images from a previous release locally so the test doesn't hit a timeout
 		assert.NoError(h.T(), h.k8shelper.GetDockerImage("rook/ceph:"+h.settings.RookVersion))
 	}
@@ -499,7 +499,7 @@ func (h *CephInstaller) InstallRook() (bool, error) {
 
 	if h.settings.UseHelm {
 		err = h.CreateRookCephClusterViaHelm(map[string]interface{}{
-			"image": "rook/ceph:" + LocalBuildTag,
+			"image": "rook/ceph:master",
 		})
 		if err != nil {
 			return false, errors.Wrap(err, "failed to install ceph cluster using Helm")
@@ -901,7 +901,7 @@ spec:
           restartPolicy: Never
           containers:
               - name: rook-cleaner
-                image: rook/ceph:` + LocalBuildTag + `
+                image: rook/ceph:` + VersionMaster + `
                 securityContext:
                     privileged: true
                 volumeMounts:
@@ -931,7 +931,7 @@ spec:
           restartPolicy: Never
           containers:
               - name: rook-cleaner
-                image: rook/ceph:` + LocalBuildTag + `
+                image: rook/ceph:` + VersionMaster + `
                 securityContext:
                     privileged: true
                 volumeMounts:
diff --git a/tests/framework/installer/ceph_manifests.go b/tests/framework/installer/ceph_manifests.go
index 5b405230c..4c0704732 100644
--- a/tests/framework/installer/ceph_manifests.go
+++ b/tests/framework/installer/ceph_manifests.go
@@ -56,7 +56,7 @@ type CephManifestsMaster struct {
 // NewCephManifests gets the manifest type depending on the Rook version desired
 func NewCephManifests(settings *TestCephSettings) CephManifests {
 	switch settings.RookVersion {
-	case LocalBuildTag:
+	case VersionMaster:
 		return &CephManifestsMaster{settings}
 	case Version1_6:
 		return &CephManifestsV1_6{settings}
diff --git a/tests/framework/installer/installer.go b/tests/framework/installer/installer.go
index 719a148a1..929b10e90 100644
--- a/tests/framework/installer/installer.go
+++ b/tests/framework/installer/installer.go
@@ -28,8 +28,8 @@ import (
 )
 
 const (
-	// LocalBuildTag tag for the latest manifests
-	LocalBuildTag = "local-build"
+	// VersionMaster tag for the latest manifests
+	VersionMaster = "master"
 
 	// test suite names
 	CassandraTestSuite = "cassandra"
diff --git a/tests/framework/installer/settings.go b/tests/framework/installer/settings.go
index 7bb7a4c23..033cf957a 100644
--- a/tests/framework/installer/settings.go
+++ b/tests/framework/installer/settings.go
@@ -21,15 +21,12 @@ import (
 	"io/ioutil"
 	"net/http"
 	"path"
-	"regexp"
 	"time"
 
 	"github.com/pkg/errors"
 	"github.com/rook/rook/tests/framework/utils"
 )
 
-var imageMatch = regexp.MustCompile(`image: rook\/ceph:[a-z0-9.-]+`)
-
 func readManifest(provider, filename string) string {
 	rootDir, err := utils.FindRookRoot()
 	if err != nil {
@@ -41,7 +38,7 @@ func readManifest(provider, filename string) string {
 	if err != nil {
 		panic(errors.Wrapf(err, "failed to read manifest at %s", manifest))
 	}
-	return imageMatch.ReplaceAllString(string(contents), "image: rook/ceph:"+LocalBuildTag)
+	return string(contents)
 }
 
 func readManifestFromGithub(rookVersion, provider, filename string) string {
diff --git a/tests/integration/ceph_base_file_test.go b/tests/integration/ceph_base_file_test.go
index 2f4743926..f2acba483 100644
--- a/tests/integration/ceph_base_file_test.go
+++ b/tests/integration/ceph_base_file_test.go
@@ -385,24 +385,9 @@ func createFilesystem(helper *clients.TestClient, k8sh *utils.K8sHelper, s suite
 	logger.Infof("Create file System")
 	fscErr := helper.FSClient.Create(filesystemName, settings.Namespace, activeCount)
 	require.Nil(s.T(), fscErr)
-	var err error
+	logger.Infof("File system %s created", filesystemName)
 
-	var filesystemList []cephclient.CephFilesystem
-	for i := 1; i <= 10; i++ {
-		filesystemList, err = helper.FSClient.List(settings.Namespace)
-		if err != nil {
-			logger.Errorf("failed to list fs. trying again. %v", err)
-			continue
-		}
-		logger.Debugf("filesystemList is %+v", filesystemList)
-		if len(filesystemList) == 1 {
-			logger.Infof("File system %s created", filesystemList[0].Name)
-			break
-		}
-		logger.Infof("Waiting for file system %s to be created", filesystemName)
-		time.Sleep(time.Second * 5)
-	}
-	logger.Debugf("filesystemList is %+v", filesystemList)
+	filesystemList, _ := helper.FSClient.List(settings.Namespace)
 	require.Equal(s.T(), 1, len(filesystemList), "There should be one shared file system present")
 }
 
diff --git a/tests/integration/ceph_flex_test.go b/tests/integration/ceph_flex_test.go
index 3949264cd..d41c1934b 100644
--- a/tests/integration/ceph_flex_test.go
+++ b/tests/integration/ceph_flex_test.go
@@ -94,7 +94,7 @@ func (s *CephFlexDriverSuite) SetupSuite() {
 		SkipOSDCreation:    false,
 		UseCSI:             false,
 		DirectMountToolbox: true,
-		RookVersion:        installer.LocalBuildTag,
+		RookVersion:        installer.VersionMaster,
 		CephVersion:        installer.OctopusVersion,
 	}
 	s.settings.ApplyEnvVars()
diff --git a/tests/integration/ceph_helm_test.go b/tests/integration/ceph_helm_test.go
index 9fdccf62a..46e7fee64 100644
--- a/tests/integration/ceph_helm_test.go
+++ b/tests/integration/ceph_helm_test.go
@@ -73,7 +73,7 @@ func (h *HelmSuite) SetupSuite() {
 		SkipOSDCreation:           false,
 		EnableAdmissionController: false,
 		EnableDiscovery:           true,
-		RookVersion:               installer.LocalBuildTag,
+		RookVersion:               installer.VersionMaster,
 		CephVersion:               installer.OctopusVersion,
 	}
 	h.settings.ApplyEnvVars()
diff --git a/tests/integration/ceph_mgr_test.go b/tests/integration/ceph_mgr_test.go
index e7c1baebb..1ae55807f 100644
--- a/tests/integration/ceph_mgr_test.go
+++ b/tests/integration/ceph_mgr_test.go
@@ -95,7 +95,7 @@ func (s *CephMgrSuite) SetupSuite() {
 		UseCSI:            true,
 		SkipOSDCreation:   true,
 		EnableDiscovery:   true,
-		RookVersion:       installer.LocalBuildTag,
+		RookVersion:       installer.VersionMaster,
 		CephVersion:       installer.MasterVersion,
 	}
 	s.settings.ApplyEnvVars()
@@ -224,7 +224,7 @@ func (s *CephMgrSuite) TestStatus() {
 	assert.Equal(s.T(), status, "Backend: rook\nAvailable: Yes")
 }
 
-func logBytesInfo(bytesSlice []byte) {
+func logBytesInfo(bytesSlice []byte){
 	logger.Infof("---- bytes slice info ---")
 	logger.Infof("bytes: %v\n", bytesSlice)
 	logger.Infof("length: %d\n", len(bytesSlice))
diff --git a/tests/integration/ceph_multi_cluster_test.go b/tests/integration/ceph_multi_cluster_test.go
index aa1676e26..9879b8c53 100644
--- a/tests/integration/ceph_multi_cluster_test.go
+++ b/tests/integration/ceph_multi_cluster_test.go
@@ -87,7 +87,7 @@ func (s *MultiClusterDeploySuite) SetupSuite() {
 		UseCSI:                    true,
 		MultipleMgrs:              true,
 		EnableAdmissionController: true,
-		RookVersion:               installer.LocalBuildTag,
+		RookVersion:               installer.VersionMaster,
 		CephVersion:               installer.NautilusVersion,
 	}
 	s.settings.ApplyEnvVars()
diff --git a/tests/integration/ceph_object_test.go b/tests/integration/ceph_object_test.go
index 5c8a1106a..f59eb9853 100644
--- a/tests/integration/ceph_object_test.go
+++ b/tests/integration/ceph_object_test.go
@@ -75,7 +75,7 @@ func (s *ObjectSuite) SetupSuite() {
 		UseCSI:                    true,
 		EnableAdmissionController: true,
 		UseCrashPruner:            true,
-		RookVersion:               installer.LocalBuildTag,
+		RookVersion:               installer.VersionMaster,
 		CephVersion:               installer.PacificVersion,
 	}
 	s.settings.ApplyEnvVars()
@@ -123,12 +123,7 @@ func runObjectE2ETest(helper *clients.TestClient, k8sh *utils.K8sHelper, s suite
 	}
 
 	logger.Infof("Running on Rook Cluster %s", namespace)
-	createCephObjectStore(s.T(), helper, k8sh, namespace, storeName, 3, tlsEnable)
-
-	// test that a second object store can be created (and deleted) while the first exists
-	testSecondObjectStore(s.T(), helper, k8sh, namespace, storeName+"-second", tlsEnable)
-
-	// now test operation of the first object store
+	createCephObjectStore(s, helper, k8sh, namespace, storeName, 3, tlsEnable)
 	testObjectStoreOperations(s, helper, k8sh, namespace, storeName)
 }
 
@@ -193,41 +188,53 @@ func checkCephObjectUser(
 	}
 }
 
-// create a CephObjectStore and wait for it to report ready status
-func createCephObjectStore(t *testing.T, helper *clients.TestClient, k8sh *utils.K8sHelper, namespace, storeName string, replicaSize int, tlsEnable bool) {
+func createCephObjectStore(s suite.Suite, helper *clients.TestClient, k8sh *utils.K8sHelper, namespace, storeName string, replicaSize int, tlsEnable bool) {
 	logger.Infof("Create Object Store %q with replica count %d", storeName, replicaSize)
 	rgwServiceName := "rook-ceph-rgw-" + storeName
 	if tlsEnable {
-		t.Run("generate TLS certs", func(t *testing.T) {
-			generateRgwTlsCertSecret(t, helper, k8sh, namespace, storeName, rgwServiceName)
-		})
+		generateRgwTlsCertSecret(s, helper, k8sh, namespace, storeName, rgwServiceName)
 	}
+	t := s.T()
 	t.Run("create CephObjectStore", func(t *testing.T) {
 		err := helper.ObjectClient.Create(namespace, storeName, 3, tlsEnable)
-		assert.Nil(t, err)
-	})
+		assert.Nil(s.T(), err)
 
-	t.Run("wait for RGWs to be running", func(t *testing.T) {
 		// check that ObjectStore is created
 		logger.Infof("Check that RGW pods are Running")
 		for i := 0; i < 24 && k8sh.CheckPodCountAndState("rook-ceph-rgw", namespace, 1, "Running") == false; i++ {
 			logger.Infof("(%d) RGW pod check sleeping for 5 seconds ...", i)
 			time.Sleep(5 * time.Second)
 		}
-		assert.True(t, k8sh.CheckPodCountAndState("rook-ceph-rgw", namespace, 1, "Running"))
+		assert.True(s.T(), k8sh.CheckPodCountAndState("rook-ceph-rgw", namespace, 1, "Running"))
 		logger.Info("RGW pods are running")
 		logger.Infof("Object store %q created successfully", storeName)
 	})
+}
 
+func testObjectStoreOperations(s suite.Suite, helper *clients.TestClient, k8sh *utils.K8sHelper, namespace, storeName string) {
 	ctx := context.TODO()
+	clusterInfo := client.AdminClusterInfo(namespace)
+	t := s.T()
+	t.Run(fmt.Sprintf("create CephObjectStoreUser %q", storeName), func(t *testing.T) {
+		createCephObjectUser(s, helper, k8sh, namespace, storeName, userid, true, true)
+		i := 0
+		for i = 0; i < 4; i++ {
+			if helper.ObjectUserClient.UserSecretExists(namespace, storeName, userid) {
+				break
+			}
+			logger.Info("waiting 5 more seconds for user secret to exist")
+			time.Sleep(5 * time.Second)
+		}
+		assert.NotEqual(t, 4, i)
+	})
 
 	// Check object store status
-	t.Run("verify object store status", func(t *testing.T) {
+	t.Run(fmt.Sprintf("verify ceph object store %q status", storeName), func(t *testing.T) {
 		retryCount := 30
 		i := 0
 		for i = 0; i < retryCount; i++ {
 			objectStore, err := k8sh.RookClientset.CephV1().CephObjectStores(namespace).Get(ctx, storeName, metav1.GetOptions{})
-			assert.Nil(t, err)
+			assert.Nil(s.T(), err)
 			if objectStore.Status == nil || objectStore.Status.BucketStatus == nil {
 				logger.Infof("(%d) object status check sleeping for 5 seconds ...%+v", i, objectStore.Status)
 				time.Sleep(5 * time.Second)
@@ -239,47 +246,16 @@ func createCephObjectStore(t *testing.T, helper *clients.TestClient, k8sh *utils
 				time.Sleep(5 * time.Second)
 				continue
 			}
-			assert.Equal(t, cephv1.ConditionConnected, objectStore.Status.BucketStatus.Health)
+			assert.Equal(s.T(), cephv1.ConditionConnected, objectStore.Status.BucketStatus.Health)
 			// Info field has the endpoint in it
-			assert.NotEmpty(t, objectStore.Status.Info)
-			assert.NotEmpty(t, objectStore.Status.Info["endpoint"])
+			assert.NotEmpty(s.T(), objectStore.Status.Info)
+			assert.NotEmpty(s.T(), objectStore.Status.Info["endpoint"])
 			break
 		}
 		if i == retryCount {
 			t.Fatal("bucket status check failed. status is not connected")
 		}
 	})
-}
-
-// test creating a second object store while an object store already exists, once the second store
-// is found to be ready, it will be deleted. "storeName" should be unique.
-func testSecondObjectStore(t *testing.T, helper *clients.TestClient, k8sh *utils.K8sHelper, namespace, storeName string, tlsEnable bool) {
-	t.Run("run a second object store", func(t *testing.T) {
-		createCephObjectStore(t, helper, k8sh, namespace, storeName, 1, tlsEnable)
-
-		t.Run("delete the second object store", func(t *testing.T) {
-			deleteObjectStore(t, k8sh, namespace, storeName)
-			assertObjectStoreDeletion(t, k8sh, namespace, storeName)
-		})
-	})
-}
-
-func testObjectStoreOperations(s suite.Suite, helper *clients.TestClient, k8sh *utils.K8sHelper, namespace, storeName string) {
-	ctx := context.TODO()
-	clusterInfo := client.AdminClusterInfo(namespace)
-	t := s.T()
-	t.Run(fmt.Sprintf("create CephObjectStoreUser %q", storeName), func(t *testing.T) {
-		createCephObjectUser(s, helper, k8sh, namespace, storeName, userid, true, true)
-		i := 0
-		for i = 0; i < 4; i++ {
-			if helper.ObjectUserClient.UserSecretExists(namespace, storeName, userid) {
-				break
-			}
-			logger.Info("waiting 5 more seconds for user secret to exist")
-			time.Sleep(5 * time.Second)
-		}
-		assert.NotEqual(t, 4, i)
-	})
 
 	context := k8sh.MakeContext()
 	objectStore, err := k8sh.RookClientset.CephV1().CephObjectStores(namespace).Get(ctx, storeName, metav1.GetOptions{})
@@ -321,9 +297,9 @@ func testObjectStoreOperations(s suite.Suite, helper *clients.TestClient, k8sh *
 		s3AccessKey, _ := helper.BucketClient.GetAccessKey(obcName)
 		s3SecretKey, _ := helper.BucketClient.GetSecretKey(obcName)
 		if objectStore.Spec.IsTLSEnabled() {
-			s3client, err = rgw.NewTestOnlyS3Agent(s3AccessKey, s3SecretKey, s3endpoint, region, true)
+			s3client, err = rgw.NewTestOnlyS3Agent(s3AccessKey, s3SecretKey, s3endpoint, true)
 		} else {
-			s3client, err = rgw.NewS3Agent(s3AccessKey, s3SecretKey, s3endpoint, region, true, nil)
+			s3client, err = rgw.NewS3Agent(s3AccessKey, s3SecretKey, s3endpoint, true, nil)
 		}
 
 		assert.Nil(s.T(), err)
@@ -384,7 +360,11 @@ func testObjectStoreOperations(s suite.Suite, helper *clients.TestClient, k8sh *
 	})
 
 	t.Run("delete CephObjectStore should be blocked by OBC bucket and CephObjectStoreUser", func(t *testing.T) {
-		deleteObjectStore(t, k8sh, namespace, storeName)
+		err := k8sh.DeleteResourceAndWait(false, "-n", namespace, "CephObjectStore", storeName)
+		assert.NoError(t, err)
+		// wait initially for the controller to detect deletion. Almost always enough, but not
+		// waiting will almost always fail the first check in the loop
+		time.Sleep(2 * time.Second)
 
 		store := &cephv1.CephObjectStore{}
 		i := 0
@@ -469,76 +449,58 @@ func testObjectStoreOperations(s suite.Suite, helper *clients.TestClient, k8sh *
 		// wait initially since it will almost never detect on the first try without this.
 		time.Sleep(3 * time.Second)
 
-		assertObjectStoreDeletion(t, k8sh, namespace, storeName)
-	})
-
-	// TODO : Add case for brownfield/cleanup s3 client}
-}
-
-func deleteObjectStore(t *testing.T, k8sh *utils.K8sHelper, namespace, storeName string) {
-	err := k8sh.DeleteResourceAndWait(false, "-n", namespace, "CephObjectStore", storeName)
-	assert.NoError(t, err)
-	// wait initially for the controller to detect deletion. Almost always enough, but not
-	// waiting immediately after this will almost always fail the first check in the loop
-	time.Sleep(4 * time.Second)
-}
+		store := &cephv1.CephObjectStore{}
+		i := 0
+		retry := 10
+		for i = 0; i < retry; i++ {
+			storeStr, err := k8sh.GetResource("-n", namespace, "CephObjectStore", storeName, "-o", "json")
+			assert.NoError(t, err)
+			logger.Infof("store: \n%s", storeStr)
 
-func assertObjectStoreDeletion(t *testing.T, k8sh *utils.K8sHelper, namespace, storeName string) {
-	store := &cephv1.CephObjectStore{}
-	i := 0
-	retry := 10
-	sleepTime := 3 * time.Second
-	for i = 0; i < retry; i++ {
-		storeStr, err := k8sh.GetResource("-n", namespace, "CephObjectStore", storeName, "-o", "json")
-		assert.NoError(t, err)
-		logger.Infof("store: \n%s", storeStr)
+			err = json.Unmarshal([]byte(storeStr), &store)
+			assert.NoError(t, err)
 
-		err = json.Unmarshal([]byte(storeStr), &store)
-		assert.NoError(t, err)
+			cond := cephv1.FindStatusCondition(store.Status.Conditions, cephv1.ConditionDeletionIsBlocked)
+			if cond.Status == v1.ConditionFalse && cond.Reason == cephv1.ObjectHasNoDependentsReason {
+				// Let's give some time to the object to be updated
+				time.Sleep(5 * time.Second)
+				break
+			}
+			logger.Info("waiting 3 more seconds for CephObjectStore to be unblocked by dependents")
+			time.Sleep(3 * time.Second)
+		}
+		assert.NotEqual(t, retry, i)
 
+		assert.Equal(t, cephv1.ConditionDeleting, store.Status.Phase) // phase == "Deleting"
+		// verify deletion is NOT blocked b/c object has dependents
 		cond := cephv1.FindStatusCondition(store.Status.Conditions, cephv1.ConditionDeletionIsBlocked)
-		if cond == nil {
-			logger.Infof("waiting for CephObjectStore %q to have a deletion condition", storeName)
-			time.Sleep(sleepTime)
-			continue
-		}
-		if cond.Status == v1.ConditionFalse && cond.Reason == cephv1.ObjectHasNoDependentsReason {
-			// no longer blocked by dependents
-			time.Sleep(5 * time.Second) // Let's give some time to the object to be updated
-			break
-		}
-		logger.Infof("waiting 3 more seconds for CephObjectStore %q to be unblocked by dependents", storeName)
-		time.Sleep(sleepTime)
-	}
-	assert.NotEqual(t, retry, i)
+		assert.Equal(t, v1.ConditionFalse, cond.Status)
+		assert.Equal(t, cephv1.ObjectHasNoDependentsReason, cond.Reason)
 
-	assert.Equal(t, cephv1.ConditionDeleting, store.Status.Phase) // phase == "Deleting"
-	// verify deletion is NOT blocked b/c object has dependents
-	cond := cephv1.FindStatusCondition(store.Status.Conditions, cephv1.ConditionDeletionIsBlocked)
-	assert.Equal(t, v1.ConditionFalse, cond.Status)
-	assert.Equal(t, cephv1.ObjectHasNoDependentsReason, cond.Reason)
+		err := k8sh.WaitUntilResourceIsDeleted("CephObjectStore", namespace, storeName)
+		assert.NoError(t, err)
+	})
 
-	err := k8sh.WaitUntilResourceIsDeleted("CephObjectStore", namespace, storeName)
-	assert.NoError(t, err)
+	// TODO : Add case for brownfield/cleanup s3 client}
 }
 
-func generateRgwTlsCertSecret(t *testing.T, helper *clients.TestClient, k8sh *utils.K8sHelper, namespace, storeName, rgwServiceName string) {
+func generateRgwTlsCertSecret(s suite.Suite, helper *clients.TestClient, k8sh *utils.K8sHelper, namespace, storeName, rgwServiceName string) {
 	ctx := context.TODO()
 	root, err := utils.FindRookRoot()
-	require.NoError(t, err, "failed to get rook root")
+	require.NoError(s.T(), err, "failed to get rook root")
 	tlscertdir, err := ioutil.TempDir(root, "tlscertdir")
-	require.NoError(t, err, "failed to create directory for TLS certs")
+	require.NoError(s.T(), err, "failed to create directory for TLS certs")
 	defer os.RemoveAll(tlscertdir)
 	cmdArgs := utils.CommandArgs{Command: filepath.Join(root, "tests/scripts/generate-tls-config.sh"),
 		CmdArgs: []string{tlscertdir, rgwServiceName, namespace}}
 	cmdOut := utils.ExecuteCommand(cmdArgs)
-	require.NoError(t, cmdOut.Err)
+	require.NoError(s.T(), cmdOut.Err)
 	tlsKeyIn, err := ioutil.ReadFile(filepath.Join(tlscertdir, rgwServiceName+".key"))
-	require.NoError(t, err)
+	require.NoError(s.T(), err)
 	tlsCertIn, err := ioutil.ReadFile(filepath.Join(tlscertdir, rgwServiceName+".crt"))
-	require.NoError(t, err)
+	require.NoError(s.T(), err)
 	tlsCaCertIn, err := ioutil.ReadFile(filepath.Join(tlscertdir, rgwServiceName+".ca"))
-	require.NoError(t, err)
+	require.NoError(s.T(), err)
 	secretCertOut := fmt.Sprintf("%s%s%s", tlsKeyIn, tlsCertIn, tlsCaCertIn)
 	tlsK8sSecret := &v1.Secret{
 		ObjectMeta: metav1.ObjectMeta{
@@ -550,5 +512,5 @@ func generateRgwTlsCertSecret(t *testing.T, helper *clients.TestClient, k8sh *ut
 		},
 	}
 	_, err = k8sh.Clientset.CoreV1().Secrets(namespace).Create(ctx, tlsK8sSecret, metav1.CreateOptions{})
-	require.Nil(t, err)
+	require.Nil(s.T(), err)
 }
diff --git a/tests/integration/ceph_smoke_test.go b/tests/integration/ceph_smoke_test.go
index ae91f4af9..6eb47fac8 100644
--- a/tests/integration/ceph_smoke_test.go
+++ b/tests/integration/ceph_smoke_test.go
@@ -99,7 +99,7 @@ func (s *SmokeSuite) SetupSuite() {
 		UseCSI:                    true,
 		EnableAdmissionController: true,
 		UseCrashPruner:            true,
-		RookVersion:               installer.LocalBuildTag,
+		RookVersion:               installer.VersionMaster,
 		CephVersion:               installer.PacificVersion,
 	}
 	s.settings.ApplyEnvVars()
diff --git a/tests/integration/ceph_upgrade_test.go b/tests/integration/ceph_upgrade_test.go
index 69aea4124..beb3aec14 100644
--- a/tests/integration/ceph_upgrade_test.go
+++ b/tests/integration/ceph_upgrade_test.go
@@ -183,7 +183,7 @@ func (s *UpgradeSuite) TestUpgradeToMaster() {
 	s.gatherLogs(s.settings.OperatorNamespace, "_before_master_upgrade")
 	s.upgradeToMaster()
 
-	s.verifyOperatorImage(installer.LocalBuildTag)
+	s.verifyOperatorImage(installer.VersionMaster)
 	s.verifyRookUpgrade(numOSDs)
 	err = s.installer.WaitForToolbox(s.namespace)
 	assert.NoError(s.T(), err)
@@ -359,15 +359,15 @@ func (s *UpgradeSuite) verifyFilesAfterUpgrade(fsName, newFileToWrite, messageFo
 // verify the upgrade but merely starts the upgrade process.
 func (s *UpgradeSuite) upgradeToMaster() {
 	// Apply the CRDs for the latest master
-	s.settings.RookVersion = installer.LocalBuildTag
+	s.settings.RookVersion = installer.VersionMaster
 	m := installer.NewCephManifests(s.settings)
 	require.NoError(s.T(), s.k8sh.ResourceOperation("apply", m.GetCRDs(s.k8sh)))
 
 	require.NoError(s.T(), s.k8sh.ResourceOperation("apply", m.GetCommon()))
 
 	require.NoError(s.T(),
-		s.k8sh.SetDeploymentVersion(s.settings.OperatorNamespace, operatorContainer, operatorContainer, installer.LocalBuildTag))
+		s.k8sh.SetDeploymentVersion(s.settings.OperatorNamespace, operatorContainer, operatorContainer, installer.VersionMaster))
 
 	require.NoError(s.T(),
-		s.k8sh.SetDeploymentVersion(s.settings.Namespace, "rook-ceph-tools", "rook-ceph-tools", installer.LocalBuildTag))
+		s.k8sh.SetDeploymentVersion(s.settings.Namespace, "rook-ceph-tools", "rook-ceph-tools", installer.VersionMaster))
 }
diff --git a/tests/scripts/github-action-helper.sh b/tests/scripts/github-action-helper.sh
index 0f0dcb598..a388f6e65 100755
--- a/tests/scripts/github-action-helper.sh
+++ b/tests/scripts/github-action-helper.sh
@@ -121,7 +121,7 @@ function build_rook() {
   tests/scripts/validate_modified_files.sh build
   docker images
   if [[ "$build_type" == "build" ]]; then
-    docker tag "$(docker images | awk '/build-/ {print $1}')" rook/ceph:local-build
+    docker tag "$(docker images | awk '/build-/ {print $1}')" rook/ceph:master
   fi
 }
 
@@ -250,15 +250,10 @@ function write_object_to_cluster1_read_from_cluster2() {
   BASE64_SECRET_KEY=$(kubectl -n rook-ceph get secrets realm-a-keys -o jsonpath="{.data.secret-key}")
   ACCESS_KEY=$(echo ${BASE64_ACCESS_KEY} | base64 --decode)
   SECRET_KEY=$(echo ${BASE64_SECRET_KEY} | base64 --decode)
-  s3cmd -v -d --config=s3cfg --access_key=${ACCESS_KEY} --secret_key=${SECRET_KEY} --host=${CLUSTER_1_IP_ADDR} mb s3://bkt
-  s3cmd -v -d --config=s3cfg --access_key=${ACCESS_KEY} --secret_key=${SECRET_KEY} --host=${CLUSTER_1_IP_ADDR} put ./1M.dat s3://bkt
+  s3cmd --config=s3cfg --access_key=${ACCESS_KEY} --secret_key=${SECRET_KEY} --host=${CLUSTER_1_IP_ADDR} mb s3://bkt
+  s3cmd --config=s3cfg --access_key=${ACCESS_KEY} --secret_key=${SECRET_KEY} --host=${CLUSTER_1_IP_ADDR} put ./1M.dat s3://bkt
   CLUSTER_2_IP_ADDR=$(kubectl -n rook-ceph-secondary get svc rook-ceph-rgw-zone-b-multisite-store -o jsonpath="{.spec.clusterIP}")
-  timeout 60 bash <<EOF
-until s3cmd -v -d --config=s3cfg --access_key=${ACCESS_KEY} --secret_key=${SECRET_KEY} --host=${CLUSTER_2_IP_ADDR} get s3://bkt/1M.dat 1M-get.dat --force; do
-  echo "waiting for object to be replicated"
-  sleep 5
-done
-EOF
+  s3cmd --config=s3cfg --access_key=${ACCESS_KEY} --secret_key=${SECRET_KEY} --host=${CLUSTER_2_IP_ADDR} get s3://bkt/1M.dat 1M-get.dat --force
   diff 1M.dat 1M-get.dat
 }