Repository: rook/rook
Pull request #9578 (2022-01-16 09:36:26)

rgw: gateway deployment strategy RollingUpdate on Pacific
<!-- Please take a look at our [Contributing](https://rook.io/docs/rook/latest/development-flow.html)
documentation before submitting a Pull Request!
Thank you for contributing to Rook! -->

**Description of your changes:**

When the strategy type of deployment is `Recreate`, it terminate at once as follows.

pods
```
NAME                     READY   STATUS        RESTARTS   AGE
echo-bc6fd5bf7-6hlld     1/1     Terminating   0          66d
echo-bc6fd5bf7-9sd2k     1/1     Terminating   0          66d
echo-bc6fd5bf7-bgxvb     1/1     Terminating   0          66d
echo-bc6fd5bf7-cqclp     1/1     Terminating   0          66d
echo-bc6fd5bf7-h4v7k     1/1     Terminating   0          66d
```

endpoints
```
NAME         ENDPOINTS                           AGE
echo         <none>                              689d
```

After that, when all pods are terminated, a new one is created at the same time.

pods
```
NAME                     READY   STATUS    RESTARTS   AGE
echo-97b48fcc6-98k4r     1/1     Running   0          89s
echo-97b48fcc6-hsk7b     1/1     Running   0          89s
echo-97b48fcc6-hwtdd     1/1     Running   0          89s
echo-97b48fcc6-wqbvj     1/1     Running   0          89s
echo-97b48fcc6-x85qq     1/1     Running   0          89s
```

In this case, from the moment the end is started until the new pod is normally running, there is no pod in the kubernetes service endpoints, so it goes to the service unavailable state.

Therefore, it is necessary to change it to `type: RollingUpdate` instead of `type: Recreate` so that rolling updates can be made without downtime.

Before Pacific, creating a deployment as much as the instance was okay, but after Pacific, the replicas was increased in one deployment, so rolling updates were set up one by one as in the previous version.

After changing the strategy, it can be seen that it is restarted normally one by one as follows.

pods
```
NAME                     READY   STATUS    RESTARTS   AGE
echo-85f97cf674-ghb8x    1/1     Running   0          2m13s
echo-85f97cf674-lqsw8    1/1     Running   0          88s
echo-85f97cf674-mcfxd    1/1     Running   0          49s
echo-85f97cf674-zqbcl    1/1     Running   0          3m32s
echo-85f97cf674-zvpsn    1/1     Running   0          2m52s
```

**Which issue is resolved by this Pull Request:**
Resolves #

**Checklist:**

- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/latest/development-flow.html#commit-structure).
- [ ] **Skip Tests for Docs**: Add the flag for skipping the build if this is only a documentation change. See [here](https://github.com/rook/rook/blob/master/INSTALL.md#skip-ci) for the flag.
- [ ] **Skip Unrelated Tests**: Add a flag to run tests for a specific storage provider. See [test options](https://github.com/rook/rook/blob/master/INSTALL.md#test-storage-provider).
- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/latest/development-flow.html#submitting-a-pull-request)
- [ ] Documentation has been updated, if necessary.
- [ ] Unit tests have been added, if necessary.
- [ ] Integration tests have been added, if necessary.
- [ ] Pending release notes updated with breaking and/or notable changes, if necessary.
- [ ] Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
- [ ] Code generation (`make codegen`) has been run to update object specifications, if necessary.


leseb: @micalgenus please update your git commit message and we should be good to go. Thanks!

Patch:
diff --git a/.github/workflows/daily-nightly-jobs.yml b/.github/workflows/daily-nightly-jobs.yml
index 7c7df6e28..7c7abb156 100644
--- a/.github/workflows/daily-nightly-jobs.yml
+++ b/.github/workflows/daily-nightly-jobs.yml
@@ -326,7 +326,6 @@ jobs:
 
     - name: run Encryption KMS IBM Key Protect
       uses: ./.github/workflows/encryption-pvc-kms-ibm-kp
-      if: "env.IBM_KP_INSTANCE_ID != '' && env.IBM_KP_SERVICE_API_KEY != ''"
       with:
         github-token: ${{ secrets.GITHUB_TOKEN }}
         ibm-instance-id: ${{ secrets.IBM_KP_INSTANCE_ID }}
diff --git a/Documentation/ceph-cluster-crd.md b/Documentation/ceph-cluster-crd.md
index 00dc85f01..f72625754 100755
--- a/Documentation/ceph-cluster-crd.md
+++ b/Documentation/ceph-cluster-crd.md
@@ -516,15 +516,13 @@ Annotations and Labels can be specified so that the Rook components will have th
 
 You can set annotations / labels for Rook components for the list of key value pairs:
 
-* `all`: Set annotations / labels for all components except `clusterMetadata`.
+* `all`: Set annotations / labels for all components
 * `mgr`: Set annotations / labels for MGRs
 * `mon`: Set annotations / labels for mons
 * `osd`: Set annotations / labels for OSDs
 * `prepareosd`: Set annotations / labels for OSD Prepare Jobs
 * `monitoring`: Set annotations / labels for service monitor
 * `crashcollector`: Set annotations / labels for crash collectors
-* `clusterMetadata`: Set annotations  only to `rook-ceph-mon-endpoints` configmap and the  `rook-ceph-mon` and `rook-ceph-admin-keyring` secrets. These annotations will not be merged with the `all` annotations. The common usage is for backing up these critical resources with `kubed`.
-Note the clusterMetadata annotation will not be merged with the `all` annotation.
 When other keys are set, `all` will be merged together with the specific component.
 
 ### Placement Configuration Settings
@@ -603,11 +601,10 @@ Priority class names can be specified so that the Rook components will have thos
 
 You can set priority class names for Rook components for the list of key value pairs:
 
-* `all`: Set priority class names for MGRs, Mons, OSDs, and crashcollectors.
+* `all`: Set priority class names for MGRs, Mons, OSDs.
 * `mgr`: Set priority class names for MGRs.
 * `mon`: Set priority class names for Mons.
 * `osd`: Set priority class names for OSDs.
-* `crashcollector`: Set priority class names for crashcollectors.
 
 The specific component keys will act as overrides to `all`.
 
diff --git a/Documentation/ceph-monitoring.md b/Documentation/ceph-monitoring.md
index 579ad4c25..97b553ec2 100644
--- a/Documentation/ceph-monitoring.md
+++ b/Documentation/ceph-monitoring.md
@@ -213,11 +213,11 @@ spec:
 
 ### Horizontal Pod Scaling using Kubernetes Event-driven Autoscaling (KEDA)
 
-Using metrics exported from the Prometheus service, the horizontal pod scaling can use the custom metrics other than CPU and memory consumption. It can be done with help of Prometheus Scaler provided by the [KEDA](https://keda.sh/docs/2.5/scalers/prometheus/). See the [KEDA deployment guide](https://keda.sh/docs/2.5/deploy/) for details.
+Using metrics exported from the Prometheus service, the horizontal pod scaling can use the custom metrics other than CPU and memory consumption. It can be done with help of Prometheus Scaler provided by the [KEDA](https://keda.sh/docs/2.4/scalers/prometheus/). See the [KEDA deployment guide](https://keda.sh/docs/2.4/deploy/) for details.
 
 Following is an example to autoscale RGW:
 ```YAML
-apiVersion: keda.sh/v1alpha1
+apiVersion: keda.k8s.io/v1alpha1
 kind: ScaledObject
 metadata:
  name: rgw-scale
@@ -225,7 +225,7 @@ metadata:
 spec:
  scaleTargetRef:
    kind: Deployment
-   name: rook-ceph-rgw-my-store-a # deployment for the autoscaling
+   deploymentName: rook-ceph-rgw-my-store-a # deployment for the autoscaling
  minReplicaCount: 1
  maxReplicaCount: 5
  triggers:
diff --git a/Documentation/ceph-upgrade.md b/Documentation/ceph-upgrade.md
index feb8ed786..cb903ea20 100644
--- a/Documentation/ceph-upgrade.md
+++ b/Documentation/ceph-upgrade.md
@@ -545,14 +545,13 @@ kubectl -n $ROOK_OPERATOR_NAMESPACE edit configmap rook-ceph-operator-config
 The default upstream images are included below, which you can change to your desired images.
 
 ```yaml
-ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.5.0"
-ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0"
-ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0"
-ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.4.0"
+ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.4.0"
+ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
+ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
+ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.3.0"
 ROOK_CSI_RESIZER_IMAGE: "k8s.gcr.io/sig-storage/csi-resizer:v1.3.0"
 ROOK_CSI_SNAPSHOTTER_IMAGE: "k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.0"
 CSI_VOLUME_REPLICATION_IMAGE: "quay.io/csiaddons/volumereplication-operator:v0.1.0"
-ROOK_CSIADDONS_IMAGE: "quay.io/csiaddons/k8s-sidecar:v0.1.0"
 ```
 
 ### **Use default images**
@@ -571,12 +570,11 @@ kubectl --namespace rook-ceph get pod -o jsonpath='{range .items[*]}{range .spec
 ```
 
 ```
-k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
-k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0
-k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
+k8s.gcr.io/sig-storage/csi-attacher:v3.3.0
+k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
+k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
 k8s.gcr.io/sig-storage/csi-resizer:v1.3.0
 k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.0
-quay.io/cephcsi/cephcsi:v3.5.0
+quay.io/cephcsi/cephcsi:v3.4.0
 quay.io/csiaddons/volumereplication-operator:v0.1.0
-quay.io/csiaddons/k8s-sidecar:v0.1.0
 ```
diff --git a/Documentation/helm-operator.md b/Documentation/helm-operator.md
index 919c5cfc3..644e327de 100644
--- a/Documentation/helm-operator.md
+++ b/Documentation/helm-operator.md
@@ -130,24 +130,21 @@ The following tables lists the configurable parameters of the rook-operator char
 | `csi.cephfsGrpcMetricsPort`         | CSI CephFS driver GRPC metrics port.                                                                                        | `9091`                                                    |
 | `csi.cephfsLivenessMetricsPort`     | CSI CephFS driver metrics port.                                                                                             | `9081`                                                    |
 | `csi.rbdGrpcMetricsPort`            | Ceph CSI RBD driver GRPC metrics port.                                                                                      | `9090`                                                    |
-| `csi.csiAddonsPort`            | CSI Addons server port.                                                                                      | `9070`                                                    |
 | `csi.rbdLivenessMetricsPort`        | Ceph CSI RBD driver metrics port.                                                                                           | `8080`                                                    |
 | `csi.forceCephFSKernelClient`       | Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs.                                                | `true`                                                    |
 | `csi.kubeletDirPath`                | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag)                                | `/var/lib/kubelet`                                        |
-| `csi.cephcsi.image`                 | Ceph CSI image.                                                                                                             | `quay.io/cephcsi/cephcsi:v3.5.0`                          |
+| `csi.cephcsi.image`                 | Ceph CSI image.                                                                                                             | `quay.io/cephcsi/cephcsi:v3.4.0`                          |
 | `csi.rbdPluginUpdateStrategy`       | CSI Rbd plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.                                  | `OnDelete`                                                |
 | `csi.cephFSPluginUpdateStrategy`    | CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.                               | `OnDelete`                                                |
-| `csi.registrar.image`               | Kubernetes CSI registrar image.                                                                                             | `k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0` |
+| `csi.registrar.image`               | Kubernetes CSI registrar image.                                                                                             | `k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0` |
 | `csi.resizer.image`                 | Kubernetes CSI resizer image.                                                                                               | `k8s.gcr.io/sig-storage/csi-resizer:v1.3.0`               |
-| `csi.provisioner.image`             | Kubernetes CSI provisioner image.                                                                                           | `k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0`           |
+| `csi.provisioner.image`             | Kubernetes CSI provisioner image.                                                                                           | `k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0`           |
 | `csi.snapshotter.image`             | Kubernetes CSI snapshotter image.                                                                                           | `k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.0`           |
-| `csi.attacher.image`                | Kubernetes CSI Attacher image.                                                                                              | `k8s.gcr.io/sig-storage/csi-attacher:v3.4.0`              |
+| `csi.attacher.image`                | Kubernetes CSI Attacher image.                                                                                              | `k8s.gcr.io/sig-storage/csi-attacher:v3.3.0`              |
 | `csi.cephfsPodLabels`               | Labels to add to the CSI CephFS Pods.                                                                                       | <none>                                                    |
 | `csi.rbdPodLabels`                  | Labels to add to the CSI RBD Pods.                                                                                          | <none>                                                    |
 | `csi.volumeReplication.enabled`     | Enable Volume Replication.                                                                                                  | `false`                                                   |
 | `csi.volumeReplication.image`       | Volume Replication Controller image.                                                                                        | `quay.io/csiaddons/volumereplication-operator:v0.1.0`     |
-| `csi.csiAddons.enabled`     | Enable CSIAddons                                                                                                  | `false`                                                   |
-| `csi.csiAddons.image`       | CSIAddons Sidecar image.                                                                                        | `quay.io/csiaddons/k8s-sidecar:v0.1.0`     |
 | `admissionController.tolerations`   | Array of tolerations in YAML format which will be added to admission controller deployment.                                 | <none>                                                    |
 | `admissionController.nodeAffinity`  | The node labels for affinity of the admission controller deployment (***)                                                   | <none>                                                    |
 | `allowMultipleFilesystems`          | **(experimental in  Octopus (v15))** Allows multiple filesystems to be deployed to a Ceph cluster.                          | `false`                                                   |
diff --git a/Makefile b/Makefile
index 1487ade6b..4ece242dc 100644
--- a/Makefile
+++ b/Makefile
@@ -12,14 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Linux doesn't guarantee file ordering, so sort the files to make sure order is deterministic.
-# And in order to handle file paths with spaces, it's easiest to read the file names into an array.
-# Set locale `LC_ALL=C` because different OSes have different sort behavior;
-# `C` sorting order is based on the byte values,
-# Reference: https://blog.zhimingwang.org/macos-lc_collate-hunt
-LC_ALL=C
-export LC_ALL
-
 include build/makelib/common.mk
 include build/makelib/helm.mk
 
diff --git a/build/crds/build-crds.sh b/build/crds/build-crds.sh
index 06c652a8c..1f40effe2 100755
--- a/build/crds/build-crds.sh
+++ b/build/crds/build-crds.sh
@@ -97,6 +97,13 @@ fi
 
 generating_main_crd
 
+# Linux doesn't guarantee file ordering, so we must sort the output to make sure it's deterministic.
+# In order to handle file paths with spaces in the yq command below, it's easiest to read the
+# file names into an array.
+# Set locale `LC_ALL=C` because different OSes have different sort behavior;
+# `C` sorting order is based on the byte values,
+# Reference: https://blog.zhimingwang.org/macos-lc_collate-hunt
+LC_ALL=C
 CRD_FILES=()
 while read -r line; do
   CRD_FILES+=("$line")
diff --git a/build/rbac/keep-rbac-yaml.sh b/build/rbac/keep-rbac-yaml.sh
index 098df8364..b235846c3 100755
--- a/build/rbac/keep-rbac-yaml.sh
+++ b/build/rbac/keep-rbac-yaml.sh
@@ -35,6 +35,12 @@ sed '/^# Source: /d' | # helm adds '# Source: <file>' comments atop of each yaml
 $YQ eval --split-exp '.kind + " " + .metadata.name + " "' - # split into files by <kind> <name> .yaml
 # outputting the filenames with spaces after kind and name keeps the same sorting from before
 
+# Linux doesn't guarantee file ordering, so sort the files to make sure order is deterministic.
+# And in order to handle file paths with spaces, it's easiest to read the file names into an array.
+# Set locale `LC_ALL=C` because different OSes have different sort behavior;
+# `C` sorting order is based on the byte values,
+# Reference: https://blog.zhimingwang.org/macos-lc_collate-hunt
+LC_ALL=C
 RBAC_FILES=()
 while read -r line; do
   RBAC_FILES+=("$line")
diff --git a/build/rbac/rbac.yaml b/build/rbac/rbac.yaml
index 8e372e4bb..ab502c7fe 100644
--- a/build/rbac/rbac.yaml
+++ b/build/rbac/rbac.yaml
@@ -757,9 +757,6 @@ spec:
     # Ceph mgr Prometheus Metrics
     - min: 9283
       max: 9283
-    # port for CSIAddons
-    - min: 9070
-      max: 9070
 ---
 kind: Role
 apiVersion: rbac.authorization.k8s.io/v1
diff --git a/deploy/charts/rook-ceph-cluster/templates/volumesnapshotclass.yaml b/deploy/charts/rook-ceph-cluster/templates/volumesnapshotclass.yaml
deleted file mode 100644
index 704eb43f9..000000000
--- a/deploy/charts/rook-ceph-cluster/templates/volumesnapshotclass.yaml
+++ /dev/null
@@ -1,48 +0,0 @@
-{{- $filesystemvsc := .Values.cephFileSystemVolumeSnapshotClass -}}
-{{- $blockpoolvsc := .Values.cephBlockPoolsVolumeSnapshotClass -}}
-
----
-{{- if default false $filesystemvsc.enabled }}
-{{- if .Capabilities.APIVersions.Has "snapshot.storage.k8s.io/v1" }}
-apiVersion: snapshot.storage.k8s.io/v1
-{{- else  }}
-apiVersion: snapshot.storage.k8s.io/v1beta1
-{{- end }}
-kind: VolumeSnapshotClass
-metadata:
-  name: {{ $filesystemvsc.name }}
-  annotations:
-    snapshot.storage.kubernetes.io/is-default-class: "{{ if default false $filesystemvsc.isDefault }}true{{ else }}false{{ end }}"
-driver: {{ .Values.operatorNamespace }}.cephfs.csi.ceph.com
-parameters:
-  clusterID: {{ .Release.Namespace }}
-  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-cephfs-provisioner
-  csi.storage.k8s.io/snapshotter-secret-namespace: {{ .Release.Namespace }}
-{{- if $filesystemvsc.parameters }}
-{{ toYaml $filesystemvsc.parameters | indent 2 }}
-{{- end }}
-deletionPolicy: {{ default "Delete" $filesystemvsc.deletionPolicy }}
-{{- end }}
-
----
-{{- if default false $blockpoolvsc.enabled }}
-{{- if .Capabilities.APIVersions.Has "snapshot.storage.k8s.io/v1" }}
-apiVersion: snapshot.storage.k8s.io/v1
-{{- else  }}
-apiVersion: snapshot.storage.k8s.io/v1beta1
-{{- end }}
-kind: VolumeSnapshotClass
-metadata:
-  name: {{ $blockpoolvsc.name }}
-  annotations:
-    snapshot.storage.kubernetes.io/is-default-class: "{{ if default false $blockpoolvsc.isDefault }}true{{ else }}false{{ end }}"
-driver: {{ .Values.operatorNamespace }}.rbd.csi.ceph.com
-parameters:
-  clusterID: {{ .Release.Namespace }}
-  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner
-  csi.storage.k8s.io/snapshotter-secret-namespace: {{ .Release.Namespace }}
-{{- if $blockpoolvsc.parameters }}
-{{ toYaml $blockpoolvsc.parameters | indent 2 }}
-{{- end }}
-deletionPolicy: {{ default "Delete" $blockpoolvsc.deletionPolicy }}
-{{- end }}
diff --git a/deploy/charts/rook-ceph-cluster/values.yaml b/deploy/charts/rook-ceph-cluster/values.yaml
index 68c27158b..1ca0664ec 100644
--- a/deploy/charts/rook-ceph-cluster/values.yaml
+++ b/deploy/charts/rook-ceph-cluster/values.yaml
@@ -238,7 +238,6 @@ cephClusterSpec:
   #   mon: rook-ceph-mon-priority-class
   #   osd: rook-ceph-osd-priority-class
   #   mgr: rook-ceph-mgr-priority-class
-  #   crashcollector: rook-ceph-crashcollector-priority-class
 
   storage: # cluster level storage configuration and selection
     useAllNodes: true
@@ -398,22 +397,6 @@ cephFileSystems:
         # in hyperconverged settings where the volume is mounted on the same node as the osds.
         csi.storage.k8s.io/fstype: ext4
 
-cephFileSystemVolumeSnapshotClass:
-  enabled: false
-  name: ceph-filesystem
-  isDefault: true
-  deletionPolicy: Delete
-  # see https://rook.io/docs/rook/latest/ceph-csi-snapshot.html#cephfs-snapshots for available configuration
-  parameters: {}
-
-cephBlockPoolsVolumeSnapshotClass:
-  enabled: false
-  name: ceph-block
-  isDefault: false
-  deletionPolicy: Delete
-  # see https://rook.io/docs/rook/latest/ceph-csi-snapshot.html#rbd-snapshots for available configuration
-  parameters: {}
-
 cephObjectStores:
   - name: ceph-objectstore
     # see https://github.com/rook/rook/blob/master/Documentation/ceph-object-store-crd.md#object-store-settings for available configuration
diff --git a/deploy/charts/rook-ceph/templates/deployment.yaml b/deploy/charts/rook-ceph/templates/deployment.yaml
index 446e59b24..ae639f883 100644
--- a/deploy/charts/rook-ceph/templates/deployment.yaml
+++ b/deploy/charts/rook-ceph/templates/deployment.yaml
@@ -91,8 +91,6 @@ spec:
           value: {{ .Values.csi.enableOMAPGenerator | quote }}
         - name: CSI_ENABLE_VOLUME_REPLICATION
           value: {{ .Values.csi.volumeReplication.enabled | quote }}
-        - name: CSI_ENABLE_CSIADDONS
-          value: {{ .Values.csi.csiAddons.enabled | quote }}
         - name: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT
           value: {{ .Values.csi.enablePluginSelinuxHostMount | quote }}
 {{- if .Values.csi.enableCSIHostNetwork }}
@@ -163,16 +161,6 @@ spec:
           value: {{ .Values.csi.volumeReplication.image | quote }}
 {{- end }}
 {{- end }}
-{{- if .Values.csi.csiAddons }}
-{{- if .Values.csi.csiAddons.image }}
-        - name: ROOK_CSIADDONS_IMAGE
-          value: {{ .Values.csi.csiAddons.image | quote }}
-{{- end }}
-{{- if .Values.csi.csiAddonsPort }}
-        - name: CSIADDONS_PORT
-          value: {{ .Values.csi.csiAddonsPort | quote }}
-{{- end }}
-{{- end }}
 {{- if .Values.csi.cephfsPodLabels }}
         - name: ROOK_CSI_CEPHFS_POD_LABELS
           value: {{ .Values.csi.cephfsPodLabels | quote }}
diff --git a/deploy/charts/rook-ceph/templates/psp.yaml b/deploy/charts/rook-ceph/templates/psp.yaml
index 23c5a415b..f0924be50 100644
--- a/deploy/charts/rook-ceph/templates/psp.yaml
+++ b/deploy/charts/rook-ceph/templates/psp.yaml
@@ -76,9 +76,6 @@ spec:
     # Ceph mgr Prometheus Metrics
     - min: 9283
       max: 9283
-    # port for CSIAddons
-    - min: 9070
-      max: 9070
 {{- if .Values.rbacEnable }}
 ---
 apiVersion: rbac.authorization.k8s.io/v1
diff --git a/deploy/charts/rook-ceph/templates/role.yaml b/deploy/charts/rook-ceph/templates/role.yaml
index 7182e71b1..9fa89d843 100644
--- a/deploy/charts/rook-ceph/templates/role.yaml
+++ b/deploy/charts/rook-ceph/templates/role.yaml
@@ -61,18 +61,6 @@ rules:
     resources: ["leases"]
     verbs: ["get", "watch", "list", "delete", "update", "create"]
 ---
-{{- if .Values.csi.csiAddons }}
-kind: Role
-apiVersion: rbac.authorization.k8s.io/v1
-metadata:
-  name: rbd-csi-nodeplugin
-  namespace: {{ .Release.Namespace }} # namespace:operator
-rules:
-  - apiGroups: ["csiaddons.openshift.io"]
-    resources: ["csiaddonsnodes"]
-    verbs: ["create"]
----
-{{- end }}
 kind: Role
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
@@ -88,9 +76,4 @@ rules:
   - apiGroups: ["coordination.k8s.io"]
     resources: ["leases"]
     verbs: ["get", "watch", "list", "delete", "update", "create"]
-  {{- if .Values.csi.csiAddons }}
-  - apiGroups: ["csiaddons.openshift.io"]
-    resources: ["csiaddonsnodes"]
-    verbs: ["create"]
-  {{- end }}
 {{- end }}
diff --git a/deploy/charts/rook-ceph/templates/rolebinding.yaml b/deploy/charts/rook-ceph/templates/rolebinding.yaml
index 5be43f9ca..4025df5b6 100644
--- a/deploy/charts/rook-ceph/templates/rolebinding.yaml
+++ b/deploy/charts/rook-ceph/templates/rolebinding.yaml
@@ -32,22 +32,6 @@ roleRef:
   name: cephfs-external-provisioner-cfg
   apiGroup: rbac.authorization.k8s.io
 ---
-{{- if .Values.csi.csiAddons }}
-kind: RoleBinding
-apiVersion: rbac.authorization.k8s.io/v1
-metadata:
-  name: rbd-csi-nodeplugin-role-cfg
-  namespace: {{ .Release.Namespace }} # namespace:operator
-subjects:
-  - kind: ServiceAccount
-    name: rook-csi-rbd-plugin-sa
-    namespace: {{ .Release.Namespace }} # namespace:operator
-roleRef:
-  kind: Role
-  name: rbd-csi-nodeplugin
-  apiGroup: rbac.authorization.k8s.io
----
-{{- end -}}
 kind: RoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
diff --git a/deploy/charts/rook-ceph/values.yaml b/deploy/charts/rook-ceph/values.yaml
index 8a869b761..80d42d088 100644
--- a/deploy/charts/rook-ceph/values.yaml
+++ b/deploy/charts/rook-ceph/values.yaml
@@ -268,7 +268,6 @@ csi:
   #cephfsGrpcMetricsPort: 9091
   #cephfsLivenessMetricsPort: 9081
   #rbdGrpcMetricsPort: 9090
-  #csiAddonsPort: 9070
   # Enable Ceph Kernel clients on kernel < 4.17. If your kernel does not support quotas for CephFS
   # you may want to disable this setting. However, this will cause an issue during upgrades
   # with the FUSE client. See the upgrade guide: https://rook.io/docs/rook/v1.2/ceph-upgrade.html
@@ -276,15 +275,15 @@ csi:
   #rbdLivenessMetricsPort: 9080
   #kubeletDirPath: /var/lib/kubelet
   #cephcsi:
-    #image: quay.io/cephcsi/cephcsi:v3.5.0
+    #image: quay.io/cephcsi/cephcsi:v3.4.0
   #registrar:
-    #image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0
+    #image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
   #provisioner:
-    #image: k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
+    #image: k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
   #snapshotter:
     #image: k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.0
   #attacher:
-    #image: k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
+    #image: k8s.gcr.io/sig-storage/csi-attacher:v3.3.0
   #resizer:
     #image: k8s.gcr.io/sig-storage/csi-resizer:v1.3.0
   # Labels to add to the CSI CephFS Deployments and DaemonSets Pods.
@@ -297,10 +296,7 @@ csi:
   volumeReplication:
     enabled: false
     #image: "quay.io/csiaddons/volumereplication-operator:v0.1.0"
-  # Enable the CSIAddons sidecar.
-  csiAddons:
-    enabled: false
-    #image: "quay.io/csiaddons/k8s-sidecar:v0.1.0"
+
 enableDiscoveryDaemon: false
 cephCommandsTimeoutSeconds: "15"
 
diff --git a/deploy/examples/cluster.yaml b/deploy/examples/cluster.yaml
index 619c8aaaf..365908df3 100644
--- a/deploy/examples/cluster.yaml
+++ b/deploy/examples/cluster.yaml
@@ -166,10 +166,6 @@ spec:
 #    osd:
 #    cleanup:
 #    prepareosd:
-# clusterMetadata annotations will be applied to only `rook-ceph-mon-endpoints` configmap and the `rook-ceph-mon` and `rook-ceph-admin-keyring` secrets.
-# And clusterMetadata annotations will not be merged with `all` annotations.
-#    clusterMetadata:
-#       kubed.appscode.com/sync: "true"
 # If no mgr annotations are set, prometheus scrape annotations will be set by default.
 #    mgr:
   labels:
@@ -211,7 +207,6 @@ spec:
 #    mon: rook-ceph-mon-priority-class
 #    osd: rook-ceph-osd-priority-class
 #    mgr: rook-ceph-mgr-priority-class
-#    crashcollector: rook-ceph-crashcollector-priority-class
   storage: # cluster level storage configuration and selection
     useAllNodes: true
     useAllDevices: true
diff --git a/deploy/examples/common.yaml b/deploy/examples/common.yaml
index 254a03b2e..bd7c91767 100644
--- a/deploy/examples/common.yaml
+++ b/deploy/examples/common.yaml
@@ -841,9 +841,6 @@ spec:
     # Ceph mgr Prometheus Metrics
     - min: 9283
       max: 9283
-    # port for CSIAddons
-    - min: 9070
-      max: 9070
 ---
 kind: Role
 apiVersion: rbac.authorization.k8s.io/v1
@@ -863,16 +860,6 @@ rules:
 ---
 kind: Role
 apiVersion: rbac.authorization.k8s.io/v1
-metadata:
-  name: rbd-csi-nodeplugin
-  namespace: rook-ceph # namespace:operator
-rules:
-  - apiGroups: ["csiaddons.openshift.io"]
-    resources: ["csiaddonsnodes"]
-    verbs: ["create"]
----
-kind: Role
-apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: rbd-external-provisioner-cfg
   namespace: rook-ceph # namespace:operator
@@ -886,9 +873,6 @@ rules:
   - apiGroups: ["coordination.k8s.io"]
     resources: ["leases"]
     verbs: ["get", "watch", "list", "delete", "update", "create"]
-  - apiGroups: ["csiaddons.openshift.io"]
-    resources: ["csiaddonsnodes"]
-    verbs: ["create"]
 ---
 kind: Role
 apiVersion: rbac.authorization.k8s.io/v1
@@ -1061,20 +1045,6 @@ roleRef:
 ---
 kind: RoleBinding
 apiVersion: rbac.authorization.k8s.io/v1
-metadata:
-  name: rbd-csi-nodeplugin-role-cfg
-  namespace: rook-ceph # namespace:operator
-subjects:
-  - kind: ServiceAccount
-    name: rook-csi-rbd-plugin-sa
-    namespace: rook-ceph # namespace:operator
-roleRef:
-  kind: Role
-  name: rbd-csi-nodeplugin
-  apiGroup: rbac.authorization.k8s.io
----
-kind: RoleBinding
-apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: rbd-csi-provisioner-role-cfg
   namespace: rook-ceph # namespace:operator
diff --git a/deploy/examples/create-external-cluster-resources.py b/deploy/examples/create-external-cluster-resources.py
index 44a8e78d9..688d9e5f2 100644
--- a/deploy/examples/create-external-cluster-resources.py
+++ b/deploy/examples/create-external-cluster-resources.py
@@ -93,12 +93,12 @@ class DummyRados(object):
         self.cmd_output_map[self.cmd_names['mgr services']
                             ] = '''{"dashboard":"https://ceph-dashboard:8443/","prometheus":"http://ceph-dashboard-db:9283/"}'''
         self.cmd_output_map['''{"caps": ["mon", "allow r, allow command quorum_status", "osd", "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow x pool=default.rgw.buckets.index"], "entity": "client.healthchecker", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.healthchecker","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon":"allow r, allow command quorum_status","osd":"allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow x pool=default.rgw.buckets.index"}}]'''
-        self.cmd_output_map['''{"caps": ["mon", "profile rbd, allow command 'osd blocklist'", "osd", "profile rbd"], "entity": "client.csi-rbd-node", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-rbd-node","key":"AQBOgrNeHbK1AxAAubYBeV8S1U/GPzq5SVeq6g==","caps":{"mon":"profile rbd, allow command 'osd blocklist'","osd":"profile rbd"}}]'''
-        self.cmd_output_map['''{"caps": ["mon", "profile rbd, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "profile rbd"], "entity": "client.csi-rbd-provisioner", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-rbd-provisioner","key":"AQBNgrNe1geyKxAA8ekViRdE+hss5OweYBkwNg==","caps":{"mgr":"allow rw","mon":"profile rbd, allow command 'osd blocklist'","osd":"profile rbd"}}]'''
-        self.cmd_output_map['''{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs *=*", "mds", "allow rw"], "entity": "client.csi-cephfs-node", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-node","key":"AQBOgrNeENunKxAAPCmgE7R6G8DcXnaJ1F32qg==","caps":{"mds":"allow rw","mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs *=*"}}]'''
-        self.cmd_output_map['''{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=*"], "entity": "client.csi-cephfs-provisioner", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-provisioner","key":"AQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs metadata=*"}}]'''
-        self.cmd_output_map['''{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=*"], "entity": "client.csi-cephfs-provisioner-openshift-storage", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-provisioner-openshift-storage","key":"BQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs metadata=*"}}]'''
-        self.cmd_output_map['''{"caps": ["mon", "allow r, allow command 'osd blocklist'", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=myfs"], "entity": "client.csi-cephfs-provisioner-openshift-storage-myfs", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-provisioner-openshift-storage-myfs","key":"CQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r, allow command 'osd blocklist'","osd":"allow rw tag cephfs metadata=myfs"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "profile rbd", "osd", "profile rbd"], "entity": "client.csi-rbd-node", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-rbd-node","key":"AQBOgrNeHbK1AxAAubYBeV8S1U/GPzq5SVeq6g==","caps":{"mon":"profile rbd","osd":"profile rbd"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "profile rbd", "mgr", "allow rw", "osd", "profile rbd"], "entity": "client.csi-rbd-provisioner", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-rbd-provisioner","key":"AQBNgrNe1geyKxAA8ekViRdE+hss5OweYBkwNg==","caps":{"mgr":"allow rw","mon":"profile rbd","osd":"profile rbd"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "allow r", "mgr", "allow rw", "osd", "allow rw tag cephfs *=*", "mds", "allow rw"], "entity": "client.csi-cephfs-node", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-node","key":"AQBOgrNeENunKxAAPCmgE7R6G8DcXnaJ1F32qg==","caps":{"mds":"allow rw","mgr":"allow rw","mon":"allow r","osd":"allow rw tag cephfs *=*"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "allow r", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=*"], "entity": "client.csi-cephfs-provisioner", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-provisioner","key":"AQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r","osd":"allow rw tag cephfs metadata=*"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "allow r", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=*"], "entity": "client.csi-cephfs-provisioner-openshift-storage", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-provisioner-openshift-storage","key":"BQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r","osd":"allow rw tag cephfs metadata=*"}}]'''
+        self.cmd_output_map['''{"caps": ["mon", "allow r", "mgr", "allow rw", "osd", "allow rw tag cephfs metadata=myfs"], "entity": "client.csi-cephfs-provisioner-openshift-storage-myfs", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.csi-cephfs-provisioner-openshift-storage-myfs","key":"CQBOgrNeAFgcGBAAvGqKOAD0D3xxmVY0R912dg==","caps":{"mgr":"allow rw","mon":"allow r","osd":"allow rw tag cephfs metadata=myfs"}}]'''
         self.cmd_output_map['''{"caps": ["mon", "allow r, allow command quorum_status, allow command version", "mgr", "allow command config", "osd", "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index"], "entity": "client.healthchecker", "format": "json", "prefix": "auth get-or-create"}'''] = '''[{"entity":"client.healthchecker","key":"AQDFkbNeft5bFRAATndLNUSEKruozxiZi3lrdA==","caps":{"mon": "allow r, allow command quorum_status, allow command version", "mgr": "allow command config", "osd": "allow rwx pool=default.rgw.meta, allow r pool=.rgw.root, allow rw pool=default.rgw.control, allow rx pool=default.rgw.log, allow x pool=default.rgw.buckets.index"}}]'''
         self.cmd_output_map['''{"format": "json", "prefix": "mgr services"}'''] = '''{"dashboard": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:7000/", "prometheus": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:9283/"}'''
         self.cmd_output_map['''{"entity": "client.healthchecker", "format": "json", "prefix": "auth get"}'''] = '''{"dashboard": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:7000/", "prometheus": "http://rook-ceph-mgr-a-57cf9f84bc-f4jnl:9283/"}'''
@@ -525,23 +525,20 @@ class RadosJSON:
                 entity = "client.csi-cephfs-provisioner-{}".format(cluster_name)
                 cmd_json = {"prefix": "auth get-or-create",
                             "entity": entity,
-                            "caps": ["mon", "allow r, allow command 'osd blocklist'", 
-                                     "mgr", "allow rw",
-                                     "osd", "allow rw tag cephfs metadata=*"],
+                            "caps": ["mon", "allow r", "mgr", "allow rw",
+                                    "osd", "allow rw tag cephfs metadata=*"],
                             "format": "json"}
             else:
                 entity = "client.csi-cephfs-provisioner-{}-{}".format(cluster_name,cephfs_filesystem)
                 cmd_json = {"prefix": "auth get-or-create",
                             "entity": entity,
-                            "caps": ["mon", "allow r, allow command 'osd blocklist'",
-                                     "mgr", "allow rw",
-                                     "osd", "allow rw tag cephfs metadata={}".format(cephfs_filesystem)],
+                            "caps": ["mon", "allow r", "mgr", "allow rw",
+                                    "osd", "allow rw tag cephfs metadata={}".format(cephfs_filesystem)],
                             "format": "json"}
         else:
             cmd_json = {"prefix": "auth get-or-create",
                         "entity": entity,
-                        "caps": ["mon", "allow r, allow command 'osd blocklist'",
-                                 "mgr", "allow rw",
+                        "caps": ["mon", "allow r", "mgr", "allow rw",
                                  "osd", "allow rw tag cephfs metadata=*"],
                         "format": "json"}
         if self._arg_parser.dry_run:
@@ -567,25 +564,25 @@ class RadosJSON:
                 entity = "client.csi-cephfs-node-{}".format(cluster_name)
                 cmd_json = {"prefix": "auth get-or-create",
                             "entity": entity,
-                            "caps": ["mon", "allow r, allow command 'osd blocklist'",
-                                     "mgr", "allow rw",
-                                     "osd", "allow rw tag cephfs *=*",
-                                     "mds", "allow rw"],
+                            "caps": ["mon", "allow r",
+                                    "mgr", "allow rw",
+                                    "osd", "allow rw tag cephfs *=*",
+                                    "mds", "allow rw"],
                             "format": "json"}
             else:
                 entity = "client.csi-cephfs-node-{}-{}".format(cluster_name,cephfs_filesystem)
                 cmd_json = {"prefix": "auth get-or-create",
                             "entity": entity,
-                            "caps": ["mon", "allow r, allow command 'osd blocklist'",
-                                     "mgr", "allow rw",
-                                     "osd", "allow rw tag cephfs data={}".format(
+                            "caps": ["mon", "allow r",
+                                    "mgr", "allow rw",
+                                    "osd", "allow rw tag cephfs data={}".format(
                                         cephfs_filesystem),
-                                     "mds", "allow rw"],
+                                    "mds", "allow rw"],
                             "format": "json"}
         else:
             cmd_json = {"prefix": "auth get-or-create",
                         "entity": entity,
-                        "caps": ["mon", "allow r, allow command 'osd blocklist'",
+                        "caps": ["mon", "allow r",
                                  "mgr", "allow rw",
                                  "osd", "allow rw tag cephfs *=*",
                                  "mds", "allow rw"],
@@ -613,14 +610,14 @@ class RadosJSON:
             entity = "client.csi-rbd-provisioner-{}-{}-{}".format(cluster_name,rbd_pool_name,rados_namespace)
             cmd_json = {"prefix": "auth get-or-create",
                         "entity": entity,
-                        "caps": ["mon", "profile rbd, allow command 'osd blocklist'",
+                        "caps": ["mon", "profile rbd",
                                  "mgr", "allow rw",
                                  "osd", "profile rbd pool={}".format(rbd_pool_name)],
                         "format": "json"}
         else:
             cmd_json = {"prefix": "auth get-or-create",
                         "entity": entity,
-                        "caps": ["mon", "profile rbd, allow command 'osd blocklist'",
+                        "caps": ["mon", "profile rbd",
                                  "mgr", "allow rw",
                                  "osd", "profile rbd"],
                         "format": "json"}
@@ -736,13 +733,13 @@ class RadosJSON:
             entity = "client.csi-rbd-node-{}-{}-{}".format(cluster_name,rbd_pool_name,rados_namespace)
             cmd_json = {"prefix": "auth get-or-create",
                         "entity": entity,
-                        "caps": ["mon", "profile rbd, allow command 'osd blocklist'",
+                        "caps": ["mon", "profile rbd",
                                  "osd", "profile rbd pool={}".format(rbd_pool_name)],
                         "format": "json"}
         else:
             cmd_json = {"prefix": "auth get-or-create",
                         "entity": entity,
-                        "caps": ["mon", "profile rbd, allow command 'osd blocklist'",
+                        "caps": ["mon", "profile rbd",
                                  "osd", "profile rbd"],
                         "format": "json"}
         if self._arg_parser.dry_run:
diff --git a/deploy/examples/images.txt b/deploy/examples/images.txt
index dac22620e..f084ab20f 100644
--- a/deploy/examples/images.txt
+++ b/deploy/examples/images.txt
@@ -1,10 +1,9 @@
- k8s.gcr.io/sig-storage/csi-attacher:v3.4.0
- k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0
- k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0
+ k8s.gcr.io/sig-storage/csi-attacher:v3.3.0
+ k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
+ k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0
  k8s.gcr.io/sig-storage/csi-resizer:v1.3.0
  k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.0
  quay.io/ceph/ceph:v16.2.7
- quay.io/cephcsi/cephcsi:v3.5.0
- quay.io/csiaddons/k8s-sidecar:v0.1.0
+ quay.io/cephcsi/cephcsi:v3.4.0
  quay.io/csiaddons/volumereplication-operator:v0.1.0
  rook/ceph:master
diff --git a/deploy/examples/monitoring/keda-rgw.yaml b/deploy/examples/monitoring/keda-rgw.yaml
index 915cea4d2..de24eea28 100644
--- a/deploy/examples/monitoring/keda-rgw.yaml
+++ b/deploy/examples/monitoring/keda-rgw.yaml
@@ -1,4 +1,4 @@
-apiVersion: keda.sh/v1alpha1
+apiVersion: keda.k8s.io/v1alpha1
 kind: ScaledObject
 metadata:
   name: rgw-scale
@@ -6,14 +6,14 @@ metadata:
 spec:
   scaleTargetRef:
     kind: Deployment
-    name: rook-ceph-rgw-my-store-a
+    deploymentName: rook-ceph-rgw-my-store-a
   minReplicaCount: 1
   maxReplicaCount: 5
   triggers:
     - type: prometheus
       metadata:
-        serverAddress: http://rook-prometheus.rook-ceph.svc:9090
-        metricName: ceph_rgw_put_collector
-        query: |
-          sum(rate(ceph_rgw_put[2m]))
-        threshold: "90"
+      serverAddress: http://rook-prometheus.rook-ceph.svc:9090
+      metricName: ceph_rgw_put_collector
+      query: |
+        sum(rate(ceph_rgw_put[2m]))
+      threshold: "90"
diff --git a/deploy/examples/operator-openshift.yaml b/deploy/examples/operator-openshift.yaml
index d1340d6b4..cf2939f4b 100644
--- a/deploy/examples/operator-openshift.yaml
+++ b/deploy/examples/operator-openshift.yaml
@@ -161,12 +161,12 @@ data:
   # The default version of CSI supported by Rook will be started. To change the version
   # of the CSI driver to something other than what is officially supported, change
   # these images to the desired release of the CSI driver.
-  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.5.0"
-  # ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0"
+  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.4.0"
+  # ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
   # ROOK_CSI_RESIZER_IMAGE: "k8s.gcr.io/sig-storage/csi-resizer:v1.3.0"
-  # ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0"
+  # ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
   # ROOK_CSI_SNAPSHOTTER_IMAGE: "k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.0"
-  # ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.4.0"
+  # ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.3.0"
 
   # (Optional) set user created priorityclassName for csi plugin pods.
   # CSI_PLUGIN_PRIORITY_CLASSNAME: "system-node-critical"
@@ -403,7 +403,6 @@ data:
   # Configure CSI RBD grpc and liveness metrics port
   # CSI_RBD_GRPC_METRICS_PORT: "9090"
   # CSI_RBD_LIVENESS_METRICS_PORT: "9080"
-  # CSIADDONS_PORT: "9070"
 
   # Whether the OBC provisioner should watch on the operator namespace or not, if not the namespace of the cluster will be used
   ROOK_OBC_WATCH_OPERATOR_NAMESPACE: "true"
@@ -418,9 +417,6 @@ data:
   # The timeout value (in seconds) of Ceph commands. It should be >= 1. If this variable is not set or is an invalid value, it's default to 15.
   ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS: "15"
   # CSI_VOLUME_REPLICATION_IMAGE: "quay.io/csiaddons/volumereplication-operator:v0.1.0"
-  # Enable the csi addons sidecar.
-  CSI_ENABLE_CSIADDONS: "false"
-  # ROOK_CSIADDONS_IMAGE: "quay.io/csiaddons/k8s-sidecar:v0.1.0"
 ---
 # The deployment for the rook operator
 # OLM: BEGIN OPERATOR DEPLOYMENT
diff --git a/deploy/examples/operator.yaml b/deploy/examples/operator.yaml
index c81ad7a9e..0c76546dc 100644
--- a/deploy/examples/operator.yaml
+++ b/deploy/examples/operator.yaml
@@ -79,12 +79,12 @@ data:
   # The default version of CSI supported by Rook will be started. To change the version
   # of the CSI driver to something other than what is officially supported, change
   # these images to the desired release of the CSI driver.
-  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.5.0"
-  # ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0"
+  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.4.0"
+  # ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
   # ROOK_CSI_RESIZER_IMAGE: "k8s.gcr.io/sig-storage/csi-resizer:v1.3.0"
-  # ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0"
+  # ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
   # ROOK_CSI_SNAPSHOTTER_IMAGE: "k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.0"
-  # ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.4.0"
+  # ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.3.0"
 
   # (Optional) set user created priorityclassName for csi plugin pods.
   # CSI_PLUGIN_PRIORITY_CLASSNAME: "system-node-critical"
@@ -321,7 +321,6 @@ data:
   # Configure CSI RBD grpc and liveness metrics port
   # CSI_RBD_GRPC_METRICS_PORT: "9090"
   # CSI_RBD_LIVENESS_METRICS_PORT: "9080"
-  # CSIADDONS_PORT: "9070"
 
   # Whether the OBC provisioner should watch on the operator namespace or not, if not the namespace of the cluster will be used
   ROOK_OBC_WATCH_OPERATOR_NAMESPACE: "true"
@@ -336,9 +335,6 @@ data:
   # See https://rook.io/docs/rook/latest/ceph-csi-drivers.html#rbd-mirroring
   CSI_ENABLE_VOLUME_REPLICATION: "false"
   # CSI_VOLUME_REPLICATION_IMAGE: "quay.io/csiaddons/volumereplication-operator:v0.1.0"
-  # Enable the csi addons sidecar.
-  CSI_ENABLE_CSIADDONS: "false"
-  # ROOK_CSIADDONS_IMAGE: "quay.io/csiaddons/k8s-sidecar:v0.1.0"
 ---
 # OLM: BEGIN OPERATOR DEPLOYMENT
 apiVersion: apps/v1
diff --git a/pkg/apis/ceph.rook.io/v1/annotations.go b/pkg/apis/ceph.rook.io/v1/annotations.go
index 4f611abf4..d68ab9991 100644
--- a/pkg/apis/ceph.rook.io/v1/annotations.go
+++ b/pkg/apis/ceph.rook.io/v1/annotations.go
@@ -57,10 +57,6 @@ func GetCleanupAnnotations(a AnnotationsSpec) Annotations {
 	return mergeAllAnnotationsWithKey(a, KeyCleanup)
 }
 
-func GetClusterMetadataAnnotations(a AnnotationsSpec) Annotations {
-	return a[KeyClusterMetadata]
-}
-
 func mergeAllAnnotationsWithKey(a AnnotationsSpec, name KeyType) Annotations {
 	all := a.All()
 	if all != nil {
diff --git a/pkg/apis/ceph.rook.io/v1/keys.go b/pkg/apis/ceph.rook.io/v1/keys.go
index 3c4bf633b..3257a312f 100644
--- a/pkg/apis/ceph.rook.io/v1/keys.go
+++ b/pkg/apis/ceph.rook.io/v1/keys.go
@@ -17,15 +17,14 @@ limitations under the License.
 package v1
 
 const (
-	KeyAll                     = "all"
-	KeyMds             KeyType = "mds"
-	KeyMon             KeyType = "mon"
-	KeyMonArbiter      KeyType = "arbiter"
-	KeyMgr             KeyType = "mgr"
-	KeyOSDPrepare      KeyType = "prepareosd"
-	KeyOSD             KeyType = "osd"
-	KeyCleanup         KeyType = "cleanup"
-	KeyMonitoring      KeyType = "monitoring"
-	KeyCrashCollector  KeyType = "crashcollector"
-	KeyClusterMetadata KeyType = "clusterMetadata"
+	KeyAll                    = "all"
+	KeyMds            KeyType = "mds"
+	KeyMon            KeyType = "mon"
+	KeyMonArbiter     KeyType = "arbiter"
+	KeyMgr            KeyType = "mgr"
+	KeyOSDPrepare     KeyType = "prepareosd"
+	KeyOSD            KeyType = "osd"
+	KeyCleanup        KeyType = "cleanup"
+	KeyMonitoring     KeyType = "monitoring"
+	KeyCrashCollector KeyType = "crashcollector"
 )
diff --git a/pkg/apis/ceph.rook.io/v1/priorityclasses.go b/pkg/apis/ceph.rook.io/v1/priorityclasses.go
index c984b852c..d60ed1acd 100644
--- a/pkg/apis/ceph.rook.io/v1/priorityclasses.go
+++ b/pkg/apis/ceph.rook.io/v1/priorityclasses.go
@@ -55,11 +55,3 @@ func GetCleanupPriorityClassName(p PriorityClassNamesSpec) string {
 	}
 	return p[KeyCleanup]
 }
-
-// GetCrashCollectorPriorityClassName returns the priority class name for the crashcollector
-func GetCrashCollectorPriorityClassName(p PriorityClassNamesSpec) string {
-	if _, ok := p[KeyCrashCollector]; !ok {
-		return p.All()
-	}
-	return p[KeyCrashCollector]
-}
diff --git a/pkg/operator/ceph/cluster/crash/crash.go b/pkg/operator/ceph/cluster/crash/crash.go
index b0db4acfc..d5836a02f 100644
--- a/pkg/operator/ceph/cluster/crash/crash.go
+++ b/pkg/operator/ceph/cluster/crash/crash.go
@@ -113,11 +113,10 @@ func (r *ReconcileNode) createOrUpdateCephCrash(node corev1.Node, tolerations []
 				Containers: []corev1.Container{
 					getCrashDaemonContainer(cephCluster, *cephVersion),
 				},
-				Tolerations:       tolerations,
-				RestartPolicy:     corev1.RestartPolicyAlways,
-				HostNetwork:       cephCluster.Spec.Network.IsHost(),
-				Volumes:           volumes,
-				PriorityClassName: cephv1.GetCrashCollectorPriorityClassName(cephCluster.Spec.PriorityClassNames),
+				Tolerations:   tolerations,
+				RestartPolicy: corev1.RestartPolicyAlways,
+				HostNetwork:   cephCluster.Spec.Network.IsHost(),
+				Volumes:       volumes,
 			},
 		}
 
diff --git a/pkg/operator/ceph/cluster/mon/mon.go b/pkg/operator/ceph/cluster/mon/mon.go
index 070ce2edc..c0c30865f 100644
--- a/pkg/operator/ceph/cluster/mon/mon.go
+++ b/pkg/operator/ceph/cluster/mon/mon.go
@@ -493,11 +493,6 @@ func (c *Cluster) initClusterInfo(cephVersion cephver.CephVersion, clusterName s
 		return errors.Wrap(err, "failed to get cluster info")
 	}
 
-	err = keyring.ApplyClusterMetadataToSecret(c.ClusterInfo, AppName, c.context, c.spec.Annotations)
-	if err != nil {
-		return errors.Wrap(err, "failed to apply annotation")
-	}
-
 	c.ClusterInfo.CephVersion = cephVersion
 	c.ClusterInfo.OwnerInfo = c.ownerInfo
 	c.ClusterInfo.Context = context
@@ -514,7 +509,7 @@ func (c *Cluster) initClusterInfo(cephVersion cephver.CephVersion, clusterName s
 		return errors.Wrap(err, "failed to save mon keyring secret")
 	}
 	// also store the admin keyring for other daemons that might need it during init
-	if err := k.Admin().CreateOrUpdate(c.ClusterInfo, c.context, c.spec.Annotations); err != nil {
+	if err := k.Admin().CreateOrUpdate(c.ClusterInfo); err != nil {
 		return errors.Wrap(err, "failed to save admin keyring secret")
 	}
 
@@ -1057,8 +1052,6 @@ func (c *Cluster) persistExpectedMonDaemons() error {
 			Finalizers: []string{DisasterProtectionFinalizerName},
 		},
 	}
-	cephv1.GetClusterMetadataAnnotations(c.spec.Annotations).ApplyToObjectMeta(&configMap.ObjectMeta)
-
 	err := c.ownerInfo.SetControllerReference(configMap)
 	if err != nil {
 		return errors.Wrapf(err, "failed to set owner reference mon configmap %q", configMap.Name)
diff --git a/pkg/operator/ceph/cluster/osd/spec.go b/pkg/operator/ceph/cluster/osd/spec.go
index 2f57fa52a..63cb3a83a 100644
--- a/pkg/operator/ceph/cluster/osd/spec.go
+++ b/pkg/operator/ceph/cluster/osd/spec.go
@@ -233,7 +233,7 @@ var defaultTuneFastSettings = []string{
 	"--osd-delete-sleep=0",                        // Time in seconds to sleep before next removal transaction for SSDs
 	"--bluestore-min-alloc-size=4096",             // Default min_alloc_size value for SSDs
 	"--bluestore-prefer-deferred-size=0",          // Default value of bluestore_prefer_deferred_size for SSDs
-	"--bluestore-compression-min-blob-size=8192",  // Default value of bluestore_compression_min_blob_size for SSDs
+	"--bluestore-compression-min-blob-size=8912",  // Default value of bluestore_compression_min_blob_size for SSDs
 	"--bluestore-compression-max-blob-size=65536", // Default value of bluestore_compression_max_blob_size for SSDs
 	"--bluestore-max-blob-size=65536",             // Default value of bluestore_max_blob_size for SSDs
 	"--bluestore-cache-size=3221225472",           // Default value of bluestore_cache_size for SSDs
diff --git a/pkg/operator/ceph/config/keyring/admin.go b/pkg/operator/ceph/config/keyring/admin.go
index cd3ac2d13..fa793f6a7 100644
--- a/pkg/operator/ceph/config/keyring/admin.go
+++ b/pkg/operator/ceph/config/keyring/admin.go
@@ -19,11 +19,7 @@ package keyring
 import (
 	"fmt"
 
-	"github.com/pkg/errors"
-	v1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
-	"github.com/rook/rook/pkg/clusterd"
 	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
 const (
@@ -52,34 +48,7 @@ func (s *SecretStore) Admin() *AdminStore {
 }
 
 // CreateOrUpdate creates or updates the admin keyring secret with cluster information.
-func (a *AdminStore) CreateOrUpdate(c *cephclient.ClusterInfo, context *clusterd.Context, annotation v1.AnnotationsSpec) error {
+func (a *AdminStore) CreateOrUpdate(c *cephclient.ClusterInfo) error {
 	keyring := fmt.Sprintf(adminKeyringTemplate, c.CephCred.Secret)
-	err := a.secretStore.CreateOrUpdate(adminKeyringResourceName, keyring)
-	if err != nil {
-		return err
-	}
-	err = ApplyClusterMetadataToSecret(c, keyringSecretName(adminKeyringResourceName), context, annotation)
-	if err != nil {
-		return errors.Errorf("failed to update admin secrets. %v", err)
-	}
-
-	return nil
-}
-
-func ApplyClusterMetadataToSecret(c *cephclient.ClusterInfo, secretName string, context *clusterd.Context, annotation v1.AnnotationsSpec) error {
-	// Get secret to update annotation
-	secret, err := context.Clientset.CoreV1().Secrets(c.Namespace).Get(c.Context, secretName, metav1.GetOptions{})
-	if err != nil {
-		return errors.Wrapf(err, "failed to get %s secrets", secretName)
-	}
-	// We would need to reset the annotations back to empty, then reapply the annotations this is because the in some rook-ceph-mon secret is retrieved
-	// and then updated, instead of a new secret being generated.
-	secret.Annotations = map[string]string{}
-	v1.GetClusterMetadataAnnotations(annotation).ApplyToObjectMeta(&secret.ObjectMeta)
-
-	_, err = context.Clientset.CoreV1().Secrets(c.Namespace).Update(c.Context, secret, metav1.UpdateOptions{})
-	if err != nil {
-		return errors.Wrapf(err, "failed to update %s secret.", secretName)
-	}
-	return nil
+	return a.secretStore.CreateOrUpdate(adminKeyringResourceName, keyring)
 }
diff --git a/pkg/operator/ceph/csi/csi.go b/pkg/operator/ceph/csi/csi.go
index 17ce6c466..246d615bf 100644
--- a/pkg/operator/ceph/csi/csi.go
+++ b/pkg/operator/ceph/csi/csi.go
@@ -94,7 +94,6 @@ func (r *ReconcileCSI) setParams() error {
 	CSIParam.SnapshotterImage = k8sutil.GetValue(r.opConfig.Parameters, "ROOK_CSI_SNAPSHOTTER_IMAGE", DefaultSnapshotterImage)
 	CSIParam.KubeletDirPath = k8sutil.GetValue(r.opConfig.Parameters, "ROOK_CSI_KUBELET_DIR_PATH", DefaultKubeletDirPath)
 	CSIParam.VolumeReplicationImage = k8sutil.GetValue(r.opConfig.Parameters, "CSI_VOLUME_REPLICATION_IMAGE", DefaultVolumeReplicationImage)
-	CSIParam.CSIAddonsImage = k8sutil.GetValue(r.opConfig.Parameters, "ROOK_CSIADDONS_IMAGE", DefaultCSIAddonsImage)
 	csiCephFSPodLabels := k8sutil.GetValue(r.opConfig.Parameters, "ROOK_CSI_CEPHFS_POD_LABELS", "")
 	CSIParam.CSICephFSPodLabels = k8sutil.ParseStringToLabels(csiCephFSPodLabels)
 	csiRBDPodLabels := k8sutil.GetValue(r.opConfig.Parameters, "ROOK_CSI_RBD_POD_LABELS", "")
diff --git a/pkg/operator/ceph/csi/spec.go b/pkg/operator/ceph/csi/spec.go
index cad77a8f1..30a20b6e3 100644
--- a/pkg/operator/ceph/csi/spec.go
+++ b/pkg/operator/ceph/csi/spec.go
@@ -51,19 +51,16 @@ type Param struct {
 	PluginPriorityClassName        string
 	ProvisionerPriorityClassName   string
 	VolumeReplicationImage         string
-	CSIAddonsImage                 string
 	EnablePluginSelinuxHostMount   bool
 	EnableCSIHostNetwork           bool
 	EnableOMAPGenerator            bool
 	EnableRBDSnapshotter           bool
 	EnableCephFSSnapshotter        bool
 	EnableVolumeReplicationSideCar bool
-	EnableCSIAddonsSideCar         bool
 	LogLevel                       uint8
 	CephFSGRPCMetricsPort          uint16
 	CephFSLivenessMetricsPort      uint16
 	RBDGRPCMetricsPort             uint16
-	CSIAddonsPort                  uint16
 	RBDLivenessMetricsPort         uint16
 	ProvisionerReplicas            int32
 	CSICephFSPodLabels             map[string]string
@@ -101,14 +98,13 @@ var (
 // manually challenging.
 var (
 	// image names
-	DefaultCSIPluginImage         = "quay.io/cephcsi/cephcsi:v3.5.0"
-	DefaultRegistrarImage         = "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.4.0"
-	DefaultProvisionerImage       = "k8s.gcr.io/sig-storage/csi-provisioner:v3.1.0"
-	DefaultAttacherImage          = "k8s.gcr.io/sig-storage/csi-attacher:v3.4.0"
+	DefaultCSIPluginImage         = "quay.io/cephcsi/cephcsi:v3.4.0"
+	DefaultRegistrarImage         = "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0"
+	DefaultProvisionerImage       = "k8s.gcr.io/sig-storage/csi-provisioner:v3.0.0"
+	DefaultAttacherImage          = "k8s.gcr.io/sig-storage/csi-attacher:v3.3.0"
 	DefaultSnapshotterImage       = "k8s.gcr.io/sig-storage/csi-snapshotter:v4.2.0"
 	DefaultResizerImage           = "k8s.gcr.io/sig-storage/csi-resizer:v1.3.0"
 	DefaultVolumeReplicationImage = "quay.io/csiaddons/volumereplication-operator:v0.1.0"
-	DefaultCSIAddonsImage         = "quay.io/csiaddons/k8s-sidecar:v0.1.0"
 
 	// Local package template path for RBD
 	//go:embed template/rbd/csi-rbdplugin.yaml
@@ -166,7 +162,6 @@ const (
 	DefaultCephFSLivenessMerticsPort uint16 = 9081
 	DefaultRBDGRPCMerticsPort        uint16 = 9090
 	DefaultRBDLivenessMerticsPort    uint16 = 9080
-	DefaultCSIAddonsPort             uint16 = 9070
 
 	detectCSIVersionName = "rook-ceph-csi-detect-version"
 	// default log level for csi containers
@@ -275,10 +270,6 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 	if err != nil {
 		return errors.Wrap(err, "error getting CSI RBD GRPC metrics port.")
 	}
-	tp.CSIAddonsPort, err = getPortFromConfig(r.opConfig.Parameters, "CSIADDONS_PORT", DefaultCSIAddonsPort)
-	if err != nil {
-		return errors.Wrap(err, "failed to get CSI Addons port")
-	}
 	tp.RBDLivenessMetricsPort, err = getPortFromConfig(r.opConfig.Parameters, "CSI_RBD_LIVENESS_METRICS_PORT", DefaultRBDLivenessMerticsPort)
 	if err != nil {
 		return errors.Wrap(err, "error getting CSI RBD liveness metrics port.")
@@ -314,11 +305,6 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 		tp.EnableVolumeReplicationSideCar = true
 	}
 
-	tp.EnableCSIAddonsSideCar = false
-	if strings.EqualFold(k8sutil.GetValue(r.opConfig.Parameters, "CSI_ENABLE_CSIADDONS", "false"), "true") {
-		tp.EnableCSIAddonsSideCar = true
-	}
-
 	if strings.EqualFold(k8sutil.GetValue(r.opConfig.Parameters, "CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY", rollingUpdate), onDelete) {
 		tp.CephFSPluginUpdateStrategy = onDelete
 	} else {
diff --git a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
index b586ffb94..8ac752b3d 100644
--- a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
+++ b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
@@ -127,49 +127,6 @@ spec:
             - name: socket-dir
               mountPath: /csi
         {{ end }}
-        {{ if .EnableCSIAddonsSideCar }}
-        - name: csi-addons
-          image: {{ .CSIAddonsImage }}
-          args :
-            - "--node-id=$(NODE_ID)"
-            - "--v={{ .LogLevel }}"
-            - "--csi-addons-address=$(CSIADDONS_ENDPOINT)"
-            - "--controller-ip=$(POD_IP)"
-            - "--controller-port={{ .CSIAddonsPort }}"
-            - "--pod=$(POD_NAME)"
-            - "--namespace=$(POD_NAMESPACE)"
-            - "--pod-uid=$(POD_UID)"
-            - "--stagingpath={{ .KubeletDirPath }}/plugins/kubernetes.io/csi/pv/"
-          ports:
-            - containerPort: {{ .CSIAddonsPort }}
-          env:
-            - name: POD_IP
-              valueFrom:
-                fieldRef:
-                  fieldPath: status.podIP
-            - name: NODE_ID
-              valueFrom:
-                fieldRef:
-                  fieldPath: spec.nodeName
-            - name: POD_UID
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.uid
-            - name: POD_NAME
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.name
-            - name: POD_NAMESPACE
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.namespace
-            - name: CSIADDONS_ENDPOINT
-              value: unix:///csi/csi-addons.sock
-          imagePullPolicy: "IfNotPresent"
-          volumeMounts:
-            - name: socket-dir
-              mountPath: /csi
-        {{ end }}
         - name: csi-rbdplugin
           image: {{ .CSIPluginImage }}
           args :
@@ -183,9 +140,6 @@ spec:
             - "--metricsport={{ .RBDGRPCMetricsPort }}"
             - "--metricspath=/metrics"
             - "--enablegrpcmetrics={{ .EnableCSIGRPCMetrics }}"
-            {{ if .EnableCSIAddonsSideCar }}
-            - "--csi-addons-endpoint=$(CSIADDONS_ENDPOINT)"
-            {{ end }}
           env:
             - name: POD_IP
               valueFrom:
@@ -201,10 +155,6 @@ spec:
                   fieldPath: metadata.namespace
             - name: CSI_ENDPOINT
               value: unix:///csi/csi-provisioner.sock
-            {{ if .EnableCSIAddonsSideCar }}
-            - name: CSIADDONS_ENDPOINT
-              value: unix:///csi/csi-addons.sock
-            {{ end }}
           imagePullPolicy: "IfNotPresent"
           volumeMounts:
             - name: socket-dir
diff --git a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin.yaml b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin.yaml
index 2b1ae4fcb..b6d9767d2 100644
--- a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin.yaml
+++ b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin.yaml
@@ -68,9 +68,6 @@ spec:
             - "--metricspath=/metrics"
             - "--enablegrpcmetrics={{ .EnableCSIGRPCMetrics }}"
             - "--stagingpath={{ .KubeletDirPath }}/plugins/kubernetes.io/csi/pv/"
-            {{ if .EnableCSIAddonsSideCar }}
-            - "--csi-addons-endpoint=$(CSIADDONS_ENDPOINT)"
-            {{ end }}
           env:
             - name: POD_IP
               valueFrom:
@@ -86,10 +83,6 @@ spec:
                   fieldPath: metadata.namespace
             - name: CSI_ENDPOINT
               value: unix:///csi/csi.sock
-            {{ if .EnableCSIAddonsSideCar }}
-            - name: CSIADDONS_ENDPOINT
-              value: unix:///csi/csi-addons.sock
-            {{ end }}
           imagePullPolicy: "IfNotPresent"
           volumeMounts:
             - name: plugin-dir
@@ -118,51 +111,6 @@ spec:
               mountPath: /etc/selinux
               readOnly: true
             {{ end }}
-        {{ if .EnableCSIAddonsSideCar }}
-        - name: csi-addons
-          securityContext:
-            privileged: true
-          image: {{ .CSIAddonsImage }}
-          args :
-            - "--node-id=$(NODE_ID)"
-            - "--v={{ .LogLevel }}"
-            - "--csi-addons-address=$(CSIADDONS_ENDPOINT)"
-            - "--controller-ip=$(POD_IP)"
-            - "--controller-port={{ .CSIAddonsPort }}"
-            - "--pod=$(POD_NAME)"
-            - "--namespace=$(POD_NAMESPACE)"
-            - "--pod-uid=$(POD_UID)"
-            - "--stagingpath={{ .KubeletDirPath }}/plugins/kubernetes.io/csi/pv/"
-          ports:
-            - containerPort: {{ .CSIAddonsPort }}
-          env:
-            - name: POD_IP
-              valueFrom:
-                fieldRef:
-                  fieldPath: status.podIP
-            - name: NODE_ID
-              valueFrom:
-                fieldRef:
-                  fieldPath: spec.nodeName
-            - name: POD_UID
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.uid
-            - name: POD_NAME
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.name
-            - name: POD_NAMESPACE
-              valueFrom:
-                fieldRef:
-                  fieldPath: metadata.namespace
-            - name: CSIADDONS_ENDPOINT
-              value: unix:///csi/csi-addons.sock
-          imagePullPolicy: "IfNotPresent"
-          volumeMounts:
-            - name: plugin-dir
-              mountPath: /csi
-        {{ end }}
         - name: liveness-prometheus
           securityContext:
             privileged: true
diff --git a/pkg/operator/ceph/csi/version.go b/pkg/operator/ceph/csi/version.go
index c8e059610..bcc8166b3 100644
--- a/pkg/operator/ceph/csi/version.go
+++ b/pkg/operator/ceph/csi/version.go
@@ -25,15 +25,15 @@ import (
 )
 
 var (
-	//minimum supported version is 3.4.0
-	minimum = CephCSIVersion{3, 4, 0}
+	//minimum supported version is 3.3.0
+	minimum = CephCSIVersion{3, 3, 0}
 	//supportedCSIVersions are versions that rook supports
+	releasev330          = CephCSIVersion{3, 3, 0}
 	releasev340          = CephCSIVersion{3, 4, 0}
-	releasev350          = CephCSIVersion{3, 5, 0}
 	supportedCSIVersions = []CephCSIVersion{
 		minimum,
+		releasev330,
 		releasev340,
-		releasev350,
 	}
 	// for parsing the output of `cephcsi`
 	versionCSIPattern = regexp.MustCompile(`v(\d+)\.(\d+)\.(\d+)`)
diff --git a/pkg/operator/ceph/object/spec.go b/pkg/operator/ceph/object/spec.go
index a654abc7b..ea7a7babd 100644
--- a/pkg/operator/ceph/object/spec.go
+++ b/pkg/operator/ceph/object/spec.go
@@ -61,9 +61,19 @@ func (c *clusterConfig) createDeployment(rgwConfig *rgwConfig) (*apps.Deployment
 		return nil, err
 	}
 	replicas := int32(1)
-	// On Pacific, we can use the same keyring and have dedicated rgw instances reflected in the service map
+	strategy := apps.DeploymentStrategy{
+		Type: apps.RecreateDeploymentStrategyType,
+	}
 	if c.clusterInfo.CephVersion.IsAtLeastPacific() {
+		// On Pacific, we can use the same keyring and have dedicated rgw instances reflected in the service map
 		replicas = c.store.Spec.Gateway.Instances
+
+		// On Pacific, rgw gateway deployments rolling update
+		strategy.Type = apps.RollingUpdateDeploymentStrategyType
+		strategy.RollingUpdate = &apps.RollingUpdateDeployment{
+			MaxUnavailable: &intstr.IntOrString{IntVal: int32(1)},
+			MaxSurge:       &intstr.IntOrString{IntVal: int32(0)},
+		}
 	}
 	d := &apps.Deployment{
 		ObjectMeta: metav1.ObjectMeta{
@@ -77,9 +87,7 @@ func (c *clusterConfig) createDeployment(rgwConfig *rgwConfig) (*apps.Deployment
 			},
 			Template: pod,
 			Replicas: &replicas,
-			Strategy: apps.DeploymentStrategy{
-				Type: apps.RecreateDeploymentStrategyType,
-			},
+			Strategy: strategy,
 		},
 	}
 	k8sutil.AddRookVersionLabelToDeployment(d)


Test patch:
diff --git a/.github/workflows/canary-integration-test.yml b/.github/workflows/canary-integration-test.yml
index fc1ea7be8..f331e0dc6 100644
--- a/.github/workflows/canary-integration-test.yml
+++ b/.github/workflows/canary-integration-test.yml
@@ -57,13 +57,6 @@ jobs:
         kubectl -n rook-ceph exec $toolbox -- mkdir -p /etc/ceph/test-data
         kubectl -n rook-ceph cp tests/ceph-status-out $toolbox:/etc/ceph/test-data/
         kubectl -n rook-ceph cp deploy/examples/create-external-cluster-resources.py $toolbox:/etc/ceph
-        # remove the existing client auth that will be re-created by external script
-        kubectl -n rook-ceph exec $toolbox -- ceph auth del client.csi-cephfs-node
-        kubectl -n rook-ceph exec $toolbox -- ceph auth del client.csi-cephfs-provisioner
-        kubectl -n rook-ceph exec $toolbox -- ceph auth del client.csi-rbd-node
-        kubectl -n rook-ceph exec $toolbox -- ceph auth del client.csi-rbd-provisioner
-        # print existing client auth
-        kubectl -n rook-ceph exec $toolbox -- ceph auth ls
         timeout 10 sh -c "until kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool; do echo 'waiting for script to succeed' && sleep 1; done"
 
     - name: dry run external script create-external-cluster-resources.py
@@ -840,7 +833,6 @@ jobs:
 
     - name: run Encryption KMS IBM Key Protect
       uses: ./.github/workflows/encryption-pvc-kms-ibm-kp
-      if: "env.IBM_KP_INSTANCE_ID != '' && env.IBM_KP_SERVICE_API_KEY != ''"
       with:
         ibm-instance-id: ${{ secrets.IBM_INSTANCE_ID }}
         ibm-service-api-key: ${{ secrets.IBM_SERVICE_API_KEY }}
diff --git a/pkg/apis/ceph.rook.io/v1/priorityclasses_test.go b/pkg/apis/ceph.rook.io/v1/priorityclasses_test.go
index cc2f018ac..a92e584bc 100644
--- a/pkg/apis/ceph.rook.io/v1/priorityclasses_test.go
+++ b/pkg/apis/ceph.rook.io/v1/priorityclasses_test.go
@@ -30,7 +30,6 @@ all: all-class
 mgr: mgr-class
 mon: mon-class
 osd: osd-class
-crashcollector: crashcollector-class
 `)
 
 	// convert the raw spec yaml into JSON
@@ -44,11 +43,10 @@ crashcollector: crashcollector-class
 
 	// the unmarshalled priority class names spec should equal the expected spec below
 	expected := PriorityClassNamesSpec{
-		"all":            "all-class",
-		"mgr":            "mgr-class",
-		"mon":            "mon-class",
-		"osd":            "osd-class",
-		"crashcollector": "crashcollector-class",
+		"all": "all-class",
+		"mgr": "mgr-class",
+		"mon": "mon-class",
+		"osd": "osd-class",
 	}
 	assert.Equal(t, expected, priorityClassNames)
 }
diff --git a/pkg/operator/ceph/cluster/crash/crash_test.go b/pkg/operator/ceph/cluster/crash/crash_test.go
index 55e233b5e..3ead1f591 100644
--- a/pkg/operator/ceph/cluster/crash/crash_test.go
+++ b/pkg/operator/ceph/cluster/crash/crash_test.go
@@ -18,7 +18,6 @@ package crash
 
 import (
 	"context"
-	"fmt"
 	"testing"
 
 	cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
@@ -26,18 +25,14 @@ import (
 	"github.com/rook/rook/pkg/client/clientset/versioned/scheme"
 	"github.com/rook/rook/pkg/clusterd"
 	cephver "github.com/rook/rook/pkg/operator/ceph/version"
-	"github.com/rook/rook/pkg/operator/k8sutil"
 	"github.com/rook/rook/pkg/operator/test"
 	"github.com/stretchr/testify/assert"
-	appsv1 "k8s.io/api/apps/v1"
 	v1 "k8s.io/api/batch/v1"
 	"k8s.io/api/batch/v1beta1"
-	corev1 "k8s.io/api/core/v1"
 	kerrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
-	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 	cntrlutil "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 )
 
@@ -110,61 +105,3 @@ func TestCreateOrUpdateCephCron(t *testing.T) {
 	assert.Error(t, err)
 	assert.True(t, kerrors.IsNotFound(err))
 }
-
-func TestCreateOrUpdateCephCrash(t *testing.T) {
-	cephCluster := cephv1.CephCluster{
-		ObjectMeta: metav1.ObjectMeta{Namespace: "rook-ceph"},
-	}
-	cephCluster.Spec.Labels = cephv1.LabelsSpec{}
-	cephCluster.Spec.PriorityClassNames = cephv1.PriorityClassNamesSpec{}
-	cephVersion := &cephver.CephVersion{Major: 16, Minor: 2, Extra: 0}
-	ctx := context.TODO()
-	context := &clusterd.Context{
-		Clientset:     test.New(t, 1),
-		RookClientset: rookclient.NewSimpleClientset(),
-	}
-
-	s := scheme.Scheme
-	err := appsv1.AddToScheme(s)
-	if err != nil {
-		assert.Fail(t, "failed to build scheme")
-	}
-	r := &ReconcileNode{
-		scheme:  s,
-		client:  fake.NewClientBuilder().WithScheme(s).WithRuntimeObjects().Build(),
-		context: context,
-	}
-
-	node := corev1.Node{}
-	nodeSelector := map[string]string{corev1.LabelHostname: "testnode"}
-	node.SetLabels(nodeSelector)
-	tolerations := []corev1.Toleration{{}}
-	res, err := r.createOrUpdateCephCrash(node, tolerations, cephCluster, cephVersion)
-	assert.NoError(t, err)
-	assert.Equal(t, controllerutil.OperationResult("created"), res)
-	name := k8sutil.TruncateNodeName(fmt.Sprintf("%s-%%s", AppName), "testnode")
-	deploy := appsv1.Deployment{}
-	err = r.client.Get(ctx, types.NamespacedName{Namespace: "rook-ceph", Name: name}, &deploy)
-	assert.NoError(t, err)
-	podSpec := deploy.Spec.Template
-	assert.Equal(t, nodeSelector, podSpec.Spec.NodeSelector)
-	assert.Equal(t, "", podSpec.ObjectMeta.Labels["foo"])
-	assert.Equal(t, tolerations, podSpec.Spec.Tolerations)
-	assert.Equal(t, false, podSpec.Spec.HostNetwork)
-	assert.Equal(t, "", podSpec.Spec.PriorityClassName)
-
-	cephCluster.Spec.Labels[cephv1.KeyCrashCollector] = map[string]string{"foo": "bar"}
-	cephCluster.Spec.Network.HostNetwork = true
-	cephCluster.Spec.PriorityClassNames[cephv1.KeyCrashCollector] = "test-priority-class"
-	tolerations = []corev1.Toleration{{Key: "key", Operator: "Equal", Value: "value", Effect: "NoSchedule"}}
-	res, err = r.createOrUpdateCephCrash(node, tolerations, cephCluster, cephVersion)
-	assert.NoError(t, err)
-	assert.Equal(t, controllerutil.OperationResult("updated"), res)
-	err = r.client.Get(ctx, types.NamespacedName{Namespace: "rook-ceph", Name: name}, &deploy)
-	assert.NoError(t, err)
-	podSpec = deploy.Spec.Template
-	assert.Equal(t, "bar", podSpec.ObjectMeta.Labels["foo"])
-	assert.Equal(t, tolerations, podSpec.Spec.Tolerations)
-	assert.Equal(t, true, podSpec.Spec.HostNetwork)
-	assert.Equal(t, "test-priority-class", podSpec.Spec.PriorityClassName)
-}
diff --git a/pkg/operator/ceph/cluster/mon/mon_test.go b/pkg/operator/ceph/cluster/mon/mon_test.go
index 14e6f37db..31938d307 100644
--- a/pkg/operator/ceph/cluster/mon/mon_test.go
+++ b/pkg/operator/ceph/cluster/mon/mon_test.go
@@ -181,21 +181,11 @@ func TestStartMonPods(t *testing.T) {
 	assert.NoError(t, err)
 	c := newCluster(context, namespace, true, v1.ResourceRequirements{})
 	c.ClusterInfo = clienttest.CreateTestClusterInfo(1)
-	c.spec.Annotations = cephv1.AnnotationsSpec{
-		cephv1.KeyClusterMetadata: cephv1.Annotations{
-			"key": "value",
-		},
-	}
 
 	// start a basic cluster
 	_, err = c.Start(c.ClusterInfo, c.rookVersion, cephver.Octopus, c.spec)
 	assert.NoError(t, err)
 
-	// test annotations
-	secret, err := c.context.Clientset.CoreV1().Secrets(c.Namespace).Get(c.ClusterInfo.Context, AppName, metav1.GetOptions{})
-	assert.NoError(t, err)
-	assert.Equal(t, map[string]string{"key": "value"}, secret.Annotations)
-
 	validateStart(t, c)
 
 	// starting again should be a no-op
@@ -273,7 +263,7 @@ func validateStart(t *testing.T, c *Cluster) {
 func TestPersistMons(t *testing.T) {
 	clientset := test.New(t, 1)
 	ownerInfo := cephclient.NewMinimumOwnerInfoWithOwnerRef()
-	c := New(&clusterd.Context{Clientset: clientset}, "ns", cephv1.ClusterSpec{Annotations: cephv1.AnnotationsSpec{cephv1.KeyClusterMetadata: cephv1.Annotations{"key": "value"}}}, ownerInfo, &sync.Mutex{})
+	c := New(&clusterd.Context{Clientset: clientset}, "ns", cephv1.ClusterSpec{}, ownerInfo, &sync.Mutex{})
 	setCommonMonProperties(c, 1, cephv1.MonSpec{Count: 3, AllowMultiplePerNode: true}, "myversion")
 
 	// Persist mon a
@@ -283,7 +273,6 @@ func TestPersistMons(t *testing.T) {
 	cm, err := c.context.Clientset.CoreV1().ConfigMaps(c.Namespace).Get(context.TODO(), EndpointConfigMapName, metav1.GetOptions{})
 	assert.NoError(t, err)
 	assert.Equal(t, "a=1.2.3.1:6789", cm.Data[EndpointDataKey])
-	assert.Equal(t, map[string]string{"key": "value"}, cm.Annotations)
 
 	// Persist mon b, and remove mon a for simply testing the configmap is updated
 	c.ClusterInfo.Monitors["b"] = &cephclient.MonInfo{Name: "b", Endpoint: "4.5.6.7:3300"}
diff --git a/pkg/operator/ceph/config/keyring/admin_test.go b/pkg/operator/ceph/config/keyring/admin_test.go
index fa5261dbe..beee9a2e7 100644
--- a/pkg/operator/ceph/config/keyring/admin_test.go
+++ b/pkg/operator/ceph/config/keyring/admin_test.go
@@ -22,7 +22,6 @@ import (
 	"path"
 	"testing"
 
-	v1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
 	"github.com/rook/rook/pkg/clusterd"
 	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
 	clienttest "github.com/rook/rook/pkg/daemon/ceph/client/test"
@@ -54,18 +53,13 @@ func TestAdminKeyringStore(t *testing.T) {
 
 	// create key
 	clusterInfo.CephCred.Secret = "adminsecretkey"
-	err := k.Admin().CreateOrUpdate(clusterInfo, ctx, v1.AnnotationsSpec{v1.KeyClusterMetadata: v1.Annotations{"key": "value"}})
+	err := k.Admin().CreateOrUpdate(clusterInfo)
 	assert.NoError(t, err)
 	assertKeyringData(fmt.Sprintf(adminKeyringTemplate, "adminsecretkey"))
 
-	// test annotation
-	secret, err := ctx.Clientset.CoreV1().Secrets(clusterInfo.Namespace).Get(clusterInfo.Context, keyringSecretName(adminKeyringResourceName), metav1.GetOptions{})
-	assert.NoError(t, err)
-	assert.Equal(t, map[string]string{"key": "value"}, secret.Annotations)
-
 	// update key
 	clusterInfo.CephCred.Secret = "differentsecretkey"
-	err = k.Admin().CreateOrUpdate(clusterInfo, ctx, v1.AnnotationsSpec{})
+	err = k.Admin().CreateOrUpdate(clusterInfo)
 	assert.NoError(t, err)
 	assertKeyringData(fmt.Sprintf(adminKeyringTemplate, "differentsecretkey"))
 }
@@ -80,7 +74,7 @@ func TestAdminVolumeAndMount(t *testing.T) {
 	s := GetSecretStore(ctx, clusterInfo, ownerInfo)
 
 	clusterInfo.CephCred.Secret = "adminsecretkey"
-	err := s.Admin().CreateOrUpdate(clusterInfo, ctx, v1.AnnotationsSpec{})
+	err := s.Admin().CreateOrUpdate(clusterInfo)
 	assert.NoError(t, err)
 
 	v := Volume().Admin()
diff --git a/pkg/operator/ceph/csi/version_test.go b/pkg/operator/ceph/csi/version_test.go
index 2cca86b9f..c69ae0c4c 100644
--- a/pkg/operator/ceph/csi/version_test.go
+++ b/pkg/operator/ceph/csi/version_test.go
@@ -23,9 +23,9 @@ import (
 )
 
 var (
-	testMinVersion         = CephCSIVersion{3, 4, 0}
+	testMinVersion         = CephCSIVersion{3, 3, 0}
+	testReleaseV330        = CephCSIVersion{3, 3, 0}
 	testReleaseV340        = CephCSIVersion{3, 4, 0}
-	testReleaseV350        = CephCSIVersion{3, 5, 0}
 	testVersionUnsupported = CephCSIVersion{4, 0, 0}
 )
 
@@ -40,12 +40,12 @@ func TestIsAtLeast(t *testing.T) {
 	assert.Equal(t, true, ret)
 
 	// Test version which is equal
-	ret = testReleaseV340.isAtLeast(&testReleaseV340)
+	ret = testReleaseV330.isAtLeast(&testReleaseV330)
 	assert.Equal(t, true, ret)
 
-	// Test for 3.5.0
+	// Test for 3.4.0
 	// Test version which is lesser
-	ret = testReleaseV350.isAtLeast(&testReleaseV340)
+	ret = testReleaseV340.isAtLeast(&testReleaseV330)
 	assert.Equal(t, true, ret)
 
 }
@@ -58,10 +58,10 @@ func TestSupported(t *testing.T) {
 	ret = testVersionUnsupported.Supported()
 	assert.Equal(t, false, ret)
 
-	ret = testReleaseV340.Supported()
+	ret = testReleaseV330.Supported()
 	assert.Equal(t, true, ret)
 
-	ret = testReleaseV350.Supported()
+	ret = testReleaseV340.Supported()
 	assert.Equal(t, true, ret)
 }