Repository: rook/rook
Pull request #13997 (2024-03-29 20:52:20)

doc: update release notes and upgrade docs for v1.14
Update pending release notes, upgrade docs, and supplementary documentation relevant for upgrades for the upcoming Rook v1.14 release.

<!-- Thank you for contributing to Rook! -->

<!-- STEPS TO FOLLOW:
  1. Add a description of the changes (frequently the same as the commit description)
  2. Enter the issue number next to "Resolves #" below (if there is no tracking issue resolved, **remove that section**)
  3. Review our Contributing documentation at https://rook.io/docs/rook/latest/Contributing/development-flow/
  4. Follow the steps in the checklist below, starting with the **Commit Message Formatting**.
-->

<!-- Uncomment this section with the issue number if an issue is being resolved
**Issue resolved by this Pull Request:**
Resolves #
--->

**Checklist:**

- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/latest/Contributing/development-flow/#commit-structure).
- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/latest/Contributing/development-flow/#submitting-a-pull-request)
- [ ] [Pending release notes](https://github.com/rook/rook/blob/master/PendingReleaseNotes.md) updated with breaking and/or notable changes for the next minor release.
- [ ] Documentation has been updated, if necessary.
- [ ] Unit tests have been added, if necessary.
- [ ] Integration tests have been added, if necessary.



Patch:
diff --git a/.github/workflows/commitlint.yml b/.github/workflows/commitlint.yml
index 26bc72368..28ab3af37 100644
--- a/.github/workflows/commitlint.yml
+++ b/.github/workflows/commitlint.yml
@@ -31,7 +31,7 @@ jobs:
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
-      - uses: wagoid/commitlint-github-action@v6.0.0
+      - uses: wagoid/commitlint-github-action@v5.4.5
         with:
           configFile: "./.commitlintrc.json"
           helpURL: https://rook.io/docs/rook/latest/Contributing/development-flow/#commit-structure
diff --git a/Documentation/CRDs/Cluster/ceph-cluster-crd.md b/Documentation/CRDs/Cluster/ceph-cluster-crd.md
index ae3c2deed..d32edd605 100755
--- a/Documentation/CRDs/Cluster/ceph-cluster-crd.md
+++ b/Documentation/CRDs/Cluster/ceph-cluster-crd.md
@@ -195,11 +195,11 @@ Configure the network that will be enabled for the cluster and services.
 
 * `provider`: Specifies the network provider that will be used to connect the network interface. You can choose between `host`, and `multus`.
 * `selectors`: Used for `multus` provider only. Select NetworkAttachmentDefinitions to use for Ceph networks.
-    * `public`: Select the NetworkAttachmentDefinition to use for the public network.
-    * `cluster`: Select the NetworkAttachmentDefinition to use for the cluster network.
+  * `public`: Select the NetworkAttachmentDefinition to use for the public network.
+  * `cluster`: Select the NetworkAttachmentDefinition to use for the cluster network.
 * `addressRanges`: Used for `host` or `multus` providers only. Allows overriding the address ranges (CIDRs) that Ceph will listen on.
-    * `public`: A list of individual network ranges in CIDR format to use for Ceph's public network.
-    * `cluster`: A list of individual network ranges in CIDR format to use for Ceph's cluster network.
+  * `public`: A list of individual network ranges in CIDR format to use for Ceph's public network.
+  * `cluster`: A list of individual network ranges in CIDR format to use for Ceph's cluster network.
 * `ipFamily`: Specifies the network stack Ceph daemons should listen on.
 * `dualStack`: Specifies that Ceph daemon should listen on both IPv4 and IPv6 network stacks.
 * `connections`: Settings for network connections using Ceph's msgr2 protocol
diff --git a/Documentation/CRDs/Cluster/network-providers.md b/Documentation/CRDs/Cluster/network-providers.md
index f98f44eea..bb3752cbe 100644
--- a/Documentation/CRDs/Cluster/network-providers.md
+++ b/Documentation/CRDs/Cluster/network-providers.md
@@ -155,7 +155,7 @@ understand and implement these requirements.
     need to be an order of magnitude larger (or more) than the host address space to allow the
     storage cluster to grow in the future.
 
-If these prerequisites are not achievable, the remaining option is to set the Rook operator config
+If these prerequisites are not achievable, plan to set the Rook operator config
 `CSI_ENABLE_HOST_NETWORK: "false"` as documented in the [CSI Host Networking](#csi-host-networking)
 section.
 
@@ -418,6 +418,24 @@ spec:
     }'
 ```
 
+## Holder Pod Deprecation
+
+Rook plans to remove CSI "holder" pods in Rook v1.16. CephCluster with `csi-*plugin-holder-*` pods
+present in the Rook operator namespace must plan to set `CSI_DISABLE_HOLDER_PODS` to `"true"` after
+Rook v1.14 is installed and before v1.16 is installed by following the migration sections below.
+CephClusters with no holder pods do not need to follow migration steps.
+
+CephClusters that do not use `network.provider: multus` can follow the
+[Disabling Holder Pods](#disabling-holder-pods) section.
+
+CephClusters that use `network.provider: multus` will need to plan the migration more carefully.
+Read the [Disabling Holder Pods with Multus and CSI Host Networking](#disabling-holder-pods-with-multus-and-csi-host-networking)
+section. Decide whether to use CSI host networking or not following the sections below.
+
+!!! hint
+    To determine if holder pods are deployed, use
+    `kubectl --namespace $ROOK_OPERATOR get pods | grep plugin-holder`
+
 ## Modifying CSI networking
 
 ### Disabling Holder Pods with Multus and CSI Host Networking
@@ -515,7 +533,8 @@ If any CephClusters have Multus enabled (`network.provider: "multus"`), follow t
 steps above before continuing.
 
 **Step 2**
-Begin by setting `CSI_DISABLE_HOLDER_PODS: "true"` (and `CSI_ENABLE_HOST_NETWORK: "true"` if desired).
+Begin by setting `CSI_DISABLE_HOLDER_PODS: "true"` -- also set the desired value of
+`CSI_ENABLE_HOST_NETWORK` if needed.
 
 After this, `csi-*plugin-*` pods will restart, and `csi-*plugin-holder-*` pods will remain running.
 
diff --git a/Documentation/Helm-Charts/operator-chart.md b/Documentation/Helm-Charts/operator-chart.md
index c8905f381..4a53af3ef 100644
--- a/Documentation/Helm-Charts/operator-chart.md
+++ b/Documentation/Helm-Charts/operator-chart.md
@@ -59,7 +59,7 @@ The following table lists the configurable parameters of the rook-operator chart
 | `csi.cephFSKernelMountOptions` | Set CephFS Kernel mount options to use https://docs.ceph.com/en/latest/man/8/mount.ceph/#options. Set to "ms_mode=secure" when connections.encrypted is enabled in CephCluster CR | `nil` |
 | `csi.cephFSPluginUpdateStrategy` | CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate | `RollingUpdate` |
 | `csi.cephFSPluginUpdateStrategyMaxUnavailable` | A maxUnavailable parameter of CSI cephFS plugin daemonset update strategy. | `1` |
-| `csi.cephcsi.image` | Ceph CSI image | `quay.io/cephcsi/cephcsi:v3.11.0` |
+| `csi.cephcsi.image` | Ceph CSI image | `quay.io/cephcsi/cephcsi:v3.10.2` |
 | `csi.cephfsLivenessMetricsPort` | CSI CephFS driver metrics port | `9081` |
 | `csi.cephfsPodLabels` | Labels to add to the CSI CephFS Deployments and DaemonSets Pods | `nil` |
 | `csi.clusterName` | Cluster name identifier to set as metadata on the CephFS subvolume and RBD images. This will be useful in cases like for example, when two container orchestrator clusters (Kubernetes/OCP) are using a single ceph cluster | `nil` |
@@ -81,7 +81,7 @@ The following table lists the configurable parameters of the rook-operator chart
 | `csi.csiRBDPluginVolumeMount` | The volume mounts of the CephCSI RBD plugin DaemonSet | `nil` |
 | `csi.csiRBDProvisionerResource` | CEPH CSI RBD provisioner resource requirement list csi-omap-generator resources will be applied only if `enableOMAPGenerator` is set to `true` | see values.yaml |
 | `csi.disableCsiDriver` | Disable the CSI driver. | `"false"` |
-| `csi.disableHolderPods` | Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network without needing hosts to the network. Holder pods are being deprecated. See issue for details: https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true". | `true` |
+| `csi.disableHolderPods` | Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network without needing hosts to the network. Holder pods are being removed. See issue for details: https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true". | `true` |
 | `csi.enableCSIEncryption` | Enable Ceph CSI PVC encryption support | `false` |
 | `csi.enableCSIHostNetwork` | Enable host networking for CSI CephFS and RBD nodeplugins. This may be necessary in some network configurations where the SDN does not provide access to an external cluster or there is significant drop in read/write performance | `true` |
 | `csi.enableCephfsDriver` | Enable Ceph CSI CephFS driver | `true` |
diff --git a/Documentation/Storage-Configuration/Ceph-CSI/custom-images.md b/Documentation/Storage-Configuration/Ceph-CSI/custom-images.md
index 834728149..b63ddb073 100644
--- a/Documentation/Storage-Configuration/Ceph-CSI/custom-images.md
+++ b/Documentation/Storage-Configuration/Ceph-CSI/custom-images.md
@@ -18,7 +18,7 @@ kubectl -n $ROOK_OPERATOR_NAMESPACE edit configmap rook-ceph-operator-config
 The default upstream images are included below, which you can change to your desired images.
 
 ```yaml
-ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.11.0"
+ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.10.2"
 ROOK_CSI_REGISTRAR_IMAGE: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0"
 ROOK_CSI_PROVISIONER_IMAGE: "registry.k8s.io/sig-storage/csi-provisioner:v4.0.0"
 ROOK_CSI_ATTACHER_IMAGE: "registry.k8s.io/sig-storage/csi-attacher:v4.5.0"
@@ -32,7 +32,7 @@ ROOK_CSIADDONS_IMAGE: "quay.io/csiaddons/k8s-sidecar:v0.8.0"
 If image version is not passed along with the image name in any of the variables above,
 Rook will add the corresponding default version to that image.
 Example: if `ROOK_CSI_CEPH_IMAGE: "quay.io/private-repo/cephcsi"` is passed,
-Rook will add internal default version and consume it as `"quay.io/private-repo/cephcsi:v3.11.0"`.
+Rook will add internal default version and consume it as `"quay.io/private-repo/cephcsi:v3.10.2"`.
 
 ### **Use default images**
 
diff --git a/Documentation/Upgrade/rook-upgrade.md b/Documentation/Upgrade/rook-upgrade.md
index ad1b4aba6..8819f1847 100644
--- a/Documentation/Upgrade/rook-upgrade.md
+++ b/Documentation/Upgrade/rook-upgrade.md
@@ -14,7 +14,7 @@ We welcome feedback and opening issues!
 
 ## Supported Versions
 
-This guide is for upgrading from **Rook v1.12.x to Rook v1.13.x**.
+This guide is for upgrading from **Rook v1.13.x to Rook v1.14.x**.
 
 Please refer to the upgrade guides from previous releases for supported upgrade paths.
 Rook upgrades are only supported between official releases.
@@ -22,6 +22,7 @@ Rook upgrades are only supported between official releases.
 For a guide to upgrade previous versions of Rook, please refer to the version of documentation for
 those releases.
 
+* [Upgrade 1.12 to 1.13](https://rook.io/docs/rook/v1.13/Upgrade/rook-upgrade/)
 * [Upgrade 1.11 to 1.12](https://rook.io/docs/rook/v1.12/Upgrade/rook-upgrade/)
 * [Upgrade 1.10 to 1.11](https://rook.io/docs/rook/v1.11/Upgrade/rook-upgrade/)
 * [Upgrade 1.9 to 1.10](https://rook.io/docs/rook/v1.10/Upgrade/rook-upgrade/)
@@ -47,22 +48,18 @@ those releases.
     official releases. Builds from the master branch can have functionality changed or removed at any
     time without compatibility support and without prior notice.
 
-## Breaking changes in v1.13
-
-* The minimum supported version of Kubernetes is v1.23.
-  Upgrade to Kubernetes v1.23 or higher before upgrading Rook.
-* The minimum supported version of Ceph is v17.2.0.
-  If a lower version is currently deployed, [Upgrade Ceph](./ceph-upgrade.md) before upgrading Rook.
-* CephCSI CephFS driver introduced a breaking change in v3.9.0. If any existing CephFS storageclass in
-  the cluster has `MountOptions` parameter set, follow the steps mentioned in the
-  [CephCSI upgrade guide](https://github.com/ceph/ceph-csi/blob/v3.9.0/docs/ceph-csi-upgrade.md/#upgrading-cephfs)
-  to ensure a smooth upgrade.
-  This became the default CSI version in Rook v1.12.1, and may have already been resolved.
-* Support for the admission controller has been removed. CRD validation is now enabled with
-  [Validating Admission Policies](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/).
-  Validating Admission Policy rules are ignored in Kubernetes v1.24 and lower.
-  If the admission controller is enabled, it is advised to upgrade to Kubernetes v1.25 or higher before upgrading Rook.
-  For more info, see https://github.com/rook/rook/pull/11532.
+## Breaking changes in v1.14
+
+* The minimum supported version of Kubernetes is v1.25.
+  Upgrade to Kubernetes v1.25 or higher before upgrading Rook.
+* The Rook operator config `CSI_ENABLE_READ_AFFINITY` was removed. v1.13 clusters that have modified
+  this value to be `"true"` must set the option as desired in each CephCluster as documented
+  [here](https://rook.github.io/docs/rook/v1.14/CRDs/Cluster/ceph-cluster-crd/#csi-driver-options)
+  before upgrading to v1.14.
+* Rook is beginning the process of deprecating CSI network "holder" pods.
+  If there are pods named `csi-*plugin-holder-*` in the Rook operator namespace, see the
+  [detailed documentation](../CRDs/Cluster/network-providers.md#holder-pod-deprecation)
+  to disable them. This is optional for v1.14, but will be required in a future release.
 
 ## Considerations
 
@@ -78,11 +75,11 @@ With this upgrade guide, there are a few notes to consider:
 
 Unless otherwise noted due to extenuating requirements, upgrades from one patch release of Rook to
 another are as simple as updating the common resources and the image of the Rook operator. For
-example, when Rook v1.13.1 is released, the process of updating from v1.13.0 is as simple as running
+example, when Rook v1.14.1 is released, the process of updating from v1.14.0 is as simple as running
 the following:
 
 ```console
-git clone --single-branch --depth=1 --branch v1.13.1 https://github.com/rook/rook.git
+git clone --single-branch --depth=1 --branch v1.14.1 https://github.com/rook/rook.git
 cd rook/deploy/examples
 ```
 
@@ -90,11 +87,11 @@ If the Rook Operator or CephCluster are deployed into a different namespace than
 `rook-ceph`, see the [Update common resources and CRDs](#1-update-common-resources-and-crds)
 section for instructions on how to change the default namespaces in `common.yaml`.
 
-Then, apply the latest changes from v1.13, and update the Rook Operator image.
+Then, apply the latest changes from v1.14, and update the Rook Operator image.
 
 ```console
 kubectl apply -f common.yaml -f crds.yaml
-kubectl -n rook-ceph set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.13.1
+kubectl -n rook-ceph set image deploy/rook-ceph-operator rook-ceph-operator=rook/ceph:v1.14.1
 ```
 
 As exemplified above, it is a good practice to update Rook common resources from the example
@@ -109,6 +106,10 @@ Also update optional resources like Prometheus monitoring noted more fully in th
 If Rook is installed via the Helm chart, Helm will handle some details of the upgrade itself.
 The upgrade steps in this guide will clarify what Helm handles automatically.
 
+!!! important
+    If there are pods named `csi-*plugin-holder-*` in the Rook operator namespace, set the new
+    config `disableHolderPods: false` in the values.yaml before upgrading to v1.14.
+
 The `rook-ceph` helm chart upgrade performs the Rook upgrade.
 The `rook-ceph-cluster` helm chart upgrade performs a [Ceph upgrade](#ceph-version-upgrades) if the Ceph image is updated.
 The `rook-ceph` chart should be upgraded before `rook-ceph-cluster`, so the latest operator has the opportunity to update
@@ -128,9 +129,9 @@ In order to successfully upgrade a Rook cluster, the following prerequisites mus
 
 ## Rook Operator Upgrade
 
-The examples given in this guide upgrade a live Rook cluster running `v1.12.9` to
-the version `v1.13.0`. This upgrade should work from any official patch release of Rook v1.12 to any
-official patch release of v1.13.
+The examples given in this guide upgrade a live Rook cluster running `v1.13.7` to
+the version `v1.14.0`. This upgrade should work from any official patch release of Rook v1.13 to any
+official patch release of v1.14.
 
 Let's get started!
 
@@ -191,7 +192,7 @@ kubectl apply -f deploy/examples/monitoring/rbac.yaml
 !!! hint
     The operator is automatically updated when using Helm charts.
 
-The largest portion of the upgrade is triggered when the operator's image is updated to `v1.13.x`.
+The largest portion of the upgrade is triggered when the operator's image is updated to `v1.14.x`.
 When the operator is updated, it will proceed to update all of the Ceph daemons.
 
 ```console
@@ -225,18 +226,18 @@ watch --exec kubectl -n $ROOK_CLUSTER_NAMESPACE get deployments -l rook_cluster=
 ```
 
 As an example, this cluster is midway through updating the OSDs. When all deployments report `1/1/1`
-availability and `rook-version=v1.13.0`, the Ceph cluster's core components are fully updated.
+availability and `rook-version=v1.14.0`, the Ceph cluster's core components are fully updated.
 
 ```console
 Every 2.0s: kubectl -n rook-ceph get deployment -o j...
 
-rook-ceph-mgr-a         req/upd/avl: 1/1/1      rook-version=v1.13.0
-rook-ceph-mon-a         req/upd/avl: 1/1/1      rook-version=v1.13.0
-rook-ceph-mon-b         req/upd/avl: 1/1/1      rook-version=v1.13.0
-rook-ceph-mon-c         req/upd/avl: 1/1/1      rook-version=v1.13.0
-rook-ceph-osd-0         req/upd/avl: 1//        rook-version=v1.13.0
-rook-ceph-osd-1         req/upd/avl: 1/1/1      rook-version=v1.12.9
-rook-ceph-osd-2         req/upd/avl: 1/1/1      rook-version=v1.12.9
+rook-ceph-mgr-a         req/upd/avl: 1/1/1      rook-version=v1.14.0
+rook-ceph-mon-a         req/upd/avl: 1/1/1      rook-version=v1.14.0
+rook-ceph-mon-b         req/upd/avl: 1/1/1      rook-version=v1.14.0
+rook-ceph-mon-c         req/upd/avl: 1/1/1      rook-version=v1.14.0
+rook-ceph-osd-0         req/upd/avl: 1//        rook-version=v1.14.0
+rook-ceph-osd-1         req/upd/avl: 1/1/1      rook-version=v1.13.7
+rook-ceph-osd-2         req/upd/avl: 1/1/1      rook-version=v1.13.7
 ```
 
 An easy check to see if the upgrade is totally finished is to check that there is only one
@@ -245,23 +246,21 @@ An easy check to see if the upgrade is totally finished is to check that there i
 ```console
 # kubectl -n $ROOK_CLUSTER_NAMESPACE get deployment -l rook_cluster=$ROOK_CLUSTER_NAMESPACE -o jsonpath='{range .items[*]}{"rook-version="}{.metadata.labels.rook-version}{"\n"}{end}' | sort | uniq
 This cluster is not yet finished:
-  rook-version=v1.12.9
-  rook-version=v1.13.0
+  rook-version=v1.13.7
+  rook-version=v1.14.0
 This cluster is finished:
-  rook-version=v1.13.0
+  rook-version=v1.14.0
 ```
 
 ### **5. Verify the updated cluster**
 
-At this point, the Rook operator should be running version `rook/ceph:v1.13.0`.
+At this point, the Rook operator should be running version `rook/ceph:v1.14.0`.
 
 Verify the CephCluster health using the [health verification doc](health-verification.md).
 
-### **6. Disable CSI holder pods**
-CSI "holder" pods are frequently reported objects of confusion and struggle in Rook. Because of
-this, they are being deprecated and will be removed in Rook v1.16.
+### **6. Disable holder pods**
 
-If there are any CephClusters that use the non-default network setting `network.provider: "multus"`,
-or if the operator config `CSI_ENABLE_HOST_NETWORK` is set to `"false"`, perform migration steps to
-remove holder pods by setting `CSI_REMOVE_HOLDER_PODS: "true"` after following this migration guide:
-[Modifying CSI Networking](./../CRDs/Cluster/network-providers.md#modifying-csi-networking)
+Rook is beginning the process of deprecating CSI network "holder" pods. If there are pods named
+`csi-*plugin-holder-*` in the Rook operator namespace, see the
+[detailed documentation](../CRDs/Cluster/network-providers.md#holder-pod-deprecation)
+to disable them. This is optional for v1.14, but will be required in a future release.
diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md
index 726825351..41f12b6b9 100644
--- a/PendingReleaseNotes.md
+++ b/PendingReleaseNotes.md
@@ -2,15 +2,16 @@
 
 ## Breaking Changes
 
-- The removal of `CSI_ENABLE_READ_AFFINITY` option and its replacement with per-cluster
-read affinity setting in cephCluster CR (CSIDriverOptions section) in [PR](https://github.com/rook/rook/pull/13665)
-- updating `netNamespaceFilePath` for all clusterIDs in rook-ceph-csi-config configMap in [PR](https://github.com/rook/rook/pull/13613)
-  - Issue: The netNamespaceFilePath isn't updated in the CSI config map for all the clusterIDs when `CSI_ENABLE_HOST_NETWORK` is set to false in `operator.yaml`
-  - Impact: This results in the unintended network configurations, with pods using the host networking instead of pod networking.
-- Rook is beginning the process of deprecating holder pods. This is especially important for Multus
-  users. Helm chart users should take care to set the new config `disableHolderPods: false` if they
-  are using Multus and still using the `rook-ceph` chart's default `values.yaml`. Special upgrade
-  docs for multus can be found [here](Documentation/CRDs/Cluster/network-providers.md#migrating-to-remove-multus-holder-pods).
+- The minimum supported version of Kubernetes is v1.25.
+  Upgrade to Kubernetes v1.25 or higher before upgrading Rook.
+- The Rook operator config `CSI_ENABLE_READ_AFFINITY` was removed. v1.13 clusters that have modified
+  this value to be `"true"` must set the option as desired in each CephCluster as documented
+  [here](https://rook.github.io/docs/rook/v1.14/CRDs/Cluster/ceph-cluster-crd/#csi-driver-options)
+  before upgrading to v1.14.
+- Rook is beginning the process of deprecating CSI network "holder" pods.
+  If there are pods named `csi-*plugin-holder-*` in the Rook operator namespace, see the
+  [detailed documentation](./Documentation/CRDs/Cluster/network-providers.md#holder-pod-deprecation)
+  to disable them. This is optional for v1.14, but will be required in a future release.
 
 ## Features
 
diff --git a/deploy/charts/rook-ceph/values.yaml b/deploy/charts/rook-ceph/values.yaml
index a0decbd12..487f0d1df 100644
--- a/deploy/charts/rook-ceph/values.yaml
+++ b/deploy/charts/rook-ceph/values.yaml
@@ -89,7 +89,7 @@ csi:
   # there is significant drop in read/write performance
   enableCSIHostNetwork: true
   # -- Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network
-  # without needing hosts to the network. Holder pods are being deprecated. See issue for details:
+  # without needing hosts to the network. Holder pods are being removed. See issue for details:
   # https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true".
   disableHolderPods: true
   # -- Enable Snapshotter in CephFS provisioner pod
@@ -479,7 +479,7 @@ csi:
 
   cephcsi:
     # -- Ceph CSI image
-    # @default -- `quay.io/cephcsi/cephcsi:v3.11.0`
+    # @default -- `quay.io/cephcsi/cephcsi:v3.10.2`
     image:
 
   registrar:
diff --git a/deploy/examples/images.txt b/deploy/examples/images.txt
index a576d55ca..4d4ae0f49 100644
--- a/deploy/examples/images.txt
+++ b/deploy/examples/images.txt
@@ -1,7 +1,7 @@
  gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20230130-v0.1.0-24-gc0cf995
  quay.io/ceph/ceph:v18.2.2
  quay.io/ceph/cosi:v0.1.1
- quay.io/cephcsi/cephcsi:v3.11.0
+ quay.io/cephcsi/cephcsi:v3.10.2
  quay.io/csiaddons/k8s-sidecar:v0.8.0
  registry.k8s.io/sig-storage/csi-attacher:v4.5.0
  registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0
diff --git a/deploy/examples/operator-openshift.yaml b/deploy/examples/operator-openshift.yaml
index eb53859a9..4d6a03d5a 100644
--- a/deploy/examples/operator-openshift.yaml
+++ b/deploy/examples/operator-openshift.yaml
@@ -140,7 +140,7 @@ data:
   # CSI_ENABLE_HOST_NETWORK: "true"
 
   # Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network
-  # without needing hosts to the network. Holder pods are being deprecated. See issue for details:
+  # without needing hosts to the network. Holder pods are being removed. See issue for details:
   # https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true".
   CSI_DISABLE_HOLDER_PODS: "true"
 
@@ -197,7 +197,7 @@ data:
   # The default version of CSI supported by Rook will be started. To change the version
   # of the CSI driver to something other than what is officially supported, change
   # these images to the desired release of the CSI driver.
-  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.11.0"
+  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.10.2"
   # ROOK_CSI_REGISTRAR_IMAGE: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0"
   # ROOK_CSI_RESIZER_IMAGE: "registry.k8s.io/sig-storage/csi-resizer:v1.10.0"
   # ROOK_CSI_PROVISIONER_IMAGE: "registry.k8s.io/sig-storage/csi-provisioner:v4.0.0"
diff --git a/deploy/examples/operator.yaml b/deploy/examples/operator.yaml
index 84a2fcf1a..8bc5b6ca1 100644
--- a/deploy/examples/operator.yaml
+++ b/deploy/examples/operator.yaml
@@ -47,7 +47,7 @@ data:
   # CSI_ENABLE_HOST_NETWORK: "true"
 
   # Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network
-  # without needing hosts to the network. Holder pods are being deprecated. See issue for details:
+  # without needing hosts to the network. Holder pods are being removed. See issue for details:
   # https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true".
   CSI_DISABLE_HOLDER_PODS: "true"
 
@@ -122,7 +122,7 @@ data:
   # The default version of CSI supported by Rook will be started. To change the version
   # of the CSI driver to something other than what is officially supported, change
   # these images to the desired release of the CSI driver.
-  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.11.0"
+  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.10.2"
   # ROOK_CSI_REGISTRAR_IMAGE: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0"
   # ROOK_CSI_RESIZER_IMAGE: "registry.k8s.io/sig-storage/csi-resizer:v1.10.0"
   # ROOK_CSI_PROVISIONER_IMAGE: "registry.k8s.io/sig-storage/csi-provisioner:v4.0.0"
diff --git a/go.mod b/go.mod
index 3a1701ebb..8d9b1dddf 100644
--- a/go.mod
+++ b/go.mod
@@ -14,7 +14,7 @@ replace (
 
 require (
 	github.com/IBM/keyprotect-go-client v0.12.2
-	github.com/aws/aws-sdk-go v1.51.11
+	github.com/aws/aws-sdk-go v1.51.7
 	github.com/banzaicloud/k8s-objectmatcher v1.8.0
 	github.com/ceph/go-ceph v0.26.0
 	github.com/coreos/pkg v0.0.0-20230601102743-20bbbf26f4d8
diff --git a/go.sum b/go.sum
index f346002c2..7b351b756 100644
--- a/go.sum
+++ b/go.sum
@@ -138,8 +138,8 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkY
 github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
 github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
 github.com/aws/aws-sdk-go v1.44.164/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI=
-github.com/aws/aws-sdk-go v1.51.11 h1:El5VypsMIz7sFwAAj/j06JX9UGs4KAbAIEaZ57bNY4s=
-github.com/aws/aws-sdk-go v1.51.11/go.mod h1:LF8svs817+Nz+DmiMQKTO3ubZ/6IaTpq3TjupRn3Eqk=
+github.com/aws/aws-sdk-go v1.51.7 h1:RRjxHhx9RCjw5AhgpmmShq3F4JDlleSkyhYMQ2xUAe8=
+github.com/aws/aws-sdk-go v1.51.7/go.mod h1:LF8svs817+Nz+DmiMQKTO3ubZ/6IaTpq3TjupRn3Eqk=
 github.com/banzaicloud/k8s-objectmatcher v1.8.0 h1:Nugn25elKtPMTA2br+JgHNeSQ04sc05MDPmpJnd1N2A=
 github.com/banzaicloud/k8s-objectmatcher v1.8.0/go.mod h1:p2LSNAjlECf07fbhDyebTkPUIYnU05G+WfGgkTmgeMg=
 github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
diff --git a/pkg/operator/ceph/csi/spec.go b/pkg/operator/ceph/csi/spec.go
index 0a93c166e..1a8c5bba8 100644
--- a/pkg/operator/ceph/csi/spec.go
+++ b/pkg/operator/ceph/csi/spec.go
@@ -139,7 +139,7 @@ var (
 // manually challenging.
 var (
 	// image names
-	DefaultCSIPluginImage   = "quay.io/cephcsi/cephcsi:v3.11.0"
+	DefaultCSIPluginImage   = "quay.io/cephcsi/cephcsi:v3.10.2"
 	DefaultRegistrarImage   = "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0"
 	DefaultProvisionerImage = "registry.k8s.io/sig-storage/csi-provisioner:v4.0.0"
 	DefaultAttacherImage    = "registry.k8s.io/sig-storage/csi-attacher:v4.5.0"
diff --git a/pkg/operator/ceph/csi/version.go b/pkg/operator/ceph/csi/version.go
index a3447f4bf..01b10f5bb 100644
--- a/pkg/operator/ceph/csi/version.go
+++ b/pkg/operator/ceph/csi/version.go
@@ -25,14 +25,14 @@ import (
 )
 
 var (
-	//minimum supported version is 3.10.0
-	minimum = CephCSIVersion{3, 10, 0}
+	//minimum supported version is 3.9.0
+	minimum = CephCSIVersion{3, 9, 0}
 	//supportedCSIVersions are versions that rook supports
-	releasev311 = CephCSIVersion{3, 11, 0}
+	releasev310 = CephCSIVersion{3, 10, 0}
 
 	supportedCSIVersions = []CephCSIVersion{
 		minimum,
-		releasev311,
+		releasev310,
 	}
 
 	// for parsing the output of `cephcsi`
diff --git a/pkg/operator/ceph/pool/controller.go b/pkg/operator/ceph/pool/controller.go
index 8d7981f09..394d2b4e7 100644
--- a/pkg/operator/ceph/pool/controller.go
+++ b/pkg/operator/ceph/pool/controller.go
@@ -222,7 +222,6 @@ func (r *ReconcileCephBlockPool) reconcile(request reconcile.Request) (reconcile
 		}
 	}
 
-	poolSpec := cephBlockPool.ToNamedPoolSpec()
 	// DELETE: the CR was deleted
 	if !cephBlockPool.GetDeletionTimestamp().IsZero() {
 		deps, err := cephBlockPoolDependents(r.context, r.clusterInfo, cephBlockPool)
@@ -240,6 +239,7 @@ func (r *ReconcileCephBlockPool) reconcile(request reconcile.Request) (reconcile
 
 		r.recorder.Event(cephBlockPool, corev1.EventTypeNormal, string(cephv1.ReconcileStarted), "starting blockpool deletion")
 
+		poolSpec := cephBlockPool.ToNamedPoolSpec()
 		logger.Infof("deleting pool %q", poolSpec.Name)
 		err = deletePool(r.context, clusterInfo, &poolSpec)
 		if err != nil {
@@ -295,6 +295,8 @@ func (r *ReconcileCephBlockPool) reconcile(request reconcile.Request) (reconcile
 	if err := configureRBDStats(r.context, clusterInfo, ""); err != nil {
 		return reconcile.Result{}, *cephBlockPool, errors.Wrap(err, "failed to enable/disable stats collection for pool(s)")
 	}
+
+	poolSpec := cephBlockPool.ToNamedPoolSpec()
 	checker := newMirrorChecker(r.context, r.client, r.clusterInfo, request.NamespacedName, &poolSpec)
 	// ADD PEERS
 	logger.Debug("reconciling create rbd mirror peer configuration")
@@ -339,9 +341,9 @@ func (r *ReconcileCephBlockPool) reconcile(request reconcile.Request) (reconcile
 		// If not mirrored there is no Status Info field to fulfil
 	} else {
 		// disable mirroring
-		err := r.disableMirroring(poolSpec.Name)
+		err := r.disableMirroring(cephBlockPool)
 		if err != nil {
-			logger.Warningf("failed to disable mirroring on pool %q. %v", poolSpec.Name, err)
+			return reconcile.Result{}, *cephBlockPool, errors.Wrapf(err, "failed to disable mirroring on pool %q", cephBlockPool.Name)
 		}
 		// update ObservedGeneration in status at the end of reconcile
 		// Set Ready status, we are done reconciling
@@ -501,20 +503,20 @@ func (r *ReconcileCephBlockPool) cancelMirrorMonitoring(cephBlockPool *cephv1.Ce
 	}
 }
 
-func (r *ReconcileCephBlockPool) disableMirroring(pool string) error {
-	mirrorInfo, err := cephclient.GetPoolMirroringInfo(r.context, r.clusterInfo, pool)
+func (r *ReconcileCephBlockPool) disableMirroring(pool *cephv1.CephBlockPool) error {
+	mirrorInfo, err := cephclient.GetPoolMirroringInfo(r.context, r.clusterInfo, pool.Name)
 	if err != nil {
-		return errors.Wrapf(err, "failed to get mirroring info for the pool %q", pool)
+		return errors.Wrapf(err, "failed to get mirroring info for the pool %q", pool.Name)
 	}
 
 	if mirrorInfo.Mode == "image" {
-		mirroredPools, err := cephclient.GetMirroredPoolImages(r.context, r.clusterInfo, pool)
+		mirroredPools, err := cephclient.GetMirroredPoolImages(r.context, r.clusterInfo, pool.Name)
 		if err != nil {
-			return errors.Wrapf(err, "failed to list mirrored images for pool %q", pool)
+			return errors.Wrapf(err, "failed to list mirrored images for pool %q", pool.Name)
 		}
 
 		if len(*mirroredPools.Images) > 0 {
-			msg := fmt.Sprintf("there are images in the pool %q. Please manually disable mirroring for each image", pool)
+			msg := fmt.Sprintf("there are images in the pool %q. Please manually disable mirroring for each image", pool.Name)
 			logger.Errorf(msg)
 			return errors.New(msg)
 		}
@@ -523,20 +525,20 @@ func (r *ReconcileCephBlockPool) disableMirroring(pool string) error {
 	// Remove storage cluster peers
 	for _, peer := range mirrorInfo.Peers {
 		if peer.UUID != "" {
-			err := cephclient.RemoveClusterPeer(r.context, r.clusterInfo, pool, peer.UUID)
+			err := cephclient.RemoveClusterPeer(r.context, r.clusterInfo, pool.Name, peer.UUID)
 			if err != nil {
-				return errors.Wrapf(err, "failed to remove cluster peer with UUID %q for the pool %q", peer.UUID, pool)
+				return errors.Wrapf(err, "failed to remove cluster peer with UUID %q for the pool %q", peer.UUID, pool.Name)
 			}
-			logger.Infof("successfully removed peer site %q for the pool %q", peer.UUID, pool)
+			logger.Infof("successfully removed peer site %q for the pool %q", peer.UUID, pool.Name)
 		}
 	}
 
 	// Disable mirroring on pool
-	err = cephclient.DisablePoolMirroring(r.context, r.clusterInfo, pool)
+	err = cephclient.DisablePoolMirroring(r.context, r.clusterInfo, pool.Name)
 	if err != nil {
-		return errors.Wrapf(err, "failed to disable mirroring for pool %q", pool)
+		return errors.Wrapf(err, "failed to disable mirroring for pool %q", pool.Name)
 	}
-	logger.Infof("successfully disabled mirroring on the pool %q", pool)
+	logger.Infof("successfully disabled mirroring on the pool %q", pool.Name)
 
 	return nil
 }


Test patch:
diff --git a/pkg/operator/ceph/csi/util_test.go b/pkg/operator/ceph/csi/util_test.go
index 190524684..1ee73d58c 100644
--- a/pkg/operator/ceph/csi/util_test.go
+++ b/pkg/operator/ceph/csi/util_test.go
@@ -45,7 +45,7 @@ func TestDeploymentTemplate(t *testing.T) {
 }
 
 func TestGetPortFromConfig(t *testing.T) {
-	key := "TEST_CSI_PORT_ENV"
+	var key = "TEST_CSI_PORT_ENV"
 	var defaultPort uint16 = 8000
 	data := map[string]string{}
 
@@ -156,8 +156,7 @@ func Test_applyVolumeToPodSpec(t *testing.T) {
 	assert.Len(t, ds.Spec.Template.Spec.Volumes, defaultVolumes+2)
 	// override existing volume configuration
 	volumes[1].VolumeSource = corev1.VolumeSource{
-		HostPath: &corev1.HostPathVolumeSource{Path: "/run/test/run"},
-	}
+		HostPath: &corev1.HostPathVolumeSource{Path: "/run/test/run"}}
 	volumeRaw, err = yaml.Marshal(volumes)
 	assert.Nil(t, err)
 	config[configKey] = string(volumeRaw)
@@ -179,6 +178,7 @@ func Test_applyVolumeToPodSpec(t *testing.T) {
 	assert.Nil(t, err)
 	applyVolumeToPodSpec(config, configKey, &ds.Spec.Template.Spec)
 	assert.Len(t, ds.Spec.Template.Spec.Volumes, defaultVolumes+1)
+
 }
 
 func Test_applyVolumeMountToContainer(t *testing.T) {
@@ -248,6 +248,7 @@ func Test_applyVolumeMountToContainer(t *testing.T) {
 	assert.Nil(t, err)
 	applyVolumeMountToContainer(config, configKey, rbdContainerName, &ds.Spec.Template.Spec)
 	assert.Len(t, ds.Spec.Template.Spec.Containers[1].VolumeMounts, defaultVolumes+1)
+
 }
 
 func Test_getImage(t *testing.T) {
@@ -266,7 +267,7 @@ func Test_getImage(t *testing.T) {
 			args: args{
 				data:         map[string]string{},
 				settingName:  "ROOK_CSI_CEPH_IMAGE",
-				defaultImage: "quay.io/cephcsi/cephcsi:v3.11.0",
+				defaultImage: "quay.io/cephcsi/cephcsi:v3.10.2",
 			},
 			want: DefaultCSIPluginImage,
 		},
@@ -277,7 +278,7 @@ func Test_getImage(t *testing.T) {
 					"ROOK_CSI_CEPH_IMAGE": "registry.io/private/cephcsi:v8",
 				},
 				settingName:  "ROOK_CSI_CEPH_IMAGE",
-				defaultImage: "quay.io/cephcsi/cephcsi:v3.11.0",
+				defaultImage: "quay.io/cephcsi/cephcsi:v3.10.2",
 			},
 			want: "registry.io/private/cephcsi:v8",
 		},
@@ -288,9 +289,9 @@ func Test_getImage(t *testing.T) {
 					"ROOK_CSI_CEPH_IMAGE": "registry.io/private/cephcsi",
 				},
 				settingName:  "ROOK_CSI_CEPH_IMAGE",
-				defaultImage: "quay.io/cephcsi/cephcsi:v3.11.0",
+				defaultImage: "quay.io/cephcsi/cephcsi:v3.10.2",
 			},
-			want: "registry.io/private/cephcsi:v3.11.0",
+			want: "registry.io/private/cephcsi:v3.10.2",
 		},
 	}
 	for _, tt := range tests {
diff --git a/pkg/operator/ceph/csi/version_test.go b/pkg/operator/ceph/csi/version_test.go
index 9ccb79d4b..8877104fc 100644
--- a/pkg/operator/ceph/csi/version_test.go
+++ b/pkg/operator/ceph/csi/version_test.go
@@ -23,13 +23,14 @@ import (
 )
 
 var (
-	testMinVersion   = CephCSIVersion{3, 10, 0}
+	testMinVersion   = CephCSIVersion{3, 9, 0}
+	testReleaseV380  = CephCSIVersion{3, 8, 0}
+	testReleaseV381  = CephCSIVersion{3, 8, 1}
 	testReleaseV390  = CephCSIVersion{3, 9, 0}
 	testReleaseV391  = CephCSIVersion{3, 9, 1}
 	testreleasev310  = CephCSIVersion{3, 10, 0}
 	testReleaseV3101 = CephCSIVersion{3, 10, 1}
 	testReleaseV3102 = CephCSIVersion{3, 10, 2}
-	testReleaseV3110 = CephCSIVersion{3, 11, 0}
 
 	testVersionUnsupported = CephCSIVersion{4, 0, 0}
 )
@@ -44,12 +45,12 @@ func TestIsAtLeast(t *testing.T) {
 	ret = testMinVersion.isAtLeast(&testMinVersion)
 	assert.Equal(t, true, ret)
 
-	// Test for 3.9.0
-	ret = testReleaseV390.isAtLeast(&testMinVersion)
+	// Test for 3.8.0
+	ret = testReleaseV380.isAtLeast(&testMinVersion)
 	assert.Equal(t, false, ret)
 
-	// Test for 3.9.1
-	ret = testReleaseV391.isAtLeast(&testMinVersion)
+	// Test for 3.8.1
+	ret = testReleaseV381.isAtLeast(&testMinVersion)
 	assert.Equal(t, false, ret)
 
 	// Test for 3.10.0
@@ -64,10 +65,6 @@ func TestIsAtLeast(t *testing.T) {
 	ret = testReleaseV3102.isAtLeast(&testReleaseV3102)
 	assert.Equal(t, true, ret)
 
-	// Test for 3.11.0
-	ret = testReleaseV3110.isAtLeast(&testReleaseV3110)
-	assert.Equal(t, true, ret)
-
 }
 
 func TestSupported(t *testing.T) {
@@ -78,20 +75,23 @@ func TestSupported(t *testing.T) {
 	ret = testVersionUnsupported.Supported()
 	assert.Equal(t, false, ret)
 
-	// 3.9.x is not supported after 3.11.0 release
-	ret = testReleaseV390.Supported()
+	// 3.8.x is not supported after 3.10.0 release
+	ret = testReleaseV380.Supported()
 	assert.Equal(t, false, ret)
 
-	ret = testReleaseV391.Supported()
+	ret = testReleaseV381.Supported()
 	assert.Equal(t, false, ret)
 
-	ret = testreleasev310.Supported()
+	ret = testReleaseV390.Supported()
 	assert.Equal(t, true, ret)
 
-	ret = testReleaseV3101.Supported()
+	ret = testReleaseV391.Supported()
+	assert.Equal(t, true, ret)
+
+	ret = testreleasev310.Supported()
 	assert.Equal(t, true, ret)
 
-	ret = testReleaseV3110.Supported()
+	ret = testReleaseV3101.Supported()
 	assert.Equal(t, true, ret)
 }