Repository: rook/rook
Pull request #14819 (2024-10-08 17:23:15)

multus: finish deprecating holder pods
Finish the process of deprecating holder pods by removing Rook's ability to deploy them. The intent of this change is to make the most superficial changes possible to accomplish this. There are still remnants of code in Rook (particularly the CSI controller) that helped configure or deploy holder pods. Due to the risk of breaking some features, cleanup work of hose remnants will be deferred for future work.

Note for @Madhu-1 @subhamkrai @travisn others

I have essentially only allowed changes to cascade so far in this PR. I am particularly concerned about causing any breakage in logic that keeps the CSI config configmap updated. Since the CSI controller is expected to be removed in an upcoming release, it seems safest to only remove surface level code that has relation to holder pods and allow the rest of the code that supports holder deployment and config to be removed at that time. The only remaining bit of code related to holders is in the RadosNamespace and FSSubvolumeGroup CRs where they configure the net namespace. Otherwise, I believe (and intend) that all remaining holder-related code remains in the CSI controller.

I'm happy to take suggestions for other things to remove as long as the CSI config configmap isn't affected. As mentioned, I have strong reservations about the risk of breaking that code.

<!-- Thank you for contributing to Rook! -->

<!-- STEPS TO FOLLOW:
  1. Add a description of the changes (frequently the same as the commit description)
  2. Enter the issue number next to "Resolves #" below (if there is no tracking issue resolved, **remove that section**)
  3. Review our Contributing documentation at https://rook.io/docs/rook/latest/Contributing/development-flow/
  4. Follow the steps in the checklist below, starting with the **Commit Message Formatting**.
-->

<!-- Uncomment this section with the issue number if an issue is being resolved
**Issue resolved by this Pull Request:**
Resolves #
--->

**Checklist:**

- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/latest/Contributing/development-flow/#commit-structure).
- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/latest/Contributing/development-flow/#submitting-a-pull-request)
- [ ] [Pending release notes](https://github.com/rook/rook/blob/master/PendingReleaseNotes.md) updated with breaking and/or notable changes for the next minor release.
- [ ] Documentation has been updated, if necessary.
- [ ] Unit tests have been added, if necessary.
- [ ] Integration tests have been added, if necessary.


Madhu-1: @BlaineEXE I agree we have seen in the past many times that we failed to add/update all the required fields in the configmap and easy to cause this problem when we touch some part of it, as you mentioned it would be good to get rid of this code once we move to the csi-operator managing CSI
mergify[bot]: This pull request has merge conflicts that must be resolved before it can be merged. @BlaineEXE please rebase it. https://rook.io/docs/rook/latest/Contributing/development-flow/#updating-your-fork
BlaineEXE: I stripped out as much holder related code as I felt I could safely. The remaining code is (or should be) all in service of ensuring `pkg/operator/ceph/csi/cluster_config.go` code that manages the CSI config configmap (including removing net namespace file paths) remains working without accidental bugs added.

Patch:
diff --git a/.mergify.yml b/.mergify.yml
index 6dae0ac98..28efa4c2c 100644
--- a/.mergify.yml
+++ b/.mergify.yml
@@ -272,7 +272,7 @@ pull_request_rules:
       - "check-success=canary-tests / multi-cluster-mirroring (quay.io/ceph/ceph:v18)"
       - "check-success=canary-tests / rgw-multisite-testing (quay.io/ceph/ceph:v18)"
       - "check-success=canary-tests / encryption-pvc-kms-ibm-kp (quay.io/ceph/ceph:v18)"
-      - "check-success=canary-tests / multus-cluster-network (quay.io/ceph/ceph:v18)"
+      - "check-success=canary-tests / multus-cluster-network (quay.io/ceph/ceph:v18)" # note: changed name for 1.16 (multus-public-and-cluster)
       - "check-success=canary-tests / csi-hostnetwork-disabled (quay.io/ceph/ceph:v18)"
       - "check-success=TestCephSmokeSuite (v1.26.15)"
       - "check-success=TestCephSmokeSuite (v1.31.0)"
diff --git a/Documentation/CRDs/Cluster/network-providers.md b/Documentation/CRDs/Cluster/network-providers.md
index 142d8cda4..c38607e4e 100644
--- a/Documentation/CRDs/Cluster/network-providers.md
+++ b/Documentation/CRDs/Cluster/network-providers.md
@@ -414,182 +414,3 @@ spec:
       }
     }'
 ```
-
-## Holder Pod Deprecation
-
-Rook plans to remove CSI "holder" pods in Rook v1.16. CephCluster with `csi-*plugin-holder-*` pods
-present in the Rook operator namespace must plan to set `CSI_DISABLE_HOLDER_PODS` to `"true"` after
-Rook v1.14 is installed and before v1.16 is installed by following the migration sections below.
-CephClusters with no holder pods do not need to follow migration steps.
-
-Helm users will set `csi.disableHolderPods: true` in values.yaml instead of `CSI_DISABLE_HOLDER_PODS`.
-
-CephClusters that do not use `network.provider: multus` can follow the
-[Disabling Holder Pods](#disabling-holder-pods) section.
-
-CephClusters that use `network.provider: multus` will need to plan the migration more carefully.
-Read the [Disabling Holder Pods with Multus](#disabling-holder-pods-with-multus) section in full
-before beginning.
-
-!!! hint
-    To determine if holder pods are deployed, use
-    `kubectl --namespace $ROOK_OPERATOR get pods | grep plugin-holder`
-
-### Disabling Holder Pods with Multus
-
-This migration section applies when any CephCluster `network.provider` is `"multus"`. If the
-scenario does not apply, skip ahead to the [Disabling Holder Pods](#disabling-holder-pods) section.
-
-**Step 1**
-
-Before setting `CSI_ENABLE_HOST_NETWORK: "true"` and `CSI_DISABLE_HOLDER_PODS: "true"`, thoroughly
-read through the [Multus Prerequisites section](#multus-prerequisites). Use the prerequisites
-section to develop a plan for modifying host configurations as well as the public
-NetworkAttachmentDefinition.
-
-Once the plan is developed, execute the plan by following the steps below.
-
-**Step 2**
-
-First, modify the public NetworkAttachmentDefinition as needed. For example, it may be necessary to
-add the `routes` directive to the Whereabouts IPAM configuration as in
-[this example](#macvlan-whereabouts-node-static-ips).
-
-**Step 3**
-
-Next, modify the host configurations in the host configuration system. The host configuration system
-may be something like PXE, ignition config, cloud-init, Ansible, or any other such system. A node
-reboot is likely necessary to apply configuration updates, but wait until the next step to reboot
-nodes.
-
-If desired, check that the NetworkAttachmentDefinition modification and host configurations are
-compatible using the [Multus validation tool](#validating-multus-configuration). For the upgrade
-case, use the `hostCheckOnly: true` config option or `--host-check-only` CLI flag.
-
-**Step 4**
-
-After the NetworkAttachmentDefinition is modified, OSD pods must be restarted. It is easiest to
-complete this requirement at the same time nodes are being rebooted to apply configuration updates.
-
-For each node in the Kubernetes cluster:
-
-1. `cordon` and `drain` the node
-2. Wait for all pods to drain
-3. Reboot the node, ensuring the new host configuration will be applied
-4. `uncordon` and `undrain` the node
-5. Wait for the node to be rehydrated and stable
-6. Proceed to the next node
-
-By following this process, host configurations will be updated, and OSDs are also automatically
-restarted as part of the `drain` and `undrain` process on each node.
-
-OSDs can be restarted manually if node configuration updates do not require reboot.
-
-**Step 5**
-
-Once all nodes are running the new configuration and all OSDs have been restarted, check that the
-new node and NetworkAttachmentDefinition configurations are compatible. To do so, verify that each
-node can `ping` OSD pods via the public network.
-
-Use the [toolbox](../../Troubleshooting/ceph-toolbox.md) or the
-[kubectl plugin](../../Troubleshooting/kubectl-plugin.md) to list OSD IPs.
-
-The example below uses
-the kubectl plugin, and the OSD public network is 192.168.20.0/24.
-
-```console
-$ kubectl rook-ceph ceph osd dump | grep 'osd\.'
-osd.0 up   in  weight 1 up_from 7 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:192.168.20.19:6800/213587265,v1:192.168.20.19:6801/213587265] [v2:192.168.30.1:6800/213587265,v1:192.168.30.1:6801/213587265] exists,up 7ebbc19a-d45a-4b12-8fef-0f9423a59e78
-osd.1 up   in  weight 1 up_from 24 up_thru 24 down_at 20 last_clean_interval [8,23) [v2:192.168.20.20:6800/3144257456,v1:192.168.20.20:6801/3144257456] [v2:192.168.30.2:6804/3145257456,v1:192.168.30.2:6805/3145257456] exists,up 146b27da-d605-4138-9748-65603ed0dfa5
-osd.2 up   in  weight 1 up_from 21 up_thru 0 down_at 20 last_clean_interval [18,20) [v2:192.168.20.21:6800/1809748134,v1:192.168.20.21:6801/1809748134] [v2:192.168.30.3:6804/1810748134,v1:192.168.30.3:6805/1810748134] exists,up ff3d6592-634e-46fd-a0e4-4fe9fafc0386
-```
-
-Now check that each node (NODE) can reach OSDs over the public network:
-
-```console
-$ ssh user@NODE
-$ user@NODE $> ping -c3 192.168.20.19
-# [truncated, successful output]
-$ user@NODE $> ping -c3 192.168.20.20
-# [truncated, successful output]
-$ user@NODE $> ping -c3 192.168.20.21
-# [truncated, successful output]
-```
-
-If any node does not get a successful `ping` to a running OSD, it is not safe to proceed. A problem
-may arise here for many reasons. Some reasons include: the host may not be properly attached to the
-Multus public network, the public NetworkAttachmentDefinition may not be properly configured to
-route back to the host, the host may have a firewall rule blocking the connection in either
-direction, or the network switch may have a firewall rule blocking the connection. Diagnose and fix
-the issue, then return to **Step 1**.
-
-**Step 6**
-
-If the above check succeeds for all nodes, proceed with the
-[Disabling Holder Pods](#disabling-holder-pods) steps below.
-
-### Disabling Holder Pods
-
-**Step 1**
-
-If any CephClusters have Multus enabled (`network.provider: "multus"`), follow the
-[Disabling Holder Pods with Multus](#disabling-holder-pods-with-multus)
-steps above before continuing.
-
-**Step 2**
-
-Begin by setting `CSI_DISABLE_HOLDER_PODS: "true"`. If `CSI_ENABLE_HOST_NETWORK` is set to
-`"false"`, also set this value to `"true"` at the same time.
-
-After this, `csi-*plugin-*` pods will restart, and `csi-*plugin-holder-*` pods will remain running.
-
-**Step 3**
-
-Check that CSI pods are using the correct host networking configuration using the example below as
-guidance (in the example, `CSI_ENABLE_HOST_NETWORK` is `"true"`):
-
-```console
-$ kubectl -n rook-ceph get -o yaml daemonsets.apps csi-rbdplugin | grep -i hostnetwork
-      hostNetwork: true
-$ kubectl -n rook-ceph get -o yaml daemonsets.apps csi-cephfsplugin | grep -i hostnetwork
-      hostNetwork: true
-$ kubectl -n rook-ceph get -o yaml daemonsets.apps csi-nfsplugin | grep -i hostnetwork
-      hostNetwork: true
-```
-
-**Step 4**
-
-At this stage, PVCs for running applications are still using the holder pods. These PVCs must be
-migrated from the holder to the new network. Follow the below process to do so.
-
-For each node in the Kubernetes cluster:
-
-1. `cordon` and `drain` the node
-2. Wait for all pods to drain
-3. Delete all `csi-*plugin-holder*` pods on the node (a new holder will take it's place)
-4. `uncordon` and `undrain` the node
-5. Wait for the node to be rehydrated and stable
-6. Proceed to the next node
-
-**Step 5**
-
-After this process is done for all Kubernetes nodes, it is safe to delete the `csi-*plugin-holder*`
-daemonsets.
-
-Delete the holder daemonsets using the example below as guidance:
-
-```console
-$ kubectl -n rook-ceph get daemonset -o name | grep plugin-holder
-daemonset.apps/csi-cephfsplugin-holder-my-cluster
-daemonset.apps/csi-rbdplugin-holder-my-cluster
-
-$ kubectl -n rook-ceph delete daemonset.apps/csi-cephfsplugin-holder-my-cluster
-daemonset.apps "csi-cephfsplugin-holder-my-cluster" deleted
-
-$ kubectl -n rook-ceph delete daemonset.apps/csi-rbdplugin-holder-my-cluster
-daemonset.apps "csi-rbdplugin-holder-my-cluster" deleted
-```
-
-**Step 6**
-
-The migration is now complete! Congratulations!
diff --git a/Documentation/Helm-Charts/operator-chart.md b/Documentation/Helm-Charts/operator-chart.md
index a78ddc1ad..63ec4f542 100644
--- a/Documentation/Helm-Charts/operator-chart.md
+++ b/Documentation/Helm-Charts/operator-chart.md
@@ -84,7 +84,6 @@ The following table lists the configurable parameters of the rook-operator chart
 | `csi.csiRBDPluginVolumeMount` | The volume mounts of the CephCSI RBD plugin DaemonSet | `nil` |
 | `csi.csiRBDProvisionerResource` | CEPH CSI RBD provisioner resource requirement list csi-omap-generator resources will be applied only if `enableOMAPGenerator` is set to `true` | see values.yaml |
 | `csi.disableCsiDriver` | Disable the CSI driver. | `"false"` |
-| `csi.disableHolderPods` | Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network without needing hosts to the network. Holder pods are being removed. See issue for details: https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true". | `true` |
 | `csi.enableCSIEncryption` | Enable Ceph CSI PVC encryption support | `false` |
 | `csi.enableCSIHostNetwork` | Enable host networking for CSI CephFS and RBD nodeplugins. This may be necessary in some network configurations where the SDN does not provide access to an external cluster or there is significant drop in read/write performance | `true` |
 | `csi.enableCephfsDriver` | Enable Ceph CSI CephFS driver | `true` |
diff --git a/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-volume-group-snapshot.md b/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-volume-group-snapshot.md
index 33d58b807..f569ac151 100644
--- a/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-volume-group-snapshot.md
+++ b/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-volume-group-snapshot.md
@@ -23,12 +23,97 @@ In short, as the documentation describes it:
     Created by cluster administrators to describe how volume group snapshots
     should be created. including the driver information, the deletion policy, etc.
 
-## Volume Group Snapshots
+## RBD Volume Group Snapshots
+
+### RBD VolumeGroupSnapshotClass
+
+In [VolumeGroupSnapshotClass](https://github.com/rook/rook/tree/master/deploy/examples/csi/rbd/groupsnapshotclass.yaml),
+the `csi.storage.k8s.io/group-snapshotter-secret-name` parameter references the
+name of the secret created for the rbd-plugin and `pool` to reflect the Ceph pool name.
+
+In the `VolumeGroupSnapshotClass`, update the value of the `clusterID` field to match the namespace
+that Rook is running in. When Ceph CSI is deployed by Rook, the operator will automatically
+maintain a configmap whose contents will match this key. By default this is
+"rook-ceph".
+
+```console
+kubectl create -f deploy/examples/csi/rbd/groupsnapshotclass.yaml
+```
+
+### RBD VolumeGroupSnapshot
+
+In [VolumeGroupSnapshot](https://github.com/rook/rook/tree/master/deploy/examples/csi/rbd/groupsnapshot.yaml),
+`volumeGroupSnapshotClassName` is the name of the `VolumeGroupSnapshotClass`
+previously created. The labels inside `matchLabels` must be present on the
+PVCs that are already created by the RBD CSI driver.
+
+```console
+kubectl create -f deploy/examples/csi/rbd/groupsnapshot.yaml
+```
+
+### Verify RBD GroupSnapshot Creation
+
+```console
+$ kubectl get volumegroupsnapshotclass
+NAME                              DRIVER                          DELETIONPOLICY   AGE
+csi-rbdplugin-groupsnapclass      rook-ceph.rbd.csi.ceph.com      Delete           21m
+```
+
+```console
+$ kubectl get volumegroupsnapshot
+NAME                       READYTOUSE   VOLUMEGROUPSNAPSHOTCLASS          VOLUMEGROUPSNAPSHOTCONTENT                              CREATIONTIME   AGE
+rbd-groupsnapshot          true         csi-rbdplugin-groupsnapclass      groupsnapcontent-d13f4d95-8822-4729-9586-4f222a3f788e   5m37s          5m39s
+```
+
+The snapshot will be ready to restore to a new PVC when `READYTOUSE` field of the
+`volumegroupsnapshot` is set to true.
+
+### Restore the RBD volume group snapshot to a new PVC
+
+Find the name of the snapshots created by the `VolumeGroupSnapshot` first by running:
+
+```console
+$ kubectl get volumegroupsnapshot/rbd-groupsnapshot -o=jsonpath='{range .status.pvcVolumeSnapshotRefList[*]}PVC: {.persistentVolumeClaimRef.name}, Snapshot: {.volumeSnapshotRef.name}{"\n"}{end}'
+PVC: rbd-pvc, Snapshot: snapshot-9d21b143904c10f49ddc92664a7e8fe93c23387d0a88549c14337484ebaf1011-2024-09-12-3.49.13
+```
+
+It will list the PVC's name followed by its snapshot name.
+
+In
+[pvc-restore](https://github.com/rook/rook/tree/master/deploy/examples/csi/rbd/pvc-restore.yaml),
+`dataSource` is one of the `Snapshot` that we just
+found. The `dataSource` kind must be the `VolumeSnapshot`.
+
+Create a new PVC from the snapshot
+
+```console
+kubectl create -f deploy/examples/csi/rbd/pvc-restore.yaml
+```
+
+### Verify RBD Restore PVC Creation
+
+```console
+$ kubectl get pvc
+rbd-pvc           Bound    pvc-9ae60bf9-4931-4f9a-9de1-7f45f31fe4da   1Gi        RWO            rook-cephfs    <unset>                 171m
+rbd-pvc-restore   Bound    pvc-b4b73cbb-5061-48c7-9ac8-e1202508cf97   1Gi        RWO            rook-cephfs    <unset>                 46s
+```
+
+### RBD volume group snapshot resource Cleanup
+
+To clean the resources created by this example, run the following:
+
+```console
+kubectl delete -f deploy/examples/csi/rbd/pvc-restore.yaml
+kubectl delete -f deploy/examples/csi/rbd/groupsnapshot.yaml
+kubectl delete -f deploy/examples/csi/rbd/groupsnapshotclass.yaml
+```
+
+## CephFS Volume Group Snapshots
 
 ### CephFS VolumeGroupSnapshotClass
 
 In [VolumeGroupSnapshotClass](https://github.com/rook/rook/tree/master/deploy/examples/csi/cephfs/groupsnapshotclass.yaml),
-the `csi.storage.k8s.io/group-snapshotter-secret-name` parameter should reference the
+the `csi.storage.k8s.io/group-snapshotter-secret-name` parameter references the
 name of the secret created for the cephfs-plugin.
 
 In the `VolumeGroupSnapshotClass`, update the value of the `clusterID` field to match the namespace
@@ -43,8 +128,8 @@ kubectl create -f deploy/examples/csi/cephfs/groupsnapshotclass.yaml
 ### CephFS VolumeGroupSnapshot
 
 In [VolumeGroupSnapshot](https://github.com/rook/rook/tree/master/deploy/examples/csi/cephfs/groupsnapshot.yaml),
-`volumeGroupSnapshotClassName` should be the name of the `VolumeGroupSnapshotClass`
-previously created. The labels inside `matchLabels` should be present on the
+`volumeGroupSnapshotClassName` is the name of the `VolumeGroupSnapshotClass`
+previously created. The labels inside `matchLabels` must be present on the
 PVCs that are already created by the CephFS CSI driver.
 
 ```console
@@ -81,8 +166,8 @@ It will list the PVC's name followed by its snapshot name.
 
 In
 [pvc-restore](https://github.com/rook/rook/tree/master/deploy/examples/csi/cephfs/pvc-restore.yaml),
-`dataSource` should be one of the `Snapshot` that we just
-found. The `dataSource` kind should be the `VolumeSnapshot`.
+`dataSource` is one of the `Snapshot` that we just
+found. The `dataSource` kind must be the `VolumeSnapshot`.
 
 Create a new PVC from the snapshot
 
@@ -98,7 +183,7 @@ cephfs-pvc           Bound    pvc-9ae60bf9-4931-4f9a-9de1-7f45f31fe4da   1Gi
 cephfs-pvc-restore   Bound    pvc-b4b73cbb-5061-48c7-9ac8-e1202508cf97   1Gi        RWO            rook-cephfs    <unset>                 46s
 ```
 
-## CephFS volume group snapshot resource Cleanup
+### CephFS volume group snapshot resource Cleanup
 
 To clean the resources created by this example, run the following:
 
diff --git a/Documentation/Upgrade/rook-upgrade.md b/Documentation/Upgrade/rook-upgrade.md
index e9197240f..9d6bc6e6b 100644
--- a/Documentation/Upgrade/rook-upgrade.md
+++ b/Documentation/Upgrade/rook-upgrade.md
@@ -49,22 +49,12 @@ those releases.
     official releases. Builds from the master branch can have functionality changed or removed at any
     time without compatibility support and without prior notice.
 
-## Breaking changes in v1.15
-
-* Rook has deprecated CSI network "holder" pods.
-    If there are pods named `csi-*plugin-holder-*` in the Rook operator namespace, see the
-    [detailed documentation](../CRDs/Cluster/network-providers.md#holder-pod-deprecation)
-    to disable them. This deprecation process is required before upgrading to the future Rook v1.16.
-
-* Ceph COSI driver images have been updated. This impacts existing COSI Buckets, BucketClaims, and
-    BucketAccesses. Update existing clusters following the guide
-    [here](https://github.com/rook/rook/discussions/14297).
-
-* CephObjectStore, CephObjectStoreUser, and OBC endpoint behavior has changed when CephObjectStore
-    `spec.hosting` configurations are set. Use the new `spec.hosting.advertiseEndpoint` config to
-    define required behavior as
-    [documented](../Storage-Configuration/Object-Storage-RGW/object-storage.md#object-store-endpoint).
+## Breaking changes in v1.16
 
+* Rook fully deprecated CSI "holder" pods in Rook v1.16. Any Rook v1.15 CephCluster with
+    `csi-*plugin-holder-*` pods present in the Rook operator namespace must follow holder pod
+    removal migration steps outlined in
+    [v1.15 documentation](https://rook.io/docs/rook/v1.15/CRDs/Cluster/network-providers/?h=depre#holder-pod-deprecation).
 
 ## Considerations
 
@@ -111,10 +101,6 @@ Also update optional resources like Prometheus monitoring noted more fully in th
 If Rook is installed via the Helm chart, Helm will handle some details of the upgrade itself.
 The upgrade steps in this guide will clarify what Helm handles automatically.
 
-!!! important
-    If there are pods named `csi-*plugin-holder-*` in the Rook operator namespace, set the new
-    config `csi.disableHolderPods: false` in the values.yaml before upgrading to v1.15.
-
 The `rook-ceph` helm chart upgrade performs the Rook upgrade.
 The `rook-ceph-cluster` helm chart upgrade performs a [Ceph upgrade](./ceph-upgrade.md) if the Ceph image is updated.
 The `rook-ceph` chart should be upgraded before `rook-ceph-cluster`, so the latest operator has the opportunity to update
@@ -262,10 +248,3 @@ This cluster is finished:
 At this point, the Rook operator should be running version `rook/ceph:v1.15.0`.
 
 Verify the CephCluster health using the [health verification doc](health-verification.md).
-
-### **6. Disable holder pods**
-
-Rook has deprecated CSI network "holder" pods. If there are pods named
-`csi-*plugin-holder-*` in the Rook operator namespace, see the
-[detailed documentation](../CRDs/Cluster/network-providers.md#holder-pod-deprecation)
-to disable them. This deprecation process is required before upgrading to the future Rook v1.16.
diff --git a/deploy/charts/rook-ceph/templates/configmap.yaml b/deploy/charts/rook-ceph/templates/configmap.yaml
index c29dbcd74..98b99741d 100644
--- a/deploy/charts/rook-ceph/templates/configmap.yaml
+++ b/deploy/charts/rook-ceph/templates/configmap.yaml
@@ -38,7 +38,6 @@ data:
   CSI_ENABLE_ENCRYPTION: {{ .Values.csi.enableCSIEncryption | quote }}
   CSI_ENABLE_OMAP_GENERATOR: {{ .Values.csi.enableOMAPGenerator | quote }}
   CSI_ENABLE_HOST_NETWORK: {{ .Values.csi.enableCSIHostNetwork | quote }}
-  CSI_DISABLE_HOLDER_PODS: {{ .Values.csi.disableHolderPods | quote }}
   CSI_ENABLE_METADATA: {{ .Values.csi.enableMetadata | quote }}
   CSI_ENABLE_VOLUME_GROUP_SNAPSHOT: {{ .Values.csi.enableVolumeGroupSnapshot | quote }}
 {{- if .Values.csi.csiDriverNamePrefix }}
diff --git a/deploy/charts/rook-ceph/values.yaml b/deploy/charts/rook-ceph/values.yaml
index 7e3d47bb3..5b48fc868 100644
--- a/deploy/charts/rook-ceph/values.yaml
+++ b/deploy/charts/rook-ceph/values.yaml
@@ -88,10 +88,6 @@ csi:
   # in some network configurations where the SDN does not provide access to an external cluster or
   # there is significant drop in read/write performance
   enableCSIHostNetwork: true
-  # -- Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network
-  # without needing hosts to the network. Holder pods are being removed. See issue for details:
-  # https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true".
-  disableHolderPods: true
   # -- Enable Snapshotter in CephFS provisioner pod
   enableCephfsSnapshotter: true
   # -- Enable Snapshotter in NFS provisioner pod
@@ -640,7 +636,7 @@ enforceHostNetwork: false
 disableDeviceHotplug: false
 
 # -- The revision history limit for all pods created by Rook. If blank, the K8s default is 10.
-revisionHistoryLimit: 
+revisionHistoryLimit:
 
 # -- Blacklist certain disks according to the regex provided.
 discoverDaemonUdev:
diff --git a/deploy/examples/csi/cephfs/pvc.yaml b/deploy/examples/csi/cephfs/pvc.yaml
index cf3057de3..656d46859 100644
--- a/deploy/examples/csi/cephfs/pvc.yaml
+++ b/deploy/examples/csi/cephfs/pvc.yaml
@@ -3,6 +3,8 @@ apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: cephfs-pvc
+  # Use this with the example `groupsnapshotclass.yaml`.
+  # Not needed if the volume group snapshots are not required.
   labels:
     group: snapshot-test
 spec:
diff --git a/deploy/examples/csi/rbd/groupsnapshot.yaml b/deploy/examples/csi/rbd/groupsnapshot.yaml
new file mode 100644
index 000000000..b0f329125
--- /dev/null
+++ b/deploy/examples/csi/rbd/groupsnapshot.yaml
@@ -0,0 +1,13 @@
+---
+apiVersion: groupsnapshot.storage.k8s.io/v1alpha1
+kind: VolumeGroupSnapshot
+metadata:
+  name: rbd-groupsnapshot
+spec:
+  source:
+    selector:
+      matchLabels:
+        # The PVCs require this label for them to be
+        # included in the VolumeGroupSnapshot
+        group: snapshot-test
+  volumeGroupSnapshotClassName: csi-rbdplugin-groupsnapclass
diff --git a/deploy/examples/csi/rbd/groupsnapshotclass.yaml b/deploy/examples/csi/rbd/groupsnapshotclass.yaml
new file mode 100644
index 000000000..05a64852f
--- /dev/null
+++ b/deploy/examples/csi/rbd/groupsnapshotclass.yaml
@@ -0,0 +1,15 @@
+---
+apiVersion: groupsnapshot.storage.k8s.io/v1alpha1
+kind: VolumeGroupSnapshotClass
+metadata:
+  name: csi-rbdplugin-groupsnapclass
+driver: rook-ceph.rbd.csi.ceph.com # csi-provisioner-name
+parameters:
+  # Specify a string that identifies your cluster. Ceph CSI supports any
+  # unique string. When Ceph CSI is deployed by Rook use the Rook namespace,
+  # for example "rook-ceph".
+  clusterID: rook-ceph # namespace: cluster
+  pool: replicapool
+  csi.storage.k8s.io/group-snapshotter-secret-name: rook-csi-rbd-provisioner
+  csi.storage.k8s.io/group-snapshotter-secret-namespace: rook-ceph
+deletionPolicy: Delete
diff --git a/deploy/examples/csi/rbd/pvc.yaml b/deploy/examples/csi/rbd/pvc.yaml
index 516a5aa87..edb8b1ba8 100644
--- a/deploy/examples/csi/rbd/pvc.yaml
+++ b/deploy/examples/csi/rbd/pvc.yaml
@@ -3,6 +3,10 @@ apiVersion: v1
 kind: PersistentVolumeClaim
 metadata:
   name: rbd-pvc
+  # Use this with the example `groupsnapshotclass.yaml`.
+  # Not needed if the volume group snapshots are not required.
+  labels:
+    group: snapshot-test
 spec:
   accessModes:
     - ReadWriteOnce
diff --git a/deploy/examples/operator-openshift.yaml b/deploy/examples/operator-openshift.yaml
index 876a029f0..239b75b16 100644
--- a/deploy/examples/operator-openshift.yaml
+++ b/deploy/examples/operator-openshift.yaml
@@ -139,11 +139,6 @@ data:
   # there is significant drop in read/write performance.
   # CSI_ENABLE_HOST_NETWORK: "true"
 
-  # Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network
-  # without needing hosts to the network. Holder pods are being removed. See issue for details:
-  # https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true".
-  CSI_DISABLE_HOLDER_PODS: "true"
-
   # Set logging level for cephCSI containers maintained by the cephCSI.
   # Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity.
   # CSI_LOG_LEVEL: "0"
diff --git a/deploy/examples/operator.yaml b/deploy/examples/operator.yaml
index 0533051ff..1a90a5193 100644
--- a/deploy/examples/operator.yaml
+++ b/deploy/examples/operator.yaml
@@ -51,11 +51,6 @@ data:
   # there is significant drop in read/write performance.
   # CSI_ENABLE_HOST_NETWORK: "true"
 
-  # Deprecation note: Rook uses "holder" pods to allow CSI to connect to the multus public network
-  # without needing hosts to the network. Holder pods are being removed. See issue for details:
-  # https://github.com/rook/rook/issues/13055. New Rook deployments should set this to "true".
-  CSI_DISABLE_HOLDER_PODS: "true"
-
   # Set to true to enable adding volume metadata on the CephFS subvolume and RBD images.
   # Not all users might be interested in getting volume/snapshot details as metadata on CephFS subvolume and RBD images.
   # Hence enable metadata is false by default.
diff --git a/design/ceph/multus-network.md b/design/ceph/multus-network.md
index 045cba88b..901904683 100644
--- a/design/ceph/multus-network.md
+++ b/design/ceph/multus-network.md
@@ -93,50 +93,6 @@ network.
 The design for mitigating the issue is to add a new DaemonSet that will own the network for all CephFS mounts
 as well as RBD mapped devices. The `csi-{cephfs,rbd}plugin` DaemonSet are left untouched.
 
-#### New "holder" DaemonSet
-The Rook-Ceph Operator's CSI controller creates a `csi-plugin-holder` DaemonSet configured to use the
-`network.selectors.public` network specified for the CephCluster. This DaemonSet runs on all the
-nodes along side the `csi-{cephfs,rbd}plugin` DaemonSet.
-
-This Pod should only be stopped and restarted when a node is stopped so that volume operations do
-not become blocked. The Rook-Ceph Operator's CSI controller should set the DaemonSet's update
-strategy to `OnDelete` so that the pods do not get deleted if the DaemonSet is updated while also
-ensuring that the pods will be updated on the next node reboot (or node drain).
-
-The new holder DaemonSet only contains a single container called `holder`, the container
-responsible for pinning the network for filesystem mounts and mapped block devices. It is used as a passthrough by
-the Ceph-CSI plugin pod which when mounting or mapping will use the network namespace of that holder
-pod.
-
-One task of the Holder container is to pass the required PID into the `csi-plugin` pod. To
-solve this, the container will create a symlink to an equivalent of `/proc/$$/ns/net` in the
-`hostPath` volume shared with the `csi-plugin` pod. Then it will sleep forever.
-In order to support multiple Ceph-CSI consuming multiple Ceph clusters, the name of the symlink should be based on the Ceph
-cluster FSID. The complete name of the symlink can be decided during the implementation phase of the
-design.
-That symlink name will be stored in the Ceph-CSI configmap with a key name that we will define
-during the implementation.
-
-#### Interactions between components
-When the Ceph-CSI plugin mounts or maps a block device it will use the network namespace of the
-Holder pod. This is achieved by using `nsenter` to enter the network namespace of the Holder pod.
-Some changes are expected as part of the mounting/mapping method in today's `csi-plugin` pod.
-Instead of invoking `rbd map ...`, invoke `nsenter --net=<net
-ns file of pause in long-running pod> rbd map ...` (and same for `mount -t ceph ...`).  The `csi-plugin`
-pod is already privileged, it already adds `CAP_SYS_ADMIN`. The only missing piece is to expose the host
-PID namespace with `hostPID: true` on the `csi-cephfsplugin` DaemonSet (RBD already has it).
-
-Cleaning up the symlink is not a requirement nor a big issue. Once the Holder container stops, its
-network namespace goes away with it. So at worst, we end up with a broken symlink.
-
-When a new node is added, both DaemonSets are started, and the
-process described above occurs on the node.
-
-The initial implementation of this design is limited to supporting a single CephCluster with
-Multus.
-Until we stop using HostNetwork entirely it is impossible to support multiple CephClusters with and
-without Multus enabled.
-
 ## Accepted proposal
 
 So far, the team has decided to go with the [whereabouts](https://github.com/dougbtv/whereabouts) IPAM.
diff --git a/pkg/operator/ceph/cluster/cephstatus.go b/pkg/operator/ceph/cluster/cephstatus.go
index 8c8bfe207..7f7e86a1a 100644
--- a/pkg/operator/ceph/cluster/cephstatus.go
+++ b/pkg/operator/ceph/cluster/cephstatus.go
@@ -314,11 +314,8 @@ func (c *cephStatusChecker) getRookPodsOnNode(node string) ([]v1.Pod, error) {
 	appLabels := []string{
 		"csi-rbdplugin-provisioner",
 		"csi-rbdplugin",
-		"csi-rbdplugin-holder",
-		"csi-cephfsplugin-holder",
 		"csi-cephfsplugin-provisioner",
 		"csi-cephfsplugin",
-		"csi-nfsplugin-holder",
 		"csi-nfsplugin-provisioner",
 		"csi-nfsplugin",
 		"rook-ceph-operator",
diff --git a/pkg/operator/ceph/csi/controller.go b/pkg/operator/ceph/csi/controller.go
index 31c9d7924..e2fc9e39f 100644
--- a/pkg/operator/ceph/csi/controller.go
+++ b/pkg/operator/ceph/csi/controller.go
@@ -52,23 +52,15 @@ const (
 
 // ReconcileCSI reconciles a ceph-csi driver
 type ReconcileCSI struct {
-	scheme             *runtime.Scheme
-	client             client.Client
-	context            *clusterd.Context
-	opManagerContext   context.Context
-	opConfig           opcontroller.OperatorConfig
-	clustersWithHolder []ClusterDetail
+	scheme           *runtime.Scheme
+	client           client.Client
+	context          *clusterd.Context
+	opManagerContext context.Context
+	opConfig         opcontroller.OperatorConfig
 	// the first cluster CR which will determine some settings for the csi driver
 	firstCephCluster *cephv1.ClusterSpec
 }
 
-// ClusterDetail is a struct that holds the information of a cluster, it knows its internals (like
-// FSID through clusterInfo) and the Kubernetes object that represents it (like the CephCluster CRD)
-type ClusterDetail struct {
-	cluster     *cephv1.CephCluster
-	clusterInfo *cephclient.ClusterInfo
-}
-
 // Add creates a new Ceph CSI Controller and adds it to the Manager. The Manager will set fields on the Controller
 // and Start it when the Manager is Started.
 func Add(mgr manager.Manager, context *clusterd.Context, opManagerContext context.Context, opConfig opcontroller.OperatorConfig) error {
@@ -78,12 +70,11 @@ func Add(mgr manager.Manager, context *clusterd.Context, opManagerContext contex
 // newReconciler returns a new reconcile.Reconciler
 func newReconciler(mgr manager.Manager, context *clusterd.Context, opManagerContext context.Context, opConfig opcontroller.OperatorConfig) reconcile.Reconciler {
 	return &ReconcileCSI{
-		scheme:             mgr.GetScheme(),
-		client:             mgr.GetClient(),
-		context:            context,
-		opConfig:           opConfig,
-		opManagerContext:   opManagerContext,
-		clustersWithHolder: []ClusterDetail{},
+		scheme:           mgr.GetScheme(),
+		client:           mgr.GetClient(),
+		context:          context,
+		opConfig:         opConfig,
+		opManagerContext: opManagerContext,
 	}
 }
 
@@ -250,22 +241,6 @@ func (r *ReconcileCSI) reconcile(request reconcile.Request) (reconcile.Result, e
 	}
 	CustomCSICephConfigExists = exists
 
-	csiHostNetworkEnabled, err := strconv.ParseBool(k8sutil.GetValue(r.opConfig.Parameters, "CSI_ENABLE_HOST_NETWORK", "true"))
-	if err != nil {
-		return reconcile.Result{}, errors.Wrap(err, "failed to parse value for 'CSI_ENABLE_HOST_NETWORK'")
-	}
-
-	csiDisableHolders, err := strconv.ParseBool(k8sutil.GetValue(r.opConfig.Parameters, "CSI_DISABLE_HOLDER_PODS", "false"))
-	if err != nil {
-		return reconcile.Result{}, errors.Wrap(err, "failed to parse value for 'CSI_DISABLE_HOLDER_PODS'")
-	}
-
-	// begin each reconcile with the assumption that holder pods won't be deployed
-	// the loop below will determine with each reconcile if they need deployed
-	// without this, holder pods won't be removed unless the operator is restarted
-	r.clustersWithHolder = []ClusterDetail{}
-	holderEnabled = false
-
 	for i, cluster := range cephClusters.Items {
 		if !cluster.DeletionTimestamp.IsZero() {
 			logger.Debugf("ceph cluster %q is being deleting, no need to reconcile the csi driver", request.NamespacedName)
@@ -295,29 +270,14 @@ func (r *ReconcileCSI) reconcile(request reconcile.Request) (reconcile.Result, e
 		}
 		clusterInfo.OwnerInfo = k8sutil.NewOwnerInfo(&cephClusters.Items[i], r.scheme)
 
-		// is holder enabled for this cluster?
-		thisHolderEnabled := (!csiHostNetworkEnabled || cluster.Spec.Network.IsMultus()) && !csiDisableHolders
-
-		// Do we have a multus cluster or csi host network disabled?
-		// If so deploy the plugin holder with the fsid attached
-		if thisHolderEnabled {
-			logger.Debugf("cluster %q: deploying the ceph-csi plugin holder", cluster.Name)
-			r.clustersWithHolder = append(r.clustersWithHolder, ClusterDetail{cluster: &cephClusters.Items[i], clusterInfo: clusterInfo})
-
-			// holder pods are enabled globally if any cluster needs a holder pod
-			holderEnabled = true
-		} else {
-			logger.Debugf("cluster %q: not deploying the ceph-csi plugin holder", request.NamespacedName)
-		}
-
-		// if holder pods were disabled, the controller needs to update the configmap for each
-		// cephcluster to remove the net namespace file path
+		// ensure any remaining holder-related configs are cleared
+		holderEnabled = false
 		err = reconcileSaveCSIDriverOptions(r.context.Clientset, cluster.Namespace, clusterInfo)
 		if err != nil {
 			return opcontroller.ImmediateRetryResult, errors.Wrapf(err, "failed to update CSI driver options for cluster %q", cluster.Name)
 		}
 
-		// Skip the new CSI-operator creation when holder pod is enabled until multus support is added in the CSI operator
+		// disable Rook-managed CSI drivers if CSI operator is enabled
 		if EnableCSIOperator() {
 			logger.Info("disabling csi-driver since EnableCSIOperator is true")
 			err := r.stopDrivers(serverVersion)
diff --git a/pkg/operator/ceph/csi/spec.go b/pkg/operator/ceph/csi/spec.go
index de30e74a8..59af7f601 100644
--- a/pkg/operator/ceph/csi/spec.go
+++ b/pkg/operator/ceph/csi/spec.go
@@ -21,7 +21,6 @@ import (
 	_ "embed"
 	"fmt"
 	"path"
-	"strconv"
 	"strings"
 	"time"
 
@@ -30,7 +29,6 @@ import (
 	opcontroller "github.com/rook/rook/pkg/operator/ceph/controller"
 	"github.com/rook/rook/pkg/operator/k8sutil"
 	"github.com/rook/rook/pkg/operator/k8sutil/cmdreporter"
-	"sigs.k8s.io/controller-runtime/pkg/client"
 
 	"github.com/pkg/errors"
 	apps "k8s.io/api/apps/v1"
@@ -38,11 +36,8 @@ import (
 	k8scsi "k8s.io/api/storage/v1beta1"
 	kerrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/version"
 	"k8s.io/client-go/kubernetes"
-
-	cephcsi "github.com/ceph/ceph-csi/api/deploy/kubernetes"
 )
 
 type Param struct {
@@ -115,15 +110,6 @@ type templateParam struct {
 	Namespace string
 }
 
-type driverDetails struct {
-	name           string
-	fullName       string
-	holderTemplate string
-	toleration     string
-	nodeAffinity   string
-	resource       string
-}
-
 var (
 	CSIParam Param
 
@@ -164,8 +150,6 @@ var (
 	// Local package template path for RBD
 	//go:embed template/rbd/csi-rbdplugin.yaml
 	RBDPluginTemplatePath string
-	//go:embed template/rbd/csi-rbdplugin-holder.yaml
-	RBDPluginHolderTemplatePath string
 	//go:embed template/rbd/csi-rbdplugin-provisioner-dep.yaml
 	RBDProvisionerDepTemplatePath string
 	//go:embed template/rbd/csi-rbdplugin-svc.yaml
@@ -174,8 +158,6 @@ var (
 	// Local package template path for CephFS
 	//go:embed template/cephfs/csi-cephfsplugin.yaml
 	CephFSPluginTemplatePath string
-	//go:embed template/cephfs/csi-cephfsplugin-holder.yaml
-	CephFSPluginHolderTemplatePath string
 	//go:embed template/cephfs/csi-cephfsplugin-provisioner-dep.yaml
 	CephFSProvisionerDepTemplatePath string
 	//go:embed template/cephfs/csi-cephfsplugin-svc.yaml
@@ -186,8 +168,6 @@ var (
 	NFSPluginTemplatePath string
 	//go:embed template/nfs/csi-nfsplugin-provisioner-dep.yaml
 	NFSProvisionerDepTemplatePath string
-	//go:embed template/nfs/csi-nfsplugin-holder.yaml
-	NFSPluginHolderTemplatePath string
 
 	//go:embed template/csi-logrotate-sidecar.yaml
 	LogrotateTemplatePath string
@@ -333,8 +313,6 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 		csiDriverobj                                                                    v1CsiDriver
 	)
 
-	enabledDrivers := make([]driverDetails, 0)
-
 	tp := templateParam{
 		Param:     CSIParam,
 		Namespace: r.opConfig.OperatorNamespace,
@@ -391,14 +369,6 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 			}
 			rbdService.Namespace = r.opConfig.OperatorNamespace
 		}
-		enabledDrivers = append(enabledDrivers, driverDetails{
-			name:           RBDDriverShortName,
-			fullName:       RBDDriverName,
-			holderTemplate: RBDPluginHolderTemplatePath,
-			nodeAffinity:   rbdPluginNodeAffinityEnv,
-			toleration:     rbdPluginTolerationsEnv,
-			resource:       rbdPluginResource,
-		})
 	}
 	if EnableCephFS {
 		tp.CsiComponentName = nodePlugin
@@ -429,14 +399,6 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 			}
 			cephfsService.Namespace = r.opConfig.OperatorNamespace
 		}
-		enabledDrivers = append(enabledDrivers, driverDetails{
-			name:           CephFSDriverShortName,
-			fullName:       CephFSDriverName,
-			holderTemplate: CephFSPluginHolderTemplatePath,
-			nodeAffinity:   cephFSPluginNodeAffinityEnv,
-			toleration:     cephFSPluginTolerationsEnv,
-			resource:       cephFSPluginResource,
-		})
 	}
 
 	if EnableNFS {
@@ -459,15 +421,6 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 			applyLogrotateSidecar(&nfsProvisionerDeployment.Spec.Template, "csi-nfs-deployment-log-collector", LogrotateTemplatePath, tp)
 		}
 		nfsProvisionerDeployment.Spec.Template.Spec.HostNetwork = opcontroller.EnforceHostNetwork()
-
-		enabledDrivers = append(enabledDrivers, driverDetails{
-			name:           NFSDriverShortName,
-			fullName:       NFSDriverName,
-			holderTemplate: NFSPluginHolderTemplatePath,
-			nodeAffinity:   nfsPluginNodeAffinityEnv,
-			toleration:     nfsPluginTolerationsEnv,
-			resource:       nfsPluginResource,
-		})
 	}
 
 	// get common provisioner tolerations and node affinity
@@ -478,13 +431,6 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 	pluginTolerations := getToleration(r.opConfig.Parameters, pluginTolerationsEnv, []corev1.Toleration{})
 	pluginNodeAffinity := getNodeAffinity(r.opConfig.Parameters, pluginNodeAffinityEnv, &corev1.NodeAffinity{})
 
-	// Deploy the CSI Holder DaemonSet if Multus is enabled or
-	// EnableCSIHostNetwork is disabled.
-	err = r.configureHolders(enabledDrivers, tp, pluginTolerations, pluginNodeAffinity)
-	if err != nil {
-		return errors.Wrap(err, "failed to configure holder")
-	}
-
 	if rbdPlugin != nil {
 		// get RBD plugin tolerations and node affinity, defaults to common tolerations and node affinity if not specified
 		rbdPluginTolerations := getToleration(r.opConfig.Parameters, rbdPluginTolerationsEnv, pluginTolerations)
@@ -505,9 +451,6 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 		if err != nil {
 			return errors.Wrapf(err, "failed to apply network config to rbd plugin daemonset %q", rbdPlugin.Name)
 		}
-		if holderEnabled {
-			rbdPlugin.Spec.Template.Spec.HostNetwork = false
-		}
 		err = k8sutil.CreateDaemonSet(r.opManagerContext, r.opConfig.OperatorNamespace, r.context.Clientset, rbdPlugin)
 		if err != nil {
 			return errors.Wrapf(err, "failed to start rbdplugin daemonset %q", rbdPlugin.Name)
@@ -577,11 +520,6 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 		if err != nil {
 			return errors.Wrapf(err, "failed to apply network config to cephfs plugin daemonset %q", cephfsPlugin.Name)
 		}
-		if holderEnabled {
-			cephfsPlugin.Spec.Template.Spec.HostNetwork = false
-			// HostPID is used to communicate with the network namespace
-			cephfsPlugin.Spec.Template.Spec.HostPID = true
-		}
 
 		err = k8sutil.CreateDaemonSet(r.opManagerContext, r.opConfig.OperatorNamespace, r.context.Clientset, cephfsPlugin)
 		if err != nil {
@@ -652,11 +590,6 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 		if err != nil {
 			return errors.Wrapf(err, "failed to apply network config to nfs plugin daemonset %q", nfsPlugin.Name)
 		}
-		if holderEnabled {
-			nfsPlugin.Spec.Template.Spec.HostNetwork = false
-			// HostPID is used to communicate with the network namespace
-			nfsPlugin.Spec.Template.Spec.HostPID = true
-		}
 
 		err = k8sutil.CreateDaemonSet(r.opManagerContext, r.opConfig.OperatorNamespace, r.context.Clientset, nfsPlugin)
 		if err != nil {
@@ -865,166 +798,6 @@ func (r *ReconcileCSI) validateCSIVersion(ownerInfo *k8sutil.OwnerInfo) (*CephCS
 	return version, nil
 }
 
-func (r *ReconcileCSI) configureHolders(enabledDrivers []driverDetails, tp templateParam, pluginTolerations []corev1.Toleration, pluginNodeAffinity *corev1.NodeAffinity) error {
-	for _, cluster := range r.clustersWithHolder {
-		for _, driver := range enabledDrivers {
-			err := r.configureHolder(driver, cluster, tp, pluginTolerations, pluginNodeAffinity)
-			if err != nil {
-				return errors.Wrapf(err, "failed to configure holder %q for %q/%q", driver.name, cluster.cluster.Name, cluster.cluster.Namespace)
-			}
-		}
-	}
-
-	return nil
-}
-
-func (r *ReconcileCSI) configureHolder(driver driverDetails, c ClusterDetail, tp templateParam, pluginTolerations []corev1.Toleration, pluginNodeAffinity *corev1.NodeAffinity) error {
-	cephPluginHolder, err := templateToDaemonSet("cephpluginholder", driver.holderTemplate, tp)
-	if err != nil {
-		return errors.Wrapf(err, "failed to load ceph %q plugin holder template", driver.fullName)
-	}
-
-	// As the plugin holder daemonset is created in the operator namespace, we
-	// need to set the owner reference to the cluster namespace only if the
-	// operator and cluster are created in same namespace so that the
-	// plugin holder daemonset is deleted when the cluster is deleted.
-	if r.opConfig.OperatorNamespace == c.cluster.Namespace {
-		err = c.clusterInfo.OwnerInfo.SetControllerReference(cephPluginHolder)
-		if err != nil {
-			return errors.Wrapf(err, "failed to set owner reference to plugin holder %q", driver.fullName)
-		}
-	}
-	holderPluginTolerations := getToleration(r.opConfig.Parameters, driver.toleration, pluginTolerations)
-	holderPluginNodeAffinity := getNodeAffinity(r.opConfig.Parameters, driver.nodeAffinity, pluginNodeAffinity)
-	// apply driver's plugin tolerations and node affinity
-	applyToPodSpec(&cephPluginHolder.Spec.Template.Spec, holderPluginNodeAffinity, holderPluginTolerations)
-
-	// apply resource request and limit from corresponding plugin container
-	applyResourcesToContainers(r.opConfig.Parameters, driver.resource, &cephPluginHolder.Spec.Template.Spec)
-
-	// Append the CEPH_CLUSTER_NAMESPACE env var so that the main container can use it to create the network
-	// namespace symlink to the Kubelet plugin directory
-	cephPluginHolder.Spec.Template.Spec.Containers[0].Env = append(
-		cephPluginHolder.Spec.Template.Spec.Containers[0].Env,
-		corev1.EnvVar{
-			Name:  "CEPH_CLUSTER_NAMESPACE",
-			Value: c.cluster.Namespace,
-		},
-	)
-
-	// Append the driver name so that the symlink file goes into the right location on the
-	// kubelet plugin directory
-	cephPluginHolder.Spec.Template.Spec.Containers[0].Env = append(
-		cephPluginHolder.Spec.Template.Spec.Containers[0].Env,
-		corev1.EnvVar{
-			Name:  "ROOK_CEPH_CSI_DRIVER_NAME",
-			Value: driver.fullName,
-		},
-	)
-
-	// Make the DS name unique per Ceph cluster
-	cephPluginHolder.Name = fmt.Sprintf("%s-%s", cephPluginHolder.Name, c.cluster.Name)
-	cephPluginHolder.Spec.Template.Name = cephPluginHolder.Name
-	cephPluginHolder.Spec.Template.Spec.Containers[0].Name = fmt.Sprintf("%s-%s", cephPluginHolder.Spec.Template.Spec.Containers[0].Name, c.cluster.Name)
-
-	// Add default labels
-	k8sutil.AddRookVersionLabelToDaemonSet(cephPluginHolder)
-
-	// If multus is enabled, add the multus plugin label
-	if c.cluster.Spec.Network.IsMultus() {
-		// Apply Multus annotations to daemonset spec
-		err = k8sutil.ApplyMultus(c.cluster.GetNamespace(), &c.cluster.Spec.Network, &cephPluginHolder.Spec.Template.ObjectMeta)
-		if err != nil {
-			return errors.Wrapf(err, "failed to apply multus configuration for holder %q in cluster %q", cephPluginHolder.Name, c.cluster.Namespace)
-		}
-	}
-	// Finally create the DaemonSet
-	_, err = r.context.Clientset.AppsV1().DaemonSets(r.opConfig.OperatorNamespace).Create(r.opManagerContext, cephPluginHolder, metav1.CreateOptions{})
-	if err != nil {
-		if kerrors.IsAlreadyExists(err) {
-			_, err = r.context.Clientset.AppsV1().DaemonSets(r.opConfig.OperatorNamespace).Update(r.opManagerContext, cephPluginHolder, metav1.UpdateOptions{})
-			if err != nil {
-				return errors.Wrapf(err, "failed to update ceph plugin holder daemonset %q", cephPluginHolder.Name)
-			}
-			logger.Debugf("holder %q already exists for cluster %q, updating it, restart holder pods to take effect of update", cephPluginHolder.Name, c.cluster.Namespace)
-		} else {
-			return errors.Wrapf(err, "failed to start ceph plugin holder daemonset %q", cephPluginHolder.Name)
-		}
-	}
-
-	clusterConfigEntry := &CSIClusterConfigEntry{
-		ClusterInfo: cephcsi.ClusterInfo{
-			Monitors: MonEndpoints(c.clusterInfo.Monitors, c.cluster.Spec.RequireMsgr2()),
-			RBD:      cephcsi.RBD{},
-			CephFS: cephcsi.CephFS{
-				FuseMountOptions:   c.clusterInfo.CSIDriverSpec.CephFS.FuseMountOptions,
-				KernelMountOptions: c.clusterInfo.CSIDriverSpec.CephFS.KernelMountOptions,
-			},
-			NFS: cephcsi.NFS{},
-			ReadAffinity: cephcsi.ReadAffinity{
-				Enabled:             c.clusterInfo.CSIDriverSpec.ReadAffinity.Enabled,
-				CrushLocationLabels: c.clusterInfo.CSIDriverSpec.ReadAffinity.CrushLocationLabels,
-			},
-		},
-	}
-
-	netNamespaceFilePath := generateNetNamespaceFilePath(CSIParam.KubeletDirPath, driver.fullName, c.cluster.Namespace)
-	if driver.name == RBDDriverShortName {
-		clusterConfigEntry.RBD.NetNamespaceFilePath = netNamespaceFilePath
-	}
-	if driver.name == CephFSDriverShortName {
-		clusterConfigEntry.CephFS.NetNamespaceFilePath = netNamespaceFilePath
-	}
-	if driver.name == NFSDriverShortName {
-		clusterConfigEntry.NFS.NetNamespaceFilePath = netNamespaceFilePath
-	}
-
-	// Save the path of the network namespace file for ceph-csi to use
-	clusterId := c.cluster.Namespace // cluster ID is same as cluster namespace for CephClusters
-	err = SaveClusterConfig(r.context.Clientset, clusterId, c.cluster.Namespace, c.clusterInfo, clusterConfigEntry)
-	if err != nil {
-		return errors.Wrapf(err, "failed to save cluster config for csi holder %q", driver.fullName)
-	}
-	return nil
-}
-
-func GenerateNetNamespaceFilePath(ctx context.Context, client client.Client, clusterNamespace, opNamespace, driverName string) (string, error) {
-	var driverSuffix string
-	opNamespaceName := types.NamespacedName{Name: opcontroller.OperatorSettingConfigMapName, Namespace: opNamespace}
-	opConfig := &corev1.ConfigMap{}
-	err := client.Get(ctx, opNamespaceName, opConfig)
-	if err != nil && !kerrors.IsNotFound(err) {
-		return "", errors.Wrap(err, "failed to get operator's configmap")
-	}
-
-	// net namespace file path is empty string if holder pods are disabled
-	csiDisableHolders, err := strconv.ParseBool(k8sutil.GetValue(opConfig.Data, "CSI_DISABLE_HOLDER_PODS", "false"))
-	if err != nil {
-		return "", errors.Wrap(err, "failed to parse value for 'CSI_DISABLE_HOLDER_PODS'")
-	}
-	if csiDisableHolders {
-		return "", nil
-	}
-
-	switch driverName {
-	case RBDDriverShortName:
-		driverSuffix = rbdDriverSuffix
-	case CephFSDriverShortName:
-		driverSuffix = cephFSDriverSuffix
-	default:
-		return "", errors.Errorf("unsupported driver name %q", driverName)
-	}
-
-	kubeletDirPath := k8sutil.GetValue(opConfig.Data, "ROOK_CSI_KUBELET_DIR_PATH", DefaultKubeletDirPath)
-	driverFullName := fmt.Sprintf("%s.%s", opNamespace, driverSuffix)
-
-	return generateNetNamespaceFilePath(kubeletDirPath, driverFullName, clusterNamespace), nil
-}
-
-func generateNetNamespaceFilePath(kubeletDirPath, driverFullName, clusterNamespace string) string {
-	return fmt.Sprintf("%s/plugins/%s/%s.net.ns", kubeletDirPath, driverFullName, clusterNamespace)
-}
-
 func validateCSIDriverNamePrefix(ctx context.Context, clientset kubernetes.Interface, namespace, driverNamePrefix string) error {
 	if EnableRBD {
 		rbdDriverNamePrefix, err := getCSIDriverNamePrefixFromDeployment(ctx, clientset, namespace, csiRBDProvisioner, "csi-rbdplugin")
@@ -1106,5 +879,5 @@ func getPrefixFromArg(arg string) (string, bool) {
 }
 
 func EnableCSIOperator() bool {
-	return enableCSIOperator && !IsHolderEnabled()
+	return enableCSIOperator
 }
diff --git a/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-holder.yaml b/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-holder.yaml
deleted file mode 100644
index f29659b29..000000000
--- a/pkg/operator/ceph/csi/template/cephfs/csi-cephfsplugin-holder.yaml
+++ /dev/null
@@ -1,67 +0,0 @@
-# This DaemonSet is only active when the cluster is deployed with Multus enabled or EnableCSIHostNetwork
-# is set to false. Hopefully in the future it will be part of every Ceph-CSI deployment. Its role is to
-# "hold" the Ceph Filesystem mounts that are created by the CSI driver. Essentially, it exposes its
-# network namespace into the Kubelet plugin directly and Ceph-CSI picks it up. When mounting, the Ceph-CSI
-# driver will use the network namespace of the DaemonSet instead of its own (the plugin pod). This pod
-# is not expected to be updated nor restarted unless the node reboots.
-kind: DaemonSet
-apiVersion: apps/v1
-metadata:
-  name: csi-cephfsplugin-holder
-  namespace: {{ .Namespace }}
-spec:
-  selector:
-    matchLabels:
-      app: csi-cephfsplugin-holder
-  updateStrategy:
-    # DO NOT CHANGE THE STRATEGY
-    # The pod is not supposed to be restarted on updates, it should never receive updates too.
-    # If it does then the connectivity to the Ceph cluster will be lost and the client I/O will
-    # fail.
-    type: OnDelete
-  template:
-    metadata:
-      labels:
-        app: csi-cephfsplugin-holder
-        {{ range $key, $value := .CSICephFSPodLabels }}
-        {{ $key }}: "{{ $value }}"
-        {{ end }}
-    spec:
-      # HostPID is needed to expose the correct process ID network namespace and not the process namespace
-      hostPID: true
-      securityContext: {}
-      serviceAccountName: rook-csi-cephfs-plugin-sa
-      {{ if .PluginPriorityClassName }}
-      priorityClassName: {{ .PluginPriorityClassName }}
-      {{ end }}
-      containers:
-        - name: csi-cephfsplugin-holder
-          # This is necessary for the Bidirectional mount propagation
-          securityContext:
-            privileged: true
-            capabilities:
-              drop: ["ALL"]
-          image: {{ .CSIPluginImage }}
-          command:
-            - "/bin/sh"
-            - "-c"
-          # the third $ allows the "$$" to be interpreted so it can return the current bash process ID
-          args:
-            - "ln
-              --force
-              --symbolic
-              --verbose
-              /proc/$$$/ns/net
-              {{ .KubeletDirPath }}/plugins/$ROOK_CEPH_CSI_DRIVER_NAME/$CEPH_CLUSTER_NAMESPACE.net.ns
-              &&
-              sleep
-              infinity"
-          volumeMounts:
-            - name: plugin-mount-dir
-              mountPath: "{{ .KubeletDirPath }}/plugins"
-              mountPropagation: "Bidirectional"
-      volumes:
-        - name: plugin-mount-dir
-          hostPath:
-            path: "{{ .KubeletDirPath }}/plugins"
-            type: Directory
diff --git a/pkg/operator/ceph/csi/template/nfs/csi-nfsplugin-holder.yaml b/pkg/operator/ceph/csi/template/nfs/csi-nfsplugin-holder.yaml
deleted file mode 100644
index 240d4b6b5..000000000
--- a/pkg/operator/ceph/csi/template/nfs/csi-nfsplugin-holder.yaml
+++ /dev/null
@@ -1,67 +0,0 @@
-# This DaemonSet is only active when the cluster is deployed with Multus enabled or EnableCSIHostNetwork
-# is set to false. Hopefully in the future it will be part of every Ceph-CSI deployment. Its role is to
-# "hold" the Ceph Filesystem mounts that are created by the CSI driver. Essentially, it exposes its
-# network namespace into the Kubelet plugin directly and Ceph-CSI picks it up. When mounting, the Ceph-CSI
-# driver will use the network namespace of the DaemonSet instead of its own (the plugin pod). This pod
-# is not expected to be updated nor restarted unless the node reboots.
-kind: DaemonSet
-apiVersion: apps/v1
-metadata:
-  name: csi-nfsplugin-holder
-  namespace: {{ .Namespace }}
-spec:
-  selector:
-    matchLabels:
-      app: csi-nfsplugin-holder
-  updateStrategy:
-    # DO NOT CHANGE THE STRATEGY
-    # The pod is not supposed to be restarted on updates, it should never receive updates too.
-    # If it does then the connectivity to the Ceph cluster will be lost and the client I/O will
-    # fail.
-    type: OnDelete
-  template:
-    metadata:
-      labels:
-        app: csi-nfsplugin-holder
-        {{ range $key, $value := .CSINFSPodLabels }}
-        {{ $key }}: "{{ $value }}"
-        {{ end }}
-    spec:
-      # HostPID is needed to expose the correct process ID network namespace and not the process namespace
-      hostPID: true
-      securityContext: {}
-      serviceAccountName: rook-csi-nfs-plugin-sa
-      {{ if .PluginPriorityClassName }}
-      priorityClassName: {{ .PluginPriorityClassName }}
-      {{ end }}
-      containers:
-        - name: csi-nfsplugin-holder
-          # This is necessary for the Bidirectional mount propagation
-          securityContext:
-            privileged: true
-            capabilities:
-              drop: ["ALL"]
-          image: {{ .CSIPluginImage }}
-          command:
-            - "/bin/sh"
-            - "-c"
-          # the third $ allows the "$$" to be interpreted so it can return the current bash process ID
-          args:
-            - "ln
-              --force
-              --symbolic
-              --verbose
-              /proc/$$$/ns/net
-              {{ .KubeletDirPath }}/plugins/$ROOK_CEPH_CSI_DRIVER_NAME/$CEPH_CLUSTER_NAMESPACE.net.ns
-              &&
-              sleep
-              infinity"
-          volumeMounts:
-            - name: plugin-mount-dir
-              mountPath: "{{ .KubeletDirPath }}/plugins"
-              mountPropagation: "Bidirectional"
-      volumes:
-        - name: plugin-mount-dir
-          hostPath:
-            path: "{{ .KubeletDirPath }}/plugins"
-            type: Directory
diff --git a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-holder.yaml b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-holder.yaml
deleted file mode 100644
index c8f11c5cc..000000000
--- a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-holder.yaml
+++ /dev/null
@@ -1,67 +0,0 @@
-# This DaemonSet is only active when the cluster is deployed with Multus enabled or EnableCSIHostNetwork
-# is set to false. Hopefully in the future it will be part of every Ceph-CSI deployment. Its role is
-# to "hold" the network RBD block mapping that are created by the CSI driver. Essentially, it exposes
-# its network namespace into the Kubelet plugin directly and Ceph-CSI picks it up. When mapping an RBD
-# image, the Ceph-CSI driver will use the network namespace of the DaemonSet instead of its own (the plugin pod).
-# This pod is not expected to be updated nor restarted unless the node reboots.
-kind: DaemonSet
-apiVersion: apps/v1
-metadata:
-  name: csi-rbdplugin-holder
-  namespace: {{ .Namespace }}
-spec:
-  selector:
-    matchLabels:
-      app: csi-rbdplugin-holder
-  updateStrategy:
-    # DO NOT CHANGE THE STRATEGY
-    # The pod is not supposed to be restarted on updates, it should never receive updates too.
-    # If it does then the connectivity to the Ceph cluster will be lost and the client I/O will
-    # fail.
-    type: OnDelete
-  template:
-    metadata:
-      labels:
-        app: csi-rbdplugin-holder
-        {{ range $key, $value := .CSIRBDPodLabels }}
-        {{ $key }}: "{{ $value }}"
-        {{ end }}
-    spec:
-      # HostPID is needed to expose the correct process ID network namespace and not the process namespace
-      hostPID: true
-      securityContext: {}
-      serviceAccountName: rook-csi-rbd-plugin-sa
-      {{ if .PluginPriorityClassName }}
-      priorityClassName: {{ .PluginPriorityClassName }}
-      {{ end }}
-      containers:
-        - name: csi-rbdplugin-holder
-          # This is necessary for the Bidirectional mount propagation
-          securityContext:
-            privileged: true
-            capabilities:
-              drop: ["ALL"]
-          image: {{ .CSIPluginImage }}
-          command:
-            - "/bin/sh"
-            - "-c"
-          # the third $ allows the "$$" to be interpreted so it can return the current bash process ID
-          args:
-            - "ln
-              --force
-              --symbolic
-              --verbose
-              /proc/$$$/ns/net
-              {{ .KubeletDirPath }}/plugins/$ROOK_CEPH_CSI_DRIVER_NAME/$CEPH_CLUSTER_NAMESPACE.net.ns
-              &&
-              sleep
-              infinity"
-          volumeMounts:
-            - name: plugin-mount-dir
-              mountPath: "{{ .KubeletDirPath }}/plugins"
-              mountPropagation: "Bidirectional"
-      volumes:
-        - name: plugin-mount-dir
-          hostPath:
-            path: "{{ .KubeletDirPath }}/plugins"
-            type: Directory
diff --git a/pkg/operator/ceph/file/subvolumegroup/controller.go b/pkg/operator/ceph/file/subvolumegroup/controller.go
index 5e7cb80f7..d74abe149 100644
--- a/pkg/operator/ceph/file/subvolumegroup/controller.go
+++ b/pkg/operator/ceph/file/subvolumegroup/controller.go
@@ -311,15 +311,7 @@ func (r *ReconcileCephFilesystemSubVolumeGroup) updateClusterConfig(cephFilesyst
 		},
 	}
 
-	// If the cluster has Multus enabled we need to append the network namespace of the driver's
-	// holder DaemonSet in the csi configmap
-	if cephCluster.Spec.Network.IsMultus() {
-		netNamespaceFilePath, err := csi.GenerateNetNamespaceFilePath(r.opManagerContext, r.client, cephCluster.Namespace, r.opConfig.OperatorNamespace, csi.CephFSDriverShortName)
-		if err != nil {
-			return errors.Wrap(err, "failed to generate cephfs net namespace file path")
-		}
-		csiClusterConfigEntry.CephFS.NetNamespaceFilePath = netNamespaceFilePath
-	}
+	csiClusterConfigEntry.CephFS.NetNamespaceFilePath = ""
 
 	err := csi.SaveClusterConfig(r.context.Clientset, buildClusterID(cephFilesystemSubVolumeGroup), cephCluster.Namespace, r.clusterInfo, &csiClusterConfigEntry)
 	if err != nil {
diff --git a/pkg/operator/ceph/pool/radosnamespace/controller.go b/pkg/operator/ceph/pool/radosnamespace/controller.go
index 7bae0b595..790fd6f61 100644
--- a/pkg/operator/ceph/pool/radosnamespace/controller.go
+++ b/pkg/operator/ceph/pool/radosnamespace/controller.go
@@ -326,14 +326,7 @@ func (r *ReconcileCephBlockPoolRadosNamespace) updateClusterConfig(cephBlockPool
 		},
 	}
 
-	if cephCluster.Spec.Network.IsMultus() {
-		// Build the network namespace config to be injected into the csi config
-		netNamespaceFilePath, err := csi.GenerateNetNamespaceFilePath(r.opManagerContext, r.client, cephCluster.Namespace, r.opConfig.OperatorNamespace, csi.RBDDriverShortName)
-		if err != nil {
-			return errors.Wrap(err, "failed to generate rbd net namespace file path")
-		}
-		csiClusterConfigEntry.RBD.NetNamespaceFilePath = netNamespaceFilePath
-	}
+	csiClusterConfigEntry.RBD.NetNamespaceFilePath = ""
 
 	// Save cluster config in the csi config map
 	err := csi.SaveClusterConfig(r.context.Clientset, buildClusterID(cephBlockPoolRadosNamespace), cephCluster.Namespace, r.clusterInfo, &csiClusterConfigEntry)


Test patch:
diff --git a/.github/workflows/canary-integration-test.yml b/.github/workflows/canary-integration-test.yml
index 55dd34b0a..19157c7fc 100644
--- a/.github/workflows/canary-integration-test.yml
+++ b/.github/workflows/canary-integration-test.yml
@@ -1632,7 +1632,7 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           artifact-name: ${{ github.job }}-${{ matrix.ceph-image }}
 
-  multus-cluster-network:
+  multus-public-and-cluster:
     runs-on: ubuntu-22.04
     if: "!contains(github.event.pull_request.labels.*.name, 'skip-ci')"
     strategy:
@@ -1649,45 +1649,50 @@ jobs:
         with:
           use-tmate: ${{ secrets.USE_TMATE }}
 
-      - name: setup golang
-        uses: actions/setup-go@0a12ed9d6a96ab950c8f026ed9f722fe0da7ef32 # v5.0.2
-        with:
-          go-version: "1.22"
-
       - name: setup cluster resources
         uses: ./.github/workflows/canary-test-config
 
       - name: set Ceph version in CephCluster manifest
-        run: tests/scripts/github-action-helper.sh replace_ceph_image  "deploy/examples/cluster-test.yaml" "${{ matrix.ceph-image }}"
+        run: tests/scripts/github-action-helper.sh replace_ceph_image "deploy/examples/cluster-multus-test.yaml" "${{ matrix.ceph-image }}"
 
-      - name: allow holder pod deployment
-        run: sed -i "s|CSI_DISABLE_HOLDER_PODS|# CSI_DISABLE_HOLDER_PODS|g" "deploy/examples/operator.yaml"
+      - name: Setup multus
+        run: ./tests/scripts/multus/setup-multus.sh
 
-      - name: validate-yaml
-        run: tests/scripts/github-action-helper.sh validate_yaml
+      - name: Set up multus prerequisite host routing
+        run: kubectl create -f tests/scripts/multus/host-cfg-ds.yaml
+
+      - name: create cluster prerequisites
+        run: tests/scripts/github-action-helper.sh create_cluster_prerequisites
+
+      - name: Install public and cluster NADs in rook-ceph namespace
+        run: kubectl create -f tests/scripts/multus/rook-ceph-public-cluster-nads.yaml
 
       - name: use local disk and create partitions for osds
         run: |
           tests/scripts/github-action-helper.sh use_local_disk
           tests/scripts/github-action-helper.sh create_partitions_for_osds
 
-      - name: deploy multus
-        run: tests/scripts/github-action-helper.sh deploy_multus
+      - name: deploy cluster
+        run: |
+          tests/scripts/github-action-helper.sh deploy_manifest_with_local_build deploy/examples/operator.yaml
+
+          export BLOCK="$(tests/scripts/github-action-helper.sh find_extra_block_dev)"
+          yq write -i deploy/examples/cluster-multus-test.yaml "spec.storage.deviceFilter" "${BLOCK}"
+          kubectl create -f deploy/examples/cluster-multus-test.yaml
 
-      - name: deploy multus cluster
-        run: tests/scripts/github-action-helper.sh deploy_multus_cluster
+          tests/scripts/github-action-helper.sh deploy_toolbox
+
+          kubectl create -f deploy/examples/filesystem-test.yaml
+          kubectl create -f deploy/examples/nfs-test.yaml
 
       - name: wait for prepare pod
         run: tests/scripts/github-action-helper.sh wait_for_prepare_pod 2
 
       - name: wait for ceph to be ready
-        run: IS_POD_NETWORK=true IS_MULTUS=true tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready osd 2
-
-      - name: wait for ceph-csi configmap to be updated with network namespace
-        run: tests/scripts/github-action-helper.sh wait_for_ceph_csi_configmap_to_be_updated
+        run: tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready osd 2
 
       - name: wait for cephnfs to be ready
-        run: IS_POD_NETWORK=true IS_MULTUS=true tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready nfs 1
+        run: tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready nfs 1
 
       - name: check multus connections
         run: tests/scripts/github-action-helper.sh test_multus_connections
@@ -1708,64 +1713,6 @@ jobs:
           name: ${{ github.job }}-${{ matrix.ceph-image }}
           additional-namespace: kube-system
 
-  csi-hostnetwork-disabled:
-    runs-on: ubuntu-22.04
-    if: "!contains(github.event.pull_request.labels.*.name, 'skip-ci')"
-    strategy:
-      matrix:
-        ceph-image: ${{ fromJson(inputs.ceph_images) }}
-    steps:
-      - name: checkout
-        uses: actions/checkout@eef61447b9ff4aafe5dcd4e0bbf5d482be7e7871 # v4.2.1
-        with:
-          fetch-depth: 0
-
-      - name: consider debugging
-        uses: ./.github/workflows/tmate_debug
-        with:
-          use-tmate: ${{ secrets.USE_TMATE }}
-
-      - name: setup cluster resources
-        uses: ./.github/workflows/canary-test-config
-
-      - name: allow holder pod deployment
-        run: sed -i "s|CSI_DISABLE_HOLDER_PODS|# CSI_DISABLE_HOLDER_PODS|g" "deploy/examples/operator.yaml"
-
-      - name: set Ceph version in CephCluster manifest
-        run: tests/scripts/github-action-helper.sh replace_ceph_image  "deploy/examples/cluster-test.yaml" "${{ matrix.ceph-image }}"
-
-      - name: use local disk and create partitions for osds
-        run: |
-          tests/scripts/github-action-helper.sh use_local_disk
-          tests/scripts/github-action-helper.sh create_partitions_for_osds
-
-      - name: deploy CSI hostNetworking disabled cluster
-        run: tests/scripts/github-action-helper.sh deploy_csi_hostnetwork_disabled_cluster
-
-      - name: wait for prepare pod
-        run: tests/scripts/github-action-helper.sh wait_for_prepare_pod 2
-
-      - name: wait for ceph to be ready
-        run: IS_POD_NETWORK=true tests/scripts/github-action-helper.sh wait_for_ceph_to_be_ready osd 2
-
-      - name: wait for ceph-csi configmap to be updated with network namespace
-        run: tests/scripts/github-action-helper.sh wait_for_ceph_csi_configmap_to_be_updated
-
-      - name: test ceph-csi-rbd plugin restart
-        run: tests/scripts/github-action-helper.sh test_csi_rbd_workload
-
-      - name: test ceph-csi-cephfs plugin restart
-        run: tests/scripts/github-action-helper.sh test_csi_cephfs_workload
-
-      - name: test ceph-csi-nfs plugin restart
-        run: tests/scripts/github-action-helper.sh test_csi_nfs_workload
-
-      - name: collect common logs
-        if: always()
-        uses: ./.github/workflows/collect-logs
-        with:
-          name: ${{ github.job }}-${{ matrix.ceph-image }}
-
   two-object-one-zone:
     runs-on: ubuntu-22.04
     if: "!contains(github.event.pull_request.labels.*.name, 'skip-ci')"
diff --git a/pkg/operator/ceph/cluster/cephstatus_test.go b/pkg/operator/ceph/cluster/cephstatus_test.go
index d464f3a52..183a4b4c5 100644
--- a/pkg/operator/ceph/cluster/cephstatus_test.go
+++ b/pkg/operator/ceph/cluster/cephstatus_test.go
@@ -324,13 +324,10 @@ func TestGetRookPodsOnNode(t *testing.T) {
 		{"app": "rook-ceph-osd"},
 		{"app": "csi-rbdplugin-provisioner"},
 		{"app": "csi-rbdplugin"},
-		{"app": "csi-rbdplugin-holder"},
 		{"app": "csi-cephfsplugin-provisioner"},
 		{"app": "csi-cephfsplugin"},
-		{"app": "csi-cephfsplugin-holder"},
 		{"app": "csi-nfsplugin-provisioner"},
 		{"app": "csi-nfsplugin"},
-		{"app": "csi-nfsplugin-holder"},
 		{"app": "rook-ceph-operator"},
 		{"app": "rook-ceph-crashcollector"},
 		{"app": "rook-ceph-mgr"},
@@ -366,7 +363,7 @@ func TestGetRookPodsOnNode(t *testing.T) {
 	pods, err := c.getRookPodsOnNode("node0")
 	assert.NoError(t, err)
 	// A pod is having two matching labels and its returned only once
-	assert.Equal(t, 17, len(pods))
+	assert.Equal(t, 14, len(pods))
 
 	podNames := []string{}
 	for _, pod := range pods {
diff --git a/pkg/operator/ceph/csi/controller_test.go b/pkg/operator/ceph/csi/controller_test.go
index 2860c6d67..b1998bab4 100644
--- a/pkg/operator/ceph/csi/controller_test.go
+++ b/pkg/operator/ceph/csi/controller_test.go
@@ -264,12 +264,10 @@ func TestCephCSIController(t *testing.T) {
 
 		ds, err := c.Clientset.AppsV1().DaemonSets(namespace).List(ctx, metav1.ListOptions{})
 		assert.NoError(t, err)
-		assert.Equal(t, 4, len(ds.Items), ds)
+		assert.Equal(t, 2, len(ds.Items), ds)
 		assert.Equal(t, "csi-cephfsplugin", ds.Items[0].Name)
-		assert.Equal(t, "csi-cephfsplugin-holder-rook-ceph", ds.Items[1].Name)
-		assert.Equal(t, "csi-rbdplugin", ds.Items[2].Name)
-		assert.Equal(t, "csi-rbdplugin-holder-rook-ceph", ds.Items[3].Name)
-		assert.Equal(t, `[{"name":"public-net","namespace":"rook-ceph"}]`, ds.Items[1].Spec.Template.Annotations["k8s.v1.cni.cncf.io/networks"], ds.Items[1].Spec.Template.Annotations)
+		assert.Equal(t, "csi-rbdplugin", ds.Items[1].Name)
+		assert.Equal(t, ``, ds.Items[1].Spec.Template.Annotations["k8s.v1.cni.cncf.io/networks"], ds.Items[1].Spec.Template.Annotations)
 
 		assert.Equal(t, []string{namespace}, saveCSIDriverOptionsCalledForClusterNS)
 	})
diff --git a/pkg/operator/ceph/csi/operator_config_test.go b/pkg/operator/ceph/csi/operator_config_test.go
index 4beb6a9a5..33f10c2a5 100644
--- a/pkg/operator/ceph/csi/operator_config_test.go
+++ b/pkg/operator/ceph/csi/operator_config_test.go
@@ -46,7 +46,6 @@ func TestReconcileCSI_createOrUpdateOperatorConfig(t *testing.T) {
 		opConfig: opcontroller.OperatorConfig{
 			OperatorNamespace: "test",
 		},
-		clustersWithHolder: []ClusterDetail{},
 	}
 	cluster := &cephv1.CephCluster{
 		ObjectMeta: metav1.ObjectMeta{
diff --git a/pkg/operator/ceph/csi/operator_driver_test.go b/pkg/operator/ceph/csi/operator_driver_test.go
index c1c0c2831..647cd6fa2 100644
--- a/pkg/operator/ceph/csi/operator_driver_test.go
+++ b/pkg/operator/ceph/csi/operator_driver_test.go
@@ -48,7 +48,6 @@ func TestReconcileCSI_createOrUpdateDriverResources(t *testing.T) {
 		opConfig: opcontroller.OperatorConfig{
 			OperatorNamespace: "test",
 		},
-		clustersWithHolder: []ClusterDetail{},
 	}
 	cluster := &cephv1.CephCluster{
 		ObjectMeta: metav1.ObjectMeta{
diff --git a/pkg/operator/ceph/csi/spec_test.go b/pkg/operator/ceph/csi/spec_test.go
index 5e75fdfbf..603ec9f46 100644
--- a/pkg/operator/ceph/csi/spec_test.go
+++ b/pkg/operator/ceph/csi/spec_test.go
@@ -21,117 +21,13 @@ import (
 	_ "embed"
 	"testing"
 
-	cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
-	rookclient "github.com/rook/rook/pkg/client/clientset/versioned/fake"
-	"github.com/rook/rook/pkg/clusterd"
-	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
-	opcontroller "github.com/rook/rook/pkg/operator/ceph/controller"
-	"github.com/rook/rook/pkg/operator/k8sutil"
-	testop "github.com/rook/rook/pkg/operator/test"
 	"github.com/stretchr/testify/assert"
 	apps "k8s.io/api/apps/v1"
 	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/runtime"
 	kfake "k8s.io/client-go/kubernetes/fake"
-	"sigs.k8s.io/controller-runtime/pkg/client/fake"
 )
 
-func TestReconcileCSI_configureHolders(t *testing.T) {
-	r := &ReconcileCSI{
-		context: &clusterd.Context{
-			Clientset:     testop.New(t, 1),
-			RookClientset: rookclient.NewSimpleClientset(),
-		},
-		opManagerContext:   context.TODO(),
-		opConfig:           opcontroller.OperatorConfig{},
-		clustersWithHolder: []ClusterDetail{},
-	}
-
-	t.Run("no clusters, noop", func(t *testing.T) {
-		err := r.configureHolders([]driverDetails{}, templateParam{}, []v1.Toleration{}, nil)
-		assert.NoError(t, err)
-	})
-
-	t.Run("one multus cluster", func(t *testing.T) {
-		r.opConfig.OperatorNamespace = "rook-ceph"
-		driverDetails := []driverDetails{
-			{
-				name:           "rbd",
-				fullName:       "rbd.csi.ceph.com",
-				holderTemplate: CephFSPluginHolderTemplatePath,
-				nodeAffinity:   cephFSPluginNodeAffinityEnv,
-				toleration:     cephFSPluginTolerationsEnv,
-			},
-		}
-		tp := templateParam{
-			Param:     CSIParam,
-			Namespace: r.opConfig.OperatorNamespace,
-		}
-
-		r.clustersWithHolder = []ClusterDetail{
-			{
-				cluster: &cephv1.CephCluster{
-					ObjectMeta: metav1.ObjectMeta{Namespace: "rook-ceph", Name: "my-cluster"},
-					Spec:       cephv1.ClusterSpec{},
-				},
-				clusterInfo: &cephclient.ClusterInfo{
-					Monitors:  map[string]*cephclient.MonInfo{"a": {Name: "a", Endpoint: "10.0.0.1:6789"}},
-					OwnerInfo: cephclient.NewMinimumOwnerInfoWithOwnerRef()},
-			},
-		}
-
-		t.Setenv(k8sutil.PodNamespaceEnvVar, "rook-ceph")
-		_, err := r.context.Clientset.CoreV1().ConfigMaps("rook-ceph").Create(r.opManagerContext, &v1.ConfigMap{ObjectMeta: metav1.ObjectMeta{Name: ConfigName, Namespace: "rook-ceph"}, Data: map[string]string{}}, metav1.CreateOptions{})
-		assert.NoError(t, err)
-
-		err = r.configureHolders(driverDetails, tp, []v1.Toleration{}, nil)
-		assert.NoError(t, err)
-	})
-}
-
-func TestGenerateNetNamespaceFilePath(t *testing.T) {
-	ctx := context.TODO()
-
-	t.Run("generate with no op configmap available and non supported driver name", func(t *testing.T) {
-		client := fake.NewClientBuilder().Build()
-		netNsFilePath, err := GenerateNetNamespaceFilePath(ctx, client, "rook-ceph", "rook-ceph", "foo")
-		assert.Error(t, err)
-		assert.Empty(t, "", netNsFilePath)
-	})
-
-	t.Run("generate with no op configmap available for rbd", func(t *testing.T) {
-		client := fake.NewClientBuilder().Build()
-		netNsFilePath, err := GenerateNetNamespaceFilePath(ctx, client, "rook-ceph", "rook-ceph", "rbd")
-		assert.NoError(t, err)
-		assert.Equal(t, "/var/lib/kubelet/plugins/rook-ceph.rbd.csi.ceph.com/rook-ceph.net.ns", netNsFilePath)
-	})
-
-	t.Run("generate with no op configmap available for cephfs", func(t *testing.T) {
-		client := fake.NewClientBuilder().Build()
-		netNsFilePath, err := GenerateNetNamespaceFilePath(ctx, client, "rook-ceph", "rook-ceph", "cephfs")
-		assert.NoError(t, err)
-		assert.Equal(t, "/var/lib/kubelet/plugins/rook-ceph.cephfs.csi.ceph.com/rook-ceph.net.ns", netNsFilePath)
-	})
-
-	t.Run("generate with op configmap for cephfs", func(t *testing.T) {
-		opCm := &v1.ConfigMap{
-			ObjectMeta: metav1.ObjectMeta{
-				Name:      opcontroller.OperatorSettingConfigMapName,
-				Namespace: "rook-ceph",
-			},
-			Data: map[string]string{"ROOK_CSI_KUBELET_DIR_PATH": "/foo"},
-		}
-		object := []runtime.Object{
-			opCm,
-		}
-		client := fake.NewClientBuilder().WithRuntimeObjects(object...).Build()
-		netNsFilePath, err := GenerateNetNamespaceFilePath(ctx, client, "rook-ceph", "rook-ceph", "cephfs")
-		assert.NoError(t, err)
-		assert.Equal(t, "/foo/plugins/rook-ceph.cephfs.csi.ceph.com/rook-ceph.net.ns", netNsFilePath)
-	})
-}
-
 func Test_getCSIDriverNamePrefixFromDeployment(t *testing.T) {
 	namespace := "test"
 	deployment := func(name, containerName, drivernameSuffix string) *apps.Deployment {
diff --git a/tests/scripts/github-action-helper.sh b/tests/scripts/github-action-helper.sh
index c2735d260..558c315a5 100755
--- a/tests/scripts/github-action-helper.sh
+++ b/tests/scripts/github-action-helper.sh
@@ -615,58 +615,6 @@ function create_helm_tag() {
   docker tag "${build_image}" "rook/ceph:${helm_tag}"
 }
 
-function deploy_multus() {
-  # download the multus daemonset, and remove mem and cpu limits that cause it to crash on minikube
-  curl https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml |
-    sed -e 's/cpu: /# cpu: /g' -e 's/memory: /# memory: /g' |
-    kubectl apply -f -
-
-  # install whereabouts
-  kubectl apply \
-    -f https://raw.githubusercontent.com/k8snetworkplumbingwg/whereabouts/master/doc/crds/daemonset-install.yaml \
-    -f https://github.com/k8snetworkplumbingwg/whereabouts/raw/master/doc/crds/whereabouts.cni.cncf.io_ippools.yaml \
-    -f https://github.com/k8snetworkplumbingwg/whereabouts/raw/master/doc/crds/whereabouts.cni.cncf.io_overlappingrangeipreservations.yaml
-
-  # create the rook-ceph namespace if it doesn't exist, the NAD will go in this namespace
-  kubectl create namespace rook-ceph || true
-
-  # install network attachment definitions
-  IFACE="eth0" # the runner has eth0 so we don't need any heuristics to find the interface
-  kubectl apply -f - <<EOF
----
-apiVersion: k8s.cni.cncf.io/v1
-kind: NetworkAttachmentDefinition
-metadata:
-  name: public-net
-  namespace: rook-ceph
-  labels:
-  annotations:
-spec:
-  config: '{ "cniVersion": "0.3.0", "type": "macvlan", "master": "$IFACE", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.20.0/24" } }'
----
-apiVersion: k8s.cni.cncf.io/v1
-kind: NetworkAttachmentDefinition
-metadata:
-  name: cluster-net
-  namespace: rook-ceph
-  labels:
-  annotations:
-spec:
-  config: '{ "cniVersion": "0.3.0", "type": "macvlan", "master": "$IFACE", "mode": "bridge", "ipam": { "type": "whereabouts", "range": "192.168.21.0/24" } }'
-EOF
-}
-
-function deploy_multus_cluster() {
-  cd "${REPO_DIR}/deploy/examples"
-  sed -i 's/.*ROOK_CSI_ENABLE_NFS:.*/  ROOK_CSI_ENABLE_NFS: \"true\"/g' operator.yaml
-  deploy_manifest_with_local_build operator.yaml
-  deploy_toolbox
-  sed -i "s|#deviceFilter:|deviceFilter: $(block_dev_basename)|g" cluster-multus-test.yaml
-  kubectl create -f cluster-multus-test.yaml
-  kubectl create -f filesystem-test.yaml
-  kubectl create -f nfs-test.yaml
-}
-
 function test_multus_connections() {
   EXEC='kubectl -n rook-ceph exec -t deploy/rook-ceph-tools -- ceph --connect-timeout 10'
   # each OSD should exist on both public and cluster network
diff --git a/tests/scripts/multus/default-public-cluster-nads.yaml b/tests/scripts/multus/default-public-cluster-nads.yaml
index 28ccd184a..2747741cc 100644
--- a/tests/scripts/multus/default-public-cluster-nads.yaml
+++ b/tests/scripts/multus/default-public-cluster-nads.yaml
@@ -22,6 +22,8 @@ spec:
       }
     }
 ---
+# IPv6 address here (with IPv4 above) makes this unsuitable for CephCluster use, but it is okay for
+# testing validation tool
 apiVersion: k8s.cni.cncf.io/v1
 kind: NetworkAttachmentDefinition
 metadata:
diff --git a/tests/scripts/multus/host-cfg-ds.yaml b/tests/scripts/multus/host-cfg-ds.yaml
index 113db80e9..9fbedcfa3 100644
--- a/tests/scripts/multus/host-cfg-ds.yaml
+++ b/tests/scripts/multus/host-cfg-ds.yaml
@@ -64,5 +64,6 @@ spec:
               sleep infinity
           resources: {}
           securityContext:
+            privileged: true
             capabilities:
               add: ["NET_ADMIN"]
diff --git a/tests/scripts/multus/rook-ceph-public-cluster-nads.yaml b/tests/scripts/multus/rook-ceph-public-cluster-nads.yaml
new file mode 100644
index 000000000..a45eb7224
--- /dev/null
+++ b/tests/scripts/multus/rook-ceph-public-cluster-nads.yaml
@@ -0,0 +1,43 @@
+---
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: public-net
+  namespace: rook-ceph
+  labels:
+  annotations:
+spec:
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "type": "macvlan",
+      "master": "eth0",
+      "mode": "bridge",
+      "ipam": {
+        "type": "whereabouts",
+        "range": "192.168.20.0/24",
+        "routes": [
+          {"dst": "192.168.29.0/24"}
+        ]
+      }
+    }
+---
+apiVersion: k8s.cni.cncf.io/v1
+kind: NetworkAttachmentDefinition
+metadata:
+  name: cluster-net
+  namespace: rook-ceph
+  labels:
+  annotations:
+spec:
+  config: |
+    {
+      "cniVersion": "0.3.1",
+      "type": "macvlan",
+      "master": "eth0",
+      "mode": "bridge",
+      "ipam": {
+        "type": "whereabouts",
+        "range": "192.168.21.0/24"
+      }
+    }
diff --git a/tests/scripts/validate_cluster.sh b/tests/scripts/validate_cluster.sh
index 091e83caa..eeac1344b 100755
--- a/tests/scripts/validate_cluster.sh
+++ b/tests/scripts/validate_cluster.sh
@@ -103,25 +103,10 @@ function test_demo_pool {
 
 function test_csi {
   timeout 360 bash -x <<-'EOF'
-    echo $IS_POD_NETWORK
-    echo $IS_MULTUS
-    if [ -z "$IS_POD_NETWORK" ]; then
-      until [[ "$(kubectl -n rook-ceph get pods --field-selector=status.phase=Running|grep -c ^csi-)" -eq 6 ]]; do
-        echo "waiting for csi pods to be ready"
-        sleep 5
-      done
-    else
-      until [[ "$(kubectl -n rook-ceph get pods --field-selector=status.phase=Running|grep -c ^csi-)" -eq 9 ]]; do
-        echo "waiting for csi pods to be ready with multus or pod networking"
-        sleep 5
-      done
-    fi
-    if [ -n "$IS_MULTUS" ]; then
-      echo "verifying csi holder interfaces (multus ones must be present)"
-      kubectl -n rook-ceph exec -t ds/csi-rbdplugin-holder-my-cluster -- grep eth0 /proc/net/dev
-      kubectl -n rook-ceph exec -t ds/csi-cephfsplugin-holder-my-cluster -- grep eth0 /proc/net/dev
-      kubectl -n rook-ceph exec -t ds/csi-nfsplugin-holder-my-cluster -- grep eth0 /proc/net/dev
-    fi
+    until [[ "$(kubectl -n rook-ceph get pods --field-selector=status.phase=Running|grep -c ^csi-)" -eq 6 ]]; do
+      echo "waiting for csi pods to be ready"
+      sleep 5
+    done
 EOF
 }