Repository: rook/rook
Pull request #13200 (2023-11-14 12:37:26)

csi: add CSIDriverOptions section in cephCluster CR
This commit adds new CSIDriverOptions section in
cephCluster CR. This section contains settings
for read affinity and kernel+fuse Mount options
These settings will be injected directly into
rook-ceph-csi-config cm to be applicable per
ceph cluster.

**Which issue is resolved by this Pull Request:**
Resolves #

**Checklist:**

- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/latest/Contributing/development-flow/#commit-structure).
- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/latest/Contributing/development-flow/#submitting-a-pull-request)
- [ ] [Pending release notes](https://github.com/rook/rook/blob/master/PendingReleaseNotes.md) updated with breaking and/or notable changes for the next minor release.
- [ ] Documentation has been updated, if necessary.
- [ ] Unit tests have been added, if necessary.
- [ ] Integration tests have been added, if necessary.


travisn: > We recently added COSI to Rook, and in that, we decided to have the COSI driver options be its own CR. I would suggest that we take this approach for CSI as well. This will keep us from over-complicating the CephCluster CR more than it already is. It will keep consistency between CSI and COSI drivers.
> 
> Also, I think this makes more sense when considering Rook as a whole: Rook runs one CSI driver per k8s cluster regardless of how many CephClusters are present, so I think it makes most logical sense to have a separate CR for it. We wouldn't want to have the CephCluster that configured the CSI driver to be deleted and then to have an additional CephCluster then have its CSI settings changed or removed as a result.

The CSI settings being considered are cluster-specific. Any configuration of these settings would not affect other clusters upon creation or deletion. During the cluster reconcile, Rook creates/updates a configmap specific for a given cluster with settings related to that cluster. For example, the mon endpoints are included in the configmap. The proposal with these settings is that they would also be specific to the cluster, and would be added to the configmap only for that cluster. Note the configmap is in the operator namespace, but has a section for each cluster. 

Since the configmap is already updated during the cephcluster reconcile, and these proposed settings would be cluster-specific, we could choose either to have a CRD for the csi settings, or add them to the cephcluster CR. If we define a new CRD, the cluster controller could reconcile it, rather than implementing a new controller. I could see the new CRD being more clear from the user perspective, even if under the covers the two approaches are functionally the same.
Rakshith-R: > > We recently added COSI to Rook, and in that, we decided to have the COSI driver options be its own CR. I would suggest that we take this approach for CSI as well. This will keep us from over-complicating the CephCluster CR more than it already is. It will keep consistency between CSI and COSI drivers.
> > Also, I think this makes more sense when considering Rook as a whole: Rook runs one CSI driver per k8s cluster regardless of how many CephClusters are present, so I think it makes most logical sense to have a separate CR for it. We wouldn't want to have the CephCluster that configured the CSI driver to be deleted and then to have an additional CephCluster then have its CSI settings changed or removed as a result.
> 
> The CSI settings being considered are cluster-specific. Any configuration of these settings would not affect other clusters upon creation or deletion. During the cluster reconcile, Rook creates/updates a configmap specific for a given cluster with settings related to that cluster. For example, the mon endpoints are included in the configmap. The proposal with these settings is that they would also be specific to the cluster, and would be added to the configmap only for that cluster. Note the configmap is in the operator namespace, but has a section for each cluster.
> 
> Since the configmap is already updated during the cephcluster reconcile, and these proposed settings would be cluster-specific, we could choose either to have a CRD for the csi settings, or add them to the cephcluster CR. If we define a new CRD, the cluster controller could reconcile it, rather than implementing a new controller. I could see the new CRD being more clear from the user perspective, even if under the covers the two approaches are functionally the same.

Apart from the above points: CSI settings being cluster-specific and configmap already being filled during cephcluster reconcile. 

Having a separate CR would lead to unnecessary duplication of code and complicate the logic further. 
We just inject the settings into the configmap and there's no other functionality for this separate CR. 
It would complicate current code that injects mon ips and radosnamespace/subvolumegroup too.

I'd prefer we keep the csi driver settings within cephcluster CR (a single source of truth).
BlaineEXE: I chatted with Travis and now understand that there are CephCluster-specific settings for CSI. I misunderstood initially, so thanks for your patience @Rakshith-R .

I also think that keeping the configs with the CephCluster makes sense in that light. I don't think we gain anything by using a separate CRD.
Rakshith-R: >We couldn't do it automatically in the past because of the issue with ceph clusters that might have different settings. Now with the cluster settings being independent, seems like we can set it automatically when encryption is enabled. This setting would just allow them to override the defaults. 

I am not sure how it might affect certain cases like upgrades or toggling network encryption status. 
I've opened a new issue #13227 to discuss this. We can take up this change in a follow-up pr.

Patch:
diff --git a/Documentation/CRDs/specification.md b/Documentation/CRDs/specification.md
index 99155d028..4283a8f9e 100644
--- a/Documentation/CRDs/specification.md
+++ b/Documentation/CRDs/specification.md
@@ -1099,6 +1099,20 @@ LogCollectorSpec
 <p>Logging represents loggings settings</p>
 </td>
 </tr>
+<tr>
+<td>
+<code>csi</code><br/>
+<em>
+<a href="#ceph.rook.io/v1.CSIDriverSpec">
+CSIDriverSpec
+</a>
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>CSI Driver Options applied per cluster.</p>
+</td>
+</tr>
 </table>
 </td>
 </tr>
@@ -2794,6 +2808,94 @@ case where the range spec is forgotten (e.g., /24). Rook does in-depth validatio
 </td>
 </tr></tbody>
 </table>
+<h3 id="ceph.rook.io/v1.CSICephFSSpec">CSICephFSSpec
+</h3>
+<p>
+(<em>Appears on:</em><a href="#ceph.rook.io/v1.CSIDriverSpec">CSIDriverSpec</a>)
+</p>
+<div>
+<p>CSICephFSSpec defines the settings for CephFS CSI driver.</p>
+</div>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>kernelMountOptions</code><br/>
+<em>
+string
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>KernelMountOptions defines the mount options for kernel mounter.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>fuseMountOptions</code><br/>
+<em>
+string
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>FuseMountOptions defines the mount options for ceph fuse mounter.</p>
+</td>
+</tr>
+</tbody>
+</table>
+<h3 id="ceph.rook.io/v1.CSIDriverSpec">CSIDriverSpec
+</h3>
+<p>
+(<em>Appears on:</em><a href="#ceph.rook.io/v1.ClusterSpec">ClusterSpec</a>)
+</p>
+<div>
+<p>CSIDriverSpec defines CSI Driver settings applied per cluster.</p>
+</div>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>readAffinity</code><br/>
+<em>
+<a href="#ceph.rook.io/v1.ReadAffinitySpec">
+ReadAffinitySpec
+</a>
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>ReadAffinity defines the read affinity settings for CSI driver.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>cephfs</code><br/>
+<em>
+<a href="#ceph.rook.io/v1.CSICephFSSpec">
+CSICephFSSpec
+</a>
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>CephFS defines CSI Driver settings for CephFS driver.</p>
+</td>
+</tr>
+</tbody>
+</table>
 <h3 id="ceph.rook.io/v1.Capacity">Capacity
 </h3>
 <p>
@@ -4249,6 +4351,20 @@ LogCollectorSpec
 <p>Logging represents loggings settings</p>
 </td>
 </tr>
+<tr>
+<td>
+<code>csi</code><br/>
+<em>
+<a href="#ceph.rook.io/v1.CSIDriverSpec">
+CSIDriverSpec
+</a>
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>CSI Driver Options applied per cluster.</p>
+</td>
+</tr>
 </tbody>
 </table>
 <h3 id="ceph.rook.io/v1.ClusterState">ClusterState
@@ -10309,6 +10425,50 @@ optional</p>
 </tr>
 </tbody>
 </table>
+<h3 id="ceph.rook.io/v1.ReadAffinitySpec">ReadAffinitySpec
+</h3>
+<p>
+(<em>Appears on:</em><a href="#ceph.rook.io/v1.CSIDriverSpec">CSIDriverSpec</a>)
+</p>
+<div>
+<p>ReadAffinitySpec defines the read affinity settings for CSI driver.</p>
+</div>
+<table>
+<thead>
+<tr>
+<th>Field</th>
+<th>Description</th>
+</tr>
+</thead>
+<tbody>
+<tr>
+<td>
+<code>enabled</code><br/>
+<em>
+bool
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>Enables read affinity for CSI driver.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>crushLocationLabels</code><br/>
+<em>
+[]string
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>CrushLocationLabels defines which node labels to use
+as CRUSH location. This should correspond to the values set in
+the CRUSH map.</p>
+</td>
+</tr>
+</tbody>
+</table>
 <h3 id="ceph.rook.io/v1.ReplicatedSpec">ReplicatedSpec
 </h3>
 <p>
diff --git a/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md b/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md
index 43eed829f..451ce7a63 100644
--- a/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md
+++ b/Documentation/Storage-Configuration/Ceph-CSI/ceph-csi-drivers.md
@@ -249,22 +249,23 @@ OSD locations defined in the CRUSH map and topology labels on nodes.
 Refer to the [krbd-options](https://docs.ceph.com/en/latest/man/8/rbd/#kernel-rbd-krbd-options)
 for more details.
 
-Execute the following steps:
+Execute the following step to enable read affinity for a specific ceph cluster:
 
-* Patch the `rook-ceph-operator-config` configmap using the following
-command.
+* Patch the ceph cluster CR to enable read affinity:
 
 ```console
-kubectl patch cm rook-ceph-operator-config -nrook-ceph -p $'data:\n "CSI_ENABLE_READ_AFFINITY": "true"'
+kubectl patch cephclusters.ceph.rook.io <cluster-name> -n rook-ceph -p '{"spec":{"csi":{"readAffinity":{"enabled": true}}}}'
+```
+
+```yaml
+  csi:
+    readAffinity:
+      enabled: true
 ```
 
 * Add topology labels to the Kubernetes nodes. The same labels may be used as mentioned in the
 [OSD topology](../../CRDs/Cluster/ceph-cluster-crd.md#osd-topology) topic.
 
-* (optional) Rook will pass the labels mentioned in [osd-topology](../../CRDs/Cluster/ceph-cluster-crd.md#osd-topology)
-as the default set of labels. This can overridden to supply custom labels by updating the
-`CSI_CRUSH_LOCATION_LABELS` value in the `rook-ceph-operator-config` configmap.
-
 Ceph CSI will extract the CRUSH location from the topology labels found on the node
 and pass it though krbd options during mapping RBD volumes.
 
diff --git a/deploy/charts/rook-ceph/templates/resources.yaml b/deploy/charts/rook-ceph/templates/resources.yaml
index 103301d0b..e13a005cd 100644
--- a/deploy/charts/rook-ceph/templates/resources.yaml
+++ b/deploy/charts/rook-ceph/templates/resources.yaml
@@ -962,6 +962,32 @@ spec:
                       description: Disable determines whether we should enable the crash collector
                       type: boolean
                   type: object
+                csi:
+                  description: CSI Driver Options applied per cluster.
+                  properties:
+                    cephfs:
+                      description: CephFS defines CSI Driver settings for CephFS driver.
+                      properties:
+                        fuseMountOptions:
+                          description: FuseMountOptions defines the mount options for ceph fuse mounter.
+                          type: string
+                        kernelMountOptions:
+                          description: KernelMountOptions defines the mount options for kernel mounter.
+                          type: string
+                      type: object
+                    readAffinity:
+                      description: ReadAffinity defines the read affinity settings for CSI driver.
+                      properties:
+                        crushLocationLabels:
+                          description: CrushLocationLabels defines which node labels to use as CRUSH location. This should correspond to the values set in the CRUSH map.
+                          items:
+                            type: string
+                          type: array
+                        enabled:
+                          description: Enables read affinity for CSI driver.
+                          type: boolean
+                      type: object
+                  type: object
                 dashboard:
                   description: Dashboard settings
                   nullable: true
diff --git a/deploy/examples/cluster.yaml b/deploy/examples/cluster.yaml
index 2a5f565f3..af2078737 100644
--- a/deploy/examples/cluster.yaml
+++ b/deploy/examples/cluster.yaml
@@ -281,6 +281,22 @@ spec:
     # No values or 0 means that the operator will wait until the placement groups are healthy before unblocking the next drain.
     pgHealthCheckTimeout: 0
 
+  # csi defines CSI Driver settings applied per cluster.
+  csi:
+    readAffinity:
+      # Enable read affinity to enable clients to optimize reads from an OSD in the same topology.
+      # Enabling the read affinity may cause the OSDs to consume some extra memory.
+      # For more details see this doc:
+      # https://rook.io/docs/rook/latest/Storage-Configuration/Ceph-CSI/ceph-csi-drivers/#enable-read-affinity-for-rbd-volumes
+      enabled: false
+
+    # cephfs driver specific settings.
+    cephfs:
+      # Set CephFS Kernel mount options to use https://docs.ceph.com/en/latest/man/8/mount.ceph/#options.
+      # kernelMountOptions: ""
+      # Set CephFS Fuse mount options to use https://docs.ceph.com/en/quincy/man/8/ceph-fuse/#options.
+      # fuseMountOptions: ""
+
   # healthChecks
   # Valid values for daemons are 'mon', 'osd', 'status'
   healthCheck:
diff --git a/deploy/examples/crds.yaml b/deploy/examples/crds.yaml
index 0ff268656..da0e5adaa 100644
--- a/deploy/examples/crds.yaml
+++ b/deploy/examples/crds.yaml
@@ -960,6 +960,32 @@ spec:
                       description: Disable determines whether we should enable the crash collector
                       type: boolean
                   type: object
+                csi:
+                  description: CSI Driver Options applied per cluster.
+                  properties:
+                    cephfs:
+                      description: CephFS defines CSI Driver settings for CephFS driver.
+                      properties:
+                        fuseMountOptions:
+                          description: FuseMountOptions defines the mount options for ceph fuse mounter.
+                          type: string
+                        kernelMountOptions:
+                          description: KernelMountOptions defines the mount options for kernel mounter.
+                          type: string
+                      type: object
+                    readAffinity:
+                      description: ReadAffinity defines the read affinity settings for CSI driver.
+                      properties:
+                        crushLocationLabels:
+                          description: CrushLocationLabels defines which node labels to use as CRUSH location. This should correspond to the values set in the CRUSH map.
+                          items:
+                            type: string
+                          type: array
+                        enabled:
+                          description: Enables read affinity for CSI driver.
+                          type: boolean
+                      type: object
+                  type: object
                 dashboard:
                   description: Dashboard settings
                   nullable: true
diff --git a/pkg/apis/ceph.rook.io/v1/types.go b/pkg/apis/ceph.rook.io/v1/types.go
index d2ae6ed98..89bc399a9 100755
--- a/pkg/apis/ceph.rook.io/v1/types.go
+++ b/pkg/apis/ceph.rook.io/v1/types.go
@@ -221,6 +221,42 @@ type ClusterSpec struct {
 	// +optional
 	// +nullable
 	LogCollector LogCollectorSpec `json:"logCollector,omitempty"`
+
+	// CSI Driver Options applied per cluster.
+	// +optional
+	CSI CSIDriverSpec `json:"csi,omitempty"`
+}
+
+// CSIDriverSpec defines CSI Driver settings applied per cluster.
+type CSIDriverSpec struct {
+	// ReadAffinity defines the read affinity settings for CSI driver.
+	// +optional
+	ReadAffinity ReadAffinitySpec `json:"readAffinity"`
+	// CephFS defines CSI Driver settings for CephFS driver.
+	// +optional
+	CephFS CSICephFSSpec `json:"cephfs,omitempty"`
+}
+
+// CSICephFSSpec defines the settings for CephFS CSI driver.
+type CSICephFSSpec struct {
+	// KernelMountOptions defines the mount options for kernel mounter.
+	// +optional
+	KernelMountOptions string `json:"kernelMountOptions,omitempty"`
+	// FuseMountOptions defines the mount options for ceph fuse mounter.
+	// +optional
+	FuseMountOptions string `json:"fuseMountOptions,omitempty"`
+}
+
+// ReadAffinitySpec defines the read affinity settings for CSI driver.
+type ReadAffinitySpec struct {
+	// Enables read affinity for CSI driver.
+	// +optional
+	Enabled bool `json:"enabled"`
+	// CrushLocationLabels defines which node labels to use
+	// as CRUSH location. This should correspond to the values set in
+	// the CRUSH map.
+	// +optional
+	CrushLocationLabels []string `json:"crushLocationLabels,omitempty"`
 }
 
 // LogCollectorSpec is the logging spec
diff --git a/pkg/daemon/ceph/client/info.go b/pkg/daemon/ceph/client/info.go
index b6930cf31..70e49bc48 100644
--- a/pkg/daemon/ceph/client/info.go
+++ b/pkg/daemon/ceph/client/info.go
@@ -48,6 +48,7 @@ type ClusterInfo struct {
 	name              string
 	OsdUpgradeTimeout time.Duration
 	NetworkSpec       cephv1.NetworkSpec
+	CSIDriverSpec     cephv1.CSIDriverSpec
 	// A context to cancel the context it is used to determine whether the reconcile loop should
 	// exist (if the context has been cancelled). This cannot be in main clusterd context since this
 	// is a pointer passed through the entire life cycle or the operator. If the context is
diff --git a/pkg/operator/ceph/cluster/cluster.go b/pkg/operator/ceph/cluster/cluster.go
index f0a453714..b3ea61502 100755
--- a/pkg/operator/ceph/cluster/cluster.go
+++ b/pkg/operator/ceph/cluster/cluster.go
@@ -220,6 +220,11 @@ func (c *ClusterController) initializeCluster(cluster *cluster) error {
 		go cluster.reportTelemetry()
 	}
 
+	err := csi.SaveCSIDriverOptions(c.context.Clientset, cluster.Namespace, cluster.ClusterInfo)
+	if err != nil {
+		return errors.Wrap(err, "failed to save CSI driver options")
+	}
+
 	// Populate ClusterInfo with the last value
 	cluster.mons.ClusterInfo = cluster.ClusterInfo
 	cluster.mons.ClusterInfo.SetName(c.namespacedName.Name)
diff --git a/pkg/operator/ceph/controller/cluster_info.go b/pkg/operator/ceph/controller/cluster_info.go
index 67882d7e6..eb85146ad 100644
--- a/pkg/operator/ceph/controller/cluster_info.go
+++ b/pkg/operator/ceph/controller/cluster_info.go
@@ -32,6 +32,7 @@ import (
 	cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
 	"github.com/rook/rook/pkg/clusterd"
 	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
+	"github.com/rook/rook/pkg/operator/ceph/cluster/osd/topology"
 	"github.com/rook/rook/pkg/operator/k8sutil"
 	"github.com/rook/rook/pkg/util/exec"
 	"github.com/rook/rook/pkg/util/sys"
@@ -154,6 +155,12 @@ func CreateOrLoadClusterInfo(clusterdContext *clusterd.Context, context context.
 	// can run on remote executor
 	clusterInfo.NetworkSpec = cephClusterSpec.Network
 
+	// update clusterInfo with cephClusterSpec.CSI.
+	clusterInfo.CSIDriverSpec = cephClusterSpec.CSI
+	if len(clusterInfo.CSIDriverSpec.ReadAffinity.CrushLocationLabels) == 0 {
+		clusterInfo.CSIDriverSpec.ReadAffinity.CrushLocationLabels = strings.Split(topology.GetDefaultTopologyLabels(), ",")
+	}
+
 	// If an admin key was provided we don't need to load the other resources
 	// Some people might want to give the admin key
 	// The necessary users/keys/secrets will be created by Rook
diff --git a/pkg/operator/ceph/csi/cluster_config.go b/pkg/operator/ceph/csi/cluster_config.go
index 7461ae5be..97928b9bf 100644
--- a/pkg/operator/ceph/csi/cluster_config.go
+++ b/pkg/operator/ceph/csi/cluster_config.go
@@ -26,6 +26,7 @@ import (
 
 	"github.com/coreos/pkg/capnslog"
 	"github.com/pkg/errors"
+	cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
 	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
 	"github.com/rook/rook/pkg/operator/k8sutil"
 	v1 "k8s.io/api/core/v1"
@@ -40,18 +41,21 @@ var (
 )
 
 type CsiClusterConfigEntry struct {
-	ClusterID      string         `json:"clusterID"`
-	Monitors       []string       `json:"monitors"`
-	Namespace      string         `json:"namespace"`
-	CephFS         *CsiCephFSSpec `json:"cephFS,omitempty"`
-	NFS            *CsiNFSSpec    `json:"nfs,omitempty"`
-	RBD            *CsiRBDSpec    `json:"rbd,omitempty"`
-	RadosNamespace string         `json:"radosNamespace,omitempty"`
+	ClusterID      string                   `json:"clusterID"`
+	Monitors       []string                 `json:"monitors"`
+	Namespace      string                   `json:"namespace"`
+	CephFS         *CsiCephFSSpec           `json:"cephFS,omitempty"`
+	NFS            *CsiNFSSpec              `json:"nfs,omitempty"`
+	RBD            *CsiRBDSpec              `json:"rbd,omitempty"`
+	RadosNamespace string                   `json:"radosNamespace,omitempty"`
+	ReadAffinity   *cephv1.ReadAffinitySpec `json:"readAffinity,omitempty"`
 }
 
 type CsiCephFSSpec struct {
 	NetNamespaceFilePath string `json:"netNamespaceFilePath,omitempty"`
 	SubvolumeGroup       string `json:"subvolumeGroup,omitempty"`
+	KernelMountOptions   string `json:"kernelMountOptions,omitempty"`
+	FuseMountOptions     string `json:"fuseMountOptions,omitempty"`
 }
 
 type CsiNFSSpec struct {
@@ -278,3 +282,75 @@ func SaveClusterConfig(clientset kubernetes.Interface, clusterNamespace string,
 
 	return nil
 }
+
+// updateCSIDriverOptions updates the CSI driver options, including read affinity, kernel mount options
+// and fuse mount options, for all entries belonging to the same cluster.
+func updateCSIDriverOptions(curr, clusterKey string,
+	csiDriverOptions *cephv1.CSIDriverSpec) (string, error) {
+	cc, err := parseCsiClusterConfig(curr)
+	if err != nil {
+		return "", errors.Wrap(err, "failed to parse current csi cluster config")
+	}
+
+	for i := range cc {
+		// If the clusterID belongs to the same cluster, update the entry.
+		if clusterKey == cc[i].Namespace {
+			cc[i].ReadAffinity = &csiDriverOptions.ReadAffinity
+			if cc[i].CephFS == nil {
+				cc[i].CephFS = &CsiCephFSSpec{
+					KernelMountOptions: csiDriverOptions.CephFS.KernelMountOptions,
+					FuseMountOptions:   csiDriverOptions.CephFS.FuseMountOptions,
+				}
+			} else {
+				cc[i].CephFS.KernelMountOptions = csiDriverOptions.CephFS.KernelMountOptions
+				cc[i].CephFS.FuseMountOptions = csiDriverOptions.CephFS.FuseMountOptions
+			}
+		}
+	}
+
+	return formatCsiClusterConfig(cc)
+}
+
+// SaveCSIDriverOptions, similar to SaveClusterConfig, updates the config map used by ceph-csi
+// with CSI driver options such as read affinity, kernel mount options and fuse mount options.
+func SaveCSIDriverOptions(clientset kubernetes.Interface, clusterNamespace string, clusterInfo *cephclient.ClusterInfo) error {
+	// csi is deployed into the same namespace as the operator
+	csiNamespace := os.Getenv(k8sutil.PodNamespaceEnvVar)
+	if csiNamespace == "" {
+		logger.Warningf("cannot save csi config due to missing env var %q", k8sutil.PodNamespaceEnvVar)
+		return nil
+	}
+	logger.Debugf("using %q for csi configmap namespace", csiNamespace)
+
+	configMutex.Lock()
+	defer configMutex.Unlock()
+
+	// fetch current ConfigMap contents
+	configMap, err := clientset.CoreV1().ConfigMaps(csiNamespace).Get(clusterInfo.Context, ConfigName, metav1.GetOptions{})
+	if err != nil {
+		return errors.Wrap(err, "failed to fetch current csi config map")
+	}
+
+	// update ConfigMap contents for current cluster
+	currData := configMap.Data[ConfigKey]
+	if currData == "" {
+		currData = "[]"
+	}
+
+	newData, err := updateCSIDriverOptions(currData, clusterNamespace, &clusterInfo.CSIDriverSpec)
+	if err != nil {
+		return errors.Wrap(err, "failed to update csi config map data")
+	}
+	if currData == newData {
+		// no change
+		return nil
+	}
+
+	// update ConfigMap with new contents
+	configMap.Data[ConfigKey] = newData
+	if _, err := clientset.CoreV1().ConfigMaps(csiNamespace).Update(clusterInfo.Context, configMap, metav1.UpdateOptions{}); err != nil {
+		return errors.Wrap(err, "failed to update csi config map with csi driver options")
+	}
+
+	return nil
+}
diff --git a/pkg/operator/ceph/csi/spec.go b/pkg/operator/ceph/csi/spec.go
index e1d12cd0c..65ebaba2e 100644
--- a/pkg/operator/ceph/csi/spec.go
+++ b/pkg/operator/ceph/csi/spec.go
@@ -868,8 +868,12 @@ func (r *ReconcileCSI) configureHolder(driver driverDetails, c ClusterDetail, tp
 	clusterConfigEntry := &CsiClusterConfigEntry{
 		Monitors: MonEndpoints(c.clusterInfo.Monitors, c.cluster.Spec.RequireMsgr2()),
 		RBD:      &CsiRBDSpec{},
-		CephFS:   &CsiCephFSSpec{},
-		NFS:      &CsiNFSSpec{},
+		CephFS: &CsiCephFSSpec{
+			FuseMountOptions:   c.clusterInfo.CSIDriverSpec.CephFS.FuseMountOptions,
+			KernelMountOptions: c.clusterInfo.CSIDriverSpec.CephFS.KernelMountOptions,
+		},
+		NFS:          &CsiNFSSpec{},
+		ReadAffinity: &c.clusterInfo.CSIDriverSpec.ReadAffinity,
 	}
 	netNamespaceFilePath := generateNetNamespaceFilePath(CSIParam.KubeletDirPath, driver.fullName, c.cluster.Namespace)
 	if driver.name == RBDDriverShortName {
diff --git a/pkg/operator/ceph/file/subvolumegroup/controller.go b/pkg/operator/ceph/file/subvolumegroup/controller.go
index 2cfd10f3e..867d8e39e 100644
--- a/pkg/operator/ceph/file/subvolumegroup/controller.go
+++ b/pkg/operator/ceph/file/subvolumegroup/controller.go
@@ -269,8 +269,11 @@ func (r *ReconcileCephFilesystemSubVolumeGroup) updateClusterConfig(cephFilesyst
 		Namespace: r.clusterInfo.Namespace,
 		Monitors:  csi.MonEndpoints(r.clusterInfo.Monitors, cephCluster.Spec.RequireMsgr2()),
 		CephFS: &csi.CsiCephFSSpec{
-			SubvolumeGroup: cephFilesystemSubVolumeGroup.Name,
+			SubvolumeGroup:     cephFilesystemSubVolumeGroup.Name,
+			KernelMountOptions: r.clusterInfo.CSIDriverSpec.CephFS.KernelMountOptions,
+			FuseMountOptions:   r.clusterInfo.CSIDriverSpec.CephFS.FuseMountOptions,
 		},
+		ReadAffinity: &r.clusterInfo.CSIDriverSpec.ReadAffinity,
 	}
 
 	// If the cluster has Multus enabled we need to append the network namespace of the driver's
diff --git a/pkg/operator/ceph/pool/radosnamespace/controller.go b/pkg/operator/ceph/pool/radosnamespace/controller.go
index 9990e5241..04f30d4f0 100644
--- a/pkg/operator/ceph/pool/radosnamespace/controller.go
+++ b/pkg/operator/ceph/pool/radosnamespace/controller.go
@@ -264,6 +264,11 @@ func (r *ReconcileCephBlockPoolRadosNamespace) updateClusterConfig(cephBlockPool
 			RadosNamespace: cephBlockPoolRadosNamespace.Name,
 		},
 		RadosNamespace: cephBlockPoolRadosNamespace.Name,
+		CephFS: &csi.CsiCephFSSpec{
+			KernelMountOptions: r.clusterInfo.CSIDriverSpec.CephFS.KernelMountOptions,
+			FuseMountOptions:   r.clusterInfo.CSIDriverSpec.CephFS.FuseMountOptions,
+		},
+		ReadAffinity: &r.clusterInfo.CSIDriverSpec.ReadAffinity,
 	}
 
 	if cephCluster.Spec.Network.IsMultus() {


Test patch:
diff --git a/pkg/operator/ceph/csi/cluster_config_test.go b/pkg/operator/ceph/csi/cluster_config_test.go
index 5f778a16d..bc406f8df 100644
--- a/pkg/operator/ceph/csi/cluster_config_test.go
+++ b/pkg/operator/ceph/csi/cluster_config_test.go
@@ -20,6 +20,7 @@ import (
 	"strings"
 	"testing"
 
+	cephv1 "github.com/rook/rook/pkg/apis/ceph.rook.io/v1"
 	cephclient "github.com/rook/rook/pkg/daemon/ceph/client"
 	"github.com/stretchr/testify/assert"
 )
@@ -391,3 +392,148 @@ func verifyEndpointPort(t *testing.T, endpoints []string, expectedPort string) {
 		assert.True(t, strings.HasSuffix(endpoint, expectedPort))
 	}
 }
+
+func TestUpdateCSIDriverOptions(t *testing.T) {
+	type args struct {
+		clusterConfig    csiClusterConfig
+		clusterKey       string
+		csiDriverOptions *cephv1.CSIDriverSpec
+	}
+	tests := []struct {
+		name    string
+		args    args
+		want    csiClusterConfig
+		wantErr bool
+	}{
+		{
+			name: "empty current config",
+			args: args{
+				clusterConfig:    []CsiClusterConfigEntry{},
+				clusterKey:       "rook-ceph",
+				csiDriverOptions: &cephv1.CSIDriverSpec{},
+			},
+			want:    []CsiClusterConfigEntry{},
+			wantErr: false,
+		},
+		{
+			name: "single matching current config",
+			args: args{
+				clusterConfig: []CsiClusterConfigEntry{
+					{
+						ClusterID: "rook-ceph",
+						Namespace: "rook-ceph",
+						Monitors:  []string{"1.1.1.1"},
+					},
+				},
+				clusterKey: "rook-ceph",
+				csiDriverOptions: &cephv1.CSIDriverSpec{
+					ReadAffinity: cephv1.ReadAffinitySpec{
+						Enabled:             true,
+						CrushLocationLabels: []string{"topology.rook.io/rack"},
+					},
+					CephFS: cephv1.CSICephFSSpec{
+						KernelMountOptions: "rw,noatime",
+						FuseMountOptions:   "debug",
+					},
+				},
+			},
+			want: []CsiClusterConfigEntry{
+				{
+					ClusterID: "rook-ceph",
+					Namespace: "rook-ceph",
+					Monitors:  []string{"1.1.1.1"},
+					ReadAffinity: &cephv1.ReadAffinitySpec{
+						Enabled:             true,
+						CrushLocationLabels: []string{"topology.rook.io/rack"},
+					},
+					CephFS: &CsiCephFSSpec{
+						KernelMountOptions: "rw,noatime",
+						FuseMountOptions:   "debug",
+					},
+				},
+			},
+			wantErr: false,
+		},
+		{
+			name: "multiple matching current config",
+			args: args{
+				clusterConfig: []CsiClusterConfigEntry{
+					{
+						ClusterID: "rook-ceph",
+						Namespace: "rook-ceph",
+						Monitors:  []string{"1.1.1.1"},
+					},
+					{
+						ClusterID: "rook-ceph-2",
+						Namespace: "rook-ceph",
+						Monitors:  []string{"1.1.1.1"},
+					},
+					{
+						ClusterID: "rook-ceph-3",
+						Namespace: "rook-ceph-1",
+						Monitors:  []string{"1.1.1.1"},
+					},
+				},
+				clusterKey: "rook-ceph",
+				csiDriverOptions: &cephv1.CSIDriverSpec{
+					ReadAffinity: cephv1.ReadAffinitySpec{
+						Enabled:             true,
+						CrushLocationLabels: []string{"topology.rook.io/rack"},
+					},
+					CephFS: cephv1.CSICephFSSpec{
+						KernelMountOptions: "rw,noatime",
+						FuseMountOptions:   "debug",
+					},
+				},
+			},
+			want: []CsiClusterConfigEntry{
+				{
+					ClusterID: "rook-ceph",
+					Namespace: "rook-ceph",
+					Monitors:  []string{"1.1.1.1"},
+					ReadAffinity: &cephv1.ReadAffinitySpec{
+						Enabled:             true,
+						CrushLocationLabels: []string{"topology.rook.io/rack"},
+					},
+					CephFS: &CsiCephFSSpec{
+						KernelMountOptions: "rw,noatime",
+						FuseMountOptions:   "debug",
+					},
+				},
+				{
+					ClusterID: "rook-ceph-2",
+					Namespace: "rook-ceph",
+					Monitors:  []string{"1.1.1.1"},
+					ReadAffinity: &cephv1.ReadAffinitySpec{
+						Enabled:             true,
+						CrushLocationLabels: []string{"topology.rook.io/rack"},
+					},
+					CephFS: &CsiCephFSSpec{
+						KernelMountOptions: "rw,noatime",
+						FuseMountOptions:   "debug",
+					},
+				},
+				{
+					ClusterID: "rook-ceph-3",
+					Namespace: "rook-ceph-1",
+					Monitors:  []string{"1.1.1.1"},
+				},
+			},
+			wantErr: false,
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			dataString, err := formatCsiClusterConfig(tt.args.clusterConfig)
+			assert.NoError(t, err)
+			got, err := updateCSIDriverOptions(dataString, tt.args.clusterKey, tt.args.csiDriverOptions)
+			if (err != nil) != tt.wantErr {
+				t.Errorf("updateCSIDriverOptions() error = %v, wantErr %v", err, tt.wantErr)
+				return
+			}
+			expectedString, err := formatCsiClusterConfig(tt.want)
+			assert.NoError(t, err)
+			assert.Equal(t, expectedString, got)
+		})
+	}
+}