Repository: rook/rook
Pull request #15696: operator: specify ceph config via secret
Author: login: patrostkowski
Labels: name: backport-release-1.17
Created: 2025-04-12T11:13:48Z | Merged: 2025-04-23T17:48:25Z

Adds .spec.cephConfigFromSecret to CephCluster for loading Ceph config parameters from a Kubernetes Secret.

Adresses: https://github.com/rook/rook/issues/15678

<!-- Thank you for contributing to Rook! -->

<!-- STEPS TO FOLLOW:
  1. Add a description of the changes (frequently the same as the commit description)
  2. Enter the issue number next to "Resolves #" below (if there is no tracking issue resolved, **remove that section**)
  3. Review our Contributing documentation at https://rook.io/docs/rook/latest/Contributing/development-flow/
  4. Follow the steps in the checklist below, starting with the **Commit Message Formatting**.
-->

<!-- Uncomment this section with the issue number if an issue is being resolved
**Issue resolved by this Pull Request:**
Resolves #
--->

**Checklist:**

- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/latest/Contributing/development-flow/#commit-structure).
- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/latest/Contributing/development-flow/#submitting-a-pull-request)
- [ ] [Pending release notes](https://github.com/rook/rook/blob/master/PendingReleaseNotes.md) updated with breaking and/or notable changes for the next minor release.
- [ ] Documentation has been updated, if necessary.
- [ ] Unit tests have been added, if necessary.
- [ ] Integration tests have been added, if necessary.



Diff:
diff --git a/Documentation/CRDs/Cluster/ceph-cluster-crd.md b/Documentation/CRDs/Cluster/ceph-cluster-crd.md
index 130f589861d5..0d62ad9abfef 100755
--- a/Documentation/CRDs/Cluster/ceph-cluster-crd.md
+++ b/Documentation/CRDs/Cluster/ceph-cluster-crd.md
@@ -108,6 +108,7 @@ For more details on the mons and when to choose a number other than `3`, see the
 * `cleanupPolicy`: [cleanup policy settings](#cleanup-policy)
 * `security`: [security page for key management configuration](../../Storage-Configuration/Advanced/key-management-system.md)
 * `cephConfig`: [Set Ceph config options using the Ceph Mon config store](#ceph-config)
+* `cephConfigFromSecret`: [Set Ceph config options using the Ceph Mon config store via Kubernetes secret reference](#ceph-config-from-secret)
 * `csi`: [Set CSI Driver options](#csi-driver-options)
 
 ### Ceph container images
@@ -887,6 +888,29 @@ should be used instead.
 
 The operator does not unset any removed config options, it is the user's responsibility to unset or set the default value for each removed option manually using the Ceph CLI.
 
+## Ceph Config From Secret
+
+In addition to `cephConfig`, Ceph configuration values can be provided via Kubernetes Secrets using `cephConfigFromSecret`. This is useful for referencing sensitive values such as passwords or tokens that shouldn't be stored directly in the CR.
+
+The structure mirrors `cephConfig`, but instead of string values, each config entry references a Kubernetes `SecretKeySelector`, specifying:
+
+- name: the name of the secret (in the same namespace as the `CephCluster`)
+- key: the key inside the secret that holds the desired value
+
+```yaml
+spec:
+  cephConfigFromSecret:
+    mgr:
+      mgr/dashboard/GRAFANA_API_PASSWORD:
+        name: grafana-api-credential # name of the Kubernetes secret
+        key: password                # key of the secret value in Kubernetes secret.Data map[string]string
+```
+
+If a given config key is defined in both `cephConfigFromSecret` and `cephConfig`, the value from `cephConfig` takes precedence.
+
+!!! warning
+    If a value from `cephConfigFromSecret` cannot be retrieved — for example, if the referenced Secret or key is missing — Rook will return a reconciliation error. This ensures that configuration provided via `cephConfigFromSecret` is applied reliably, as it is treated as a declarative and intentional configuration by the admin.
+
 ## CSI Driver Options
 
 The CSI driver options mentioned here are applied per Ceph cluster. The following options are available:
diff --git a/Documentation/CRDs/specification.md b/Documentation/CRDs/specification.md
index dcd30478018f..a77799846d67 100644
--- a/Documentation/CRDs/specification.md
+++ b/Documentation/CRDs/specification.md
@@ -1048,6 +1048,18 @@ map[string]map[string]string
 <p>Ceph Config options</p>
 </td>
 </tr>
+<tr>
+<td>
+<code>cephConfigFromSecret</code><br/>
+<em>
+map[string]map[string]k8s.io/api/core/v1.SecretKeySelector
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>CephConfigFromSecret works exactly like CephConfig but takes config value from Secret Key reference.</p>
+</td>
+</tr>
 </table>
 </td>
 </tr>
@@ -4905,6 +4917,18 @@ map[string]map[string]string
 <p>Ceph Config options</p>
 </td>
 </tr>
+<tr>
+<td>
+<code>cephConfigFromSecret</code><br/>
+<em>
+map[string]map[string]k8s.io/api/core/v1.SecretKeySelector
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>CephConfigFromSecret works exactly like CephConfig but takes config value from Secret Key reference.</p>
+</td>
+</tr>
 </tbody>
 </table>
 <h3 id="ceph.rook.io/v1.ClusterState">ClusterState
diff --git a/Documentation/Storage-Configuration/Advanced/ceph-configuration.md b/Documentation/Storage-Configuration/Advanced/ceph-configuration.md
index f9c9ee9e8a08..ca07266fcec8 100644
--- a/Documentation/Storage-Configuration/Advanced/ceph-configuration.md
+++ b/Documentation/Storage-Configuration/Advanced/ceph-configuration.md
@@ -209,7 +209,7 @@ ceph osd pool set rbd pg_num 512
 ## Custom `ceph.conf` Settings
 
 !!! info
-    The advised method for controlling Ceph configuration is to use the [`cephConfig:` structure](../../CRDs/Cluster/ceph-cluster-crd.md#ceph-config)
+    The advised method for controlling Ceph configuration is to use the [`cephConfig`](../../CRDs/Cluster/ceph-cluster-crd.md#ceph-config) and [`cephConfigFromSecret`](../../CRDs/Cluster/ceph-cluster-crd.md#ceph-config-from-secret)
     in the `CephCluster` CRD.
     <br><br>It is highly recommended that this only be used when absolutely necessary and that the `config` be
     reset to an empty string if/when the configurations are no longer necessary. Configurations in the
diff --git a/deploy/charts/rook-ceph/templates/resources.yaml b/deploy/charts/rook-ceph/templates/resources.yaml
index 3048672a6f22..6be152d56e4d 100644
--- a/deploy/charts/rook-ceph/templates/resources.yaml
+++ b/deploy/charts/rook-ceph/templates/resources.yaml
@@ -1265,6 +1265,34 @@ spec:
                   description: Ceph Config options
                   nullable: true
                   type: object
+                cephConfigFromSecret:
+                  additionalProperties:
+                    additionalProperties:
+                      description: SecretKeySelector selects a key of a Secret.
+                      properties:
+                        key:
+                          description: The key of the secret to select from.  Must be a valid secret key.
+                          type: string
+                        name:
+                          default: ""
+                          description: |-
+                            Name of the referent.
+                            This field is effectively required, but due to backwards compatibility is
+                            allowed to be empty. Instances of this type with an empty value here are
+                            almost certainly wrong.
+                            More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+                          type: string
+                        optional:
+                          description: Specify whether the Secret or its key must be defined
+                          type: boolean
+                      required:
+                        - key
+                      type: object
+                      x-kubernetes-map-type: atomic
+                    type: object
+                  description: CephConfigFromSecret works exactly like CephConfig but takes config value from Secret Key reference.
+                  nullable: true
+                  type: object
                 cephVersion:
                   description: The version information that instructs Rook to orchestrate a particular version of Ceph.
                   nullable: true
diff --git a/deploy/examples/crds.yaml b/deploy/examples/crds.yaml
index 7b317bb66c96..fdae3f1c1a57 100644
--- a/deploy/examples/crds.yaml
+++ b/deploy/examples/crds.yaml
@@ -1263,6 +1263,34 @@ spec:
                   description: Ceph Config options
                   nullable: true
                   type: object
+                cephConfigFromSecret:
+                  additionalProperties:
+                    additionalProperties:
+                      description: SecretKeySelector selects a key of a Secret.
+                      properties:
+                        key:
+                          description: The key of the secret to select from.  Must be a valid secret key.
+                          type: string
+                        name:
+                          default: ""
+                          description: |-
+                            Name of the referent.
+                            This field is effectively required, but due to backwards compatibility is
+                            allowed to be empty. Instances of this type with an empty value here are
+                            almost certainly wrong.
+                            More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
+                          type: string
+                        optional:
+                          description: Specify whether the Secret or its key must be defined
+                          type: boolean
+                      required:
+                        - key
+                      type: object
+                      x-kubernetes-map-type: atomic
+                    type: object
+                  description: CephConfigFromSecret works exactly like CephConfig but takes config value from Secret Key reference.
+                  nullable: true
+                  type: object
                 cephVersion:
                   description: The version information that instructs Rook to orchestrate a particular version of Ceph.
                   nullable: true
diff --git a/pkg/apis/ceph.rook.io/v1/types.go b/pkg/apis/ceph.rook.io/v1/types.go
index 0c80afaaa0fb..0d9cc90e9372 100755
--- a/pkg/apis/ceph.rook.io/v1/types.go
+++ b/pkg/apis/ceph.rook.io/v1/types.go
@@ -239,6 +239,11 @@ type ClusterSpec struct {
 	// +optional
 	// +nullable
 	CephConfig map[string]map[string]string `json:"cephConfig,omitempty"`
+
+	// CephConfigFromSecret works exactly like CephConfig but takes config value from Secret Key reference.
+	// +optional
+	// +nullable
+	CephConfigFromSecret map[string]map[string]v1.SecretKeySelector `json:"cephConfigFromSecret,omitempty"`
 }
 
 // CSIDriverSpec defines CSI Driver settings applied per cluster.
diff --git a/pkg/apis/ceph.rook.io/v1/zz_generated.deepcopy.go b/pkg/apis/ceph.rook.io/v1/zz_generated.deepcopy.go
index 36c747f75a03..5ad7a281d7f0 100644
--- a/pkg/apis/ceph.rook.io/v1/zz_generated.deepcopy.go
+++ b/pkg/apis/ceph.rook.io/v1/zz_generated.deepcopy.go
@@ -2046,6 +2046,23 @@ func (in *ClusterSpec) DeepCopyInto(out *ClusterSpec) {
 			(*out)[key] = outVal
 		}
 	}
+	if in.CephConfigFromSecret != nil {
+		in, out := &in.CephConfigFromSecret, &out.CephConfigFromSecret
+		*out = make(map[string]map[string]corev1.SecretKeySelector, len(*in))
+		for key, val := range *in {
+			var outVal map[string]corev1.SecretKeySelector
+			if val == nil {
+				(*out)[key] = nil
+			} else {
+				in, out := &val, &outVal
+				*out = make(map[string]corev1.SecretKeySelector, len(*in))
+				for key, val := range *in {
+					(*out)[key] = *val.DeepCopy()
+				}
+			}
+			(*out)[key] = outVal
+		}
+	}
 	return
 }
 
diff --git a/pkg/operator/ceph/cluster/cluster.go b/pkg/operator/ceph/cluster/cluster.go
index 400d7df3327d..7069864c7840 100755
--- a/pkg/operator/ceph/cluster/cluster.go
+++ b/pkg/operator/ceph/cluster/cluster.go
@@ -567,7 +567,17 @@ func (c *cluster) shouldSetClusterFullSettings() bool {
 
 func (c *cluster) updateConfigStoreFromCRD() error {
 	monStore := config.GetMonStore(c.context, c.ClusterInfo)
-	return monStore.SetAllMultiple(c.Spec.CephConfig)
+	cephConfigFromSecret, err := c.fetchCephConfigFromSecrets()
+	if err != nil {
+		return err
+	}
+	if err := monStore.SetAllMultiple(cephConfigFromSecret); err != nil {
+		return err
+	}
+	if err := monStore.SetAllMultiple(c.Spec.CephConfig); err != nil {
+		return err
+	}
+	return nil
 }
 
 func (c *cluster) reportTelemetry() {
@@ -741,3 +751,43 @@ func (c *cluster) configureMsgr2() error {
 
 	return nil
 }
+
+func (c *cluster) fetchCephConfigFromSecrets() (map[string]map[string]string, error) {
+	result := make(map[string]map[string]string)
+
+	for module, keys := range c.Spec.CephConfigFromSecret {
+		result[module] = make(map[string]string)
+
+		for key, selector := range keys {
+			val, err := c.fetchSecretValue(selector)
+			if err != nil {
+				return nil, fmt.Errorf("failed to get value for key %q in module %q from secret %q: %w",
+					key, module, selector.LocalObjectReference.Name, err)
+			}
+
+			logger.Debugf("setting Ceph config key %q in module %q from secret %q",
+				key, module, selector.LocalObjectReference.Name)
+			logger.Tracef("setting Ceph config key %q in module %q to value %q from secret %q",
+				key, module, val, selector.LocalObjectReference.Name)
+			result[module][key] = val
+		}
+	}
+
+	return result, nil
+}
+
+func (c *cluster) fetchSecretValue(selector v1.SecretKeySelector) (string, error) {
+	secret, err := c.context.Clientset.CoreV1().Secrets(c.ClusterInfo.Namespace).Get(
+		c.ClusterInfo.Context, selector.LocalObjectReference.Name, metav1.GetOptions{},
+	)
+	if err != nil {
+		return "", fmt.Errorf("failed to get secret %q: %w", selector.LocalObjectReference.Name, err)
+	}
+
+	val, ok := secret.Data[selector.Key]
+	if !ok {
+		return "", fmt.Errorf("secret %q is missing key %q", selector.LocalObjectReference.Name, selector.Key)
+	}
+
+	return string(val), nil
+}
diff --git a/pkg/operator/ceph/cluster/cluster_test.go b/pkg/operator/ceph/cluster/cluster_test.go
index 6758aa21f24b..71d4f2108043 100644
--- a/pkg/operator/ceph/cluster/cluster_test.go
+++ b/pkg/operator/ceph/cluster/cluster_test.go
@@ -18,6 +18,7 @@ limitations under the License.
 package cluster
 
 import (
+	"context"
 	"flag"
 	"fmt"
 	"strings"
@@ -33,11 +34,14 @@ import (
 	"github.com/rook/rook/pkg/operator/ceph/cluster/telemetry"
 	"github.com/rook/rook/pkg/operator/ceph/csi"
 	cephver "github.com/rook/rook/pkg/operator/ceph/version"
+	"github.com/rook/rook/pkg/operator/test"
 	testop "github.com/rook/rook/pkg/operator/test"
 	exectest "github.com/rook/rook/pkg/util/exec/test"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 	"gopkg.in/ini.v1"
+	v1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
 func TestPreClusterStartValidation(t *testing.T) {
@@ -431,3 +435,163 @@ func TestClusterFullSettings(t *testing.T) {
 		assert.False(t, setBackfillFullRatio)
 	})
 }
+
+func TestFetchCephConfigFromSecrets(t *testing.T) {
+	tests := []struct {
+		name                 string
+		cephConfigFromSecret map[string]map[string]v1.SecretKeySelector
+		secrets              []*v1.Secret
+		expectedOutput       map[string]map[string]string
+		expectErr            bool
+	}{
+		{
+			name: "module with valid secret and key (foo)",
+			cephConfigFromSecret: map[string]map[string]v1.SecretKeySelector{
+				"foo": {
+					"foo/bar": {
+						LocalObjectReference: v1.LocalObjectReference{Name: "secret-foo"},
+						Key:                  "val",
+					},
+				},
+			},
+			secrets: []*v1.Secret{
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "secret-foo", Namespace: "rook-ceph"},
+					Data:       map[string][]byte{"val": []byte("alpha")},
+				},
+			},
+			expectedOutput: map[string]map[string]string{
+				"foo": {"foo/bar": "alpha"},
+			},
+			expectErr: false,
+		},
+		{
+			name: "module with missing key in existing secret (bar)",
+			cephConfigFromSecret: map[string]map[string]v1.SecretKeySelector{
+				"bar": {
+					"bar/baz": {
+						LocalObjectReference: v1.LocalObjectReference{Name: "secret-bar"},
+						Key:                  "val",
+					},
+				},
+			},
+			secrets: []*v1.Secret{
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "secret-bar", Namespace: "rook-ceph"},
+					Data:       map[string][]byte{"not-val": []byte("noop")},
+				},
+			},
+			expectErr: true,
+		},
+		{
+			name: "module with missing secret (bar)",
+			cephConfigFromSecret: map[string]map[string]v1.SecretKeySelector{
+				"bar": {
+					"bar/baz": {
+						LocalObjectReference: v1.LocalObjectReference{Name: "missing-secret"},
+						Key:                  "val",
+					},
+				},
+			},
+			secrets:   []*v1.Secret{},
+			expectErr: true,
+		},
+		{
+			name: "multiple modules with valid keys",
+			cephConfigFromSecret: map[string]map[string]v1.SecretKeySelector{
+				"foo": {
+					"foo/key1": {
+						LocalObjectReference: v1.LocalObjectReference{Name: "secret-1"},
+						Key:                  "k1",
+					},
+				},
+				"bar": {
+					"bar/key2": {
+						LocalObjectReference: v1.LocalObjectReference{Name: "secret-2"},
+						Key:                  "k2",
+					},
+				},
+			},
+			secrets: []*v1.Secret{
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "secret-1", Namespace: "rook-ceph"},
+					Data:       map[string][]byte{"k1": []byte("v1")},
+				},
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "secret-2", Namespace: "rook-ceph"},
+					Data:       map[string][]byte{"k2": []byte("v2")},
+				},
+			},
+			expectedOutput: map[string]map[string]string{
+				"foo": {"foo/key1": "v1"},
+				"bar": {"bar/key2": "v2"},
+			},
+			expectErr: false,
+		},
+		{
+			name: "multiple modules with one valid key and one missing key",
+			cephConfigFromSecret: map[string]map[string]v1.SecretKeySelector{
+				"foo": {
+					"foo/key": {
+						LocalObjectReference: v1.LocalObjectReference{Name: "secret-foo"},
+						Key:                  "valid-key",
+					},
+				},
+				"bar": {
+					"bar/key": {
+						LocalObjectReference: v1.LocalObjectReference{Name: "secret-bar"},
+						Key:                  "missing-key",
+					},
+				},
+			},
+			secrets: []*v1.Secret{
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "secret-foo", Namespace: "rook-ceph"},
+					Data:       map[string][]byte{"valid-key": []byte("ok")},
+				},
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "secret-bar", Namespace: "rook-ceph"},
+					Data:       map[string][]byte{"not-the-key": []byte("noop")},
+				},
+			},
+			expectErr: true,
+		},
+		{
+			name:                 "cephConfigFromSecret is nil",
+			cephConfigFromSecret: nil,
+			secrets:              []*v1.Secret{},
+			expectedOutput:       map[string]map[string]string{},
+			expectErr:            false,
+		},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			clientset := test.New(t, 1)
+			clusterInfo := cephclient.AdminTestClusterInfo("rook-ceph")
+
+			for _, s := range tc.secrets {
+				_, err := clientset.CoreV1().Secrets(s.Namespace).Create(context.TODO(), s, metav1.CreateOptions{})
+				assert.NoError(t, err)
+			}
+
+			c := &cluster{
+				context: &clusterd.Context{
+					Clientset: clientset,
+				},
+				ClusterInfo: clusterInfo,
+				Spec: &cephv1.ClusterSpec{
+					CephConfigFromSecret: tc.cephConfigFromSecret,
+				},
+			}
+
+			result, err := c.fetchCephConfigFromSecrets()
+			if tc.expectErr {
+				assert.Error(t, err)
+			} else {
+				assert.NoError(t, err)
+				assert.Equal(t, tc.expectedOutput, result)
+			}
+		})
+	}
+}


Reviews:
user: login: travisn
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-16T16:54:55Z
user: login: BlaineEXE
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T17:44:09Z
user: login: BlaineEXE
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-16T17:46:39Z
user: login: patrostkowski
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T20:26:47Z
user: login: patrostkowski
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T20:28:54Z
user: login: BlaineEXE
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-18T15:51:58Z
user: login: travisn
body: The changes look good to me overall, mainly just a request for testing if you wouldn't mind.
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-21T18:07:31Z
user: login: BlaineEXE
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-04-22T15:02:15Z
user: login: patrostkowski
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T05:49:02Z
user: login: patrostkowski
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T05:50:07Z
user: login: travisn
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-23T17:46:08Z
user: login: BlaineEXE
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-23T17:47:53Z

Review comments:
diff_hunk: @@ -567,7 +568,12 @@ func (c *cluster) shouldSetClusterFullSettings() bool {
 
 func (c *cluster) updateConfigStoreFromCRD() error {
 	monStore := config.GetMonStore(c.context, c.ClusterInfo)
-	return monStore.SetAllMultiple(c.Spec.CephConfig)
+	cephConfig, err := c.getMergedCephConfig()
+	if err != nil {
+		cephConfig = c.Spec.CephConfig
+		logger.Errorf("failed to parse CephConfig and CephConfigSecretRef %q, continue without CephConfigSecretRef: %s", c.Spec.CephConfigSecretRef.Name, err)
+	}
+	return monStore.SetAllMultiple(cephConfig)
path: pkg/operator/ceph/cluster/cluster.go
user: login: travisn
body: We'll need to review the logs to make sure the secrets are not leaked in this code path. For example, if debug logging is enabled (`ROOK_LOG_LEVEL: DEBUG`), they might show up. IIRC the values are not logged, only the keys, but let's confirm.
created_at: 2025-04-14T17:33:44Z
author_association: MEMBER
diff_hunk: @@ -741,3 +742,75 @@ func (c *cluster) configureMsgr2() error {
 
 	return nil
 }
+
+// getMergedCephConfig merges both maps from CephConfig and CephConfigFromSecret
+func (c *cluster) getMergedCephConfig() map[string]map[string]string {
+	secretConfig := c.fetchCephConfigFromSecrets()
+	return mergeConfigs(secretConfig, c.Spec.CephConfig)
+}
+
+// fetchCephConfigFromSecrets fetches and parses the cephConfig YAML from the referenced secret
+func (c *cluster) fetchCephConfigFromSecrets() map[string]map[string]string {
+	result := make(map[string]map[string]string)
+
+	for module, keys := range c.Spec.CephConfigFromSecret {
+		if result[module] == nil {
+			result[module] = map[string]string{}
+		}
+
+		for key, selector := range keys {
+			val, err := c.fetchSecretValue(selector)
+			if err != nil {
+				logger.Errorf("failed to get value for key %q in module %q from secret %q: %v",
+					key, module, selector.LocalObjectReference.Name, err)
+				continue
+			}
+			result[module][key] = val
+		}
+	}
+
+	return result
+}
+
+func (c *cluster) fetchSecretValue(selector v1.SecretKeySelector) (string, error) {
+	secret, err := c.context.Clientset.CoreV1().Secrets(c.ClusterInfo.Namespace).Get(
+		c.ClusterInfo.Context, selector.LocalObjectReference.Name, metav1.GetOptions{},
+	)
+	if err != nil {
+		return "", fmt.Errorf("failed to get secret %q: %w", selector.LocalObjectReference.Name, err)
+	}
+
+	val, ok := secret.Data[selector.Key]
+	if !ok {
+		return "", fmt.Errorf("secret %q is missing key %q", selector.LocalObjectReference.Name, selector.Key)
+	}
+
+	return string(val), nil
+}
+
+// mergeConfigs merges base with override where
+// values from override take precedence over values from base.
+func mergeConfigs(base, override map[string]map[string]string) map[string]map[string]string {
path: pkg/operator/ceph/cluster/cluster.go
user: login: travisn
body: Why do we need to merge the configs from secrets? Why can't we just apply the two sources of configs separately? It seems like complexity that doesn't really change the behavior. Whether the settings are merged or not, if you have conflicting settings, the last one set will win. 
created_at: 2025-04-16T16:53:10Z
author_association: MEMBER
diff_hunk: @@ -567,7 +568,12 @@ func (c *cluster) shouldSetClusterFullSettings() bool {
 
 func (c *cluster) updateConfigStoreFromCRD() error {
 	monStore := config.GetMonStore(c.context, c.ClusterInfo)
-	return monStore.SetAllMultiple(c.Spec.CephConfig)
+	cephConfig, err := c.getMergedCephConfig()
+	if err != nil {
+		cephConfig = c.Spec.CephConfig
+		logger.Errorf("failed to parse CephConfig and CephConfigSecretRef %q, continue without CephConfigSecretRef: %s", c.Spec.CephConfigSecretRef.Name, err)
+	}
+	return monStore.SetAllMultiple(cephConfig)
path: pkg/operator/ceph/cluster/cluster.go
user: login: BlaineEXE
body: Artem's PR to do this for RGW used 2 logs. One Trace level (insecure) log where configs are logged, and normal debug logging where only the secret name was logged. I think using the same pattern here would make it easiest to maintain both features in the future: https://github.com/rook/rook/pull/15426/files#diff-59805cd31b1b8454d5983925c49b66741543cea7be9801f865518a9ea2e2ea98R219-R221
created_at: 2025-04-16T17:44:09Z
author_association: MEMBER
diff_hunk: @@ -741,3 +742,75 @@ func (c *cluster) configureMsgr2() error {
 
 	return nil
 }
+
+// getMergedCephConfig merges both maps from CephConfig and CephConfigFromSecret
+func (c *cluster) getMergedCephConfig() map[string]map[string]string {
+	secretConfig := c.fetchCephConfigFromSecrets()
+	return mergeConfigs(secretConfig, c.Spec.CephConfig)
+}
+
+// fetchCephConfigFromSecrets fetches and parses the cephConfig YAML from the referenced secret
+func (c *cluster) fetchCephConfigFromSecrets() map[string]map[string]string {
+	result := make(map[string]map[string]string)
+
+	for module, keys := range c.Spec.CephConfigFromSecret {
+		if result[module] == nil {
+			result[module] = map[string]string{}
+		}
+
+		for key, selector := range keys {
+			val, err := c.fetchSecretValue(selector)
+			if err != nil {
+				logger.Errorf("failed to get value for key %q in module %q from secret %q: %v",
+					key, module, selector.LocalObjectReference.Name, err)
+				continue
+			}
+			result[module][key] = val
+		}
+	}
+
+	return result
+}
+
+func (c *cluster) fetchSecretValue(selector v1.SecretKeySelector) (string, error) {
+	secret, err := c.context.Clientset.CoreV1().Secrets(c.ClusterInfo.Namespace).Get(
+		c.ClusterInfo.Context, selector.LocalObjectReference.Name, metav1.GetOptions{},
+	)
+	if err != nil {
+		return "", fmt.Errorf("failed to get secret %q: %w", selector.LocalObjectReference.Name, err)
+	}
+
+	val, ok := secret.Data[selector.Key]
+	if !ok {
+		return "", fmt.Errorf("secret %q is missing key %q", selector.LocalObjectReference.Name, selector.Key)
+	}
+
+	return string(val), nil
+}
+
+// mergeConfigs merges base with override where
+// values from override take precedence over values from base.
+func mergeConfigs(base, override map[string]map[string]string) map[string]map[string]string {
path: pkg/operator/ceph/cluster/cluster.go
user: login: BlaineEXE
body: I agree. We can most likely just set these without complex logic. If we need more protection in the future, we can address it then.

In the RGW PR we agreed to always apply the spec configs over top of any configs Rook previously applied. The idea was that this allows the most user flexibility when they know what they are doing, even allowing them to override what Rook believes to be the best options.
created_at: 2025-04-16T17:46:39Z
author_association: MEMBER
diff_hunk: @@ -741,3 +742,75 @@ func (c *cluster) configureMsgr2() error {
 
 	return nil
 }
+
+// getMergedCephConfig merges both maps from CephConfig and CephConfigFromSecret
+func (c *cluster) getMergedCephConfig() map[string]map[string]string {
+	secretConfig := c.fetchCephConfigFromSecrets()
+	return mergeConfigs(secretConfig, c.Spec.CephConfig)
+}
+
+// fetchCephConfigFromSecrets fetches and parses the cephConfig YAML from the referenced secret
+func (c *cluster) fetchCephConfigFromSecrets() map[string]map[string]string {
+	result := make(map[string]map[string]string)
+
+	for module, keys := range c.Spec.CephConfigFromSecret {
+		if result[module] == nil {
+			result[module] = map[string]string{}
+		}
+
+		for key, selector := range keys {
+			val, err := c.fetchSecretValue(selector)
+			if err != nil {
+				logger.Errorf("failed to get value for key %q in module %q from secret %q: %v",
+					key, module, selector.LocalObjectReference.Name, err)
+				continue
+			}
+			result[module][key] = val
+		}
+	}
+
+	return result
+}
+
+func (c *cluster) fetchSecretValue(selector v1.SecretKeySelector) (string, error) {
+	secret, err := c.context.Clientset.CoreV1().Secrets(c.ClusterInfo.Namespace).Get(
+		c.ClusterInfo.Context, selector.LocalObjectReference.Name, metav1.GetOptions{},
+	)
+	if err != nil {
+		return "", fmt.Errorf("failed to get secret %q: %w", selector.LocalObjectReference.Name, err)
+	}
+
+	val, ok := secret.Data[selector.Key]
+	if !ok {
+		return "", fmt.Errorf("secret %q is missing key %q", selector.LocalObjectReference.Name, selector.Key)
+	}
+
+	return string(val), nil
+}
+
+// mergeConfigs merges base with override where
+// values from override take precedence over values from base.
+func mergeConfigs(base, override map[string]map[string]string) map[string]map[string]string {
path: pkg/operator/ceph/cluster/cluster.go
user: login: patrostkowski
body: indeed merging both maps was not necessary, I went with the simplest approach and executed setting one config after another
created_at: 2025-04-16T20:26:47Z
author_association: CONTRIBUTOR
diff_hunk: @@ -567,7 +568,12 @@ func (c *cluster) shouldSetClusterFullSettings() bool {
 
 func (c *cluster) updateConfigStoreFromCRD() error {
 	monStore := config.GetMonStore(c.context, c.ClusterInfo)
-	return monStore.SetAllMultiple(c.Spec.CephConfig)
+	cephConfig, err := c.getMergedCephConfig()
+	if err != nil {
+		cephConfig = c.Spec.CephConfig
+		logger.Errorf("failed to parse CephConfig and CephConfigSecretRef %q, continue without CephConfigSecretRef: %s", c.Spec.CephConfigSecretRef.Name, err)
+	}
+	return monStore.SetAllMultiple(cephConfig)
path: pkg/operator/ceph/cluster/cluster.go
user: login: patrostkowski
body: I don't think I understood - should the code also validate overwritten keys and print debug and trace messages? 
created_at: 2025-04-16T20:28:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -567,7 +568,12 @@ func (c *cluster) shouldSetClusterFullSettings() bool {
 
 func (c *cluster) updateConfigStoreFromCRD() error {
 	monStore := config.GetMonStore(c.context, c.ClusterInfo)
-	return monStore.SetAllMultiple(c.Spec.CephConfig)
+	cephConfig, err := c.getMergedCephConfig()
+	if err != nil {
+		cephConfig = c.Spec.CephConfig
+		logger.Errorf("failed to parse CephConfig and CephConfigSecretRef %q, continue without CephConfigSecretRef: %s", c.Spec.CephConfigSecretRef.Name, err)
+	}
+	return monStore.SetAllMultiple(cephConfig)
path: pkg/operator/ceph/cluster/cluster.go
user: login: BlaineEXE
body: I'd follow Artem's PR as a pattern since we already went through a lot of discussions on the implementation. Don't worry about validating overwritten keys. Do please print debug messages that say the config 'who' and 'key' being written. And also print trace messages that also include the (possibly insecure) config 'value', which can be used if corner cases need to be debugged while Rook is running.
created_at: 2025-04-18T15:51:58Z
author_association: MEMBER
diff_hunk: @@ -887,6 +888,29 @@ should be used instead.
 
 The operator does not unset any removed config options, it is the user's responsibility to unset or set the default value for each removed option manually using the Ceph CLI.
 
+## Ceph Config From Secret
+
+In addition to `cephConfig`, you can provide Ceph configuration values via Kubernetes Secrets using `cephConfigFromSecret`. This is useful for referencing sensitive values such as passwords or tokens that shouldn't be stored directly in the CR.
path: Documentation/CRDs/Cluster/ceph-cluster-crd.md
user: login: travisn
body: nit: We try to avoid "you" in the docs
```suggestion
In addition to `cephConfig`, Ceph configuration values can be provided via Kubernetes Secrets using `cephConfigFromSecret`. This is useful for referencing sensitive values such as passwords or tokens that shouldn't be stored directly in the CR.
```
created_at: 2025-04-21T18:02:28Z
author_association: MEMBER
diff_hunk: @@ -741,3 +748,45 @@ func (c *cluster) configureMsgr2() error {
 
 	return nil
 }
+
+func (c *cluster) fetchCephConfigFromSecrets() map[string]map[string]string {
path: pkg/operator/ceph/cluster/cluster.go
user: login: travisn
body: Would you mind adding unit tests for this method? 
created_at: 2025-04-21T18:06:59Z
author_association: MEMBER
diff_hunk: @@ -887,6 +888,29 @@ should be used instead.
 
 The operator does not unset any removed config options, it is the user's responsibility to unset or set the default value for each removed option manually using the Ceph CLI.
 
+## Ceph Config From Secret
+
+In addition to `cephConfig`, you can provide Ceph configuration values via Kubernetes Secrets using `cephConfigFromSecret`. This is useful for referencing sensitive values such as passwords or tokens that shouldn't be stored directly in the CR.
+
+The structure mirrors `cephConfig`, but instead of string values, each config entry references a Kubernetes `SecretKeySelector`, specifying:
+
+- name: the name of the secret (in the same namespace as the `CephCluster`)
+- key: the key inside the secret that holds the desired value
+
+```yaml
+spec:
+  cephConfigFromSecret:
+    mgr:
+      mgr/dashboard/GRAFANA_API_PASSWORD:
+        name: grafana-api-credential # name of the Kubernetes secret
+        key: password                # key of the secret value in Kubernetes secret.Data map[string]string
+```
+
+If a given config key is defined in both `cephConfigFromSecret` and `cephConfig`, the value from `cephConfig` takes precedence.
+
+!!! warning
+    If a value from `cephConfigFromSecret` cannot be retrieved — for example, if the referenced Secret or key is missing — Rook will log the issue and continue reconciling without applying that specific configuration entry.
path: Documentation/CRDs/Cluster/ceph-cluster-crd.md
user: login: BlaineEXE
body: This behavior doesn't match what we did for the object store. In our conversation on that work, we decided that applying configs is a stated intent by the admin and that any failure to reach the desired state should be a reconcile error.

Here is that implementation: https://github.com/rook/rook/pull/15426/files#diff-59805cd31b1b8454d5983925c49b66741543cea7be9801f865518a9ea2e2ea98R231-R239
created_at: 2025-04-21T18:22:18Z
author_association: MEMBER
diff_hunk: @@ -887,6 +888,29 @@ should be used instead.
 
 The operator does not unset any removed config options, it is the user's responsibility to unset or set the default value for each removed option manually using the Ceph CLI.
 
+## Ceph Config From Secret
+
+In addition to `cephConfig`, you can provide Ceph configuration values via Kubernetes Secrets using `cephConfigFromSecret`. This is useful for referencing sensitive values such as passwords or tokens that shouldn't be stored directly in the CR.
+
+The structure mirrors `cephConfig`, but instead of string values, each config entry references a Kubernetes `SecretKeySelector`, specifying:
+
+- name: the name of the secret (in the same namespace as the `CephCluster`)
+- key: the key inside the secret that holds the desired value
+
+```yaml
+spec:
+  cephConfigFromSecret:
+    mgr:
+      mgr/dashboard/GRAFANA_API_PASSWORD:
path: Documentation/CRDs/Cluster/ceph-cluster-crd.md
user: login: BlaineEXE
body: Did you test this config as written? I'm assuming this is fine, but I'm not 100% certain that the YAML parser would know what to do with keys that contain `/`.
created_at: 2025-04-21T18:25:12Z
author_association: MEMBER
diff_hunk: @@ -887,6 +888,29 @@ should be used instead.
 
 The operator does not unset any removed config options, it is the user's responsibility to unset or set the default value for each removed option manually using the Ceph CLI.
 
+## Ceph Config From Secret
+
+In addition to `cephConfig`, you can provide Ceph configuration values via Kubernetes Secrets using `cephConfigFromSecret`. This is useful for referencing sensitive values such as passwords or tokens that shouldn't be stored directly in the CR.
+
+The structure mirrors `cephConfig`, but instead of string values, each config entry references a Kubernetes `SecretKeySelector`, specifying:
+
+- name: the name of the secret (in the same namespace as the `CephCluster`)
+- key: the key inside the secret that holds the desired value
+
+```yaml
+spec:
+  cephConfigFromSecret:
+    mgr:
+      mgr/dashboard/GRAFANA_API_PASSWORD:
+        name: grafana-api-credential # name of the Kubernetes secret
+        key: password                # key of the secret value in Kubernetes secret.Data map[string]string
+```
+
+If a given config key is defined in both `cephConfigFromSecret` and `cephConfig`, the value from `cephConfig` takes precedence.
+
+!!! warning
+    If a value from `cephConfigFromSecret` cannot be retrieved — for example, if the referenced Secret or key is missing — Rook will log the issue and continue reconciling without applying that specific configuration entry.
path: Documentation/CRDs/Cluster/ceph-cluster-crd.md
user: login: patrostkowski
body: sure! I was following the user suggested approach, where Rook would only log the issue and continue without setting the key. Let me patch it then.
created_at: 2025-04-23T05:49:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -887,6 +888,29 @@ should be used instead.
 
 The operator does not unset any removed config options, it is the user's responsibility to unset or set the default value for each removed option manually using the Ceph CLI.
 
+## Ceph Config From Secret
+
+In addition to `cephConfig`, you can provide Ceph configuration values via Kubernetes Secrets using `cephConfigFromSecret`. This is useful for referencing sensitive values such as passwords or tokens that shouldn't be stored directly in the CR.
+
+The structure mirrors `cephConfig`, but instead of string values, each config entry references a Kubernetes `SecretKeySelector`, specifying:
+
+- name: the name of the secret (in the same namespace as the `CephCluster`)
+- key: the key inside the secret that holds the desired value
+
+```yaml
+spec:
+  cephConfigFromSecret:
+    mgr:
+      mgr/dashboard/GRAFANA_API_PASSWORD:
path: Documentation/CRDs/Cluster/ceph-cluster-crd.md
user: login: patrostkowski
body: yes! works on my machine and the config is applied correctly:

```
2025-04-23 05:43:43.362867 D | ceph-spec: CephCluster "rook-ceph" status: "Progressing". "failed to create cluster: failed to execute post actions after all the ceph managers started: failed to set config store options: failed to get value for key \"mgr/dashboard/GRAFANA_API_PASSWORD\" in module \"mgr\" from secret \"ceph-config-custom\": secret \"ceph-config-custom\" is missing key \"grafana_api_nonexistent\""
...
2025-04-23 05:44:56.956836 I | ceph-cluster-controller: CR has changed for "rook-ceph". diff=  v1.ClusterSpec{
  	... // 25 identical fields
  	CSI:        {},
  	CephConfig: {"global": {"mon_warn_on_pool_no_redundancy": "false", "osd_pool_default_size": "1"}},
  	CephConfigFromSecret: map[string]map[string]v1.SecretKeySelector{
  		"global": {"osd_pool_default_size": {LocalObjectReference: {Name: "ceph-config-custom"}, Key: "osd_pool_default_size"}},
  		"mgr": {
  			"mgr/dashboard/GRAFANA_API_PASSWORD": {
  				LocalObjectReference: {Name: "ceph-config-custom"},
- 				Key:                  "grafana_api_nonexistent",
+ 				Key:                  "grafana_api_password",
  				Optional:             nil,
  			},
  		},
  	},
  }
...
2025-04-23 05:45:14.232788 D | ceph-cluster-controller: setting Ceph config key "mgr/dashboard/GRAFANA_API_PASSWORD" in module "mgr" from secret "ceph-config-custom"
```

I also think users depend on it, as I had seen a few issues (https://github.com/rook/rook/issues/15678, https://github.com/rook/rook/issues/15559) mentioning using `mgr` modules in CephCluster CRs
created_at: 2025-04-23T05:50:07Z
author_association: CONTRIBUTOR

PR comments:
user: login: patrostkowski
created_at: 2025-04-16T08:07:46Z
body: @travisn I cannot run `make codegen` on my M1 Mac, tried to clean up golang cache and repo cache but without success, any ideas why?

```bash
$ make codegen
Generating deepcopy code for 1 targets
2025/04/16 10:05:58 internal error: package "golang.org/x/sys/unix" without types was imported from "github.com/sirupsen/logrus"
make: *** [codegen] Error 1
```
author_association: CONTRIBUTOR
user: login: travisn
created_at: 2025-04-16T16:49:46Z
body: @patrostkowski Does it help to set the `GOPROXY` as suggested in the [dev guide](https://rook.io/docs/rook/latest-release/Contributing/development-flow/#build)?
author_association: MEMBER
user: login: patrostkowski
created_at: 2025-04-16T16:56:00Z
body: > @patrostkowski Does it help to set the `GOPROXY` as suggested in the [dev guide](https://rook.io/docs/rook/latest-release/Contributing/development-flow/#build)?

sadly no, same outcome :(
author_association: CONTRIBUTOR
user: login: travisn
created_at: 2025-04-16T17:03:50Z
body: > > @patrostkowski Does it help to set the `GOPROXY` as suggested in the [dev guide](https://rook.io/docs/rook/latest-release/Contributing/development-flow/#build)?
> 
> sadly no, same outcome :(

hmm not sure why, but after everything else in the PR is finished, I could run it and push the fix. 
author_association: MEMBER
user: login: BlaineEXE
created_at: 2025-04-16T17:48:43Z
body: > travisn I cannot run `make codegen` on my M1 Mac, tried to clean up golang cache and repo cache but without success, any ideas why?

This works locally for me. I still have the Rook source checked out into my GOPATH at `~/go/src/github.com/rook/rook`, which in theory shouldn't be needed with go modules, but it's possible some part of Rook's tooling still requires the GOPATH location
author_association: MEMBER
user: login: patrostkowski
created_at: 2025-04-16T20:34:06Z
body: @travisn @BlaineEXE I tried running `make codegen` on my 2nd Mac but still ran into issues. I might need to ask one of you to generate the code and push it to this PR for now. Once I get back from holidays I will clean up my setup.

EDIT: managed to run codegen successfully on my old Linux machine
author_association: CONTRIBUTOR
user: login: BlaineEXE
created_at: 2025-04-23T17:48:20Z
body: Looks great! Thanks, @patrostkowski !
author_association: MEMBER