Repository: rook/rook (Go)

Issue:
Block pool deserialization is failing when a custom crush rule was created

rook-ceph v1.15.0 doesn't create CephBlockPool "rook-ceph/replicapool"

rook-ceph-operator error log is as follows

```
2024-09-04 07:40:02.608111 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 07:40:02.836318 E | ceph-block-pool-controller: failed to reconcile CephBlockPool "rook-ceph/replicapool". invalid pool CR "replicapool" spec: failed to get crush map: failed to unmarshal crush map: json: cannot unmarshal number -1 into Go struct field stepSpec.rules.steps.num of type uint
```

```
# cat cephblockpool.yaml
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: replicapool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 3

# kubectl create -f cephblockpool.yaml 
cephblockpool.ceph.rook.io/replicapool created

```

ceph didn't find replicapool pool

```
bash-5.1$ ceph osd pool ls 
.mgr


bash-5.1$ ceph status
  cluster:
    id:     15475c67-ab60-4fa2-aa80-92ed018fa93c
    health: HEALTH_OK
 
  services:
    mon: 3 daemons, quorum a,c,b (age 23h)
    mgr: a(active, since 20h), standbys: b
    osd: 21 osds: 21 up (since 4h), 21 in (since 4h)
 
  data:
    pools:   1 pools, 1 pgs
    objects: 2 objects, 449 KiB
    usage:   630 MiB used, 1.4 TiB / 1.4 TiB avail
    pgs:     1 active+clean

bash-5.1$ ceph -v
ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)

sh-5.1$ rook version
2024/09/04 08:06:09 maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
rook: v1.15.0
go: go1.22.6

```

OS and k8s information

```
# cat /etc/os-release 
NAME="AlmaLinux"
VERSION="9.2 (Turquoise Kodkod)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.2 (Turquoise Kodkod)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.2"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"


# uname -a
Linux test-alma9-kvm167 5.14.0-284.11.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Tue May 9 05:49:00 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

# kubectl version 
Client Version: v1.30.0
```




Discussion:
The complete rook-ceph-operator log is as follows:
```
# kubectl -n rook-ceph logs rook-ceph-operator-b798d7d74-wtfkk 
2024/09/04 10:32:43 maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
2024-09-04 10:32:43.050729 I | rookcmd: starting Rook v1.15.0 with arguments '/usr/local/bin/rook ceph operator'
2024-09-04 10:32:43.050799 I | rookcmd: flag values: --enable-machine-disruption-budget=false, --help=false, --kubeconfig=, --log-level=INFO
2024-09-04 10:32:43.050810 I | cephcmd: starting Rook-Ceph operator
2024-09-04 10:32:43.133536 I | cephcmd: base ceph version inside the rook operator image is "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)"
2024-09-04 10:32:43.139141 I | op-k8sutil: ROOK_CURRENT_NAMESPACE_ONLY="false" (env var)
2024-09-04 10:32:43.139204 I | operator: watching all namespaces for Ceph CRs
2024-09-04 10:32:43.139291 I | operator: setting up schemes
2024-09-04 10:32:43.146904 I | op-k8sutil: ROOK_OPERATOR_METRICS_BIND_ADDRESS="0" (configmap)
2024-09-04 10:32:43.146924 I | operator: setting up the controller-runtime manager
2024-09-04 10:32:43.147268 I | ceph-cluster-controller: successfully started
2024-09-04 10:32:43.150392 I | op-k8sutil: ROOK_DISABLE_DEVICE_HOTPLUG="false" (configmap)
2024-09-04 10:32:43.150410 I | ceph-cluster-controller: enabling hotplug orchestration
2024-09-04 10:32:43.150469 I | ceph-nodedaemon-controller: successfully started
2024-09-04 10:32:43.150498 I | ceph-block-pool-controller: successfully started
2024-09-04 10:32:43.150572 I | ceph-object-store-user-controller: successfully started
2024-09-04 10:32:43.150674 I | ceph-object-realm-controller: successfully started
2024-09-04 10:32:43.150763 I | ceph-object-zonegroup-controller: successfully started
2024-09-04 10:32:43.150826 I | ceph-object-zone-controller: successfully started
2024-09-04 10:32:43.150983 I | ceph-object-controller: successfully started
2024-09-04 10:32:43.151031 I | ceph-file-controller: successfully started
2024-09-04 10:32:43.151084 I | ceph-nfs-controller: successfully started
2024-09-04 10:32:43.151148 I | ceph-rbd-mirror-controller: successfully started
2024-09-04 10:32:43.151195 I | ceph-client-controller: successfully started
2024-09-04 10:32:43.151235 I | ceph-filesystem-mirror-controller: successfully started
2024-09-04 10:32:43.151309 I | operator: rook-ceph-operator-config-controller successfully started
2024-09-04 10:32:43.151354 I | ceph-csi: rook-ceph-operator-csi-controller successfully started
2024-09-04 10:32:43.151516 I | op-bucket-prov: rook-ceph-operator-bucket-controller successfully started
2024-09-04 10:32:43.151554 I | ceph-bucket-topic: successfully started
2024-09-04 10:32:43.151566 I | ceph-bucket-notification: successfully started
2024-09-04 10:32:43.151578 I | ceph-bucket-notification: successfully started
2024-09-04 10:32:43.151615 I | ceph-fs-subvolumegroup-controller: successfully started
2024-09-04 10:32:43.151702 I | blockpool-rados-namespace-controller: successfully started
2024-09-04 10:32:43.153244 I | ceph-cosi-controller: successfully started
2024-09-04 10:32:43.153287 I | operator: starting the controller-runtime manager
2024-09-04 10:32:43.258936 I | op-k8sutil: ROOK_CEPH_COMMANDS_TIMEOUT_SECONDS="15" (configmap)
2024-09-04 10:32:43.258976 I | op-k8sutil: ROOK_LOG_LEVEL="INFO" (configmap)
2024-09-04 10:32:43.259006 I | op-k8sutil: ROOK_ENABLE_DISCOVERY_DAEMON="false" (configmap)
2024-09-04 10:32:43.264483 I | op-k8sutil: ROOK_CEPH_ALLOW_LOOP_DEVICES="false" (configmap)
2024-09-04 10:32:43.264500 I | operator: rook-ceph-operator-config-controller done reconciling
2024-09-04 10:32:43.264697 I | op-k8sutil: ROOK_WATCH_FOR_NODE_FAILURE="true" (configmap)
2024-09-04 10:32:43.267562 I | op-k8sutil: ROOK_CSI_DISABLE_DRIVER="false" (configmap)
2024-09-04 10:32:43.268089 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:43.282190 I | ceph-cluster-controller: reconciling ceph cluster in namespace "rook-ceph"
2024-09-04 10:32:43.282593 I | clusterdisruption-controller: deleted all legacy node drain canary pods
2024-09-04 10:32:43.324217 I | ceph-block-pool-controller: skipping reconcile since operator is still initializing
2024-09-04 10:32:43.655269 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:43.655333 I | op-k8sutil: ROOK_OBC_WATCH_OPERATOR_NAMESPACE="true" (configmap)
2024-09-04 10:32:43.655341 I | op-k8sutil: ROOK_OBC_PROVISIONER_NAME_PREFIX="" (default)
2024-09-04 10:32:43.655345 I | op-bucket-prov: ceph bucket provisioner launched watching for provisioner "rook-ceph.ceph.rook.io/bucket"
2024-09-04 10:32:43.655903 I | op-bucket-prov: successfully reconciled bucket provisioner
I0904 10:32:43.656004       1 manager.go:135] "msg"="starting provisioner" "logger"="objectbucket.io/provisioner-manager" "name"="rook-ceph.ceph.rook.io/bucket"
I0904 10:32:44.451809       1 request.go:700] Waited for 1.169571269s due to client-side throttling, not priority and fairness, request: GET:https://172.23.4.1:443/api/v1/namespaces/rook-ceph/secrets/rook-ceph-mon
2024-09-04 10:32:45.253972 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:45.307753 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/s3-store". failed to retrieve current ceph "mon" version: failed to get ceph daemons versions: failed to run 'ceph versions'. . Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)'): exit status 1
2024-09-04 10:32:45.456144 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:45.469742 I | ceph-cluster-controller: enabling ceph mon monitoring goroutine for cluster "rook-ceph"
2024-09-04 10:32:45.469783 I | op-osd: ceph osd status in namespace "rook-ceph" check interval "1m0s"
2024-09-04 10:32:45.469788 I | ceph-cluster-controller: enabling ceph osd monitoring goroutine for cluster "rook-ceph"
2024-09-04 10:32:45.469816 I | ceph-cluster-controller: ceph status check interval is 1m0s
2024-09-04 10:32:45.469823 I | ceph-cluster-controller: enabling ceph status monitoring goroutine for cluster "rook-ceph"
2024-09-04 10:32:45.550038 I | ceph-cluster-controller: skipping ceph status since operator is still initializing
2024-09-04 10:32:45.657785 I | op-k8sutil: ROOK_USE_CSI_OPERATOR="false" (configmap)
2024-09-04 10:32:46.466449 I | ceph-spec: detecting the ceph image version for image quay.io/ceph/ceph:v18.2.4...
2024-09-04 10:32:46.654716 I | op-k8sutil: CSI_ENABLE_HOST_NETWORK="true" (default)
2024-09-04 10:32:46.654754 I | op-k8sutil: CSI_DISABLE_HOLDER_PODS="true" (configmap)
2024-09-04 10:32:47.254397 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:47.320851 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/s3-store". failed to retrieve current ceph "mon" version: failed to get ceph daemons versions: failed to run 'ceph versions'. . Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)'): exit status 1
2024-09-04 10:32:48.655115 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:49.254245 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:49.311330 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/s3-store". failed to retrieve current ceph "mon" version: failed to get ceph daemons versions: failed to run 'ceph versions'. . Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)'): exit status 1
2024-09-04 10:32:49.657560 E | ceph-csi: failed to delete CSI-operator Ceph Connection "". no matches for kind "CephConnection" in version "csi.ceph.io/v1alpha1"
2024-09-04 10:32:49.658534 E | ceph-csi: failed to delete CSI-operator client profile "". no matches for kind "ClientProfile" in version "csi.ceph.io/v1alpha1"
2024-09-04 10:32:49.659694 E | ceph-csi: failed to delete CSI-operator Ceph Connection "". no matches for kind "CephConnection" in version "csi.ceph.io/v1alpha1"
2024-09-04 10:32:49.660857 E | ceph-csi: failed to delete CSI-operator client profile "". no matches for kind "ClientProfile" in version "csi.ceph.io/v1alpha1"
2024-09-04 10:32:49.661942 E | ceph-csi: failed to delete CSI-operator driver config "". no matches for kind "Driver" in version "csi.ceph.io/v1alpha1"
2024-09-04 10:32:49.662815 E | ceph-csi: failed to delete CSI-operator operator config "". no matches for kind "OperatorConfig" in version "csi.ceph.io/v1alpha1"
2024-09-04 10:32:49.662828 I | op-k8sutil: ROOK_CSI_ENABLE_RBD="true" (configmap)
2024-09-04 10:32:49.662833 I | op-k8sutil: ROOK_CSI_ENABLE_CEPHFS="false" (configmap)
2024-09-04 10:32:49.662836 I | op-k8sutil: ROOK_CSI_ENABLE_NFS="false" (configmap)
2024-09-04 10:32:49.662916 I | op-k8sutil: ROOK_CSI_ALLOW_UNSUPPORTED_VERSION="false" (configmap)
2024-09-04 10:32:49.662925 I | op-k8sutil: CSI_FORCE_CEPHFS_KERNEL_CLIENT="true" (configmap)
2024-09-04 10:32:49.662928 I | op-k8sutil: CSI_GRPC_TIMEOUT_SECONDS="150" (configmap)
2024-09-04 10:32:49.662932 I | op-k8sutil: CSI_CEPHFS_LIVENESS_METRICS_PORT="9081" (default)
2024-09-04 10:32:49.662935 I | op-k8sutil: CSIADDONS_PORT="9070" (default)
2024-09-04 10:32:49.662938 I | op-k8sutil: CSI_RBD_LIVENESS_METRICS_PORT="9080" (default)
2024-09-04 10:32:49.662943 I | op-k8sutil: CSI_ENABLE_LIVENESS="false" (configmap)
2024-09-04 10:32:49.662946 I | op-k8sutil: CSI_PLUGIN_PRIORITY_CLASSNAME="system-node-critical" (configmap)
2024-09-04 10:32:49.662951 I | op-k8sutil: CSI_PROVISIONER_PRIORITY_CLASSNAME="system-cluster-critical" (configmap)
2024-09-04 10:32:49.662957 I | op-k8sutil: CSI_ENABLE_OMAP_GENERATOR="false" (default)
2024-09-04 10:32:49.662960 I | op-k8sutil: CSI_ENABLE_RBD_SNAPSHOTTER="true" (configmap)
2024-09-04 10:32:49.662966 I | op-k8sutil: CSI_ENABLE_CEPHFS_SNAPSHOTTER="false" (configmap)
2024-09-04 10:32:49.662972 I | op-k8sutil: CSI_ENABLE_NFS_SNAPSHOTTER="true" (configmap)
2024-09-04 10:32:49.665069 I | op-k8sutil: CSI_ENABLE_CSIADDONS="false" (configmap)
2024-09-04 10:32:49.665083 I | op-k8sutil: CSI_ENABLE_TOPOLOGY="false" (configmap)
2024-09-04 10:32:49.665087 I | op-k8sutil: CSI_ENABLE_ENCRYPTION="false" (configmap)
2024-09-04 10:32:49.665091 I | op-k8sutil: CSI_ENABLE_METADATA="false" (default)
2024-09-04 10:32:49.665094 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2024-09-04 10:32:49.665097 I | op-k8sutil: CSI_CEPHFS_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2024-09-04 10:32:49.665100 I | op-k8sutil: CSI_NFS_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2024-09-04 10:32:49.665104 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY="RollingUpdate" (default)
2024-09-04 10:32:49.665106 I | op-k8sutil: CSI_RBD_PLUGIN_UPDATE_STRATEGY_MAX_UNAVAILABLE="1" (default)
2024-09-04 10:32:49.665110 I | op-k8sutil: CSI_PLUGIN_ENABLE_SELINUX_HOST_MOUNT="false" (configmap)
2024-09-04 10:32:49.665113 I | ceph-csi: Kubernetes version is 1.30
2024-09-04 10:32:49.665117 I | op-k8sutil: CSI_LOG_LEVEL="" (default)
2024-09-04 10:32:49.665120 I | op-k8sutil: CSI_SIDECAR_LOG_LEVEL="" (default)
2024-09-04 10:32:49.665123 I | op-k8sutil: CSI_LEADER_ELECTION_LEASE_DURATION="" (default)
2024-09-04 10:32:49.665136 I | op-k8sutil: CSI_LEADER_ELECTION_RENEW_DEADLINE="" (default)
2024-09-04 10:32:49.665139 I | op-k8sutil: CSI_LEADER_ELECTION_RETRY_PERIOD="" (default)
2024-09-04 10:32:50.457986 I | ceph-spec: detected ceph image version: "18.2.4-0 reef"
2024-09-04 10:32:50.458002 I | ceph-cluster-controller: validating ceph version from provided image
2024-09-04 10:32:50.659460 I | op-k8sutil: CSI_PROVISIONER_REPLICAS="2" (configmap)
2024-09-04 10:32:50.659479 I | op-k8sutil: ROOK_CSI_CEPH_IMAGE="quay.io/cephcsi/cephcsi:v3.12.0" (default)
2024-09-04 10:32:50.659483 I | op-k8sutil: ROOK_CSI_REGISTRAR_IMAGE="registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.11.1" (default)
2024-09-04 10:32:50.659488 I | op-k8sutil: ROOK_CSI_PROVISIONER_IMAGE="registry.k8s.io/sig-storage/csi-provisioner:v5.0.1" (default)
2024-09-04 10:32:50.659490 I | op-k8sutil: ROOK_CSI_ATTACHER_IMAGE="registry.k8s.io/sig-storage/csi-attacher:v4.6.1" (default)
2024-09-04 10:32:50.659493 I | op-k8sutil: ROOK_CSI_SNAPSHOTTER_IMAGE="registry.k8s.io/sig-storage/csi-snapshotter:v8.0.1" (default)
2024-09-04 10:32:50.659496 I | op-k8sutil: ROOK_CSI_RESIZER_IMAGE="registry.k8s.io/sig-storage/csi-resizer:v1.11.1" (default)
2024-09-04 10:32:50.659503 I | op-k8sutil: ROOK_CSI_KUBELET_DIR_PATH="/var/lib/kubelet" (default)
2024-09-04 10:32:50.659509 I | op-k8sutil: ROOK_CSIADDONS_IMAGE="quay.io/csiaddons/k8s-sidecar:v0.9.0" (default)
2024-09-04 10:32:50.659513 I | op-k8sutil: CSI_TOPOLOGY_DOMAIN_LABELS="" (default)
2024-09-04 10:32:50.659517 I | op-k8sutil: ROOK_CSI_CEPHFS_POD_LABELS="" (default)
2024-09-04 10:32:50.659520 I | op-k8sutil: ROOK_CSI_NFS_POD_LABELS="" (default)
2024-09-04 10:32:50.659533 I | op-k8sutil: ROOK_CSI_RBD_POD_LABELS="" (default)
2024-09-04 10:32:50.659535 I | op-k8sutil: CSI_CLUSTER_NAME="" (default)
2024-09-04 10:32:50.659537 I | op-k8sutil: ROOK_CSI_IMAGE_PULL_POLICY="IfNotPresent" (default)
2024-09-04 10:32:50.659539 I | op-k8sutil: CSI_CEPHFS_KERNEL_MOUNT_OPTIONS="" (default)
2024-09-04 10:32:50.659542 I | op-k8sutil: CSI_CEPHFS_ATTACH_REQUIRED="true" (configmap)
2024-09-04 10:32:50.659545 I | op-k8sutil: CSI_RBD_ATTACH_REQUIRED="true" (configmap)
2024-09-04 10:32:50.659549 I | op-k8sutil: CSI_NFS_ATTACH_REQUIRED="true" (configmap)
2024-09-04 10:32:50.659551 I | op-k8sutil: CSI_DRIVER_NAME_PREFIX="rook-ceph" (default)
2024-09-04 10:32:50.661606 I | op-k8sutil: CSI_ENABLE_VOLUME_GROUP_SNAPSHOT="true" (configmap)
2024-09-04 10:32:50.661616 I | op-k8sutil: CSI_KUBE_API_BURST="" (default)
2024-09-04 10:32:50.661619 I | op-k8sutil: CSI_KUBE_API_QPS="" (default)
2024-09-04 10:32:50.661623 I | ceph-csi: detecting the ceph csi image version for image "quay.io/cephcsi/cephcsi:v3.12.0"
2024-09-04 10:32:50.661699 I | op-k8sutil: CSI_PROVISIONER_TOLERATIONS="" (default)
2024-09-04 10:32:50.661706 I | op-k8sutil: CSI_PROVISIONER_NODE_AFFINITY="" (default)
2024-09-04 10:32:51.256375 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:51.315930 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/s3-store". failed to retrieve current ceph "mon" version: failed to get ceph daemons versions: failed to run 'ceph versions'. . Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)'): exit status 1
2024-09-04 10:32:52.454298 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:53.255773 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:53.311007 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/s3-store". failed to retrieve current ceph "mon" version: failed to get ceph daemons versions: failed to run 'ceph versions'. . Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)'): exit status 1
2024-09-04 10:32:53.454387 I | cephclient: writing config file /var/lib/rook/rook-ceph/rook-ceph.config
2024-09-04 10:32:53.454528 I | cephclient: generated admin config in /var/lib/rook/rook-ceph
2024-09-04 10:32:53.735331 I | ceph-cluster-controller: cluster "rook-ceph": version "18.2.4-0 reef" detected for image "quay.io/ceph/ceph:v18.2.4"
2024-09-04 10:32:54.657519 I | ceph-csi: Detected ceph CSI image version: "v3.12.0"
2024-09-04 10:32:54.662334 I | op-k8sutil: CSI_PLUGIN_TOLERATIONS="" (default)
2024-09-04 10:32:54.662354 I | op-k8sutil: CSI_PLUGIN_NODE_AFFINITY="" (default)
2024-09-04 10:32:54.662363 I | op-k8sutil: CSI_RBD_PLUGIN_TOLERATIONS="- key: ceph-taint\n  operator: Equal\n  value: \"osd\"\n  effect: NoExecute\n- key: ceph-taint\n  operator: Equal\n  effect: NoExecute\n  value: \"mon\"\n  effect: NoExecute\n" (configmap)
2024-09-04 10:32:54.662397 I | op-k8sutil: CSI_RBD_PLUGIN_NODE_AFFINITY="ceph-rbd-plug=enabled" (configmap)
2024-09-04 10:32:54.662442 I | op-k8sutil: CSI_RBD_PLUGIN_RESOURCE="" (default)
2024-09-04 10:32:54.662451 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME="" (default)
2024-09-04 10:32:54.662453 I | op-k8sutil: CSI_RBD_PLUGIN_VOLUME_MOUNT="" (default)
2024-09-04 10:32:54.678917 I | op-k8sutil: CSI_RBD_PROVISIONER_TOLERATIONS="- key: ceph-taint\n  operator: Equal\n  value: \"osd\"\n  effect: NoExecute\n- key: ceph-taint\n  operator: Equal\n  value: \"mon\"\n  effect: NoExecute\n" (configmap)
2024-09-04 10:32:54.678991 I | op-k8sutil: CSI_RBD_PROVISIONER_NODE_AFFINITY="ceph-rbd-plug=enabled" (configmap)
2024-09-04 10:32:54.679017 I | op-k8sutil: CSI_RBD_PROVISIONER_RESOURCE="" (default)
2024-09-04 10:32:54.698407 I | ceph-csi: successfully started CSI Ceph RBD driver
2024-09-04 10:32:54.698423 I | op-k8sutil: CSI_RBD_FSGROUPPOLICY="File" (configmap)
2024-09-04 10:32:54.704077 I | ceph-csi: CSIDriver object updated for driver "rook-ceph.rbd.csi.ceph.com"
2024-09-04 10:32:54.704095 I | op-k8sutil: removing daemonset csi-cephfsplugin if it exists
2024-09-04 10:32:54.705913 I | op-k8sutil: removing deployment csi-cephfsplugin-provisioner if it exists
I0904 10:32:54.852380       1 request.go:700] Waited for 1.098166946s due to client-side throttling, not priority and fairness, request: GET:https://172.23.4.1:443/api/v1/namespaces/rook-ceph/configmaps/rook-config-override
2024-09-04 10:32:54.863914 I | op-mon: start running mons
2024-09-04 10:32:55.453716 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:55.655151 I | ceph-spec: parsing mon endpoints: c=192.168.0.158:6789,a=192.168.0.159:6789,b=192.168.0.160:6789
2024-09-04 10:32:55.681810 E | ceph-block-pool-controller: failed to reconcile CephBlockPool "rook-ceph/replicapool". invalid pool CR "replicapool" spec: failed to get crush map: failed to unmarshal crush map: json: cannot unmarshal number -1 into Go struct field stepSpec.rules.steps.num of type uint
2024-09-04 10:32:55.858371 I | ceph-csi: successfully removed CSI CephFS driver
2024-09-04 10:32:55.858396 I | op-k8sutil: removing daemonset csi-nfsplugin if it exists
2024-09-04 10:32:55.860737 I | op-k8sutil: removing deployment csi-nfsplugin-provisioner if it exists
2024-09-04 10:32:56.094627 E | ceph-object-controller: failed to reconcile CephObjectStore "rook-ceph/s3-store". failed to get admin ops API context: failed to create or retrieve rgw admin ops user: failed to create object user "rgw-admin-ops-user". error code 1 for object store "s3-store": failed to create s3 user. 2024-09-04T10:32:56.087+0000 7ff0ee347840  0 ERROR: could not find zonegroup (s3-store)
2024-09-04T10:32:56.087+0000 7ff0ee347840  0 ERROR: failed to start notify service ((2) No such file or directory
2024-09-04T10:32:56.087+0000 7ff0ee347840  0 ERROR: failed to init services (ret=(2) No such file or directory)
couldn't init storage provider. : exit status 5
```
Do commands related to OSDs and the crush map succeed from the toolbox? The command that is failing related to the pools is: `ceph osd crush dump`. Does that command fail in the toolbox as well?
> Do commands related to OSDs and the crush map succeed from the toolbox? The command that is failing related to the pools is: `ceph osd crush dump`. Does that command fail in the toolbox as well?

The "ceph osd tree" and "ceph osd crush dump" commands are executed successfully in the tools pod

```
#   kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-tools" -o jsonpath='{.items[0].metadata.name}') bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

bash-5.1$ ceph versions
{
    "mon": {
        "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 3
    },
    "mgr": {
        "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 2
    },
    "osd": {
        "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 21
    },
    "overall": {
        "ceph version 18.2.4 (e7ad5345525c7aa95470c26863873b581076945d) reef (stable)": 26
    }
}


bash-5.1$ ceph osd crush dump
{
    "devices": [
        {
            "id": 0,
            "name": "osd.0",
            "class": "hdd"
        },
...

bash-5.1$ ceph osd tree
ID   CLASS  WEIGHT   TYPE NAME                   STATUS  REWEIGHT  PRI-AFF
-26         0.75163  root test-mix                                        
-43         0.10738      host hdd-kvm164                                  
 12    hdd  0.04880          osd.12                  up   1.00000  1.00000
 13    hdd  0.05858          osd.13                  up   1.00000  1.00000
-46         0.10738      host hdd-kvm165                                  
  0    hdd  0.04880          osd.0                   up   1.00000  1.00000
  1    hdd  0.05858          osd.1                   up   1.00000  1.00000
-49         0.10738      host hdd-kvm166                                  
 10    hdd  0.04880          osd.10                  up   1.00000  1.00000
 11    hdd  0.05858          osd.11                  up   1.00000  1.00000
-52         0.10738      host hdd-kvm167                                  
  6    hdd  0.04880          osd.6                   up   1.00000  1.00000
  7    hdd  0.05858          osd.7                   up   1.00000  1.00000

```
[ceph-osd-crush-dump.txt](https://github.com/user-attachments/files/16882737/ceph-osd-crush-dump.txt)

Ok, I see now that the error is about how rook is deserializing the results of the crush dump. Rook's definition [here](https://github.com/rook/rook/blob/master/pkg/daemon/ceph/client/crush.go#L78) expects a `uint` for the `num` value, but since the value is `-1`, the deserialization is failing. Would you mind attaching the full crush dump to analyze further? I'm curious what is unique about your env that caused this `-1` value. 

Before creating CephBlockPool, I manually created two rules in ceph: rbd-mix-rule and mixed_replicated_rule

Attached file is the complete ceph osd getcrushmap result

[crush_map_decompile.txt](https://github.com/user-attachments/files/16901874/crush_map_decompile.txt)
[crush_map.tar.gz](https://github.com/user-attachments/files/16901877/crush_map.tar.gz)

Ah, you are delving into custom crush rules, which is why others haven't hit this. The serialization error is easy enough to fix. But what is the custom crush rule trying to accomplish? We'd like rook to generate the crush rules according to settings in the blockpool CR, if possible, in case others want to do the same thing. 
1. I want to use custom crush to achieve mixed crush of SSD/HDD, This allows customizing the root bucket and achieving low cost and high performance,for example

```
rule mixed_replicated_rule_two {
       id 1
       type replicated
       step take ssd_hosts class ssd
       step chooseleaf firstn 1 type host
       step emit
       step take hdd_hosts class hdd
       step chooseleaf firstn -1 type host
       step emit
}

rule mixed-s3-store.rgw.buckets.data {
        id 13
        type erasure
        step set_chooseleaf_tries 5
        step set_choose_tries 100
        step take test-ssd-data class ssd
        step chooseleaf indep 1 type host
        step emit
        step take test-hdd class hdd
        step chooseleaf indep -1 type host
        step emit
}
```

reference: https://docs.ceph.com/en/latest/rados/operations/crush-map/#custom-crush-rules


2. I suggest rook-ceph CephBlockPool CephObjectStore to add parameters for selecting custom crush

3. Is there any way to solve this problem? Delete the custom crush? But I don't want to delete the custom crush.


In the pool spec there is a setting: `enableCrushUpdates`, which is `false` by default. This means Rook will not change the crush rules after initial creation. So if you keep that default, I would expect the following to work:
1. Create your custom crush rules as you currently are, before the pools are created in Rook. 
   - Name the crush rules the same as the name of the pool (this is the convention Rook requires). 
2. Create the pool. Rook will find the crush rules already exist, and will not update them since `enableCrushUpdates: false`

With the `-1` step num, you'll just need the fix from #14709, which we will plan on in the next release.
My enableCrushUpdates parameter is the default value false, but modifying crush.go requires recompiling the rook image. I will wait for the next version of rook v1.15.2. Thank you
> My enableCrushUpdates parameter is the default value false, but modifying crush.go requires recompiling the rook image. I will wait for the next version of rook v1.15.2. Thank you

The release is planned by end of next week.



Patch:
diff --git a/pkg/daemon/ceph/client/crush.go b/pkg/daemon/ceph/client/crush.go
index 2f1c4d4631cd..75bb6fc01caf 100644
--- a/pkg/daemon/ceph/client/crush.go
+++ b/pkg/daemon/ceph/client/crush.go
@@ -75,7 +75,7 @@ type ruleSpec struct {
 
 type stepSpec struct {
 	Operation string `json:"op"`
-	Number    uint   `json:"num"`
+	Number    int    `json:"num"`
 	Item      int    `json:"item"`
 	ItemName  string `json:"item_name"`
 	Type      string `json:"type"`
diff --git a/pkg/daemon/ceph/client/crush_rule.go b/pkg/daemon/ceph/client/crush_rule.go
index b79d5470ac5c..97575b2a8164 100644
--- a/pkg/daemon/ceph/client/crush_rule.go
+++ b/pkg/daemon/ceph/client/crush_rule.go
@@ -150,7 +150,7 @@ func buildTwoStepCrushSteps(pool cephv1.PoolSpec) []stepSpec {
 	// Step three
 	stepTakeSubFailureDomain := &stepSpec{
 		Operation: "chooseleaf_firstn",
-		Number:    pool.Replicated.ReplicasPerFailureDomain,
+		Number:    int(pool.Replicated.ReplicasPerFailureDomain),
 		Type:      pool.Replicated.SubFailureDomain,
 	}
 	steps = append(steps, *stepTakeSubFailureDomain)


Test patch:
diff --git a/pkg/daemon/ceph/client/crush_rule_test.go b/pkg/daemon/ceph/client/crush_rule_test.go
index ce799bfdf138..c48e52b7355d 100644
--- a/pkg/daemon/ceph/client/crush_rule_test.go
+++ b/pkg/daemon/ceph/client/crush_rule_test.go
@@ -56,7 +56,7 @@ func TestBuildCrushSteps(t *testing.T) {
 	assert.Equal(t, 4, len(steps))
 	assert.Equal(t, cephv1.DefaultCRUSHRoot, steps[0].ItemName)
 	assert.Equal(t, "datacenter", steps[1].Type)
-	assert.Equal(t, uint(2), steps[2].Number)
+	assert.Equal(t, 2, steps[2].Number)
 }
 
 func TestCompileCRUSHMap(t *testing.T) {