Project: Rook (Runtime / Cloud Native Storage)
File: osd-migration.md

Configure the default OSD backend via the `cephCluster` CR so that new OSDs are built with it. Existing OSDs should be able to migrate to use a new backend store configured via `cephCluster` CR. The migration process should include destroying the existing OSDs one by one, wiping the drive, then recreating a new OSD on that drive with the same ID. Ceph uses BlueStore as a special-purpose storage backend designed specifically for Ceph OSD workloads. The upstream Ceph team is working on improving BlueStore in ways that will significantly improve performance, especially in disaster recovery scenarios. This new OSD backend is known as `bluestore-rdr`. In the future, other backends will also need to be supported, such as `seastore` from the Crimson effort. To employ a new torage backend, the existing OSD needs to be destroyed, drives wiped, and new OSD created using the specified storage backend. Newly provisioned clusters should be able to use the specific storage backend when creating OSDs. Existing clusters should be able to migrate existing OSDs to a specific backend, without downtime and without risking data loss. Note that this necessarily entails redeploying and backfilling each OSD in turn. Migration of OSDs with following configurations is deferred for now and will considered in the future: Hybrid OSDs where metadata (RocksDB+WAL) is placed on faster storage media and application data on slower media. Setups with multiple OSDs per drive, though with recent Ceph releases the motivation for deploying this way is mostly obviated. Add `spec.storage.store` in the Ceph cluster YAML. ```yaml storage: store: type: bluestore-rdr updateStore: yes-really-update-store ``` `type`: The backend to be used for OSDs: `bluestore`, `bluestore-rdr`, etc. The default type will be `bluestore` `updateStore`: Allows the operator to migrate existing OSDs to a different backend. This field can only take the value `yes-really-update-store`. If the user wants to change the `store.type` field for an existing cluster, they will also need to update `spec.storage.store.updateStore` with `yes-really-update-store`. Add `status.storage.osd` to the Ceph cluster status. This will help convey the progress of OSD migration ``` yaml status: storage: osd: storeType: bluestore: 3 bluestore-rdr: 5 ``` `storeType.bluestore`: Total number of BlueStore OSDs running `storeType.bluestore-rdr`: Total number of BlueStore-rdr OSDs running The cluster's `phase` should be set to `progressing` while OSDs are migrating The migration process will involve destroying existing OSDs one by one, wiping the drives, deploying a new OSD with the same ID, then waiting for all PGs to be `active+clean` before migrating the next OSD. Since this operation involves possible impact or downtime, users should be exercise caution before proceeding with this action. NOTE: Once the OSDs are migrated to a new backend, say `bluestore-rdr`, they won't be allowed to be migrated back to the legacy store (BlueStore). Add a new label `osd-store:<osd store type>` to all OSD

This label will help to identify the current backend being used for the OSD and enable the operator to determine if the OSD should be migrated if the the admin changes the backend type in the spec (`spec.Storage.store.type`). Set the OSD backend provided by the user in `spec.storage.store.type` as an environment variable in the OSD prepare job. If no OSD store is provided in the spec, then set the environment variable to `bluestore`. The prepare pod will use this environment variable when preparing OSDs with the `ceph-volume` command. RAW MODE ``` ceph-volume raw prepare <OSDSTOREENV_VARIABLE> --data /dev/vda ``` LVM MODE ``` ceph-volume lvm prepare <OSDSTOREENV_VARIABLE> --data /dev/vda ``` The ceph-volume `activate` command doesn't require the OSD backend to be passed as an argument. It auto-detects the backend that was used during when the OSD was prepared. After upgrading an existing Rook Ceph cluster to a Ceph release that supports `bluestore-rdr`, admins can migrate `bluestore` OSDs to `bluestore-rdr`. The backend of OSDs can not be overridden, so this update will require the OSDs to be replaced one by one. In order to migrate OSDs to use `bluestore-rdr`, admins must patch the Ceph cluster spec as below: ```yaml storage: store: type: bluestore-rdr updateStore: yes-really-update-store ``` The operator's reconciler will replace one OSD at a time. A configmap will be used to store the OSD ID currently being migrated. OSD replacement steps: List all OSDs where `osd-store:<osd store type>` does not match `spec.storage.store.type`. If all PGs are not `active+clean`, do not proceed. If all PGs are `active+clean` but a previous OSD replacement is not completed, do not proceed. If all PGs are `active+clean` and no replacement is in progress, then select an OSD to be migrated. Delete the OSD deployment. Create an OSD prepare job with an environment variable indictating the OSD ID to be replaced. The OSD prepare pod will destroy the OSD and prepare it again using the same OSD ID. Refer for details. Once the destroyed OSD pod is recreated, delete the configmap. If there is any error during the OSD migration, then preserve the OSD ID being replaced in the configmap for next reconcile. Reconcile the operator and perform as same steps until all the OSDs have migrated to the new backend. The OSD prepare pod job will destroy an OSD using following steps: Check for OSD ID environment variable of the OSD to be destroyed. Use `ceph volume list` to fetch the OSD path. Destroy the OSD using following command: ``` ceph osd destroy <OSD_ID> --yes-i-really-mean-it ``` Wipe the OSD drive. This removes all the data on the device. Prepare the OSD with the new store type by using the same OSD ID. This is done by passing the OSD ID as `--osd-id` flag to `ceph-volume` command. ``` ceph-volume lvm prepare --osd-id <OSDID> --data <OSDPATH> ``` These changes require significant development effort to migrate existing OSDs to use a new backend. They will be divided into following phases: New OSDs (greenfield). Migrating existing OSDs on PVC without metadata devices. Migrating existing OSDs on PVC with metadata devices. Existing node-based OSDs (multiple OSDs are created at once via `ceph-volume batch` which adds additional complications).