Chapter 9. BlueStore Starting with Red Hat Ceph Storage 4, BlueStore is the default object store for the OSD daemons. The earlier object store, FileStore, requires a file system on top of raw block devices. Objects are then written to the file system. BlueStore does not require an initial file system, because BlueStore puts objects directly on the block device. Important BlueStore provides a high-performance backend for OSD daemons in a production environment. By default, BlueStore is configured to be self-tuning. If you determine that your environment performs better with BlueStore tuned manually, please contact Red Hat support and share the details of your configuration to help us improve the auto-tuning capability. Red Hat looks forward to your feedback and appreciates your recommendations. 9.1. Ceph BlueStore The following are some of the main features of using BlueStore: Direct management of storage devices BlueStore consumes raw block devices or partitions. This avoids any intervening layers of abstraction, such as local file systems like XFS, that might limit performance or add complexity. Metadata management with RocksDB BlueStore uses the RocksDB' key-value database to manage internal metadata, such as the mapping from object names to block locations on a disk. Full data and metadata checksumming By default all data and metadata written to BlueStore is protected by one or more checksums. No data or metadata are read from disk or returned to the user without verification. Efficient copy-on-write The Ceph Block Device and Ceph File System snapshots rely on a copy-on-write clone mechanism that is implemented efficiently in BlueStore. This results in efficient I/O both for regular snapshots and for erasure coded pools which rely on cloning to implement efficient two-phase commits. No large double-writes BlueStore first writes any new data to unallocated space on a block device, and then commits a RocksDB transaction that updates the object metadata to reference the new region of the disk. Only when the write operation is below a configurable size threshold, it falls back to a write-ahead journaling scheme, similar to how FileStore operates. Multi-device support BlueStore can use multiple block devices for storing different data. For example: Hard Disk Drive (HDD) for the data, Solid-state Drive (SSD) for metadata, Non-volatile Memory (NVM) or Non-volatile random-access memory (NVRAM) or persistent memory for the RocksDB write-ahead log (WAL). See Ceph BlueStore devices for details. Note The ceph-disk utility does not yet provision multiple devices. To use multiple devices, OSDs must be set up manually. Efficient block device usage Because BlueStore does not use any file system, it minimizes the need to clear the storage device cache. 9.2. Ceph BlueStore devices This section explains what block devices the BlueStore back end uses. BlueStore manages either one, two, or three storage devices. Primary WAL DB In the simplest case, BlueStore consumes a single (primary) storage device. The storage device is partitioned into two parts that contain: OSD metadata : A small partition formatted with XFS that contains basic metadata for the OSD. This data directory includes information about the OSD, such as its identifier, which cluster it belongs to, and its private keyring. Data : A large partition occupying the rest of the device that is managed directly by BlueStore and that contains all of the OSD data. This primary device is identified by a block symbolic link in the data directory. You can also use two additional devices: A WAL (write-ahead-log) device : A device that stores BlueStore internal journal or write-ahead log. It is identified by the block.wal symbolic link in the data directory. Consider using a WAL device only if the device is faster than the primary device. For example, when the WAL device uses an SSD disk and the primary devices uses an HDD disk. A DB device : A device that stores BlueStore internal metadata. The embedded RocksDB database puts as much metadata as it can on the DB device instead of on the primary device to improve performance. If the DB device is full, it starts adding metadata to the primary device. Consider using a DB device only if the device is faster than the primary device. Warning If you have only a less than a gigabyte storage available on fast devices. Red Hat recommends using it as a WAL device. If you have more fast devices available, consider using it as a DB device. The BlueStore journal is always placed on the fastest device, so using a DB device provides the same benefit that the WAL device while also allows for storing additional metadata. 9.3. Ceph BlueStore caching The BlueStore cache is a collection of buffers that, depending on configuration, can be populated with data as the OSD daemon does reading from or writing to the disk. By default in Red Hat Ceph Storage, BlueStore will cache on reads, but not writes. This is because the bluestore_default_buffered_write option is set to false to avoid potential overhead associated with cache eviction. If the bluestore_default_buffered_write option is set to true , data is written to the buffer first, and then committed to disk. Afterwards, a write acknowledgement is sent to the client, allowing subsequent reads faster access to the data already in cache, until that data is evicted. Read-heavy workloads will not see an immediate benefit from BlueStore caching. As more reading is done, the cache will grow over time and subsequent reads will see an improvement in performance. How fast the cache populates depends on the BlueStore block and database disk type, and the client's workload requirements. Important Please contact Red Hat support before enabling the bluestore_default_buffered_write option. 9.4. Sizing considerations for Ceph BlueStore When mixing traditional and solid state drives using BlueStore OSDs, it is important to size the RocksDB logical volume ( block.db ) appropriately. Red Hat recommends that the RocksDB logical volume be no less than 4% of the block size with object, file and mixed workloads. Red Hat supports 1% of the BlueStore block size with RocksDB and OpenStack block workloads. For example, if the block size is 1 TB for an object workload, then at a minimum, create a 40 GB RocksDB logical volume. When not mixing drive types, there is no requirement to have a separate RocksDB logical volume. BlueStore will automatically manage the sizing of RocksDB. BlueStore's cache memory is used for the key-value pair metadata for RocksDB, BlueStore metadata and object data. Note The BlueStore cache memory values are in addition to the memory footprint already being consumed by the OSD. 9.5. Adding Ceph BlueStore OSDs This section describes how to install a new Ceph OSD node with the BlueStore back end object store. Prerequisites A running Red Hat Ceph Storage cluster. Root-level access to the node. Procedure Add a new OSD node to the [osds] section in Ansible inventory file, by default located at /etc/ansible/hosts . Replace: HOST_NAME with the name of the OSD node Example Navigate to the /usr/share/ceph-ansible/ directory. Create the host_vars directory. Create the configuration file for the newly added OSD in host_vars . Replace: HOST_NAME with the host name of the newly added OSD Example Add the following setting to the newly created file: Note To use BlueStore for all OSDs, add osd_objectstore:bluestore to the group_vars/all.yml file. Configure the BlueStore OSDs, in host_vars/ HOST_NAME .yml : Syntax Replace: DATALV with the data logical volume name DATAVG with the data logical volume group name Example Optional. If you want to store the block.wal and block.db on dedicated logical volumes, edit the host_vars/ HOST_NAME .yml file as follows: Replace: DATALV with the logical volume where the data should be contained WALLV with the logical volume where the write-ahead-log should be contained VG with the volume group the WAL and/or DB device LVs are on DBLV with the logical volume the BlueStore internal metadata should be contained Example Note When using lvm_volumes: with osd_objectstore: bluestore the lvm_volumes YAML dictionary must contain at least data . When defining wal or db , it must have both the LV name and VG name ( db and wal are not required). This allows for four combinations: just data, data and wal, data and wal and db, or data and db. Data can be a raw device, lv or partition. The wal and db can be a lv or partition. When specifying a raw device or partition ceph-volume will put logical volumes on top of them. Note Currently, ceph-ansible does not create the volume groups or the logical volumes. This must be done before running the Anisble playbook. Optional: You can override the block.db default size in the group_vars/all.yml file: Syntax Example Note The value of bluestore_block_db_size should be greater than 2 GB. Open and edit the group_vars/all.yml file, and uncomment the osd_memory_target option. Adjust the value on how much memory you want the OSD to consume. Note The default value for the osd_memory_target option is 4000000000 , which is 4 GB. This option pins the BlueStore cache in memory. Important The osd_memory_target option only applies to BlueStore-backed OSDs. Run the following Ansible playbook: From a Ceph Monitor node, verify that the new OSD has been successfully added: 9.6. Tuning Ceph BlueStore for small writes In BlueStore, the raw partition is allocated and managed in chunks of bluestore_min_alloc_size . By default, bluestore_min_alloc_size is 64 KB for HDDs, and 4 KB for SSDs. The unwritten area in each chunk is filled with zeroes when it is written to the raw partition. This can lead to wasted unused space when not properly sized for your workload, for example when writing small objects. It is best practice to set bluestore_min_alloc_size to match the smallest write so this can write amplification penalty can be avoided. For example, if your client writes 4 KB objects frequently, use ceph-ansible to configure the following setting on OSD nodes: bluestore_min_alloc_size = 4096 Note The settings bluestore_min_alloc_size_ssd and bluestore_min_alloc_size_hdd are specific to SSDs and HDDs, respectively, but setting them is not necessary because setting bluestore_min_alloc_size overrides them. Prerequisites A running Red Hat Ceph Storage cluster. New servers that can be freshly provisioned as OSD nodes, or: OSD nodes that can be redeployed. The admin keyring for the Ceph Monitor node, if you are redeploying an existing Ceph OSD node. Procedure Optional: If redeploying an existing OSD node, use the shrink-osd.yml Ansible playbook to remove the OSD from the cluster. Example If redeploying an existing OSD node, wipe the OSD drives and reinstall the OS. Prepare the node for OSD provisioning using Ansible. Examples of preparation tasks include enabling Red Hat Ceph Storage repositories, adding an Ansible user, and enabling password-less SSH login. Add the bluestore_min_alloc_size to the ceph_conf_overrides section of the group_vars/all.yml Ansible playbook: If deploying a new node, add it to the Ansible inventory file, normally /etc/ansible/hosts : Example If redeploying an existing OSD, copy the admin keyring file in the Ceph Monitor node to the node where you want to deploy the OSD. Provision the OSD node using Ansible: Example After the playbook finishes, verify the setting using the ceph daemon command: Example You can see bluestore_min_alloc_size is set to 4096 bytes, which is equivalent to 4 KiB. Additional Resources See the ` Red Hat Ceph Storage Installation Guide for more information. 9.7. The BlueStore fragmentation tool As a storage administrator, you will want to periodically check the fragmentation level of your BlueStore OSDs. You can check fragmentation levels with one simple command for offline or online OSDs. 9.7.1. Prerequisites A running Red Hat Ceph Storage 3.3 or higher storage cluster. BlueStore OSDs. 9.7.2. What is the BlueStore fragmentation tool? For BlueStore OSDs, the free space gets fragmented over time on the underlying storage device. Some fragmentation is normal, but when there is excessive fragmentation this causes poor performance. The BlueStore fragmentation tool generates a score on the fragmentation level of the BlueStore OSD. This fragmentation score is given as a range, 0 through 1. A score of 0 means no fragmentation, and a score of 1 means severe fragmentation. Table 9.1. Fragmentation scores' meaning Score Fragmentation Amount 0.0 - 0.4 None to tiny fragmentation. 0.4 - 0.7 Small and acceptable fragmentation. 0.7 - 0.9 Considerable, but safe fragmentation. 0.9 - 1.0 Severe fragmentation and that causes performance issues. Important If you have severe fragmentation, and need some help in resolving the issue, contact Red Hat Support . 9.7.3. Checking for fragmentation Checking the fragmentation level of BlueStore OSDs can be done either online or offline. Prerequisites A running Red Hat Ceph Storage 3.3 or higher storage cluster. BlueStore OSDs. Online BlueStore fragmentation score Inspect a running BlueStore OSD process: Simple report: Syntax Example A more detailed report: Syntax Example Offline BlueStore fragmentation score Inspect a non-running BlueStore OSD process: Simple report: Syntax Example A more detailed report: Syntax Example Additional Resources See the Red Hat Ceph Storage 4.1 BlueStore Fragmentation Tool for details on the fragmentation score. 9.8. How to Migrate the Object Store from FileStore to BlueStore As a storage administrator, you can migrate from the traditional object store, FileStore, to the new object store, BlueStore. 9.8.1. Prerequisites A healthy and running Red Hat Ceph Storage cluster. 9.8.2. Migrating from FileStore to BlueStore BlueStore improves performance and robustness, compared to the traditional FileStore. A single Red Hat Ceph Storage cluster can contain a mix of both FileStore and BlueStore devices. Converting an individual OSD cannot be done in place, or in isolation. The conversion process will rely either on the storage cluster's normal replication and healing process or tools and strategies that copy OSD content from an old (FileStore) device to a new (BlueStore) device. There are two approach to migrate from FileStore to BlueStore. First Approach The first approach is to mark out each device in turn, wait for the data to replicate across the storage cluster, reprovision the OSD, and mark it back "in" again. Here are the advantages and disadvantage to this approach: Advantages Simple. Can be done on a device-by-device basis. No spare devices or nodes are required. Disadvantages Copying data over the network happens twice. Note One copy to some other OSD in the storage cluster, allowing you to maintain the desired number of replicas, and then another copy back to the reprovisioned BlueStore OSD. Second Approach The second approach is doing a whole node replacement. You need to have an empty node that has no data. There are two ways to do this: * Starting with a new, empty node that is not part of the storage cluster. * By offloading data from an existing node in the storage cluster. Advantages Data is copied over the network only once. Converts an entire node's OSDs at once. Can parallelize to converting multiple nodes at a time. No spare devices are required on each node. Disadvantages A spare node is required. An entire node's worth of OSDs will be migrating data at a time. This is like likely to impact overall cluster performance. All migrated data still makes one full hop over the network. 9.8.3. Migrating from FileStore to BlueStore using Ansible Migrating from FileStore to BlueStore using Ansible will shrinks and redeploys all OSDs on the node. The Ansible playbook does a capacity check before starting the migration. The ceph-volume utility then redeploys the OSDs. Prerequisites A healthy and running Red Hat Ceph Storage 4 cluster. The ansible user account for use with the Ansible application. Procedure Log in as the ansible user on the Ansible administration node. Edit the group_vars/osd.yml file, add and set the following options: Run the following Ansible playbook: Syntax Example Warning If you explicitly set osd_crush_update_on_start = False in your Ceph configuration file, the conversion fails. It creates a new OSD with a different ID and misplaces it in the CRUSH rule. Additionally, it does not clear the old OSD data directory. Wait for the migration to complete before starting on the OSD node in the storage cluster. 9.8.4. Migrating from FileStore to BlueStore using the mark out and replace approach The simplest approach to migrate from FileStore to BlueStore is to mark out each device in turn, wait for the data to replicate across the storage cluster, reprovision the OSD, and mark it back "in" again. Prerequisites A running Red Hat Ceph Storage cluster. root access to the node. Procedure Replace the variable OSD_ID below with the ODS identification number. Find a FileStore OSD to replace. Get the OSD identification number: Identify whether an OSD is using FileStore or BlueStore: Syntax Example To view the current count of FileStore devices versus BlueStore devices: Mark the FileStore OSD out: Wait for the data to migrate off the OSD: Stop the OSD: Capture which device this OSD is using: Unmount the OSD: Destroy the OSD data, using the value from step 5 as DEVICE : Important Be EXTREMELY CAREFUL as this will destroy the contents of the device. Be certain the data on the device is not needed, that is the storage cluster is healthy, before proceeding.`` Note If the OSD is encrypted, then the unmount the osd-lockbox and remove the encryption before zapping the OSD using dmsetup remove . Note If the OSD contains logical volumes, then use the --destroy option on the ceph-volume lvm zap command. Make the storage cluster aware that the OSD has been destroyed: Reprovision the OSD as a BlueStore OSD, using DEVICE from step 5, and the same OSD_ID : Repeat this procedure. Note The refilling of the new BlueStore OSD can happen concurrently with the draining of the FileStore OSD, as long as you ensure the storage cluster is HEALTH_OK before destroying any OSDs. Failure to do so will reduce the redundancy of your data and increase the risk of, or the potentially of data loss. 9.8.5. Migrating from FileStore to BlueStore using the whole node replacement approach Migrating from FileStore to BlueStore can be done on a node-by-node basis by transferring each stored copy of the data only once. This migration can be done with a spare node in the storage cluster, or having the sufficient free space to evacuate an entire node from the storage cluster in order to use it as a spare. Ideally, the node must have roughly the same capacity as the other nodes you will be migrating. Prerequisites A running Red Hat Ceph Storage cluster. root access to the node. An empty node that has no data. Procedure Replace the variable NEWNODE below with the new node name. Replace the variable EXISTING_NODE_TO_CONVERT below with the node name already existing in the storage cluster. Replace the variable OSD_ID below with the OSD identification number. Using a new node that is not in the storage cluster. For using an existing node already in the storage cluster, skip to step 3. Add the node to the CRUSH hierarchy: Important Do not attach it to the root. Install the Ceph software packages: Note Copy the Ceph configuration file, by default /etc/ceph/ceph.conf , and keyrings to the new node. Skip to step 5. If you are using an existing node already in the storage cluster, use the following command: Note Where default is the immediate ancestor in the CRUSH map. Skip to step 8. Provision new BlueStore OSDs for all devices: Verify that OSDs joined the cluster: You should see the new node name with all of the OSDs underneath the node name, but the node must not be nested underneath any other node in hierarchy. Example Swap the new node into the old node's position in the cluster: At this point, all data on the EXISTING_NODE_TO_CONVERT will start migrating to OSDs on the NEWNODE . Note If there is a difference in the total capacity of the old and new nodes you might also see some data migrate to or from other nodes in the storage cluster, but as long as the nodes are similarly sized this will be a relatively small amount of data. Wait for data migration to complete: Log into the EXISTING_NODE_TO_CONVERT , stop and unmount all old OSDs on the now-empty EXISTING_NODE_TO_CONVERT : Destroy and purge the old OSDs: Wipe the old OSD devices. This requires you do identify which devices are to be wiped manually. Do the following command for each device: Important Be EXTREMELY CAREFUL as this will destroy the contents of the device. Be certain the data on the device is not needed, that is the storage cluster is healthy, before proceeding. Note If the OSD is encrypted, then the unmount the osd-lockbox and remove the encryption before zapping the OSD using dmsetup remove . Note If the OSD contains logical volumes, then use the --destroy option on the ceph-volume lvm zap command. Use the now-empty old node as the new node, and repeat the process.

Commands:
[osds] node1 node2 node3 HOST_NAME
[osds] node1 node2 node3 node4
[user@admin ~]USD cd /usr/share/ceph-ansible
[root@admin ceph-ansible] mkdir host_vars
[root@admin ceph-ansible] touch host_vars/ HOST_NAME .yml
[root@admin ceph-ansible] touch host_vars/node4.yml
osd_objectstore: bluestore
lvm_volumes: - data: DATALV data_vg: DATAVG
lvm_volumes: - data: data-lv1 data_vg: vg1
lvm_volumes: - data: DATALV wal: WALLV wal_vg: VG db: DBLV db_vg: VG
lvm_volumes: - data: data-lv3 wal: wal-lv1 wal_vg: vg3 db: db-lv3 db_vg: vg3
ceph_conf_overrides: osd: bluestore_block_db_size: VALUE
ceph_conf_overrides: osd: bluestore_block_db_size: 24336000000
[user@admin ceph-ansible]USD ansible-playbook site.yml
ceph osd tree
ansible-playbook -v infrastructure-playbooks/shrink-osd.yml -e osd_to_kill= OSD_ID
[admin@admin ceph-ansible]USD ansible-playbook -v infrastructure-playbooks/shrink-osd.yml -e osd_to_kill=1
ceph_conf_overrides: osd: bluestore_min_alloc_size: 4096
[osds] OSD_NODE_NAME
[osds] osd1 devices="[ '/dev/sdb' ]"
ansible-playbook -v site.yml -l OSD_NODE_NAME
[admin@admin ceph-ansible]USD ansible-playbook -v site.yml -l osd1
ceph daemon OSD.ID config get bluestore_min_alloc_size
ceph daemon osd.1 config get bluestore_min_alloc_size { "bluestore_min_alloc_size": "4096" }
ceph daemon OSD_ID bluestore allocator score block
ceph daemon osd.123 bluestore allocator score block
ceph daemon OSD_ID bluestore allocator dump block
ceph daemon osd.123 bluestore allocator dump block
ceph-bluestore-tool --path PATH_TO_OSD_DATA_DIRECTORY --allocator block free-score
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-score
ceph-bluestore-tool --path PATH_TO_OSD_DATA_DIRECTORY --allocator block free-dump
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-123 --allocator block free-dump
nb_retry_wait_osd_up: 50 delay_wait_osd_up: 30
ansible-playbook infrastructure-playbooks/filestore-to-bluestore.yml --limit OSD_NODE_TO_MIGRATE
[ansible@admin ~]USD ansible-playbook infrastructure-playbooks/filestore-to-bluestore.yml --limit osd1
ceph osd tree
ceph osd metadata OSD_ID | grep osd_objectstore
ceph osd metadata 0 | grep osd_objectstore "osd_objectstore": "filestore",
ceph osd count-metadata osd_objectstore
ceph osd out OSD_ID
while ! ceph osd safe-to-destroy OSD_ID ; do sleep 60 ; done
systemctl stop ceph-osd@ OSD_ID
mount | grep /var/lib/ceph/osd/ceph- OSD_ID
umount /var/lib/ceph/osd/ceph- OSD_ID
ceph-volume lvm zap DEVICE
ceph osd destroy OSD_ID --yes-i-really-mean-it
ceph-volume lvm create --bluestore --data DEVICE --osd-id OSD_ID
ceph osd crush add-bucket NEWNODE node
yum install ceph-osd
ceph osd crush unlink EXISTING_NODE_TO_CONVERT default
ceph-volume lvm create --bluestore --data /dev/ DEVICE
ceph osd tree
ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -5 0 node newnode 10 ssd 1.00000 osd.10 up 1.00000 1.00000 11 ssd 1.00000 osd.11 up 1.00000 1.00000 12 ssd 1.00000 osd.12 up 1.00000 1.00000 -1 3.00000 root default -2 3.00000 node oldnode1 0 ssd 1.00000 osd.0 up 1.00000 1.00000 1 ssd 1.00000 osd.1 up 1.00000 1.00000 2 ssd 1.00000 osd.2 up 1.00000 1.00000
ceph osd crush swap-bucket NEWNODE EXISTING_NODE_TO_CONVERT
while ! ceph osd safe-to-destroy USD(ceph osd ls-tree EXISTING_NODE_TO_CONVERT ); do sleep 60 ; done
systemctl stop ceph-osd@ OSD_ID umount /var/lib/ceph/osd/ceph- OSD_ID
for osd in ceph osd ls-tree EXISTING_NODE_TO_CONVERT ; do ceph osd purge USDosd --yes-i-really-mean-it ; done
ceph-volume lvm zap DEVICE