Chapter 3. Configuring OpenStack to use Ceph block devices As a storage administrator, you must configure the Red Hat OpenStack Platform to use the Ceph block devices. The Red Hat OpenStack Platform can use Ceph block devices for Cinder, Cinder Backup, Glance, and Nova. 3.1. Prerequisites A new or existing Red Hat Ceph Storage cluster. A running Red Hat OpenStack Platform environment. 3.2. Configuring Cinder to use Ceph block devices The Red Hat OpenStack Platform can use Ceph block devices to provide back-end storage for Cinder volumes. Prerequisites Root-level access to the Cinder node. A Ceph volume pool. The user and UUID of the secret to interact with Ceph block devices. Procedure Edit the Cinder configuration file: In the [DEFAULT] section, enable Ceph as a backend for Cinder: Ensure that the Glance API version is set to 2. If you are configuring multiple cinder back ends in enabled_backends , the glance_api_version = 2 setting must be in the [DEFAULT] section and not the [ceph] section. Create a [ceph] section in the cinder.conf file. Add the Ceph settings in the following steps under the [ceph] section. Specify the volume_driver setting and set it to use the Ceph block device driver: Specify the cluster name and Ceph configuration file location. In typical deployments the Ceph cluster has a cluster name of ceph and a Ceph configuration file at /etc/ceph/ceph.conf . If the Ceph cluster name is not ceph , specify the cluster name and configuration file path appropriately: By default, Red Hat OpenStack Platform stores Ceph volumes in the rbd pool. To use the volumes pool created earlier, specify the rbd_pool setting and set the volumes pool: Red Hat OpenStack Platform does not have a default user name or a UUID of the secret for volumes. Specify rbd_user and set it to the cinder user. Then, specify the rbd_secret_uuid setting and set it to the generated UUID stored in the uuid-secret.txt file: Specify the following settings: When you configure Cinder to use Ceph block devices, the configuration file might look similar to this: Example Note Consider removing the default [lvm] section and its settings. 3.3. Configuring Cinder backup to use Ceph block devices The Red Hat OpenStack Platform can configure Cinder backup to use Ceph block devices. Prerequisites Root-level access to the Cinder node. Procedure Edit the Cinder configuration file: Go to the [ceph] section of the configuration file. Specify the backup_driver setting and set it to the Ceph driver: Specify the backup_ceph_conf setting and specify the path to the Ceph configuration file: Note The Cinder backup Ceph configuration file may be different from the Ceph configuration file used for Cinder. For example, it can point to a different Ceph storage cluster. Specify the Ceph pool for backups: Note The Ceph configuration file used for Cinder backup might be different from the Ceph configuration file used for Cinder. Specify the backup_ceph_user setting and specify the user as cinder-backup : Specify the following settings: When you include the Cinder options, the [ceph] section of the cinder.conf file might look similar to this: Example Verify if Cinder backup is enabled: If enable_backup is set to False , then edit the local_settings file and set it to True . Example 3.4. Configuring Glance to use Ceph block devices The Red Hat OpenStack Platform can configure Glance to use Ceph block devices. Prerequisites Root-level access to the Glance node. Procedure To use Ceph block devices by default, edit the /etc/glance/glance-api.conf file. If you used different pool, user or Ceph configuration file settings apply the appropriate values. Uncomment the following settings if necessary and change their values accordingly: To enable copy-on-write (CoW) cloning set show_image_direct_url to True . Important Enabling CoW exposes the back end location via Glance's API, so the endpoint should not be publicly accessible. Disable cache management if necessary. The flavor should be set to keystone only, not keystone+cachemanagement . Red Hat recommends the following properties for images: The virtio-scsi controller gets better performance and provides support for discard operations. For systems using SCSI/SAS drives, connect every Cinder block device to that controller. Also, enable the QEMU guest agent and send fs-freeze/thaw calls through the QEMU guest agent. 3.5. Configuring Nova to use Ceph block devices The Red Hat OpenStack Platform can configure Nova to use Ceph block devices. You must configure each Nova node to use ephemeral back-end storage devices, which allows all virtual machines to use the Ceph block devices. Prerequisites Root-level access to the Nova nodes. Procedure Edit the Ceph configuration file: Add the following section to the [client] section of the Ceph configuration file: Create new directories for the admin socket and log file, and change the directory permissions to use the qemu user and libvirtd group: Note The directories must be allowed by SELinux or AppArmor. On each Nova node, edit the /etc/nova/nova.conf file. Under the [libvirt] section, configure the following settings: Example Replace the UUID in rbd_user_secret with the UUID in the uuid-secret.txt file. 3.6. Restarting the OpenStack services Restarting the Red Hat OpenStack Platform services enables you to activate the Ceph block device drivers. Prerequisites Root-level access to the Red Hat OpenStack Platform nodes. Procedure Load the block device pool names and Ceph user names into the configuration file. Restart the appropriate OpenStack services after modifying the corresponding configuration files:

Commands:
vim /etc/cinder/cinder.conf
enabled_backends = ceph
glance_api_version = 2
volume_driver = cinder.volume.drivers.rbd.RBDDriver
rbd_cluster_name = us-west rbd_ceph_conf = /etc/ceph/us-west.conf
rbd_pool = volumes
rbd_user = cinder rbd_secret_uuid = 4b5fd580-360c-4f8c-abb5-c83bb9a3f964
rbd_flatten_volume_from_snapshot = false rbd_max_clone_depth = 5 rbd_store_chunk_size = 4 rados_connect_timeout = -1
[DEFAULT] enabled_backends = ceph glance_api_version = 2 ... [ceph] volume_driver = cinder.volume.drivers.rbd.RBDDriver rbd_cluster_name = ceph rbd_pool = volumes rbd_user = cinder rbd_ceph_conf = /etc/ceph/ceph.conf rbd_flatten_volume_from_snapshot = false rbd_secret_uuid = 4b5fd580-360c-4f8c-abb5-c83bb9a3f964 rbd_max_clone_depth = 5 rbd_store_chunk_size = 4 rados_connect_timeout = -1
vim /etc/cinder/cinder.conf
backup_driver = cinder.backup.drivers.ceph
backup_ceph_conf = /etc/ceph/ceph.conf
backup_ceph_pool = backups
backup_ceph_user = cinder-backup
backup_ceph_chunk_size = 134217728 backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true
[ceph] volume_driver = cinder.volume.drivers.rbd.RBDDriver rbd_cluster_name = ceph rbd_pool = volumes rbd_user = cinder rbd_ceph_conf = /etc/ceph/ceph.conf rbd_flatten_volume_from_snapshot = false rbd_secret_uuid = 4b5fd580-360c-4f8c-abb5-c83bb9a3f964 rbd_max_clone_depth = 5 rbd_store_chunk_size = 4 rados_connect_timeout = -1 backup_driver = cinder.backup.drivers.ceph backup_ceph_user = cinder-backup backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true
cat /etc/openstack-dashboard/local_settings | grep enable_backup
OPENSTACK_CINDER_FEATURES = { 'enable_backup': True, }
vim /etc/glance/glance-api.conf
stores = rbd default_store = rbd rbd_store_chunk_size = 8 rbd_store_pool = images rbd_store_user = glance rbd_store_ceph_conf = /etc/ceph/ceph.conf
show_image_direct_url = True
flavor = keystone
hw_scsi_model=virtio-scsi hw_disk_bus=scsi hw_qemu_guest_agent=yes os_require_quiesce=yes
vim /etc/ceph/ceph.conf
[client] rbd cache = true rbd cache writethrough until flush = true rbd concurrent management ops = 20 admin socket = /var/run/ceph/guests/USDcluster-USDtype.USDid.USDpid.USDcctid.asok log file = /var/log/ceph/qemu-guest-USDpid.log
mkdir -p /var/run/ceph/guests/ /var/log/ceph/ chown qemu:libvirt /var/run/ceph/guests /var/log/ceph/
[libvirt] images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/ceph.conf rbd_user = cinder rbd_secret_uuid = 4b5fd580-360c-4f8c-abb5-c83bb9a3f964 disk_cachemodes="network=writeback" inject_password = false inject_key = false inject_partition = -2 live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED" hw_disk_discard = unmap
systemctl restart openstack-cinder-volume systemctl restart openstack-cinder-backup systemctl restart openstack-glance-api systemctl restart openstack-nova-compute