Chapter 11. Scaling the Ceph Storage cluster 11.1. Scaling up the Ceph Storage cluster You can scale up the number of Ceph Storage nodes in your overcloud by re-running the deployment with the number of Ceph Storage nodes you need. Before doing so, ensure that you have enough nodes for the updated deployment. These nodes must be registered with the director and tagged accordingly. Registering New Ceph Storage Nodes To register new Ceph storage nodes with the director, follow these steps: Log in to the undercloud as the stack user and initialize your director configuration: Define the hardware and power management details for the new nodes in a new node definition template; for example, instackenv-scale.json . Import this file to the OpenStack director: Importing the node definition template registers each node defined there to the director. Assign the kernel and ramdisk images to all nodes: Note For more information about registering new nodes, see Section 2.2, "Registering nodes" . Manually Tagging New Nodes After you register each node, you must inspect the hardware and tag the node into a specific profile. Use profile tags to match your nodes to flavors, and then assign flavors to deployment roles. To inspect and tag new nodes, complete the following steps: Trigger hardware introspection to retrieve the hardware attributes of each node: The --all-manageable option introspects only the nodes that are in a managed state. In this example, all nodes are in a managed state. The --provide option resets all nodes to an active state after introspection. Important Ensure that this process completes successfully. This process usually takes 15 minutes for bare metal nodes. Retrieve a list of your nodes to identify their UUIDs: Add a profile option to the properties/capabilities parameter for each node to manually tag a node to a specific profile. The addition of the profile option tags the nodes into each respective profile. Note As an alternative to manual tagging, use the Automated Health Check (AHC) Tools to automatically tag larger numbers of nodes based on benchmarking data. For example, the following commands tag three additional nodes with the ceph-storage profile: Tip If the nodes you just tagged and registered use multiple disks, you can set the director to use a specific root disk on each node. See Section 2.4, "Defining the root disk for multi-disk clusters" for instructions on how to do so. Re-deploying the Overcloud with Additional Ceph Storage Nodes After registering and tagging the new nodes, you can now scale up the number of Ceph Storage nodes by re-deploying the overcloud. When you do, set the CephStorageCount parameter in the parameter_defaults of your environment file (in this case, ~/templates/storage-config.yaml ). In Section 7.1, "Assigning nodes and flavors to roles" , the overcloud is configured to deploy with 3 Ceph Storage nodes. To scale it up to 6 nodes instead, use: Upon re-deployment with this setting, the overcloud should now have 6 Ceph Storage nodes instead of 3. 11.2. Scaling down and replacing Ceph Storage nodes In some cases, you might need to scale down your Ceph cluster, or even replace a Ceph Storage node, for example, if a Ceph Storage node is faulty. In either situation, you must disable and rebalance any Ceph Storage node that you want to remove from the overcloud to avoid data loss. Note This procedure uses steps from the Red Hat Ceph Storage Administration Guide to manually remove Ceph Storage nodes. For more in-depth information about manual removal of Ceph Storage nodes, see Starting, stopping, and restarting Ceph daemons that run in containers and Removing a Ceph OSD using the command-line interface . Procedure Log in to a Controller node as the heat-admin user. The director's stack user has an SSH key to access the heat-admin user. List the OSD tree and find the OSDs for your node. For example, the node you want to remove might contain the following OSDs: Disable the OSDs on the Ceph Storage node. In this case, the OSD IDs are 0 and 1. The Ceph Storage cluster begins rebalancing. Wait for this process to complete. Follow the status by using the following command: After the Ceph cluster completes rebalancing, log in to the Ceph Storage node you are removing, in this case overcloud-cephstorage-0 , as the heat-admin user and stop the node. Stop the OSDs. While logged in to the Controller node, remove the OSDs from the CRUSH map so that they no longer receive data. Remove the OSD authentication key. Remove the OSD from the cluster. Leave the node and return to the undercloud as the stack user. Disable the Ceph Storage node so that director does not reprovision it. Removing a Ceph Storage node requires an update to the overcloud stack in director with the local template files. First identify the UUID of the overcloud stack: Identify the UUIDs of the Ceph Storage node you want to delete: Delete the node from the stack and update the plan accordingly: Important If you passed any extra environment files when you created the overcloud, pass them again here using the -e option to avoid making undesired changes to the overcloud. For more information, see Modifying the overcloud environment in the Director Installation and Usage guide. Wait until the stack completes its update. Use the heat stack-list --show-nested command to monitor the stack update. Add new nodes to the director node pool and deploy them as Ceph Storage nodes. Use the CephStorageCount parameter in the parameter_defaults of your environment file, in this case, ~/templates/storage-config.yaml , to define the total number of Ceph Storage nodes in the overcloud. Note For more information about how to define the number of nodes per role, see Section 7.1, "Assigning nodes and flavors to roles" . After you update your environment file, redeploy the overcloud: Director provisions the new node and updates the entire stack with the details of the new node. Log in to a Controller node as the heat-admin user and check the status of the Ceph Storage node: Confirm that the value in the osdmap section matches the number of nodes in your cluster that you want. The Ceph Storage node that you removed is replaced with a new node. 11.3. Adding an OSD to a Ceph Storage node This procedure demonstrates how to add an OSD to a node. For more information about Ceph OSDs, see Ceph OSDs in the Red Hat Ceph Storage Operations Guide . Procedure Notice the following heat template deploys Ceph Storage with three OSD devices: To add an OSD, update the node disk layout as described in Section 5.3, "Mapping the Ceph Storage node disk layout" . In this example, add /dev/sde to the template: Run openstack overcloud deploy to update the overcloud. Note This example assumes that all hosts with OSDs have a new device called /dev/sde . If you do not want all nodes to have the new device, update the heat template. For for information about how to define hosts with a differing devices list, see Section 5.5, "Mapping the disk layout to non-homogeneous Ceph Storage nodes" . 11.4. Removing an OSD from a Ceph Storage node This procedure demonstrates how to remove an OSD from a node. It assumes the following about the environment: A server ( ceph-storage0 ) has an OSD ( ceph-osd@4 ) running on /dev/sde . The Ceph monitor service ( ceph-mon ) is running on controller0 . There are enough available OSDs to ensure the storage cluster is not at its near-full ratio. For more information about Ceph OSDs, see Ceph OSDs in the Red Hat Ceph Storage Operations Guide . Procedure SSH into ceph-storage0 and log in as root . Disable and stop the OSD service: Disconnect from ceph-storage0 . SSH into controller0 and log in as root . Identify the name of the Ceph monitor container: Enable the Ceph monitor container to mark the undesired OSD as out : Note This command causes Ceph to rebalance the storage cluster and copy data to other OSDs in the cluster. The cluster temporarily leaves the active+clean state until rebalancing is complete. Run the following command and wait for the storage cluster state to become active+clean : Remove the OSD from the CRUSH map so that it no longer receives data: Remove the OSD authentication key: Remove the OSD: Disconnect from controller0 . SSH into the undercloud as the stack user and locate the heat environment file in which you defined the CephAnsibleDisksConfig parameter. Notice the heat template contains four OSDs: Modify the template to remove /dev/sde . Run openstack overcloud deploy to update the overcloud. Note This example assumes that you removed the /dev/sde device from all hosts with OSDs. If you do not remove the same device from all nodes, update the heat template. For for information about how to define hosts with a differing devices list, see Section 5.5, "Mapping the disk layout to non-homogeneous Ceph Storage nodes" .

Commands:
source ~/stackrc
openstack overcloud node import ~/instackenv-scale.json
openstack overcloud node configure
openstack overcloud node introspect --all-manageable --provide
openstack baremetal node list
openstack baremetal node set 551d81f5-4df2-4e0f-93da-6c5de0b868f7 --property capabilities="profile:ceph-storage,boot_option:local" openstack baremetal node set 5e735154-bd6b-42dd-9cc2-b6195c4196d7 --property capabilities="profile:ceph-storage,boot_option:local" openstack baremetal node set 1a2b090c-299d-4c20-a25d-57dd21a7085b --property capabilities="profile:ceph-storage,boot_option:local"
parameter_defaults: ControllerCount: 3 OvercloudControlFlavor: control ComputeCount: 3 OvercloudComputeFlavor: compute CephStorageCount: 6 OvercloudCephStorageFlavor: ceph-storage CephMonCount: 3 OvercloudCephMonFlavor: ceph-mon
-2 0.09998 host overcloud-cephstorage-0 0 0.04999 osd.0 up 1.00000 1.00000 1 0.04999 osd.1 up 1.00000 1.00000
[heat-admin@overcloud-controller-0 ~]USD sudo podman exec ceph-mon-<HOSTNAME> ceph osd out 0 [heat-admin@overcloud-controller-0 ~]USD sudo podman exec ceph-mon-<HOSTNAME> ceph osd out 1
[heat-admin@overcloud-controller-0 ~]USD sudo podman exec ceph-mon-<HOSTNAME> ceph -w
[heat-admin@overcloud-cephstorage-0 ~]USD sudo systemctl disable ceph-osd@0 [heat-admin@overcloud-cephstorage-0 ~]USD sudo systemctl disable ceph-osd@1
[heat-admin@overcloud-cephstorage-0 ~]USD sudo systemctl stop ceph-osd@0 [heat-admin@overcloud-cephstorage-0 ~]USD sudo systemctl stop ceph-osd@1
[heat-admin@overcloud-controller-0 ~]USD sudo podman exec ceph-mon-<HOSTNAME> ceph osd crush remove osd.0 [heat-admin@overcloud-controller-0 ~]USD sudo podman exec ceph-mon-<HOSTNAME> ceph osd crush remove osd.1
[heat-admin@overcloud-controller-0 ~]USD sudo podman exec ceph-mon-<HOSTNAME> ceph auth del osd.0 [heat-admin@overcloud-controller-0 ~]USD sudo podman exec ceph-mon-<HOSTNAME> ceph auth del osd.1
[heat-admin@overcloud-controller-0 ~]USD sudo podman exec ceph-mon-<HOSTNAME> ceph osd rm 0 [heat-admin@overcloud-controller-0 ~]USD sudo podman exec ceph-mon-<HOSTNAME> ceph osd rm 1
[heat-admin@overcloud-controller-0 ~]USD exit [stack@director ~]USD
[stack@director ~]USD openstack baremetal node list [stack@director ~]USD openstack baremetal node maintenance set UUID
openstack stack list
openstack server list
openstack overcloud node delete --stack overcloud <NODE_UUID>
parameter_defaults: ControllerCount: 3 OvercloudControlFlavor: control ComputeCount: 3 OvercloudComputeFlavor: compute CephStorageCount: 3 OvercloudCephStorageFlavor: ceph-storage CephMonCount: 3 OvercloudCephMonFlavor: ceph-mon
openstack overcloud deploy --templates -e <ENVIRONMENT_FILE>
[heat-admin@overcloud-controller-0 ~]USD sudo ceph status
parameter_defaults: CephAnsibleDisksConfig: devices: - /dev/sdb - /dev/sdc - /dev/sdd osd_scenario: lvm osd_objectstore: bluestore
parameter_defaults: CephAnsibleDisksConfig: devices: - /dev/sdb - /dev/sdc - /dev/sdd - /dev/sde osd_scenario: lvm osd_objectstore: bluestore
systemctl disable ceph-osd@4 systemctl stop ceph-osd@4
podman ps | grep ceph-mon ceph-mon-controller0
podman exec ceph-mon-controller0 ceph osd out 4
podman exec ceph-mon-controller0 ceph -w
podman exec ceph-mon-controller0 ceph osd crush remove osd.4
podman exec ceph-mon-controller0 ceph auth del osd.4
podman exec ceph-mon-controller0 ceph osd rm 4
parameter_defaults: CephAnsibleDisksConfig: devices: - /dev/sdb - /dev/sdc - /dev/sdd - /dev/sde osd_scenario: lvm osd_objectstore: bluestore
parameter_defaults: CephAnsibleDisksConfig: devices: - /dev/sdb - /dev/sdc - /dev/sdd osd_scenario: lvm osd_objectstore: bluestore