Chapter 15. Scaling the Ceph Storage cluster You can scale the size of your Ceph Storage cluster by adding or removing storage nodes. 15.1. Scaling up the Ceph Storage cluster As capacity and performance requirements change, you can scale up your Ceph Storage cluster to meet increased demands. Before doing so, ensure that you have enough nodes for the updated deployment. Then you can register and tag the new nodes in your Red Hat OpenStack Platform (RHOSP) environment. This procedure results in the following actions: The storage networks and firewall rules are configured on the new CephStorage nodes. The ceph-admin user is created on the new CephStorage nodes. The ceph-admin user public SSH key is distributed to the new CephStorage nodes so that cephadm can use SSH to add extra nodes. If a new CephMon or CephMgr node is added, the ceph-admin private SSH key is also distributed to that node. The updated Ceph specification is applied and cephadm schedules the new nodes to join the Ceph cluster. Procedure Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: Modify the ~/overcloud-baremetal-deploy.yaml to add the CephStorage nodes to the deployment. The following example file represents an original deployment with three CephStorage nodes. The following example modifies this file to add three additional nodes. Use the openstack overcloud node provision command with the updated ~/overcloud-baremetal-deploy.yaml file. Note This command will provision the configured nodes and and output an updated copy of ~/overcloud-baremetal-deployed.yaml . The new version updates the CephStorage role. The DeployedServerPortMap and HostnameMap also contains the new storage nodes. Use the openstack overcloud ceph spec command to generate a Ceph specification file. Note The files used in the openstack overcloud ceph spec should already be available for use. They are created in the following locations: The overcloud-baremetal-deployed.yaml file was created in the step of this procedure. The osd_spec.yaml file was created in Configuring advanced OSD specifications . Providing the OSD specification with the --osd-spec parameter is optional. The roles_data.yaml file was created in Designating nodes for Red Hat Ceph Storage . It is assumed the new nodes are assigned to one of the roles in this file. The output of this command will be the ceph_spec.yaml file. Use the openstack overcloud ceph user enable command to create the ceph-admin user on all nodes in the cluster. The ceph-admin user must be present on all nodes to enable SSH access to a node by the Ceph orchestrator. Note Use the ceph_spec.yaml file created in the step. Copy the ceph_spec.yaml file to the controller-0 node. Log in to the controller-0 node. Mount the ceph_spec.yaml file : Replace <spec_file_path> with the fully qualified path and file name of the ceph_spec.yaml file. Use the orchestrator to apply ceph_spec.yaml : Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: Optional: Use the openstack overcloud deploy command with the updated ~/overcloud-baremetal-deployed.yaml file. Note This step is only necessary if you are scaling out a ComputeHCI node. It is not necessary if you are scaling out a CephStorage node. 15.2. Scaling down and replacing Red Hat Ceph Storage nodes In some cases, you might need to scale down your Red Hat Ceph Storage cluster or replace a Red Hat Ceph Storage node. In either situation, you must disable and rebalance the Red Hat Ceph Storage nodes that you want to remove from the overcloud to prevent data loss. Procedure Do not proceed with this procedure if the Red Hat Ceph Storage cluster does not have the capacity to lose OSDs. Log in to the overcloud Controller node as the tripleo-admin user. Use the sudo cephadm shell command to start a Ceph shell. Use the ceph osd tree command to identify OSDs to be removed by server. In the following example we want to identify the OSDs of ceph-2 host. Exit the cephadm shell Export the Ceph cluster specification to a YAML file. Edit the specification file exported in the step. Remove all the occurrences of the scaled down node from the placement:hosts section of the spec.yaml file. Save the edited file. Apply the modified Ceph specification file. Important If you do not export and edit the Ceph specification file before removing the OSDs, the Ceph Manager will attempt to recreate the OSDs. Use the sudo cephadm shell command to start a Ceph shell. Use the command ceph orch osd rm --zap <osd_list> to remove the OSDs. Use the command ceph orch osd status to check the status of OSD removal. Warning Do not proceed with the step until this command returns no results. Use the command ceph orch host drain <HOST> to drain any remaining daemons. Use the command ceph orch host rm <HOST> to remove the host. Note This node is no longer used by the Ceph cluster but is still managed by director as a bare-metal node. End the Ceph shell session. Note If scaling down the Ceph cluster is temporary and the nodes removed will be restored later, the scaling up action can increment the count and set provisioned: true on nodes that were previously set provisioned: false . If the node will never reused, it can be set provisioned: false indefinitely and the scaling up action can specify a new instances entry. The following file sample provides some examples of each instance. To remove the node from director, see Scaling down bare-metal nodes in Installing and managing Red Hat OpenStack Platform with director .

Commands:
source ~/stackrc
- name: CephStorage count: 3 instances: - hostname: ceph-0 name: ceph-0 - hostname: ceph-1 name: ceph-2 - hostname: ceph-2 name: ceph-2
- name: CephStorage count: 6 instances: - hostname: ceph-0 name: ceph-0 - hostname: ceph-1 name: ceph-2 - hostname: ceph-2 name: ceph-2 - hostname: ceph-3 name: ceph-3 - hostname: ceph-4 name: ceph-4 - hostname: ceph-5 name: ceph-5
openstack overcloud node provision --stack overcloud --network-config --output ~/overcloud-baremetal-deployed.yaml ~/overcloud-baremetal-deploy.yaml
openstack overcloud ceph spec ~/overcloud-baremetal-deployed.yaml --osd-spec osd_spec.yaml --roles-data roles_data.yaml -o ceph_spec.yaml
openstack overcloud ceph user enable ceph_spec.yaml
cephadm shell -m <spec_file_path>
ceph orch apply -i /mnt/ceph_spec.yaml
source ~/stackrc
openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm.yaml -e deployed_ceph.yaml -e overcloud-baremetal-deploy.yaml
ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.58557 root default -7 0.19519 host ceph-2 5 hdd 0.04880 osd.5 up 1.00000 1.00000 7 hdd 0.04880 osd.7 up 1.00000 1.00000 9 hdd 0.04880 osd.9 up 1.00000 1.00000 11 hdd 0.04880 osd.11 up 1.00000 1.00000
sudo cephadm shell -- ceph orch ls --export > spec.yaml
sudo cephadm shell -m spec.yaml -- ceph orch apply -i /mnt/spec.yaml
ceph orch osd rm --zap 5 7 9 11 Scheduled OSD(s) for removal ceph orch osd rm status OSD_ID HOST STATE PG_COUNT REPLACE FORCE DRAIN_STARTED_AT 7 ceph-2 draining 27 False False 2021-04-23 21:35:51.215361 9 ceph-2 draining 8 False False 2021-04-23 21:35:49.111500 11 ceph-2 draining 14 False False 2021-04-23 21:35:50.243762
ceph orch osd rm status OSD_ID HOST STATE PG_COUNT REPLACE FORCE DRAIN_STARTED_AT 7 ceph-2 draining 34 False False 2021-04-23 21:35:51.215361 11 ceph-2 draining 14 False False 2021-04-23 21:35:50.243762
ceph orch host drain ceph-2
ceph orch host rm ceph-2
- name: Compute count: 2 instances: - hostname: overcloud-compute-0 name: node10 # Removed from deployment due to disk failure provisioned: false - hostname: overcloud-compute-1 name: node11 - hostname: overcloud-compute-2 name: node12