Chapter 6. Upgrading an overcloud with director-deployed Ceph deployments If your environment includes director-deployed Red Hat Ceph Storage deployments with or without hyperconverged infrastructure (HCI) nodes, you must upgrade your deployments to Red Hat Ceph Storage 5. With an upgrade to version 5, cephadm now manages Red Hat Ceph Storage instead of ceph-ansible . Note If you are using the Red Hat Ceph Storage Object Gateway (RGW), ensure that all RGW pools have the application label rgw as described in Why are the RGW services crashing after running the cephadm adoption playbook? . Implementing this configuration change addresses a common issue encountered when upgrading from Red Hat Ceph Storage Release 4 to 5. 6.1. Installing ceph-ansible If you deployed Red Hat Ceph Storage using director, you must complete this procedure. The ceph-ansible package is required to upgrade Red Hat Ceph Storage with Red Hat OpenStack Platform. Procedure Enable the Ceph 5 Tools repository: Install the ceph-ansible package: 6.2. Downloading Red Hat Ceph Storage containers to the undercloud from Satellite If the Red Hat Ceph Storage container image is hosted on a Red Hat Satellite Server, then you must download a copy of the image to the undercloud before starting the Red Hat Ceph Storage upgrade using Red Hat Satellite. Prerequisite The required Red Hat Ceph Storage container image is hosted on the Satellite Server. Procedure Log in to the undercloud node as the stack user. Download the Red Hat Ceph Storage container image from the Satellite Server: Replace <ceph_image_file> with the Red Hat Ceph Storage container image file hosted on the Satellite Server. The following is an example of this command: 6.3. Upgrading to Red Hat Ceph Storage 5 Upgrade the following nodes from Red Hat Ceph Storage version 4 to version 5: Red Hat Ceph Storage nodes Hyperconverged infrastructure (HCI) nodes, which contain combined Compute and Ceph OSD services For information about the duration and impact of this upgrade procedure, see Upgrade duration and impact . Note Red Hat Ceph Storage 5 uses Prometheus v4.10, which has the following known issue: If you enable Red Hat Ceph Storage dashboard, two data sources are configured on the dashboard. For more information about this known issue, see BZ#2054852 . Red Hat Ceph Storage 6 uses Prometheus v4.12, which does not include this known issue. Red Hat recommends upgrading from Red Hat Ceph Storage 5 to Red Hat Ceph Storage 6 after the upgrade from Red Hat OpenStack Platform (RHOSP) 16.2 to 17.1 is complete. To upgrade from Red Hat Ceph Storage version 5 to version 6, begin with one of the following procedures for your environment: Director-deployed Red Hat Ceph Storage environments: Updating the cephadm client External Red Hat Ceph Storage cluster environments: Updating the Red Hat Ceph Storage container image Procedure Log in to the undercloud host as the stack user. Source the stackrc undercloud credentials file: Run the Red Hat Ceph Storage external upgrade process with the ceph tag: Replace <stack> with the name of your stack. If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter. Run the ceph versions command to confirm all Red Hat Ceph Storage daemons have been upgraded to version 5. This command is available in the ceph monitor container that is hosted by default on the Controller node. Important The command in the step runs the ceph-ansible rolling_update.yaml playbook to update the cluster from version 4 to 5. It is important to confirm all daemons have been updated before proceeding with this procedure. The following example demonstrates the use and output of this command. As demonstrated in the example, all daemons in your deployment should show a package version of 16.2.* and the keyword pacific . Note The output of the command sudo podman ps | grep ceph on any server hosting Red Hat Ceph Storage should return a version 5 container. Create the ceph-admin user and distribute the appropriate keyrings: Update the packages on the Red Hat Ceph Storage nodes: If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter. Note By default, the Ceph Monitor service (CephMon) runs on the Controller nodes unless you have used the composable roles feature to host them elsewhere. This command includes the ceph_mon tag, which also updates the packages on the nodes hosting the Ceph Monitor service (the Controller nodes by default). Configure the Red Hat Ceph Storage nodes to use cephadm : If you are running this command at a DCN deployed site, add the value skip-tag cleanup_cephansible to the provided comma-separated list of values for the --skip-tags parameter. Run the ceph -s command to confirm all processes are now managed by Red Hat Ceph Storage orchestrator. This command is available in the ceph monitor container that is hosted by default on the Controller node. Important The command in the step runs the ceph-ansible cephadm-adopt.yaml playbook to move future management of the cluster from ceph-ansible to cephadm and the Red Hat Ceph Storage orchestrator. It is important to confirm all processes are now managed by the orcestrator before proceeding with this procedure. The following example demonstrates the use and output of this command. As demonstrated in this example, there are 63 daemons that are not managed by cephadm . This indicates there was a problem with the running of the ceph-ansible cephadm-adopt.yml playbook. Contact Red Hat Ceph Storage support to troubleshoot these errors before proceeding with the upgrade. When the adoption process has been completed successfully, there should not be any warning about stray daemons not managed by cephadm . Modify the overcloud_upgrade_prepare.sh file to replace the ceph-ansible file with a cephadm heat environment file: Important Do not include ceph-ansible environment or deployment files, for example, environments/ceph-ansible/ceph-ansible.yaml or deployment/ceph-ansible/ceph-grafana.yaml , in openstack deployment commands such as openstack overcloud upgrade prepare and openstack overcloud deploy . For more information about replacing ceph-ansible environment and deployment files with cephadm files, see Implications of upgrading to Red Hat Ceph Storage 5 . Note This example uses the environments/cephadm/cephadm-rbd-only.yaml file because RGW is not deployed. If you plan to deploy RGW, use environments/cephadm/cephadm.yaml after you finish upgrading your RHOSP environment, and then run a stack update. Modify the overcloud_upgrade_prepare.sh file to remove the following environment file if you added it earlier when you ran the overcloud upgrade preparation: Save the file. Run the upgrade preparation command: If your deployment includes HCI nodes, create a temporary hci.conf file in a cephadm container of a Controller node: Log in to a Controller node: Replace <controller_ip> with the IP address of the Controller node. Retrieve a cephadm shell from the Controller node: Example In the cephadm shell, create a temporary hci.conf file: Example Apply the configuration: Example For more information about adjusting the configuration of your HCI deployment, see Ceph configuration overrides for HCI in Deploying a hyperconverged infrastructure . Important You must upgrade the operating system on all HCI nodes to RHEL 9. For more information on upgrading Compute and HCI nodes, see Upgrading Compute nodes to RHEL 9.2 . Important If Red Hat Ceph Storage Rados Gateway (RGW) is used for object storage, complete the steps in Ceph config overrides set for the RGWs on the RHCS 4.x does not get reflected after the Upgrade to RHCS 5.x to ensure your Red Hat Ceph Storage 4 configuration is reflected completely in Red Hat Ceph Storage 5. Important If the Red Hat Ceph Storage Dashboard is installed, complete the steps in After FFU 16.2 to 17.1, Ceph Grafana dashboard failed to start due to incorrect dashboard configuration to ensure it is properly configured. 6.4. Implications of upgrading to Red Hat Ceph Storage 5 The Red Hat Ceph Storage cluster is now upgraded to version 5. This has the following implications: You no longer use ceph-ansible to manage Red Hat Ceph Storage. Instead, the Ceph Orchestrator manages the Red Hat Ceph Storage cluster. For more information about the Ceph Orchestrator, see The Ceph Operations Guide . You no longer need to perform stack updates to make changes to the Red Hat Ceph Storage cluster in most cases. Instead, you can run day two Red Hat Ceph Storage operations directly on the cluster as described in The Ceph Operations Guide . You can also scale Red Hat Ceph Storage cluster nodes up or down as described in Scaling the Ceph Storage cluster in Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director . You can inspect the Red Hat Ceph Storage cluster's health. For more information about monitoring your cluster's health, see Monitoring Red Hat Ceph Storage nodes in Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director . Do not include environment files or deployment files, for example, environments/ceph-ansible/ceph-ansible.yaml or deployment/ceph-ansible/ceph-grafana.yaml , in openstack deployment commands such as openstack overcloud upgrade prepare and openstack overcloud deploy . If your deployment includes ceph-ansible environment or deployment files, replace them with one of the following options: Red Hat Ceph Storage deployment Original ceph-ansible file Cephadm file replacement Ceph RADOS Block Device (RBD) only environments/ceph-ansible/ceph-ansible.yaml environments/cephadm/cephadm-rbd-only.yaml RBD and the Ceph Object Gateway (RGW) environments/ceph-ansible/ceph-rgw.yaml environments/cephadm/cephadm.yaml Ceph Dashboard environments/ceph-ansible/ceph-dashboard.yaml Respective file in environments/cephadm/ Ceph MDS environments/ceph-ansible/ceph-mds.yaml Respective file in environments/cephadm/

Commands:
[stack@director ~]USD sudo subscription-manager repos --enable=rhceph-5-tools-for-rhel-8-x86_64-rpms
[stack@director ~]USD sudo dnf install -y ceph-ansible
sudo podman pull <ceph_image_file>
sudo podman pull satellite.example.com/container-images-osp-17_1-rhceph-5-rhel8:latest
source ~/stackrc
openstack overcloud external-upgrade run --skip-tags "ceph_ansible_remote_tmp" --stack <stack> --tags ceph,facts 2>&1
sudo podman exec ceph-mon-USD(hostname -f) ceph versions { "mon": { "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3 }, "mgr": { "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3 }, "osd": { "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 180 }, "mds": {}, "rgw": { "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 3 }, "overall": { "ceph version 16.2.10-248.el8cp (0edb63afd9bd3edb333364f2e0031b77e62f4896) pacific (stable)": 189 } }
ANSIBLE_LOG_PATH=/home/stack/cephadm_enable_user_key.log ANSIBLE_HOST_KEY_CHECKING=false ansible-playbook -i /home/stack/overcloud-deploy/<stack>/config-download/<stack>/tripleo-ansible-inventory.yaml -b -e ansible_python_interpreter=/usr/libexec/platform-python /usr/share/ansible/tripleo-playbooks/ceph-admin-user-playbook.yml -e tripleo_admin_user=ceph-admin -e distribute_private_key=true --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd
openstack overcloud upgrade run --stack <stack> --skip-tags ceph_ansible_remote_tmp --tags setup_packages --limit Undercloud,ceph_mon,ceph_mgr,ceph_rgw,ceph_mds,ceph_nfs,ceph_grafana,ceph_osd --playbook /home/stack/overcloud-deploy/<stack>/config-download/<stack>/upgrade_steps_playbook.yaml 2>&1
openstack overcloud external-upgrade run --skip-tags ceph_ansible_remote_tmp --stack <stack> --tags cephadm_adopt 2>&1
sudo cephadm shell -- ceph -s cluster: id: f5a40da5-6d88-4315-9bb3-6b16df51d765 health: HEALTH_WARN 63 stray daemon(s) not managed by cephadm
#!/bin/bash openstack overcloud upgrade prepare --yes --timeout 460 --templates /usr/share/openstack-tripleo-heat-templates --ntp-server 192.168.24.1 --stack <stack> -r /home/stack/roles_data.yaml -e /home/stack/templates/internal.yaml ... -e /usr/share/openstack-tripleo-heat-templates/environments/cephadm/cephadm-rbd-only.yaml -e ~/containers-prepare-parameter.yaml
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/manila-cephfsganesha-config.yaml
source stackrc chmod 755 /home/stack/overcloud_upgrade_prepare.sh sh /home/stack/overcloud_upgrade_prepare.sh
ssh cloud-admin@<controller_ip>
[cloud-admin@controller-0 ~]USD sudo cephadm shell
cat <<EOF > hci.conf [osd] osd_memory_target_autotune = true osd_numa_auto_affinity = true [mgr] mgr/cephadm/autotune_memory_target_ratio = 0.2 EOF ...
ceph config assimilate-conf -i hci.conf