Chapter 3. Upgrade a Red Hat Ceph Storage cluster using cephadm As a storage administrator, you can use the cephadm Orchestrator to upgrade Red Hat Ceph Storage 5.0 and later. The automated upgrade process follows Ceph best practices. For example: The upgrade order starts with Ceph Managers, Ceph Monitors, then other daemons. Each daemon is restarted only after Ceph indicates that the cluster will remain available. The storage cluster health status is likely to switch to HEALTH_WARNING during the upgrade. When the upgrade is complete, the health status should switch back to HEALTH_OK. Upgrading directly from Red Hat Ceph Storage 5 to Red Hat Ceph Storage 7 is supported. Warning Upgrading to Red Hat Ceph Storage 5.2 on Ceph Object Gateway storage clusters (single-site or multi-site) is supported but you must set the ceph config set mgr mgr/cephadm/no_five_one_rgw true --force option prior to upgrading your storage cluster. For more information, see the knowledge base article Support Restrictions for upgrades for RADOS Gateway (RGW) on Red Hat Red Hat Ceph Storage 5.2 and Known issues section in the Red Hat Ceph Storage 5.2 Release Notes. Note ceph-ansible is currently not supported with Red Hat Ceph Storage 5. This means that once you have migrated your storage cluster to Red Hat Ceph Storage 5, you must use cephadm and cephadm-ansible to perform subsequent updates. Important Red Hat Enterprise Linux 9 and later does not support the cephadm-ansible playbook. Note You do not get a message once the upgrade is successful. Run ceph versions and ceph orch ps commands to verify the new image ID and the version of the storage cluster. 3.1. Upgrading the Red Hat Ceph Storage cluster You can use ceph orch upgrade command for upgrading a Red Hat Ceph Storage 5.0 cluster. Prerequisites A running Red Hat Ceph Storage cluster 5. Red Hat Enterprise Linux 8.4 EUS or later. Root-level access to all the nodes. Ansible user with sudo and passwordless ssh access to all nodes in the storage cluster. At least two Ceph Manager nodes in the storage cluster: one active and one standby. Note Red Hat Ceph Storage 5 also includes a health check function that returns a DAEMON_OLD_VERSION warning if it detects that any of the daemons in the storage cluster are running multiple versions of RHCS. The warning is triggered when the daemons continue to run multiple versions of Red Hat Ceph Storage beyond the time value set in the mon_warn_older_version_delay option. By default, the mon_warn_older_version_delay option is set to 1 week. This setting allows most upgrades to proceed without falsely seeing the warning. If the upgrade process is paused for an extended time period, you can mute the health warning: After the upgrade has finished, unmute the health warning: Procedure Update the cephadm and cephadm-ansible package: Example Navigate to the /usr/share/cephadm-ansible/ directory: Example If the buckets are created or have the num_shards = 0 , manually reshard the buckets, before planning an upgrade to Red Hat Ceph Storage 5.3: Syntax Example Run the preflight playbook with the upgrade_ceph_packages parameter set to true on the bootstrapped host in the storage cluster: Syntax Example This package upgrades cephadm on all the nodes. Log into the cephadm shell: Example Ensure all the hosts are online and that the storage cluster is healthy: Example Set the OSD noout , noscrub , and nodeep-scrub flags to prevent OSDs from getting marked out during upgrade and to avoid unnecessary load on the cluster: Example Check service versions and the available target containers: Syntax Example Note The image name is applicable for both Red Hat Enterprise Linux 8 and Red Hat Enterprise Linux 9. Upgrade the storage cluster: Syntax Example Note To perform a staggered upgrade, see Performing a staggered upgrade . While the upgrade is underway, a progress bar appears in the ceph status output. Example Verify the new IMAGE_ID and VERSION of the Ceph cluster: Example Note If you are not using the cephadm-ansible playbooks, after upgrading your Ceph cluster, you must upgrade the ceph-common package and client libraries on your client nodes. Example Verify you have the latest version: Example When the upgrade is complete, unset the noout , noscrub , and nodeep-scrub flags: Example 3.2. Upgrading the Red Hat Ceph Storage cluster in a disconnected environment You can upgrade the storage cluster in a disconnected environment by using the --image tag. You can use ceph orch upgrade command for upgrading a Red Hat Ceph Storage 5 cluster. Important Red Hat Enterprise Linux 9 and later does not support the cephadm-ansible playbook. Prerequisites A running Red Hat Ceph Storage cluster 5. Red Hat Enterprise Linux 8.4 EUS or later. Root-level access to all the nodes. Ansible user with sudo and passwordless ssh access to all nodes in the storage cluster. At least two Ceph Manager nodes in the storage cluster: one active and one standby. Register the nodes to CDN and attach subscriptions. Check for the customer container images in a disconnected environment and change the configuration, if required. See the Changing configurations of custom container images for disconnected installations section in the Red Hat Ceph Storage Installation Guide for more details. By default, the monitoring stack components are deployed based on the primary Ceph image. For disconnected environment of the storage cluster, you have to use the latest available monitoring stack component images. Table 3.1. Custom image details for monitoring stack Monitoring stack component Red Hat Ceph Storage version Image details Prometheus Red Hat Ceph Storage 5.0 and 5.1 registry.redhat.io/openshift4/ose-prometheus:v4.6 Red Hat Ceph Storage 5.2 onwards registry.redhat.io/openshift4/ose-prometheus:v4.10 Grafana All Red Hat Ceph Storage 5 versions registry.redhat.io/rhceph/rhceph-5-dashboard-rhel8:latest Node-exporter Red Hat Ceph Storage 5.0 registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.5 Red Hat Ceph Storage 5.0z1 and 5.1 registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.6 Red Hat Ceph Storage 5.2 onwards registry.redhat.io/openshift4/ose-prometheus-node-exporter:v4.10 AlertManager Red Hat Ceph Storage 5.0 registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.5 Red Hat Ceph Storage 5.0z1 and 5.1 registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.6 Red Hat Ceph Storage 5.2 onwards registry.redhat.io/openshift4/ose-prometheus-alertmanager:v4.10 HAProxy Red Hat Ceph Storage 5.1 onwards registry.redhat.io/rhceph/rhceph-haproxy-rhel8:latest Keepalived Red Hat Ceph Storage 5.1 onwards registry.redhat.io/rhceph/keepalived-rhel8:latest SNMP Gateway Red Hat Ceph Storage 5.0 onwards registry.redhat.io/rhceph/snmp-notifier-rhel8:latest Procedure Update the cephadm and cephadm-ansible package. Example Run the preflight playbook with the upgrade_ceph_packages parameter set to true and the ceph_origin parameter set to custom on the bootstrapped host in the storage cluster: Syntax Example This package upgrades cephadm on all the nodes. Log into the cephadm shell: Example Ensure all the hosts are online and that the storage cluster is healthy: Example Set the OSD noout , noscrub , and nodeep-scrub flags to prevent OSDs from getting marked out during upgrade and to avoid unnecessary load on the cluster: Example Check service versions and the available target containers: Syntax Example Upgrade the storage cluster: Syntax Example While the upgrade is underway, a progress bar appears in the ceph status output. Example Verify the new IMAGE_ID and VERSION of the Ceph cluster: Example When the upgrade is complete, unset the noout , noscrub , and nodeep-scrub flags: Example Additional Resources See the Registering Red Hat Ceph Storage nodes to the CDN and attaching subscriptions section in the Red Hat Ceph Storage Installation Guide . See the Configuring a private registry for a disconnected installation section in the Red Hat Ceph Storage Installation Guide . 3.3. Staggered upgrade As a storage administrator, you can upgrade Red Hat Ceph Storage components in phases rather than all at once. Starting with Red Hat Ceph Storage 5.2, the ceph orch upgrade command enables you to specify options to limit which daemons are upgraded by a single upgrade command. Note If you want to upgrade from a version that does not support staggered upgrades, you must first manually upgrade the Ceph Manager ( ceph-mgr ) daemons. For more information on performing a staggered upgrade from releases, see Performing a staggered upgrade from releases . 3.3.1. Staggered upgrade options Starting with Red Hat Ceph Storage 5.2, the ceph orch upgrade command supports several options to upgrade cluster components in phases. The staggered upgrade options include: --daemon_types : The --daemon_types option takes a comma-separated list of daemon types and will only upgrade daemons of those types. Valid daemon types for this option include mgr , mon , crash , osd , mds , rgw , rbd-mirror , cephfs-mirror , iscsi , and nfs . --services : The --services option is mutually exclusive with --daemon-types , only takes services of one type at a time, and will only upgrade daemons belonging to those services. For example, you cannot provide an OSD and RGW service simultaneously. --hosts : You can combine the --hosts option with --daemon_types , --services , or use it on its own. The --hosts option parameter follows the same format as the command line options for orchestrator CLI placement specification. --limit : The --limit option takes an integer greater than zero and provides a numerical limit on the number of daemons cephadm will upgrade. You can combine the --limit option with --daemon_types , --services , or --hosts . For example, if you specify to upgrade daemons of type osd on host01 with a limit set to 3 , cephadm will upgrade up to three OSD daemons on host01. 3.3.2. Performing a staggered upgrade As a storage administrator, you can use the ceph orch upgrade options to limit which daemons are upgraded by a single upgrade command. Cephadm strictly enforces an order for the upgrade of daemons that is still present in staggered upgrade scenarios. The current upgrade order is: Ceph Manager nodes Ceph Monitor nodes Ceph-crash daemons Ceph OSD nodes Ceph Metadata Server (MDS) nodes Ceph Object Gateway (RGW) nodes Ceph RBD-mirror node CephFS-mirror node Ceph iSCSI gateway node Ceph NFS nodes Note If you specify parameters that upgrade daemons out of order, the upgrade command blocks and notes which daemons you need to upgrade before you proceed. Example Prerequisites A cluster running Red Hat Ceph Storage 5.2 or later. Root-level access to all the nodes. At least two Ceph Manager nodes in the storage cluster: one active and one standby. Procedure Log into the cephadm shell: Example Ensure all the hosts are online and that the storage cluster is healthy: Example Set the OSD noout , noscrub , and nodeep-scrub flags to prevent OSDs from getting marked out during upgrade and to avoid unnecessary load on the cluster: Example Check service versions and the available target containers: Syntax Example Upgrade the storage cluster: To upgrade specific daemon types on specific hosts: Syntax Example To specify specific services and limit the number of daemons to upgrade: Syntax Example Note In staggered upgrade scenarios, if using a limiting parameter, the monitoring stack daemons, including Prometheus and node-exporter , are refreshed after the upgrade of the Ceph Manager daemons. As a result of the limiting parameter, Ceph Manager upgrades take longer to complete. The versions of monitoring stack daemons might not change between Ceph releases, in which case, they are only redeployed. Note Upgrade commands with limiting parameters validates the options before beginning the upgrade, which can require pulling the new container image. As a result, the upgrade start command might take a while to return when you provide limiting parameters. To see which daemons you still need to upgrade, run the ceph orch upgrade check or ceph versions command: Example To complete the staggered upgrade, verify the upgrade of all remaining services: Syntax Example When the upgrade is complete, unset the noout , noscrub , and nodeep-scrub flags: Example Verification Verify the new IMAGE_ID and VERSION of the Ceph cluster: Example Additional Resources For more information about upgrading a Red Hat Ceph Storage cluster using cephadm , see Upgrade a Red Hat Ceph Storage cluster using cephadm . 3.3.3. Performing a staggered upgrade from releases Starting with Red Hat Ceph Storage 5.2, you can perform a staggered upgrade on your storage cluster by providing the necessary arguments. If you want to upgrade from a version that does not support staggered upgrades, you must first manually upgrade the Ceph Manager ( ceph-mgr ) daemons. Once you have upgraded the Ceph Manager daemons, you can pass the limiting parameters to complete the staggered upgrade. Important Verify you have at least two running Ceph Manager daemons before attempting this procedure. Prerequisites A cluster running Red Hat Ceph Storage 5.0 or later. At least two Ceph Manager nodes in the storage cluster: one active and one standby. Procedure Log into the Cephadm shell: Example Determine which Ceph Manager is active and which are standby: Example Manually upgrade each standby Ceph Manager daemon: Syntax Example Fail over to the upgraded standby Ceph Manager: Example Check that the standby Ceph Manager is now active: Example Verify that the active Ceph Manager is upgraded to the new version: Syntax Example Repeat steps 2 - 6 to upgrade the remaining Ceph Managers to the new version. Check that all Ceph Managers are upgraded to the new version: Example Once you upgrade all your Ceph Managers, you can specify the limiting parameters and complete the remainder of the staggered upgrade. Additional Resources For more information about performing a staggered upgrade and staggered upgrade options, see Performing a staggered upgrade . 3.4. Monitoring and managing upgrade of the storage cluster After running the ceph orch upgrade start command to upgrade the Red Hat Ceph Storage cluster, you can check the status, pause, resume, or stop the upgrade process. The health of the cluster changes to HEALTH_WARNING during an upgrade. If the host of the cluster is offline, the upgrade is paused. Note You have to upgrade one daemon type after the other. If a daemon cannot be upgraded, the upgrade is paused. Prerequisites A running Red Hat Ceph Storage cluster 5. Root-level access to all the nodes. At least two Ceph Manager nodes in the storage cluster: one active and one standby. Upgrade for the storage cluster initiated. Procedure Determine whether an upgrade is in process and the version to which the cluster is upgrading: Example Note You do not get a message once the upgrade is successful. Run ceph versions and ceph orch ps commands to verify the new image ID and the version of the storage cluster. Optional: Pause the upgrade process: Example Optional: Resume a paused upgrade process: Example Optional: Stop the upgrade process: Example 3.5. Troubleshooting upgrade error messages The following table shows some cephadm upgrade error messages. If the cephadm upgrade fails for any reason, an error message appears in the storage cluster health status. Error Message Description UPGRADE_NO_STANDBY_MGR Ceph requires both active and standby manager daemons to proceed, but there is currently no standby. UPGRADE_FAILED_PULL Ceph was unable to pull the container image for the target version. This can happen if you specify a version or container image that does not exist (e.g., 1.2.3), or if the container registry is not reachable from one or more hosts in the cluster.

Commands:
ceph health mute DAEMON_OLD_VERSION --sticky
ceph health unmute DAEMON_OLD_VERSION
dnf update cephadm dnf update cephadm-ansible
cd /usr/share/cephadm-ansible
radosgw-admin bucket reshard --num-shards 11 --bucket BUCKET_NAME
radosgw-admin bucket reshard --num-shards 11 --bucket mybucket
ansible-playbook -i INVENTORY_FILE cephadm-preflight.yml --extra-vars "ceph_origin=rhcs upgrade_ceph_packages=true"
[ceph-admin@admin cephdm-ansible]USD ansible-playbook -i /etc/ansible/hosts cephadm-preflight.yml --extra-vars "ceph_origin=rhcs upgrade_ceph_packages=true"
cephadm shell
ceph -s
ceph osd set noout ceph osd set noscrub ceph osd set nodeep-scrub
ceph orch upgrade check IMAGE_NAME
ceph orch upgrade check registry.redhat.io/rhceph/rhceph-5-rhel8:latest
ceph orch upgrade start IMAGE_NAME
ceph orch upgrade start registry.redhat.io/rhceph/rhceph-5-rhel8:latest
ceph status [...] progress: Upgrade to 16.2.0-146.el8cp (1s) [............................]
ceph versions ceph orch ps
[root@client01 ~] dnf update ceph-common
[root@client01 ~] ceph --version
ceph osd unset noout ceph osd unset noscrub ceph osd unset nodeep-scrub
dnf update cephadm dnf update cephadm-ansible
ansible-playbook -i INVENTORY_FILE cephadm-preflight.yml --extra-vars "ceph_origin=custom upgrade_ceph_packages=true"
[ceph-admin@admin ~]USD ansible-playbook -i /etc/ansible/hosts cephadm-preflight.yml --extra-vars "ceph_origin=custom upgrade_ceph_packages=true"
cephadm shell
ceph -s
ceph osd set noout ceph osd set noscrub ceph osd set nodeep-scrub
ceph orch upgrade check IMAGE_NAME
ceph orch upgrade check LOCAL_NODE_FQDN :5000/rhceph/rhceph-5-rhel8
ceph orch upgrade start IMAGE_NAME
ceph orch upgrade start LOCAL_NODE_FQDN :5000/rhceph/rhceph-5-rhel8
ceph status [...] progress: Upgrade to 16.2.0-115.el8cp (1s) [............................]
ceph version ceph versions ceph orch ps
ceph osd unset noout ceph osd unset noscrub ceph osd unset nodeep-scrub
ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-5-rhel8:latest --hosts host02 Error EINVAL: Cannot start upgrade. Daemons with types earlier in upgrade order than daemons on given host need upgrading. Please first upgrade mon.ceph-host01 NOTE: Enforced upgrade order is: mgr -> mon -> crash -> osd -> mds -> rgw -> rbd-mirror -> cephfs-mirror -> iscsi -> nfs
cephadm shell
ceph -s
ceph osd set noout ceph osd set noscrub ceph osd set nodeep-scrub
ceph orch upgrade check IMAGE_NAME
ceph orch upgrade check registry.redhat.io/rhceph/rhceph-5-rhel8:latest
ceph orch upgrade start --image IMAGE_NAME --daemon-types DAEMON_TYPE1 , DAEMON_TYPE2 --hosts HOST1 , HOST2
ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-5-rhel8:latest --daemon-types mgr,mon --hosts host02,host03
ceph orch upgrade start --image IMAGE_NAME --services SERVICE1 , SERVICE2 --limit LIMIT_NUMBER
ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-5-rhel8:latest --services rgw.example1,rgw1.example2 --limit 2
ceph orch upgrade check --image registry.redhat.io/rhceph/rhceph-5-rhel8:latest
ceph orch upgrade start --image IMAGE_NAME
ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-5-rhel8:latest
ceph osd unset noout ceph osd unset noscrub ceph osd unset nodeep-scrub
ceph versions ceph orch ps
cephadm shell
ceph -s cluster: id: 266ee7a8-2a05-11eb-b846-5254002d4916 health: HEALTH_OK services: mon: 2 daemons, quorum host01,host02 (age 92s) mgr: host01.ndtpjh(active, since 16h), standbys: host02.pzgrhz
ceph orch daemon redeploy mgr.ceph- HOST . MANAGER_ID --image IMAGE_ID
ceph orch daemon redeploy mgr.ceph-host02.pzgrhz --image registry.redhat.io/rhceph/rhceph-5-rhel8:latest
ceph mgr fail
ceph -s cluster: id: 266ee7a8-2a05-11eb-b846-5254002d4916 health: HEALTH_OK services: mon: 2 daemons, quorum host01,host02 (age 1h) mgr: host02.pzgrhz(active, since 25s), standbys: host01.ndtpjh
ceph tell mgr.ceph- HOST . MANAGER_ID version
ceph tell mgr.host02.pzgrhz version { "version": "16.2.8-12.el8cp", "release": "pacific", "release_type": "stable" }
ceph mgr versions { "ceph version 16.2.8-12.el8cp (600e227816517e2da53d85f2fab3cd40a7483372) pacific (stable)": 2 }
ceph orch upgrade status
ceph orch upgrade pause
ceph orch upgrade resume
ceph orch upgrade stop