Chapter 4. Staggered upgrade As a storage administrator, you can upgrade Red Hat Ceph Storage components in phases rather than all at once. The ceph orch upgrade command enables you to specify options to limit which daemons are upgraded by a single upgrade command. Note If you want to upgrade from a version that does not support staggered upgrades, you must first manually upgrade the Ceph Manager ( ceph-mgr ) daemons. For more information on performing a staggered upgrade from releases, see Performing a staggered upgrade from releases . 4.1. Staggered upgrade options The ceph orch upgrade command supports several options to upgrade cluster components in phases. The staggered upgrade options include: --daemon_types : The --daemon_types option takes a comma-separated list of daemon types and will only upgrade daemons of those types. Valid daemon types for this option include mgr , mon , crash , osd , mds , rgw , rbd-mirror , cephfs-mirror , and nfs . --services : The --services option is mutually exclusive with --daemon-types , only takes services of one type at a time, and will only upgrade daemons belonging to those services. For example, you cannot provide an OSD and RGW service simultaneously. --hosts : You can combine the --hosts option with --daemon_types , --services , or use it on its own. The --hosts option parameter follows the same format as the command line options for orchestrator CLI placement specification. --limit : The --limit option takes an integer greater than zero and provides a numerical limit on the number of daemons cephadm will upgrade. You can combine the --limit option with --daemon_types , --services , or --hosts . For example, if you specify to upgrade daemons of type osd on host01 with a limit set to 3 , cephadm will upgrade up to three OSD daemons on host01. 4.1.1. Performing a staggered upgrade As a storage administrator, you can use the ceph orch upgrade options to limit which daemons are upgraded by a single upgrade command. Cephadm strictly enforces an order for the upgrade of daemons that is still present in staggered upgrade scenarios. The current upgrade order is: Ceph Manager nodes Ceph Monitor nodes Ceph-crash daemons Ceph OSD nodes Ceph Metadata Server (MDS) nodes Ceph Object Gateway (RGW) nodes Ceph RBD-mirror node CephFS-mirror node Ceph NFS nodes Note If you specify parameters that upgrade daemons out of order, the upgrade command blocks and notes which daemons you need to upgrade before you proceed. Example Note There is no required order for restarting the instances. Red Hat recommends restarting the instance pointing to the pool with primary images followed by the instance pointing to the mirrored pool. Prerequisites A cluster running Red Hat Ceph Storage 5.3 or 6.1. Root-level access to all the nodes. At least two Ceph Manager nodes in the storage cluster: one active and one standby. Procedure Log into the cephadm shell: Example Ensure all the hosts are online and that the storage cluster is healthy: Example Set the OSD noout , noscrub , and nodeep-scrub flags to prevent OSDs from getting marked out during upgrade and to avoid unnecessary load on the cluster: Example Check service versions and the available target containers: Syntax Example Upgrade the storage cluster: To upgrade specific daemon types on specific hosts: Syntax Example To specify specific services and limit the number of daemons to upgrade: Syntax Example Note In staggered upgrade scenarios, if using a limiting parameter, the monitoring stack daemons, including Prometheus and node-exporter , are refreshed after the upgrade of the Ceph Manager daemons. As a result of the limiting parameter, Ceph Manager upgrades take longer to complete. The versions of monitoring stack daemons might not change between Ceph releases, in which case, they are only redeployed. Note Upgrade commands with limiting parameters validates the options before beginning the upgrade, which can require pulling the new container image. As a result, the upgrade start command might take a while to return when you provide limiting parameters. To see which daemons you still need to upgrade, run the ceph orch upgrade check or ceph versions command: Example To complete the staggered upgrade, verify the upgrade of all remaining services: Syntax Example Verification Verify the new IMAGE_ID and VERSION of the Ceph cluster: Example When the upgrade is complete, unset the noout , noscrub , and nodeep-scrub flags: Example 4.1.2. Performing a staggered upgrade from releases You can perform a staggered upgrade on your storage cluster by providing the necessary arguments You can perform a staggered upgrade on your storage cluster by providing the necessary arguments. If you want to upgrade from a version that does not support staggered upgrades, you must first manually upgrade the Ceph Manager ( ceph-mgr ) daemons. Once you have upgraded the Ceph Manager daemons, you can pass the limiting parameters to complete the staggered upgrade. Important Verify you have at least two running Ceph Manager daemons before attempting this procedure. Prerequisites A cluster running Red Hat Ceph Storage 5.2 or lesser. At least two Ceph Manager nodes in the storage cluster: one active and one standby. Procedure Log into the Cephadm shell: Example Determine which Ceph Manager is active and which are standby: Example Manually upgrade each standby Ceph Manager daemon: Syntax Example Fail over to the upgraded standby Ceph Manager: Example Check that the standby Ceph Manager is now active: Example Verify that the active Ceph Manager is upgraded to the new version: Syntax Example Repeat steps 2 - 6 to upgrade the remaining Ceph Managers to the new version. Check that all Ceph Managers are upgraded to the new version: Example Once you upgrade all your Ceph Managers, you can specify the limiting parameters and complete the remainder of the staggered upgrade. Additional Resources For more information about performing a staggered upgrade and staggered upgrade options, see Performing a staggered upgrade .

Commands:
ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-7-rhel9:latest --hosts host02 Error EINVAL: Cannot start upgrade. Daemons with types earlier in upgrade order than daemons on given host need upgrading. Please first upgrade mon.ceph-host01
cephadm shell
ceph -s
ceph osd set noout ceph osd set noscrub ceph osd set nodeep-scrub
ceph orch upgrade check IMAGE_NAME
ceph orch upgrade check registry.redhat.io/rhceph/rhceph-7-rhel9:latest
ceph orch upgrade start --image IMAGE_NAME --daemon-types DAEMON_TYPE1 , DAEMON_TYPE2 --hosts HOST1 , HOST2
ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-7-rhel9:latest --daemon-types mgr,mon --hosts host02,host03
ceph orch upgrade start --image IMAGE_NAME --services SERVICE1 , SERVICE2 --limit LIMIT_NUMBER
ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-7-rhel9:latest --services rgw.example1,rgw1.example2 --limit 2
ceph orch upgrade check --image registry.redhat.io/rhceph/rhceph-7-rhel9:latest
ceph orch upgrade start --image IMAGE_NAME
ceph orch upgrade start --image registry.redhat.io/rhceph/rhceph-7-rhel9:latest
ceph versions ceph orch ps
ceph osd unset noout ceph osd unset noscrub ceph osd unset nodeep-scrub
cephadm shell
ceph -s cluster: id: 266ee7a8-2a05-11eb-b846-5254002d4916 health: HEALTH_OK services: mon: 2 daemons, quorum host01,host02 (age 92s) mgr: host01.ndtpjh(active, since 16h), standbys: host02.pzgrhz
ceph orch daemon redeploy mgr.ceph- HOST . MANAGER_ID --image IMAGE_ID
ceph orch daemon redeploy mgr.ceph-host02.pzgrhz --image registry.redhat.io/rhceph/rhceph-7-rhel9:latest
ceph mgr fail
ceph -s cluster: id: 266ee7a8-2a05-11eb-b846-5254002d4916 health: HEALTH_OK services: mon: 2 daemons, quorum host01,host02 (age 1h) mgr: host02.pzgrhz(active, since 25s), standbys: host01.ndtpjh
ceph tell mgr.ceph- HOST . MANAGER_ID version
ceph tell mgr.host02.pzgrhz version { "version": "18.2.0-128.el8cp", "release": "reef", "release_type": "stable" }
ceph mgr versions { "ceph version 18.2.0-128.el8cp (600e227816517e2da53d85f2fab3cd40a7483372) pacific (stable)": 2 }