Chapter 4. Rebooting the overcloud After you perform a minor Red Hat OpenStack Platform (RHOSP) update to the latest 16.2 version, reboot your overcloud. The reboot refreshes the nodes with any associated kernel, system-level, and container component updates. These updates provide performance and security benefit. Plan downtime to perform the reboot procedures. Use the following guidance to understand how to reboot different node types: If you reboot all nodes in one role, reboot each node individually. If you reboot all nodes in a role simultaneously, service downtime can occur during the reboot operation. Complete the reboot procedures on the nodes in the following order: Section 4.1, "Rebooting Controller and composable nodes" Section 4.2, "Rebooting a Ceph Storage (OSD) cluster" Section 4.3, "Rebooting Compute nodes" 4.1. Rebooting Controller and composable nodes Reboot Controller nodes and standalone nodes based on composable roles, and exclude Compute nodes and Ceph Storage nodes. Procedure Log in to the node that you want to reboot. Optional: If the node uses Pacemaker resources, stop the cluster: [heat-admin@overcloud-controller-0 ~]USD sudo pcs cluster stop Reboot the node: [heat-admin@overcloud-controller-0 ~]USD sudo reboot Wait until the node boots. Verfication Verify that the services are enabled. If the node uses Pacemaker services, check that the node has rejoined the cluster: [heat-admin@overcloud-controller-0 ~]USD sudo pcs status If the node uses Systemd services, check that all services are enabled: [heat-admin@overcloud-controller-0 ~]USD sudo systemctl status If the node uses containerized services, check that all containers on the node are active: [heat-admin@overcloud-controller-0 ~]USD sudo podman ps 4.2. Rebooting a Ceph Storage (OSD) cluster Complete the following steps to reboot a cluster of Ceph Storage (OSD) nodes. Prerequisites On a Ceph Monitor or Controller node that is running the ceph-mon service, check that the Red Hat Ceph Storage cluster status is healthy and the pg status is active+clean : USD sudo podman exec -it ceph-mon-controller-0 ceph -s If the Ceph cluster is healthy, it returns a status of HEALTH_OK . If the Ceph cluster status is unhealthy, it returns a status of HEALTH_WARN or HEALTH_ERR . For troubleshooting guidance, see the Red Hat Ceph Storage 4 Troubleshooting Guide . Procedure Log in to a Ceph Monitor or Controller node that is running the ceph-mon service, and disable Ceph Storage cluster rebalancing temporarily: USD sudo podman exec -it ceph-mon-controller-0 ceph osd set noout USD sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance Note If you have a multistack or distributed compute node (DCN) architecture, you must specify the cluster name when you set the noout and norebalance flags. For example: sudo podman exec -it ceph-mon-controller-0 ceph osd set noout --cluster <cluster_name> Select the first Ceph Storage node that you want to reboot and log in to the node. Reboot the node: Wait until the node boots. Log into the node and check the cluster status: USD sudo podman exec -it ceph-mon-controller-0 ceph status Check that the pgmap reports all pgs as normal ( active+clean ). Log out of the node, reboot the node, and check its status. Repeat this process until you have rebooted all Ceph Storage nodes. When complete, log in to a Ceph Monitor or Controller node that is running the ceph-mon service and re-enable cluster rebalancing: USD sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout USD sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance Note If you have a multistack or distributed compute node (DCN) architecture, you must specify the cluster name when you unset the noout and norebalance flags. For example: sudo podman exec -it ceph-mon-controller-0 ceph osd set noout --cluster <cluster_name> Perform a final status check to verify that the cluster reports HEALTH_OK : USD sudo podman exec -it ceph-mon-controller-0 ceph status 4.3. Rebooting Compute nodes To ensure minimal downtime of instances in your Red Hat OpenStack Platform environment, the Migrating instances workflow outlines the steps you must complete to migrate instances from the Compute node that you want to reboot. Note If you do not migrate the instances from the source Compute node to another Compute node, the instances might be restarted on the source Compute node, which might cause the upgrade to fail. This is related to the known issue around changes to Podman and the libvirt service: BZ#2009106 - podman panic after tripleo_nova_libvirt restart two times BZ#2010135 - podman panic after tripleo_nova_libvirt restart two times Migrating instances workflow Decide whether to migrate instances to another Compute node before rebooting the node. Select and disable the Compute node that you want to reboot so that it does not provision new instances. Migrate the instances to another Compute node. Reboot the empty Compute node. Enable the empty Compute node. Prerequisites Before you reboot the Compute node, you must decide whether to migrate instances to another Compute node while the node is rebooting. Review the list of migration constraints that you might encounter when you migrate virtual machine instances between Compute nodes. For more information, see Migration constraints in Configuring the Compute Service for Instance Creation . If you cannot migrate the instances, you can set the following core template parameters to control the state of the instances after the Compute node reboots: NovaResumeGuestsStateOnHostBoot Determines whether to return instances to the same state on the Compute node after reboot. When set to False , the instances remain down and you must start them manually. The default value is False . NovaResumeGuestsShutdownTimeout Number of seconds to wait for an instance to shut down before rebooting. It is not recommended to set this value to 0 . The default value is 300 . For more information about overcloud parameters and their usage, see Overcloud Parameters . Procedure Log in to the undercloud as the stack user. List all Compute nodes and their UUIDs: USD source ~/stackrc (undercloud) USD openstack server list --name compute Identify the UUID of the Compute node that you want to reboot. From the undercloud, select a Compute node and disable it: USD source ~/overcloudrc (overcloud) USD openstack compute service list (overcloud) USD openstack compute service set <hostname> nova-compute --disable List all instances on the Compute node: (overcloud) USD openstack server list --host <hostname> --all-projects Optional: If you decide to migrate the instances to another Compute node, complete the following steps: If you decide to migrate the instances to another Compute node, use one of the following commands: To migrate the instance to a different host, run the following command: (overcloud) USD openstack server migrate <instance_id> --live <target_host> --wait Let nova-scheduler automatically select the target host: (overcloud) USD nova live-migration <instance_id> Live migrate all instances at once: USD nova host-evacuate-live <hostname> Note The nova command might cause some deprecation warnings, which are safe to ignore. Wait until migration completes. Confirm that the migration was successful: (overcloud) USD openstack server list --host <hostname> --all-projects Continue to migrate instances until none remain on the Compute node. Log in to the Compute node and reboot the node: [heat-admin@overcloud-compute-0 ~]USD sudo reboot Wait until the node boots. Re-enable the Compute node: USD source ~/overcloudrc (overcloud) USD openstack compute service set <hostname> nova-compute --enable Check that the Compute node is enabled: (overcloud) USD openstack compute service list

Commands:
[heat-admin@overcloud-controller-0 ~]USD sudo pcs cluster stop
[heat-admin@overcloud-controller-0 ~]USD sudo reboot
[heat-admin@overcloud-controller-0 ~]USD sudo pcs status
[heat-admin@overcloud-controller-0 ~]USD sudo systemctl status
[heat-admin@overcloud-controller-0 ~]USD sudo podman ps
sudo podman exec -it ceph-mon-controller-0 ceph -s
sudo podman exec -it ceph-mon-controller-0 ceph osd set noout sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance
sudo reboot
sudo podman exec -it ceph-mon-controller-0 ceph status
sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance
sudo podman exec -it ceph-mon-controller-0 ceph status
source ~/stackrc (undercloud) USD openstack server list --name compute
source ~/overcloudrc (overcloud) USD openstack compute service list (overcloud) USD openstack compute service set <hostname> nova-compute --disable
(overcloud) USD openstack server list --host <hostname> --all-projects
(overcloud) USD openstack server migrate <instance_id> --live <target_host> --wait
(overcloud) USD nova live-migration <instance_id>
nova host-evacuate-live <hostname>
(overcloud) USD openstack server list --host <hostname> --all-projects
[heat-admin@overcloud-compute-0 ~]USD sudo reboot
source ~/overcloudrc (overcloud) USD openstack compute service set <hostname> nova-compute --enable
(overcloud) USD openstack compute service list