Chapter 3. Restoring the undercloud and control plane nodes If your undercloud or control plane nodes become corrupted or if an error occurs during an update or upgrade, you can restore the undercloud or overcloud control plane nodes from a backup to their state. If the restore process fails to automatically restore the Galera cluster or nodes with colocated Ceph monitors, you can restore these components manually. 3.1. Preparing a control plane with colocated Ceph monitors for the restore process Before you restore a control plane nodes with colocated Ceph monitors, prepare your environment by creating a script that mounts the Ceph monitor backup file to the node file system and another script that ReaR uses to locate the backup file. Important If you cannot back up the /var/lib/ceph directory, you must contact the Red Hat Technical Support team to rebuild the ceph-mon index. For more information, see Red Hat Technical Support Team . Prerequisites You have created a backup of the undercloud node. For more information, see Section 1.8, "Creating a backup of the undercloud node" . You have created a backup of the control plane nodes. For more information, see Section 2.6, "Creating a backup of the control plane nodes" . You have access to the backup node. If you use an OVS bridge for your network interfaces, you have access to the network configuration information that you set in the NETWORKING_PREPARATION_COMMANDS parameter. For more information, see see Section 1.7, "Configuring Open vSwitch (OVS) interfaces for backup" . Procedure On each node that you want to restore, create the script /usr/share/rear/setup/default/011_backup_ceph.sh and add the following content: mount -t <file_type> <device_disk> /mnt/local cd /mnt/local [ -d "var/lib/ceph" ] && tar cvfz /tmp/ceph.tar.gz var/lib/ceph --xattrs --xattrs-include='.' --acls cd / umount <device_disk> Replace <file_type> and <device_disk> with the type and location of the backup file. Normally, the file type is xfs and the location is /dev/vda2 . On the same node, create the script /usr/share/rear/wrapup/default/501_restore_ceph.sh and add the following content: if [ -f "/tmp/ceph.tar.gz" ]; then rm -rf /mnt/local/var/lib/ceph/* tar xvC /mnt/local -f /tmp/ceph.tar.gz var/lib/ceph --xattrs --xattrs-include='.' fi Additional resources Section 3.2, "Restoring the undercloud node" Section 3.3, "Restoring the control plane nodes" 3.2. Restoring the undercloud node You can restore the undercloud node to its state using the backup ISO image that you created using ReaR. You can find the backup ISO images on the backup node. Burn the bootable ISO image to a DVD or download it to the undercloud node through Integrated Lights-Out (iLO) remote access. Prerequisites You have created a backup of the undercloud node. For more information, see Section 1.8, "Creating a backup of the undercloud node" . You have access to the backup node. If you use an OVS bridge for your network interfaces, you have access to the network configuration information that you set in the NETWORKING_PREPARATION_COMMANDS parameter. For more information, see see Section 1.7, "Configuring Open vSwitch (OVS) interfaces for backup" . If you configured backup encryption, you must decrypt the backup before you begin the restoration process. Run the following decrypt step in the system where the backup file is located: USD dd if=backup.tar.gz | /usr/bin/openssl des3 -d -k "<encryption key>" | tar -C <backup_location> -xzvf - '*.conf' Replace <encryption key> with your encryption key. Replace <backup_location> with the folder in which you want to save the backup.tar.gz file, for example, /ctl_plane_backups/undercloud-0/ . Procedure Log in to the hypervisor: USD ssh root@<hypervisor> Power off the undercloud node. Ensure that the undercloud node is powered off completely before you proceed: USD virsh destroy undercloud-0 Boot the undercloud node with the backup ISO image. When the Relax-and-Recover boot menu displays, select Recover <undercloud_node> . Replace <undercloud_node> with the name of your undercloud node. Note If your system uses UEFI, select the Relax-and-Recover (no Secure Boot) option. Log in as the root user and restore the node: The following message displays: Welcome to Relax-and-Recover. Run "rear recover" to restore your system! RESCUE <undercloud_node>:~ # rear recover When the undercloud node restoration process completes, the console displays the following message: Finished recovering your system Exiting rear recover Running exit tasks Power off the node: RESCUE <undercloud_node>:~ # poweroff On boot up, the node resumes its state. 3.3. Restoring the control plane nodes If an error occurs during an update or upgrade, you can restore the control plane nodes to their state using the backup ISO image that you have created using ReaR. To restore the control plane, you must restore all control plane nodes to ensure state consistency. You can find the backup ISO images on the backup node. Burn the bootable ISO image to a DVD or download it to the undercloud node through Integrated Lights-Out (iLO) remote access. Note Red Hat supports backups of Red Hat OpenStack Platform with native SDNs, such as Open vSwitch (OVS) and the default Open Virtual Network (OVN). For information about third-party SDNs, refer to the third-party SDN documentation. Prerequisites You have created a backup of the control plane nodes. For more information, see Section 2.6, "Creating a backup of the control plane nodes" . You have access to the backup node. If you use an OVS bridge for your network interfaces, you have access to the network configuration information that you set in the NETWORKING_PREPARATION_COMMANDS parameter. For more information, see see Section 2.5, "Configuring Open vSwitch (OVS) interfaces for backup" . Procedure Power off each control plane node. Ensure that the control plane nodes are powered off completely before you proceed. Boot each control plane node with the corresponding backup ISO image. When the Relax-and-Recover boot menu displays, on each control plane node, select Recover <control_plane_node> . Replace <control_plane_node> with the name of the corresponding control plane node. Note If your system uses UEFI, select the Relax-and-Recover (no Secure Boot) option. On each control plane node, log in as the root user and restore the node: The following message displays: When the control plane node restoration process completes, the console displays the following message: When the command line console is available, restore the config-drive partition of each control plane node: # once completed, restore the config-drive partition (which is ISO9660) RESCUE <control_plane_node>:~ USD dd if=/mnt/local/mnt/config-drive of=<config_drive_partition> Power off the node: Set the boot sequence to the normal boot device. On boot up, the node resumes its state. To ensure that the services are running correctly, check the status of pacemaker. Log in to a Controller node as the root user and enter the following command: To view the status of the overcloud, use the OpenStack Integration Test Suite (tempest). For more information, see Validating your OpenStack cloud with the Integration Test Suite (tempest) . Troubleshooting Clear resource alarms that are displayed by pcs status by running the following command: Clear STONITH fencing action errors that are displayed by pcs status by running the following commands: 3.4. Restoring the Galera cluster manually If the Galera cluster does not restore as part of the restoration procedure, you must restore Galera manually. Note In this procedure, you must perform some steps on one Controller node. Ensure that you perform these steps on the same Controller node as you go through the procedure. Procedure On Controller-0 , retrieve the Galera cluster virtual IP: USD sudo hiera -c /etc/puppet/hiera.yaml mysql_vip Disable the database connections through the virtual IP on all Controller nodes: USD sudo iptables -I INPUT -p tcp --destination-port 3306 -d USDMYSQL_VIP -j DROP On Controller-0 , retrieve the MySQL root password: USD sudo hiera -c /etc/puppet/hiera.yaml mysql::server::root_password On Controller-0 , set the Galera resource to unmanaged mode: USD sudo pcs resource unmanage galera-bundle Stop the MySQL containers on all Controller nodes: USD sudo podman container stop USD(sudo podman container ls --all --format "{{.Names}}" --filter=name=galera-bundle) Move the current directory on all Controller nodes: USD sudo mv /var/lib/mysql /var/lib/mysql-save Create the new directory /var/lib/mysq on all Controller nodes: USD sudo mkdir /var/lib/mysql USD sudo chown 42434:42434 /var/lib/mysql USD sudo chcon -t container_file_t /var/lib/mysql USD sudo chmod 0755 /var/lib/mysql USD sudo chcon -r object_r /var/lib/mysql USD sudo chcon -u system_u /var/lib/mysql Start the MySQL containers on all Controller nodes: USD sudo podman container start USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) Create the MySQL database on all Controller nodes: USD sudo podman exec -i USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) bash -c "mysql_install_db --datadir=/var/lib/mysql --user=mysql --log_error=/var/log/mysql/mysql_init.log" Start the database on all Controller nodes: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) bash -c "mysqld_safe --skip-networking --wsrep-on=OFF --log-error=/var/log/mysql/mysql_safe.log" & Move the .my.cnf Galera configuration file on all Controller nodes: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) bash -c "mv /root/.my.cnf /root/.my.cnf.bck" Reset the Galera root password on all Controller nodes: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) bash -c "mysql -uroot -e'use mysql;update user set password=PASSWORD(\"USDROOTPASSWORD\")where User=\"root\";flush privileges;'" Restore the .my.cnf Galera configuration file inside the Galera container on all Controller nodes: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) bash -c "mv /root/.my.cnf.bck /root/.my.cnf" On Controller-0 , copy the backup database files to /var/lib/MySQL : USD sudo cp USDBACKUP_FILE /var/lib/mysql USD sudo cp USDBACKUP_GRANT_FILE /var/lib/mysql Note The path to these files is /home/tripleo-admin/. On Controller-0 , restore the MySQL database: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) bash -c "mysql -u root -pUSDROOT_PASSWORD < \"/var/lib/mysql/USDBACKUP_FILE\" " USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) bash -c "mysql -u root -pUSDROOT_PASSWORD < \"/var/lib/mysql/USDBACKUP_GRANT_FILE\" " Shut down the databases on all Controller nodes: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) bash -c "mysqladmin shutdown" On Controller-0 , start the bootstrap node: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) \ /usr/bin/mysqld_safe --pid-file=/var/run/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock --datadir=/var/lib/mysql \ --log-error=/var/log/mysql/mysql_cluster.log --user=mysql --open-files-limit=16384 \ --wsrep-cluster-address=gcomm:// & Verification: On Controller-0, check the status of the cluster: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) bash -c "clustercheck" Ensure that the following message is displayed: "Galera cluster node is synced", otherwise you must recreate the node. On Controller-0 , retrieve the cluster address from the configuration: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) bash -c "grep wsrep_cluster_address /etc/my.cnf.d/galera.cnf" | awk '{print USD3}' On each of the remaining Controller nodes, start the database and validate the cluster: Start the database: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) /usr/bin/mysqld_safe --pid-file=/var/run/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock \ --datadir=/var/lib/mysql --log-error=/var/log/mysql/mysql_cluster.log --user=mysql --open-files-limit=16384 \ --wsrep-cluster-address=USDCLUSTER_ADDRESS & Check the status of the MYSQL cluster: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" \ --filter=name=galera-bundle) bash -c "clustercheck" Ensure that the following message is displayed: "Galera cluster node is synced", otherwise you must recreate the node. Stop the MySQL container on all Controller nodes: USD sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) \ /usr/bin/mysqladmin -u root shutdown On all Controller nodes, remove the following firewall rule to allow database connections through the virtual IP address: USD sudo iptables -D INPUT -p tcp --destination-port 3306 -d USDMYSQL_VIP -j DROP Restart the MySQL container on all Controller nodes: USD sudo podman container restart USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) Restart the clustercheck container on all Controller nodes: USD sudo podman container restart USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=clustercheck) On Controller-0 , set the Galera resource to managed mode: USD sudo pcs resource manage galera-bundle Verification To ensure that services are running correctly, check the status of pacemaker: USD sudo pcs status To view the status of the overcloud, use the OpenStack Integration Test Suite (tempest). For more information, see Validating your OpenStack cloud with the Integration Test Suite (tempest) . If you suspect an issue with a particular node, check the state of the cluster with clustercheck : USD sudo podman exec clustercheck /usr/bin/clustercheck 3.5. Restoring the undercloud node database manually If the undercloud database does not restore as part of the undercloud restore process, you can restore the database manually. You can only restore the database if you previously created a standalone database backup. Prerequisites You have created a standalone backup of the undercloud database. For more information, see Section 1.6, "Creating a standalone database backup of the undercloud nodes" . Procedure Log in to the director undercloud node as the root user. Stop all tripleo services: [root@director ~]# systemctl stop tripleo_* Ensure that no containers are running on the server by entering the following command: [root@director ~]# podman ps If any containers are running, enter the following command to stop the containers: Create a backup of the current /var/lib/mysql directory and then delete the directory: [root@director ~]# cp -a /var/lib/mysql /var/lib/mysql_bck [root@director ~]# rm -rf /var/lib/mysql Recreate the database directory and set the SELinux attributes for the new directory: [root@director ~]# mkdir /var/lib/mysql [root@director ~]# chown 42434:42434 /var/lib/mysql [root@director ~]# chmod 0755 /var/lib/mysql [root@director ~]# chcon -t container_file_t /var/lib/mysql [root@director ~]# chcon -r object_r /var/lib/mysql [root@director ~]# chcon -u system_u /var/lib/mysql Create a local tag for the mariadb image. Replace <image_id> and <undercloud.ctlplane.example.com> with the values applicable in your environment: [root@director ~]# podman images | grep mariadb <undercloud.ctlplane.example.com>:8787/rh-osbs/rhosp16-openstack-mariadb 16.2_20210322.1 <image_id> 3 weeks ago 718 MB [root@director ~]# podman tag <image_id> mariadb [root@director ~]# podman images | grep maria localhost/mariadb latest <image_id> 3 weeks ago 718 MB <undercloud.ctlplane.example.com>:8787/rh-osbs/rhosp16-openstack-mariadb 16.2_20210322.1 <image_id> 3 weeks ago 718 MB Initialize the /var/lib/mysql directory with the container: [root@director ~]# podman run --net=host -v /var/lib/mysql:/var/lib/mysql localhost/mariadb mysql_install_db --datadir=/var/lib/mysql --user=mysql Copy the database backup file that you want to import to the database: [root@director ~]# cp /root/undercloud-all-databases.sql /var/lib/mysql Start the database service to import the data: [root@director ~]# podman run --net=host -dt -v /var/lib/mysql:/var/lib/mysql localhost/mariadb /usr/libexec/mysqld Import the data and configure the max_allowed_packet parameter: Log in to the container and configure it: [root@director ~]# podman exec -it <container_id> /bin/bash ()[mysql@5a4e429c6f40 /]USD mysql -u root -e "set global max_allowed_packet = 1073741824;" ()[mysql@5a4e429c6f40 /]USD mysql -u root < /var/lib/mysql/undercloud-all-databases.sql ()[mysql@5a4e429c6f40 /]USD mysql -u root -e 'flush privileges' ()[mysql@5a4e429c6f40 /]USD exit exit Stop the container: [root@director ~]# podman stop <container_id> Check that no containers are running: [root@director ~]# podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES [root@director ~]# Restart all tripleo services: [root@director ~]# systemctl start multi-user.target

Commands:
mount -t <file_type> <device_disk> /mnt/local cd /mnt/local [ -d "var/lib/ceph" ] && tar cvfz /tmp/ceph.tar.gz var/lib/ceph --xattrs --xattrs-include='.' --acls cd / umount <device_disk>
if [ -f "/tmp/ceph.tar.gz" ]; then rm -rf /mnt/local/var/lib/ceph/* tar xvC /mnt/local -f /tmp/ceph.tar.gz var/lib/ceph --xattrs --xattrs-include='.' fi
dd if=backup.tar.gz | /usr/bin/openssl des3 -d -k "<encryption key>" | tar -C <backup_location> -xzvf - '*.conf'
ssh root@<hypervisor>
virsh destroy undercloud-0
Welcome to Relax-and-Recover. Run "rear recover" to restore your system! RESCUE <undercloud_node>:~ # rear recover
Finished recovering your system Exiting rear recover Running exit tasks
RESCUE <undercloud_node>:~ # poweroff
Welcome to Relax-and-Recover. Run "rear recover" to restore your system! RESCUE <control_plane_node>:~ # rear recover
Finished recovering your system Exiting rear recover Running exit tasks
once completed, restore the config-drive partition (which is ISO9660) RESCUE <control_plane_node>:~ USD dd if=/mnt/local/mnt/config-drive of=<config_drive_partition>
RESCUE <control_plane_node>:~ # poweroff
pcs status
pcs resource clean
pcs resource clean pcs stonith history cleanup
sudo hiera -c /etc/puppet/hiera.yaml mysql_vip
sudo iptables -I INPUT -p tcp --destination-port 3306 -d USDMYSQL_VIP -j DROP
sudo hiera -c /etc/puppet/hiera.yaml mysql::server::root_password
sudo pcs resource unmanage galera-bundle
sudo podman container stop USD(sudo podman container ls --all --format "{{.Names}}" --filter=name=galera-bundle)
sudo mv /var/lib/mysql /var/lib/mysql-save
sudo mkdir /var/lib/mysql sudo chown 42434:42434 /var/lib/mysql sudo chcon -t container_file_t /var/lib/mysql sudo chmod 0755 /var/lib/mysql sudo chcon -r object_r /var/lib/mysql sudo chcon -u system_u /var/lib/mysql
sudo podman container start USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle)
sudo podman exec -i USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) bash -c "mysql_install_db --datadir=/var/lib/mysql --user=mysql --log_error=/var/log/mysql/mysql_init.log"
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) bash -c "mysqld_safe --skip-networking --wsrep-on=OFF --log-error=/var/log/mysql/mysql_safe.log" &
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) bash -c "mv /root/.my.cnf /root/.my.cnf.bck"
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) bash -c "mysql -uroot -e'use mysql;update user set password=PASSWORD(\"USDROOTPASSWORD\")where User=\"root\";flush privileges;'"
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) bash -c "mv /root/.my.cnf.bck /root/.my.cnf"
sudo cp USDBACKUP_FILE /var/lib/mysql sudo cp USDBACKUP_GRANT_FILE /var/lib/mysql
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) bash -c "mysql -u root -pUSDROOT_PASSWORD < \"/var/lib/mysql/USDBACKUP_FILE\" " sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) bash -c "mysql -u root -pUSDROOT_PASSWORD < \"/var/lib/mysql/USDBACKUP_GRANT_FILE\" "
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) bash -c "mysqladmin shutdown"
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) /usr/bin/mysqld_safe --pid-file=/var/run/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock --datadir=/var/lib/mysql --log-error=/var/log/mysql/mysql_cluster.log --user=mysql --open-files-limit=16384 --wsrep-cluster-address=gcomm:// &
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) bash -c "clustercheck"
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) bash -c "grep wsrep_cluster_address /etc/my.cnf.d/galera.cnf" | awk '{print USD3}'
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) /usr/bin/mysqld_safe --pid-file=/var/run/mysql/mysqld.pid --socket=/var/lib/mysql/mysql.sock --datadir=/var/lib/mysql --log-error=/var/log/mysql/mysql_cluster.log --user=mysql --open-files-limit=16384 --wsrep-cluster-address=USDCLUSTER_ADDRESS &
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) bash -c "clustercheck"
sudo podman exec USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle) /usr/bin/mysqladmin -u root shutdown
sudo iptables -D INPUT -p tcp --destination-port 3306 -d USDMYSQL_VIP -j DROP
sudo podman container restart USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=galera-bundle)
sudo podman container restart USD(sudo podman container ls --all --format "{{ .Names }}" --filter=name=clustercheck)
sudo pcs resource manage galera-bundle
sudo pcs status
sudo podman exec clustercheck /usr/bin/clustercheck
systemctl stop tripleo_*
podman ps
podman stop <container_name>
cp -a /var/lib/mysql /var/lib/mysql_bck rm -rf /var/lib/mysql
mkdir /var/lib/mysql chown 42434:42434 /var/lib/mysql chmod 0755 /var/lib/mysql chcon -t container_file_t /var/lib/mysql chcon -r object_r /var/lib/mysql chcon -u system_u /var/lib/mysql
podman images | grep mariadb <undercloud.ctlplane.example.com>:8787/rh-osbs/rhosp16-openstack-mariadb 16.2_20210322.1 <image_id> 3 weeks ago 718 MB
podman tag <image_id> mariadb
podman images | grep maria localhost/mariadb latest <image_id> 3 weeks ago 718 MB <undercloud.ctlplane.example.com>:8787/rh-osbs/rhosp16-openstack-mariadb 16.2_20210322.1 <image_id> 3 weeks ago 718 MB
podman run --net=host -v /var/lib/mysql:/var/lib/mysql localhost/mariadb mysql_install_db --datadir=/var/lib/mysql --user=mysql
cp /root/undercloud-all-databases.sql /var/lib/mysql
podman run --net=host -dt -v /var/lib/mysql:/var/lib/mysql localhost/mariadb /usr/libexec/mysqld
podman exec -it <container_id> /bin/bash ()[mysql@5a4e429c6f40 /]USD mysql -u root -e "set global max_allowed_packet = 1073741824;" ()[mysql@5a4e429c6f40 /]USD mysql -u root < /var/lib/mysql/undercloud-all-databases.sql ()[mysql@5a4e429c6f40 /]USD mysql -u root -e 'flush privileges' ()[mysql@5a4e429c6f40 /]USD exit exit
podman stop <container_id>
podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
systemctl start multi-user.target