Chapter 4. Executing the backup procedure Before you perform a fast forward upgrade, back up the undercloud and the overcloud control plane nodes so that you can restore them to their state if an error occurs. Note Before you back up the undercloud and overcloud, ensure that you do not perform any operations on the overcloud from the undercloud. 4.1. Performing prerequisite tasks before backing up the undercloud Do not perform an undercloud backup when you deploy the undercloud or when you make changes to an existing undercloud. To prevent data corruptions, confirm that there are no stack failures, ongoing tasks, and that all OpenStack services except for mariadb are stopped before you back up the undercloud node. Procedure List failures for all available stacks: Verify that there are no ongoing tasks in the cloud: If the command returns no results, there are no ongoing tasks. Stop all OpenStack services in the cloud: Start the tripleo_mysql service: Verify that the tripleo_mysql service is running: 4.2. Backing up the undercloud To back up the undercloud node, you must log in as the root user on the undercloud node. As a precaution, you must back up the database to ensure that you can restore it. Prerequisites You have created and exported the backup directory. For more information, see Creating and exporting the backup directory . You have performed prerequisite tasks before backing up the undercloud. For more information, see Performing prerequisite tasks before backing up the undercloud . You have installed and configured ReaR on the undercloud node. For more information, see Install and Configure ReaR . Procedure Locate the database password. Back up the databases: Stop the mariadb database service: Create the backup: You can find the backup ISO file that you create with ReaR on the backup node in the /ctl_plane_backups directory. 4.3. Backing up the control plane To back up the control plane, you must first stop the pacemaker cluster and all containers operating on the control plane nodes. Do not operate the stack to ensure state consistency. After you complete the backup procedure, start the pacemaker cluster and the containers. As a precaution, you must back up the database to ensure that you can restore the database after you restart the pacemaker cluster and containers. Back up the control plane nodes simultaneously. Prerequisites You have created and exported the backup directory. For more information, see Creating and exporting the backup directory . You have installed and configured ReaR on the undercloud node. For more information, see Install and Configure ReaR . Procedure Locate the database password: Back up the databases: On one of control plane nodes, stop the pacemaker cluster: Important Do not operate the stack. When you stop the pacemaker cluster and the containers, this results in the temporary interruption of control plane services to Compute nodes. There is also disruption to network connectivity, Ceph, and the NFS data plane service. You cannot create instances, migrate instances, authenticate requests, or monitor the health of the cluster until the pacemaker cluster and the containers return to service following the final step of this procedure. On each control plane node, stop the containers. Stop the containers: Stop the ceph-mon@controller.service container: Stop the ceph-mgr@controller.service container: To back up the control plane, run the following command as root in the command line interface of each control plane node: You can find the backup ISO file that you create with ReaR on the backup node in the /ctl_plane_backups directory. Note When you execute the backup command, you might see warning messages regarding the tar command and sockets that are ignored during the tar process, similar to the following: When the backup procedure generates ISO images for each of the control plane nodes, restart the pacemaker cluster and the containers: On one of the control plane nodes, enter the following command: On each control plane node, start the containers. Start the ceph-mon@controller.service container: Start the ceph-mgr@controller.service container:

Commands:
(undercloud) [stack@undercloud-0 ~]USD source stackrc && for i in `openstack stack list -c 'Stack Name' -f value`; do openstack stack failures list USDi; done
(undercloud) [stack@undercloud-0 ~]USD openstack stack list --nested | grep -v "_COMPLETE"
systemctl stop tripleo_*
systemctl start tripleo_mysql
systemctl status tripleo_mysql
PASSWORD=USD(/bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)
podman exec mysql bash -c "mysql -uroot -pUSDPASSWORD -s -N -e \"SELECT CONCAT('\\\"SHOW GRANTS FOR ''',user,'''@''',host,''';\\\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')\" | xargs -n1 mysql -uroot -pUSDPASSWORD -s -N -e | sed 's/USD/;/' " > openstack-backup-mysql-grants.sql
podman exec mysql bash -c "mysql -uroot -pUSDPASSWORD -s -N -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\" | xargs mysqldump -uroot -pUSDPASSWORD --single-transaction --databases" > openstack-backup-mysql.sql
systemctl stop tripleo_mysql
rear -d -v mkbackup
PASSWORD=USD(/bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)
podman exec galera-bundle-podman-X bash -c "mysql -uroot -pUSDPASSWORD -s -N -e \"SELECT CONCAT('\\\"SHOW GRANTS FOR ''',user,'''@''',host,''';\\\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')\" | xargs -n1 mysql -uroot -pUSDPASSWORD -s -N -e | sed 's/USD/;/' " > openstack-backup-mysql-grants.sql
podman exec galera-bundle-podman-X bash -c "mysql -uroot -pUSDPASSWORD -s -N -e \"select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';\" | xargs mysqldump -uroot -pUSDPASSWORD --single-transaction --databases" > openstack-backup-mysql.sql
pcs cluster stop --all
systemctl stop tripleo_*
sudo systemctl stop ceph-mon@USD(hostname -s)
sudo systemctl stop ceph-mgr@USD(hostname -s)
rear -d -v mkbackup
WARNING: tar ended with return code 1 and below output: ---snip--- tar: /var/spool/postfix/public/qmgr: socket ignored This message indicates that files have been modified during the archiving process and the backup might be inconsistent. Relax-and-Recover continues to operate, however, it is important that you verify the backup to ensure that you can use this backup to recover your system.
pcs cluster start --all
systemctl start ceph-mon@USD(hostname -s)
systemctl start ceph-mgr@USD(hostname -s)