Chapter 4. Troubleshooting the Block Storage backup service You can diagnose many issues by verifying that the Block Storage services are running correctly and then by examining the log files for error messages. 4.1. Verifying the Block Storage backup service deployment After a deployment or when troubleshooting issues, it is important to verify that the necessary Block Storage services are running correctly on their hosts. Ensure that the Block Storage backup service is running on every Controller node, like the Block Storage scheduler service. After you verify that the necessary Block Storage services are running correctly, then you must verify that the Block Storage backup service is deployed successfully. Procedure Run the openstack volume service list command: Verify that the State entry of every service is up . If not, examine the relevant log files. For more information about the location of these log files, see Block Storage (cinder) Log Files in Managing overcloud observability . Verify that the Block Storage backup service is deployed successfully, by backing up any Block Storage volume and ensuring that the backup succeeds. For more information, see Troubleshooting backups . Additional resources Examining the Block Storage backup service log file 4.2. Troubleshooting backups The Block Storage backup service performs static checks when receiving a request to back up a Block Storage (cinder) volume. If these checks fail then you will immediately be notified: Check for an invalid volume reference ( missing ). Check if the volume is in-use or attached to an instance. The in-use case requires you to use the --force option to perform a backup. For more information, see Creating a backup of an in-use volume . When you use the --force volume backup option, you create a crash-consistent, but not an application-consistent, backup because the volume is not quiesced before performing the backup. Therefore, the data is intact but the backup does not have an awareness of which applications were running when the backup was performed. When these checks succeed: the Block Storage backup service accepts the request to backup this volume, the CLI backup command returns immediately, and the volume is backed up in the background. Therefore the CLI backup command returns even if the backup fails. You can use the openstack volume backup list command to verify that the volume backup is successful, when the Status of the backup entry is available . If a backup fails, examine the Block Storage backup service log file for error messages to discover the cause. For more information, see Examining the Block Storage backup service log file . 4.3. Examining the Block Storage backup service log file When a backup or restore does not succeed, you can examine the Block Storage backup service log file for error messages that can help you to determine the reason. Procedure Find the Block Storage backup service log file on the Controller node where the backup service is running. This log file is located in the following path: /var/log/containers/cinder/cinder-backup.log . 4.4. Volume backup workflow The following diagram and explanation describe the steps that occur when the user requests the cinder API to backup a Block Storage (cinder) volume. Figure 4.1. Creating a backup of a Block Storage volume The user issues a request to the cinder API, which is a REST API, to back up a Block Storage volume. The cinder API receives the request from HAProxy and validates the request, the user credentials, and other information. The cinder API creates the backup record in the SQL database. The cinder API makes an asynchronous RPC call to the cinder-backup service via AMQP to back up the volume. The cinder API returns the current backup record, with an ID, to the API caller. An RPC create message arrives on one of the backup services. The cinder-backup service performs a synchronous RPC call to get_backup_device . The cinder-volume service ensures that the correct device is returned to the caller. Normally, it is the same volume, but if the volume is in use, the service returns a temporary cloned volume or a temporary snapshot, depending on the configuration. The cinder-backup service issues another synchronous RPC to cinder-volume to expose the source device. The cinder-volume service exports and maps the source device (volume or snapshot) and returns the appropriate connection information. The cinder-backup service attaches the source device by using the connection information. The cinder-backup service calls the backup back end driver, with the device already attached, which begins the data transfer to the backup repository. The source device is detached from the Backup host. The cinder-backup service issues a synchronous RPC to cinder-volume to disconnect the source device. The cinder-volume service unmaps and removes the export for the device. If a temporary volume or temporary snapshot was created, cinder-backup calls cinder-volume to remove it. The cinder-volume service removes the temporary volume. When the backup is completed, the backup record is updated in the database. 4.5. Volume restore workflow The following diagram and explanation describe the steps that occur when the user requests the cinder API to restore a Block Storage service (cinder) backup. Figure 4.2. Restoring a Block Storage backup The user issues a request to the cinder API, which is a REST API, to restore a Block Storage backup. The cinder API receives the request from HAProxy and validates the request, the user credentials, and other information. If the request does not contain an existing volume as the destination, the cinder API makes an asynchronous RPC call to create a new volume and polls the status of the volume until it becomes available. The cinder-scheduler selects a volume service and makes the RPC call to create the volume. The selected cinder-volume service creates the volume. When the cinder API detects that the volume is available, the backup record is created in the database. The cinder API makes an asynchronous RPC call to the backup service via AMQP to restore the backup. The cinder API returns the current volume ID, backup ID, and volume name to the API caller. An RPC create message arrives on one of the backup services. The cinder-backup service performs a synchronous RPC call to cinder-volume to expose the volume. The cinder-volume service exports and maps the volume returning the appropriate connection information. The cinder-backup service attaches the volume by using the connection information. The cinder-backup service calls the back end driver with the volume already attached, which begins the data restoration to the volume. The volume is detached from the backup host. The cinder-backup service issues a synchronous RPC to cinder-volume to disconnect the volume. The cinder-volume service unmaps and removes the export for the volume. When the volume is restored, the backup record is updated in the database.

Commands:
openstack volume service list +------------------+-------------------------+------+---------+-------+----------------------------+ | Binary | Host | Zone | Status | State | Updated At | +------------------+-------------------------+------+---------+-------+----------------------------+ | cinder-scheduler | controller-0 | nova | enabled | up | 2023-06-21T13:07:42.000000 | | cinder-scheduler | controller-1 | nova | enabled | up | 2023-06-21T13:07:42.000000 | | cinder-scheduler | controller-2 | nova | enabled | up | 2023-06-21T13:07:42.000000 | | cinder-backup | controller-0 | nova | enabled | up | 2023-06-21T13:07:46.000000 | | cinder-backup | controller-1 | nova | enabled | up | 2023-06-21T13:07:46.000000 | | cinder-backup | controller-2 | nova | enabled | up | 2023-06-21T13:07:46.000000 | | cinder-volume | hostgroup@tripleo_iscsi | nova | enabled | up | 2023-06-21T13:07:47.000000 | +------------------+-------------------------+------+---------+-------+----------------------------+