Chapter 3. Performing operations with the Block Storage backup service You can use the Block Storage service (cinder) backup service to perform, protect, restore, and troubleshoot backups. Note To execute openstack client commands on the cloud, you must specify the name of the cloud detailed in your clouds.yaml file. You can specify the name of the cloud by using one of the following methods: Use the --os-cloud option with each command: Use this option if you access more than one cloud. Create an environment variable for the cloud name in your bashrc file: Prerequisites The administrator has created a project for you, and they have provided you with a clouds.yaml file for you to access the cloud. You have installed the python-openstackclient package. Only administrators and volume owners with access can perform and access backups. 3.1. Using the Block Storage backup service You can use the Block Storage backup service to perform full or incremental backups, to protect your backups, and to restore a backup to a volume. 3.1.1. Authenticating volume owners for access to volume backups Administrators can back up any volume belonging to the project. To ensure that the volume owner can also access the volume backup, administrators must provide arguments to authenticate the volume owner when backing up the volume. Procedure Provide the following arguments to authenticate a volume owner for access to volume backups: Replace <projectname> with the name of the project (tenant) of the owner of the volume. Replace <username> and <password> with the username and password credentials of the user that is the owner of the volume within this project. Note [--name <backup_name>] <volume> are the typical arguments when creating a volume backup. 3.1.2. Creating backups Create a backup of your Block Storage volume to protect your data from being lost if there is an issue with the volume. For more information, see Creating a full volume backup . You can also create a backup directly from a snapshot of a volume. In addition to the volume data, a backup stores the volume metadata, such as the name and description. If data compression is enabled for the storage back end for your backups, then your backups are compressed, which can reduce the performance of backup operations. Full backups are easier to manage but they can become resource intensive when the size of the volume increases over time. With incremental backups, you can capture periodic changes to volumes and minimize resource usage. When you create a backup of a Block Storage volume, the metadata for this backup is stored in the Block Storage service database. The metadata is used if you restore the volume backup. To ensure that a backup survives a catastrophic loss of the Block Storage service database, you can manually export and store the metadata of this backup. 3.1.2.1. Creating a full volume backup You can create one or more full backups of a volume. Prerequisites Only volume owners and administrators can backup volumes. The storage back end for backups must have the necessary space. The backup quotas specified for your project have not been exceeded. Procedure Access the remote shell for the OpenStackClient pod from your workstation: List the volumes to obtain the ID or name of the volume you want to back up: Note Usually, you can only back up a volume that has an available status, but you can backup a volume with an in-use status if required. For more information, see Creating a backup of an in-use volume . Back up the volume: Replace <volume> with the ID or name of the volume you want to back up. Optional: replace <backup_name> with the name of this backup. This command immediately provides the ID of this backup but the volume is backed up asynchronously, in the background. For example: Verification List the backups: The volume backup is created when this backup has an available status. For example: Exit the openstackclient pod: Additional resources Creating backups 3.1.2.2. Creating a full backup of a snapshot You can create a full backup from a snapshot by using the ID of the volume associated with the snapshot. The backup is created by directly attaching to the snapshot, which is faster than cloning the snapshot into a volume and then backing up this volume. But this feature can affect the backup performance because of the extra step of creating the volume from a snapshot. Prerequisites: Your backup repository must have the necessary space. The backup quotas specified for your project have not been exceeded. Procedure Access the remote shell for the OpenStackClient pod from your workstation: List the snapshots to obtain the name or ID of the snapshot you want to backup: List the details of this snapshot to obtain the ID of the volume associated with this snapshot: Replace <snapshot> with the name or ID of the snapshot you want to backup. The value of the volume_id field is the ID of the volume associated with this snapshot. For example: Backup the snapshot: Replace <volume_id> with the ID of the volume associated with this snapshot. Optional: Replace <backup_name> with the name of this backup. This command immediately provides the ID of this backup but the snapshot is backed up asynchronously, in the background. For example: Verification List the backups: The snapshot backup is created when this backup has an available status. For example: Exit the openstackclient pod: 3.1.2.3. Creating a backup of an in-use volume Usually you can only backup a volume that has an available status. But you can use the --force option when creating a backup, to back up a volume that has an in-use status. When you use the --force volume backup option, you create a crash-consistent, but not an application-consistent, backup because the volume is not quiesced before performing the backup. Therefore, the data is intact but the backup does not have an awareness of which applications were running when the backup was performed. Prerequisites Only volume owners and project administrators can backup volumes. Your backup repository must have the necessary space. The backup quotas specified for your project have not been exceeded. Procedure Access the remote shell for the OpenStackClient pod from your workstation: List the volumes to obtain the ID or name of the volume you want to back up: For example: Force the back up, if the volume that you want to backup has an in-use status: Replace <volume> with the ID or name of the volume you want to back up. Optional: replace <backup_name> with the name of this backup. This command immediately provides the ID of this backup but the volume is backed up asynchronously, in the background. For example: Verification List the backups: The volume backup is created when this backup has an available status. For example: Exit the openstackclient pod: 3.1.2.4. Incremental backups If a volume has at least one full backup, you can use the Block Storage backup service to create an incremental backup. For more information, see Creating an incremental backup . Full backups are easier to manage but they can become resource intensive when the size of the volume increases over time. With incremental backups, you can capture periodic changes to volumes and minimize your resource usage. An incremental backup only stores the changes made to the volume since the last full or incremental backup. Incremental backups increase the administrative overhead required for managing your backups. For instance, you cannot delete a full backup if it already has one or more incremental backups, you can only delete the latest incremental backup. Incremental backups have a lower performance than full backups: When you create an incremental backup, all of the data in the volume must first be read and compared with the data in both the full backup and each subsequent incremental backup. 3.1.2.5. Creating an incremental backup You can create an incremental backup to only store the changes made to the volume since the last full or incremental backup. Prerequisites: At least one full backup of the volume. For more information, see Creating a full volume backup . Only volume owners and project administrators can backup volumes. Your backup repository must have the necessary space. The backup quotas specified for your project have not been exceeded. Procedure Access the remote shell for the OpenStackClient pod from your workstation: List the volumes to obtain the ID or name of the volume you want to back up: Back up the volume and use the --incremental option: Replace <volume> with the ID or name of the volume you want to back up. Optional: replace <backup_name> with the name of this backup. This command immediately provides the ID of this backup but the volume is backed up asynchronously, in the background. For example: Verification List the backups: The volume backup is created when this backup has an available status. For example: Exit the openstackclient pod: 3.1.2.6. Canceling a backup You must request a force delete on the backup to cancel it. Important You cannot cancel backups if you use the Red Hat Ceph Storage back end for your backup repository. Procedure Access the remote shell for the OpenStackClient pod from your workstation: List the backups to obtain the ID or name of the backup you want to cancel: Cancel the backup: Replace <backup> with the ID or name of the volume backup that you want to cancel. There can be a slight delay for the backup to be successfully canceled. Verification The backup is canceled when the backup record is not listed by this command: Exit the openstackclient pod: 3.1.3. Protecting your backups When you create a backup of a Block Storage volume, the metadata for this backup is stored in the Block Storage service database, which is used to restore this volume. To ensure that a backup survives a catastrophic loss of the Block Storage service database, you can manually export and store the metadata of this backup in a safe location, such as an offsite backup. For more information, see Exporting backup metadata . When the Block Storage service database experiences a catastrophic loss, you cannot restore any of your backups because this database contains the backup metadata used when restoring backups. But if you manually exported and saved the metadata of a backup, then you can import this metadata into the new Block Storage database, so that you can use this backup to restore the volume. For more information see Importing backup metadata . Note For incremental backups, you must import the metadata of all the preceding backups before you can use it to restore the volume. 3.1.3.1. Exporting backup metadata You can export the metadata of a backup and store it in a file so that you can restore the volume backup even if the Block Storage database suffers a catastrophic loss. Note For an incremental backup, you must export the metadata of all the preceding backups. Access the remote shell for the OpenStackClient pod from your workstation: List the backups to obtain the ID or name of the backup: Export the metadata of the backup and store it in an appropriately named YAML file: Replace <backup> with the ID or name of the volume backup. Replace <filename> with the name of the YAML file to save the exported backup_service and backup_url values for this backup. For example: Copy the file to a safe location, such as an offsite backup. Verification Edit the file to see that the values of the backup_service and backup_url , match the values provided by this command: For example: Exit the openstackclient pod: 3.1.3.2. Importing backup metadata If you export and save the metadata of a volume backup, then you can import this metadata and use the backup if there is a loss of the Block Storage service database. You can also use this procedure to recreate a backup that was deleted. Note For incremental backups, you must also import the metadata of all the preceding backups. Prerequisites You must provide the backup_service and backup_url metadata values of this backup. For more information see Exporting backup metadata . A Block Storage database that does not already contain this backup. Procedure Access the remote shell for the OpenStackClient pod from your workstation: Locate the file in which you stored the exported backup_service and backup_url metadata values of this backup. Import the metadata values of this volume backup to the Block Storage database: Replace <backup_service> with the backup_service metadata value of this volume backup. Replace <backup_url> with the backup_url metadata value of this volume backup. This command provides the name and the ID of this backup. For example: Exit the openstackclient pod: steps Restoring backups 3.1.4. Restoring backups After you create a Block Storage volume backup, you can restore this backed up data if needed. You can use one of the following methods to restore your backups: Restore the backup to a volume that you specify. For more information, see Restoring a backup to a specific volume . Note If you choose Red Hat Ceph Storage as the back end for your backup repository, then you can only restore backed up volumes to a RBD-based Block Storage back end. Restore the backup to a new volume. For more information, see Restoring a backup to a new volume . Important When the Block Storage service database experiences a catastrophic loss you cannot restore any of your backups, unless you have exported and saved their metadata. Only administrators can cancel restoring a volume backup. 3.1.4.1. Restoring a backup to a specific volume You can restore a volume backup to an available volume that you have already created. If you restore a volume from an encrypted volume backup, then the destination volume type must also be encrypted. Important If you choose Red Hat Ceph Storage as the back end for your backup repository, then you can only restore backed up volumes to a RBD-based Block Storage back end. Procedure Access the remote shell for the OpenStackClient pod from your workstation: List the backups to obtain the name or ID of the backup you want to restore: For example: List the volumes: Ensure that the status of the required volume is available and then obtain the name or ID of this volume. For example: Restore the backup to the volume: Replace <backup> with the name or ID of the Block Storage volume backup. Replace <volume> with the name or ID of the available Block Storage volume. For example: Verify that the backup_id provided by this command corresponds to the ID of the backup that was restored and that the volume_name and volume_id values correspond to the name and ID of the specified volume. Delete the backup if you no longer need it: Exit the openstackclient pod: 3.1.4.2. Restoring a backup to a new volume You can create a new volume when you restore a backup of a Block Storage volume. Procedure Access the remote shell for the OpenStackClient pod from your workstation: List the backups to obtain the name or ID of the backup you want to restore: For example: Restore the backup to a new volume: Replace <backup> with the ID of the Block Storage volume backup. Optional: Replace <volume> with the ID of the Block Storage volume. Verify that the backup_id provided by this command corresponds to the ID of the backup that was restored. The volume_id value is the ID of the created volume. But the volume_name can be a temporary name that is replaced with the name of the backed up volume. List the volumes to verify that the volume with an ID of volume_id has been created and to obtain this volume name: For example: Delete the backup if you no longer need it: Exit the openstackclient pod: 3.1.4.3. Canceling restoring a backup You can cancel restoring a volume backup by changing the status of the backup to error . But you cannot cancel restoring a backup when Red Hat Ceph Storage is the back end of the backup repository. Warning If you cancel restoring a backup after it starts, the destination volume is useless, because there is no way of knowing how much data, if any, was actually restored. Prerequisites Ensure that the back end of your backup repository is not Red Hat Ceph Storage. Only administrators can cancel restoring a volume backup. Procedure Access the remote shell for the OpenStackClient pod from your workstation: List the backups to obtain the name or ID of the backup that you want to stop restoring: Change the status of this backup to error to cancel its restore operation: Replace <backup> with the name or ID of the volume backup that you do not want to restore. Canceling a restore is an asynchronous action, because the back end of the backup repository must detect the change in the backup status before it cancels the restore. Verification List the volume backups to verify that the restore is canceled: When the status of the backup changes to available , then the restore is canceled. Exit the openstackclient pod: 3.2. Troubleshooting the Block Storage backup service You can diagnose many issues by verifying that the Block Storage services are running correctly and then by examining the log files for error messages. 3.2.1. Troubleshooting backups The Block Storage backup service performs static checks when receiving a request to back up a Block Storage (cinder) volume. If these checks fail then you will immediately be notified: Check for an invalid volume reference ( missing ). Check if the volume is in-use or attached to an instance. The in-use case requires you to use the --force option to perform a backup. For more information, see Creating a backup of an in-use volume . When you use the --force volume backup option, you create a crash-consistent, but not an application-consistent, backup because the volume is not quiesced before performing the backup. Therefore, the data is intact but the backup does not have an awareness of which applications were running when the backup was performed. When these checks succeed: the Block Storage backup service accepts the request to backup this volume, the CLI backup command returns immediately, and the volume is backed up in the background. Therefore the CLI backup command returns even if the backup fails. You can use the openstack volume backup list command to verify that the volume backup is successful, when the Status of the backup entry is available . If a backup fails, examine the Block Storage backup service log file for error messages to discover the cause. 3.2.2. Examining the Block Storage backup service log file When a backup or restore does not succeed, you can examine the Block Storage backup service log file for error messages that can help you to determine the reason. Procedure Find the Block Storage backup service log file on the Controller node where the backup service is running. This log file is located in the following path: /var/log/containers/cinder/cinder-backup.log . 3.2.3. Volume backup workflow The following diagram and explanation describe the steps that occur when the user requests the cinder API to back up a Block Storage (cinder) volume. Figure 3.1. Creating a backup of a Block Storage volume The user issues a request to the cinder API, which is a REST API, to back up a Block Storage volume. The cinder API receives the request from HAProxy and validates the request, the user credentials, and other information. The cinder API creates the backup record in the SQL database. The cinder API makes an RPC call to the cinder-scheduler . The cinder-scheduler makes an asynchronous RPC call to the cinder-backup service via AMQP to back up the volume. The cinder API returns the current backup record, with an ID, to the API caller. An RPC create message arrives on one of the backup services. The cinder-backup service performs a synchronous RPC call to get_backup_device . The cinder-volume service ensures that the correct device is returned to the caller. Normally, it is the same volume, but if the volume is in use, the service returns a temporary cloned volume or a temporary snapshot, depending on the configuration. The cinder-backup service issues another synchronous RPC to cinder-volume to expose the source device. The cinder-volume service exports and maps the source device (volume or snapshot) and returns the appropriate connection information. The cinder-backup service attaches the source device by using the connection information. The cinder-backup service calls the backup back end driver, with the device already attached, which begins the data transfer to the backup repository. The source device is detached from the Backup host. The cinder-backup service issues a synchronous RPC to cinder-volume to disconnect the source device. The cinder-volume service unmaps and removes the export for the device. If a temporary volume or temporary snapshot was created, cinder-backup calls cinder-volume to remove it. The cinder-volume service removes the temporary volume. When the backup is completed, the backup record is updated in the database. 3.2.4. Volume restore workflow The following diagram and explanation describe the steps that occur when the user requests the cinder API to restore a Block Storage service (cinder) backup. Figure 3.2. Restoring a Block Storage backup The user issues a request to the cinder API, which is a REST API, to restore a Block Storage backup. The cinder API receives the request from HAProxy and validates the request, the user credentials, and other information. If the request does not contain an existing volume as the destination, the cinder API makes an asynchronous RPC call to create a new volume and polls the status of the volume until it becomes available. The cinder-scheduler selects a volume service and makes the RPC call to create the volume. The selected cinder-volume service creates the volume. When the cinder API detects that the volume is available, the backup record is updated in the database. The cinder API makes an asynchronous RPC call to the backup service via AMQP to restore the backup. The cinder API returns the current volume ID, backup ID, and volume name to the API caller. An RPC create message arrives on one of the backup services. The cinder-backup service performs a synchronous RPC call to cinder-volume to expose the volume. The cinder-volume service exports and maps the volume returning the appropriate connection information. The cinder-backup service attaches the volume by using the connection information. The cinder-backup service calls the back end driver with the volume already attached, which begins the data restoration to the volume. The volume is detached from the backup host. The cinder-backup service issues a synchronous RPC to cinder-volume to disconnect the volume. The cinder-volume service unmaps and removes the export for the volume. When the volume is restored, the backup record is updated in the database.

Commands:
openstack flavor list --os-cloud <cloud_name>
`export OS_CLOUD=<cloud_name>`
openstack --os-project-name <projectname> --os-username <username> --os-password <password> volume backup create [--name <backup_name>] <volume>
oc rsh -n openstack openstackclient
openstack volume list
openstack volume backup create [--name <backup_name>] <volume>
openstack volume backup create --name vol1bu2 vol_1 +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | | name | vol1bu2 | +-------+--------------------------------------+
openstack volume backup list
+--------------------------------------+---------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+---------+-------------+-----------+------+ | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | vol1bu2 | None | available | 1 | | b604a932-94a5-468e-bf6b-841ac16f69a8 | None | None | available | 1 | +--------------------------------------+---------+-------------+-----------+------+
exit
oc rsh -n openstack openstackclient
openstack volume snapshot list
openstack volume snapshot show <snapshot>
openstack volume snapshot show snap_1 +--------------------------------------------+--------------------------------------+ | Field | Value | +--------------------------------------------+--------------------------------------+ | created_at | 2023-07-18T08:14:16.000000 | | description | None | | id | 2d95b707-6657-49af-ac8f-f9a7417d4650 | | name | snap_1 | | os-extended-snapshot-attributes:progress | 100% | | os-extended-snapshot-attributes:project_id | c2c1da89ed1648fc8b4f35a045f8d34c | | properties | | | size | 1 | | status | available | | updated_at | 2023-07-18T08:14:17.000000 | | volume_id | 6841e3d1-8a1a-4496-bc51-f7c04b787e8f | +--------------------------------------------+--------------------------------------+
openstack volume backup create [--name <backup_name>] --snapshot <snapshot> <volume_id>
openstack volume backup create --name snap1bu1 --snapshot snap_1 6841e3d1-8a1a-4496-bc51-f7c04b787e8f +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 867e6cfb-9be7-47fa-8a79-221b0e80c757 | | name | snap1bu1 | +-------+--------------------------------------+
openstack volume backup list
+--------------------------------------+------------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+------------+-------------+-----------+------+ | 867e6cfb-9be7-47fa-8a79-221b0e80c757 | snap1bu1 | None | available | 1 | +--------------------------------------+------------+-------------+-----------+------+
exit
oc rsh -n openstack openstackclient
openstack volume list
+--------------------------------------+----------------+-----------+------+--------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------------+-----------+------+--------------------------------+ | 6841e3d1-8a1a-4496-bc51-f7c04b787e8f | vol_1 | available | 1 | | | 92800cf6-82ae-448a-a2bb-872fa4d98099 | Pansible_vol_2 | in-use | 1 | Attached to inst1 on /dev/vdc | +--------------------------------------+----------------+-----------+------+--------------------------------+
openstack volume backup create [--name <backup_name>] --force <volume>
openstack volume backup create --name panvol2bu1 --force Pansible_vol_2 +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 8c72bbf3-eb8e-4459-83e9-c7654ebe6343 | | name | panvol2bu1 | +-------+--------------------------------------+
openstack volume backup list
+--------------------------------------+------------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+------------+-------------+-----------+------+ | 8c72bbf3-eb8e-4459-83e9-c7654ebe6343 | panvol2bu1 | None | available | 1 | +--------------------------------------+------------+-------------+-----------+------+
exit
oc rsh -n openstack openstackclient
openstack volume list
openstack volume backup create --incremental [--name <backup_name>] <volume>
openstack volume backup create --name vol3incbu1 --incremental vol_3 +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | f1681313-b5ed-4520-9b63-5b533f7cdc11 | | name | vol3incbu1 | +-------+--------------------------------------+
openstack volume backup list
+--------------------------------------+---------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+---------+-------------+-----------+------+ | f1681313-b5ed-4520-9b63-5b533f7cdc11 | vol3incbu1 | None | available | 1 | | f0e9ba67-67e1-4c2c-96ce-221df75bf2c2 | vol3bu1 | None | available | 1 | +--------------------------------------+---------+-------------+-----------+------+
exit
oc rsh -n openstack openstackclient
openstack volume backup list
openstack volume backup delete --force <backup>
openstack volume backup show <backup>
exit
oc rsh -n openstack openstackclient
openstack volume backup list
openstack volume backup record export -f yaml <backup> > <filename>.yaml
openstack volume backup record export -f yaml vol1bu2 > vol1bu2.yaml
openstack volume backup record export -f yaml <backup>
openstack volume backup record export -f yaml vol1bu2 backup_service: cinder.backup.drivers.ceph.CephBackupDriver backup_url: eyJkcml2 ... YWxzZX0=
exit
oc rsh -n openstack openstackclient
openstack volume backup record import <backup_service> <backup_url>
openstack volume backup record import cinder.backup.drivers.ceph.CephBackupDriver eyJkcml2 ... YWxzZX0= +-------+--------------------------------------+ | Field | Value | +-------+--------------------------------------+ | id | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | | name | vol1bu2 | +-------+--------------------------------------+
exit
oc rsh -n openstack openstackclient
openstack volume backup list
+--------------------------------------+---------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+---------+-------------+-----------+------+ | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | vol1bu2 | None | available | 1 |
openstack volume list
+--------------------------------------+----------------+-----------+------+--------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------------+-----------+------+--------------------------------+ | 654e2be8-bc79-4528-96a7-5f773d31c201 | vol_3 | available | 1 | |
openstack volume backup restore <backup> <volume>
openstack volume backup restore vol1bu2 vol_3 +-------------+--------------------------------------+ | Field | Value | +-------------+--------------------------------------+ | backup_id | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | | volume_id | 654e2be8-bc79-4528-96a7-5f773d31c201 | | volume_name | vol_3 | +-------------+--------------------------------------+
openstack volume backup delete <backup>
exit
oc rsh -n openstack openstackclient
openstack volume backup list
+--------------------------------------+---------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+---------+-------------+-----------+------+ | 83dadc43-2aa9-4c0b-bc05-a12203a8f4cb | vol1bu2 | None | available | 1 |
openstack volume backup restore <backup> [<volume>]
openstack volume list
+--------------------------------------+----------------+-----------+------+--------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------------+-----------+------+--------------------------------+ | 296c853c-c749-4eb6-857a-57ec182232a6 | vol_1 | available | 1 | |
openstack volume backup delete <backup>
exit
oc rsh -n openstack openstackclient
openstack volume backup list
openstack volume backup set --state error <backup>
openstack volume backup list
exit