Project: Longhorn (Runtime / Cloud Native Storage)
File: 20230809-support-backup-and-restore-for-volumes-with-v2-data-engine.md

Support backup and full restore for volumes with v2 data engine. Leverage the existing backup and restore frameworks Back up volumes with v2 data engine Fully restore volumes with v2 data engine Support fully restoring v2 volume backup to v1/v2 volume Incremental restore volumes with v2 data engine Backup and restore are essential for data recovery. Longhorn system can back up volumes with v1 data engine. However, it does not support snapshot backup for v2 data engine yet. For better user experience, backup and restore operations should be consistent across both data engines. Additionally, a v2 volume can support restoring to an either v1 or v2 volume. Backup and restore operations should be consistent across both data engines. engine-proxy Add `BackupRestoreFinish` for wrapping up the volume restore spdk Add `bdevlvolget_fragmap` API Get a fragmap for a specific segment of a logical volume (lvol) using the provided offset and size. A fragmap is a bitmap that records the allocation status of clusters. A value of "1" indicates that a cluster is allocated, whereas "0" signifies that a cluster is unallocated. Add `creation_time` xattr in lvols The `creation_time` is represented as a snapshot creation time. Add `bdevlvolgetxattr` and `bdevlvolsetxattr` spdk-go-helper Add `BdevLvolGetFragmap` for `bdevlvolget_fragmap` Add `BdevGetXattr` for `bdevlvolget_xattr` Add `BdevSetXattr` for `bdevlvolset_xattr` In `UI > Backup > Volume Operations > Restore` page, add `Data Engine` option. Longhorn can incrementally back up volumes that use the v1 data engine. Incremental backups are based on fiemap, which are file extent mappings of the disk files. This means that only the changes to the volume since the last backup are backed up, which can save time and space. However, snapshots of volumes that use the v2 data engine are SPDK lvols, which do not support fiemap. Instead, we can use the concept of fiemap to leverage the fragmentation map (fragmap) instead. A fragmap is a mapping of data and hole segments in a volume. This means that we can use the fragmap to identify the changed segments of a v2 volume and back them up, even though fiemap is not supported. Here are some additional details about fiemap and fragmaps: fiemap is a Linux kernel API that allows applications to query the file system for the extents of a file. This can be used to efficiently back up files, as only the changed extents need to be backed up. fragmap is a data structure that stores the fragmentation information for a volume. This information includes the size and location of each data segment in the volume, as well as the size and location of any holes in the

spdk_tgt Add `bdevlvolget_fragmap` JSON-RPC method: retrieve the fragmap for a specific portion of the specified lvol bdev Parameters lvs_name: The name of the lvstore name lvol_name: The name of the lvol. offset: offset the a specific portion of the lvol. Unit: bytes. size: size the a specific portion of the lvol. Unit: bytes. The fragmap is internally constructed by incorporating with `spdkbdevseekhole` and `spdkbdevseekdata` The size of each data segment or hole segment is the same as the cluster size of a lvstore An example of a fragmap of a lvol Backup Flow The backup flow is explained by the an example. A snapshot chain of a volume is ``` volume-head -> snap003 -> snap002 -> snap001 (backed up) ``` `snap001` is already backed up, and we are going to back up `snap003`. The steps are longhorn-manager's backup controller is aware of the backup resource for the snapshot `snapshot3` and then issues a backup request to the instance-manager by the proxy function `SnapshotBackup()` with the specified volume and snapshot names. After the instance-manager proxy service receives the request, the proxy function transfers the request to the SPDK service by `EngineBackupCreate()`. The SPDK service randomly picks one of the replicas and executes the backup by `ReplicaBackupCreate()`. Expose snapshot (`snap003`) to be a block device locally via NVMe over TCP. The data in the block device is the accumulated data of `snap003` and its descendants. Get the fragmaps of the snapshot to be backed up and its descendants prior to the last backup. In the example, we need to get the fragmaps of `snap003` and `snap002`. Retrieve the fragmap of each lvol using `bdevlvolget_fragmap` JSON-RPC method. Overlay all the fragmaps, iterate each bit, and if it's set to 1, read data from the corresponding cluster on the block device and perform a backup. Full Restore Flow The engine controller within Longhorn Manager is informed of a volume restore request. Subsequently, it initiates a restore operation to the instance manager using the BackupRestore() proxy function. This operation includes essential details such as the volume name, backup name, credentials, and more. Once the instance manager's proxy service receives the request, the `EngineRestoreCreate()` proxy function is employed to transfer the request to the SPDK service. Within `EngineRestoreCreate()`, the raid bdev is removed, followed by issuing a replica restore command to the associated replicas through the `ReplicaRestoreCreate()` function. In the `ReplicaRestoreCreate()`, the lvol of the replica is locally exposed as a block device using NVMe over TCP. The subsequent steps involve downloading data blocks and writing them to the block device according to the block mappings. Upon the successful completion of the restoration process, Longhorn Manager sends a request to finalize the restoration to the instance manager via the `EngineRestoreFinish()` function. This function facilitates the recreation of the raid bdev. Concurrent volume backup and full restore