Project: Longhorn (Runtime / Cloud Native Storage)
File: 20230417-extend-csi-snapshot-to-support-backingimage.md

Extend CSI snapshot to support Longhorn BackingImage In Longhorn, we have BackingImage for VM usage. We would like to extend the CSI Snapshotter to support BackingImage management. Extend the CSI snapshotter to support: Create Longhorn BackingImage Delete Longhorn BackingImage Creating a new PVC from CSI snapshot that is associated with a Longhorn BackingImage Can support COW over each relative base image for delta data transfer for better space efficiency. (Will be in next improvement) User can backup a BackingImage based volume and restore it in another cluster without manually preparing BackingImage in a new cluster. With this improvement, users can use standard CSI VolumeSnapshot as the unified interface for BackingImage creation, deletion and restoration of a Volume. To use this feature, users need to deploy the CSI snapshot CRDs and related Controller The instructions are already on our document: https://longhorn.io/docs/1.4.1/snapshots-and-backups/csi-snapshot-support/enable-csi-snapshot-support/ Create a VolumeSnapshotClass with type `bi` which refers to BackingImage ```yaml kind: VolumeSnapshotClass apiVersion: snapshot.storage.k8s.io/v1 metadata: name: longhorn-snapshot-vsc driver: driver.longhorn.io deletionPolicy: Delete parameters: type: bi export-type: qcow2 # default to raw if it is not provided ``` Users can create a BackingImage of a Volume by creation of VolumeSnapshot. Example below for a Volume named `test-vol` ```yaml apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: test-snapshot-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc source: persistentVolumeClaimName: test-vol ``` Longhorn will create a BackingImage exported from this Volume. Users can create a volume based on a prior created VolumeSnapshot. Example below for a Volume named `test-vol-restore` ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-vol-restore spec: storageClassName: longhorn dataSource: name: test-snapshot-backing kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: ReadWriteOnce resources: requests: storage: 5Gi ``` Longhorn will create a Volume based on the BackingImage associated with the VolumeSnapshot. Users can request the creation of a Volume based on a prior BackingImage which was not created via the CSI VolumeSnapshot. With the BackingImage already existing, users need to create the VolumeSnapshotContent with an associated VolumeSnapshot. The `snapshotHandle` of the VolumeSnapshotContent needs to point to an existing BackingImage. Example below for a Volume named `test-restore-existing-backing` and an existing BackingImage `test-bi` For pre-provisioning, users need to provide following query parameters: `backingImageDataSourceType`: `sourceType` of existing BackingImage, e.g. `export-from-volume`, `download` `backingImage`: Name of the BackingImage you should also provide the `sourceParameters` of existing BackingImage in the `snapshotHandle` for validation. `export-from-volume`: you should provide `volume-name` `export-type` `download`: you should proviide `url` `checksum`: optional ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotContent metadata: name: test-existing-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc driver: driver.longhorn.io deletionPolicy: Delete source: snapshotHandle: bi://backing?backingImageDataSourceType=export-from-volume&backingImage=test-bi&volume-name=vol-export-src&export-type=qcow2 volumeSnapshotRef: name: test-snapshot-existing-backing namespace: default ``` ```yaml apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: test-snapshot-existing-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc source: volumeSnapshotContentName: test-existing-backing ``` ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-restore-existing-backing spec: storageClassName: longhorn dataSource: name: test-snapshot-existing-backing kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: ReadWriteOnce resources: requests: storage: 5Gi ``` Longhorn will create a Volume based on the BackingImage associated with the VolumeSnapshot and the VolumeSnapshotContent. Users can request the creation of a Volume based on a BackingImage which was not created yet with following 2 kinds of data sources. `download`: Download a file from a URL as a BackingImage. `export-from-volume`: Export an existing in-cluster volume as a backing

Users need to create the VolumeSnapshotContent with an associated VolumeSnapshot. The `snapshotHandle` of the VolumeSnapshotContent needs to provide the parameters for the data source. Example below for a volume named `test-on-demand-backing` and an non-existing BackingImage `test-bi` with two different data sources. `download`: Users need to provide following parameters `backingImageDataSourceType`: `download` for on-demand download. `backingImage`: Name of the BackingImage `url`: The file from a URL as a BackingImage. `backingImageChecksum`: Optional. Used for checking the checksum of the file. example yaml: ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotContent metadata: name: test-on-demand-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc driver: driver.longhorn.io deletionPolicy: Delete source: snapshotHandle: bi://backing?backingImageDataSourceType=download&backingImage=test-bi&url=https%3A%2F%2Flonghorn-backing-image.s3-us-west-1.amazonaws.com%2Fparrot.qcow2&backingImageChecksum=bd79ab9e6d45abf4f3f0adf552a868074dd235c4698ce7258d521160e0ad79ffe555b94e7d4007add6e1a25f4526885eb25c53ce38f7d344dd4925b9f2cb5d3b volumeSnapshotRef: name: test-snapshot-on-demand-backing namespace: default ``` `export-from-volume`: Users need to provide following parameters `backingImageDataSourceType`: `export-form-volume` for on-demand export. `backingImage`: Name of the BackingImage `volume-name`: Volume to be exported for the BackingImage `export-type`: Currently Longhorn supports `raw` or `qcow2` example yaml: ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotContent metadata: name: test-on-demand-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc driver: driver.longhorn.io deletionPolicy: Delete source: snapshotHandle: bi://backing?backingImageDataSourceType=export-from-volume&backingImage=test-bi&volume-name=vol-export-src&export-type=qcow2 volumeSnapshotRef: name: test-snapshot-on-demand-backing namespace: default ``` Users then can create corresponding VolumeSnapshot and PVC ```yaml apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: test-snapshot-on-demand-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc source: volumeSnapshotContentName: test-on-demand-backing ``` ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-restore-on-demand-backing spec: storageClassName: longhorn dataSource: name: test-snapshot-on-demand-backing kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: ReadWriteOnce resources: requests: storage: 5Gi ``` No changes necessary We add a new type `bi` to the parameter `type` in the VolumeSnapshotClass. It means that the CSI VolumeSnapshot created with this VolumeSnapshotClass is associated with a Longhorn BackingImage. When the users create VolumeSnapshot and the volumeSnapshotClass `type` is `bi` ```yaml apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: test-snapshot-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc source: persistentVolumeClaimName: test-vol ``` We do: Get the name of the Volume The name of the BackingImage will be same as the VolumeSnapshot `test-snapshot-backing`. Check if a BackingImage with the same name as the requested VolumeSnapshot already exists. Return success without creating a new BackingImage. Create a BackingImage. Get `export-type` from VolumeSnapshotClass parameter `export-type`, default to `raw.` Encode the `snapshotId` as `bi://backing?backingImageDataSourceType=export-from-volume&backingImage=test-snapshot-backing&volume-name=${VolumeName}&export-type=raw` This `snaphotId` will be used in the later CSI CreateVolume and DeleteSnapshot call. If VolumeSource type is `VolumeContentSource_Snapshot`, decode the `snapshotId` to get the parameters. `bi://backing?backingImageDataSourceType=${TYPE}&backingImage=${BACKINGIMAGENAME}&backingImageChecksum=${backingImageChecksum}&${OTHERPARAMETES}` If BackingImage with the given name already exists, create the volume. If BackingImage with the given name does not exists, we prepare it first. There are 2 kinds of types which are `export-from-volume` and `download`. For `download`, it means we have to prepare the BackingImage before creating the Volume. We first decode other parameters from `snapshotId` and create the BackingImage. For `export-from-volume`, it means we have to prepare the BackingImage before creating the Volume. We first decode other parameters from `snapshotId` and create the BackingImage. NOTE: we already have related code for preparing the BackingImage with type `download` or `export-from-volume` before creating a Volume, Decode the `snapshotId` to get the name of the BackingImage. Then we delete the BackingImage directly. Integration test plan. Deploy the csi snapshot CRDs, Controller as instructed at https://longhorn.io/docs/1.4.1/snapshots-and-backups/csi-snapshot-support/enable-csi-snapshot-support/ Create a VolumeSnapshotClass with type `bi` ```yaml kind: VolumeSnapshotClass apiVersion: snapshot.storage.k8s.io/v1 metadata: name: longhorn-snapshot-vsc driver: driver.longhorn.io deletionPolicy: Delete parameters: type: bi ``` Success Create a Volume `test-vol` of 5GB. Create PV/PVC for the

Create a workload using the Volume. Write some data to the Volume. Create a VolumeSnapshot with following yaml: ```yaml apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: test-snapshot-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc source: persistentVolumeClaimName: test-vol ``` Verify that BacingImage is created. Verify the properties of BackingImage `sourceType` is `export-from-volume` `volume-name` is `test-vol` `export-type` is `raw` Delete the VolumeSnapshot `test-snapshot-backing` Verify the BacingImage is deleted Create a Volume `test-vol` of 5GB. Create PV/PVC for the Volume. Create a workload using the Volume. Write some data to the Volume. Create a VolumeSnapshot with following yaml: ```yaml apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: test-snapshot-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc source: persistentVolumeClaimName: test-vol ``` Verify that BacingImage is created. Create a new PVC with following yaml: ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-restore-pvc spec: storageClassName: longhorn dataSource: name: test-snapshot-backing kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: ReadWriteOnce resources: requests: storage: 5Gi ``` Attach the PVC `test-restore-pvc` to a workload and verify the data Delete the PVC Create a BackingImage `test-bi` using longhorn test raw image `https://longhorn-backing-image.s3-us-west-1.amazonaws.com/parrot.qcow2` Create a VolumeSnapshotContent with `snapshotHandle` pointing to BackingImage `test-bi` and provide the parameters. ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotContent metadata: name: test-existing-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc driver: driver.longhorn.io deletionPolicy: Delete source: snapshotHandle: bi://backing?backingImageDataSourceType=download&backingImage=test-bi&url=https%3A%2F%2Flonghorn-backing-image.s3-us-west-1.amazonaws.com%2Fparrot.qcow2&backingImageChecksum=bd79ab9e6d45abf4f3f0adf552a868074dd235c4698ce7258d521160e0ad79ffe555b94e7d4007add6e1a25f4526885eb25c53ce38f7d344dd4925b9f2cb5d3b volumeSnapshotRef: name: test-snapshot-existing-backing namespace: default ``` Create a VolumeSnapshot associated with the VolumeSnapshotContent ```yaml apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: test-snapshot-existing-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc source: volumeSnapshotContentName: test-existing-backing ``` Create a PVC with the following yaml ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-restore-existing-backing spec: storageClassName: longhorn dataSource: name: test-snapshot-existing-backing kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: ReadWriteOnce resources: requests: storage: 5Gi ``` Attach the PVC `test-restore-existing-backing` to a workload and verify the data Type `download` Create a VolumeSnapshotContent with `snapshotHandle` providing the required parameters and BackingImage name `test-bi` ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotContent metadata: name: test-on-demand-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc driver: driver.longhorn.io deletionPolicy: Delete source: snapshotHandle: bi://backing?backingImageDataSourceType=download&backingImage=test-bi&url=https%3A%2F%2Flonghorn-backing-image.s3-us-west-1.amazonaws.com%2Fparrot.qcow2&backingImageChecksum=bd79ab9e6d45abf4f3f0adf552a868074dd235c4698ce7258d521160e0ad79ffe555b94e7d4007add6e1a25f4526885eb25c53ce38f7d344dd4925b9f2cb5d3b volumeSnapshotRef: name: test-snapshot-on-demand-backing namespace: default ``` Create a VolumeSnapshot associated with the VolumeSnapshotContent ```yaml apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: test-snapshot-on-demand-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc source: volumeSnapshotContentName: test-on-demand-backing ``` Create a PVC with the following yaml ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-restore-on-demand-backing spec: storageClassName: longhorn dataSource: name: test-snapshot-on-demand-backing kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: ReadWriteOnce resources: requests: storage: 5Gi ``` Verify BackingImage `test-bi` is created Attach the PVC `test-restore-on-demand-backing` to a workload and verify the data Type `export-from-volume` Success Create a Volme `test-vol` and write some data to it. Create a VolumeSnapshotContent with `snapshotHandle` providing the required parameters and BackingImage name `test-bi` ```yaml apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotContent metadata: name: test-on-demand-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc driver: driver.longhorn.io deletionPolicy: Delete source: snapshotHandle: bi://backing?backingImageDataSourceType=export-from-volume&backingImage=test-bi&volume-name=test-vol&export-type=qcow2 volumeSnapshotRef: name: test-snapshot-on-demand-backing namespace: default ``` Create a VolumeSnapshot associated with the VolumeSnapshotContent ```yaml apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: test-snapshot-on-demand-backing spec: volumeSnapshotClassName: longhorn-snapshot-vsc source: volumeSnapshotContentName: test-on-demand-backing ``` Create a PVC with the following yaml ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: test-restore-on-demand-backing spec: storageClassName: longhorn dataSource: name: test-snapshot-on-demand-backing kind: VolumeSnapshot apiGroup: snapshot.storage.k8s.io accessModes: ReadWriteOnce resources: requests: storage: 5Gi ``` Verify BackingImage `test-bi` is created Attach the PVC `test-restore-on-demand-backing` to a workload and verify the data No upgrade strategy needed We need to update the docs and examples to reflect the new type of parameter `type` in the VolumeSnapshotClass.