Project: Longhorn (Runtime / Cloud Native Storage)
File: 20210216-volume-live-migration.md

To enable Harvester to utilize Kubevirts live migration support, we need to allow for volume live migration, so that a Kubevirt triggered migration will lead to a volume migration from the old node to the new node. https://github.com/longhorn/longhorn/issues/2127 https://github.com/rancher/harvester/issues/384 https://github.com/longhorn/longhorn/issues/87 Support Harvester VM live migration Using multiple engines for faster volume failover for other scenarios than live migration We want to add volume migration support so that we can use the VM live migration support of Kubevirt via Harvester. By limiting this feature to that specific use case we can use the csi drivers attach / detach flow to implement migration interactions. To do this, we need to be able to start a second engine for a volume on a different node that uses matching replicas of the first engine. We only support this for a volume while it is used with `volumeMode=BLOCK`, since we don't support concurrent writes and having kubernetes mount a filesystem even in read only mode can potentially lead to a modification of the filesystem (metadata, access time, journal replay, etc). Previously the only way to support live migration in Harvester was using a Longhorn RWX volume that meant dealing with NFS and it's problems, instead we want to add support for live migration for a traditional Longhorn volume this was previously implemented for the old RancherVM. After this enhancement Longhorn will support a special `migratable` flag that allows for a Longhorn volume to be live migrated from one node to another. The assumption here is that the initial consumer will never write again to the block device once the new consumer takes over. To test one needs to create a storage class with `migratable: "true"` set as a parameter. Afterwards an RWX PVC is necessary since migratable volumes need to be able to be attached to multiple nodes. ```yaml kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: longhorn-migratable provisioner: driver.longhorn.io allowVolumeExpansion: true parameters: numberOfReplicas: "3" staleReplicaTimeout: "2880" # 48 hours in minutes fromBackup: "" migratable: "true" ``` We use CirOS as our test image for the live migration. The login account is `cirros` and the password is `gocubsgo`. To test with Harvester one can use the below example yamls as a quick start. Deploy the below yaml so that Harvester will download the CirrOS image into the local Minio store. NOTE: The CirrOS servers don't support the range request which Kubevirt importer uses, which is why we let harvester download the image first. ```yaml apiVersion: harvester.cattle.io/v1alpha1 kind: VirtualMachineImage metadata: name: image-jxpnq namespace: default spec: displayName: cirros-0.4.0-x86_64-disk.img url: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img ``` Afterwards deploy the

to create a live migratable virtual machine. ```yaml apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachine metadata: labels: harvester.cattle.io/creator: harvester name: cirros-rwx-blk spec: dataVolumeTemplates: apiVersion: cdi.kubevirt.io/v1alpha1 kind: DataVolume metadata: annotations: cdi.kubevirt.io/storage.import.requiresScratch: "true" name: cirros-rwx-blk spec: pvc: accessModes: ReadWriteMany resources: requests: storage: 8Gi storageClassName: longhorn-migratable volumeMode: Block source: http: certConfigMap: importer-ca-none url: http://minio.harvester-system:9000/vm-images/image-jxpnq # locally downloaded cirros image running: true template: metadata: annotations: harvester.cattle.io/diskNames: '["cirros-rwx-blk"]' harvester.cattle.io/imageId: default/image-jxpnq labels: harvester.cattle.io/creator: harvester harvester.cattle.io/vmName: cirros-rwx-blk spec: domain: cpu: cores: 1 sockets: 1 threads: 1 devices: disks: disk: bus: virtio name: disk-0 inputs: [] interfaces: masquerade: {} model: virtio name: default machine: type: q35 resources: requests: memory: 128M hostname: cirros-rwx-blk networks: name: default pod: {} volumes: dataVolume: name: cirros-rwx-blk name: disk-0 ``` Once the `cirros-rwx` virtual machine is up and running deploy the `cirros-rwx-migration.yaml` to initiate a virtual machine live migration. ```yaml apiVersion: kubevirt.io/v1alpha3 kind: VirtualMachineInstanceMigration metadata: name: cirros-rwx-blk spec: vmiName: cirros-rwx-blk ``` volume detach call now expects a `detachInput { hostId: "" }` if `hostId==""` it will be treated as detach from all nodes same behavior as before. csi driver now calls volume attach/detach for all volume types: RWO, RWX (NFS), RWX (Migratable). the api volume-manager now determines, whether attach/detach is necessary and valid instead of the csi driver. If a volume is already attached (to the requested node) we will return the current volume. If a volume is mode RWO, it will be attached to the requested node, unless it's attached already to a different node. If a volume is mode RWX (NFS), it will only be attached when requested in maintenance mode. Since in other cases the volume is controlled by the share-manager. If a volume is mode RWX (Migratable), will initially be attached to the requested node unless already attached, at which point a migration will be initiated to the new node. If a volume is already detached (from all, from the requested node) we will return the current volume. If a volume is mode RWO, It will be detached from the requested node. If a volume is mode RWX (NFS), it will only be detached if it's currently attached in maintenance mode. Since in other cases the volume is controlled by the share-manager. If a volume is mode RWX (Migratable) It will be detached from the requested node. if a migration is in progress then depending on the requested node to detach different migration actions will happen. A migration confirmation will be triggered if the detach request is for the first node. A migration rollback will be triggered if the detach request is for the second

The live migration intention is triggered and evaluated via the attach/detach calls. The expectation is that Kubernetes will bring up a new pod that requests attachment of the already attached volume. This will initiate the migration start, after this there are two things that can happen. Either Kubernetes will terminate the new pod which is equivalent to a migration rollback, or the old pod will be terminated which is equivalent to a migration complete operation. Users launch a new VM with a new migratable Longhorn volume -> A migratable volume is created then attached to node1. Similar to regular attachment, Longhorn will set `v.spec.nodeID` to `node1` here. Users launch the 2nd VM (pod) with the same Longhorn volume -> Kubernetes requests that the volume (already attached) be attached to node2. Then Longhorn receives the attach call and set `v.spec.migrationNodeID` to `node2` with `v.spec.nodeID = node1`. Longhorn volume-controller brings up the new engine on node2, with inactive matching replicas (same as live engine upgrade) Longhorn CSI driver polls for the existence of the second engine on node2 before acknowledging attachment success. Once the migration is started (running engines on both nodes), the following detach decides whether migration is completed successfully, or a migration rollback is desired: If succeeded: Kubevirt will remove the original pod on `node1`, this will lead to requesting detachment from node1, which will lead to longhorn setting `v.spec.nodeID` to `node2` and unsetting `v.spec.migrationNodeID` If failed: Kubevirt will terminate the new pod on `node2`, this will lead to requesting detachment from node2, which will lead to longhorn keeping `v.spec.nodeID` to `node1` and unsetting `v.spec.migrationNodeID` Longhorn volume controller then cleans up the second engine and switches the active replicas to be the current engine ones. In summary: ``` n1 | vm1 has the volume attached (v.spec.nodeID = n1) n2 | vm2 requests attachment [migrationStart] -> (v.spec.migrationNodeID = n2) volume-controller brings up new engine on n2, with inactive matching replicas (same as live engine upgrade) csi driver polls for existence of second engine on n2 before acknowledging attach The following detach decides whether a migration is completed successfully, or a migration rollback is desired. n1 | vm1 requests detach of n1 [migrationComplete] -> (v.spec.nodeID = n2, v.spec.migrationNodeID = "") n2 | vm2 requests detach of n2 [migrationRollback] -> (v.spec.NodeID = n1, v.spec.migrationNodeID = "") The volume controller then cleans up the second engine and switches the active replicas to be the current engine ones. ``` E2E test for migration successful E2E test for migration rollback Requires using a storage class with `migratable: "true"` parameter for the harvester volumes as well as an RWX PVC to allow live migration in Kubernetes/Kubevirt.