Project: StratoVirt (Runtime / Container Runtime)
File: migration.md

Virtual machine live migration is the key feature provided by StratoVirt. It needs to execute virtual machine migration when any of the following happens: Server overload: when a source server is overloaded, a set of the VMs from this server is migrated to an underloaded server using VM migration technique. Server maintenance: if there is a need for server maintenance, VMs from the source server are migrated to another server. Server fault: whenever there is server fault, VMs are migrated from the faulty server to the target server. The migration stream can be passed over any transport as following: TCP mode migration: using tcp sockets to do the migration. UNIX mode migration: using unix sockets to do the migration. Note: UNIX mode only supports migrate two VMs on the same host OS. TCP mode supports migrate both on the same or different host OS. Launch the source VM: ```shell ./stratovirt \ -machine q35 \ -kernel ./vmlinux.bin \ -append "console=ttyS0 pci=off reboot=k quiet panic=1 root=/dev/vda" \ -drive file=path/to/rootfs,id=rootfs,readonly=off,direct=off \ -device virtio-blk-pci,drive=rootfs,id=rootfs,bus=pcie.0,addr=0 \ -qmp unix:path/to/socket1,server,nowait \ -serial stdio \ ``` Launch the destination VM: ```shell ./stratovirt \ -machine q35 \ -kernel ./vmlinux.bin \ -append "console=ttyS0 pci=off reboot=k quiet panic=1 root=/dev/vda" \ -drive file=path/to/rootfs,id=rootfs,readonly=off,direct=off \ -device virtio-blk-pci,drive=rootfs,id=rootfs,bus=pcie.0,addr=0 \ -qmp unix:path/to/socket2,server,nowait \ -serial stdio \ -incoming tcp:192.168.0.1:4446 \ ``` Note: The destination VM command line parameter needs to be consistent with the source VM. If it is necessary to change the data transmission from tcp network protocol to unix socket, the parameter `-incoming tcp:192.168.0.1:4446` needs to be replaced with `-incoming unix:/tmp/stratovirt-migrate.socket`. Unix socket protocol only supports migrate two VMs on the same host OS. Start to send migration for the source VM: ```shell $ ncat -U path/to/socket1 <- {"QMP":{"version":{"StratoVirt":{"micro":1,"minor":0,"major":0},"package":""},"capabilities":[]}} -> {"execute":"migrate",

<- {"return":{}} ``` Note: If using unix socket protocol to migrate vm, you need to modify QMP command of `"uri":"tcp:192.168.0.1:4446"` to `"uri":"unix:/tmp/stratovirt-migrate.socket"`. When finish executing the command line, the live migration is start. in a moment, the source VM should be successfully migrated to the destination VM. If you want to cancel the live migration, executing the following command: ```shell $ ncat -U path/to/socket1 <- {"QMP":{"version":{"StratoVirt":{"micro":1,"minor":0,"major":0},"package":""},"capabilities":[]}} -> {"execute":"migrate_cancel"} <- {"return":{}} ``` Use QMP command `query-migrate` to check migration state: ```shell $ ncat -U path/to/socket <- {"QMP":{"version":{"StratoVirt":{"micro":1,"minor":0,"major":0},"package":""},"capabilities":[]}} -> {"execute":"query-migrate"} <- {"return":{"status":"completed"}} ``` Now there are 5 states during migration: `None`: Resource is not prepared all. `Setup`: Resource is setup, ready to migration. `Active`: In migration. `Completed`: Migration completed. `Failed`: Migration failed. `Canceled`: Migration canceled. Migration supports machine type: `q35` (on x86_64 platform) `virt` (on aarch64 platform) Some devices and feature don't support to be migration yet: `vhost-net` `vhost-user-net` `vfio` devices `balloon` `mem-shared`,`backend file of memory` `pmu` `sve` `gic-version=2` Some device attributes can't be changed: `virtio-net`: mac `virtio-blk`: file(only ordinary file or copy file), serial_num `device`: bus, addr `smp` `m` If hot plug device before migrate source vm, add newly replaced device command should be add to destination vm. Before live migration: source and destination host CPU needs to be the same architecture. the VMs image needs to be shared by source and destination. live migration may fail if the VM is performing lifecycle operations, such as reboot, shutdown. the command to startup the VM needs to be consistent on source and destination host. During live migration: source and destination networks cannot be disconnected. it is banned to operate VM lifecycle, includes using the QMP command and executing in the VM. live migration time is affected by network performance, total memory of VM and applications. After live migration: it needs to wait for the source VM to release resources before fetching back the live migration operation.