Chapter 15. MTV performance recommendations The purpose of this section is to share recommendations for efficient and effective migration of virtual machines (VMs) using Migration Toolkit for Virtualization (MTV), based on findings observed through testing. The data provided here was collected from testing in Red Hat Labs and is provided for reference only. Overall, these numbers should be considered to show the best-case scenarios. The observed performance of migration can differ from these results and depends on several factors. 15.1. Ensure fast storage and network speeds Ensure fast storage and network speeds, both for VMware and Red Hat OpenShift (OCP) environments. To perform fast migrations, VMware must have fast read access to datastores. Networking between VMware ESXi hosts should be fast, ensure a 10 GiB network connection, and avoid network bottlenecks. Extend the VMware network to the OCP Workers Interface network environment. It is important to ensure that the VMware network offers high throughput (10 Gigabit Ethernet) and rapid networking to guarantee that the reception rates align with the read rate of the ESXi datastore. Be aware that the migration process uses significant network bandwidth and that the migration network is utilized. If other services utilize that network, it may have an impact on those services and their migration rates. For example, 200 to 325 MiB/s was the average network transfer rate from the vmnic for each ESXi host associated with transferring data to the OCP interface. 15.2. Ensure fast datastore read speeds to ensure efficient and performant migrations Datastores read rates impact the total transfer times, so it is essential to ensure fast reads are possible from the ESXi datastore to the ESXi host. Example in numbers: 200 to 300 MiB/s was the average read rate for both vSphere and ESXi endpoints for a single ESXi server. When multiple ESXi servers are used, higher datastore read rates are possible. 15.3. Endpoint types MTV 2.6 allows for the following vSphere provider options: ESXi endpoint (inventory and disk transfers from ESXi), introduced in MTV 2.6 vCenter Server endpoint; no networks for the ESXi host (inventory and disk transfers from vCenter) vCenter endpoint and ESXi networks are available (inventory from vCenter, disk transfers from ESXi). When transferring many VMs that are registered to multiple ESXi hosts, using the vCenter endpoint and ESXi network is suggested. Note As of vSphere 7.0, ESXi hosts can label which network to use for NBD transport. This is accomplished by tagging the desired virtual network interface card (NIC) with the appropriate vSphereBackupNFC label. When this is done, MTV will be able to utilize the ESXi interface for network transfer to Openshift as long as the worker and ESXi host interfaces are reachable. This is especially useful when migration users may not have access to the ESXi credentials yet would like to be able to control which ESXi interface is used for migration. For more details, see: (MTV-1230) You can use the following ESXi command, which designates interface vmk2 for NBD backup: esxcli network ip interface tag add -t vSphereBackupNFC -i vmk2 15.4. Set ESXi hosts BIOS profile and ESXi Host Power Management for High Performance Where possible, ensure that hosts used to perform migrations are set with BIOS profiles related to maximum performance. Hosts which use Host Power Management controlled within vSphere should check that High Performance is set. Testing showed that when transferring more than 10 VMs with both BIOS and host power management set accordingly, migrations had an increase of 15 MiB in the average datastore read rate. 15.5. Avoid additional network load on VMware networks You can reduce the network load on VMware networks by selecting the migration network when using the ESXi endpoint. By incorporating a virtualization provider, MTV enables the selection of a specific network, which is accessible on the ESXi hosts, for the purpose of migrating virtual machines to OCP. Selecting this migration network from the ESXi host in the MTV UI will ensure that the transfer is performed using the selected network as an ESXi endpoint. It is imperative to ensure that the network selected has connectivity to the OCP interface, has adequate bandwidth for migrations, and that the network interface is not saturated. In environments with fast networks, such as 10GbE networks, migration network impacts can be expected to match the rate of ESXi datastore reads. 15.6. Control maximum concurrent disk migrations per ESXi host Set the MAX_VM_INFLIGHT MTV variable to control the maximum number of concurrent VMs transfers allowed for the ESXi host. MTV allows for concurrency to be controlled using this variable; by default, it is set to 20. When setting MAX_VM_INFLIGHT , consider the number of maximum concurrent VMs transfers are required for ESXi hosts. It is important to consider the type of migration to be transferred concurrently. Warm migrations, which are defined by migrations of a running VM that will be migrated over a scheduled time. Warm migrations use snapshots to compare and migrate only the differences between snapshots of the disk. The migration of the differences between snapshots happens over specific intervals before a final cut-over of the running VM to OpenShift occurs. In MTV 2.6, MAX_VM_INFLIGHT reserves one transfer slot per VM, regardless of current migration activity for a specific snapshot or the number of disks that belong to a single vm. The total set by MAX_VM_INFLIGHT is used to indicate how many concurrent VM tranfers per ESXi host is allowed. Examples MAX_VM_INFLIGHT = 20 and 2 ESXi hosts defined in the provider mean each host can transfer 20 VMs. 15.7. Migrations are completed faster when migrating multiple VMs concurrently When multiple VMs from a specific ESXi host are to be migrated, starting concurrent migrations for multiple VMs leads to faster migration times. Testing demonstrated that migrating 10 VMs (each containing 35 GiB of data, with a total size of 50 GiB) from a single host is significantly faster than migrating the same number of VMs sequentially, one after another. It is possible to increase concurrent migration to more than 10 virtual machines from a single host, but it does not show a significant improvement. Examples 1 single disk VMs took 6 minutes, with migration rate of 100 MiB/s 10 single disk VMs took 22 minutes, with migration rate of 272 MiB/s 20 single disk VMs took 42 minutes, with migration rate of 284 MiB/s Note From the aforementioned examples, it is evident that the migration of 10 virtual machines simultaneously is three times faster than the migration of identical virtual machines in a sequential manner. The migration rate was almost the same when moving 10 or 20 virtual machines simultaneously. 15.8. Migrations complete faster using multiple hosts Using multiple hosts with registered VMs equally distributed among the ESXi hosts used for migrations leads to faster migration times. Testing showed that when transferring more than 10 single disk VMS, each containing 35 GiB of data out of a total of 50G total, using an additional host can reduce migration time. Examples 80 single disk VMs, containing 35 GiB of data each, using a single host took 2 hours and 43 minutes, with a migration rate of 294 MiB/s. 80 single disk VMs, containing 35 GiB of data each, using 8 ESXi hosts took 41 minutes, with a migration rate of 1,173 MiB/s. Note From the aforementioned examples, it is evident that migrating 80 VMs from 8 ESXi hosts, 10 from each host, concurrently is four times faster than running the same VMs from a single ESXi host. Migrating a larger number of VMs from more than 8 ESXi hosts concurrently could potentially show increased performance. However, it was not tested and therefore not recommended. 15.9. Multiple migration plans compared to a single large migration plan The maximum number of disks that can be referenced by a single migration plan is 500. For more details, see (MTV-1203) . When attempting to migrate many VMs in a single migration plan, it can take some time for all migrations to start. By breaking up one migration plan into several migration plans, it is possible to start them at the same time. Comparing migrations of: 500 VMs using 8 ESXi hosts in 1 plan, max_vm_inflight=100 , took 5 hours and 10 minutes. 800 VMs using 8 ESXi hosts with 8 plans, max_vm_inflight=100 , took 57 minutes. Testing showed that by breaking one single large plan into multiple moderately sized plans, for example, 100 VMS per plan, the total migration time can be reduced. 15.10. Maximum values tested for cold migrations Maximum number of ESXi hosts tested: 8 Maximum number of VMs in a single migration plan: 500 Maximum number of VMs migrated in a single test: 5000 Maximum number of migration plans performed concurrently: 40 Maximum single disk size migrated: 6 T disks, which contained 3 Tb of data Maximum number of disks on a single VM migrated: 50 Highest observed single datastore read rate from a single ESXi server: 312 MiB/second Highest observed multi-datastore read rate using eight ESXi servers and two datastores: 1,242 MiB/second Highest observed virtual NIC transfer rate to an OpenShift worker: 327 MiB/second Maximum migration transfer rate of a single disk: 162 MiB/second (rate observed when transferring nonconcurrent migration of 1.5 Tb utilized data) Maximum cold migration transfer rate of the multiple VMs (single disk) from a single ESXi host: 294 MiB/s (concurrent migration of 30 VMs, 35/50 GiB used, from Single ESXi) Maximum cold migration transfer rate of the multiple VMs (single disk) from multiple ESXi hosts: 1173MB/s (concurrent migration of 80 VMs, 35/50 GiB used, from 8 ESXi servers, 10 VMs from each ESXi) 15.11. Warm migration recommendations The following recommendations are specific to warm migrations: 15.11.1. Migrate up to 400 disks in parallel Testing involved migrating 200 VMs in parallel, with 2 disks each using 8 ESXi hosts, for a total of 400 disks. No tests were run on migration plans migrating over 400 disks in parallel, so it is not recommended to migrate over this number of disks in parallel. 15.11.2. Migrate up to 200 disks in parallel for the fastest rate Testing was successfully performed on parallel disk migrations with 200, 300, and 400 disks. There was a decrease in the precopy migration rate, approximately 25%, between the tests migrating 200 disks and those migrating 300 and 400 disks. Therefore, it is recommended to perform parallel disk migrations in groups of 200 or fewer, instead of 300 to 400 disks, unless a decline of 25% in precopy speed does not affect your cutover planning. 15.11.3. When possible, set cutover time to be immediately after a migration plan starts To reduce the overall time of warm migrations, it is recommended to set the cutover to occur immediately after the migration plan is started. This causes MTV to run only one precopy per VM. This recommendation is valid, no matter how many VMs are in the migration plan. 15.11.4. Increase precopy intervals between snapshots If you are creating many migration plans with a single VM and have enough time between the migration start and the cutover, increase the value of the controller_precopy_interval parameter to between 120 and 240 minutes, inclusive. The longer setting will reduce the total number of snapshots and disk transfers per VM before the cutover. 15.12. Maximum values tested for warm migrations Maximum number of ESXi hosts tested: 8 Maximum number of worker nodes: 12 Maximum number of VMs in a single migration plan: 200 Maximum number of total parallel disk transfers: 400, with 200 VMs, 6 ESXis, and a transfer rate of 667 MB/s Maximum number of disks on a single VM migrated: 3 Maximum number of parallel disk transfers per ESXi host: 68 Maximum transfer rate observed of a single disk with no concurrent migrations: 76.5 MB/s Maximum transfer rate observed of multiple disks from a single ESXi host: 253 MB/s (concurrent migration of 10 VMs, 1 disk each, 35/50 GiB used per disk) Total transfer rate observed of multiple disks (210) from 8 ESXi hosts: 802 MB/s (concurrent migration of 70 VMs, 3 disks each, 35/50 GiB used per disk) 15.13. Increasing asynchronous I/O (AIO) sizes and buffer counts for NBD transport mode This document describes how to change NBD transport NFC parameters for increased migration performance when using the Migration Toolkit for Virtualization (MTV) product. Warning Using AIO buffering is only suitable for Cold Migration use cases. Disable AIO settings before initializing Warm Migration. For more details, see Disabling AIO Buffer Configuration . Additional information Best Practices for NBD Transport Solution for poor NBD performance 15.13.1. Key findings The best migration performance was achieved by migrating using multiple VMs (10) on a single ESXi host with the following values: VixDiskLib.nfcAio.Session.BufSizeIn64KB=16 vixDiskLib.nfcAio.Session.BufCount=4 The following improvements were noted when using AIO buffer (Asynchronous Buffer Counts) settings: Migration time was reduced by 31.1% , from 0:24:32 to 0:16:54 . Read rate was increased from 347.83 MB/s to 504.93 MB/s . There was no significant improvement observed when using AIO buffer settings with a single VM. There was no significant improvement observed when using AIO buffer settings with multiple VMs from multiple hosts. 15.13.2. Enabling AIO buffer configuration Validating Controller Pod support for AIO values Ensure that the forklift-controller pod in the openshift-mtv namespace supports the AIO buffer values. Since the pod name prefix is dynamic, check the pod name first by running the following command: oc get pods -n openshift-mtv | grep forklift-controller | awk '{print USD1}' The example output is as follows: forklift-controller-667f57c8f8-qllnx Note This is the pod name prefix from the example: forklift-controller-667f57c8f8-qllnx Check the environment variables of the pod by running: oc get pod forklift-controller-667f57c8f8-qllnx -n openshift-mtv -o yaml Check for the following lines in the output: ... \- name: VIRT\_V2V\_EXTRA\_ARGS \- name: VIRT\_V2V\_EXTRA\_CONF\_CONFIG\_MAP ... Editing ForkliftController Configuration In the openshift-mtv namespace , edit the ForkliftController object to include the AIO buffer values by running the following command: oc edit forkliftcontroller -n openshift-mtv Add the following under the spec section: virt_v2v_extra_args: "--vddk-config /mnt/extra-v2v-conf/input.conf" virt_v2v_extra_conf_config_map: "perf" Creating a ConfigMap named perf Create the required ConfigMap using the following command: oc -n openshift-mtv create cm perf Preparing the ConfigMap content Convert the desired buffer configuration values to Base64. For example, for 16/4: echo -e "VixDiskLib.nfcAio.Session.BufSizeIn64KB=16\nvixDiskLib.nfcAio.Session.BufCount=4" | base64 The output will be similar to the following: Vml4RGlza0xpYi5uZmNBaW8uU2Vzc2lvbi5CdWZTaXplSW42NEtCPTE2CnZpeERpc2tMaWIubmZjQWlvLlNlc3Npb24uQnVmQ291bnQ9NAo= Editing the ConfigMap Update the perf ConfigMap with the Base64 string under the binaryData section, for example: apiVersion: v1 kind: ConfigMap binaryData: input.conf: Vml4RGlza0xpYi5uZmNBaW8uU2Vzc2lvbi5CdWZTaXplSW42NEtCPTE2CnZpeERpc2tMaWIubmZjQWlvLlNlc3Npb24uQnVmQ291bnQ9NAo= metadata: name: perf namespace: openshift-mtv Restarting the Forklift Controller Pod Restart the forklift-controller pod to apply the new configuration. Ensure the VIRT_V2V_EXTRA_ARGS environment variable reflects the updated settings. Verifying migration logs Run a migration plan and check the logs of the migration pod. Confirm that the AIO buffer settings are passed as parameters, particularly the --vddk-config value . For example: exec: /usr/bin/virt-v2v ... --vddk-config /mnt/extra-v2v-conf/input.conf Sample log excerpt: Buffer size calc for 16 value: (16 * 64 * 1024 = 1048576) nbdkit: vddk[1]: debug: [NFC VERBOSE] NfcAio_OpenSession: Opening an AIO session. nbdkit: vddk[1]: debug: [NFC INFO] NfcAioInitSession: Disabling read-ahead buffer since the AIO buffer size of 1048576 is >= the read-ahead buffer size of 65536. Explicitly setting flag '`NFC_AIO_SESSION_NO_NET_READ_AHEAD`' nbdkit: vddk[1]: debug: [NFC VERBOSE] NfcAioInitSession: AIO Buffer Size is 1048576 nbdkit: vddk[1]: debug: [NFC VERBOSE] NfcAioInitSession: AIO Buffer Count is 4 Note The above logs were when using debug_level = 4 Inspecting ConfigMap values Content are in the Migration Pod Log in to the migration pod and verify the buffer settings using the following command: cat /mnt/extra-v2v-conf/input.conf The example output is as follows: VixDiskLib.nfcAio.Session.BufSizeIn64KB=16 vixDiskLib.nfcAio.Session.BufCount=4 Enabling Debugging (optional) To enable debug logs, convert the configuration to Base64, including a high log level: echo -e "`VixDiskLib.nfcAio.Session.BufSizeIn64KB=16\nVixDiskLib.nfcAio.Session.BufCount=4\nVixDiskLib.nfc.LogLevel=4`" | base64 Note Adding a high log level will degrade performance and is for debugging purposes only. 15.13.3. Disabling AIO Buffer Configuration To disable the AIO buffer configuration, complete the following steps: Edit the ForkliftController Object: Remove the previously added lines from the spec section in the ForkliftController object: oc edit forkliftcontroller -n openshift-mtv Remove the following lines: virt_v2v_extra_args: "`-vddk-config /mnt/extra-v2v-conf/input.conf`" virt_v2v_extra_conf_config_map: "`perf`" Delete the ConfigMap: Remove the perf ConfigMap that was created earlier: oc delete cm perf -n openshift-mtv Restart the Forklift Controller Pod (Optional). If needed, ensure the changes take effect by restarting the forklift-controller pod. 15.13.4. Key requirements for AIO Buffer (Asynchronous Buffer Counts) support VDDK and vSphere Versions Support is based upon tests performed using the following versions: vSphere : 7.0.3 VDDK : 7.0.3 For other VDDK and vSphere versions, please check the AIO buffer support in the official VMware documentation .

Commands:
esxcli network ip interface tag add -t vSphereBackupNFC -i vmk2
get pods -n openshift-mtv | grep forklift-controller | awk '{print USD1}'
forklift-controller-667f57c8f8-qllnx
get pod forklift-controller-667f57c8f8-qllnx -n openshift-mtv -o yaml
\- name: VIRT\_V2V\_EXTRA\_ARGS \- name: VIRT\_V2V\_EXTRA\_CONF\_CONFIG\_MAP
edit forkliftcontroller -n openshift-mtv
virt_v2v_extra_args: "--vddk-config /mnt/extra-v2v-conf/input.conf" virt_v2v_extra_conf_config_map: "perf"
-n openshift-mtv create cm perf
echo -e "VixDiskLib.nfcAio.Session.BufSizeIn64KB=16\nvixDiskLib.nfcAio.Session.BufCount=4" | base64
Vml4RGlza0xpYi5uZmNBaW8uU2Vzc2lvbi5CdWZTaXplSW42NEtCPTE2CnZpeERpc2tMaWIubmZjQWlvLlNlc3Npb24uQnVmQ291bnQ9NAo=
apiVersion: v1 kind: ConfigMap binaryData: input.conf: Vml4RGlza0xpYi5uZmNBaW8uU2Vzc2lvbi5CdWZTaXplSW42NEtCPTE2CnZpeERpc2tMaWIubmZjQWlvLlNlc3Npb24uQnVmQ291bnQ9NAo= metadata: name: perf namespace: openshift-mtv
exec: /usr/bin/virt-v2v ... --vddk-config /mnt/extra-v2v-conf/input.conf
Buffer size calc for 16 value: (16 * 64 * 1024 = 1048576) nbdkit: vddk[1]: debug: [NFC VERBOSE] NfcAio_OpenSession: Opening an AIO session. nbdkit: vddk[1]: debug: [NFC INFO] NfcAioInitSession: Disabling read-ahead buffer since the AIO buffer size of 1048576 is >= the read-ahead buffer size of 65536. Explicitly setting flag '`NFC_AIO_SESSION_NO_NET_READ_AHEAD`' nbdkit: vddk[1]: debug: [NFC VERBOSE] NfcAioInitSession: AIO Buffer Size is 1048576 nbdkit: vddk[1]: debug: [NFC VERBOSE] NfcAioInitSession: AIO Buffer Count is 4
cat /mnt/extra-v2v-conf/input.conf
VixDiskLib.nfcAio.Session.BufSizeIn64KB=16 vixDiskLib.nfcAio.Session.BufCount=4
echo -e "`VixDiskLib.nfcAio.Session.BufSizeIn64KB=16\nVixDiskLib.nfcAio.Session.BufCount=4\nVixDiskLib.nfc.LogLevel=4`" | base64
edit forkliftcontroller -n openshift-mtv
virt_v2v_extra_args: "`-vddk-config /mnt/extra-v2v-conf/input.conf`" virt_v2v_extra_conf_config_map: "`perf`"
delete cm perf -n openshift-mtv