Chapter 16. Troubleshooting This section provides information for troubleshooting common migration issues. 16.1. Error messages This section describes error messages and how to resolve them. warm import retry limit reached The warm import retry limit reached error message is displayed during a warm migration if a VMware virtual machine (VM) has reached the maximum number (28) of changed block tracking (CBT) snapshots during the precopy stage. To resolve this problem, delete some of the CBT snapshots from the VM and restart the migration plan. Unable to resize disk image to required size The Unable to resize disk image to required size error message is displayed when migration fails because a virtual machine on the target provider uses persistent volumes with an EXT4 file system on block storage. The problem occurs because the default overhead that is assumed by CDI does not completely include the reserved place for the root partition. To resolve this problem, increase the file system overhead in CDI to more than 10%. 16.2. Using the must-gather tool You can collect logs and information about MTV custom resources (CRs) by using the must-gather tool. You must attach a must-gather data file to all customer cases. You can gather data for a specific namespace, migration plan, or virtual machine (VM) by using the filtering options. Note If you specify a non-existent resource in the filtered must-gather command, no archive file is created. Prerequisites You must be logged in to the OpenShift Virtualization cluster as a user with the cluster-admin role. You must have the Red Hat OpenShift CLI ( oc ) installed. Collecting logs and CR information Navigate to the directory where you want to store the must-gather data. Run the oc adm must-gather command: USD oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 The data is saved as /must-gather/must-gather.tar.gz . You can upload this file to a support case on the Red Hat Customer Portal . Optional: Run the oc adm must-gather command with the following options to gather filtered data: Namespace: USD oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 \ -- NS=<namespace> /usr/bin/targeted Migration plan: USD oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 \ -- PLAN=<migration_plan> /usr/bin/targeted Virtual machine: USD oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 \ -- VM=<vm_id> NS=<namespace> /usr/bin/targeted 1 1 Specify the VM ID as it appears in the Plan CR. 16.3. Architecture This section describes MTV custom resources, services, and workflows. 16.3.1. MTV custom resources and services The Migration Toolkit for Virtualization (MTV) is provided as an Red Hat OpenShift Operator. It creates and manages the following custom resources (CRs) and services. MTV custom resources Provider CR stores attributes that enable MTV to connect to and interact with the source and target providers. NetworkMapping CR maps the networks of the source and target providers. StorageMapping CR maps the storage of the source and target providers. Plan CR contains a list of VMs with the same migration parameters and associated network and storage mappings. Migration CR runs a migration plan. Only one Migration CR per migration plan can run at a given time. You can create multiple Migration CRs for a single Plan CR. MTV services The Inventory service performs the following actions: Connects to the source and target providers. Maintains a local inventory for mappings and plans. Stores VM configurations. Runs the Validation service if a VM configuration change is detected. The Validation service checks the suitability of a VM for migration by applying rules. The Migration Controller service orchestrates migrations. When you create a migration plan, the Migration Controller service validates the plan and adds a status label. If the plan fails validation, the plan status is Not ready and the plan cannot be used to perform a migration. If the plan passes validation, the plan status is Ready and it can be used to perform a migration. After a successful migration, the Migration Controller service changes the plan status to Completed . The Populator Controller service orchestrates disk transfers using Volume Populators. The Kubevirt Controller and Containerized Data Import (CDI) Controller services handle most technical operations. 16.3.2. High-level migration workflow The high-level workflow shows the migration process from the point of view of the user: You create a source provider, a target provider, a network mapping, and a storage mapping. You create a Plan custom resource (CR) that includes the following resources: Source provider Target provider, if MTV is not installed on the target cluster Network mapping Storage mapping One or more virtual machines (VMs) You run a migration plan by creating a Migration CR that references the Plan CR. If you cannot migrate all the VMs for any reason, you can create multiple Migration CRs for the same Plan CR until all VMs are migrated. For each VM in the Plan CR, the Migration Controller service records the VM migration progress in the Migration CR. Once the data transfer for each VM in the Plan CR completes, the Migration Controller service creates a VirtualMachine CR. When all VMs have been migrated, the Migration Controller service updates the status of the Plan CR to Completed . The power state of each source VM is maintained after migration. 16.3.3. Detailed migration workflow You can use the detailed migration workflow to troubleshoot a failed migration. The workflow describes the following steps: Warm Migration or migration to a remote OpenShift cluster: When you create the Migration custom resource (CR) to run a migration plan, the Migration Controller service creates a DataVolume CR for each source VM disk. For each VM disk: The Containerized Data Importer (CDI) Controller service creates a persistent volume claim (PVC) based on the parameters specified in the DataVolume CR. If the StorageClass has a dynamic provisioner, the persistent volume (PV) is dynamically provisioned by the StorageClass provisioner. The CDI Controller service creates an importer pod. The importer pod streams the VM disk to the PV. After the VM disks are transferred: The Migration Controller service creates a conversion pod with the PVCs attached to it when importing from VMWare. The conversion pod runs virt-v2v , which installs and configures device drivers on the PVCs of the target VM. The Migration Controller service creates a VirtualMachine CR for each source virtual machine (VM), connected to the PVCs. If the VM ran on the source environment, the Migration Controller powers on the VM, the KubeVirt Controller service creates a virt-launcher pod and a VirtualMachineInstance CR. The virt-launcher pod runs QEMU-KVM with the PVCs attached as VM disks. Cold migration from RHV or OpenStack to the local OpenShift cluster: When you create a Migration custom resource (CR) to run a migration plan, the Migration Controller service creates for each source VM disk a PersistentVolumeClaim CR, and an OvirtVolumePopulator when the source is RHV, or an OpenstackVolumePopulator CR when the source is OpenStack. For each VM disk: The Populator Controller service creates a temporarily persistent volume claim (PVC). If the StorageClass has a dynamic provisioner, the persistent volume (PV) is dynamically provisioned by the StorageClass provisioner. The Migration Controller service creates a dummy pod to bind all PVCs . The name of the pod contains pvcinit . The Populator Controller service creates a populator pod. The populator pod transfers the disk data to the PV. After the VM disks are transferred: The temporary PVC is deleted, and the initial PVC points to the PV with the data. The Migration Controller service creates a VirtualMachine CR for each source virtual machine (VM), connected to the PVCs. If the VM ran on the source environment, the Migration Controller powers on the VM, the KubeVirt Controller service creates a virt-launcher pod and a VirtualMachineInstance CR. The virt-launcher pod runs QEMU-KVM with the PVCs attached as VM disks. Cold migration from VMWare to the local OpenShift cluster: When you create a Migration custom resource (CR) to run a migration plan, the Migration Controller service creates a DataVolume CR for each source VM disk. For each VM disk: The Containerized Data Importer (CDI) Controller service creates a blank persistent volume claim (PVC) based on the parameters specified in the DataVolume CR. If the StorageClass has a dynamic provisioner, the persistent volume (PV) is dynamically provisioned by the StorageClass provisioner. For all VM disks: The Migration Controller service creates a dummy pod to bind all PVCs . The name of the pod contains pvcinit . The Migration Controller service creates a conversion pod for all PVCs. The conversion pod runs virt-v2v , which converts the VM to the KVM hypervisor and transfers the disks' data to their corresponding PVs. After the VM disks are transferred: The Migration Controller service creates a VirtualMachine CR for each source virtual machine (VM), connected to the PVCs. If the VM ran on the source environment, the Migration Controller powers on the VM, the KubeVirt Controller service creates a virt-launcher pod and a VirtualMachineInstance CR. The virt-launcher pod runs QEMU-KVM with the PVCs attached as VM disks. 16.3.4. How MTV uses the virt-v2v tool The Migration Toolkit for Virtualization (MTV) uses the virt-v2v tool to convert the disk image of a VM into a format compatible with OpenShift Virtualization. The tool makes migrations easier because it automatically performs the tasks needed to make your VMs work with OpenShift Virtualization, such as enabling paravirtualized VirtIO drivers in the converted virtual machine, if possible, and installing the QEMU guest agent. virt-v2v is included in Red Hat Enterprise Linux (RHEL) versions 7 and later. 16.3.4.1. Main functions of virt-v2v in MTV migrations During migration, MTV uses virt-v2v to collect metadata about VMs, make necessary changes to VM disks, and copy the disks containing the VMs to OpenShift Virtualization. virt-v2v makes the following changes to VM disks to prepare them for migration: Additions: Injection of VirtIO drivers, for example, network or disk drivers. Preparation of hypervisor-specific tools or agents, for example, a QEMU guest agent installation. Modification of boot configuration, for example, updated bootloader or boot entries. Removals: Unnecessary or former hypervisor-specific files, for example, VMware tools or VirtualBox additions. Old network driver configurations, for example, removing VMware-specific NIC drivers. Configuration settings that are incompatible with the target system, for example, old boot settings. If you are migrating from VMware or from OVA files, virt-v2v also sets their IP addresses either during the migration or during the first reboot of the VMs after migration. Note You can also run pre-defined Ansible hooks before or after a migration using MTV. For more information, see Adding hooks to an MTV migration plan . These hooks do not necessarily use virt-v2v . 16.3.4.2. Customizing, removing, and installing files MTV uses virt-v2v to perform additional guest customizations during the conversion, such as the following actions: Customization to preserve IP addresses Customization to preserve drive letters Note For RHEL-based guests, virt-v2v attempts to install the guest agent from the Red Hat registry. If the migration is run in a detached environment, the installer will fail, and you must use hooks, or other automation, to install the guest agent. For more information, see the man reference pages: virt-v2v - Convert a guest to use KVM virt-customize - Customize a virtual machine 16.3.4.3. Permissions and virt-v2v virt-v2v does not require permissions or access credentials for the guest operating system itself because virt-v2v is not run against a running VM, but only against the disks of a VM. 16.4. Logs and custom resources You can download logs and custom resource (CR) information for troubleshooting. For more information, see the detailed migration workflow . 16.4.1. Collected logs and custom resource information You can download logs and custom resource (CR) yaml files for the following targets by using the Red Hat OpenShift web console or the command-line interface (CLI): Migration plan: Web console or CLI. Virtual machine: Web console or CLI. Namespace: CLI only. The must-gather tool collects the following logs and CR files in an archive file: CRs: DataVolume CR: Represents a disk mounted on a migrated VM. VirtualMachine CR: Represents a migrated VM. Plan CR: Defines the VMs and storage and network mapping. Job CR: Optional: Represents a pre-migration hook, a post-migration hook, or both. Logs: importer pod: Disk-to-data-volume conversion log. The importer pod naming convention is importer-<migration_plan>-<vm_id><5_char_id> , for example, importer-mig-plan-ed90dfc6-9a17-4a8btnfh , where ed90dfc6-9a17-4a8 is a truncated RHV VM ID and btnfh is the generated 5-character ID. conversion pod: VM conversion log. The conversion pod runs virt-v2v , which installs and configures device drivers on the PVCs of the VM. The conversion pod naming convention is <migration_plan>-<vm_id><5_char_id> . virt-launcher pod: VM launcher log. When a migrated VM is powered on, the virt-launcher pod runs QEMU-KVM with the PVCs attached as VM disks. forklift-controller pod: The log is filtered for the migration plan, virtual machine, or namespace specified by the must-gather command. forklift-must-gather-api pod: The log is filtered for the migration plan, virtual machine, or namespace specified by the must-gather command. hook-job pod: The log is filtered for hook jobs. The hook-job naming convention is <migration_plan>-<vm_id><5_char_id> , for example, plan2j-vm-3696-posthook-4mx85 or plan2j-vm-3696-prehook-mwqnl . Note Empty or excluded log files are not included in the must-gather archive file. Example must-gather archive structure for a VMware migration plan 16.4.2. Downloading logs and custom resource information from the web console You can download logs and information about custom resources (CRs) for a completed, failed, or canceled migration plan or for migrated virtual machines (VMs) from the Red Hat OpenShift web console. Procedure In the Red Hat OpenShift web console, click Migration Plans for virtualization . Click Get logs beside a migration plan name. In the Get logs window, click Get logs . The logs are collected. A Log collection complete message is displayed. Click Download logs to download the archive file. To download logs for a migrated VM, click a migration plan name and then click Get logs beside the VM. 16.4.3. Accessing logs and custom resource information from the command line You can access logs and information about custom resources (CRs) from the command line by using the must-gather tool. You must attach a must-gather data file to all customer cases. You can gather data for a specific namespace, a completed, failed, or canceled migration plan, or a migrated virtual machine (VM) by using the filtering options. Note If you specify a non-existent resource in the filtered must-gather command, no archive file is created. Prerequisites You must be logged in to the OpenShift Virtualization cluster as a user with the cluster-admin role. You must have the Red Hat OpenShift CLI ( oc ) installed. Procedure Navigate to the directory where you want to store the must-gather data. Run the oc adm must-gather command: USD oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 The data is saved as /must-gather/must-gather.tar.gz . You can upload this file to a support case on the Red Hat Customer Portal . Optional: Run the oc adm must-gather command with the following options to gather filtered data: Namespace: USD oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 \ -- NS=<namespace> /usr/bin/targeted Migration plan: USD oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 \ -- PLAN=<migration_plan> /usr/bin/targeted Virtual machine: USD oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 \ -- VM=<vm_name> NS=<namespace> /usr/bin/targeted 1 1 You must specify the VM name , not the VM ID, as it appears in the Plan CR.

Commands:
oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11
oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 -- NS=<namespace> /usr/bin/targeted
oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 -- PLAN=<migration_plan> /usr/bin/targeted
oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 -- VM=<vm_id> NS=<namespace> /usr/bin/targeted 1
must-gather └── namespaces ├── target-vm-ns │ ├── crs │ │ ├── datavolume │ │ │ ├── mig-plan-vm-7595-tkhdz.yaml │ │ │ ├── mig-plan-vm-7595-5qvqp.yaml │ │ │ └── mig-plan-vm-8325-xccfw.yaml │ │ └── virtualmachine │ │ ├── test-test-rhel8-2disks2nics.yaml │ │ └── test-x2019.yaml │ └── logs │ ├── importer-mig-plan-vm-7595-tkhdz │ │ └── current.log │ ├── importer-mig-plan-vm-7595-5qvqp │ │ └── current.log │ ├── importer-mig-plan-vm-8325-xccfw │ │ └── current.log │ ├── mig-plan-vm-7595-4glzd │ │ └── current.log │ └── mig-plan-vm-8325-4zw49 │ └── current.log └── openshift-mtv ├── crs │ └── plan │ └── mig-plan-cold.yaml └── logs ├── forklift-controller-67656d574-w74md │ └── current.log └── forklift-must-gather-api-89fc7f4b6-hlwb6 └── current.log
oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11
oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 -- NS=<namespace> /usr/bin/targeted
oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 -- PLAN=<migration_plan> /usr/bin/targeted
oc adm must-gather --image=registry.redhat.io/migration-toolkit-virtualization/mtv-must-gather-rhel8:2.7.11 -- VM=<vm_name> NS=<namespace> /usr/bin/targeted 1