# Everything You Need to Know About Virtual Machine Live Migration Between Heterogeneous Processors

Kenta Ishiguro, Fonyuy-Asheri Caleb, Elouan Barraud, Renaud Lachaize, Yérom-David Bromberg, Alain Tchana

► To cite this version:

Kenta Ishiguro, Fonyuy-Asheri Caleb, Elouan Barraud, Renaud Lachaize, Yérom-David Bromberg, et al.. Everything You Need to Know About Virtual Machine Live Migration Between Heterogeneous Processors. EuroSys 2026 - 21st European Conference on Computer Systems, Apr 2026, Edinburgh, United Kingdom. hal-05513543

HAL Id: hal-05513543

https://hal.science/hal-05513543v1

Submitted on 16 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Everything You Need to Know About Virtual Machine Live Migration Between Heterogeneous Processors

**Kenta Ishiguro**

Univ. Grenoble Alpes, INRIA, CNRS, Grenoble INP, LIG, 38000 Grenoble, France

**Fonyuy-Asheri Caleb**

Univ. Rennes, CNRS, INRIA, IRISA, Rennes, France

**Elouan Barraud**

Univ. Grenoble Alpes, INRIA, CNRS, Grenoble INP, LIG, 38000 Grenoble, France

**Renaud Lachaize**

Univ. Grenoble Alpes, INRIA, CNRS, Grenoble INP, LIG, 38000 Grenoble, France

**Yérom-David Bromberg**

Univ. Rennes, CNRS, INRIA, IRISA, Rennes, France

**Alain Tchana**

Univ. Grenoble Alpes, INRIA, CNRS, Grenoble INP, LIG, 38000 Grenoble, France

## Abstract

This paper focuses on virtual machines (VMs) migrations across semi-heterogeneous processors (SH), which share the same ISA but expose different features. Processor incompatibilities in such settings can block or break migrations, limiting resource utilization. We provide the first detailed study of live migration feasibility across SH processors, yielding several findings valuable to cloud tenants, operators and hypervisor vendors. Building on these insights, we present MIGCHECK, a simulation tool that predicts the feasibility and outcome of VM migrations across SH processors without requiring costly real migrations. We demonstrate its accuracy and effectiveness through multiple use cases, including machine addition/removal, hypervisor updates, and VM platform adjustments.

**CCS Concepts: Software and its engineering → Virtual machines.**

**Keywords:** hypervisors, live migration, heterogeneous processors

## 1 Introduction

Despite its maturity, virtual machine (VM) [4] live migration still raises significant challenges in practice, given that the machines in a datacenter are often heterogeneous, due to several factors [14, 19, 24]. The diversity of I/O devices (such as NICs) is a well-known source of heterogeneity, which is already partially addressed (e.g., using paravirtual devices [25]). In this paper, we focus on another source of heterogeneity, which has received less attention thus far: CPU heterogeneity. More precisely, in many datacenters, the physical machines have semi-heterogenous (SH) processors: even if they are all based on the same chip vendor and product lines (e.g., Intel

x86 Xeons), these machines are equipped with processors corresponding to different generations (e.g., Intel Skylake, Ice Lake, etc.). We use the notion of semi-heterogeneity to denote that such processors from different generations typically have (1) differences in their microarchitectures and resources (e.g., cache sizes and number of CPU cores), and, more importantly, (2) *differences in CPU features* (e.g., AVX). In this paper, we study the consequence of processor semi-heterogeneity on live VM migration.

Understanding the feasibility of live migration across SH processors is crucial for cloud management systems and cloud operators to fully utilize computing resources, as processor incompatibility can prevent migrations or, conversely, cause migration failures. Suppose that the feasibility of migrations can be reliably identified without actually conducting them; such a capability would be extremely valuable as it may improve the efficiency of automatic destination selection [8, 36], expand the number of available servers [39], and avoid unnecessary network traffic and memory usage on destination servers (see $O_5$ in §4.3).

However, precisely assessing the feasibility of a migration is not straightforward. While migration from older to newer processors is generally considered possible [39], this is not always the case, as CPU features can be removed in newer processors [33]. Moreover, the problem remains complex and error-prone even in the case of migration between physical machines equipped with identical physical processors [5, 9–11], as CPU feature heterogeneity can stem from various factors, including hypervisor setup and versions [6], as well as BIOS and CPU firmware updates.

The complexity of live migrations across SH processors raises three challenging research questions regarding the management of VMs in datacenters. First (**RQ1**), under which conditions is live migration feasible or not across SH processors? Second (**RQ2**), to what extent does SH influence the feasibility of live migration and VM maintenance tasks in real-world datacenters? Third (**RQ3**), even after a successful migration, can we guarantee the available CPU features on the destination server do not affect the migrated VM's behavior?
To answer these research questions, we conducted three different types of studies. First, we performed reverse engineering of the migration algorithms used by two popular hypervisors, Xen and KVM. Next, we analyzed datacenters and carried out live migrations in two research cloud-testbeds (Grid5K [23] and CloudLab [17]) to assess the feasibility of live migrations in practice. Finally, we identified bugs in virtualization stacks arising from the complexities of migration across SH processors and analyzed how these bugs affect migrations.

Our study reveals three main findings: (F1) Even if restoring a VM's registers and memory on the destination server is possible, hypervisors can block live migration by validating CPU compatibility between the VM configuration and destination servers based on virtualized CPU features. The goal of this validation is the same in both Xen and KVM, as they ensure that the guest VM can be configured properly after migration; however, the design of the validation process significantly differs between these two hypervisors (§3); (F2) Datacenters indeed consist of SH processors. This semi-heterogeneity reduces live migration opportunities within the datacenters, and the feasibility of migration differs between hypervisors, even when the same servers are used. Our simulation indicates that this reduction in migration opportunities increases the duration of maintenance campaigns. Furthermore, our results indicate that prior knowledge of migration feasibility is crucial in mitigating this increase by expanding migration opportunities (§4); (F3) Live migrations across SH processors are error-prone. One possible outcome is an inconsistency between the pre- and post-migration states of VMs. This inconsistency may affect the behavior of the VMs. In the worst case, user-space applications running in the VMs could be terminated due to illegal instruction exceptions if certain instructions are missing on the destination server. Additionally, due to a lack of security considerations, migrations can expose VMs to vulnerabilities, such as Ret bleed [44], when migrating from non-vulnerable processors to those with known vulnerabilities. (§5).

Based on our findings, we propose MIGCHECK, a live migration simulation tool which determines the feasibility and outcome of migrations across SH processors without the burden and overhead of conducting actual live migrations. MIGCHECK collects necessary information from the data center and provides the operators with detailed information about possible migrations and impossible ones as well as reasons why the impossible migrations are classified as such for a given hypervisor and hypervisor version.

To facilitate the extensibility of MIGCHECK to newer hypervisor versions, we introduced a semi-automatic method that leverages Coccinelle [31] semantic patches to isolate hypervisor code blocks related to VM live migration, together with an annotation language that links the hypervisor code base to the corresponding MIGCHECK implementation. We demonstrated both the accuracy of MIGCHECK (§7.1) and

its usefulness on three usecases (§7.2–§7.4), as well as the effectiveness of our semi-automatic extension approach on six Xen versions (§7.5).

# 2 Background and Motivation

Our work focuses on x86-based CPUs, although we believe that the same analysis methodology can be applied to other instruction set architectures.

## 2.1 CPUID

CPU Identification (CPUID) is an x86 instruction designed to retrieve detailed information about a CPU, including its capabilities, hardware features, and specific component attributes. It functions by setting a *leaf number* in the EAX register and, optionally, a *subleaf number* in the ECX register. Depending on the leaf and subleaf values provided, the CPU returns the requested information through the EAX, EBX, ECX, and EDX registers. Due to the complexity and extensive evolution of the x86 architecture, the information accessible via CPUID has expanded considerably, now encompassing over 900 bitfields distributed across over 63 CPUID leaves.

The range of leaf values available on a given CPU is determined by the CPU vendor. Typically, this range is divided into two subcategories: the first covers *basic* CPU features, while the second pertains to *extended* features. Additionally, within the basic range, a sub-leaf is dedicated to listing *structured extended* features.

## 2.2 VM Live Migration

VM live migration is an operation extensively utilized by both cloud providers [16, 21, 34, 37, 38, 42, 45] and cloud tenants [18]. It transfers a VM from one hypervisor (*Hyp*<sub>src</sub>) to another (*Hyp*<sub>dst</sub>) with minimal downtime. The process begins with *Hyp*<sub>src</sub> sending the VM configuration (e.g., number of vCPUs, memory size, required CPU features) to *Hyp*<sub>dst</sub>, which then determines whether it can satisfy these requirements and correctly run the migrated VM. If this validation succeeds, the migration proceeds by copying the VM's state to *Hyp*<sub>dst</sub>, including its **memory pages** and **CPU registers**. The VM's disk image is generally shared across all compute nodes via a distributed file system such as Ceph [43].

Restoring the VM's CPU state on *Hyp*<sub>dst</sub> is a critical step in the live migration process. This phase heavily depends on the compatibility between the CPU features required by the migrated VM and those supported by *Hyp*<sub>dst</sub>.

## 2.3 Understanding Migration Feasibility

Cloud management systems use information about the feasibility of live migration across their computing resources consisting of SH processors to optimize utilization. For instance, OpenStack [35] features an automatic destination selection mechanism for live migration, which relies on migration feasibility data to identify an available server among
computing resources [36]. If this information is lacking, live migration will be aborted if the initial attempt is deemed infeasible [8]. In another example, XenCenter [26] manages CPU compatibility only within the same computing pool, requiring operators to ensure CPU compatibility between different pools for successful migration [39]. By accurately identifying the feasibility, operators can expand the available computing resources for their migration destinations. Additionally, information about the feasibility of live migration is valuable for reducing unnecessary memory transfers due to live migration failures. As we detail later in §3.2, not all live migration verifications are performed before memory transfer, making failures in such cases quite costly in terms of network bandwidth and memory usage.

Understanding the feasibility of live migration across SH processors is important but remains a challenging task due to the complexities of identifying CPU compatibility. A potential solution to this challenge is to leverage processor generations as a means of assessing compatibility. In general, live migration from older processors to newer ones is considered feasible, as it is commonly believed that newer ones support a superset of the CPU features supported by older ones. For example, XenCenter's documentation [39] suggests using the same or a newer processor as the migration destination for CPU compatibility. However, this belief is not always true. For example, a bug report for Libvirt [33] indicates that users were confused because live migration from an older processor to a newer one was not feasible. This infeasibility arises from the removal of Intel MPX since 2019 [29].

Even worse, host CPU information alone is insufficient to determine CPU compatibility for live migration. Since hypervisors virtualize CPUID execution, the CPU features available to VMs depend on the combination of the host CPU and the hypervisor version. Consequently, live migration can be infeasible even when two servers have identical CPUs if the destination hypervisor is older than the source hypervisor.

A series of OpenStack bugs highlights this complexity. The first bug [6] stems from using native CPUID results to determine CPU compatibility, which overlooks incompatibilities when different KVM versions are used across servers. To fix this issue, OpenStack began considering only the virtualized CPUID at the destination site. While this change enables OpenStack to detect CPU incompatibilities when KVM versions differ, it also leads to unnecessary blocking of live migrations [9–11]. This occurs because OpenStack assumes that CPU features masked by KVM are available on the source server (ignoring virtualization), while it assumes that those same masked features are unavailable on the destination server (taking virtualization into account). Consequently, OpenStack has implemented a workaround that bypasses CPU compatibility checks for migrations [7], despite the importance of prior knowledge of migration feasibility for safe and efficient VM management in datacenters.

This analysis reveals that a comprehensive understanding of the implementation of hypervisors' live migration algorithms is essential to determine the **feasibility** of migration across SH processors.

## 3 Reverse Engineering

We performed reverse engineering of the *CPU compatibility verifications* done by Xen and KVM, the two most widely used open-source hypervisors. For KVM, we consider the QEMU/Libvirt stack, which is popular in cloud management systems such as OpenStack [35] and Proxmox [20]. Libvirt provides a generic interface to interact with the user-space virtual machine monitor (VMM), which in this case is QEMU. Hereafter, we use “KVM” to refer to the entire virtualization stack and “KVM hypervisor” to specifically denote the KVM kernel module.

To our knowledge, there is no existing public documentation that describes how these systems handle live migration in SH environments. A detailed understanding of these mechanisms is essential to determine under what conditions live migration across SH is feasible or not. Moreover, this knowledge is crucial to preventing bugs related to the improper handling of CPU features during migration.

### 3.1 Virtual CPUID Construction

Before starting a VM, the hypervisor determines the set of CPU features that the VM is allowed to observe during CPUID execution from within the VM. This set is referred to as the *virtual CPUID* (or vCPUID for short).

**Xen.** The construction of Xen's vCPUID (also referred to as *virtual CPU policies* in Xen's terminology) is as follows. Being a type-I hypervisor, Xen operates directly on the hardware and is the first kernel loaded by the bootloader during system startup. At boot time, Xen collects information about the CPU using the CPUID instruction. Then it constructs a data structure, called *native CPU policy* (denoted as natPol), which aggregates this information. Given the critical role of CPU management, Xen performs several basic adjustments to natPol to ensure that only features that Xen can **properly manage** are potentially exposed to VMs. These adjustments consist mainly in trimming the number of CPUID leaves to retain only those Xen knows how to manage.

A second step known as *CPU feature leveling* is performed by Xen to determine the list of features for each guest type (PV, HVM, PVH, and Dom0) using natPol. The list of *known features* is defined in the header file cpufeatureset.h alongside the VM type to which each feature can be exposed. This list depends on the version of Xen and the security patches applied. The file cpufeatureset.h is processed by Xen's compilation toolchain to generate cpuid-autogen.h. Xen then uses the information contained in this file, along with natPol, to define the *host CPU policy* (denoted as hostPol).
The latter, together with natPol, serves as the basis for con-
structing default *virtual CPU policies* (*virtPol*) for each type
of VM. Optionally, users can customize the actual *virtPol*
of a specific VM according to their application needs.

**KVM.** This is a multi-layer virtualization system where each layer retrieves vCPUIDs from a lower layer and masks certain features. KVM hypervisor allows QEMU to access its vCPUIDs through the GET_SUPPORTED_CPUID ioctl. These vCPUIDs represent the CPU capabilities from KVM hypervisor's perspective, similar to hostPol in Xen. In this API, KVM hypervisor collects information about the host CPU using the CPUID instruction and modifies the results based on recognized features listed in kvm/cpuid.c. This feature list depends on the Linux/KVM hypervisor version because KVM is a type-II hypervisor that collects the availability of each feature through a function provided by Linux.

QEMU provides the query-cpu-model-expansion API
through QEMU Machine Protocol (QMP) to advertise its
vCPUIDs to upper-layer software stacks. After retrieving
vCPUIDs from KVM hypervisor, QEMU modifies them by
masking certain features based on their type and the default
CPU configuration assigned to the VM.

Through QEMU's API, Libvirt constructs its vCPUIDs by
masking certain CPU features for migration and security
considerations. These vCPUIDs are represented as a spe-
cial vCPU model known as the *host model*, which aims to
maximize CPU features while ensuring migration capability.
The vCPU model in Libvirt incorporates the *named models*
predefined in QEMU, which correspond to specific processor
generations (e.g., Ice Lake) and include essential CPU infor-
mation such as a set of available features and CPU signatures.
Libvirt addresses minor variations within a single CPU gen-
eration by adding or removing individual CPU features from
the base named model, to represent the host model.

**Take-away 1:** *The construction of the vCPUID in both virtualization systems depends on four factors: (F₁) the CPU features presented by the physical machine at boot time, (F₂) the features recognized by the virtualization stack version, (F₃) security and migration considerations, and (F₄) the specific needs of users.*

## 3.2 CPU Features Consideration In Live Migration

**Xen.** The *virtPol* of the VM being migrated is part of the configuration sent by the source server (*Xen*<sub>src</sub>) to the destination server (*Xen*<sub>dst</sub>) during the initial step of the migration process (§2.2). Let us denote *natPol*<sub>src</sub> and *natPol*<sub>dst</sub> as the native CPU policies of *Xen*<sub>src</sub> and *Xen*<sub>dst</sub>, respectively.

Xen<sub>dst</sub> performs five major verifications on the virtPol to ensure its compatibility with natPol<sub>dst</sub>. The first four verifications (V<sub>1</sub>-V<sub>4</sub>) are carried out before Xen<sub>src</sub> transmits

the VM's memory pages. The final verification (V<sub>5</sub>) is con-
ducted **after** all memory pages of the migrated VM have
been received. Xen<sub>dst</sub> stops the live migration process as
soon as any of these checks fail. The five verifications are
performed in the following exact order: (V<sub>1</sub>) the maximum
input value accepted by CPUID for obtaining basic CPU in-
formation in virtPol must be less than or equal to that
in natPol<sub>dst</sub>; (V<sub>2</sub>) the number of structured extended fea-
ture enumerations in virtPol must be less than or equal
to that in natPol<sub>dst</sub>; (V<sub>3</sub>) the number of extended function
enumerations in virtPol must be less than or equal to
that in natPol<sub>dst</sub>; (V<sub>4</sub>) if virtPol includes CPUID faulting,
natPol<sub>dst</sub> must support it as well; (V<sub>5</sub>) natPol<sub>dst</sub> must sup-
port all CPU states saved by the XSAVE instruction (e.g., SSE,
AVX512) available to virtPol.

Checks (V₁), (V₂) and (V₃) ensure that a VM does not mi-
grate into a server whose natPol dst does not have at least
as many CPUID leaves as there are in virtPol. Given that
CPUID faulting allows the hypervisor absolute control on
what it offers as virtPol, check (V₄) ensures a VM with a
possibly custom virtPol does not migrate into a server that
may not offer support for this customized CPUID policy. The
last check, (V₅), ensures that the target server has the neces-
sary CPU instructions to manipulate all CPU states saved by
the XSAVE instruction with which the VM may arrive.

**KVM.** Unlike Xen, neither the source nor the destination KVM hypervisor verifies the vCPUID configuration. This responsibility is handled by Libvirt, in coordination with QEMU, prior to the VM's memory transfer. Using QEMU APIs, Libvirt performs a CPU feature-level comparison to ensure compatibility between the two servers. To validate compatibility, the Libvirt instance running on the source server sends the VM's vCPU model, representing its vCPU configuration, to the Libvirt instance on the destination server. The destination Libvirt then launches a QEMU process based on the received vCPU model and retrieves the enabled and disabled features from the launched QEMU using the QEMU Object Model (QOM) APIs. If QEMU forcibly enables or disables any features during launch (e.g., due to a lack of support at the destination), Libvirt considers the servers incompatible and blocks live migration.

**Take-away 2: (Answer to RQ1)** *Both virtualization systems check the feasibility of live migration across SH processors. They rely on virtualized CPU features rather than the host CPU features when evaluating CPU compatibility for live migration. However, their validation processes differ, leading to variations in migration feasibility and outcomes even when the same physical resources are used. KVM adopts a more fine-grained approach, considering the availability of features in each vCPUID leaf, whereas Xen uses a less granular method, comparing the number of vCPUID leaves.*
# 4 Empirical Study

This section examines datacenter heterogeneity through three examples: two research infrastructures (Grid5K [23] and CloudLab [17]) and a commercial cloud (referred to as Company-A). Our analysis uses data collected from these datacenters between July 2024 and January 2025. We then demonstrate how CPU heterogeneity significantly restricts opportunities for VM migration.

## 4.1 Datacenter Composition

Grid5K comprises 73 clusters and features 46 distinct processor types. Among these, 53 clusters are equipped with Intel CPUs, while 16 clusters use AMD CPUs. Each cluster consists of homogeneous CPUs, deployed simultaneously. Cluster sizes vary significantly, ranging from a single server to as many as 124 servers.

CloudLab provides a variety of computing resources organized into *resource groups*, each of which typically has a unique processor model. For clarity and consistency in our analysis, we refer to these resource groups as *clusters*. We considered a total of 22 clusters with 18 distinct processor types. Among these, 18 are equipped with Intel CPUs and the remaining 4 with AMD CPUs.

Company-A consists of 27 clusters, each built with processors from the same generation. Among them, 25 clusters are equipped with Intel CPUs spanning nine different models, while the remaining two clusters use AMD CPUs.

## 4.2 Experimentation Methodology

We carried out these experiments only on CloudLab and Grid5K, as Company-A is a commercial cloud. We used Xen (version 4.18.3-pre) and paravirtualized (PV) VMs that boot Ubuntu 18.04. Regarding KVM, we used Linux kernel 6.8.0-59-generic, QEMU 8.2.2 and Libvirt 10.0.0. Due to time constraints, we limit our experiments to KVM on CloudLab.

Let $C$ be the set of all clusters in a given datacenter. From each cluster $C_i \in C$, we randomly select a machine $M^i$, and denote by $\mathcal{M}$ the set of all selected machines. Due to time constraints, we randomly selected a subset of machine pairs $(M^i, M^k) \in \mathcal{M} \times \mathcal{M}$, with $M^i \neq M^k$, instead of evaluating all possible pairs. We start a VM on $M^i$ and attempt to migrate it to $M^k$. For Grid5K we performed 1,228 cluster-to-cluster VM migrations, out of a theoretical total of 2,070 possible pairs (46×45). As for CloudLab, we ran 78 of the 306 possible migrations (18 × 17). During each migration, we note whether the migration was successful - a migration is deemed successful if the hypervisor reports that it succeeded.

## 4.3 Experimentation Results

We made several key observations:

(O₁) We measured an overall migration success rate of 46.17% on Grid5K and 61.54% on CloudLab with Xen, and 45.7% on CloudLab with KVM. (O₂) Migrations between Intel

CPUs show lower success rates (66.84% on Grid5K and 69.70% on CloudLab with Xen, and 51.7% on CloudLab with KVM) compared to AMD CPUs, which achieved 100% success on both platforms with both hypervisors. This stark difference can be attributed to the broader variety and generational diversity of Intel processors present on both testbeds. (O₃) All cross-vendor migration attempts failed. (O₄) Migrations from older to newer CPUs tend to be more successful, with success rates of 61.20% on Grid5K and 67.7% on CloudLab using Xen, and 96.4% on CloudLab using KVM. In contrast, migrations from newer to older CPUs have success rates of 19.75% on Grid5K and 67.7% on CloudLab with Xen, and 0.04% on CloudLab with KVM. (O₅) In our Xen experiments, 139 (21%) of all failed migrations on Grid5K and 10 (33%) on CloudLab failed at the verification step $V_5$ (§ 3.2), which is performed after memory transfer. Consequently, these migration failures result in unnecessary memory transfer. In our KVM experiments, all failed migrations were blocked by KVM before memory transfer. (O₆) A VM started on node $M$ may not be migratable to node $N$. However, if the VM is initially booted on node $N$ and migrated to node $M$, it can later be migrated back from $M$ to $N$. (O₇) We identify three categories of nodes based on migration behavior: *centrifugal*, *centripetal*, and *intermediate* nodes. Centrifugal nodes cannot receive incoming VM migrations, but VMs started on them can be migrated out. This behaviour is mostly observed in nodes running the oldest CPUs in a given data center. Grid5K and CloudLab each have a single centrifugal cluster. Conversely, centripetal nodes can receive incoming migrations but do not allow VMs to be migrated out. This is the behaviour with nodes running the latest CPUs. This is because these CPUs have more features resulting in greater challenges when migrating to other CPUs that have lesser features. Neither Grid5K nor CloudLab have centripetal nodes. Intermediate nodes support both incoming and outgoing migrations. This is the most common behaviour observed in data center nodes. A node will generally accept migrations from nodes with older CPUs and successfully migrate its VMs to nodes with newer CPUs. In addition, migrations are generally possible between nodes running CPUs released the same year. Aside the Centrifugal clusters, the remaining clusters on Grid5K and CloudLab are intermediate clusters.

**Take-away 3:** Combining observations (O₆) and (O₇), we recommend avoiding the direct deployment of VMs on centripetal nodes if future migrations are expected. To host a VM on a centripetal node while preserving migration flexibility, the VM should first be booted on a non-centripetal node and then migrated to the centripetal one.
## 4.4 Focus on Hypervisor Vendor Impact

Our analysis focuses on results obtained on CloudLab. Xen and KVM yield the same results for 44 migrations (24 successful and 20 failed) out of a total of 70. However, they exhibit different results among the remaining 26 migrations.

In eight cases, migrations are feasible with KVM but blocked by Xen. The differences arise from the presence of microcode-level protection against the Gather Data Sampling (GDS) vulnerability. We observed that the model-specific register value, GDS_CTRL, which indicates the presence of protection, differs between KVM and Xen on several servers, even though they use the same processor. As a result, KVM can leverage this protection, while Xen cannot in such situations. When the protection is absent, Xen turns off AVX to mitigate GDS. In our experiments, we found that Xen running on newer processors (e.g., Skylake and Ice Lake) is unable to leverage the protection, leading to AVX being turned off in those cases. Consequently, Xen blocks migrations from older to newer servers due to $V_5$ (§3.2) that fail in the absence of AVX, whereas KVM allows these migrations since it does not disable AVX and the newer servers have greater CPU capabilities than the older ones.

In 18 cases, migrations are feasible with Xen but blocked by KVM. These differences arise because KVM employs CPU feature-level validation to block migrations, which is a more conservative approach than that used by Xen. To provide more detail, we categorize the CPU features that cause migration blockages in KVM.

The first category includes CPU features enumerated by CPUID. By only checking the number of CPUID leaves, Xen sometimes overlooks several absent CPU features on target servers, whereas KVM can detect them. For example, AVX and MOVBE belong to the same CPUID leaf (CPUID with EAX=1). An Ivy Bridge processor enumerates AVX but does not enumerate MOVBE. Since the target Ivy Bridge server allows executing CPUID with EAX=1, Xen does not check for the availability of MOVBE on the target server. In contrast, KVM considers whether MOVBE is available. This variation can lead to different migration results between the two hypervisors.

The second category includes CPU features enumerated by model-specific registers (MSR). Several CPU features are enumerated by MSR, such as processor capabilities that indicate vulnerability to certain attacks or which VM-execution control fields can be configured. While KVM takes these features into account for its migration validation, Xen does not consider them.

The third category includes AVX features that are disabled by Xen to mitigate GDS. These AVX features can cause migration blockages in Xen due to $V_5$, but they are disabled as a mitigation against GDS. By narrowing down the CPU feature set, migrations become feasible in Xen. In contrast, these features remain enabled in KVM by using microcode mitigation

and KVM blocks migrations when AVX features are available for the VM but the target server does not support them.

The fourth category consists of CPU features that are not exposed to PVMs in Xen. Several CPU features (e.g., hardware lock elision) are not made available to VMs in Xen, while KVM exposes them. This difference can lead to inconsistent results between the two hypervisors, even when using the same pairs of servers.

**Take-away 4:** *The choice of hypervisor (Xen or KVM) impacts the feasibility of migrations, even if identical pairs of servers are used. The differences stem from how hypervisors virtualize CPU features, specifically which features are exposed to VMs, and their methods to ensure CPU compatibility in live migration.*

## 4.5 Focus on CPU Generations Impact

As demonstrated in $O_4$ in §4.3, processor generation is not a reliable indicator for determining migration feasibility across SH processors.

Using Xen, migrations from older to newer servers are generally successful when the processors are from the same vendor. However, exceptions occur when mitigation against GDS is applied to the destination processor, resulting in the absence of AVX. Conversely, some migrations from newer to older servers are successful mostly when the gap between the release years is less than 5 years. This can also be observed as a result of mitigation applied to newer processors, which present the same or fewer features than older processors, thereby allowing migrations to go through.

Using KVM, each case had one exception: one failure of migration from older to newer servers and one success from newer to older. The migration from the older Intel Xeon Gold 6142 (Skylake) to the newer Intel Xeon Platinum 8360Y (Ice Lake) was affected by the removal of certain CPU features among Intel processors. Specifically, the newer processor does not support TSX-related features (e.g., hardware lock elision [29]) and does not enumerate the RSB alternative [28]. Since KVM validates live migration feasibility through feature-level comparison, the absence of these features on the newer processor results in migration blockage. In contrast, migration from the newer Intel Xeon E5-2640 v4 to the older Intel Xeon D-1548 is successful. Both processors, launched in Q1 2016 and Q4 2015 respectively, belong to the Broadwell microarchitecture and have the same set of CPU features, which allows for successful migration.

**Take-away 5:** *Relying solely on processor generation as an indicator of CPU compatibility can lead to unexpected migration failures or overlooked opportunities for migration.*

## 4.6 Impact on Maintenance Duration

To estimate the impact of processor heterogeneity on maintenance duration, we conduct the following experiments through simulation. We consider a dummy datacenter consisting of two heterogeneous clusters, $C_1$ and $C_2$. Each cluster
contains 50 machines, and each machine can host up to 15 VMs with identical resource requirements. Suppose the datacenter operator plans to **upgrade** the operating systems (OS) of the machines in cluster $C_1$. We consider two scenarios depending on whether the hypervisor is capable of successfully performing VM live migration between heterogeneous CPUs (i.e., from $C_1$ to $C_2$) or not. We suppose that the VMs have access to all the CPU features.

Without heterogeneous migration support, if $C_1$ is operating at 50% capacity (i.e., 25 machines are fully utilized while the remaining 25 are idle), the upgrade campaign can be completed in four rounds of VM migrations, requiring $4 \times T$ in total, where $T$ represents the duration of a VM migration. If $C_1$ is at 75% capacity, the upgrade time doubles to $8 \times T$. A slight increase in utilization to 80% raises the upgrade time to $10 \times T$.

In contrast, when the hypervisor supports live migration across heterogeneous CPUs, the total maintenance campaign time is significantly reduced—approximately halved.

**Take-away 6: (Answer to RQ2)** *Datacenters are composed of SH processors. This heterogeneity reduces live migration opportunities, resulting in maintenance inefficiency. Prior knowledge of the feasibility of migration between clusters can alleviate this inefficiency. However, determining the feasibility remains challenging because it requires consideration of the virtualization software stack, not just processor information.*

# 5 CPU Heterogeneity is a Source of Bugs

This section demonstrates that CPU heterogeneity is a source of bugs, which can have severe consequences for both cloud providers and VM tenants. The result consists of bugs we identified **ourselves** through experimentation.

## 5.1 Inconsistency of Pre- and Post-migration

**Xen.** Our experimental setup consisted of two servers: Server $S_A$, equipped with an Intel Xeon Silver 4314 @ 2.40GHz processor, and Server $S_B$, equipped with an Intel Xeon Gold 6126 @ 2.60GHz processor. $S_A$ supports the Intel feature AVX512_VNNI whearas $S_B$ does not.

In our first experiment, we launch a VM on $S_A$. Within this VM, we execute an application that leverages the AVX512_VNNI feature to perform matrix multiplication in an infinite loop. We then migrate the VM to $S_B$. Upon arrival, the running application crashes with an *Illegal Instruction* error. This issue arises because Xen only considers the maximum values of the CPUID's basic functions, structured extended features, and extended functions ($V_1-V_3$ in §3.2), as well as the availability of some instructions ($V_5$ in §3.2). However, for a given leaf value within this range, not all CPUs necessarily support all listed features. We also observe that the vCPUID information is adjusted to reflect the capabilities of $S_B$. However, the

contents of the /proc/cpuinfo file inside the VM remain unchanged, still reporting the original CPU features.

Subsequently, we migrated the VM back to $S_A$. We find that the vCPUID output still reflects the adjusted (reduced) capabilities from $S_B$, rather than reverting to those of $S_A$. Nonetheless, despite the fact that the vCPUID now suggests that the AVX512_VNNI feature is unavailable, the application once again executes successfully without any issues. The /proc/cpuinfo output continues to reflect the same data as when the VM was originally booted.

In a second experiment, we created a similar VM on $S_B$. Initially, the matrix multiplication application could not be executed (as expected) due to the absence of AVX512_VNNI. We then migrated the VM to $S_A$. After migration, we observed that the information returned by the CPUID instruction and the content of /proc/cpuinfo remained identical to those obtained on $S_B$. However, despite the CPUID and the /proc/cpuinfo file not indicating the presence of AVX512_VNNI, our application—which could not previously be executed—was now able to run successfully.

**KVM.** In the first experiment presented above, Libvirt prevents live migration from $S_A$ to $S_B$ due to CPU incompatibility, which is identified through its feature-level comparison.

Regarding the second experiment, similar to the Xen experiment presented earlier, the VM migrated from $S_B$ to $S_A$ can execute AVX512_VNNI even though CPUID indicates the absence of AVX512_VNNI in the VM.

## 5.2 Lack of Security Consideration

While the current live migration validation focuses on ensuring that no features are lost from the guest VM after migration, this validation is insufficient to protect guest VMs from processor vulnerabilities, specifically Retbleed [44], a family of speculative execution attacks. Retbleed is relevant to processors that have the specific behavior on return stack buffer (RSB) underflows. Skylake and its close derivatives are known to have this behavior, and these processors enumerate RSB Alternative (RSBA) [28] to indicate that they are susceptible to Retbleed. To examine this security concern, we conducted using KVM a live migration from Server ($S_C$), equipped with an Intel Xeon E5-2660 v3 (Haswell), to Server ($S_D$), equipped with an Intel Xeon Silver 4114 (Skylake). $S_D$ is susceptible to Retbleed, whereas $S_C$ is not.

This migration from $S_C$ to $S_D$ is feasible because the set of CPU features on $S_D$ is a superset of those on $S_C$. However, it exposes the VM to the risk of Retbleed, as the migrated VM runs on the vulnerable Skylake processor and the available software mitigation is not enabled. Guest OSes enable a software mitigation against Retbleed when they detect at boot time that the processor is vulnerable. For instance, Linux detects the risk of Retbleed by using the CPU signature or the presence of RSBA as indicators. In this experiment, the VM is booted on the Haswell processor in $S_C$. Thus, the
guest OS does not enable its software mitigation because this processor is not vulnerable to Retbleed. While Libvirt treats RSBA as a CPU feature, the feature-level comparison in Libvirt does not block this migration, as it only detects the absence of required features and does not consider the presence of features that are not used by the VM.

Intel recommends advertising RSBA to VMs if they may run on a vulnerable processor in the future, indicating to guest OSes the need for mitigation [28]. However, QEMU on $S_C$ does not enable RSBA, even when requested, because the Haswell processor in $S_C$ does not enumerate it. Since RSBA is treated as a CPU feature, QEMU interprets the request to set RSBA for VMs on $S_C$ as an attempt to enable an unsupported CPU feature. To avoid the security risks associated with migration, it is also important to be aware of CPU features that are only available on the destination server.

**Take-away 7: (Answer to RQ3)** Even if migration is feasible, the outcome of the migration is critical, as the migrated VM can be affected by CPU feature discrepancies and security risks.

## 5.3 A Bug can Reduce Migration Opportunities

We identified a bug in the Libvirt's base CPU model selection code. This bug can reduce migration opportunities. As described in §3.1, Libvirt selects a named model as the base CPU model for the host model, but the bug leads to suboptimal choices. The primary goal of this selection is to choose a base model that minimizes the need for feature subtraction to represent the host model. To achieve this goal, Libvirt's heuristic algorithm prioritizes named models that require fewer feature subtractions. However, due to the bug, Libvirt tends to select newer models that necessitate a greater number of feature subtractions than intended. This suboptimal selection exacerbates the severity of the bug present in OpenStack 22 and earlier. Specifically, these versions of OpenStack rely on Libvirt's information to assess CPU compatibility but fail to account for subtracted features [5]. Consequently, they overestimate the features utilized by VMs on certain CPUs compared to those on the destination server and incorrectly block live migration. After notifying Libvirt maintainers about this bug, it was acknowledged and fixed almost immediately.

**Take-away 8:** All these findings pave the way for several new research directions in virtualization: a clear definition and formalization of the roles of the different virtualization layers, fuzzing of migration algorithms, solutions for handling certain CPU incompatibilities, and the development of tools for cloud operators to anticipate the impact of heterogeneity, among others. In this paper, we focus on the latter direction, leaving the others to future work.

<figure><img src="image_2.png" /><figcaption>**Figure 1.** Workflow for end-user utilization of MIGCHECK.</figcaption></figure>

# 6 MIGCHECK

## 6.1 Goals

Based on our findings, a thorough understanding of the impact of processor heterogeneity on the feasibility of cross-cluster migrations in datacenters is essential for improving the efficiency of VM management. A global view of inter-cluster relationships with respect to migration feasibility provides cloud operators with valuable insights to expand available resources during maintenance campaigns and workload consolidation for instance. However, existing methods for assessing migration feasibility are heuristic and prone to errors (§2.3), limiting their usability for cloud operators.

To address this limitation, we propose MIGCHECK, a tool that abstracts processor heterogeneity in a target datacenter as a **migration graph**. This offers the cloud operator a concise overview of the potential migration opportunities within both their **current** and **future** infrastructure. We designed MIGCHECK with four key requirements in mind: ($R_1$) flexibility, ($R_2$) usability and deployability, ($R_3$) scalability (large scale clouds), and ($R_4$) extensibility. The first three requirements target the end users of MIGCHECK (i.e., cloud operators), while the last one is intended for the developers of MIGCHECK. As a proof-of-concept, we design and implement MIGCHECK for Xen, but we believe that the same methodology can be applied to other hypervisors, such as KVM.

## 6.2 Overview

**From the perspective of MIGCHECK end users.** Fig. 1 illustrates the utilization workflow of MIGCHECK. ① It first ($R_2$ and $R_3$) automatically collects processor information (primarily CPUID) and hypervisor version data from servers managed by the cloud management system (e.g., OpenStack) of the target datacenter, and then ② simulates the CPU compatibility verification heuristic corresponding to the target hypervisor version on this data to provide actionable migration insights. Finally, ③ MIGCHECK constructs a graph whose nodes represent clusters and whose directed edges
capture the migration details of a VM launched with the default vCPUID on one node to another. Specifically, (R₂) the migration details include both the **feasibility** of migrations and the **reason** for CPU incompatibility in cases of infeasible migrations. Finally, MIGCHECK enables cloud operators to make (R₁) **informed decisions at low cost** in three critical scenarios related to VM configuration and datacenter updates:

(S₁) **vCPUID adjustment impact on migrations.** With MIGCHECK, the cloud operator can evaluate the impact of different vCPUID configurations on VM migration feasibility. In particular, MIGCHECK identifies the reasons for CPU incompatibility that prevent certain migrations, indicating which vCPUID features block live migration. By configuring vCPUIDs according to their needs, cloud operators can balance the trade-off between exposing CPU features to VMs and preserving migration opportunities.

(S₂) **Hypervisor updates impact on migrations.** As shown in §3, hypervisor implementation significantly affects migration feasibility. We designed MIGCHECK to facilitate the integration of new CPU compatibility verification heuristics introduced by emerging hypervisor versions. Therefore, MIGCHECK can be used to visualize the impact of hypervisor updates on migration feasibility within the datacenter, enabling operators to evaluate potential effects before performing the updates.

(S₃) **Hardware addition or removal impact on migrations.** As servers in datacenters have a limited service life, cloud operators must periodically replace clusters. MIGCHECK can reconstruct the migration graph by adding or removing nodes, providing a visualization that helps operators understand the impact of changes on migration feasibility, such as avoiding accidental overlook of infeasible migrations.

**From the perspective of MIGCHECK developers.** Another crucial requirement of MIGCHECK is its extensibility (R₄). Accurate assessment of migration feasibility relies on **detailed knowledge** of CPU compatibility verification heuristics, which MIGCHECK leverages in its simulations. However, this approach introduces a new challenge: reducing the manual effort required to support additional hypervisor versions.

We introduce a **semi-automatic** solution which leverages a *reference live-migration* simulator together with a new *annotation language*. The reference simulator contains the CPU compatibility verification heuristic, implemented in Python, corresponding to that of a baseline hypervisor version (implemented in C, for instance). The reference simulator is enriched with human-readable annotations that describe both the semantics and the relationship between each line or block of simulation code and the corresponding line or block of code in the actual algorithm. By exploiting these annotations together with a given target hypervisor version, MIGCHECK generates an implementation guide that

highlights all relevant code modifications required to update the reference simulator in order to support live migration for the target hypervisor version.

## 6.3 CPU and Hypervisor Information Collection

To meet requirements R₂ and R₃, MIGCHECK collects the necessary information without requiring extra effort. To achieve this, we designed MIGCHECK to be compatible with the infrastructure as code (IaC) concept. MIGCHECK reads files that include target servers within the datacenter and tasks to be performed on those servers. These tasks, specified by MIGCHECK developers, retrieve necessary information such as processor information and hypervisor version through remote command-line access. In large-scale datacenters managed by cloud management systems, remote command-line access to a control domain (e.g., Dom0 for Xen, host Linux for KVM) is typically used for adding servers to existing managed clusters when scaling out the datacenter [12, 13]. MIGCHECK leverages server definitions used for scaling out to collect the necessary information from servers. For instance, on a given server, MIGCHECK for Xen collects the processor family, available CPU features relevant to V₅ (§3.2), and available range for querying basic, extended, and structured extended features by invoking CPUID, as well as the presence of CPUID faulting and GDS mitigation by reading model-specific registers, and the Xen version using x1, which is a command-line tool for managing Xen.

## 6.4 Migration Feasibility Graph

MIGCHECK provides a visualized inter-cluster relationship with respect to migration feasibility as a migration graph to address R₁ and R₂. Let G = (V, E) be the migration graph in a given datacenter, where V denotes a set of clusters, and each cluster v ∈ V can be defined as v = {p, h}. Here, p represents processor information and h represents hypervisor version. A set of migrations between clusters is defined as E ⊆ {(x, y) | (x, y) ∈ V² and x ≠ y}. MIGCHECK provides migration details for each directed edge e ∈ E using simulation. These migration details indicate whether a VM launched with a given vCPUID on a machine in cluster x can be migrated to a machine in cluster y. If the migration is infeasible, the details include the reason for CPU incompatibility. For instance, MIGCHECK reports which verification (V₁-V₅ in §3.2) detects CPU incompatibility.

MIGCHECK end users can assess the impact of changes to their infrastructure by adding, removing, or modifying clusters. To see the impact, end users can update the value of parameters in the processor information p or the hypervisor version h as needed. Once the set of clusters is updated, MIGCHECK simulates with updated information and constructs an updated migration graph.
<figure><img src="image_3.png" /><figcaption>**Figure 2.** Semi-automatic update workflow of MIGCHECK.</figcaption></figure>

## 6.5 Semi-automatic MIGCHECK Update

We release MIGCHECK with several CPU compatibility verification heuristics corresponding to different versions of Xen. The first prototype of MIGCHECK, referred to as the *reference MIGCHECK*, was built from a *reference version of Xen* (4.18.3) through reverse engineering (§3), as shown in part A of Fig. 2. The other heuristics implemented in MIGCHECK were then derived using the *semi-automatic method* presented in this section.

**Listing 1.** A snippet of migration code as implemented in MIGCHECK, augmented with semantic information via our annotation language.

1. `@mp_start`
2. `@notetype` modification
3. `@description` This note modifies the maximum extended CPUID leaf for Xen 4.18.3.
4. `@finelame` xen/arch/x86/cpu-policy.c
5. `@function` calculate_host_policy
6. `@block` 376-380
7. `@@notedata`
8. `@description` Assignment of the maximum CPUID extended leaf for VM.
9. `@finelame` xen/arch/x86/cpu-policy.c
10. `@function` guest_common_default_leaves
11. `@line` 413
12. `@mp_end`

Our semi-automatic solution leverages an annotation language to generate implementation guides for new MIGCHECK versions. These guides highlight all relevant code modifications required to implement support for a new Xen version in MIGCHECK, derived from the reference implementation. Each annotation specifies an annotation type, a description, a code location, and the relevant function or structure name; see Listing 1 for an illustration.

In summary, the integration of a new Xen version into MIGCHECK follows the steps illustrated in lines B and C of Fig. 2. The process takes as input the original code of the reference Xen version, its annotated implementation within MIGCHECK, the original code of the target Xen version, and semantic patches. Our system employs semantic patches with Coccinelle [31] and the function call graph of Xen to delimit the code segments related to VM migration involving

CPUID consideration in both the reference and target Xen codebases. This step automates the reverse engineering process to extract the necessary operations for implementing MIGCHECK (shown in part B of Fig. 2).

The semantic patches we developed specify the keywords and expressions associated with cpu_policy, an internal structure in Xen that stores information for CPUID virtualization. This process extracts all code blocks that access a field of cpu_policy, which are operations relevant to CPUID virtualization and CPU compatibility checks for migration. The function call graph serves two purposes: (1) trimming extracted operations that are not relevant to VM migration and (2) recovering the semantics of the extracted operations. For (1), we use two functions, `__start_xen` and `main_migrate`, as the entry points for boot and migration, respectively. Functions not included in these invocation chains are identified as irrelevant, allowing us to eliminate the associated operations from consideration. For (2), we analyze the invocation order and contexts of functions. The invocation order indicates the execution sequence of the extracted operations across interfunctions, while the invocation context reveals the operation type, whether manipulation or check, for each operation. Our system identifies that all operations during boot are manipulations, while those during migration are checks. The annotations provide additional information. For example, our system tracks updates of constant values defined with macros used in the operations through these annotations.

The generation of this extension guide relies on an internal operation, namely an *intelligent diff* between the reference and target versions of Xen, as shown in line C of Fig. 2. This diff is “intelligent” in the sense that it is restricted to code segments relevant to VM migration, rather than the entire Xen codebase. These segments are described by the annotations and semantic patches we developed. The generated guide provides detailed for four types of update instructions: 1) the code has been moved but not changed, 2) the code has been moved between functions, 3) the code has changed partially, or 4) the code section no longer exists.

# 7 Evaluation

We evaluate MIGCHECK to answer the following questions: (§7.1) Can MIGCHECK accurately identify the feasibility of migration and failure reasons? (§7.2-§7.4) Is MIGCHECK effective for the three usecases ($S_1-S_3$) outlined in §6.2? (§7.5) How does the semi-automatic update of MIGCHECK facilitate the implementation of new hypervisor versions? Otherwise indicated, we use the reference MIGCHECK (Xen 4.18.3).

## 7.1 MIGCHECK Accuracy

To validate the accuracy of the MIGCHECK simulation results, we compare them with the actual migration results from CloudLab and Grid5K obtained by using the methodology described in §4.2.
<figure><img src="image_4.png" /><figcaption>**Figure 3.** Comparison of migration feasibility and failure reasons between MIGCHECK simulations and actual results in CloudLab and Grid5K.</figcaption></figure>

Fig. 3 shows the number of feasible and infeasible migrations, categorized by their six failure reasons, in the two cloud testbeds. We can see that MIGCHECK accurately identifies migration feasibility and failure reasons for all (1,306) migrations in the two cloud testbeds, as the generated results align with those obtained from actual migrations.

## 7.2 Impact of vCPUID Adjustment ($S_1$)

We use MIGCHECK to evaluate the number of available migration destinations in a target datacenter under different vCPUID configurations. For the latter, we consider the default vCPUID configuration for para-virtualized VMs (PVMs) of each machine type. This scenario corresponds to the capability of selecting a CPU Platform, as offered by Google Compute Engine to tenants [22]. Fig. 4 presents the results, where the x-axis shows each machine type along with the launch year of its processor.

These results illustrate the trade-off between exposing more CPU features to VMs and preserving migration opportunities. In both CloudLab (pc3000 to x1170) and Grid5K, the number of available destinations decreases gradually from left to right, showing that using the default vCPUID configuration of servers equipped with newer processors—typically with more CPU features—reduce migration flexibility. Notable exceptions appear with c220g5, c6420, and r650 in CloudLab, and with abacus8 in Grid5K. These anomalies are explained by the disabling of AVX as a mitigation for Gather Data Sampling (GDS). In fact, since all four machines lack microcode-level protection, Xen 4.18.3 disables AVX, which reduces the set of exposed CPU features and thereby increases the number of migration destinations.

## 7.3 Impact of Hypervisor Updates ($S_2$)

We use MIGCHECK to evaluate the number of available destinations in CloudLab across two Xen versions, specifically

4.17.1 and 4.18.3. A key difference between these versions is their handling of the GDS vulnerability. While Xen 4.18.3 mitigates GDS by disabling AVX in the absence of microcode-level protection, Xen 4.17.1 does not. Fig. 5 reports the results, when a VM is launched on each machine type using the default vCPUID configuration of Xen 4.17.1 for PVMs for this machine. We then evaluate migration both between Xen 4.17.1 machines and from Xen 4.17.1 to Xen 4.18.3, thereby capturing the impact of the hypervisor upgrade.

We identify three classes of machines, depending on how GDS is mitigated or whether the processor is vulnerable: First, machines pc3000 and d710 lack AVX capability. As a result, the hypervisor upgrade has no effect on them (the first four bars of Fig. 5 have the same height). Second, machines d820-x1170 support AVX, but are either not vulnerable to GDS or their CPUs handle it at the microcode level. Therefore, they do not require the software mitigation introduced in Xen 4.18.3, and VM migration among them remains possible after the hypervisor upgrade. However, migration from these machines to c220g5-r650 is no longer possible after the upgrade, since the latter machines are vulnerable to GDS and Xen 4.18.3 mitigates it by disabling AVX512. This explains why the second bar is shorter for machines d820-x1170. Finally, machines c220g5-r650 already exhibit fewer migration destinations even under Xen 4.17.1. This is because they support AVX512, while older machines only support AVX2. After the upgrade, there are no migration destinations for VMs launched on these machines under Xen 4.17.1, since Xen 4.18.3 detects the lack of microcode-level GDS mitigation and must disable AVX as well.

## 7.4 Impact of Machine Replacement ($S_3$)

In Grid5K, VMs launched on any machine cannot be migrated to uvb, as its processor is too old. We use MIGCHECK to evaluate the number of available migration destinations when replacing uvb with a more recent machine. For this purpose, we selected each CloudLab machine whose processor was released after that of uvb, namely d820-r650. Fig. 6 reports the results, using the default vCPUID configuration for PVMs of each introduced machine.

The best results are obtained when replacing uvb with machines whose processors were launched between 2013 and 2016. These processors expose more CPU features than those released before 2013 and, importantly, do not require GDS mitigation, thereby significantly expanding migration opportunities. Although processors released before 2013 provide fewer features—particularly fewer basic CPUID leaves—they still expand migration opportunities more than processors launched after 2016. This is because, unlike the latest processors, they support AVX without requiring GDS mitigation.
<figure><img src="image_5.png" /><figcaption>**Figure 4.** Number of available migration destinations in CloudLab and Grid5K under different vCPUID configurations.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Figure 5.** Number of available migration destinations between Xen 4.17.1 machines and from Xen 4.17.1 to 4.18.3 machines in CloudLab.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 6.** Number of migrations made possible by replacing the oldest cluster (uvb) with one of newer CloudLab machines in Grid5K.</figcaption></figure>

## 7.5 Effectiveness of Semi-automatic Update Method

We use the reference MIGCHECK for Xen 4.18.3 to generate implementation guides for currently available newer versions (v4.18.4-4.20.0). Fig. 7 shows the number of update instructions in the implementation guides for each version and context. This figure demonstrates that the maximum number of changes to code segments related to VM migration involving CPUID consideration per version is at most

<figure><img src="image_8.png" /><figcaption>**Figure 7.** Number of update guides generated by the semi-automatic update method starting from the reference MIGCHECK.</figcaption></figure>

11 among the six versions. We observe that the Xen relevant code blocks are quite stable among these versions, as the reported changes (shown in Fig. 7) do not necessitate modifications to the MIGCHECK simulation code.

## 8 Related Work

Processor heterogeneity has been explored in the context of application migration between different servers. Several research works propose mechanisms to ensure application execution on CPU with different ISAs [15, 30, 32, 40] or leverage different architectures on several servers to speed up execution times [2, 3].

However, few works target VM migration among heterogeneous processors. Intel FlexMigration [27] and AMD Extended Migration [1, 41] propose CPU downgrading to enforce CPU compatibility within a migration pool. The CPU downgrading approach turns off CPU features to align with the lowest common denominator of processor features in the migration pool. It allows VMs to be migrated among SH
processors, but essential CPU features are lost in the process. This approach reduces the potential benefits of incorporating new-generation servers into a datacenter.

To alleviate the maintenance complexity caused by SH processors, cloud providers often employ one or a combination of the following two approaches. (A₁) VM Reboot. This strategy involves shutting down the VM during the maintenance window and restarting it afterwards. Although typically coordinated with the VM owner, it inevitably leads to service downtime and can adversely affect Quality of Service, sometimes lasting up to an hour. Google Cloud adopts this approach for maintenance operations in multi-tenant clusters [21]. (A₂) Holdback machines. It consists in maintaining a pool of idle holdback machines (excluded from the sales pool), resulting in resource underutilization and financial waste. Google Cloud uses this approach for sole-tenant nodes [21].

## 9 Conclusion

VM live migration remains an unavoidable task in data centers, with its success highly limited by processor heterogeneity. Our work provides an explanation to this limitation and presents its impact on the migration spectrum of a data center. Our ongoing and future work aims at providing an improved live migration facility supporting migrations between heterogeneous processors.

## Acknowledgments

We thank our shepherd, Joel Wolfrath, and the anonymous reviewers for their insightful comments. This work was supported by the French Agence Nationale de la Recherche under the ANR projects "SecondChance" (ANR-23-CE25-0015) and "PEPR DiVa" (ANR-23-PECL-0004), by Inria "Défi OS", and by the CNRS MLNS2 International Research Project (IRP). Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations [23]. We are also grateful to CloudLab for providing the experimental infrastructure.

## References

[1] AMD. Live migration with amd-v™ extended migration technology, 2008. http://developer.amd.com/wordpress/media/2013/02/ livevirtualmachinemigrationonamdprocessors.pdf.

[2] Antonio Barbalace, Robert Lyerly, Christopher Jelesnianski, Anthony Carno, Ho-Ren Chuang, Vincent Legout, and Binoy Ravindran. Breaking the boundaries in heterogeneous-isa datacenters. SIGPLAN Not., 52(4):645-659, apr 2017.

[3] Antonio Barbalace, Marina Sadini, Saif Ansary, Christopher Jelesnianski, Akshay Ravichandran, Cagil Kendir, Alastair Murray, and Binoy Ravindran. Popcorn: Bridging the programmability gap in heterogeneous-isa platforms. In Proceedings of the Tenth European Conference on Computer Systems, EuroSys '15, New York, NY, USA, 2015. Association for Computing Machinery.

[4] Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hansen, Eric Jul, Christian Limpach, Ian Pratt, and Andrew Warfield. Live migration of virtual machines. In Proceedings of the 2nd Conference on Symposium on Networked Systems Design & Implementation - Volume 2, NSDI'05, page 273-286, USA, 2005. USENIX Association.

[5] OpenStack Contributors. Live migration fails despite matching CPUs. https://bugsuraunpad.net/nova/+bug/1898715, 2020.

[6] OpenStack Contributors. Live migration fails due to host cpu features different. https://bugs launchpad.net/nova/+bug/1903822, 2020.

[7] OpenStack Contributors. libvirt: Add a workaround to skip comparecpu() on destination. https://opendev.org/openstack/nova/ commit/267a40663cd8d0b94bbc5ebda4ece55a45753b64, 2021.

[8] OpenStack Contributors. Live-migrating an instance from 'Queens' (CentOS-7) to 'Train' (CentOS-8) fails during libvirt's compareCPU() check. https://bugs launchpad.net/nova/+bug/1913716/comments/3, 2021.

[9] OpenStack Contributors. compareHypervisorCPU() incompatibility during live migration. https://bugs launchpad.net/nova/+bug/2039803, 2023.

[10] OpenStack Contributors. Live migration fails with new compare_hypervisor_cpu method. https://bugs launchpad.net/nova/+bug/ 2023035, 2023.

[11] OpenStack Contributors. Nova fails to live migrate VMs even though source and destination hosts are the same. https://bugs launchpad.net/nova-lxd/+bug/2049231, 2024.

[12] OpenStack Contributors. Scaling the cluster out - Canonical OpenStack documentation. https://canonical-openstack.readthedocs-hosted. com/en/latest/how-to/operations/scaling-the-cluster-out/, 2025.

[13] XenServer Contributors. Install | XenServer 8.4. https://docs.xenserver. com/en-us/xenserver/8/install, 2025.

[14] Steve Crago, Kyle Dunn, Patrick Eads, Lorin Hochstein, Dong-In Kang, Mikyung Kang, Devendra Modium, Karandeep Singh, Jinwoo Suh, and John Paul Walters. Heterogeneous cloud computing. In 2011 IEEE International Conference on Cluster Computing, pages 378-385, 2011.

[15] Matthew DeVuyst, Ashish Venkat, and Dean M. Tullsen. Execution migration in a heterogeneous-isa chip multiprocessor. SIGARCH Comput. Archit. News, 40(1):261-272, mar 2012.

[16] Tu Dinh Ngoc, Boris Teabe, Alain Tchana, Gilles Muller, and Daniel Hagimont. Hypertp: A unified approach for live hypervisor replacement in datacenters. Journal of Parallel and Distributed Computing, 181:104733, 2023.

[17] Dmitry Duplyakin, Robert Ricci, Aleksander Maricq, Gary Wong, Jonathon Duerig, Eric Eide, Leigh Stoller, Mike Hibler, David Johnson, Kirk Webb, Aditya Akella, Kuangching Wang, Glenn Ricart, Larry Landweber, Chip Elliott, Michael Zink, Emmanuel Cecchet, Snighaswin Kar, and Prabodh Mishra. The design and operation of CloudLab. In Proceedings of the USENIX Annual Technical Conference (ATC), pages 1-14, July 2019.

[18] Google Cloud Engine. Sole-tenant node clusters. https://cloud.google.com/dataproc/docs/concepts/configuring-clusters/sole-tenant-nodes.

[19] Chongzhou Fang, Najmeh Nazari, Behnam Omidi, Han Wang, Aditya Puri, Manish Arora, Setareh Rafatirad, Houman Homayoun, and Khaled Khasawneh. Assessing and mitigating heterogeneity-driven security threats in the cloud. ACM Trans. Internet Technol., 25(4), September 2025.

[20] Proxmox Server Solutions GmbH. Proxmox - Powerful open-source server solutions. https://www.proxmox.com/en/. Accessed 2025-07-07.

[21] Google. Live migration process during maintenance events. https://cloud.google.com/compute/docs/instances/live-migration-process.

[22] Google. Specify a minimum cpu platform for vm instances. https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform.

[23] Grid5000. Grid5000. https://www.grid5000.fr.
[24] Ori Hadary, Luke Marshall, Ishai Menache, Abhisek Pan, Esaias E Greeff, David Dion, Star Dorminey, Shailesh Joshi, Yang Chen, Mark Russinovich, and Thomas Moscibroda. Protean: VM allocation service at scale. In *14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)*, pages 845–861. USENIX Association, November 2020.

[25] Nadav Har'El, Abel Gordon, Alex Landau, Muli Ben-Yehuda, Avishay Traeger, and Razya Ladelsky. Efficient and scalable paravirtual I/O system. In *2013 USENIX Annual Technical Conference (USENIX ATC 13)*, pages 231–242, San Jose, CA, June 2013. USENIX Association.

[26] Cloud Software Group Inc. About XenCenter. https://docs.xenserver.com/en-us/xencenter/current-release/intro-welcome. Accessed 2025-07-07.

[27] Intel. Intel® virtualization technology flexmigration application note, 2012. https://www.intel.com/content/dam/www/public/us/en/documents/application-notes/virtualization-technology-flexmigration-application-note.pdf.

[28] Intel. Retpline: A branch target injection mitigation. https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/retpline-branch-target-injection-mitigation.html#inpage-nav-5-1, 2022. Accessed 2025-06-17.

[29] Intel. Intel 64 and IA-32 Architectures Software Developer's Manual Combined Volumes: 1, 2A, 2B, 2C, 2D, 3A, 3B, 3C, 3D, and 4. https://cdrdv2.intel.com/v1/dl/getContent/671200, 2025. Accessed 2025-06-17.

[30] R. Kumar, D.M. Tullsen, P. Ranganathan, N.P. Jouppi, and K.I. Farkas. Single-isa heterogeneous multi-core architectures for multithreaded workload performance. In *Proceedings. 31st Annual International Symposium on Computer Architecture, 2004.*, pages 64–75, 2004.

[31] Julia Lawall and Gilles Muller. Coccinelle: 10 years of automated evolution in the linux kernel. In *2018 USENIX Annual Technical Conference (USENIX ATC 18)*, pages 601–614, 2018.

[32] Tong Li, Paul Brett, Rob Knauerhase, David Koufaty, Dheeraj Reddy, and Scott Hahn. Operating system support for overlapping-isa heterogeneous multi-core architectures. In *HPCA - 16 2010 The Sixteenth International Symposium on High-Performance Computer Architecture*, pages 1–12, 2010.

[33] Libvirt and OpenStack Contributors. Unacceptable cpu info: Cpu doesn't have compatibility. https://bugzilla.redhat.com/show_bug.cgi?id=2138381, 2022.

[34] Jialun Lyu, Marisa You, Celine Irvine, Mark Jung, Tyler Narmore, Jacob Shapiro, Luke Marshall, Savyasachi Samal, Ioannis Manousakis, Lisa Hsu, Preetha Subbarayalu, Ashish Raniwala, Brijesh Warrier, Ricardo Bianchini, Bianca Schroeder, and Daniel S. Berger. Hyrax: Fail-in-Place server operation in cloud platforms. In *17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23)*, pages 287–304, Boston, MA, July 2023. USENIX Association.

[35] OpenStack. Open Source Cloud Computing Infrastructure - OpenStack. https://www.openstack.org/. Accessed 2025-06-17.

[36] OpenStack. Automatic selection of the destination host. https://docs.openstack.org/nova/2025.1/admin/live-migration-usage.html#automatic-selection-of-the-destination-host, 2023.

[37] Adam Ruprecht, Danny Jones, Dmitry Shiraev, Greg Harmon, Maya Spivak, Michael Krebs, Miche Baker-Harvey, and Tyler Sanderson. Vm live migration at scale. SIGPLAN Not., 53(3):45–56, March 2018.

[38] Mark Russinovich, Naga Govindaraju, Melur Raghuraman, David Hepkin, Jamie Schwartz, and Arun Kishan. Virtual machine preserving host updates for zero day patching in public cloud. In *Proceedings of the Sixteenth European Conference on Computer Systems*, EuroSys '21, page 114–129, New York, NY, USA, 2021. Association for Computing Machinery.

[39] XenServer Staff. Migrate virtual machines. https://docs.xenserver.com/en-us/xencenter/current-release/vms-relocate.html#compatibility-requirements, 2025.

[40] Ashish Venkat and Dean M. Tullsen. Harnessing isa diversity: Design of a heterogeneous-isa chip multiprocessor. In *2014 ACM/IEEE 41st International Symposium on Computer Architecture (ISCA)*, pages 121–132, 2014.

[41] Elsie Wahlig. Amd extended migration, 2007. https://www.linux-kvm.org/images/2/24/KvmForum2007%24AMD_Extended_Migration-KVMForum-ElsieWahlig.pdf.

[42] Xin Wang, Deguo Li, Zhihong Wang, Lidong Jiang, Shubo Wen, Daxiang Kang, Engin Arslan, Peng He, Xinyu Qian, Bin Niu, Jianwen Pi, Xiaoning Ding, Ke Lin, and Hao Luo. Byte vswitch: A high-performance virtual switch for cloud networking. In *Proceedings of the Twentieth European Conference on Computer Systems*, EuroSys '25, page 1417–1432, New York, NY, USA, 2025. Association for Computing Machinery.

[43] Sage A Weil. Ceph: reliable, scalable, and high-performance distributed storage. 2007.

[44] Johannes Wikner and Kaveh Razavi. RETBLEED: Arbitrary speculative code execution with return instructions. In *31st USENIX Security Symposium (USENIX Security 22)*, pages 3825–3842, Boston, MA, August 2022. USENIX Association.

[45] Xiantao Zhang, Xiao Zheng, Zhi Wang, Qi Li, Junkang Fu, Yang Zhang, and Yibin Shen. Fast and scalable vmm live upgrade in large cloud infrastructure. In *Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems*, ASPLOS '19, page 93–105, New York, NY, USA, 2019. Association for Computing Machinery.