# UTwinVM: Reliable hints on the effects of hypervisor updates on VMs in the Cloud

Djob Mvondo, Tong Xing, Antonio Barbalace

► To cite this version:

Djob Mvondo, Tong Xing, Antonio Barbalace. UTwinVM: Reliable hints on the effects of hypervisor updates on VMs in the Cloud. Middleware 2024 - 25th ACM/IFIP International Middleware Conference, ACM Association For Computing Machinery, Dec 2024, Hong Kong, Hong Kong SAR China. pp.103-116, 10.1145/3652892.3700752 . hal-04894609

HAL Id: hal-04894609

https://hal.science/hal-04894609v1

Submitted on 17 Jan 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# UTwinVM: Reliable hints on the effects of hypervisor updates on VMs in the Cloud

Djob Mvondo

barbe-thystere.mvondo-djob@irisa.fr

Univ Rennes, CNRS, INRIA, IRISA

Rennes, France

Tong Xing

tong.xing@ed.ac.uk

University of Edinburgh

Scotland, UK

Antonio Barbalace

antonio.barbalace@ed.ac.uk

University of Edinburgh

Scotland, UK

## ABSTRACT

We investigate the problem of getting hints on the effects of virtualization system (aka hypervisor) updates impact on virtual machines (VMs). System administrators can be reluctant to apply updates due to vague hints regarding the updates' impact on running applications. The problem is challenging since VMs are black boxes by design, reducing the scope of the data that can be retrieved and analyzed. Additionally, cloning VMs is only sometimes possible for obvious legal and privacy concerns.

In this paper, we present UTwinVM for Updated Twin VM, a mechanism to obtain valuable hints concerning the impact of updates on applications running in VMs. UTwinVM key idea is to generate a digital twin of running VMs that mimics the original VMs workloads behavior if they were running on with the updated virtualization stack. To achieve that, UTwinVM records several metrics regarding running VMs and the virtualization system on the initial server and on another server where the intended updated system runs. Then, it leverages a non-linear negative squared solver to determine how the initial system differs from the updated one. Based on that, it runs through specific scripts, workloads in VMs will match workloads running in the production VMs as if they were running on the updated system. Consequently, system administrators can observe these recreated workloads to obtain hints on the potential performance impact on production VMs. UTwinVM is non-intrusive for VMs and does not require modifications to the virtualization system.

We implement an open-source prototype of UTwinVM atop the Linux/KVM virtualization system (v6.6.4). Our evaluation shows that UTwinVM incurs no overhead on running VMs and less than 11% performance impact on host services while providing accurate hints for several update scenarios – up to 99%.

## ACM Reference Format:

Djob Mvondo, Tong Xing, and Antonio Barbalace. 2025. UTwinVM: Reliable hints on the effects of hypervisor updates on VMs in the Cloud. In *Proceedings of ACM Conference (Conference’17)*. ACM, Hong Kong, China, 14 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

## 1 INTRODUCTION

Virtualization systems are subject to updates. Virtualization is widely used in computing infrastructures to maximize resource utilization and ensure isolation among different applications. With virtualization, each application runs in a isolated unit called virtual machine (VM) with the help of a virtualization software (a.k.a.,

hypervisors). As every software, hypervisors are subject to updates. These updates are of utmost importance to the life cycle of hypervisors. In the form of patches, these updates are applied to hypervisors to fix security issues or improve the performance of some hypervisor subsystems. To illustrate the importance of those on Xen [6] – a mainstream hypervisor, in Figure 1 we plot the number of patches released during a specific version and merged in the next release.

<figure><img src="image_2.png" /><figcaption>Figure 1: Number of patches that emerged after the release of a version and were merged with the next release. We plot the values for the recent 10 versions of the Xen hypervisor as of December 2021. The values are retrieved from Xen official website https://xenproject.org/.</figcaption></figure>

The effects of the patches should be known. However, before applying patches, their effects on the running VMs in terms of performance (CPU, disk, and memory usage) and overhead on the hypervisor should be known, since patches can undermine performance [29, 36, 64], or even lead to failures in some situations [48, 56]. To test the effect of patches on running VMs, the traditional method is to set up test servers, where the characteristics and behaviour of VMs is reproduced with carefully chosen applications [70]. Once the test environment is up, a batch of experiments inside the recreated VMs or at hypervisor level are launched, to evaluate potential overheads that could affect the running VMs. Unfortunately, mimicking VMs behaviour is a difficult task, principally due to their black box nature. Simply cloning VMs to create the test environment is not always an option, mainly due to:

(a) legal reasons, such as privacy [13, 22, 49, 58, 71]) in Cloud;

(b) it is time and resource consuming [40, 52, 57]; and

(c) in most of the cases, the input that triggers workload is not known (for example, because encrypted).

The limits of existing approaches. Previous solutions propose to log some externalized state of running VMs and then synthetically reproduce them [2, 20, 23, 39, 51] to obtain a similar workload behavior. These approaches, which we refer to as record-replay are composed of two main modules. The first module logs VMs and hypervisor servers state periodically or per-event occurrence
(e.g., the arrival of a packet on the NIC). The second module reads each log line and triggers an action or a set of actions inside a recreated VM. An example of an action is a memory allocation followed by random accesses and updates to mimic memory usage. These techniques, however, cannot be used to predict what will be the behavior of VMs on an updated system. This is due to two main reasons. The first reason is that each subsystem (e.g., memory or disk) is considered as independent, thus, it misses the correlation among different subsystems (Section 3). Thus, when one subsystem is updated, the record-replay mechanism will not be able to show its effect. The second reason is that the mechanism is strongly tied to system parameters. The epoch in the logs heavily depends on system parameters such as scheduling frequency. Thus, recreating VM workloads for an updated system will not be accurate.

Some forensic research works [5, 11, 19] present mechanisms to determine the exact applications running inside VMs. The problem with these approaches is that even if we correctly predict the applications inside the VMs, we must accurately identify the configuration and the inputs with which the application runs. Otherwise, replaying it in a test VM will make no sense since the results we obtain may differ depending on the application's actions. However, the exploration space for the configuration and input of applications is too large to be predicted correctly. Additionally, the application prediction is imperfect, which means a poor prediction will cause the recreated VMs to execute potentially wrong workloads and lead to wrong observations during the changes affecting the test.

**Our approach:** *UTwinVM*. In this paper, we propose Updated Twin VM (*UTwinVM*) that takes a different approach to get hints on potential virtualization system updates effects on VMs' applications. Instead of trying to reproduce exactly what is running inside the VM and perform manipulations to match the updated system, *UTwinVM* core idea is to determine how the updated system changes from the initial system. Thus, it can determine how VM applications could be affected. In a nutshell, *UTwinVM* records several metrics regarding the VMs and the virtualized system. We choose to observe metrics that can be observed from outside the VM to prevent intrusiveness and limit overhead on VMs. It also runs a set of specific workloads in a VM on the initial system and on a server with the updated system. These workloads are designed to benchmark all subsystems. With this information, we use a non-linear negative square solver (trained on several samples) to determine how the metrics obtained on the updated system differ from the initial. Finally, using specific scripts, *UTwinVM* replays workloads in VMs by using the recorded metrics on the VMs and the information provided by our solver. With this mechanism, the replayed workload is similar to an extent to the workload of the VMs if it were running on the updated system. The workload behavior can be observed to get hints on the potential impact of the system updates on VMs. Since *UTwinVM* relies on metrics already exposed by mainstream hypervisors, *UTwinVM* does not require virtualization system codebase modifications.

**Contributions.** Overall, this paper investigates the mechanisms of replaying a production environment composed of several VMs to measure the impact of a new environment on the applications running in the VMs. This paper makes at least the following contributions:

* A study of the efficiency of the standard *record-replay* techniques for reproducing at least VMs' CPU, memory, and IO consumption for an updated system;

* *UTwinVM*, an open source reproduction system that overcomes the limitations of a standard *record-replay*<fnref n="1" />;

* An extensive evaluation of *UTwinVM* that shows that *UTwinVM* incurs no overhead on VMs, less than 11% overhead on host services while providing accurate hints for several scenarios.

**Paper structure.** The rest of the paper is organized as follows: Section 2 presents background about patching and software testing in production environments, Section 3 presents the *record-replay* design and its shortcomings. Then, Section 4 introduces *UTwinVM*: its design and internal mechanisms. Section 5 presents our *UTwinVM* evaluation results. We then discuss some limitations and future work in Section 6. Lastly, Section 7 presents related work, and we conclude in Section 8.

## 2 BACKGROUND

**Lack of hints can cause reluctance to patch.** Companies must update their software stack to stay updated regarding security and performance. However, updating the software stack can have effects on running applications. Thus, it is essential to have hints of the potential effects of updates on running applications. These hints provide enough information for system administrators to decide if the resulting trade-off is worth updating. However, recent reports highlight the dilemma for companies to patch their systems. For example, the Cybersecurity and Infrastructure Agency (CISA) issued an alert regarding companies not applying the necessary patches to mitigate a security issue [17] that resulted in massive vulnerability exploitation. One of the reasons for the previous observations is the fear of side effects [18] — *how will the patches impact running services/containers/VMs*. Thus, there is a need to propose efficient and accurate ways to achieve the test process.

**Lack of hints can also slow down VM migration in data centers.** Knowing the effects of a system update on running applications and system resources is helpful in live migration. Live migration is a mechanism that enables the movement of a running VM from one server to another with a minimized downtime for the applications running within the latter. Thus, it is a critical task during fault tolerance [16, 35]. However, a considerable time is used to determine a suitable node for the migration to be effective and harmless for VMs [67]<fnref n="2" />. However, it is difficult to predict if minor changes between the initial and target system may cause performance degradation since there is no view of what the VMs are doing. Overall, having an accurate way to obtain hints can improve the decision of which node is suitable to host an incoming VM.

**Existing techniques overview.** The traditional method for testing updates (or patches) effects on production applications as illustrated in Figure 2 is to set up a testbed with the characteristics of your target environment: the triplet `<server physical resources, operating system, and applications> - setup` on Figure 2. Then,

<footnote n="1">¹All artifacts are available at https://github.com/djobii2078/UTwinVM/</footnote>

<footnote n="2">²In this paper, Alibaba mentions up to 15 days to find a suitable migration target for a large number of VMs</footnote>
reproduce the production environment state by launching appli-
cations with the same inputs and parameters as the production
servers — *rep* on Figure 2. The *rep* step enables the recording of
applications metrics to compare them later. The last step consists
of applying the updates to the host system and rerunning the appli-
cations — *upd* on Figure 2. The applications' metrics obtained at
phase *upd* can be compared with those at *rep* phase to get hints
on the potential effects of the updates.

<figure><img src="image_3.png" /><figcaption>Figure 2: Patch traditional testing process overview.</figcaption></figure>

However, when dealing with a virtualization environment, i.e.,
applications are not running on bare metal but inside virtual ma-
chines. Virtual machines are black boxes by design, which means
there's no direct way to understand what is happening inside. To
tackle this problem, two main classes of research work compete.
The first class consists of trying to predict which application is
running in the VM as illustrated with Bolt [19], KairosVM [11], or
CloudVMI [5]. If the application is correctly predicted, then the
information can be used to recreate the production environment.
However, these solutions have shortcomings since knowing the
applications running the VMs is not enough. To reproduce the exact
environment (the *rep* phase on Figure 2), the applications inputs,
sizes, and parameters must be known. Otherwise, the accuracy of
the process considerably drops.

The second class methods can be referred to as *record-replay*, which consists of logging metrics regarding several subsystems (I/O, memory, etc.) in a periodic or event-based manner. Then, the metrics and the associated timestamps are used by scripts to reproduce the behavior corresponding to the observed metrics (e.g., a memory usage of x1 MB and CPU usage of x2% at epoch *yyy*). This method is ideal for reproducing VM behaviors on a system similar to the original with identical physical characteristics. However, the updated metrics are no longer valid on an updated system since they were strongly tied to the system configuration on the original system.

In the next section, we show why *record-replay* techniques are
insufficient for our problem and derive a set of observations.

3 A RECORD-REPLAY APPROACH ?

A standard *record-replay* [39] for our problem will consist of logging
several metrics related to the VMs observable from outside the
VM. Then, it tries to reproduce these metrics via some controlled
programs. The key idea behind this approach is that reproducing
these coarse-grained metrics in a VM will generate the same load
as on the initial system. Then, the recreated VM can be audited
on the updated system to provide hints about the potential impact

of the system changes. To check whether existing *record-replay*
mechanisms can solve our problem, we evaluate a prototype of
*record-replay* following the previous works' architecture to evaluate
its accuracy and efficiency.

## 3.1 Implementation.

Several research works explore record-replay frameworks for vir-
tual machines such as ClusterRR [62], BluePrint [12], CrossCut [15],
and ReVirt [20]. They follow the same architecture but differ re-
garding memory overhead optimization, replay time, or storage
cost. Unfortunately, these works are not open-source. VMWare
WorkStation proposed a record replay feature but is no longer
supported since 2011 (v8.0) [60]. QEMU proposes a record replay
feature but does not support Linux/KVM, a mainstream virtualiza-
tion system. For all these reasons, we implement a custom prototype
of *record-replay*, which follows the same design described in the
aforementioned record-replay frameworks.

Our prototype is divided into two parts. A *record* module
triggered by the hypervisor runs on the initial system to fetch the
necessary metrics related to the system and the VMs. These metrics
include virtual CPUs (vCPUs) usage, disk read and write requests-
sizes provided by the virtual disk driver statistics, and network
receive and transmit requests-sizes provided by the virtual network
drivers statistics. To get memory usage from outside the VM (i.e.,
we do not use ballooning [25, 26, 61] as it is disabled in most public
Cloud VM offerings), we leverage the hardware feature Intel Page
Modification Logging (PML) feature which provides information
on memory pages updated by VMs. This method has already been
showcased in previous works [8]; we do not claim its contribution.
Additionally, it records the host CPU and memory usage. The metric
is collected periodically. The period corresponds to the tick of the
host scheduler. In our case, it corresponds to 3ms. We choose the
scheduler tick as the period to capture when there is a change in
the execution pipeline. The metrics are logged in a local file. The
second part of our prototype is a *replay* module that replays the
logs for each observed VMs and the host. It creates VMs with the
same configuration, reads the log files produced by the *record*
module line by line, and triggers the necessary actions in the VMs
and hypervisor to generate the load described by each line.

## 3.2 Evaluation.

To evaluate our *record-replay* prototype, we set up a server with 3
VMs with Linux/KVM (characteristics and testbed detailed in Sec-
tion 5). Then, we run a CPU benchmark (*SYSBENCH*) inside each VM.
We aim to determine if we can use the *record-replay* prototype to ob-
tain hints regarding a hypervisor update effect on VMs applications.
We consider the following scenario. We are considering a patch that
changes the hypervisor logic and want to know its potential effects
on our VMs. We consider two patches: the first is a simple patch
that changes the scheduling algorithm of the hypervisor to FIFO,
and the second is a patchset of 10 patches proposed by Yu Zhao
that target the hypervisor handling of VM page tables for better
performance [32, 69]. We replay using the recorded metrics of the
VMs on the corresponding updated systems (with FIFO and page
table handling). We then compare the performance of the recreated
VMs and the original VMs when the patch is applied.
<figure><img src="image_4.png" /><figcaption>Figure 3: *record-replay* results for the three VMs for the FIFO scheduling patch. The baseline corresponds to the original VMs metrics running on the updated virtualization system (FIFO scheduling). A value of 1 means that recreated VM metrics match the expected values.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4: *record-replay* results for the three VMs for the page table management patch. The baseline corresponds to the original VMs metrics running on the updated virtualization system (page table management). A value of 1 means that recreated VM metrics match the expected values.</figcaption></figure>

## 3.3 Observation and limitations

Figure 3 and 4 report the results obtained for the updated system with FIFO scheduling and page table handling improvement, respectively. For each VM, we plot the mean, 95th, and 99th percentiles of the difference between the recreated VM (with *record-replay*) and the original VMs running on the updated virtualization system. The tuples *netrx* : *nettx* and *diskrx* : *disktx* values represent the data sizes exchanged (packets or block requests) for the network and disk during the experiment. CPU and memory represent the vCPU usage and memory usage of each VM. A value of 1 (100%) means that the recreated VM metrics match the expected values.

For both patch scenarios, the results obtained with *record-replay* are bad, on average < 70%. The low value is due to the *rigidity* of the *record-replay* approach. These approaches do not evolve when the system from which the log originates changes. It cannot express the correlations between the virtualization system changes and the logs replaying. As a result, it leads to different scheduling decisions, cache usage, disk, and memory access patterns compared to the real VM configuration. Thus, they need to be improved to solve the problem we are trying to solve.

In the next section, we present *UTwinVM* that aims to overcome the limitations of approaches similar to *record-replay* to achieve better accuracy.

## 4 UTwinVM

*UTwinVM* reproduces the workloads observed in VMs on one server (the initial server hereafter denoted I-SERVER) on another server (the new server hereafter denoted N-SERVER) without the need to clone the latter VMs. Additionally, *UTwinVM* updates the workloads in the reproduced VMs to match the potential changes that could occur if VMs were running on a I-SERVER with an updated virtualization system. Thus, *UTwinVM* is meant to provide hints on the effect of changes to I-SERVER on its running VMs.

### 4.1 Virtualisation system representation in

*UTwinVM*

In our context, a server will run a virtualization system denoted by *V*, which powers the virtual environments (*VEs*)<fnref n="3" />. Thus, *S* can be described as a tuple *S* = < *hw*, *V*, *VEs* >, where *hw* represents the hardware characteristics of the server (total memory, number of cores, processor speeds, caches L1d & L1i, NIC, etc.), is the *V* the virtualization system that powers the virtual environments (*VEs*).

With *UTwinVM*, we consider that the virtualization system denoted *V* is characterized by several sub-systems that symbiotically interact with each other to ensure execution of virtual machines.

<footnote n="3">³In this paper, we just consider VMs.</footnote>
The sub-systems are as follows: the memory management sub-system denoted *mem<sub>mgt</sub>* , process management sub-system denoted *proc<sub>mgt</sub>* , block I/O management sub-system *disk<sub>mgt</sub>* , and network I/O management sub-system *net<sub>mgt</sub>* . The *mem<sub>mgt</sub>* sub-system represents the components responsible for ensuring virtual memory address translations and memory protection with the memory page tables by maintaining a shadow page table layer or leveraging hardware virtualization features such as AMD NPT or Intel EPT [44, 45, 54, 63, 66, 68]. The *proc<sub>mgt</sub>* handles the scheduling of VE's virtual CPUs (vCPUs) on the physical CPUs. The *disk<sub>mgt</sub>* management sub-system handles all the interaction with block devices (e.g., hard drives) on behalf of the VE's to read and write blocks. The *net<sub>mgt</sub>* is similar to *disk<sub>mgt</sub>* with the difference that it concerns network devices. In this work, we do not consider other devices such as FPGAs or GPUs, but the formalization can easily be extended to consider them. Thus, in the context of this work, V boils down to V =< *mem<sub>mgt</sub>* , *proc<sub>mgt</sub>* , *disk<sub>mgt</sub>* , *net<sub>mgt</sub>* >.

As mentioned previously, *VEs* represents the different virtual environment running in the server. Each *VE* is defined by the virtual characteristics it has been attached: memory, vCPUs, running OS, virtual block device driver, and network device driver. Consequently, we can represent a *VE* as follows: $\forall V \in V_Es, V = [ memory \ vCPUs \ OS \ vbd\_driver \ net\_driver ]$. All the applications running inside a *VE* are not supposed to be known since *VE* are black boxes.

## 4.2 UTwinVM architecture

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 5: Architecture of UTwinVM.** UTwinVM consists of mainly 4 modules. logger for collecting the necessary metrics at runtime, sysget for getting insights on the initial and target environment, ml for resource usage prediction, and exec to generate the resource usage.</figcaption></figure>

As shown in Figure 5, UTwinVM comprises four main modules. The first module, logger that runs on I-server, monitors events and metrics associated with the VMs and hypervisor's subsystems and records them in a log file. The second module, sysget, is a VM that runs once on both I-server and N-server (the one with the updated virtualization stack) and collects metrics related to the virtualization system. These metrics represent the performance level of the virtualization system. Thus collecting them on the I-server and N-server provides an understanding of the differences between virtualization systems on both servers. The third module, ml, is a machine learning model trained to determine, for a resource usage log entry collected on I-server, the new resource usage based on the virtualization system on N-server. ml goal is to understand the

correlations between the virtualization sub-systems layers and the resource logs and to correctly predict the resource usage values depending on the changes observed in the virtualization system. The last module, exec, spawns VMs and launch specialized programs inside each VM, to generate the new observed resource values. Thus, the reproduced system can be audited to provide hints regarding the effects of a given change on I-server virtualization system.

In the next sections, we dive into each module to describe their core functionality and their implementation.

## 4.3 logger

<figure><img src="image_8.png" /><figcaption>**Figure 6: Overview of logger**</figcaption></figure>

**Description.** This module collects the necessary information that will be used to generate the workload in N-server. As shown in Figure 6, it monitors each VM and collects runtime metrics on each virtualization subsystem (as described in Section 4.1), and records them in a log file. logger populates the log file on an event-periodic basis. It captures all the events related to a virtualization system, e.g., non-maskable interrupt (NMI) sent to a VM or a VM EXIT. Additionally, it periodically captures VM-related metrics. To capture most of the system, we use the scheduler tick as the default period. At each tick, for each VM, logger logs the:

* cpu usage — expressed in percentage, which is the percentage of CPU time slice used (concerning the vCPUs allocated),

* memory usage — expressed in percentage, which is the percentage of memory actively (the working set),

* disk read and disk write — expressed in bytes, representing the number of bytes read/written from/to sectors on the attached block device. These values are exposed by the virtual block interface attached to the VM.

* disk read rqs and disk write rqs represent the number of read/write requests sent to the attached block device.

* net tx and net rcv — expressed in bytes, represent the number of bytes transmitted/received to/from the attached network device. These values are exposed by the virtual network interface attached to the VM.

* net tx rqs and net rcv rqs, which represents the number of transmit and write requests sent to the attached network device.

As mentioned previously, in this work, we focus on CPU, memory, disk, and network devices. However, it can be extended to GPUs, floppy disks, or audio devices.

**Optimization.** While designing logger, we kept two essential parameters in mind. The first is overhead on running VMs. The mechanisms used to collect these metrics are lightweight and are delayed/postponed whenever server resources usage is high to limit
the burden imposed on the latter. This is crucial as we don't want to affect the execution state of applications running in VMs. The second aspect is off-the-shelf usage. This implies that the tools we use already exist in default packages of mainstream virtualization systems. A tool that will require patching the virtualization or operating system would have a low impact since it will require a reboot of the virtualization system to support UTwinVM.

## 4.4 sysget

**Description.** `sysget` provides information on the characteristics of the virtualization system on both I-server and N-server (with the updated virtualization layer). `sysget` is essential to capture the differences between the two environments. To achieve this, `sysget` is basically a VM that runs on both the I-server and N-server, and runs a set of specialized benchmarks. For accuracy, `sysget` should be attached to a virtual network and block device interface similar to VMs already running. Thus, the results reported will reflect those of the VMs already present on the server. The benchmarks inside `sysget` reports metrics related to each virtualization subsystem. The metrics are:

* `mem_{alloc, dealloc}`, which is an array that represents the CPU time required to allocate/free 1k, 4k, and 1MB of memory. These values (`mem_alloc` and `mem_dealloc`) capture the `mem_mgt` subsystem as presented in Section 4.1.

* `schedsurf`, which represents the average CPU time required to yield a vCPU in the VM.

* `sched enter`, which represents the average CPU time required to schedule a new task on a VM vCPU. The values `schedsurf` and `sched enter` captures the `proc_mgt` subsystem.

* `disk_{read, write}`, which is an array that represents the CPU time required to read/write a block of 10MiB and 100MiB. This value captures the `disk_mgt` subsystem.

* `net_{tx, rcv}`, which is an array that represents the CPU time required to transmit/send a network packet of 1K and 4K. This value captures the `net_mgt` subsystem.

Getting these values on both I-server and N-server provides enough insights into each server's characteristics and what changed between the two. These values are then used as input along with the logs collected by `parser` in the prediction module `ml`.

**Optimization.** For a more accurate model, we introduce several levels of background CPU load when running `sysget`. This allows the model to have data on the behavior of the VM when there is CPU contention.

## 4.5 ml

**Description.** `ml` is the core module of UTwinVM. It is responsible for converting the logs obtained on the initial system and converting them on the basis of the changes observed on the updated system.

Let us recall the notation introduced in Section 4.1. If we consider the initial system, $S = \langle hw, V, VEs \rangle$, $W_{loads}$ the workloads running inside the VEs, and $\alpha_o$ the metrics obtained with `logger`. Given a new system $S' = \langle hw, V', VEs \rangle$, $V' \neq V$, we need to obtain $\alpha_x$ i.e., the logs that the VMs would have generated on $S'$ while running $W_{loads}$. With this representation, $\alpha_x$ can be perceived as

derived from $\alpha_o$, and added to the observed difference between V and $V'$. This gives us:

$$ \alpha_x = \alpha_o + \sum_{i=1}^{n} \omega_i \| X_i^S - X_i^{S'} \| $$

where $X_i^S$ is a feature obtained by `sysget` on the server S. Finding the suitable coefficient is now a matter of finding the right algorithm. We use a non-negative least squares (NNLS) [9, 55] solver to find the model that best fits our training data. NNLS fits our use case very well, as it ensures that each parameter contributes some non-negative amount to the overall result. NNLS is also useful for feature selection, as it sets coefficients that are not relevant to a particular job to zero, thus helping identify what features we need.

**Optimization.** Since we use nonnegative least squares (NNLS), the residual errors are not normally distributed and simple techniques for computing confidence limits are not applicable. Thus, we use cross-validation, where subsets of the training data can be used to check if the does not overfit and generalizes well. Several cross-validation techniques exist, but we use a leave-one-out-cross-validation scheme as our training data size is relatively small (16686 observations). Concretely, for m training samples, we perform m cross-validation runs where each run uses m - 1 points as training data and tests the model at the left-out data point. Using the scheduler tick as the default period with `logger` comes from the $R^2$ score and accuracy obtained with our model (see Table 1). Since the 3ms (our system's scheduler tick) captures more data per unit of time, it can be inferred more accurately than the other periods.

Notice that `ml` does not need to run on the initial server, and thus does not introduce overhead on the VMs.

<table><caption>Table 1: $R^2$ and accuracy of `ml` based on the period of the training samples.</caption><thead><tr><td>Period</td><td>R<sup>2</sup> (%)</td><td>Accuracy(%)</td></tr></thead><tbody><tr><td>3ms</td><td>95.2</td><td>98.3</td></tr><tr><td>50ms</td><td>92.3</td><td>95.1</td></tr><tr><td>100ms</td><td>91.2</td><td>94.9</td></tr><tr><td>250ms</td><td>91</td><td>94.8</td></tr><tr><td>500ms</td><td>91</td><td>94.8</td></tr><tr><td>1s</td><td>91</td><td>94.8</td></tr></tbody></table>

## 4.6 exec

**Description.** `exec` is the last module of UTwinVM, and is in charge of replaying the transformed logs on the new server. Simply put, it spawns a replay process for each VM. Each process reads its corresponding log file and, for each line, triggers the corresponding action to generate CPU usage. We leverage `cgroups` inside the VM to easily control the resource usage levels. For the specific case of network receive generation, we use a client on the host that sends packets of variable size to generate the network receive load in the target VMs. Thus, during the execution of this module, other programs can run to collect useful metrics or diagnose the running system, providing the necessary insights on the effects of the changes performed (on the virtualization layer) on the running VMs and overall system.
**Optimization.** Replaying the CPU and memory can be tricky since they are components that are affected by the others. Thus, for more accuracy, we perform a ahead-of-time computation to predict the CPU and memory usage that will generate all other resources (network and block devices). And based on the prediction, the replay process will just replay the remaining CPU and memory usage obtained by subtracting the predicted one from the log value. Thus, the replay is more accurate regarding the log values.

## 4.7 Prototype implementation details

We built a *UTwinVM* prototype that targets the Linux/KVM [30] and the Xen v4.17.0 [6] hypervisors, used atop of the Libvirt toolkit v10.1.0 [33]. The *logger* leverages the Libvirt OCaml library to collect all the necessary information about the VMs running, apart from memory usage within the VM. Several techniques have been developed for memory usage to perform a non-intrusive working set evaluation of VMs [8, 28, 38, 46]. We use a technique similar to the one described in [8] with *UTwinVM* as mentioned in Section 3 that leverages the Intel PML feature.

*sysget* is an Ubuntu server 16.04.7 LTS virtual image that contains the set of benchmarks (mostly written in C language) necessary to assess the virtualization system underneath. *ml* is mostly powered by the Python SciPy library [50], trained over 16686 samples obtained from several controlled scenarios, and evaluated with cross-validation (as described in Section 4.5). *exec* combines *cgroups* with *iperf* [27] and *fio* [4] for network and disk load generation, respectively, in addition to custom C applications for memory and CPU load generation.

# 5 EVALUATION

We evaluate *UTwinVM* to determine (1) the additional overhead of using *UTwinVM* on production VMs and host services and (2) if it can provide correct hints regarding the impact of an update on the virtualization system. To answer these two questions, we design several experimentation scenarios to evaluate *UTwinVM'* overhead(s) and accuracy.

**Testbed and benchmarks.** Our experiments are carried out on 2 identical Supermicro X11DPI-N based servers. Each with dual socket Intel® Xeon® Gold 6230R @ 2.10GHz, with 768GB of RAM, dual 1Gbps Ethernet, dual 25Gbps Ethernet, and 1TB HDD. Each server has 104 cores. We use the Linux/KVM hypervisor as our virtualization system alongside Ubuntu Server 18.04.5 LTS with Linux Kernel 6.6.4. Note that since *UTwinVM* does not rely on a specific kernel feature of the Linux kernel 6.6.4, its usage on more recent kernels is not compromised. Each VM runs Ubuntu Server 14.04.6 LTS and, except otherwise specified, is configured with 2vCPUs and 1GB of memory. We chose this number of cores for our VMs as it corresponds to one of the most popular configurations in Cloud platforms [24]. Unless otherwise specified, the results reported are the average of 10 runs of the experiment, which are enough to achieve 95% confidence intervals $\le 3\%$ on all runs. All benchmarks used during the evaluations are presented in Table 2.

## 5.1 Overhead

*UTwinVM* can introduce overhead during the metric collection phase with the *logger*. Note that *sysget* can run on a server similar

to the one hosting VMs and is not forced to run alongside running VMs. However, in this evaluation, we consider the worst case, where *sysget* is also running on the initial server. Concretely, we are interested in assessing the overhead induced on the server in terms of resources used and the impact on running VMs. To this goal, we design the following experiments.

**Experimental procedure.** In a server running several VMs, we run *logger* to collect metrics for 5 minutes (300 s) with different periods for the periodic metrics. Each VM runs *SYSBENCH* for 6 minutes (360s) to stress the CPU. We record the overhead generated on the server resources (memory and CPU) when running *logger* as we increase the number of VMs (1 – 80). Additionally, we record the overhead regarding the sysbench running inside the VMs when *logger* runs. We repeat the same experiment with *sysget*. *sysget* is allocated 2vCPUs and 2GB of memory.

Furthermore, to assess the potential overhead on the administration services<fnref n="4" /> running on the host, we also measure the overhead generated by *logger* on three different benchmarks when they run on the host: STREAM, 7ZIP, and GEEKBENCH<fnref n="5" />. We run these benchmarks in the host for the most demanding scenario (i.e., 80 VMs and scheduler tick - 3ms - as the period).

**Results regarding overhead on host resources and services.** Figure 8 reports the CPU usage generated by *logger* while varying the number of VMs on the host. We observe that *logger* generates a low overhead ranging from 0.9% to 11.8% on average over the 104 cores. The usage of the default OCaml library of libvirt can explain this low overhead. Additionally, the collection process CPU affinity is fixed on two cores to limit synchronization between a large number of cores. Furthermore, *logger* overhead does not increase concerning the number of VMs to monitor, which makes it suitable for large-scale systems. As a result of low CPU overhead, the applications running in the host are lightly affected, as shown in Figure 7. We observe less than 11% overhead for the different applications of the three benchmarks.

**Results regarding the impact on VMs.** *logger* effect on VMs is negligible. The overhead of the *SYSBENCH* benchmark results running in the VMs when *logger* is running is 0.003%, therefore, almost zero. This matches the low CPU overhead generated on the host, independently of the period used. Similarly, *sysget* does not hinge on the VMs activities. Being a VM, *sysget* is allocated a fixed amount of resources and does not affect the colocated VMs. Another advantage of *sysget* is its short lifespan. Although VMs can affect each other by generating enough load to stress the host [14, 42, 43, 53, 65], the execution window of *sysget* coupled to its low resource needs results in a negligible overhead. Notice that *sysget* can also be executed on a dummy server having the same characteristics as the one in production.

## 5.2 Hints accuracy

The final goal of using *UTwinVM* is to obtain accurate hints. We evaluate if *UTwinVM* can provide accurate hints regarding the potential impact of an update of the virtualization system on VMs.

<footnote n="4">⁴Like datacenter frameworks services, e.g., OpenStack Compute (Nova) [47]</footnote>

<footnote n="5">⁵We just run three benchmarks of the GeekBench suite since they span the range of applications we needed</footnote>
<table><caption>Table 2: List of benchmarks used</caption><thead><tr><td>Benchmarks</td><td>Description</td></tr></thead><tbody><tr><td>Sysbench [31]</td><td>Sysbench is an open-source and multi-purpose benchmark utility to evaluate CPU, memory, I/O, and also databases (MySQL) performance. We principally use Sysbench for stressing the CPU.</td></tr><tr><td>Stream [37]</td><td>STREAM is a simple synthetic benchmark program that measures sustainable memory bandwidth (in MB/s) and the corresponding computation rate for simple vector kernels.</td></tr><tr><td>7zip [1]</td><td>Compression program and comes with a benchmark built-in which utilizes LZMA method to compress and decompress 32MB blocks of data.</td></tr><tr><td>Geekbench [21]</td><td>Cross-platform benchmark that includes workloads that model real-world tasks and applications. Geekbench is a benchmark that reflects what actual users face on their devices. We use Geekbench 6.</td></tr><tr><td>LAMP</td><td>Linux Apache MySQL PHP. Form a web server coupled with a transactional database. We use it to simulate a real-world web server hosted in a VM.</td></tr><tr><td>Spec CPU 2017 [10]</td><td>It provides a comparative measure of integer and/or floating point compute intensive performance</td></tr></tbody></table>

<figure><img src="image_9.png" /><figcaption>Figure 7: Performance of three benchmarks running in the host w/o the logger. We consider the most demanding scenario in our test bed, i.e., 80VMs and the scheduler tick as period (3ms).</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 8: CPU usage of the server w/o the logger per simultaneous running VMs.</figcaption></figure>

To assess the accuracy, we design the following experiments.

a. **Experimental procedure.** On a server composed of 80VMs, we run SYSBENCH in each VM for a fixed amount of time, 10 minutes (600s). We denote this configuration, $S_b$, $b$ for base. Then, we patch the virtualization system to update the scheduling algorithm used. We apply two patches, with each patch changing the scheduling algorithm from Credit Fair Scheduler (CFS) to Round-Robin (RR) and from CFS to First-in First-Out (FIFO). For each patch, We rerun SYSBENCH in the VMs for the updated virtualization systems and collect, for each VM, the following metrics:

* CPU usage (% relative to the number of vCPUs allocated to the VM),

* memory usage (% relative to the total memory of the virtual machine),

* network transmission throughput (nettx),

* network reception throughput (netrx),

* disk read throughput (diskrd),

* disk write throughput (diskwr),

* number of TLB_FLUSH, and

* number of VM_EXITS
We denote this configuration $S_p$, $p$ for patch.

We now use *UTwinVM* to get hints on the effects of applying the two patches on our VMs and compare them to the actual effects observed in $S_p$. Thus, we use *UTwinVM* to launch on a similar server, VMs whose behavior is supposed to mimic the initial VMs as if the patch was applied. We denote the results obtained with *UTwinVM* for each patch: UTWINVM-RR and UTWINVM-FIFO. We repeat the experiment with the *record-replay* approach which we denote *RecRp* on the graphs.

We repeat the same experiment concerning a patch that changes the I/O scheduler from *MQ-DEADLINE* to a *NOOP* scheduler. We denote the results obtained with *UTwinVM* for the *NOOP* patch by UTWINVM-NOOP.

Furthermore, we perform the same experiment regarding the patchset that updates page table (PT) management by the hypervisor mentioned in Section 3 [32, 69]. We denote the results obtained with *UTwinVM* for this configuration by UTWINVM-PT.

**b. Results for the task scheduler.** Figure 9a and Figure 9b plots the difference in the metrics obtained in $S_p$ and *UTwinVM* regarding $S_b$ for one VM for the RR and FIFO patch respectively. A value of 1 means that the results of the VM with $S_b$ and $S_p$ (or *UTwinVM*) are the same (the patch did not affect the VM). Overall, on average, *UTwinVM* results slightly differ from $S_p$ by 3%. However, the tendency is similar, and the overall observation matches *UTwinVM* results. The *record-replay* approach achieves a precision value of 0.65 up to 0.82 which shows that *UTwinVM* outperforms *record-replay*. The results can be used to have hints about the effects of these patches on VMs.

**c. Results for the I/O scheduler.** The same trend is observed with the I/O schedulers. As shown in Figure 10, the difference of *UTwinVM* - *NOOP* regarding $S_b$ *mq* - *deadline* slightly differs on average by 7% higher compared to the difference observed by $S_p$ *NOOP*. However, as with the task schedulers, the tendency is the same; thus, the hints obtained by analyzing the graph are accurate. The *record-replay* approach achieves a precision of 0.72 and up to 0.84, showing the limitations of this approach.

**d. Results for the page table update patchset.** Figure 11 reports the difference obtained in $S_p$ and *UTwinVM* regarding $S_b$ for the corresponding patchset. The results follow the same trends as described in the previous sections. The difference between *UTwinVM* and *PT* is less than 6.2%, highlighting the accuracy of *UTwinVM* and strengthening the fact that *UTwinVM* is effective in providing hints on hypervisor-level patch effects. The *record-replay* accuracy is worse for this scenario, with precision values between 0.34 up to 0.57.

**e. More results for the tasks' scheduler.** Figure 12 provides more insights on the difference between the $S_p$ RR and *UTwinVM* version for 6 VMs of our setup regarding the RR task scheduler. Regarding resource usage, for brevity and better visibility, we plot the average, 95th, and 99th percentile of the difference between each resource usage for each epoch between the different configurations. Apart from VM5, where the accuracy is in the interval [93%, 96%], all other VMs exhibit an accuracy > 97% for all metrics. The *record-replay* approach achieves precision between 77% and 89%.

<figure><img src="image_11.png" /><figcaption>(a) A VM resource usage difference when using RR as the virtualization system task scheduler. Baseline is $S_b$ with *CFS*. We compare the difference obtained with RR, *UTwinVM* - RR replay, and *RecRp* - RR replay.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>(b) A VM resource usage difference when using FIFO as the virtualization system task scheduler. Baseline is $S_b$ with *CFS*. We compare the difference obtained with *FIFO*, *UTwinVM* - *FIFO* replay, and *RecRp* - *FIFO* replay.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 10: A VM resource usage difference when using *NOOP* as the virtualization system I/O scheduler. Baseline is $S_b$ with *mq*-deadline. We compare the difference obtained with *NOOP*, *UTwinVM* - *NOOP* replay, and *RecRp*-NOOP replay.</figcaption></figure>

**f. Results for inner metrics.** However, despite *UTwinVM* being able to reproduce with very high accuracy macro metrics such as CPU usage or network throughput, the accuracy drops when it
<figure><img src="image_14.png" /><figcaption>Figure 11: A VM resource usage difference when using the patchset we denote PT. Baseline is $S_p$. We compare the difference obtained with PT, *UTwinVM* – PT replay, and *RecRp-PT* replay.</figcaption></figure>

comes to VM internal metrics. As shown in Figure 13, the number MMU_CACHE MISS, TLB FLUSH, and VM EXITS observed for a *UTwinVM* VM is on average 21% lower than the values $S_p$.$^6$. Reproducing the inner behavior of the VM is more complex as it depends on the nature of the applications (CPU-intensive, memory, spinning app, etc.) running inside the VM. Similarly to *UTwinVM*, the record-replay approach does not perform well for inner metrics with precision values going down to 61%. However, the overall analysis from the macro metrics shows that *UTwinVM* provides accurate hints to understand the effects of the patches on VMs.

## 5.3 LAMP and Spec CPU 2017

We also evaluate *UTwinVM* using two applications that represent real-world use cases. The first application is the LAMP bundle – i.e., Linux, Apache, MySQL, and PHP. It represents a 3-tier web server serving dynamic pages per incoming HTTP requests. We leverage the `AB` tool [3] to generate load on the web server.

The second application is Spec CPU 2017, which is a CPU-intensive suite for measuring and comparing compute-intensive performance, stressing a system’s processor, memory subsystem, and compiler [10]. Spec CPU 2017 contains a variety of 47 benchmarks that span a wide range of usecases.

**Experimental procedure.** We reproduce the experimental procedure of Section 5.2 while using LAMP and Spec CPU 2017 benchmarks as our applications. We consider all the patching scenarios: RR, FIFO, PT, NOOP, and MQ-DEADLINE and compare *UTwinVM* results with $S_p$ with respect to the baseline $S_b$.

**Results.** Due to space constraints, we report results for only the three patching scenarios RR, FIFO, and PT. Additionally, we only report the result for 2 benchmarks randomly selected from the 47 benchmarks proposed by Spec CPU 2017. The two benchmarks are `blender_r` and `gcc_r`, respectively, representing 3D rendering and GNU C compiling. However, the results and observations are similar to the following observations.

Figure 14, 15, and 16, reports the results for the recreated VM for each patch scenario for the LAMP benchmark, and `BLENDER_R` and `GCC_R` benchmarks respectively. Overall, independently of the

benchmark and patching scenario, the precision ranges between 95.3 and 99.4%. The record-replay approach achieves precision between 82% and 85%. This highlights the efficiency of *UTwinVM* for complex applications encountered in the real world.

## 5.4 Evaluation summary

Our evaluation focused on the overhead and accuracy of *UTwinVM*. The results showed that the *UTwinVM* data collection process on VMs at runtime incurs no overhead for applications running inside the VMs and less than 11% on the host CPU. As a result, the overhead of running the services on the host remains low. Furthermore, the results showed that *UTwinVM* could effectively reproduce workloads while considering the changes in the virtualization system. The accuracy compared to our baseline is > 95%, and an administrator can analyze the results to have accurate hints.

*UTwinVM* can also reproduce, with an accuracy of ≈ 89%, internal metrics such as TLB FLUSH or VM EXITS, but this aspect can be improved as discussed in the next section. Finally, the mechanism and results shown by these evaluation results are promising and will be helpful for companies willing to achieve such operations without peeking into their VMs.

# 6 LIMITATIONS AND DISCUSSIONS

Despite *UTwinVM* results, it faces some limitations, which we hereby discuss alongside potential future works around them.

**Inner Metrics Reproduction.** A significant limitation of *UTwinVM* is its accuracy regarding internal VM metrics. As discussed in §5, *UTwinVM* inner metrics reproduction accuracy is around 82%. An ideal approach is to observe what tasks are running inside the VM, their parameters, and how they are spawned. However, this is not legally possible. An approach we are currently exploring to improve our current ML model to understand the correlations between the virtualization system events (VM EXITS, non-maskable interrupts (NMI) injection, hypercalls, etc.) and the type of request running in the VM OS. The resulting model could understand the changes in the virtualization system and use the latter model to refine the applications used to replay the resource usage.

**Hardware dependency.** To be complete, *UTwinVM* should consider the changes in the hardware configuration between the initial and replayed environment. It will be a handy tool for migration operations or testing mechanisms before purchasing new hardware. However, the complexity increases with a different hardware setup. The clock speed of CPUs, the different NUMA nodes and sockets, and the bus and cache sizes, to name a few, are all variables that can change between two hardware setups and drastically affect the replay process. An approach worth exploring is modeling the performance of an application based on the hardware characteristics and being able to guess what the output will be. The idea is similar to using hardware simulators such as Gem5 [7] but without the cost of emulation and the possibility to run big datasets very fast.

# 7 RELATED WORK

**Black box introspection.** There exist recent efforts at achieving black box introspection for several purposes, notably forensics. The most recent related work is Bolt [19]. Bolt identifies the characteristics of VM applications by leveraging shared resources (such

<footnote>$^6$Independently of the scheduling algorithm being used</footnote>
<figure><img src="image_15.png" /><figcaption>Figure 12: Resource usage accuracy compared of UTwinVM for RR. The baseline is $S_p$ with RR scheduling algorithm. For brevity, we show 6VMs.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 13: A VM MMU_CACHE MISS, TLB FLUSH, & VM EXITS between the baseline $S_p$, UTwinVM, and RecRp recreated environment. We report values for the NOOP I/O scheduler, FIFO and RR task scheduler, and the PT patchset.</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 14: VM running LAMP resource's usage difference for different patching scenarios: RR, FIFO, and PT. The baseline is $S_b$ with CFS, and we compare $S_p$ with UTwinVM, and RecRp.</figcaption></figure>

as caches) information. Malicious users can later exploit the latter knowledge to design a security attack. While Bolt can identify, to some extent, the applications running inside a VM, the idea isn't easy to reuse in this work. Indeed, to be helpful, we will need to know the applications' input parameters and how they contribute to the VM's total load. UTwinVM takes a different approach by extracting existing external information to understand how they

will change accordingly to the virtualization system. Kairos [11] and Cloudvmi [5] propose introspection techniques to predict the future needs in terms of computing resources of a VM, thus reducing the latency of such future operations. UTwinVM do not use introspection, as this usually requires patching the virtualization system to leverage the necessary information. One of our objectives is to reuse existing interfaces to ease adoption and limit overhead.
<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /><figcaption>Figure 15: VM running *blender_r* benchmark (from Spec CPU 2017) resource's usage difference for different patching scenarios: RR, FIFO, and PT. The baseline is $S_b$ with CFS, and we compare $S_p$ with *UTwinVM*, and *RecRp*.</figcaption></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /><figcaption>Figure 16: VM running *gcc_r* benchmark (from Spec CPU 2017) resource's usage difference for different patching scenarios: RR, FIFO, and PT. The baseline is $S_b$ with CFS, and we compare $S_p$ with *UTwinVM*, and *RecRp*.</figcaption></figure>

Other research works mainly focus on efficiently replicating VMs workloads based on pre-recorded traces [2, 20, 23, 39, 51]. They aim to replay a VM's behaviour to check or confirm whether it follows a given pattern. However, *UTwinVM* extends the idea by predicting how this behaviour changes if we change the environment's settings where the traces were collected. Thus, *UTwinVM* provides a mechanism to perform forensics and predict behaviour on updated virtualized systems.

**VM benchmarking.** Tools to ease VM benchmarking have been emerging in recent years. Their main objective is to provide fast, scalable, and efficient methods for testing an application's configuration and obtaining results (inside a VM). Justin et al. [41] state that: *a fundamental challenge facing these projects, however, is how to conduct practical scientific experiments that provide insight into these new environments, referring to the challenges of setting up new environments for an IT company. Besson et al. [59] propose a ranking method to provide the best configuration needed (in terms of CPU and memory) for your application to yield the best performance. UTwinVM does not use ranking or clustering methods, as the set of possible configurations is unknown beforehand. However, using UTwinVM, an administrator can know if new system configurations will permit the existing applications to run correctly. Yuhui et al. [34] also aim to classify VMs to find the suitable configuration for an application. It leverages semi-supervised non-negative matrix factorization.

# 8 CONCLUSION

In this paper, we describe *UTwinVM*, a quick and effective way to mimic the behaviour of VMs and replay the changes as they would when faced with an update in the virtualization system. *UTwinVM*

leverages existing tools to gather information about the VMs and apply NNLS to predict the behaviour on the target system. Our evaluation shows up to 99% accuracy in the behaviour reproduction while changing several components of the virtualization system. Additionally, the cost of *UTwinVM* in the production environment is very low, thus preserving VMs' response time. Despite room for improvement, *UTwinVM* can be used by system administrators to get valuable and accurate hints regarding the effects of updates on the virtualization system.

# 9 ACKNOWLEDGMENTS

We thank our shepherd, Xin Jin, and the anonymous reviewers for their insightful comments. This work was partially supported by Rennes Métropole (Program "Allocation d'installation scientifique", AIS, Rennes, France), along with the French ANR project sGOV (ANR-23-CE25-0007-01).

# REFERENCES

[1] 7zip. 7-benchmark, 2008. https://www.7-cpu.com/util.shtml - Last accessed on 15/07/2021.

[2] Steve Herrod (VMWare Alumus). The amazing vm record/replay feature in vmware workstation 6, 2007. http://tiny.cc/aximuz - Last accessed on 10/07/2021.

[3] Apache. ab - apache http server benchmarking tool. https://httpd.apache.org/docs/2.4/programs/ab.html.

[4] Jens Axboe. Fio-flexible i/o tester synthetic benchmark. https://github.com/axboe/fio (Accessed: 2021-06-13), 2005.

[5] Hyun wook Baek, Abhinav Srivastava, and Jacobus Van der Merwe. Cloudvmi: Virtual machine introspection as a cloud service. In *2014 IEEE International Conference on Cloud Engineering*, pages 153–158, 2014.

[6] Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt, and Andrew Warfield. Xen and the art of virtualization. *ACM SIGOPS operating systems review*, 37(5):164–177, 2003.

[7] Nathan Binkert, Bradford Beckmann, Gabriel Black, Steven K Reinhardt, Ali Saidi, Arkaprava Basu, Joel Hestness, Derek R Hower, Tushar Krishna, Somayeh
Sardashti, et al. The gem5 simulator. *ACM SIGARCH computer architecture news*, 39(2):1-7, 2011.

[8] Stella Bitchebe, Djob Mvondo, Laurent Réveillère, Noël de Palma, and Alain Tchana. Extending intel pml for hardware-assisted working set size estimation of vms. In *Proceedings of the 17th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments*, VEE 2021, page 111-124, New York, NY, USA, 2021. Association for Computing Machinery.

[9] Rasmus Bro and Sijmen Jong. A fast non-negativity-constrained least squares algorithm. *Journal of Chemometrics*, 11:393-401, 09 1997.

[10] James Bucek, Klaus-Dieter Lange, and Jóakim v. Kistowski. Spec cpu2017: Next-generation compute benchmark. In *Companion of the 2018 ACM/SPEC International Conference on Performance Engineering*, pages 41-42, 2018.

[11] Kevin Burns, Antonio Barbalace, Vincent Legout, and Binoy Ravindran. Kairosvm: Deterministic introspection for real-time virtual machine hierarchical scheduling. In *Proceedings of the 2014 IEEE Emerging Technology and Factory Automation (ETFA)*, pages 1-8, 2014.

[12] Anton Burtsev, David Johnson, Mike Hibler, Eric Eide, and John Regehr. Abstracts for practical virtual machine replay. In *Proceedings of The 12th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments*, VEE '16, page 93-106, New York, NY, USA, 2016. Association for Computing Machinery.

[13] Deyan Chen and Hong Zhao. Data security and privacy protection issues in cloud computing. In *2012 International Conference on Computer Science and Electronics Engineering*, volume 1, pages 647-651, 2012.

[14] Young-ri Choi and Hoon Choi. Evaluating the performance of resource overcommitted virtualized systems. In *2010 Proceedings of the 5th International Conference on Ubiquitous Information Technologies and Applications*, pages 1-6, 2010.

[15] Jim Chow, Dominic Lucchetti, Tal Garfinkel, Geoffrey Lefebvre, Ryan Gardner, Joshua Mason, Sam Small, and Peter M. Chen. Multi-stage replay with crosscut. *SIGPLAN Not.*, 45(7):13-24, mar 2010.

[16] Christopher Clark, Keir Fraser, Steven Hand, Jacob Gorm Hansen, Eric Jul, Christian Limpach, Ian Pratt, and Andrew Warfield. Live migration of virtual machines. In *Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation-Volume 2*, pages 273-286, 2005.

[17] Cybersecurity and Infrastructure Security Agency. Continued threat actor exploitation post pulse secure vpn patching, 2020. https://us-cert.cisa.gov/ncas/alerts/aa20-107a.

[18] DataCenterKnowledge. Why some data centers don't patch and why they should, 2020. http://tiny.cc/6ximuz.

[19] Christina Delimitrou and Christos Kozyrakis. Bolt: I know what you did last summer... in the cloud. In *Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS '17*, page 599-613, New York, NY, USA, 2017. Association for Computing Machinery.

[20] George W. Dunlap, Samuel T. King, Sukru Cinar, Murtaza A. Basrai, and Peter M. Chen. Revirt: Enabling intrusion analysis through virtual-machine logging and replay. *SIGOPS Oper. Syst. Rev.*, 36(SI):211-224, December 2003.

[21] Geekbench. Geekbench 5.4. https://www.geekbench.com/ - Last accessed on 4/04/2021.

[22] Michelle Goddard. The eu general data protection regulation (gdpr): European regulation that has a global impact. *International Journal of Market Research*, 59(6):703-705, 2017.

[23] J.B. Grizzard and R.W. Gardner. Analysis of virtual machine record and replay for trustworthy computing. 32:528-535, 09 2013.

[24] Ori Hadary, Luke Marshall, Ishai Menache, Abhisek Pan, Esaias E Greeff, David Dion, Star Dorminey, Shailesh Joshi, Yang Chen, Mark Russinovich, and Thomas Moscibroda. Protean: VM allocation service at scale. In *14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)*, pages 845-861. USENIX Association, November 2020.

[25] Michael R. Hines, Umesh Deshpande, and Kartik Gopalan. Post-copy live migration of virtual machines. *SIGOPS Oper. Syst. Rev.*, 43(3):14-26, jul 2009.

[26] Michael R. Hines and Kartik Gopalan. Post-copy based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning. In *Proceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments*, VEE '09, page 51-60, New York, NY, USA, 2009. Association for Computing Machinery.

[27] Chung-Hsing Hsu and Ulrich Kremer. Iperf: A framework for automatic construction of performance prediction models. In *Workshop on Profile and Feedback-Directed Compilation (PFDC)*, Paris, France. Citeseer, 1998.

[28] Jingyuan HU, Xiaokuang Bai, Sai Sha, Yingwei Luo, Xiaolin Wang, and Zhenlin Wang. Working set size estimation with hugepages in virtualization. In *2018 IEEE Intl Conf on Parallel Distributed Processing with Applications, Ubiquitous Computing Communications, Big Data Cloud Computing, Social Computing Networking, Sustainable Computing Communications (ISPA/IUCC/BDCloud/SocialCom/SustainCom)*, pages 501-508, 2018.

[29] Zhichao Hua, Dong Du, Yubin Xia, Haibo Chen, and Binyu Zang. EPTI: Efficient defence against meltdown attack for unpatched VMs. In *2018 USENIX Annual Technical Conference (USENIX ATC 18)*, pages 255-266, Boston, MA, July 2018.

USENIX Association.

[30] Avi Kivity, Yaniv Kamay, Dor Laor, Uri Lublin, and Anthony Liguori. kvm: the linux virtual machine monitor. In *Proceedings of the Linux symposium*, volume 1, pages 225-230. Dttawa, Dntorio, Canada, 2007.

[31] Alexey Kopytov. Sysbench: a system performance benchmark. http://sysbench.sourceforge.net/, 2004.

[32] Michael Larabel. Linux patches improve vm guest performance when the host encounters memory pressure. https://www.phoronix.com/news/Better-Guest-For-Host-RAM-Press.

[33] Libvirt. Libvirt virtualization api, 2006. https://libvirt.org/ - Last accessed on 09/06/2021.

[34] Yuhui Lin, Adam Barker, and John Thomson. Modelling vm latent characteristics and predicting application performance using semi-supervised non-negative matrix factorization. In *2020 IEEE 13th International Conference on Cloud Computing (CLOUD)*, pages 470-474, 2020.

[35] Haikun Liu, Cheng-Zhong Xu, Hai Jin, Jiayu Gong, and Xiaofei Liao. Performance and energy modeling for live migration of virtual machines. In *Proceedings of the 20th International Symposium on High Performance Distributed Computing, HPDC '11*, page 171-182, New York, NY, USA, 2011. Association for Computing Machinery.

[36] Steffen Maass, Mohan Kumar Kumar, Taesoo Kim, Tushar Krishna, and Abhishek Bhattacharjee. <span class="smallcaps">smallcaps smallercapital">eco</span><span class="smallcaps">tlb</span>: Eventually consistent tlbs. *ACM Trans. Archit. Code Optim.*, 17(4), sep 2020.

[37] John D McCalpin. Stream benchmark. Link: www.cs.virginia.edu/stream/ref.html# what, 22:7, 1995.

[38] Anna Melekhova. Machine learning in virtualization: Estimate a virtual machine's working set size. In *2013 IEEE Sixth International Conference on Cloud Computing*, pages 863-870, 2013.

[39] Michael P. Mesnier. //trace: Parallel trace replay with approximate causal events. In *5th USENIX Conference on File and Storage Technologies (FAST 07)*, San Jose, CA, February 2007. USENIX Association.

[40] A B M Moniruzzaman, Kawser Wazed Nafi, and Syed Akther Hossain. Virtual memory streaming technique for virtual machines (vms) for rapid scaling and high performance in cloud environment. In *2014 International Conference on Informatics, Electronics Vision (ICIEV)*, pages 1-6, 2014.

[41] Justin Moore, Jeff Chase, Keith Farkas, and Parthasarathy Ranganathan. Data center workload monitoring, analysis, and emulation. *CAECW '05: Proceedings of the Eighth Workshop on Computer Architecture Evaluation Using Commercial Workloads*, 01 2005.

[42] Djob Mvondo, Alain Tchana, Renaud Lachaize, Daniel Hagimont, and Noël De Palma. Fine-grained fault tolerance for resilient pvm-based virtual machine monitors. In *2020 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)*, pages 197-208, 2020.

[43] Djob Mvondo, Boris Teabe, Alain Tchana, Daniel Hagimont, and Noel De Palma. Closer: A new design principle for the privileged virtual machine os. In *2019 IEEE 27th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS)*, pages 49-60, 2019.

[44] Jun Nakajima. Intel updates. *Xen Summit November*, 2007.

[45] Neel Natu and Peter Grehan. Nested paging in bhyve. *The FreeBSD Project*, http://people.freebsd.org/neel/bhyve/bhyvenestedpatching.pdf, 2014.

[46] Vlad Nitu, Aram Kocharyan, Hannas Yaya, Alain Tchana, Daniel Hagimont, and Hrachya Astsatryan. Working set size estimation techniques in virtualized environments: One size does not fit all. *Proc. ACM Meas. Anal. Comput. Syst.*, 2(1), April 2018.

[47] OpenStack. Openstack compute (nova), 2020. https://docs.openstack.org/nova/latest/ - Last accessed on 10/08/2021.

[48] Oracle. End-to-end use case: Patching your data center, 2015. https://docs.oracle.com/cd/E24628_01/em.121/e27046/appdx_patchung_use_case.htm#EMLCM94012.

[49] Michael Pearce, Sherali Zeadally, and Ray Hunt. Virtualization: Issues, security threats, and solutions. *ACM Comput. Surv.*, 45(2), March 2013.

[50] SciPy. Scipy.org, 2001. https://www.scipy.org/ - Last accessed on 10/07/2021.

[51] Yasser Shalabi, Mengjia Yan, Nima Honarmand, Ruby B. Lee, and Josep Torrelles. Record-replay architecture as a general security framework. In *2018 IEEE International Symposium on High Performance Computer Architecture (HPCA)*, pages 180-193, 2018.

[52] Prateek Sharma, Lucas Chaufournier, Prashant Shenoy, and Y. C. Tay. Containers and virtual machines at scale: A comparative study. In *Proceedings of the 17th International Middleware Conference*, Middleware '16, New York, NY, USA, 2016. Association for Computing Machinery.

[53] Pezhman Sheinidashtegol and Michael Galloway. Performance impact of ddos attacks on three virtual machine hypervisors. In *2017 IEEE International Conference on Cloud Engineering (IC2E)*, pages 204-214, 2017.

[54] Ronak Singhal. Inside intel® core microarchitecture (nehalem). In *2008 IEEE Hot Chips 20 Symposium (HCS)*, pages 1-25, 2008.

[55] Martin Slawski and Matthias Hein. Sparse recovery by thresholded non-negative least squares. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K. Q. Weinberger, editors, *Advances in Neural Information Processing Systems*, volume 24.
Curran Associates, Inc., 2011.

[56] Xudong Sun, Runxiang Cheng, Jianyan Chen, Elaine Ang, Owolabi Legunsen, and Tianyin Xu. Testing configuration changes in context to prevent production failures. In *14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20)*, pages 735–751. USENIX Association, November 2020.

[57] Yifeng Sun, Yingwei Luo, Xiaolin Wang, Zhenlin Wang, Binbin Zhang, Haogang Chen, and Xiaoming Li. Fast live cloning of virtual machine based on xen. In *2009 11th IEEE International Conference on High Performance Computing and Communications*, pages 392–399, 2009.

[58] Hassan Takabi, James B.D. Joshi, and Gail-Joon Ahn. Security and privacy challenges in cloud computing environments. *IEEE Security Privacy*, 8(6):24–31, 2010.

[59] Blesson Varghese, Lawan Thamsuhang Subba, Long Thai, and Adam Barker. Container-based cloud virtual machine benchmarking. In *2016 IEEE International Conference on Cloud Engineering (IC2E)*, pages 192–201, 2016.

[60] VMWare. Goodbye, replay debugging. http://www.replaydebugging.com/2011/09/goodbye-replay-debugging.html.

[61] Carl A. Waldspurger. Memory resource management in vmware esx server. *SIGOPS Oper. Syst. Rev.*, 36(SI):181–194, dec 2003.

[62] Wei Wang, Zhiyu Hao, and Lei Cui. Clusterr: a record and replay framework for virtual machine cluster. In *Proceedings of the 18th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments*, VEE 2022, page 31–44, New York, NY, USA, 2022. Association for Computing Machinery.

[63] Xiaolin Wang, Jiarui Zang, Zhenlin Wang, Yingwei Luo, and Xiaoming Li. Selective hardware/software memory virtualization. In *Proceedings of the 7th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments*, VEE ’11, page 217–226, New York, NY, USA, 2011. Association for Computing Machinery.

[64] Song Wei, Kun Zhang, and Bibo Tu. Performance impact of host kernel page table isolation on virtualized servers. In *2021 IEEE Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC)*, pages 912–919, 2021.

[65] Philip M. Wells, Koushik Chakraborty, and Gurindar S. Sohi. Hardware support for spin management in overcommitted virtual machines. In *Proceedings of the 15th International Conference on Parallel Architectures and Compilation Techniques, PACT ’06*, page 124–133, New York, NY, USA, 2006. Association for Computing Machinery.

[66] Sheng Yang. Extending kvm with new intel virtualization technology. In *KVM forum*, 2008.

[67] Xiantao Zhang, Xiao Zheng, Zhi Wang, Qi Li, Junkang Fu, Yang Zhang, and Yibin Shen. Fast and scalable vmm live upgrade in large cloud infrastructure. In *Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’19*, page 93–105, New York, NY, USA, 2019. Association for Computing Machinery.

[68] Yu Zhang, Renéa Oertel, and Wolfgang Rehm. Paging method switching for qemu-kvm guest machine. In *Proceedings of the 2014 International Conference on Big Data Science and Computing*, BigDataScience ’14, New York, NY, USA, 2014. Association for Computing Machinery.

[69] Yu Zhao. [patch mm-unstable v2 00/10] mm/kvm: locklessly clear the accessed bit. https://lore.kernel.org/linux-mm/20230526234435.662652-1-yuzhao@google.com/T/.

[70] Wei Zheng, Ricardo Bianchini, G. John Janakiraman, Jose Renato Santos, and Yoshio Turner. Justrunit: Experiment-based management of virtualized data centers. In *Proceedings of the 2009 Conference on USENIX Annual Technical Conference, USENIX’09*, page 18, USA, 2009. USENIX Association.

[71] Minqi Zhou, Rong Zhang, Wei Xie, Weining Qian, and Aoying Zhou. Security and privacy in cloud computing: A survey. In *2010 Sixth International Conference on Semantics, Knowledge and Grids*, pages 105–112, 2010.