# Revisiting virtual machine consolidation to save resources and energy in heterogeneous production cloud infrastructures

Simon Lambert, Vladimir Ostapenco, Laurent Lefèvre, Eddy Caron,
Anne-Cécile Orgerie, Benjamin Fichel, Rémi Grivel

<figure><img src="image_1.png" /></figure>

## To cite this version:

Simon Lambert, Vladimir Ostapenco, Laurent Lefèvre, Eddy Caron, Anne-Cécile Orgerie, et al. Revisiting virtual machine consolidation to save resources and energy in heterogeneous production cloud infrastructures. International Journal of High Performance Computing Applications, 2025, 10.1177/10943420251399694. hal-05457159

HAL Id: hal-05457159

https://hal.science/hal-05457159v1

Submitted on 13 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Revisiting virtual machine consolidation to save resources and energy in heterogeneous production cloud infrastructures

Simon Lambert<fnref n="1" />,<fnref n="3" />, Vladimir Ostapenco<fnref n="1" />, Laurent Lefèvre <fnref n="1" />, Eddy Caron <fnref n="1" />, Anne-Cécile Orgerie <fnref n="2" />, Benjamin Fichel <fnref n="4" />, Rémi Grivel <fnref n="3" />

## Abstract

Due to some overprovisioning policies and variable usage, data centers in production can face low average resource utilization. This can result in a waste of underused servers and energy. In this context, virtual machine (VM) consolidation combined with shutdown policies can be a pertinent approach for improving resource utilization and reducing energy consumption of the entire cloud infrastructure. However, VM consolidation requires expensive migration techniques, which can potentially affect performance. Consolidation of workload has been proposed and studied as a core capability since the invention of the Cloud. But after two decades of deployment of Cloud infrastructures, VM consolidation is still rarely used in production for small and large-scale environments. In this article, we explore and revisit the potential of savings that can be achieved through a versatile and efficient Virtual Machine consolidation in small and large-scale production infrastructures through usage analysis of two Cloud providers infrastructures. We show that potential benefits in terms of saved cloud resources and energy usage reduction can occur for systems in production.

## Keywords

Energy Efficiency, Cloud, Workload Consolidation, Virtual Machines, Migration

## Introduction

Data centers can have low average resource utilization, ranging from 10% to 50% Shehabi et al. (2018). The main reasons for this underutilization are workload variability and strict SLAs, which involve provisioning infrastructure to handle theoretical peak loads, resulting in resource waste and a large amount of unused resources.

In this context, virtual machine (VM) consolidation combined with shutdown policies is a pertinent approach Lefèvre and Orgerie (2010) for improving resource utilization and reducing energy consumption of the entire cloud infrastructure Srikantaiah et al. (2008). The goal of this VM consolidation is to reduce the number of physical compute nodes required to run VMs. The unused compute nodes are then turned off using shutdown policies to save energy.

By leveraging the virtualization capabilities in cloud environment, VM consolidation can be performed on par with resource oversubscription (overcommitment) techniques to further increase the resource utilization and save more energy. Oversubscription can even help reduce the carbon footprint of cloud infrastructures. As fewer physical resources are required to handle customer workloads, freed up resources can be used to handle future workloads, reducing the need to deploy new equipment if no rebound effects occur.

However, VM consolidation requires expensive migration techniques which can potentially affect performance. In addition, excessive consolidation can lead to significant

performance degradations, especially in oversubscribed environments.

Consolidation of work has been proposed and studied as a core capability since the inventing of the Cloud Helali and Omri (2021). But after two decades of deployment of Cloud infrastructures, VM consolidation is still rarely used in production for small and large scale environments Zou et al. (2024). In this article, we explore and revisit the potential of savings that can be achieved through a versatile and efficient Virtual Machine consolidation in small and large-scale production infrastructures through usage analysis of two Cloud providers infrastructures. Exploring consolidation on these clusters allows us to identify similarities in the implementation of this type of methodology in heterogeneous infrastructures. The methodology is also used on recent traces of production clusters to provide evidence that consolidation can be applied on production virtualization clusters.

<footnote n="1">¹Ensl, UCBL, CNRS, Inria, LIP, University of Lyon, Lyon, France</footnote>

<footnote n="2">²Inria, CNRS, IRISA, University of Rennes, Rennes, France</footnote>

<footnote n="3">³Ciril GROUP, FRANCE</footnote>

<footnote n="4">⁴OVHCloud, FRANCE</footnote>
It covers a wide spectrum of Clouds configurations by considering several heterogeneous operational infrastructures deployed by Cloud providers. We analyze the virtual machines usage from two different Cloud providers (CirilGroup and OVHcloud) of varying sizes. We show that potential benefits in terms of saved cloud resources and energy usage reduction can occur for systems in production.

This article is organized as follows. Section Related works covers the existing works on workload consolidation and shutdown policies. After briefly presenting the considered infrastructures in section Studied infrastructures, the section A versatile methodology presents the versatile methodology proposed in this article and adapted both for small-scale and large scale production infrastructures. We analyze main simulation and experimental results in section Result analysis while considered limits are presented in section Discussion. The section Conclusion and future works concludes this article.

## Related works

### Workload consolidation

The primary goal of the workload consolidation is to put customer workloads onto a minimal number of physical compute nodes in order to improve the global resource utilization and to reduce the energy consumption.

The unused compute nodes are then usually turned off or placed in an idle state to save energy. Even though workload consolidation is applicable to any type of cloud workloads, we narrow down its scope to VM consolidation.

The deployment of sophisticated virtualization solutions made workload consolidation more straightforward. Current virtualization solutions support live VM migration, which allow for nearly seamless movement of VMs across compute nodes with small downtime Voorsluys et al. (2009). This provides a wide range of opportunities for dynamic VM consolidation to reduce energy consumption while meeting performance requirements.

However, VM consolidation can also be static, which means that consolidation is carried out using allocation policies without VM migrations. This work focuses on dynamic VM consolidation, which we refer to as VM consolidation for simplicity.

As previously stated, workload consolidation is an pertinent approach for combining with other technics such as shutdown policies or sleep states to maximize its impact Lefèvre and Orgerie (2010).

However, the dynamic VM consolidation can have significant time and energy costs, impacts performance by adding an overhead that must be considered, and can induce downtimes Liu et al. (2011). The performance degradation and downtime during live VM migration depend on application behavior (memory pages updates during migration) Voorsluys et al. (2009). Downtimes can range from tens of milliseconds for low resource utilization workloads to several seconds for highly intensive HPC workloads Voorsluys et al. (2009). For applications with dynamic workloads, the average performance degradation during live VM migration, including downtime, is estimated to be around 10% of CPU utilization Beloglazov (2013).

Furthermore, the cost of migration depends on the size of the VM and the available network bandwidth. VMs can have varying memory footprints ranging from several hundreds of megabytes to hundreds of gigabytes, which are transferred during the migration process to the destination compute node. Migration costs also depend on the workload intensity, because for live migration the memory is fully copied and synchronized. Therefore, intensive memory usage can extend migration time significantly. Additionally, if the source and destination compute nodes do not have shared storage, the VM storage must also be migrated.

The cloud provider has to find a trade-off between using workload consolidation to increase energy efficiency and meeting QoS requirements. Since cloud resources are usually underutilized, resulting in significant resource waste, workload consolidation can be performed on par with resource oversubscription techniques to increase resource utilization and save even more energy Dabbagh et al. (2015). However, excessive consolidation can result in performance degradations and increased energy consumption due to longer execution times, which can lead to significant QoS degradations Kim et al. (2013). Oversubscription techniques should therefore be used with caution.

Workload consolidation in the context of a cloud data center is performed at the management system level. Workload consolidation algorithms in cloud environment can have a wide range of objectives such as maximizing resource utilization, minimizing energy consumption, or maximizing cloud provider profits Khan et al. (2018). They can take into account the current placement of VMs for decision-making, or simply choose a destination compute node for each VM, providing a solution for initial VM placement. They can also be centralized or distributed, having the entire view of cloud infrastructure and taking global decisions or just exchanging information with its neighbors and taking local decisions. They can be predictive or not, basing their decisions on the estimation of future resource usage or simply on current resource demand. Khan et al. (2018) provide an extensive overview of all these types of workload consolidation algorithms.

Workload consolidation algorithms are usually composed of three components Khan et al. (2018):

* Source Host Selection involves selecting a set of hosts from which the VMs should be migrated out. This could be, for instance, overloaded or underloaded hosts. VMs should be migrated from overloaded hosts to avoid QoS violations. VMs can be migrated from underloaded hosts to power them off or put them into low-power mode to save energy.

* VM Selection entails selecting a set of VMs to migrate from the source hosts. This could be, for instance, a set of resource-intensive VMs to avoid overloading the host, or all VMs to free up an underloaded host.

* Destination Host Selection or VM Placement involves choosing destination hosts for each VM selected for migration. It can be determined based on the availability and utilization of destination host resources, as well as on a wide set of network, application, or infrastructure constraints.
One of the main components of workload consolidation algorithms is destination host selection, also known as VM placement, which plays a major role in the consolidation outcome. The main goal of VM placement is to select the host machines on which to place the VMs while minimizing the number of active hosts machines without violating any constraints. The optimal VM placement is an NP-hard problem, so various heuristic and meta-heuristic algorithms have been proposed Khan et al. (2018). Heuristic and meta-heuristic algorithms allow solving VM placement problem in a reasonable time, usually yielding non-optimal solutions. In reality, workload consolidation is much more complex than a classical bin-packing problem, as it can have numerous objectives and constraints Mann (2015). To be efficient, VM placement must consider a variety of supplementary constraints such as host heterogeneity leading to non-homogenous performance, application constraints in the case of communicating tasks to avoid latency issues, network constraints such as IP address availability, operational constraints such as client requests and workload compatibility, and cooling constraints to avoid the formation of hotspots Orgerie et al. (2014); Buyya et al. (2024). Unfortunately, these constraints are difficult to incorporate into heuristic and meta-heuristic algorithms. Furthermore, VM consolidation usually involves a multi-objective optimization in which, for instance, the number of active hosts is minimized while also minimizing the cost of VMs migrations. These types of problems are even more challenging to model and solve.

## Shutdown policies

Since the static part of the energy consumed (consumption even when no workload is executed) accounts for a substantial portion of the total energy consumed by data center equipment, shutting down unused equipment can result in significant energy savings.

Shutdown policies are already available in multiple management systems, such as slurm¹ and VMware VSphere², allowing powering down unused resources. In a cloud environment, due to elasticity requirements, providers tend to overprovision infrastructure to accommodate the potential arrival of workloads that may be unpredictable. The shutdown policies can be applied to the overprovisioned infrastructure with almost no risk. However, cloud providers need to keep a portion of these unused resources running to meet immediate elasticity requirements.

Shutdown policies can be combined with workload consolidation and scheduling algorithms to be more efficient. In this configuration, the workload is consolidated or scheduled across fewer compute nodes, and the freed nodes are then powered off. Using shutdown policies combined with workload consolidation and scheduling usually results in greater energy savings. We explore workload consolidation and scheduling policies in more depth later in this section.

However, shutting down and powering on compute nodes incurs significant time, energy, and power costs Benoit et al. (2017). For instance, a Dell PowerEdge R640 compute node can take more than 3 minutes and consume over 20000 Joules to power on. Therefore, shutdown policies are suitable for use during periods of inactivity

to be beneficial for energy efficiency. They can also be used in conjunction with workload prediction techniques to minimize unnecessary power on and power off sequences and lower the energy consumption. Furthermore, shutdown policies affect not only power consumption but also temperature and thus cooling systems Raïs et al. (2018). The systematic intense usage of shutdown policies is not always the best approach, because compute node boots can induce significant consumption spikes and these spikes must be handled by the global power supply and cooling system. This can result in violations of data center power limits and exceeding cooling capacity, potentially causing power outages and even damaging IT equipment.

The shutdown policies can result in significant energy savings by acting on static power consumption. It can be used in conjunction with workload consolidation and scheduling to increase its effectiveness, and is suitable for dealing with long idle periods. However, due to its significant costs in terms of time and energy, as well as impacts on power supply and cooling systems, it should be used with caution.

## Studied infrastructures

There are nowadays many data centers types, that can be classified based on their size, location, quality of service they provide or their peak power consumption for example. A commonly used classification combines these different characteristics, without necessarily setting fixed values for them. It includes edge data centers, that are typically small-sized data centers that provide computation and storage close to the user terminals and devices, collocation data centers, that are small to medium-sized data centers where companies rent space in a facility owned by a third party to host their servers and equipments, or hyperscale data centers, which are large data centers that contain thousands to millions of servers, used in particular for their efficiency and the flexibility they provide to companies. A new type of data center is also emerging since the democratization of generative AI services, with data centers dedicated to hosting AI-specific hardware having a higher energy density than conventional computing servers. Globally, the sector is evolving, and the question of its electricity consumption, as well as its other environmental impacts, is always a pressing issue.

This study combines consolidation works performed in 2 heterogeneous cloud provider infrastructures with their own specificities and challenges.

**Large-scale infrastructure presentation** The large-scale infrastructure is the infrastructure of the leading European cloud provider OVHcloud, having nearly half a million physical servers spread across several dozen data centers, hosting more than 18 million applications and serving more than one and a half million customers. OVHcloud provides a wide range of cloud services in the public cloud offering, including SaaS (Software as a Service), PaaS (Platform as a Service) and IaaS (Infrastructure as a Service), as well as bare metal instances with the flexibility of cloud to their customers. The OVHcloud public cloud infrastructure is divided into several clusters, each located in a single data center.
In this work, rather than studying the entire OVHcloud public cloud infrastructure, we focus on a single cluster (named *Cluster L1*) containing 985 physical hosts running 7376 VMs located in the same data center, which is one of the largest clusters. In Ostapenco (2024), we also study smaller OVHcloud clusters.

The studied cluster L1 provides guaranteed resources to its customers, thus no oversubscription is applied, which means that each resource allocated to VMs corresponds to a resource on a physical host.

In the OVHcloud public cloud infrastructure, the storage provided to VMs is integrated into the hypervisor hosts rather than being centralized in storage arrays as is often the case in other data centers. Thus, both storage and memory should be transferred to migrate a VM from one host to another, which significantly increases the cost of VM migrations. Therefore, VM migrations must be properly scheduled to avoid network congestions and performance degradations.

**Small-scale infrastructure presentation** The considered small-scale infrastructure comes from a software development company and a cloud service provider located in France called Ciril GROUP. It provides software packages to private and public clients such as local authorities, as well as housing, IaaS, PaaS and SaaS offers hosted in its own data center. The presented work focuses on the IaaS and PaaS scope of the data center, which hosts heterogeneous software and applications. This infrastructure is relatively small sized compared to the large-scale infrastructure studied in this article, and composed of more than 50 computing servers, multiple storage arrays for storage or archiving, as well as multiple networking equipments to interconnect all these devices. It hosts all or part of infrastructures of hundreds of clients on thousands of VMs separated on multiple virtualization clusters.

In contrast to the large-scale one, the small-scale infrastructure is located within one single data center. It is widely mutualized, meaning that some equipments, especially for storage and network, will be physically used by multiple virtualization clusters. This enables for example to reduce costs for the cloud operator without compromising Quality of Service (QoS). These specificities have an impact on consolidation performance, for example for VM migration. As the VM storage is centralized on storage arrays, there is no need to migrate the VM storage from one Physical Machine (PM) to the other, reducing drastically the VM migration time. Storage arrays are connected to computing servers through a high speed network (Storage Area Network) based on iSCSI (Internet Small Computer System Interface) which encapsulates SCSI commands in Ethernet frames and uses IP Ethernet for transport. Thanks to this kind of architecture, we are able to reduce VM migration times to tenths of a second or seconds, making this operation transparent from the user perspective.

The smaller size of the infrastructure and its many virtualization clusters has other benefits. For instance, it enables a more detailed analysis and knowledge of events and infrastructure elements, both by operational teams and monitoring systems. With a smaller number of equipments and due to the performance of storage arrays, VM and PM monitoring can be achieved with a shorter sampling time and

a longer retention time. This will eventually better capture usage variation of the elements of the cluster, allowing us to tune our consolidation periods and consequently limit potential performance degradation on the virtual machines.

In this work, we focus on 2 clusters of the Ciril GROUP data center: *Cluster S1* is composed of 6 PMs, 400 VMs and used for internal company usage. This cluster was also the object of a study where consolidation was combined with user involvement in Lambert et al. (2024). *Cluster S2* is a cluster composed of 16 PMs and more than 1000 active VMs hosting Ciril GROUP customer's production applications.

## A versatile methodology

This section covers the methodology proposed to study and apply the VM consolidation on heterogeneous production cloud infrastructures.

In this work, we perform an offline consolidation based on a snapshot of infrastructure metrics. The offline VM consolidation is accomplished by first analyzing the initial VM placement and infrastructure metrics, and then by exploring a set of VM migration strategies attempts to find a better VM placement to reduce the amount of hosts required to run VMs. The placement is then applied through live VM migrations.

## Detecting consolidation potential

The consolidation potential can be revealed by studying the initial VM positioning alongside CPU, RAM and Disk allocation metrics. The allocation metrics represent the percentage of available hardware resources on hosts that are allocated to VMs.

Initial VM positioning is usually handled by the cloud management system scheduler, and to be efficient from a resource usage perspective, it should fill as much of the available space on used hosts (with at least one VM running) as possible, before scheduling a VM on an unused host (with no VMs running). Thus, if the average allocation rates of used hosts are low, it indicates that numerous hosts have a large amount of empty space that can be used to reduce the number of used hosts by filling these empty spaces.

Metrics, such as average CPU usage of hosts, can be used to reveal the potential for more aggressive VM consolidation with oversubscription that has no or minimal impact on performance.

## Low Resource Usage Virtual Machines detection

In order to detect LRU VMs (Low Resource Usage Virtual Machines), we propose to use thresholds on a set of VM usage metrics. The set of VM usage metrics we choose to detect LRU VMs is inspired by the idle VMs detection performed by existing solutions, such as Veeam³, Google Cloud⁴ and Eg Reporter⁵. Each solution uses its own set of VM metrics to detect idle VMs. They all use CPU and network usage metrics, with some also using memory and disk usage metrics.

We develop and deploy a simple generic detection algorithm (Algorithm 1). The set of VM usage metrics we
select for large- and small-scale infrastructures can vary due to differences in VM storage implementation.

**Algorithm 1** LRU-VM detection algorithm - for small and large scale infrastructures

```pascal
function IS_USED vm, t, cpuRate, netRate, diskRate
  if cpu_usage vm, t ≤ cpuRate and
    net_usage vm, t ≤ netRate [ and disk_usage vm, t] ≤
    diskRate ] then
    return 0
  else
    return 1
end if
end function
```

Algorithm 1 is executed for each selected threshold, each timestamp, and each VM to calculate the fraction of time during which VMs are considered used. Then, using the LRU ratios, the LRU VMs are identified.

**LRU VM detection on large-scale infrastructure** For large-scale infrastructure, we use CPU, disk, and network usage metrics to detect LRU VMs because these metrics cover various VM usage patterns, and are available in widely used cloud virtualization solutions, such as *libvirt*.

In order to select thresholds for these usage metrics, their values are first calibrated with the default values used by previously introduced solutions to detect idle VMs. Then they are adjusted to reflect various usage patterns, including not only no usage but also low resource usage patterns. A threshold sensibility analysis using production traces allows for the final threshold adjustment.

Therefore, for the large-scale infrastructure, we study the following thresholds for detecting LRU VMs:

* CPU usage rate metric thresholds of 1%, 2%, 5% and 10%;

* Network exchange metric thresholds of 200kB/s, 2MB/s and no threshold for both send and receive metrics;

* Disk usage metric thresholds of 200kB/s, 2MB/s and no threshold for both disk read and write metrics;

* LRU ratios of 99% and 100% indicating the percentage of time during the reporting period when the maximum resource usage of the VM is less than the chosen thresholds.

We apply the algorithm 1 to detect if a Virtual Machine is considered used in timestamp *t*.

**LRU VM detection on small-scale infrastructure** On the small-scale infrastructure, *iSCSI* is used to ensure the storage of the VMs. The read and write operations occurs through a specific network interface that uses Ethernet networks. Disk usage of virtual machines can therefore be monitored with this interface, in the same way as network exchanges on other interfaces. As for the large scale infrastructure, VM usage evaluation is conducted CPU, network, and disk usage. We use Algorithm 1 to detect LRU-VMs on the small-scale infrastructure using a combined threshold for network and disk usage referred as *Input/Output (I/O) threshold*. The algorithm considers a VM as *idle* if it uses less than

a specific *cpuRate* and if the average bandwidth on its network interfaces is below a given *netRate*.

The considered thresholds for the small-scale infrastructure are the following:

* CPU usage rate metric thresholds of 1%, 2%, 5% and 10%;

* I/O thresholds of 1kb/s, 200kb/s, 2Mb/s;

* LRU ratios of 99% and 100% indicating the percentage of time during the reporting period when the maximum resource usage of the VM is less than the chosen thresholds.

Considering the risks of using oversubscription in terms of performance degradation, default CPU and I/O usage rates used in Algorithm 1 are set to 1 in the small-scale infrastructure.

## Collecting metrics and traces

Analysis of the consolidation potential as well as the quantification of LRU VMs require to gather metrics and usage traces of multiple elements in the clusters.

**Large-scale infrastructure metrics and traces** agraphIn-frastructure description The infrastructure description, which includes hosts and VMs with their characteristics and initial placement, is first obtained from the production OVH-cloud Prometheus database and then saved in a dedicated *TimescaleDB* database.

The infrastructure description is provided in a custom set of Prometheus metrics. We collect a comprehensive set of hosts and VMs characteristics relevant to our study, such as available and used CPUs, RAM, and storage.

**Usage metrics** CPU, disk, and network usage metrics used to detect LRU VMs are also retrieved from the production OVHcloud Prometheus database and stored in the *TimescaleDB* database.

We base our study on a single day of traces, having 1 point every 5 minutes for each metric.

We retrieve the following VM usage metrics:

* CPU usage rate - 1 average value per CPU core;

* Network exchange in bytes per second for both transmission and reception metrics - 1 average value per network interface and metric;

* Disk usage in bytes per second for both read and write metrics - 1 average value per disk and metric.

To assess the average host CPU usage, we also collect CPU usage rate metrics for each host.

VM CPU usage metrics are also used to simulate CPU usage to estimate energy consumption of infrastructure.

We recognize that using a single day of metrics and an average value every 5 minutes for each metric is insufficient to accurately detect LRU VMs. In reality, the metrics may be much more variable, which can be hidden by averaging. Additionally, analyzing a single day of metrics may lead to false detections, as the VM may have low resource utilization only during the selected day with high activity on other days. However, even using a single day of averaged metrics can be useful to provide an idea of the state of the production infrastructure and the amount of LRU VMs.
**Small-scale infrastructure metrics and traces** In the small-scale infrastructure, usage metrics are collected using a vSphere Telegraf agent configured to gather metrics every 30 seconds. Metrics from the different clusters are then stored into a VictoriaMetrics database with a retention period of one year.

**Usage metrics** CPU and I/O (network and storage) metrics are used to detect LRU VMs.

VM usage evaluation is typically conducted over periods of one week, using a resampled usage matrix containing mean values over the consolidation period.

We retrieve the following VM usage metrics:

* Network and storage exchange : average bandwidth observed over the collection period - 1 average value per VM

* CPU usage rate: 1 average value per VM

The amount and share of LRU-VMs can be computed using the native metrics collection rate, i.e 30 seconds, which will provide a binary matrix containing the state of each VM at each iteration. The resampled matrix that matches the consolidation period will therefore provide the share of time in which the VM will be considered as Low Resource Usage during the consolidation period.

## Consolidation strategies

Consolidation strategies define how the VM consolidation is performed.

**Consolidation strategies for large-scale infrastructure** For large-scale consolidation, we explore a series of VM consolidation strategies, ranging from low to high risk in terms of potential negative performance impacts. Therefore, we explore the following VM consolidation strategies:

* LRU VMs migration: only LRU VMs can be migrated, and the oversubscription is not used;

* LRU VMs migration with oversubscription: only LRU VMs can be migrated, and the oversubscription is used for LRU VMs only;

* General VM migration: both LRU and non-LRU VMs can be migrated, and the oversubscription is not used;

* General VM migration with oversubscription: both LRU and non-LRU VMs can be migrated, and the oversubscription is used for LRU VMs only.

**Consolidation strategies for small-scale infrastructure** For small-scale consolidation, only the equivalent of General VM migration with oversubscription is studied. Considering the smaller amount of physical machines in the infrastructure, as well as the fact that CPU oversubscription is already used in the small-scale infrastructure, both LRU and non-LRU VMs will be migrated to obtain better results in terms of energy savings.

The architecture of the small-scale infrastructure also enables a higher number of VM migration without expecting performance degradation, as the storage is centralized and the amount of memory to transfer during VM migration is drastically reduced.

## VM placement

Once VMs to migrate have been identified, the destination hosts for each VM selected for migration must be chosen.

**VM placement on large-scale infrastructure** Due to the large number of resources, we explore some solver-based approaches. It allows solving the VM placement problem while taking into account conditions and constraints necessary for the implementation of the previously introduced consolidation strategies. In this approach, the VM placement problem with all necessary conditions and constraints is first formulated as a bin-packing problem using constraint or linear programming, and then solved using an existing solver such as CP-SAT⁶ or Gurobi⁷. However, because the bin-packing problem is NP-hard, constraint and linear programming solvers looking for optimal solutions frequently fail to deliver satisfactory results in a reasonable amount of time. Another option is to use a solver using local search techniques, such as Hexaly⁸, which has a rich modeling interface and is able to provide close to optimal solutions relatively quickly. Therefore, for large-scale infrastructure, we chose to use the Hexaly solver for VM placement.

We consider the CPU, RAM, and disk characteristics of both hosts and VMs, as well as other constraints required to perform VM placement following previously discussed consolidation strategies.

Each consolidation strategy execution requires two solver resolutions. The first resolution aims to minimize the number of hosts accommodating VMs, while the second aims to minimize migration costs (minimize the amount of data to be transferred).

For this purpose, we create a Hexaly model, which includes the following constraints and objectives.

### Constraints

* Each VM must be on a single host and on a single host only.

* The sum of resources required by all VMs on the host must not exceed its capacity for CPU, RAM and Disk resources.

* For LRU VMs migration strategies only, force non-LRU VMs positions.

### Objectives

* Minimize the number of hosts used, where a host is considered used if it contains at least one VM.

* Minimize migration costs, where migration costs are calculated for each VM, taking into account its initial placement.

- If a VM is migrated, its migration cost is equal to the sum of the RAM and Disk sizes.

- If a VM is not migrated, its migration cost is equal to 0.

**VM placement on small-scale infrastructure** For the small-scale infrastructure, consolidation is conducted using a group based approach. VMs can be separated into two or more consolidation groups depending on their computed usage over the consolidation period. Each consolidation group will be assigned one oversubscription ratio in order to maximize
the potential of energy savings. Table 1 provides an example of a 3 groups consolidation where each group gets a specific subscription ratio depending on the VMs usage.

<table><caption>**Table 1.** Example of group consolidation - Small scale</caption><thead><tr><th>Group number</th><th>G1</th><th>G2</th><th>G3</th></tr></thead><tbody><tr><td>Usage rate (r)</td><td>r&lt;10%</td><td>10%&lt;r&lt;20%</td><td>r&gt;20%</td></tr><tr><td>Subscription ratio</td><td>3</td><td>2</td><td>1</td></tr></tbody></table>

In simulation, the consolidation solver first computes the number of PMs to use for each consolidation group depending on the total amount of vCPUs and RAM, weighted by the oversubscription ratio. Once calculated, a First-Fit Decreasing (FFD) heuristic is used to place VMs on PMs for each consolidation group. This approach was selected in simulation to avoid additional complexity in the solver. Indeed, the experimental VM placement will be handled by the hypervisor orchestrator, using features such as *VM/Host affinity rules*⁹.

A Bin-Packing solver can however be implemented to optimize VM placement globally or in each consolidation group using these conditions:

### Constraints

* Each VM must be on a single host and on a single host only.

* The sum of weighted resources required by all VMs on the host must not exceed its capacity for CPU, RAM.

* VMs of different consolidation group must not be on a similar host.

### Objectives

* Minimize the number of hosts used, where a host is considered used if it contains at least one VM.

## Selecting allocation ratios

As this methodology combines consolidation with oversubscription, the selection of pertinent allocation ratios is crucial to enhance the energy savings while limiting the potential performance degradations.

In the article, allocation and subscription ratios are used interchangeably to define the ratio between the amount of resources allocated to VMs and the amount of physical resources available. Oversubscription ratio is commonly used when this ratio is greater than 1.

**Large-scale infrastructure allocation ratios** In a production environment, the allocation ratios are usually statically defined on the cluster level or host machine level.

In OpenStack by default, allocation ratios appear to be used solely by the scheduler to assign VMs to hosts, without pinning (binding) virtual CPU cores used by the VMs to specific physical CPU cores. VMs are considered as processes which execute on the host and are therefore scheduled by the hypervisor, floating from one physical CPU core to another¹⁰. In the context where ratios are only used by the scheduler to place VMs, we can assume that they can be defined on a per-VM basis. Therefore, we consider that the allocation ratios can be configured independently for each VM. Thus, two VMs on the same host can have different allocation ratios.

Because oversubscription carries a non-negligible risk of performance degradation, we limit its use to LRU VMs.

To select allocation ratios, we first calibrated them with the values found in the default OpenStack configuration, which has default allocation ratios of 16 for CPU and 1.5 for RAM¹¹. Then, the CPU allocation ratios were adjusted based on the previously selected CPU usage metric thresholds. We assume that LRU VMs with CPU usage below 10% can be oversubscribed with a maximum CPU allocation ratio of 10 with low risk of performance degradation. Since the cumulative CPU usage of 10 VMs with the CPU usage always below 10% should never exceed the CPU capacity and cause resource contentions, they can be co-hosted with almost no impact on performance. However, to further reduce the risk of performance degradation, the maximum CPU allocation ratio investigated is 8.

For the RAM allocation ratios, we study the allocation ratios of 1.5, as found in the default OpenStack configuration, as well as of 2 to broaden the scope of our study.

With this in mind, we explore the combinations of the following allocation ratios:

* CPU: 1, 2, 4, 8;

* RAM: 1, 1.5, 2.

**Small-scale infrastructure allocation ratios** On the small-scale infrastructure, the oversubscription ratio to identify will be focused on memory. As the VM usage evaluation algorithm considers VMs as active if they use more than 1% of their CPU by default, this metric is not considered when identifying the oversubscription ratio. On the memory side, however, the heterogeneity of VMs makes it a relevant metric to assess. A manual experimentation is conducted to identify a usable memory overcommitment ratio on this infrastructure. First, a set of idle VMs was identified from a production cluster. Duplicates of these VMs were created and manually placed on a single physical machine. Memory overcommitment mechanisms related metrics as well as the PM CPU usage were monitored during this manipulation.

<figure><img src="image_3.png" /><figcaption>**Figure 1.** Measured ballooned, swapped, shared memory and CPU usage on the PM</figcaption></figure>

Figure 1 shows the amount of memory retrieved by three memory overcommitment mechanisms on the PM during the experimentation, as well the PM CPU usage. The studied memory overcommitment mechanisms are respectively ballooning, swapping and Transparent Page Sharing (TPS). Ballooning is a response to PM memory shortage where a PM asks VMs to free inactive or unused memory to provide it to other VMs. Swapping also occurs in the context of memory shortage. Here, the PM will write
the memory pages allocated to the VMs on an external storage (i.e. its physical disks or a storage controller). TPS is a memory scan of running VMs that identifies identical memory pages on each VMs. For each similar memory page on a specific VM, TPS creates a single copy of the page that is shared between the identical pages.

One can observe that the CPU usage reduces and stabilizes itself after migrating solely the idle VMs on the PM, which validates the hypothesis that CPU oversubscription will not cause performance degradation on the small-scale infrastructure clusters (Clusters S1 and S2).

Multiple other metrics were observed during the experimentation, which are presented in Table 2. Two values are to be identified here: the first one is the total amount vCPU placed on the server. The theoretical number of vCPU to be placed on the PM is here almost reached, as the hypervisor documentation indicates a theoretical maximum value of 32 vCPU per core ¹². The PM having 12 physical cores, the maximum number of vCPU that can be allocated here is 384. Considering this limitation, we identified the memory oversubscription reached here that will be considered as an upper bound in the remaining of the study: $O_{max} = \frac{1168}{384} = 3.04$.

<table><caption>**Table 2.** RAM and CPUs observed metrics on a PM after VMs manual placement</caption><thead><tr><th></th><th>Physical<br>RAM PM<br>(GB)</th><th>RAM Usage<br>PM (GB)</th><th>Sum of VMs<br>configured<br>RAM (GB)</th><th>Sum of VMs<br>configured<br>CPUs</th></tr></thead><tbody><tr><th>Total</th><td>384</td><td>322.79</td><td>1168</td><td>382</td></tr></tbody></table>

This experimentation on the small-scale cluster enabled us to identify an upper bound for the memory oversubscription ratio of 3.04. It also validated the hypervisors' maximal configurations on the CPU perspective as well as the non-necessity of studying CPU oversubscription, given the 1% threshold used in the VM usage evaluation algorithm.

## Simulating consolidation on infrastructures

With all the previous elements collected and determined in the methodology, it is possible to run simulations using different tools and parameter to assess the amount of saved resources and power consumption reduction of the clusters.

**Simulation process on large-scale infrastructure** For large-scale infrastructure, all migration strategies are implemented and executed using the SimGrid Casanova et al. (2014) (version 3.35) simulator.

SimGrid is a mature and renowned framework for simulating distributed computing systems. It has an active community and is used in hundreds of scientific publications. SimGrid allows for fine-grained platform simulation, including hosts, networking and storage. It can be easily extended using the plugin mechanism and is distributed with an energy plugin that enables energy consumption accounting. The energy plugin for host energy calculation is based on a CPU load dependent energy model described in Heinrich et al. (2017). Additionally, SimGrid supports VMs and VM migrations out of the box, making it suitable for IaaS cloud studies. All of this makes SimGrid an excellent choice for conducting our VM consolidation study.

The entire VM consolidation study is implemented with SimGrid in C++ language. The infrastructure description,

including hosts and VMs with their characteristics and initial location, is retrieved from a dedicated *TimescaleDB*¹³ database. For hosts, we consider characteristics such as the number of available cores, as well as the amount of RAM and storage available. For VMs, we consider CPU, RAM and storage requirements. All of these characteristics, along with the initial placement of VMs on hosts, are provided to SimGrid as a simulated platform.

The network topology is a star zone, with the first host being the gateway and all other hosts connected with split duplex links having a link bandwidth of 10 GB/s. We use this basic network topology for simplification purposes, which is suitable for our scenario because we do not account for migration costs in terms of time and energy, and also we perform the VM migration only once.

We use the SimGrid host energy model to estimate energy consumption of hosts. It takes 4 parameters: *Idle* wattage, referring to power consumption when a host is up and running but has nothing to do; *Epsilon* wattage when all cores are at 0 or epsilon load, but not in idle state; *AllCores* wattage when all cores of the host are used at 100%; *Off* wattage when the host is turned off. The energy model requires these parameters to function properly. Initially, we intended to calibrate the energy model using a set of the OVHcloud compute nodes. However, due to the insufficient frequency of power measurements and the complexity of power metrics retrieval, we use a Troll cluster node of large-scale test bed for experimental research called Grid'5000 Balouek et al. (2013), presenting the closest hardware profile to a part of OVHcloud infrastructure and having a high-performance external power measurement available. Therefore, to perform the energy calculations for this scenario, we calibrate the energy model with values retrieved from an external power meter Omegawatt OmegaWatt (2018) for a Troll cluster node. Ideally, to accurately reflect the energy consumption of the infrastructure, we should calibrate the energy model for each compute node individually. However, it is not straightforward in the context of a large-scale infrastructure containing several hundred thousand compute nodes. Thus, we choose to perform the energy model calibration for one compute node and apply it to all compute nodes. In this case, the energy model provides energy consumption estimates assuming that all compute nodes have the same consumption, which is sufficient in our case to provide an idea of energy savings. To simulate the workload and get consumption for various levels of CPU usage, we execute the stress-ng¹⁴ tool with factorial stress method for 120 seconds. The result of energy model calibration can be found in Table 3. All

<table><caption>**Table 3.** SimGrid energy model calibration</caption><thead><tr><th>Model Parameter (wattage)</th><th>Power consumption (average)</th><th>Calibration conditions</th></tr></thead><tbody><tr><td>Off</td><td>10 Watts</td><td>The host is powered down</td></tr><tr><td>Idle</td><td>161.8 Watts</td><td>Nothing is running except the OS</td></tr><tr><td>Epsilon</td><td>186.7 Watts</td><td>Loading all cores at 1%</td></tr><tr><td>AllCores</td><td>392.8 Watts</td><td>Loading all cores at 100%</td></tr></tbody></table>
simulated hosts use the same energy model, regardless of their configuration, such as the number of cores or other resources available.

To calculate the energy savings that can be achieved through consolidation strategies, we first replay the VM CPU usage traces with SimGrid using the initial VM placement and retrieve the energy consumption of hosts estimated by the SimGrid energy model. Then we carry out the consolidation strategy, shut down the freed hosts, replay the VM CPU usage traces with the new VM placement, and retrieve the energy consumption. We define energy savings from a consolidation strategy as the difference between initial energy consumption and energy consumption when replaying CPU usage traces with the VM placement obtained through this strategy.

**Simulation process on small-scale infrastructure** Consid- ering the smaller amount of PMs and VMs in the small-scale infrastructure, an efficient Python simulator was developed to compute consolidation using 15 minutes periods over one week on different clusters. These simulation parameters are used to anticipate production deployment of the consolida- tion methodology for the small-scale infrastructure: the one week period will enable us to train smaller prediction models that are periodically updated. The 15-minute consolidation period is considered as a good trade-off between the number of migrations the consolidation can lead to, while being a reasonable period for potential performance degradation in the event of a consolidation error. The consolidation solver is executed every 15 minutes to:

1. Assign the VMs to their consolidation group.

2. Compute the number of necessary PMs for each consolidation group.

3. Place the VMs on PMs using a FFD heuristic.

The simulator logs the used PMs as well as the VM placement at each iteration, and computes the saved energy at the end of the simulation. The power consumption reduction is computed using the base power consumption of the studied cluster over the simulated week and the cumulated power consumption of used hosts for each iteration. This enables to identify periods during which the consolidation with oversubscription might be more efficient, meaning that there is a higher number of LRU VMs.

This temporal simulation approach may also enable us to compare consolidation results using collected metrics and metrics obtained from prediction models. This would allow us to compare power consumption reduction as well as potential consolidation errors caused by prediction models.

## Result analysis

In this section, we analyze results of VM consolidation for both small and large-scale infrastructure.

### Results for large-scale infrastructure

For large-scale infrastructure, we perform experiments for each previously introduced migration strategy, employing all previously presented allocation ratios and LRU VMs detection thresholds where applicable. Instead of presenting the results of all the experiments, we focus on the most appealing ones.

**Analyzing the consolidation potential** We begin by examining infrastructure statistics to reveal the potential for VM consolidation. Table 4 shows overall infrastructure statistics extracted from production infrastructure traces.

<table><caption>**Table 4.** Large-scale Cluster L1 statistics</caption><thead><tr><td>Hosts</td><td>VMs</td><td>Unused hosts</td><td>Average host CPU usage</td></tr></thead><tbody><tr><td>985</td><td>7376</td><td>20</td><td>9.3%</td></tr></tbody></table>

It displays the total number of hosts and VMs, as well as the number of unused hosts, and the average CPU usage of all hosts in the cluster. We observe that the studied cluster has 20 unused powered-on hosts that do not contain any VMs. These hosts could be turned off to save energy. Additionally, we can mention that the average CPU usage of the hosts is very low (< 10%). Hosts are significantly underutilized, indicating substantial resource waste and opportunity for VM consolidation.

Table 5 displays resource usage statistics for used hosts with at least one VM. The percentage of CPUs, RAM, and disk resources used represents the portion of resources allocated to VMs. There is a significant amount of space available on used hosts, especially on disk usage. This space could potentially be filled through VM consolidation, reducing the number of required hosts.

<table><caption>**Table 5.** Large-scale infrastructure resource usage statistics for used hosts</caption><thead><tr><th colspan="2">CPUs</th><th colspan="2">RAM</th><th colspan="2">Disk</th></tr><tr><th>Total</th><th>Allocated</th><th>Total</th><th>Allocated</th><th>Total</th><th>Allocated</th></tr></thead><tbody><tr><td>51.1k</td><td>66.5%</td><td>145.8 TB</td><td>90.7%</td><td>1234.7 TB</td><td>64.2%</td></tr></tbody></table>

### Detecting Low Resource Usage Virtual Machines

Figure 2 shows the percentage of LRU VMs for LRU ratios of 99% and 100% (percentage of time when the maximum resource usage of the VM is less than the chosen thresholds) on Cluster L1. The LRU ratio of 99% is the least restrictive, allowing VMs to exceed the threshold for 1% of the time, while 100% is the most restrictive, allowing no threshold exceedances.

Depending on the LRU VM detection thresholds, we observe that LRU VMs represent between 2% and 49.1% of VMs.

As a result, the thresholds chosen have a significant influence on the quantity of LRU VMs detected. Using more relaxed thresholds increases this quantity. However, using too relaxed thresholds increases the risk that a VM will be mistakenly identified as LRU. The more relaxed a threshold is, the higher its value. Thus, LRU VM detection thresholds must be carefully chosen to avoid false detections and potential excessive QoS degradations.

**LRU VMs migration strategy** In Figure 3, we show the number of hosts that can be freed when consolidating LRU VMs detected with various CPU, disk and network thresholds and LRU ratio fixed at 99%. We can mention that the number of freed hosts varies depending on the threshold used for LRU VM detection.
<figure><img src="image_4.png" /><figcaption>**Figure 2.** LRU VMs percentage. Total VM number: 7376</figcaption></figure>

With the most relaxed thresholds, LRU VMs migration strategy can be used to free 58 hosts, representing 5.9% of the total number of hosts.

In the following sections, to simplify the representation of the results, for consolidation strategies involving LRU VMs, we study only the most relaxed thresholds, which means the LRU ratio of 99%, the CPU usage threshold of 10%, and no constraint on disk usage and network exchange.

<figure><img src="image_5.png" /><figcaption>**Figure 3.** Number of freed hosts for the LRU VMs migration strategy and the LRU ratio of 99%. Total hosts number: 985</figcaption></figure>

**LRU VMs migration with oversubscription** After executing the LRU VMs migration with oversubscription strategy using the most relaxed thresholds to detect LRU VMs, we found that for any allocation ratio, no additional hosts can be freed when compared to the LRU VMs migration strategy without oversubscription. The most relaxed thresholds here correspond to 10% CPU usage and no disk or network threshold. The oversubscription always presents risks of performance degradation, therefore it should be used with caution. So, if the goal is to free up as many hosts as possible, the LRU VMs migration strategy without oversubscription should be preferred.

However, the oversubscription can be useful to save CPU and RAM resources. Table 6 shows the quantity of resources

that can be saved by oversubscribing LRU VMs detected with the most relaxed thresholds. We can note that even with low allocation ratios, a significant amount of resources can be saved. Thus, if the goal is to save CPU and RAM resources, it is preferable to oversubscribe the LRU VMs in place rather than migrating them. The CPU/RAM ratio combinations are chosen to highlight the impact of CPU oversubscription alone (lines 3 and 4), memory oversubscription allocation alone (line 1), and a combination of the two (line 2).

In summary, the LRU VMs migration with oversubscription strategy provides no additional benefit in terms of freed hosts in the studied context when compared to the LRU VMs migration strategy without oversubscription, regardless of the allocation ratios used. Although this result may seem peculiar, it should be due to the outcome of the LRU VMs migration strategy without oversubscription, which results in almost no hosts with LRU VMs only. Therefore, near no additional hosts can be freed by moving LRU VMs while applying oversubscription. Depending on the goal, it can be replaced by either the LRU VMs migration without oversubscription strategy or an in-place oversubscription.

<table><caption>**Table 6.** Cores and RAM saved with oversubscription. Total cores allocated to VMs: 33.4k. Total RAM allocated to VMs: 126.4TB</caption><thead><tr><th colspan="2">Oversubscription ratios</th><th rowspan="2">Saved cores</th><th rowspan="2">RAM saved</th></tr><tr><th>CPU</th><th>RAM</th></tr></thead><tbody><tr><td>1</td><td>1.5</td><td>0</td><td>19.5TB</td></tr><tr><td>2</td><td>2</td><td>7.8k</td><td>29.3TB</td></tr><tr><td>4</td><td>1</td><td>11.6k</td><td>0</td></tr><tr><td>8</td><td>1</td><td>13.6k</td><td>0</td></tr></tbody></table>

**General VM migration strategy** Table 7 show the general VM migration strategy execution results. Comparing to LRU VMs migration strategy, this strategy allows saving some additional hosts.

It can be used to free 66 hosts for large cluster, representing 6.7% of the total number of hosts.

The general VM migration strategy poses a higher risk of performance degradation because it migrates all types of VMs, including those that are heavily used. Furthermore, migrating more heavily used VMs is usually more challenging. It can take more time and cause more downtime during the live migration process. Thus, the general VM migration strategy must be used when the benefit is significant.

<table><caption>**Table 7.** VM placement results for the general VM migration strategy and the most relaxed thresholds.</caption><thead><tr><th></th><th>Freed hosts</th><th>VMs migrated</th><th>Migration cost *</th></tr></thead><tbody><tr><td>Result (Total)</td><td>66 (985)</td><td>1934 (7376)</td><td>177 TB (904 TB)</td></tr></tbody></table>

**General VM migration with oversubscription** For the general VM migration with oversubscription strategy, the amount of CPU and RAM saved is the same as for LRU VM migration with oversubscription shown in Table 6, which is expected since the same allocation ratios are applied.

<footnote>* - Including both memory and disk size of VMs.</footnote>
However, looking at Figure 4, which shows the number of freed hosts for the general VM migration with oversubscription strategy, we can observe that for most allocation ratios, the number of freed hosts is more important when compared to the general VM migration without oversubscription strategy (represented in the figure by allocation ratios of 1 for both CPU and RAM).

If only the CPU is oversubscribed, we see only a slight increase in the number of freed hosts, which have only one additional host freed.

When RAM becomes oversubscribed, we achieve significantly greater gains in terms of freed hosts. Even with allocation ratios of 2.0 for CPU and 1.5 for RAM, which can be viewed as “safe” allocation ratios because only LRU VMs are oversubscribed, there is a significant increase in the number of freed hosts when compared to the non-oversubscribed strategy. For these allocation ratios, we see a nearly four-fold increase.

Furthermore, increasing the oversubscription ratio over 2.0 for both CPU and RAM results in only one extra freed host. Here, a CPU ratio over 2 does not improve the results for any given RAM oversubscription ratio. In the studied cluster L1, using allocation ratios of 2 for both CPU and RAM results in nearly 31.5% of freed hosts that can be switched off. However, oversubscribing RAM, even with the allocation ratio of 2, can be risky and lead to significant performance degradation. On the large-scale infrastructure, we do not consider a RAM oversubscription ratio over 2.0 as it is considered too risky.

<figure><img src="image_6.png" /><figcaption>**Figure 4.** Number of freed hosts for the general VM migration with oversubscription strategy. The allocation ratios of 1 for both CPU and RAM represent the number of freed hosts for the general VM migration without oversubscription strategy. Total hosts number: 985</figcaption></figure>

*Energy savings* We estimate the possible energy savings by replaying the CPU usage traces of the VMs before and after executing the migration strategy, as well as by shutting down the freed hosts. The traces used contain one day of CPU usage metrics for each VM.

In Figure 5 we show the percentage of energy saved by three VM migration strategies. The first migration strategy is the LRU VM migration for the most relaxed thresholds,

the second is the general VM migration strategy without oversubscription, and the third is the general VM migration with oversubscription and allocation ratios of 2 for both CPU and RAM. The energy savings are mainly due to powering down the hosts freed by the migration strategy.

The general VM migration strategy with oversubscription and allocation ratios of 2 for both CPU and RAM allowed to save 29.5% of energy, which is not negligible. The general VM Migration strategy without oversubscription resulted in energy savings of 7.6%, which represent only a slight increase over the LRU VMs migration strategy for the most relaxed thresholds, representing 6.8% of energy savings.

<figure><img src="image_7.png" /><figcaption>**Figure 5.** Percentage of energy saved. Strategy A - LRU VMs migration strategy for the most relaxed thresholds. Strategy B - General VM migration strategy without oversubscription. Strategy C - General VM migration with oversubscription and allocation ratios of 2 for both CPU and RAM</figcaption></figure>

*Migration costs* The previously discussed consolidation strategies can result in significant savings in terms of freed hosts, resources, and reduced energy consumption. However, before applying a migration strategy, we need to understand its costs.

In this section, we do not study migration costs in terms of time and energy, as they depend on how the migration is performed, as well as the state and usage of the infrastructure during the migration process. Instead, we investigate migrations costs in terms of migration volumes, including the number of VMs to be migrated and the amount of data to be transferred. These metrics are useful for estimating the potential network impacts and risks of service unavailability induced by the migration process, as well as for properly scheduling it.

Table 8 shows the migration volumes for three consolidation strategies for the general VM type and each cluster. We can mention that some consolidation strategies require transferring hundreds of terabytes of data between hosts, which can cause network congestion if not executed correctly. Furthermore, massive VM migrations can cause significant performance issues.

Therefore, due to their costs, VM migrations must be properly scheduled to avoid network congestions and performance degradations.

## Results for small-scale infrastructure

*Analyzing the consolidation potential* We collected PMs and VMs usage data of 2 virtualization clusters (S1 and
<table><caption>**Table 8.** Migration volumes for three migration strategies for general VM type.</caption><thead><tr><td>Migration strategy</td><td>VMs migrated</td><td>Migration size</td></tr></thead><tbody><tr><td>LRU VMs migration<br>Most relaxed thresholds</td><td>1348</td><td>141 TB</td></tr><tr><td>General VM migration<br>without oversubscription</td><td>1934</td><td>176.8 TB</td></tr><tr><td>General VM migration<br>with oversubscription and<br>allocation ratios of 2<br>for both CPU and RAM</td><td>4148</td><td>474.2 TB</td></tr></tbody></table>

S2) located in the small-scale infrastructure over one year and computed the average usage over a period of one week to assess the potential of LRU-VMs consolidation on these clusters. The first cluster, referred as *Cluster S1* is composed of 6 PMs and 400 daily active VMs, and hosts internal company tools as well as development, testing or integration virtual machines. The second cluster, *Cluster S2*, is a larger cluster composed of 16 PMs and more than 1000 active VMs hosting Ciril GROUP customer's production applications.

On both clusters, there are no unused hosts. The average CPU usage vary between 30 and 60% depending on the cluster and the time of the day, and RAM usage vary between 55 and 70%. CPU oversubscription is already used on these clusters. *Cluster S1* has a virtual CPU (vCPU) per physical CPU (pCPU) ratio of 4 and 90% of the physical memory is allocated to the VMs.

We also assess the consolidation potential by conducting an overall analysis of VM usage, using CPU and I/O thresholds of 1% and 1kbps. We analyze the VM usage over periods of one week and present here relevant results. For each analyzed week, VMs not returning any values over the previous 2 months are removed from the scope of the analysis as they are potentially shutdown or even deleted.

Figure 6 shows the usage analysis of the 2 small-scale infrastructure clusters over one week. Figure 6a shows the VM usage of *Cluster S1*. Among the 399 monitored VMs, we can observe 2 main usage groups: one with highly used VMs (>90% of the time) and one with VMs being used between 30 and 50% of the time.

Figure 6b shows the same analysis conducted on *Cluster S2*. Here we can observe also two main groups of VMs with, on one side, 51.1% of frequently used VMs and on the other side, 18.8% of VMs used less than 10% of the time. Table 9 summarizes a first brief analysis of the number and share of LRU-VMs for each cluster using the CPU and I/O thresholds of 1.

<table><caption>**Table 9.** Number and share of LRU-VMs in each cluster using CPU and I/O thresholds of 1</caption><thead><tr><th></th><th>Cluster S1</th><th>Cluster S2</th></tr></thead><tbody><tr><th>LRU VMs</th><td># of VMs : 399<br>LRU-99: 5 (1.25%)<br>LRU-100: 1 (0.25%)</td><td># of VMs : 1014<br>LRU-99: 60 (5.92%)<br>LRU-100: 2 (0.2%)</td></tr></tbody></table>

This VM usage analysis enables us to get an overall idea of the consolidation potential over one week, as it highlights the share or number of VMs having low resource usage and that could be subject to consolidation

<figure><img src="image_8.png" /><figcaption>(a) Small scale: Cluster S1</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(b) Small scale: Cluster S2</figcaption></figure>

<caption>**Figure 6.** VM usage analysis on the small-scale infrastructure - CPU threshold: 1% and I/O threshold: 1kbps</caption>

with oversubscription. However, VMs idle periods may not coincide, making it impossible to consolidate enough VMs with oversubscription and therefore to shut down servers. In addition to this global usage evaluation, a temporal analysis was conducted to assess periods during which consolidation with oversubscription may perform well, as well as identifying the share of VMs to include in the consolidation.

Figure 7a shows the hourly average usage of a specific group of VMs in *Cluster S1*. From Figure 6a, we extract the group of VMs which usage is between 30 and 50%. This represents a group of 105 VMs, whose usage is not constant. The term usage here refers to the fact that a machine will have CPU or I/O usage above the selected thresholds: 1% for CPU and 1kbps for threshold. Therefore, even a VM considered as used may not consume a significant amount of resources on the PMs. These VMs have a mean usage under 20% at night. The figure also shows the quantile range between the 25th and 75th percentile, and we can observe that 75% of the selected VMs have a mean usage below 25%. If we combine these VMs with those whose overall usage is below 20%, we can expect to have enough VMs on which we can use consolidation with oversubscription. We conduct the same analysis on *Cluster 2*, with results shown in Figure 7b. Here, VMs whose overall usage is under 10% are selected for temporal analysis. On these 184 VMs, we can see a stable mean usage around 7%, with 75% of these 184
<figure><img src="image_10.png" /><figcaption>(a) Temporal analysis - Cluster S1</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>(b) Temporal analysis - Cluster S2</figcaption></figure>

<caption>**Figure 7.** Temporal analysis of specific group of VMs</caption>

VMs having a mean usage under 12%. Considering the 1014 VMs, this represents 13.6% of VMs that can be consolidated with oversubscription.

This consolidation potential analysis translates to a good knowledge of the virtualization infrastructure and may be more difficult to conduct over clusters as *Cluster L1*. This VM usage analysis enables us to identify the potential of VM consolidation on both clusters, with one cluster having a significant proportion of its VMs lightly used during specific periods, and the other cluster having a large number of VMs that are constantly lightly used.

*Analyzing the number of Low Resource Usage Virtual Machines* Figure 8 shows the share of LRU VMs with LRU ratios of 99.0% and 100.0% on the small-scale infrastructure clusters. For both clusters, and similarly to the large-scale infrastructure cluster, we vary the CPU and I/O thresholds.

Depending on the used thresholds, the share of LRU VMs vary between 0.24% and 53.37% for *Cluster S1*, and between 0.2% and 58.09% for *Cluster S2*.

Like for the large-scale infrastructure, the chosen thresholds may have a major role on the potential energy savings after the consolidation, but also on the overall performance degradation on the infrastructures. Considering higher thresholds will naturally increase the number of VMs considered in the consolidation, which may result in further energy consumption reduction. However, it is fair to question the value of a threshold for *Low Resource Usage* VMs. For example, a VM using 10% of its CPU may, depending on different parameters, not be considered idle. Applying oversubscription on this VM will generate stress in its available resources and, therefore, can generate performance degradation on the applications it hosts. We here explore a wide range of thresholds for the LRU VM detection algorithms on both clusters to anticipate this potential performance degradation.

Considering the low amount of LRU VMs using the default combination of CPU and I/O thresholds, and given the fact that consolidation will be applied over period of 15 minutes, we suggest to explore other LRU ratios when conducting the consolidation simulations on these clusters.

<figure><img src="image_12.png" /><figcaption>(a) Small scale: Cluster 1</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>(b) Small scale: Cluster 2</figcaption></figure>

<caption>**Figure 8.** Share of LRU VMs on the small-scale clusters</caption>

*Energy savings* Simulations were run on the two small-scale infrastructure clusters, varying the CPU and I/O thresholds to detect LRU VMs over the consolidation period of 15 minutes. We run the simulations over one week to take into account the evolution of virtual machines usage.

The consolidation follows the group based approach defined in section VM placement on small-scale infrastructure, with a simple separation. VMs being considered as LRU over 90% of the consolidation period will be assigned a RAM oversubscription ratio of 2. All the other VMs will be not be oversubscribed.

We conduct a first global analysis of consolidation results in terms of energy savings using the different CPU and I/O thresholds. Figure 9 shows the results, with power consumption reduction varying from 13.52 to more than 43.79% power consumption depending on the cluster, CPU threshold and I/O threshold. Power consumption reduction is lower for *Cluster S1* (Figure 9a), as it is already smaller sized than *Cluster S2* (Figure 9b). To assess the relevance of the temporal analysis, as well as for the variation in oversubscription ratio, we conduct another simulation analysis on *Cluster S1*. Here, we consolidate LRU-80 VMs with a RAM oversubscription ratio of 3, using the same consolidation period as for the previous simulations.
<figure><img src="image_14.png" /><figcaption>(a) Small scale: Cluster S1</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>(b) Small scale: Cluster S2</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>**Figure 10.** Power consumption reduction on Cluster S1 over one week using LRU-80 VMs with an oversubscription ratio of 3</figcaption></figure>

Figure 10 shows the average power consumption evolution of Cluster S1 with and without VM consolidation. Over one week, we obtain a power consumption reduction of 22.05%. We can clearly see the variations in power consumption, especially during low resource usage periods highlighted in Figure 7a.

Discussion

VM consolidation in heterogeneous cloud infrastructures is far from being straightforward and presents numerous challenges. These challenges depend not only on the size of infrastructure, but also on how it is deployed. For instance, if the storage provided to the VMs is integrated into hosts, VM consolidation is less straightforward due to the need to transfer storage to migrate VMs.

Exploring various VM consolidation strategies allowed us to demonstrate the potential to save a varying amount of resources and energy on both large- and small-scale infrastructures.

For the large-scale infrastructure, the LRU VM migration strategy allows freeing up and shutting down 5.9% of the hosts for the most relaxed LRU VM detection thresholds. Adding oversubscription to the LRU VMs migration strategy does not result in a significant increase in the number of freed up hosts. However, oversubscription done in-place without migrating any VMs saves a significant amount of CPU and RAM resources. The explored allocation ratios result in resource savings ranging from 23.4% to 40.7% for CPU and 15.4% to 23.2% for RAM of total resources used by VMs. Migrating both LRU and non-LRU VMs (general VM migration strategy) can free up some additional hosts. However, the share of freed hosts is 6.7% of the total number of hosts, which is only a slight increase compared to the LRU VM migration strategy. Since migrating heavily used VMs can result in significant performance degradation, the general VM migration strategy alone is of limited benefit for the studied cluster. The general VM migration with oversubscription yields a greater number of freed up hosts. Using CPU and RAM allocation ratios of 2 allows nearly 31.5% of hosts to be freed and switched off. VM consolidation allowed for significant energy savings, ranging from 6.8% to 29.5% of total energy consumed. Although the savings are significant, due to the specificity of VM storage provisioning in the large-scale infrastructure, migration strategies require transferring hundreds of terabytes of data between hosts. Therefore, the migrations must be properly scheduled to avoid network congestions and significant performance degradations.

However, the large-scale infrastructure study presents several limitations. First, it is based only on a single day of traces, which may not reflect the overall behavior of VMs, potentially leading to an excessive number of false LRU detections. To overcome this, more traces need to be studied and if the volume of traces becomes unmanageable, averaging techniques can be employed. Furthermore, since in large-scale infrastructure the storage provided to the VMs is integrated into hosts, migration costs can be extremely high, reaching hundreds of terabytes and requiring proper scheduling to avoid network congestions and performance degradations. In this study, we explored only a single execution of VM consolidation based on previous infrastructure metrics. Further study is required to assess the validity of resulting VM placement over time, as an LRU VM may become non-LRU after consolidation, inducing a risk of performance degradation. This issue can be partially addressed by employing prediction models to predict the behavior of LRU VMs over time and reduce
risks of unexpected changes in resource usage. In addition, the periodic applicability of VM consolidation should be explored.

For the small-scale infrastructure clusters, the share of saved resources also varies depending on the combination of LRU ratio, CPU threshold and I/O threshold. While the absolute amount of saved resources is drastically lower than for the big-scale infrastructure cluster, the averaged results show good saving potentials. The value ranges between 16.67% to 37.5% depending on the cluster. These numbers consider an identical PM configuration across all the hosts of one cluster as they are based on the average number of PM used in each simulation. In terms of energy savings, we obtain a power consumption reduction varying between 13.59% to 43.79%, still depending on the cluster and CPU and I/O thresholds. Both clusters show good potential of saved resources and energy consumption reduction, but other aspects need to be considered such as fault tolerance. For example for the Cluster S1, the average reduction of used PM for most of the threshold combination is 1. If using a fault tolerance rate, this freed-up host could be used in production implementation to avoid service interruption.

On the small-scale infrastructure, CPU oversubscription is already used without performance degradation on the hosted VMs. Considering the small CPU threshold (1%) in the detection algorithm, it is assumed that increasing this CPU oversubscription will not cause new performance degradation on the oversubscribed LRU VMs. Complete performance degradation analysis would require to have access to the VM operating system or the application itself, which is not always the case for public or private cloud providers. Due to their relatively small size, the potential in resource and energy savings is naturally smaller for small-scale infrastructure. Additional features such as considering fault tolerance for production implementation may reduce drastically or even cancel the benefits of consolidation for these small clusters. Similarly to the large scale infrastructure, other consolidation periods may be explored to strike a right balance between benefits and risks of using consolidation with oversubscription.

For both clusters, the presented numbers are considered to be upper bound values for the number of freed hosts and the energy savings. Indeed, different parameters may affect the performance of the consolidation when deployed in production. Using proactive consolidation with prediction models may affect and reduce the energy savings depending on the quality of the models. Other features, such as fault tolerance, that requires running extra PMs to ensure continuity of service in case of a PM crash, can also reduce the benefits obtained through the methodology.

## Conclusion and future works

In this work, we revisited VM consolidation by exploring a set of VM consolidation strategies on a small- and large-scale infrastructures using real production traces. These two infrastructures not only have differences in terms of size, but also in the way they are deployed. Small-scale infrastructure has VM storage centralized on storage arrays, so there is no need to migrate VM storage from one PM to other, reducing drastically VM migration costs.

Whereas in the large-scale infrastructure, storage is local to physical machines, requiring storage to be transferred for each VM migration. Differences in infrastructure size impact the amount of metrics that can be retrieved and analyzed, potentially impacting the accuracy of LRU VMs detection. Moreover, the infrastructure size impacts the performance and execution time of VM placement algorithms, as VM placement is an NP-hard problem and optimal VM placement is difficult to achieve at large-scale within a reasonable amount of time.

For large-scale infrastructure, we define and explore four migration strategies, with the goal of minimizing the number of physical hosts required to host VMs while performing as few migrations as possible using a solved-based approach that provides close to optimal VM placements. The migration strategies address both LRU and non-LRU VMs migration, as well as LRU VMs oversubscription. We explore the resource and energy savings potential offered by each migration strategy by progressively moving from the most safe strategy to the least safe in terms of performance degradation risk. Before delving into migration strategies, we evaluate the consolidation potential by analyzing initial VM placement and resource allocation metrics, and identify LRU VMs by applying a set of defined thresholds on their usage metrics. We highlight that by carrying out certain VM migration strategies, we manage to free up and shut down a significant number of compute nodes and save a considerable amount of energy.

On the small-scale infrastructure, the consolidation potential and number of LRU VMs are assessed in a similar way to large-scale infrastructures. An additional analysis is conducted over a period of one week with temporal usage variation to understand the VM usage profiles and validate the consolidation potential. Due to the limited size of the clusters in the small-scale infrastructure, only consolidation with general VM migration, i.e. LRU and non-LRU, and oversubscription is studied. Oversubscription is only applied to LRU-VMs, using a wider range of VMs than LRU-99 or LRU-100 VMs. Indeed, VMs are consolidated every 15 minutes and oversubscription is applied to VMs used for less than 10% of the time of this consolidation period. We evaluate our consolidation strategy on two different clusters that both show gains in energy consumption, even with an initial small number of PMs.

The VM consolidation problem, although it has been widely studied in the literature, is still composed of challenges to assess especially to be effectively used in production on heterogeneous cloud infrastructures. In this work, we propose a versatile consolidation methodology that can be applied on multiple infrastructure varying in size and architecture. We highlight the importance of studying the consolidation potential on these heterogeneous infrastructures and explore different consolidation strategies in simulations, both by varying the number and considered VMs to migrate on one side and by applying temporal consolidation on the other. The proposed methodology reveals itself applicable to the 2 heterogeneous studied infrastructure, with specificity to consider on each side. In both cases, it results in a considerable amount of saved resources and energy consumption reduction.
In this work, we investigated the potential of VM consolidation to save cloud resources and reduce energy consumption. The study was carried out using simulation. Therefore, it opens up a variety of perspectives. First, workloads in a cloud environment can be highly dynamic because clients can create and remove VMs at any time. Thus, workload prediction models can be used to optimize VM consolidation strategies. Next, after experimenting with simulation, an appealing option is to implement and execute VM consolidation strategies in a real production environment. Also, integrating end users as well as operational teams in the implementation of consolidation on production infrastructures may increase its potential and appears as a option to consider. Finally, evaluating the benefits of VM consolidation on other environmental impacts on data centers is appealing, considering the concerns they raise in terms of water consumption or raw material usage for example.

## Acknowledgements

This work is supported by the "FrugalCloud" Inria and OVHcloud partnership. This work was funded by ANRT (CIFRE N°2021/1297). Experiments presented in this paper were carried out using the Grid'5000 / Slices-fr testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER, and several Universities as well as other organizations (see https://www.grid5000.fr).

## Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

## Notes

1. Slurm Power Saving Guide - https://slurm.schedmd.com/power_save.html

2. VM Ware Managing Power Resources - https://docs.vmware.com/en/VMware-vSphere/8.0/vsphere-resource-management/GUID-5E5E349A-4644-4C9C-B434-1C0243EBDC80.html

3. Veeam Idle VMs - https://helpcenter.veeam.com/docs/mp/reports/idle_vms_vmware.html?version=9a

4. Google Cloud Idle VM recommendations - https://cloud.google.com/compute/docs/instances/idle-vm-recommendations-overview

5. Eg Reporter Idle VMs Report - https://www.eginnovations.com/documentation/The-eG-Reporter/Virtual-Machines-Idle-VMs-Report.htm

6. CP-SAT Solver - https://developers.google.com/optimization/cp/cpSolver

7. Gurobi Optimized - https://www.gurobi.com/

8. Hexaly Optimizer - https://www.hexaly.com/

9. VM-Host Affinity Rules with vSphere DRS - https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.resmgmt.doc/GUID-2FB90EF5-7733-4095-8B66-F10D6C57B820.html

10. OpenStack CPU topologies - https://docs.openstack.org/nova/train/admin/cpu-topologies.html

11. OpenStack Overcommitting CPU and RAM - https://docs.openstack.org/arch-design/design-compute/design-compute-overcommit.html

12. vSphere Configuration Limits - https://configmax.broadcom.com/guest?vmwareproduct=vSphere&release=vSphere%207.0&categories=2-0

13. Timescale DB - https://github.com/timescale/timescaledb

14. stress-ng - https://github.com/ColinIanKing/stress-ng

## References

Balouek D, Amarie AC, Charrier G, Desprez F, Jeannot E, Jeanvoine E, Lèbre A, Margery D, Niclausse N, Nussbaum L et al. (2013) Adding virtualization capabilities to the Grid’5000 testbed. In: Cloud Computing and Services Science: Second International Conference, CLOSER 2012, Porto, Portugal, April 18-21, 2012. Revised Selected Papers 2. Springer, pp. 3–20.

Beloglazov A (2013) Energy-efficient management of virtual machines in data centers for cloud computing. PhD Thesis, University of Melbourne, Department of Computing and Information Systems.

Benoit A, Lefèvre L, Orgerie AC and Raïs I (2017) Shutdown policies with power capping for large scale computing systems. In: Euro-Par 2017: Parallel Processing: 23rd International Conference on Parallel and Distributed Computing, Santiago de Compostela, Spain, August 28–September 1, 2017, Proceedings 23. Springer, pp. 134–146.

Buyya R, Ilager S and Arroba P (2024) Energy-efficiency and sustainability in new generation cloud computing: A vision and directions for integrated management of data centre resources and workloads. Software: Practice and Experience 54(1): 24–38. DOI:10.1002/spe.3248.

Casanova H, Giersch A, Legrand A, Quinson M and Suter F (2014) Versatile, scalable, and accurate simulation of distributed applications and platforms. Journal of Parallel and Distributed Computing 74(10): 2899–2917.

Dabbagh M, Hamdaoui B, Guizani M and Rayes A (2015) Toward energy-efficient cloud computing: Prediction, consolidation, and overcommitment. IEEE network 29(2): 56–61.

Heinrich FC, Cornebize T, Degomme A, Legrand A, Carpen-Amarie A, Hunold S, Orgerie AC and Quinson M (2017) Predicting the energy-consumption of MPI applications at scale using only a single node. In: 2017 IEEE international conference on cluster computing (CLUSTER). IEEE, pp. 92–102.

Helali L and Omri MN (2021) A survey of data center consolidation in cloud computing systems. Computer Science Review 39: 100366. DOI:https://doi.org/10.1016/j.cosrev.2021.100366. URL https://www.sciencedirect.com/science/article/pii/S157401372100006X.

Khan MA, Paplinski A, Khan AM, Murshed M and Buyya R (2018) Dynamic Virtual Machine Consolidation Algorithms for Energy-Efficient Cloud Resource Management: A Review.
In: Rivera W (ed.) *Sustainable Cloud and Energy Services: Principles and Practice*. Cham: Springer International Publishing. ISBN 978-3-319-62238-5, pp. 135–165. DOI:10.1007/978-3-319-62238-5-6.

Kim Sg, Eom H and Yeom HY (2013) Virtual machine consolidation based on interference modeling. *the journal of Supercomputing* 66: 1489–1506.

Lambert S, Caron E, Lefèvre L and Grivel R (2024) S-ORCA: A social-based consolidation approach to reduce Cloud infrastructures energy consumption. In: *15th IEEE International Conference on Cloud Computing Technology and Science - Cloudcom 2024*. URL https://inria.hal.science/hal-04797304.

Lefèvre L and Orgerie AC (2010) Designing and evaluating an energy efficient cloud. *The Journal of Supercomputing* 51: 352–373.

Liu H, Xu CZ, Jin H, Gong J and Liao X (2011) Performance and energy modeling for live migration of virtual machines. In: *Proceedings of the 20th international symposium on High performance distributed computing*. pp. 171–182.

Mann ZÁ (2015) Approximability of virtual machine allocation: much harder than bin packing. In: *9th Hungarian-Japanese Symposium on Discrete Mathematics and Its Applications*.

OmegaWatt (2018) Omegawatt. URL http://www.omegawatt.fr/.

Orgerie AC, Assuncao Md and Lefèvre L (2014) A Survey on Techniques for Improving the Energy Efficiency of Large-Scale Distributed Systems. *ACM Computing Surveys* 46(4): 1–31.

Ostapenco V (2024) *Modeling, evaluating and orchestrating heterogeneous leverages for large-scale cloud data centers management*. PhD Thesis, Universite de Lyon, FR.

Raïs I, Orgerie AC, Quinson M and Lefèvre L (2018) Quantifying the impact of shutdown techniques for energy-efficient data centers. *Concurrency and Computation: Practice and Experience* 30(17): e4471.

Shehabi A, Smith SJ, Masanet E and Koomey J (2018) Data center growth in the United States: decoupling the demand for services from electricity use. *Environmental Research Letters* 13(12): 124030.

Srikantaiah S, Kansal A and Zhao F (2008) Energy aware consolidation for cloud computing. In: *Proceedings of the 2008 Conference on Power Aware Computing and Systems, HotPower’08. USA: USENIX Association*, p. 10.

Voorsluys W, Broberg J, Venugopal S and Buyya R (2009) Cost of virtual machine live migration in clouds: A performance evaluation. In: *Cloud Computing: First International Conference, CloudCom 2009, Beijing, China, December 1-4, 2009. Proceedings 1*. Springer, pp. 254–265.

Zou J, Wang K, Zhang K and Kassim M (2024) Perspective of virtual machine consolidation in cloud computing: a systematic survey. *Telecommun. Syst.* 87(2): 257–285. DOI:10.1007/s11235-024-01184-9. URL https://doi.org/10.1007/s11235-024-01184-9.

Distributed and HPC Platforms) from the LIP laboratory in Ecole Normale Supérieure of Lyon, France. He also works for Ciril GROUP, a french software-developing company and a cloud provider. He works on the environmental impacts of cloud data centers.

Vladimir Ostapenco received a PhD degree in computer science from Ecole Normale Supérieure of Lyon, France, in December 2024. During his PhD, he was a member of the Avalon team (Algorithms and Software Architectures for Distributed and HPC Platforms) from the LIP laboratory in Ecole Normale Supérieure of Lyon, France and worked on energy efficiency and environmental impacts of cloud data center infrastructures as part of the Inria-OVHcloud (FrugalCloud) challenge.

Laurent Lefèvre is a permanent researcher in computer science at Inria (the French Institute for Research in Computer Science and Control). He is a member of the Avalon team (Algorithms and Software Architectures for Distributed and HPC Platforms) from the LIP laboratory in Ecole Normale Supérieure of Lyon, France. He has organized several conferences in high-performance networking and computing. He has co-authored more than 200 articles published in refereed journals and conference proceedings. Since 18 years, he works on improving energy efficiency and reducing environmental impacts of large-scale systems (HPC centers, data centers, clouds, and big wired networks).

Eddy Caron is a Professor at University of Lyon (ISFA department) and holds a position with the LIP laboratory (ENS Lyon, France). He is a member of AVALON project from INRIA. He received his PhD in C.S. from University de Picardie Jules Verne in 2000 and his HDR (Habilitation à Diriger les Recherches) from the Ecole Normale Supérieure de Lyon in 2010. He was co-founder and scientific advisor for two start’up SysFera and Qirinus. His research focuses on distributed computing environment. At the middleware level, he deals with a large scope of subjects: scheduling, workflow management, data management, energy management and, security, etc. He published more than 200 publications.

Anne-Cécile Orgerie received a PhD degree in computer science from Ecole Normale Superieure de Lyon, France, in September 2011. She has been a full time researcher at CNRS in the IRISA laboratory, Rennes, France, since October 2012. Her research is in the broad area of distributed systems, and in particular focuses on energy efficiency and environmental impact of cloud computing infrastructures and smart grids.

Benjamin Fichel has been working for OVHcloud as a Telecom and Network engineer for the last 15 years. Within the infrastructure group, he leads several transformation projects in order to modernize the control plane and change the paradigms of infrastructure consumption through the use of standardized APIs.

Rémi Grivel is Managing Director of Ciril GROUP and SynAApS. He took part in the creation of the SynAApS datacenter in 2010, and now heads the teams responsible for maintaining it in operational condition. Since the creation of the SynAApS datacenter, he has been working with the teams to provide the highest standards of security, while minimizing the datacenter’s impact on the environment.

## Author Biographies

Simon Lambert is a third-year PhD student in computer science at Ecole Normale Supérieure of Lyon and Inria (the French Institute for Research in Computer Science and Control). He is a member of the Avalon team (Algorithms and Software Architectures for