# xPUE: Extending Power Usage Effectiveness Metrics for Cloud Infrastructures

Guillaume Fieni, Romain Rouvoy, Lionel Seinturier

► To cite this version:

Guillaume Fieni, Romain Rouvoy, Lionel Seinturier. xPUE: Extending Power Usage Effectiveness Metrics for Cloud Infrastructures. IEEE Transactions on Sustainable Computing, In press, 10.1109/TSUSC.2025.3549687. [hal-04980997](https://hal.science/hal-04980997v1)

HAL Id: hal-04980997

https://hal.science/hal-04980997v1

Submitted on 6 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# xPUE: Extending Power Usage Effectiveness Metrics for Cloud Infrastructures

Guillaume Fieni,*,† Romain Rouvoy,†,* Lionel Seinturier,†,*

* Inria, France

† Univ. Lille, CRISTAL, UMR CNRS 9189, France

***Abstract***—The energy consumption analysis and optimization of data centers have been an increasingly popular topic over the past few years. It is widely recognized that several effective metrics exist to capture the efficiency of hardware and/or software hosted in these infrastructures. Unfortunately, choosing the corresponding metrics for specific infrastructure and assessing its efficiency over time is still considered an open problem. For this purpose, energy efficiency metrics, such as the Power Usage Effectiveness (PUE), assess the efficiency of the computing equipment of the infrastructure. However, this metric stops at the power supply of hosted servers and fails to offer a finer granularity to bring a deeper insight into the Power Usage Effectiveness of hardware and software running in cloud infrastructure.

Therefore, we propose to leverage complementary PUE metrics, coined xPUE, to compute the energy efficiency of the computing continuum from hardware components, up to the running software layers. Our contribution aims to deliver real-time energy efficiency metrics from different perspectives for cloud infrastructure, hence helping cloud ecosystems—from cloud providers to their customers—to experiment and optimize the energy usage of cloud infrastructures at large.

## I. INTRODUCTION

Modern Data Centers (DCs) are continuously trying to maximize the Power Usage Effectiveness (PUE) of their infrastructure to reduce their operating cost, and eventually their carbon emissions [12]. In the context of cloud providers, PUE is increasingly adopted and communicated as a Key Performance Indicator (KPI) reflecting the energy efficiency of the delivered solution. While the optimal PUE is 1.0, most infrastructures in 2023 stagnate at a PUE of 1.58 on average, according to a survey from the Uptime Institute [17]. For years, reducing the PUE of DC has become an active research area where actors compete to propose the most efficient cooling techniques, consider renewable energies as part of their electricity mix, and improve the utilization of IT resources.

However, the emergence of the cloud computing paradigm has strengthened the role and the impact of software layers on the energy consumption of these DCs. While the server PUE (sPUE) has only been mentioned in the literature [2], its scope remains limited to hardware layers, hence failing to address the software layers spectrum. In particular, we believe that both virtualization technologies and control planes may shuffle reported energy efficiency indicators, if not appropriately optimized. One could, for example, envision scenarios where the PUE of a DC is reported as optimal, but the combined inefficiency of the software layers deployed in the hosted servers does not reflect the savings achieved by

the building infrastructure, hence misleading cloud customers. This is all the more prevalent as cloud provider are intensively communicating on PUE as a marketing argument to attract customers.<fnref n="1" />,<fnref n="2" />

We, therefore, propose to push the limits of the state-of-the-art hardware metrics—namely, the PUE and sPUE—and to introduce a novel software metric—coined virtual PUE (vPUE)—to better reflect the end-to-end energy consumption of cloud computing operators by reporting on the PUE of software layers, which can be indefinitely stacked. More specifically, we introduce xPUE as a family of composable metrics that can be used to study and assess the energy efficiency of cloud infrastructures across the hardware-software continuum.

Using xPUE, the cloud ecosystem—including cloud providers and their customers—can investigate energy wastes in the software layers of their infrastructure and take appropriate actions to reduce them. For example, our experiments highlight the impact of platform control planes as well as the importance of operating large-scale infrastructure at their maximum capacity.

To summarize, the four contributions reported in this article are:

1) A formalisation of the sPUE metric, which has been mentioned by [2], but never properly defined,

2) The introduction of xPUE as a family of new and complementary PUE metrics to cover all the software layers of cloud infrastructures,

3) The implementation of xPUE metrics using the POWERAPI framework [5],

4) The study of xPUE metrics across multiple cloud deployments to deliver insightful feedback on the key factors impacting the Power Usage Effectiveness of cloud infrastructures.

This article starts by delivering background on state-of-the-art PUE metrics and their limitations (cf. Section II) before introducing our contribution (cf. Section III). Then, we detail the implementation of xPUE as a real-time energy effectiveness metric for cloud infrastructures (cf. Section IV) and we assess its validity across multiple cloud deployments (cf. Section V). We conclude in Section VII.

<footnote n="1">¹https://www.google.com/intl/en/about/datacenters/efficiency/</footnote>

<footnote n="2">²https://datacenters.microsoft.com/sustainability/efficiency/</footnote>
## II. RELATED WORK

While the literature in our area of DC effectiveness is abundant, we choose to focus on the metrics of growing interest in the cloud industry: the PUE and its variants.

a) Power Usage Effectiveness (PUE): The standard ISO/IEC 30134-2:2016 [12] defines the PUE as a metric reflecting the energy efficiency of a DC. More specifically, the PUE is defined as the ratio of the overall energy consumed at the DC level to the energy consumed by hosted IT equipment, thus estimated as:

$$PUE = \frac{\sum Energy(DC)}{\sum Energy(IT)} = 1 + \frac{\sum Energy(non-IT)}{\sum Energy(IT)} \quad (1)$$

where $\sum Energy(DC)$ is the sum of energies drawn by what is not considered as a computing device, but is required to operate a DC (so-called non-IT), such as lighting, air conditioning, etc., and the IT equipment.

The PUE is a widely-adopted metric, often cited by major cloud providers to demonstrate their progress in DC efficiency.<fnref n="3" /> While the ideal PUE is 1.0, the average PUE for a DC in 2020 was 1.58 [17]. For example, Google publishes quarterly and trailing 12-month PUE values going back to 2008 for their DC hosted globally and reports a fleet-wide PUE of 1.10 for 2021 [13].

Nonetheless, PUE has been criticized when adopted as a measure of efficiency because it only considers energy and does not consider the effective usage of the computational resources [4], [7]. This means that a PUE can mechanically decrease by artificially increasing the IT workload, hence increasing the energy consumption of servers to reduce the impact of non-IT equipments.

b) Data Center Infrastructure Efficiency (DCIE): The DCIE is the reciprocal of the PUE, defined as:

$$DCIE = \frac{1}{PUE} = \frac{\sum Energy(IT)}{\sum Energy(DC)} \quad (2)$$

One can observe that, although PUE and DCIE are the most commonly-used metrics to compare the energy efficiency of DC, they only assess the global energy efficiency and fail to provide any insight into the IT efficiency in particular [24].

Furthermore, the scopes of PUE and DCIE do only capture the building efficiency of a DC, hence stopping at the power outlet of hosted computing units. This partial coverage is particularly critical in the context of cloud computing, which does not limit itself to building infrastructures, but heavily rely on computing resources (servers, routers, etc.) and software platforms to operate a wide diversity of Everything-as-a-Service (XaaS) offers—ranging from Infrastructure-as-a-Service (IaaS) to Software-as-a-Service (SaaS). By focusing and communicating on PUE, cloud providers might hinder the overhead of these computing resources and software platforms, hence misleading stakeholders about the end-to-end efficiency of their products.

c) Carbon Usage Effectiveness (CUE): The CUE (in kilograms of carbon dioxide per kilowatt-hours: kgCO2eq) aims to assess the efficiency of the energy used for the DC:

$$CUE = \frac{\sum Emission_{CO2}(DC)}{\sum Energy(DC)} \quad (3)$$

The CUE does not take into account the embodied emissions accountable to the manufacturing of the DC or its equipment.

However, the CUE includes the carbon emissions due to the mix of energy being used by the DC in production.

d) sPUE: Introduced by Barroso et al. in 2013 [2], the sPUE is computed as the ratio of the server input power to its useful component power, including all the parts directly involved in the computations, namely motherboard, disks, CPUs, DRAM, GPU, I/O, etc. sPUE aims to quantify the efficiency of individual servers and the authors report that state-of-the-art sPUE should be less than 1.2 at the time of writing their book. In particular, low sPUE should result from efforts in delivering optimized supply and cooling steps.

However, to the best of our knowledge, neither the cloud operators nor the literature adopted this indicator to report on best practices in the design of hardware servers and, eventually, cloud computing offers.

e) DWPE: Data center Workload Power Efficiency (DWPE) is another metrics introduced by Wilde et al. in 2019 [26] to assess the energy efficiency of the High Performance Computing (HPC) infrastructures. While DWPE complements sPUE by considering application workloads, it fails to appropriately evaluate the various software layers of cloud infrastructures by focusing on performance objectives of HPC applications.

*Limitations & Opportunities:* Most energy efficiency indicators aim to evaluate global—at a DC granularity—or specific hardware parts—at a baremetal granularity—of the cloud infrastructures. However, none of them proposes an end-to-end approach from the scale of DC to individual servers and even the hosted software services, no matter their location and potentially distributed nature. In particular, in the context of cloud infrastructures, providing an end-to-end energy indicator should allow deeper analysis and tuning of the infrastructure hardware and software components to optimize energy usage at large.

In this article, we therefore propose complementary metrics to analyze, in-depth, the PUE of cloud infrastructures, specifically the virtual layers introduced by these infrastructures. Such a family of complementary PUE metrics are intended to quantify, for 1 Watt consumed by a cloud native service, how many Watts are effectively required by the hosting DC.

## III. xPUE: A NEW FAMILY OF PUE METRICS

xPUE groups a family of PUE-related metrics that can be easily adopted by cloud providers, and more generally service providers, to estimate the PUE of their infrastructure in the deep, including the software platforms they may operate. xPUE extends standard PUE and sPUE metrics by delivering insights beyond the power supply of physical servers.

<footnote n="3">³How Microsoft measures datacenter water and energy use to improve Azure Cloud sustainability</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. Efficiency coverage of state-of-the-art and xPUE metrics (highlighted in grey and yellow, respectively).</figcaption></figure>

## A. Overview of xPUE

In this article, we specifically leverage the sPUE mentioned—but never formalized—by Barroso et al. [2] to introduce the vPUE metrics and highlight resource usage effectiveness at the scales of hardware and virtual layers, respectively. Figure 1 depicts the complementarity of xPUE metrics with state-of-the-art metrics, including PUE, sPUE, and CUE. One can observe that, while PUE and CUE are delivering an in-breadth coverage of the resource usage effectiveness of DC buildings along different perspectives—energy and carbon, respectively—sPUE and vPUE brings more in-depth insights by investigating how these resources are consumed within hardware clusters and servers and, ultimately, cloud software services and platforms. As the energy consumption of the hardware servers and software services is proportional to their usage, they can exhibit significant variability when it comes to energy efficiency. In the context of this article, we are therefore interested in investigating the end-to-end resource usage effectiveness of cloud services, by covering hardware and software layers. By doing so, we intend to expose the real effectiveness of cloud infrastructures and to contribute to a more transparent exposure of how much energy is consumed by cloud providers for each functional unit, whenever operating a Metal-as-a-Service (MAAS), IAAS, or a Container-as-a-Service (CAAS) offer. By focusing on the software layers that are not covered by existing metrics, we intend to raise new environmental challenges for cloud infrastructures, thus encouraging the ecosystem to maximize the end-to-end power usage effectiveness of cloud services, hence going beyond the sole effectiveness of DC buildings, as currently covered by the PUE. This more holistic analysis of the energy and carbon efficiency of cloud infrastructures is brought by the definition of global PUE (GPUE) and global CUE (GCUE) indicators, which are introduced in the following sections.

## B. sPUE: Assessing Cloud Servers PUE

The sPUE aims to estimate the Power Usage Effectiveness at the scale of a server [2]. This metric is particularly relevant in the cloud when operating a MAAS (also known as bare-metal) offer, which consists in delivering a hardware server to the customer. In this context, the cloud provider is in charge of hosting and eventually assembling the delivered server. While the PUE stops at the power outlet of hosted servers, the

sPUE metric intends to dive into the integrated components to capture the overhead imposed by the cooling systems and power supplies inside the server, as mentioned by [2]. In this context, we formalize the sPUE as the ratio between the energy consumed by the IT equipment (cf. Equation 1) and the energy directly consumed by the hardware components contributing the computations of the cloud infrastructure (incl. CPU, GPU, memory, disk, controllers, etc.), as follows:

$$sPUE = \frac{\sum Energy(IT)}{\sum Energy(hardware)} \quad (4)$$

This metric, therefore, indicates how much energy is consumed by the physical server for each unit of computation delivered by the CPU and other critical components. By formalizing the sPUE, we are interested in highlighting the efforts spent by cloud providers to deliver energy-proportional bare-metal solutions—i.e., limiting the cost of cooling components and optimizing the supply of direct current. We believe that this additional dimension is important to capture, as there is no guarantee that the PUE reflects this overhead.<fnref n="4" /> While the PUE is usually computed at the scale of a DC, the sPUE can be computed at the scale of a single server, a rack, a cluster, a room or a DC, depending on the scope of the analysis. Server-scale sPUE can be used to improve the design of individual servers to reduce potential energy waste, which DC-scale sPUE is intended to report on the average efficiency of a farm of, potentially heterogeneous, production servers. Therefore, beyond the reported numbers, we strongly recommend indicating the scale at which the sPUE has been analyzed. Additionally, we recommend listing the hardware components that have been considered in the computation of the sPUE, as the sPUE can be computed for different hardware configurations.

sPUE covers general-purpose computations that can be delivered to cloud customers, but also exploited internally to operate a cloud platform. However, no matter their nature—being IAAS, CAAS, Platform-as-a-Service (PAAS), etc.—these cloud platforms are hosting a large number of software services, which are in charge of managing the cloud infrastructure and delivering the hosted services to the customers. These software services are generally implemented as Virtual Machine (VM) or containers, and they are often deployed on top of a hypervisor or a container engine. In this context, we are interested in investigating how much energy is consumed by the cloud software services for each unit of computation delivered to the hosted software, no matter their nature—i.e., VM or containers.

## C. vPUE: Assessing Cloud Services PUE

The metric vPUE dives into the software layers of cloud infrastructures by investigating the cost of operating large and complex software platforms, such as OPENSTACK for a IAAS or KUBERNETES for a CAAS. These cloud solutions generally share two key concepts: virtualization techniques to control access to the computational resources, and control planes to

<footnote n="4">⁴The PUE might eventually reflect, indirectly, side-effects of server emissions like a reduced consumption of the air cooling system of a DC.</footnote>
deploy and manage the hosted services. To capture this metric, we define the vPUE as follows:

$$vPUE = \frac{\sum Energy(\text{hardware})}{\sum Energy(\text{software})} \qquad (5)$$

where hardware (taken from Equation 4) refers to all the server components required to operate the software hosted by the considered cloud platform. The list of included software covers all the processes running on the hosting server, which encompass the operating system, the cloud platform, as well as other daemon processes that can be deployed by the cloud operator. At first, the resulting ratio is intended to measure the overhead imposed on the execution of general-purpose software processes. From this general-purpose indicator, we can leverage the principle of partial PUE (pPUE) [12] to introduce partial sPUE indicators that capture the overhead imposed by different aspects of the cloud infrastructure. This can include the overhead of hypervisors and the associated control planes that we consider as a specific scope (S), which is used to filter out the software process of interest together with the hardware components they are running on:

$$vPUE(S) = \frac{\sum_{s \in S} Energy(\text{hardware}_s)}{\sum_{s \in S} Energy(\text{software}_s)} \qquad (6)$$

where $s \in S$ selects all the hardware and software parts that are related to the scope S. As a result, vPUE(KUBERNETES) is intended to capture the overhead of the KUBERNETES cloud platform, while vPUE(OPENSTACK) is intended to capture the overhead of the OPENSTACK. This scope can also be further refined to capture the overhead of dedicated planes of a cloud platform, such as comparing the efficiency of the control plane compared to data plane, as illustrated in Section V.

One should also keep in mind that, unlike hardware layers, software layers can be stacked by cloud providers and/or their customers. For example, the deployment of a KUBERNETES cluster atop VM hosted by a IAAS infrastructure is commonly adopted by practitioners to offer more flexibility when it comes to adjusting resource usage.

### D. cPUE: Applying xPUE Metrics to Cloud Infrastructures

To account for nested virtualization practices and to better reflect the end-to-end efficiency of a cloud platform, we introduce the cloudPUE (cPUE) metric, which is a metric computed from sPUE and vPUE. Concretely, to report this compound metric, one needs to compute the product of xPUE metrics, depending on the spectrum of the deployed infrastructure, as follows:

$$cPUE = \prod_{x \in L} xPUE \text{ with } L \text{ the selected cloud layers} \qquad (7)$$

where the list of selected layers, L, depends on the context and the owner of the cloud infrastructure. For example, a IAAS provider operating an OPENSTACK cloud platform will compute $cPUE = sPUE \times vPUE(\text{OpenStack})$. Similarly, a KUBERNETES platform hosted on-premise will rather be reported as $cPUE = sPUE \times vPUE(\text{Kubernetes})$. Finally, because of the recursive property of virtualization techniques, one can imagine computing $cPUE = sPUE \times$

$vPUE(\text{OpenStack}) \times vPUE(\text{Kubernetes})$ to capture the end-to-end PUE a multi-layers CAAS or PAAS platform leveraging several cloud technologies. Interestingly, stakeholders of a cloud ecosystem do not need full control of the infrastructure to estimate the cPUE of their solution. For example, any customer of a IAAS provider can compute the cPUE of their hosted software by multiplying the vPUE of their VM by the cPUE reported by their cloud provider.

Furthermore, beyond being a product of xPUE metrics, any cPUE metric can also be multiplied by the PUE of the DC hosting the software services as follows:

$$gPUE = cPUE \times PUE \qquad (8)$$

which is reported as the GPUE revealing, for each unit of computation performed by cloud software, how much energy is effectively consumed by the whole DC hosting this software. Given that xPUE metrics share the same properties as the PUE—i.e., the ideal value is 1.0—one can observe that any waste of resource in any of the covered layers may severely impact the global PUE of the infrastructure, thus challenging the cloud ecosystem to pay attention to the effectiveness of their hardware and software solutions.

At this point, one can suspect that the PUE reported by the cloud industry does not reflect the real cost of the efficiency of infrastructures by only focusing on the optimizations operated on the hosting facilities and omitting the cost of the software running on top of these facilities. We, therefore, recommend adopting the GPUE metric to share more transparently the efficiency of individual cloud offers.

### E. Revisiting State-of-the-Art Metrics with cPUE

Beyond the GPUE, one can also revisit the state-of-the-art metrics to consider their global impact, beyond the DC building optimizations. In particular, the GCUE metric can be more accurately reported by cloud infrastructures as follows:

$$gCUE = cPUE \times CUE \qquad (9)$$

where the CUE metric is extended with the cost of the cloud platform, hence better reflecting the carbon footprint of each unit of computation operated by a cloud platform.

xPUE can be extended to other environmental metrics, such as DCIE, Green Power Usage Effectiveness (GPUE), Green Energy Coefficient (GEC), Energy Reuse Factor (ERF) and Water Usage Effectiveness (WUE) to cover a larger scope of the environmental impact of cloud infrastructures [22].

## IV. IMPLEMENTATION DETAILS OF xPUE METRICS

The implementation of the xPUE metrics requires reporting power measurements at a finer granularity than a power outlet. While any software-defined power meter, like SCAPHANDRE<fnref n="5" /> or KEPLER,<fnref n="6" /> can be used to implement the xPUE metrics we propose, we leverage the POWERAPI toolkit [5], [9] to implement the runtime support xPUE.

<footnote n="5">⁵https://github.com/hubblo-org/scaphandre</footnote>

<footnote n="6">⁶https://sustainable-computing.io</footnote>
## A. Introducing the POWERAPI Toolkit

POWERAPI is a middleware toolkit for building software-defined power meters [5], [9]. Software-defined power meters are configurable software libraries that can estimate the power consumption at the scale of individual software processes in real-time. POWERAPI supports the acquisition of raw metrics from a wide diversity of sensors (e.g., physical meters, processor interfaces, hardware counters, OS counters) and the delivery of power consumptions via different channels (including file system, network, web, graphical).

To implement xPUE, we more specifically leverage the POWERAPI toolkit to collect energy metrics from multiple sources of energy measurements and to aggregate them. For the global energy measurements required by the sPUE—i.e., $\sum Energy(IT)$ in Equation 4—we leverage hardware power meters plugged into the power supply and Intelligent Platform Management Interface (IPMI) for the global power measurements of the servers. As IPMI measurements have a low refresh rate and a low accuracy, we recommend using hardware power meters whenever possible. When the server combines multiple power supplies, the power measurements of all active power supplies are aggregated.

For hardware-specific measurements (CPU, DRAM, GPU...) required by the sPUE and vPUE—i.e., $\sum Energy(hardware)$ in Equations 4 and 5—we consider the hardware power interfaces, such as Running Average Power Limit (RAPL) for the CPU, as it is widely available and accurate on recent CPUs [8], [23].

Regarding software power measurements—i.e., $\sum Energy(software)$ in Equation 5—we use the SMARTWATTS power meter [10], [11], which automatically infers power models from hardware measurements and disaggregates power consumption among software processes. SMARTWATTS supports power estimation both at the granularities of VMs and containers, as well as it succeeds in estimating CPU and DRAM power consumptions in real-time with high accuracy. Furthermore, as SMARTWATTS offers process-scale power measurements, we can therefore easily separate the power consumption of any software or service running in the infrastructure (from control/master nodes to worker/compute nodes). Concretely, to differentiate the hosted cloud software from the control plane services, we can select and tag pre-defined groups of software. For example, in a KUBERNETES cluster, some services are directly related to the infrastructure, like the container runtime, control plane, networking, and monitoring services. Subgroups can also be defined for the services: one can compute dedicated vPUE for network and monitoring services to further analyze their energy efficiency.

## B. Implementing the xPUE Formulas

We implemented each xPUE metric as a Python formula in the open-source library POWERAPI (cf. Figure 2). The POWERAPI toolkit exposes a software agent that consumes input measurements from various sources (databases, message queues), processes the samples, and produces estimations through the same or another database(s). We also leverage

<figure><img src="image_3.png" /><figcaption>Fig. 2. Deployment of xPUE metrics using POWERAPI</figcaption></figure>

the de facto standard libraries in Python: PANDAS for the manipulation and analysis of samples, as well as SCIPY for the computation of the xPUE estimations.

This results in the development of a family of dedicated xPUE formulas atop POWERAPI that continuously estimate sPUE and vPUE indicators and can even be further combined to report on the vPUE and gPUE compound metrics in real-time. As xPUE metrics rely on power measurements coming from multiple sources (RAPL, IPMI, hardware, and software power meters) that are not synchronized, we resample the power measurements to a common time base. By default, xPUE formula detects the shortest time window possible to resample the PANDAS `DataFrame`. Finally, our implementation of xPUE leverages a database to store the power measurements received from hardware and software power meters. These measurements are then handled by the formula components and xPUE metrics are then stored in the database for further analysis and visualization (e.g., as a GRAFANA dashboard).

## C. Deploying the xPUE Metrics

To simplify the deployment process, xPUE is available as containers, which provide an environment-agnostic deployment and ease the lifecycle management of its related services. The formula components can be deployed on any host of the cluster, or a remote server, as it only requires access to the power measurements through a message queue to work. For example, one can use a MONGODB instance as a message queue to communicate the power-meters measurements through a publish-subscribe pattern and then store the xPUE metrics into an INFLUXDB Time Series DataBase (TSDB) or a PROMETHEUS agent. The xPUE metrics can be exposed as GRAFANA dashboards for environment/service-specific metrics reporting and real-time/offline analysis. Figure 2 depicts a typical deployment of xPUE to monitor the vPUE of multiple parts of a cloud infrastructure involving both IAAS (OPENSTACK) and CAAS (KUBERNETES) platforms.

## V. EMPIRICAL VALIDATION

This section builds on our implementation of xPUE to study the various factors that may contribute to improving or
degrading the Power Usage Effectiveness of cloud infrastruc-
tures. We, therefore, start by investigating the impact of sPUE
on different hardware configurations (cf. Section V-B), then
exploring the vPUE in the context of a IAAS infrastructure
(based on OPENSTACK) and a CAAS infrastructure (based
on KUBERNETES). We conclude by illustrating the cPUE
in the context of a CAAS deployed in a IAAS, a widely-
adopted architecture in the cloud industry (cf. Section V-D).
Throughout this section, we assess the relevance of xPUE to
evaluate the Power Usage Effectiveness into deeper layers of
KUBERNETES and OPENSTACK based cloud infrastructures.

A. Evaluation Methodology

We follow the experimental guidelines reported by [25] to
enforce the quality of our empirical results. For the sake of
reproducible research, xPUE, the necessary tools, deployment
scripts, and resulting datasets are open-source and publicly
available on GitHub.<fnref n="7" />

Hardware Settings: Most of our experiments are deployed on the Grid’5000 testbeds infrastructure [1], which is a large-scale and flexible testbed for experiment-driven research in all areas of computer science, with a focus on parallel and distributed computing—including cloud, HPC, Big Data, and AI. We deploy our experimental infrastructure on 5 nodes of the cluster gros located in the site of Nancy. The description of the considered servers is reported in Table I (cf. 3 first columns). This cluster is particularly interesting as each node has its power supply monitored by a hardware power meter to monitor the power consumption of the node with high accuracy and at a high frequency.

We also consider the provisioning of additional bare-metal
servers from a production-scale cloud infrastructure provided
by OVHCLOUD (2 last columns of Table I). We chose this
cloud provider as all the hosted servers are cooled using
an advanced water cooling system, which may contribute
favorably to the sPUE, compared to traditional air cooling.<fnref n="8" />
Furthermore, OVHCLOUD proposes a MAAS offer with a
wide diversity of hardware architectures (including Intel and
AMD processors) and access to server-scale power measure-
ments.

*Software Settings:* All machines of our experiment are using the Ubuntu 20.04 LTS Linux distribution with a kernel version 5.4.0-121-generic, where only a minimal set of daemons are running in the background.

For the KUBERNETES cluster, the deployment is done using
kubectl and the version deployed is the 1.21. The container
runtime is Containerd version 1.6.6 and the Container Net-
work Interface (CNI) deployed is FLANNEL version 0.18.1.

For the OPENSTACK cluster, the deployment is done using MICROSTACK, which deploys OPENSTACK version Ussuri through the SNAP package manager. This allows us to quickly deploy an OPENSTACK instance in self-contained packages that support individual monitoring through Linux Cgroups. This choice considerably eases the deployment and the re-

producibility of our results, while remaining representative of
real-world deployments.

**Input Workloads:** For both deployments, we use a state-of-the-art benchmark tool for Linux, STRESS-NG, to simulate various resource-intensive workloads that stress various parts of the system. Therefore, during each experiment, containers and VM will be started with a random resource workload that will be maintained until it is stopped. This allows us to generate a base workload on the infrastructure and to stress multiple parts of the software services and underlying hardware under varying loads, from resource-scarce to resource-intensive CPU workloads.

Our approach does not prevent to use additional benchmarks
to stress the system, like SPEC<fnref n="9" /> or PHORONIX,¹⁰ to further
investigate the impact of the input workload on the energy
efficiency of the cloud infrastructure, as well as memory-
intensive benchmarks.

While synthetic, these input workloads are intended to mimic real-world workloads that can be found in cloud infrastructures [18], and are used to stress the system to its maximum capacity, therefore exploring a wide spectrum of values for the sPUE and vPUE indicators we introduced.

*Power Meters:* To monitor the power consumption of the servers, we use the available hardware power meters attached to the input of the power supply of every machine in the cluster. The measurements are automatically reported to an INFLUXDB at a frequency 50 measurements per second, which is more than required for our experiments.

Regarding software measurements, SMARTWATTS requires the deployment of two components, the *Sensor* which monitors the *Hardware Performance Counters* (HwPC) and needs to be deployed on every node, and the *Formula* component which computes the power estimations. In all our experiments, we configure the *Sensor* components to report on power estimations once per second ($\beta$ = 1 HZ), and the *FORMULA* component with an error threshold of $\alpha$ = 5 W at the scale of the CPU package (PKG).

B. sPUE Experiments

a) *Server setting.:* To evaluate the relevance of sPUE, we run our benchmark designed to stress twice the maximum CPU load of the Intel Xeon Gold server. This aims to investigate the behavior of the hardware infrastructure in a situation of resource over-commitment. Figure 3 reports on the evolution of the sPUE over time, when increasing CPU workload. One can first observe that idle servers seriously degrade the energy efficiency of cloud infrastructures, with a $sPUE > 4$.

Beyond this first observation, the sPUE tends to decrease with higher CPU usage, no matter the number of concurrent containers, as acknowledged by Figure 4. With an average value of 3.1 throughout the experimentation, one can observe that the optimal and lowest value reached for the sPUE of such a standard server caps at 2.7, which is high above the values commonly communicated for PUE. As the sPUE

<footnote n="7">⁷Anonymized</footnote>

<footnote n="9">⁹https://www.spec.org/benchmarks.html</footnote>

<footnote n="8">⁸https://blog.ovhcloud.com/water-cooling-from-innovation-to-disruption-part-i/ ¹⁰https://openbenchmarking.org/tests/pts</footnote>
<table><caption>TABLE I
TESTBED HARDWARE SETTINGS</caption><thead><tr><th rowspan="2">Provider Model</th><th colspan="3">Grid'5000</th><th colspan="2">OVHCLOUD</th></tr><tr><th>Dell PowerEdge R640</th><th>Dell PowerEdge R7525</th><th>Dell PowerEdge R640</th><th>Intel bare-metal server</th><th>AMD bare-metal server</th></tr></thead><tbody><tr><td>CPU</td><td>Intel Xeon Gold 5220</td><td>AMD EPYC 7352</td><td>Intel Xeon Gold 5218</td><td>Intel Xeon Silver 4214R</td><td>AMD EPYC 7413</td></tr><tr><td>Generation</td><td>Cascade Lake</td><td>Zen 2</td><td>Cascade Lake</td><td>Cascade Lake</td><td>Zen 3</td></tr><tr><td>Socket(s)</td><td>1</td><td>1</td><td>2</td><td>2</td><td>1</td></tr><tr><td>Cores per socket</td><td>18</td><td>24</td><td>16</td><td>12</td><td>24</td></tr><tr><td>Threads per socket</td><td>36</td><td>48</td><td>32</td><td>24</td><td>48</td></tr><tr><td>Total threads per server</td><td>36</td><td>48</td><td>64</td><td>48</td><td>48</td></tr><tr><td>Memory</td><td>96 GB</td><td>128 GB</td><td>384 GB</td><td>32 GB</td><td>64 GB</td></tr><tr><td>TDP</td><td>125 W</td><td>155 W</td><td>125 W</td><td>100 W</td><td>180 W</td></tr><tr><td>Cooling</td><td></td><td>Air</td><td></td><td></td><td>Water</td></tr></tbody></table>

<figure><img src="image_4.png" /><figcaption>Fig. 3. Evolution of the sPUE over time and increasing workload</figcaption></figure>

should be combined with the PUE of any cloud provider to report the GPUE reflecting the physical power consumption induced by any Joule consumed by the computing units (cf. Equation 8), it can only degrade the value of the standard PUE. These observations thus challenge cloud infrastructures and MAAS providers to deliver more energy-efficient servers by investing in energy-proportional techniques to power linearly a server with the triggered activities and computations. This is particularly critical as DC and cloud providers are known to be under-utilized [3], [6], which let us expect higher sPUE in production than the one we measured in our testbed under stress conditions.

b) Hardware impact.: To further investigate this hardware overhead of servers, we hypothesize like [2] that the sPUE might be influenced by the CPU architecture and the cooling system of a server. Therefore, using the same input workload as in the previous experiment, we compare the sPUE of Intel and AMD servers that are deployed in Grid'5000 (based on air cooling) and OVHCLOUD (based on water cooling) infrastructures (cf. Table I). While water cooling systems directly contribute to improving the PUE of the DC, they may also indirectly impact the sPUE by removing the CPU/GPU fans from the server frame, hence saving energy consumption induced by the equipment embedded in most of the servers. Figure 5 thus depicts the Cumulative Distribution Function (CDF) of sPUE for each hardware configuration of Table I. Each configuration is stressed in the same conditions, by executing a workload of twice the maximum CPU load of each configuration to offer a fair comparison. One can observe that the best sPUE (1.4) results from the combination of a

water cooling system and a single AMD CPU. Interestingly, the optimal value for this configuration is quickly reached and maintained, compared to the other hardware configurations that report a wider distribution of sPUE values.

c) Other impacts.: While we could not change the AC-DC power supply of our testbed or include alternative supply designs, we believe that the sPUE can also capture the efficiency of this hardware component and contribute to adopting a more energy-efficient solution. We are also confident that this separation of concerns could be captured by a partial sPUE, inspired by the pPUE [12], to isolate the overhead imposed by this hardware component.

One can therefore observe that optimizing the sPUE does not only require adopting energy-saving strategies to power and cool down hardware components, but also maximizing the utilization of provisioned resources. This observation thus challenges the hardware configuration of servers to be carefully sized to the closest number of CPU threads, and other hardware components, which are required to support a target workload. In this context, elasticity mechanisms should be deployed at the hardware level by cloud infrastructure to implement energy proportionality and deal with the variability of workloads, hence preventing the over-provisioning of resources that require to be kept always on.

d) Cluster sPUE.: Beyond single node deployments, cloud infrastructures are often considered to provision a cluster of several nodes that are then assembled to deploy a IAAS (e.g., OPENSTACK) or CAAS (e.g., KUBERNETES) platform. These clusters are typically composed of, at least, a control node and several worker nodes. We, therefore, consider the injection of a cluster-wide input workload to both OPENSTACK and KUBERNETES platforms, deployed atop 5 Intel Xeon Gold 5220 servers, to study the average sPUE at the scale of a cluster, as mentioned in Section III-B. Table II and Figure 6 compare the sPUE distribution of all the nodes involved in the cluster for both experiments. Similarly to the case of idle servers, idle platforms may be the root cause of a critical sPUE observed at the scale of a cluster, with observed factors above 20. Such situations typically witness the over-provisioning, and under-allocation, of a cloud infrastructure that allocates much more computing resources than required. While the optimal sPUE observed for a cluster of 5 nodes reaches 2.9, one can observe that even a stressful scenario like one of our scenario results in an average sPUE of 5 to 6.5—depending on platforms—thus highlighting the critical
<figure><img src="image_5.png" /><figcaption>Fig. 4. Correlation of the sPUE and the CPU average load</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 5. Comparing the sPUE of all the hardware configurations under test.</figcaption></figure>

<table><caption>TABLE II
CONTROL-WIDE SPUE STATISTICS.</caption><thead><tr><th>Platform</th><th>min</th><th>max</th><th>mean</th><th>median</th></tr></thead><tbody><tr><td>KUBERNETES</td><td>2.9</td><td>22.6</td><td>5</td><td>3.8</td></tr><tr><td>OPENSTACK</td><td>2.9</td><td>27.5</td><td>6.5</td><td>4.4</td></tr></tbody></table>

<figure><img src="image_7.png" /><figcaption>Fig. 6. Comparing the sPUE of a cluster used to host OPENSTACK and KUBERNETES platforms.</figcaption></figure>

<table><caption>TABLE III
CONTROL PLANE IMPACT ON SPUE.</caption><thead><tr><th>Node</th><th>Platform</th><th>min</th><th>max</th><th>mean</th><th>median</th></tr></thead><tbody><tr><td>control node</td><td>KUBERNETES</td><td>2.9</td><td>4.4</td><td>4</td><td>4</td></tr><tr><td></td><td>OPENSTACK</td><td>2</td><td>6.2</td><td>3.3</td><td>2.9</td></tr><tr><td>worker nodes</td><td>KUBERNETES</td><td>1.4</td><td>6.7</td><td>2.1</td><td>1.8</td></tr><tr><td></td><td>OPENSTACK</td><td>1.7</td><td>14.9</td><td>3.5</td><td>2.8</td></tr></tbody></table>

impact of the power usage efficiency of the nodes composing the cluster required to host a cloud infrastructure.

*Network impact.:* In the context of a distributed setting, like a cluster, the question of including network equipment might arise. As part of our experiment, we consider the impact of the network—i.e., a single *Top of Rack (TOR)* switch in our context—as part of this cluster setting, assuming that a switch is required to connect several nodes.<fnref n="11" /> However, the power consumption of most hardware network equipment (switch, routers, etc.) is known to be stable, no matter the workload (115.3 W in the context of our TOR switch), thus we advocate including the energy consumption of network

equipment within the IT part of the sPUE, and not as part of the hardware (cf. Equation 4). The motivation for doing such is that sPUE could be easily reduced by adding more and more network equipment in the hardware part (divisor), which would go against the objective of optimizing the energy efficiency of the overall cloud infrastructure.

*Control plane impact.:* Then, we compute the sPUE of the control node and the worker nodes separately for each configuration (cf. Table III and Figure 7). One can observe that no matter the cloud platform, the sPUE of the control node is always higher than the sPUE of the worker nodes, as the control plane is consuming fewer resources than the worker nodes, on average, and thus, fails to exploit the full efficiency of the underlying hardware configuration. One can nonetheless observe that an OPENSTACK control plane reports

<footnote n="11">¹¹Production-scale cloud infrastructures may involve multiple TOR switches to ensure failover.</footnote>
<figure><img src="image_8.png" /><figcaption>Fig. 7. Comparing the sPUE of control & workers nodes for OPENSTACK & KUBERNETES platforms.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Fig. 8. Evolution of the vPUE of OPENSTACK when increasing the number of hosted VM.</figcaption></figure>

a slightly lower sPUE on average, compared to its worker nodes. This can be explained by the consumption of this control node, which tends to be relatively higher than other nodes, due to the number of control services that are involved in the OPENSTACK platform. To reduce the impact of the control plane on the sPUE, one should therefore consider maximizing the number of active worker nodes and consider the deployment of carefully sized control nodes, involving potentially smaller servers.

One can also observe that the sPUE of KUBERNETES and OPENSTACK platforms seem to differ, which highlights that both technologies are not stressing the hardware components in the same way. To better understand the root cause of these differences, we further explore the vPUE of both platforms in the following section.

## C. vPUE Experiments

We then move to the study of vPUE, first in the context of an IAAS platform, based on OPENSTACK, and then on a CAAS platform, based on KUBERNETES. Through the following experiments, we intend to study the resource overhead imposed by the software platform and all the services it provides to deploy and control VM and/or containers. We believe that such software layers represent another key efficiency factor to carefully consider when delivering cloud infrastructures. Furthermore, unlike hardware layers, software layers can be embedded to deliver, for example, a CAAS platform atop IAAS. Thus, considering the impact of such common practices is also another insightful feedback that we intend to cover thanks to the vPUE indicator.

### Estimating the vPUE of OPENSTACK:

a) Platform setting.: To estimate the vPUE of OPENSTACK, we used the same input workload as in the sPUE experiments, namely, we run a benchmark designed to start twice the maximum CPU load of the provisioned cluster. This aims to investigate the efficiency of the cloud infrastructure ranging from an idle state to a situation of resource overcommitment. We use a separate control node for hosting cluster-wide OPENSTACK services in addition to services that are required to be deployed in the worker nodes. The CPU and DRAM overcommitment parameters are kept to default: 16:1

and 1.5:1, respectively. Each allocated VM uses a profile m1.exp, which consists of 1 vCPU and 256 MB of DRAM. Given that we use 4 worker nodes, summing to 144 threads and about 384 GB of DRAM, we can allocate—in theory—up to 2,304 vCPUs and 576 GB of vRAM, which represent about 2,000 VM with the profile m1.exp. However, as previously mentioned, we stop our benchmark when reaching 288 VM, which represents twice the hardware threads made available at the scale of the cluster.

The vPUE is computed as the ratio of the total energy consumption of all the cluster services (including VM) to the energy consumption of hosted VM. As introduced in Section IV-A, we estimate the energy consumption of individual VM by using the SMARTWATTS software-defined power meter [10].

b) Overcommitment impact.: Figure 8 illustrates the evolution of the vPUE over time when increasing the number of hosted VM. One can observe that the more VM, the better vPUE, as one could expect.

More precisely, Figure 9 shows that the vPUE converges towards its optimal value when reaching more than 100 hosted VM, which roughly corresponds to the total amount of physical resources (CPU & memory) available in the cluster we provisioned. This average value is estimated to 1.3 in the context of our deployment of OPENSTACK, involving 1 control node and 4 worker nodes. This first part of the experiment demonstrates that, as does the sPUE, the optimal vPUE is reached when fully stressing the worker nodes, which encourages the optimization of the overcommitment parameters as a way to maximize the resource utilization of cloud infrastructures, hence favorably contributing to both indicators.

c) Control plane impact.: Regarding the control plane of OPENSTACK, one can leverage the SMARTWATTS power measurements reported at the scale of individual services. In particular, one can observe that the major power-consuming service is neutron-api, which is the central service for managing virtual machines in OPENSTACK. In our testbed, NEUTRON-API consumed 5× more than the average power consumption of a hosted VM, which can be partially explained by the benchmark we run to stress the OPENSTACK platform.

Yet, when considering the overall energy consumption for
<figure><img src="image_10.png" /><figcaption>Fig. 9. Correlation of the vPUE with the amount of provisioned VM</figcaption></figure>

the scenario we executed, one can observe that almost 80% of the energy is consumed by the virtual machines, and 20% are imposed by the OPENSTACK services, resulting in an aggregated vPUE of 1.25.

We believe that such detailed reports offered by partial vPUE and SMARTWATTS software-defined power meter can be further exploited by the operators of OPENSTACK platforms to highlight platform energy hotspots that require to be carefully considered, hence challenging the relevance and benefit of deployed control services, beyond the standard configurations.

One should also note that, while the vPUE reports a lower value (1.25) than the sPUE (2.7, in Figure 6), it keeps representing a factor that needs to be combined with other indicators—i.e., cPUE = 2.7 × 1.25 = 3.375 in this experiment—to reason upon global indicators and not partial ones. This cPUE roughly indicates that 1 Watt consumed by any hosted VM effectively imposes 3.375 Watts at the scale of the cluster, in the best case.

Estimating the vPUE of KUBERNETES: Beyond the specific case of OPENSTACK, we also investigate the vPUE of a KUBERNETES platform in this section.

d) Platform setting.: We keep using our benchmark designed to start twice the maximum CPU load of the cluster. To keep the configuration of the KUBERNETES cluster as close as possible to a production environment, we follow the best practices and do not allow the scheduling of containers to the control plane. This means that, in a cluster composed of 4 worker nodes, there are 144 available threads. During this experimentation, we observe an average vPUE of 1.3 with an optimal value of 1.1. Interestingly, one can observe that the control plane of KUBERNETES consumes less energy than the one of OPENSTACK to execute the same workload. One can observe in Table IV that the services composing the control plane of OPENSTACK impose an overhead of 151.1 kJ (910%) compared to the control plane of KUBERNETES, while the expectable overhead imposed by virtual machines over containers is limited to 109.6 kJ (23%). The control plane of KUBERNETES thus leaves 96.88% of the total energy consumed for the execution of hosted containers, compared to 78.08% in the case of OPENSTACK.

<table><caption>TABLE IV
ENERGY CONSUMPTION OF KUBERNETES AND OPENSTACK SERVICES.</caption><thead><tr><th>Node</th><th>Platform</th><th>Energy (kJ)</th><th>diff (kJ)</th></tr></thead><tbody><tr><td>control</td><td>KUBERNETES</td><td>15.6</td><td></td></tr><tr><td>services</td><td>OPENSTACK</td><td><strong>166.7</strong></td><td><strong>+151.1</strong></td></tr><tr><td>hosted</td><td>KUBERNETES</td><td>484.3</td><td></td></tr><tr><td>jobs</td><td>OPENSTACK</td><td><strong>593.9</strong></td><td><strong>+109.6</strong></td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Fig. 10. Comparing the cPUE of hardware/software configurations.</figcaption></figure>

## D. cPUE & GPUE Experiments

As previously mentioned, KUBERNETES clusters can be provisioned atop a set of VM hosted by a IAAS. In such a situation, the vPUE of the IAAS has to be combined with the one of KUBERNETES to better reflect the resulting efficiency of the platform.

To evaluate the ability of xPUE to assess the energy efficiency of a KUBERNETES infrastructure provisioned in an OPENSTACK cluster, we explore different configurations of IAAS/CAAS technologies and compute the associated cPUE. The estimated cPUE metrics are summarized in Figure 10. By combining sPUE and vPUE metrics, one can observe that both the hardware server and the software stack can have a strong influence on the energy consumption of cloud infrastructures. In our setup, at the scale of a single server, one can observe that the cPUE can range from 1.68 for a baremetal KUBERNETES cluster hosted by AMD servers provisioned by OVHCLOUD to 3.19 for a virtualized KUBERNETES cluster provisioned atop a set of OPENSTACK virtual machines deployed in a private IAAS (Grid5000). This observation strengthens our claim that the optimization of energy efficiency of cloud infrastructures requires a holistic approach covering all hardware and software layers, beyond the sole optimization of the *Data Center* and its PUE.

To further reveal the concrete energy footprint of a cloud service, one should therefore combine the cPUE of the cloud infrastructure with the PUE of the data center hosting the servers. Figure 11 estimates GPUE of different deployments of OPENSTACK and KUBERNETES hosted on an AMD server provisioned by OVHCLOUD, according to the selected DC. We use the PUE publicly published as a reference for the DC
<figure><img src="image_12.png" /><figcaption>Fig. 11. Comparing the GPUE of DC/platform configurations.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Fig. 12. Comparing the GCUE of OVHCLOUD DC of various countries.</figcaption></figure>

built by OVHCLOUD—i.e., Gravelines in France, Beauharnois in Canada, and Limburg in Germany. As for the DC leased to OVHCLOUD—i.e. Big Data Exchange (previously Telstra) for Singapore and NextDC for Sydney in Australia—we use the data publicly published by the operators as a reference.

One can observe that all the DCs owned by OVHCLOUD report on a similar GPUE, no matter the operated cloud platform, which is more efficient than traditional DCs. The GPUE reported in Figure 11 highlights the limitations of exposing the PUE in isolation, with values that are all above 1.8, while the PUE is claimed to stagnate at 1.5, according to the Uptime Institute [17].

Given the worldwide presence of OVHCLOUD and other cloud providers, we are also studying the impact of the effective DC location. This location has a direct impact on the energy mix adopted to power the DC, which can be considered to estimate the related carbon emissions. Therefore, Figure 12 estimates the GCUE of several DCs of OVHCLOUD across the world. As there is no data officially published by OVHCLOUD at the DC granularity for the CUE, we choose to rely on the data published by ELECTRICITY MAP as a reference for the *Carbon dioxide Emission Factor* (CEF) of the countries where are based each DC.

By extending the CUE with the CPUE we described in this article, we observe that the range of GCUE is further extended, with 48.75 eqCO2 per unit of computation for operating software in a KUBERNETES cluster to 1,262 eqCO2 for the same unit of computation operated in Sydney in a KUBERNETES cluster that is deployed atop OPENSTACK. This figure shows that the combined choice of DC location, server configuration and software stacks can have a strong impact on carbon emissions, as illustrated by the factor 26 observed between the two extreme values of Figure 12.

We believe that the adoption of xPUE metrics can not only help cloud providers to better understand the impact of their choices on the energy and carbon footprint of their services but also help their customers to make more informed decisions when selecting a cloud provider. We also hypothesize that xPUE metrics can stimulate the hardware and software industries to design and implement more energy-efficient solutions, which can be further validated by xPUE metrics.

## VI. RECOMMENDATIONS

Based on the above experiments, we can formulate the following recommendations for the design of energy efficient cloud platforms. These recommendations are targeting both cloud providers and cloud users, depending on the level of control they have over the cloud infrastructure:

* **Impact of cooling.** We observed that liquid-cooled servers report better energy efficiency than traditional air-cooled servers (cf. Figure 5). Liquid-cooling is one example of passive cooling techniques, which can dissipate more heat per watt, compared to active techniques. We, therefore, recommend to prioritize the provisioning of passively-cooled servers operated by cloud operators.

* **Impact of hardware.** Although they exhibit higher TDP properties (cf. Table I), AMD processors reported better energy efficiency than Intel processors, on average. As observed in Figure 5, reported numbers can be influenced by the model of processors, its generation, its chip technology. Beyond the impact of the cooling system, the chassis can also influence the energy efficiency of the servers. Given this combination of factors, we recommend to report *partial* sPUE of individual hardware components, such as CPU, GPU, as well as the cooling system, the power supply and the chassis in order to encourage the provisioning of energy-efficient hardware components.

* **Impact of usage.** Server usage should be maximized to optimize the energy efficiency. This can be done by consolidating workloads on fewer servers [14], turning off idle VMs and servers [27], and using power management features to reduce power consumption. Cluster-wide oversubscription [20], as well as dynamic [19] and vertical [21] oversubscription are alternative techniques to increase the utilization rate of cloud servers and VMs, while preserving their quality of service.

* **Impact of clusters.** We observed that the control plane may induce a non-negligible impact on the energy efficiency of a cloud platform (cf. Figure 7): the more worker node, the lower impact of the control plane. We, therefore, advocate considering the adoption of large clusters in
cloud data centers, possibly subscribing public/managed cloud offers, instead of private cloud deployments. Given this observation, and the above-reported impacts, increasing the size of any cluster can only be beneficial if the usage of individual servers is maximized. Additionally, the hardware configuration of control plane servers can be optimized to reduce their energy consumption, as well as the number of control plane servers, according to our observations.

* **Impact of virtualization.** We observed that software containers are reporting better energy efficiency than VMs. Nonetheless, the energy efficiency of VMs remains competitive, as long as it is not considered as a means to implement nested virtualization, like hosting a container runtime inside a VM (cf. Figure 10). When possible, we recommend the deployment of containers in public cloud infrastructures.

* **Impact of energy mix.** Furthermore, when going beyond energy efficiency, and considering carbon emissions of cloud infrastructures, we observed that the physical location of DCs has a critical impact on carbon emissions (cf. Figure 12). Our end-to-end xPUE metrics highlight that the choice of physical location is of utmost importance due to the amplification the emissions imposed by the combination of energy-efficiency factors.

* **Impact on water consumption.** As mentioned in Section III-E, one can observe that nothing prevent to combine the cPUE compound metric we propose (cf. Section III-D) with the state-of-the-art Water Usage Effectiveness (WUE) metric in order to compute the global WUE (GWUE) indicator, which can be used to assess the water consumption of cloud infrastructures as follows:

$$g_{WUE} = c_{PUE} \times WUE = \frac{c_{PUE} \times \sum AnnualWater(DC)}{\sum Energy(IT)} \quad (10)$$

By optimizing their cPUE, cloud stakeholders can effectively assess the water consumption of cloud infrastructures with the resulting indicator, GWUE, and compare the water consumption of different cloud infrastructures, which is another environmental KPI that is increasingly adopted by cloud providers.

## VII. CONCLUSION

The energy efficiency of cloud infrastructures is a critical concern for service deployments and a lot of work has been made to accurately evaluate this efficiency. While multiple indicators aim to assess the efficiency of data centers, none of them takes into account the efficiency of the hosted software. They mostly treat the hosted software as a black box and report yearly feedback about the global efficiency of the infrastructure, which does not allow the evaluation of specific parts of the infrastructure.

To address this issue, we propose xPUE, a family of Power Usage Effectiveness metrics that deliver a complementary perspective on the energy efficiency of cloud infrastructures, which remain uncovered by the state of the art. In this paper, we report on the implementation of xPUE metrics using the

POWERAPI framework and its SMARTWATTS power meter to assess the impact of all the layers of a cloud infrastructure, from hardware settings to the software platforms. While we demonstrate our approach on KUBERNETES and OPENSTACK test clusters, we strongly believe that it can be deployed and used across a large variety of infrastructures. We took care to allow high flexibility to the users to easily adapt our solution to their specific infrastructure control plane and tools.

Furthermore, thanks to xPUE, cloud infrastructure operators can assess in real-time the efficiency of their infrastructure down to the software level. This allows for more experiments on energy management policies, and faster feedback about their efficiency for the cloud operator. Instead of relying on metrics that impose a long delay (mostly yearly) before getting feedback, we believe that xPUE provides a reliable indicator of the energy efficiency of the different parts of cloud infrastructure, while allowing more flexibility for the cloud operator. In particular, we believe that cloud software architectures, like microservices, can leverage these online indicators to better guide the optimization of the resource consumption [15], [16] and influence the deployment and resources.

Beyond the optimization of the infrastructure, we believe that the end-to-end estimation of energy/carbon impacts of cloud software paves the way to reporting and tracking the environmental emissions of cloud software. This is a critical step towards more sustainable cloud computing by raising awareness of customers on the hidden impact of their online activities.

## ACKNOWLEDGMENTS

This work is supported by the “FrugalCloud” Inria and OVHCLOUD partnership. Additionally, this work also received partial support from the French government through the Agence Nationale de la Recherche (ANR) under the France 2030 program, including partial funding from the CARECLOUD (ANR-23-PECL-0003), DISTILLER (ANR-21-CE25-0022), and GREENACT (ANR-21-HDF1-0006) grants.

The maintenance of the PowerAPI toolkit is currently funded by Inria, Orange Labs, OVHCLOUD and LA POSTE Group.

## REFERENCES

[1] Daniel Balouek, Alexandra Carpen Amarie, Ghislain Charrier, Frédéric Desprez, Emmanuel Jeannot, Emmanuel Jeanvoine, Adrien Lèbre, David Margery, Nicolas Niclausse, Lucas Nussbaum, Olivier Richard, Christian Pérez, Flavien Quesnel, Cyril Rohr, and Luc Sarzyniec. Adding virtualization capabilities to the Grid’5000 testbed. In *Cloud Computing and Services Science*, volume 367 of *Communications in Computer and Information Science*, pages 3–20. Springer International Publishing, 2013. doi:10.1007/978-3-319-04519-1_1.

[2] Luiz André Barroso, Jimmy Clidaras, and Urs Hölzle. The datacenter as a computer: An introduction to the design of warehouse-scale machines. *Synthesis lectures on computer architecture*, 8(3):1–154, 2013.

[3] Luiz André Barroso and Urs Hölzle. The Case for Energy-Proportional Computing. *Computer*, 40(12):33–37, 2007. doi:10.1109/MC.2007.443.

[4] Gemma A. Brady, Nikil Kapur, Jonathan L. Summers, and Harvey M. Thompson. A case study and critical assessment in calculating power usage effectiveness for a data centre. *Energy Conversion and Management*, 76:155–161, 2013. doi:10.1016/j.enconman.2013.07.035.
[5] Maxime Colmant, Mascha Kurpicz, Pascal Felber, Loïc Huertas, Romain Rouvoy, and Anita Sobe. Process-level power estimation in vm-based systems. In Proceedings of the Tenth European Conference on Computer Systems, EuroSys 2015, Bordeaux, France, April 21-24, 2015, pages 14:1-14:14. ACM, 2015. doi:10.1145/2741948.2741971.

[6] Eli Cortez, Anand Bonde, Alexandre Muzio, Mark Russinovich, Marcus Fontoura, and Ricardo Bianchini. Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms. In Proceedings of the 26th Symposium on Operating Systems Principles, SOSP'17, page 153-167. ACM, 2017. doi:10.1145/3132747.3132772.

[7] Miyuru Dayarathna, Yonggang Wen, and Rui Fan. Data center energy consumption modeling: A survey. IEEE Commun. Surv. Tutorials, 18(1):732-794, 2016. doi:10.1109/COMST.2015.2481183.

[8] Spencer Desrochers, Chad Paradis, and Vincent M. Weaver. A validation of DRAM RAPL power measurements. In Proceedings of the Second International Symposium on Memory Systems, pages 455-470. ACM, 2016. doi:10.1145/2989081.2989088.

[9] Guillaume Fieni, Daniel Romero Acero, Pierre Rust, and Romain Rouvoy. PowerAPI: A Python framework for building software-defined power meters. J. Open Source Softw., 9(98):6670, 2024. URL: https://doi.org/10.21105/joss.06670, doi:10.21105/JOSS.06670.

[10] Guillaume Fieni, Romain Rouvoy, and Lionel Seinturier. Smartwatts: Self-calibrating software-defined power meter for containers. In 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing, CCGRID 2020, Melbourne, Australia, May 11-14, 2020, pages 479-488. IEEE, 2020. doi:10.1109/CCGrid49817.2020.00-45.

[11] Guillaume Fieni, Romain Rouvoy, and Lionel Seinturier. SelfWatts: On-the-fly Selection of Performance Events to Optimize Software-defined Power Meters. In 21st IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing, CCGrid 2021, Melbourne, Australia, May 10-13, 2021, pages 324-333. IEEE, 2021. URL: https://doi.org/10.1109/CCGrid51090.2021.00042, doi:10.1109/CCGRID51090.2021.00042.

[12] International Organization for Standardization. Information technology – Data centres – Key performance indicators – Part 2: Power usage effectiveness (PUE). Standard, International Organization for Standardization, Geneva, CH, April 2016. URL: https://www.iso.org/standard/63451.html.

[13] Google. Data centers efficiency. https://www.google.com/about/datacenters/efficiency/, 2021.

[14] Aurelien Havet, Valerio Schiavoni, Pascal Felber, Maxime Colmant, Romain Rouvoy, and Christof Fetzer. GENPACK: A Generational Scheduler for Cloud Data Centers. In 2017 IEEE International Conference on Cloud Engineering, IC2E 2017, Vancouver, BC, Canada, April 4-7, 2017, pages 95-104. IEEE Computer Society, 2017. doi:10.1109/IC2E.2017.15.

[15] Xiaofeng Hou, Chao Li, Jiacheng Liu, Lu Zhang, Yang Hu, and Minyi Guo. Ant-man: towards agile power management in the microservice era. In SC, page 78. IEEE/ACM, 2020.

[16] Xiaofeng Hou, Chao Li, Jiacheng Liu, Lu Zhang, Shaolei Ren, Jingwen Leng, Quan Chen, and Minyi Guo. Alphar: Learning-powered resource management for irregular, dynamic microservice graph. In IPDPS, pages 797-806. IEEE, 2021.

[17] Uptime Institute. Global pures — are they going anywhere? https://journal.uptimeinstitute.com/global-pues-are-they-going-anywhere/, December 2023.

[18] Pierre Jacquet, Thomas Ledoux, and Romain Rouvoy. Cloudfactory: An open toolkit to generate production-like workloads for cloud infrastructures. In IEEE International Conference on Cloud Engineering, IC2E 2023, Boston, MA, USA, September 25-29, 2023, pages 81-91. IEEE, 2023. doi:10.1109/IC2E59103.2023.00017.

[19] Pierre Jacquet, Thomas Ledoux, and Romain Rouvoy. SCROOGEVM: Boosting Cloud Resource Utilization With Dynamic Oversubscription. IEEE Trans. Sustain. Comput., 9(5):754-765, 2024. doi:10.1109/TSUSC.2024.3369333.

[20] Pierre Jacquet, Thomas Ledoux, and Romain Rouvoy. SlackVM: Packing Virtual Machines in Oversubscribed Cloud Infrastructures. In IEEE International Conference on Cluster Computing, CLUSTER 2024, Kobe, Japan, September 24-27, 2024, pages 190-201. IEEE, 2024. doi:10.1109/CLUSTER59578.2024.00024.

[21] Pierre Jacquet, Thomas Ledoux, and Romain Rouvoy. SweetspotVM: Oversubscribing CPU without Sacrificing VM Performance. In 24th IEEE International Symposium on Cluster, Cloud and Internet Computing, CCGrid 2024, Philadelphia, PA, USA, May 6-9, 2024, pages 148-157. IEEE, 2024. URL: https://doi.org/10.1109/CCGrid59990.2024.00026, doi:10.1109/CCGRID59990.2024.00026.

[22] Bhanu Priya, Emmanuel S Pilli, and Ramesh C Joshi. A survey on energy and power consumption models for greener cloud. In 2013 3rd IEEE International Advance Computing Conference (IACC), pages 76-82. IEEE, 2013.

[23] Efraim Rotem, Alon Naveh, Avinash Ananthakrishnan, Eliezer Weissmann, and Doron Rajwan. Power-Management Architecture of the Intel Microarchitecture Code-Named Sandy Bridge. Micro, 2012.

[24] Bernd Schaeppi, Thomas Bogner, Alexander Schloesser, Lutz Stobbe, and Marcos Dias de Asuncao. Metrics for energy efficiency assessment in data centers and server rooms. In 2012 Electronics Goes Green 2012+, pages 1-6. IEEE, 2012. URL: https://ieeexplore.ieee.org/document/6360442.

[25] Erik van der Kouwe, Dennis Andriesse, Herbert Bos, Cristiano Giuffrida, and Gernot Heiser. Benchmarking Crimes: An Emerging Threat in Systems Security. CoRR, abs/1801.02381, 2018.

[26] Torsten Wilde, Axel Auweter, Michael K. Patterson, Hayk Shoukourian, Herbert Huber, Arndt Bode, Detlef Labrenz, and Carlo Cavazzoni. Dwpe, a new data center energy-efficiency metric bridging the gap between infrastructure and workload. In International Conference on High Performance Computing & Simulation, HPCS 2014, Bologna, Italy, 21-25 July, 2014, pages 893-901. IEEE, 2014. URL: https://doi.org/10.1109/HPCSIM.2014.6903784, doi:10.1109/HPCSIM.2014.6903784.

[27] Bo Zhang, Yahya Al-Dhuraibi, Romain Rouvoy, Fawaz Paraiso, and Lionel Seinturier. CloudGC: Recycling Idle Virtual Machines in the Cloud. In 2017 IEEE International Conference on Cloud Engineering, IC2E 2017, Vancouver, BC, Canada, April 4-7, 2017, pages 105-115. IEEE Computer Society, 2017. doi:10.1109/IC2E.2017.26.