# Evaluating Federated Learning Beyond Simulation: A Deployment-Aware Methodology

Mathis Valli, Alexandru Costan, Cédric Tedeschi, Gabriel Antoniu, Loïc Cudennec

► To cite this version:

Mathis Valli, Alexandru Costan, Cédric Tedeschi, Gabriel Antoniu, Loïc Cudennec. Evaluating Federated Learning Beyond Simulation: A Deployment-Aware Methodology. 2026. hal-05477787

HAL Id: hal-05477787

https://hal.science/hal-05477787v1

Preprint submitted on 26 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Evaluating Federated Learning Beyond Simulation: A Deployment-Aware Methodology

Mathis Valli, Alexandru Costan, Cédric Tedeschi, Gabriel Antoniu*, Loïc Cudennec†

*University of Rennes, Inria, CNRS, IRISA [mathis.valli, alexandru costan, cedric.tedeschi]@irisa.fr

†AMIAD loic.cudennec@intradef.gouv.fr

**Abstract**—Federated Learning (FL) is often evaluated in simulation, which overlooks network variability, system heterogeneity, and energy costs in geo-distributed settings. We propose a deployment-aware methodology that triangulates analytical modeling, simulation, and real-world deployments within a unified FL evaluation framework. For a given series of experimental scenarios, the methodology allows to assess the consistency of performance trends across the three evaluation approaches, quantifying deviations in key metrics such as run time, communication overhead, and energy consumption. This further enables cross-validation of the reliability of multiple measurement tools, highlighting discrepancies in commonly reported metrics such as the energy usage.

The methodology is validated on FL workloads by comparing analytical predictions and simulations against large-scale deployments on the Grid'5000 testbed, spanning 51 nodes across four geographically distant sites. By varying key FL components such as aggregation algorithms, client sampling rates, and datasets, we characterize how different FL design choices affect the reliability of the three evaluation approaches. Our findings reveal significant divergences: analytical models accurately capture communication patterns and preserve the relative performance of the scenarios, simulations reflect broad trends but often lead to performance rankings of different configurations inconsistent with those found through actual deployment, while only the latter uncovers hidden costs, such as increased energy consumption due to data imbalances. We conclude with practical guidance on what simulations can and cannot capture, the complementary role of analytical models, and which aspects require deployment-level validation.

**Index Terms**—FL, benchmark, Continuum, ML, data privacy

## I. INTRODUCTION

Federated Learning (FL) is transforming machine learning (ML) by shifting the training paradigm from centralized high-performance computing or cloud-based clusters to distributed execution at the edge. Unlike traditional centralized ML pipelines, FL avoids raw data transmission and instead relies on iterative model updates shared between clients and a central coordinator. It enables collaborative model training across the Computing Continuum [1], while preserving data locality and privacy. As a result, FL has been adopted in diverse domains, ranging from medical imaging [2] and fraud detection [3] to in-situ training of large datasets (e.g., images, videos) [4].

Despite this promise, most evaluations of FL remain grounded in simulations [5], [6] that abstract away deployment constraints and platform characteristics. Such simulations, often executed on a single machine, provide controlled environments for benchmarking algorithm performance, but fail

to capture the operational complexity and variability of real-world deployments [7], [8]. In practice, and particularly across the Computing Continuum, FL clients are geographically distributed, subject to fluctuating bandwidth and latency, and equipped with heterogeneous compute and energy resources. These factors introduce a gap between simulation results and real-world behaviors, hindering evaluations with respect to FL metrics like convergence, resource usage and scalability [9].

This paper addresses this discrepancy by proposing a methodology combining **analytical estimates**, **simulations** and **geo-distributed deployments** into a **unified evaluation workflow**. For a given set of experimental scenarios, the three approaches are systematically compared to determine whether performance trends and relative orderings remain consistent. The methodology quantifies deviations in execution time, communication overhead, and energy consumption across platforms, while also assessing the reliability of monitoring and profiling tools used for FL in heterogeneous environments, particularly with respect to energy measurement accuracy.

This methodology serves both researchers and practitioners. It provides guidance on when simulation is adequate to capture algorithmic behavior, when deployment-specific effects invalidate conclusions from simulation, and under which conditions analytical models can yield fast yet accurate approximations.

Several challenges arise: ensuring *reproducibility* across heterogeneous infrastructures, guaranteeing the reliability of *energy and network monitoring* in geo-distributed environments, and establishing the *comparability* of results across analytical, simulated, and deployed settings. This work takes a first step towards addressing these challenges by introducing a unified experimentation methodology, integrated with a well-established experimental framework (e.g., E2Clab [10]) that ensures reproducibility and performance optimization. The methodology can be used across multiple existing testbeds (e.g., Grid'5000, Chameleon Cloud, CHI@Edge, FIT IoT [11]–[13]), and is extensible to new testbeds. It provides transparent access to both experimental artifacts and results, supporting explainability, verification, and reproducibility to enable a holistic understanding of performance of FL on the Continuum. Our contributions are as follows:

1. A unified methodology for quantifying the gap between simulations and real-world deployments in FL.

2. A systematic assessment of the predictive power of analytical models for FL run time, network, and energy.
3) An extensive experimental evaluation including: simula-
tions using the Flower [14] framework, geo-distributed
deployments on the Grid'5000 testbed with 51 nodes
across four sites. By varying key FL parameters such
as aggregation algorithms and client sampling rates, we
demonstrate how these design choices differently affect
analytical estimates, simulations, and deployed systems.

4) A characterization of FL monitoring challenges, includ-
ing measurement inaccuracies and tool inconsistencies.

5) Evidence-based recommendations for reproducible FL experimentation, clarifying the capabilities and limitations of simulation in capturing real-world behavior.

The remainder of the paper is organized as follows. Sec-
tion II presents the principles of FL and reviews the limitations
of existing evaluation approaches. Section III introduces the
proposed unified evaluation methodology, while Section IV
presents a representative set of FL experiments applying the
methodology. Section V compares the outcomes across ana-
lytical models, simulation, and deployments, while Section VI
discusses the practical implications for the FL community.

II. BACKGROUND AND MOTIVATION

A. *Federated Learning workflows*

In FL, participating devices collaboratively update a shared
model by exchanging only model updates, instead of trans-
mitting data to a central server. This approach enables the
learning of high-quality models across potentially millions of
clients [15], while preserving user privacy and reducing data
transfer overhead. In practice, due to the vast scale and hetero-
geneity of participating clients, each round of training typically
involves only a subset of clients (often in the hundreds) [16].

Recent large-scale FL deployments, such as Google's Gboard [15] and Apple's FL-based systems [17], highlight the complexities of deploying FL in real-world environments [9]. These production use cases have to deal with partial client participation, congested cellular networks, and frequent disconnections. Moreover, real-world FL deployments face diverse (possibly conflicting) challenges such as protection against external conditions, access to reliable power sources, network connectivity, and deployment authorization processes [8]. These operational challenges differ radically from the controlled conditions typically assumed in simulations.

B. FL evaluation approaches

Most FL research relies on approaches that differ in their
ability to capture deployment constraints. We identify three
complementary approaches: analytical estimates, simulations
and geo-distributed deployments.

1) Analytical estimates: Analytical models provide a means to estimate systems costs such as communication, run time or energy consumption when actual deployment is not possible. Communication is typically expressed as a function of model size, client participation rate, and the number of training rounds [18]. Run time is often approximated by the latency of the slowest participating client combined with network delays

[19], while energy consumption can be inferred from workload
characteristics and device-specific power profiles [20].

In practice, however, these formulas rely on parameters that
are difficult to obtain and require careful calibration for a
specific infrastructure. As a result, such models are not widely
adopted in the FL community, although simplified versions can
still offer valuable complementary insights to simulations. Our
work examines how these models compare to simulations and
geo-distributed deployments, and under which conditions they
can provide reliable information.

2) *Simulations:* By simulation, we refer to single-machine or small-cluster experiments that execute many clients. Most studies rely heavily on controlled simulations [5], [9], and a recent review [21] found that up to 60% of FL studies are evaluated exclusively via simulation; about 20% remain ambiguous in their evaluation setting, and roughly 20% are deployed in distributed scenarios. Simulation allows rapid prototyping and testing with a large number of virtual clients on a single machine. While effective for algorithmic exploration, such setups assume idealized conditions, including stable connectivity, synchronous and homogeneous clients. A few studies expand simulation with emulation, in which client behavior is based on real traces. Frameworks such as FedScale [22] and LEAF [23] support this approach. Although emulation provides more realism than pure simulation, these frameworks remain limited: they primarily address statistical heterogeneity and trace-based variability, but do not fully capture system heterogeneity or the complex network dynamics that affect real deployments. Moreover, trace logs are not always publicly accessible. For these reasons, emulation remains less common and is not the focus of this work.

3) *Geo-distributed deployments:* Deployments on real infrastructures provide the most accurate FL behavior. They capture network variability, hardware heterogeneity and energy constraints. However, in practice, these deployments are rare and typically limited in scale. For example, [24] uses 3 Raspberry Pi and 1 Jetson Nano in a small physical testbed. [25] deploys Raspberry Pi 3, Raspberry Pi 4, Jetson Nano and Jetson TX2, testing on up to 64 edge devices, but all nodes are co-located in the same site rather than geo-distributed across distinct locations. The study in [8] deploys a real-world sensor network for air quality forecasting to discuss the practical challenges of real-world FL setups. Such deployments are harder to orchestrate and reproduce, but they reveal costs and effects that simulations or emulations cannot capture.

C. Monitoring FL on geo-distributed infrastructures

Evaluating FL on real testbeds requires reliable monitoring
of energy consumption, communication and run time. This
is challenging because existing tools vary widely in granu-
larity, precision, and measurement methodology. In practice,
researchers have used various techniques to quantify energy
consumption including software profilers and physical power
measurement instruments such as wattmeters [26], each with
their inherent limitations. For network monitoring, tools such
as psutil and API-level counters such as Kwollect [27]
<figure><img src="image_2.png" /><figcaption>Fig. 1: Federated Learning Evaluation Methodology.</figcaption></figure>

may provide divergent results due to differences in sampling frequency and measurement granularity.

Monitoring in FL is particularly challenging due to irregular workloads and heterogeneous devices. Computation, communication and waiting phases alternate in short bursts, causing coarse-grained measurement tools to smooth over or entirely miss transient variations. When workloads are light, due to small models or limited data, consumption may fall below the sensitivity threshold of certain sensors. Hardware heterogeneity further complicates measurements, as the accuracy of a given tool can vary across platforms (e.g., RAPL accuracy depends on the CPU model). Drawing reliable conclusions about system efficiency requires fine-grained measurements of computation, communication and energy consumption, otherwise certain trade-offs may be misinterpreted [28]. These factors make monitoring a major source of uncertainty in FL experiments and a significant obstacle to reproducibility.

### D. Insights from other communities

The challenge of accurate evaluation is not unique to FL. Studies in IoT and wireless sensor networks have demonstrated significant divergences between simulation and physical deployments, caused by factors such as radio interference, hardware variability, and unpredictable environmental conditions. Best practices in these domains recommend a staged evaluation approach: initial simulation, calibration with empirical traces, and final validation on physical testbeds [29]–[32]. These lessons highlight the importance of an evaluation grounded in real geo-distributed deployments and motivate the adoption of similar methodologies in the context of FL.

### E. Problem statement

While FL has been extensively studied, its validation mostly relies on simulations, which simplify experimentation but fail to capture deployment realities. As discussed in Section II-B, a few works report experiments on small testbeds [33], [34]. However, to the best of our knowledge, only [8] directly compares simulation with deployment, and no study includes analytical estimates. Reproducibility remains a challenge due

to the inherent randomness and client heterogeneity of FL as well as inconsistent monitoring techniques. These gaps advocate for a unified methodology, systematically integrating analytics, simulation, and distributed deployment to provide a comprehensive and reproducible evaluation framework.

This work addresses the following research questions:

1. To what extent do analytical estimates and simulation results diverge from real deployments in terms of run time, communication overhead and energy consumption?

2. Do simulations preserve global performance trends and relative performance of different configurations across different FL parameters, or do deployment realities invalidate their predictions?

3. Which FL components contribute most to the gap between analytical estimates, simulations and real deployments? How do monitoring practices and reproducibility challenges such as randomness and client heterogeneity impact the reliability of results across these three evaluation approaches?

## III. METHODOLOGY OVERVIEW

We here propose a generic methodology and then apply it to typical FL scenarios. This methodology, illustrated in Figure 1, defines a workflow for evaluating FL beyond simulation by combining three complementary perspectives, namely analytical estimates, simulations, and geo-distributed deployments, and enabling their comparison within a common protocol.

Analytical estimates provide fast, low-cost approximations for communication bounds, run time, and energy consumption. Simulations enable large-scale testing of algorithmic ideas and, despite simplifying system behavior, are often able to capture global performance trends and preserve relative performance of different configurations. Finally, geo-distributed deployments expose the impact of network variability, hardware heterogeneity, and real-world constraints that cannot get captured by simplified approaches.

The methodology consists of several stages. At each stage, run time, communication and energy consumption are measured with standard monitoring tools, cross-checking multiple sources when available to improve reliability. Experiments are orchestrated with E2Clab [10], a framework that automates reservation, configuration, execution, and log collection and enables reproducible runs.

### A. Scenario definition

As a first step, a baseline task (defined by a dataset, a model architecture, and some parameters) is defined. This task will be used to estimate the run-to-run variance of key metrics in both simulation and real deployments. Based on this baseline, an experiment plan builds multiple scenario instances by varying different factors (e.g., sampling rate, dataset).

### B. Analytical estimates

Once the experimental scenarios are defined, the second step consists in applying analytical models to approximate the system costs. The models presented below target centralized,
synchronous FL, the most common setting, which reduces the number of parameters requiring calibration. Extending these models to decentralized or asynchronous FL would require additional considerations such as staleness, overlap, and peer-to-peer exchanges, yet the overall methodology remains applicable. Note that these models rely on infrastructure parameters such as latency, device throughput, and power profiles, which may be difficult to obtain. Yet, simplified formulations still provide useful insights. Below, we summarize the models used for communication, run time, and energy.

a) **Communication:** Communication is the most established cost metric in FL, as it directly depends on model size, sampling rate and number of rounds, while remaining largely independent of hardware or scheduling details. As a result, most prior work adopts simple formulas to estimate traffic during training [35]–[37]. Following this common practice, we model communication as:

$$Comm_{th} = 2 S_{model} \phi N_{clients} R (1 + \delta_{ser}) \quad (1)$$

where $S_{model}$ is the model size, $\phi$ the client sampling rate, $N_{clients}$ the number of clients, $R$ the number of rounds, and $\delta_{ser}$ a small serialization overhead (generally ~2%).

b) **Run time:** Run time has received little attention in FL, as most studies focus on accuracy versus number of rounds rather than execution cost [38]. Yet run time is crucial for deployment feasibility. When modeled, the duration of a training round is generally expressed as the sum of client computation and communication. In synchronous FL, the duration is determined by the slowest participating client [39], [40]. Computation time can be approximated by converting FLOPs into execution time using sustained device throughput, consistent with the Roofline model [41]. The cost of model aggregation on the server is typically negligible for standard model sizes [9], [38]. Under these assumptions, the run time can be defined as:

$$T_{round} = \mathbb{E} \left[ \max_{i} (t_i + RTT_{g(i)}) \right] + t_{serv}. \quad (2)$$

Here, $g$ indexes client types (“sites”), i.e., homogeneous groups by hardware and network, and $g(i)$ maps client $i$ to its site; $i$ refers to clients selected in the current round. With $t_i = E a_{g(i)} D_i$, $a_g = F_{train}/(\rho_g R_g^{peak})$, and $t_{serv} = \frac{n_{eval} F_{eval}}{\rho_{serv} R_{serv}^{peak}}$. The term $RTT_{g(i)}$ denotes the average round-trip communication latency for site $g(i)$.

c) **Energy consumption:** Energy is critical for cost-aware FL deployments, as it accounts not only for computation and communication but also for idle periods during which devices wait for each other. A few prior studies have proposed models or performed direct measurements of FL energy consumption [42], [43]. Following these works, we model energy consumption as the sum of client training, server evaluation, communication, and idle phases:

$$E_{round} = E_{ci} + E_{ct} + E_{si} + E_{st} + E_{comm} \quad (3)$$

<table><caption>TABLE I: Run time model symbols.</caption><thead><tr><th>Symbol</th><th>Definition</th></tr></thead><tbody><tr><td>E</td><td>Local epochs</td></tr><tr><td>D<sub>i</sub></td><td>Samples of client <i>i</i></td></tr><tr><td>a<sub>g</sub></td><td>Per-sample compute time at site <i>g</i></td></tr><tr><td>F<sub>train</sub></td><td>FLOPs per training sample</td></tr><tr><td>ρ<sub>g</sub></td><td>Sustained utilization of site <i>g</i></td></tr><tr><td>R<sup>peak</sup><sub>g</sub></td><td>Peak compute rate at site <i>g</i></td></tr><tr><td>t<sub>serv</sub></td><td>Server evaluation duration</td></tr><tr><td>n<sub>eval</sub></td><td>Evaluation samples</td></tr><tr><td>F<sub>eval</sub></td><td>FLOPs per evaluation sample</td></tr><tr><td>R<sup>peak</sup><sub>serv</sub></td><td>Server peak compute rate</td></tr><tr><td>ρ<sub>srv</sub></td><td>Sustained server utilization</td></tr><tr><td>RTT<sub>g</sub></td><td>Round-trip latency between site <i>g</i> and the server</td></tr></tbody></table>

<table><caption>TABLE II: Definitions of energy model terms.</caption><thead><tr><th>Term</th><th>Definition</th></tr></thead><tbody><tr><td>E<sub>ci</sub></td><td>Σ<sub>g</sub> K<sub>g</sub> P<sup>idle</sup><sub>g</sub> T<sub>round</sub> (client idle)</td></tr><tr><td>E<sub>ct</sub></td><td>E[Σ<sub>i</sub> E F<sub>train</sub> D<sub>i</sub> e<sup>train</sup><sub>g(i)</sub>] (client training)</td></tr><tr><td>E<sub>si</sub></td><td>P<sup>idle</sup><sub>srv</sub> T<sub>round</sub> (server idle)</td></tr><tr><td>E<sub>st</sub></td><td>t<sub>serv</sub> (P<sup>act</sup><sub>srv</sub> - P<sup>idle</sup><sub>srv</sub>) (server evaluation)</td></tr><tr><td>E<sub>comm</sub></td><td>Comm<sub>th</sub> ε<sub>bit</sub> (communication)</td></tr></tbody></table>

Here, following Table II, $\epsilon_{g}^{train} = (P_{g}^{act} - P_{g}^{idle})/(\rho_{g} R_{g}^{peak})$ is the energy per training FLOP at site $g$, $\epsilon_{bit}$ the energy per transmitted bit, $K_g$ the number of clients at site $g$, $P_{srv}^{idle}$ the idle server power, $P_{srv}^{act}$ the active server power.

### C. Simulations

The third step involves running simulations. Several FL clients run on a single node using an FL framework such as Flower [14]. Each client is launched as a process on the same node as the server. Since all clients share the same resources, this type of configuration reproduces the training logic but it does not capture the parallelism or network variability of a true distributed deployment. Consequently, the duration of a round can be as long as $\sum_{k=1}^{K} t_k$ instead of $\max_k t_k$ in a real deployment (where $t_k$ is the compute time of client $k$).

In this setup, the following metrics are measured: number and size of messages sent to and from logical clients, run time, energy consumption from node telemetry (BMC, PDUs), CPU RAPL counters, and GPU power sensors.

The logged communications should match exactly the values predicted by Equation (1). Simulated run time and energy consumption are not expected to match deployment or analytical estimates. We still include them in our methodology as simulations are expected to preserve overall trends and relative performance of configurations. They are also the most common approach used in FL research to validate and compare configurations.

### D. Geo-distributed deployments

The fourth step consists of running experiments on real, geo-distributed infrastructure. Deployments are orchestrated via E2Clab [44], an edge-to-cloud framework that automates
resource reservation, environment configuration, execution, and log collection while preserving the training logic. This ensures that experiments can be repeated with consistent configurations, supporting reproducibility. Experiments are monitored similarly to simulations, with additional measurements provided by the underlying testbed.

The following metrics are recorded in a similar way to the previous step: run time, energy consumption from node telemetry (BMC, PDUs), RAPL counters and GPU power sensors, and network metrics (RTT/bandwidth) via passive counters or occasional active probes.

Run-to-run variance is measured on the baseline task for both simulation and deployment, providing a reference for interpreting all subsequent comparisons.

### E. Gap analysis

The final step compares the results of the three evaluation approaches (analytical estimates, simulation and real deployment) to assess the reliability of the proxies (analytical and simulation) compared to the real deployment. Three aspects are considered when relevant:

* **Approximation error:** The deviation of analytical estimates and simulations from deployment, reported as mean absolute percentage error (MAPE).

* **Trend similarity:** Whether metrics increase, decrease, or plateau consistently as configurations vary. This is quantified with Spearman's $ρ$, which captures whether the series move together in a monotone way. $ρ = 1$ means the two series follow the exact same monotone trend and $ρ = 0$ means there is no monotone association.

* **Ranking fidelity:** We assess whether configurations preserve the same order as in deployment. This is quantified using Kendall's $τ$, which counts the fraction of configuration pairs that preserve their order between two series. $τ = 1$ means all pairs are ranked identically and $τ = 0$ means the ranking is random.

When multiple measurements exist for the same metric (e.g., wall power vs. device counters), we assess the reliability of each source and report the most consistent. A metric is considered reliable if it remains within predefined error bounds, follows the same trends, and preserve rankings. Otherwise, deviations are signaled and possible causes (e.g., model assumptions, simulator limits, network effects) are discussed.

Beyond evaluation, this gap analysis can help refine analytical models and interpret simulations by linking their deviations from deployment to specific system behaviors.

## IV. ILLUSTRATION

To demonstrate the applicability of the proposed methodology, we instantiate it on a representative set of FL experiments. By applying the workflow to these configurations, we aim to address the research questions outlined in Section II-E.

### A. Scenario definition

We design a baseline task that is sufficiently long and difficult to stress the system, while remaining reasonable in

terms of resources used so that the actual deployment is feasible within an acceptable timeframe. The baseline task is centralized, synchronous FL and involves 50 clients. The dataset used is CIFAR-10 [45], partitioned with a Dirichlet distribution (with $α = 0.1$) to emulate a non-IID setting [46]. Training is performed with a ResNet-18 [47] model with partial fine-tuning [48]. Model evaluation is done on the server using a fixed test set to ensure consistency and reproducibility of metrics. The other hyperparameters follow standard practice, with a learning rate set to 0.01 and a batch size of 64.

Building on this baseline, we vary a set of commonly studied components and parameters that are expected to significantly influence performance results in geo-distributed settings:

* **Aggregation algorithms:** FedAvg [49] (simple average of client updates), FedProx [6] (proximal term to stabilize under heterogeneity), and FedYogi [5] (adaptive server optimizer using Yogi);

* **Client selection methods:** random sampling vs. adaptive selection, as advocated by the Oort approach [16], prioritizing high label-entropy clients with lower estimated energy cost;

* **Client sampling rate ($\phi$) and Local epochs ($E$);

* **Data cap:** capped at 512 samples per epoch per client vs. uncapped;

* **Model architectures:** ResNet-18, ResNet-50, MobileNetV3 [47], [50], EfficientNet-B0 [51], and CLIP (ViT-B/32) [52];

* **Datasets:** CIFAR-10 (standard 10-class images, synthetic non-IID split) and FLAIR (real non-IID dataset) [45], [53].

These parameters span the main dimensions that affect the cost and behavior of FL systems. They encompass **algorithmic choices** (aggregation method, client selection), **system-level parameters** (client sampling rate, number of local epochs, data capping), and **workload diversity** (different models and datasets). By varying these factors, we obtain a broad spectrum of representative FL configurations, allowing us to assess the sensitivity of analytical estimates and simulations to changes in algorithms, data heterogeneity, and computational intensity. This provides a representative benchmark space for analyzing the gap between simulations and real deployments.

### B. Analytical estimates

We instantiate the models presented in Section III-B in our deployment context (detailed later in Section IV-D) consisting of 50 clients and a central server. The server is located in Lille, while the clients are distributed across three sites with distinct hardware characteristics and round-trip times (RTTs) to Lille: Nantes (10 clients, 10.6 ms), Rennes (10 clients, 9.1 ms), and Nancy (30 clients, 8.3 ms). Since the exact mapping of dataset sizes to individual clients is unknown, we consider the allocation as being random in our estimates, while ensuring that the site-level client counts are preserved.
1) *Communication:* This is the simplest formula to apply, as it requires no parameter calibration. In our study, $N_{clients}$ and $R$ are fixed, so only $\phi$ and $S_{model}$ vary. For the baseline task, this yields a communication cost of $Comm_{th} = 83.7216$ MB.

2) *Run time:* Round duration is approximated as the sum of slowest client's computation, server evaluation, and network latency. We estimated the slowest client time using Monte-Carlo sampling (20,000 draws of $m = \phi K$ clients, with site assignment consistent with the 10/10/30 split). Due to client heterogeneity, the slowest selected client is not necessarily the one with the largest dataset. Network latency is set to that of the slowest site included in each cohort. For the baseline configuration, the model predicts $T_{total} \approx 25$ min 19 s.

3) *Energy:* Energy consumption is approximated as the sum of client computation, server evaluation, communication, and idle terms (i.e., energy spent while the server waits for client updates or while clients wait for server signals). FLOPs are converted to joules using site-specific efficiency factors $\epsilon_g$, derived from average power values: Nantes (110 W), Rennes (170 W), Nancy (125 W). Communication energy is estimated using a J/bit model ($\epsilon_{bit} = 2P_{NIC}/R_{link}$). Idle energy is computed as the product of waiting times and idle power (70 W, 80 W, and 60 W for Nantes, Rennes, and Nancy, respectively). For the baseline task, the estimated total energy is $E_{total} = 1.605$ kWh.

### C. Simulations

We use the Flower [14] simulation engine to run multiple logical clients on a single machine. Flower relies on Ray [54] for client scheduling: we specify the GPU fraction allocated per client, and the engine executes as many clients in parallel as resources allow while queuing the rest. In our setup, each client is assigned 10% of a GPU, with the server process colocated on the same host.

Simulations are executed on a node equipped with an Intel Xeon Gold 6126 CPU and an NVIDIA Tesla P100 GPU. We record total run time and model performance using the same evaluation protocol that is later applied to the deployment. Energy consumption is measured through CPU RAPL counters, GPU power readings from nvidia-smi, and node-level measurements from the BMC interface.

### D. Geo-distributed deployment

Experiments were deployed on the Grid'5000 testbed [11], a geo-distributed infrastructure spanning multiple sites in France. In our case, the central server runs at Lille on the chifflot cluster [55] on a dual Intel Xeon Gold 6126 with one NVIDIA Tesla P100, 192 GiB RAM, and a 25 Gbps Ethernet link, while clients were distributed on dedicated nodes across Rennes (10 nodes on the parasilo cluster [56], 2 Intel Xeon E5-2630 v3), Nancy (30 nodes on the gros cluster [57], Intel Xeon Gold 5220), and Nantes (10 nodes on the ecotype cluster [58], 2 Intel Xeon E5-2630L v4). The Grid'5000 backbone network connects its sites through a dedicated Layer 2 Ethernet infrastructure [59]. As a reference, the round trip time between Lille and Rennes is about 10 ms. Each client uses

its own dedicated node, to isolate resource and performance monitoring.

Deployments are orchestrated with E2Clab [44], which in our case reserves nodes through the Grid'5000 API, installs dependencies and uploads experiment artifacts, starts the FL server and clients, awaits for the experiment's completion, and collects logs and artifacts from a single experiment specification. E2Clab defines the experiment process through configuration, seeds, and dataset partitions, ensuring that simulations and deployments are both comparable and shareable.

We collect energy, network, and run time metrics with local sensors and the Grid'5000 monitoring stack [27]. Model performance uses the same evaluation protocol as in simulation. Energy is monitored via node-level sensors (BMC/PDUs), RAPL and nvidia-smi. Network traffic is measured using psutil and Grid'5000 counters.

## V. RESULTS AND GAP ANALYSIS

In this section, we first validate our monitoring stack for network, run time, and energy, to ensure measurement fidelity and then, we quantify baseline variance. Finally, we analyze how FL components (i.e., aggregation, client selection, sampling rate, local epochs, data capping, and models) influence performance in order to compare the ability of analytical estimates, simulations, and deployments to capture FL behavior. Approximation error (MAPE), trend correlation (Spearman $\rho$), and ranking fidelity (Kendall $\tau$) are used for this objective.

### A. Validation of the FL monitoring stack

1) *Network usage:* In simulation, all messages sent and received using Flower's internal logging mechanism are recorded, and the counts matched exactly the prediction of the analytical estimates (Eq. 1), thus validating the formula. For deployment, we use both psutil, which records one value per communication round, and the Grid'5000 API, which samples every 10 seconds. Figure 2 compares traces from rounds 50-90 of the baseline scenario with the theoretical estimate. The API counter sometimes aggregates several rounds, which causes spikes. Measured traffic in real deployment exceeds theoretical estimates due to protocol overhead.

2) *Run time logging:* In addition to logging the total and per-round run time of the FL process, we use cProfile to record the time spent in each function. On the server, we distinguish four categories: client updates aggregation, data handling (dataset preparation), model evaluation, and idle time (waiting for clients to finish training). On the client, we track data handling, model training, and idle time. This allows us to compare the time distribution predicted by analytical estimates with that observed in simulation and in real deployment.

3) *Energy monitoring:* We record energy in simulation to observe qualitative trends, but these measurements are not directly comparable to analytical or deployment values. Because all simulated clients share the same compute node and power sensors capture only aggregate consumption, simulated energy reflects relative variation rather than an actual per-node
<figure><img src="image_3.png" /><figcaption>Fig. 2: Server network traffic (sent + received) during rounds 50–90 of the baseline scenario. Measured with psutil (per round), Grid’5000 API (10 s interval) and compared to theoretical payload values. Really measured traffic exceeds theoretical estimates due to protocol and keep-alive overheads.</figcaption></figure>

energy use. In deployment, Grid’5000 nodes provide three
monitoring sources:

*   **Board Management Controller (BMC):** always available, but coarse in granularity and masking short-term fluctuations.

* External wattmeters and PDUs: accurate to within about 2% [60] and considered ground truth, but only available on a subset of nodes.

*   Running Average Power Limit (RAPL): on-chip sensor whose accuracy depends strongly on CPU model and workload [61].

In our measurements, AMD CPUs (EPYC 7301) produced implausible RAPL readings (500–800 W) compared to BMC values of 200–250 W, making RAPL unusable on this platform. In contrast, CPUs such as the Intel Xeon Gold 5220 in Figure 3 produced RAPL measurements closely aligned with PDU and BMC, so we restricted deployment to nodes with reliable RAPL data. While PDUs and wattmeters are only available on a subset of nodes, BMC is available system-wide and provides the only comprehensive view. We also observe that BMC tends to smooth small fluctuations, producing staircase power curves, whereas PDUs and wattmeters capture these variations more accurately. For consistency across nodes and experiments, all energy results reported in the remainder of this paper are derived from BMC measurements.

B. Run-to-run variance

We quantify variance by running our baseline configuration five times via simulation and deployment. The relative standard deviations are: run time 0.3% vs. 1.5%, energy (BMC) 1.5% vs. 1.0%, and communication 0% (no real transfer) vs. 0.11% (psutil) for simulation and deployment, respectively.

Simulations exhibit more stable run times, while geo-
distributed deployments experience additional jitter from net-
work variability [62]. RAPL and NVIDIA-SMI readings are
consistent across environments, and BMC-based energy is
slightly more stable in deployment due to aggregation over
multiple nodes.

<figure><img src="image_4.png" /><figcaption>Fig. 3: Power readings (RAPL CPU, BMC node, PDU outlet) vs. stressed threads on an Intel Xeon Gold 5220 node.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 4: Run time profiling by stages in simulation (left) and in deployment (right). The client bar represents the average times across all participating clients. The client selection and model aggregation phases appear with a visible size for clarity, although their actual durations appear negligible compared to the other categories.</figcaption></figure>

C. Aggregation algorithms and client selection

Figure 4 shows the run time breakdown of the baseline scenario. The dominant steps are client-side model training and server-side evaluation. The analytical model predicts a total run time of 1519 s, of which 933 s corresponds to server waiting during client training. This distribution is similar to both simulation and deployment in this scenario. The aggregation itself takes negligible time (rounded down to 0%).

Model evaluation accounts for a substantial fraction of
run time: 32% in deployment and 39% in simulation. This
creates a bottleneck because the next round begins only after
evaluation completes. While acceptable in simulation, real
deployments could benefit from decoupling evaluation from
training: clients can start the next round while the server is
still evaluating the previous model, since evaluation results
are rarely required to start the next round. Evaluation could
also occur less frequently, e.g., every few rounds or only on
significant model updates. These adaptations reduce overhead
without compromising monitoring accuracy.
<table><caption>TABLE III: Run time and node-level energy across FL algorithms, on CIFAR-10 and FLAIR.</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Algo</th><th colspan="3">Run time (s)</th><th colspan="4">Energy (kWh)</th></tr><tr><th>AE</th><th>Sim</th><th>Deploy</th><th>AE</th><th>Sim</th><th>Deploy</th><th>Perf.</th></tr></thead><tbody><tr><td>CIFAR-10</td><td>Avg</td><td>1519</td><td>1350</td><td>1490</td><td>1.605</td><td>0.169</td><td>1.624</td><td>0.67</td></tr><tr><td>CIFAR-10</td><td>Prox</td><td>1519</td><td>1355</td><td>1429</td><td>1.605</td><td>0.170</td><td>1.587</td><td>0.73</td></tr><tr><td>CIFAR-10</td><td>Yogi</td><td>1519</td><td>1350</td><td>1464</td><td>1.605</td><td>0.168</td><td>1.615</td><td>0.67</td></tr><tr><td>FLAIR</td><td>Avg</td><td>232</td><td>2084</td><td>454</td><td>0.234</td><td>0.220</td><td>0.327</td><td>0.56</td></tr><tr><td>FLAIR</td><td>Prox</td><td>232</td><td>2113</td><td>439</td><td>0.234</td><td>0.221</td><td>0.333</td><td>0.57</td></tr><tr><td>FLAIR</td><td>Yogi</td><td>232</td><td>2082</td><td>374</td><td>0.234</td><td>0.220</td><td>0.311</td><td>0.584</td></tr></tbody></table>

<table><caption>TABLE IV: Relative change (%) with improved client selection vs. the same strategy without it, under geo-distributed deployment conditions. Negative run time/energy is better; positive accuracy/μF1 is better.</caption><thead><tr><th>Dataset</th><th>Algo</th><th>run time Δ%</th><th>energy Δ%</th><th>performance Δ%</th></tr></thead><tbody><tr><td>CIFAR-10</td><td>FedAvg</td><td>-6.7</td><td>-9.4</td><td>-0.9</td></tr><tr><td>CIFAR-10</td><td>FedProx</td><td>2.0</td><td>-2.5</td><td>-2.2</td></tr><tr><td>CIFAR-10</td><td>FedYogi</td><td>-5.7</td><td>-8.0</td><td>3.3</td></tr><tr><td>FLAIR</td><td>FedAvg</td><td>-45.1</td><td>-30.6</td><td>4.8</td></tr><tr><td>FLAIR</td><td>FedProx</td><td>-44.4</td><td>-30.6</td><td>-0.3</td></tr><tr><td>FLAIR</td><td>FedYogi</td><td>-46.1</td><td>-32.5</td><td>3.9</td></tr></tbody></table>

**Takeaway:** Aggregation costs are negligible.

Aggregation contributes negligibly to run time in both simulation and deployment, whereas model evaluation accounts for roughly one-third of total run time.

Table III compares analytical estimates (AE), simulation, and deployment across the three aggregation algorithms. As expected, run time and energy values are similar as aggregation contributes negligibly to round costs, leading to overlapping curves. Traditional correlation metrics (Spearman's $\rho$ and Kendall's $\tau$) are uninformative in this scenario<fnref n="1" />; what matters is the gap between AE, simulation and deployment. For CIFAR-10, AE is highly accurate (MAPE ≈6% for run time, ≈1% for energy), and simulation closely follows (≈8% for run time, ≈2% for energy). For FLAIR, however, AE significantly underestimates run time (by approximately 50%) and energy (by approximately 30%), and simulation diverges even further, by 370% for the run time, and approximately 40% for energy.

This discrepancy stems from the analytical model's assumption that round time is dominated by training, evaluation, and communication. This assumption holds for CIFAR-10, where clients train for about 4 s and evaluate for about 2.5 s per round, making orchestration overheads negligible. In contrast, most FLAIR clients process only a few samples leading to only 0.3 s training, and 0.5 s evaluation. In this lightweight scenario, fixed costs from scheduling, setup and communication dominate and are not captured by the analytical model.

As expected, simulation run times and energy consumption do not match deployment values due to hardware differences, leading to the most striking effect of inversion between

<figure><img src="image_6.png" /><figcaption>Fig. 5: Communication cost vs. sampling rate $\phi$. Measured on-wire traffic (psutil, Grid5000 API) vs. simulation vs. theoretical payload-only communication. Percentages show (measured – theoretical)/theoretical.</figcaption></figure>

datasets: in deployment, FLAIR is much faster than CIFAR-10 (≈2 s vs. ≈8 s per round), while in simulation, FLAIR appears slower (≈10 s vs. ≈7 s). This occurs since Flower multiplexes clients on a single GPU: small per-client workloads like FLAIR are disproportionately affected by GPU context switching and dataloader overhead. In deployment, clients train independently on CPUs with few samples, completing almost instantly, while CIFAR-10 remains slower due to larger per-client workloads.

**Takeaway:** Workload profiles impact the approaches gap.

Analytical estimates and simulations accurately predict CIFAR-10 behavior but not for FLAIR, where overheads dominate. This leads to an inversion (FLAIR appears slower in simulation but faster in deployment) highlighting the risk of generalizing conclusions from simulation alone.

**Effect of improved client selection:** Table IV compares adaptive and random client selection under geo-distributed deployment. For CIFAR-10, improved client selection slightly reduces run time and energy consumption without affecting accuracy. For FLAIR, it substantially lowers both metrics while maintaining or even slightly improving model performance.

### D. Impact of client sampling rate and local epochs

Figure 5 shows that as the client sampling rate $\phi$ increases from 0.1 to 1.0, measured communication consistently exceeds the theoretical (payload only) estimate. Overhead is highest at low participation (44% at $\phi$ = 0.1), where per-message and background communication costs dominate when few updates are exchanged. As $\phi$ increases, these fixed costs become less significant, reducing the gap to 27%.

Figure 6 extends the analysis to energy consumption. For CIFAR-10, AE remain close to deployment (MAPE of approximately 18%), preserving both ranking ($\tau$ = 0.93) and trend ($\rho$ = 0.98). Simulations capture global trends ($\rho$ = 0.94) but introduce more rank inversions ($\tau$ = 0.83). For FLAIR, AE deviate more (MAPE ≈35%) but still preserve ranking ($\tau$ = 0.91), while simulations show poor fidelity with frequent

<footnote n="1">¹Spearman's $\rho$ = 1 and Kendall's $\tau$ = 1 for AE vs. deployment, but negative for simulation vs. deployment due to inversions.</footnote>
<figure><img src="image_7.png" /><figcaption>Fig. 6: Energy consumption over local epochs *E* and client participation rate *φ* for CIFAR-10 (top) and FLAIR (bottom), across AE, simulation, and deployment.</figcaption></figure>

inversions ($\tau$ = 0.61, $\rho$ = 0.78). Notably, deployment shows that Configuration ($E$=2, $\phi$=0.1) consumes more energy than Configuration ($E$=1, $\phi$=0.2) (0.681 vs. 0.580), but simulation reverses this ordering (0.181 vs. 0.220).

The gap is larger for FLAIR for the same reasons discussed in Section V-C: small client workloads finish almost instantly in deployment but are penalized by scheduling in simulation, which can reverse the ordering of configurations. With heavier workloads like CIFAR-10, simulation and deployment results remain more consistent.

Model performance is similar between simulation and deployment because the training logic is identical.

### *Takeaway: Analytical estimates complement simulations.*

When varying local epochs and participation rates, analytical estimates preserve deployment rankings and provide reliable cost estimates, while simulations may misrank configurations but remain useful for model performance. Together, they enable approximate deployment outcomes without requiring an actual deployment.

## E. Data capping effect

Table V summarizes the impact of the 512-sample cap compared to full-dataset training. For CIFAR-10, removing the cap nearly doubles costs, and analytical estimates capture this trend with ≈6% error. For FLAIR, the divergence is more pronounced: deployment costs increase sharply (run time 454 to 1704s, energy 0.327 to 1.775 kWh), while simulations show only a limited increase (run time 2084 to 2601s). Analytical estimates follow the trend but with larger deviation (MAPE ≈40%).

During deployment, removing the cap amplifies client heterogeneity: clients with larger datasets take longer, forcing

<table><caption>TABLE V: Effect of capping (512 samples/client) vs. uncapping on run time, energy, performance. Data processed is given in images, run time in seconds, energy in kWh. Performance is accuracy for CIFAR-10 and micro-F1 for FLAIR.</caption><thead><tr><th rowspan="2">Dataset</th><th rowspan="2">Cap</th><th rowspan="2">Data proc.<br>(images)</th><th colspan="3">Run time (s)</th><th colspan="3">Energy (kWh)</th><th rowspan="2">Perf.</th></tr><tr><th>AE</th><th>Sim</th><th>Deploy</th><th>AE</th><th>Sim</th><th>Deploy</th></tr></thead><tbody><tr><td>CIFAR-10</td><td>512</td><td>822k</td><td>1519</td><td>1350</td><td>1490</td><td>1.605</td><td>0.169</td><td>1.624</td><td>0.666</td></tr><tr><td>CIFAR-10</td><td>∞</td><td>1560k</td><td>3292</td><td>2045</td><td>3720</td><td>3.416</td><td>0.249</td><td>3.739</td><td>0.590</td></tr><tr><td>FLAIR</td><td>512</td><td>49k</td><td>232</td><td>2084</td><td>454</td><td>0.234</td><td>0.220</td><td>0.327</td><td>0.560</td></tr><tr><td>FLAIR</td><td>∞</td><td>161k</td><td>979</td><td>2601</td><td>1704</td><td>0.972</td><td>0.260</td><td>1.775</td><td>0.530</td></tr></tbody></table>

<table><caption>TABLE VI: Model performance with and without data capping. Accuracy for CIFAR-10, micro-F1 for FLAIR.</caption><thead><tr><th rowspan="2">Algorithm</th><th colspan="2">CIFAR-10 (Accuracy)</th><th colspan="2">FLAIR (μF1)</th></tr><tr><th>Cap</th><th>Uncap</th><th>Cap</th><th>Uncap</th></tr></thead><tbody><tr><td>Avg</td><td>0.66</td><td>0.59</td><td>0.56</td><td>0.53</td></tr><tr><td>Prox</td><td>0.73</td><td>0.79</td><td>0.56</td><td>0.57</td></tr><tr><td>Yogi</td><td>0.67</td><td>0.61</td><td>0.57</td><td>0.54</td></tr></tbody></table>

others to wait. Simulations mask this effect because multiple logical clients share the same node, resulting in run time increasing more uniformly with total data load.

As shown in Table VI, FedAvg performance deteriorates when the cap is removed, as increased heterogeneity harms its operation. In contrast, algorithms such as FedProx can leverage the additional data to improve results. In practice, deployments often enforce a cap or similar limits for privacy, fairness, or policy reasons. As a result, performance gains observed in experimental scenarios without caps may not translate to real-world deployments, where a cap may negate or even reverse them.
<figure><img src="image_8.png" /><figcaption>Fig. 7: Communication cost vs. model size.</figcaption></figure>

<table><caption>TABLE VII: Run time and accuracy for different models.</caption><thead><tr><th rowspan="2">model</th><th colspan="4">Run time (s)</th></tr><tr><th>AE</th><th>Simulation</th><th>Deployment</th><th>Accuracy</th></tr></thead><tbody><tr><td>MobileNetV3</td><td>222</td><td>1365</td><td>1449</td><td>0.757</td></tr><tr><td>ResNet-18</td><td>1519</td><td>1380</td><td>1585</td><td>0.666</td></tr><tr><td>Efficientnetb0</td><td>362</td><td>1690</td><td>1884</td><td>0.830</td></tr><tr><td>Clip</td><td>3628</td><td>2822</td><td>2479</td><td>0.844</td></tr><tr><td>ResNet-50</td><td>3406</td><td>2517</td><td>3269</td><td>0.828</td></tr></tbody></table>

**Takeaway:** Algorithmic gains may be offset by real-world constraints.

The impact of data cap illustrates how evaluations that ignore real deployment constraints can overestimate gains. In practice, caps or similar limits can negate or even reverse these benefits, highlighting the importance of incorporating realistic constraints in deployment-oriented evaluations.

## F. Models comparison

Figure 7 compares communication costs across models under different evaluation approaches. Communication load at the central server scales with model size, from 20 KB for ResNet-18 to 80 KB for ResNet-50. Analytical estimates and simulations closely follow this growth, while deployment reveals additional fixed background costs. The relative difference is more pronounced for smaller models, as those fixed costs become relatively less significant.

Table VII details run time and accuracy for these different models. Analytical estimates, calibrated on ResNet-18, generalize to ResNet-50 but require adaptation when the model family changes due to differences in efficiency and memory profiles. For example, EfficientNet-B0, despite using fewer FLOPs than ResNet-18 [63], exhibits lower operational efficiency, making it slower than expected [64]. Analytical estimates preserve relative ordering ($\tau = 0.91$) and trends ($\rho = 0.96$) well, though absolute differences can be substantial. Simulations ($\tau = 0.83$, $\rho = 0.89$) are generally faster than deployment because of stronger hardware and lower synchronization costs, except for CLIP, whose transformer layers (dominated by self-attention) appear more sensitive to memory behavior and resource contention on shared nodes.

**Takeaway:** Model architectures impact the gaps between the approaches.

Analytical estimates remain reliable only for models similar to those used for calibration, losing accuracy when efficiency or memory behavior changes. Model architecture also scales differently in simulation and deployment, since each environment stresses different hardware resources.

## VI. CONCLUSION

This work introduces and instantiates a methodology for evaluating FL across three complementary approaches: analytical estimates, simulations, and geo-distributed deployments. Applied to representative FL scenarios, it quantifies deviations between these approaches in run time, communication, and energy, extending evaluation beyond simulation-only studies.

Analytical estimates bring fast, explainable predictions of total cost and its main components, allowing reasoning about how computation, communication, and idle phases contribute to overall efficiency. Simulations add experimental flexibility: they reproduce the training logic, preserve performance trends, and enable rapid exploration of algorithms, hyperparameters, and models. Deployments provide realism, exposing system effects: idle energy, stragglers, protocol and network overheads, deployment constraints such as data caps or scheduling delays, that determine practical run time and energy use.

**Guidance:** The methodology can be applied via a staged strategy: (i) start with analytical estimates to obtain quick, low-cost approximations of cost components; (ii) use simulations to explore algorithmic trends and filter promising configurations; (iii) validate critical scenarios in real deployments to capture system-level constraints.

This staged process allows researchers and system designers to balance cost, accuracy, and realism when evaluating FL systems. Gap analysis further helps interpret simulation results and refine analytical estimates.

**Limitations:** The deployments used for illustration were executed on a geo-distributed testbed with heterogeneous hardware, but this represents only one type of possible FL infrastructure. Other scenarios may involve stronger hardware imbalance, intermittent connectivity, or resource-constrained edge devices, leading to different performance and energy behaviors. The datasets, models, and aggregation algorithms used cover representative but not exhaustive FL workloads. However, new scenarios can be easily added through new instances of the first step of the methodology (i.e., scenario definition) while the underlying building block used for deployments - the E2Clab framework - allows to explore new configurations and testbeds in a plug-and-play manner, extending the analysis beyond the scenarios presented here. Ultimately, broadening the range of infrastructures, datasets, and algorithms will help generalize the methodology and make it a practical tool for guiding efficient, realistic FL design.

**Outlook:** A promising direction is the joint optimization of model performance and system cost using analytical estimates and simulations for large-scale exploration, and deployments for focused validation. Incorporating distributed system
simulators could make simulations more realistic by modeling network variability and resource contention. Automated calibration of analytical estimates from small deployment samples would further improve predictive accuracy across hardware and workloads.

For the work presented here, we plan to release artifacts (logs, analysis scripts etc.) to foster reproducibility and enable follow-up studies on additional hardware and workloads.

## REFERENCES

[1] D. Balouek-Thomert, E. G. Renart, A. R. Zamani, A. Simonet, and M. Parashar, "Towards a computing continuum: Enabling edge-to-cloud integration for data-driven workflows," *The International Journal of High Performance Computing Applications*, vol. 33, no. 6, pp. 1159-1174, Nov. 2019, publisher: SAGE Publications Ltd STM. [Online]. Available: https://doi.org/10.1177/1094342019877383

[2] D. C. Nguyen, Q.-V. Pham, P. N. Pathirana, M. Ding, A. Seneviratne, Z. Lin, O. Dobre, and W.-J. Hwang, "Federated Learning for Smart Healthcare: A Survey," *ACM Comput. Surv.*, vol. 55, no. 3, pp. 60:1-60:37, Feb. 2022. [Online]. Available: https://dl.acm.org/doi/10.1145/3501296

[3] S. K. Aljunaid, S. J. Almheiri, H. Dawood, and M. A. Khan, "Secure and Transparent Banking: Explainable AI-Driven Federated Learning Model for Financial Fraud Detection," *Journal of Risk and Financial Management*, vol. 18, no. 4, p. 179, Apr. 2025, publisher: Multidisciplinary Digital Publishing Institute. [Online]. Available: https://www.mdpi.com/1911-8074/18/4/179

[4] M. Y. Lu, D. Kong, J. Lipkova, R. J. Chen, R. Singh, D. F. K. Williamson, T. Y. Chen, and F. Mahmood, "Federated Learning for Computational Pathology on Gigapixel Whole Slide Images," Sep. 2020, arXiv:2009.10190 [eess]. [Online]. Available: http://arxiv.org/abs/2009.10190

[5] S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečný, S. Kumar, and H. B. McMahan, "Adaptive Federated Optimization," Sep. 2021. [Online]. Available: http://arxiv.org/abs/2003.00295

[6] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, "Federated Learning: Challenges, Methods, and Future Directions," *IEEE Signal Processing Magazine*, vol. 37, no. 3, pp. 50-60, May 2020.

[7] M. Valli, A. Costan, C. Tedeschi, and L. Cudennec, "Towards Efficient Learning on the Computing Continuum: Advancing Dynamic Adaptation of Federated Learning," in *FlexScience'24: Proceedings of the 14th Workshop on AI and Scientific Computing at Scale using Flexible Computing Infrastructures*. Pisa, Italy: ACM, Jun. 2024, pp. 42-49. [Online]. Available: https://hal.science/hal-04698619

[8] C. Prigent, K. Keahey, A. Costan, L. Cudennec, and G. Antoniu, "On the Reproducibility Challenges of Federated Learning: Investigating the Gap between Simulation, Emulation and Real-World Deployments," in *CCGrid 2025 - IEEE 25th International Symposium on Cluster, Cloud and Internet Computing*, Tromso, Norway, May 2025. [Online]. Available: https://hal.science/hal-04997547

[9] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Konečný, S. Mazzocchi, H. B. McMahan, T. Van Overveldt, D. Petrou, D. Ramage, and J. Roselander, "Towards Federated Learning at Scale: System Design," Mar. 2019, arXiv:1902.01046 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1902.01046

[10] D. Rosendo, A. Costan, G. Antoniu, and P. Valduriez, "E2Clab: Reproducible Analysis of Complex Workflows on the Edge-to-Cloud Continuum," May 2021, pages: 1-2 Published: IPDPS 2021 - IEEE 35th International Parallel and Distributed Processing Symposium. [Online]. Available: https://hal.science/hal-03269852

[11] F. Cappello, E. Caron, M. Dayde, F. Desprez, Y. Jegou, P. Primet, E. Jeannot, S. Lanteri, J. Leduc, N. Melab, G. Mornet, R. Namyst, B. Quetier, and O. Richard, "Grid'5000: a large scale and highly reconfigurable grid experimental testbed," in *The 6th IEEE/ACM International Workshop on Grid Computing*, 2005., Nov. 2005, pp. 8 pp., iSSN: 2152-1093. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/1542730

[12] J. Caposiena, O. Carrillo, F. Le Mouël, B. Jonglez, P. Neyron, and T. Arrabal, "Towards a flexible Network Operating System Testbed for the Computing Continuum," in *2025 IEEE 25th International Symposium on Cluster, Cloud and Internet Computing Workshops (CCGridW)*, ser. 2025 IEEE 25th International Symposium on Cluster, Cloud and Internet Computing Workshops (CCGridW), IEEE, Ed., Tromsø Norway, Norway, May 2025, pp. 148-155. [Online]. Available: https://hal.science/hal-05154217

[13] E. Fleury, N. Mitton, T. Noel, and C. Adjih, "FIT IoT-LAB: The Largest IoT Open Experimental Testbed," *ERCIM News*, no. 101, p. 4, Apr. 2015. [Online]. Available: https://inria.hal.science/hal-01138038

[14] D. J. Beutel, T. Topal, A. Mathur, X. Qiu, J. Fernandez-Marques, Y. Gao, L. Sani, K. H. Li, T. Parcollet, P. P. B. de Gusmão, and N. D. Lane, "Flower: A Friendly Federated Learning Research Framework," Mar. 2022. [Online]. Available: http://arxiv.org/abs/2007.14390

[15] Z. Xu, Y. Zhang, G. Andrew, C. A. Choquette-Choo, P. Kairouz, H. B. McMahan, J. Rosenstock, and Y. Zhang, "Federated Learning of Gboard Language Models with Differential Privacy," Jul. 2023, arXiv:2305.18465 [cs]. [Online]. Available: http://arxiv.org/abs/2305.18465

[16] F. Lai, X. Zhu, H. V. Madhyastha, and M. Chowdhury, "Oort: Efficient Federated Learning via Guided Participant Selection," May 2021, arXiv:2010.06081 [cs]. [Online]. Available: http://arxiv.org/abs/2010.06081

[17] M. Paulik, M. Seigel, H. Mason, D. Telaar, J. Kluivers, R. v. Dalen, C. W. Lau, L. Carlson, F. Granqvist, C. Vandevelde, S. Agarwal, J. Freuderiger, A. Byde, A. Bhowmick, G. Kapoor, S. Beaumont, Cahill, D. Hughes, O. Javidbakht, F. Dong, R. Rishi, and S. Hung, "Federated Evaluation and Tuning for On-Device Personalization: System Design & Applications," Feb. 2021, arXiv:2102.08503 [cs]. [Online]. Available: http://arxiv.org/abs/2102.08503

[18] M. Hosseinzadeh, N. Hudson, S. Heshmati, and H. Khamfroush, "Communication-Loss Trade-Off in Federated Learning: A Distributed Client Selection Algorithm," in *2022 IEEE 19th Annual Consumer Communications & Networking Conference (CCNC)*, Jan. 2022, pp. 1-6, iSSN: 2331-9860. [Online]. Available: https://ieeexplore.ieee.org/document/9700601

[19] A. Hard, A. M. Girgis, E. Amid, S. Augenstein, L. McConnaughey, R. Mathews, and R. Anil, "Learning from straggler clients in federated learning," Mar. 2024, arXiv:2403.09086 [cs]. [Online]. Available: http://arxiv.org/abs/2403.09086

[20] C. W. Zaw, S. R. Pandey, K. Kim, and C. S. Hong, "Energy-aware Resource Management for Federated Learning in Multi-access Edge Computing Systems," Jan. 2021, arXiv:2103.13293 [cs]. [Online]. Available: http://arxiv.org/abs/2103.13293

[21] C. Prigent, A. Costan, G. Antoniu, and L. Cudennec, "Enabling Federated Learning across the Computing Continuum: Systems, Challenges and Future Directions," *Future Generation Computer Systems*, vol. 160, pp. 767-783, Jun. 2024, publisher: Elsevier. [Online]. Available: https://hal.science/hal-04659211

[22] F. Lai, Y. Dai, S. Singapuram, J. Liu, X. Zhu, H. Madhyastha, and M. Chowdhury, "FedScale: Benchmarking Model and System Performance of Federated Learning at Scale," in *Proceedings of the 39th International Conference on Machine Learning*. PMLR, Jun. 2022, pp. 11814-11827, iSSN: 2640-3498. [Online]. Available: https://proceedings.mlr.press/v162/lai22a.html

[23] S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Konečný, H. B. McMahan, V. Smith, and A. Talwalkar, "LEAF: A Benchmark for Federated Settings," Dec. 2019, arXiv:1812.01097 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1812.01097

[24] C. Feng, N. Huber, A. H. Celdran, G. Bovet, and B. Stiller, "Demo: A Practical Testbed for Decentralized Federated Learning on Physical Edge Devices," May 2025, arXiv:2505.08033 [cs]. [Online]. Available: http://arxiv.org/abs/2505.08033

[25] K.-S. Wong, M. Nguyen-Duc, K. Le-Huy, L. Ho-Tuan, C. Do-Danh, and D. Le-Phuoc, "An Empirical Study of Federated Learning on IoT-Edge Devices: Resource Allocation and Heterogeneity," May 2023, arXiv:2305.19831 [cs]. [Online]. Available: http://arxiv.org/abs/2305.19831

[26] Y. Gao, M. Kim, S. Abuadbba, Y. Kim, C. Thapa, K. Kim, S. A. Camtepe, H. Kim, and S. Nepal, "End-to-End Evaluation of Federated Learning and Split Learning for Internet of Things," Aug. 2020, arXiv:2003.13376 [cs]. [Online]. Available: http://arxiv.org/abs/2003.13376
[27] S. Delamare and L. Nussbaum, "Monitoring étendu d'infrastructures avec Kwallect," in JRES (Journées réseaux de l'enseignement et de la recherche) 2021. Marseille, France: Renater, May 2022. [Online].
Available: https://hal.science/hal-04807316

[28] L. Barbieri, S. Savazzi, S. Kianoush, M. Nicoli, and L. Serio, "A Carbon Tracking Model for Federated Learning: Impact of Quantization and Sparsification," in 2023 IEEE 28th International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD). Nov. 2023, pp. 213–218, arXiv:2310.08087 [eess]. [Online]. Available: http://arxiv.org/abs/2310.08087

[29] K. Tan, D. Wu, A. J. Chan, and P. Mohapatra, "Comparing simulation tools and experimental testbeds for wireless mesh networks," Pervasive and Mobile Computing, vol. 7, no. 4, pp. 434–448, Aug. 2011. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S157411921100040X

[30] J. Gomez, E. F. Kfoury, J. Crichigno, and G. Srivastava, "A survey on network simulators, emulators, and testbeds used for research and education," Computer Networks, vol. 237, p. 110054, Dec. 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1389128623004991

[31] S. Silmi, Z. Doukha, R. Kemcha, and S. Moussaoui, "Wireless sensor networks simulators and testbeds," Computer Science and Information Technology, 2020. [Online]. Available: https://wwwsemanticscholar.org/paper/Wireless-sensor-networks-simulators-and-testbeds-Silmi-Doukha/fa86f23a88944d0acba6d34c7592160536102ae8

[32] N. D. Patel, B. M. Mehtre, and R. Wankar, "Simulators, Emulators, and Test-beds for Internet of Things: A Comparison," in 2019 Third International conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), Dec. 2019, pp. 139–145. [Online]. Available: https://ieeexplore.ieee.org/document/9032519

[33] L. Ridolfi, D. Naseh, S. S. Shinde, and D. Tarchi, "Implementation and Evaluation of a Federated Learning Framework on Raspberry Pi Platforms for IoT 6G Applications," Future Internet, vol. 15, no. 11, p. 358, Nov. 2023, publisher: Multidisciplinary Digital Publishing Institute. [Online]. Available: https://www.mdpi.com/1999-5903/15/11/358

[34] S. Sebbio, G. Morabito, A. Catalfamo, L. Carnevale, and M. Fazio, "Federated Learning on Raspberry Pi 4: A Comprehensive Power Consumption Analysis," in Proceedings of the IEEE/ACM 16th International Conference on Utility and Cloud Computing, ser. UCC '23. New York, NY, USA: Association for Computing Machinery, Apr. 2024, pp. 1–6. [Online]. Available: https://dl.acm.org/doi/10.1145/3603166.3632545

[35] G. A. Baumgart, J. Shin, A. Payani, M. Lee, and R. R. Kompella, "Not All Federated Learning Algorithms Are Created Equal: A Performance Evaluation Study," Mar. 2024, arXiv:2403.17287 [cs]. [Online]. Available: http://arxiv.org/abs/2403.17287

[36] M. Asad, A. Moustafa, T. Ito, and M. Aslam, "Evaluating the Communication Efficiency in Federated Learning Algorithms," Apr. 2020, arXiv:2004.02738 [cs]. [Online]. Available: http://arxiv.org/abs/2004.02738

[37] V. Hayyolalam and Özkasap, "FedBWO: Enhancing Communication Efficiency in Federated Learning," May 2025, arXiv:2505.04435 [cs]. [Online]. Available: http://arxiv.org/abs/2505.04435

[38] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, R. G. L. D'Oliveira, H. Eichner, S. E. Rouayheb, D. Evans, J. Gardner, Z. Garrett, A. Gascón, B. Ghazi, P. B. Gibbons, M. Gruteser, Z. Harchaoui, C. He, L. He, Z. Huo, B. Hutchinson, J. Hsu, M. Jaggi, T. Javidi, G. Joshi, M. Khodak, J. Konečný, A. Korolova, F. Koushanfar, S. Koyejo, T. Lepoint, Y. Liu, P. Mittal, M. Mohri, R. Nock, A. Özgür, R. Pagh, M. Raykova, H. Qi, D. Ramage, R. Raskar, D. Song, W. Song, S. U. Stich, Z. Sun, A. T. Suresh, F. Tramèr, P. Vepakomma, J. Wang, L. Xiong, Z. Xu, Q. Yang, F. X. Yu, H. Yu, and S. Zhao, "Advances and Open Problems in Federated Learning," Mar. 2021, arXiv:1912.04977 [cs, stat]. [Online]. Available: http://arxiv.org/abs/1912.04977

[39] J. Mills, J. Hu, and G. Min, "Faster Federated Learning with Decaying Number of Local SGD Steps," May 2023, arXiv:2305.09628 [cs]. [Online]. Available: http://arxiv.org/abs/2305.09628

[40] T. Nishio and R. Yonetani, "Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edge," in ICC 2019 - 2019 IEEE International Conference on Communications (ICC), May 2019, pp. 1–7, iSSN: 1938-1883. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/8761315

[41] S. Williams, A. Waterman, and D. Patterson, "Roofline: an insightful visual performance model for multicore architectures," Commun. ACM, vol. 52, no. 4, pp. 65–76, Apr. 2009. [Online]. Available: https://dl.acm.org/doi/10.1145/1498765.1498785

[42] X. Qiu, T. Parcollet, J. Fernandez-Marques, P. P. B. Gusmao, Y. Gao, D. J. Beutel, T. Topal, A. Mathur, and N. D. Lane, "A First Look into the Carbon Footprint of Federated Learning," Journal of Machine Learning Research, vol. 24, no. 129, pp. 1–23, 2023. [Online]. Available: http://jmlr.org/papers/v24/21-0445.html

[43] Z. Yang, M. Chen, W. Saad, C. S. Hong, and M. Shikh-Bahaei, "Energy Efficient Federated Learning Over Wireless Communication Networks," Nov. 2020, arXiv:1911.02417 [cs]. [Online]. Available: http://arxiv.org/abs/1911.02417

[44] D. Rosendo, P. Silva, M. Simonin, A. Costan, and G. Antoniu, "E2Clab: Exploring the Computing Continuum through Repeatable, Replicable and Reproducible Edge-to-Cloud Experiments," in Cluster 2020 - IEEE International Conference on Cluster Computing, Kobe, Japan, Sep. 2020, pp. 1–11. [Online]. Available: https://hal.science/hal-02916032

[45] A. Krizhevsky, "Learning multiple layers of features from tiny images," Tech. Rep., 2009.

[46] Q. Li, Y. Diao, Q. Chen, and B. He, "Federated Learning on Non-IID Data Silos: An Experimental Study," in 2022 IEEE 38th International Conference on Data Engineering (ICDE), May 2022, pp. 965–978, iSSN: 2375-026X. [Online]. Available: https://ieeexplore.ieee.org/abstract/document/9835537

[47] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," Dec. 2015, arXiv:1512.03385 [cs]. [Online]. Available: http://arxiv.org/abs/1512.03385

[48] G. Legate, N. Bernier, L. Caccia, E. Oyallon, and E. Belilovsky, "Guiding The Last Layer in Federated Learning with Pre-Trained Models," Nov. 2023, arXiv:2306.03937 [cs]. [Online]. Available: http://arxiv.org/abs/2306.03937

[49] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas, "Communication-Efficient Learning of Deep Networks from Decentralized Data," Feb. 2016. [Online]. Available: http://arxiv.org/abs/1602.05629

[50] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Q. V. Le, and H. Adam, "Searching for MobileNetV3," Nov. 2019, arXiv:1905.02244 [cs]. [Online]. Available: http://arxiv.org/abs/1905.02244

[51] M. Tan and Q. V. Le, "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks," Sep. 2020, arXiv:1905.11946 [cs]. [Online]. Available: http://arxiv.org/abs/1905.11946

[52] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, "Learning Transferable Visual Models From Natural Language Supervision," Feb. 2021, arXiv:2103.00020 [cs]. [Online]. Available: http://arxiv.org/abs/2103.00020

[53] C. Song, F. Granqvist, and K. Talwar, "FLAIR: Federated Learning Annotated Image Repository," Jul. 2022. [Online]. Available: http://arxiv.org/abs/2207.08869

[54] P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, and I. Stoica, "Ray: A Distributed Framework for Emerging AI Applications," Sep. 2018, arXiv:1712.05889 [cs]. [Online]. Available: http://arxiv.org/abs/1712.05889

[55] "Lille:Hardware - Grid5000." [Online]. Available: https://www.grid5000.fr/w/Lille:Hardwarechifflot

[56] "Rennes:Hardware - Grid5000." [Online]. Available: https://www.grid5000.fr/w/Rennes:Hardwareparasilo

[57] "Nancy:Hardware - Grid5000." [Online]. Available: https://www.grid5000.fr/w/Nancy:Hardwaregros

[58] "Nantes:Hardware - Grid5000." [Online]. Available: https://www.grid5000.fr/w/Nantes:Hardwareecotype

[59] "Grid5000:Network - Grid5000." [Online]. Available: https://www.grid5000.fr/w/Grid5000:Network

[60] D. Hackenberg, T. Ilsche, R. Schöne, D. Molka, M. Schmidt, and W. E. Nagel, "Power measurement techniques on standard compute nodes: A quantitative comparison," in 2013 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), Apr. 2013, pp. 194–204. [Online]. Available: https://ieeexplore.ieee.org/document/6557170

[61] K. N. Khan, M. Hirki, T. Niemi, J. K. Nurminen, and Z. Ou, "RAPL in Action: Experiences in Using RAPL for Power Measurements," ACM
*Trans. Model. Perform. Eval. Comput. Syst.*, vol. 3, no. 2, pp. 9:1–9:26, Mar. 2018. [Online]. Available: https://dl.acm.org/doi/10.1145/3177754

[62] H. Ballani, P. Costa, T. Karagiannis, and A. Rowstron, “Towards predictable datacenter networks,” *SIGCOMM Comput. Commun. Rev.*, vol. 41, no. 4, pp. 242–253, Aug. 2011. [Online]. Available: https://dl.acm.org/doi/10.1145/2043164.2018465

[63] “Models and pre-trained weights — Torchvision main documentation.” [Online]. Available: https://pytorch.org/vision/stable/models.html

[64] S. Li, M. Tan, R. Pang, A. Li, L. Cheng, Q. Le, and N. P. Jouppi, “Searching for Fast Model Families on Datacenter Accelerators,” Feb. 2021, arXiv:2102.05610 [cs]. [Online]. Available: http://arxiv.org/abs/2102.05610