# LivingBench: an IoT/Edge Platform Benchmark Based on an Environmental Observation Use-Case

Haraesh Jr, Ammar Kazem, Matthieu Nicolas, François Lemercier, Guillaume

Pierre, Christoff Andermann

► To cite this version:

Haraesh Jr, Ammar Kazem, Matthieu Nicolas, François Lemercier, Guillaume Pierre, et al. Living-Bench: an IoT/Edge Platform Benchmark Based on an Environmental Observation Use-Case. ICIN 2026 - 29th Conference on Innovation in Clouds, Internet and Networks, Mar 2026, Athens, Greece. hal-05454254v2

HAL Id: hal-05454254

https://inria.hal.science/hal-05454254v2

Submitted on 13 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# LivingBench: an IoT/Edge Platform Benchmark
Based on an Environmental Observation Use-Case

Haraesh JR Ⓓ Ⓒ Ammar Kazem Ⓓ Ⓒ Matthieu Nicolas Ⓓ Ⓒ François Lemercier Ⓓ Ⓒ Guillaume Pierre Ⓓ Ⓒ

Univ Rennes, Inria, CNRS, IRISA

{haraesh.jr, ammar.kazem, matthieu.nicolas, francois.lemercier, guillaume.pierre}@irisa.fr

Christoff Andermann Ⓒ

Géosciences Rennes, Université de Rennes

Helmholtz Centre Potsdam, GFZ German Research Center for Geosciences

christoff.andermann@univ-rennes.fr

**Abstract**—In recent years, a number of edge computing platforms have been proposed to process data produced by IoT sensors. Performing computation close to the sources of data allows faster insight and greater reliability at a lower cost compared to traditional cloud-based deployments. However, designers of IoT/edge platforms face difficult issues. In particular, exercising and testing a new platform in conditions that approach a real deployment requires a sufficient number of standard benchmarking systems capable of generating realistic workloads. In this paper, we propose LivingBench, a benchmarking tool with the capability of incorporating real or synthetic workload injection, developed to exercise edge computing systems. LivingBench integrates a real-world data trace captured in an environmental observatory, together with a collection of actual applications designed for processing these data, and a load injector tool capable of replaying a (possibly pre-processed) trace to benchmark an MQTT-based edge system. We describe the architecture of LivingBench and show how it may be used to evaluate the maximum data processing capacity of an edge system under test.

**Index Terms**—IoT, edge computing, performance, overloading, workload injection, benchmarking.

## I. INTRODUCTION

Designing and experimentally evaluating IoT/edge cluster systems is difficult. Because the domain is relatively new, few actual deployments have been made and even fewer public datasets and complete benchmarks are available to support researchers in evaluating the behavior and performance of new IoT/Edge platforms.

Researchers who design a new edge platform and who want to evaluate it under a realistic workload must currently exploit one of a handful of public datasets such as the famous New York taxi trace [5]. However, this dataset represents only a single type of data that an edge platform might process. Furthermore, it does not include examples of real applications which may need to process these data so researchers are obliged to invent new applications to process the available data. This raises difficult questions regarding the level of

realism of the scientific experimentation procedures based on such applications and datasets, considering they might include conscious or unconscious biases which potentially influence the evaluation results.

We argue that a strong benchmarking tool designed to evaluate current and future IoT/edge platforms should have the following properties. First, it should be based on an actual IoT/edge use-case and a publicly-available dataset derived from this very use-case. Second, it must include a set of realistic applications capable of exploiting parts or all of the available data. Under these conditions it becomes possible to evaluate new edge platforms and to compare multiple platforms under a standardized set of conditions and metrics. Finally, if so desired by the users of the benchmark, it should be possible to manipulate the standard workloads, for instance to generate greater or lower workload intensity to match the processing capacity of the IoT/Edge system under test.

In recent years, organizations such as the SPEC RG Cloud Working Group have produced high-quality benchmarks to exercise different types of cloud systems [6]. Unfortunately, similar initiatives in the domain of IoT/Edge platforms have so far been lacking.

In this paper we propose LivingBench, a benchmarking tool for edge computing systems based on an actual IoT/edge use-case in the domain of natural environment observation. Specifically, the Kaligandaki observatory is dedicated to the hydrological observation of the Kali Gandaki river [18] set in the Himalaya mountains in Nepal. This river is known to experience floods in the monsoon season which may sometimes have catastrophic consequences [4]. The observatory therefore exploits a variety of sensors to generate data related to weather, water level and similar metrics. Due to local constraints these data are transmitted to the cloud using cellular networks at a frequency such as once per day. Although this is sufficient to support data collection and archival for later analysis, a different approach is necessary to quickly identify and/or predict floods and to issue timely alerts to the local population. The observatory is therefore augmented with an *in-situ* edge platform where real-time data processing can take place [14].

LivingBench comprises three main elements. First, it con-

<footnote>This research was supported by funding from the French government, administered by the National Agency for Research (ANR) through the Future Investment Program, integrated into France 2030, with grant reference ANR-21-ESRE-0014.</footnote>
tains a real dataset produced by the sensors in the Kaligandaki
observatory. The data include in particular environmental mea-
surements captured from the Kali Gandaki river, which include
the river level measured by gauge and rain dipper sensors,
weather-related data and the battery module that powers the
rest of the sensors. Second, LivingBench contains a number
of applications designed to process the sensor data: some
applications are dedicated to detecting sensor failures, whereas
others focus on validating and cleaning input data before fur-
ther processing, and ultimately on identifying relevant events
such as a flood. These applications are designed to integrate
easily in the LivingFog edge platform which is being deployed
in the Kaligandaki observatory [14], but they should also be
easily portable to other edge platforms compatible with the
MQTT standard for IoT messaging [13]. Finally, LivingBench
provides a workload injector capable of pre-processing the
trace (e.g., to emulate a greater number of sensor than actually
deployed in the observatory) and injecting it in the edge
platform to exercise it with real data and real applications.

The remainder of this paper is organized as follows. Sec-
tion II presents background information about natural envi-
ronment observatories and the way they may process IoT data
using edge platforms. Then, Section III presents the design
of the LivingBench system. Section IV evaluates this design
and shows how the tool may be used to conduct a number
of performance analyses. Finally, Section V discusses related
work in the domain of IoT workloads and edge platform
benchmarking tools, and Section VI concludes the paper.

## II. BACKGROUND: DATA ANALYSIS IN NATURAL ENVIRONMENT OBSERVATORIES

Natural environment observatories exploit dozens to hundreds of sensors of various types and locations to characterize the behavior of the natural environment in a well-defined zone according to the requirements of environmental scientists. The produced data are typically transmitted to the cloud for long-term archival using a variety of networking technologies. The same data may also be processed by a local edge platform to extract real-time insights and/or to trigger automatic reactions to specific predicates based on these data [8].

LivingFog is an open-source edge computing platform originally developed as part of the FogGuru European Project [2]. Although it was initially designed for smart-city scenarios, later work repurposed it to specifically address natural environment monitoring scenarios [14]. When sensor data are being produced, they get streamed to LivingFog following the MQTT protocol [13]. Sensor data can then be processed by one or more applications which consume data streams from certain MQTT topics and produce new data streams in other topics.

The architecture of LivingFog is described in Figure 1. Its main component are the Mosquitto MQTT message broker, together with a set of data processing applications. Data are also forwarded to an InfluxDB time-series database for short-term storage and visualization. Prometheus is used to monitor the system and generate alerts using the AlertManager in case

<figure><img src="image_2.png" /><figcaption>Fig. 1: LivingFog system architecture.</figcaption></figure>

of system dysfunction. The system is designed to execute
within a Kubernetes orchestrator executing on one or more
single-board computer(s) such as Raspberry Pis.

The set of applications was chosen according to the nature
of IoT data being collected as well as the objectives of the en-
vironmental scientists. Typically, data entering the system are
first checked for syntactic correctness (i.e., ensuring that they
follow the expected formats), then for semantic correctness
(i.e., ensuring that the values seem to make sense), and later
for identifying specific events of interests such as the flood of a
monitored river. The event detection can also trigger additional
functions such as changing the data sampling frequencies.

### III. SYSTEM DESIGN

LivingBench is a benchmarking tool designed to exercise
IoT/edge platforms with a realistic data set and suite of
applications<fnref n="1" />. Although it was designed to fit the architecture
of LivingFog, it should be easily portable to also exercise other
MQTT-compatible edge platforms.

LivingBench provides three components: (i) a real dataset captured by the sensors deployed along the Kali Gandaki river; (ii) a set of applications capable of processing parts of these data; and (iii) a workload injector which can inject trace data into the edge system under test, possibly after a trace pre-processing phase. We describe these components next.

#### A. Dataset

The LivingBench dataset was obtained from a real-life sen-
sor network deployment along the Kali Gandaki river in Nepal
from March 2024 until September 2024. This observatory
comprises two stations in different locations along the river,
and the following list of sensors.

* The Lufft WS100 [10] sensor monitors the precipitation level during the past 5 minutes and the type of precipitation (drizzle, snow, hail).

* The Lufft WS501 [9] sensor monitors the weather, including temperature, relative humidity and wind speed.

* The Vaisala WXT520 [17] sensor measures wind speed and direction, precipitation and air temperature.

* The OTT HACH RLS300 sensor monitors the water level of the river.

<footnote n="1">¹LivingBench is available in open-source at
https://gitlab.inria.fr/livingfog/livingbench</footnote>
* Other sensors include different types of rain meters, seismometers, cameras, and a measurement of the charge level of the battery powering the overall system.

The trace data is made available in raw format, as it was originally produced by the sensors, as well as reformatted using the InfluxDB Line Protocol [7]. Considering its usage in data ingestion in InfluxDB, this latter option is the most convenient format for workload injection. As any given sensor device may be measuring multiple metrics, each record in this trace comprises the following fields: (i) the sensor type; (ii) the name of the station where the sensor is deployed; (iii) The ID of the sensor; (iv) a CSV-formatted list specifying the units of the measured values; (v) a CSV-formatted list with the values captured by the sensor; and (vi) a timestamp of the measurement. The actual data were captured at intervals ranging from 15 sec to 5 min depending on the sensor types.

## B. Data processing applications

LivingBench comprises a set of simple applications capable of processing parts of the dataset following a realistic use case, and therefore of exercising the edge platform under test with a realistic workload. Each application is designed to consume input data from one or more MQTT topics, and to produce output data in one or more other topics where they can be consumed by other applications. Each application is containerized such that it can easily be deployed in a Docker-based or Kubernetes-based edge platform.

The provided applications provide a representative set of functions that a real observatory may want to execute in their edge platform. They belong to two main categories:

* **Syntactic validation applications** are used to receive a "raw" MQTT topic and ensuring that the data actually follow the prescribed syntax before being further processed by other applications. Invalid data being produced by failing sensors can thus be forwarded to a separate MQTT topic dedicated to error management.

* **Semantic validation applications** ensure that the input data is plausible. For instance, a sensor reporting a river water temperature under 0°C is probably failing and its data should be discarded. Here as well, bogus data can be sent to a specific MQTT topic, for instance in order to send alerts to the observatory managers.

Future applications to be added to the list include:

* **Event detection applications** aim to identify specific predicates which constitute a relevant event for the observatory managers or the environmental scientists. Predicates themselves may be defined as simple threshold conditions, for instance in case water level raises above certain limits. Others may be more complex and require integrating multiple data sources. When a predicate becomes true, a message is sent to a dedicated topic where it may trigger reactions such as an alert, and an automatic analysis of the ongoing event.

* **Prediction applications** observe the trends of certain data time series and issue predictions about their future values. Predictors may be based on simple statistical methods such as linear extrapolation, or be generated using validated models of the natural environment.

<figure><img src="image_3.png" /><figcaption>Fig. 2: LivingBench components.</figcaption></figure>

## C. Workload injector

The workload injector is in charge of reading a (possibly pre-processed) trace and injecting it in the form of messages delivered to the edge system's MQTT broker where data can be processed by one or more applications. Its main components are depicted in Figure 2:

* **Configurations** describe the properties of the workload to be injected, such as the size and type of the workload to be injected, the list of measurements captured in a single data line and their individual properties. This information is abstracted in *experiment configurations* (which specify experiment-specific properties) and *sensor configurations* (which define sensor properties that would ideally capture the data in real life).

* **The Data Maker** is in charge of preparing a synthetic workload that can be injected into an edge/IoT system. This may require filtering only specific parts of the full dataset as well as randomly altering the actual measurements using a range which can be defined using the configurations, with the aim of preserving the "real" nature of the source data.

* **The Data Injector** retrieves a specific workload generated by the Data Maker and injects it into the MQTT broker of an edge system. It can be visualized as mirroring a real-life sensor network comprised of *n* sensors, each emitting *m* data lines to the edge platform.

# IV. EVALUATION

LivingBench is intended as a generic tool capable of performing a wide range of experiments to assess the behavior and performance of an edge system under test. We now illustrate its usage by showing how it can be used to stress an edge platform up to its saturation point.

## A. Experimentation Parameters

The following experiments exercise an edge system composed of two Raspberry Pi 4 single-board computers where the LivingFog platform has been deployed. These two nodes
<table><caption>TABLE I: CPU usage under different pod, workload, and message transmission configurations for the worker node.</caption><thead><tr><th>Round number</th><th># pods</th><th>Emitting interval (s)</th><th>Dataset size</th><th>Max CPU usage (%)</th><th>Overload duration</th></tr></thead><tbody><tr><td>1</td><td>150</td><td>30</td><td>75,000</td><td>99.9-100</td><td>6 min</td></tr><tr><td>2</td><td>150</td><td>30</td><td>100,000</td><td>99.9-100</td><td>8 min</td></tr><tr><td>3</td><td>150</td><td>60</td><td>75,000</td><td>99.9-100</td><td>5 min</td></tr><tr><td>4</td><td>150</td><td>60</td><td>100,000</td><td>99.9-100</td><td>7 min</td></tr><tr><td>5</td><td>200</td><td>30</td><td>75,000</td><td>99.9-100</td><td>10 min</td></tr><tr><td>6</td><td>200</td><td>30</td><td>100,000</td><td>99.9-100</td><td>11 min</td></tr><tr><td>7</td><td>200</td><td>60</td><td>75,000</td><td>99.9-100</td><td>11 min</td></tr><tr><td>8</td><td>200</td><td>60</td><td>100,000</td><td>99.9-100</td><td>9 min</td></tr></tbody></table>

respectively host the Kubernetes control plane (in charge of controlling the system) and one worker node where applications actually run. An HP notebook with SSD storage, 32 GB RAM and an Intel® Core™ Ultra 7 265U CPU is utilized to connect to and configure the edge nodes and for orchestration of software setup, and a gigabit Netgear switch connects all three computing systems to the same network.

The experiments rely on two applications: (i) a threshold checker that alerts the user when the river level goes beyond a specific threshold; and (ii) a predictor that extrapolates the battery voltage values from recent data samples.

To stress the edge platform we injected data streams with eight different configurations which either emulate a system with 150 or 200 pods running multiple instances of the same applications. These configurations were chosen to identify the maximum load that can be taken by the system under test. In each evaluation round, either 75,000 or 100,000 lines of data were transmitted by the LivingBench's load injector, emulating a network of 150-200 sensors emitting data. A monitoring script was deployed on both edge nodes in order to monitor their CPU usage, RAM usage and temperature periodically. We derived two performance metrics: (i) the maximum percentage of CPU usage observed during the experiment; and (ii) how long (in minutes) the edge system was observed to be overloaded during this maximum CPU usage.

## B. Evaluation Results and Discussion

Table I shows the results during the eight different scenarios. Additionally, Figure 3 depicts the CPU utilization of both nodes while executing the 3rd and 7th rounds of experiments, respectively with 150 and 200 pods. In every round with 150 pods, the edge system was able to process the incoming load and return to a normal state of behavior after 5-8 minutes. However, in every round based on 200 pods, the overload period is much longer: it was only after 9-11 minutes that the utilization goes back to normal.

In all rounds of experimentation, both the control plane and worker node experienced a spike in CPU performance every 5 minutes, even after the end of the load injection process. We hypothesize a background process such as database compaction performed by InfluxDB is the root cause. A deeper analysis is out of the scope of this paper.

In all rounds with 200-pod deployments, we witnessed a reduction in the transmission speed of messages to the MQTT broker, following which pod deployments started to crash and restart. We also observed an increase in delay when trying to access the Kubernetes cluster or its pods, along with the MQTT broker itself. This indicates clear system overload, which in turn gives useful indications about the maximum capacity of the edge system under test. On the other hand, the control plane returns to a normal state much quicker.

We argue that this demonstrates LivingBench's capability to serve as a benchmarking tool for edge computing systems, in this case to drive the edge system under test to its limits and to estimate its maximum processing capacity. We leave further usage of the LivingBench system to study other types of behavior for future work.

## V. RELATED WORK

While numerous works have explored performance benchmarking in IoT, edge, and fog computing contexts, few have focused on the availability and characterization of representative workload datasets. The scarcity of such datasets prevents the reproducibility and fairness of benchmarking studies, and moreover, limits the ability to compare system performance across different architectures.

Tadakamalla et al. define a workload as the set of all inputs a system receives from its environment over one or more time intervals, and explore its characterization within IoT networks [16]. They emphasize that leveraging IoT-generated data requires understanding the workload's fundamental components, such as device requests, to assess system utilization. The study tackles multiple datasets outlining their specifications and identifying factors that contribute to dataset accuracy. However, it does not propose any benchmarking framework capable of using these datasets as input sources.

Shishira et al. compare synthetic and real workloads: real workloads are publicly available for experimentation while retaining real-world traces from them, whereas synthetic workloads are defined as that which is representative of the real world and preserves its important characteristics [15]. The authors also identify two main strategies for generating synthetic workloads: an empirical approach, which replays traces of real workloads to reproduce observed patterns, and an analytical approach, which relies on mathematical models to describe and generate representative workload characteristics.

McChesney et al introduce DeFog, a benchmarking suite which addresses the lack of standardized methods for evaluating the performance of cloud, fog, and edge systems [11]. The framework provides a benchmarking methodology and defines a set of performance metrics applied to six applications. The study highlights that workloads processed across cloud, fog, and edge environments can exhibit significantly different latency behaviors when subjected to stress conditions. Depending on the application, communication latency (here defined as the time taken for asset transfer inside the environment throughout the benchmarking process) has been observed in the study to be between 1.75-2.75 seconds when applications were deployed on a cloud-edge system using a Raspberry Pi 3, whereas it increases to 3.5-5.1 seconds when deployed on a
<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 3: CPU utilization of LivingFog system visualized for the third and seventh round of experiments.</figcaption></figure>

cloud-only environment. We note that this study does not take into account the real or synthetic nature of workloads.

A similar approach is taken by Aslanpour et al., who employ IBM's Monitor, Analyze, Plan, Execute (MAPE) framework to classify a comprehensive set of performance metrics relevant to cloud, fog, and edge systems [1]. The authors organize these metrics into four main categories. An evaluation of existing simulators for evaluating and measuring performance metrics is also presented, but the study does not mention the use of real workloads or tools that can incorporate them.

Batz et al. [3] propose an IoT benchmark designed to model the performance of a data center in Boden, Sweden. The benchmark is composed of a set of application primitives, themselves based on the core components of the Microsoft IoT reference architecture [12]. Its goal is to emulate an IoT application and its associated workload, allowing simulation of the performance of specific decomposed use cases. While this modeling approach is comprehensive, it does not test or monitor an actual edge computing system connected to an IoT network, nor does it support fully generic use cases.

## VI. CONCLUSION

In this paper, we have presented LivingBench as a benchmarking tool designed to evaluate edge computing system performance by providing them a workload and an associated architecture for applications which exploit them. We have discussed its architecture and integration with LivingFog in the context of environmental observation. Our results demonstrate that LivingBench is capable of overloading a edge computing system and hence establish a benchmark for it.

Future work will focus on evaluating a wider variety of edge platforms, evaluating LivingBench's integration with non-Kubernetes orchestrators and more intuitive integration of real workloads.

## REFERENCES

[1] Mohammad S. Aslanpour, Sukhpal Singh Gill, and Adel N. Toosi. Performance evaluation metrics for cloud, fog and edge computing: A review, taxonomy, benchmarks and standards for future research. *Internet of Things*, 12, 2020. https://doi.org/10.1016/j.iot.2020.100273.

[2] Davaadorj Battulga, Mozhdeh Farhadi, Mulugeta Ayalew Tamiru, Li Wu, and Guillaume Pierre. LivingFog: Leveraging fog computing and lorawan technologies for smart marina management (experience paper). In *Proc. ICIN*, 2022. https://doi.org/10.1109/ICIN53892.2022.9758124.

[3] Thomas Batz, Reinhard Herzog, Jon Summers, and Kym Watson. IoT workload emulation for data centers. *Open Research Europe*, 1, 2021. https://doi.org/10.12688/openreseurope.13079.1.

[4] Rainer Bell, Monique Fort, Joachim Götz, Heidi Bernsteiner, Christoff Andermann, Jürgen Etzlstorfer, Eva Posch, Narayan Gurung, and Sonam Gurung. Major geomorphic events and natural hazards during monsoonal precipitation 2018 in the Kali Gandaki Valley, Nepal Himalaya. *Geomorphology*, 372, 2021. https://doi.org/10.1016/j geomorph.2020.107451.

[5] Brian Donovan and DB Work. New York City taxi trip data (2010-2013). Univ. Illinois Urbana-Champaign, 2016. https://doi.org/10.13012/J8PN93H8.

[6] Nikolas Herbst et al. Ready for rain? A view from SPEC research on the future of cloud metrics. CoRR, abs/1604.03470, 2016. https://doi.org/10.48550/arXiv.1604.03470.

[7] InfluxData Inc. Line protocol. https://docs.influxdata.com/influxdb/cloud-serverless/reference/syntax/line-protocol/.

[8] Ammar Kazem, Guillaume Pierre, and Laurent Longuevergne. Enhancing environmental observatories with fog computing. *Frontiers in Environmental Science*, 13, 2025. https://doi.org/10.3389/fenvs.2025.1568016.

[9] Lufft. Compact weather sensors - WS501-UMB smart weather sensor. https://www.lufft.com/products/compact-weather-sensors-293/ws501-umb-smart-weather-sensor-1839/.

[10] Lufft. Precipitation sensors - WS100 radar precipitation sensor / smart disdrometer. https://www.lufft.com/products/precipitation-sensors-287/ws100-radar-precipitation-sensor-smart-disdrometer-2361/.

[11] Jonathan McChesney, Nan Wang, Ashish Tanwer, Eyal De Lara, and Besson Varghese. DeFog: fog computing benchmarks. In *Proc. ACM SEC*, 2019. https://doi.org/10.1145/3318216.3363299.

[12] Microsoft. Microsoft Azure IoT reference architecture, 2018. https://download.microsoft.com/download/A/4/D/A4DAD253-BC21-41D3-B9D9-87D2AE6F0719/Microsoft_Azure_IoT_Reference_Architecture.pdf.

[13] MQTT Authors. MQTT: The standard for IoT messaging. https://$mQT.org/.

[14] Matthieu Nicolas, Elsa Mühl, Christoff Andermann, and Guillaume Pierre. Urgent Computing across the Device-Edge-Cloud Continuum: Concepts, Applications, and Future Directions, chapter "LivingFog: In-Situ Environmental Data Processing for Urgent Event Detection and Analysis". Springer Nature, 2026, to appear.

[15] S.R. Shishira, A. Kandasamy, and K. Chandrasekaran. In *Proc. ICCCT*, 2017. https://doi.org/10.1145/3154979.3155003.

[16] Uma Tadakamalla and Daniel A Menascé. Characterization of IoT workloads. In *Proc. EDGE*, 2019. https://doi.org/10.1007/978-3-030-23374-7_1.

[17] Vaisala. Vaisala weather transmitter WXT520. https://www.vaisala.com/sites/default/files/documents/M210906EN-C.pdf.

[18] Wikipedia. Gandaki river. https://en.wikipedia.org/wiki/Gandaki_River.