# Kopernic Dynamic Benchmarks (KDBench) - open source measurement-based benchmarks

Yves Sorel, Ismail Hawila, Liliana Cucu-Grosjean, Mehdi Mezouak, Hadrien Clarke, Slim Ben-Amor

## ► To cite this version:

Yves Sorel, Ismail Hawila, Liliana Cucu-Grosjean, Mehdi Mezouak, Hadrien Clarke, et al., Kopernic Dynamic Benchmarks (KDBench) - open source measurement-based benchmarks. RTSS@work 2024 - organized as part of the IEEE Real-Time Systems Symposium (RTSS) 2024, Oct 2024, York, United Kingdom. hal-05004638

HAL Id: hal-05004638

https://inria.hal.science/hal-05004638v1

Submitted on 27 Mar 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Kopernic Dynamic Benchmarks (KDBench) - open source measurement-based benchmarks

Y. Sorel, I. Hawila, L. Cucu-Grosjean, M. Mezouak, H. Clarke and S. Ben Amor

StatInf and INRIA

Paris, France

firstname.lastname@namecompanyoruniv.fr

***Abstract***—The real-time systems community is facing the lack of benchmarks adapted for probabilistic and statistical analyses. In this paper we provide three classes of benchmarks: (i) benchmarks for statistical worst-case execution time estimators, (ii) benchmarks for probabilistic schedulability analyses of independent tasks and (iii) benchmarks for probabilistic schedulability analysis of dependent tasks. The proposed benchmarks are obtained by monitoring the evolution of different parameters during the execution of open-source drone autopilot programs on a microcontroller. Measurements are both software-based and hardware-based.

***Index Terms***—open source, measurement-based, benchmarks

## I. INTRODUCTION

In this demo, we propose the presentation of a measurement-based benchmark that contains the following items:

* a set of programs $A_i$ with their source code and binary code as well as the description of the relations between these programs, i.e., precedence constraints and/or periodicity of execution. In our case, the programs are PX4-RT programs of an autopilot (see more details in Section II);

* a processor description $p$. In our case, the processor is the ARM Cortex-M4 (see more details in Section II);

* a description of a measurement protocol $\mathcal{M}$ with 4 ordered sequences of execution times, response times, input variables and output variables;

* a description of the operating system (if any) as well as the scheduling algorithm implemented within the operating system scheduler and the associated protocol of sharing data among programs;

* WCET estimation obtained with the commercial tool RocqStat as an input for the task parameters for the measurement-based dependent task benchmark.

## II. KDBENCH ENVIRONMENT

In this section we present an overview of the KDBench environment considered to collect data during the simulated execution of the KDBench programs, PX4-RT programs [1]. The PX4-RT programs are real-time versions of the PX4 autopilot, which is an open source flight control software [4]. PX4 runs on top of NuttX, a Unix-like OS, developed by Gregory Nutt [5]. We execute the autonomous flight mission with PX4-RT running on the single core ARM Cortex-M4 included in the Pixhawk board. Pixhawk is an independent

project that aims to provide high-end pilot control (autopilot) hardware at low or reasonable costs to academic, recreational and industrial user communities [2]. This is the most common board for autopilot PX4-based drones. We measure execution times of PX4-RT tasks inside a hardware-in-the-loop (HitL) simulation in order to avoid extremely time consuming real flight missions. We understand by HitL that the execution of the benchmark is done on a microcontroller while sensors and actuators of a CPS, as well as its physical environment, are simulated. The simulated part is based on the widely-used simulator for drones, named Gazebo [6] (see Figure 1), which is a powerful 3D simulation environment for autonomous robots that is particularly suitable for testing object-avoidance and computer vision.

<figure><img src="image_2.png" /><figcaption>Fig. 1: PX4-RT data dependent real-time tasks graph and the simulation environment</figcaption></figure>

The real-time tasks of PX4-RT are scheduled by the NuttX scheduler, according to a fixed-priority scheduling with FIFO tie breaking, i.e., when several modules have the same priority they are executed FIFO, otherwise the module with the highest priority is executed.

More precisely, we have 9 real-time tasks of PX4-RT and their list is detailed in Table I together with their periods of activation. In the first line of Table I, tasks are ordered according to their priority from the highest priority (on the left of the table) to the lowest priority (on the right of the table). For instance, the task Sensors has the highest priority while Commander has the lowest priority. The last 4 lines of Table I contain the periods of activation of each task within
the 4 considered measuring flight scenarios, where the periods $T_i$ are indexed according to the scenario number i.

<table><caption>TABLE I: List of real-time control tasks with their associated periods</caption><thead><tr><th>Periods</th><th>Sensors</th><th>EKF2</th><th>Rate</th><th>Attitude</th><th>Position</th><th>Flight manager</th><th>Hover thrust</th><th>Navigator</th><th>Commander</th></tr></thead><tbody><tr><td>T<sub>1</sub> (ms)</td><td>10</td><td>10</td><td>15</td><td>15</td><td>15</td><td>15</td><td>15</td><td>25</td><td>50</td></tr><tr><td>T<sub>2</sub> (ms)</td><td>4</td><td>4</td><td>4</td><td>4</td><td>4</td><td>4</td><td>4</td><td>4</td><td>4</td></tr><tr><td>T<sub>3</sub> (ms)</td><td>8</td><td>8</td><td>8</td><td>8</td><td>8</td><td>8</td><td>8</td><td>8</td><td>8</td></tr><tr><td>T<sub>4</sub> (ms)</td><td>12</td><td>12</td><td>12</td><td>12</td><td>12</td><td>12</td><td>12</td><td>12</td><td>12</td></tr></tbody></table>

### III. KDBENCH DATA

In this section we present the flight missions used for collecting the KDBench data. We consider 3 flight missions, where each flight mission is done with periods defined by the 4 scenarios. This means that we obtain 12 sets of collected data, 4 sets for each flight mission. Moreover, we present the data that are extracted from the benchmark after the flight mission completes.

We collect data from three different flight missions that we generate by using the QGroundControl [3] software which provides mission planning for MAVlink enabled drones. The three missions are described as follows:

(i) during the first mission *short_mission*, the drone follows a simple line trajectory and the duration of the flight is 23s,

(ii) for the second *medium_mission* we choose a mission with a sudden change of the drone orientation, in addition to other waypoints where we change the altitude. The duration of this flight mission is around 43s,

(iii) the third *long_mission* has more waypoints than the second flight mission and its duration is 2min 48s.

### IV. ACKNOWLEDGMENTS

There results are partially funded by FR INRIA ADT KDBench and the results are obtained during the stay of M. Mezouak and H. Clarke within the Kopernic team at INRIA, Paris.

### REFERENCES

[1] Marwan Wehaiba el Khazen, Kevin Zagalo, Hadrien Clarke, Mehdi Mezouak, Yasmina Abdeddaïm, Avner Bar-Hen, Slim Ben-Amor, Rihab Bennour, Adriana Gogonel, Kossivi Kougblenou, Yves Sorel, and Liliana Cucu-Grosjean. Work in progress: KDBench - towards open source benchmarks for measurement-based multicore WCET estimators. In 28th IEEE Real-Time and Embedded Technology and Applications Symposium, RTAS, pages 309–312. IEEE, 2022.

[2] DroneCode Project Inc. Pixhawk. https://pixhawk.org/.

[3] DroneCode Project Inc. Qgroundcontrol software. http://qgroundcontrol.com.

[4] L. Meier. PX4 Development Guide. https://dev.px4.io/en/.

[5] G. Nuttx. Nuttx operating system, user's manual. http://www.nuttx.org/doku.php?id=documentation:userguide.

[6] Open Robotics. Gazebo simulator. http://gazebosim.org.