Sebastian Gallenmüller, Georg Carle
Yuri Demchenko

SLICES-RI Plain Orchestrating System (pos)Reproducible Experiment Workflows by Design

Our solution: plain orchestrating service [3]Our goals:
How to limit the effort spent on reproducibility?
▶ Reduce effort for researchers▶ Integrate reproducibility into experiment design: Automate entire experiment

How to create robust, reproducible experiments?
▶ Documentation of all relevant parameters: Well-structured experiment workflow
serving as experiment documentation▶ Live images (clean slate on reboot): Experiments become repeatable(setup, execution, evaluation)▶ Automate the documentation of experiments

Achieving Repeatability▶ Full experiment automation

Achieving Reproducibility▶ Sharing access to testbed▶ Other researchers (re-)run experiment: Experiments become reproducible

Measurement Tools

DuT

Controller

LoadGenExperiment
Local
Vars

Global
Vars

Setup

Run N
Run II
Run I

Loop
Vars N
Loop
Vars II
Loop
Vars I

Task
Description

Setup

Local
Vars

Task
Description
Task
Description
Task
Description

Results N
Results II
Results I

Evaluation
ScriptParameters
Result Data

Publication

pos experiment workflow

Reproducibility Across Heterogeneous Testbeds

MoonGen [2] packet generator for 100 Gbit/s Ethernet and beyond▶ High performance based on DPDK– Supports bandwidths of ≥100 Gbit/s– Over 100 million packets per second▶ Hardware timestamping functionality– Utilizing off-the-shelf NICs
– Timestamps with ns-resolution– High accuracy and precision

Evaluation
Measurement Phase Setup Phase
Phase

Reproducibility by Design

Host Controller
pos Controller
LoadGenDuT

pos experiment

MoonGenhost experiment▶ Limitation: Experiment workflow depends on pos controller▶ Flexible configuration by Lua user scripts– Support for new protocols can be added easily– Modification of packets before sending via user-defined scripts: Solution: Export pos testbed controller to other testbeds▶ MoonGen has been used for 350+ scientific publications since 2015

▶ Ongoing work:– Porting pos to other testbeds such as CloudLab or Chameleon

SLICES & EOSC [1]1. Create experiment in host testbed
2. Deploy pos inside the experiment of host testbed
3. Deploy pos workflow inside the pos environment

SLICES-RI▶ EU initiative with 15 member countries

Offering SLICES services to EOSC
▶ Providing state-of-the-art digital research infrastructure (5G/6G networks, IoT, cloud infrastructures)▶ Powerful experiment APIs to utilize different infrastuctures▶ Services such as reproducible experiment workflows
Embedding SLICES into EOSC ecosystem▶ Application of EOSC-supported principles and methodologies (e.g.,RO-Crate)▶ FAIR Research Lifecycle Management (cf. RELIANCE project)▶ Goal: Creation of a large-scale digital
research infrastructure▶ Shared pos-inspired API:– Cross-reproduction of experiments
on participating testbeds– Establishing a widely-accepted
template for reproducible research▶ Contact us:– Sebastian Gallenmüller
<gallenmu@net.in.tum.de>– Yuri Demchenko<y.demchenko@uva.nl>[1] Y. Demchenko, S. Gallenmüller, S. Fdida, P. Andreou, C. Crettaz, and M. Kirkeng. Experimental Research Reproducibility and Experiment Workflow
Management. In 15th International Conference on COMmunication Systems & NETworkS, COMSNETS 2023, Bangalore, India, 2023.
[2] P. Emmerich, S. Gallenmüller, D. Raumer, F. Wohlfart, and G. Carle. MoonGen: A Scriptable High-Speed Packet Generator. In Internet Measurement
Conference 2015 (IMC’15), Tokyo, Japan, Oct. 2015.
[3] S. Gallenmüller, D. Scholz, H. Stubbe, and G. Carle. The pos Framework: A Methodology and Toolchain for Reproducible Network Experiments. In
Conference on emerging Networking EXperiments and Technologies (CoNEXT), Munich, Germany, 2021.
Funded by the European Union’s Horizon 2020 research and innovation programme, grant agreement No 101008468 and 101079774 (SLICES-SC, SLICES-PP).