DOI: 10.5281/zenodo.17639772
title: Benchmarking the Overhead of Distributed Tracing Agents - Dataset

description: Contents

This dataset contains the measurement results of our overhead measurement study "Benchmarking the Overhead of Distributed Tracing Agents" (currently under review). It will be de-anonymized after acceptance.

By using this dataset, you can





reproduce our measurements or try to repeat the measurements on different infrastructure (different CPU, OS, JVM version, …) in order to compare the results, or




analyse our measurement results (mainly .csv files) in depth.



It contains



The state of the microbenchmark MooBench (commit e475465), the state with async profiler integrated (moobench-asprof.tar) and async profiler itself (async-profiler-4.1-linux-x64.tar.gz)

The default parameter measurement results (default-results.tar.xz)

The scalability experiments measurement results (exp-results-elasticapm-java.tar, exp-results-inspectIT-java.tar, exp-results-Kieker-java.tar, exp-results-OpenTelemetry-java.tar, exp-results-pinpoint-java.tar, exp-results-Scouter-java.tar, exp-results-Skywalking-java.tar)

The flamegraph of the async-profiler analysis (flamegraphs.tar)


Reproduction

We executed three main experiments: Single user execution, scalability analysis, and time consumption analysis. The reproduction of these results is described in the following.

Prerequisites

The execution requires



A Linux system, capable of running bash (our experiments were executed on Rocky Linux 9.6, other distributions are expected to work)

A recent JDK installation (we used OpenJDK 21.0.8, OpenJDK 17 and above are expected to work)

Docker 28.0.0 or newer for benchmarking Elastic APM, Skywalking, and Pinpoint (other frameworks can still be benchmarked without docker)


Before starting the experiments, do the following:



Extract MooBench: tar -xvf moobench.tar.xz

Build MooBench: cd moobench && ./setup.sh


Single User Execution

To run the single user execution, execute the following steps (in the moobench-folder):



cd frameworks

export NUM_OF_LOOPS=30; ./benchmarkAll.sh (Number of loops can be set differently; our experiments were executed with 30 loopss

To obtain the table from the paper, execute cd ../analysis/; ./getStatistics.sh ../frameworks/


The process is expected to take ~2 days on a recent machine. It can be sped up specifying a list of frameworks (e.g., ./benchmarkAll.sh inspectIT-java OpenTelemetry-java Kieker-java), or by reducing the number of loops.

The file default-results.tar.xz contains our results of this experiment.

Scalability Experiment

To run the scalability benchmarks, execute the following steps (in the moobench-folder):



cd frameworks

export NUM_OF_LOOPS=30; ./runBenchmarkAll.sh ALL DEPTH (Number of loops can be adapted; instead of all, *one* framework name can be specified, e.g., inspectit-java) 


The process is expected to take multiple days per framework, so if it is executed for all 7 frameworks, the experiments need to be distributed.

The files exp-results-*.tar.xz contain our results of this experiments.

Time Consumption Analysis

To execute the experiments, execute the following steps:



Extract the adapted moobench version: tar -xvf moobench-asprof.tar && cd moobench && ./setup.sh

Extract async profiler (we used version 4.1; it is expected that this also works with other versions, but for archival purposes, we uploaded version 4.1 here): tar -xvf async-profiler-4.1-linux-x64.tar.gz and export the path: export ASYNC_PROFILER_HOME=$(pwd)/async-profiler-4.1-linux-x64

Run the benchmark:cd moobench/framework && ./runAllWithAsyncProfiler.sh

In every framework folder, individual results can be plotted, e.g., for Kieker using $ASYNC_PROFILER_HOME/bin/jfrconv flamegraph_1_10_5.collapsed flamegraph.html

To get the mapping to time consumption sources, execute inside the moobench-folder: cd tools/compile-results/build/distributions/ && tar -xvf compile-results.tar && cd compile-results/lib; java -cp "$(printf '%s:' *.jar)" moobench.tools.flamegraphs.ClassifyNodes $DATAFOLDER/flamegraph_X.collapsed where $DATAFOLDER/flamegraph_X.collapsed points to the file you want to analyze; please note that performance measurements will differ depending on the speed of your hard disk etc., so while the general tendendency is expected the same, you will most likely see gradually different values.


The file flamegraphs.tar.xz contain our results.