DOI: 10.17617/3.uqgk4a
title: Magma Experiment for Code/Bug-based Coverage Benchmarking

description: &lt;h1&gt;Magma Experiment for Code- and Bug-based Coverage Benchmarking&lt;/h1&gt;

&lt;h2&gt;Overview&lt;/h2&gt;
&lt;p&gt;
This dataset contains the &lt;b&gt;raw experimental artifacts&lt;/b&gt; for the Magma portion of our study on the
concordance (split-half reliability) of &lt;b&gt;coverage-based&lt;/b&gt; and &lt;b&gt;bug-based&lt;/b&gt; fuzzer benchmarking procedures. [oai_citation:0‡FSE2026.pdf](sediment://file_000000006394720c8478a4c3a5cd9f3e)
&lt;/p&gt;

&lt;p&gt;
The artifacts are intentionally released at a low level (queues, logs, and bug information) so that others can
&lt;b&gt;reproduce&lt;/b&gt;, &lt;b&gt;audit&lt;/b&gt;, and &lt;b&gt;recompute&lt;/b&gt; outcomes under alternative analysis choices.
&lt;/p&gt;

&lt;p&gt;
Note: This dataset is &lt;b&gt;not linked directly from the paper&lt;/b&gt;. The paper links to a companion code repository, and
that repository links to this dataset.
&lt;/p&gt;

&lt;hr/&gt;

&lt;h2&gt;Experimental Setup (Summary)&lt;/h2&gt;
&lt;ul&gt;
 &lt;li&gt;&lt;b&gt;Benchmark suite:&lt;/b&gt; Magma v1.2.0&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Benchmarks used:&lt;/b&gt; 18 fuzz drivers (benchmarks) across multiple programs&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Fuzzers:&lt;/b&gt; 8 fuzzers&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Trials:&lt;/b&gt; 20 independent trials per (fuzzer, benchmark) combination (subject to supported combinations)&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Campaign length:&lt;/b&gt; 23 hours per trial&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Bug evaluation:&lt;/b&gt; Magma canary-based ground truth; we count &lt;b&gt;triggered bugs&lt;/b&gt;&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Coverage evaluation:&lt;/b&gt; branch coverage (LLVM tooling)&lt;/li&gt;
&lt;/ul&gt;

&lt;hr/&gt;

&lt;h2&gt;What This Dataset Contains&lt;/h2&gt;
&lt;p&gt;
This dataset includes the raw outputs needed to compute bug-based results directly and to compute coverage-based
results via offline replay:
&lt;/p&gt;

&lt;ul&gt;
 &lt;li&gt;&lt;b&gt;Execution queues&lt;/b&gt; produced during fuzzing campaigns (per fuzzer, benchmark, trial)&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Raw bug information&lt;/b&gt; from Magma’s canary-based bug reporting (including logs/outputs per run)&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Per-run metadata&lt;/b&gt; required to group results and reproduce the analysis workflow&lt;/li&gt;
&lt;/ul&gt;

&lt;hr/&gt;

&lt;h2&gt;Important: How Coverage Results Are Obtained&lt;/h2&gt;
&lt;p&gt;
&lt;b&gt;Coverage is not precomputed in this dataset.&lt;/b&gt; This dataset includes the execution queues and other artifacts,
but &lt;b&gt;coverage values must be derived&lt;/b&gt; using the companion code repository.
&lt;/p&gt;

&lt;p&gt;
To extract coverage:
&lt;/p&gt;
&lt;ol&gt;
 &lt;li&gt;Obtain the companion code repository referenced by the paper.&lt;/li&gt;
 &lt;li&gt;Configure it to point to this dataset on disk.&lt;/li&gt;
 &lt;li&gt;Run the provided pipeline to replay the execution queues and compute &lt;b&gt;branch coverage&lt;/b&gt; via LLVM tooling.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;
If you download only this dataset without the companion code, you can still analyze &lt;b&gt;bug-based&lt;/b&gt; outcomes,
but you &lt;b&gt;cannot&lt;/b&gt; reproduce the paper’s coverage-based results.
&lt;/p&gt;

&lt;hr/&gt;

&lt;h2&gt;Directory Layout&lt;/h2&gt;
&lt;p&gt;
The dataset is organized hierarchically by &lt;b&gt;fuzzer&lt;/b&gt;, &lt;b&gt;benchmark&lt;/b&gt;, and &lt;b&gt;trial&lt;/b&gt;.
The exact naming is intended to match the assumptions of the companion analysis scripts.
&lt;/p&gt;

&lt;p&gt;
Conceptually:
&lt;/p&gt;
&lt;pre&gt;
&lt;fuzzer&gt;/&lt;benchmark&gt;/&lt;trial&gt;/...
&lt;/pre&gt;

&lt;p&gt;
Each trial directory corresponds to one 23-hour campaign run and contains the artifacts produced by that run
(e.g., queues, logs, bug-triggering information, and auxiliary metadata).
&lt;/p&gt;

&lt;hr/&gt;

&lt;h2&gt;File Count&lt;/h2&gt;
&lt;p&gt;
At release time, this dataset contains exactly &lt;b&gt;2,860 files&lt;/b&gt;.
&lt;/p&gt;

&lt;hr/&gt;

&lt;h2&gt;Known Omissions / Unsupported Combinations&lt;/h2&gt;
&lt;ul&gt;
 &lt;li&gt;&lt;b&gt;SymCC × exif&lt;/b&gt; is not present because SymCC does not compile on the &lt;b&gt;exif&lt;/b&gt; benchmark; therefore, no such run exists.&lt;/li&gt;
&lt;/ul&gt;

&lt;hr/&gt;

&lt;h2&gt;Data Completeness Note (Minimal)&lt;/h2&gt;
&lt;p&gt;
A small number of runs (seven) were re-executed after a temporary server failure to complete the expected trial matrix, but were excluded from the paper.
The released dataset is intended to be &lt;b&gt;complete&lt;/b&gt; for the supported fuzzer–benchmark combinations.
&lt;/p&gt;

&lt;hr/&gt;

&lt;h2&gt;Relationship to Paper and Code&lt;/h2&gt;
&lt;ul&gt;
 &lt;li&gt;&lt;b&gt;Paper:&lt;/b&gt; defines the research questions, metrics, and statistical analysis. [oai_citation:1‡FSE2026.pdf](sediment://file_000000006394720c8478a4c3a5cd9f3e)&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;Companion code repository:&lt;/b&gt; performs coverage extraction (queue replay), aggregation, ranking, and concordance computations.&lt;/li&gt;
 &lt;li&gt;&lt;b&gt;This dataset:&lt;/b&gt; provides the raw artifacts consumed by that code.&lt;/li&gt;
&lt;/ul&gt;

&lt;hr/&gt;

&lt;h2&gt;Intended Use&lt;/h2&gt;
&lt;ul&gt;
 &lt;li&gt;Reproduction of bug-based outcomes and reanalysis of trial variability&lt;/li&gt;
 &lt;li&gt;Recomputation of coverage-based outcomes via replay using the companion code&lt;/li&gt;
 &lt;li&gt;Alternative analyses of bug and code coverage.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;
This dataset is &lt;b&gt;not&lt;/b&gt; a pre-aggregated results table; it is raw experimental output.
&lt;/p&gt;

&lt;hr/&gt;

&lt;h2&gt;License&lt;/h2&gt;
&lt;p&gt;
This dataset is released under &lt;b&gt;CC BY 4.0&lt;/b&gt;.
&lt;/p&gt;

&lt;hr/&gt;

&lt;h2&gt;Citation&lt;/h2&gt;
&lt;p&gt;
Please cite this dataset via its DOI (doi:10.17617/3.UQGK4A). When referencing the methodology or results, also cite the accompanying paper, &lt;i&gt;In Bugs We Trust? On Measuring the Randomness of a Fuzzer Benchmarking Outcome&lt;i&gt;
&lt;/p&gt;