# PyroBuildS: Speeding up the exploration of large configuration spaces with incremental build

Georges Aaron Randrianaina, Djamel Eddine Khelladi, Olivier Zendra, Mathieu Acher

► To cite this version:

Georges Aaron Randrianaina, Djamel Eddine Khelladi, Olivier Zendra, Mathieu Acher. PyroBuildS: Speeding up the exploration of large configuration spaces with incremental build. Journal of Systems and Software, 2026, 231, pp.112592. 10.1016/j.jss.2025.112592. hal-05213625

HAL Id: hal-05213625

https://hal.science/hal-05213625v1

Submitted on 18 Aug 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# PyroBuildS: Speeding Up the Exploration of Large Configuration Spaces with Incremental Build

Georges Aaron Randrianaina<sup>a</sup>, Djamel Eddine Khelladi<sup>a</sup>, Olivier Zendra<sup>a</sup>, Mathieu Acher<sup>a,b</sup>

<sup>a</sup>Univ Rennes, CNRS, Inria, IRISA - UMR 6074, F-35000 Rennes, France

<sup>b</sup>Institut Universitaire de France (IUF), France

## Abstract

Software developers are acutely aware that software build is an essential but resource-intensive step in any software development process, all the more when building large and/or highly configurable systems, whose vast number of configuration options leads to an explosion in the number of variants to build and evaluate. A potential approach to speed up the builds of multiple configurations is to do *incremental build*, i.e., to not clean the build environment and reuse previous builds when building a new configuration. Previous exploratory studies showed some benefits and limitations of incremental build, but mainly on small configurable software systems and on a limited set of close configurations. However, for *large configuration spaces*, little is known whether the large distance across configurations impacts the correctness and efficiency of incremental build.

This paper presents **PyroBuildS**, a new approach to speed up incremental builds while keeping reproducibility, featuring a configuration variation operator parameterized by two deny lists of problematic options and a mutation size (diversity).

We evaluate **PyroBuildS** through an empirical study on three large complex configurable systems, namely Linux, BusyBox, and ToyBox, with respectively 18637, 1078, 330 configuration options. We first show that for all configurations **PyroBuildS** produces the exact same binaries as a clean

Email addresses: georges-aaron.randrianaina@irisa.fr (Georges Aaron Randrianaina), djamel-eddine.khelladi@irisa.fr (Djamel Eddine Khelladi), olivier.zendra@inria.fr (Olivier Zendra), mathieu.acher@irisa.fr (Mathieu Acher)

Preprint submitted to Journal of Systems and Software

August 18, 2025
build, except for Linux with some non-reproducible random configurations. We identify the reasons why incremental build speeds up or slows down the build of large configuration spaces – a knowledge that can be integrated into **PyroBuildS**. Incremental build systematically pays off, since problematic options are avoided in the first place – something only **PyroBuildS** does. We also show that a naive use of incremental build on random Linux configurations backfires, taking more time than clean builds. Thus, **PyroBuildS** controls diversity to avoid too many differences across configurations to perform efficient incremental builds.

Thanks to its ability to operate over non-problematic options and close enough configurations, **PyroBuildS** significantly speeds up the exploration of large configuration spaces, with a gain in build time from 16% to 22% in all three systems with mutated configurations. Finally, with random configurations, **PyroBuildS** also speeds up the build time from 15% to 20% for **ToyBox** and **BusyBox**.

*Keywords:* Software Product Line, Highly-configurable Systems, Incremental Build, Build Systems

# 1. Introduction

Building software is a crucial activity for project developers and maintainers, in which various artifacts are assembled, compiled, tested, then deployed. The emergence of continuous integration (CI) has accelerated this trend with the integration of build services into major code platforms (e.g., GitHub, GitLab). CI aims at continuously ensuring the quality of software products, both in terms of functional and non-functional properties (e.g., security, execution time).

Software configurations add further complexity to software building. Different artifact variants may be assembled (e.g., due to conditional compilation directives `#ifdef-s` in the source code). Different external libraries may be compiled and integrated as well. The way build is performed can also change, e.g., with different compiler flags. Developers and maintainers want to ensure that throughout the project evolution all or at least a subset of software configurations build well. Since most of today's software is configurable in order to fit constraints, functional and performance requirements, many organizations build different software configurations of their projects. For example, JHipster builds dozens of configurations at each commit, involving
different technologies (Docker, Maven, grunt, etc.) Halin et al. (2018).

Highly configurable systems need intensive testing, especially those used
in critical systems. For example, Linux version 6.1 has over 20,000 con-
figuration options, each of which can be either enabled, enabled as module
or disabled by the user, shaping the generated executables. Assuming that
each of these options can take only two values, enabled or disabled, leads to
2^20000 possible configurations. The actual number is lower, due to constraints
among features, but remains huge. It is thus practically infeasible to test all
the possible configurations, but only a small subset. Multiple initiatives ex-
ist that intensively build Linux kernel configurations: KernelCI ker (2021),
Intel 0-day zer (2021); Kerrisk (2012), Tuxmake Team (2023); Rue (2021),
TuxML Martin et al. (2022); Acher et al. (2019a). However, building soft-
ware is increasingly complex and costly in terms of time and resources, such
as energy Edge (2020); Beller et al. (2017a); Lebeuf et al. (2018a); Hilton
et al. (2017a). For instance, the TuxML project took more than 15K hours
of computation to build 90K+ random configurations and derive an accurate
prediction model for size prediction Martin et al. (2022). Building multi-
ple configurations has various applications Lyu et al. (2024); Randrianaina
et al. (2024). First, ensuring that the configuration builds without compi-
lation errors or warnings<fnref n="1" />. Second, bug finding and automated testing are
major use-cases. Typical bug finding involves tools such as fuzzers that op-
erate over a kernel binary, built out of the code source Hasanov et al. (2025).
For testing, KernelCI runs the built kernels on real hardware to run test
suites and regression tests. Third, performance learning involves finding the
best configuration, predicting performance, or identifying important options
Alves Pereira et al. (2021); Jamshidi et al. (2017); Guo et al. (2013); Lesoil
et al. (2021); Jamshidi et al. (2018). Both use cases require to observe and
measure properties out of numerous builds. Fourth, testing patches involves
finding configurations that cover specific commits and then build it Yıldıran
et al. (2024); Lawall and Muller (2017a). Finally, debugging involves un-
derstanding the source of errors and identifying responsible options Yıldıran
et al. (2024); Melo et al. (2016); Mordahl et al. (2019); Acher et al. (2019a);
Halin et al. (2018). All scenarios imply building multiple configurations,

<footnote n="1">¹Linus Torvalds enables `-Werror` by default for all kernel builds https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=3fe617ccafd6f5bb33c2391d6f4eeb41c1fd0151</footnote>
hence significant computational costs.

*Reducing the cost of configuration builds* is an important topic that has only recently been addressed through the concept of *incremental build*. Although incremental compilation and build have lead to much research Cao et al. (2017a,b); Feldman (1979a); Tichy (1986); Schwanke and Kaiser (1988); Mitchell (2012); baz ([n. d.]); Erdweg et al. (2015); Konat et al. (2018); Hammer et al. (2015); Maudoux and Mens (2018); McIntosh (2024), most of the works consider the classical scenario of editing and *committing a few changes*, as part of software maintenance and evolution. The premise is that rather than building from scratch and cleaning the build’s artifacts, a configuration can be built from an already completed configuration build, incrementally on top of previous builds. It aims to reuse artifacts of previous configurations’ build, thus saving computation resources.

As far as we know, only three studies Randrianaina et al. (2022b,a); Lyu et al. (2024) explored the benefits of incremental build of software *configurations*. Randrianaina et al. Randrianaina et al. (2022b) investigated incremental build on small configurable systems with only 16 to 100 options and a few configurations. Randrianaina et al. Randrianaina et al. (2022a) showed its limits on only a few examples of a large configurable system (Linux). Lyu et al. Lyu et al. (2024) propose an approach to determine an efficient ordering for incremental builds of multiple configurations, but not at the scale of Linux, Toybox and Busybox. Modifying a configuration can cause multiple changes in the artifacts, especially for systems with numerous options (thus configurations). Thus, building one configuration after another challenges the effectiveness of an incremental strategy. Incremental build is first expected to produce a binary that is *correct and consistent* with a traditional build (i.e., from a clean copy of the codebase). Then, incremental build should reduce the amount of time, by not re-building artifacts that have already been produced. Ideally, incremental build should pay off for all configurations. When only one or two options are changed incremental build is expected to be effective, but prior exploratory works Randrianaina et al. (2022b,a) suggest it is not always the case, due to complex interplay across options, artifacts, and the build system. When dozens or hundreds of options are modified, the differences between configurations cause significant changes and hinder build artifacts reusability, with studies Randrianaina et al. (2022b,a) reporting mixed, negative results on reproducibility and speed-up. Why incremental build is effective or not is unclear, especially for large configuration spaces that exacerbate the reported issues.
Our goal is to fill this gap. First, we design a new approach called PyroBuildS, able to reduce computational cost without trading diversity of configurations and correctness. An important feature of PyroBuildS is a mutation operator that chooses the next configuration to incrementally build. This operator takes as input a deny list of options hindering reproducibility, a deny list of options hindering cost reduction, and a parameter to control the number of mutations to perform, hence diversity. The design of PyroBuildS is directly informed by our empirical findings, which revealed specific configuration options and conditions under which incremental builds are ineffective or counterproductive. Second, we run an in-depth empirical study of incremental build on three large complex configurable systems, namely Linux, BusyBox, and Toybox, with respectively 18637, 1078, 330 configuration options. We rely on random sampling to generate random configurations and (randomly) mutated options. As state-of-the-art sampling methods did not scale for our experiments (see Section 4.4) we rely on the randconfig utility, widely used as part of Kconfig-based systems. We build numerous default and random configurations, to serve as bases from which we incrementally build sets of random or mutated configurations.

We first observe that incrementally building the random configurations significantly reduces build time for BusyBox, and Toybox, with an average gain of 15% to 20%; incremental build thus pays off as such. However, we show that some options can be detrimental to build time; PyroBuildS is able to avoid them. For Linux, however, incremental build on random configurations just backfires, taking more time than with clean builds.

When incrementally building mutated configurations, we observe a systematic gain in execution time from 16% to 22%, and a diversity in terms of options that are changed between configurations.

Overall, PyroBuildS significantly reduces large configuration spaces exploration times, provided adequate mutation strategies are used to operate only with non-problematic options and close enough configurations.

To summarize, our contributions are as follows:

* An empirical study that investigates the general challenges of incrementally building large, complex configurable systems. A key insight is that incremental builds can sometimes be more costly than traditional builds – in particular, specific problematic options hinder incremental build effectiveness and randomly selected Linux configurations often result in worse performance when built incrementally. These findings
provide a foundation for the design of PyroBuildS.

* **PyroBuildS**, an approach and tool designed to incrementally build sets of configurations – either randomly generated or systematically mutated – while controlling the generation process to favor cases where incremental builds are effective. Like fireworks, it launches base configurations as "rockets" and explores their "sparks" (mutated configurations) with the ability to switch bases across mutation rounds.

* A new dataset of 12K incrementally build random and mutated configurations, built and repeated three times making a total of 36k builds.

* Systematic gains up to 20% when incrementally building random configurations, except on Linux where random configurations can augment incremental build time.

* Significant and systematic gains up to 22% when incrementally building mutated configurations, while diversifying the size of diffs between configurations.

* Identification of new options hampering incremental build. This identification is the first to explain why incremental build pays off or not, and is also actionable as part of PyroBuildS to make it effective.

* Our replication package, described in Section 9

This paper is organised as follows. Section 2 provides background on MAKE, on the Linux kernel build system and on incremental build, and outlines the motivation for our work. Section 3 explains our PyroBuildS approach. Section 4 presents our experimental settings. Section 5 reports and analyses results. Section 6 discusses threats to validity. Related work is analyzed in Section 7. Section 8 concludes and outlines future work directions.

## 2. Background and Motivation

This section provides some background on the MAKE build system, Kconfig language to specify configurations, and describes the promises and challenges of incremental build.
## 2.1. Build and MAKE

A build system specifies how to translate source code into a deliverable. The developer typically declares: 1) rules on how to compile a source code, 2) which libraries to include, and 3) how to link them all together in order to obtain an executable.

MAKE Feldman (1979b) is one of the most popular build systems Mcintosh et al. (2015). Build specifications are declared in *Makefiles* that are spread out in the subdirectories of a project. Makefiles contain 1) (leaf-) rules whose trigger is a filename to generate a library or object/executable file, with its "recipe" depending on source files 2) rules that trigger other rules. To build a project, MAKE starts by reading the *Makefile* at the root of the project, then recursively goes into subdirectories. On its way, MAKE builds a dependency graph of the relations between rules and spawns a build process to satisfy each rule. On its first invocation, *Make* checks that all rules are satisfied. If a target file does not exist, *Make* produces it according to the specification. Otherwise, it checks the timestamps of its dependencies: if the target file is older than one of its dependencies, it must be recompiled. This process is propagated on the whole rule dependency graph. For a first build of a project, *Make* does a full build because none of the target files exist yet. Importantly, a second invocation of *Make* does nothing if no modification was done. Typically, a call to *Make* checks timestamps and triggers the rules impacted by the last file modification. If the specifications are correctly written Bezemer et al. (2017), this process does not rebuild the whole project every time, thus saving resources. This process of rebuilding only the minimum necessary parts of the build is called an *incremental build*. It has been well-known and widely used for decades.

## 2.2. Kconfig and Configuration

Kconfig is a language and tool used in several projects (e.g., Linux kernel, Busybox) to describe a software configuration options and their constraints. For instance, the Linux kernel offers more than 18.000 options, documented in several Kconfig files, spanning different aspects of the kernel (e.g., file system, security). Kconfig allows developers to choose options to build a custom kernel, either in an interface or by editing a `.config` file options' values. Additional tools exist to control the logical validity w.r.t. Kconfig constraints, or to generate (1) a default configuration (`defconfig`) with pre-set values, for instance to run a vanilla Linux kernel; (2) a random configuration with
<figure><img src="image_2.png" /><figcaption>Figure 1: Three configurations and various clean and incremental builds.</figcaption></figure>
random answers to options (*randconfig*), typically to test different configurations.

Figure 1 shows excerpts of three configurations $c_1, c_2, c_3$. Each configuration specifies values for all options. Boolean options take value 'y' (activated) or 'n' (deactivated), tristate options can take value 'y', 'n' or 'm' (for module); string, int, or hex options also exist. In a `.config` file, an option associated with no value means the option is deactivated. In Figure 1, the three configurations have commonalities and differences, e.g $c_1, c_2$ and $c_3$ all have 'y' value for `CONFIG_X86`, while `CONFIG_BRTFS_FS` and `CONFIG_BRTFS_DEBUG` are activated with 'y' values in $c_2$ but not in $c_1$.

Out of a `.config` file, the build process through `MAKE` is executed to eventually produce a binary. The executed steps vary depending on configuration and options values. The right part of Figure 1 shows the steps and compiled files (CC) for configurations $c_1, c_2, c_3$. Compared to $c_1$, $c_2$ contains additional steps, the two new activated options causing the build to compile three new files (shown in **bold**). Note that compilation step 'inode' also differs between $c_1$ and $c_2$: the option value changes in $c_2$ affect `inode.c`, hence causing the artifact to be rebuilt. Thus, a key takeaway is that *options can impact even shared targeted artifacts*. Other steps are identical, other options values being the same.

The two independent builds of $c_1$ and $c_2$ suggest a potential missed opportunity. Indeed, it is unfortunate that the build of $c_2$ does not reuse the already built artifacts of $c_1$. Some steps are redundantly re-executed, leading to a waste of time.

## 2.3. Incremental Build of Configurations

In the examples of Figure 1, the clean builds of $c_1$ and $c_2$ respectively take 2 and 3 minutes. But *incremental build* could improve this: instead of building from scratch out of a clean build environment, $c_2$ can be built out of $c_1$' artefacts. This would save some build steps, which `MAKE` is able of identifying based on file timestamps. In Figure 1, the incremental build of $c_2$ by reusing $c_1$ build results in a 21 seconds build time for $c_2$, compared to an initial 3 minutes with a clean build. However, the incremental build of $c_3$ reusing $c_2$ build takes 20 seconds more than a clean build.

An incremental build is thus characterized by two configurations: a previous configuration that has already been built – either with clean build (e.g., $c_1$) or with a previous incremental build (e.g., $c_2$), and the new configuration to incrementally build (e.g., $c_2$ or $c_3$) In Kconfig-based systems, incremental
build is trivial to implement, with two successive calls to MAKE, the second operating over a different .config file and avoiding re-compiling many artefacts provided by the first call. So *the main challenge of incremental build of configurations is not to operationalize the build and reuse, but to pick the next configuration worth building in an incremental way*.

## 2.4. Challenges and Motivation

A number of works and techniques exist to optimize the compilation (i.e., build) of systems *after commits* Cao et al. (2017b); Feldman (1979a); Tichy (1986); Schwanke and Kaiser (1988); Mitchell (2012); baz ([n. d.]); Erdweg et al. (2015); Konat et al. (2018); Hammer et al. (2015); Maudoux and Mens (2018). Building numerous varied configurations of large configurable systems is an important but expensive task. However, a much lower number of works pertain to optimising (i.e., decreasing the cost of) *massive builds of large configurations*. The most commonly deployed solution is quite costly, since it just consists in applying successive, independent clean builds, not decreasing resources' usage at all and missing the opportunity to reuse already build artefacts.

Incremental build with MAKE has the potential to speeding up build and saving resources. However, these benefits have only been shown on build after commits: changing a configuration and thus options' values can lead to significant files' differences. Furthermore, exploratory studies on configuration build on very small configurable systems with up to 40 configuration options, and showed mixed effective results Randrianaina et al. (2022b).

*Whether those benefits also occur on configuration build of large complex configurable systems with thousands of options thus remains largely unexplored. To the best of our knowledge, no thorough investigation exists on the use of incremental build on large complex configurable systems.* The next sections detail our approach and empirical study aiming at filling this gap.

# 3. PyroBuildS

This section details each step of our proposed approach, aiming for the following properties:

* **reproducible builds**: incremental builds should produce the same binaries as clean builds;
* **reduction of build time**: incremental builds should take less time than clean builds;

* **diversity**: incremental builds should allow exploring and handling configurations with different options values.

A number of challenges and open questions thus have to be addressed. The example of Figure 1 notably shows that the reduction in build time between $c_1$ and $c_2$ is effective, but not between $c_2$ and $c_3$. This raises the question of whether this is an outlier case or stems from a significant difference (*delta*) between $c_2$ and $c_3$, possibly caused by the use of *randconfig* for selecting $c_3$. We also need to investigate whether specific option values contribute to this negative effect. For example, options that affect the entire system, such as debugging options, can increase build time as they require recompiling numerous files.

The key idea behind **PyroBuildS** is to carefully control the variations applied to the configurations being built to ensure reproducibility and reduce build time. We design an automated strategy for selecting configurations that differ in option values but still offer benefits for incremental builds. We first develop a variation operator capable of mutating a given configuration based on three key parameters: a deny list of options that should not be changed, a deny list of options that should not be selected, and a change size that determines the number of options to change. Multiple configurations are then built, in an incremental way and clean way, to gather actionable insights. This data-based approach serves two purposes: (1) understanding the specifics of each subject system (e.g., determining if *randconfig* poses problems for a given system); and (2) parameterizing the change magnitude and the deny lists for these systems.

## 3.1. Overall Approach

**Fireworks Analogy.** Very much like fireworks, **PyroBuildS** starts from several *base configurations* (the "rockets") and generates *mutated configurations* (the "sparks") derived from each base. This enables exploring the configuration space with an efficient *incremental build* of the mutants, while preserving a good amount of diversity. Figure 2 illustrates this principle, using black circles for base configurations and white circles for their incremental variants.

**Local variations with incremental build.** Our goal is to explore a potentially vast configuration space while reducing computational cost when
<figure><img src="image_3.png" /><figcaption>Figure 2: Illustration of PyroBuildS’s “fireworks” approach. Black circles are base configurations (the “rockets”) and white circles are incremental mutations (the “sparks”). Dashed arrows indicate local variations, while solid arrows represent base changes in the exploration space.</figcaption></figure>

building multiple configurations. **PyroBuildS** leverages the idea that “close” configurations can be built incrementally. In such cases, the build system reuses prior build artifacts thus saving time by *not* rebuilding everything from scratch. To select "close" configurations, **PyroBuildS** employs a variation mechanism that *mutates* a given configuration repeatedly, creating a *chain* of local changes. After *m* such variations, the final configuration can differ substantially from the original base configuration because these small changes accumulate, akin to a spark traveling further and further from the rocket. This approach produces an interesting diversity of configurations while retaining the efficiency of incremental builds. While it would be possible to generate each variation directly from the base configuration, chaining allows more distant configurations to emerge through incremental changes. In other words, the *m*-th variation can accumulate all prior modifications, effectively reaching parts of the configuration space that a single-step mutation might not.

**Incremental Build vs. Clean Build.** Once PyroBuildS finishes one local chain, it selects another base configuration (typically randomly sampled)
to explore a more distant region of the configuration space. When transitioning from one base to another, incremental builds might still reuse some artifacts, but if the two configurations are truly distant, the savings could be marginal. In these cases, PyroBuildS may resort to a clean rebuild. Thus, PyroBuildS always benefits from incremental build within a local chain of variations, and optionally uses it across base configurations when it remains cost-effective. Moreover, this covers the scenario of building specific configurations required by developers (e.g., for a specific customer or hardware) that may not benefit from incremental build. In such cases, PyroBuildS can perform a clean build for the desired configuration and then use it as a new base for incremental exploration, ensuring both targeted support and broader efficiency gains.

**Instrumentation.** As stated, the PyroBuildS approach, shown at the bottom of Figure 3, aims at reducing build time when exploring a large configuration space. To do this, PyroBuildS uses *configuration variation* for configuration space exploration and *incremental build* for build time reduction.

PyroBuildS allows building *n* base configurations and a sequence of *m* configurations per base. The first configurations `.config1` ... `.confign` are built with a clean build, resulting in their binaries (i.e., Build artefacts) [1]. Then, PyroBuildS takes the previous configuration `.configj.i` and applies a variation operator to result in a new configuration `.configj.i+1` [2]. After that, the new configuration `.configj.i+1` is build incrementally on the binary build artefact of `.configj.i` [3]. This process is repeated for *m* configurations until `.config1.m` ... `.confign.m`.

At the core of PyroBuildS is the variation step [2], which can be done in several ways that we explore in our evaluation. The first is to randomly generate a new `.configj.i+1`. A second way is to control the changes to apply to a `.configj.i`; thus, we designed and implemented a variation step that mutates a given configuration based on different parameters, as shown at the top of Figure 3.

## 3.2. Parameters to control PyroBuildS exploration strategy

PyroBuildS accepts a number of parameters (see Figure 2 and 3):

* **Number of base configurations (n).** Multiple base “rockets” let PyroBuildS explore large portions of the space, potentially quite distant from one another.
<figure><img src="image_4.png" /><figcaption>Figure 3: Overall approach of PyroBuildS on *n* base configurations with *m* configurations incrementally built per base.</figcaption></figure>
* **Number of local variations (m).** Each base configuration spawns an incremental chain of *m* mutants. Over successive steps, these accumulated changes can reach new, distant points in the configuration space.

* **Base-configuration selection.** Bases may be chosen by random sampling (e.g. uniform), by t-wise combinatorial strategies, via specialized configuration tools (e.g. `randconfig` for the Linux kernel), or even with solver-based sampling methods, provided they scale adequately.

### 3.3. Parameters to control the mutations of configurations

PyroBuildS also features parameters to control in a fine way how a configuration `.config` is mutated. A first parameter is the change size, i.e., the number of options values to change. Once some options values have been modified, a ripple effect may cause other dependent options values to change as well due to Kconfig logical constraints. Thus the change size is an internal parameter that may eventually produce much more impacts on options (see the empirical evaluation for more details).

Two deny lists of options exist, to prevent some options from being selected for the next configuration. These are known from documentation, expert knowledge, pre-training, or even previous executions of PyroBuildS.

The *non-reproducible deny list* contains options known to cause non-reproducible builds. The Linux documentation<fnref n="2" /> provides information about such options. Randrianaina et al. Randrianaina et al. (2024) also proposed a learning-based approach to identify such options. Some options may also lead to non-reproducible builds only in the context of incremental build.

The *performance deny list* contains options causing abnormal increase in incremental build time compared to traditional build. These options may be identified with pre-training or warm-up executions of PyroBuildS that compare clean builds times with incremental builds times. From this data, we can identify modified options values that favor of hinder build time reduction. Options in the performance deny list can also come from expert knowledge, for example, Linux developers knowing about specific options that cause security issues, Debug or compiler-related options that crosscut the whole system thus increase build time, or options influencing the size of the built artefact Kuo et al. (2020b); Acher et al. (2019b, 2022); Ghavamnia et al. (2020); Kuo

<footnote n="2">²https://www.kernel.org/doc/html/v5.4/kbuild/reproducible-builds.html</footnote>
et al. (2020a), etc. While constructing such lists can be costly and incomplete, they can benefit PyroBuildS and pay off during the exploration with incremental builds.

### 3.4. Configuration variation step

The configuration variation step, shown in zoomed box [2] atop Figure 3, consists in taking an existing configuration `.config`, and parameters to produce a new configuration `.config`, only considers configuration options that have boolean values (or tristate in some cases) and are not in the deny lists. From these options, a random set of options equal to the size given for the variation is then selected (`O = {o₁, ..., oₙ}`). Value assignation is simple: an option that was enabled in the original configuration becomes disabled in the new configuration, and vice versa.

Then next, crucial, step is to verify the consistency of the new configuration [2.2]. In particular, it needs to include the dependencies for the selected options `O` to mutate. Indeed, configuration options have dependencies among them that must be solved to include all the configuration options required to integrate `O`. Thus PyroBuildS reuses an existing state-of-the-art solver<fnref n="3" /> to find the proper dependencies and apply them to the configuration. A valid configuration (i.e., pass) is obtained when the solver finds the right constraints to add all the chosen options to the configuration. Otherwise, the configuration being invalid (i.e., fail), the process of choosing the changed options has to be re-applied until a valid configuration is found.

Algorithm 1 below further details the variation step. Initially, the base configuration, denoted as `config`, is added to the set of configurations to be built (Line 2), starting with a clean build of the base. Its hash is then computed to ensure that this configuration is not proposed again in subsequent steps (Line 3). While the number of steps `N` (i.e., configurations to build) has not been reached, PyroBuildS applies a variation of size `V` to generate a new configuration (Line 6), after which its hash is computed (Line 7). If this hash is not present in the set of previously generated configurations, the new configuration and its corresponding hash are added to the set (Lines 8–11). The final sequence of configurations to be built is returned, where the first is a clean build and the rest are incrementally built on top of their immediate

<footnote n="3">³The quality and internal workings of this solver are out of the scope of this paper.</footnote>
predecessor.

<caption>**Algorithm 1:** Incremental build with PyroBuildS</caption>

**input:** Base configuration *config*, steps *N*, variation size *V*, deny list *D*

1 **begin**
2 &nbsp;&nbsp;&nbsp;&nbsp;*out_config* ← {*config*}
3 &nbsp;&nbsp;&nbsp;&nbsp;*hashes* ← {*checksum(*config*)}
4 &nbsp;&nbsp;&nbsp;&nbsp;*i* ← 1
5 &nbsp;&nbsp;&nbsp;&nbsp;**while** *i* ≤ *N* **do**
6 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;*new_config* ← *vary*(*config*, *V*, *D*)
7 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;*new_config_hash* ← *checksum*(*new_config*)
8 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**if** (*new_config_hash*) ∉ *hashes* **then**
9 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;*out_config* ← *out_config* + {*new_config*}
10 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;*hashes* ← *hashes* + {*new_config_hash*}
11 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;*i* ← *i* + 1
12 &nbsp;&nbsp;&nbsp;&nbsp;**return** *out_config*

Algorithm 2 describes how the mutated options are selected. It starts by drawing random options respecting the specified change size and deny lists (Line 4). Then, each option is mutated by either removing it or adding it in the configuration (Lines 5-9). Finally, the dependencies are resolved with a solver (Line 10). This process is repeated until a valid configuration is found with V mutations and their dependencies, which is always the case in practice.
<caption>**Algorithm 2:** Configuration variation</caption>

**input:** Configuration *config*, *V*, *deny list* *D*
**begin**
2 &nbsp;&nbsp;&nbsp;&nbsp;found ← false
3 &nbsp;&nbsp;&nbsp;&nbsp;**while** ¬*found* **do**
4 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;*candidates* ← *draw_random_options*(*V*, *D*)
5 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**foreach** *option* ∈ *candidates* **do**
6 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**if** *option* ∈ *config* **then**
7 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[ *config.remove*(*option*) ]
8 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**else**
9 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[ *config.add*(*option*) ]
10 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;found ← *DependencySolver*(*config*)
11 &nbsp;&nbsp;&nbsp;&nbsp;**return** *config*

We reused the SMT solver developed as part of Kmax Gazzillo (2017) and recently updated for testing Linux kernel patches Yıldıran et al. (2024). This solver encodes all Kconfig constraints, translating them into SMT constraints, enabling various operations, such as checking a configuration validity by verifying its satisfiability. Kmax also provides the krepair tool which adds options on demand, while getting as close as possible to a base configuration. In our approach, we input a list of options and their values and a base configuration into krepair to obtain a new configuration with the new options and their respective values. If the requested options require additional dependencies, the configuration is not satisfiable, so krepair uses the *unsat* core provided by the solver to resolve the dependencies and adds the missing options. More details on Kmax are provided in Gazzillo (2017).

Finally, since PyroBuildS aims to build *m* incremental configurations on top of *n* base configurations, we simply call *n* times the Algorithm 1.

## 4. Evaluation Methodology

This section describes our empirical study on the incremental build on large configurable system. It first presents the methodology we follow, then our dataset, research questions, and build process. Finally, it describes how we implemented incremental build within our PyroBuildS tool.
4.1. *Research Questions*

This section presents the research questions for empirical study. In partic-
ular, some of the research questions address aspects of incremental building
in general (RQ1, RQ4, and RQ5) and others address aspects specific to Py-
roBuildS (RQ2 and RQ3).

RQ₁ (Reproducibility) To what extent are PyroBuildS incremental builds re-producible? This research question quantifies the reproducibility of incremental build. We compare the output of an incremental build with a clean build. Prior works Randrianaina et al. (2022b) compare name and size of each symbol in the binary. We use a stricter comparison and a bit-by-bit diff between two binaries for Linux, Toybox, and Busybox.

RQ₂ (Cost reduction with mutated configurations) What are the performance gains of PyroBuildS with incremental builds of mutated configurations compared to clean builds? This research question quantifies the gains in build time with a sequence of mutated configurations, measuring the build time of an incremental build, then comparing it to the clean build time.

RQ₃ (Cost reduction with random configurations) What are the performance gains of PyroBuildS with incremental builds of random configurations compared to clean builds? This research question quantifies the gains in build time with a sequence of random configurations, by measuring the build time of an incremental build then comparing it to the clean build time.

RQ₄ (Problematic options for incremental build) Are there options negatively impacting the incremental build? This research question investigates in more details the cases where incremental builds result in longer build times, so as to identify the options causing this. We hypothesize the existence of such cases, but to what extent this is true for the three systems remain unclear.

RQ₅ (Diversity induced by incremental build) What is the diversity brought by incremental build in the configuration space exploration? This research question quantifies the built configurations diversity, in terms of activated options, source code files, and subsystems modules.
<table><caption>Table 1: Studied systems</caption><thead><tr><td>System</td><td>Version</td><td>LoC</td><td>#Options</td></tr></thead><tbody><tr><td>Linux</td><td>5.13</td><td>21605254</td><td>18637</td></tr><tr><td>Busybox</td><td>1.36.1</td><td>220304</td><td>1078</td></tr><tr><td>Toybox</td><td>0.8.11</td><td>90768</td><td>352</td></tr></tbody></table>

## 4.2. Dataset and Variables

This section presents our case studies, selected for being open-source with an accessible project, exhibiting compile-time configuration options, being popular, and most importantly being large configurable systems. We chose three configurable systems: Linux<fnref n="4" />, BusyBox<fnref n="5" />, and ToyBox<fnref n="6" />. They vary in size from 69k lines of code (LOC) to 21M LOC, and have respectively 18637, 1087, and 352 configuration options. Table 1 details the selected case studies<fnref n="7" />.

In this study, we aim to investigate the use of incremental build versus clean build on large configurable systems. Hence, the type of build (i.e., incremental and clean) of configurations is the *independent variable* we controlled. To answer our research questions, we observed three *dependent variables*: the *build time* of a configuration after both clean and incremental builds, the build *reproducibility*, and the *diff size* in terms of number of options.

## 4.3. Build Instrumentation

To address our research questions, we evaluate the effectiveness of incremental builds in two complementary settings. First, we assess incremental builds on a diverse set of random configurations. Since these configurations may differ significantly, they can challenge the reuse of build artifacts and, in turn, the effectiveness of incremental builds compared to traditional clean builds. This is one of our key hypotheses. Second, we examine incremental builds for "local" changes, that is, relatively small variations over a given base configuration. Here, we expect that PyroBuildS produces configurations that make incremental builds particularly effective.

<footnote n="4">⁴ https://www.kernel.org/</footnote>

<footnote n="5">⁵ https://www.busybox.net/</footnote>

<footnote n="6">⁶ http://landley.net/toybox/</footnote>

<footnote n="7">⁷ LoC computed with cloc — https://github.com/AlDanial/cloc</footnote>
In our experimental setup, we randomly select a set of $n = 100$ base configurations from the project. These include default setups such as Linux's `defconfig` and configurations generated via the `Kconfig` utility using the `make randconfig` command. For each base configuration, we generate variations in two ways:

* **Random variations:** For each base, we generate $m = 10$ fully random configurations.

* **Mutated variations:** For each base, we also generate $m = 10$ configurations by mutating the base configuration with `PyroBuildS`. In these mutated sequences, the change size is controlled: we use a change size of 10 for `Toybox` and `Busybox`, and a change size of 1 for `Linux`. Although a change size of 1 may seem minimal, the many options and dependencies in `Linux` lead to widespread configuration differences, resulting in high diversity. In this context, the change size is a hyper-parameter whose impact is empirically validated by our results.

Thus, for each types (random or mutated), we generate 100 base configurations × 10 variations = 1,000 configurations per project. For each configuration variation (whether random or mutated), we perform two builds: an incremental build (using `PyroBuildS`) and a corresponding clean build from scratch. This yields for our three case studies a total of

$$ (100 \times 10 \times 2) \times 2 \times 3 = 12000 $$

builds per run, where:

* $100 \times 10$ represents the number of variations per type,

* the first factor of 2 corresponds to the two build types (incremental and clean),

* the second factor of 2 accounts for the two configuration types (random and mutated), and

* the factor of 3 corresponds to our three case studies (Linux, `Busybox`, and `Toybox`).

We repeat the entire process in three separate runs for reproducibility, using the same hardware. In total, this experimental design yields a dataset
comprising 12,000 builds per run, amounting to 36,000 build instances across all runs.

Finally, during each build (both clean and incremental), we measure the build time and record other relevant variables, as well as the state of the build system's directory after each build, to facilitate further analysis.

### 4.4. Experimental Settings

**Build Environment.** All builds are performed inside a Docker container containing all necessary dependencies to builds each of our subject systems. A build can pollute a binary by embedding useless metadata (e.g., timestamps), thus leading to unreproducible builds. To favor reproducibility, we override parts of the build environment during the build, following the guidelines offered by the documentation of the software project or searching into its mailing list/issues for the best practice to achieve reproducible builds. Finally, for all three systems, when a clean build resulted in a build failure, incrementally building the same configuration using PyroBuildS gave the same result. Therefore, when a bug or misconfiguration occurs, the failure is reproducible with incremental builds, providing the same outcomes as clean builds. However, since failures have shorter build time, they bias the gain measurements of incremental build, which is our main objective in this work. Thus, we only keep successful builds in our dataset to measure time differences. We conduct our experiments on a server running Debian GNU/Linux 12 with 2xAMD Epyc 7532 CPU and 512 Go of RAM.

**Configuration.** We leverage Kconfig's built-in command *randconfig* coupled with our custom preset to generate random configurations. For instance, we only target configurations for the x86_64 architecture for the Linux kernel hence this value is preset to be enabled. In addition, we disable options that could lead to unreproducible builds or simply options that are not yet implemented and leading to build failures (e.g., Toybox). Kconfig also offers the built-in command *defconfig* which we use to generate the default configuration for each project.

We use *randconfig* to generate configurations, so the uniform distribution of the generated configurations may not be guaranteed. Other techniques like t-wise and uniform random sampling (URS) tackle this problem. However, as far as we know, they don't scale at all on Linux. Heradio et al. Heradio et al. (2022) do not apply their "scalable sampling" to Linux. Recently, Baranov-et
al. Baranov and Legay (2025) proposed BAITAL, a t-wise sampling claim-
ing scalability to thousands of options. We have launched the sampling of
only 500 configurations with BAITAL for Linux, Busybox, and Toybox with,
respectively, 27045, 1218, and 84 variables and 16463, 684, and 59 clauses
in their formulas. For Toybox and Busybox, we generated SAT formulas in
dimacs format with kmax that unfortunately cannot be translated back to
a valid .config due to missing options<sup>8</sup>. For Linux, after running since 25th
of November 2024 (still running at the time of submission), only 10% of the
work had been performed by BAITAL, confirming poor scalability. There-
fore, for practicality we use randconfig, which showed it was scalable and
yielding diverse configurations.

5. Results

This section presents and discusses the obtained results.

5.1. **RQ₁ – Reproducibility**

Reproducible Builds are verified by performing a bit-by-bit comparison of
two binaries using the command-line utility cmp<sup>9</sup>. For a given configuration,
we compare the binary obtained by its clean build to the binary obtained by
an incremental build. We consider the build reproducible if and only if the
two binaries are exactly the same.

All builds created by PyroBuildS are 100% reproducible for Toybox and
Busybox. For Linux, we have the following cases. First, incremental builds
of random configurations have an unreproducible builds rate varying from
41% to 56%. This is consistent with the results of Randrianaina et al. Ran-
drianaina et al. (2024) who also observed a rate of 47.5% of unreproducible
builds. This is a complex problem, since even the Linux community have not
yet identify all options causing non-reproducible builds, but only a fraction of
the 18K+ options<sup>10</sup>. Solving this reproducibility issue for Linux is extremely
hard and out of our scope. However, for the remaining builds performed with
controlled, mutated configurations over base configurations (e.g., defconfig),

<footnote n="8"><sup>8</sup>This Kmax limitation is currently labeled as a bug being addressed in an issue on the Kmax repository: https://github.com/paulgazz/kmax/issues/226</footnote>

<footnote n="9">⁹https://www.gnu.org/software/diffutils/manual/diffutils.html#Invoking-cmp</footnote>

<footnote n="10">¹⁰https://www.kernel.org/doc/html/latest/kbuild/reproducible-builds.html</footnote>
<figure><img src="image_5.png" /><figcaption>Figure 4: Time gain per system with PyroBuildS.</figcaption></figure>

we attain a rate of 100% of reproducible builds, which shows the importance
of PyroBuildS deny lists.

**RQ₁ insights:** Incremental build with PyroBuildS produced 100% reproducible builds in Toybox and Busybox in all situations. For Linux, only controlled mutations resulted in 100% reproducible builds.

## 5.2. RQ₂ – *Cost reduction with mutated configurations*

In this research question, we report the build time obtained with Py-
roBuildS by combining incremental builds and configuration variation over
a default configuration of each system. We compare the incremental build time of each configuration with their own clean build time.

As depicted in Figure 4a, our results show that PyroBuildS has improved overall build time for each system. We observe an average build time improvement of 16% for Toybox, 22% for Busybox and 23% for Linux. The maximum gain for each system are up to 19% for Toybox, 47% for Busybox and 59% for Linux.

Linux is the only system in which, for certain configuration pairs, incremental builds are slower than clean builds. Specifically, 23% of configurations exhibit this behavior, with performance gains dropping as low as -9%. Configurations with significant differences from the previous build tend to have slower incremental builds, particularly when the difference exceeds 400 (e.g., 640 for the slowest builds). While diff size is a relevant indicator of slower incremental build times, it does not account for everything. The kind of options included by the configuration also impacts build time. The negative impact of so-called problematic options is discussed in Sec 5.4.

For Linux, we thus recommend to set the change size value to 1. This significantly reduces build time while providing good diversity, as it modifies more than 100 option values. For large change size causing distant configurations (as for pairs of randconfig, see RQ3), incremental build may become ineffective. In such cases, using a traditional, clean build with PyroBuildS is better. This adjustment is simple to implement in PyroBuildS. For Toybox and Busybox, PyroBuildS speeds-up, whatever the distances across configurations. We thus recommend a change size of 10, as in our experiments, that provides good diversity and speed-up factor.

**RQ₂ insights:** PyroBuildS always gave a positive gain with incremental build over clean build with randomly mutated configurations. The mean gains for Toybox, Busybox, and Linux, are respectively 16%, 22%, and 19%. For Linux, a loss in build time starts with mutations of large diff sizes – something we can control in PyroBuildS.

### 5.3. RQ₃ – Cost reduction with random configurations

To answer this research question, we report the time gains from applying PyroBuildS over fully randomly generated configurations built incrementally when compared with their own clean build time.

Results are shown in Figure 4b. We observe that the time saved for the Linux kernel is always negative, with a mean of -23% (a time loss). However,
for Toybox and BusyBox, PyroBuildS always saves time, the mean positive gain being respectively 20% and 14%.

Given the size of the configuration space of the Linux kernel, every call of `randconfig` generates a random configuration that is completely different from the previous one. Thus, during the supposed incremental build, the build system is building a whole new configuration. Whereas for Toybox and BusyBox, we can still observe some similarities among configurations due to their configuration space being smaller than Linux.

The results of RQ2 and RQ3 have an impact on PyroBuildS and its parameterization. For Toybox and Busybox, PyroBuildS speeds-up whatever exploration and pairs of configurations. To explore in the large, we recommend using `randconfig`. To explore more locally, the change size of 10 of our experiments provides good diversity and speed-up, but can be increased if needs be. It is then up to continuous integration systems and developers to find the right balance and possibly to combine random with mutations. In any case, PyroBuildS is effective and can reduce cost.

For Linux, PyroBuildS should rely on traditional, clean build in the case of random configurations or pairs of configurations with large differences (see RQ₂). It is straightforward to detect such cases and avoid negative gains. For all other cases (i.e., mutated configurations), incremental build should be used and is effective. An interesting strategy is to use random configurations as bases (with clean build) and then incrementally build mutated configurations around, as they change an average of 3328 options (Min:Max [1795:4975]), thus diversifying the configuration space and gaining overall build time with PyroBuildS. This strategy guarantees computational cost reduction, while providing a good balance between global and local diversity.

**RQ₃ insights:** PyroBuildS always gave a positive gain with incremental build over clean build with randomly generated configurations only in Toybox, Busybox with a mean gain of, respectively, 20% and 14%. For Linux, there was a significant loss of -23%, calling to control diversity and the configuration diffs, since picking random configurations is not effective.

### 5.4. RQ₄ – Problematic options for incremental build

The previous results show that in some cases incremental build does not deliver a gain for Linux. For BusyBox and Toybox, we always found gains, but in few cases build time is significantly less reduced with incremental
build, being close to a clean build time. We thus investigated to find the
causes for this.

We first selected the subset of configurations pairs incrementally built
with a loss or no gain in build time. Then, we split the diff between the two
configurations to replay it with a granularity of one change in the options,
e.g., one option activated or dis-activated at a time. We further incrementally
built all those intermediate configurations to identify the problematic options
w.r.t. build execution time. To confirm that the identified list of options are
indeed the cause of the loss or no gain in time, we took again the same
selected subset of pairs of configurations, and we built them incrementally
but without the identified problematic options. This process took a few hours
of qualitative analysis.

A crucial aspect of our methodology is the use of mutated configurations,
thanks to PyroBuildS. By making relatively small modifications, we are able
to more precisely identify the options responsible for the negative effects on
incremental builds. In contrast, using random configurations would make
identifying problematic options far more difficult due to the large number of
variables, complicating the process of drawing reliable conclusions.

Following this methodology, we identified some problematic options. For
instance, for Busybox, options `LOCALE_SUPPORT` and `DEBUG` crosscut nu-
merous files, while `PAM` is related to the authentication aspect that affects
many utilities. Hence, it is no surprise that this kind of options can force
the recompilation of many artefacts and limit reuse in incremental build.
For Linux, some options like `COMPILE_TEST` can increase build time. This
option makes the build system build all the modules of the configuration
even though they are not supported by the current architecture. In addi-
tion, we observe configuration options that impacts directly the core part
of the kernel (options located in the `kernel` subsystem of Linux), such as
`GENERAL_IRQ CHIP`. In fact, modification on the core part of the kernel
triggers the rebuild of the whole codebase. Some values on the target archi-
tecture also impact the whole codebase, for instance `X86_X32` that enables
the ABI for 32 bits architecture. All these, coupled with a large amount of
drivers picked by our variation strategy, has a significant impact on build
time.

Interestingly, the identified problematic options in Linux, Toybox and
BusyBox are either cross-cutting options (*e.g.*, `LOCALE_SUPPORT` and `DE-
BUG`) or compiler-related options (*e.g.*, `COMPILE_TEST`). These options are
likely to remain problematic whatever the version, because of their signifi-
cant impact on the re-build process. Hence, this knowledge and performance
deny list can be reused by PyroBuildS when processing further versions of
these three systems.

It is worth noting that prior works Randrianaina et al. (2022b,a); Lyu et al. (2024) observed variability in incremental build performance but did not identify its root causes, thereby missing the opportunity to proactively avoid problematic options as done in PyroBuildS.

**RQ₄ insights:** We found evidence that certain options are limiting the time gains of incremental builds for Busybox and Toybox. These options can even result in a scenario where a clean build is faster than an incremental build for Linux. Therefore, the performance deny list in PyroBuildS is crucial to avoid these problematic options and consistently speed up the build process.

5.5. **RQ<sub>5</sub> – Diversity induced by incremental build**

To answer this question, we observe the various sizes of diffs between
configurations in terms of activated and dis-activated options.

Figures 5b and 5a show the overall distribution of the various diffs be-
tween the configurations we build in our three case studies and for each of
the 10 batches. For randomly generated configurations in Figure 5b, the
mean diff size for Toybox, Busybox and Linux is 125 (Min:Max[96:150]),
425 (Min:Max[354:482]), and 3328 (Min:Max[1795:4975]) of changed options.
For the diff sizes between randomly mutated configurations in Figure 5a, the
mean diff size for Toybox, Busybox and Linux is 19 (Min:Max[10:79]), 29
(Min:Max[10:310]), and 124 (Min:Max[2:640]) of changed options.

These distributions of diff sizes indicate that our considered configurations within each batch are diverse enough covering small, medium and large diffs in terms of options that are changed between configurations, especially when put in ratio w.r.t. their total number of options: respectively, 352, 1078, and 18637 for Toybox, Busybox, and Linux.

**RQ₅ insights:** For Busybox and Toybox, PyroBuildS can reach a high diversity while speeding up the builds by picking random configurations. For Linux, a proper parameterization of change size is needed and delivers interesting diversity (120 options are changed on average across successive configurations).
<figure><img src="image_6.png" /><figcaption>Figure 5: Number of options that differ between two configurations in the build order.</figcaption></figure>
# 6. Threats to validity

We now discuss internal and external threats to validity Wohlin et al. (2012).

**Internal validity.** We first measured the time performance of clean and incremental builds. The main risk lies in the interference of measurements with other running software. To counter this risk, we used a dedicated server where only our experiment was executed. We further isolated the build environment Docker image with only the required build tools and dependencies.

We repeated the builds three times to reduce variability and ensure more reliable averages. As part of our experiments, a campaign of Linux builds took several days. The longer build times naturally mitigate minor fluctuations, reducing the need for multiple iterations. Additionally, as part of our exploratory experiments, we also observed consistent build times when repeating, further supporting our approach. Overall, we are confident that our experimental data are reliable for Linux. We considered diversity by looking at the number of options that are activated during the build. Other metrics could also be considered, such as the impacted source code. However, the diff size of options is a good indicator for the diversity of a given configuration.

**External validity.** We experimented on three systems (Toybox, Busybox, Linux kernel) that are C-based, highly configurable software systems with the MAKE build system. We purposely focused MAKE in order to replicate the previous work of Randrianaina et al. Randrianaina et al. (2022b,a) on a different scale of complex configurable software system. Further results of PyroBuildS with mutation-based incremental build cannot be generalized to other small-sized configurable software system, and more experimentation remaining necessary in this area. We are however confident that the principles and key components of PyroBuildS (e.g., deny lists to avoid problematic options) can be reused.

# 7. Related Work

We now discuss main related work to PyroBuildS.

**Build systems.** Many works exist on (incremental) build systems Cao et al. (2017b); Feldman (1979a); Tichy (1986); Schwanke and Kaiser (1988); Mitchell (2012); baz ([n. d.]); Erdweg et al. (2015); Konat et al. (2018); Hammer et al. (2015); Maudoux and Mens (2018) but they focus on code changes through the evolution of software (e.g. commits) rather than configurations.
Cserep et al. Cserép and Fekete (2020) introduce how to detect only the necessary files to build with incremental parsing of the codebase. In Konat et al. ([n. d.], 2018), Konat et al. provide a DSL to increase the effectiveness of writing build scripts. With such an expressive language, analysis and error detection could be performed beforehand. Maudoux et al. Maudoux and Mens (2017) show that incremental build could help speed up builds of continuous integration (CI). Gallaba et al. Gallaba et al. (2020) proposed to accelerate the continuous integration build. They infer data from which build acceleration decisions can be made by caching the build environment and skipping unaffected build steps. An open issue is to adapt these techniques over software configurations. When considering configurations, the differences can be much larger than across commits, spanning numerous files and drastically challenging build tasks. As empirically shown, large configuration spaces exacerbate the differences. Configurations thus challenge the effectiveness of incremental compilation and build in this context.

Several empirical studies on build systems have been performed (e.g., Hilton et al. (2016, 2017b); Zhao et al. (2017); Lebeuf et al. (2018b); Maudoux and Mens (2019, 2018); Lawall and Muller (2017b)). Beller et al. Beller et al. (2017b) performed an analysis of builds with Travis CI on top of GitHub. About 10% of builds show different behavior when different environments are used. In our case, we are considering different software configurations rather than environments. Hasanov et al. Hasanov et al. (2025) systematically analyzed the challenges of configuration testing for continuous fuzzers and provided guidelines to configuration in the context of the Linux kernel. We have shown that incremental build can speed up the build of multiple configurations. Lawall et al. Lawall and Muller (2017a) proposed JMake, a mutation-based tool for signaling changed lines that are not subjected to the compiler. They aim to find an appropriate Linux configuration that can compile and thus covers the files in which changes have been made. In contrast, we design a configurable mutation operator to diversify an existing base configuration and synthesize several related configurations that can then be incrementally build.

Yıldıran et al. Yıldıran et al. (2024) propose krepair, a more efficient solution than JMake to achieve high patch coverage, leveraging this time advanced constraint-solving techniques and targeted configuration repairs. Once krepair has synthesized a configuration file, incremental build can be used to speed up the build process compared to a clean build. We leave a comprehensive evaluation of incremental build performance – following
krepair’s configuration synthesis – as future work.

Lyu et al. Lyu et al. (2024) developed BUDDI to support the idea of *op-
timal ordering* proposed by Randrianaina et al. Randrianaina et al. (2022b),
which consists in finding the fastest order to incrementally build a given
set of configurations. In contrast, **PyroBuildS** does not take the configura-
tion set as input but rather generates configurations dynamically through a
mutation strategy. This makes **PyroBuildS** more suitable for exploring large
configuration spaces without prior knowledge of all configurations.

It is worth noting that BUDDI demonstrates promising results on relatively small C, Make-based systems. In contrast, our work targets large-scale, Kconfig/Kbuild-based systems such as Linux, which involve thousands of configuration options and a more complex dependency structure. Furthermore, we observed that certain configurations – especially random ones – consistently lead to increased build times when paired. More precisely, let $c_{base}$ and $c_{rand}$ be two randomly sampled configurations; building $c_{rand}$ incrementally from $c_{base}$ often results in a higher build time than a clean build of $c_{rand}$ alone. Thus, for Linux, $\forall (c_{base}, c_{rand})$ in *RandPairs: Incremental*($c_{base}, c_{rand}$) > *Clean*($c_{rand}$). This empirical finding highlights a fundamental limitation: no matter how configurations are ordered, incremental build should not be applied across arbitrary random configurations. As such, while the idea of finding an optimal order is appealing, its benefits may not generalize to highly heterogeneous or structurally distant configurations – especially in systems like Linux. These findings suggest that adaptations would be required for BUDDI to scale to such settings.

**Software product line (SPL) and variability.** The SPL community develops numerous methods and techniques to manage a family of variants (or products). Configurations are used to build or execute variants and are subject to intensive research. For instance, building variants is a necessary step before deriving performance prediction models Alves Pereira et al. (2021); Jamshidi et al. (2017); Guo et al. (2013); Lesoil et al. (2021); Jamshidi et al. (2018). Formal methods and program analysis can identify some classes of configuration defects Thüm et al. (2014); Classen et al. (2013), leading to variability-aware testing approaches (e.g., Nguyen et al. (2014); Kim et al. (2013, 2011); Shi et al. (2012); von Rhein et al. (2018); Kastner and Apel (2008); Kenner et al. (2010); Fischer et al. (2021); Lima et al. (2020); Reisner et al. (2010)). The general principle is to exploit the commonalities among variants, mainly at code level. Nguyen et al. implemented Varex, a variability-aware PHP interpreter, to test WordPress by running code com-
mon to several variants only once Nguyen et al. (2014). Reisner et al. use a symbolic execution framework to evaluate how the configuration options impact the coverage of the system given a test suite Reisner et al. (2010). Static analysis and notably type-checking has been used to analyze some properties of configurations and can scale to very large code bases such as the Linux Kernel von Rhein et al. (2018); Kastner and Apel (2008); Kenner et al. (2010). Though variability-aware analysis is relevant in many engineering contexts, our interest differs and consists in concretely building a sample of (possibly distant and diverse) configurations with an unexplored approach – incremental build. Al-Hajjaji et al. Al-Hajjaji et al. (2017, 2019) focused on optimizing the order of a tested set of configurations, by selecting the most dissimilar configuration to the previous one to be built. However, while the authors incrementally select the configurations based on a similarity prioritization criterion, they do not perform incremental build. Sampling configurations is subject to intensive research Alves Pereira et al. (2020); Jamshidi et al. (2018); Souto et al. (2017); Thüm et al. (2014); Heradio et al. (2022). However, incremental build brings new challenges. As empirically shown for Linux, incremental build is not effective when picking random configurations since the distance and differences across configurations are too important. In a sense, we develop in our work a new sampling procedure to either locally or globally explore configurations, avoiding the trap of changing values of problematic options. To the best of our knowledge, we are the first to quantify and evaluate the use of incremental build on large configurable systems. We offer valuable learned lessons on when incremental build works and when not, especially in relation to the diff sizes of the configurations to be built incrementally.

## 8. Conclusion

In this paper, we introduced PyroBuildS, an innovative approach to reduce, through incremental builds, the computational cost of building large and complex configurable systems. Although incremental builds have been studied in the context of post-commit changes, their advantages in terms of configurations and large complex configuration spaces have not been fully understood. Our work and PyroBuildS address this gap.

Our empirical evaluation on Linux, BusyBox, and ToyBox demonstrated that PyroBuildS effectively maintains build correctness while achieving substantial reductions in build time. Gains with incremental build varied from
15% to 20% with random configurations and from 16% to 22% with mutated configurations. We identified specific problematic options that hinder incremental build effectiveness, allowing **PyroBuildS** to avoid these and ensure systematic gains. We have shown that "just" applying incremental builds can be counterproductive, and so that the three parameters of **PyroBuildS** that control configuration generation are crucial.

Prior studies reported exploratory results on small configurable systems, with both negative and positive effects of incremental build. Our detailed analysis now explains these mixed outcomes, and **PyroBuildS** offers a way to improve the situation on large complex configurable systems.

Our recommendations for using **PyroBuildS** are as follows.

For BusyBox and ToyBox, **PyroBuildS** speeds up builds regardless of the distances across configurations. Hence, `randconfig` can be used out of the box. To explore more locally, a change size of 10 provides a good balance of diversity and speed-up, though this value can be increased if necessary. Overall, **PyroBuildS** is capable of achieving high diversity while speeding up builds by picking random configurations.

For Linux, **PyroBuildS** relies on traditional, clean builds in the case of random configurations or pairs of configurations with large differences. These cases are straightforward to detect, enabling the avoidance of negative gains. We recommend relying on **PyroBuildS** to generate local variations of configurations where incremental builds pay off. That is, **PyroBuildS** can significantly reduce build time while still providing good diversity, as it modifies more than 100 option values per local variation. A possible exploration strategy is to use random configurations as bases (with clean builds) and then incrementally build mutated configurations around them. **PyroBuildS** can effectively implement this process.

Overall, we believe that **PyroBuildS** provides actionable insights and practical support for developers, maintainers, and CI practitioners aiming to accelerate the build processes of highly configurable systems. While it is up to these stakeholders to define an appropriate exploration strategy—e.g., choosing between random or local configuration changes—**PyroBuildS** offers (1) a tool that facilitates controlled configuration generation and incremental building, and (2) empirically grounded recommendations to avoid pitfalls and guide effective use of incremental builds.

In future work, we aim to apply incremental builds in other scenarios, such as performance learning or testing patches in configurable systems, where
both code evolution and configuration changes must be managed. For instance, Yildiran et al. Yildiran et al. (2024) addressed the challenge of determining the optimal configurations to test patches in Linux project commits. Building on their approach, we plan to integrate PyroBuildS to assess its impact and potential benefits. This requires a specific effort to carefully avoid selecting configurations that are too distant or include problematic options. Moreover, PyroBuildS could also be integrated in a CI. Indeed, we can see PyroBuildS being integrated with KernelCI and Intel 0-day for example. For example, the idea would be to build default configurations at each release and turn around them in a firework analogy with incremental builds. This would strengthen the confidence in testing the default configurations as it is done by KernelCI and extend the number of tested random configurations by Intel 0-day with small extra resources. Finally, we plan to test PyroBuildS on configurable systems with a different build system and in different programming languages for more generalizable results.

## 9. Data availability

The scripts to produce the data of this paper as well as the notebook that generated the graphs are shared in the following repository https://github.com/garandria/pyrobuilds-artifact
References

[n. d.]. A fast, scalable, multi-language and extensible build system. https://bazel.build/

2021. Intel 0-day. https://01.org/lkp/documentation/0-day-test-service

2021. KernelCI. https://kernelci.org/

Mathieu Acher, Hugo Martin, Juliana Alves Pereira, Arnaud Blouin, Djamel Eddine Khelladi, and Jean-Marc Jézéquel. 2019a. *Learning From Thousands of Build Failures of Linux Kernel Configurations*. Technical Report. Inria ; IRISA. 1–12 pages. https://hal.inria.fr/hal-02147012

Mathieu Acher, Hugo Martin, Luc Lesoil, Arnaud Blouin, Jean-Marc Jézéquel, Djamel Eddine Khelladi, Olivier Barais, and Juliana Alves Pereira. 2022. Feature subset selection for learning huge configuration spaces: the case of Linux kernel size. In *Proceedings of the 26th ACM International Systems and Software Product Line Conference-Volume A*. 85–96.

Mathieu Acher, Hugo Martin, Juliana Alves Pereira, Arnaud Blouin, Jean-Marc Jézéquel, Djamel Eddine Khelladi, Luc Lesoil, and Olivier Barais. 2019b. *Learning very large configuration spaces: What matters for Linux kernel sizes*. Ph. D. Dissertation. Inria Rennes-Bretagne Atlantique.

Mustafa Al-Hajjaji, Sascha Lity, Remo Lachmann, Thomas Thüm, Ina Schaefer, and Gunter Saake. 2017. Delta-oriented product prioritization for similarity-based product-line testing. In *2017 IEEE/ACM 2nd International Workshop on Variability and Complexity in Software Design (VACE)*. IEEE, 34–40.

Mustafa Al-Hajjaji, Thomas Thüm, Malte Lochau, Jens Meinicke, and Gunter Saake. 2019. Effective product-line testing using similarity-based product prioritization. *Software & Systems Modeling* 18 (2019), 499–521.

Juliana Alves Pereira, Mathieu Acher, Hugo Martin, and Jean-Marc Jézéquel. 2020. Sampling effect on performance prediction of configurable systems: A case study. In *Proceedings of the ACM/SPEC International Conference on Performance Engineering*. 277–288. https://doi.org/10.1145/3358960.3379137
Juliana Alves Pereira, Hugo Martin, Mathieu Acher, Jean-Marc Jézéquel, Goetz Botterweck, and Anthony Ventresque. 2021. Learning Software Configuration Spaces: A Systematic Literature Review. Journal of Systems and Software (Aug. 2021). https://doi.org/10.1016/j.jss.2021.111044

Eduard Baranov and Axel Legay. 2025. Baital: Sampling configurable systems with high t-wise coverage. Science of Computer Programming 240 (2025), 103209. https://doi.org/10.1016/j.scico.2024.103209

Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017a. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub. In Proceedings of the 14th International Conference on Mining Software Repositories (Buenos Aires, Argentina) (MSR '17). IEEE Press, 356-367. https://doi.org/10.1109/MSR.2017.62

Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017b.Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub. In Proceedings of the 14th International Conference on Mining Software Repositories (Buenos Aires, Argentina) (MSR '17). IEEE Press, Piscataway, NJ, USA, 356-367.

Cor-Paul Bezemer, Shane McIntosh, Bram Adams, Daniel M German, and Ahmed E Hassan. 2017. An empirical study of unspecified dependencies in make-based build systems. Empirical Software Engineering 22, 6 (2017), 3117-3148. https://doi.org/10.1007/s10664-017-9510-8

Qi Cao, Ruiyin Wen, and Shane McIntosh. 2017a. Forecasting the Duration of Incremental Build Jobs. In Proc. of the International Conference on Software Maintenance and Evolution (ICSME). 524-528.

Qi Cao, Ruiyin Wen, and Shane McIntosh. 2017b. Forecasting the duration of incremental build jobs. In 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 524-528. https://doi.org/10.1109/ICSME.2017.34

Andreas Classen, Maxime Cordy, Pierre-Yves Schobbens, Patrick Heymans, Axel Legay, and Jean-François Raskin. 2013. Featured Transition Systems: Foundations for Verifying Variability-Intensive Systems and Their Application to LTL Model Checking. IEEE Transactions on Software Engineering 39, 8 (aug 2013), 1069-1089.
Máté Cserép and Anett Fekete. 2020. Integration of Incremental Build Systems Into Software Comprehension Tools. In *ICAI*. 85–93. http://ceur-ws.org/Vol-2650/paper10.pdf

Jack Edge. 2020. The costs of continuous integration. https://lwn.net/Articles/813767/

Sebastian Erdweg, Moritz Lichter, and Manuel Weiel. 2015. A sound and optimal incremental build system with dynamic dependencies. *ACM Sigplan Notices* 50, 10 (2015), 89–106.

Stuart I. Feldman. 1979a. Make — a program for maintaining computer programs. *Software: Practice and Experience* 9, 4 (1979), 255–265.

Stuart I. Feldman. 1979b. Make — a program for maintaining computer programs. *Software: Practice and Experience* 9, 4 (1979), 255–265. https://doi.org/10.1002/spe.4380090402 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/spe.4380090402

Stefan Fischer, Rudolf Ramler, Claus Klammer, and Rick Rabiser. 2021. Testing of Highly Configurable Cyber-Physical Systems – A Multiple Case Study. In *15th International Working Conference on Variability Modelling of Software-Intensive Systems* (Krems, Austria) (*VaMoS'21*). Association for Computing Machinery, New York, NY, USA, Article 19, 10 pages. https://doi.org/10.1145/3442391.3442411

Keheliya Gallaba, Yves Junqueira, John Ewart, and Shane Mcintosh. 2020. Accelerating Continuous Integration by Caching Environments and Inferring Dependencies. *IEEE Transactions on Software Engineering* (2020), 1–1.

Paul Gazzillo. 2017. Kmax: Finding all configurations of kbuild makefiles statically. In *Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering*. 279–290.

Seyedhamed Ghavamnia, Tapti Palit, Shachee Mishra, and Michalis Polychronakis. 2020. Temporal system call specialization for attack surface reduction. In *29th USENIX Security Symposium (USENIX Security 20)*. 1749–1766.
Jianmei Guo, Krzysztof Czarnecki, Sven Apel, Norbert Siegmund, and Andrzej Wasowski. 2013. Variability-aware performance prediction: A statistical learning approach. In *2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE)*. IEEE, 301–311.

Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2018. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack. *Empirical Software Engineering* 24, 2 (jul 2018), 674–717. https://doi.org/10.1007/s10664-018-9635-4

Matthew A. Hammer, Joshua Dunfield, Kyle Headley, Nicholas Labich, Jeffrey S. Foster, Michael Hicks, and David Van Horn. 2015. Incremental computation with names. *Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications* (Oct 2015).

Sanan Hasanov, Stefan Nagy, and Paul Gazzillo. 2025. A Little Goes a Long Way: Tuning Configuration Selection for Continuous Kernel Fuzzing. In *Proceedings of the International Conference on Software Engineering (ICSE)*.

Ruben Heradio, David Fernandez-Amoros, José A Galindo, David Benavides, and Don Batory. 2022. Uniform and scalable sampling of highly configurable systems. *Empirical Software Engineering* 27, 2 (2022), 44.

Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017a. Trade-Offs in Continuous Integration: Assurance, Security, and Flexibility. In *Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering* (Paderborn, Germany) (*ESEC/FSE 2017*). Association for Computing Machinery, New York, NY, USA, 197–207. https://doi.org/10.1145/3106237.3106270

Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017b. Trade-offs in continuous integration: assurance, security, and flexibility. In *Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering*. 197–207.

Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-
source projects. In *2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE)*. IEEE, 426–437.

Pooyan Jamshidi, Norbert Siegmund, Miguel Velez, Christian Kästner, Akshay Patel, and Yuvraj Agarwal. 2017. Transfer learning for performance modeling of configurable systems: an exploratory analysis. In *Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, ASE 2017, Urbana, IL, USA, October 30 - November 03, 2017*, Grigore Rosu, Massimiliano Di Penta, and Tien N. Nguyen (Eds.). IEEE Computer Society, 497–508.

Pooyan Jamshidi, Miguel Velez, Christian Kästner, and Norbert Siegmund. 2018. Learning to sample: exploiting similarities across environments to learn performance models for configurable systems. In *Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering*. 71–82.

Christian Kästner and Sven Apel. 2008. Type-checking software product lines-a formal approach. In *2008 23rd IEEE/ACM International Conference on Automated Software Engineering - ASE '08*. IEEE, 258–267.

Andy Kenner, Christian Kästner, Steffen Haase, and Thomas Leich. 2010. TypeChef: Toward Type Checking \#Ifdef Variability in C. In *Proceedings of the 2Nd International Workshop on Feature-Oriented Software Development* (Eindhoven, The Netherlands) (*FOSD '10*). ACM, New York, NY, USA, 25–32.

Michael Kerrisk. 2012. KS2012: Kernel build/boot testing. https://lwn.net/Articles/514278/.

Chang Hwan Peter Kim, Don S Batory, and Sarfraz Khurshid. 2011. Reducing combinatorics in testing product lines. In *Proceedings of the tenth international conference on Aspect-oriented software development (AOSD '11)*. ACM, 57–68.

Chang Hwan Peter Kim, Darko Marinov, Sarfraz Khurshid, Don Batory, Sabrina Souto, Paulo Barros, and Marcelo d'Amorim. 2013. SPLat: lightweight dynamic analysis for reducing combinatorics in testing configurable systems - ESEC/FSE '13. In *Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering*. ACM, 257–267.
Gabriël Konat, Sebastian Erdweg, and Eelco Visser. 2018. Scalable incremental building with dynamic task dependencies. In *2018 33rd IEEE/ACM International Conference on Automated Software Engineering (ASE)*. IEEE, 76–86. https://doi.org/10.1145/3238147.3238196

Gabriël Konat, Roelof Sol, Sebastian Erdweg, and Eelco Visser. [n.d.]. Precise, Efficient, and Expressive Incremental Build Scripts with PIE. ([n. d.]).

Hsuan-Chi Kuo, Jianyan Chen, Sibin Mohan, and Tianyin Xu. 2020a. Set the configuration for the heart of the os: On the practicality of operating system kernel debloating. *Proceedings of the ACM on Measurement and Analysis of Computing Systems* 4, 1 (2020), 1–27.

Hsuan-Chi Kuo, Dan Williams, Ricardo Koller, and Sibin Mohan. 2020b. A linux in unikernel clothing. In *Proceedings of the Fifteenth European Conference on Computer Systems*. 1–15.

Julia Lawall and Gilles Muller. 2017a. JMake: Dependable Compilation for Kernel Janitors. In *The 47th IEEE/IFIP International Conference on Dependable Systems and Networks*. IEEE/IFIP, Denver, Colorado, United States. https://doi.org/10.1109/DSN.2017.62

Julia Lawall and Gilles Muller. 2017b. JMake: Dependable Compilation for Kernel Janitors. In *2017 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)*. IEEE, 357–366.

Carlene Lebeuf, Elena Voyloshnikova, Kim Herzig, and Margaret-Anne Storey. 2018a. Understanding, Debugging, and Optimizing Distributed Software Builds: A Design Study. In *2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)*. 496–507. https://doi.org/10.1109/ICSME.2018.00060

Carlene Lebeuf, Elena Voyloshnikova, Kim Herzig, and Margaret-Anne Storey. 2018b. Understanding, debugging, and optimizing distributed software builds: A design study. In *2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)*. IEEE, 496–507.

Luc Lesoil, Mathieu Acher, Xhevahire Tërnava, Arnaud Blouin, and Jean-Marc Jézéquel. 2021. The interplay of compile-time and run-time options for performance prediction. In *SPLC '21: 25th ACM International Systems and Software Product Line Conference, Leicester, United Kingdom*,
*September 6-11, 2021, Volume A, Mohammad Mousavi and Pierre-Yves Schobbens (Eds.). ACM, 100-111.*

Jackson A. Prado Lima, Willian Douglas Ferrari Mendonça, Silvia R. Vergilio, and Wesley K. G. Assunção. 2020. Learning-based prioritization of test cases in continuous integration of highly-configurable software. In *SPLC '20: 24th ACM International Systems and Software Product Line Conference*, Roberto Erick Lopez-Herrejon (Ed.). ACM, 31:1-31:11.

Jun Lyu, Shanshan Li, He Zhang, Lanxin Yang, Bohan Liu, and Manuel Rigger. 2024. Towards Efficient Build Ordering for Incremental Builds with Multiple Configurations. *Proc. ACM Softw. Eng.* 1, FSE, Article 67 (jul 2024), 24 pages. https://doi.org/10.1145/3660774

Hugo Martin, Mathieu Acher, Juliana Alves Pereira, Luc Lesoil, Jean-Marc Jézéquel, and Djamel Eddine Khelladi. 2022. Transfer Learning Across Variants and Versions: The Case of Linux Kernel Size. *IEEE Trans. Software Eng.* 48, 11 (2022), 4274-4290. https://doi.org/10.1109/TSE.2021.3116768

Guillaume Maudoux and Kim Mens. 2017. Bringing incremental builds to continuous integration. In *Proc. 10th Seminar Series Advanced Techniques & Tools for Software Evolution*. 1-6.

Guillaume Maudoux and Kim Mens. 2018. Correct, efficient, and tailored: The future of build systems. *IEEE Software* 35, 2 (2018), 32-37.

Guillaume Maudoux and Kim Mens. 2019. Lessons and Pitfalls in Building Firefox with Tup.. In *SATToSE*.

Shane McIntosh. 2024. Mining Our Way Back to Incremental Builds for DevOps Pipelines. In *Proc. of the International Conference on Mining Software Repositories (MSR)*. To appear.

Shane McIntosh, Meiyappan Nagappan, Bram Adams, Audris Mockus, and Ahmed E. Hassan. 2015. A Large-Scale Empirical Study of the Relationship between Build Technology and Build Maintenance. *Empirical Softw. Engg.* 20, 6 (dec 2015), 1587-1633. https://doi.org/10.1007/s10664-014-9324-x
Jean Melo, Elvis Flesborg, Claus Brabrand, and Andrzej Wąsowski. 2016. A Quantitative Analysis of Variability Warnings in Linux. In *Proceedings of the Tenth International Workshop on Variability Modelling of Software-intensive Systems* (Salvador, Brazil) (VaMoS '16). ACM, 3–8.

Neil Mitchell. 2012. Shake before building. *ACM SIGPLAN Notices* 47 (10 2012), 55. https://doi.org/10.1145/2398856.2364538

Austin Mordahl, Jeho Oh, Ugur Koc, Shiyi Wei, and Paul Gazzillo. 2019. An empirical study of real-world variability bugs detected by variability-oblivious tools. In *Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering* (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 50–61. https://doi.org/10.1145/3338906.3338967

Hung Viet Nguyen, Christian Kästner, and Tien N Nguyen. 2014. Exploring variability-aware execution for testing plugin-based web applications. In *Proceedings of the 36th International Conference on Software Engineering - ICSE '14*. ACM, 907–918.

Georges Aaron Randrianaina, Djamel Eddine Khelladi, Olivier Zendra, and Mathieu Acher. 2022a. Towards incremental build of software configurations. In *Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results*. 101–105.

Georges Aaron Randrianaina, Djamel Eddine Khelladi, Olivier Zendra, and Mathieu Acher. 2024. Options Matter: Documenting and Fixing Non-Reproducible Builds in Highly-Configurable Systems. In *Mining Software Repository (MSR)*.

Georges Aaron Randrianaina, Xhevahire Tërnava, Djamel Eddine Khelladi, and Mathieu Acher. 2022b. On the benefits and limits of incremental build of software configurations: an exploratory study. In *Proceedings of the 44th International Conference on Software Engineering*. 1584–1596.

Elnatan Reisner, Charles Song, Kin-Keung Ma, Jeffrey S Foster, and Adam Porter. 2010. Using symbolic evaluation to understand behavior in configurable software systems. In *Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering (ICSE '10, Vol. 1)*. ACM Press, 445.
Dan Rue. 2021. Portable and reproducible kernel builds with TuxMake. https://lwn.net/Articles/841624/.

Robert W. Schwanke and Gail E. Kaiser. 1988. Smarter Recompilation. *ACM Trans. Program. Lang. Syst.* 10, 4 (Oct. 1988), 627–632.

Jiangfan Shi, Myra B. Cohen, and Matthew B. Dwyer. 2012. Integration Testing of Software Product Lines Using Compositional Symbolic Execution. In *Proceedings of the 15th International Conference on Fundamental Approaches to Software Engineering (LNCS, Vol. 7212)*. Springer, 270–284.

Sabrina Souto, Marcelo d'Amorim, and Rohit Gheyi. 2017. Balancing soundness and efficiency for practical testing of configurable systems. In *2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)*. IEEE, 632–642.

Tuxmake Team. Online; accessed 2023. TuxMake. https://tuxmake.org/.

Thomas Thüm, Sven Apel, Christian Kästner, Ina Schaefer, and Gunter Saake. 2014. A Classification and Survey of Analysis Strategies for Software Product Lines. *Comput. Surveys* 47, 1 (2014), 6:1–6:45.

Walter F. Tichy. 1986. Smart Recompilation. *ACM Trans. Program. Lang. Syst.* 8, 3 (June 1986), 273–291. https://doi.org/10.1145/5956.5959

Alexander von Rhein, Jörg Liebig, Andreas Janker, Christian Kästner, and Sven Apel. 2018. Variability-Aware Static Analysis at Scale: An Empirical Study. *ACM Trans. Softw. Eng. Methodol.* 27, 4 (2018), 18:1–18:33.

Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. *Experimentation in software engineering*. Springer Science & Business Media.

Necip Fazıl Yıldıran, Jeho Oh, Julia Lawall, and Paul Gazzillo. 2024. Maximizing Patch Coverage for Testing of Highly-Configurable Software without Exploding Build Times. In *Proceedings of the ACM International Conference on the Foundations of Software Engineering (FSE)*. ACM. arXiv:arXiv:2404.17966
Yangyang Zhao, Alexander Serebrenik, Yuming Zhou, Vladimir Filkov, and Bogdan Vasilescu. 2017. The impact of continuous integration on other software development practices: a large-scale empirical study. In *2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE)*. IEEE, 60–71.