# In the Search for Truth: Navigating Variability in Neuroimaging Software Pipelines

Youenn Merel Jourdan, Mathieu Acher, Camille Maumet

► To cite this version:

Youenn Merel Jourdan, Mathieu Acher, Camille Maumet. In the Search for Truth: Navigating Variability in Neuroimaging Software Pipelines. SPLC 2025 - 29th ACM International Systems and Software Product Line Conference, Association for Computing Machinery (ACM), Sep 2025, Coruna, Spain, Spain. pp.129-135, 10.1145/3744915.3748470. hal-05158426

HAL Id: hal-05158426

https://inria.hal.science/hal-05158426v1

Submitted on 11 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# In the Search for Truth: Navigating Variability in Neuroimaging Software Pipelines

Youenn Merel Jourdan

Mathieu Acher

Camille Maumet

Univ Rennes, Inria, CNRS, Inserm
Rennes, France

CNRS, INSA Rennes, IUF, IRISA, Inria
Rennes, France

Univ Rennes, Inria, CNRS, Inserm
Rennes, France

youenn.merel@inria.fr

## Abstract

Neuroimaging pipelines – software-driven analysis workflows of brain images – are characterized by a wide range of tools, parameters, and configuration choices. Such flexibility, while enabling diverse scientific inquiries, gives rise to analytical variability: different pipeline variants can lead to different outcomes. In practice, each neuroimaging pipeline variant produces a statistic map – a complex, structured 3D output whose relevance can only be assessed with specific domain expertise, unlike simple metrics such as execution time. And, in most cases, there is no ground truth against which to judge these outputs, making it unclear which variant yields the best result.

In this paper, we introduce a "sampling, variant scoring, learning" methodology to study variability in the absence of a quantitative target – i.e. ground truth. We report our experience in developing an Universal Variability Language (UVL) feature model of 90 features representing the configuration space of a well-established open source neuroimaging analysis software (SPM). We sample and run 1000 valid configurations generating 1000 statistic maps as outputs.

We studied various candidate proxy ground truths and computed Spearman correlations as a quantitative metric of the performance of each pipeline. We tested the following 12 proxy ground truths: the average statistic map (across variants), the output of an expert-derived configuration, and a set of randomly selected outputs (as baseline). Then, we used a decision tree learning approach to inspect variability. We evaluated the sensitivity of our method to the choice of (proxy) ground truth, both in terms of predictive accuracy and in the identification of important features. These first results outline the challenge of choosing and validating a referential to assess our understanding of variability in the absence of ground truth.

## CCS Concepts

* Software and its engineering → Feature interaction; • Computing methodologies → Classification and regression trees; Modeling methodologies; • Applied computing → Imaging; Bioinformatics; • Computer systems organization → Pipeline computing.

## Keywords

Software Variability, Feature Model, Neuroimaging, Pipelines, Sampling, Decision Tree, Performance, Variability Analytics

<figure><img src="image_2.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License.
SPLC-A '25, A Coruña, Spain

© 2025 Copyright held by the owner/author(s).

ACM ISBN 979-8-4007-2024-6/2025/09

https://doi.org/10.1145/3744915.3748470

## ACM Reference Format:

Youenn Merel Jourdan, Mathieu Acher, and Camille Maumet. 2025. In the Search for Truth: Navigating Variability in Neuroimaging Software Pipelines. In *29th ACM International Systems and Software Product Line Conference - Volume A (SPLC-A '25)*, September 1-5, 2025, A Coruña, Spain. ACM, New York, NY, USA, 7 pages. https://doi.org/10.1145/3744915.3748470

## 1 Introduction

Neuroimaging pipelines – i.e. software-driven workflows used to process and analyze brain data – are essential to transform raw neuroimaging data (e.g. MRI, PET) into structured outputs that provide information about the brain anatomy and function. Their use spans from diagnostics in clinical routine to the study of cognitive processes and neurological disorders.

In practice, the design and implementation of these pipelines involve the integration of multiple tools and algorithms, each configurable via a range of parameters. This configurability is essential for tailoring the analyses to different experimental or participant-specific conditions but also introduces *analytical variability*: distinct pipeline variants, though equally valid from a scientific point of view, may eventually produce different findings [5, 14].

Variability thus plays a dual role: it enables extensive configurations, yet, simultaneously, it provides a myriad of possible outcomes. This is especially challenging in the absence of a definitive ground truth that may be used to probe the configurations, which is typical in biomedical studies. In the end, it effectively makes it unclear and uncertain which configuration yields the most reliable and interpretable results. Our goal is to navigate the myriad of pipeline configurations, ultimately ensuring that pipeline outputs remain both reproducible and clinically or scientifically meaningful.

In practice, neuroimaging pipeline outputs are complex representations of brain activity. There are several challenges when considering those: First, no single definitive ground truth exists given the indirect nature of brain activity measurement. Second, unlike simple metrics such as execution time, the output is a 3D structure – not a scalar or a quantitative label – requiring expert interpretation. Overall, we are in the situation where different variants may lead to different truths. This short paper introduces a "sampling, variant scoring, learning" approach to navigate variability in the absence of a ground truth and reports on our experience. We address four research questions:

**RQ1 (Variability Implementation):** How feasible is it to model and automate the exploration of neuroimaging pipeline configurations using a software product line approach?

**RQ2 (Accuracy):** How accurate is our learning approach in inferring quantitative performance against different proxy ground truths for unseen pipeline configurations?
**RQ3 (Interpretability):** How efficient is our method in identifying the key features affecting neuroimaging pipelines?

**RQ4 (Cost):** What are the computational costs and performance trade-offs of our approach?

# 2 Background

## 2.1 Imaging brain function

Task-based functional magnetic resonance imaging (fMRI) is a powerful neuroimaging technique used to study brain activity by tracking changes in blood flow. Human participants perform pre-defined tasks while in the MRI scanner – tasks that can range from simple motor tasks (e.g. finger taping) to more complex cognitive tasks (e.g. gambling). Such experiments aim to provide insight on the brain functions, by detecting which area of the brain are activated during these tasks. fMRI output data take the form of 3D matrices associating each voxel (i.e. volume pixel) with a level of activation.

In addition, like in many biomedical fields [7, 23], the data produced cannot be evaluated against a directly observable ground truth. Indeed the only alternative to measure brain functional activity is to use deep electrodes a process that is both invasive and cannot be performed across the full brain.

## 2.2 fMRI pipelines

Following the data acquisition, results are analyzed through a structured, software-driven workflow – i.e. pipeline. This process begins by preprocessing the data including: correcting for artifacts such as head motion or differences in acquisition times across brain slices, aligning to a standard brain template and smoothing (to enhance the signal-to-noise ratio and reduce any misalignment). Statistical analysis follows, typically employing a general linear model in each voxel (i.e. volume pixel) to estimate task-related activation.

At each of these stages, practitioners can modulate many parameters: algorithm to use for interpolation or for realignment, size of the smoothing kernel, shape of the hemodynamic response function (HRF), etc. Furthermore, typical neuroimaging pipelines can be composed using a variety of tools. The main open source software for fMRI include: SPM [27], FSL [35], and AFNI [9].

## 2.3 Motivation

In software product line (SPL) studies, evaluating variability and predicting the effect of different configurations generally relies on metrics to assess the impact of different features. For each configuration, qualitative or quantitative, functional and non-functional properties are measured. Quantitative measurement includes properties such as response time, runtime, number of defects, while qualitative metrics includes indicators such as output quality, reliability, compliance. Consequently, applying SPL methods typically relies on identifying properties that can objectively be measured. Looking at Figure 1 and the configuration matrix at the top, there is usually an associated quantitative or qualitative value.

In the context of neuroimaging the situation is different since the output is a set of statistic maps of the brain (see middle of Figure 1). Moreover, it brings us face to face with one of the major challenge of method development in this field: the absence of ground truth. Another related issue is that statistic maps are inherently complex and high-dimensional, making them far more difficult to interpret

and assess individually than scalar metrics like execution time or memory usage. Among the thousands of possible configurations, how can we determine which ones yield meaningful, reliable, and trustworthy results? As a result of an indirect observation, the statistic maps produced by neuroimaging pipelines can only be evaluated in relation to each other or using proxy ground truths, as there is no a gold standard against which to judge them.

# 3 Method

<figure><img src="image_3.png" /><figcaption>Figure 1: Our method to navigate variability</figcaption></figure>

In our quest to understand software variability in neuroimaging pipelines, we undertake to identify and evaluate a set of proxy ground truths against which we can learn about the configuration space. Figure 1 provides an overview of the stages of the proposed
method. Associated code is available at https://github.com/Inria-Empenn/neuro_variability_splc25.

## 3.1 Modeling

The first step is to model the fMRI pipeline configuration space in a representative and actionable way. As a case study, we choose to work on a single subject analysis with the SPM software.

To create a *feature model (FM)* expressing the functional pre-processing and single subject analysis steps, we relied at first on a reference handbook of fMRI analysis [30] to identify common steps across fMRI pipelines and best practices. Additionally, we refer to the SPM documentation [1] for implementation details specific to our case study. Finally, we review the FM with an expert (co-author), that provides us with practical insights and best practices. Throughout, we use the Universal Variability Language (UVL) [4] formalism, a community-driven language for variability models.

## 3.2 Sampling and Running

Several sampling strategies and criterion can be considered, e.g., [17, 20, 28, 37]. We implement a uniform random sampling strategy. To this aim, we use flamapy [19], a python framework dedicated to FM analysis and natively compatible with UVL. Specifically, we use the "Sampling" operation from flamapy, using Uniform Random Sampling through Binary Decision Diagrams (BDDs) to sample valid configurations.

As a sanity check, we assess the representativity of our 1000 features sample. To this aim – and thanks to the relatively limited complexity of the FM – we sample all of the 27,648 valid configurations and compare the distribution of features in the sample versus the full set of configurations.

From the sampled configurations, we build SPM-based pipelines using the Nipype framework [15]. With glue code, we transcribe configuration into a set of connected Nipype nodes, and configure the parameters according to selected features.

Each pipeline produces a statistic map, expressing the spatial patterns of activation levels across the brain.

## 3.3 Variant scoring

Each pipeline variant produces a full three-dimensional statistic map rather than a single numeric value. By contrast, the usual machine-learning formulation seeks an approximate function $f^*: X \to y$ with X the feature space (here: configuration options of the fMRI pipeline) and y a scalar target. Our approach is therefore to transform this high-dimensional output into a single quality score.

The goal of *variant scoring* is to produce a one-dimensional score that measures the quality of a given variant. Here, we propose to use a score that measures the adequacy between the statistic map produced by the variant and a predefined ground truth. The challenge is therefore two-fold: (i) to specify an appropriate ground truth for the comparison, and (ii) to design a distance function that collapses the three-dimensional discrepancy between a variant's map and the ground truth into a single, interpretable score.

In the search of such ground truth, we choose to work with a set of candidate proxy ground truths. As a first candidate, we compute the average statistic map across all variants, seen as a consensus

across configurations. Second, we craft a configuration based on expert knowledge. Finally, as a baseline, we randomly select 10 statistic maps among our 1000 outputs and use each of them as a candidate ground truth.

To determine each statistic map performance independently, we compute the Spearman correlations with each of the 12 proxy ground truths.

In addition, we calculate Spearman correlation between each pair of output statistic maps. This provides an assessment of the two-by-two agreements and disagreements across all the outputs.

## 3.4 Learning

Having computed quantitative measures derived from proxy ground truths to use as target variables, we implement a decision tree learning approaches.

### 3.4.1 Quantitative measures.
Through the use of the scikit-learn [26], we iteratively fit regression decision tree models (with default hyper-parameters) against the 12 target variables – i.e. the correlations computed in the previous variant scoring step against the 12 proxy ground truths (average, expert-based and 10 random). We train our model 10 times for each metric, with training sets from 100 to 700 configurations for each iteration. Average statistic maps and associated correlations are recomputed independently for each test and train subsets as to avoid data leakage.

We average and compute evaluation metrics such as mean absolute percentage error (MAPE) across all iterations, and generate learning curves and the decision tree graph.

### 3.4.2 Clustering.
We divide our dataset into 4 distinct clusters, using hierarchical clustering (with complete linkage criteria). Distance between configuration is based on the pairwise correlations.

We then fit a classifier decision tree models – with default hyper-parameters – to predict the cluster of each configuration. We train our model 10 times, with training sets from 100 to 700 configurations – tests set from 900 to 300 – for each iteration. Clustering is computed independently for each test and train subsets as to avoid data leakage. We use Hungarian matching to align our cluster across all subsets through their centroids.

# 4 Evaluation

## 4.1 Case study

We use a dataset from a single participant performing a simple auditory task [31]. Ninety-six brain volumes were acquired (TR [Repetition Time]=7 s), in 16 blocks of 6 acquisitions (i.e. 42 s in total). The condition for successive blocks alternated between rest and listening, starting with rest.

## 4.2 RQ1 (Variability Implementation)

Our UVL model contains 90 categorical features, including 30 cores (mandatory) features and 61 leaf features. In total, this represents 27,648 different configuration.

Our model has a maximum depth of 5. The 1st level is abstract ('pipeline'), and the 2nd represents the preprocessing and statistical analysis. The 3rd level represents processing steps such as realignment, smoothing or signal modeling. The parameters for each of these steps are on the 4th level (e.g. size of the smoothing kernel),
and the 5th level contains the range of parameters values (e.g. 2, 4, etc). An extract of the model (smoothing step) is shown in listing 1.

```
"spatial_smoothing"
optional
"spatial_smoothing/fwhm"
alternative
    "spatial_smoothing/fwhm/2"
    "spatial_smoothing/fwhm/4"
    "spatial_smoothing/fwhm/6" {reference 'yes'}
    "spatial_smoothing/fwhm/8"
    "spatial_smoothing/fwhm/10"
```

### Listing 1: Excerpt from UVL model (smoothing)

Individual components within Nipype are configured using glue code. In Nipype, the different steps are implemented as “Node”, chained together into a “Workflow” (i.e. the full pipeline). Each “Node” is setup according to the selected features. Listing 2 presents an example of code for the smoothing step.

```python
def get_smoothing(self, features: list):
    node = Node(interface=Smooth(), name="spatial_smoothing")
    node inputs.fwhm = 0
    if f"{name}/fwhm" in features:
        node inputs.fwhm = float(self.getFeatureEnd(f"{name}/fwhm", features))
    return node
```

### Listing 2: Python function for smoothing node configuration

In our current model, we do not take advantage of the advanced language levels offered by UVL, using the syntax only at Boolean level. Due to limitation in flamapy with the sampling of continuous value, features are always represented as categorical values. For some cases, the categories must be translated into continuous values – e.g. the smoothing kernel (in mm) must be expressed as an integer between 0 and n.

## 4.3 RQ2 (Accuracy)

As shown in Figure 2, among the tested proxy ground truths in the regression decision tree, the best performing target for prediction was the average statistic map over the configurations with an averaged MAPE – across ten iterations – to predict correlation to this target of 4.32% for a training set of 700 (70%) configurations.

<figure><img src="image_4.png" /><figcaption>Figure 2: Regressor decision tree learning curve</figcaption></figure>

Interestingly, correlation to the expert configuration statistic map MAPE (10.31%) is among the two worst performing of the ten randomly selected configurations (10.86% to 7.28% MAPE).

<table><caption>Table 1: Feature importances (correlations to average statistic maps)</caption><thead><tr><td>Feature</td><td>Importance [min-max]</td></tr></thead><tbody><tr><td>Norm. cross correlation (coreg. cost fct)</td><td>62.4% [60.2-64.9]</td></tr><tr><td>FWHM (smoothing)</td><td>31.0% [28.6-33.0]</td></tr><tr><td>First ref. slice (slice timing)</td><td>4.7% [4.1-5.3]</td></tr><tr><td>HRF derivatives</td><td>1.7% [1.1-1.9]</td></tr><tr><td>HRF temporal deriv.</td><td>0.2% [0.2-0.2]</td></tr></tbody></table>

## 4.4 RQ3 (Interpretability)

4.4.1 Regression decision tree. We evaluate and compare the feature importances for the model fitted on the correlations to the average statistic map (i.e. the best performing proxy ground truth in RQ2). We averaged feature importances produced with a 70% training set for the ten iterations (see table 1).

The feature corresponding to a normalized cross-correlation used as a cost function for coregistration has the highest importance value. This feature hence has the greatest impact for discriminating configurations with respect to our proxy ground truths. Similarly, the feature defining the size of the spatial smoothing kernel is the second most important in both datasets. This feature corresponds to the smoothing value applied to inputs during preprocessing.

Overall, we note that a large majority of features (84/90) in our model have an importance equal to zero – in particular these include all of spatial normalization or motion correction features.

4.4.2 Clustering decision tree. To evaluate the accuracy of the clustering decision tree, we use the accuracy, F1 and precision scores, averaging these values across ten iterations. For the largest training set (700 configurations), we obtained values of 89.8% accuracy, 89.5% F1 score and 93.3% precision.

Figure 3 shows the average statistic maps for each cluster. The visual assessment of those maps is consistent with the classification and decision tree obtained. Indeed, configurations with a larger smoothing kernel (FWHM > 3mm, clusters 2 and 4 on Figure 3) produce smoother outputs, whereas for lower FWHM values (clusters 1 and 3 on Figure 3), the outputs are noisier but also more spatially detailed. Similarly, the output from the configurations using normalised cross correlation as the cost function for coregistration (clusters 3 and 4) shows a misalignment of the statistic map with the background anatomical image.

These observations help to explain the high importance of both features. Spatial smoothing tends to reinforce spatially large areas of signal present on the maps while eliminating smaller clusters. In contrast, the normalised cross correlation method fails to align the brain correctly. This undesirable behaviour leads to output that are further away from the average of the statistic maps.

## 4.5 RQ4 (Cost)

4.5.1 Sampling. The sampling phase takes only a few seconds on a standard workstation, as expected given the relatively low complexity of the FM.
<figure><img src="image_5.png" /><figcaption>Figure 3: Decision tree for 4 clusters</figcaption></figure>

**4.5.2 Pipeline execution.** However, the pipelines execution phase is significantly more demanding, requiring some 50 hours of total runtime distributed across 40 nodes on a large computation cluster.

This corresponds to an average execution time per pipeline of 3 minutes – a relatively short time for a fMRI analysis, that can be explained by the small size of the dataset used in our experiment.

**4.5.3 Postprocessing.** The post-processing phase, which includes generating the average statistic maps and calculating a matrix of more than a million pairwise correlations takes approximately 30 hours on a single node. The generation of the average statistic map being relatively quick (a few minutes for 1000 outputs), the main cost of this step is the calculation of the pairwise correlations.

# 5 Related work

**Variability in workflows and neuroimaging pipelines.** Scientific workflows are increasingly used to automate complex data analysis processes across domains. These workflows often involve configurable steps, tools, and parameters, resulting in a large configuration space. Some studies have investigated variability in the specific context of neuroimaging. Botvinik-Nezer et al. [5] showed that different teams analyzing the same dataset with different pipelines reached substantially different conclusions. Existing works have explored sources of pipeline-induced variability, including software choice [6, 10], operating systems [14], and preprocessing methods [8, 16]. Recent studies also aim to map and understand these differences using clustering or metadata analysis [12]. However, very few contributions (including [10]) focuses on proposing methods to explore, analyze and understand it in the absence of ground truth. Our work contributes a principled, automated methodology

for studying variability in complex workflows, with applicability in the neuroimaging domain.

**Variability modeling and implementation.** Modelling and implementing variability has been studied in various contexts, including (scientific) workflows, data pipelines, business processes [21, 24]. The Common Variability Language [18] offers a framework for separating variability from base models and has been instantiated across domains. Techniques such as feature models [3], superimposition [2], and delta modeling [33] provide compositional and annotative mechanisms to generate and manage variants. We report on our experience with UVL [4] and implementation of variability on top of Nipype (see Section 4.2).

**Variability and learning.** Numerous studies in software engineering have explored configuration spaces using statistical machine learning [22, 25, 29, 32, 34]. These works typically frame the problem as one of classification or regression, predicting scalar outputs such as execution time or binary success/failure outcomes. In contrast, neuroimaging pipelines produce structured, non-scalar outputs that are challenging to assess and not directly quantifiable. In our work, we recast this challenge as a regression problem by computing Spearman correlations between each output and several proxy ground truths acting as reference. Our sampling, measuring, and learning methodology is, to the best of our knowledge, novel in this context. Moreover, we believe the underlying problem of structured output comparison without ground truth has broader implications beyond neuroimaging, applicable to any domain involving complex, high-dimensional outputs (e.g., logs, 3D models, videos).
# 6 Discussion

**Main results.** Our experiments have shown that our ninety-feature configuration space can be summarized in five features of importance using a regression decision tree. Similarly, two features – the two most important in the previous experiment – are sufficient to predict the cluster of a configuration among four. Another positive result of our method is that instead of reviewing potential "truths" from 1000 results we were able to identify 4 clusters of interest that are informative for domain experts and capable of explaining the underlying variability.

A comparison of the performance of our different target variables clearly shows correlation with the mean as an indicator of interest. The expert configuration performed worse than the randomly selected configuration, which calls for further investigation. Nonetheless, visual inspection of the statistic map shows as expected an activations in the auditory cortex demonstrating the quality of the expert-based configuration. Overall, the mean and expert references could be seen as complementary: While the expert reference can help identify key clusters and broader trends grounded in domain knowledge, it may have limitations in explaining all sources of variability. In contrast, the mean serves as a more robust statistical reference, revealing patterns across configurations.

As expected, computational costs show that hoping to execute and analyses all the configurations in the current modeled space is unrealistic, and that sampling is the way forward. Specifically, our learning method does not need to observe the whole 1000 configurations: 100 configurations already lead to highly accurate results that generalize to the whole variability space.

**Limitations.** A first limit to our work is the dataset we used in this study. Indeed, the use of data from a single subject and task introduces several limitations: 1/ the focus on an auditory task limits the exploration of variability across different cognitive processes 2/ the absence of inter-subject variability precludes the examination of how analytical choices interact with biological diversity.

We expect computation times to significantly increase with more realistic use cases (multi-subject, more complex tasks, etc.). In future work, it will be essential to pay close attention to the scalability of the setup. Furthermore, our FM focus on SPM, a single tool in a much larger and complex analytical space that encompasses a variety of tools, each with its own algorithms, parameters, and assumptions. Consequently, our findings may not be directly transferable to other pipelines. Besides, decision trees may struggle to capture complex, non-linear relationships in the data, leading to oversimplified models. They can also overfit, especially with small datasets. Additionally, they may not perform as well as more complex models. Other learning techniques may well be considered.

**Future work.** In our future work, we plan to expand our research by leveraging datasets from the Human Connectome Project (HCP) [36], a large open dataset including high-resolution, multi-paradigm fMRI data from over 1200 participants (see [13] for an example of data reuse). We aim to integrate additional tools into our variability model – e.g. fMRIPrep [11] – to broaden the scope of our analysis and enhance its applicability across different software environments. By better leveraging the various language levels within the UVL framework, we seek to capture more nuanced

aspects of variability and expand the coverage of our sampling. Additionally, we will explore methods to automatically detect invalid configurations. To better evaluate the accuracy and relevance of the pipelines outputs, we intend to develop automated quality assessment techniques, such as using masks of specific brain areas. Indeed, randomly sampling the configuration space allows to explore a wide variety of feature combinations but does not offer any guarantees in terms of correctness of the outputs of all the tested configurations. By excluding configurations producing incorrect outputs, we would like to constrain future experiments towards configurations that are more relevant and even representative of the results obtained in the field of neuroimaging. In particular, we believe that it would be useful to study how this impacts the performance of the expert-based candidate ground truth.

In the search for truth, navigating in large variability spaces remains a fundamental and open challenge, as diverse tools, configurations, and interpretations continue to yield potentially divergent outcomes. This calls for a broader community effort to develop principled, systematic approaches to characterize and manage such variability; our work represents a first step in that direction.

# Acknowledgments

YMJ, MA and CM were supported by ANR JCJC VICUNA (ANR-23-CE45-0022). YMJ was also supported by Region Bretagne (ARED GUANACO). Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several Universities as well as other organizations (see https://www.grid5000.fr).

# References

[1] The FIL Methods Group (and honorary members). 2021. *SPM12 Manual*. Functional Imaging Laboratory, Wellcome Centre for Human Neuroimaging, UCL Queen Square Institute of Neurology. Available at https://www.fil.ion.ucl.ac.uk/spm/doc/manual.pdf.

[2] Sven Apel, Christian Kästner, and Christian Lengauer. 2013. Language-Independent and Automated Software Composition: The FeatureHouse Experience. *IEEE Transactions on Software Engineering (TSE)* 39 (2013), 63–79.

[3] David Benavides, Sergio Segura, and Antonio Ruiz-Cortes. 2010. Automated Analysis of Feature Models 20 years Later: a Literature Review. *Information Systems* 35, 6 (2010).

[4] David Benavides, Chico Sundermann, Kevin Feichtinger, José A. Galindo, Rick Rabiser, and Thomas Thüm. 2025. UVL: Feature modelling with the Universal Variability Language. *Journal of Systems and Software* 225 (2025), 112326. https://doi.org/10.1016/j.jsv.2024.112326

[5] Rotem Botvinik-Nezer, Felix Holzmeister, Colin F. Camerer, Anna Dreber, Juergen Huber, Magnus Johannesson, Michael Kirchler, Roni Iwanir, Jeanette A. Mumford, R. Alison Adcock, Paolo Avesani, Blazej M. Baczkowski, Aahana Bajracharya, Leah Bakst, Sheryl Ball, Marco Barilari, Nadège Bault, Derek Beaton, Julia Beitner, Roland G. Benoit, Ruud M. W. J. Berkers, Jamil P. Bhanji, Bharat B. Biswal, Sebastian Bobadilla-Suarez, Tiago Bortolini, Katherine L. Bottenhorn, Alexander Bowring, Senne Braem, Hayley R. Brooks, Emily G. Brudner, Cristian B. Calderon, Julia A. Camilleri, Jaime J. Castrellon, Luca Cecchetti, Edna C. Cieslik, Zachary J. Cole, Olivier Collignon, Robert W. Cox, William A. Cunningham, Stefan Czoschke, Kamalaker Dadi, Charles P. Davis, Alberto De Luca, Mauricio R. Delgado, Lysia Demetriou, Jeffrey B. Dennison, Xin Di, Erin W. Dickie, Ekaterina Dobryakova, Claire L. Donnat, Juergen Dukart, Niall W. Duncan, Joke Durnez, Amr Eed, Simon B. Eickhoff, Andrew Erhart, Laura Fontanesi, G. Matthew Fricke, Shiguang Fu, Adriana Galván, Remi Gau, Sarah Genon, Tristan Glatard, Enrico Glerean, Jelle J. Goeman, Sergej A. E. Golowin, Carlos González-García, Krzysztof J. Gorgolewski, Cheryl L. Grady, Mikella A. Green, João F. Guassi Moréira, Olivia Guest, Shabnam Hakimi, J. Paul Hamilton, Roeland Hancock, Giacomo Handjaras, Bronson B. Harry, Colin Hawco, Peer Herholz, Gabrielle Herman, Stephan Heunis, Felix Hoffstaedter, Jeremy Hogeveen, Susan Holmes, Chuan-Peng Hu, Scott A. Huettel, Matthew E. Hughes, Vittorio Iacovella, Alexandru D. Iordan, Peder M. Isager, Ayse I. Isik, Andrew Jahn, Matthew R. Johnson, Tom Johnstone, Michael J. E. Joseph, Anthony C. Juliano, Joseph W. Kable, Michalis
Kassinopoulos, Cemal Koba, Xiang-Zhen Kong, Timothy R. Koscik, Nuri Erkut Kucukboyaci, Brice A. Kuhl, Sebastian Kupek, Angela R. Laird, Claus Lamm, Robert Langner, Nina Lauharatanahirun, Hongmi Lee, Sangil Lee, Alexander Leemans, Andrea Leo, Elise Lesage, Flora Li, Monica Y. C. Li, Phui Cheng Lim, Evan N. Lintz, Schuyler W. Liphardt, Annabel B. Losecaat Vermeer, Bradley C. Love, Michael L. Mack, Norberto Malpica, Theo Marinas, Camille Maumet, Kelsey McDonald, Joseph T. McGuire, Helena Melero, Adriana S. Méndez Leal, Benjamin Meyer, Kristin N. Meyer, Glad Mihai, Georgios D. Mitsis, Jorge Moll, Dyuan M. Nielson, Gustav Nilsonne, Michael P. Notter, Manuele Olivetti, Adrian I. Oicas, Paolo Papale, Kaustubh R. Patil, Jonathan E. Peelle, Alexandre Pérez, Doris Pischedda, Jean-Baptiste Poline, Yanina Prystauka, Shruti Ray, Patricia A. Reuter-Lorenz, Richard C. Reynolds, Emiliano Ricciardi, Jenny R. Rieck, Anais M. Rodriguez-Thompson, Anthony Romyn, Taylor Salo, Gregory R. Samanez-Larkin, Emilio Sanz-Morales, Margaret L. Schlichting, Douglas H. Schultz, Qiang Shen, Margaret A. Sheridan, Jennifer A. Silvers, Kenny Skagerlund, Alec Smith, David V. Smith, Peter Sokol-Hessner, Simon R. Steinkamp, Sarah M. Tashjian, Bertrand Thirion, John N. Thorp, Gustav Tinghög, Loreen Tisdall, Steven H. Tompson, Claudio Toro-Serey, Juan Jesus Torre Tresols, Leonardo Tozzi, Vuong Truong, Luca Turella, Anna E. Van 'T Veer, Tom Verguts, Jean M. Vettel, Sagana Vijayarajah, Khoi Vo, Matthew B. Wall, Wouter D. Weeda, Susanne Weis, David J. White, David Wisniewski, Alba Xifora-Porxas, Emily A. Yearling, David S. Yoon, Rui Yuan, Kenneth S. L. Yuen, Lei Zhang, Xu Zhang, Joshua E. Zosky, Thomas E. Nichols, Russell A. Poldrack, and Tom Schonberg. 2020. Variability in the analysis of a single neuroimaging dataset by many teams. *Nature* 582, 781 (June 2020), 84–88. https://doi.org/10.1038/s41586-020-2314-9

[6] Alexander Bowring, Thomas E. Nichols, and Camille Maumet. 2022. Isolating the sources of pipeline-variability in group-level task-fMRI results. 43, 3 (2022), 1112–1128. https://doi.org/10.1002/hbm.25713 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/hbm.25713.

[7] Mikaela Cashman, Myra B. Cohen, Priya Ranjan, and Robert W. Cottingham. 2018. Navigating the maze: the impact of configurability in bioinformatics software. In *Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering*. ACM, Montpellier France, 757–767. https://doi.org/10.1145/3238147.3240466

[8] Nathan W. Churchill, Robyn Spring, Babak Afshin-Pour, Fan Dong, and Stephen C. Strother. 2015. An Automated, Adaptive Framework for Optimizing Preprocessing Pipelines in Task-Based Functional MRI. 10, 7 (2015), e0131520. https://doi.org/10.1371/journal.pone.0131520 Publisher: Public Library of Science.

[9] Robert W Cox. 1996. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. *Computers and Biomedical research* 29, 3 (1996), 162–173.

[10] Jessica Dafflon, Pedro F. Da Costa, František Váša, Ricardo Pio Monti, Danilo Bzdok, Peter J. Hellyer, Federico Turkheimer, Jonathan Wood, Emily Jones, and Robert Leech. 2022. A guided multiverse study of neuroimaging analyses. 13, 1 (2022), 3758. https://doi.org/10.1038/s41467-022-31347-8 Number: 1 Publisher: Nature Publishing Group.

[11] Oscar Esteban, Christopher J. Markiewicz, Ross W. Blair, Craig A. Moodie, A. Ilkay Isik, Asier Erramuzpe, James D. Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, Hiroyuki Oya, Satrajit S. Ghosh, Jessey Wright, Joke Durnez, Russell A. Poldrack, and Krzysztof J. Gorgolewski. 2019. fMRIPrep: a robust preprocessing pipeline for functional MRI. *Nature Methods* 16, 1 (Jan. 2019), 111–116. https://doi.org/10.1038/s41592-018-0235-4

[12] Elodie Germani, Elisa Fromont, and Camille Maumet. 2024. Uncovering communities of pipelines in the task-fMRI analytical space. In *IEEE International Conference on Image Processing*.

[13] Elodie Germani, Elisa Fromont, Pierre Maurel, and Camille Maumet. 2024. The HCP multi-pipeline dataset: an opportunity to investigate analytical variability in fMRI data analysis. arXiv:2312.14493 [q-bio.NC] https://arxiv.org/abs/2312.14493

[14] Tristan Glatard, Lindsay B. Lewis, Rafael Ferreira Da Silva, Reza Adalat, Natacha Beck, Claude Lepage, Pierre Rioux, Marc-Etienne Rousseau, Tarek Sherif, Ewa Deelman, Najmeh Khalili-Mahani, and Alan C. Evans. 2015. Reproducibility of neuroimaging analyses across operating systems. *Frontiers in Neuroinformatics* 9 (April 2015). https://doi.org/10.3389/fninf.2015.00012

[15] Krzysztof Gorgolewski, Christopher D. Burns, Cindee Madison, Dav Clark, Yaroslav O. Halchenko, Michael L. Waskom, and Satrajit S. Ghosh. 2011. Nipype: A Flexible, Lightweight and Extensible Neuroimaging Data Processing Framework in Python. *Frontiers in Neuroinformatics* Volume 5 - 2011 (2011). https://doi.org/10.3389/fninf.2011.00013

[16] Ed H. B. M. Gronenschild, Petra Habets, Heidi I. L. Jacobs, Ron Mengelers, Nico Rozendaal, Jim van Os, and Machteld Marcelis. 2012. The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements. (2012). https://doi.org/10.1371/journal.pone.0038234

[17] Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. 2019. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack. *Empir. Softw. Eng.* 24, 2 (2019), 674–717. https://doi.org/10.1007/S10664-018-9635-4

[18] Øystein Haugen, Andrzej Wasowski, and Krzysztof Czarnecki. 2012. CVL: common variability language. In *Proceedings of the 16th International Software Product Line Conference-Volume 2*. 266–267.

[19] JM Horcas, JA Galindo, and D Benavides. 2022. Flamapy: Python-based AAFM framework.

[20] Christian Kalteneger, Alexander Grebhahn, Norbert Siegmund, Jianmei Guo, and Sven Apel. 2019. Distance-Based Sampling of Software Configuration Spaces. In *Proceedings of the International Conference on Software Engineering (ICSE)*.

[21] Jacob Krüger, Gül Çalıklı, Thorsten Berger, Thomas Leich, and Gunter Saake. 2019. Effects of explicit feature traceability on program comprehension. In *Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering*. 338–349.

[22] Chengjie Lu, Yize Shi, Huihui Zhang, Man Zhang, Tiexin Wang, Tao Yue, and Shaukat Ali. 2022. Learning configurations of operating environment of autonomous vehicles to maximize their collisions. *IEEE Transactions on Software Engineering* 49, 1 (2022), 384–402.

[23] Anders Lundgren and Upulee Kanewala. 2016. Experiences of testing bioinformatics programs for detecting subtle faults. In *Proceedings of the International Workshop on Software Engineering for Science*. ACM, Austin Texas, 16–22. https://doi.org/10.1145/2897676.2897678

[24] Wardah Mahmood, Daniel Strüber, Anthony Anjorin, and Thorsten Berger. 2022. Effects of Variability in Models: A Family of Experiments. *Empirical Softw. Engg.* 27, 3 (may 2022), 38 pages. https://doi.org/10.1007/s10664-021-10112-3

[25] Vivek Nair, Tim Menzies, Norbert Siegmund, and Sven Apel. 2017. Using Bad Learners to Find Good Configurations. In *ESECFSE*. 257–267.

[26] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Courapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-Learn: Machine Learning in Python. *Journal of Machine Learning Research* 12 (2011), 2825–2830.

[27] William D Penny, Karl J Friston, John T Ashburner, Stefan J Kiebel, and Thomas E Nichols. 2006. *Statistical parametric mapping: the analysis of functional brain images*. Elsevier.

[28] Juliana Alves Pereira, Mathieu Acher, Hugo Martin, and Jean-Marc Jézéquel. 2020. Sampling Effect on Performance Prediction of Configurable Systems: A Case Study. In *ICPE '20: ACM/SPEC International Conference on Performance Engineering, Edmonton, AB, Canada, April 20-24, 2020*, José Nelson Amaral, Anne Koziolek, Catia Trubiani, and Alexandru Iosup (Eds.). ACM, 277–288. https://doi.org/10.1145/3358960.3379137

[29] Juliana Alves Pereira, Mathieu Acher, Hugo Martin, Jean-Marc Jézéquel, Goetz Botterweck, and Anthony Ventresque. 2021. Learning software configuration spaces: A systematic literature review. *J. Syst. Softw.* 182 (2021), 111044. https://doi.org/10.1016/j.jss.2021.111044

[30] Russell A Poldrack, Jeanette A Mumford, and Thomas E Nichols. 2011. *Handbook of functional MRI data analysis*. Cambridge University Press.

[31] Geraint Rees and Karl Friston. 1999. Mother of All Experiments. https://www.fil.ion.ucl.ac.uk/spm/data/auditory.

[32] A. Sarkar, Jianmei Guo, N. Siegmund, S. Apel, and K. Czarnecki. 2015. Cost-Efficient Sampling for Performance Prediction of Configurable Systems (T). In *ASE '15*.

[33] Ina Schaefer, Lorenzo Bettini, Viviana Bono, Ferruccio Damiani, and Nico Tazzarella. 2010. Delta-Oriented Programming of Software Product Lines. In *Software Product Lines: Going Beyond*, Jan Bosch and Jaejoon Lee (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 77–91.

[34] Norbert Siegmund, Alexander Grebhahn, Christian Kästner, and Sven Apel. 2015. Performance-Influence Models for Highly Configurable Systems. In *ESEC/FSE '15*.

[35] Stephen M Smith, Mark Jenkinson, Mark W Woolrich, Christian F Beckmann, Timothy EJ Behrens, Heidi Johansen-Berg, Peter R Bannister, Marilena De Luca, Ivana Drobnjak, David E Flitney, et al. 2004. Advances in functional and structural MR image analysis and implementation as FSL. *Neuroimage* 23 (2004), S208–S219.

[36] David C. Van Essen, Stephen M. Smith, Deanna M. Barch, Timothy EJ. Behrens, Essa Yacoub, and Kamil Ugurbil. 2013. The WU-Minn Human Connectome Project: An overview. *NeuroImage* 80 (Oct. 2013), 62–79. https://doi.org/10.1016/j.neuroimage.2013.05.041

[37] Mahsa Varshosaz, Mustafa Al-Hajjaji, Thomas Thüm, Tobias Runge, Mohammad Reza Mousavi, and Ina Schaefer. 2018. A classification of product sampling for software product lines. In *Proceedings of the 22nd International Systems and Software Product Line Conference - Volume 1, SPLC 2018, Gothenburg, Sweden, September 10-14, 2018*, Thorsten Berger, Paulo Borba, Goetz Botterweck, Tomi Männistö, David Benavides, Sarah Nadi, Timo Kehrer, Rick Rabiser, Christoph Elsner, and Mukelabai Mukelabai (Eds.). ACM, 1–13. https://doi.org/10.1145/3233027.3233035