# Software Variability for Replicable Science

Mathieu Acher

► To cite this version:

Mathieu Acher. Software Variability for Replicable Science. CBSoft 2025 - Congresso Brasileiro de Software: Teoria e Prática, Sep 2025, Rio, Brazil. hal-05334168

HAL Id: hal-05334168

https://hal.science/hal-05334168v1

Submitted on 28 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Software Variability for
Replicable Science

Mathieu Acher

@acherm https://mathieuacher.com/
# Software Variability for Replicable Science

**Abstract:** The ability to recreate computational results provides a solid foundation for scientific research. Yet, reproducibility and getting identical results with the original data, code, and environment are challenging, due to many uncontrolled or undocumented variability factors. When reproduction is achieved, replicability can be envisioned to check whether conclusions still hold or generalize when well-controlled changes—created by inevitable software variability—are introduced. In this talk, I will first provide evidence that deep software variability — spanning operating systems, compilers, input data, versions, and configurations, etc. — is impacting reproducibility and replicability in numerous fields of computational science. I will then present an approach based on "modelling, sampling, measuring, learning" to systematically explore variability spaces of neuroimaging pipelines. I will also illustrate how Masters` students et INSA Rennes leverage variability to reproduce and replicate studies in soccer, chess, and energy consumption. Throughout this talk, I will try to convince the audience that software-engineering and variability researchers have a key role to play for truly replicable science.

https://cbsoft.sbc.org.br/2025/sbcars/palestras/?lang=en

Brazilian Symposium on Software Components, Architectures, and Reuse

<figure><img src="image_2.png" /></figure>
# AGENDA

## Frictionless Reproducibility and (Deep) Software (Variability)

- Modelling, Sampling, Measuring, Learning Variability Spaces

- Students leverage variability to reproduce and replicate studies

- Mastering variability is key for truly replicable science
# Science is changing: Computation-based research

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

PHOTO: THE VOORHIES

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>
# Computational science depends on software and its engineering

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

design of mathematical model
mining and analysis of data
executions of large simulations
problem solving
executable paper

from a set of scripts to automate the deployment to... a
comprehensive system containing several features that
help researchers exploring various hypotheses

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>
“Insanity is doing the same thing over and over again
and expecting different results”

http://throwgrammarfromthetrain.blogspot.com/2010/10/definition-of-insanity.html

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

CMake
Cross-platform Make

<figure><img src="image_29.png" /></figure>

NumPy

<figure><img src="image_30.png" /></figure>

puppet

<figure><img src="image_31.png" /></figure>

docker

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

GCC

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>
# Reproducibility

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

(Claerbout/Donoho/Peng definition)

> "The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures." (~executable paper)

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>
# Reproducibility

Data

<figure><img src="image_40.png" /></figure>

**Analysis**
(Code+Computing Environment)

<figure><img src="image_41.png" /></figure>

Results

<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

**Analysis**
(Code+Computing Environment)

<figure><img src="image_44.png" /></figure>

Results

<figure><img src="image_45.png" /></figure>
# Reproducibility

Data

<figure><img src="image_46.png" /></figure>

**Analysis**
(Code+Computing Environment)

<figure><img src="image_47.png" /></figure>

Results

<figure><img src="image_48.png" /></figure>

Data

<figure><img src="image_49.png" /></figure>

**Analysis**
(Code+Computing Environment)

<figure><img src="image_50.png" /></figure>

Results

<figure><img src="image_51.png" /></figure>
# Reproducibility and Variability

<figure><img src="image_52.png" /></figure>

**Analysis**
(Code+Computing Environment)

<figure><img src="image_53.png" /></figure>

**Results**

**Which parameters of the brain mask were used?**

**something else?**

**Which release/version of SPM was used?**

<figure><img src="image_54.png" /></figure>

**Analysis**
(Code+Computing Environment)

**Data**

**Did we understand the method correctly?**

**Was it a different computing platform?**

<figure><img src="image_55.png" /></figure>

**Results**

<figure><img src="image_56.png" /></figure>
# Reproducible science

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

**Socio-technical issues:** open science, open source software, multi-disciplinary collaboration, incentives/rewards, initiatives, etc. with many challenges related to data acquisition, knowledge organization/sharing, etc.

<figure><img src="image_57.png" /></figure>

RR
FR Reproducible Research network

<figure><img src="image_58.png" /></figure>

https://www.recherche-reproductible.fr

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>

Software Heritage

https://www.softwareheritage.org

<figure><img src="image_61.png" /></figure>

ACM REP '25

Participate • Attend • Program • ACM REP • ACM EIGREP • Q

2025 ACM Conference on Reproducibility and Replicability

https://acm-rep.github.io/2025/

<figure><img src="image_62.png" /></figure>

Reproducible
Builds

Reproducible builds are a set of software development practices that create an
independently- verifiable path from source to binary code. (MIT)

https://reproducible-builds.org/
# Replicability and Variability

Data

<figure><img src="image_63.png" /></figure>

**Analysis**
(Code+Computing Environment)

<figure><img src="image_64.png" /></figure>

Results

<figure><img src="image_65.png" /></figure>
# Reproducibility, Replicability, and Variability

**Reproducible:** Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

"Terminologies for Reproducible Research", Lorena A. Barba, 2018

**=> finding one variant (and fixing some variability) to retrieve exact/similar results**

**Replication:** A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses.

**=> exploring variants' and embracing variability**

<figure><img src="image_66.png" /></figure>

<figure><img src="image_67.png" /></figure>

CMake
Cross-platform Make

<figure><img src="image_68.png" /></figure>

NumPy

<figure><img src="image_69.png" /></figure>

puppet

<figure><img src="image_70.png" /></figure>

docker

<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /></figure>

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>
# Frictionless Reproducibility?

Reading a paper in 2024 is sometimes like in 1970:

* Where is the source code? (eg implementation of the solution, scripts to compute metrics)

* Where is the data? (eg to test the solution)

* Contacting authors?

* no response?

* code not consistent with the PDF

* ...

* It does not work on my machine; results are completely different...

**There are lots of socio-technical frictions... even when you have the code and data!**

=> When people can replicate an analysis at the touch of a button using open-source software, open data, and methods to assess and compare proposals, it significantly eases verification of results, engagement with a diverse range of contributors, and progress
# Frictionless reproducibility (an example)

Cutting through buggy adversarial example defenses:
fixing 1 line of code breaks SABRE

Nicholas Carlini

Google DeepMind

<figure><img src="image_75.png" /></figure>

## Abstract

SABRE is a defense to adversarial examples that was accepted at IEEE S&P 2024. We first reveal significant flaws in the evaluation that point to clear signs of gradient masking. We then show the cause of this gradient masking: a bug in the original evaluation code. By fixing a single line of code in the original repository, we reduce SABRE's robust accuracy to 0%. In response to this, the authors modify the defense and introduce a new defense component not described in the original paper. But this fix contains a second bug; modifying one more line of code reduces robust accuracy to *below* baseline levels. After we released the first version of our paper online, the authors introduced another change to the defense; by commenting out one line of code during attack we reduce the robust accuracy to 0% again.

```bash
diff --git a/core/defenses/sabre.py b/core/defenses/sabre.py
```

2 index fe509e6..bf13629 100644

3 --- a/core/defenses/sabre.py

4 +++ b/core/defenses/sabre.py

5 @@ -165,7 +165,7 @@ class SabreWrapper nn.Module):

model = Sabre eps=eps, wave=wave, use_random=use_random, n

6 self.core = model

7 self.base_model = base_model

8 - self transform = BPDAWrapper lambda x, lambda r: model transform x, lambda r float())

9 + self transform = (lambda x, lambda r: model transform x, lambda r).float())

10 @property

11 def lambda_r(self):

## Submission history

From: Nicholas Carlini [view email]

[v1] Mon, 6 May 2024 17:48:24 UTC (19 KB)

[v2] Mon, 27 May 2024 17:41:06 UTC (20 KB)
# Reproducible/replicable science as a (deep) software variability problem

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

Despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results.

<figure><img src="image_76.png" /></figure>

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses

<figure><img src="image_77.png" /></figure>

Climate is changing

<figure><img src="image_78.png" /></figure>

<figure><img src="image_79.png" /></figure>

<figure><img src="image_80.png" /></figure>

The sea level is rising

<figure><img src="image_81.png" /></figure>

<figure><img src="image_82.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹, Xavier Yepes-Arbós¹,⁵, Eleftheria Exarchou¹,⁵,², and Francisco J. Doblas-Reyes¹,⁵,²,⁴

(a) Model code

EC-EARTH

(b) Different environments

<figure><img src="image_83.png" /></figure>

<figure><img src="image_84.png" /></figure>

Computing environment A

(c) Five-member 20-year ensemble simulations (constant forcing)

<figure><img src="image_85.png" /></figure>

(d) Calculation of standard indices of performance

<figure><img src="image_86.png" /></figure>

(e) Statistical testing of differences in performance

<figure><img src="image_87.png" /></figure>

(f) Conclusion on replicability

<figure><img src="image_88.png" /></figure>

Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? Using two versions of EC-Earth: one “non-replicable” case (see below) and one replicable case.

12m difference between five-member experiments m05e and e011. Black dotted regions indicate where the difference is significant according to a Kolmogorov-Smirnov test (39% of grid points show a significant difference)

<figure><img src="image_89.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹,², Xavier Yepes-Arbós²,⁵ Eleftheria Exarchou¹,², and Francisco J. Doblas-Reyes¹,²,⁴

<figure><img src="image_90.png" /></figure>

Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? Using two versions of EC-Earth: one “non-replicable” case (see below) and one replicable case.

<table><caption>Table 2. The four experiments considered in this study.</caption><thead><tr><th>Experiment ID</th><th>e011</th><th>m06e</th><th>a0gi</th><th>a0go</th></tr></thead><tbody><tr><td>Computing environment</td><td>ECMWF-CCA</td><td>MareNostrum3</td><td>ECMWF-CCA</td><td>MareNostrum3</td></tr><tr><td>EC-Earth version</td><td>3.1</td><td>3.1</td><td>3.2</td><td>3.2</td></tr><tr><td>Processors (IFS+NEMO+OASIS)</td><td>598</td><td>512</td><td>432 (288 + 144)</td><td>416 (288 + 128)</td></tr><tr><td rowspan="4">F flags</td><td>(480 + 96 + 22)</td><td>(384 + 96 + 22)</td><td>(OASIS: library)</td><td>(OASIS: library)</td></tr><tr><td>-02-g</td><td>-02-g</td><td>-02-g</td><td>-02-fp-model</td></tr><tr><td>-traceback</td><td>-traceback</td><td>-traceback -r8</td><td>precise -xHost</td></tr><tr><td>-vec-report0 -r8</td><td>-vec-report0 -r8</td><td>-fp-model strict</td><td>-g -traceback</td></tr><tr><td rowspan="2">C flags</td><td>-vec-report0 -r8</td><td>-vec-report0 -r8</td><td>-fp-model strict</td><td>-g -traceback</td></tr><tr><td>-02-g</td><td>-02-g</td><td>-xHost</td><td>-r8</td></tr><tr><td rowspan="3">LD flags</td><td>-traceback</td><td>-traceback</td><td>-02-g</td><td>-02-fp-model</td></tr><tr><td>-02-g</td><td>-02-g</td><td>-traceback</td><td>precise -xHost</td></tr><tr><td>-traceback</td><td>-traceback</td><td>-fp-model strict</td><td>-g -traceback</td></tr><tr><td>Output size</td><td>141.8 GB</td><td>141.6 GB</td><td>101.3 GB</td><td>101.3 GB</td></tr></tbody></table>

<figure><img src="image_91.png" /><figcaption>Table 1. The two computing environments considered in this study.</figcaption></figure>

<table><tbody><tr><td>Computing environment</td><td>ECMWF-CCA</td><td>MareNostrum3</td></tr><tr><td>Location</td><td>Reading, UK</td><td>Barcelona, Spain</td></tr><tr><td>Motherboard</td><td>Cray XC30 system</td><td>IBM dx360 M4</td></tr><tr><td>Processor</td><td>Dual 12-core E5-2697 v2 (Ivy Bridge) series processors (2.7 GHz), 24 cores per node</td><td>2x Intel SandyBridge-EP E5-2670/1600 20M 8-core at 2.6GHz, 16 cores per node</td></tr><tr><td>Operating system</td><td>Cray Linux Environment (CLE) 5.2</td><td>Linux - SuSe distribution 11 SP2</td></tr><tr><td>Compiler</td><td>Intel(R) 64 Compiler XE for applications running on Intel(R) 64 version 14.0.1.106 build 20131008</td><td>Intel(R) 64 Compiler XE for applications running on Intel(R) 64 version 13.0.1.117 build 20121010</td></tr><tr><td>MPI version</td><td>Cray mpich2 v6.2.0</td><td>Intel MPI v4.1.3.049</td></tr><tr><td>LAPACK version</td><td>Cray libsci v12.2.0</td><td>Intel MKL v11.0.1</td></tr><tr><td>SZIP, HDF5, NetCDF4</td><td>v2.1, v1.8.11, v4.3.0</td><td>v2.1, v1.8.14, v4.2</td></tr><tr><td>GribAPI, GribEX</td><td>v1.13.0, v000395</td><td>v1.14.0, v000370</td></tr></tbody></table>

<figure><img src="image_92.png" /></figure>
# Should software version numbers determine science?

> PLoS One. 2012;7(6):e38234. doi: 10.1371/journal.pone.0038234. Epub 2012 Jun 1.

The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements

Ed H B M Gronenschild ¹, Petra Habets, Heidi I L Jacobs, Ron Mengelers, Nico Rozendaal, Jim van Os, Machteld Marcelis

Significant differences were revealed between FreeSurfer version v5.0.0 and the two earlier **versions**. [...] About a factor two smaller differences were detected between Macintosh and Hewlett-Packard **workstations** and between **OSX 10.5 and OSX 10.6**. The observed differences are similar in magnitude as effect sizes reported in **accuracy** evaluations and neurodegenerative-studies.

<figure><img src="image_93.png" /></figure>

see also Krefting, D., Scheel, M., Freing, A., Specovius, S., Paul, F., and Brandt, A. (2011). “Reliability of quantitative neuroimage analysis using freesurfer in distributed environments,” in *MICCAI Workshop on High-Performance and Distributed Computing for Medical Imaging*. (Toronto, ON)

<figure><img src="image_94.png" /></figure>
"Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed."

Reproducibility of neuroimaging analyses across operating systems, Glatard et al., Front. Neuroinform., 24 April 2015

The implementation of mathematical functions manipulating single-precision **floating-point numbers in libmath** has evolved during the last years, leading to numerical differences in computational results. While these differences have little or no impact on simple analysis pipelines such as brain extraction and cortical tissue classification, their **accumulation** creates **important differences in longer pipelines** such as the subcortical tissue classification, RSfMRI analysis, and cortical thickness extraction.

<figure><img src="image_95.png" /></figure>

<table><thead><tr><th></th><th>Cluster A</th><th>Cluster B</th></tr></thead><tbody><tr><th>Applications</th><td>Freesurfer 5.3.0, build 1<br>FSL 5.0.6, build 1<br>CIVET 1.1.12-UCSF, build 1</td><td>Freesurfer 5.3.0, build 1 and 2<br>FSL 5.0.6, build 1 and 2<br>CIVET 1.1.12-UCSF, build 1</td></tr><tr><th>Interpreters</th><td>Python 2.4.3, bash 3.2.25,<br>Perl 5.8.8, tcsh 6.14.00</td><td>Python 2.7.5, bash 4.2.47,<br>Perl 5.18.2, tcsh 6.18.01</td></tr><tr><th>glibc version</th><td>2.5</td><td>2.18</td></tr><tr><th>OS</th><td>CentOS 5.10</td><td>Fedora 20</td></tr><tr><th>Hardware</th><td>x86_64 CPUs (Intel Xeon)</td><td>x86_64 CPUs (Intel Xeon)</td></tr></tbody></table>
"Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed."

Reproducibility of neuroimaging analyses across operating systems, Glatard et al., Front. Neuroinform., 24 April 2015

Statically building programs improves reproducibility across OSes, but small differences may still remain when dynamic libraries are loaded by static executables[...]. When static builds are not an option, software heterogeneity might be addressed using virtual machines. However, such solutions are only workarounds: differences may still arise between **static executables built on different OSes**, or between **dynamic executables executed in different VMs**.

<figure><img src="image_96.png" /></figure>

<table><thead><tr><th></th><th>Cluster A</th><th>Cluster B</th></tr></thead><tbody><tr><th scope="row">Applications</th><td>Freesurfer 5.3.0, build 1<br>FSL 5.0.6, build 1<br>CIVET 1.1.12-UCSF, build 1</td><td>Freesurfer 5.3.0, build 1 and 2<br>FSL 5.0.6, build 1 and 2<br>CIVET 1.1.12-UCSF, build 1</td></tr><tr><th scope="row">Interpreters</th><td>Python 2.4.3, bash 3.2.25,<br>Perl 5.8.8, tcsh 6.14.00</td><td>Python 2.7.5, bash 4.2.47,<br>Perl 5.18.2, tcsh 6.18.01</td></tr><tr><th scope="row">glibc version</th><td>2.5</td><td>2.18</td></tr><tr><th scope="row">OS</th><td>CentOS 5.10</td><td>Fedora 20</td></tr><tr><th scope="row">Hardware</th><td>x86_64 CPUs (Intel Xeon)</td><td>x86_64 CPUs (Intel Xeon)</td></tr></tbody></table>

<figure><img src="image_97.png" /></figure>
<figure><img src="image_98.png" /></figure>

Despite the availability of data and code, several studies report that the same data analyzed with **different software** can lead to **different results**

Many **layers** (operating system, third-party libraries, versions, workloads, compile-time options and flags, etc.) themselves subject to variability can alter the results.

DEEP VARIABILITY

<figure><img src="image_99.png" /></figure>

**Reproducible science and **deep** software variability:** a threat and opportunity for scientific knowledge!

<figure><img src="image_100.png" /></figure>
Deep Reinforcement Learning that Matters

Peter Henderson¹*, Riashat Islam¹,²*, Philip Bachman²
Joelle Pineau¹, Doina Precup¹, David Meger¹

What is the magnitude of the effect
**hyperparameter** settings can have on baseline
performance?

How does the choice of **network architecture** for
the policy and value function approximation affect
performance?

How can the **reward scale** affect results?

Can **random seeds** drastically alter performance?

How do the **environment properties** affect
variability in reported RL algorithm performance?

Are commonly used baseline **implementations**
comparable?

torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision

David Picard

LIGM, École des Ponts, 77455 Marnes la vallée, France

DAVID.PICARD@ENPC.FR

Abstract

In this paper I investigate the effect of random seed selection on the accuracy when using popular deep learning architectures for computer vision. I scan a large amount of seeds (up to 10⁴) on CIFAR 10 and I also scan fewer seeds on Imagenet using pre-trained models to investigate large scale datasets. The conclusions are that even if the variance is not very large, it is surprisingly easy to find an outlier that performs much better or much worse than the average.

<figure><img src="image_101.png" /></figure>
Reproducible and replicable CFD: it's harder than you think

Olivier Mesnard, Lorena A. Barba

Mechanical and Aerospace Engineering, George Washington University, Washington DC 20052

> "Completing a full replication study of our previously published findings on bluff-body aerodynamics was harder than we thought. Despite the fact that we have good reproducible-research practices, sharing our code and data openly."

**Story 1: Meshing and boundary conditions can ruin everything**

**Story 3: All linear algebra libraries are not created equal**

**Story 4: Different versions of your code, external libraries or even compilers may challenge reproducibility**

<figure><img src="image_102.png" /></figure>
# Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, ... Tom Schonberg

+ Show authors

*Nature* **582**, 84–88 (2020) | Cite this article

42k Accesses

203 Citations

2056 Altmetric

Metrics

Data analysis workflows in many scientific domains have become increasingly complex and *flexible* (=<u>subject to variability</u>). Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. *This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology.*

Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that *analytical flexibility can have substantial effects on scientific conclusions*, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to *analytical variability* are discussed.
REAL WORLD EXAMPLE (x264)

<figure><img src="image_103.png" /></figure>
# AGENDA

Frictionless Reproducibility and (Deep) Software (Variability)

**Modelling, Sampling, Measuring, Learning Variability Spaces**

Students leverage variability to reproduce and replicate studies

Mastering variability is key for truly replicable science
# Sampling and learning (regression, classification)

<figure><img src="image_104.png" /></figure>

<figure><img src="image_105.png" /></figure>

Whole
Population of
Configurations

<figure><img src="image_106.png" /></figure>

Training
Sample

<figure><img src="image_107.png" /></figure>

Performance
Measurements

<figure><img src="image_108.png" /></figure>

Prediction
Model

<figure><img src="image_109.png" /></figure>

Performance
Prediction

<figure><img src="image_110.png" /></figure>

$f: \mathbb{C} \rightarrow \mathbb{R}^+$ the function affecting to any configuration $c \in \mathbb{C}$ its performance $f(c) \in \mathbb{R}^+$,

<figure><img src="image_111.png" /></figure>

J. Alves Pereira, H. Martin, M. Acher, J.-M. Jézéquel, G. Botterweck and A. Ventresque
"Learning Software Configuration Spaces: A Systematic Literature Review" JSS, 2021²⁸
# Modelling Variability Spaces

* Abstractions are definitely needed to...

* reason about logical constraints and interactions

* integrate domain knowledge

* synthesize domain knowledge

* automate and guide the exploration of variants

* scope and prioritize experiments

* Language and formalism: **feature model** (widely applicable!)

* translation to logics

* reasoning with SAT/CP/SMT solvers

FAMiliAR

<figure><img src="image_112.png" /></figure>

<figure><img src="image_113.png" /></figure>

Fig. 1: Web UI for the model

<figure><img src="image_114.png" /></figure>

<figure><img src="image_115.png" /></figure>
# Modelling Variability Spaces

* Abstractions are definitely needed...

* Yes, but how to obtain a feature model?

* modelling

* reverse engineering (out of command-line parameters, source code, logs, configurations, etc.)

* learning (next slide!)

* modeling+reverse engineering+learning (Acher et al. 2021)

<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>
# Performance Prediction

Modelling

$f: \mathbb{C} \rightarrow \mathbb{R}^+$ the function affecting to any configuration

$c \in \mathbb{C}$ its performance $f(c) \in \mathbb{R}^+$,

x264 --no-mbtree
--rc-lookahead 40
--ref 9
-o trailer_480p24.x264
trailer_2k_480p24.y4m

<figure><img src="image_118.png" /></figure>

??? seconds

Sampling/Measuring

<table><thead><tr><th>no_8x8dct</th><th>no_asm</th><th>no_cabac</th><th>no_deblock</th><th>no_fast_pskip</th><th>no_mbtree</th><th>no_mixed_refs</th><th>no_weighttb</th><th>rc_lookahead</th><th>ref</th><th>size</th><th>elapsedtime</th></tr></thead><tbody><tr><td>True</td><td>False</td><td>False</td><td>True</td><td>True</td><td>False</td><td>True</td><td>True</td><td>20</td><td>9</td><td>1718492</td><td>3.444</td></tr><tr><td>True</td><td>False</td><td>True</td><td>False</td><td>True</td><td>False</td><td>False</td><td>True</td><td>40</td><td>9</td><td>1962957</td><td>4.744</td></tr><tr><td>True</td><td>False</td><td>False</td><td>True</td><td>False</td><td>True</td><td>True</td><td>False</td><td>40</td><td>1</td><td>3657562</td><td>2.427</td></tr><tr><td>True</td><td>False</td><td>True</td><td>False</td><td>True</td><td>True</td><td>True</td><td>False</td><td>40</td><td>9</td><td>3436410</td><td>3.447</td></tr><tr><td>False</td><td>False</td><td>False</td><td>True</td><td>False</td><td>False</td><td>True</td><td>False</td><td>60</td><td>5</td><td>2066645</td><td>2.957</td></tr></tbody></table>

## Regression problem (linear regression, regression tree, random forest, gradient boosting, SVM, etc.)

Guo et al. ASE 2013, Apel et al. ASE’15, Czarnecki et al. SPLC’15, Siegmund et al. FSE’15, Kastner et al. ASE’17, Menzies et al. FSE’17, Batory et al. FSE’17

Learning
# Options Matter: Documenting and Fixing Non-Reproducible Builds in Highly-Configurable Systems Randrianaina, Khelladi, Zendra, Acher MSR'2024

also at FOSDEM 2024 https://fosdem.org/2024/schedule/event/fosdem-2024-2848-documenting-and-fixing-non-reproducible-builds-due-to-configuration-options/

<figure><img src="image_119.png" /></figure>

## Measuring

## Modelling+Sampling

<figure><img src="image_120.png" /><figcaption>Table 2: Identified options and their category.</figcaption></figure>

<figure><img src="image_121.png" /></figure>

## Learning
# Neuroimaging pipelines: Variability everywhere!

<figure><img src="image_122.png" /></figure>

Input data

<figure><img src="image_123.png" /></figure>

<figure><img src="image_124.png" /></figure>

Results

... and many more...

<figure><img src="image_125.png" /></figure>

≠ algorithm

<figure><img src="image_126.png" /></figure>

≠ software

<figure><img src="image_127.png" /></figure>

≠ software version

<figure><img src="image_128.png" /></figure>

≠ parameters

<figure><img src="image_129.png" /></figure>

≠ environment

Camille Maumet keynote @ VaMoS
2025

We have already noticed the impact of **analytical variability** studied across:

* Different operating systems [Glatard 2015]

* Different versions [Gronenschild 2012]

* Many-analysis by different teams (NARPS, Nature 2020) and there are other examples: [Carp, 2012] [Botvinik-Nezer, 2020] [Nørgaard 2020] [Trübutschek 2024] [Bowring 2019]
# Neuroimaging pipelines: Variability everywhere

<figure><img src="image_130.png" /></figure>

Input data

<figure><img src="image_131.png" /></figure>

<figure><img src="image_132.png" /></figure>

Results

... and many more...

<figure><img src="image_133.png" /></figure>

≠ algorithm

<figure><img src="image_134.png" /></figure>

≠ software

<figure><img src="image_135.png" /></figure>

≠ software version

<figure><img src="image_136.png" /></figure>

≠ parameters

<figure><img src="image_137.png" /></figure>

≠ environment

Modelling, Sampling, Measuring, Learning
<figure><img src="image_138.png" /></figure>

<figure><img src="image_139.png" /></figure>

<figure><img src="image_140.png" /></figure>

<figure><img src="image_141.png" /></figure>

<figure><img src="image_142.png" /></figure>

<figure><img src="image_143.png" /></figure>

<figure><img src="image_144.png" /></figure>

<figure><img src="image_145.png" /></figure>

## Approach

### Modeling

<figure><img src="image_146.png" /></figure>

Expert

<figure><img src="image_147.png" /></figure>

Manual

<figure><img src="image_148.png" /></figure>

Handbook

<figure><img src="image_149.png" /></figure>

UVL

Feature

Model

(25 000+ config.)

Uniform

random

sampling

<figure><img src="image_150.png" /></figure>

### Sampling

<table><thead><tr><th>slice_timingCorrection/<br>ref_slice/middle</th><th>signal_modeling/hrf/<br>temporal_dispersion_derivs</th><th>...</th><th>spatial_smothing/<br>fwhm</th><th>signal_modeling/hrf/<br>canonical</th></tr></thead><tbody><tr><td>False</td><td>False</td><td>...</td><td>0</td><td>True</td></tr><tr><td>False</td><td>False</td><td>...</td><td>12</td><td>False</td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr><tr><td>True</td><td>False</td><td>...</td><td>2</td><td>False</td></tr><tr><td>True</td><td>False</td><td>...</td><td>6</td><td>False</td></tr></tbody></table>

Sample (1000 valid configurations)

### Running

```python
def get_smothing(self, features: list):
    node = Node(interface=Smooth(), name="spatial_smothing")
    node inputs, fwhm = 0
    if 'name' in fwhm in features:
        node inputs, fwhm = float(self.getFeatureEnd(f'
            name')/fwhm', features))
    return node
```

Nipype framework
Auditory task dataset

Pipelines execution

## Software Product Line approach

* Model our configuration space

* Sample from it

* Build pipelines

* Run them on a task-fMRI dataset
# Modelling, Sampling, Measuring, Learning...

**Issue #1:** The measure here is **not** a label or a scalar quantity (like runtime, number of defects, etc.) but statistic maps (complex, high-dimensional)

It is not the “usual” classification or regression problem

<figure><img src="image_151.png" /></figure>

<table><thead><tr><th>slice_timingCorrection/<br>ref_slice/middle</th><th>signal_modeling/hrf/<br>temporal_dispersion_derivs</th><th>...</th><th>spatial_smoothing/<br>fwhm</th><th>signal_modeling/hrf/<br>canonical</th></tr></thead><tbody><tr><td>False</td><td>False</td><td>...</td><td>0</td><td>True</td></tr><tr><td>False</td><td>False</td><td>...</td><td>12</td><td>False</td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr><tr><td>True</td><td>False</td><td>...</td><td>2</td><td>False</td></tr><tr><td>True</td><td>False</td><td>...</td><td>6</td><td>False</td></tr></tbody></table>

Measurement/
metric

<figure><img src="image_152.png" /></figure>

Sample (1000 valid configurations)
# Modelling, Sampling, Measuring, Learning...

**Issue #2:** Outputs cannot be evaluated against a directly observable ground truth (we have no direct measure of whole brain activity)

Which pipeline variant is right? Which variants compute the truth?

<figure><img src="image_153.png" /></figure>

<figure><img src="image_154.png" /></figure>

<figure><img src="image_155.png" /></figure>
# Modelling, Sampling, Measuring Scoring, Learning

We boil down to a classification problem (class = cluster of “closed” brain maps); we compare maps to a reference (proxy ground-truth)

## Approach

* Reduce high dimensional statistic maps to a **single score: Spearman correlation**

* Divide configurations into 4 clusters

* Try to **predict** correlations, cluster

* Learn from it

<figure><img src="image_156.png" /></figure>
Modelling, Sampling, Measuring Scoring, Learning

We boil down to a classification problem (class = cluster of “closed” brain maps); we compare maps to a reference (proxy ground-truth)

<figure><img src="image_157.png" /><figcaption>Regression decision tree learning curve</figcaption></figure>

* Predict correlations (mean, expert, random) for any given configuration

* Evaluate relevance of proxy ground truths

* Infer feature importances
We boil down to a classification problem (class = cluster of “closed” brain maps); we compare maps to a reference (proxy ground-truth)

## Main results

* Predict cluster for any given configuration

* Infer feature importances

* Coregistration (misalignment)

* Smoothing

<figure><img src="image_158.png" /><figcaption>Clustering decision tree feature importances</figcaption></figure>
# Modelling, Sampling, Measuring Scoring, Learning

Our approach to model the neuroimaging pipeline space was
successful.

This was done using an UVL Feature Model of a (single-software) neuroimaging
configuration space at manageable computational cost

Our "sampling, variants scoring, and learning" method
accurately generalized to whole configuration space.

We explained clusters of configuration space with a few features.

Steps towards establishing possible, relevant “truths” thanks to variants
pipelines
# Learning variability can be easy...

<figure><img src="image_159.png" /></figure>

<figure><img src="image_160.png" /></figure>

<figure><img src="image_161.png" /></figure>

## ... but look **deeply**.
## Otherwise...

Prediction models and configuration knowledge may not **generalize** (ie misleading/inaccurate/pointless for developers and users) if the software evolves, is compiled differently, a different hardware is used, input data fed differs, etc.

software application
variability

input data variability
build/compiler
variability

operating system
variability

hardware variability⁴²

<figure><img src="image_162.png" /></figure>
# Predicting performance of a configuration without considering other variability layers? Beware of deep variability!

**Threats** to **variability knowledge**: predicting, tuning, or understanding configurable systems without being aware of inputs can be inaccurate and... pointless

eg effectiveness of sampling strategies (random, 2-wise, etc.) is input specific (see also Pereira et al. ICPE 2020)

**Opportunities:** for some performance properties (P) and subject systems, some stability is observed and performance remains consistent!

Stefan Mühlbauer, Florian Sattler, Christian Kaltenecker, Johannes Dorn, Sven Apel, Norbert Siegmund "Analyzing the Impact of Workloads on Modeling the Performance of Configurable Software Systems" ICSE 2023

**Insight:** Workload sensitivity challenges the robustness and generalizability of single-workload performance models, yet it is neglected in state-of-the-art approaches. Worse, robust techniques using only rankings or relative importance of options are inapplicable for certain workload variations.

<table><tr><td>gcc</td><td>100</td></tr><tr><td>ImageMagick</td><td>51</td></tr><tr><td>lingeling</td><td>39</td></tr><tr><td>nodeJS</td><td>180</td></tr><tr><td>poppler</td><td>50</td></tr><tr><td>SQLite</td><td>97</td></tr><tr><td>x264</td><td>8</td></tr></table>

<table><thead><tr><th>System</th><th>#M</th><th>Performance(s) P</th><th>Docker</th><th>Dataset</th></tr></thead><tbody><tr><td>gcc</td><td>2400</td><td>size, ctime, exec</td><td>Link</td><td>Link</td></tr><tr><td>ImageMagick</td><td>100 000</td><td>size, time</td><td>Link</td><td>Link</td></tr><tr><td>lingeling</td><td>35 100</td><td>#confL., #reduc.</td><td>Link</td><td>Link</td></tr><tr><td>nodeJS</td><td>96 950</td><td>#operations/s</td><td>Link</td><td>Link</td></tr><tr><td>poppler</td><td>23 680</td><td>size, time</td><td>Link</td><td>Link</td></tr><tr><td>SQLite</td><td>7500</td><td>15 query times q1-q15</td><td>Link</td><td>Link</td></tr><tr><td>x264</td><td>280 797</td><td>cpu, fps, kbs, size, time</td><td>Link</td><td>Link</td></tr><tr><td>xz</td><td>1440</td><td>size, time</td><td>Link</td><td>Link</td></tr></tbody></table>

L. Lesoil, M. Acher, A. Blouin and J.-M. Jézéquel "Input sensitivity on the performance of configurable systems an empirical study" JSS 2023

<figure><img src="image_163.png" /></figure>
* Linux as a subject software system (not as an OS interacting with other layers)

* Targeted non-functional, quantitative property: **binary size**

* interest for maintainers/users of the Linux kernel (embedded systems, cloud, etc.)

* challenging to predict (cross-cutting options, interplay with compilers/build systems, etc./)

* Dataset: version 4.13.3 (september 2017), x86_64 arch, measurements of 95K+ random configurations

* paranoiac about deep variability since 2017, Docker to control the build environment and scale

* diversity of binary sizes: from 7Mb to 1.9Gb

* 6% MAPE errors: quite good, though costly...

<figure><img src="image_164.png" /></figure>

<figure><img src="image_165.png" /></figure>

<figure><img src="image_166.png" /></figure>

<figure><img src="image_167.png" /></figure>
# TUXML: Sampling, Measuring, Learning

Heraldo Borges, Juliana Alves Pereira, Djamel Eddine Khelladi, Mathieu Acher. Linux Kernel Configurations at Scale: A Dataset for Performance and Evolution Analysis. EASE 2025

<table><thead><tr><th>cid</th><th>compilation_date</th><th>compilation_time</th><th>config_file</th><th>stdout_log_file</th><th>stderr_log_file</th><th>user_output_file</th><th>compiled_kernel_size</th><th>compressed_compiled_kernel_size</th></tr></thead><tbody><tr><td>74882</td><td>2019-08-12 17:09:42</td><td>399.856</td><td>[BLOB - 24,3 Kio]</td><td>[BLOB - 33,7 Kio]</td><td>[BLOB - 14 o]</td><td>[BLOB - 702 o]</td><td>74559280</td><td>GZIP-bzImage: 8855504, GZIP-vmlinux: 10943304, ...</td></tr><tr><td>74881</td><td>2019-08-12 16:58:09</td><td>460.392</td><td>[BLOB - 25,8 Kio]</td><td>[BLOB - 34,7 Kio]</td><td>[BLOB - 14 o]</td><td>[BLOB - 704 o]</td><td>81377768</td><td>GZIP-bzImage: 18375632, GZIP-vmlinux: 20462408 ...</td></tr><tr><td>74880</td><td>2019-08-12 16:47:28</td><td>301.775</td><td>[BLOB - 22 Kio]</td><td>[BLOB - 24,2 Kio]</td><td>[BLOB - 14 o]</td><td>[BLOB - 705 o]</td><td>83004496</td><td>GZIP-bzImage: 14365648, GZIP-vmlinux: 16452424 ...</td></tr><tr><td>74879</td><td>2019-08-12 16:46:14</td><td>1393.61</td><td>[BLOB - 24,1 Kio]</td><td>[BLOB - 50 Kio]</td><td>[BLOB - 571 o]</td><td>[BLOB - 712 o]</td><td>109098328</td><td>GZIP-bzImage: 17183792, GZIP-vmlinux: 19272160 ...</td></tr><tr><td>74878</td><td>2019-08-12 16:46:03</td><td>305.705</td><td>[BLOB - 26,1 Kio]</td><td>[BLOB - 28,8 Kio]</td><td>[BLOB - 14 o]</td><td>[BLOB - 703 o]</td><td>55523752</td><td>GZIP-bzImage: 14767568, GZIP-vmlinux: 16852088 ...</td></tr></tbody></table>
<figure><img src="image_168.png" /></figure>

<table><caption>Table I: Dataset properties for each version. The number of deleted/new features, delta commits, files changes are w.r.t. 4.13.</caption><thead><tr><th>Version</th><th>Release Date</th><th>LOC</th><th>Files</th><th>Examples</th><th>Seconds/config</th><th>Options</th><th>Features</th><th>Deleted features</th><th>New features</th><th>ΔCommits</th><th>Files changes</th></tr></thead><tbody><tr><td>4.13</td><td>2017/09/03</td><td>16,616,534</td><td>60,530</td><td>92,562</td><td>not available</td><td>12,776</td><td>9,468</td><td>-</td><td>-</td><td>-</td><td>934,628</td></tr><tr><td>4.15</td><td>2018/01/28</td><td>17,073,368</td><td>62,249</td><td>39,391</td><td>not available</td><td>12,998</td><td>9,425</td><td>342</td><td>299</td><td>31,052</td><td>1,972,020</td></tr><tr><td>4.20</td><td>2018/12/23</td><td>17,526,171</td><td>62,423</td><td>23,489</td><td>225</td><td>13,533</td><td>10,189</td><td>468</td><td>1,189</td><td>104,691</td><td>2,170,935</td></tr><tr><td>5.0</td><td>2019/03/03</td><td>17,679,372</td><td>63,076</td><td>19,952</td><td>247</td><td>13,673</td><td>10,293</td><td>494</td><td>1,319</td><td>118,778</td><td>3,827,025</td></tr><tr><td>5.4</td><td>2019/10/24</td><td>19,358,903</td><td>67,915</td><td>25,847</td><td>285</td><td>14,159</td><td>10,813</td><td>663</td><td>2,008</td><td>181,308</td><td>4,393,117</td></tr><tr><td>5.7</td><td>2020/05/31</td><td>19,358,903</td><td>67,915</td><td>20,159</td><td>258</td><td>14,586</td><td>11,338</td><td>715</td><td>2,585</td><td>225,804</td><td>4,681,313</td></tr><tr><td>5.8</td><td>2020/08/02</td><td>19,729,197</td><td>69,303</td><td>21,923</td><td>289</td><td>14,817</td><td>11,530</td><td>730</td><td>2,792</td><td>242,381</td><td>4,681,313</td></tr></tbody></table>

4.13 version (sep 2017): 6%. What about **evolution?** Can we reuse the 4.13 Linux prediction model? No, accuracy quickly decreases: **4.15 (5 months after): 20%; 5.7 (3 years after): 35%**

<figure><img src="image_169.png" /></figure>

<figure><img src="image_170.png" /></figure>
# Transfer learning (reuse of knowledge)

* Mission Impossible: Saving variability knowledge and prediction model 4.13 (15K hours of computation)

* Heterogeneous transfer learning: the feature space is different

* TEAMS: transfer evolution-aware model shifting

<figure><img src="image_171.png" /></figure>

<figure><img src="image_172.png" /></figure>

<figure><img src="image_173.png" /></figure>
# Transfer learning (con't)

Luc Lesoil, Helge Spieker, Arnaud Gotlieb, Mathieu Acher, Paul Temple, Arnaud Blouin, Jean-Marc Jézéquel:

Learning input-aware performance models of configurable systems: An empirical evaluation. J. Syst. Softw. 208: 111883 (2024)

<figure><img src="image_174.png" /><figcaption>Figure 1: The performance prediction problem: how to predict software performance considering both configurations and inputs?</figcaption></figure>

<table><thead><tr><td>Approach</td><td>Description of the approach</td><td>Offline cost (Offshore Organization)</td><td>Online measurement cost (User)</td><td>Input properties</td></tr></thead><tbody><tr><td>Supervised online learning</td><td>Train performance model on demand, from scratch, each time a new input is fed to the configurable system</td><td>None</td><td>High</td><td>No</td></tr><tr><td>Offline learning</td><td>Use a pre-trained model over measurements of multiple configurations and inputs. Input properties are used to make the prediction (in an online setting).</td><td>High</td><td>None</td><td>Yes</td></tr><tr><td>Transfer learning</td><td>Adapt a pre-trained model for a new targeted input. It requires to gather fresh measurements of some configurations over the input (in an online setting).</td><td>Medium</td><td>Medium</td><td>Yes</td></tr></tbody></table>
# Is there an interplay between compile-time and runtime options?

<figure><img src="image_175.png" /></figure>

L. Lesoil, M. Acher, X. Tèrnava, A. Blouin and J.-M. Jézéquel "The Interplay of Compile-time and Run-time Options for Performance Prediction" in SPLC '21

<figure><img src="image_176.png" /></figure>

This paper investigates how compile-time options can affect software performances and how compile-time options interact with run-time options.

<figure><img src="image_177.png" /><figcaption>**Figure 1: Cross-layer variability of x264**</figcaption></figure>
# Leverage stability across variability layers!

**L. Lesoil**, M. Acher, X. Tèrnava, A. Blouin and J.-M. Jézéquel "The Interplay of Compile-time and Run-time Options for Performance Prediction" in SPLC '21

<figure><img src="image_178.png" /><figcaption>(c) x264, I#4 (lyric video), encoding time</figcaption></figure>

**First good news: Worth tuning software at compile-time!**

**Second good news:** For all the execution time distributions of x264 and all the input videos, the worst correlation is greater than 0.97. If the compile-time options change the scale of the distribution, they do not change the rankings of run-time configurations (i.e., they do not truly interact with the run-time options).

**It has three practical implications:**

1. **Reuse of configuration knowledge:** transfer learning of prediction models boils down to apply a linear transformation among distributions. Users can also trust the documentation of run-time options, consistent whatever the compile-time configuration is.

2. **Tuning at lower cost:** finding the best compile-time configuration among all the possible ones allows one to immediately find the best configuration at run time. We can remove away

3. **Measuring at lower cost:** do not use a default compile-time configuration, use it and will generalize!

Did we recommend to use two binaries? YES, one for measuring, another for performances!

<figure><img src="image_179.png" /></figure>
Key results (Poppler)

First good new. Worth the

Second good new. Worth the

It has three practical implications:

<figure><img src="image_180.png" /></figure>

xz

performances!

L. Lesoil, M. Acher, X. Tèrnava, A. Blouin and J.-M. Jézéquel "The Interplay of Compile-time and Run-time Options for Performance Prediction" in SPLC '21

<figure><img src="image_181.png" /></figure>

(c) x264, I #4 (lyric video), encoding time

interplay between
compile-time and runtime
options and even input!

<figure><img src="image_182.png" /></figure>

<figure><img src="image_183.png" /></figure>

<figure><img src="image_184.png" /></figure>

<figure><img src="image_185.png" /></figure>
# AGENDA

Frictionless Reproducibility and (Deep) Software (Variability)

Modelling, Sampling, Measuring, Learning Variability Spaces

**Students leverage variability to reproduce and replicate studies**

Mastering variability is key for truly replicable science
# Design of a reproducibility course, two complementary parts

## From Floating-Point Experiments...

Students first explored the non-associativity of floating-point arithmetic as a reproducibility "Hello World" using Docker, GitHub Actions, and templated experimentation to analyze sources of **variability across programming languages, compiler flags, and numerical precision**.

> "Teaching Reproducibility and
> Embracing Variability: From
> Floating-Point Experiments to
> Replicating Research" Acher et al. REP
> 2025

## ...to Replicating Research

The second half of the course focused on **reproducing and replicating actual research papers**, including studies on large language models playing chess, home advantage in football during COVID19, and energy efficiency across programming languages.
How often (x+y)+z == x+(y+z) ?

<figure><img src="image_186.png" /><figcaption>Figure 2: Feature model (excerpt). Inverse (resp. Relation-</figcaption></figure>

onds to checking the non-arity (x * y) / (y * x) =

<table><thead><tr><th>Parameters, Input Data Programming Style</th><th>e.g., random seed selection</th><th>Language</th><th>Library</th><th>System</th><th>Compiler</th><th>VariabilityMac</th><th>EqualityCheck</th><th>NumberGenerations</th><th>Repeat</th><th>min</th><th>max</th><th>std</th><th>mean</th></tr></thead><tbody><tr><td rowspan="2">Language</td><td rowspan="2">python®</td><td rowspan="2">Java</td><td rowspan="2">C++</td><td rowspan="2">F</td><td rowspan="2"></td><td>seed none</td><td>ASSOCATMTY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>seed none</td><td>MULT_JW</td><td>100</td><td>10</td><td>60.0</td><td>71.0</td><td>3.562026281111375</td><td>65.1</td></tr><tr><td rowspan="2">Compiler & VM</td><td rowspan="2">GCC</td><td rowspan="2">LLVM</td><td rowspan="2">JVM</td><td rowspan="2"></td><td rowspan="2"></td><td>seed none</td><td>ASSOCATMTY</td><td>100</td><td>10</td><td>60.0</td><td>63.0</td><td>3.307161581699343</td><td>55.9</td></tr><tr><td>seed none</td><td>MULT_JW</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td rowspan="2">Library</td><td rowspan="2">NumPy</td><td rowspan="2">blas</td><td rowspan="2">jblas</td><td rowspan="2"></td><td rowspan="2"></td><td>seed none</td><td>associativity</td><td>100</td><td>10</td><td>71.0</td><td>82.0</td><td>3.3466401901363023</td><td>76.9</td></tr><tr><td>seed none</td><td>mult-inverse</td><td>100</td><td>10</td><td>58.0</td><td>78.0</td><td>6.0</td><td>68.9</td></tr><tr><td rowspan="2">Platform</td><td rowspan="2"></td><td rowspan="2">Windows</td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td>seed none</td><td>mult-inverse ge</td><td>100</td><td>10</td><td>62.0</td><td>64.0</td><td>5.80557537624865</td><td>53.4</td></tr><tr><td>seed none</td><td>associativity</td><td>100</td><td>10</td><td>81.0</td><td>81.0</td><td>0.0</td><td>81.0</td></tr><tr><td rowspan="2">Processor</td><td rowspan="2"></td><td rowspan="2">Intel®</td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td>seed none</td><td>mult-inverse ge</td><td>100</td><td>10</td><td>70.0</td><td>75.0</td><td>0.0</td><td>70.0</td></tr><tr><td>mult-inverse ge</td><td>ASSOCATMTY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td rowspan="2">Micro-architecture</td><td rowspan="2">Inner state of</td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td>seed none</td><td>seed none equality</td><td>100</td><td>10</td><td>72.0</td><td>72.0</td><td>2.74043494</td><td>100.0</td></tr><tr><td>seed none equality</td><td>MULT_JW</td><td>100</td><td>10</td><td>47.0</td><td>57.0</td><td>2.80081434</td><td>100.0</td></tr><tr><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td>seed none</td><td>ASSOCATMTY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>seed none equality</td><td>MULT_JW</td><td>100</td><td>10</td><td>60.0</td><td>79.0</td><td>6.147971513</td><td>100.0</td></tr><tr><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td>seed none</td><td>seed none equality</td><td>100</td><td>10</td><td>49.0</td><td>59.0</td><td>2.8301943</td><td>100.0</td></tr><tr><td>seed none equality</td><td>ASSOCATMTY</td><td>100</td><td>10</td><td>89.0</td><td>89.0</td><td>0.0</td><td>100.0</td></tr><tr><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td>seed none</td><td>seed none equality</td><td>100</td><td>10</td><td>73.0</td><td>73.0</td><td>0.0</td><td>100.0</td></tr><tr><td>seed none equality</td><td>MULT_JW</td><td>100</td><td>10</td><td>55.0</td><td>55.0</td><td>0.0</td><td>55.0</td></tr><tr><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td rowspan="2"></td><td>seed none</td><td>seed none equality of jula rang</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>seed none</td><td>MULT_JW</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr></tbody></table>

<figure><img src="image_187.png" /></figure>
<figure><img src="image_188.png" /></figure>

"Teaching Reproducibility and
Embracing Variability: From
Floating-Point Experiments to
Replicating Research" Acher et al.
REP 2025

**Each group shared their solutions/results through a Git (with Dockerfile, README.md, Github actions, Notebooks, etc.)**
Choose 1 out of 3

* "Debunking the Chessboard: Confronting GPTs Against Chess Engines to Estimate Elo Ratings and Assess Legal Move Abilities"

* https://blog.mathieuacher.com/GPTsChessEloRatingLegalMoves/

<figure><img src="image_189.png" /></figure>

* "COVID and Home Advantage in Football: An Analysis of Result and xG Data in European Leagues"

* https://blog.mathieuacher.com/FootballAnalysis-xG-COVIDHome/

<figure><img src="image_190.png" /></figure>

<figure><img src="image_191.png" /></figure>

<figure><img src="image_192.png" /></figure>

* "Ranking Programming Languages by Energy Efficiency" SCP 2021

* https://github.com/greensoftwarelab/RosettaExamples

=> XXX (link)

<figure><img src="image_193.png" /></figure>
# Debunking the Chessboard: Confronting GPTs Against Chess Engines to Estimate Elo Ratings and Assess Legal Move Abilities

https://blog.mathieuacher.com/GPTsChessEloRatingLegalMoves/

https://www.youtube.com/watch?v=6D1Xlbkm4JE (video with MonsieurPhi)

* what about GPT4o or Claude or Llama?

* prompt sensitivity?

* temperature?

* what about confronting to other chess engines?

* ...

<figure><img src="image_194.png" /></figure>

<figure><img src="image_195.png" /></figure>
<figure><img src="image_196.png" /></figure>

MrPhi

@MonsieurPhi

Petite note: j'ai remarqué que jusqu'à 1800 Elo, sa sensibilité au prompt est un peu bizarre. Par exemple avec "1-0" (Blanc gagne) il joue le coup correct 6. Mais avec "0-1" (Noir gagne) ou "1/2-1/2", il joue Nf6 (ce qui est illogique vu que Nf6 fait justement perdre les Noirs).

Translate post

<figure><img src="image_197.png" /></figure>

11:55 AM - Apr 19, 2024 447 Views

<figure><img src="image_198.png" /></figure>

<figure><img src="image_199.png" /></figure>

<figure><img src="image_200.png" /></figure>

<figure><img src="image_201.png" /></figure>

<figure><img src="image_202.png" /></figure>

<figure><img src="image_203.png" /></figure>

Post your reply

Reply

<figure><img src="image_204.png" /></figure>

MrPhi @MonsieurPhi - Apr 19

Mais il n'a plus ce comportement bizarre quand le Elo dépasse 2000. Enfin bref ! Il faudrait faire d'autres tests pour déterminer à quel point cette information sur le niveau Elo influence son niveau de jeu, mais c'est déjà intéressant comme petite expérience.

<figure><img src="image_205.png" /></figure>

<figure><img src="image_206.png" /></figure>

<figure><img src="image_207.png" /></figure>

<figure><img src="image_208.png" /></figure>

<figure><img src="image_209.png" /></figure>

```python
gpt_config = GPTConfig(
    model_gpt="gpt-3.5-turbo-instruct",
    temperature=0.0,
    max_tokens=5,
    chat_gpt=False,
    system_role_message=None  # Since it wasn't provided in the original call
)
```

```python
# Optional GM titles
black title = "[BlackTitle \"GM\\"]" if include_black_title else ""
white title = "[WhiteTitle \"GM\\"]" if include white title else ""
```

```python
# Construct the PGN header with the configurable options
pgn_headers = f"""[Event "FIDE World Championship Match 2024"]
[Site "Los Angeles, USA"]
[Date "2024.12.01"]
[Round "5"]
[White "{white_name}"]
[Black "{black_name}"]
[Result "{result}"]
[WhiteElo "{white elo}"]
{white title}
[BlackElo "{black elo}"]
{black title}
[TimeControl "40/7200:20/3600:900+30"]
[UTCDate "2024.11.27"]
[UTCTime "09:01:25"]
[Variant "Standard"]
""
```

Modelling+Sampling

<figure><img src="image_210.png" /></figure>

<figure><img src="image_211.png" /></figure>
<figure><img src="image_212.png" /></figure>

MrPhi

@MonsieurPhi

...

Petite note : j'ai remarqué que jusqu'à 1800 Elo, sa sensibilité au prompt est un peu bizarre. Par exemple avec "1-0" (Blanc gagne) il joue le coup correct g6. Mais avec "0-1" (Noir gagne) ou "1/2-1/2", il joue Nf6 (ce qui est illogique vu que Nf6 fait justement perdre les Noirs).

Translate post

<figure><img src="image_213.png" /></figure>

11:55 AM · Apr 19, 2024 · 447 Views

<figure><img src="image_214.png" /></figure>

1

<figure><img src="image_215.png" /></figure>

<figure><img src="image_216.png" /></figure>

<figure><img src="image_217.png" /></figure>

<figure><img src="image_218.png" /></figure>

<figure><img src="image_219.png" /></figure>

Post your reply

Reply

<figure><img src="image_220.png" /></figure>

MrPhi @MonsieurPhi · Apr 19

Mais il n'a plus ce comportement bizarre quand le Elo dépasse 2000. Enfin bref ! Il faudrait faire d'autres tests pour déterminer à quel point cette information sur le niveau Elo influence son niveau de jeu, mais c'est déjà intéressant comme petite expérience.

<figure><img src="image_221.png" /></figure>

<figure><img src="image_222.png" /></figure>

<figure><img src="image_223.png" /></figure>

<figure><img src="image_224.png" /></figure>

441

<figure><img src="image_225.png" /></figure>

<figure><img src="image_226.png" /></figure>

<table><thead><tr><th>Result</th><th>WhiteName</th><th>BlackName</th><th>WhiteElo</th><th>BlackElo</th><th>IncludeWhiteTitle</th><th>IncludeBlackTitle</th><th>Move</th></tr></thead><tbody><tr><td>0-1</td><td>XXX</td><td>Louapre, David</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>Nf6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>XXX</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Louapre, David</td><td>XXX</td><td>1400</td><td>1400</td><td>False</td><td>False</td><td>Nf6</td></tr><tr><td>1-0</td><td>XXX</td><td>Nepomniachtchi, Ian</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Nepomniachtchi, Ian</td><td>Louapre, David</td><td>2900</td><td>2900</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>Nepomniachtchi, Ian</td><td>1800</td><td>1800</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Louapre, David</td><td>Nepomniachtchi, Ian</td><td>1000</td><td>1000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Nepomniachtchi, Ian</td><td>XXX</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>XXX</td><td>Nepomniachtchi, Ian</td><td>2000</td><td>2000</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>0-1</td><td>Nepomniachtchi, Ian</td><td>XXX</td><td>1700</td><td>1700</td><td>False</td><td>False</td><td>g6</td></tr><tr><td>1-0</td><td>Louapre, David</td><td>XXX</td><td>1700</td><td>1700</td><td>False</td><td>False</td><td>Nf6</td></tr></tbody></table>

<figure><img src="image_227.png" /></figure>

<figure><img src="image_228.png" /></figure>

<figure><img src="image_229.png" /></figure>
# Home advantage in football (soccer) before and during COVID https://blog.mathieuacher.com/FootballAnalysis-xG-COVIDHome/

<figure><img src="image_230.png" /><figcaption>Ligue_1 xPoints change significance with mann</figcaption></figure>

<table><thead><tr><th></th><th>2014</th><th>2015</th><th>2016</th><th>2017</th><th>2018</th><th>2019</th><th>2020</th></tr></thead><tbody><tr><td><strong>2014</strong></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>2015</strong></td><td>0.251745</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>2016</strong></td><td>0.207138</td><td>0.448466</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td><strong>2017</strong></td><td>0.321279</td><td>0.486951</td><td>0.416316</td><td></td><td></td><td></td><td></td></tr><tr><td><strong>2018</strong></td><td>0.460318</td><td>0.307324</td><td>0.263694</td><td>0.357313</td><td></td><td></td><td></td></tr><tr><td><strong>2019</strong></td><td>0.475235</td><td>0.281316</td><td>0.229452</td><td>0.329968</td><td>0.452614</td><td></td><td></td></tr><tr><td><strong>2020</strong></td><td><strong>0.012854</strong></td><td><strong>0.003026</strong></td><td><strong>0.001609</strong></td><td><strong>0.005677</strong></td><td><strong>0.010800</strong></td><td><strong>0.022462</strong></td><td></td></tr></tbody></table>

<figure><img src="image_231.png" /></figure>

<figure><img src="image_232.png" /></figure>

<figure><img src="image_233.png" /></figure>

xPoints vs xG vs result; inter-season comparison; single-season comparison; variability in COVID management across leagues

**ard; The results show a significant decrease in home advantage during the COVID period, particularly in Ligue 1 and the Premier League.**
# My students reproduced results and found... differences!

## Key finding #1: Bug or feature? scipy 1.6.0 changes p-values on MannU

<figure><img src="image_234.png" /></figure>

* The importance of reproducing results (with other teams/people!)

* The difficulty of reproducing results: the **version** of a library can change everything!

* Be careful about non-choices of **configurations** (here on the alternative hypothesis: two-sided vs one-sided)

Download SciPy 1.6.0 Reference Guide Scientific Functions (scipy stats)

This is documentation for an old release of SciPy (version 1.6.0). Read this page in the documentation of the latest stable release (version 1.14.1).

scipy.stats.mannwhitneyu

scipy.stats.mannwhitneyu(x, y, use=' continuity True', alternative='None')

[source]

Compute the Mann-Whitney rank test on samples x and y.

Parameters: x, y: array-like

Use of arrays should be one-dimensional.

use continuity: bool optional

Whether a continuity correction (IV/2) should be taken into account. Default is True.

alternative: (None, two-sided, 'less', 'greater') optional

Defines the alternative hypothesis. The following options are available (default is 'none'):

* None: computes p-value half the size of the 'two-sided' p-value and a different U statistic. The default behavior is not the same as using 'less' or 'greater', it only exists for tests under backward compatibility and is deprecated.

* 'two-sided'

* 'less': one-sided

* 'greater': one-sided

Use of the None option is deprecated.

Returns:

statistic: float

The Mann-Whitney U statistic, equal to min(x, y) if alternatives equal to None (deprecated); exists for backward compatibility), and U for y otherwise.

p-value: float

p-value assuming an asymptotic normal distribution. One-sided or two-sided, depending on the choice of alternative.

Notes:

Use only when the number of observations in each sample is ≥ 20 and you have 2 independent samples of ranks. Mann-Whitney U is significant if the U obtained is LESS THAN or equal to the critical value of U.

This test corrects for ties and by default uses a continuity correction.

References

1. https://en.wikipedia.org/wiki/Mann-Whitney_U_test

2. H.B. Mann and D.R. Whitney, "On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other," The Annals of Mathematical Statistics, vol. 18, no. 1, pp. 50-60, 1947.

This is documentation for an old release of SciPy version 1.7.0. Read this page in the documentation of the latest stable release (version 1.14.1).

scipy.stats.mannwhitneyu

scipy.stats.mannwhitneyu(x, y, continuity=True, alternative='two-sided', est='t')

[source]

Fit the Mann-Whitney U rank test on two independent samples.

The Mann-Whitney U is a nonparametric test of the null hypothesis that the distribution underlying sample x is the same as the distribution underlying sample y. It is often used as a test of difference in location between distributions.

Parameters: x, y: array-like

Not all arrays of samples. The arrays must be broadcastable except along the dimension given by `x`.

use continuity: bool optional

Whether a continuity correction (IV/2) should be applied. Default is True when `use continuity` is `True`.

alternative: (two-sided, 'less', 'greater') optional

Defines the alternative hypothesis. Default is 'two-sided'. Let `df` and `df` for the cumulative distribution functions of the distribution underlying `x` and `y` respectively. Then the following alternative hypotheses are available:

* 'less': the distributions are not equal. (i.e. `df` for `y` is less than `df` for `x`.)

* 'test': the distribution underlying `x` is stochastically less than the distribution underlying `y. (i.e. `df` for `x` is less than `df` for `y`.)

Under a more restrictive set of assumptions, the alternative hypotheses can be expressed in terms of the locations of the distributions; see [S] section 5.1.

axis: int, optional

Are along which to perform the test. Default is `0`.

method: (note, 'asymmetric', 'two-sided') optional

Selects the method used to calculate the p-value. Default is `ztest`, the following options are available:

* 'asymmetric': computes the standardized test statistic against the normal distribution, corresponding to `df`.

* 'ztest': computes the z-statistic by comparing the standardized U statistic against the expected distribution of the statistics under the null hypothesis. No correction is made for ties.

* 'chi2': (chooses 'ztest') when the size of one of the samples is less than 8.

Returns:

res: (list, 'asymmetric', 'two-sided')

An object containing attributes:

statistic: float

The Mann-Whitney U statistic, corresponding with sample x. See factors for the test statistic, corresponding with sample y.

註釋: 註釋

<table><caption>EPL xPoints change significance with mann</caption><thead><tr><th></th><th>2014</th><th>2015</th><th>2016</th><th>2017</th><th>2018</th><th>2019</th><th>2020</th></tr></thead><tbody><tr><td>2014</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>2015</td><td>0.061728</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>2016</td><td>0.207185</td><td>0.260411</td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>2017</td><td>0.408533</td><td>0.119547</td><td>0.291484</td><td></td><td></td><td></td><td></td></tr><tr><td>2018</td><td>0.104307</td><td>0.418444</td><td>0.392634</td><td>0.208557</td><td></td><td></td><td></td></tr><tr><td>2019</td><td>0.032631</td><td>0.373903</td><td>0.193936</td><td>0.077736</td><td>0.259500</td><td></td><td></td></tr><tr><td>2020</td><td>0.000772</td><td>0.047791</td><td>0.015721</td><td>0.003938</td><td>0.032873</td><td>0.102317</td><td></td></tr></tbody></table>

### (a) Original

<table><caption>English Premier League xPoints Change Significance with Mann-Whitney U Test</caption><thead><tr><th></th><th>2014</th><th>2015</th><th>2016</th><th>2017</th><th>2018</th><th>2019</th><th>2020</th></tr></thead><tbody><tr><td>2014</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td></tr><tr><td>2015</td><td>0.123055</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td></tr><tr><td>2016</td><td>0.414370</td><td>0.519108</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td></tr><tr><td>2017</td><td>0.817066</td><td>0.238897</td><td>0.582968</td><td>nan</td><td>nan</td><td>nan</td><td>nan</td></tr><tr><td>2018</td><td>0.208613</td><td>0.838586</td><td>0.785267</td><td>0.417115</td><td>nan</td><td>nan</td><td>nan</td></tr><tr><td>2019</td><td>0.065263</td><td>0.748807</td><td>0.387872</td><td>0.155471</td><td>0.519001</td><td>nan</td><td>nan</td></tr><tr><td>2020</td><td>0.001544</td><td>0.095714</td><td>0.031442</td><td>0.007875</td><td>0.065746</td><td>0.204634</td><td>nan</td></tr></tbody></table>

### (b) Reproduction
# My students _replicated_ and found new insights!

## Key finding #2

* Original study was conducted in 2021... opportunities to extend it to 2022 and 2023 and 2024

* and ask new questions: has the post-COVID period restored the home advantage?

* Answer/teaser: yes!

* New findings on **other national championships and European cups** (Champions League, the Turkish Süper Lig) on post-COVID seasons (2022–2024)

<figure><img src="image_235.png" /></figure>

INSA

INSTITUT NATIONAL
DES SCIENCES
APPLIQUÉES
RENNES

<figure><img src="image_236.png" /></figure>
# AGENDA

Frictionless Reproducibility and (Deep) Software (Variability)

Modelling, Sampling, Measuring, Learning Variability Spaces

Students leverage variability to reproduce and replicate studies

**Mastering variability is key for truly replicable science**
# Our Vision

## Embrace deep variability!

Explicit modeling of the variability points and their relationships, such as:

1. Get insights into the variability “factors” and their possible interactions

2. Capture and document configurations for the sake of **reproducibility**

3. Explore diverse configurations to **replicate**, and hence optimize, validate, increase the robustness, or provide better resilience

## Embracing Deep Variability For Reproducibility & Replicability

Mathieu Acher, Benoit Combemale, Georges Aaron Randrianaina, Jean-Marc Jézéquel

IRISA, Université de Rennes

Rennes, France

### ABSTRACT

Reproducibility (a.k.a., determinism in some cases) constitutes a fundamental aspect in various fields of computer science, such as floating-point computations in numerical analysis and simulation, concurrency models in parallelism, reproducible builds for third parties integration and packaging, and containerization for execution environments. These concepts, while pervasive across diverse concerns, often exhibit intricate inter-dependencies, making it challenging to achieve a comprehensive understanding. In this short and vision paper we delve into the application of software engineering techniques, specifically variability management, to systematically identify and explicit points of variability that may give rise to reproducibility issues (e.g., language, libraries, compiler, virtual machine, OS, environment variables, etc.). The primary objectives are: i) gaining insights into the variability layers and their possible interactions, ii) capturing and documenting configurations for the sake of reproducibility, and iii) exploring diverse configurations to replicate, and hence validate and ensure the robustness of results. By adopting these methodologies, we aim to address the complexities associated with reproducibility and replicability in modern software systems and environments, facilitating a more comprehensive and nuanced perspective on these critical aspects.

In this paper we propose to characterize both intended and unintended variability of any software-intensive system in order to support reproducibility and replicability, and eventually estimate its robustness, uncertainty profile, and explore different hypotheses.

### 2 DEEP SOFTWARE VARIABILITY

Uncertainty in informatics comes from many different origins [16, 36], either ontological (i.e., inherent unpredictability, e.g., aleatory) or epistemic (i.e., due to insufficient knowledge).

Ontological causes include noise in the input data of a program, its memory layout, network delays, the internal state of the processor, the ambient temperature and even the age of the processor¹.

Epistemic causes include misunderstanding of the user's needs, variable behavior of conceptually similar resolution methods, choice of threshold parameters, unexpected behavior of APIs, variable behavior among functionally similar libraries, or subtle differences in the semantics of programming languages (e.g., ~3% evaluates to ~1 in Java but to 1 in Python, or even inside the same programming language (for instance x/0 is an undefined behavior in C)).

<table><tr><td>Parameters, Input Data</td><td>e.g., random seed selection</td></tr><tr><td>Programming Style</td><td>e.g., x+(y+z) vs. (x+y+z)</td></tr></table>

ACM REP 2024

⇒ We aim to **address the complexities associated with reproducibility and replicability in modern software systems and environments**, facilitating a more comprehensive and nuanced perspective on these critical “factors”.
References

Barba, L. A. (2018). "Terminologies for Reproducible Research."

Donoho, D. (2024). "Data Science at the Singularity." *Harvard Data Science Review*.

Carlini, N. (2024). "Cutting through buggy adversarial example defenses: fixing 1 line of code breaks SABRE." (preprint).

Massonnet, F., et al. "Replicability of the EC-Earth3 Earth system model under a change in computing environment." *Geoscientific Model Development*, 2020

Gronenschild, E. H. B. M., et al. (2012). "The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements." *PLoS ONE*.

Krefting, D., et al. (2011). "Reliability of quantitative neuroimage analysis using FreeSurfer in distributed environments." *MICCAI Workshop on HPDC for Medical Imaging*.

Glatard, T., et al. (2015). "Reproducibility of neuroimaging analyses across operating systems." *Frontiers in Neuroinformatics*.
References

Pineau, J. (2019). "Building Reproducible, Reusable, and Robust Machine Learning Software." (ICSE keynote).

Henderson, P., et al. "Deep Reinforcement Learning that Matters.", 2017

Mesnard, O., & Barba, L. A. "Reproducible and replicable CFD: it's harder than you think.", 2016

Botvinik-Nezer, R., et al. (2020). "Variability in the analysis of a single neuroimaging dataset by many teams." Nature.

Lesoil, L., Acher, M., Blouin, A., & Jézéquel, J.-M. (2023). "The Interaction between Inputs and Configurations fed to Software Systems: an Empirical Study." Journal of Systems and Software.

Alves Pereira, J., Martin, H., Acher, M., Jézéquel, J.-M., Botterweck, G., & Ventresque, A. (2021). "Learning Software Configuration Spaces: A Systematic Literature Review." Journal of Systems and Software.
References

Randrianaina, L., Khelladi, D. E., Zendra, O., & Acher, M. (2024). "Options Matter: Documenting and Fixing Non-Reproducible Builds in Highly-Configurable Systems." MSR.

Merel Jourdan, Y., Acher, M., & Maumet, C. (2025). "In the Search for Truth: Navigating Variability in Neuroimaging Software Pipelines." SPLC 2025

Juliana Alves Pereira, Mathieu Acher, Hugo Martin, and Jean-Marc Jézéquel. Sampling Effect on Performance Prediction of Configurable Systems: A Case Study (2020). In International Conference on Performance Engineering (ICPE 2020)

Martin, H., Acher, M., Alves Pereira, J., Lesoil, L., Jézéquel, J.-M., & Khelladi, D. E. (2021). "Transfer learning across variants and versions: The case of Linux kernel size." IEEE TSE.

Borges, H., Alves Pereira, J., Khelladi, D. E., & Acher, M. (2025). "Linux Kernel Configurations at Scale: A Dataset for Performance and Evolution Analysis" EASE.
References

Lesoil, L., Spieker, H., Gotlieb, A., Acher, M., Temple, P., Blouin, A., & Jézéquel, J.-M. (2024).
"Learning input-aware performance models of configurable systems: An empirical evaluation."
*Journal of Systems and Software*, 208:111883.

Lesoil, L., Acher, M., Tèrnava, X., Blouin, A., & Jézéquel, J.-M. (2021). "The Interplay of Compile-time and Run-time Options for Performance Prediction." SPLC.

Acher, M., et al. (2025). "Teaching Reproducibility and Embracing Variability: From Floating-Point Experiments to Replicating Research." REP.

Acher, M. "Debunking the Chessboard: Confronting GPTs Against Chess Engines to Estimate Elo Ratings and Assess Legal Move Abilities."
https://blog.mathieuacher.com/GPTsChessEloRatingLegalMoves/

Acher, M. "COVID and Home Advantage in Football: An Analysis of Results and xG Data in European Leagues."
https://blog.mathieuacher.com/FootballAnalysis-xG-COVIDHome/
References

Mathieu Acher, Benoît Combemale, Georges Aaron Randrianaina, Jean-Marc Jézéquel "Embrace
Deep Variability for Reproducibility & Replicability." ACM REP 2024

Georges Aaron Randrianaina, Xhevahire Tërnava, Djamel Eddine Khelladi, Mathieu Acher "On the
Benefits and Limits of Incremental Build of Software Configurations: An Exploratory Study" ICSE 2022

Xhevahire Tërnava, Mathieu Acher, Benoît Combemale: Specialization of Run-time Configuration
Space at Compile-time: An Exploratory Study. SAC 2023: 1459-1468

Mathieu Acher, Hugo Martin, Juliana Alves Pereira, Luc Lesoil, Arnaud Blouin, Jean-Marc Jézéquel, Djamel Eddine Khelladi, and Olivier Barais. Feature Subset Selection for Learning Huge Configuration Spaces: The case of Linux Kernel Size (2022). In SPLC 2022

Valov, P., et al. (2017). “Transferring Performance Prediction Models Across Different Hardware Platforms.” ICPE.

Jamshidi, P., et al. (2017). “Transfer Learning for Software Performance Analysis: An Exploratory Analysis.” ASE.
► Frictionless reproducibility: code+data+metrics

► Deep variability is a problem (frictions!)

▶ evidence in many scientific domains

► Deep variability is a solution (exploration!)

▶ fixing variability once and for all is not

► Replicability is the holy grail!

▶ explore variants for robustness, validation, optimization and knowledge finding

<figure><img src="image_237.png" /></figure>

► Some solutions

▶ abstractions/models

▶ learning and sampling

▶ reuse of configuration knowledge

▶ leveraging stability

▶ systematic exploration

▶ identification of root causes

▶ LLMs to support exploration of variants' space

▶ incremental build of configuration space (Randrianaina et al. ICSE’22)

▶ debloating variability (Ternava et al. SAC’23)

▶ feature subset selection (Martin et al. SPLC’23)

<figure><img src="image_238.png" /></figure>

► Software engineering and variability researchers
have a key role to play for truly replicable science.
Backup slides (disclaimer: don't try to understand everything ;))
<figure><img src="image_239.png" /></figure>

# What can we do? (robustness)

## Robustness (trustworthiness) of scientific results to sources of variability

I have shown many examples of sources of variations and non-robust results...

Robustness should be rigorously defined (hint: it's not the definition as given in computer science)

How to verify the effect of sources of variations on the robustness of given conclusions?

* actionable metrics?

* methodology? (eg when to stop?)

* variability can actually be leveraged to augment confidence

<figure><img src="image_240.png" /></figure>
## Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, ... Tom Schonberg

+ Show authors

Nature 582, 84–88 (2020) | Cite this article

42k Accesses | 203 Citations | 2056 Altmetric | Metrics

different
assumptions

<figure><img src="image_241.png" /></figure>

different data

<figure><img src="image_242.png" /></figure>

<figure><img src="image_243.png" /></figure>

deep

software

variability

<figure><img src="image_244.png" /></figure>

different
methods

## Increasing Transparency Through a Multiverse Analysis

Sara Steegen ¹, Francis Tuerlinckx ¹, Andrew Gelman ², Wolf Vanpaemel ³

Affiliations + expand

PMID: 27694465 DOI: 10.1177/1745691616658637

different analyses

<figure><img src="image_245.png" /></figure>

## Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

R. Silberzahn, E. L. Uhlmann, D. P. Martin, more...

First Published August 23, 2018 | Research Article

https://doi.org/10.1177/2515245917747646

<figure><img src="image_246.png" /></figure>

Show all authors v
Deep software variability is...

a threat for reproducible research

"Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

an opportunity for replication

"A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses."

"A study that refutes some scientific findings of another study, through the collection of new data (possibly with different methods) and completion of new analyses."

robustifying and augmenting
scientific knowledge

<figure><img src="image_247.png" /></figure>
# Reproducible Science as a Testing Problem

## #1 Test Generation Problem (input)

inputs: computing environment, parameters of an algorithm, versions of a library or tool, choice of a programming language

## #2 Oracle Problem (output)

we usually ignore the outcome! (open problems; open questions; new knowledge)

Input

System under
Study
.replicable)

Output
(scientific
result)
# Reproduction vs replication http://rescience.github.io/faq/

> **Reproduction** of a computational study means running the same computation on the same input data, and then checking if the results are the same, or at least "close enough" when it comes to numerical approximations. Reproduction can be considered as **software testing** at the level of a complete study.

> We don't “test” in one run, in one computing environment, with one kind of input data, etc.

> **Replication** of a scientific study (computational or other) means repeating a published protocol, respecting its spirit and intentions but varying the technical details. For computational work, this would mean using different software, running a simulation from different initial conditions, etc. The idea is to change something that everyone believes shouldn't matter, and see if the scientific conclusions are affected or not.

> It is the most interesting direction, basically for synthesizing new scientific knowledge!

In both cases, there is the need to
harness the combinatorial explosion
of deep software variability

<figure><img src="image_248.png" /></figure>
# Reproducible Science and Software Engineering
@acherm

aka Deep Software Variability for Replicability in Computational Science

## Deep Questions?

### Reproducibility and Replicability

**Reproducible:** Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

**Replication:** A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses.

"Terminologies for Reproducible Research", Lorena A. Barba, 2018

<figure><img src="image_249.png" /></figure>

The Claerbout/Donoho/Peng terminology is broadly disseminated across disciplines (see Table 2). But the recent adoption of an opposing terminology by two large professional groups—ACM and FASEB—make standardization awkward. The ACM publicizes its rationale for adoption as based on the International Vocabulary of Metrology, but a close reading of the sources makes this justification tenuous. The source of the FASEB adoption is unclear, but there's a chance that Casadevall and Fang (2010) had an influence there. They, in turn, based their definitions on the emphatic but essentially flawed work of Drummond (2009).

<table><caption>Table 2: Grouping of terminologies, as in Table 1, but by discipline.</caption><thead><tr><th>A</th><th>B1</th><th>B2</th></tr></thead><tbody><tr><td>political science<br>economics</td><td>signal processing<br>scientific computing<br>economy<br>epidemiology<br>clinical studies<br>internal medicine<br>physiology (neuro)<br>computational biology<br>biomedical research<br>statistics</td><td>microbiology, immunology (FASEB)<br>computer science (ACM)</td></tr></tbody></table>

<figure><img src="image_250.png" /></figure>

Sometimes, variability is consistent/stable across layers and knowledge transfer is immediate.

But there are also interactions among variability layers and variability knowledge may not generalize.
<figure><img src="image_251.png" /></figure>

Reproducible
Builds

Reproducible builds are a set of software development practices that create an
independently-verifiable path from source to binary code. (more)

<figure><img src="image_252.png" /></figure>

Why does it matter?

Whilst anyone may inspect the source code of free and open source software for malicious flaws, most software is distributed pre-compiled
with no method to confirm whether they correspond.

This incentivises attacks on developers who release software, not only via traditional exploitation, but also in the forms of political influence, blackmail or even threats of violence.

This is particularly a concern for developers collaborating on privacy or security software: attacking these typically result in compromising
particularly politically-sensitive targets such as dissidents, journalists and whistleblowers, as well as anyone wishing to communicate securely
under a repressive regime.

Whilst individual developers are a natural target, it additionally encourages attacks on build infrastructure as a successful attack would
provide access to a large number of downstream computer systems. By modifying the generated binaries here instead of modifying the
upstream source code, illicit changes are essentially invisible to its original authors and users alike.

The motivation behind the **Reproducible Builds** project is therefore to allow verification that no vulnerabilities or backdoors have been
introduced during this compilation process. By promising identical results are always generated from a given source, this allows multiple third
parties to come to a consensus on a "correct" result, highlighting any deviations as suspect and worthy of scrutiny.

This ability to notice if a developer or build system has been compromised then prevents such threats or attacks occurring in the first place, as
any compromise can be quickly detected. As a result, front-liners cannot be threatened/coerced into exploiting or exposing their colleagues.

Several free software projects already, or will soon, provide reproducible builds.
<table><caption>Table 3: Summary of measured systems; N<sub>f</sub> – Number of features; NM – Number of machines on which systems were measured; NMC – Number of measured configurations</caption><thead><tr><th>System</th><th>N<sub>f</sub></th><th>NM</th><th>NMC</th></tr></thead><tbody><tr><td>XZ</td><td>7</td><td>7</td><td>154</td></tr><tr><td>x264</td><td>7</td><td>11</td><td>165</td></tr><tr><td>SQLite</td><td>5</td><td>10</td><td>32</td></tr></tbody></table>

<table><caption>Table 2: Summary of hardware platforms on which configurable software systems were measured; MID – Machine ID in DataMill cluster; NC – Number of CPUs; IS – Instruction set; CCR – CPU clock rate (MHz); RAM – RAM memory size (MB)</caption><thead><tr><th colspan="3">Systems</th><th colspan="5">Machines</th></tr><tr><th>XZ</th><th>x264</th><th>SQLite</th><th>MID</th><th>NC</th><th>IS</th><th>CCR</th><th>RAM</th></tr></thead><tbody><tr><td>✓</td><td></td><td></td><td>73</td><td>2</td><td>i686</td><td>1733</td><td>1771</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td>75</td><td>2</td><td>i686</td><td>3200</td><td>977</td></tr><tr><td>✓</td><td></td><td></td><td>77</td><td>2</td><td>i686</td><td>2992</td><td>2024</td></tr><tr><td>✓</td><td></td><td></td><td>78</td><td>1</td><td>i686</td><td>1495</td><td>755</td></tr><tr><td>✓</td><td></td><td></td><td>79</td><td>4</td><td>x86_64</td><td>3291</td><td>7961</td></tr><tr><td>✓</td><td></td><td></td><td>80</td><td>8</td><td>x86_64</td><td>3401</td><td>7907</td></tr><tr><td>✓</td><td>✓</td><td></td><td>81</td><td>16</td><td>x86_64</td><td>2411</td><td>32193</td></tr><tr><td>✓</td><td></td><td></td><td>87</td><td>1</td><td>i686</td><td>1595</td><td>249</td></tr><tr><td>✓</td><td></td><td></td><td>88</td><td>1</td><td>i686</td><td>1700</td><td>978</td></tr><tr><td>✓</td><td></td><td>✓</td><td>90</td><td>2</td><td>i686</td><td>3200</td><td>977</td></tr><tr><td>✓</td><td></td><td></td><td>91</td><td>1</td><td>i686</td><td>2400</td><td>1009</td></tr><tr><td>✓</td><td>✓</td><td></td><td>97</td><td>2</td><td>i686</td><td>2992</td><td>873</td></tr><tr><td>✓</td><td></td><td></td><td>98</td><td>2</td><td>i686</td><td>2992</td><td>873</td></tr><tr><td>✓</td><td></td><td></td><td>99</td><td>2</td><td>i686</td><td>2793</td><td>880</td></tr><tr><td>✓</td><td></td><td></td><td>103</td><td>2</td><td>i686</td><td>3200</td><td>881</td></tr><tr><td>✓</td><td></td><td></td><td>104</td><td>1</td><td>i686</td><td>1800</td><td>502</td></tr><tr><td>✓</td><td>✓</td><td></td><td>105</td><td>2</td><td>i686</td><td>3200</td><td>881</td></tr><tr><td>✓</td><td></td><td></td><td>106</td><td>2</td><td>i686</td><td>3192</td><td>494</td></tr><tr><td>✓</td><td></td><td>✓</td><td>125</td><td>4</td><td>x86_64</td><td>3301</td><td>7960</td></tr><tr><td>✓</td><td></td><td></td><td>128</td><td>2</td><td>i686</td><td>2993</td><td>2024</td></tr><tr><td></td><td>✓</td><td></td><td>130</td><td>2</td><td>i686</td><td>3198</td><td>880</td></tr><tr><td></td><td></td><td>✓</td><td>146</td><td>2</td><td>i686</td><td>2998</td><td>872</td></tr><tr><td></td><td></td><td></td><td>157</td><td>36</td><td>x86_64</td><td>2301</td><td>15954</td></tr></tbody></table>

“Linear model provides a good approximation of transformation between performance distributions of a system deployed in different hardware environments”

what about
variability of
input data?
compile-time options?
version?

<figure><img src="image_253.png" /></figure>
Transfer Learning for Software Performance Analysis: An Exploratory Analysis
Jamshidi et al. ASE 2017

SPEAR (SAT Solver)

Analysis time

14 options

16,384 configurations

SAT problems

3 hardware

2 versions

X264 (video encoder)

Encoding time

16 options

4,000 configurations

Video quality/size

2 hardware

3 versions

SQLite (DB engine)

Query time

14 options

1,000 configurations

DB Queries

2 hardware

2 versions

SaC (Compiler)

Execution time

50 options

71,267 configurations

10 Demo programs

<table><tr><td>ec1 : [h2 → h1, w3, v3]</td><td>SM</td><td>0.97</td></tr><tr><td>ec2 : [h2 → h1, w1, v3]</td><td>S</td><td>0.96</td></tr><tr><td>ec3 : [h1, w1 → w2, v3]</td><td>M</td><td>0.65</td></tr><tr><td>ec4 : [h1, w1 → w3, v3]</td><td>M</td><td>0.67</td></tr><tr><td>ec5 : [h1, w3, v2 → v3]</td><td>L</td><td>0.05</td></tr><tr><td>ec6 : [h1, w3, v1 → v3]</td><td>L</td><td>0.06</td></tr><tr><td>ec7 : [h1, w1 → w3, v2 → v3]</td><td>L</td><td>0.08</td></tr><tr><td>ec8 : [h2 → h1, w1 → w3, v2 → v3]</td><td>VL</td><td>0.09</td></tr></table>

**Insight.** For non-severe hardware changes, we can linearly transfer performance models across environments.

**Insight.** The strength of the influence of configuration options is typically preserved across environments.

**Insight.** A large percentage of configurations are typically invalid in both source and target environments.

<figure><img src="image_254.png" /></figure>
# Transfer Learning for Software Performance Analysis: An Exploratory Analysis

## Jamshidi et al. ASE 2017

**SPEAR (SAT Solver)**

**Analysis time**

14 options
16,384 configurations
SAT problems
3 hardware
2 versions

**X264 (video encoder)**

**Encoding time**

16 options
4,000 configurations
Video quality/size
2 hardware
3 versions

**SQLite (DB engine)**

**Query time**

14 options
1,000 configurations
DB Queries
2 hardware
2 versions

**SaC (Compiler)**

**Execution time**

50 options
71,267 configurations
10 Demo programs

<table><tr><td>ec1 : [h2 → h1, w3, v3]</td><td>SM</td><td>0.97</td></tr><tr><td>ec2 : [h2 → h1, w1, v3]</td><td>S</td><td>0.96</td></tr><tr><td>ec3 : [h1, w1 → w2, v3]</td><td>M</td><td>0.65</td></tr><tr><td>ec4 : [h1, w1 → w3, v3]</td><td>M</td><td>0.67</td></tr><tr><td>ec5 : [h1, w3, v2 → v3]</td><td>L</td><td>0.05</td></tr><tr><td>ec6 : [h1, w3, v1 → v3]</td><td>L</td><td>0.06</td></tr><tr><td>ec7 : [h1, w1 → w3, v2 → v3]</td><td>L</td><td>0.08</td></tr><tr><td>ec8 : [h2 → h1, w1 → w3, v2 → v3]</td><td>VL</td><td>0.09</td></tr></table>

<figure><img src="image_255.png" /></figure>

**mixing deep variability:** hard to assess the specific influence of each layer

**very few hardware, version, and input data... but lots of runtime configurations (variants)**

**Let's go deep with **input data!****
# Practical impacts for users, developers, scientists, and self-adaptive systems

Threats to **variability knowledge** for performance property bitrate

<figure><img src="image_256.png" /></figure>

* optimal configuration is specific to an input; a good configuration can be a bad one

* some options' values have an opposite effect depending on the input

* effectiveness of sampling strategies (random, 2-wise, etc.) is input specific (somehow confirming Pereira et al. ICPE 2020)

* predicting, tuning, or understanding configurable systems without being aware of inputs can be inaccurate and... pointless

<figure><img src="image_257.png" /></figure>

<figure><img src="image_258.png" /></figure>
# Computational science depends on software and its engineering

<figure><img src="image_259.png" /></figure>

- multi-million line of code base

- multi-dependencies

- multi-systems

- multi-layer

- multi-version

- multi-person

- multi-variant

## Climate is changing

<figure><img src="image_260.png" /></figure>

Observed warming 2009 - 2019 relative to 1961-1990

<figure><img src="image_261.png" /></figure>

<figure><img src="image_262.png" /></figure>

<figure><img src="image_263.png" /></figure>

## The sea level is rising

<figure><img src="image_264.png" /></figure>

<figure><img src="image_265.png" /></figure>

<figure><img src="image_266.png" /></figure>

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses
# x264 video encoder (compilation/build)

~ x264 --bframes 1 --ref 3 --cabac DiverSE-meeting.mp4 -o meeting13.webm

mathieuacher

localhost.localdomain

../x264-SLIMFAST/x264

x264-gcov

<figure><img src="image_267.png" /></figure>

./configure --help

--disable-thread
--disable-win32thread
--disable-interlaced
--bit-depth=BIT_DEPTH
--chroma-format=FORMAT

disable multithreaded encoding
disable win32threads (windows only)
disable interlaced encoding support
set output bit depth (8, 10, all) [all]
output chroma format (400, 420, 422, 444, all) [all]

Advanced options:

--disable-asm
--enable-lto
--enable-debug
--enable-gprof
--enable-strip
--enable-pic
disable platform-specific assembly optimizations
enable link-time optimization
add -g
add -pg
add -s
build position-independent code

Cross-compilation:

--host=HOST
--cross-prefix=PREFIX
--sysroot=SYSROOT
build programs to run on HOST
use PREFIX for compilation tools
root of cross-build tree

External library support:

--disable-avs
disable avisynth support
--disable-swscale
disable swscale support
--disable-lavf
disable lavformat support
--disable-ffms
disable fmmegsource support
--disable-gpac
disable gpac support
--disable-lsmash
disable lsmash support

compile-time
options

<figure><img src="image_268.png" /></figure>
# What can we do? (#1 studies)

## Empirical studies about deep software variability

* more subject systems

* more variability layers, including interactions

* more quantitative (e.g., performance) properties

with challenges for gathering measurements data:

* how to scale experiments? Variant space is huge!

* how to fix/isolate some layers? (eg hardware)

* how to measure in a reliable way?

Expected outcomes:

* significance of deep software variability in the wild

* identification of **stable** layers: sources of variability that should not affect the conclusion and that can be eliminated/forgotten

* identification/quantification of **sensitive** layers and interactions that matter

* **variability knowledge**

<figure><img src="image_269.png" /></figure>
# What can we do? (#2 cost)

## Reducing the cost of exploring the variability spaces

Many directions here (references at the end of the slides):

* **learning**
  * many algorithms/techniques with tradeoffs interpretability/accuracy
  * transfer learning (instead of learning from scratch)

* **sampling strategies**
  * uniform random sampling? t-wise? distance-based? ...
  * sample of hardware? input data?

* incremental build of configurations

* white-box approaches

* ...

<figure><img src="image_270.png" /></figure>
# Key results (for x264)

<figure><img src="image_271.png" /></figure>

**Worth tuning software at compile-time:** gain about 10 % of execution time with the tuning of compile-time options (compared to the default compile-time configuration). The improvements can be larger for some inputs and some runtime configurations.

**Stability of variability knowledge:** For all the execution time distributions of x264 and all the input videos, the worst correlation is greater than 0.97. If the compile-time options change the scale of the distribution, they do not change the rankings of run-time configurations (i.e., they do not truly interact with the run-time options).

**Reuse of configuration knowledge:** $f_1 = \beta \times f_2 + \alpha$

* Linear transformation among distributions

* Users can also trust the documentation of run-time options, consistent whatever the compile-time configuration is.

<figure><img src="image_272.png" /></figure>
# Our Vision

## Embrace deep variability!

Explicit modeling of the variability points and their relationships, such as:

1. Get insights into the variability “factors” and their possible interactions

2. Capture and document configurations for the sake of **reproducibility**

3. Explore diverse configurations to **replicate**, and hence optimize, validate, increase the robustness, or provide better resilience

https://hal.science/hal-04582287

## Embracing Deep Variability For Reproducibility & Replicability

Mathieu Acher

Univ Rennes, Inria, CNRS, IRISA, IUF
Rennes, France

Benoit Combemale

Univ Rennes, Inria, CNRS, IRISA
Rennes, France

Jean-Marc Jézéquel

Univ Rennes, Inria, CNRS, IRISA, IUF
Rennes, France

Georges Aaron Randrianaina

Univ Rennes, Inria, CNRS, IRISA
Rennes, France

### ABSTRACT

Reproducibility (a.k.a. determinism in some cases) constitutes a fundamental aspect in various fields of computer science, such as floating-point computations in numerical analysis and simulation, concurrency models in parallelism, reproducible builds for third parties integration and packaging, and containerization for execution environments. These concepts, while pervasive across diverse concerns, often exhibit intricate inter-dependencies, making it challenging to achieve a comprehensive understanding. In this short and vision paper we delve into the application of software engineering techniques, specifically variability management, to systematically identify and explicit points of variability that may give rise to reproducibility issues (e.g. language, libraries, compiler, virtual machine, OS, environment variables, etc). The primary objectives are: i) gaining insights into the variability layers and their possible interactions, ii) capturing and documenting configurations for the sake of reproducibility, and iii) exploring diverse configurations to replicate, and hence validate and ensure the robustness of results. By adopting these methodologies, we aim to address the complexities associated with reproducibility and replicability in modern software systems and environments, facilitating a more comprehensive and nuanced perspective on these critical aspects.

### 1 INTRODUCTION

Many scientific domains need to process large amount of data with more and more complex computations. For instance, studies about climate modelling and change involve the design of mathematical model, the mining and analysis of data, the executions of large simulations, etc. [16, 24, 29]. These computational tasks rely on various kinds of software, from a set of scripts to automate the deployment to a comprehensive system containing several features that help researchers exploring various hypotheses. It is not an overstatement to say that computational science depends on software and its engineering [2, 34, 54].

One of the main promise of software is that a result obtained by an experiment (e.g. a simulation) can be achieved again with a high degree of agreement. But despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results [6, 9, 15, 19, 22, 31, 41, 42, 53]. For instance, applications of different analysis pipelines, alterations

community [22]. As a result, software can threaten the scientific knowledge and recommendations built on top of these computations and studies.

In this paper we propose to characterize both intended and unintended variability of any software-intensive system in order to support reproducibility and replicability, and eventually estimate its robustness, uncertainty profile, and explore different hypotheses.

### 2 DEEP SOFTWARE VARIABILITY

Uncertainty in informatics comes from many different origins [17, 39], either ontological (i.e., inherent unpredictability, e.g., aleatory) or epistemic (i.e., due to insufficient knowledge).

Ontological causes include noise in the input data of a program, its memory layout, network delays, the internal state of the processor, the ambient temperature and even the age of the processor¹.

Epistemic causes include misunderstanding of the user's needs, variable behavior of conceptually similar resolution methods, choice of threshold parameters, unexpected behavior of APIs, variable behavior among functionally similar libraries, or subtle differences in the semantics of programming languages (e.g., ~3% evaluates to ~1 in Java but 1 in Python), or even inside the same programming language (for instance x/0 is an undefined behavior in C).

<figure><img src="image_273.png" /><figcaption>Figure 1: Deep Variability</figcaption></figure>

⇒ We aim to **address the complexities associated with reproducibility and replicability in modern software systems and environments**, facilitating a more comprehensive and nuanced perspective on these critical “factors”.
Multi-Level Analysis of Compiler-Induced Variability and Performance Tradeoffs

Michael Bentley
Ian Briggs
Ganesh Gopalakrishnan
mbentley@cs.utah.edu
ianbriggsutah@gmail.com
ganesh@cs.utah.edu
University of Utah

Dong H. Ahn
Ignacio Laguna
Gregory L. Lee
Holger E. Jones
ahn1@llnl.gov
lagunaperalt1@llnl.gov
lee218@llnl.gov
jones19@llnl.gov
Lawrence Livermore National Laboratory

**Definition of Reproducibility.** Given the growing heterogeneity of hardware and software, one cannot always define reproducibility as achieving bitwise reproducible results. Instead, we view a reproducible computation as one that produces a result within an “acceptable range” of a trusted baseline answer. In FLiT, we rely on the application developer to provide an acceptance testing function that (indirectly) defines this range.

<table><caption>Table 1: Compilers used in the MFEM study with summary statistics. The best flags are chosen by the best average speedup across all MFEM examples. The average speedup over all 19 MFEM examples is reported and is calculated relative to the speed of g++ -O2.</caption><thead><tr><td>Compiler</td><td>Released</td><td># Variable Runs</td><td>Best Flags</td><td>Speedup</td></tr></thead><tbody><tr><td>gcc-8.2.0</td><td>26 July 2018</td><td>78 of 1,288 (6.0%)</td><td>-02 -funsafe-math-optimizations</td><td>1.097</td></tr><tr><td>clang-6.0.1</td><td>05 July 2018</td><td>24 of 1,368 (1.8%)</td><td>-03 -funsafe-math-optimizations</td><td>1.042</td></tr><tr><td>icpc-18.0.3</td><td>16 May 2018</td><td>984 of 1,976 (49.8%)</td><td>-02 -fp-model fast=2</td><td>1.056</td></tr></tbody></table>

<figure><img src="image_274.png" /></figure>

<figure><img src="image_275.png" /><figcaption>Figure 1: Multi-level workflow. Levels are (1) determine variability-inducing compilations, (2) analyze the space of reproducibility and performance, and (3) debug variability by identifying files and functions causing variability.</figcaption></figure>
`exec(software) = exec_repro(software)`

or

`exec(software) =~ exec(software_repro)`

(difference: `exec_repro` is another execution environment... and so somehow differs or not with `exec`; or we consider that software differs...)

(`exec`: execution? what's the outcome then? in fact execution can be replaced by "build"... which is another kind of execution)

`exec(software) ?= exec_repro(software)`

`software =~ software_repro`

`exec(software, hardware)`

`exec(software, hardware, compiler, input_data, operating_system, bios, container, hypervisor, dependencies_versions)`

`exec (v1, v2, ..., vN) =~ exec_repro (v1', v2', ..., vN')`

`for i in [1, n], v_{i} =~ v_{i} (or not!)`

`~= is specific to a domain, to a usage, etc.`

`~= can be over the N layers or over N' layers (N' < N)`

`~= can be specific to some pairs elements (eg we know that with this hardware, the exec time is multiplied by 2)`

for instance, we know the `~=` between a software configuration with any hardware (but if the compiler changes, then the `~=` should be "tuned" accordingly)

also `~=` can be defined between a configuration set and an hardware set (eg performance distribution)
## Exact same results? No

**Definition of Reproducibility.** Given the growing heterogeneity of hardware and software, one cannot always define reproducibility as achieving bitwise reproducible results. Instead, we view a reproducible computation as one that produces a result within an “acceptable range” of a trusted baseline answer. In FLiT, we rely on the application developer to provide an acceptance testing function that (indirectly) defines this range.

## Multi-Level Analysis of Compiler-Induced Variability and Performance Tradeoffs

Michael Bentley
Ian Briggs
Ganesh Gopalakrishnan
mbentley@cs.utah.edu
ianbriggsutah@gmail.com
ganesh@cs.utah.edu
University of Utah

Dong H. Ahn
Ignacio Laguna
Gregory L. Lee
Holger E. Jones
ahn1@llnl.gov
lagunaperalt1@llnl.gov
lee218@llnl.gov
jones19@llnl.gov
Lawrence Livermore National Laboratory

<figure><img src="image_276.png" /><figcaption>Table 1: Compilers used in the MFEM study with summary statistics. The best flags are chosen by the best average speedup across all MFEM examples. The average speedup over all 19 MFEM examples is reported and is calculated relative to the speedup of g** -02.</figcaption></figure>

<table><caption>Figure 1: Multi-level workflow. Levels are (1) determine variability-inducing compilations, (2) analyze the space of reproducible and performance, and (3) debug variability by identifying files and functions causing variability.</caption><thead><tr><td>Compiler</td><td>Released</td><td># Variable Runs</td><td>Best Flags</td><td>Speedup</td></tr></thead><tbody><tr><td>gcc-8.2.0</td><td>26 July 2018</td><td>78 of 1,288 (6.0%)</td><td>-02 -funsafe-math-optimizations</td><td>1.097</td></tr><tr><td>clang-6.0.1</td><td>05 July 2018</td><td>24 of 1,368 (1.8%)</td><td>-03 -funsafe-math-optimizations</td><td>1.042</td></tr><tr><td>icp-18.0.3</td><td>16 May 2018</td><td>984 of 1,978 (49.8%)</td><td>-02 -fp-model fast-2</td><td>1.056</td></tr></tbody></table>
Frictionless reproducibility (annotated bibliography; grey literature)

https://hdsr.mitpress.mit.edu/pub/8dqqwqiu/release/1 The Mechanics of Frictionless
Reproducibility, B Recht

interesting historical perspective on research in neural networks (NeurIPs 87 titles are shockingly
still relevant); really love some parts about random experiments, science as a “massively parallel
genetic algorithm” or the discussions on the difficulty of using ML/DL software (completely
aligned with my terrible experience of Weka GUI in ~2006)

https://www.argmin.net/p/the-department-of-frictionless-reproducibility

https://statmodeling.stat.columbia.edu/2023/10/13/frictionless-reproducibility-methods-as-proto-al
gorithms-division-of-labor-as-a-characteristic-of-statistical-methods-statistics-as-the-science-of-d
efaults-statisticians-well-prepared-to-think-abo/
Progress and frictionless reproducibility

Inspired by Thomas Kuhn (1962), we can think of the scientific and engineering process as a massively parallel genetic algorithm. If we want to improve upon the systems we currently have, we might try a small perturbation to see if we get an improvement. If we can find a small change that improves some desired outcome, we could change our systems to reflect this improvement. If we continually search for these improvements and work hard to demonstrate their value, we may head in a better direction over time.

For scientific endeavors, we could perhaps gauge ‘better’ or ‘worse’ by performing random experiments—not randomized experiments per se, but random experiments in the sense of trying potentially surprising improvements. If our small tweak results in better outcomes, we can attempt to convince a journal editor or conference program committee to publish it. And this communication gives everyone else a new starting point for their own random experimentation.

A single investigator can only make so much progress by random searching alone, but random search is pleasantly parallelizable. Competing scientists can independently try their own random ideas and publish their results. Sometimes an individual result is so promising that the herd of experimenters all flock around the good idea, hoping to strike gold on a nearby improvement and bring home bragging rights. To some, this looks like an inefficient mess. To others, it looks like science.

https://hdsr.mitpress.mit.edu/pub/8dqqwqiu/release/1 The Mechanics of Frictionless Reproducibility, B Recht
# Data sharing and frictions

> "Data set benchmarking and competitive testing took over machine learning in the late 1980s. Email and file transfer were becoming more accessible. The current specification of FTP was finalized in 1985. In 1987, a PhD student at UC Irvine named David Aha put up an FTP server to host data sets for empirically testing machine learning methods. Aha was motivated by service to the community, but he also wanted to show his nearest-neighbor methods would outperform Ross Quinlan's decision tree induction algorithms. He formatted his data sets using the 'attribute-value' representation that a rival researcher, Ross Quinlan (1986), had used. And, so, the UC Irvine Machine Learning Repository was born."

> "Improvements in computing greased the wheels, giving us faster computers, faster data transfer, and smaller storage footprints. But computing technology alone was not sufficient to drive progress. Friendly competition with Quinlan inspired Aha to build the UCI repository. **And more explicit competitions were also crucial components of the success.**"

The Mechanics of Frictionless Reproducibility, B Recht, 2024

https://hdsr.mitpress.mit.edu/pub/8dqgwqiu/release/1
In [1]: import torch, struct
...: def binary_double(num):
...:     print("'.join(f'{c:0>8b}' for c in struct.pack('!d', num)))
...:     binary_double(torch.tensor(9, device='cuda')/10)
...:     binary_double(torch.tensor(9, device='cpu')/10)
001111111101100110011001100110011110000000000000000000000000000
00111111111101100110011001100111100000000000000000000000000000

<figure><img src="image_277.png" /></figure>

Stas Bekman

@StasBekman · Jan 20

Floating point math discrepancies with some pretrained LM models can be an issue.

I was debugging today a weird discrepancy between Llama-2-7b inference
results which proved to be triggered by whether `from_pretrained` was called ...

Show more

<figure><img src="image_278.png" /></figure>

This is from the mps device:

<figure><img src="image_279.png" /></figure>
<figure><img src="image_280.png" /><figcaption>Figure 2: Feature model (excerpt). Inverse (resp. Relation-WithPi) corresponds to checking the property $(x*z)/(y*z) = x/y$ (resp. $(x*z*\pi)/(y*z*\pi) = x/y$) with $z, y \neq 0$</figcaption></figure>

```python
def equality_test(equality_check: EqualityCheck, x, y, z) -> bool:
    if equality_check == EqualityCheck.ASSOCIATIVITY:
        return x+(y+z) == (x+y)+z
    elif equality_check == EqualityCheck.MULT_INV:
        return (x * z) / (y * z) == x / y
    elif equality_check == EqualityCheck.MULT_INV_PI:
        return (x * z * math.pi) / (y * z * math.pi) == (x / y)
```

```python
fn check_ratio(config: &Config, x: f64, y: f64, z: f64) -> bool:
    if let Some,errormargin) = config.errormargin {
        #[cfg(feature = "associativity")]
        ((x + y) + z - x - (y + z)).abs() < errormargin
    }
    #[cfg(feature = "multInverse")]
    ((x * z) / (y * z) - x / y).abs() < errormargin
}
#[cfg(feature = "multInverse_pi")]
((x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) - x / y).abs() < errormargin
}
}
else {
    #[cfg(feature = "associativity")]
    (x + y) + z == x + (y + z)
}
#[cfg(feature = "multInverse")]
(x * z) / (y * z) == x / y
}
#[cfg(feature = "multInverse_pi")]
(x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) == (x / y)
}
}
```
<figure><img src="image_281.png" /><figcaption>Figure 2: Feature model (excerpt). Inverse (resp. Relation-WithPi) corresponds to checking the property $(x*z)/(y*z) = x/y$ (resp. $(x*z*\pi)/(y*z*\pi) = x/y$) with $z, y \neq 0$</figcaption></figure>

```c
fn associativity_test(config: &Config) -> bool {
    let mut rng = thread rng();
    // TODO: this variant for generating random
    // let x = rng.gen::<f64>();
    // let y = rng.gen::<f64>();
    // let z = rng.gen::<f64>();
    let x = rng.gen_range(0.000_000_000_000_001..100.0); // TODO: variation point for range min, max value
    let y = rng.gen_range(0.000_000_000_000_001..100.0);
    let z = rng.gen_range(0.000_000_000_000_001..100.0);
    check_ratio(config, x, y, z)
}

fn proportion(config: &Config, number: i32, seed_val: u64) -> i32 {
    StdRng::seed_from_u64(seed_val);
    let mut ok = 0;
    for _ in 0..number {
        if associativity_test(config) {
            ok += 1;
        }
    }
    ok * 100 / number
}
```

```c
if errorMargin:
    variability_misc = f"--errorMargin {errorMargin}"
    cmd_args = [
        "cargo",
        "run",
        "--features",
        feature,
        "-q",
        "--",
        "--errorMargin",
        errorMargin,
    ]
```

```c
fn check_ratio(config: &Config, x: f64, y: f64, z: f64) -> bool {
    if let Some errorMargin = config.errorMargin {
        &[cfg(feature = "associativity")]
        {
            [(x + y) + z * x * (y + z)].abs() < errorMargin
        }
        &[cfg(feature = "multInverse")]
        {
            [(x * z) / (y * z) - x / y].abs() < errorMargin
        }
        &[cfg(feature = "multInversePi")]
        {
            [(x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) - x / y].abs() < errorMargin
        }
    } else {
        &[cfg(feature = "associativity")]
        {
            (x + y) + z == x + (y + z)
        }
        &[cfg(feature = "multInverse")]
        {
            (x * z) / (y * z) == x / y
        }
        &[cfg(feature = "multInversePi")]
        {
            (x * z * std::f64::constants::PI) / (y * z * std::f64::constants::PI) == (x / y)
        }
    }
}
```
<table><thead><tr><th>Language</th><th>Library</th><th>System</th><th>Compiler</th><th>VariabilityMisc</th><th>EqualityCheck</th><th>NumberGenerations</th><th>Repeat</th><th>min</th><th>max</th><th>std</th><th>mean</th></tr></thead><tbody><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV</td><td>100</td><td>10</td><td>60.0</td><td>71.0</td><td>3.562302626111375</td><td>65.1</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>51.0</td><td>63.0</td><td>3.330165161069343</td><td>55.9</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV</td><td>100</td><td>10</td><td>62.0</td><td>62.0</td><td>0.0</td><td>62.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>47.0</td><td>47.0</td><td>0.0</td><td>47.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>associativity</td><td>100</td><td>10</td><td>71.0</td><td>82.0</td><td>3.3466401061363023</td><td>76.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>mult-inverse</td><td>100</td><td>10</td><td>58.0</td><td>78.0</td><td>6.0</td><td>66.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>mult-inverse-pi</td><td>100</td><td>10</td><td>42.0</td><td>64.0</td><td>5.885575587824865</td><td>53.4</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>associativity</td><td>100</td><td>10</td><td>81.0</td><td>81.0</td><td>0.0</td><td>81.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse</td><td>100</td><td>10</td><td>70.0</td><td>70.0</td><td>0.0</td><td>70.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse-pi</td><td>100</td><td>10</td><td>56.0</td><td>56.0</td><td>0.0</td><td>56.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV</td><td>100</td><td>10</td><td>62.0</td><td>72.0</td><td>2.764054992217051</td><td>66.6</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>47.0</td><td>57.0</td><td>2.808914381037628</td><td>53.1</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV</td><td>100</td><td>10</td><td>67.0</td><td>67.0</td><td>0.0</td><td>67.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>53.0</td><td>53.0</td><td>0.0</td><td>53.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>74.0</td><td>90.0</td><td>4.6097722286464435</td><td>82.5</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>MULT_INV</td><td>100</td><td>10</td><td>60.0</td><td>79.0</td><td>6.16765757804371</td><td>68.6</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>49.0</td><td>59.0</td><td>2.8301943396169813</td><td>54.3</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>89.0</td><td>89.0</td><td>0.0</td><td>89.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_INV</td><td>100</td><td>10</td><td>73.0</td><td>73.0</td><td>0.0</td><td>73.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>55.0</td><td>55.0</td><td>0.0</td><td>55.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None approximate equality of Julia lang</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None approximate equality of Julia lang</td><td>MULT_INV</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr></tbody></table>

https://github.com/FAMILIAR-project/reproducibility-associativity/