<figure><img src="image_1.png" /></figure>

HAL
open science

Deep Software Variability and Frictionless
Reproducibility

Mathieu Acher

<figure><img src="image_2.png" /></figure>

To cite this version:

Mathieu Acher. Deep Software Variability and Frictionless Reproducibility. 2024, pp.1-121. hal-04601752

HAL Id: hal-04601752

https://hal.science/hal-04601752

Submitted on 5 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
<figure><img src="image_1.png" /></figure>

HAL
open science

# Deep Software Variability and Frictionless Reproducibility

Mathieu Acher

► To cite this version:

Mathieu Acher. Deep Software Variability and Frictionless Reproducibility. 2024, pp.1-121. hal-04601752

HAL Id: hal-04601752

https://hal.science/hal-04601752v1

Submitted on 5 Jun 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
<figure><img src="image_3.png" /></figure>

# Deep Software Variability and Frictionless Reproducibility

Mathieu Acher @acherm
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

Groupement
de recherche

GPL Génie de la programmation
et du logiciel

GDR GPL

Génie de la Programmation et du Logiciel

<figure><img src="image_4.png" /></figure>

cnrs

# Deep Software Variability and Frictionless Reproducibility

Mathieu Acher @acherm

<figure><img src="image_5.png" /></figure>

Inria

DiverSE
Diversity-Centric Software Engineering

UNIVERSITE DE RENNES

<figure><img src="image_6.png" /></figure>

cnrs

<figure><img src="image_7.png" /></figure>

Université
de Rennes

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

institut
universitaire
de France

<figure><img src="image_10.png" /></figure>

UMR

IRISA
# Deep Software Variability and Frictionless Reproducibility

**Abstract:** The ability to recreate computational results with minimal effort and actionable metrics provides a solid foundation for scientific research and software development. When people can replicate an analysis at the touch of a button using open-source software, open data, and methods to assess and compare proposals, it significantly eases verification of results, engagement with a diverse range of contributors, and progress. However, we have yet to fully achieve this; there are still many sociotechnical frictions.

Inspired by David Donoho's vision, this talk aims to revisit the three crucial pillars of frictionless reproducibility (data sharing, code sharing, and competitive challenges) with the perspective of deep software variability.

Our observation is that multiple layers — hardware, operating systems, third-party libraries, software versions, input data, compile-time options, and parameters — are subject to variability that exacerbates frictions but is also essential for achieving robust, generalizable results and fostering innovation. I will first review the literature, providing evidence of how the complex variability interactions across these layers affect qualitative and quantitative software properties, thereby complicating the reproduction and replication of scientific studies in various fields.

I will then present some software engineering and AI techniques that can support the strategic exploration of variability spaces. These include the use of abstractions and models (e.g., feature models), sampling strategies (e.g., uniform, random), cost-effective measurements (e.g., incremental build of software configurations), and dimensionality reduction methods (e.g., transfer learning, feature selection, software bloating).

I will finally argue that deep variability is both the problem and solution of frictionless reproducibility, calling the software science community to develop new methods and tools to manage variability and foster reproducibility in software systems.
# Deep Software Variability and Frictionless Reproducibility

**Abstract:** The ability to recreate computational results with minimal effort and actionable metrics provides a solid foundation for scientific research and software development. When people can replicate an analysis at the touch of a button using open-source software, open data, and methods to assess and compare proposals, it significantly eases verification of results, engagement with a diverse range of contributors, and progress. However, we have yet to fully achieve this; there are still many sociotechnical frictions.

Inspired by David Donoho's vision, this talk aims to revisit the three crucial pillars of frictionless reproducibility (data sharing, code sharing, and competitive challenges) with the perspective of deep software variability.

Our observation is that multiple layers — hardware, operating systems, third-party libraries, software versions, input data, compile-time options, and parameters — are subject to variability that exacerbates frictions but is also essential for achieving robust, generalizable results and fostering innovation. I will first review the literature, providing evidence of how the complex variability interactions across these layers affect qualitative and quantitative software properties, thereby complicating the reproduction and replication of scientific studies in various fields.

I will then present some software engineering and AI techniques that can support the strategic exploration of variability spaces. These include the use of abstractions and models (e.g., feature models), sampling strategies (e.g., uniform, random), cost-effective measurements (e.g., incremental build of software configurations), and dimensionality reduction methods (e.g., transfer learning, feature selection, software bloating).

I will finally argue that deep variability is both the problem and solution of frictionless reproducibility, calling the software science community to develop new methods and tools to manage variability and foster reproducibility in software systems.
Special thanks to* Aaron Randrianaina, Jean-Marc Jézéquel, Benoit Combemale, Luc Lesoil, Arnaud Gotlieb, Helge Spieker, Quentin Mazouni, Paul Temple, Gauthier Le Bartz Lyan, Xhevahire Tërnava, Olivier Barais, and the whole DiverSE and RIPOST teams

*random order, incomplete
Special thanks to* Aaron Randrianaina, Jean-Marc Jézéquel, Benoit Combemale, Luc Lesoil, Arnaud Gotlieb, Helge Spieker, Quentin Mazouni, Paul Temple, Gauthier Le Bartz Lyan, Xhevahire Tërnava, Olivier Barais, and the whole DiverSE and RIPOST teams

*random order, incomplete
# AGENDA

## Frictionless Reproducibility and (Deep) Software (Variability)

**Problem:** Variability and Frictions

**Solution:** Variability and Exploration

**Discussions**
# AGENDA

## Frictionless Reproducibility and (Deep) Software (Variability)

**Problem:** Variability and Frictions

**Solution:** Variability and Exploration

**Discussions**
SOFTWARE VARIANTS
ARE EATING THE WORLD

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

OBS
Open Broadcaster Software*

NETFLIX

twitch

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

SQLite

node

<figure><img src="image_12.png" /></figure>

kubernetes

x265

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

Poppler

money

<figure><img src="image_15.png" /></figure>

GCC

<figure><img src="image_16.png" /></figure>

ANdROId

<figure><img src="image_17.png" /></figure>

zoom

<figure><img src="image_18.png" /></figure>

OPENVPN™

Dropbox

<figure><img src="image_19.png" /></figure>

JHipster

<figure><img src="image_20.png" /></figure>

git

<figure><img src="image_21.png" /></figure>

git

<figure><img src="image_22.png" /></figure>

docker

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

Thunderbird

Polaris

<figure><img src="image_26.png" /></figure>

VLC

<figure><img src="image_27.png" /></figure>

MPlayer

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

Fleep

<figure><img src="image_31.png" /></figure>

h265

<figure><img src="image_32.png" /></figure>

Microsoft Teams

<figure><img src="image_33.png" /></figure>

Microsoft
SQL Azure

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

WavePad
Bundled Sound

Stockfish

MySQL

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>

unity

@FileSender

<figure><img src="image_38.png" /></figure>

MS
Paint

<figure><img src="image_39.png" /></figure>

skype

<figure><img src="image_40.png" /></figure>

avast

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

Lichess 2021
SOFTWARE VARIANTS
ARE EATING THE WORLD

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

OBS
Open Broadcaster Software*

NETFLIX

twitch

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

SQLite

node

<figure><img src="image_19.png" /></figure>

kubernetes

x265

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

Poppler

montopop

GCC

<figure><img src="image_22.png" /></figure>

ANONYMOUS

Dropbox

<figure><img src="image_23.png" /></figure>

JHipster

<figure><img src="image_24.png" /></figure>

Twitter

<figure><img src="image_25.png" /></figure>

git

<figure><img src="image_26.png" /></figure>

docker

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>

Thunderbird

Polaris SQL

<figure><img src="image_31.png" /></figure>

VLC

<figure><img src="image_32.png" /></figure>

MPlayer

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

F

Fleep

<figure><img src="image_35.png" /></figure>

zoom

<figure><img src="image_36.png" /></figure>

OPENVPN™

<figure><img src="image_37.png" /></figure>

h265

<figure><img src="image_38.png" /></figure>

Microsoft Teams

<figure><img src="image_39.png" /></figure>

Microsoft
SQL Azure

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

Stockfish

<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

unity

@FileSender

<figure><img src="image_45.png" /></figure>

MS
Paint

<figure><img src="image_46.png" /></figure>

skype

<figure><img src="image_47.png" /></figure>

avast

<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /></figure>

Lichess 2021
# Science is changing: Computation-based research

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>

PHOTO: THE VOORHIES

<figure><img src="image_49.png" /></figure>

<figure><img src="image_50.png" /></figure>

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>
Science is changing:
Computation-based research

<figure><img src="image_50.png" /></figure>

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

<figure><img src="image_54.png" /></figure>

<figure><img src="image_55.png" /></figure>

PHOTO: THE VOORHIES

<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>

<figure><img src="image_61.png" /></figure>

<figure><img src="image_62.png" /></figure>
# Computational science depends on software and its engineering

<figure><img src="image_56.png" /></figure>

<figure><img src="image_57.png" /></figure>

design of mathematical model
mining and analysis of data
executions of large simulations
problem solving
executable paper

from a set of scripts to automate the deployment to... a
comprehensive system containing several features that
help researchers exploring various hypotheses

<figure><img src="image_58.png" /></figure>

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>
# Computational science depends on software and its engineering

<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>

design of mathematical model
mining and analysis of data
executions of large simulations
problem solving
executable paper

from a set of scripts to automate the deployment to... a
comprehensive system containing several features that
help researchers exploring various hypotheses

<figure><img src="image_65.png" /></figure>

<figure><img src="image_66.png" /></figure>

<figure><img src="image_67.png" /></figure>
# Computational science depends on software and its engineering

multi-million line of code base

multi-dependencies

multi-systems

multi-layer

multi-version

multi-person

multi-variant

<figure><img src="image_61.png" /></figure>

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

## Dealing with software collapse: software stops working eventually

Konrad Hinsen 2019

Configuration failures represent one of the most common types of software failures Sayagh et al. TSE 2018
# Computational science depends on software and its engineering

* multi-million line of code base

* multi-dependencies

* multi-systems

* multi-layer

* multi-version

* multi-person

* multi-variant

<figure><img src="image_68.png" /></figure>

<figure><img src="image_69.png" /></figure>

<figure><img src="image_70.png" /></figure>

## Dealing with software collapse: software stops working eventually

Konrad Hinsen 2019

Configuration failures represent one of the most common types of software failures Sayagh et al. TSE 2018
“Insanity is doing the same thing over and over again
and expecting different results”

http://throwgrammarfromthetrain.blogspot.com/2010/10/definition-of-insanity.html

<figure><img src="image_64.png" /></figure>

<figure><img src="image_65.png" /></figure>

<figure><img src="image_66.png" /></figure>

<figure><img src="image_67.png" /></figure>

<figure><img src="image_68.png" /></figure>

<figure><img src="image_69.png" /></figure>

<figure><img src="image_70.png" /></figure>

<figure><img src="image_71.png" /></figure>

CMake
Cross-platform Make

<figure><img src="image_72.png" /></figure>

NumPy

<figure><img src="image_73.png" /></figure>

puppet

<figure><img src="image_74.png" /></figure>

docker

<figure><img src="image_75.png" /></figure>

<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>

GCC

<figure><img src="image_78.png" /></figure>

<figure><img src="image_79.png" /></figure>
“Insanity is doing the same thing over and over again
and expecting different results”

http://throwgrammarfromthetrain.blogspot.com/2010/10/definition-of-insanity.html

<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /></figure>

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>

<figure><img src="image_75.png" /></figure>

<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>

<figure><img src="image_78.png" /></figure>

CMake
Cross-platform Make

<figure><img src="image_79.png" /></figure>

NumPy

<figure><img src="image_80.png" /></figure>

puppet

<figure><img src="image_81.png" /></figure>

docker

<figure><img src="image_82.png" /></figure>

<figure><img src="image_83.png" /></figure>

<figure><img src="image_84.png" /></figure>

GCC

<figure><img src="image_85.png" /></figure>

<figure><img src="image_86.png" /></figure>
# Reproducibility

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

(Claerbout/Donoho/Peng definition)

> "The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures." (~executable paper)

<figure><img src="image_80.png" /></figure>

<figure><img src="image_81.png" /></figure>

<figure><img src="image_82.png" /></figure>
# Reproducibility

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

(Claerbout/Donoho/Peng definition)

> "The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures." (~executable paper)

<figure><img src="image_87.png" /></figure>

<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /></figure>
# Reproducibility and Replicability

**Reproducible:** Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

**Replication:** A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses.

"Terminologies for Reproducible Research", Lorena A. Barba, 2018

<figure><img src="image_83.png" /></figure>

CMake
Cross-platform Make

<figure><img src="image_84.png" /></figure>

NumPy

<figure><img src="image_85.png" /></figure>

puppet

docker

<figure><img src="image_86.png" /></figure>
# Reproducibility and Replicability

**Reproducible:** Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

**Replication:** A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses.

"Terminologies for Reproducible Research", Lorena A. Barba, 2018

<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /></figure>

<figure><img src="image_92.png" /></figure>
# Reproducibility and Replicability

**Reproducible:** Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

**Replication:** A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses.

"Terminologies for Reproducible Research", Lorena A. Barba, 2018

The Claerbout/Donoho/Peng terminology is broadly disseminated across disciplines (see Table 2). But the recent adoption of an opposing terminology by two large professional groups—ACM and FASEB—make standardization awkward. The ACM publicizes its rationale for adoption as based on the International Vocabulary of Metrology, but a close reading of the sources makes this justification tenuous. The source of the FASEB adoption is unclear, but there's a chance that Casadevall and Fang (2010) had an influence there. They, in turn, based their definitions on the emphatic but essentially flawed work of Drummond (2009).

<table><caption>Table 2: Grouping of terminologies, as in Table 1, but by discipline.</caption><thead><tr><th>A</th><th>B1</th><th>B2</th></tr></thead><tbody><tr><td>political science<br>economics</td><td>signal processing<br>scientific computing<br>econometry<br>epidemiology<br>clinical studies<br>internal medicine<br>physiology (neuro)<br>computational biology<br>biomedical research<br>statistics</td><td>microbiology, immunology (FASEB)<br>computer science (ACM)</td></tr></tbody></table>

*As a result of discussions with the National Information Standards Organization (NISO), it was recommended that ACM harmonize its terminology and definitions with those used in the broader scientific research community, and ACM agreed with NISO's recommendation to swap the terms "reproducibility" and "replication" with the existing definitions used by ACM as part of its artifac review and badging initiative. ACM took action to update all prior badging to ensure consistency.

<figure><img src="image_87.png" /></figure>
# Reproducibility and Replicability

**Reproducible:** Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

**Replication:** A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses.

"Terminologies for Reproducible Research", Lorena A. Barba, 2018

The Claerbout/Donoho/Peng terminology is broadly disseminated across disciplines (see Table 2). But the recent adoption of an opposing terminology by two large professional groups—ACM and FASEB—make standardization awkward. The ACM publicizes its rationale for adoption as based on the International Vocabulary of Metrology, but a close reading of the sources makes this justification tenuous. The source of the FASEB adoption is unclear, but there's a chance that Casadevall and Fang (2010) had an influence there. They, in turn, based their definitions on the emphatic but essentially flawed work of Drummond (2009).

<table><caption>Table 2: Grouping of terminologies, as in Table 1, but by discipline.</caption><thead><tr><th>A</th><th>B1</th><th>B2</th></tr></thead><tbody><tr><td>political science<br>economics</td><td>signal processing<br>scientific computing<br>econometry<br>epidemiology<br>clinical studies<br>internal medicine<br>physiology (neuro)<br>computational biology<br>biomedical research<br>statistics</td><td>microbiology, immunology (FASEB)<br>computer science (ACM)</td></tr></tbody></table>

*As a result of discussions with the National Information Standards Organization (NISO), it was recommended that ACM harmonize its terminology and definitions with those used in the broader scientific research community, and ACM agreed with NISO's recommendation to swap the terms "reproducibility" and "replication" with the existing definitions used by ACM as part of its artifac review and badging initiative. ACM took action to update all prior badging to ensure consistency.

<figure><img src="image_93.png" /></figure>
# Reproducibility and Replicability

**Methods Reproducibility:** A method is reproducible if reusing the original code leads to the same results.

**Results Reproducibility:** A result is reproducible if a reimplementation of the method generates statistically similar values.

**Inferential Reproducibility:** A finding or a conclusion is reproducible if one can draw it from a different experimental setup.

“Unreproducible Research is Reproducible”, Bouthillier et al., ICML 2019

<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /></figure>
# Reproducibility and Replicability

**Methods Reproducibility:** A method is reproducible if reusing the original code leads to the same results.

**Results Reproducibility:** A result is reproducible if a reimplementation of the method generates statistically similar values.

**Inferential Reproducibility:** A finding or a conclusion is reproducible if one can draw it from a different experimental setup.

“Unreproducible Research is Reproducible”, Bouthillier et al., ICML 2019

<figure><img src="image_94.png" /></figure>

<figure><img src="image_95.png" /></figure>
# Reproducible science

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

**Socio-technical issues:** open science, open source software, multi-disciplinary collaboration, incentives/rewards, initiatives, etc.

with many challenges related to data acquisition, knowledge organization/sharing, etc.

<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /></figure>

<figure><img src="image_92.png" /></figure>
# Reproducible science

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

**Socio-technical issues:** open science, open source software, multi-disciplinary collaboration, incentives/rewards, initiatives, etc.

with many challenges related to data acquisition, knowledge organization/sharing, etc.

<figure><img src="image_96.png" /></figure>

<figure><img src="image_97.png" /></figure>

<figure><img src="image_98.png" /></figure>
# Reproducible science

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

**Socio-technical issues:** open science, open source software, multi-disciplinary collaboration, incentives/rewards, initiatives, etc. with many challenges related to data acquisition, knowledge organization/sharing, etc.

## EMSE Open Science Initiative

Openness in science is key to fostering progress via transparency, reproducibility, and replicability. Especially open data and open source are two fundamental pillars in open science as both build the core for excellence in evidence-based research. The Empirical Software Engineering Journal (EMSE) has therefore decided to explicitly foster open science and reproducible research by encouraging and supporting authors to share their (anonymised and curated) empirical data and source code in form of replication packages. The overall goals are:

* Increasing the transparency, reproducibility, and replicability of research endeavours. This supports the immediate credibility of authors' work, and it also provides a common basis for joint community efforts grounded on shared data.

* Building up an overall body of knowledge in the community leading to widely accepted and well-formed software engineering theories in the long run.
https://github.com/emsejournal/openscience

**Reproducible Science is good. Replicated Science is better.**

ReScience C is a *platinum open-access* peer-reviewed journal that targets computational research and encourages the explicit *replication* of already published research, promoting new and open-source implementations in order to ensure that the original research is *reproducible*. You can read about the ideas behind ReScience C in the article *Sustainable computational science: the ReScience initiative*

https://rescience.github.io/

https://reproducible-research.inria.fr/

OpenReview.net

<figure><img src="image_93.png" /></figure>

Reproducible
Builds

Reproducible builds are a set of software development practices that create an independently- verifiable path from source to binary code. [see:]

<figure><img src="image_94.png" /></figure>

Software Heritage

<figure><img src="image_95.png" /></figure>

GitLab

GitHub

Entreprise

<figure><img src="image_96.png" /></figure>

<figure><img src="image_97.png" /></figure>
# Reproducible science

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

**Socio-technical issues:** open science, open source software, multi-disciplinary collaboration, incentives/rewards, initiatives, etc. with many challenges related to data acquisition, knowledge organization/sharing, etc.

## EMSE Open Science Initiative

Openness in science is key to fostering progress via transparency, reproducibility, and replicability. Especially open data and open source are two fundamental pillars in open science as both build the core for excellence in evidence-based research. The Empirical Software Engineering Journal (EMSE) has therefore decided to explicitly foster open science and reproducible research by encouraging and supporting authors to share their (anonymised and curated) empirical data and source code in form of replication packages. The overall goals are:

* Increasing the transparency, reproducibility, and replicability of research endeavours. This supports the immediate credibility of authors' work, and it also provides a common basis for joint community efforts grounded on shared data.

* Building up an overall body of knowledge in the community leading to widely accepted and well-formed software engineering theories in the long run.
https://github.com/emsejournal/openscience

**Reproducible Science is good. Replicated Science is better.**

ReScience C is a *platinum open-access* peer-reviewed journal that targets computational research and encourages the explicit *replication* of already published research, promoting new and open-source implementations in order to ensure that the original research is *reproducible*. You can read about the ideas behind ReScience C in the article *Sustainable computational science: the ReScience initiative*

https://rescience.github.io/

https://reproducible-research.inria.fr/

OpenReview.net

<figure><img src="image_99.png" /></figure>

Reproducible
Builds

Reproducible builds are a set of software development practices that create an independently- verifiable path from source to binary code. [see:]

<figure><img src="image_100.png" /></figure>

Software Heritage

<figure><img src="image_101.png" /></figure>

GitLab

GitHub

Entreprise

<figure><img src="image_102.png" /></figure>

<figure><img src="image_103.png" /></figure>
# Reproducible science

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

**Socio-technical issues:** open science, open source software, multi-disciplinary collaboration, incentives/rewards, initiatives, etc.
with many challenges related to data acquisition, knowledge organization/sharing, etc.

<figure><img src="image_98.png" /></figure>

<figure><img src="image_99.png" /></figure>

## CALL FOR CHALLENGE CASES

Home / Call for Papers / Call for challenge cases

## ARTIFACT EVALUATION

Authors of accepted research papers are invited to submit the artifacts associated with paper for evaluation. To do so, they should submit a PDF via EasyChair (select the Research or Development or Other artifacts track). The PDF should contain a stable URL (or DOI) to the artifacts. The URL contain the steps or general instructions to execute/analyze the artifact. Each artifact submission will be reviewed by at least two reviewers.

According to ACM's "Result and Artifact Review and Badging" policy, an "artifact" is "a digital object that was either created by the authors to be used as part of the study or generated by the experiment itself [... which can include] software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results."

In this track, an industrial performance dataset will be provided. The participants are invited to come up with research questions about the dataset, and study those. The challenge is open-ended: participants can choose the research questions that they find most interesting. The proposed approaches and/or tools and their findings are discussed in short papers, and presented in the main conference.

ICPE 2022

13th ACM/SPEC International Conference on
Performance Engineering
# Reproducible science

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

**Socio-technical issues:** open science, open source software, multi-disciplinary collaboration, incentives/rewards, initiatives, etc.
with many challenges related to data acquisition, knowledge organization/sharing, etc.

<figure><img src="image_104.png" /></figure>

<figure><img src="image_105.png" /></figure>

## CALL FOR CHALLENGE CASES

Home / Call for Papers / Call for challenge cases

ICPE 2022

13th ACM/SPEC International Conference on
Performance Engineering

## ARTIFACT EVALUATION

Authors of accepted research papers are invited to submit the artifacts associated with paper for evaluation. To do so, they should submit a PDF via EasyChair (select the Research Artifacts track). The PDF should contain a stable URL (or DOI) to the artifacts. The URL contain the steps or general instructions to execute/analyze the artifact. Each artifact submission will be reviewed by at least two reviewers.

According to ACM's "Result and Artifact Review and Badging" policy, an " artifact" is "a digital object that was either created by the authors to be used as part of the study or generated by the experiment itself [ ... which can include ] software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results."

In this track, an industrial performance dataset will be provided. The participants are invited to come up with research questions about the dataset, and study those. The challenge is open-ended: participants can choose the research questions that they find most interesting. The proposed approaches and/or tools and their findings are discussed in short papers, and presented in the main conference.
# Lamb and Zacchiroli “Reproducible Builds: Increasing the Integrity of Software Supply Chains” IEEE Software 2022

https://arxiv.org/pdf/2104.06020

(best paper award IEEE Software for year 2022)

> “The build process of a software product is reproducible if, after designating a specific version of its source code and all of its build dependencies, every build produces bit-for-bit identical artifacts, no matter the environment in which the build is performed.”

<figure><img src="image_100.png" /></figure>

Reproducible
Builds

Reproducible builds are a set of software development practices that create an independently-verifiable path from source to binary code. (more)
# Lamb and Zacchiroli “Reproducible Builds: Increasing the Integrity of Software Supply Chains” IEEE Software 2022

https://arxiv.org/pdf/2104.06020

(best paper award IEEE Software for year 2022)

> “The build process of a software product is reproducible if, after designating a specific version of its source code and all of its build dependencies, every build produces bit-for-bit identical artifacts, no matter the environment in which the build is performed.”

<figure><img src="image_106.png" /></figure>

Reproducible
Builds

Reproducible builds are a set of software development practices that create an independently-verifiable path from source to binary code. (more)
# Frictionless reproducibility

Statistics > Other Statistics

[Submitted on 2 Oct 2023]

## Data Science at the Singularity

David Donoho

https://arxiv.org/abs/2310.00865

https://hdsr.mitpress.mit.edu/pub/g9mau4m0/release/2

> “Computation-driven research really has changed in the last 10 years, driven by three principles of data science, which, after longstanding partial efforts, are finally available in mature form for daily practice, as frictionless open services offering **data sharing, code sharing, and competitive challenges**.”

### [FR-1: Data] + [FR-2: Re-execution] + [FR-3: Challenges]

> “We are entering an era of frictionless research exchange, in which research algorithmically builds on the digital artifacts created by earlier research, and any good ideas that are found get spread rapidly, everywhere. The collective behavior induced by frictionless research exchange is the emergent superpower driving many events that are so striking today.”
# Frictionless reproducibility

Statistics > Other Statistics

[Submitted on 2 Oct 2023]

## Data Science at the Singularity

David Donoho

https://arxiv.org/abs/2310.00865

https://hdsr.mitpress.mit.edu/pub/g9mau4m0/release/2

> "Computation-driven research really has changed in the last 10 years, driven by three principles of data science, which, after longstanding partial efforts, are finally available in mature form for daily practice, as frictionless open services offering **data sharing, code sharing, and competitive challenges**."

## [FR-1: Data] + [FR-2: Re-execution] + [FR-3: Challenges]

> "We are entering an era of frictionless research exchange, in which research algorithmically builds on the digital artifacts created by earlier research, and any good ideas that are found get spread rapidly, everywhere. The collective behavior induced by frictionless research exchange is the emergent superpower driving many events that are so striking today."
# Frictionless reproducibility

**[FR-1: Data]** "Datafication of everything, with a culture of research data sharing."

**[FR-2: Re-execution (code)]:** "Research code sharing including the ability to exactly re-execute the same complete workflow by different researchers."

**[FR-3: Challenges]** "a shared public dataset, a prescribed and quantified task performance metric, a set of enrolled competitors seeking to outperform each other on the task, and a public leaderboard."

<figure><img src="image_101.png" /></figure>

<figure><img src="image_102.png" /></figure>

CODE

<figure><img src="image_103.png" /></figure>

<figure><img src="image_104.png" /></figure>

performance
metric

<figure><img src="image_105.png" /></figure>

Leaderboard

<figure><img src="image_106.png" /></figure>
# Frictionless reproducibility

**[FR-1: Data]** "Datafication of everything, with a culture of research data sharing."

**[FR-2: Re-execution (code)]:** "Research code sharing including the ability to exactly re-execute the same complete workflow by different researchers."

**[FR-3: Challenges]** "a shared public dataset, a prescribed and quantified task performance metric, a set of enrolled competitors seeking to outperform each other on the task, and a public leaderboard."

<figure><img src="image_107.png" /></figure>

<figure><img src="image_108.png" /></figure>

CODE

<figure><img src="image_109.png" /></figure>

performance
metric

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

Leaderboard

<figure><img src="image_112.png" /></figure>
# Frictionless reproducibility

**[FR-1: Data]** "Datafication of everything, with a culture of research data sharing."

**[FR-2: Re-execution (code)]:** "Research code sharing including the ability to exactly re-execute the same complete workflow by different researchers."

**[FR-3: Challenges]** "a shared public dataset, a prescribed and quantified task performance metric, a set of enrolled competitors seeking to outperform each other on the task, and a public leaderboard."

$$ \text{frictionless reproducibility} = [\text{FR-1}] + [\text{FR-2}] + [\text{FR-3}] $$

<figure><img src="image_107.png" /></figure>

<figure><img src="image_108.png" /></figure>

<figure><img src="image_109.png" /></figure>

performance
metric

<table><caption>Table 1: Leave-One-outs, and what is blocked</caption><thead><tr><th>If we only have...</th><th>We are blocked, because</th><th>Example</th></tr></thead><tbody><tr><td>[FR-1] + [FR-2]</td><td>No defined task</td><td>Exploratory Data Analysis</td></tr><tr><td>[FR-1] + [FR-3]</td><td>Can't build on code of others</td><td>Netflix Challenge;<br>DARPA Biometric Challenges</td></tr><tr><td>[FR-2] + [FR-3]</td><td>No Common Dataset</td><td>Human Subjects Clinical Research</td></tr></tbody></table>
# Frictionless reproducibility

**[FR-1: Data]** "Datafication of everything, with a culture of research data sharing."

**[FR-2: Re-execution (code)]:** "Research code sharing including the ability to exactly re-execute the same complete workflow by different researchers."

**[FR-3: Challenges]** "a shared public dataset, a prescribed and quantified task performance metric, a set of enrolled competitors seeking to outperform each other on the task, and a public leaderboard."

$$ \text{frictionless reproducibility} = [\text{FR-1}] + [\text{FR-2}] + [\text{FR-3}] $$

<figure><img src="image_113.png" /></figure>

<figure><img src="image_114.png" /></figure>

<figure><img src="image_115.png" /></figure>

performance
metric

<table><caption>Table 1: Leave-One-outs, and what is blocked</caption><thead><tr><th>If we only have...</th><th>We are blocked, because</th><th>Example</th></tr></thead><tbody><tr><td>[FR-1] + [FR-2]</td><td>No defined task</td><td>Exploratory Data Analysis</td></tr><tr><td>[FR-1] + [FR-3]</td><td>Can't build on code of others</td><td>Netflix Challenge;<br>DARPA Biometric Challenges</td></tr><tr><td>[FR-2] + [FR-3]</td><td>No Common Dataset</td><td>Human Subjects Clinical Research</td></tr></tbody></table>
# Frictionless reproducibility

frictionless reproducibility = [FR-1: Data] + [FR-2: Re-execution] + [FR-3: Challenges]

[FR-1] and [FR-2] are quite “standard” but do not come without frictions – more soon! [FR-3] is an important and original piece

**On the one hand**, [FR-3] is a way to objectively assess a contribution, compare solutions, and measure progress (if any). [FR-3] sounds legit to provide a “task definition that formalized a specific research problem and made it an object of study”. [FR-3] is “the competitive element that attracted our attention in the first place”.

**Think about the absence of [FR-3].** The “challenge paradigm” is a big ongoing shift (see Isabelle Guyon and Evelyne Viegas - "AI Competitions and the Science Behind Contests")

* Many success stories (mainly in empirical machine learning): speech processing, biometric recognition, facial recognition, protein structure prediction problem (CASP), etc.

* More and more leaderboard (eg https://evalplus.github.io/leaderboard.html or competition (eg SAT competition)

* Many platforms, services, and events supporting the shift (eg Kaggle)
# Frictionless reproducibility

frictionless reproducibility = [FR-1: Data] + [FR-2: Re-execution] + [FR-3: Challenges]

[FR-1] and [FR-2] are quite “standard” but do not come without frictions – more soon! [FR-3] is an important and original piece

**On the one hand**, [FR-3] is a way to objectively assess a contribution, compare solutions, and measure progress (if any). [FR-3] sounds legit to provide a “task definition that formalized a specific research problem and made it an object of study”. [FR-3] is “the competitive element that attracted our attention in the first place”.

**Think about the absence of [FR-3].** The “challenge paradigm” is a big ongoing shift (see Isabelle Guyon and Evelyne Viegas - "AI Competitions and the Science Behind Contests")

* Many success stories (mainly in empirical machine learning): speech processing, biometric recognition, facial recognition, protein structure prediction problem (CASP), etc.

* More and more leaderboard (eg [https://evalplus.github.io/leaderboard.html](https://evalplus.github.io/leaderboard.html)) or competition (eg SAT competition)

* Many platforms, services, and events supporting the shift (eg Kaggle)
# Frictionless reproducibility

frictionless reproducibility = [FR-1: Data] + [FR-2: Re-execution] + [FR-3: Challenges]

[FR-1] and [FR-2] are quite “standard” but do not come without frictions – more soon! [FR-3] is an important and original piece

**On the one hand**, [FR-3] is a way to objectively assess a contribution, compare solutions, and measure progress (if any).
[FR-3] sounds legit to provide a “task definition that formalized a specific research problem and made it an object of study”. [FR-3] is “the competitive element that attracted our attention in the first place”. The performance measurement crystallized a specific project’s contribution, boiling down an entire research contribution essentially to a single number, which can be reproduced. **Think about the absence of [FR-3]**

The “challenge paradigm” is a big ongoing shift (see Isabelle Guyon and Evelyne Viegas - "AI Competitions and the Science Behind Contests")

* Many success stories (mainly in empirical machine learning): speech processing, biometric recognition, facial recognition, protein structure prediction problem (CASP), etc.

* More and more leaderboard (eg [https://evalplus.github.io/leaderboard.html](https://evalplus.github.io/leaderboard.html)) or competition (eg SAT competition)

* Many platforms, services, and events supporting the shift (eg Kaggle)
# Frictionless reproducibility

frictionless reproducibility = [FR-1: Data] + [FR-2: Re-execution] + [FR-3: Challenges]

[FR-1] and [FR-2] are quite “standard” but do not come without frictions – more soon! [FR-3] is an important and original piece

**On the one hand**, [FR-3] is a way to objectively assess a contribution, compare solutions, and measure progress (if any).
[FR-3] sounds legit to provide a “task definition that formalized a specific research problem and made it an object of study”. [FR-3] is “the competitive element that attracted our attention in the first place”. The performance measurement crystallized a specific project’s contribution, boiling down an entire research contribution essentially to a single number, which can be reproduced. **Think about the absence of [FR-3]**

The “challenge paradigm” is a big ongoing shift (see Isabelle Guyon and Evelyne Viegas - "AI Competitions and the Science Behind Contests")

* Many success stories (mainly in empirical machine learning): speech processing, biometric recognition, facial recognition, protein structure prediction problem (CASP), etc.

* More and more leaderboard (eg [https://evalplus.github.io/leaderboard.html](https://evalplus.github.io/leaderboard.html)) or competition (eg SAT competition)

* Many platforms, services, and events supporting the shift (eg Kaggle)
# Frictionless reproducibility

frictionless reproducibility = [FR-1: Data] + [FR-2: Re-execution] + [FR-3: Challenges]

[FR-1] and [FR-2] are quite “standard” but do not come without frictions – more soon! [FR-3] is an important but *discussable* piece

On the other hand, we know that the power of a simple scoring function is dangerous (e.g., Goodhart's law)

"What if the metric is wrong? What if the subtleties of a complex problem are not amenable to representation by a single scalar? What happens when metrics for locally optimal solutions are apparent, but ones for globally optimal solutions are not? What happens when the community is not (yet) mature enough to rally around a consensus-scoring function? I think it is important to recognize that finding an appropriate scoring function, let alone an objectively best one, is an ongoing task and might evolve as FR-1 and FR-2 provide a deeper understanding of the problem space."

Overcoming Potential Obstacles as We Strive for Frictionless Reproducibility by Adam D. Schuyler (2024)

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

CODE

<figure><img src="image_112.png" /></figure>

performance
metric

<figure><img src="image_113.png" /></figure>

<figure><img src="image_114.png" /></figure>
# Frictionless reproducibility

frictionless reproducibility = [FR-1: Data] + [FR-2: Re-execution] + [FR-3: Challenges]

[FR-1] and [FR-2] are quite “standard” but do not come without frictions – more soon! [FR-3] is an important but *discussable* piece

**On the other hand**, we know that the power of a simple scoring function is dangerous (e.g., Goodhart's law)

"What if the metric is wrong? What if the subtleties of a complex problem are not amenable to representation by a single scalar? What happens when metrics for locally optimal solutions are apparent, but ones for globally optimal solutions are not? What happens when the community is not (yet) mature enough to rally around a consensus-scoring function? I think it is important to recognize that finding an appropriate scoring function, let alone an objectively best one, **is an ongoing task and might evolve as FR-1 and FR-2 provide a deeper understanding of the problem space**."

Overcoming Potential Obstacles as We Strive for Frictionless Reproducibility by Adam D. Schuyler (2024)

<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>

CODE

<figure><img src="image_118.png" /></figure>

performance
metric

<figure><img src="image_119.png" /></figure>

<figure><img src="image_120.png" /></figure>
# Are we frictionless?

Reading a paper in 2024 is sometimes like in 1970:

* Where is the source code? (eg implementation of the solution, scripts to compute metrics)

* Where is the data? (eg to test the solution)

* Contacting authors?

* no response?

* code not consistent with the PDF

* ...

* It does not work on my machine; results are completely different...

**There are lots of socio-technical frictions... even when you have the code and data!**

=> When people can replicate an analysis at the touch of a button using open-source software, open data, and methods to assess and compare proposals, it significantly eases verification of results, engagement with a diverse range of contributors, and progress
# Are we frictionless?

Reading a paper in 2024 is sometimes like in 1970:

* Where is the source code? (eg implementation of the solution, scripts to compute metrics)

* Where is the data? (eg to test the solution)

* Contacting authors?

* no response?

* code not consistent with the PDF

* ...

* It does not work on my machine; results are completely different...

**There are lots of socio-technical frictions... even when you have the code and data!**

=> When people can replicate an analysis at the touch of a button using open-source software, open data, and methods to assess and compare proposals, it significantly eases verification of results, engagement with a diverse range of contributors, and progress
# Frictionless reproducibility (an example)

Cutting through buggy adversarial example defenses:
fixing 1 line of code breaks SABRE

Nicholas Carlini

Google DeepMind

<figure><img src="image_115.png" /></figure>

## Abstract

SABRE is a defense to adversarial examples that was accepted at IEEE S&P 2024. We first reveal significant flaws in the evaluation that point to clear signs of gradient masking. We then show the cause of this gradient masking: a bug in the original evaluation code. By fixing a single line of code in the original repository, we reduce SABRE's robust accuracy to 0%. In response to this, the authors modify the defense and introduce a new defense component not described in the original paper. But this fix contains a second bug; modifying one more line of code reduces robust accuracy to *below* baseline levels. After we released the first version of our paper online, the authors introduced another change to the defense; by commenting out one line of code during attack we reduce the robust accuracy to 0% again.

```python
diff --git a/core/defenses/sabre.py b/core/defenses/sabre.py
2 1
3 index fe509e6..bf13629 100644
4
5 --- a/core/defenses/sabre.py
6
7 +++ b/core/defenses/sabre.py
8
9 @@ -165,7 +165,7 @@ class SabreWrapper nn.Module):
10         model = Sabre eps=eps, wave=wave, use_random=use_random, n
11         self.core = model
12         self.base_model = base_model
13
14 - self transform = BPDAWrapper lambda x, lambda_r: model transform x, lambda_r).float()
15 + self transform = (lambda x, lambda_r: model transform x, lambda_r).float()
16
17 @property
18 def lambda_r(self):
```

## Submission history

From: Nicholas Carlini [view email]

[v1] Mon, 6 May 2024 17:48:24 UTC (19 KB)

[v2] Mon, 27 May 2024 17:41:06 UTC (20 KB)
# Frictionless reproducibility (an example)

Cutting through buggy adversarial example defenses:
fixing 1 line of code breaks SABRE

Nicholas Carlini

Google DeepMind

<figure><img src="image_121.png" /></figure>

## Abstract

SABRE is a defense to adversarial examples that was accepted at IEEE S&P 2024. We first reveal significant flaws in the evaluation that point to clear signs of gradient masking. We then show the cause of this gradient masking: a bug in the original evaluation code. By fixing a single line of code in the original repository, we reduce SABRE's robust accuracy to 0%. In response to this, the authors modify the defense and introduce a new defense component not described in the original paper. But this fix contains a second bug; modifying one more line of code reduces robust accuracy to *below* baseline levels. After we released the first version of our paper online, the authors introduced another change to the defense; by commenting out one line of code during attack we reduce the robust accuracy to 0% again.

```bash
diff --git a/core/defenses/sabre.py b/core/defenses/sabre.py
```

2 index fe509e6..bf13629 100644

3 --- a/core/defenses/sabre.py

4 +++ b/core/defenses/sabre.py

5 @@ -165,7 +165,7 @@ class SabreWrapper nn.Module):

model = Sabre eps=eps, wave=wave, use_random=use_random, n

7 self.core = model

8 self.base_model = base_model

9 - self transform = BPDAWrapper (lambda x, lambda r: model. transform (x, lambda r).float())

10 + self transform = (lambda x, lambda r: model. transform (x, lambda r).float())

11 @property

12 def lambda_r (self):

## Submission history

From: Nicholas Carlini [view email]

[v1] Mon, 6 May 2024 17:48:24 UTC (19 KB)

[v2] Mon, 27 May 2024 17:41:06 UTC (20 KB)
# Reproducible science... with frictions

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

> Despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results.

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses

<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>

<figure><img src="image_118.png" /></figure>
# Reproducible science... with frictions

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

> Despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results.

<figure><img src="image_122.png" /></figure>

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses

<figure><img src="image_123.png" /></figure>

<figure><img src="image_124.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹, Xavier Yepes-Arbós¹,², Eleftheria Exarchou¹,², and Francisco J. Doblas-Reyes¹,²,⁴

<figure><img src="image_119.png" /></figure>

Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? Using two versions of EC-Earth: one “non-replicable” case (see below) and one replicable case.

<figure><img src="image_120.png" /><figcaption>12m difference between five-member experiments m06e and e011. Black dotted regions indicate where the difference is significant according to a Kolmogorov-Smirnov test (99% of grid points show a significant difference)</figcaption></figure>

<figure><img src="image_121.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹, Xavier Yepes-Arbós¹,², Eleftheria Exarchou¹,², and Francisco J. Doblas-Reyes¹,²,⁴

<figure><img src="image_125.png" /></figure>

Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? Using two versions of EC-Earth: one “non-replicable” case (see below) and one replicable case.

<figure><img src="image_126.png" /><figcaption>12m difference between five-member experiments m06e and e011. Black dotted regions indicate where the difference is significant according to a Kolmogorov-Smirnov test (99% of grid points show a significant difference)</figcaption></figure>

<figure><img src="image_127.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹, Xavier Yepes-Arbós¹,⁵, Eleftheria Exarchou¹,⁵,², and Francisco J. Doblas-Reyes¹,⁵,²,⁴

<figure><img src="image_122.png" /></figure>

Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? Using two versions of EC-Earth: one “non-replicable” case (see below) and one replicable case.

<figure><img src="image_123.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹, Xavier Yepes-Arbós¹,⁵, Eleftheria Exarchou¹,⁵,², and Francisco J. Doblas-Reyes¹,⁵,²,⁴

(a) Model code

EC-EARTH

(b) Different environments

Computing environment A

Computing environment B

(c) Five-member 20-year ensemble simulations (constant forcing)

<figure><img src="image_128.png" /></figure>

(d) Calculation of standard indices of performance

<figure><img src="image_129.png" /></figure>

(e) Statistical testing of differences in performance

<figure><img src="image_130.png" /></figure>

(f) Conclusion on replicability

<figure><img src="image_131.png" /></figure>

Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? Using two versions of EC-Earth: one “non-replicable” case (see below) and one replicable case.

12m difference between five-member experiments m05e and e011. Black dotted regions indicate where the difference is significant according to a Kolmogorov-Smirnov test (39% of grid points show a significant difference)

<figure><img src="image_132.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹,², Xavier Yepes-Arbós¹,², Eleftheria Exarchou¹,², and Francisco J. Doblas-Reyes¹,²,⁴

<figure><img src="image_124.png" /></figure>

Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? Using two versions of EC-Earth: one “non-replicable” case (see below) and one replicable case.

<table><caption>**Table 1.** The two computing environments considered in this study.</caption><thead><tr><th></th><th>ECMWF-CCA</th><th>MareNostrum3</th></tr></thead><tbody><tr><td>Computing environment</td><td>ECMWF-CCA</td><td>MareNostrum3</td></tr><tr><td>Location</td><td>Reading, UK</td><td>Barcelona, Spain</td></tr><tr><td>Motherboard</td><td>Cray XC30 system</td><td>IBM dx360 M4</td></tr><tr><td>Processor</td><td>Dual 12-core E5-2697 v2 (Ivy Bridge) series processors (2.7 GHz), 24 cores per node</td><td>2x Intel SandyBridge-EP E5-2670/1600 20M 8-core at 2.6 GHz, 16 cores per node</td></tr><tr><td>Operating system</td><td>Cray Linux Environment (CLE) 5.2</td><td>Linux - SuSe distribution 11 SP2</td></tr><tr><td>Compiler</td><td>Intel(R) 64 Compiler XE for applications running on Intel(R) 64, version 14.0.1.106 build 20131008</td><td>Intel(R) 64 Compiler XE for applications running on Intel(R) 64, version 13.0.1.117 build 20121010</td></tr><tr><td>MPI version</td><td>Cray mpich2 v6.2.0</td><td>Intel MPI v4.1.3.049</td></tr><tr><td>LAPACK version</td><td>Cray libsci v12.2.0</td><td>Intel MKL v11.0.1</td></tr><tr><td>SZIP, HDF5, NetCDF4</td><td>v2.1, v1.8.11, v4.3.0</td><td>v2.1, v1.8.14, v4.2</td></tr><tr><td>GribAPI, GribEX</td><td>v1.13.0, v000395</td><td>v1.14.0, v000370</td></tr></tbody></table>

<table><caption>**Table 2.** The four experiments considered in this study.</caption><thead><tr><th>Experiment ID</th><th>e011</th><th>m06e</th><th>a0gi</th><th>a0go</th></tr></thead><tbody><tr><td>Computing environment</td><td>ECMWF-CCA</td><td>MareNostrum3</td><td>ECMWF-CCA</td><td>MareNostrum3</td></tr><tr><td>EC-Earth version</td><td>3.1</td><td>3.1</td><td>3.2</td><td>3.2</td></tr><tr><td>Processors (IFS+NEMO+OASIS)</td><td>598</td><td>512</td><td>432 (288 + 144)</td><td>416 (288 + 128)</td></tr><tr><td>F flags</td><td>(480 + 96 + 22)</td><td>(384 + 96 + 22)</td><td>(OASIS: library)</td><td>(OASIS:library)</td></tr><tr><td rowspan="3">C flags</td><td>-O2 -g</td><td>-O2 -g</td><td>-O2 -g</td><td>-O2 -fp-model precise -xHost</td></tr><tr><td>-traceback</td><td>-traceback</td><td>-traceback -r8</td><td>-g -traceback</td></tr><tr><td>-vec-report0 -r8</td><td>-vec-report0 -r8</td><td>-fp-model strict</td><td>-g -traceback -r8</td></tr><tr><td>LD flags</td><td>-O2 -g</td><td>-O2 -g</td><td>-fp-model strict -xHost</td><td>-O2 -fp-model precise -xHost</td></tr><tr><td>Output size</td><td>141.8 GB</td><td>141.6 GB</td><td>101.3 GB</td><td>101.3 GB</td></tr></tbody></table>

<figure><img src="image_125.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹,², Xavier Yepes-Arbós¹,², Eleftheria Exarchou¹,², and Francisco J. Doblas-Reyes¹,²,⁴

<figure><img src="image_133.png" /></figure>

Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? Using two versions of EC-Earth: one “non-replicable” case (see below) and one replicable case.

<table><caption>**Table 1.** The two computing environments considered in this study.</caption><thead><tr><th></th><th>ECMWF-CCA</th><th>MareNostrum3</th></tr></thead><tbody><tr><td>Computing environment</td><td>ECMWF-CCA</td><td>MareNostrum3</td></tr><tr><td>Location</td><td>Reading, UK</td><td>Barcelona, Spain</td></tr><tr><td>Motherboard</td><td>Cray XC30 system</td><td>IBM dx360 M4</td></tr><tr><td>Processor</td><td>Dual 12-core E5-2697 v2 (Ivy Bridge) series processors (2.7 GHz), 24 cores per node</td><td>2x Intel SandyBridge-EP E5-2670/1600 20M 8-core at 2.6 GHz, 16 cores per node</td></tr><tr><td>Operating system</td><td>Cray Linux Environment (CLE) 5.2</td><td>Linux - SuSe distribution 11 SP2</td></tr><tr><td>Compiler</td><td>Intel(R) 64 Compiler XE for applications running on Intel(R) 64, version 14.0.1.106 build 20131008</td><td>Intel(R) 64 Compiler XE for applications running on Intel(R) 64, version 13.0.1.117 build 20121010</td></tr><tr><td>MPI version</td><td>Cray mpich2 v6.2.0</td><td>Intel MPI v4.1.3.049</td></tr><tr><td>LAPACK version</td><td>Cray libsci v12.2.0</td><td>Intel MKL v11.0.1</td></tr><tr><td>SZIP, HDF5, NetCDF4</td><td>v2.1, v1.8.11, v4.3.0</td><td>v2.1, v1.8.14, v4.2</td></tr><tr><td>GribAPI, GribEX</td><td>v1.13.0, v000395</td><td>v1.14.0, v000370</td></tr></tbody></table>

<table><caption>**Table 2.** The four experiments considered in this study.</caption><thead><tr><th>Experiment ID</th><th>e011</th><th>m06e</th><th>a0gi</th><th>a0go</th></tr></thead><tbody><tr><td>Computing environment</td><td>ECMWF-CCA</td><td>MareNostrum3</td><td>ECMWF-CCA</td><td>MareNostrum3</td></tr><tr><td>EC-Earth version</td><td>3.1</td><td>3.1</td><td>3.2</td><td>3.2</td></tr><tr><td>Processors (IFS+NEMO+OASIS)</td><td>598<br>(480+96+22)</td><td>512<br>(384+96+22)</td><td>432 (288 + 144)<br>(OASIS: library)</td><td>416 (288 + 128)<br>(OASIS: library)</td></tr><tr><td>F flags</td><td>-O2 -g<br>-traceback<br>-vec-report0 -r8<br>-vec-report0 -r8</td><td>-O2 -g<br>-traceback<br>-vec-report0 -r8<br>-vec-report0 -r8</td><td>-O2 -g<br>-traceback -r8<br>-fp-model strict<br>-fp-model strict<br>-xHost</td><td>-O2 -fp-model precise -xHost</td></tr><tr><td>C flags</td><td>-O2 -g<br>-traceback</td><td>-O2 -g<br>-traceback</td><td>-O2 -g<br>-traceback<br>-fp model strict -xHost</td><td>-O2 -fp-model precise -xHost<br>-g -traceback</td></tr><tr><td>LD flags</td><td>-O2 -g<br>-traceback</td><td>-O2 -g<br>-traceback</td><td>-O2 -g<br>-traceback<br>-fp-model strict<br>-xHost</td><td>-O2 -fp-model precise -xHost<br>-g -traceback</td></tr><tr><td>Output size</td><td>141.8 GB</td><td>141.6 GB</td><td>101.3 GB</td><td>101.3 GB</td></tr></tbody></table>

<figure><img src="image_134.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹,², Xavier Yepes-Arbós¹,², Eleftheria Exarchou¹,², and Francisco J. Doblas-Reyes¹,²,⁴

<figure><img src="image_126.png" /></figure>

## Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results?

A study involving eight institutions and seven different supercomputers in Europe is currently ongoing with EC-Earth. This ongoing study aims to do the following:

* evaluate different **computational environments** that are used in collaboration to produce CMIP6 experiments (can we safely create large ensembles composed of subsets that emanate from different partners of the consortium?);

* detect if the same **CMIP6 configuration** is replicable among platforms of the EC-Earth consortium (that is, can we safely exchange restarts with EC-Earth partners in order to initialize simulations and to avoid long spin-ups?); and

* systematically evaluate the impact of **different compilation flag options** (that is, what is the highest acceptable level of optimization that will not break the replicability of EC-Earth for a given environment?).

<figure><img src="image_127.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹,², Xavier Yepes-Arbós¹,², Eleftheria Exarchou¹,², and Francisco J. Doblas-Reyes¹,²,⁴

<figure><img src="image_135.png" /></figure>

## Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results?

A study involving eight institutions and seven different supercomputers in Europe is currently ongoing with EC-Earth. This ongoing study aims to do the following:

* evaluate different **computational environments** that are used in collaboration to produce CMIP6 experiments (can we safely create large ensembles composed of subsets that emanate from different partners of the consortium?);

* detect if the same **CMIP6 configuration** is replicable among platforms of the EC-Earth consortium (that is, can we safely exchange restarts with EC-Earth partners in order to initialize simulations and to avoid long spin-ups?); and

* systematically evaluate the impact of **different compilation flag options** (that is, what is the highest acceptable level of optimization that will not break the replicability of EC-Earth for a given environment?).

<figure><img src="image_136.png" /></figure>
# Should software version numbers determine science?

> PLoS One. 2012;7(6):e38234. doi: 10.1371/journal.pone.0038234. Epub 2012 Jun 1.

The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements

Ed H B M Gronenschild ¹, Petra Habets, Heidi I L Jacobs, Ron Mengelers, Nico Rozendaal, Jim van Os, Machteld Marcelis

<figure><img src="image_128.png" /></figure>

**see also** Krefting, D., Scheel, M., Freing, A., Specovius, S., Paul, F., and Brandt, A. (2011). "Reliability of quantitative neuroimage analysis using freesurfer in distributed environments," in *MICCAI Workshop on High-Performance and Distributed Computing for Medical Imaging*. (Toronto, ON)

Significant differences were revealed between FreeSurfer version v5.0.0 and the two earlier **versions**. [...] About a factor two smaller differences were detected between Macintosh and Hewlett-Packard **workstations** and between **OSX 10.5 and OSX 10.6**. The observed differences are similar in magnitude as effect sizes reported in **accuracy** evaluations and neurodegenerative-studies.
# Should software version numbers determine science?

> PLoS One. 2012;7(6):e38234. doi: 10.1371/journal.pone.0038234. Epub 2012 Jun 1.

The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements

Ed H B M Gronenschild ¹, Petra Habets, Heidi I L Jacobs, Ron Mengelers, Nico Rozendaal, Jim van Os, Machteld Marcelis

Significant differences were revealed between FreeSurfer version v5.0.0 and the two earlier **versions**. [...] About a factor two smaller differences were detected between Macintosh and Hewlett-Packard **workstations** and between **OSX 10.5 and OSX 10.6**. The observed differences are similar in magnitude as effect sizes reported in **accuracy** evaluations and neurodegenerative-studies.

<figure><img src="image_137.png" /></figure>

**see also** Krefting, D., Scheel, M., Freing, A., Specovius, S., Paul, F., and Brandt, A. (2011). “Reliability of quantitative neuroimage analysis using freesurfer in distributed environments,” in *MICCAI Workshop on High-Performance and Distributed Computing for Medical Imaging*. (Toronto, ON).
“Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed.”

Reproducibility of neuroimaging analyses across operating systems, Glatard et al., Front. Neuroinform., 24 April 2015

<figure><img src="image_129.png" /></figure>

The implementation of mathematical functions manipulating single-precision **floating-point numbers in libmath** has evolved during the last years, leading to numerical differences in computational results. While these differences have little or no impact on simple analysis pipelines such as brain extraction and cortical tissue classification, their **accumulation** creates **important differences in longer pipelines** such as the subcortical tissue classification, RSfMRI analysis, and cortical thickness extraction.

<table><thead><tr><th></th><th>Cluster A</th><th>Cluster B</th></tr></thead><tbody><tr><td>Applications</td><td>Freesurfer 5.3.0, build 1<br>FSL 5.0.6, build 1<br>CIVET 1.1.12-UCSF, build 1</td><td>Freesurfer 5.3.0, build 1 and 2<br>FSL 5.0.6, build 1 and 2<br>CIVET 1.1.12-UCSF, build 1</td></tr><tr><td>Interpreters</td><td>Python 2.4.3, bash 3.2.25,<br>Perl 5.8.8, tcsh 6.14.00</td><td>Python 2.7.5, bash 4.2.47,<br>Perl 5.18.2, tcsh 6.18.01</td></tr><tr><td>glibc version</td><td>2.5</td><td>2.18</td></tr><tr><td>OS</td><td>CentOS 5.10</td><td>Fedora 20</td></tr><tr><td>Hardware</td><td>x86_64 CPUs (Intel Xeon)</td><td>x86_64 CPUs (Intel Xeon)</td></tr></tbody></table>
> “Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed.”

Reproducibility of neuroimaging analyses across operating systems, Glatard et al., Front. Neuroinform., 24 April 2015

<figure><img src="image_138.png" /></figure>

The implementation of mathematical functions manipulating single-precision **floating-point numbers in libmath** has evolved during the last years, leading to numerical differences in computational results. While these differences have little or no impact on simple analysis pipelines such as brain extraction and cortical tissue classification, their **accumulation** creates **important differences in longer pipelines** such as the subcortical tissue classification, RSfMRI analysis, and cortical thickness extraction.

<table><thead><tr><th></th><th>Cluster A</th><th>Cluster B</th></tr></thead><tbody><tr><th>Applications</th><td>Freesurfer 5.3.0, build 1<br>FSL 5.0.6, build 1<br>CIVET 1.1.12-UCSF, build 1</td><td>Freesurfer 5.3.0, build 1 and 2<br>FSL 5.0.6, build 1 and 2<br>CIVET 1.1.12-UCSF, build 1</td></tr><tr><th>Interpreters</th><td>Python 2.4.3, bash 3.2.25,<br>Perl 5.8.8, tcsh 6.14.00</td><td>Python 2.7.5, bash 4.2.47,<br>Perl 5.18.2, tcsh 6.18.01</td></tr><tr><th>glibc version</th><td>2.5</td><td>2.18</td></tr><tr><th>OS</th><td>CentOS 5.10</td><td>Fedora 20</td></tr><tr><th>Hardware</th><td>x86_64 CPUs (Intel Xeon)</td><td>x86_64 CPUs (Intel Xeon)</td></tr></tbody></table>
> “Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed.”

Reproducibility of neuroimaging analyses across operating systems, Glatard et al., Front. Neuroinform., 24 April 2015

Statically building programs improves reproducibility across OSes, but small differences may still remain when dynamic libraries are loaded by static executables[...]. When static builds are not an option, software heterogeneity might be addressed using virtual machines. However, such solutions are only workarounds: differences may still arise between **static executables built on different OSes**, or between **dynamic executables executed in different VMs**.

<figure><img src="image_130.png" /></figure>

<table><thead><tr><th></th><th>Cluster A</th><th>Cluster B</th></tr></thead><tbody><tr><td>Applications</td><td>Freesurfer 5.3.0, build 1<br>FSL 5.0.6, build 1<br>CIVET 1.1.12-UCSF, build 1</td><td>Freesurfer 5.3.0, build 1 and 2<br>FSL 5.0.6, build 1 and 2<br>CIVET 1.1.12-UCSF, build 1</td></tr><tr><td>Interpreters</td><td>Python 2.4.3, bash 3.2.25,<br>Perl 5.8.8, tcsh 6.14.00</td><td>Python 2.7.5, bash 4.2.47,<br>Perl 5.18.2, tcsh 6.18.01</td></tr><tr><td>glibc version</td><td>2.5</td><td>2.18</td></tr><tr><td>OS</td><td>CentOS 5.10</td><td>Fedora 20</td></tr><tr><td>Hardware</td><td>x86_64 CPUs (Intel Xeon)</td><td>x86_64 CPUs (Intel Xeon)</td></tr></tbody></table>
> “Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed.”

Reproducibility of neuroimaging analyses across operating systems, Glatard et al., Front. Neuroinform., 24 April 2015

Statically building programs improves reproducibility across OSes, but small differences may still remain when dynamic libraries are loaded by static executables[...]. When static builds are not an option, software heterogeneity might be addressed using virtual machines. However, such solutions are only workarounds: differences may still arise between **static executables built on different OSes**, or between **dynamic executables executed in different VMs**.

<figure><img src="image_139.png" /></figure>

<table><thead><tr><th></th><th>Cluster A</th><th>Cluster B</th></tr></thead><tbody><tr><th scope="row">Applications</th><td>Freesurfer 5.3.0, build 1<br>FSL 5.0.6, build 1<br>CIVET 1.1.12-UCSF, build 1</td><td>Freesurfer 5.3.0, build 1 and 2<br>FSL 5.0.6, build 1 and 2<br>CIVET 1.1.12-UCSF, build 1</td></tr><tr><th scope="row">Interpreters</th><td>Python 2.4.3, bash 3.2.25,<br>Perl 5.8.8, tcsh 6.14.00</td><td>Python 2.7.5, bash 4.2.47,<br>Perl 5.18.2, tcsh 6.18.01</td></tr><tr><th scope="row">glibc version</th><td>2.5</td><td>2.18</td></tr><tr><th scope="row">OS</th><td>CentOS 5.10</td><td>Fedora 20</td></tr><tr><th scope="row">Hardware</th><td>x86_64 CPUs (Intel Xeon)</td><td>x86_64 CPUs (Intel Xeon)</td></tr></tbody></table>
# Reproducible science as a (deep) software variability problem

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

Despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results.

<figure><img src="image_131.png" /></figure>

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses

<figure><img src="image_132.png" /></figure>

Climate is changing

<figure><img src="image_133.png" /></figure>

<figure><img src="image_134.png" /></figure>

The sea level is rising

<figure><img src="image_135.png" /></figure>

<figure><img src="image_136.png" /></figure>
# Reproducible science as a (deep) software variability problem

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

Despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results.

<figure><img src="image_140.png" /></figure>

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses

<figure><img src="image_141.png" /></figure>

Climate is changing

<figure><img src="image_142.png" /></figure>

<figure><img src="image_143.png" /></figure>

The sea level is rising

<figure><img src="image_144.png" /></figure>

<figure><img src="image_145.png" /></figure>
deep software variability

software application variability

input data variability

build variability

compiler variability

container variability

hypervisor variability

operating system variability

hardware variability

Despite the availability of data and code, several studies report that the same data analyzed with **different software** can lead to **different results**

Many **layers** (operating system, third-party libraries, versions, workloads, compile-time options and flags, etc.) themselves subject to variability can alter the results.

**Reproducible science and deep software variability:** a threat and opportunity for scientific knowledge!

version variability
deep software variability

software application variability

input data variability

build variability

compiler variability

container variability

hypervisor variability

operating system variability

hardware variability

Despite the availability of data and code, several studies report that the same data analyzed with **different software** can lead to **different results**

Many **layers** (operating system, third-party libraries, versions, workloads, compile-time options and flags, etc.) themselves subject to variability can alter the results.

**Reproducible science and deep software variability:** a threat and opportunity for scientific knowledge!

version variability
How often $(x+y)+z == x+(y+z)$?

<figure><img src="image_137.png" /><figcaption>Figure 2: Feature model (excerpt). Inverse (resp. Relation-WithPi) corresponds to checking the property $(x*z)/(y*z) = x/y$ (resp. $(x*z*\pi)/(y*z*\pi) = x/y$) with $z, y \neq 0$</figcaption></figure>

<table><thead><tr><th>Parameters,<br>Input Data<br>Programming<br>Style</th><th>e.g., random seed selection</th></tr></thead><tbody><tr><td>Language</td><td>python<sup>1</sup><br>LLVM</td></tr><tr><td>Compiler & VM</td><td>GCC<br>NumPy</td></tr><tr><td>Library</td><td>blas<sup>2</sup><br>PETSc</td></tr><tr><td>Platform</td><td>intel<sup>3</sup><br>Mac OS</td></tr><tr><td>Processor</td><td>AMD<sup>4</sup><br>RISC-V</td></tr><tr><td>Micro-<br>architecture</td><td>Inner state of</td></tr></tbody></table>

<table><thead><tr><th>Language</th><th>Library</th><th>System</th><th>Compiler</th><th>Variableity</th><th>EqualityCheck</th><th>NumberGenerations</th><th>Repeat</th><th>min</th><th>max</th><th>std</th><th>mean</th></tr></thead><tbody><tr><td>Perl</td><td></td><td></td><td></td><td>seed none</td><td>ASSOCATMTV</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed none</td><td>MULT_JNV</td><td>100</td><td>10</td><td>60.0</td><td>71.0</td><td>0.0</td><td>3,562026262111375</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed none</td><td>MULT_JNV_PI</td><td>100</td><td>10</td><td>60.0</td><td>65.0</td><td>0.0</td><td>3,5016151615069943</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCATMTV</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_JNV</td><td>100</td><td>10</td><td>62.0</td><td>62.0</td><td>0.0</td><td>62.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_JNV_PI</td><td>100</td><td>10</td><td>47.0</td><td>47.0</td><td>0.0</td><td>47.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed none</td><td>associativity</td><td>100</td><td>10</td><td>71.0</td><td>82.0</td><td>0.0</td><td>3,3664041901363023</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed none</td><td>mult-inverse</td><td>100</td><td>10</td><td>58.0</td><td>78.0</td><td>0.0</td><td>68.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed none</td><td>mult-inverse</td><td>100</td><td>10</td><td>42.0</td><td>64.0</td><td>0.0</td><td>5,80557537624865</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>associativity</td><td>100</td><td>10</td><td>81.0</td><td>81.0</td><td>0.0</td><td>81.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse</td><td>100</td><td>10</td><td>70.0</td><td>75.0</td><td>0.0</td><td>70.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse</td><td>100</td><td>10</td><td>56.0</td><td>56.0</td><td>0.0</td><td>56.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed none</td><td>ASSOCATMTV</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed none</td><td>MULT_JNV</td><td>100</td><td>10</td><td>62.0</td><td>72.0</td><td>0.0</td><td>2,74045498217051</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed none</td><td>MULT_JNV_PI</td><td>100</td><td>10</td><td>47.0</td><td>57.0</td><td>0.0</td><td>2,80091438037628</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCATMTV</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_JNV</td><td>100</td><td>10</td><td>67.0</td><td>67.0</td><td>0.0</td><td>67.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_JNV_PI</td><td>100</td><td>10</td><td>53.0</td><td>53.0</td><td>0.0</td><td>53.0</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed none strict-equality</td><td>ASSOCATMTV</td><td>100</td><td>10</td><td>74.0</td><td>90.0</td><td>0.0</td><td>4,609772228646465</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed none strict-equality</td><td>MULT_JNV</td><td>100</td><td>10</td><td>60.0</td><td>79.0</td><td>0.0</td><td>6,16795737624865</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed none strict-equality</td><td>MULT_JNV_PI</td><td>100</td><td>10</td><td>49.0</td><td>59.0</td><td>0.0</td><td>2,8301943396160913</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>ASSOCATMTV</td><td>100</td><td>10</td><td>89.0</td><td>89.0</td><td>0.0</td><td>89.0</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_JNV</td><td>100</td><td>10</td><td>73.0</td><td>73.0</td><td>0.0</td><td>73.0</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_JNV_PI</td><td>100</td><td>10</td><td>55.0</td><td>55.0</td><td>0.0</td><td>55.0</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed none approximate equality of julia rang</td><td>ASSOCATMTV</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed none approximate equality of julia rang</td><td>MULT_JNV</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr></tbody></table>

<figure><img src="image_138.png" /></figure>
How often $(x+y)+z == x+(y+z)$?

<figure><img src="image_146.png" /><figcaption>Figure 2: Feature model (excerpt). Inverse (resp. Relation-WithPi) corresponds to checking the property $(x*z)/(y*z) = x/y$ (resp. $(x*z*\pi)/(y*z*\pi) = x/y$) with $z, y \neq 0$</figcaption></figure>

<table><thead><tr><th>Parameters,<br>Input Data<br>Programming<br>Style</th><th>e.g., random seed selection</th></tr></thead><tbody><tr><td>Language</td><td>python®<br>LLVM</td></tr><tr><td>Compiler & VM</td><td>GCC<br>jvm</td></tr><tr><td>Library</td><td>NumPy<br>blas</td></tr><tr><td>Platform</td><td>Intel®<br>OS X</td></tr><tr><td>Processor</td><td>AMD®<br>RISC-V</td></tr><tr><td>Micro-<br>architecture</td><td>Inner state of</td></tr></tbody></table>

<table><thead><tr><th>Language</th><th>Library</th><th>System</th><th>Compiler</th><th>VariableType</th><th>EqualityCheck</th><th>NumberGenerations</th><th>Repeat</th><th>min</th><th>max</th><th>std</th><th>mean</th></tr></thead><tbody><tr><td>Perl</td><td></td><td></td><td></td><td>seed none</td><td>ASSOCATMTVY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed none</td><td>MULT_JNV</td><td>100</td><td>10</td><td>60.0</td><td>71.0</td><td>6.0</td><td>3,5620262626111375</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed none</td><td>MULT_JNV PI</td><td>100</td><td>10</td><td>60.0</td><td>65.0</td><td>6.0</td><td>3,5016151615069943</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCATMTVY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_JNV</td><td>100</td><td>10</td><td>62.0</td><td>62.0</td><td>0.0</td><td>62.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_JNV PI</td><td>100</td><td>10</td><td>47.0</td><td>47.0</td><td>0.0</td><td>47.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed none</td><td>associativity</td><td>100</td><td>10</td><td>71.0</td><td>82.0</td><td>6.0</td><td>3,3666401901363023</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed none</td><td>mult-inverse</td><td>100</td><td>10</td><td>58.0</td><td>78.0</td><td>6.0</td><td>68.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed none</td><td>mult-inverse ge</td><td>100</td><td>10</td><td>42.0</td><td>64.0</td><td>6.0</td><td>5,805575376724865</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>associativity</td><td>100</td><td>10</td><td>81.0</td><td>81.0</td><td>0.0</td><td>81.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse</td><td>100</td><td>10</td><td>70.0</td><td>75.0</td><td>0.0</td><td>70.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse ge</td><td>100</td><td>10</td><td>56.0</td><td>56.0</td><td>0.0</td><td>56.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed none</td><td>ASSOCATMTVY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed none</td><td>MULT_JNV</td><td>100</td><td>10</td><td>62.0</td><td>72.0</td><td>6.0</td><td>2,74045498217051</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed none</td><td>MULT_JNV PI</td><td>100</td><td>10</td><td>47.0</td><td>57.0</td><td>6.0</td><td>2,80091438037628</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCATMTVY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_JNV</td><td>100</td><td>10</td><td>67.0</td><td>67.0</td><td>0.0</td><td>67.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_JNV PI</td><td>100</td><td>10</td><td>53.0</td><td>53.0</td><td>0.0</td><td>53.0</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed none strict-equality</td><td>ASSOCATMTVY</td><td>100</td><td>10</td><td>74.0</td><td>90.0</td><td>0.0</td><td>4,809772228646465</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed none strict-equality</td><td>MULT_JNV</td><td>100</td><td>10</td><td>60.0</td><td>79.0</td><td>6.0</td><td>6,167957376724865</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed none strict-equality</td><td>MULT_JNV PI</td><td>100</td><td>10</td><td>49.0</td><td>59.0</td><td>0.0</td><td>2,8301943396160913</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>ASSOCATMTVY</td><td>100</td><td>10</td><td>89.0</td><td>89.0</td><td>0.0</td><td>89.0</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_JNV</td><td>100</td><td>10</td><td>73.0</td><td>73.0</td><td>0.0</td><td>73.0</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_JNV PI</td><td>100</td><td>10</td><td>55.0</td><td>55.0</td><td>0.0</td><td>55.0</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed none approximate equality of julia rang</td><td>ASSOCATMTVY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>julia</td><td></td><td></td><td></td><td>seed none approximate equality of julia rang</td><td>MULT_JNV</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr></tbody></table>

<figure><img src="image_147.png" /></figure>
# AGENDA

Frictionless Reproducibility and (Deep) Software (Variability)

**Problem (cont'd): Variability and Frictions**

Solution: Variability and Exploration

Discussions
# AGENDA

## Frictionless Reproducibility and (Deep) Software (Variability)

### Problem (cont'd): Variability and Frictions

Solution: Variability and Exploration

Discussions
<figure><img src="image_139.png" /></figure>

OBS
Open Broadcaster Software®

<figure><img src="image_140.png" /></figure>

<figure><img src="image_141.png" /></figure>

## deep software variability

hardware variability

15,000+ options

thousands of compiler
flags and compile-time
options
dozens of
preferences

100+ command-line
parameters

1000+ feature toggles

38

Non-functional properties
<figure><img src="image_148.png" /></figure>

OBS
Open Broadcaster Software®

<figure><img src="image_149.png" /></figure>

<figure><img src="image_150.png" /></figure>

## deep software variability

- hardware variability

- 15,000+ options

- thousands of compiler flags and compile-time options
dozens of preferences

- 100+ command-line parameters

- 1000+ feature toggles

38

## Non-functional properties
<figure><img src="image_142.png" /></figure>

OBS
Open Broadcaster Software*

<figure><img src="image_143.png" /></figure>

<figure><img src="image_144.png" /></figure>

hardware variability

15,000+ options

thousands of compiler flags
and compile-time options

<figure><img src="image_145.png" /></figure>

dozens of preferences

<figure><img src="image_146.png" /></figure>

CODE

input data

100+ command-line parameters

1000+ feature toggles

deep software variability

System under
Study
(reproducible
and
replicable)

Variability
Output

(scientific result;
most of the time
**quantitative**
information)

<figure><img src="image_147.png" /></figure>
<figure><img src="image_151.png" /></figure>

hardware variability

15,000+ options

<figure><img src="image_152.png" /></figure>

264

<figure><img src="image_153.png" /></figure>

thousands of compiler flags
and compile-time options

<figure><img src="image_154.png" /></figure>

dozens of preferences

<figure><img src="image_155.png" /></figure>

CODE

input data

100+ command-line parameters

1000+ feature toggles

deep software variability

System under
Study
(reproducible
and
replicable)

Variability
Output

(scientific result;
most of the time
**quantitative**
information)

<figure><img src="image_156.png" /></figure>

performance
metric
DEEP VARIABILITY

HARDWARE

<figure><img src="image_148.png" /></figure>

AGE

<figure><img src="image_149.png" /></figure>

# CORES

<figure><img src="image_150.png" /></figure>

GPU

OPERATING
SYSTEM

<figure><img src="image_151.png" /></figure>

VERSION

<figure><img src="image_152.png" /></figure>

OPTION

<figure><img src="image_153.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_154.png" /></figure>

COMPIL.

SOFTWARE
VARIANT

<figure><img src="image_155.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_156.png" /></figure>

SIZE

<figure><img src="image_157.png" /></figure>

LENGTH

<figure><img src="image_158.png" /></figure>

RES.

<figure><img src="image_159.png" /></figure>

<figure><img src="image_160.png" /></figure>

<figure><img src="image_161.png" /></figure>
DEEP VARIABILITY

HARDWARE

<figure><img src="image_157.png" /></figure>

AGE

<figure><img src="image_158.png" /></figure>

# CORES

<figure><img src="image_159.png" /></figure>

GPU

OPERATING
SYSTEM

<figure><img src="image_160.png" /></figure>

VERSION

<figure><img src="image_161.png" /></figure>

OPTION

<figure><img src="image_162.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_163.png" /></figure>

COMPIL.

SOFTWARE
VARIANT

<figure><img src="image_164.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_165.png" /></figure>

SIZE

<figure><img src="image_166.png" /></figure>

LENGTH

<figure><img src="image_167.png" /></figure>

RES.

<figure><img src="image_168.png" /></figure>

<figure><img src="image_169.png" /></figure>

<figure><img src="image_170.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹,², Xavier Yepes-Arbós²,⁵
Eleftheria Exarchou¹,², and Francisco J. Doblas-Reyes¹,²,⁴

<figure><img src="image_162.png" /></figure>

Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? Using two versions of EC-Earth: one “non-replicable” case (see below) and one replicable case.

<table><caption>**Table 2.** The four experiments considered in this study.</caption><thead><tr><th>Experiment ID</th><th>e011</th><th>m06e</th><th>a0gi</th><th>a0go</th></tr></thead><tbody><tr><td>Computing environment</td><td>ECMWF-CCA</td><td>MareNostrum3</td><td>ECMWF-CCA</td><td>MareNostrum3</td></tr><tr><td>EC-Earth version</td><td>3.1</td><td>3.1</td><td>3.2</td><td>3.2</td></tr><tr><td>Processors (IFS+NEMO+OASIS)</td><td>598<br>(480 + 96 + 22)</td><td>512<br>(384 + 96 + 22)</td><td>432 (288 + 144)<br>(OASIS: library)</td><td>416 (288 + 128)<br>(OASIS: library)</td></tr><tr><td rowspan="4">F flags</td><td>-O2-g</td><td>-O2-g</td><td>-O2-g</td><td>-O2-fp-model</td></tr><tr><td>-traceback</td><td>-traceback</td><td>-traceback -r8</td><td>precise -xHost</td></tr><tr><td>-vec-report0 -r8</td><td>-vec-report0 -r8</td><td>-fp-model strict</td><td>-g -traceback</td></tr><tr><td>-vec-report0 -r8</td><td>-vec-report0 -r8</td><td>-fp-model strict<br>-xHost</td><td>-g -traceback<br>-r8</td></tr><tr><td rowspan="2">C flags</td><td>-O2-g</td><td>-O2-g</td><td>-O2-g</td><td>-O2-fp-model</td></tr><tr><td>-traceback</td><td>-traceback</td><td>-traceback</td><td>precise -xHost</td></tr><tr><td rowspan="2">LD flags</td><td>-O2-g</td><td>-O2-g</td><td>-fp-model<br>strict -xHost</td><td>-g -traceback</td></tr><tr><td>-traceback</td><td>-traceback</td><td>-traceback</td><td>-fp-model strict<br>-xHost</td></tr><tr><td>Output size</td><td>141.8 GB</td><td>141.6 GB</td><td>101.3 GB</td><td>101.3 GB</td></tr></tbody></table>

<figure><img src="image_163.png" /><figcaption>**Table 1.** The two computing environments considered in this study.</figcaption></figure>

<table><tbody><tr><td>Computing environment</td><td>ECMWF-CCA</td><td>MareNostrum3</td></tr><tr><td>Location</td><td>Reading, UK</td><td>Barcelona, Spain</td></tr><tr><td>Motherboard</td><td>Cray XC30 system</td><td>IBM dx360 M4</td></tr><tr><td>Processor</td><td>Dual 12-core E5-2697 v2<br>(Ivy Bridge) series<br>processors (2.7 GHz), 24<br>cores per node</td><td>2x Intel SandyBridge-EP<br>E5-2670/1600 20M 8-core<br>at 2.6GHz, 16 cores<br>per node</td></tr><tr><td>Operating system</td><td>Cray Linux Environment<br>(CLE) 5.2</td><td>Linux - SuSe distribution<br>11 SP2</td></tr><tr><td>Compiler</td><td>Intel(R) 64 Compiler XE<br>for applications running<br>on Intel(R) 64, version<br>14.0.1.106 build 20131008</td><td>Intel(R) 64 Compiler XE<br>for applications running<br>on Intel(R) 64, version<br>13.0.1.117 build 20121010</td></tr><tr><td>MPI version</td><td>Cray mpich2 v6.2.0</td><td>Intel MPI v4.1.3.049</td></tr><tr><td>LAPACK version</td><td>Cray libsci v12.2.0</td><td>Intel MKL v11.0.1</td></tr><tr><td>SZIP, HDF5, NetCDF4</td><td>v2.1, v1.8.11, v4.3.0</td><td>v2.1, v1.8.14, v4.2</td></tr><tr><td>GribAPI, GribEX</td><td>v1.13.0, v000395</td><td>v1.14.0, v000370</td></tr></tbody></table>

<figure><img src="image_164.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹,², Xavier Yepes-Arbós²,⁵
Eleftheria Exarchou¹,², and Francisco J. Doblas-Reyes¹,²,⁴

<figure><img src="image_171.png" /></figure>

Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? Using two versions of EC-Earth: one “non-replicable” case (see below) and one replicable case.

<table><caption>Table 2. The four experiments considered in this study.</caption><thead><tr><th>Experiment ID</th><th>e011</th><th>m06e</th><th>a0gi</th><th>a0go</th></tr></thead><tbody><tr><td>Computing environment</td><td>ECMWF-CCA</td><td>MareNostrum3</td><td>ECMWF-CCA</td><td>MareNostrum3</td></tr><tr><td>EC-Earth version</td><td>3.1</td><td>3.1</td><td>3.2</td><td>3.2</td></tr><tr><td>Processors (IFS+NEMO+OASIS)</td><td>598<br>(480 + 96 + 22)</td><td>512<br>(384 + 96 + 22)</td><td>432 (288 + 144)<br>(OASIS: library)</td><td>416 (288 + 128)<br>(OASIS: library)</td></tr><tr><td>F flags</td><td>-02-g<br>-traceback<br>-vec-report0 -r8<br>-vec-report0 -r8</td><td>-02-g<br>-traceback<br>-vec-report0 -r8<br>-vec-report0 -r8</td><td>-02-g<br>-traceback -r8<br>-fp-model strict<br>-fp-model strict<br>-xHost</td><td>-02-fp-model<br>precise -xHost<br>-g -traceback<br>-r8</td></tr><tr><td>C flags</td><td>-02-g<br>-traceback</td><td>-02-g<br>-traceback</td><td>-02-g<br>-traceback<br>-fp-model strict<br>-xHost</td><td>-02-fp-model<br>precise -xHost<br>-g -traceback</td></tr><tr><td>LD flags</td><td>-02-g<br>-traceback</td><td>-02-g<br>-traceback</td><td>-02-g<br>-traceback<br>-fp-model strict<br>-xHost</td><td>-02-fp-model<br>precise -xHost<br>-g -traceback</td></tr><tr><td>Output size</td><td>141.8 GB</td><td>141.6 GB</td><td>101.3 GB</td><td>101.3 GB</td></tr></tbody></table>

<figure><img src="image_172.png" /><figcaption>Table 1. The two computing environments considered in this study.</figcaption></figure>

<table><tbody><tr><td>Computing environment</td><td>ECMWF-CCA</td><td>MareNostrum3</td></tr><tr><td>Location</td><td>Reading, UK</td><td>Barcelona, Spain</td></tr><tr><td>Motherboard</td><td>Cray XC30 system</td><td>IBM dx360 M4</td></tr><tr><td>Processor</td><td>Dual 12-core E5-2697 v2<br>(Ivy Bridge) series<br>processors (2.7 GHz), 24<br>cores per node</td><td>2x Intel SandyBridge-EP<br>E5-2670/1600 20M 8-core<br>at 2.6GHz, 16 cores<br>per node</td></tr><tr><td>Operating system</td><td>Cray Linux Environment<br>(CLE) 5.2</td><td>Linux - SuSe distribution<br>11 SP2</td></tr><tr><td>Compiler</td><td>Intel(R) 64 Compiler XE<br>for applications running<br>on Intel(R) 64, version<br>14.0.1.106 build 20131008</td><td>Intel(R) 64 Compiler XE<br>for applications running<br>on Intel(R) 64, version<br>13.0.1.117 build 20121010</td></tr><tr><td>MPI version</td><td>Cray mpich2 v6.2.0</td><td>Intel MPI v4.1.3.049</td></tr><tr><td>LAPACK version</td><td>Cray libsci v12.2.0</td><td>Intel MKL v11.0.1</td></tr><tr><td>SZIP, HDF5, NetCDF4</td><td>v2.1, v1.8.11, v4.3.0</td><td>v2.1, v1.8.14, v4.2</td></tr><tr><td>GribAPI, GribEX</td><td>v1.13.0, v000395</td><td>v1.14.0, v000370</td></tr></tbody></table>

<figure><img src="image_173.png" /></figure>
<table><thead><tr><th>Explanatory variable</th><th>Meaning</th></tr></thead><tbody><tr><td>entcoef</td><td>Entrainment coefficient</td></tr><tr><td>ct</td><td>Accretion constant</td></tr><tr><td>rhcrit</td><td>Critical relative humidity</td></tr><tr><td>vfl</td><td>Ice fall speed through clouds</td></tr><tr><td>eacf</td><td>Empirically adjusted cloud fraction</td></tr><tr><td>cw</td><td>Threshold for precipitation</td></tr><tr><td>dtice</td><td>Temperature range of ice albedo variation</td></tr><tr><td>ice</td><td>Nonspherical ice</td></tr><tr><td>midware</td><td>Client middleware</td></tr><tr><td>ice_size</td><td>Ice particle size</td></tr><tr><td>alpham</td><td>Albedo at melting point of ice</td></tr><tr><td>processor_name</td><td>CPU classification</td></tr><tr><td>clock_classic</td><td>Processor clock speed recorded under classic middleware</td></tr><tr><td>ram_size</td><td>Hardware RAM</td></tr><tr><td>clock_boinc_i</td><td>Integer processor clock speed recorded under BOINC middleware</td></tr><tr><td>clock_boinc_f</td><td>Floating point processor clock speed recorded under BOINC middleware</td></tr><tr><td>os_name</td><td>Operating system</td></tr><tr><td>dtheta</td><td>Perturbations to initial conditions on a given level</td></tr></tbody></table>

Association of parameter, software, and hardware variation with large-scale behavior across 57,000 climate models

Christopher G. Knight, Sylvia H. E. Knight, Neil Massey, Tolu Aina, Carl Christensen, Dave J. F...

+ See all authors and affiliations

PNAS July 24, 2007 104 (30) 12259-12264; https://doi.org/10.1073/pnas.0608144104

We demonstrate that **effects of parameter, hardware, and software variation are detectable, complex, and interacting**. However, we find most of the effects of parameter variation are caused by a small subset of parameters. Notably, the entrainment coefficient in clouds is associated with 30% of the variation seen in climate sensitivity, although both low and high values can give high climate sensitivity. **We demonstrate that the effect of hardware and software is small relative to the effect of parameter variation** and, over the wide range of systems tested, may be treated as equivalent to that caused by changes in initial conditions.

<figure><img src="image_165.png" /></figure>

57,067 climate model runs. These runs sample parameter space for 10 parameters with between two and four levels of each, covering 12,487 parameter combinations (24% of possible combinations) and a range of initial conditions
<table><thead><tr><th>Explanatory variable</th><th>Meaning</th></tr></thead><tbody><tr><td>entcoef</td><td>Entrainment coefficient</td></tr><tr><td>ct</td><td>Accretion constant</td></tr><tr><td>rhcrit</td><td>Critical relative humidity</td></tr><tr><td>vfl</td><td>Ice fall speed through clouds</td></tr><tr><td>eacf</td><td>Empirically adjusted cloud fraction</td></tr><tr><td>cw</td><td>Threshold for precipitation</td></tr><tr><td>dtice</td><td>Temperature range of ice albedo variation</td></tr><tr><td>ice</td><td>Nonspherical ice</td></tr><tr><td>midware</td><td>Client middleware</td></tr><tr><td>ice_size</td><td>Ice particle size</td></tr><tr><td>alpham</td><td>Albedo at melting point of ice</td></tr><tr><td>processor_name</td><td>CPU classification</td></tr><tr><td>clock_classic</td><td>Processor clock speed recorded under classic middleware</td></tr><tr><td>ram_size</td><td>Hardware RAM</td></tr><tr><td>clock_boinc_i</td><td>Integer processor clock speed recorded under BOINC middleware</td></tr><tr><td>clock_boinc_f</td><td>Floating point processor clock speed recorded under BOINC middleware</td></tr><tr><td>os_name</td><td>Operating system</td></tr><tr><td>dtheta</td><td>Perturbations to initial conditions on a given level</td></tr></tbody></table>

Association of parameter, software, and hardware variation with large-scale behavior across 57,000 climate models

Christopher G. Knight, Sylvia H. E. Knight, Neil Massey, Tolu Aina, Carl Christensen, Dave J. F...

+ See all authors and affiliations

PNAS July 24, 2007 104 (30) 12259-12264; https://doi.org/10.1073/pnas.0608144104

We demonstrate that **effects of parameter, hardware, and software variation are detectable, complex, and interacting**. However, we find most of the effects of parameter variation are caused by a small subset of parameters. Notably, the entrainment coefficient in clouds is associated with 30% of the variation seen in climate sensitivity, although both low and high values can give high climate sensitivity. **We demonstrate that the effect of hardware and software is small relative to the effect of parameter variation** and, over the wide range of systems tested, may be treated as equivalent to that caused by changes in initial conditions.

<figure><img src="image_174.png" /></figure>

57,067 climate model runs. These runs sample parameter space for 10 parameters with between two and four levels of each, covering 12,487 parameter combinations (24% of possible combinations) and a range of initial conditions
Joelle Pineau "Building Reproducible, Reusable, and Robust Machine Learning Software" ICSE'19 keynote "[...] results can be brittle to even minor perturbations in the domain or experimental procedure"

Deep Reinforcement Learning that Matters

Peter Henderson¹*, Riashat Islam¹,²*, Philip Bachman²
Joelle Pineau¹, Doina Precup¹, David Meger¹

What is the magnitude of the effect
**hyperparameter** settings can have on baseline
performance?

How does the choice of **network architecture** for
the policy and value function approximation affect
performance?

How can the **reward scale** affect results?

Can **random seeds** drastically alter performance?

How do the **environment properties** affect
variability in reported RL algorithm performance?

Are commonly used baseline **implementations** comparable?

torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision

David Picard

LIGM, École des Ponts, 77455 Marnes la vallée, France

DAVID.PICARD@ENPC.FR

Abstract

In this paper I investigate the effect of random seed selection on the accuracy when using popular deep learning architectures for computer vision. I scan a large amount of seeds (up to 10⁴) on CIFAR 10 and I also scan fewer seeds on Imagenet using pre-trained models to investigate large scale datasets. The conclusions are that even if the variance is not very large, it is surprisingly easy to find an outlier that performs much better or much worse than the average.

<figure><img src="image_166.png" /></figure>
Deep Reinforcement Learning that Matters

Peter Henderson¹*, Riashat Islam¹,²*, Philip Bachman²
Joelle Pineau¹, Doina Precup¹, David Meger¹

What is the magnitude of the effect
**hyperparameter** settings can have on baseline
performance?

How does the choice of **network architecture** for
the policy and value function approximation affect
performance?

How can the **reward scale** affect results?

Can **random seeds** drastically alter performance?

How do the **environment properties** affect
variability in reported RL algorithm performance?

Are commonly used baseline **implementations**
comparable?

torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision

David Picard

LIGM, École des Ponts, 77455 Marnes la vallée, France

DAVID.PICARD@ENPC.FR

Abstract

In this paper I investigate the effect of random seed selection on the accuracy when using popular deep learning architectures for computer vision. I scan a large amount of seeds (up to 10⁴) on CIFAR 10 and I also scan fewer seeds on Imagenet using pre-trained models to investigate large scale datasets. The conclusions are that even if the variance is not very large, it is surprisingly easy to find an outlier that performs much better or much worse than the average.

<figure><img src="image_175.png" /></figure>
Reproducible and replicable CFD: it's harder than you think

Olivier Mesnard, Lorena A. Barba

Mechanical and Aerospace Engineering, George Washington University, Washington DC 20052

> "Completing a full replication study of our previously published findings on bluff-body aerodynamics was harder than we thought. Despite the fact that we have good reproducible-research practices, sharing our code and data openly."

**Story 1: Meshing and boundary conditions can ruin everything**

**Story 3: All linear algebra libraries are not created equal**

**Story 4: Different versions of your code, external libraries or even compilers may challenge reproducibility**

<figure><img src="image_167.png" /></figure>
Reproducible and replicable CFD: it's harder than you think

Olivier Mesnard, Lorena A. Barba

Mechanical and Aerospace Engineering, George Washington University, Washington DC 20052

> "Completing a full replication study of our previously published findings on bluff-body aerodynamics was harder than we thought. Despite the fact that we have good reproducible-research practices, sharing our code and data openly."

**Story 1: Meshing and boundary conditions can ruin everything**

**Story 3: All linear algebra libraries are not created equal**

**Story 4: Different versions of your code, external libraries or even compilers may challenge reproducibility**

<figure><img src="image_176.png" /></figure>
# Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, ... Tom Schonberg

+ Show authors

*Nature* **582**, 84–88 (2020) | Cite this article

42k Accesses

203 Citations

2056 Altmetric

Metrics

Data analysis workflows in many scientific domains have become increasingly complex and flexible (= subject to variability). Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology.

Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that **analytical flexibility can have substantial effects on scientific conclusions**, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to **analytical variability** are discussed.
# Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, ... Tom Schonberg

+ Show authors

*Nature* **582**, 84–88 (2020) | Cite this article

42k Accesses

203 Citations

2056 Altmetric

Metrics

Data analysis workflows in many scientific domains have become increasingly complex and flexible (= subject to variability). Here we assess the effect of this flexibility on the results of functional magnetic resonance imaging by asking 70 independent teams to analyse the same dataset, testing the same 9 ex-ante hypotheses. The flexibility of analytical approaches is exemplified by the fact that no two teams chose identical workflows to analyse the data. This flexibility resulted in sizeable variation in the results of hypothesis tests, even for teams whose statistical maps were highly correlated at intermediate stages of the analysis pipeline. Variation in reported results was related to several aspects of analysis methodology.

Notably, a meta-analytical approach that aggregated information across teams yielded a significant consensus in activated regions. Furthermore, prediction markets of researchers in the field revealed an overestimation of the likelihood of significant findings, even by researchers with direct knowledge of the dataset. Our findings show that **analytical flexibility can have substantial effects on scientific conclusions**, and identify factors that may be related to variability in the analysis of functional magnetic resonance imaging. The results emphasize the importance of validating and sharing complex analysis workflows, and demonstrate the need for performing and reporting multiple analyses of the same data. Potential approaches that could be used to mitigate issues related to **analytical variability** are discussed.
# Can Machine Learning Pipelines Be Better Configured? Wang et al. FSE'23

<figure><img src="image_168.png" /></figure>

> "A pipeline is subject to misconfiguration if it exhibits significantly inconsistent performance upon changes in the **versions** of its **configured** libraries or the combination of these libraries. We refer to such performance inconsistency as a pipeline configuration (PLC) issue."

<table><thead><tr><th>{Keras, Tensorflow}<br>Versions</th><th>Score<br>(AUC)</th><th>Ranking</th><th>Time<br>(ms)</th><th>Memory<br>(MB)</th></tr></thead><tbody><tr><td>{2.7.1, 2.7.0}</td><td>Crash</td><td>Crash</td><td>Crash</td><td>Crash</td></tr><tr><td>{2.4.3, 2.4.1}</td><td>0.768</td><td>522</td><td>1895.656</td><td>1244.446</td></tr><tr><td>{2.4.3, 2.3.1}</td><td>0.737</td><td>521</td><td>1882.099</td><td>1241.001</td></tr><tr><td>{2.4.3, 2.2.0}</td><td>0.559</td><td>523</td><td>1926.980</td><td>1248.518</td></tr><tr><td>{2.3.1, 2.4.1}</td><td>Crash</td><td>Crash</td><td>Crash</td><td>Crash</td></tr><tr><td>{2.3.1, 2.3.1}</td><td>Crash</td><td>Crash</td><td>Crash</td><td>Crash</td></tr><tr><td>{2.3.1, 2.2.0}</td><td>0.997</td><td>1</td><td>1877.330</td><td>1199.130</td></tr><tr><td>{2.3.1, 2.1.0}</td><td>0.997</td><td>1</td><td>1888.612</td><td>1202.602</td></tr><tr><td>{2.3.1, 2.0.0}</td><td>Crash</td><td>Crash</td><td>Crash</td><td>Crash</td></tr><tr><td>{2.3.1, 1.15.2}</td><td>0.997</td><td>1</td><td>1989.861</td><td>1194.425</td></tr><tr><td>{2.3.1, 1.14.0}</td><td>0.997</td><td>1</td><td>1853.423</td><td>1196.269</td></tr><tr><td>{2.3.1, 1.13.1}</td><td>0.997</td><td>1</td><td>1901.693</td><td>1183.123</td></tr></tbody></table>

In this paper, we empirically studied 11,363 ML pipelines from diverse competitions on KAGGLE to explore the impacts of different ML library version combinations on their performances. Our study reveals the pervasiveness and severity of PLC issues in ML pipelines. Our findings can motivate the establishment of a symbiotic ecosystem where researchers, tool builders, and library vendors work together to assist developers in combating PLC issues.
# Can Machine Learning Pipelines Be Better Configured? Wang et al. FSE'23

<figure><img src="image_177.png" /></figure>

> "A pipeline is subject to misconfiguration if it exhibits significantly inconsistent performance upon changes in the **versions** of its **configured** libraries or the combination of these libraries. We refer to such performance inconsistency as a pipeline configuration (PLC) issue."

<table><thead><tr><th>{Keras, Tensorflow}</th><th></th><th></th><th></th><th></th><th></th></tr><tr><th>Versions</th><th>Score (AUC)</th><th>Ranking</th><th>Time (ms)</th><th>Memory (MB)</th></tr></thead><tbody><tr><td>{2.7.1, 2.7.0}</td><td>Crash</td><td>Crash</td><td>Crash</td><td>Crash</td></tr><tr><td>{2.4.3, 2.4.1}</td><td>0.768</td><td>522</td><td>1895.656</td><td>1244.446</td></tr><tr><td>{2.4.3, 2.3.1}</td><td>0.737</td><td>521</td><td>1882.099</td><td>1241.001</td></tr><tr><td>{2.4.3, 2.2.0}</td><td>0.559</td><td>523</td><td>1926.980</td><td>1248.518</td></tr><tr><td>{2.3.1, 2.4.1}</td><td>Crash</td><td>Crash</td><td>Crash</td><td>Crash</td></tr><tr><td>{2.3.1, 2.3.1}</td><td>Crash</td><td>Crash</td><td>Crash</td><td>Crash</td></tr><tr><td>{2.3.1, 2.2.0}</td><td>0.997</td><td>1</td><td>1877.330</td><td>1199.130</td></tr><tr><td>{2.3.1, 2.1.0}</td><td>0.997</td><td>1</td><td>1888.612</td><td>1202.602</td></tr><tr><td>{2.3.1, 2.0.0}</td><td>Crash</td><td>Crash</td><td>Crash</td><td>Crash</td></tr><tr><td>{2.3.1, 1.15.2}</td><td>0.997</td><td>1</td><td>1989.861</td><td>1194.425</td></tr><tr><td>{2.3.1, 1.14.0}</td><td>0.997</td><td>1</td><td>1853.423</td><td>1196.269</td></tr><tr><td>{2.3.1, 1.13.1}</td><td>0.997</td><td>1</td><td>1901.693</td><td>1183.123</td></tr></tbody></table>

In this paper, we empirically studied 11,363 ML pipelines from diverse competitions on KAGGLE to explore the impacts of different ML library version combinations on their performances. Our study reveals the pervasiveness and severity of PLC issues in ML pipelines. Our findings can motivate the establishment of a symbiotic ecosystem where researchers, tool builders, and library vendors work together to assist developers in combating PLC issues.
# Deep software variability: Are layers/features orthogonal or are there interactions?

## Variability layers

<figure><img src="image_169.png" /></figure>

## 264 environment

### Hardware

* Age & usage
* Number of cores
* Processor

### Operating System

* Kernel options
* Linux distribution
* Dependencies

### Software

* x264 compilation
* x264 variant
* x264 version

### Input data

* Video category
* Video resolution
* Video quality

<figure><img src="image_170.png" /></figure>
# Deep software variability: Are layers/features orthogonal or are there interactions?

## Variability layers

<figure><img src="image_178.png" /></figure>

## 264 environment

### Hardware

Age & usage
Number of cores
Processor

### Operating System

Kernel options
Linux distribution
Dependencies

### Software

x264 compilation
x264 variant
x264 version

### Input data

Video category
Video resolution
Video quality

<figure><img src="image_179.png" /></figure>
<figure><img src="image_171.png" /></figure>

<figure><img src="image_172.png" /></figure>

<figure><img src="image_173.png" /></figure>

<figure><img src="image_174.png" /></figure>

<figure><img src="image_175.png" /></figure>

<figure><img src="image_176.png" /></figure>

480

Lossless:
x264 --qp 0 -o <output> <input>

Maximum PSNR at the cost of speed and visual quality:
x264 --preset placebo --tune psnr -o <output> <input>

Constant bitrate at 1000kbps with a 2 second-buffer:
x264 --vbv-bufsize 2000 --bitrate 1000 -o <output> <input>

Presets:

--profile <string> Force the limits of an H.264 profile
    Overrides all settings.
    - baseline, main, high, high10, high422

--preset <string> Use a preset to select encoding settings [m=medium,slower,veryslow,placebo]
    Overridden by user settings.
    - ultrafast,superfast,veryfast,faster,fast
    - medium,slow,slower,veryslow,placebo

--tune <string> Tune the settings for a particular type of source
    or situation
    Overridden by user settings.
    Multiple tunings are separated by commas.
    Only one psy tuning can be used at a time.

-I, --keyint <integer or "infinite"> Maximum GOP size [250]
  --tff Enable interlaced mode (top field first)
  --bff Enable interlaced
  --pulldown <string> Use soft pulldown

- none, 22, 32, 48

Ratecontrol:

-B, --bitrate <integer> Set bitrate (kbps)
  --crf <float> Quality-based VBR
  --vbv-maxrate <integer> Max local bitrate
  --vbv-bufsize <integer> Set size of the VBR buffer

-p, --pass <integer> Enable multipass
    - 1: First pass
    - 2: Last pass

HARDWARE

<figure><img src="image_177.png" /></figure>

AGE

<figure><img src="image_178.png" /></figure>

# CORES

<figure><img src="image_179.png" /></figure>

GPU

OPERATING
SYSTEM

<figure><img src="image_180.png" /></figure>

VERSION

<figure><img src="image_181.png" /></figure>

OPTION

<figure><img src="image_182.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_183.png" /></figure>

COMPIL

X264

VARIANT

<figure><img src="image_184.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_185.png" /></figure>

SIZE

LENGTH

<figure><img src="image_186.png" /></figure>

RES.
<figure><img src="image_180.png" /></figure>

<figure><img src="image_181.png" /></figure>

<figure><img src="image_182.png" /></figure>

<figure><img src="image_183.png" /></figure>

<figure><img src="image_184.png" /></figure>

<figure><img src="image_185.png" /></figure>

480

Lossless:
x264 --qp 0 -o <output> <input>

Maximum PSNR at the cost of speed and visual quality:
x264 --preset placebo --tune psnr -o <output> <input>

Constant bitrate at 1000kbps with a 2 second-buffer:
x264 --vbv-bufsize 2000 --bitrate 1000 -o <output> <input>

Presets:

--profile <string> Force the limits of an H.264 profile
    Overrides all settings.
    - baseline, main, high, high10, high422

--preset <string> Use a preset to select encoding settings [m = medium, slow = slow, veryslow = very slow, placebo]
    Overridden by user settings.
    - ultrafast, superfast, veryfast, faster, fast
    - medium, slow, slower, veryslow, placebo

--tune <string> Tune the settings for a particular type of source or situation
    Overridden by user settings.
    Multiple tunings are separated by commas.
    Only one psy tuning can be used at a time.

-I, --keyint <integer or "infinite"> Maximum GOP size [250]
  --tff Enable interlaced mode (top field first)
  --bff Enable interlaced
  --pulldown <string> Use soft pulldown

- none, 22, 32

Ratecontrol:

-B, --bitrate <integer> Set bitrate (kbps)
  --crf <float> Quality-based VBR
  --vbv-maxrate <integer> Max local bitrate
  --vbv-bufsize <integer> Set size of the VBR buffer

-p, --pass <integer> Enable multipass
    - 1: First pass
    - 2: Last pass

HARDWARE

<figure><img src="image_186.png" /></figure>

AGE

<figure><img src="image_187.png" /></figure>

# CORES

<figure><img src="image_188.png" /></figure>

GPU

OPERATING
SYSTEM

<figure><img src="image_189.png" /></figure>

VERSION

<figure><img src="image_190.png" /></figure>

OPTION

<figure><img src="image_191.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_192.png" /></figure>

COMPIL

X264

VARIANT

<figure><img src="image_193.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_194.png" /></figure>

SIZE

LENGTH

<figure><img src="image_195.png" /></figure>

RES.
# REAL WORLD EXAMPLE (x264)

## HARDWARE

<figure><img src="image_187.png" /></figure>

A

DELL LATITUDE
7400

20.04

B

<figure><img src="image_188.png" /></figure>

RASPBERRY PI
4 MODEL B

10.4

## OPERATING
SYSTEM

<figure><img src="image_189.png" /></figure>

ubuntu

<figure><img src="image_190.png" /></figure>

x264
--mbtree

<figure><img src="image_191.png" /></figure>

x264
--no-mbtree

<figure><img src="image_192.png" /></figure>

x264
--mbtree

<figure><img src="image_193.png" /></figure>

x264
--no-mbtree

## SOFTWARE

## INPUT DATA

<figure><img src="image_194.png" /></figure>

animation

<figure><img src="image_195.png" /></figure>

vertical

<figure><img src="image_196.png" /></figure>

animation

<figure><img src="image_197.png" /></figure>

vertical

<figure><img src="image_198.png" /></figure>

animation

<figure><img src="image_199.png" /></figure>

vertical

<figure><img src="image_200.png" /></figure>

animation

<figure><img src="image_201.png" /></figure>

vertical

<figure><img src="image_202.png" /></figure>

6

<figure><img src="image_203.png" /></figure>

22

<figure><img src="image_204.png" /></figure>

6

<figure><img src="image_205.png" /></figure>

25

<figure><img src="image_206.png" /></figure>

73

<figure><img src="image_207.png" /></figure>

351

<figure><img src="image_208.png" /></figure>

72

<figure><img src="image_209.png" /></figure>

359

DURATION (S)

SIZE (MB)

33

28

21

34

33

28

21

34
REAL WORLD EXAMPLE (x264)

HARDWARE

<figure><img src="image_196.png" /></figure>

A

DELL LATITUDE
7400

20.04

B

<figure><img src="image_197.png" /></figure>

RASPBERRY PI
4 MODEL B

10.4

OPERATING
SYSTEM

<figure><img src="image_198.png" /></figure>

ubuntu

<figure><img src="image_199.png" /></figure>

x264
--mbtree

<figure><img src="image_200.png" /></figure>

x264
--no-mbtree

<figure><img src="image_201.png" /></figure>

x264
--mbtree

<figure><img src="image_202.png" /></figure>

x264
--no-mbtree

SOFTWARE

INPUT DATA

<figure><img src="image_203.png" /></figure>

animation

<figure><img src="image_204.png" /></figure>

vertical

<figure><img src="image_205.png" /></figure>

animation

<figure><img src="image_206.png" /></figure>

vertical

<figure><img src="image_207.png" /></figure>

animation

<figure><img src="image_208.png" /></figure>

vertical

<figure><img src="image_209.png" /></figure>

animation

<figure><img src="image_210.png" /></figure>

vertical

<figure><img src="image_211.png" /></figure>

6

<figure><img src="image_212.png" /></figure>

22

<figure><img src="image_213.png" /></figure>

6

<figure><img src="image_214.png" /></figure>

25

<figure><img src="image_215.png" /></figure>

73

<figure><img src="image_216.png" /></figure>

351

<figure><img src="image_217.png" /></figure>

72

<figure><img src="image_218.png" /></figure>

359

DURATION (s)

SIZE (MB)

33

28

21

34

33

28

21

34
<figure><img src="image_210.png" /><figcaption>REAL WORLD EXAMPLE (x264)</figcaption></figure>

HARDWARE

OPERATING
SYSTEM

SOFTWARE

INPUT DATA
<figure><img src="image_219.png" /><figcaption>REAL WORLD EXAMPLE (x264)</figcaption></figure>

HARDWARE

OPERATING
SYSTEM

SOFTWARE

INPUT DATA
<figure><img src="image_211.png" /><figcaption>REAL WORLD EXAMPLE (x264)</figcaption></figure>

≈*16

≈*12
REAL WORLD EXAMPLE (x264)

HARDWARE

<figure><img src="image_220.png" /></figure>

A

DELL LATITUDE
7400

20.04

B

<figure><img src="image_221.png" /></figure>

RASPBERRY PI
4 MODEL B

10.4

OPERATING
SYSTEM

<figure><img src="image_222.png" /></figure>

ubuntu

SOFTWARE

x264

<figure><img src="image_223.png" /></figure>

--mbtree

...

2

x264

--no-mbtree

...

<figure><img src="image_224.png" /></figure>

x264

<figure><img src="image_225.png" /></figure>

--mbtree

...

2

x264

--no-mbtree

...

INPUT DATA

<figure><img src="image_226.png" /></figure>

animation

<figure><img src="image_227.png" /></figure>

vertical

<figure><img src="image_228.png" /></figure>

animation

<figure><img src="image_229.png" /></figure>

vertical

<figure><img src="image_230.png" /></figure>

animation

<figure><img src="image_231.png" /></figure>

vertical

<figure><img src="image_232.png" /></figure>

animation

<figure><img src="image_233.png" /></figure>

vertical

<figure><img src="image_234.png" /></figure>

6

<figure><img src="image_235.png" /></figure>

22

<figure><img src="image_236.png" /></figure>

6

<figure><img src="image_237.png" /></figure>

25

<figure><img src="image_238.png" /></figure>

73

<figure><img src="image_239.png" /></figure>

351

<figure><img src="image_240.png" /></figure>

72

<figure><img src="image_241.png" /></figure>

359

DURATION (S)

SIZE (MB)

33

28

21

34

33

28

21

34

≈*16

≈*12
DEEP VARIABILITY

HARDWARE

<figure><img src="image_212.png" /></figure>

AGE

CPU

# CORES

<figure><img src="image_213.png" /></figure>

GPU

OPERATING
SYSTEM

<figure><img src="image_214.png" /></figure>

VERSION

<figure><img src="image_215.png" /></figure>

OPTION

<figure><img src="image_216.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_217.png" /></figure>

COMPIL.

SOFTWARE
VARIANT

<figure><img src="image_218.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_219.png" /></figure>

SIZE

<figure><img src="image_220.png" /></figure>

LENGTH

<figure><img src="image_221.png" /></figure>

RES.

The "best"/default software
variant might be a bad one.

Influential software options
and their interactions vary.

Performance prediction
models and **variability**
**knowledge may not**
**generalize**

<figure><img src="image_222.png" /></figure>

BUG

<figure><img src="image_223.png" /></figure>

PERF. ↗

<figure><img src="image_224.png" /></figure>

PERF. ↗

L. Lesoil, M. Acher, A. Blouin and J.-M. Jézéquel, "Deep Software Variability: Towards Handling Cross-Layer Configuration" in VaMoS 2021
DEEP VARIABILITY

HARDWARE

<figure><img src="image_242.png" /></figure>

AGE

<figure><img src="image_243.png" /></figure>

CPU

# CORES

<figure><img src="image_244.png" /></figure>

GPU

OPERATING
SYSTEM

<figure><img src="image_245.png" /></figure>

VERSION

<figure><img src="image_246.png" /></figure>

OPTION

<figure><img src="image_247.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_248.png" /></figure>

COMPIL.

SOFTWARE
VARIANT

<figure><img src="image_249.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_250.png" /></figure>

SIZE

<figure><img src="image_251.png" /></figure>

LENGTH

<figure><img src="image_252.png" /></figure>

RES.

The "best"/default software
variant might be a bad one.

Influential software options
and their interactions vary.

Performance prediction
models and **variability**
**knowledge may not**
**generalize**

<figure><img src="image_253.png" /></figure>

BUG

<figure><img src="image_254.png" /></figure>

PERF. ↗

<figure><img src="image_255.png" /></figure>

PERF. ↗

L. Lesoil, M. Acher, A. Blouin and J.-M. Jézéquel, "Deep Software Variability: Towards Handling Cross-Layer Configuration" in VaMoS 2021
# Let's go deep with **input data!**

<figure><img src="image_225.png" /></figure>

Intuition: video encoder behavior (and thus **runtime configurations**) hugely depends on the **input video** (different compression ratio, encoding size/type etc.)

* Is the best software configuration still the best?

* Are influential options always influential?

* Does the configuration knowledge generalize?

two performance models $f_1$ and $f_2$

$$f_1 = \beta \times f_2 + \alpha \qquad ?$$

YouTube User General Content dataset: **1397 videos**

Measurements of **201 soft. configurations** (with **same** hardware, compiler, version, etc.): encoding time, bitrate, etc.

<figure><img src="image_226.png" /></figure>
# Let's go deep with **input data!**

<figure><img src="image_256.png" /></figure>

Intuition: video encoder behavior (and thus **runtime configurations**) hugely depends on the **input video** (different compression ratio, encoding size/type etc.)

Is the best software configuration still the best?

Are influential options always influential?

Does the configuration knowledge generalize?

two performance models $f_1$ and $f_2$

$$f_1 = \beta \times f_2 + \alpha$$

YouTube User General Content dataset: **1397 videos**

Measurements of **201 soft. configurations** (with **same** hardware, compiler, version, etc.): encoding time, bitrate, etc.

<figure><img src="image_257.png" /></figure>
Inputs =

<figure><img src="image_227.png" /></figure>

<figure><img src="image_228.png" /></figure>

<figure><img src="image_229.png" /></figure>

<figure><img src="image_230.png" /></figure>

<figure><img src="image_231.png" /></figure>

<figure><img src="image_232.png" /></figure>

<figure><img src="image_233.png" /></figure>

...

configurations' measurements over input_1

<table><thead><tr><td>configurationID</td><td>cabac</td><td>ref</td><td>mixed_ref</td><td>me</td><td>subme</td><td>me_range</td><td>trellis</td><td>elapsedtime</td><td>fps</td><td>rank elapse</td></tr></thead><tbody><tr><td>1</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:14</td><td>375.22</td><td>1</td></tr><tr><td>138</td><td>0</td><td>5</td><td>0</td><td>tesa</td><td>10</td><td>24</td><td>2</td><td>0:04:54</td><td>155.35</td><td>7</td></tr><tr><td>15</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:22</td><td>384.22</td><td>3</td></tr><tr><td>16</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:24</td><td>375.4</td><td>4</td></tr><tr><td>17</td><td>0</td><td>1</td><td>0</td><td>hex</td><td>0</td><td>16</td><td>0</td><td>0:02:19</td><td>385.92</td><td>2</td></tr><tr><td>21</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:84</td><td>260.65</td><td>6</td></tr><tr><td>22</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:61</td><td>303.2</td><td>5</td></tr></tbody></table>

configurations' measurements over input_42

<table><thead><tr><td>configurationID</td><td>cabac</td><td>ref</td><td>mixed_ref</td><td>me</td><td>subme</td><td>me_range</td><td>trellis</td><td>elapsedtime</td><td>fps</td><td>rank elapse</td></tr></thead><tbody><tr><td>1</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>04:37</td><td>375.22</td><td>3</td></tr><tr><td>138</td><td>0</td><td>5</td><td>0</td><td>tesa</td><td>10</td><td>24</td><td>2</td><td>07:56</td><td>155.35</td><td>7</td></tr><tr><td>15</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>07:23</td><td>384.22</td><td>6</td></tr><tr><td>16</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>04:33</td><td>375.4</td><td>2</td></tr><tr><td>17</td><td>0</td><td>1</td><td>0</td><td>hex</td><td>0</td><td>16</td><td>0</td><td>06:00</td><td>385.92</td><td>5</td></tr><tr><td>21</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>05:48</td><td>260.65</td><td>4</td></tr><tr><td>22</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>02:19</td><td>303.2</td><td>1</td></tr></tbody></table>
Inputs =

<figure><img src="image_258.png" /></figure>

<figure><img src="image_259.png" /></figure>

<figure><img src="image_260.png" /></figure>

<figure><img src="image_261.png" /></figure>

<figure><img src="image_262.png" /></figure>

<figure><img src="image_263.png" /></figure>

<figure><img src="image_264.png" /></figure>

...

configurations' measurements over input_1

<table><thead><tr><td>configurationID</td><td>cabac</td><td>ref</td><td>mixed_ref</td><td>me</td><td>subme</td><td>me_range</td><td>trellis</td><td>elapsedtime</td><td>fps</td><td>rank elapse time</td></tr></thead><tbody><tr><td>1</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:14</td><td>375.22</td><td>1</td></tr><tr><td>138</td><td>0</td><td>5</td><td>0</td><td>tesa</td><td>10</td><td>24</td><td>2</td><td>0:04:54</td><td>155.35</td><td>7</td></tr><tr><td>15</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:22</td><td>384.22</td><td>3</td></tr><tr><td>16</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:24</td><td>375.4</td><td>4</td></tr><tr><td>17</td><td>0</td><td>1</td><td>0</td><td>hex</td><td>0</td><td>16</td><td>0</td><td>0:02:19</td><td>385.92</td><td>2</td></tr><tr><td>21</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:84</td><td>260.65</td><td>6</td></tr><tr><td>22</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:61</td><td>303.2</td><td>5</td></tr></tbody></table>

configurations' measurements over input_42

<table><thead><tr><td>configurationID</td><td>cabac</td><td>ref</td><td>mixed_ref</td><td>me</td><td>subme</td><td>me_range</td><td>trellis</td><td>elapsedtime</td><td>fps</td><td>rank elapse time</td></tr></thead><tbody><tr><td>1</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>04:37</td><td>375.22</td><td>3</td></tr><tr><td>138</td><td>0</td><td>5</td><td>0</td><td>tesa</td><td>10</td><td>24</td><td>2</td><td>07:56</td><td>155.35</td><td>7</td></tr><tr><td>15</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>07:23</td><td>384.22</td><td>6</td></tr><tr><td>16</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>04:33</td><td>375.4</td><td>2</td></tr><tr><td>17</td><td>0</td><td>1</td><td>0</td><td>hex</td><td>0</td><td>16</td><td>0</td><td>06:00</td><td>385.92</td><td>5</td></tr><tr><td>21</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>05:48</td><td>260.65</td><td>4</td></tr><tr><td>22</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>02:19</td><td>303.2</td><td>1</td></tr></tbody></table>
<figure><img src="image_234.png" /></figure>

<figure><img src="image_235.png" /></figure>

<figure><img src="image_236.png" /></figure>

<figure><img src="image_237.png" /></figure>

<figure><img src="image_238.png" /></figure>

<figure><img src="image_239.png" /></figure>

<figure><img src="image_240.png" /></figure>

configurations' measurements over input_1

<table><thead><tr><th>configurationID</th><th>cabac</th><th>ref</th><th>mixed_ref</th><th>me</th><th>subme</th><th>me_range</th><th>trellis</th><th>elapsedtime</th><th>fps</th><th>rank elapse</th></tr></thead><tbody><tr><td>1</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:14</td><td>375.22</td><td>1</td></tr><tr><td>138</td><td>0</td><td>5</td><td>0</td><td>tesa</td><td>10</td><td>24</td><td>2</td><td>0:04:54</td><td>155.35</td><td>7</td></tr><tr><td>15</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:22</td><td>384.22</td><td>3</td></tr><tr><td>16</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:24</td><td>375.4</td><td>4</td></tr><tr><td>17</td><td>0</td><td>1</td><td>0</td><td>hex</td><td>0</td><td>16</td><td>0</td><td>0:02:19</td><td>385.92</td><td>2</td></tr><tr><td>21</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:84</td><td>260.65</td><td>6</td></tr><tr><td>22</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:61</td><td>303.2</td><td>5</td></tr></tbody></table>

Generalization/transfer:

what's the relationship between perf_pred_1 and perf_pred_42?

* with perf_pred_i a performance model capable of predicting performance of any configuration on input_i linear relationship?

* eg Pearson/Spearman linear correlation

* influential features/options: same?

configurations' measurements over input_42

<table><thead><tr><th>configurationID</th><th>cabac</th><th>ref</th><th>mixed_ref</th><th>me</th><th>subme</th><th>me_range</th><th>trellis</th><th>elapsedtime</th><th>fps</th><th>rank elapse</th></tr></thead><tbody><tr><td>1</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:43:37</td><td>375.22</td><td>3</td></tr><tr><td>138</td><td>0</td><td>5</td><td>0</td><td>tesa</td><td>10</td><td>24</td><td>2</td><td>0:75:56</td><td>155.35</td><td>7</td></tr><tr><td>15</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:72:23</td><td>384.22</td><td>6</td></tr><tr><td>16</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:43:33</td><td>375.4</td><td>2</td></tr><tr><td>17</td><td>0</td><td>1</td><td>0</td><td>hex</td><td>0</td><td>16</td><td>0</td><td>0:60:00</td><td>385.92</td><td>5</td></tr><tr><td>21</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:54:48</td><td>260.65</td><td>4</td></tr><tr><td>22</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:21:90</td><td>303.2</td><td>1</td></tr></tbody></table>
<figure><img src="image_265.png" /></figure>

<figure><img src="image_266.png" /></figure>

<figure><img src="image_267.png" /></figure>

<figure><img src="image_268.png" /></figure>

<figure><img src="image_269.png" /></figure>

<figure><img src="image_270.png" /></figure>

<figure><img src="image_271.png" /></figure>

configurations' measurements over input_1

<table><thead><tr><td>configurationID</td><td>cabac</td><td>ref</td><td>mixed_ref</td><td>me</td><td>subme</td><td>me_range</td><td>trellis</td><td>elapsedtime</td><td>fps</td><td>rank elapse</td></tr></thead><tbody><tr><td>1</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:14</td><td>375.22</td><td>1</td></tr><tr><td>138</td><td>0</td><td>5</td><td>0</td><td>tesa</td><td>10</td><td>24</td><td>2</td><td>0:04:54</td><td>155.35</td><td>7</td></tr><tr><td>15</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:22</td><td>384.22</td><td>3</td></tr><tr><td>16</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:24</td><td>375.4</td><td>4</td></tr><tr><td>17</td><td>0</td><td>1</td><td>0</td><td>hex</td><td>0</td><td>16</td><td>0</td><td>0:02:19</td><td>385.92</td><td>2</td></tr><tr><td>21</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:84</td><td>260.65</td><td>6</td></tr><tr><td>22</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:02:61</td><td>303.2</td><td>5</td></tr></tbody></table>

Generalization/transfer:

what's the relationship between perf_pred_1 and perf_pred_42?

* with perf_pred_i a performance model capable of predicting performance of any configuration on input_i linear relationship?

* eg Pearson/Spearman linear correlation

* influential features/options: same?

configurations' measurements over input_42

<table><thead><tr><td>configurationID</td><td>cabac</td><td>ref</td><td>mixed_ref</td><td>me</td><td>subme</td><td>me_range</td><td>trellis</td><td>elapsedtime</td><td>fps</td><td>rank elapse</td></tr></thead><tbody><tr><td>1</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:43:37</td><td>375.22</td><td>3</td></tr><tr><td>138</td><td>0</td><td>5</td><td>0</td><td>tesa</td><td>10</td><td>24</td><td>2</td><td>0:75:56</td><td>155.35</td><td>7</td></tr><tr><td>15</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:72:23</td><td>384.22</td><td>6</td></tr><tr><td>16</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:43:33</td><td>375.4</td><td>2</td></tr><tr><td>17</td><td>0</td><td>1</td><td>0</td><td>hex</td><td>0</td><td>16</td><td>0</td><td>0:60:00</td><td>385.92</td><td>5</td></tr><tr><td>21</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:54:48</td><td>260.65</td><td>4</td></tr><tr><td>22</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>0:21:90</td><td>303.2</td><td>1</td></tr></tbody></table>
# Let's go deep with **input data!**

<figure><img src="image_241.png" /></figure>

Intuition: video encoder behavior (and thus **runtime configurations**) hugely depends on the **input video** (different compression ratio, encoding size/type etc.)

Is the best software configuration still the best?

Are influential options always influential?

Does the configuration knowledge generalize?

two performance models $f_1$ and $f_2$

$$f_1 = \beta \times f_2 + \alpha$$

<figure><img src="image_242.png" /></figure>

YouTube User General Content dataset: **1397 videos**

Measurements of **201 soft. configurations** (with **same** hardware, compiler, version, etc.): encoding time, bitrate, etc.
# Let's go deep with **input data!**

<figure><img src="image_272.png" /></figure>

Intuition: video encoder behavior (and thus **runtime configurations**) hugely depends on the **input video** (different compression ratio, encoding size/type etc.)

Is the best software configuration still the best?

Are influential options always influential?

Does the configuration knowledge generalize?

two performance models $f_1$ and $f_2$

$$f_1 = \beta \times f_2 + \alpha$$

<figure><img src="image_273.png" /></figure>

YouTube User General Content dataset: **1397 videos**

Measurements of **201 soft. configurations** (with **same** hardware, compiler, version, etc.): encoding time, bitrate, etc.
<figure><img src="image_243.png" /></figure>

# Do x264 software performances stay consistent across inputs?

1397 videos x 201 software configurations

* Encoding time: very strong correlations
  ○ low input sensitivity

* FPS: very strong correlations
  ○ low input sensitivity

* CPU usage : moderate correlation, a few negative correlations
  ○ medium input sensitivity

* Bitrate: medium-low correlation, many negative correlations
  ○ High input sensitivity

* Encoding size: medium-low correlation, many negative correlations
  ○ High input sensitivity

two performance models $f_1$ and $f_2$

$$f_1 = \beta \times f_2 + \alpha$$
<figure><img src="image_274.png" /></figure>

# Do x264 software performances stay consistent across inputs?

1397 videos x 201 software configurations

* Encoding time: very strong correlations
  ○ low input sensitivity

* FPS: very strong correlations
  ○ low input sensitivity

* CPU usage : moderate correlation, a few negative correlations
  ○ medium input sensitivity

* Bitrate: medium-low correlation, many negative correlations
  ○ High input sensitivity

* Encoding size: medium-low correlation, many negative correlations
  ○ High input sensitivity

two performance models $f_1$ and $f_2$

$$f_1 = \beta \times f_2 + \alpha$$
Are there some configuration options
more sensitive to input videos? (bitrate)

<figure><img src="image_244.png" /></figure>
Are there some configuration options
more sensitive to input videos? (bitrate)

<figure><img src="image_275.png" /></figure>
<figure><img src="image_245.png" /></figure>

Are there some configuration options
more sensitive to input videos? (bitrate)

<figure><img src="image_246.png" /></figure>
<figure><img src="image_276.png" /></figure>

Are there some configuration options
more sensitive to input videos? (bitrate)

<figure><img src="image_277.png" /></figure>
# Practical impacts for users, developers, scientists, and self-adaptive systems

**Threats** to **variability knowledge**: predicting, tuning, or understanding configurable systems without being aware of inputs can be inaccurate and... pointless

**Opportunities:** for some performance properties (P) and subject systems, some stability is observed and performance remains consistent!

<table><thead><tr><th>System</th><th>Domain</th><th>Commit</th><th>Configs #C</th><th>Inputs I</th><th>#I</th></tr></thead><tbody><tr><td>gcc</td><td>Compilation</td><td>ccbe07</td><td>80</td><td>.c programs</td><td>30</td></tr><tr><td>ImageMagick</td><td>Image processing</td><td>5ee49d6</td><td>100</td><td>images</td><td>1000</td></tr><tr><td>lingeling</td><td>SAT solver</td><td>7d5db72</td><td>100</td><td>SAT formulae</td><td>351</td></tr><tr><td>nodeJS</td><td>JS runtime env.</td><td>78343bb</td><td>50</td><td>.js scripts</td><td>1939</td></tr><tr><td>poppler</td><td>PDF rendering</td><td>42dde68</td><td>16</td><td>.pdf files</td><td>1480</td></tr><tr><td>SQLite</td><td>DBMS</td><td>53fa025</td><td>50</td><td>databases</td><td>150</td></tr><tr><td>x264</td><td>Video encoding</td><td>e9a5903</td><td>201</td><td>videos</td><td>1397</td></tr><tr><td>xz</td><td>Data compression</td><td>e7da44d</td><td>30</td><td>system files</td><td>48</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>#M</th><th>Performance(s) P</th><th>Docker</th><th>Dataset</th></tr></thead><tbody><tr><td>gcc</td><td>2400</td><td>size, ctime, exec</td><td>Link</td><td>Link</td></tr><tr><td>ImageMagick</td><td>100 000</td><td>size, time</td><td>Link</td><td>Link</td></tr><tr><td>lingeling</td><td>35 100</td><td>#confl., #reduc.</td><td>Link</td><td>Link</td></tr><tr><td>nodeJS</td><td>96 950</td><td>#operations/s</td><td>Link</td><td>Link</td></tr><tr><td>poppler</td><td>23 680</td><td>size, time</td><td>Link</td><td>Link</td></tr><tr><td>SQLite</td><td>7500</td><td>15 query times q1-q15</td><td>Link</td><td>Link</td></tr><tr><td>x264</td><td>280 797</td><td>cpu, fps, kbs, size, time</td><td>Link</td><td>Link</td></tr><tr><td>xz</td><td>1440</td><td>size, time</td><td>Link</td><td>Link</td></tr></tbody></table>

L. Lesoil, M. Acher, A. Blouin and J.-M. Jézéquel “The Interaction between Inputs and Configurations fed to Software Systems: an Empirical Study”
https://arxiv.org/abs/2112.07279

<figure><img src="image_247.png" /></figure>
# Practical impacts for users, developers, scientists, and self-adaptive systems

**Threats** to **variability knowledge**: predicting, tuning, or understanding configurable systems without being aware of inputs can be inaccurate and... pointless

**Opportunities:** for some performance properties (P) and subject systems, some stability is observed and performance remains consistent!

<table><thead><tr><th>System</th><th>Domain</th><th>Commit</th><th>Configs #C</th><th>Inputs I</th><th>#I</th></tr></thead><tbody><tr><td>gcc</td><td>Compilation</td><td>ccbe07</td><td>80</td><td>.c programs</td><td>30</td></tr><tr><td>ImageMagick</td><td>Image processing</td><td>5ee49d6</td><td>100</td><td>images</td><td>1000</td></tr><tr><td>lingeling</td><td>SAT solver</td><td>7d5db72</td><td>100</td><td>SAT formulae</td><td>351</td></tr><tr><td>nodeJS</td><td>JS runtime env.</td><td>78343bb</td><td>50</td><td>.js scripts</td><td>1939</td></tr><tr><td>poppler</td><td>PDF rendering</td><td>42dde68</td><td>16</td><td>.pdf files</td><td>1480</td></tr><tr><td>SQLite</td><td>DBMS</td><td>53fa025</td><td>50</td><td>databases</td><td>150</td></tr><tr><td>x264</td><td>Video encoding</td><td>e9a5903</td><td>201</td><td>videos</td><td>1397</td></tr><tr><td>xz</td><td>Data compression</td><td>e7da44d</td><td>30</td><td>system files</td><td>48</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>#M</th><th>Performance(s) P</th><th>Docker</th><th>Dataset</th></tr></thead><tbody><tr><td>gcc</td><td>2400</td><td>size, ctime, exec</td><td>Link</td><td>Link</td></tr><tr><td>ImageMagick</td><td>100 000</td><td>size, time</td><td>Link</td><td>Link</td></tr><tr><td>lingeling</td><td>35 100</td><td>#confl.,#reduc.</td><td>Link</td><td>Link</td></tr><tr><td>nodeJS</td><td>96 950</td><td>#operations/s</td><td>Link</td><td>Link</td></tr><tr><td>poppler</td><td>23 680</td><td>size, time</td><td>Link</td><td>Link</td></tr><tr><td>SQLite</td><td>7500</td><td>15 query times q1-q15</td><td>Link</td><td>Link</td></tr><tr><td>x264</td><td>280 797</td><td>cpu, fps, kbs, size, time</td><td>Link</td><td>Link</td></tr><tr><td>xz</td><td>1440</td><td>size, time</td><td>Link</td><td>Link</td></tr></tbody></table>

L. Lesoil, M. Acher, A. Blouin and J.-M. Jézéquel “The Interaction between Inputs and Configurations fed to Software Systems: an Empirical Study”
https://arxiv.org/abs/2112.07279

<figure><img src="image_278.png" /></figure>
DEEP VARIABILITY

HARDWARE

<figure><img src="image_248.png" /></figure>

AGE

CPU

# CORES

<figure><img src="image_249.png" /></figure>

GPU

OPERATING
SYSTEM

<figure><img src="image_250.png" /></figure>

VERSION

<figure><img src="image_251.png" /></figure>

OPTION

<figure><img src="image_252.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_253.png" /></figure>

COMPIL.

SOFTWARE
 VARIANT

<figure><img src="image_254.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_255.png" /></figure>

SIZE

<figure><img src="image_256.png" /></figure>

LENGTH

<figure><img src="image_257.png" /></figure>

Sometimes, variability is
consistent/stable and
knowledge transfer is
immediate.

But there are also
interactions among
variability layers and
**variability knowledge**
**may not generalize**

<figure><img src="image_258.png" /></figure>

BUG

<figure><img src="image_259.png" /></figure>

PERF. →

<figure><img src="image_260.png" /></figure>

PERF. →

<figure><img src="image_261.png" /></figure>

Poppler

node

264

<figure><img src="image_262.png" /></figure>

xz
DEEP VARIABILITY

HARDWARE

<figure><img src="image_279.png" /></figure>

AGE

CPU

# CORES

<figure><img src="image_280.png" /></figure>

GPU

OPERATING
SYSTEM

<figure><img src="image_281.png" /></figure>

VERSION

<figure><img src="image_282.png" /></figure>

OPTION

<figure><img src="image_283.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_284.png" /></figure>

COMPIL.

SOFTWARE
 VARIANT

<figure><img src="image_285.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_286.png" /></figure>

SIZE

<figure><img src="image_287.png" /></figure>

LENGTH

<figure><img src="image_288.png" /></figure>

SIZE

Sometimes, variability is
consistent/stable and
knowledge transfer is
immediate.

But there are also
interactions among
variability layers and
**variability knowledge**
**may not generalize**

<figure><img src="image_289.png" /></figure>

BUG

<figure><img src="image_290.png" /></figure>

PERF. →

<figure><img src="image_291.png" /></figure>

PERF. →

<figure><img src="image_292.png" /></figure>

Poppler

node

264

<figure><img src="image_293.png" /></figure>

xz
# DOES DEEP SOFTWARE VARIABILITY AFFECT PREVIOUS SCIENTIFIC, SOFTWARE-BASED STUDIES? (A GRAPHICAL TEMPLATE)

## HARDWARE

<figure><img src="image_263.png" /></figure>

AGE

<figure><img src="image_264.png" /></figure>

CPU

# CORES

<figure><img src="image_265.png" /></figure>

GPU

LIST ALL DETAILS...

AND QUESTIONS:

## OPERATING SYSTEM

<figure><img src="image_266.png" /></figure>

VERSION

<figure><img src="image_267.png" /></figure>

OPTION

<figure><img src="image_268.png" /></figure>

DISTRIB.

## WHAT IF WE RUN THE EXPERIMENTS ON DIFFERENT:

OS?

## SOFTWARE

<figure><img src="image_269.png" /></figure>

COMPIL.

SOFTWARE VARIANT

<figure><img src="image_270.png" /></figure>

VERSION

VERSION/COMMIT?

## INPUT DATA

<figure><img src="image_271.png" /></figure>

SIZE

<figure><img src="image_272.png" /></figure>

LENGTH

<figure><img src="image_273.png" /></figure>

RES.

PARAMETERS?

INPUT?
# DOES DEEP SOFTWARE VARIABILITY AFFECT PREVIOUS SCIENTIFIC, SOFTWARE-BASED STUDIES? (A GRAPHICAL TEMPLATE)

## HARDWARE

<figure><img src="image_294.png" /></figure>

AGE

<figure><img src="image_295.png" /></figure>

CPU

# CORES

<figure><img src="image_296.png" /></figure>

GPU

LIST ALL DETAILS...

AND QUESTIONS:

## OPERATING SYSTEM

<figure><img src="image_297.png" /></figure>

VERSION

<figure><img src="image_298.png" /></figure>

OPTION

<figure><img src="image_299.png" /></figure>

DISTRIB.

## WHAT IF WE RUN THE EXPERIMENTS ON DIFFERENT:

OS?

## SOFTWARE

<figure><img src="image_300.png" /></figure>

COMPIL.

SOFTWARE VARIANT

<figure><img src="image_301.png" /></figure>

VERSION

VERSION/COMMIT?

## INPUT DATA

<figure><img src="image_302.png" /></figure>

SIZE

<figure><img src="image_303.png" /></figure>

LENGTH

<figure><img src="image_304.png" /></figure>

RES.

PARAMETERS?

INPUT?
# AGENDA

## Frictionless Reproducibility and (Deep) Software (Variability)

### Problem: Variability and Frictions

### Solution: Variability and Exploration

### Discussions
# AGENDA

## Frictionless Reproducibility and (Deep) Software (Variability)

### Problem: Variability and Frictions

### Solution: Variability and Exploration

### Discussions
# Deep variability problem (statement)

Fundamentally, we have a huge multi-dimensional variant space (eg 10^6000)

`run (source_code) => result`

`run (hardware, operating_system, build_environment, input_data, source_code, ...) => results`

**Fixing variability once and for all, in all dimensions/layers, is the obvious solution...**

But it is either impossible (eg the ages of processor can have an impact on execution time)...

Or not desirable

* non-robust result

* generalization/transferability of the results/findings

* kill innovation
# Deep variability problem (statement)

Fundamentally, we have a huge multi-dimensional variant space (eg 10^6000)

run (source_code) => result

run (hardware, operating_system, build_environment, input_data, source_code, ...) => results

**Fixing variability once and for all, in all dimensions/layers, is the obvious solution...**

But it is either impossible (eg the ages of processor can have an impact on execution time)...

Or not desirable

* non-robust result

* generalization/transferability of the results/findings

* kill innovation
# Replicability is the holy grail!

Exploring various configurations:

* Make more **robust** scientific findings

* Define and assess the **validity** envelope

* Enable **exploration** and **optimization**

* Innovation and new hypothesis, insights, **knowledge**

⇒ We propose to embrace deep variability for the sake of **replicability**

<figure><img src="image_274.png" /></figure>
# Replicability is the holy grail!

Exploring various configurations:

* Make more **robust** scientific findings

* Define and assess the **validity** envelope

* Enable **exploration** and **optimization**

* Innovation and new hypothesis, insights, **knowledge**

⇒ We propose to embrace deep variability for the sake of **replicability**

<figure><img src="image_305.png" /></figure>
# Our Vision

## Embrace deep variability!

Explicit modeling of the variability points and their relationships, such as:

1. Get insights into the variability “factors” and their possible interactions

2. Capture and document configurations for the sake of **reproducibility**

3. Explore diverse configurations to **replicate**, and hence optimize, validate, increase the robustness, or provide better resilience

## Embracing Deep Variability For Reproducibility & Replicability

Mathieu Acher, Benoit Combemale, Georges Aaron Randrianaina, Jean-Marc Jézéquel

IRISA, Université de Rennes

Rennes, France

### ABSTRACT

Reproducibility (a.k.a., determinism in some cases) constitutes a fundamental aspect in various fields of computer science, such as floating-point computations in numerical analysis and simulation, concurrency models in parallelism, reproducible builds for third parties integration and packaging, and containerization for execution environments. These concepts, while pervasive across diverse concerns, often exhibit intricate inter-dependencies, making it challenging to achieve a comprehensive understanding. In this short and vision paper we delve into the application of software engineering techniques, specifically variability management, to systematically identify and explicit points of variability that may give rise to reproducibility issues (e.g., language, libraries, compiler, virtual machine, OS, environment variables, etc.). The primary objectives are: i) gaining insights into the variability layers and their possible interactions, ii) capturing and documenting configurations for the sake of reproducibility, and iii) exploring diverse configurations to replicate, and hence validate and ensure the robustness of results. By adopting these methodologies, we aim to address the complexities associated with reproducibility and replicability in modern software systems and environments, facilitating a more comprehensive and nuanced perspective on these critical aspects.

In this paper we propose to characterize both intended and unintended variability of any software-intensive system in order to support reproducibility and replicability, and eventually estimate its robustness, uncertainty profile, and explore different hypotheses.

### 2 DEEP SOFTWARE VARIABILITY

Uncertainty in informatics comes from many different origins [16, 36], either ontological (i.e., inherent unpredictability, e.g., aleatory) or epistemic (i.e., due to insufficient knowledge).

Ontological causes include noise in the input data of a program, its memory layout, network delays, the internal state of the processor, the ambient temperature and even the age of the processor¹.

Epistemic causes include misunderstanding of the user's needs, variable behavior of conceptually similar resolution methods, choice of threshold parameters, unexpected behavior of APIs, variable behavior among functionally similar libraries, or subtle differences in the semantics of programming languages (e.g., ~3% evaluates to ~1 in Java but to 1 in Python, or even inside the same programming language (for instance x/0 is an undefined behavior in C)).

<table><tr><td>Parameters, Input Data</td><td>e.g., random seed selection</td></tr><tr><td>Programming Style</td><td>e.g., x+(y+z) vs. (x+y+z)</td></tr></table>

⇒ We aim to **address the complexities associated with reproducibility and replicability in modern software systems and environments**, facilitating a more comprehensive and nuanced perspective on these critical “factors”.
# Our Vision

## Embrace deep variability!

Explicit modeling of the variability points and their relationships, such as:

1. Get insights into the variability “factors” and their possible interactions

2. Capture and document configurations for the sake of **reproducibility**

3. Explore diverse configurations to **replicate**, and hence optimize, validate, increase the robustness, or provide better resilience

## Embracing Deep Variability For Reproducibility & Replicability

Mathieu Acher, Benoit Combemale, Georges Aaron Randrianaina, Jean-Marc Jézéquel

IRISA, Université de Rennes

Rennes, France

### ABSTRACT

Reproducibility (a.k.a., determinism in some cases) constitutes a fundamental aspect in various fields of computer science, such as floating-point computations in numerical analysis and simulation, concurrency models in parallelism, reproducible builds for third parties integration and packaging, and containerization for execution environments. These concepts, while pervasive across diverse concerns, often exhibit intricate inter-dependencies, making it challenging to achieve a comprehensive understanding. In this short and vision paper we delve into the application of software engineering techniques, specifically variability management, to systematically identify and explicit points of variability that may give rise to reproducibility issues (e.g., language, libraries, compiler, virtual machine, OS, environment variables, etc.). The primary objectives are: i) gaining insights into the variability layers and their possible interactions, ii) capturing and documenting configurations for the sake of reproducibility, and iii) exploring diverse configurations to replicate, and hence validate and ensure the robustness of results. By adopting these methodologies, we aim to address the complexities associated with reproducibility and replicability in modern software systems and environments, facilitating a more comprehensive and nuanced perspective on these critical aspects.

In this paper we propose to characterize both intended and unintended variability of any software-intensive system in order to support reproducibility and replicability, and eventually estimate its robustness, uncertainty profile, and explore different hypotheses.

### 2 DEEP SOFTWARE VARIABILITY

Uncertainty in informatics comes from many different origins [16, 36], either ontological (i.e., inherent unpredictability, e.g., aleatory) or epistemic (i.e., due to insufficient knowledge).

Ontological causes include noise in the input data of a program, its memory layout, network delays, the internal state of the processor, the ambient temperature and even the age of the processor¹.

Epistemic causes include misunderstanding of the user's needs, variable behavior of conceptually similar resolution methods, choice of threshold parameters, unexpected behavior of APIs, variable behavior among functionally similar libraries, or subtle differences in the semantics of programming languages (e.g., ~3% evaluates to ~1 in Java but to 1 in Python, or even inside the same programming language (for instance x/0 is an undefined behavior in C)).

<table><tr><td>Parameters, Input Data</td><td>e.g., random seed selection</td></tr><tr><td>Programming Style</td><td>e.g., x+(y+z) vs. (x+y)+z</td></tr></table>

ACM REP 2024

⇒ We aim to **address the complexities associated with reproducibility and replicability in modern software systems and environments**, facilitating a more comprehensive and nuanced perspective on these critical “factors”.
# Solution #1: Variability model

* Abstractions are definitely needed to...

* reason about logical constraints and interactions

* integrate domain knowledge

* synthesize domain knowledge

* automate and guide the exploration of variants

* scope and prioritize experiments

* Language and formalism: **feature model** (widely applicable!)

* translation to logics

* reasoning with SAT/CP/SMT solvers

FAMiliAR

<figure><img src="image_275.png" /></figure>

<figure><img src="image_276.png" /></figure>

Fig. 1: Web UI for the Floating point experiments

x/t

<figure><img src="image_277.png" /></figure>

<figure><img src="image_278.png" /></figure>
# Solution #1: Variability model

* Abstractions are definitely needed to...

* reason about logical constraints and interactions

* integrate domain knowledge

* synthesize domain knowledge

* automate and guide the exploration of variants

* scope and prioritize experiments

* Language and formalism: **feature model** (widely applicable!)

* translation to logics

* reasoning with SAT/CP/SMT solvers

FAMiliAR

<figure><img src="image_306.png" /></figure>

<figure><img src="image_307.png" /></figure>

Fig. 1: Web UI for the Floating point experiments

<figure><img src="image_308.png" /></figure>

<figure><img src="image_309.png" /></figure>
# Solution #1: Variability model

* Abstractions are definitely needed...

* Yes, but how to obtain a feature model?

* modelling

* reverse engineering (out of command-line parameters, source code, logs, configurations, etc.)

* learning (next slide!)

* modeling+reverse engineering+learning (HDR)

<figure><img src="image_279.png" /></figure>

<figure><img src="image_280.png" /></figure>
# Solution #1: Variability model

* Abstractions are definitely needed...

* Yes, but how to obtain a feature model?

* modelling

* reverse engineering (out of command-line parameters, source code, logs, configurations, etc.)

* learning (next slide!)

* modeling+reverse engineering+learning (HDR)

<figure><img src="image_310.png" /></figure>

<figure><img src="image_311.png" /></figure>
# Solution #2: sampling and learning

(regression, classification)

<figure><img src="image_281.png" /></figure>

<figure><img src="image_282.png" /></figure>

$f: \mathbb{C} \rightarrow \mathbb{R}^+$ the function affecting to any configuration $c \in \mathbb{C}$ its performance $f(c) \in \mathbb{R}^+$,

J. Alves Pereira, H. Martin, M. Acher, J.-M. Jézéquel, G. Botterweck and A. Ventresque
"Learning Software Configuration Spaces: A Systematic Literature Review" JSS, 2021<sup>69</sup>
# Solution #2: sampling and learning

(regression, classification)

<figure><img src="image_312.png" /></figure>

<figure><img src="image_313.png" /></figure>

$f: \mathbb{C} \rightarrow \mathbb{R}^+$ the function affecting to any configuration $c \in \mathbb{C}$ its performance $f(c) \in \mathbb{R}^+$,

<figure><img src="image_314.png" /></figure>

J. Alves Pereira, H. Martin, M. Acher, J.-M. Jézéquel, G. Botterweck and A. Ventresque

“Learning Software Configuration Spaces: A Systematic Literature Review” JSS, 2021<sup>69</sup>
<table><thead><tr><td>configurationID</td><td>cabac</td><td>ref</td><td>mixed_ref</td><td>me</td><td>subme</td><td>me_range</td><td>trellis</td><td>elapsedtime</td><td>fps</td><td>rank elapse time</td></tr></thead><tbody><tr><td>1</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>04:37</td><td>375.22</td><td>3</td></tr><tr><td>138</td><td>0</td><td>5</td><td>0</td><td>tesa</td><td>10</td><td>24</td><td>2</td><td>07:56</td><td>155.35</td><td>7</td></tr><tr><td>15</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>07:23</td><td>384.22</td><td>6</td></tr><tr><td>16</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>04:33</td><td>375.4</td><td>2</td></tr><tr><td>17</td><td>0</td><td>1</td><td>0</td><td>hex</td><td>0</td><td>16</td><td>0</td><td>06:00</td><td>385.92</td><td>5</td></tr><tr><td>21</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>05:48</td><td>260.65</td><td>4</td></tr><tr><td>22</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>02:19</td><td>303.2</td><td>1</td></tr></tbody></table>

x264 --me dia

--ref 5

...

-o output_1.x264

<figure><img src="image_283.png" /></figure>

<figure><img src="image_284.png" /></figure>
<table><thead><tr><td>configurationID</td><td>cabac</td><td>ref</td><td>mixed_ref</td><td>me</td><td>subme</td><td>me_range</td><td>trellis</td><td>elapsedtime</td><td>fps</td><td>rank elapse time</td></tr></thead><tbody><tr><td>1</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>04:37</td><td>375.22</td><td>3</td></tr><tr><td>138</td><td>0</td><td>5</td><td>0</td><td>tesa</td><td>10</td><td>24</td><td>2</td><td>07:56</td><td>155.35</td><td>7</td></tr><tr><td>15</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>07:23</td><td>384.22</td><td>6</td></tr><tr><td>16</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>04:33</td><td>375.4</td><td>2</td></tr><tr><td>17</td><td>0</td><td>1</td><td>0</td><td>hex</td><td>0</td><td>16</td><td>0</td><td>06:00</td><td>385.92</td><td>5</td></tr><tr><td>21</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>05:48</td><td>260.65</td><td>4</td></tr><tr><td>22</td><td>0</td><td>1</td><td>0</td><td>dia</td><td>0</td><td>16</td><td>0</td><td>02:19</td><td>303.2</td><td>1</td></tr></tbody></table>

x264 --me dia

--ref 5

...

-o output_1.x264

<figure><img src="image_315.png" /></figure>

<figure><img src="image_316.png" /></figure>
<figure><img src="image_285.png" /></figure>

hardware variability

15,000+ options

<figure><img src="image_286.png" /></figure>

OBS
Open Broadcaster Software*

<figure><img src="image_287.png" /></figure>

thousands of compiler flags
and compile-time options

dozens of preferences

input data

100+ command-line parameters

1000+ feature toggles

deep software variability

System under
Study
(reproducible)

Variability
Output
(binary)

"The build process of a software product is reproducible if, after designating a specific version of its source code and all of its build dependencies, every build produces bit-for-bit identical artifacts, no matter the environment in which the build is performed."

Lamb and Zacchiroli "Reproducible Builds: Increasing the Integrity of Software Supply Chains" IEEE Software 2022
<figure><img src="image_317.png" /></figure>

hardware variability

15,000+ options

<figure><img src="image_318.png" /></figure>

OBS
Open Broadcaster Software*

<figure><img src="image_319.png" /></figure>

thousands of compiler flags
and compile-time options

dozens of preferences

input data

100+ command-line parameters

1000+ feature toggles

deep software variability

System under
Study
(reproducible)

Variability
Output
(binary)

"The build process of a software product is reproducible if, after designating a specific version of its source code and all of its build dependencies, every build produces bit-for-bit identical artifacts, no matter the environment in which the build is performed."

Lamb and Zacchiroli "Reproducible Builds: Increasing the Integrity of Software Supply Chains" IEEE Software 2022
<figure><img src="image_288.png" /></figure>

15,000+
compile-time options

deep software variability

<figure><img src="image_289.png" /></figure>

<figure><img src="image_290.png" /></figure>

System under
Study

Variability
Output
(binary)

"The build process of a software product is reproducible if, after designating a specific version of its source code and all of its build dependencies, every build produces bit-for-bit identical artifacts, no matter the environment in which the build is performed." Lamb and Zacchiroli "Reproducible Builds: Increasing the Integrity of Software Supply Chains" IEEE Software 2022

make defconfig # configuration
make # build the kernel (binary) out of config
make # should be the same, right?
<figure><img src="image_320.png" /></figure>

15,000+
compile-time options

deep software variability

<figure><img src="image_321.png" /></figure>

<figure><img src="image_322.png" /></figure>

System under
Study

Variability
Output
(binary)

"The build process of a software product is reproducible if, after designating a specific version of its source code and all of its build dependencies, every build produces bit-for-bit identical artifacts, no matter the environment in which the build is performed." Lamb and Zacchiroli "Reproducible Builds: Increasing the Integrity of Software Supply Chains" IEEE Software 2022

make defconfig # configuration
make # build the kernel (binary) out of config
make # should be the same, right?
# Options Matter: Documenting and Fixing Non-Reproducible Builds in Highly-Configurable Systems Randrianaina, Khelladi, Zendra, Acher MSR'2024

also at FOSDEM 2024 https://fosdem.org/2024/schedule/event/fosdem-2024-2848-documenting-and-fixing-non-reproducible-builds-due-to-configuration-options/

<figure><img src="image_291.png" /><figcaption>Table 2: Identified options and their category.</figcaption></figure>

<table><thead><tr><td>Option</td><td>Category</td></tr></thead><tbody><tr><td>MODULE_SIG_SHA1</td><td>Module Signing</td></tr><tr><td>MODULE_SIG_SHA224</td><td>Module Signing</td></tr><tr><td>MODULE_SIG_SHA256</td><td>Module Signing</td></tr><tr><td>MODULE_SIG_SHA384</td><td>Module Signing</td></tr><tr><td>MODULE_SIG_SHA512</td><td>Module Signing</td></tr><tr><td>MODULE_SIG</td><td>Module Signing</td></tr><tr><td>GCOV_PROFILE_FTRACE</td><td>Profiling</td></tr><tr><td>GCOV_PROFILE_ALL</td><td>Profiling</td></tr><tr><td>DEBUG_INFOSuspIT</td><td>Debug Info</td></tr><tr><td>DEBUG_INFOREDUCED</td><td>Debug Info</td></tr></tbody></table>

<figure><img src="image_292.png" /></figure>
# Options Matter: Documenting and Fixing Non-Reproducible Builds in Highly-Configurable Systems Randrianaina, Khelladi, Zendra, Acher MSR'2024

also at FOSDEM 2024 https://fosdem.org/2024/schedule/event/fosdem-2024-2848-documenting-and-fixing-non-reproducible-builds-due-to-configuration-options/

<figure><img src="image_323.png" /><figcaption>Table 2: Identified options and their category.</figcaption></figure>

<figure><img src="image_324.png" /></figure>

<figure><img src="image_325.png" /></figure>
## #1 take away message: look at every variability layer when you want a bit-to-bit reproducibility; don't ignore compile-time options!

<figure><img src="image_293.png" /></figure>

<figure><img src="image_294.png" /></figure>

"The build process of a software product is reproducible if, after designating a specific version **and a specific variant** of its source code and all of its build dependencies, every build produces bit-for-bit identical artifacts, no matter the environment in which the build is performed." Lamb and Zacchiroli "Reproducible Builds: Increasing the Integrity of Software Supply Chains" IEEE Software 2022
## #1 take away message: look at every variability layer when you want a bit-to-bit reproducibility; don't ignore compile-time options!

<figure><img src="image_326.png" /></figure>

<figure><img src="image_327.png" /></figure>

"The build process of a software product is reproducible if, after designating a specific version **and a specific variant** of its source code and all of its build dependencies, every build produces bit-for-bit identical artifacts, no matter the environment in which the build is performed." Lamb and Zacchiroli "Reproducible Builds: Increasing the Integrity of Software Supply Chains" IEEE Software 2022
## #2 take away message: interactions across variability layers exist (eg compile-time option with build path) and may hamper reproducibility

Busybox. To pinpoint the source of non-reproducible builds, the typical workflow is to slightly vary the build environment. Changing the build path between two builds of the same configuration for Busybox impacts 49.75% of the configurations, causing their build to be non-reproducible (presented in Figure 2 under the name Busybox (alter)). The decision tree identifies the option involved which is DEBUG. In fact, this option includes some debug information in the binary including the build path. Thus, interactions exist between configuration options and build environment. This can be solved in two ways, either by disabling the option, or not changing the build environment. There, a trade-off is to be made because the developer may need DEBUG and require the build to be reproducible. Building in the same directory solved the issue and the configurations we have picked for Busybox are reproducible at 100% as shown in Figure 2. Overall, altering the build environment in Busybox identified the DEBUG option as key to achieving 100% reproducibility, either by disabling it or maintaining a consistent build path.

<figure><img src="image_295.png" /></figure>

"The build process of a software product is reproducible if, after designating a specific version **and a specific variant** of its source code and all of its build dependencies, every build produces bit-for-bit identical artifacts, no matter the environment in which the build is performed." Lamb and Zacchiroli "Reproducible Builds: Increasing the Integrity of Software Supply Chains" IEEE Software 2022
## #2 take away message: interactions across variability layers exist (eg compile-time option with build path) and may hamper reproducibility

Busybox. To pinpoint the source of non-reproducible builds, the typical workflow is to slightly vary the build environment. Changing the build path between two builds of the same configuration for Busybox impacts 49.75% of the configurations, causing their build to be non-reproducible (presented in Figure 2 under the name Busybox (alter)). The decision tree identifies the option involved which is DEBUG. In fact, this option includes some debug information in the binary including the build path. Thus, interactions exist between configuration options and build environment. This can be solved in two ways, either by disabling the option, or not changing the build environment. There, a trade-off is to be made because the developer may need DEBUG and require the build to be reproducible. Building in the same directory solved the issue and the configurations we have picked for Busybox are reproducible at 100% as shown in Figure 2. Overall, altering the build environment in Busybox identified the DEBUG option as key to achieving 100% reproducibility, either by disabling it or maintaining a consistent build path.

<figure><img src="image_328.png" /></figure>

"The build process of a software product is reproducible if, after designating a specific version **and a specific variant** of its source code and all of its build dependencies, every build produces bit-for-bit identical artifacts, no matter the environment in which the build is performed." Lamb and Zacchiroli "Reproducible Builds: Increasing the Integrity of Software Supply Chains" IEEE Software 2022
* Linux as a subject software system (not as an OS interacting with other layers)

* Targeted non-functional, quantitative property: **binary size**

* interest for maintainers/users of the Linux kernel (embedded systems, cloud, etc.)

* challenging to predict (cross-cutting options, interplay with compilers/build systems, etc./)

* Dataset: version 4.13.3 (september 2017), x86_64 arch, measurements of 95K+ random configurations

* paranoiac about deep variability since 2017, Docker to control the build environment and scale

* diversity of binary sizes: from 7Mb to 1.9Gb

* 6% MAPE errors: quite good, though costly...

<figure><img src="image_296.png" /></figure>

<figure><img src="image_297.png" /></figure>

<figure><img src="image_298.png" /></figure>

<figure><img src="image_299.png" /></figure>
* Linux as a subject software system (not as an OS interacting with other layers)

* Targeted non-functional, quantitative property: **binary size**

* interest for maintainers/users of the Linux kernel (embedded systems, cloud, etc.)

* challenging to predict (cross-cutting options, interplay with compilers/build systems, etc./)

* Dataset: version 4.13.3 (september 2017), x86_64 arch, measurements of 95K+ random configurations

* paranoiac about deep variability since 2017, Docker to control the build environment and scale

* diversity of binary sizes: from 7Mb to 1.9Gb

* 6% MAPE errors: quite good, though costly...

<figure><img src="image_329.png" /></figure>

<figure><img src="image_330.png" /></figure>

<figure><img src="image_331.png" /></figure>

<figure><img src="image_332.png" /></figure>
<figure><img src="image_300.png" /></figure>

<table><caption>Table I: Dataset properties for each version. The number of deleted/new features, delta commits, files changes are w.r.t. 4.13.</caption><thead><tr><th>Version</th><th>Release Date</th><th>LOC</th><th>Files</th><th>Examples</th><th>Seconds/config</th><th>Options</th><th>Features</th><th>Deleted features</th><th>New features</th><th>ΔCommits</th><th>Files changes</th></tr></thead><tbody><tr><td>4.13</td><td>2017/09/03</td><td>16,616,534</td><td>60,530</td><td>92,562</td><td>not available</td><td>12,776</td><td>9,468</td><td>-</td><td>-</td><td>31,052</td><td>934,628</td></tr><tr><td>4.15</td><td>2018/01/28</td><td>17,073,368</td><td>62,249</td><td>39,391</td><td>not available</td><td>12,998</td><td>9,425</td><td>342</td><td>299</td><td>104,691</td><td>1,972,020</td></tr><tr><td>4.20</td><td>2018/12/23</td><td>17,526,171</td><td>62,423</td><td>23,489</td><td>225</td><td>13,533</td><td>10,189</td><td>468</td><td>1,189</td><td>118,778</td><td>2,170,935</td></tr><tr><td>5.0</td><td>2019/03/03</td><td>17,679,372</td><td>63,076</td><td>19,952</td><td>247</td><td>13,673</td><td>10,293</td><td>494</td><td>1,319</td><td>181,308</td><td>3,827,025</td></tr><tr><td>5.4</td><td>2019/10/24</td><td>19,358,903</td><td>67,915</td><td>25,847</td><td>285</td><td>14,159</td><td>10,813</td><td>663</td><td>2,008</td><td>225,804</td><td>4,393,117</td></tr><tr><td>5.7</td><td>2020/05/31</td><td>19,358,903</td><td>67,915</td><td>20,159</td><td>258</td><td>14,586</td><td>11,338</td><td>715</td><td>2,585</td><td>242,381</td><td>4,681,313</td></tr><tr><td>5.8</td><td>2020/08/02</td><td>19,729,197</td><td>69,303</td><td>21,923</td><td>289</td><td>14,817</td><td>11,530</td><td>730</td><td>2,792</td><td></td><td></td></tr></tbody></table>

4.13 version (sep 2017): 6%. What about **evolution?** Can we reuse the 4.13 Linux prediction model? No, accuracy quickly decreases: **4.15 (5 months after): 20%; 5.7 (3 years after): 35%**

<figure><img src="image_301.png" /></figure>

<figure><img src="image_302.png" /></figure>
<figure><img src="image_333.png" /></figure>

<table><caption>Table I: Dataset properties for each version. The number of deleted/new features, delta commits, files changes are w.r.t. 4.13.</caption><thead><tr><th>Version</th><th>Release Date</th><th>LOC</th><th>Files</th><th>Examples</th><th>Seconds/config</th><th>Options</th><th>Features</th><th>Deleted features</th><th>New features</th><th>ΔCommits</th><th>Files changes</th></tr></thead><tbody><tr><td>4.13</td><td>2017/09/03</td><td>16,616,534</td><td>60,530</td><td>92,562</td><td>not available</td><td>12,776</td><td>9,468</td><td>-</td><td>-</td><td>-</td><td>934,628</td></tr><tr><td>4.15</td><td>2018/01/28</td><td>17,073,368</td><td>62,249</td><td>39,391</td><td>not available</td><td>12,998</td><td>9,425</td><td>342</td><td>299</td><td>31,052</td><td>1,972,020</td></tr><tr><td>4.20</td><td>2018/12/23</td><td>17,526,171</td><td>62,423</td><td>23,489</td><td>225</td><td>13,533</td><td>10,189</td><td>468</td><td>1,189</td><td>104,691</td><td>2,170,935</td></tr><tr><td>5.0</td><td>2019/03/03</td><td>17,679,372</td><td>63,076</td><td>19,952</td><td>247</td><td>13,673</td><td>10,293</td><td>494</td><td>1,319</td><td>118,778</td><td>3,827,025</td></tr><tr><td>5.4</td><td>2019/10/24</td><td>19,358,903</td><td>67,915</td><td>25,847</td><td>285</td><td>14,159</td><td>10,813</td><td>663</td><td>2,008</td><td>181,308</td><td>4,393,117</td></tr><tr><td>5.7</td><td>2020/05/31</td><td>19,358,903</td><td>67,915</td><td>20,159</td><td>258</td><td>14,586</td><td>11,338</td><td>715</td><td>2,585</td><td>225,804</td><td>4,681,313</td></tr><tr><td>5.8</td><td>2020/08/02</td><td>19,729,197</td><td>69,303</td><td>21,923</td><td>289</td><td>14,817</td><td>11,530</td><td>730</td><td>2,792</td><td>242,381</td><td>4,681,313</td></tr></tbody></table>

4.13 version (sep 2017): 6%. What about **evolution?** Can we reuse the 4.13 Linux prediction model? No, accuracy quickly decreases: **4.15 (5 months after): 20%; 5.7 (3 years after): 35%**

<figure><img src="image_334.png" /></figure>

<figure><img src="image_335.png" /></figure>
Solution #3 Transfer learning (reuse of knowledge)

* Mission Impossible: Saving variability knowledge and prediction model 4.13 (15K hours of computation)

* Heterogeneous transfer learning: the feature space is different

* TEAMS: transfer evolution-aware model shifting

<figure><img src="image_303.png" /></figure>

<figure><img src="image_304.png" /></figure>

<figure><img src="image_305.png" /></figure>
## Solution #3 Transfer learning (reuse of knowledge)

* Mission Impossible: Saving variability knowledge and prediction model 4.13 (15K hours of computation)

* Heterogeneous transfer learning: the feature space is different

* TEAMS: transfer evolution-aware model shifting

<figure><img src="image_336.png" /></figure>

<figure><img src="image_337.png" /></figure>

<figure><img src="image_338.png" /></figure>
# Solution #3 Transfer learning (con't)

Luc Lesoil, Helge Spieker, Arnaud Gotlieb, Mathieu Acher, Paul Temple, Arnaud Blouin, Jean-Marc Jézéquel:
Learning input-aware performance models of configurable systems: An empirical evaluation. J. Syst. Softw. 208: 111883 (2024)

<figure><img src="image_306.png" /><figcaption>Figure 1: The performance prediction problem: how to predict software performance considering both configurations and inputs?</figcaption></figure>

<table><thead><tr><td>Approach</td><td>Description of the approach</td><td>Offline cost (Offshore Organization)</td><td>Online measurement cost (User)</td><td>Input properties</td></tr></thead><tbody><tr><td>Supervised online learning</td><td>Train performance model on demand, from scratch, each time a new input is fed to the configurable system</td><td>None</td><td>High</td><td>No</td></tr><tr><td>Offline learning</td><td>Use a pre-trained model over measurements of multiple configurations and inputs. Input properties are used to make the prediction (in an online setting).</td><td>High</td><td>None</td><td>Yes</td></tr><tr><td>Transfer learning</td><td>Adapt a pre-trained model for a new targeted input. It requires to gather fresh measurements of some configurations over the input (in an online setting).</td><td>Medium</td><td>Medium</td><td>Yes</td></tr></tbody></table>
# Solution #3 Transfer learning (con't)

Luc Lesoil, Helge Spieker, Arnaud Gotlieb, Mathieu Acher, Paul Temple, Arnaud Blouin, Jean-Marc Jézéquel:
Learning input-aware performance models of configurable systems: An empirical evaluation. J. Syst. Softw. 208: 111883 (2024)

<figure><img src="image_339.png" /><figcaption>Figure 1: The performance prediction problem: how to predict software performance considering both configurations and inputs?</figcaption></figure>

<table><thead><tr><td>Approach</td><td>Description of the approach</td><td>Offline cost (Offshore Organization)</td><td>Online measurement cost (User)</td><td>Input properties</td></tr></thead><tbody><tr><td>Supervised online learning</td><td>Train performance model on demand, from scratch, each time a new input is fed to the configurable system</td><td>None</td><td>High</td><td>No</td></tr><tr><td>Offline learning</td><td>Use a pre-trained model over measurements of multiple configurations and inputs. Input properties are used to make the prediction (in an online setting).</td><td>High</td><td>None</td><td>Yes</td></tr><tr><td>Transfer learning</td><td>Adapt a pre-trained model for a new targeted input. It requires to gather fresh measurements of some configurations over the input (in an online setting).</td><td>Medium</td><td>Medium</td><td>Yes</td></tr></tbody></table>
# Is there an interplay between compile-time and runtime options?

<figure><img src="image_307.png" /></figure>

L. Lesoil, M. Acher, X. Tèrnava, A. Blouin and J.-M. Jézéquel "The Interplay of Compile-time and Run-time Options for Performance Prediction" in SPLC '21

<figure><img src="image_308.png" /></figure>

This paper investigates how compile-time options can affect software performances and how compile-time options interact with run-time options.

<figure><img src="image_309.png" /><figcaption>**Figure 1: Cross-layer variability of x264**</figcaption></figure>
# Is there an interplay between compile-time and runtime options?

<figure><img src="image_340.png" /></figure>

L. Lesoil, M. Acher, X. Tèrnava, A. Blouin and J.-M. Jézéquel "The Interplay of Compile-time and Run-time Options for Performance Prediction" in SPLC '21

<figure><img src="image_341.png" /></figure>

This paper investigates how compile-time options can affect software performances and how compile-time options interact with run-time options.

<figure><img src="image_342.png" /><figcaption>Figure 1: Cross-layer variability of x264</figcaption></figure>
# Solution #4: Leverage stability across variability layers!

**L. Lesoil**, M. Acher, X. Tèrnava, A. Blouin and J.-M. Jézéquel "The Interplay of Compile-time and Run-time Options for Performance Prediction" in SPLC '21

<figure><img src="image_310.png" /><figcaption>(c) x264, I#4 (lyric video), encoding time</figcaption></figure>

**First good news:** *Worth tuning software at compile-time!*

**Second good news:** For all the execution time distributions of x264 and all the input videos, the worst correlation is greater than 0.97. If the compile-time options change the scale of the distribution, they do not change the rankings of run-time configurations (i.e., they do not truly interact with the run-time options).

**It has three practical implications:**

1. **Reuse of configuration knowledge:** transfer learning of prediction models boils down to apply a linear transformation among distributions. Users can also trust the documentation of run-time options, consistent whatever the compile-time configuration is.

2. **Tuning at lower cost:** finding the best compile-time configuration among all the possible ones allows one to immediately find the best configuration at run time. We can remove away

3. **Measuring at lower cost:** do not use a default compile-time configuration, use it and will generalize!

Did we recommend to use two binaries? YES, one for measuring, another for performances!

<figure><img src="image_311.png" /></figure>
# Solution #4: Leverage stability across variability layers!

**L. Lesoil**, M. Acher, X. Tèrnava, A. Blouin and J.-M. Jézéquel "The Interplay of Compile-time and Run-time Options for Performance Prediction" in SPLC '21

<figure><img src="image_343.png" /><figcaption>(c) x264, I#4 (lyric video), encoding time</figcaption></figure>

**First good news:** *Worth tuning software at compile-time!*

**Second good news:** For all the execution time distributions of x264 and all the input videos, the worst correlation is greater than 0.97. If the compile-time options change the scale of the distribution, they do not change the rankings of run-time configurations (i.e., they do not truly interact with the run-time options).

**It has three practical implications:**

1. **Reuse of configuration knowledge:** transfer learning of prediction models boils down to apply a linear transformation among distributions. Users can also trust the documentation of run-time options, consistent whatever the compile-time configuration is.

2. **Tuning at lower cost:** finding the best compile-time configuration among all the possible ones allows one to immediately find the best configuration at run time. We can remove away

3. **Measuring at lower cost:** do not use a default compile-time configuration, use it and will generalize!

Did we recommend to use two binaries? YES, one for measuring, another for performances!

<figure><img src="image_344.png" /></figure>
interplay between compile-time and runtime options and even input!

**L. Lesoil**, M. Acher, X. Tèrnava, A. Blouin and J.-M. Jézéquel "The Interplay of Compile-time and Run-time Options for Performance Prediction" in SPLC '21

<figure><img src="image_312.png" /></figure>

(c) x264, I #4 (lyric video), encoding time

Key results

First good new. Worth the

Second good new correlation is created change the rankings

It has three practical implications:

<figure><img src="image_313.png" /></figure>

xz

performances!

transfer learning of a single user's data. Users can adjust the configuration is.

The best compile-time configuration among all the possible ones allows us to configure at run time. We can remove awards from default compile time configuration use

<figure><img src="image_314.png" /></figure>

<figure><img src="image_315.png" /></figure>

<figure><img src="image_316.png" /></figure>

<figure><img src="image_317.png" /></figure>
interplay between compile-time and runtime options and even input!

**L. Lesoil**, M. Acher, X. Tèrnava, A. Blouin and J.-M. Jézéquel "The Interplay of Compile-time and Run-time Options for Performance Prediction" in SPLC '21

<figure><img src="image_345.png" /></figure>

(c) x264, J #4 (lyric video), encoding time

Poppler

Key results

First good new. Worth the

Second good news: all the input videos, the worst correlation is created when you scale the distribution, they do not change the rankings even with the runtime options).

It has three practical implications:

<figure><img src="image_346.png" /></figure>

xz

transfer learning or just copy the configuration is. It does not apply a clear cut rule for choosing the best compile-time configuration among all the possible ones allows us to use a default compile time configuration, use a

default compile-time configuration at run time. We can remove awards from the results and use a default compile time configuration, use a

performances!

<figure><img src="image_347.png" /></figure>

<figure><img src="image_348.png" /></figure>

<figure><img src="image_349.png" /></figure>

<figure><img src="image_350.png" /></figure>
What
is your move?

<figure><img src="image_318.png" /></figure>

<figure><img src="image_319.png" /></figure>
What
is your move?

<figure><img src="image_351.png" /></figure>

<figure><img src="image_352.png" /></figure>
[Event "FIDE World Championship Match 2024"]

[Site "Los Angeles, USA"]

[Date "2024.12.01"]

[Round "5"]

[White "Kramnik, Vladimir"]

[Black "Nepomniachtchi, Ian"]

[Result "1-0"]

[WhiteElo "2900"]

[BlackElo "2900"]

[TimeControl

"40/7200:20/3600:900+30"]

[UTCDate "2024.11.27"]

[UTCTime "09:01:25"]

[Variant "Standard"]

[Event "FIDE World Championship Match 2024"]

[Site "Los Angeles, USA"]

[Date "2024.12.01"]

[Round "5"]

[White "Louapre, David"]

[Black "Giraud, Thibaut"]

[Result "0-1"]

[WhiteElo "1400"]

[BlackElo "1400"]

[TimeControl

"40/7200:20/3600:900+30"]

[UTCDate "2024.11.27"]

[UTCTime "09:01:25"]

[Variant "Standard"]

What is your prompt?

<figure><img src="image_320.png" /></figure>

<figure><img src="image_321.png" /></figure>
[Event "FIDE World Championship Match 2024"]
[Site "Los Angeles, USA"]
[Date "2024.12.01"]
[Round "5"]
[White "Kramnik, Vladimir"]
[Black "Nepomniachtchi, Ian"]
[Result "1-0"]
[WhiteElo "2900"]
[BlackElo "2900"]
[TimeControl
"40/7200:20/3600:900+30"]
[UTCDate "2024.11.27"]
[UTCTime "09:01:25"]
[Variant "Standard"]

[Event "FIDE World Championship Match 2024"]
[Site "Los Angeles, USA"]
[Date "2024.12.01"]
[Round "5"]
[White "Louapre, David"]
[Black "Giraud, Thibaut"]
[Result "0-1"]
[WhiteElo "1400"]
[BlackElo "1400"]
[TimeControl
"40/7200:20/3600:900+30"]
[UTCDate "2024.11.27"]
[UTCTime "09:01:25"]
[Variant "Standard"]

What is your prompt?

<figure><img src="image_353.png" /></figure>

<figure><img src="image_354.png" /></figure>
<figure><img src="image_322.png" /></figure>

MrPhi

@MonsieurPhi

...

Petite note: j'ai remarqué que jusqu'à 1800 Elo, sa sensibilité au prompt est un peu bizarre. Par exemple avec "1-0" (Blanc gagne) il joue le coup correct g6. Mais avec "0-1" (Noir gagne) ou "1/2-1/2", il joue Nf6 (ce qui est illogique vu que Nf6 fait justement perdre les Noirs).

Translate post

<figure><img src="image_323.png" /></figure>

11:55 AM · Apr 19, 2024 · 447 Views

<figure><img src="image_324.png" /></figure>

<figure><img src="image_325.png" /></figure>

<figure><img src="image_326.png" /></figure>

<figure><img src="image_327.png" /></figure>

<figure><img src="image_328.png" /></figure>

<figure><img src="image_329.png" /></figure>

Post your reply

Reply

<figure><img src="image_330.png" /></figure>

MrPhi @MonsieurPhi · Apr 19

Mais il n'a plus ce comportement bizarre quand le Elo dépasse 2000. Enfin bref ! Il faudrait faire d'autres tests pour déterminer à quel point cette information sur le niveau Elo influence son niveau de jeu, mais c'est déjà intéressant comme petite expérience.

<figure><img src="image_331.png" /></figure>

<figure><img src="image_332.png" /></figure>

<figure><img src="image_333.png" /></figure>

<figure><img src="image_334.png" /></figure>

<figure><img src="image_335.png" /></figure>

<figure><img src="image_336.png" /></figure>

<figure><img src="image_337.png" /></figure>

<figure><img src="image_338.png" /></figure>