<figure><img src="image_1.png" /></figure>

HAL
open science

Reproducible Science and Software Engineering

Mathieu Acher, Paul Temple, Olivier Barais

► To cite this version:

Mathieu Acher, Paul Temple, Olivier Barais. Reproducible Science and Software Engineering. Doctoral. Valence, EJCP 2023, France. 2023, pp.140. hal-04152637v2

HAL Id: hal-04152637

https://inria.hal.science/hal-04152637v2

Submitted on 6 Jul 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Reproducible Science and Software Engineering

Mathieu Acher @acherm

Paul Temple @pleupi22

Olivier Barais @barais
# Reproductibilité des résultats et génie logiciel empirique

**Résumé:** L'une des principales promesses des logiciels est qu'un résultat obtenu par une expérience (par exemple, une simulation) puisse être reproduit avec un degré élevé de concordance. La recherche de la reproductibilité a un impact sur différents domaines scientifiques, prend différentes formes et nécessite de rendre toutes les données et le code disponibles de manière que les calculs puissent être exécutés à nouveau avec des résultats identiques. Dans ce cours, nous passerons d'abord en revue les terminologies (par exemple, reproductible vs réplication vs répétition), les outils de base (par exemple, les systèmes de version, les "build systems", les gestionnaires de paquets, les "notebooks"), et les techniques (par exemple, les tests automatisés, l'intégration et le déploiement continus, la gestion de la configuration) liés à la science reproductible.

Par la suite, nous programmerons une fonctionnalité relativement simple dans différentes variantes et démontrerons que de nombreux facteurs (y compris les langages de programmation, les versions de bibliothèques, les compilateurs, les types de variables, la gestion de l'aléatoire, etc.) peuvent avoir un impact sur le résultat final. Grâce à cet exercice, nous souhaitons présenter et discuter des techniques logicielles, des méthodologies et des outils que les développeurs ou les scientifiques peuvent utiliser pour aborder et atténuer les problèmes de reproductibilité, en espérant que cela conduise à des résultats plus robustes et plus généraux.

École Jeunes Chercheuses et Jeunes
Chercheurs en Programmation (EJCP)

L'école du GDR GPL.
# Reproducible Science and Software Engineering

**Abstract:** One of the main promises of software is that a result obtained from an experiment (e.g. a simulation) can be reproduced with a high degree of concordance. The quest for reproducibility has an impact on different scientific fields, takes different forms and requires making all the data and code available so that calculations can be run again with identical results. In this course, we will first review terminologies (e.g. reproducible vs replication vs repeatability), basic tools (e.g. versioning systems, build systems, package managers, notebooks), and techniques (e.g. automated testing, continuous integration and deployment, configuration management) related to reproducible science.

We will then program a relatively simple feature in different variants and demonstrate that many factors (including programming languages, library versions, compilers, variable types, randomness management, etc.) can have an impact on the final result. Through this exercise, we aim to present and discuss software techniques, methodologies and tools that developers or scientists can use to address and mitigate reproducibility issues, with the goal of leading to more robust and general results.

École Jeunes Chercheuses et Jeunes
Chercheurs en Programmation (EJCP)

L'école du GDR GPL.
# About us...

Mathieu Acher https://www.mathieuacher.com/
Professeur INSA Rennes

Thème de recherche principal : la variabilité logicielle et les systèmes hautement
configurables comme le kernel Linux ou les chaînes de traitements scientifiques.

SE+AI

INSA

<figure><img src="image_3.png" /></figure>

institut
universitaire
de France

Paul Temple est maitre de conférences à l'Université de Rennes. Il a obtenu sa thèse en 2018 en étudiant le problème de prédiction de performances des systèmes configurables tout en y ajoutant une dimension supplémentaire qui est l'ensemble d'entrées à fournir à ces systèmes. Postdoc à l'UNamur, Namur, Belgique sous la supervision de Gilles Perrouin et de Patrick Heymans.

Olivier Barais, Professeur à l'Université de Rennes, responsable de l'équipe DiverSE
https://olivier.barais.fr/

<figure><img src="image_4.png" /></figure>

Université
de Rennes

<figure><img src="image_5.png" /></figure>

DiverSE

University-Centric Software Engineering

Inria

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

ENS
# AGENDA

## Reproducible Science and (Deep) Software (Variability)

- Deep Software Variability

- Evidence of Deep Software Variability in Science

- Threats and Opportunities
# AGENDA

## Interactive course: Reproducible Science and (Deep) Software (Variability)

Is $(x+y)+z == x+(y+z)$? (TP1)

A suspicious paper in machine learning... (TP2)
SOFTWARE VARIANTS
ARE EATING THE WORLD

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

OBS
Open Broadcaster Software*

NETFLIX

twitch

<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

CIOUCUD

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

Lichess 2021
SOFTWARE IS EATING SCIENCE

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /><figcaption>Climate is changing</figcaption></figure>

<figure><img src="image_28.png" /></figure>

Obtained from www.nod33.com - 2019-01-10 19:51:19

<figure><img src="image_29.png" /><figcaption>The sea level is rising</figcaption></figure>

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>
# Computational science depends on software and its engineering

<figure><img src="image_32.png" /></figure>

* design of mathematical model
* mining and analysis of data
* executions of large simulations
* problem solving
* executable paper

from a set of scripts to automate the deployment to... a
comprehensive system containing several features that
help researchers exploring various hypotheses

Climate is changing

<figure><img src="image_33.png" /></figure>

The sea level is rising

<figure><img src="image_34.png" /></figure>
# Computational science depends on software and its engineering

<figure><img src="image_35.png" /></figure>

- multi-million line of code base

- multi-dependencies

- multi-systems

- multi-layer

- multi-version

- multi-person

- multi-variant

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses

<figure><img src="image_36.png" /><figcaption>The sea level is rising</figcaption></figure>

<figure><img src="image_37.png" /></figure>
# Computational science depends on software and its engineering

* multi-million line of code base

* multi-dependencies

* multi-systems

* multi-layer

* multi-version

* multi-person

* multi-variant

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

## Dealing with software collapse: software stops working eventually

Konrad Hinsen 2019

Configuration failures represent one of the most common types of software failures Sayagh et al. TSE 2018

```
In [6]: sess.run(tf.svd(tf_natrix))
Out[6]:
array([ 9.99998987e-01,  1.48747023e-03,  4.88133628e-06,
       4.69811084e-06,  4.37980998e-06,  3.45290823e-06,
       1.14686304e-06,  3.10980795e-06,  2.9752912e-06,
       2.65099743e-06,  1.91531706e-06,  0.00000000e+00,
       0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
       0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
       0.00000000e+00],
array([ 1.00000000e+00,  9.82503479e-05, -2.52892733e-06,
       6.43756948e-07, -0.6401498e-06, -3.18551632e-06,
           nan,          nan,          nan,
       0.00000000e+00,  0.00000000e+00,  0.00000000e+00,
       8.60028546e-10,  1.93595406e-09,  0.00000000e+00,
       -1.24836730e-09,  3.83645737e-09,  -3.90316446e-09,
           nan,          -7.00323994e-07],
[-7.27595761e-11,  7.15255737e-07,  -2.68207733e-02,
       -6.68754578e-01,  1.68675050e-01,  -2.37232931e-02,
           nan,          nan,          nan]```
# “Insanity is doing the same thing over and over again and expecting different results”

http://throwgrammarfromthetrain.blogspot.com/2010/10/definition-of-insanity.html

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

<figure><img src="image_45.png" /></figure>
# Reproducibility

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

(Claerbout/Donoho/Peng definition)

> "The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures." (~executable paper)

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /></figure>
# Reproducibility and Replicability

**Reproducible:** Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

**Replication:** A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses.

"Terminologies for Reproducible Research", Lorena A. Barba, 2018

<figure><img src="image_50.png" /></figure>

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>
# Reproducibility and Replicability

**Reproducible:** Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

**Replication:** A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses.

"Terminologies for Reproducible Research", Lorena A. Barba, 2018

The Claerbout/Donoho/Peng terminology is broadly disseminated across disciplines (see Table 2). But the recent adoption of an opposing terminology by two large professional groups—ACM and FASEB—make standardization awkward. The ACM publicizes its rationale for adoption as based on the International Vocabulary of Metrology, but a close reading of the sources makes this justification tenuous. The source of the FASEB adoption is unclear, but there's a chance that Casadevall and Fang (2010) had an influence there. They, in turn, based their definitions on the emphatic but essentially flawed work of Drummond (2009).

<table><caption>Table 2: Grouping of terminologies, as in Table 1, but by discipline.</caption><thead><tr><th>A</th><th>B1</th><th>B2</th></tr></thead><tbody><tr><td>political science</td><td>signal processing</td><td>microbiology, immunology (FASEB)</td></tr><tr><td>economics</td><td>scientific computing</td><td>computer science (ACM)</td></tr><tr><td></td><td>econometry</td><td></td></tr><tr><td></td><td>epidemiology</td><td></td></tr><tr><td></td><td>clinical studies</td><td></td></tr><tr><td></td><td>internal medicine</td><td></td></tr><tr><td></td><td>physiology (neuro)</td><td></td></tr><tr><td></td><td>computational biology</td><td></td></tr><tr><td></td><td>biomedical research</td><td></td></tr><tr><td></td><td>statistics</td><td></td></tr></tbody></table>

<figure><img src="image_54.png" /></figure>
# Reproducibility and Replicability

**Methods Reproducibility:** A method is reproducible if reusing the original code leads to the same results.

**Results Reproducibility:** A result is reproducible if a reimplementation of the method generates statistically similar values.

**Inferential Reproducibility:** A finding or a conclusion is reproducible if one can draw it from a different experimental setup.

“Unreproducible Research is Reproducible”, Bouthillier et al., ICML 2019

<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>
# Reproducible science

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

**Socio-technical issues:** open science, open source software, multi-disciplinary collaboration, incentives/rewards, initiatives, etc.

with many challenges related to data acquisition, knowledge organization/sharing, etc.

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>
# Reproducible science

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

**Socio-technical issues:** open science, open source software, multi-disciplinary collaboration, incentives/rewards, initiatives, etc. with many challenges related to data acquisition, knowledge organization/sharing, etc.

## EMSE Open Science Initiative

Openness in science is key to fostering progress via transparency, reproducibility, and replicability. Especially open data and open source are two fundamental pillars in open science as both build the core for excellence in evidence-based research. The Empirical Software Engineering Journal (EMSE) has therefore decided to explicitly foster open science and reproducible research by encouraging and supporting authors to share their (anonymised and curated) empirical data and source code in form of replication packages. The overall goals are:

* Increasing the transparency, reproducibility, and replicability of research endeavours. This supports the immediate credibility of authors' work, and it also provides a common basis for joint community efforts grounded on shared data.

* Building up an overall body of knowledge in the community leading to widely accepted and well-formed software engineering theories in the long run.

https://github.com/emsejournal/openscience

**Reproducible Science is good. Replicated Science is better.**

ReScience C is a *platinum open-access* peer-reviewed journal that targets computational research and encourages the explicit *replication* of already published research, promoting new and open-source implementations in order to ensure that the original research is *reproducible*. You can read about the ideas behind ReScience C in the article *Sustainable computational science: the ReScience initiative*

https://rescience.github.io/

https://reproducible-research.inria.fr/

OpenReview.net

<figure><img src="image_61.png" /></figure>

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>
# Reproducible science

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

**Socio-technical issues:** open science, open source software, multi-disciplinary collaboration, incentives/rewards, initiatives, etc.
with many challenges related to data acquisition, knowledge organization/sharing, etc.

<figure><img src="image_65.png" /></figure>

<figure><img src="image_66.png" /></figure>

## CALL FOR CHALLENGE CASES

Home / Call for Papers / Call for challenge cases

## ARTIFACT EVALUATION

Authors of accepted research papers are invited to submit the artifacts associated with the paper for evaluation. To do so, they should submit a PDF via Easychair (select the Research or Data Evaluation, or both, in the Evaluation Category). The PDF should contain a stable URL (or DOI) to the artifacts. The URL should contain the steps or general instructions to execute/analyze the artifact. Each artifact submission will be reviewed by at least two reviewers.

According to ACM's "Result and Artifact Review and Badging" policy, an " artifact " is "a digital object that was either created by the authors to be used as part of the study or generated by the experiment itself [ ... which can include ] software systems, scripts used to run experiments, input datasets, raw data collected in the experiment, or scripts used to analyze results."

A idea of the challenge track is to provide participants with a set of case studies that tackle relevant problems and challenge the state of the art. The challenge track happens in two phases. In the first phase, there will be a call for cases. Submitted cases will be reviewed by the challenge co-chairs to ensure that the information is clearly described. Accepted cases will be part of the official conference program.

In this track, an industrial performance dataset will be provided. The participants are invited to come up with research questions about the dataset, and study those. The challenge is open-ended: participants can choose the research questions that they find most interesting. The proposed approaches and/or tools and their findings are discussed in short papers, and presented in the main conference.
# Why should you make your software (work) reproducible and replicable?

Why not: hard to trust your results; hard to innovate on top of your work; impossible to refute hypothesis or verify new theories

Easier to redo computation when eg inputs change

Easier to inspect and document

Easier to evolve and maintain

Openness and transparency boost innovation

It can be seen as **engineering high-quality software**

It is an important effort, it's very hard! But worth, not to say mandatory...
What's your experience with reproducible science so far?
# Reproducible science

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

Despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results.

<figure><img src="image_67.png" /></figure>

<figure><img src="image_68.png" /></figure>

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses

Climate is changing

<figure><img src="image_69.png" /></figure>

The sea level is rising

<figure><img src="image_70.png" /></figure>
# Should software version numbers determine science?

> PLoS One. 2012;7(6):e38234. doi: 10.1371/journal.pone.0038234. Epub 2012 Jun 1.

The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements

Ed H B M Gronenschild ¹, Petra Habets, Heidi I L Jacobs, Ron Mengelers, Nico Rozendaal, Jim van Os, Machteld Marcelis

Significant differences were revealed between FreeSurfer version v5.0.0 and the two earlier **versions**. [...] About a factor two smaller differences were detected between Macintosh and Hewlett-Packard **workstations** and between **OSX 10.5 and OSX 10.6**. The observed differences are similar in magnitude as effect sizes reported in **accuracy** evaluations and neurodegenerative-studies.

<figure><img src="image_71.png" /></figure>

**see also** Krefting, D., Scheel, M., Freing, A., Specovius, S., Paul, F., and Brandt, A. (2011). "Reliability of quantitative neuroimage analysis using freesurfer in distributed environments," in *MICCAI Workshop on High-Performance and Distributed Computing for Medical Imaging*. (Toronto, ON).
“Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed.”

Reproducibility of neuroimaging analyses across operating systems, Glatard et al., Front. Neuroinform., 24 April 2015

<figure><img src="image_72.png" /></figure>

The implementation of mathematical functions manipulating single-precision **floating-point numbers in libmath** has evolved during the last years, leading to numerical differences in computational results. While these differences have little or no impact on simple analysis pipelines such as brain extraction and cortical tissue classification, their **accumulation** creates **important differences in longer pipelines** such as the subcortical tissue classification, RSfMRI analysis, and cortical thickness extraction.

<table><thead><tr><th></th><th>Cluster A</th><th>Cluster B</th></tr></thead><tbody><tr><td>Applications</td><td>Freesurfer 5.3.0, build 1<br>FSL 5.0.6, build 1<br>CIVET 1.1.12-UCSF, build 1</td><td>Freesurfer 5.3.0, build 1 and 2<br>FSL 5.0.6, build 1 and 2<br>CIVET 1.1.12-UCSF, build 1</td></tr><tr><td>Interpreters</td><td>Python 2.4.3, bash 3.2.25,<br>Perl 5.8.8, tcsh 6.14.00</td><td>Python 2.7.5, bash 4.2.47,<br>Perl 5.18.2, tcsh 6.18.01</td></tr><tr><td>glibc version</td><td>2.5</td><td>2.18</td></tr><tr><td>OS</td><td>CentOS 5.10</td><td>Fedora 20</td></tr><tr><td>Hardware</td><td>x86_64 CPUs (Intel Xeon)</td><td>x86_64 CPUs (Intel Xeon)</td></tr></tbody></table>
> “Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed.”

Reproducibility of neuroimaging analyses across operating systems, Glatard et al., Front. Neuroinform., 24 April 2015

Statically building programs improves reproducibility across OSes, but small differences may still remain when dynamic libraries are loaded by static executables[...]. When static builds are not an option, software heterogeneity might be addressed using virtual machines. However, such solutions are only workarounds: differences may still arise between **static executables built on different OSes**, or between **dynamic executables executed in different VMs**.

<figure><img src="image_73.png" /></figure>

<table><thead><tr><th></th><th>Cluster A</th><th>Cluster B</th></tr></thead><tbody><tr><td>Applications</td><td>Freesurfer 5.3.0, build 1<br>FSL 5.0.6, build 1<br>CIVET 1.1.12-UCSF, build 1</td><td>Freesurfer 5.3.0, build 1 and 2<br>FSL 5.0.6, build 1 and 2<br>CIVET 1.1.12-UCSF, build 1</td></tr><tr><td>Interpreters</td><td>Python 2.4.3, bash 3.2.25, Perl 5.8.8, tcsh 6.14.00</td><td>Python 2.7.5, bash 4.2.47, Perl 5.18.2, tcsh 6.18.01</td></tr><tr><td>glibc version</td><td>2.5</td><td>2.18</td></tr><tr><td>OS</td><td>CentOS 5.10</td><td>Fedora 20</td></tr><tr><td>Hardware</td><td>x86_64 CPUs (Intel Xeon)</td><td>x86_64 CPUs (Intel Xeon)</td></tr></tbody></table>
# Is it possible to reproduce a paper from 1987?

<figure><img src="image_74.png" /><figcaption>## lecun1989-repro</figcaption></figure>

<figure><img src="image_75.png" /></figure>

<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>

This code tries to reproduce the 1989 Yann LeCun et al. paper: *Backpropagation Applied to Handwritten Zip Code Recognition*. To my knowledge this is the earliest real-world application of a neural net trained with backpropagation (now 33 years ago).
# Is it possible to reproduce a paper from 1987?

<figure><img src="image_78.png" /></figure>

<figure><img src="image_79.png" /></figure>

## run

Since we don't have the exact dataset that was used in the paper, we take MNIST and randomly pick examples from it to generate an approximation of the dataset, which contains only 7291 training and 2007 testing digits, only of size 16x16 pixels (standard MNIST is 28x28).

https://github.com/karpathy/lecun1989-repro
# Is it possible to reproduce a paper from 1987?

## same results? well...

Now we can attempt to reproduce the paper. The original network trained for 3 days, but my (Apple Silicon M1) MacBook Air 33 years later chunks through it in about 90 seconds. (non-emulated arm64 but CPU only, I don't believe PyTorch and Apple M1 are best friends ever just yet, but anyway still about 3000X speedup). So now that we've run prepro we can run repro! (haha):

```
$ python repro.py
```

Running this prints (on the 23rd, final pass):

```
eval: split train. loss 4.073383e-03. error 0.62%. misses: 45
```

```
eval: split test . loss 2.838382e-02. error 4.09%. misses: 82
```

This is close but not quite the same as what the paper reports. To match the paper exactly we'd expect the following instead:

```
eval: split train. loss 2.5e-3. error 0.14%. misses: 10
```

```
eval: split test . loss 1.8e-2. error 5.00%. misses: 102
```
# Is it possible to reproduce a paper from 1987?

## huge variants/decisions space!

**Open questions:**

* The 12 -> 8 connections from H2 to H1 are not described in this paper... I will assume a sensible block structure connectivity

* Not clear what exactly is the "MSE loss". Was the scaling factor of 1/2 included to simplify the gradient calculation? Will assume no.

* What is the learning rate? I will run a sweep to determine the best one manually.

* Was any learning rate decay used? Not mentioned, I am assuming no.

* Was any weight decay used? not mentioned, assuming no.

* Is there a bug in the pdf where in weight init the fan in should have a square root? The pdf's formatting is a bit messed up. Assuming yes.

* The paper does not say, but what exactly are the targets? Assuming they are +1/-1 for pos/neg, as the output units have tanh too...

## hyperparameter?

## algorithm?

## implementation?

## NN architecture?

One more note on the weight init conundrum. Eg the "Kaiming init" is:

a = gain * sqrt(3 / fan_in)
-U(-a, a)

For tanh neurons the recommended gain is 5/3. So therefore we would have $a = sqrt(3) * 5 / 3 * sqrt(1 / fan_in) = 2.89 * sqrt(1 / fan_in)$, which is close to what the paper does (gain 2.4). So if the original work in fact did use a sqrt and the pdf is just formatted wrong, then the (modern) Kaiming init and the originally used init are pretty close.
# Reproducible science as a (deep) software variability problem

> "Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

Despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results.

<figure><img src="image_80.png" /></figure>

<figure><img src="image_81.png" /></figure>

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses

Climate is changing

<figure><img src="image_82.png" /></figure>

The sea level is rising

<figure><img src="image_83.png" /></figure>
deep software variability

software application variability

input data variability

build variability

compiler variability

container variability

hypervisor variability

operating system variability

hardware variability

Despite the availability of data and code, several studies report that the same data analyzed with **different software** can lead to **different results**

Many **layers** (operating system, third-party libraries, versions, workloads, compile-time options and flags, etc.) themselves subject to variability can alter the results.

**Reproducible science and deep software variability:** a threat and opportunity for scientific knowledge!

version variability
# General advice: #1 Open source software and tools

Open format over tools

Excel? Word? Evernote?

eg CSV+Pandas, Markdown, orgmode, Joplin

SAS, Matlab, Mathematica

eg Python, R, scilab

Dropbox, Google drive

eg git/gitlab, Zenodo, SoftwareHeritage
General advice: #2 Reproducibility mindset

Try to design your project and research in such a way it will be reproducible

reusable, modular, testable, parameterizable software

Try to document your effort and trace your decisions (eg cahier de laboratoires, versioning systems)

Don't wait the last minute to prepare a shareable repository (share it with your co-authors as soon as possible!)

From notebooks (more exploratory) to engineered, scalable software
<figure><img src="image_84.png" /></figure>

<figure><img src="image_85.png" /></figure>

<figure><img src="image_86.png" /></figure>

<figure><img src="image_87.png" /></figure>

<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /></figure>

<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /></figure>

<figure><img src="image_92.png" /></figure>

<figure><img src="image_93.png" /></figure>

<figure><img src="image_94.png" /></figure>

<figure><img src="image_95.png" /></figure>

<figure><img src="image_96.png" /></figure>

<figure><img src="image_97.png" /></figure>

In [1]:
```python
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
```

In [2]:
```python
# data to be used -> UCI heart disease
# from the university of California
# data downloaded via https://github.com/sagnikghoshc7/Heart-Disease-Prediction
# seems ok with the description from UCI -> https://archive.ics.uci.edu/dataset/45/heart+disease
```

In [3]:
```python
dataset = pd.read_csv('heart.csv')
X = dataset.drop('target', axis=1)
y = dataset['target'].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=1)
```

In [4]:
```python
# look if split looks okayish
print(X)
print(y)
print(X_train)
print(X_test)

count = np.sum(y_test == 1)
print("total number of instance in test: " + str(len(y_test)))
print(y_test)
print("total number of instance in test with class 1: " + str(count))
```

<table><thead><tr><th></th><th>age</th><th>sex</th><th>cp</th><th>trestbps</th><th>chol</th><th>fbs</th><th>restecg</th><th>thalach</th><th>exang</th><th>oldpeak</th><th>\</th></tr></thead><tbody><tr><td>0</td><td>63</td><td>1</td><td>3</td><td>145</td><td>233</td><td>1</td><td>0</td><td>150</td><td>0</td><td>2.3</td><td></td></tr><tr><td>1</td><td>37</td><td>1</td><td>2</td><td>130</td><td>250</td><td>0</td><td>1</td><td>187</td><td>0</td><td>3.5</td><td></td></tr><tr><td>2</td><td>41</td><td>0</td><td>1</td><td>130</td><td>204</td><td>0</td><td>0</td><td>172</td><td>0</td><td>1.4</td><td></td></tr><tr><td>3</td><td>56</td><td>1</td><td>1</td><td>120</td><td>236</td><td>0</td><td>1</td><td>178</td><td>0</td><td>0.8</td><td></td></tr><tr><td>4</td><td>57</td><td>0</td><td>0</td><td>120</td><td>354</td><td>0</td><td>1</td><td>163</td><td>1</td><td>0.6</td><td></td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td></td></tr><tr><td>298</td><td>57</td><td>0</td><td>0</td><td>140</td><td>241</td><td>0</td><td>1</td><td>123</td><td>1</td><td>0.2</td><td></td></tr><tr><td>299</td><td>45</td><td>1</td><td>3</td><td>110</td><td>264</td><td>0</td><td>1</td><td>132</td><td>0</td><td>1.2</td><td></td></tr><tr><td>300</td><td>68</td><td>1</td><td>0</td><td>144</td><td>193</td><td>1</td><td>1</td><td>141</td><td>0</td><td>3.4</td><td></td></tr><tr><td>301</td><td>57</td><td>1</td><td>0</td><td>130</td><td>131</td><td>0</td><td>1</td><td>115</td><td>1</td><td>1.2</td><td></td></tr><tr><td>302</td><td>57</td><td>0</td><td>1</td><td>130</td><td>236</td><td>0</td><td>0</td><td>174</td><td>0</td><td>0.0</td><td></td></tr></tbody></table>

<table><thead><tr><th></th><th>slope</th><th>ca</th><th>thal</th></tr></thead><tbody><tr><td>0</td><td>0</td><td>0</td><td>1</td></tr><tr><td>1</td><td>0</td><td>0</td><td>2</td></tr><tr><td>2</td><td>2</td><td>0</td><td>2</td></tr><tr><td>3</td><td>2</td><td>0</td><td>2</td></tr><tr><td>4</td><td>2</td><td>0</td><td>2</td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td></tr><tr><td>298</td><td>1</td><td>0</td><td>3</td></tr><tr><td>299</td><td>1</td><td>0</td><td>3</td></tr><tr><td>300</td><td>1</td><td>2</td><td>3</td></tr><tr><td>301</td><td>1</td><td>1</td><td>3</td></tr><tr><td>302</td><td>1</td><td>1</td><td>2</td></tr></tbody></table>
<figure><img src="image_98.png" /><figcaption># Git/Gitlab|Github</figcaption></figure>

input_sensitivity / results / RQ5 / RQ3 / RQ3.ipynb

<figure><img src="image_99.png" /></figure>

<figure><img src="image_100.png" /></figure>

* Jupyter Notebook 99.7%

* Python 0.1%

* Dockerfile 0.0%

* Shell 0.2%

<figure><img src="image_101.png" /></figure>

## Research question results

2 study how inputs affect (1) performance distributions could change in a negligible way, with input sensitivity, it is necessary to quantify how reflecting the software to its input data is worth the cost.

## Can we ignore input sensitivity?

Now much we can lose, we first define two scenarios. In this scenario, we just train a simple performance configuration according to the model, configure the relate

* S2 - *Ignoring input sensitivity.* In this scenario, we train a model on a given input i.e. the source input, and then predict the best configuration for this source input. If we ignore the threat of input sensitivity, we can easily reuse this model for any other input, including the target input defined in S1. Finally, we execute the software with the configuration predicted by our model on the target input.
**General advice: #3 Master your computing environment**

Don't assume "it works on my machine"

Track dependencies, versions, packages, tools

Ask yourself: "What if I have to reproduce my results?"

Ask your co-authors to reproduce your results (ideally throughout the work, not at the end)

Automate the synthesis of your computing environment

Test your computing environment!
Computing environment matters!

binaries

configure; make

reprozip

Docker

Guix

Nix
What's the problem of only sharing binaries and executable?
What's the problem of only sharing binaries and executable?

What's the problem of...

./configure; make; make install
<figure><img src="image_102.png" /></figure>

<figure><img src="image_103.png" /></figure>

<figure><img src="image_104.png" /></figure>

<figure><img src="image_105.png" /></figure>

<figure><img src="image_106.png" /></figure>

<figure><img src="image_107.png" /></figure>

<figure><img src="image_108.png" /></figure>

<figure><img src="image_109.png" /></figure>

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

<figure><img src="image_112.png" /></figure>

<figure><img src="image_113.png" /></figure>

<figure><img src="image_114.png" /></figure>

<figure><img src="image_115.png" /></figure>

<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>

<figure><img src="image_118.png" /></figure>

<figure><img src="image_119.png" /></figure>

<figure><img src="image_120.png" /></figure>

In [1]:
```python
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
```

In [2]:
```python
# data to be used -> UCI heart disease
# from the university of California
# data downloaded via https://github.com/sagnikhoshc7/Heart-Disease-Prediction
# seems ok with the description from UCI -> https://archive.ics.uci.edu/dataset/45/heart+disease
```

In [3]:
```python
dataset = pd.read_csv('heart.csv')
X = dataset.drop('target', axis=1)
y = dataset['target'].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=1)
```

In [4]:
```python
# look if split looks okayish
print(X)
print(y)
print(X_train)
print(X_test)

count = np.sum(y_test == 1)
print("total number of instance in test: " + str(len(y_test)))
print(y_test)
print("total number of instance in test with class 1: " + str(count))
```

<table><thead><tr><th></th><th>age</th><th>sex</th><th>cp</th><th>trestbps</th><th>chol</th><th>fbs</th><th>restecg</th><th>thalach</th><th>exang</th><th>oldpeak</th><th>\</th></tr></thead><tbody><tr><td>0</td><td>63</td><td>1</td><td>3</td><td>145</td><td>233</td><td>1</td><td>0</td><td>150</td><td>0</td><td>2.3</td><td></td></tr><tr><td>1</td><td>37</td><td>1</td><td>2</td><td>130</td><td>250</td><td>0</td><td>1</td><td>187</td><td>0</td><td>3.5</td><td></td></tr><tr><td>2</td><td>41</td><td>0</td><td>1</td><td>130</td><td>204</td><td>0</td><td>0</td><td>172</td><td>0</td><td>1.4</td><td></td></tr><tr><td>3</td><td>56</td><td>1</td><td>1</td><td>120</td><td>236</td><td>0</td><td>1</td><td>178</td><td>0</td><td>0.8</td><td></td></tr><tr><td>4</td><td>57</td><td>0</td><td>0</td><td>120</td><td>354</td><td>0</td><td>1</td><td>163</td><td>1</td><td>0.6</td><td></td></tr><tr><td>298</td><td>57</td><td>0</td><td>0</td><td>140</td><td>241</td><td>0</td><td>1</td><td>123</td><td>1</td><td>0.2</td><td></td></tr><tr><td>299</td><td>45</td><td>1</td><td>3</td><td>110</td><td>264</td><td>0</td><td>1</td><td>132</td><td>0</td><td>1.2</td><td></td></tr><tr><td>300</td><td>68</td><td>1</td><td>0</td><td>144</td><td>193</td><td>1</td><td>1</td><td>141</td><td>0</td><td>3.4</td><td></td></tr><tr><td>301</td><td>57</td><td>1</td><td>0</td><td>130</td><td>131</td><td>0</td><td>1</td><td>115</td><td>1</td><td>1.2</td><td></td></tr><tr><td>302</td><td>57</td><td>0</td><td>1</td><td>130</td><td>236</td><td>0</td><td>0</td><td>174</td><td>0</td><td>0.0</td><td></td></tr></tbody></table>

<table><thead><tr><th></th><th>slope</th><th>ca</th><th>thal</th></tr></thead><tbody><tr><td>0</td><td>0</td><td>0</td><td>1</td></tr><tr><td>1</td><td>0</td><td>0</td><td>2</td></tr><tr><td>2</td><td>2</td><td>0</td><td>2</td></tr><tr><td>3</td><td>2</td><td>0</td><td>2</td></tr><tr><td>4</td><td>2</td><td>0</td><td>2</td></tr><tr><td>...</td><td>...</td><td>...</td><td>...</td></tr><tr><td>298</td><td>1</td><td>0</td><td>3</td></tr><tr><td>299</td><td>1</td><td>0</td><td>3</td></tr><tr><td>300</td><td>1</td><td>2</td><td>3</td></tr><tr><td>301</td><td>1</td><td>1</td><td>3</td></tr><tr><td>302</td><td>1</td><td>1</td><td>2</td></tr></tbody></table>

# What's the problem with notebooks?
# Requirements (wishlist!)

- Source code under version control tools like git.

- Versions of all its dependencies (ideally the source code of dependencies)

- Compiler/interpreter and on the options it has been passed

- Environment variables

- Parameters of your application

- Ecosystem agnostic

- ...

<figure><img src="image_121.png" /></figure>
# Full computational reproducibility!

ReproZip can automatically pack your research along with all necessary data files, libraries, environment variables and options into a self-contained bundle. Then ReproZip can use that bundle to automatically set up the same original environment so anybody can reproduce the research on a different machine, without tracking down and installing the dependencies, or even having to run the same operating system!

LEARN MORE »

## No lock-in

ReproZip works locally on your computer, server, or high performance computing clusters. Once you make an `.rpz` bundle, you can reproduce it with a variety of unpacker plug-ins (e.g. Docker, Vagrant, Singularity) or in the cloud with `ReproServer`.

TRY OUT REPROZIP TODAY »

## Free & Open Source

The development of ReproZip has always been open source and free to use since the project began in 2013. We welcome and encourage anyone to submit pull requests or issues with any feature requests or bug reports!

CHECK OUT THE GITHUB REPO »

## Extensible

ReproZip has been used to capture a wide variety of digital materials, from desktop applications and scripts in various languages to dynamic websites to high performance computing experiments. You can then reproduce your work across various OS's!

EXPLORE SOME EXAMPLES »
<figure><img src="image_122.png" /></figure>

master

<figure><img src="image_123.png" /></figure>

input_sensitivity / replication / containers / nodejs / Dockerfile

<figure><img src="image_124.png" /></figure>

<figure><img src="image_125.png" /></figure>

llesoil replication containers

Code

Blame

20 lines (18 loc) · 669 Bytes

1 FROM fedora:33

2 # update packages

3 RUN dnf -y up --exclude=kernel --exclude=kernel-core --exclude=kernel-devel --exclude=kernel-headers --exclude=python --exclude=gcc-c++ --exclude=make

4 # https://github.com/nodejs/node/blob/master/BUILDING.md

5 RUN dnf install -y python gcc-c++ make wget nano

6 RUN uname -r >> infos \

7 && gcc --version >> infos \

8 && make -v >> infos \

9 && python --version >> infos

10 # clone nodejs

11 RUN wget https://github.com/nodejs/node/archive/refs/tags/v15.14.0.tar.gz \

12 && tar -xf v15.14.0.tar.gz

13 # command to compile node with the default configurations

14 RUN cd node-15.14.0 \

15 && ./configure \

16 && make -j4

17 # Add files to run the scripts

18 COPY ./ /
```nixos
with import <nixpkgs> {}; stdenv.mkDerivation {
    name = "hello";

    # Source Code
    # See: https://nixos.org/nixpkgs/manual/#ssec-unpack-phase
    src = ./src;

    # Dependencies
    # See: https://nixos.org/nixpkgs/manual/#ssec-stdenv-dependencies
    buildInputs = [ coreutils gcc ];

    # Build Phases
    # See: https://nixos.org/nixpkgs/manual/#sec-stdenv-phases
    configurePhase = ''
        declare -xp
    '';
    buildPhase = ''
        gcc "$src/hello.c" -o ./hello
    '';
    installPhase = ''
        mkdir -p "$out/bin"
        cp ./hello "$out/bin/"
    '';
}
```
(define-public hello
    (package
        (name "hello")
        (version "2.10")
        (source (origin
            (method url-fetch)
            (uri (string-append "mirror://gnu/hello/hello-" version
                                ".tar.gz"))
            (sha256
                (base32
                    "0ssi1wpaf7plaswqqjwigppsg5fyh99vd1b9kz17c9lng89ndq1i"))))
            (build-system gnu-build-system)
            (synopsis "Hello, GNU world: An example GNU package")
            (description
                "GNU Hello prints the message \"Hello, world!\" and then exits. It
                serves as an example of standard GNU coding practices. As such, it supports
                command-line arguments, multiple languages, and so on.")
            (home-page "https://www.gnu.org/software/hello/")
            (license gpl3+))))
**General advice: #4 Test and doubt about the quality of your software**

Don't assume “it works”

Test here means **automated test**: manually checking that a function is doing what you expect is OK for prototyping/exploring, but it's limited in terms of input, number of runs, and well it's hard to redo the effort for any evolution

small-scale: “unit” test of a function; test a script (eg that preprocesses data);
interaction between “modules” of your software

large-scale: test the whole processing chains

Test even the libraries and tools you depend on!
General advice: #5 Automate everything

Automate the retrieval of data

Automate the pre-process of data

Automate the build of software

Automate the running/measurement (incl. deployment) of software

Automate the synthesis of the computing environment

Automate the production of figures (numbers)

Automate the production of Figures/Tables/...

Automate your automation: generate programs that generate what you want
General advice: #6 Threats to validity

Report on threats to validity (see deep variability)

Describe how you mitigate the threats

Discuss threats!

Internally: think about how your threats relate to your software or data
General advice: #7 Explore the variants space

Make vary your experiments

Robustness of your results

Opportunity to find new paths/hypothesis/...

Sometimes called ablation studies or “sensitivity” studies or ...

It's much easier with a reproducible software stack ;)

<figure><img src="image_126.png" /></figure>
**General advice: #4 Test and doubt about the quality of your software**

Don't assume “it works”

Test here means **automated test**: manually checking that a function is doing what you expect is OK for prototyping/exploring, but it's limited in terms of input, number of runs, and well it's hard to redo the effort for any evolution

small-scale: “unit” test of a function; test a script (eg that preprocesses data);
interaction between “modules” of your software

large-scale: test the whole processing chains

Test even the libraries and tools you depend on!
# Testing Problem in a nutshell

* #1 Test Generation Problem (input)

* #2 Oracle Problem (output)

Input

System under
Study

Output
# Testing techniques for reproducible science

* #1 Test Generation Problem (input)

* #2 Oracle Problem (output)

Important to ensure high-quality software... Test your script, your module, your component, your workflow, your generator... Small-scale and large-scale!

Input

System under
Study

Output
# Reproducible Science as a Testing Problem

## #1 Test Generation Problem (input)

inputs: computing environment, parameters of an algorithm, versions of a library or tool, choice of a programming language

## #2 Oracle Problem (output)

we usually ignore the outcome! (open problems; open questions; new knowledge)

Input

System under
Study
.replicable)

Output
(scientific
result)
Let say we inject fault on purpose (~manual mutant)

**Listing 1.** A Python implementation of `bubble_sort` affected by a bug when swapping array elements.

```python
def bubble_sort(arr):
    length = len(arr)

    for i in range(length):
        for j in range(0, length - i - 1):
            if arr[j] > arr[j+1]:
                arr[j], arr[j+1] = arr[j+1], arr[i]

    return arr
```

<figure><img src="image_127.png" /></figure>
# Assertion-based Testing

**Listing 1.** A Python implementation of `bubble_sort` affected by a bug when swapping array elements.

```python
def bubble_sort(arr):
    length = len(arr)
    for i in range(length):
        for j in range(0, length - i - 1):
            if arr[j] > arr[j+1]:
                arr[j], arr[j+1] = arr[j+1], arr[i]

    return arr
```

<figure><img src="image_128.png" /></figure>

**Listing 2.** A manually-written unit test.

```python
arr = [3, 1, 2]
bubble_sort(arr)
assert arr == [1, 2, 3]  # AssertionError, ←
actual: [1, 2, 1]
```
# Strengths and Limits of Assertion-based Testing

* Precise oracles

* Domain-specific knowledge

* Examples-based

* Input test is limited by construction

* Incomplete (cases)

* Tests subject to developers bias

* Costly

* One specific oracle per example (labelling)

<caption>**Listing 2.** A manually-written unit test.</caption>

```perl
arr = [3, 1, 2]
bubble_sort(arr)
assert arr == [1, 2, 3] # AssertionError, ←
    actual: [1, 2, 1]
```
Differential Testing

<figure><img src="image_129.png" /></figure>

De
**Listing 1.** A Python implementation of `bubble_sort` affected by a bug when swapping array elements.

```python
def bubble_sort(arr):
    length = len(arr)
    for i in range(length):
        for j in range(0, length - i - 1):
            if arr[j] > arr[j+1]:
                arr[j], arr[j+1] = arr[j+1], arr[i]

    return arr
```

**Listing 3.** Differential testing using multiple implementations of sorting algorithms.

```python
sorting_algorithms = [bubble_sort, merge_sort, insertion_sort]

while True:
    arr = get_random_array()  # e.g., [3, 1, 2]
    sorted hook = [alg(arr.copy()) for alg in sorting_algorithms]
    all_same = all sorted hook[0] for sorted hook in sorted hook
    assert all_same
```
Metamorphic Testing

<figure><img src="image_130.png" /></figure>
**Listing 1.** A Python implementation of `bubble_sort` affected by a bug when swapping array elements.

```python
def bubble_sort(arr):
    length = len(arr)
    for i in range(length):
        for j in range(0, length - i - 1):
            if arr[j] > arr[j+1]:
                arr[j], arr[j+1] = arr[j+1], arr[i]

    return arr
```

**Listing 4.** Metamorphic testing by comparing whether the relative order is maintained for a smaller array.

```python
while True:
    arr = get_random_array()
    if len(arr) >= 1:
        sorted_arr = bubble_sort(arr.copy())
        random_elem = random.choice sorted_arr)
        arr.remove(random_elem)
        sorted_smaller_arr = bubble_sort(arr)
        sorted_arr.remove(random_elem)
        assert sorted_arr == sorted_smaller_arr
```
**Listing 1.** A Python implementation of `bubble_sort` affected by a bug when swapping array elements.

```python
def bubble_sort(arr):
    length = len(arr)
    for i in range(length):
        for j in range(0, length - i - 1):
            if arr[j] > arr[j+1]:
                arr[j], arr[j+1] = arr[j+1], arr[i]
    return arr
```

As Listing 4 shows, we designed a MR that checks whether the relative order of sorted elements is maintained when an element is removed from an input array. For example, given an input array `i1 = [3, 1, 2]` and a correctly-sorted array `o1 = [1, 2, 3]`, we derive a new input by removing one element `e` from the input array, for example, `e = 2`, resulting in a new input array `i2 = [3, 1]`, for which we can infer the expected result by removing `e` from `o1`, that is, `o2 = [1, 3]`. This specific idea enables finding the bug as well. When passing `[3, 1, 2]` as input array, the incorrect output `[1, 2, 1]` is produced; when passing `[3, 1]` as input by removing 2, the output is `[1, 3]`, rather than `[1, 1]`, breaking

**Listing 4.** Metamorphic testing by comparing whether the relative order is maintained for a smaller array.

```python
while True:
    arr = get_random_array()
    if len(arr) >= 1:
        sorted_arr = bubble_sort(arr.copy())
        random_elem = random.choice sorted_arr)
        arr.remove(random_elem)
        sorted_smaller_arr = bubble_sort(arr)
        sorted_arr.remove(random_elem)
        assert sorted_arr == sorted_smaller_arr
```
**Intramorphic Testing.** In this work, we propose Intramorphic Testing to tackle the test oracle problem by changing the system under test so that, for a given input and the original system's output, an oracle for the output of the changed system can be derived. Figure 1 illustrates the approach. Given a program *P*, a new program *P'* is derived for which, given an input *I*, a known relationship between the two program's outputs (i.e., *O* = *P*(*I*) and *O*' = *P*'(*I*) is validated. Similar to metamorphic testing, Intramorphic Testing is a high-level idea and conceptualization, for which many instantiations are possible.

<figure><img src="image_131.png" /><figcaption>**Figure 1.** Differential testing, metamorphic testing, and Intramorphic Testing in comparison.</figcaption></figure>
**Listing 1.** A Python implementation of `bubble_sort` affected by a bug when swapping array elements.

```python
def bubble_sort(arr):
    length = len(arr)
    for i in range(length):
        for j in range(0, length - i - 1):
            if arr[j] > arr[j+1]:
                arr[j], arr[j+1] = arr[j+1], arr[i]

    return arr
```

<figure><img src="image_132.png" /><figcaption>Figure 1. Differential testing, metamorphic testing, and Intramorphic Testing in comparison.</figcaption></figure>

**Listing 5.** An Intramorphic Testing realization that adds an additional reverse sorting function, whose reversed output array is compared with the sorted output array of the original sorting function.

```python
while True:
    arr = get_random_array()
    sorted_arr = bubble_sort(arr.copy())
    reverse_sorted_arr = bubble_sort(arr, copy())
    sorted_arr.reverse()
    assert sorted_arr.reverse() == reverse_sorted_arr
```
**Listing 1.** A Python implementation of `bubble_sort` affected by a bug when swapping array elements.

```python
def bubble_sort(arr):
    length = len(arr)
    for i in range(length):
        for j in range(0, length - i - 1):
            if arr[j] > arr[j+1]:
                arr[j], arr[j+1] = arr[j+1], arr[i]
    return arr
```

<figure><img src="image_133.png" /></figure>

**Listing 5.** An Intramorphic Testing realization that adds an additional reverse sorting function, whose reversed output array is compared with the sorted output array of the original sorting function.

```python
while True:
    arr = get_random_array()
    sorted_arr = bubble_sort(arr.copy())
    reverse_sorted_arr = bubble_sort(arr, reverse=True)
    copy()
    sorted_arr.reverse()
    assert sorted_arr.reverse() == reverse_sorted_arr
```

To realize an Intramorphic Testing technique for our use case, we could implement another alternative sorting implementation `bubble_sort()` as a potential replacement for `bubble_sort()` that sorts the array in descending order. Thus, the expectation that we could check is that, by reversing one of the two output arrays, two equivalent arrays are obtained (see Listing 5). For example, for an input `[3, 1, 2]`, we would expect an output `[3, 2, 1]` for `bubble_sort()` rather than `[1, 2, 3]`. This concrete Intramorphic Testing realization also detects the bug. Even if `bubble_sort Reverse()` is affected by the same index bug (i.e., by replacing only the comparison operator in Listing 1), it would detect the bug, since `bubble_sort()` returns `[3, 2, 3]` as
## Abstract

A test oracle determines whether a system behaves correctly for a given input. Automatic testing techniques rely on an automated test oracle to test the system without user interaction. Important families of automated test oracles include *Differential Testing* and *Metamorphic Testing*, which are both black-box approaches; that is, they provide a test oracle that is oblivious to the system's internals. In this work, we propose *Intramorphic Testing* as a white-box methodology to tackle the test oracle problem. To realize an Intramorphic Testing approach, a modified version of the system is created, for which, given a single input, a test oracle can be provided that relates the output of the original and modified systems. As a concrete example, by replacing a greater-equals operator in the implementation of a sorting algorithm with smaller-equals, it would be expected that the output of the modified implementation is the reverse output of the original implementation. In this paper, we introduce the methodology and illustrate it via a set of use cases.

Differential Testing

<figure><img src="image_134.png" /></figure>

Metamorphic Testing

<figure><img src="image_135.png" /></figure>

Intramorphic Testing

<figure><img src="image_136.png" /><figcaption>**Figure 1.** Differential testing, metamorphic testing, and Intramorphic Testing in comparison.</figcaption></figure>
<figure><img src="image_137.png" /><figcaption>Figure 1: Differential testing, metamorphic testing, and Intramorphic Testing in comparison.</figcaption></figure>
Random Generator:
Csmith

<figure><img src="image_138.png" /></figure>

C program

gcc -O0

gcc -O2

clang -Os

results

majority

minority

vote

<figure><img src="image_139.png" /></figure>

<figure><img src="image_140.png" /></figure>
# Finding and Understanding Bugs in C Compilers

Xuejun Yang Yang Chen Eric Eide John Regehr

University of Utah, School of Computing
{jyang, chenyang, eeide, regehr}@cs.utah.edu

* Randomized differential testing has the advantage that **no oracle** for test results is needed.

* Idea: if one has multiple, deterministic implementations of the same specification, all implementations must produce the same result from the same valid input

* 300+ bugs found in compilers (eg gcc)

1
int foo (void) {
2   signed char x = 1;
3   unsigned char y = 255;
4   return x > y;
5 }

<caption>**Figure 1.** We found a bug in the version of GCC that shipped with Ubuntu Linux 8.04.1 for x86. At all optimization levels it compiles this function to return 1; the correct result is 0. The Ubuntu compiler was heavily patched; the base version of GCC did not have this bug.</caption>
Given an image, you can:

* change its colour palette

* flip it

* rotate it

* zoom...

And still know what the output should be!

**Free test-cases (input)**

**Don't need to specify the oracle (output)**

<figure><img src="image_141.png" /></figure>

<figure><img src="image_142.png" /></figure>

<figure><img src="image_143.png" /></figure>

<figure><img src="image_144.png" /></figure>

<figure><img src="image_145.png" /></figure>

<figure><img src="image_146.png" /></figure>

<figure><img src="image_147.png" /></figure>
# Metamorphic Testing for CV

Given an image, you can:

* change its colour palette

* flip it

* rotate it

* zoom...

And still know what the output should be!

**Free test-cases (input)**

**Don't need to specify the oracle (output)**

<figure><img src="image_148.png" /></figure>

<figure><img src="image_149.png" /></figure>

<figure><img src="image_150.png" /></figure>
# Testing toolbox

* Testing Problem

* Test Generation Problem (input)

* Oracle Problem (output)

* Assertion-based testing

* Mutation Testing

* Differential Testing

* Metamorphic Testing

* Multimorphic Testing

* Many testing topics (property-based testing, search-based testing, fuzzing, etc.)

* see references!
References

* Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, Shin Yoo: The Oracle Problem in Software Testing: A Survey. IEEE Trans. Software Eng. 41(5): 507-525 (2015)

* Oscar Luis Vera-Pérez, Benjamin Danglot, Martin Monperrus, Benoit Baudry: A comprehensive study of pseudo-tested methods. Empirical Software Engineering 24(3): 1195-1225 (2019)

* Xuejun Yang, Yang Chen, Eric Eide, John Regehr: Finding and understanding bugs in C compilers. PLDI 2011: 283-294

* Vu Le, Mehrdad Afshari, Zhendong Su: Compiler validation via equivalence modulo inputs. PLDI 2014: 216-226

* Paul Temple, Mathieu Acher, and Jean-Marc Jézéquel. Empirical Assessment of Multimorphic Testing (2019). In IEEE Transactions on Software Engineering (TSE)
References

* Alastair F. Donaldson, Hugues Evrard, Andrei Lascu, Paul Thomson: Automated testing of graphics shader compilers. PACMPL 1(OOPSLA): 93:1-93:29 (2017)

* José A. Galindo, Mauricio Alferez, Mathieu Acher, Benoit Baudry, and David Benavides. A Variability-based Testing Approach for Synthesizing Video Sequences (2014). In International Symposium on Software Testing and Analysis (ISSTA'14)

* Sergio Segura, José Antonio Parejo, Javier Troya, Antonio Ruiz Cortés: Metamorphic Testing of RESTful Web APIs. IEEE Trans. Software Eng. 44(11): 1083-1099 (2018)

* Axel Halin, Alexandre Nuttinck, Mathieu Acher, Xavier Devroey, Gilles Perrouin, and Benoit Baudry. Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack (2018). In Empirical Software Engineering
General advice: #7 Explore the variants space

Make vary your experiments

Robustness of your results

Opportunity to find new paths/hypothesis/...

Sometimes called ablation studies or “sensitivity” studies or ...

It's much easier with a reproducible software stack ;)

<figure><img src="image_151.png" /></figure>
# AGENDA

Reproducible Science and (Deep) Software (Variability)

**Deep Software Variability**

Evidence of Deep Software Variability in Science

Threats and Opportunities
# Reproducible Science as a Testing Problem

## #1 Test Generation Problem (input)

inputs: computing environment, parameters of an algorithm, versions of a library or tool, choice of a programming language

## #2 Oracle Problem (output)

we usually ignore the outcome! (open problems; open questions; new knowledge)

Input

System under

Study

(replicable)

Output

(scientific
result)
<figure><img src="image_152.png" /></figure>

OBS
Open Broadcaster Software®

<figure><img src="image_153.png" /></figure>

<figure><img src="image_154.png" /></figure>

## deep software variability

hardware variability

15,000+ options

thousands of compiler
flags and compile-time
options
dozens of
preferences

100+ command-line
parameters
1000+ feature toggles

Non-functional properties
<figure><img src="image_155.png" /></figure>

<figure><img src="image_156.png" /></figure>

<figure><img src="image_157.png" /></figure>

<figure><img src="image_158.png" /></figure>

ARM

<figure><img src="image_159.png" /></figure>

<figure><img src="image_160.png" /></figure>

Tribute

<figure><img src="image_161.png" /></figure>

GCC

<figure><img src="image_162.png" /></figure>

CMake
Cross-platform Make

<figure><img src="image_163.png" /></figure>

OBS
Open Broadcaster Software

<figure><img src="image_164.png" /></figure>

<figure><img src="image_165.png" /></figure>

thousands of compiler flags
and compile-time options

dozens of preferences

input data

100+ command-line parameters

1000+ feature toggles

deep software variability

System under
Study
.replicable)

Variability
Output

(scientific result
most of the time
quantitative
information)
# Deep software variability: does it matter? i.e. Are layers/features orthogonal or are there interactions?

## Variability layers

Hardware

Age & usage
Number of cores
Processor

Operating System

Kernel options
Linux distribution
Dependencies

Software

x264 compilation
x264 variant
x264 version

Input data

Video category
Video resolution
Video quality

<figure><img src="image_166.png" /></figure>

## 264 environment

Hardware

Operating System

<figure><img src="image_167.png" /></figure>

<figure><img src="image_168.png" /></figure>

Software

X264

Input data

<figure><img src="image_169.png" /></figure>

<figure><img src="image_170.png" /></figure>

<figure><img src="image_171.png" /></figure>

<figure><img src="image_172.png" /></figure>

<figure><img src="image_173.png" /></figure>

## Encoding performances
Configuration is hard: numerous options, informal knowledge

<figure><img src="image_174.png" /></figure>

<figure><img src="image_175.png" /></figure>

<figure><img src="image_176.png" /></figure>

<figure><img src="image_177.png" /></figure>

<figure><img src="image_178.png" /></figure>

<figure><img src="image_179.png" /></figure>

<figure><img src="image_180.png" /></figure>

Lossless:
x264 --qp 0 -o <output> <input>

Maximum PSNR at the cost of speed and visual quality:
x264 --preset placebo --tune psnr -o <output> <input>

Constant bitrate at 1000kbps with a 2 second-buffer:
x264 --vbv-bufsize 2000 --bitrate 1000 -o <output> <input>

Presets:

--profile <string>
Force the limits of an H.264 profile
- Overrides all settings.
- baseline, main, high, high10, high42

--preset <string>
Use a preset to select encoding settings [m = medium, slow = slow, veryslow = very slow, placebo]
- Overridden by user settings.
- ultrafast, superfast, veryfast, faster, fast
- medium, slow, slower, veryslow, placebo

--tune <string>
Tune the settings for a particular type of source
or situation
- Overridden by user settings.
Multiple tunings are separated by commas.
Only one psy tuning can be used at a time.

-I, --keyint <integer or "infinite"> Maximum GOP size [250]
  --tff
  --bff
  --pulldown <string>
Enable interlaced mode (top field first)

Ratecontrol:

-B, --bitrate <integer>
  Set bitrate (kbps)
--crf <float>
  Quality-based VBR
--vbv-maxrate <integer>
  Max local bitrate
--vbv-bufsize <integer>
  Set size of the buffer
-p, --pass <integer>
  Enable multipass
- 1: First pass
- 2: Last pass

HARDWARE

<figure><img src="image_181.png" /></figure>

AGE

<figure><img src="image_182.png" /></figure>

# CORES

<figure><img src="image_183.png" /></figure>

GPU

OPERATING
SYSTEM

<figure><img src="image_184.png" /></figure>

VERSION

<figure><img src="image_185.png" /></figure>

OPTION

<figure><img src="image_186.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_187.png" /></figure>

COMPIL.

264

VARIANT

<figure><img src="image_188.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_189.png" /></figure>

SIZE

LENGTH

<figure><img src="image_190.png" /></figure>

RES.
REAL WORLD EXAMPLE (x264)

HARDWARE

<figure><img src="image_191.png" /></figure>

A

DELL LATITUDE
7400

20.04

B

<figure><img src="image_192.png" /></figure>

RASPBERRY PI
4 MODEL B

10.4

OPERATING
SYSTEM

<figure><img src="image_193.png" /></figure>

ubuntu

<figure><img src="image_194.png" /></figure>

x264
--mbtree

<figure><img src="image_195.png" /></figure>

x264
--no-mbtree

<figure><img src="image_196.png" /></figure>

x264
--mbtree

<figure><img src="image_197.png" /></figure>

x264
--no-mbtree

SOFTWARE

INPUT DATA

<figure><img src="image_198.png" /></figure>

animation

<figure><img src="image_199.png" /></figure>

vertical

<figure><img src="image_200.png" /></figure>

animation

<figure><img src="image_201.png" /></figure>

vertical

<figure><img src="image_202.png" /></figure>

animation

<figure><img src="image_203.png" /></figure>

vertical

<figure><img src="image_204.png" /></figure>

animation

<figure><img src="image_205.png" /></figure>

vertical

DURATION (S)

6

22

6

25

73

351

72

359

SIZE (MB)

33

28

21

34

33

28

21

34
<figure><img src="image_206.png" /><figcaption>REAL WORLD EXAMPLE (x264)</figcaption></figure>

HARDWARE

OPERATING
SYSTEM

SOFTWARE

INPUT DATA
REAL WORLD EXAMPLE (x264)

HARDWARE

<figure><img src="image_207.png" /></figure>

A

DELL LATITUDE
7400

20.04

B

<figure><img src="image_208.png" /></figure>

RASPBERRY PI
4 MODEL B

10.4

OPERATING
SYSTEM

<figure><img src="image_209.png" /></figure>

ubuntu

<figure><img src="image_210.png" /></figure>

x264
--mbtree
...

<figure><img src="image_211.png" /></figure>

x264
--no-mbtree
...

<figure><img src="image_212.png" /></figure>

x264
--mbtree
...

<figure><img src="image_213.png" /></figure>

x264
--no-mbtree
...

SOFTWARE

<figure><img src="image_214.png" /></figure>

animation

<figure><img src="image_215.png" /></figure>

vertical

<figure><img src="image_216.png" /></figure>

animation

<figure><img src="image_217.png" /></figure>

vertical

<figure><img src="image_218.png" /></figure>

animation

<figure><img src="image_219.png" /></figure>

vertical

<figure><img src="image_220.png" /></figure>

animation

<figure><img src="image_221.png" /></figure>

vertical

INPUT DATA

<figure><img src="image_222.png" /></figure>

<figure><img src="image_223.png" /></figure>

<figure><img src="image_224.png" /></figure>

<figure><img src="image_225.png" /></figure>

<figure><img src="image_226.png" /></figure>

<figure><img src="image_227.png" /></figure>

<figure><img src="image_228.png" /></figure>

<figure><img src="image_229.png" /></figure>

<figure><img src="image_230.png" /></figure>

<table><tr><td>DURATION (S)</td><td>6</td><td>22</td><td>6</td><td>25</td><td>73</td><td>351</td><td>72</td><td>359</td></tr><tr><td>SIZE (MB)</td><td>33</td><td>28</td><td>21</td><td>34</td><td>33</td><td>28</td><td>21</td><td>34</td></tr></table>

≈*16

≈*12
DEEP VARIABILITY

HARDWARE

<figure><img src="image_231.png" /></figure>

AGE

CPU

# CORES

<figure><img src="image_232.png" /></figure>

GPU

OPERATING
SYSTEM

<figure><img src="image_233.png" /></figure>

VERSION

<figure><img src="image_234.png" /></figure>

OPTION

<figure><img src="image_235.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_236.png" /></figure>

COMPIL.

SOFTWARE
VARIANT

<figure><img src="image_237.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_238.png" /></figure>

SIZE

<figure><img src="image_239.png" /></figure>

LENGTH

<figure><img src="image_240.png" /></figure>

RES.

The "best"/default software
variant might be a bad one.

Influential software options
and their interactions vary.

Performance prediction
models and **variability**
**knowledge may not**
**generalize**

<figure><img src="image_241.png" /></figure>

BUG

<figure><img src="image_242.png" /></figure>

PERF. ↗

<figure><img src="image_243.png" /></figure>

PERF. ↗

L. Lesoil, M. Acher, A. Blouin and J.-M. Jézéquel, "Deep Software Variability: Towards Handling Cross-Layer Configuration" in VaMoS 2021
<table><caption>Table 3: Summary of measured systems; N<sub>f</sub> – Number of features; NM – Number of machines on which systems were measured; NMC – Number of measured configurations</caption><thead><tr><th>System</th><th>N<sub>f</sub></th><th>NM</th><th>NMC</th></tr></thead><tbody><tr><td>XZ</td><td>7</td><td>7</td><td>154</td></tr><tr><td>x264</td><td>7</td><td>11</td><td>165</td></tr><tr><td>SQLite</td><td>5</td><td>10</td><td>32</td></tr></tbody></table>

<table><caption>Table 2: Summary of hardware platforms on which configurable software systems were measured; MID – Machine ID in DataMill cluster; NC – Number of CPUs; IS – Instruction set; CCR – CPU clock rate (MHz); RAM – RAM memory size (MB)</caption><thead><tr><th colspan="3">Systems</th><th colspan="5">Machines</th></tr><tr><th>XZ</th><th>x264</th><th>SQLite</th><th>MID</th><th>NC</th><th>IS</th><th>CCR</th><th>RAM</th></tr></thead><tbody><tr><td>✓</td><td></td><td></td><td>73</td><td>2</td><td>i686</td><td>1733</td><td>1771</td></tr><tr><td>✓</td><td>✓</td><td>✓</td><td>75</td><td>2</td><td>i686</td><td>3200</td><td>977</td></tr><tr><td>✓</td><td></td><td></td><td>77</td><td>2</td><td>i686</td><td>2992</td><td>2024</td></tr><tr><td>✓</td><td></td><td></td><td>78</td><td>1</td><td>i686</td><td>1495</td><td>755</td></tr><tr><td>✓</td><td></td><td></td><td>79</td><td>4</td><td>x86_64</td><td>3291</td><td>7961</td></tr><tr><td>✓</td><td></td><td></td><td>80</td><td>8</td><td>x86_64</td><td>3401</td><td>7907</td></tr><tr><td>✓</td><td>✓</td><td></td><td>81</td><td>16</td><td>x86_64</td><td>2411</td><td>32193</td></tr><tr><td>✓</td><td></td><td></td><td>87</td><td>1</td><td>i686</td><td>1595</td><td>249</td></tr><tr><td>✓</td><td></td><td></td><td>88</td><td>1</td><td>i686</td><td>1700</td><td>978</td></tr><tr><td>✓</td><td></td><td>✓</td><td>90</td><td>2</td><td>i686</td><td>3200</td><td>977</td></tr><tr><td>✓</td><td></td><td>✓</td><td>91</td><td>1</td><td>i686</td><td>2400</td><td>1009</td></tr><tr><td>✓</td><td></td><td>✓</td><td>97</td><td>2</td><td>i686</td><td>2992</td><td>873</td></tr><tr><td>✓</td><td></td><td>✓</td><td>98</td><td>2</td><td>i686</td><td>2992</td><td>873</td></tr><tr><td>✓</td><td></td><td>✓</td><td>99</td><td>2</td><td>i686</td><td>2793</td><td>880</td></tr><tr><td>✓</td><td></td><td></td><td>103</td><td>2</td><td>i686</td><td>3200</td><td>881</td></tr><tr><td>✓</td><td></td><td></td><td>104</td><td>1</td><td>i686</td><td>1800</td><td>502</td></tr><tr><td>✓</td><td>✓</td><td></td><td>105</td><td>2</td><td>i686</td><td>3200</td><td>881</td></tr><tr><td>✓</td><td></td><td></td><td>106</td><td>2</td><td>i686</td><td>3192</td><td>494</td></tr><tr><td>✓</td><td></td><td>✓</td><td>125</td><td>4</td><td>x86_64</td><td>3301</td><td>7960</td></tr><tr><td>✓</td><td></td><td>✓</td><td>128</td><td>2</td><td>i686</td><td>2993</td><td>2024</td></tr><tr><td></td><td>✓</td><td></td><td>130</td><td>2</td><td>i686</td><td>3198</td><td>880</td></tr><tr><td></td><td></td><td>✓</td><td>146</td><td>2</td><td>i686</td><td>2998</td><td>872</td></tr><tr><td></td><td></td><td>✓</td><td>157</td><td>36</td><td>x86_64</td><td>2301</td><td>15954</td></tr></tbody></table>

“Linear model provides a good approximation of transformation between performance distributions of a system deployed in different hardware environments”

what about
variability of
input data?
compile-time options?
version?

<figure><img src="image_244.png" /></figure>
# Transfer Learning for Software Performance Analysis: An Exploratory Analysis
## Jamshidi et al. ASE 2017

SPEAR (SAT Solver)

Analysis time

14 options
16,384 configurations
SAT problems
3 hardware
2 versions

X264 (video encoder)

Encoding time

16 options
4,000 configurations
Video quality/size
2 hardware
3 versions

SQLite (DB engine)

Query time

14 options
1,000 configurations
DB Queries
2 hardware
2 versions

SaC (Compiler)

Execution time

50 options
71,267 configurations
10 Demo programs

<table><tr><td>ec1 : [h<sub>2</sub> → h<sub>1</sub>, w<sub>3</sub>, v<sub>3</sub>]</td><td>SM</td><td>0.97</td></tr><tr><td>ec2 : [h<sub>2</sub> → h<sub>1</sub>, w<sub>1</sub>, v<sub>3</sub>]</td><td>S</td><td>0.96</td></tr><tr><td>ec3 : [h<sub>1</sub>, w<sub>1</sub> → w<sub>2</sub>, v<sub>3</sub>]</td><td>M</td><td>0.65</td></tr><tr><td>ec4 : [h<sub>1</sub>, w<sub>1</sub> → w<sub>3</sub>, v<sub>3</sub>]</td><td>M</td><td>0.67</td></tr><tr><td>ec5 : [h<sub>1</sub>, w<sub>3</sub>, v<sub>2</sub> → v<sub>3</sub>]</td><td>L</td><td>0.05</td></tr><tr><td>ec6 : [h<sub>1</sub>, w<sub>3</sub>, v<sub>1</sub> → v<sub>3</sub>]</td><td>L</td><td>0.06</td></tr><tr><td>ec7 : [h<sub>1</sub>, w<sub>1</sub> → w<sub>3</sub>, v<sub>2</sub> → v<sub>3</sub>]</td><td>L</td><td>0.08</td></tr><tr><td>ec8 : [h<sub>2</sub> → h<sub>1</sub>, w<sub>1</sub> → w<sub>3</sub>, v<sub>2</sub> → v<sub>3</sub>]</td><td>VL</td><td>0.09</td></tr></table>

**Insight.** For non-severe hardware changes, we can linearly transfer performance models across environments.

**Insight.** The strength of the influence of configuration options is typically preserved across environments.

**Insight.** A large percentage of configurations are typically invalid in both source and target environments.

<figure><img src="image_245.png" /></figure>
# Transfer Learning for Software Performance Analysis: An Exploratory Analysis
## Jamshidi et al. ASE 2017

SPEAR (SAT Solver)

Analysis time

14 options
16,384 configurations
SAT problems
3 hardware
2 versions

X264 (video encoder)

Encoding time

16 options
4,000 configurations
Video quality/size
2 hardware
3 versions

SQLite (DB engine)

Query time

14 options
1,000 configurations
DB Queries
2 hardware
2 versions

SaC (Compiler)

Execution time

50 options
71,267 configurations
10 Demo programs

<table><tr><td>ec<sub>1</sub>: [h<sub>2</sub> → h<sub>1</sub>, w<sub>3</sub>, v<sub>3</sub>]</td><td>SM</td><td>0.97</td></tr><tr><td>ec<sub>2</sub>: [h<sub>2</sub> → h<sub>1</sub>, w<sub>1</sub>, v<sub>3</sub>]</td><td>S</td><td>0.96</td></tr><tr><td>ec<sub>3</sub>: [h<sub>1</sub>, w<sub>1</sub> → w<sub>2</sub>, v<sub>3</sub>]</td><td>M</td><td>0.65</td></tr><tr><td>ec<sub>4</sub>: [h<sub>1</sub>, w<sub>1</sub> → w<sub>3</sub>, v<sub>3</sub>]</td><td>M</td><td>0.67</td></tr><tr><td>ec<sub>5</sub>: [h<sub>1</sub>, w<sub>3</sub>, v<sub>2</sub> → v<sub>3</sub>]</td><td>L</td><td>0.05</td></tr><tr><td>ec<sub>6</sub>: [h<sub>1</sub>, w<sub>3</sub>, v<sub>1</sub> → v<sub>3</sub>]</td><td>L</td><td>0.06</td></tr><tr><td>ec<sub>7</sub>: [h<sub>1</sub>, w<sub>1</sub> → w<sub>3</sub>, v<sub>2</sub> → v<sub>3</sub>]</td><td>L</td><td>0.08</td></tr><tr><td>ec<sub>8</sub>: [h<sub>2</sub> → h<sub>1</sub>, w<sub>1</sub> → w<sub>3</sub>, v<sub>2</sub> → v<sub>3</sub>]</td><td>VL</td><td>0.09</td></tr></table>

<figure><img src="image_246.png" /></figure>

**mixing deep variability:** hard to assess the specific influence of each layer

**very few hardware, version, and input data... but lots of runtime configurations (variants)**

**Let's go deep with input data!**
# Let's go deep with **input data!**

<figure><img src="image_247.png" /></figure>

Intuition: video encoder behavior (and thus **runtime configurations**) hugely depends on the **input video** (different compression ratio, encoding size/type etc.)

Is the best software configuration still the best?

Are influential options always influential?

Does the configuration knowledge generalize?

two performance models $f_1$ and $f_2$

$$f_1 = \beta \times f_2 + \alpha \qquad ?$$

YouTube User General Content dataset: **1397 videos**

Measurements of **201 soft. configurations** (with **same** hardware, compiler, version, etc.): encoding time, bitrate, etc.

<figure><img src="image_248.png" /></figure>
<figure><img src="image_249.png" /></figure>

# Do x264 software performances stay consistent across inputs?

1397 videos x 201 software configurations

* Encoding time: very strong correlations
  ○ low input sensitivity

* FPS: very strong correlations
  ○ low input sensitivity

* CPU usage : moderate correlation, a few negative correlations
  ○ medium input sensitivity

* Bitrate: medium-low correlation, many negative correlations
  ○ High input sensitivity

* Encoding size: medium-low correlation, many negative correlations
  ○ High input sensitivity

two performance models $f_1$ and $f_2$

$$f_1 = \beta \times f_2 + \alpha$$
Are there some configuration options
more sensitive to input videos? (bitrate)

<figure><img src="image_250.png" /></figure>
<figure><img src="image_251.png" /></figure>

Are there some configuration options
more sensitive to input videos? (bitrate)

<figure><img src="image_252.png" /></figure>
# Practical impacts for users, developers, scientists, and self-adaptive systems

Threats to **variability knowledge** for performance property bitrate

<figure><img src="image_253.png" /></figure>

* optimal configuration is specific to an input; a good configuration can be a bad one

* some options' values have an opposite effect depending on the input

* effectiveness of sampling strategies (random, 2-wise, etc.) is input specific (somehow confirming Pereira et al. ICPE 2020)

* predicting, tuning, or understanding configurable systems without being aware of inputs can be inaccurate and... pointless

<figure><img src="image_254.png" /></figure>

<figure><img src="image_255.png" /></figure>
# Practical impacts for users, developers, scientists, and self-adaptive systems

**Threats** to **variability knowledge**: predicting, tuning, or understanding configurable systems without being aware of inputs can be inaccurate and... pointless

**Opportunities:** for some performance properties (P) and subject systems, some stability is observed and performance remains consistent!

<table><thead><tr><th>System</th><th>Domain</th><th>Commit</th><th>Configs #C</th><th>Inputs I</th><th>#I</th></tr></thead><tbody><tr><td>gcc</td><td>Compilation</td><td>ccb4e07</td><td>80</td><td>.c programs</td><td>30</td></tr><tr><td>ImageMagick</td><td>Image processing</td><td>5ee49d6</td><td>100</td><td>images</td><td>1000</td></tr><tr><td>lingeling</td><td>SAT solver</td><td>7d5db72</td><td>100</td><td>SAT formulae</td><td>351</td></tr><tr><td>nodeJS</td><td>JS runtime env.</td><td>78343bb</td><td>50</td><td>.js scripts</td><td>1939</td></tr><tr><td>poppler</td><td>PDF rendering</td><td>42dde68</td><td>16</td><td>.pdf files</td><td>1480</td></tr><tr><td>SQLite</td><td>DBMS</td><td>53fa025</td><td>50</td><td>databases</td><td>150</td></tr><tr><td>x264</td><td>Video encoding</td><td>e9a5903</td><td>201</td><td>videos</td><td>1397</td></tr><tr><td>xz</td><td>Data compression</td><td>e7da44d</td><td>30</td><td>system files</td><td>48</td></tr></tbody></table>

<table><thead><tr><th>System</th><th>#M</th><th>Performance(s) P</th><th>Docker</th><th>Dataset</th></tr></thead><tbody><tr><td>gcc</td><td>2400</td><td>size, ctime, exec</td><td>Link</td><td>Link</td></tr><tr><td>ImageMagick</td><td>100 000</td><td>size, time</td><td>Link</td><td>Link</td></tr><tr><td>lingeling</td><td>35 100</td><td>#confl., #reduc.</td><td>Link</td><td>Link</td></tr><tr><td>nodeJS</td><td>96 950</td><td>#operations/s</td><td>Link</td><td>Link</td></tr><tr><td>poppler</td><td>23 680</td><td>size, time</td><td>Link</td><td>Link</td></tr><tr><td>SQLite</td><td>7500</td><td>15 query times q1-q15</td><td>Link</td><td>Link</td></tr><tr><td>x264</td><td>280 797</td><td>cpu, fps, kbs, size, time</td><td>Link</td><td>Link</td></tr><tr><td>xz</td><td>1440</td><td>size, time</td><td>Link</td><td>Link</td></tr></tbody></table>

L. Lesoil, M. Acher, A. Blouin and J.-M. Jézéquel “The Interaction between Inputs and Configurations fed to Software Systems: an Empirical Study”
https://arxiv.org/abs/2112.07279

<figure><img src="image_256.png" /></figure>
# x264 video encoder (compilation/build)

~ x264 --bframes 1 --ref 3 --cabac DiverSE-meeting.mp4 -o meeting13.webm

mathieucahier

localhost.localdomain

.../x264-SLIMFAST/x264

x264-gcov

./configure --help

<table><tr><td>--disable-thread</td><td>disable multithreaded encoding</td></tr><tr><td>--disable-win32thread</td><td>disable win32threads (windows only)</td></tr><tr><td>--disable-interlaced</td><td>disable interlaced encoding support</td></tr><tr><td>--bit-depth=BIT_DEPTH</td><td>set output bit depth (8, 10, all) [all]</td></tr><tr><td>--chroma-format=FORMAT</td><td>output chroma format (400, 420, 422, 444, all) [all]</td></tr></table>

Advanced options:

<table><tr><td>--disable-asm</td><td>disable platform-specific assembly optimizations</td></tr><tr><td>--enable-lto</td><td>enable link-time optimization</td></tr><tr><td>--enable-debug</td><td>add -g</td></tr><tr><td>--enable-gprof</td><td>add -pg</td></tr><tr><td>--enable-strip</td><td>add -s</td></tr><tr><td>--enable-pic</td><td>build position-independent code</td></tr></table>

Cross-compilation:

<table><tr><td>--host=HOST</td><td>build programs to run on HOST</td></tr><tr><td>--cross-prefix=PREFIX</td><td>use PREFIX for compilation tools</td></tr><tr><td>--sysroot=SYSROOT</td><td>root of cross-build tree</td></tr></table>

External library support:

<table><tr><td>--disable-avs</td><td>disable avisynth support</td></tr><tr><td>--disable-swscale</td><td>disable swscale support</td></tr><tr><td>--disable-lavf</td><td>disable lavformat support</td></tr><tr><td>--disable-ffms</td><td>disable fmmpegsource support</td></tr><tr><td>--disable-gpac</td><td>disable gpac support</td></tr><tr><td>--disable-lsmash</td><td>disable lsmash support</td></tr></table>

compile-time
options

<figure><img src="image_257.png" /></figure>
# Is there an interplay between compile-time and runtime options?

<figure><img src="image_258.png" /></figure>

L. Lesoil, M. Acher, X. Tèrnava, A. Blouin and J.-M. Jézéquel "The Interplay of Compile-time and Run-time Options for Performance Prediction" in SPLC '21

<figure><img src="image_259.png" /></figure>

This paper investigates how compile-time options can affect software performances and how compile-time options interact with run-time options.

<figure><img src="image_260.png" /><figcaption>Figure 1: Cross-layer variability of x264</figcaption></figure>
Key results (for x264)

<figure><img src="image_261.png" /></figure>

**Worth tuning software at compile-time:** gain about 10 % of execution time with the tuning of compile-time options (compared to the default compile-time configuration). The improvements can be larger for some inputs and some runtime configurations.

**Stability of variability knowledge:** For all the execution time distributions of x264 and all the input videos, the worst correlation is greater than 0.97. If the compile-time options change the scale of the distribution, they do not change the rankings of run-time configurations (i.e., they do not truly interact with the run-time options).

**Reuse of configuration knowledge:** $f_1 = \beta \times f_2 + \alpha$

* Linear transformation among distributions

* Users can also trust the documentation of run-time options, consistent whatever the compile-time configuration is.

<figure><img src="image_262.png" /></figure>
interplay between compile-time and runtime options and even input!

**L. Lesoil**, M. Acher, X. Tèrnava, A. Blouin and J.-M. Jézéquel "The Interplay of Compile-time and Run-time Options for Performance Prediction" in SPLC '21

<figure><img src="image_263.png" /><figcaption>(c) x264. T+4 (lyric video), encoding time</figcaption></figure>

## Poppler

### Key results

**First good new. Worth the bobble!**

**Second good news:** all the input videos, the worst correlation is created by scaling the distribution, they do not change the rankings (except with the runtime options).

**It has three practical implications:**

transfer learning or cross-domain adaptation. Users can adjust the configuration to suit the distribution of runtime options.

The best compile-time configuration among all the possible ones allows us to configure at run time. We can remove awards from default compile-time configurations.

performances!

264

for
* Linux as a subject software system (not as an OS interacting with other layers)

* Targeted non-functional, quantitative property: **binary size**

* interest for maintainers/users of the Linux kernel (embedded systems, cloud, etc.)

* challenging to predict (cross-cutting options, interplay with compilers/build systems, etc./)

* Dataset: version 4.13.3 (september 2017), x86_64 arch, measurements of 95K+ random configurations

* paranoiac about deep variability since 2017, Docker to control the build environment and scale

* diversity of binary sizes: from 7Mb to 1.9Gb

* 6% MAPE errors: quite good, though costly...

<figure><img src="image_264.png" /></figure>

<figure><img src="image_265.png" /></figure>

<figure><img src="image_266.png" /></figure>

<figure><img src="image_267.png" /></figure>
<figure><img src="image_268.png" /></figure>

<table><caption>Table I: Dataset properties for each version. The number of deleted/new features, delta commits, files changes are w.r.t. 4.13.</caption><thead><tr><th>Version</th><th>Release Date</th><th>LOC</th><th>Files</th><th>Examples</th><th>Seconds/config</th><th>Options</th><th>Features</th><th>Deleted features</th><th>New features</th><th>ΔCommits</th><th>Files changes</th></tr></thead><tbody><tr><td>4.13</td><td>2017/09/03</td><td>16,616,534</td><td>60,530</td><td>92,562</td><td>not available</td><td>12,776</td><td>9,468</td><td>-</td><td>-</td><td>-</td><td>934,628</td></tr><tr><td>4.15</td><td>2018/01/28</td><td>17,073,368</td><td>62,249</td><td>39,391</td><td>not available</td><td>12,998</td><td>9,425</td><td>342</td><td>299</td><td>31,052</td><td>1,972,020</td></tr><tr><td>4.20</td><td>2018/12/23</td><td>17,526,171</td><td>62,423</td><td>23,489</td><td>225</td><td>13,533</td><td>10,189</td><td>468</td><td>1,189</td><td>104,691</td><td>2,170,935</td></tr><tr><td>5.0</td><td>2019/03/03</td><td>17,679,372</td><td>63,076</td><td>19,952</td><td>247</td><td>13,673</td><td>10,293</td><td>494</td><td>1,319</td><td>118,778</td><td>3,827,025</td></tr><tr><td>5.4</td><td>2019/10/24</td><td>19,358,903</td><td>67,915</td><td>25,847</td><td>285</td><td>14,159</td><td>10,813</td><td>663</td><td>2,008</td><td>181,308</td><td>4,393,117</td></tr><tr><td>5.7</td><td>2020/05/31</td><td>19,358,903</td><td>67,915</td><td>20,159</td><td>258</td><td>14,586</td><td>11,338</td><td>715</td><td>2,585</td><td>225,804</td><td>4,681,313</td></tr><tr><td>5.8</td><td>2020/08/02</td><td>19,729,197</td><td>69,303</td><td>21,923</td><td>289</td><td>14,817</td><td>11,530</td><td>730</td><td>2,792</td><td>242,381</td><td>4,681,313</td></tr></tbody></table>

4.13 version (sep 2017): 6%. What about **evolution?** Can we reuse the 4.13 Linux prediction model? No, accuracy quickly decreases: **4.15 (5 months after): 20%; 5.7 (3 years after): 35%**

<figure><img src="image_269.png" /></figure>

<figure><img src="image_270.png" /></figure>
Transfer learning to the rescue

*   **Mission Impossible: Saving variability knowledge and prediction model 4.13** (15K hours of computation)

*   **Heterogeneous transfer learning: the feature space is different**

*   **TEAMS: transfer evolution-aware model shifting**

<figure><img src="image_271.png" /></figure>

<figure><img src="image_272.png" /></figure>

<figure><img src="image_273.png" /></figure>
DEEP VARIABILITY

HARDWARE

<figure><img src="image_274.png" /></figure>

AGE

<figure><img src="image_275.png" /></figure>

CPU

# CORES

<figure><img src="image_276.png" /></figure>

GPU

OPERATING
SYSTEM

<figure><img src="image_277.png" /></figure>

VERSION

<figure><img src="image_278.png" /></figure>

OPTION

<figure><img src="image_279.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_280.png" /></figure>

COMPIL.

SOFTWARE
VARIANT

<figure><img src="image_281.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_282.png" /></figure>

SIZE

<figure><img src="image_283.png" /></figure>

LENGTH

<figure><img src="image_284.png" /></figure>

SIZE

Sometimes, variability is
consistent/stable and
knowledge transfer is
immediate.

But there are also
interactions among
variability layers and
**variability knowledge**
**may not generalize**

BUG

PERF. →

PERF. →

<figure><img src="image_285.png" /></figure>

Poppiter

node

264

<figure><img src="image_286.png" /></figure>

XZ
# AGENDA

Reproducible Science and (Deep) Software (Variability)

Deep Software Variability

**Evidence of Deep Software Variability in Computational
Science**

Threats and Opportunities
<figure><img src="image_287.png" /></figure>

# It's all about software... Where is (computational) science?

<figure><img src="image_288.png" /></figure>

from a set of scripts to automate the deployment to... a comprehensive system containing several features that help researchers exploring various hypotheses
> “Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and

”

> PLoS One. 2012;7(6):e38234. doi: 10.1371/journal.pone.0038234. Epub 2012 Jun 1.

The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements

Ed H B M Gronenschild ¹, Petra Habets, Heidi I L Jacobs, Ron Mengelers, Nico Rozendaal, Jim van Os, Machteld Marcelis

<figure><img src="image_289.png" /></figure>

see also Krefting, D., Scheel, M., Freing, A., Specovius, S., Paul, F., and Brandt, A. (2011). “Reliability of quantitative neuroimage analysis using freesurfer in distributed environments,” in *MICCAI Workshop on High-Performance and Distributed Computing for Medical Imaging*. (Toronto, ON)

Significant differences were revealed between FreeSurfer version v5.0.0 and the two earlier **versions**. [...] About a factor two smaller differences were detected between Macintosh and Hewlett-Packard **workstations** and between **OSX 10.5 and OSX 10.6**. The observed differences are similar in magnitude as effect sizes reported in **accuracy** evaluations and neurodegenerative-studies.

<figure><img src="image_290.png" /></figure>
“Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed.”

Reproducibility of neuroimaging analyses across operating systems, Glatard et al., Front. Neuroinform., 24 April 2015

<figure><img src="image_291.png" /></figure>

<table><thead><tr><th></th><th>Cluster A</th><th>Cluster B</th></tr></thead><tbody><tr><th>Applications</th><td>Freesurfer 5.3.0, build 1<br>FSL 5.0.6, build 1<br>CIVET 1.1.12-UCSF, build 1</td><td>Freesurfer 5.3.0, build 1 and 2<br>FSL 5.0.6, build 1 and 2<br>CIVET 1.1.12-UCSF, build 1</td></tr><tr><th>Interpreters</th><td>Python 2.4.3, bash 3.2.25,<br>Perl 5.8.8, tcsh 6.14.00</td><td>Python 2.7.5, bash 4.2.47,<br>Perl 5.18.2, tcsh 6.18.01</td></tr><tr><th>glibc version</th><td>2.5</td><td>2.18</td></tr><tr><th>OS</th><td>CentOS 5.10</td><td>Fedora 20</td></tr><tr><th>Hardware</th><td>x86_64 CPUs (Intel Xeon)</td><td>x86_64 CPUs (Intel Xeon)</td></tr></tbody></table>

The implementation of mathematical functions manipulating single-precision **floating-point numbers in libmath** has evolved during the last years, leading to numerical differences in computational results. While these differences have little or no impact on simple analysis pipelines such as brain extraction and cortical tissue classification, their **accumulation** creates important differences in longer pipelines such as the subcortical tissue classification, RSfMRI analysis, and cortical thickness extraction.

<figure><img src="image_292.png" /></figure>
“Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed.”

Reproducibility of neuroimaging analyses across operating systems, Glatard et al., Front. Neuroinform., 24 April 2015

Statically building programs improves reproducibility across OSes, but small differences may still remain when dynamic libraries are loaded by static executables[...]. When static builds are not an option, software heterogeneity might be addressed using virtual machines. However, such solutions are only workarounds: differences may still arise between **static executables built on different OSes**, or between **dynamic executables executed in different VMs**.

<figure><img src="image_293.png" /></figure>

<table><thead><tr><th></th><th>Cluster A</th><th>Cluster B</th></tr></thead><tbody><tr><th scope="row">Applications</th><td>Freesurfer 5.3.0, build 1<br>FSL 5.0.6, build 1<br>CIVET 1.1.12-UCSF, build 1</td><td>Freesurfer 5.3.0, build 1 and 2<br>FSL 5.0.6, build 1 and 2<br>CIVET 1.1.12-UCSF, build 1</td></tr><tr><th scope="row">Interpreters</th><td>Python 2.4.3, bash 3.2.25,<br>Perl 5.8.8, tcsh 6.14.00</td><td>Python 2.7.5, bash 4.2.47,<br>Perl 5.18.2, tcsh 6.18.01</td></tr><tr><th scope="row">glibc version</th><td>2.5</td><td>2.18</td></tr><tr><th scope="row">OS</th><td>CentOS 5.10</td><td>Fedora 20</td></tr><tr><th scope="row">Hardware</th><td>x86_64 CPUs (Intel Xeon)</td><td>x86_64 CPUs (Intel Xeon)</td></tr></tbody></table>

<figure><img src="image_294.png" /></figure>
<table><thead><tr><th>Explanatory variable</th><th>Meaning</th></tr></thead><tbody><tr><td>entcoef</td><td>Entrainment coefficient</td></tr><tr><td>ct</td><td>Accretion constant</td></tr><tr><td>rhcrit</td><td>Critical relative humidity</td></tr><tr><td>vfl</td><td>Ice fall speed through clouds</td></tr><tr><td>eacf</td><td>Empirically adjusted cloud fraction</td></tr><tr><td>cw</td><td>Threshold for precipitation</td></tr><tr><td>dtice</td><td>Temperature range of ice albedo variation</td></tr><tr><td>ice</td><td>Nonspherical ice</td></tr><tr><td>midware</td><td>Client middleware</td></tr><tr><td>ice_size</td><td>Ice particle size</td></tr><tr><td>alpham</td><td>Albedo at melting point of ice</td></tr><tr><td>processor_name</td><td>CPU classification</td></tr><tr><td>clock_classic</td><td>Processor clock speed recorded under classic middleware</td></tr><tr><td>ram_size</td><td>Hardware RAM</td></tr><tr><td>clock_boinc_i</td><td>Integer processor clock speed recorded under BOINC middleware</td></tr><tr><td>clock_boinc_f</td><td>Floating point processor clock speed recorded under BOINC middleware</td></tr><tr><td>os_name</td><td>Operating system</td></tr><tr><td>dtheta</td><td>Perturbations to initial conditions on a given level</td></tr></tbody></table>

Association of parameter, software, and hardware variation with large-scale behavior across 57,000 climate models

Christopher G. Knight, Sylvia H. E. Knight, Neil Massey, Tolu Aina, Carl Christensen, Dave J. F...

+ See all authors and affiliations

PNAS July 24, 2007 104 (30) 12259-12264; https://doi.org/10.1073/pnas.0608144104

We demonstrate that **effects of parameter, hardware, and software variation are detectable, complex, and interacting**. However, we find most of the effects of parameter variation are caused by a small subset of parameters. Notably, the entrainment coefficient in clouds is associated with 30% of the variation seen in climate sensitivity, although both low and high values can give high climate sensitivity. **We demonstrate that the effect of hardware and software is small relative to the effect of parameter variation** and, over the wide range of systems tested, may be treated as equivalent to that caused by changes in initial conditions.

<figure><img src="image_295.png" /></figure>

57,067 climate model runs. These runs sample parameter space for 10 parameters with between two and four levels of each, covering 12,487 parameter combinations (24% of possible combinations) and a range of initial conditions
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹, Xavier Yepes-Arbós¹,², Eleftheria Exarchou¹,², and Francisco J. Doblas-Reyes¹,²,⁴

<figure><img src="image_296.png" /></figure>

Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results? Using two versions of EC-Earth: one “non-replicable” case (see below) and one replicable case.

¹²m difference between five-member experiments m06e and e011. Black dotted regions indicate where the difference is significant according to a Kolmogorov-Smirnov test (90% of grid points show a significant difference)

<figure><img src="image_297.png" /></figure>

<figure><img src="image_298.png" /></figure>
# Replicability of the EC-Earth3 Earth system model under a change in computing environment

François Massonnet¹,², Martin Ménégoz¹,²,³, Mario Acosta¹,², Xavier Yepes-Arbós¹,², Eleftheria Exarchou¹,², and Francisco J. Doblas-Reyes¹,²,⁴

<figure><img src="image_299.png" /></figure>

## Can a coupled ESM simulation be restarted from a different machine without causing climate-changing modifications in the results?

A study involving eight institutions and seven different supercomputers in Europe is currently ongoing with EC-Earth. This ongoing study aims to do the following:

* evaluate different **computational environments** that are used in collaboration to produce CMIP6 experiments (can we safely create large ensembles composed of subsets that emanate from different partners of the consortium?);

* detect if the same **CMIP6 configuration** is replicable among platforms of the EC-Earth consortium (that is, can we safely exchange restarts with EC-Earth partners in order to initialize simulations and to avoid long spin-ups?); and

* systematically evaluate the impact of **different compilation flag options** (that is, what is the highest acceptable level of optimization that will not break the replicability of EC-Earth for a given environment?).

<figure><img src="image_300.png" /></figure>
Deep Reinforcement Learning that Matters

Peter Henderson¹*, Riashat Islam¹,²*, Philip Bachman²
Joelle Pineau¹, Doina Precup¹, David Meger¹

What is the magnitude of the effect
**hyperparameter** settings can have on baseline
performance?

How does the choice of **network architecture** for
the policy and value function approximation affect
performance?

How can the **reward scale** affect results?

Can **random seeds** drastically alter performance?

How do the **environment properties** affect
variability in reported RL algorithm performance?

Are commonly used baseline **implementations**
comparable?

torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision

David Picard

LIGM, École des Ponts, 77455 Marnes la vallée, France

DAVID.PICARD@ENPC.FR

Abstract

In this paper I investigate the effect of random seed selection on the accuracy when using popular deep learning architectures for computer vision. I scan a large amount of seeds (up to 10⁴) on CIFAR 10 and I also scan fewer seeds on Imagenet using pre-trained models to investigate large scale datasets. The conclusions are that even if the variance is not very large, it is surprisingly easy to find an outlier that performs much better or much worse than the average.

<figure><img src="image_301.png" /></figure>
Reproducible and replicable CFD: it's harder than you think

Olivier Mesnard, Lorena A. Barba

Mechanical and Aerospace Engineering, George Washington University, Washington DC 20052

> "Completing a full replication study of our previously published findings on bluff-body aerodynamics was harder than we thought. Despite the fact that we have good reproducible-research practices, sharing our code and data openly."

**Story 1: Meshing and boundary conditions can ruin everything**

**Story 3: All linear algebra libraries are not created equal**

**Story 4: Different versions of your code, external libraries or even compilers may challenge reproducibility**

<figure><img src="image_302.png" /></figure>
# AGENDA

- Reproducible Science and (Deep) Software (Variability)

- Deep Software Variability

- Evidence of Deep Software Variability in Computational Science

- **Threats and Opportunities**
(computational)
science

<figure><img src="image_303.png" /></figure>

<figure><img src="image_304.png" /></figure>

Deep software variability is a threat to
scientific, software-based experiments

<figure><img src="image_305.png" /></figure>

from a set of scripts to automate the deployment to... a
comprehensive system containing several features that
help researchers exploring various hypotheses

<figure><img src="image_306.png" /></figure>
# DOES DEEP SOFTWARE VARIABILITY AFFECT PREVIOUS SCIENTIFIC, SOFTWARE-BASED STUDIES? (A GRAPHICAL TEMPLATE)

## HARDWARE

<figure><img src="image_307.png" /></figure>

AGE

<figure><img src="image_308.png" /></figure>

CPU

# CORES

<figure><img src="image_309.png" /></figure>

GPU

## LIST ALL DETAILS...
AND QUESTIONS:

## OPERATING SYSTEM

<figure><img src="image_310.png" /></figure>

VERSION

<figure><img src="image_311.png" /></figure>

OPTION

<figure><img src="image_312.png" /></figure>

DISTRIB.

## WHAT IF WE RUN THE EXPERIMENTS ON DIFFERENT:

OS?

## SOFTWARE

<figure><img src="image_313.png" /></figure>

COMPIL.

## SOFTWARE VARIANT

<figure><img src="image_314.png" /></figure>

VERSION

VERSION/COMMIT?

## INPUT DATA

<figure><img src="image_315.png" /></figure>

SIZE

<figure><img src="image_316.png" /></figure>

LENGTH

<figure><img src="image_317.png" /></figure>

RES.

PARAMETERS?

INPUT?
# What can we do? (#1 studies)

Empirical studies about deep software variability

* more subject systems

* more variability layers, including interactions

* more quantitative (e.g., performance) properties

with challenges for gathering measurements data:

* how to scale experiments? Variant space is huge!

* how to fix/isolate some layers? (eg hardware)

* how to measure in a reliable way?

Expected outcomes:

* significance of deep software variability in the wild

* identification of **stable** layers: sources of variability that should not affect the conclusion and that can be eliminated/forgotten

* identification/quantification of **sensitive** layers and interactions that matter

* **variability knowledge**

<figure><img src="image_318.png" /></figure>
# What can we do? (#2 cost)

## Reducing the cost of exploring the variability spaces

Many directions here (references at the end of the slides):

* **learning**
  * many algorithms/techniques with tradeoffs interpretability/accuracy
  * transfer learning (instead of learning from scratch)

* **sampling strategies**
  * uniform random sampling? t-wise? distance-based? ...
  * sample of hardware? input data?

* incremental build of configurations

* white-box approaches

* ...

<figure><img src="image_319.png" /></figure>
# What can we do? (modelling)

## Modelling variability

* Abstractions are definitely needed to...

* reason about logical constraints and interactions

* integrate domain knowledge

* synthesize domain knowledge

* automate and guide the exploration of variants

* scope and prioritize experiments

* Challenges:

* Multiple systems, layers, concerns

* Different kinds of variability: technical vs domain, accidental vs essential, implicit vs explicit... when to stop modelling?

* reverse engineering

<figure><img src="image_320.png" /></figure>
<figure><img src="image_321.png" /></figure>

# What can we do? (robustness)

## Robustness (trustworthiness) of scientific results to sources of variability

I have shown many examples of sources of variations and non-robust results...

Robustness should be rigorously defined (hint: it's not the definition as given in computer science)

How to verify the effect of sources of variations on the robustness of given conclusions?

* actionable metrics?

* methodology? (eg when to stop?)

* variability can actually be leveraged to augment confidence

<figure><img src="image_322.png" /></figure>
(computational)
science

Deep software variability is an
opportunity to robustify and augment
scientific knowledge

from a set of scripts to automate the deployment to... a
comprehensive system containing several features that
help researchers exploring various hypotheses

<figure><img src="image_323.png" /></figure>

<figure><img src="image_324.png" /></figure>

<figure><img src="image_325.png" /></figure>

<figure><img src="image_326.png" /></figure>
## Variability in the analysis of a single neuroimaging dataset by many teams

Rotem Botvinik-Nezer, Felix Holzmeister, ... Tom Schonberg 🌐 + Show authors

*Nature* **582**, 84–88 (2020) | Cite this article

42k Accesses | 203 Citations | 2056 Altmetric | Metrics

different
assumptions

<figure><img src="image_327.png" /></figure>

different data

<figure><img src="image_328.png" /></figure>

<figure><img src="image_329.png" /></figure>

deep

software

variability

<figure><img src="image_330.png" /></figure>

different analyses

<figure><img src="image_331.png" /></figure>

different
methods

## Increasing Transparency Through a Multiverse Analysis

Sara Steegen¹, Francis Tuerlinckx¹, Andrew Gelman², Wolf Vanpaemel³

Affiliations + expand

PMID: 27694465 DOI: 10.1177/1745691616658637

## Many Analysts, One Data Set: Making Transparent How Variations in Analytic Choices Affect Results

R. Silberzahn, E. L. Uhlmann, D. P. Martin, more...

Show all authors v

First Published August 23, 2018 | Research Article

https://doi.org/10.1177/2515245917747646

<figure><img src="image_332.png" /></figure>
"When you are a researcher, you want to be the first. Research is about discovery, inventing things that others have not done. It is by definition a form of competition. You have to accept that." CNRS CEO

deep
software
variability

<figure><img src="image_333.png" /></figure>

<figure><img src="image_334.png" /></figure>

At a time of global pandemics, global warming and unprecedented public distrust of politics and democracy, research is anything but an individualistic competition.

Above all, let us value the need for reproducibility in research. Let us prefer the "discovery" that will be verified or even falsified and encourage collaboration.

Deeply exploring the variability space cannot be done alone. Collaborative, distributed effort needed.

<figure><img src="image_335.png" /></figure>
Are we in ivory
towers?

I mean: We're
studying software...
for the sake of
improving software
engineering (and it's
nice!).

But should we stay?

<figure><img src="image_336.png" /></figure>

deep

software

variability

(computational)
science

<figure><img src="image_337.png" /></figure>

<figure><img src="image_338.png" /></figure>

<figure><img src="image_339.png" /></figure>

<figure><img src="image_340.png" /></figure>
Deep software variability is...

a threat for reproducible research

"Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results."

an opportunity for replication

"A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses."

"A study that refutes some scientific findings of another study, through the collection of new data (possibly with different methods) and completion of new analyses."

robustifying and augmenting
scientific knowledge

<figure><img src="image_341.png" /></figure>
# Reproducibility and replicability: A software engineering perspective

## Practical advices:

* engineer high-quality software;

* automate as much as possible;

* track every decision;

* discuss and recognize threats (deep variability?);

* explore the variants space eg ablation studies

**Deep variability** in your papers/experiments: Are you deep?

No silver bullet... it is an open and difficult problem in science!
# DEEP QUESTIONS?
<figure><img src="image_342.png" /></figure>

Florence

VaMoS'22

Variability Modelling of Software-Intensive Systems

Experimentation users-group [Xug] Meetings

# Reproducible Science and Deep Software Variability

<figure><img src="image_343.png" /></figure>

<figure><img src="image_344.png" /></figure>

<figure><img src="image_345.png" /></figure>

<figure><img src="image_346.png" /></figure>

<figure><img src="image_347.png" /></figure>

Poppler

<figure><img src="image_348.png" /></figure>

<figure><img src="image_349.png" /></figure>

<figure><img src="image_350.png" /></figure>

XZ
<figure><img src="image_351.png" /></figure>
BACKUP SLIDES
(BEYOND x264) EMPIRICAL AND FUNDAMENTAL QUESTION

HARDWARE

<figure><img src="image_352.png" /></figure>

AGE

<figure><img src="image_353.png" /></figure>

# CORES

<figure><img src="image_354.png" /></figure>

GPU

HOW DOES DEEP
SOFTWARE VARIABILITY
MANIFEST IN THE WILD?

OPERATING
SYSTEM

<figure><img src="image_355.png" /></figure>

VERSION

<figure><img src="image_356.png" /></figure>

OPTION

<figure><img src="image_357.png" /></figure>

DISTRIB.

SOFTWARE

<figure><img src="image_358.png" /></figure>

COMPIL.

VARIANT

<figure><img src="image_359.png" /></figure>

VERSION

INPUT DATA

<figure><img src="image_360.png" /></figure>

SIZE

<figure><img src="image_361.png" /></figure>

LENGTH

<figure><img src="image_362.png" /></figure>

RES.

SOFTWARE
SCIENTISTS

SHOULD
OBSERVE THE
JUNGLE/
GALAXY!
DEEP SOFTWARE VARIABILITY

4 CHALLENGES AND OPPORTUNITIES

Luc Lesoil, Mathieu Acher, Arnaud Blouin, Jean-Marc Jézéquel:
*Deep Software Variability: Towards Handling Cross-Layer Configuration*. VaMoS 2021: 10:1-10:8
# IDENTIFY THE INFLUENTIAL LAYERS

## HARDWARE

<figure><img src="image_363.png" /></figure>

AGE

<figure><img src="image_364.png" /></figure>

CPU

# CORES

<figure><img src="image_365.png" /></figure>

GPU

## OPERATING SYSTEM

<figure><img src="image_366.png" /></figure>

VERSION

<figure><img src="image_367.png" /></figure>

OPTION

<figure><img src="image_368.png" /></figure>

DISTRIB.

## SOFTWARE

<figure><img src="image_369.png" /></figure>

COMPIL.

X264

VARIANT

<figure><img src="image_370.png" /></figure>

VERSION

## INPUT DATA

<figure><img src="image_371.png" /></figure>

SIZE

<figure><img src="image_372.png" /></figure>

LENGTH

<figure><img src="image_373.png" /></figure>

RES.

**PROBLEM**

≠ LAYERS,
≠ IMPORTANCES ON
PERFORMANCES

**CHALLENGE**
ESTIMATE THEIR EFFECTS

**OPPORTUNITY**

LEVERAGE THE USEFUL
VARIABILITY LAYERS
& VARIABLES
# TEST & BENCHMARK ENVIRONMENTS

## HARDWARE

<figure><img src="image_374.png" /></figure>

<figure><img src="image_375.png" /></figure>

<figure><img src="image_376.png" /></figure>

### PROBLEM

COMBINATORIAL EXPLOSION AND COST

## OPERATING SYSTEM

<figure><img src="image_377.png" /></figure>

<figure><img src="image_378.png" /></figure>

<figure><img src="image_379.png" /></figure>

### CHALLENGE

BUILD A REPRESENTATIVE, CHEAP SET OF ENVIRONMENTS

## SOFTWARE

<figure><img src="image_380.png" /></figure>

<figure><img src="image_381.png" /></figure>

<figure><img src="image_382.png" /></figure>

0.152.2854

X264

0.155.2917

## INPUT DATA

<figure><img src="image_383.png" /></figure>

<figure><img src="image_384.png" /></figure>

<figure><img src="image_385.png" /></figure>

### OPPORTUNITY

DIMENSIONALITY REDUCTION
# TRANSFER PERFORMANCES ACROSS ENVIRONMENTS

<figure><img src="image_386.png" /></figure>

<figure><img src="image_387.png" /></figure>

A

DELL LATITUDE
7400

<figure><img src="image_388.png" /></figure>

ubuntu

20.04

<figure><img src="image_389.png" /></figure>

B

RASPBERRY PI
4 MODEL B

<figure><img src="image_390.png" /></figure>

10.4

**PROBLEM**

OPTIONS' IMPORTANCES
CHANGE WITH ENVIRONMENTS

**CHALLENGE**

TRANSFER PERFORMANCES
ACROSS ENVIRONMENTS

**OPPORTUNITY**

REDUCE COST OF MEASURE
# CROSS-LAYER TUNING

→ BUG

→ PERF. ↗

→ PERF. ↗

## HARDWARE

<figure><img src="image_391.png" /></figure>

AGE

<figure><img src="image_392.png" /></figure>

CPU

# CORES

<figure><img src="image_393.png" /></figure>

GPU

## OPERATING SYSTEM

<figure><img src="image_394.png" /></figure>

VERSION

<figure><img src="image_395.png" /></figure>

OPTION

<figure><img src="image_396.png" /></figure>

DISTRIB.

## SOFTWARE

<figure><img src="image_397.png" /></figure>

COMPIL.

X264

VARIANT

<figure><img src="image_398.png" /></figure>

VERSION

## INPUT DATA

<figure><img src="image_399.png" /></figure>

SIZE

<figure><img src="image_400.png" /></figure>

LENGTH

<figure><img src="image_401.png" /></figure>

RES.

**PROBLEM**
(NEGATIVE) INTERACTIONS
OF LAYERS

**CHALLENGE**
FIND & FIX VALUES TO IMPROVE
PERFORMANCES

**OPPORTUNITY**
SPECIALIZE THE ENVIRONMENT
FOR A USE CASE
CHALLENGES FOR DEEP SOFTWARE VARIABILITY

- IDENTIFY THE INFLUENTIAL LAYERS

- TEST & BENCHMARK ENVIRONMENTS

- TRANSFER PERFORMANCES ACROSS ENVIRONMENTS

- CROSS-LAYER TUNING
# Wrap-up

Deep software variability is a thing...

miracle and smooth fit of variability layers?

or subtle interactions among layers?

software scientists should **systematically study the phenomenon**

Many challenges and opportunities

**cross-layer tuning** at affordable cost

configuration **knowledge** that generalizes to any context and usage

Dimensionality reduction and **transfer** learning

<figure><img src="image_402.png" /></figure>

Variability layers (in time and in space)

<figure><img src="image_403.png" /></figure>
# Impacts of deep software variability

Users/developers/scientists: missed opportunities due to (accidental) variability/complexity

Deep software variability can be seen as a **threat** to knowledge/validity

**Claim:** Deep software variability is a threat to scientific, software-based experiments

**Example #1:** in the results of neuroimaging studies

* applications of different analysis pipelines (Krefting et al. 2011)

* alterations in software version (Glatard et al. 2015)

* changes in operating system (Gronenschild 2012 et al.)

have both shown to cause variation (up to the point it can change the conclusions).

**Example #2:** on a modest scale, our ICPE 2020 (Pereira et al.) showed that a variation in inputs could change the conclusion about the effectiveness of a sampling strategy for the same software system.

**Example #3:** I'm reading Zakaria Ournani et al. "Taming Energy Consumption Variations In Systems Benchmarking" ICPE 2020 as an excellent inquiry to control deep variability factors

**Example #4:** Similar observations have been made in the machine learning community (Henderson et al. 2018)

<figure><img src="image_404.png" /></figure>
# Impacts of deep software variability

Users/developers/scientists: missed opportunities due to (accidental) complexity

Deep software variability can be seen as a **threat** to knowledge/validity

Claim: Deep software variability is a threat to scientific, software-based experiments

I propose an exercise that might be slightly disturbing:

**Does deep software variability affect (your|a known) previous scientific, software-based studies?**

(remark: it's mainly how we've investigated deep software variability so far... either as a threat to our own experiments or as a threat identified in papers)
Applications of different analysis pipelines, alterations in software version, and even changes in operating system have both shown to cause variation in the results of a neuroimaging study. Example:

Reproducibility of neuroimaging analyses across operating systems

Tristan Glatard

Lindsay B. Lewis

Rafael Ferreira da Silva

Rozza Adiaat

Natacha Beck

Claude Lepage

Alan C. Evans

Pierre Rioux

Marc-Etienne Roussouaup

Tarok Sherif

Ewa Deolman

Najmeh Khalidi-Mahani

Alan C. Evans

Neuroimaging pipelines are known to generate different results depending on the computing platform where they are compiled and executed. We quantify these differences for brain tissue classification, fMRI analysis, and cortical thickness (CT) extraction, using three of the main neuroimaging packages (FSL, Freesurfer and CIVET) and different versions of GNU/Linux. We also identify some causes of these differences using library and system call interception. We find that these packages use mathematical functions based on single-precision floating-point arithmetic whose implementations in operating systems continue to evolve. While these differences have little or no impact on simple analysis pipelines such as brain extraction and cortical tissue classification, their accumulation creates important differences in longer pipelines such as subcortical tissue classification, fMRI analysis, and cortical thickness extraction. With FSL, most Dice coefficients between subcortical

DEEP SOFTWARE
VARIABILITY

Deep software variability is a threat to
"[...] results can be brittle to even minor perturbations in the domain of experimental procedure" Example:
scientific, software-based experiments

torch.marl seed (3407) is all you need: On the influence of
autoencoders and deep learning architectures on computer
vision

David Picard

LIGM, École des Ponts, 77455 Marnes la vallée, France

DAVID.PICARD@ENPC.FR

DEEP SOFTWARE
VARIABILITY

Abstract

In this paper I investigate the effect of random seed selection on the accuracy when using popular deep learning architectures for computer vision. I scan a large amount of seeds (up to $10^4$) on CIFAR 10 and I also scan fewer seeds on Imagenet using pre-trained models to investigate large scale datasets. The conclusions are that even if the variance is not very large, it is surprisingly easy to find an outlier that performs much better or much worse than the average.
<figure><img src="image_405.png" /></figure>
# AGENDA

Interactive course: Reproducible Science and (Deep) Software (Variability)

Is $(x+y)+z == x+(y+z)?$ (TP1)

A suspicious paper in machine learning... (TP2)
Is $(x+y)+z == x+(y+z)$?
Is $(x+y)+z == x+(y+z)$?

In Informatics, not in Mathematics

$x, y, z$: number

Choose the language, the compiler/interpreter, the library, the computing environment, the computer you want...

And tell us!
Is $(x+y)+z == x+(y+z)$?

In Informatics, not in Mathematics

$x, y, z$: number

Choose the programming language, the compiler/interpreter, the library, the computing environment, the computer you want...

Is your work reproducible?
$$Is (x+y)+z == x+(y+z)?

How often $(x+y)+z == x+(y+z)$?
Is $(x+y)+z == x+(y+z)$?

How often $(x+y)+z == x+(y+z)$?

write a program that returns a percentage

again: Choose the programming language, the compiler/interpreter, the library, the computing environment, the computer you want...
<figure><img src="image_406.png" /></figure>

Code

Blame

58 lines (53 loc) · 1.2 KB

```c
#include <stdio.h>
#include <stdlib.h>
#define XMASK (char)(99)
static int randseed=-1;

/* Macro to return a uniform distributed random real number between 0 and 1 */
/* s is the seed, bb the multiplier. 'Maxint &' is used to mask the sign bit*/
#define Maxint (0x7fffffff)
#define invmaxint 4.656612875e-10
#define bb 31415821
#ifdef CUSTOM
#define uniform (double)(invmaxint*(randseed=(Maxint & ((randseed*bb)-1))))
#else
#ifdef WIN
#define uniform rand()
#else
#define uniform drand48()
#endif
#endif

int associativity_test()
{
    double x = uniform; double y = uniform; double z = uniform;
    // printf("%f %f %f %d\n",x,y,z,x+(y+z) == (x+y)+z);
    return x+(y+z) == (x+y)+z;
}

float proportion(int number)
{
    int ok=0;
    for (int i=0;i<number;i++) ok += associativity_test();
    return ok*100.0/number;
}

/* Here is the entry point of the program */
#ifdef OLD MAIN_C
void main(argc, argv)
```
```c
int main(int argc, char **argv) {
    #else
    #endif
    #ifdef WIN
    srand(123);
    #else
    srand48(123);
```

https://github.com/FAMILIAR-project/reproducibility-associativity/
<table><thead><tr><th>Language</th><th>Library</th><th>System</th><th>Compiler</th><th>VariabilityMisc</th><th>EqualityCheck</th><th>NumberGenerations</th><th>Repeat</th><th>min</th><th>max</th><th>std</th><th>mean</th></tr></thead><tbody><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV</td><td>100</td><td>10</td><td>60.0</td><td>71.0</td><td>3.562302626111375</td><td>65.1</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>51.0</td><td>63.0</td><td>3.330165161069343</td><td>55.9</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV</td><td>100</td><td>10</td><td>62.0</td><td>62.0</td><td>0.0</td><td>62.0</td></tr><tr><td>Perl</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>47.0</td><td>47.0</td><td>0.0</td><td>47.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>associativity</td><td>100</td><td>10</td><td>71.0</td><td>82.0</td><td>3.3466401061363023</td><td>76.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>mult-inverse</td><td>100</td><td>10</td><td>58.0</td><td>78.0</td><td>6.0</td><td>66.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed None</td><td>mult-inverse-pi</td><td>100</td><td>10</td><td>42.0</td><td>64.0</td><td>5.885575587824865</td><td>53.4</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>associativity</td><td>100</td><td>10</td><td>81.0</td><td>81.0</td><td>0.0</td><td>81.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse</td><td>100</td><td>10</td><td>70.0</td><td>70.0</td><td>0.0</td><td>70.0</td></tr><tr><td>Go</td><td></td><td></td><td></td><td>seed 42</td><td>mult-inverse-pi</td><td>100</td><td>10</td><td>56.0</td><td>56.0</td><td>0.0</td><td>56.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV</td><td>100</td><td>10</td><td>62.0</td><td>72.0</td><td>2.764054992217051</td><td>66.6</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed None</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>47.0</td><td>57.0</td><td>2.808914381037628</td><td>53.1</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV</td><td>100</td><td>10</td><td>67.0</td><td>67.0</td><td>0.0</td><td>67.0</td></tr><tr><td>R</td><td></td><td></td><td></td><td>seed 42</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>53.0</td><td>53.0</td><td>0.0</td><td>53.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>74.0</td><td>90.0</td><td>4.6097722286464435</td><td>82.5</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>MULT_INV</td><td>100</td><td>10</td><td>60.0</td><td>79.0</td><td>6.16765757804371</td><td>68.6</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>49.0</td><td>59.0</td><td>2.8301943396169813</td><td>54.3</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None strict-equality</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>89.0</td><td>89.0</td><td>0.0</td><td>89.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_INV</td><td>100</td><td>10</td><td>73.0</td><td>73.0</td><td>0.0</td><td>73.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed 42 strict-equality</td><td>MULT_INV_PI</td><td>100</td><td>10</td><td>55.0</td><td>55.0</td><td>0.0</td><td>55.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None approximate equality of Julia lang</td><td>ASSOCIATIVITY</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr><tr><td>Julia</td><td></td><td></td><td></td><td>seed None approximate equality of Julia lang</td><td>MULT_INV</td><td>100</td><td>10</td><td>100.0</td><td>100.0</td><td>0.0</td><td>100.0</td></tr></tbody></table>

https://github.com/FAMILIAR-project/reproducibility-associativity/
Please contribute!

new language

new compiler/interpreter

new options/flags

source code

Dockerfile, nix, guix

https://github.com/FAMILIAR-project/reproducibility-associativity/
Is $(x+y)+z == x+(y+z)$? How often $(x+y)+z == x+(y+z)$?

My banker proposed this investment to me:

You give me e ≈ 2.71828...

The following year, I take 1 euro as a fee and multiply by 1.

The next year, I take 1 euro as a fee and multiply by 2.

The next year, I take 1 euro as a fee and multiply by 3.

...

After n years, I take 1 euro as a fee and multiply by n.

To retrieve my money, there is a 1 euro fee.

In 50 years, for my retirement, how much money will I have?

Write a program that answers this question
https://www.college-de-france.fr/fr/agenda/seminaire/sema
ntiques-mecanisees-quand-la-machine-raisonne-sur-ses-l
angages/arithmetique-des-ordinateurs-et-sa-formalisation

Sylvie Boldo talk (the previous example was based on her talk, thanks!)

Reproduce everything (figures, tables, etc.) that has been
said!

Replicate! (change programming language, compiler flags
or versions, etc.)
# AGENDA

Interactive course: Reproducible Science and (Deep) Software (Variability)

Is $(x+y)+z == x+(y+z)$? (TP1)

A suspicious paper in machine learning... (TP2)
# A suspicious paper in machine learning... (TP2)

Paul (Temple) slides and material here:

https://github.com/templep/EJCP2023
<figure><img src="image_407.png" /></figure>

android

<figure><img src="image_408.png" /></figure>

debian

GNU/Linux

<figure><img src="image_409.png" /></figure>

ubuntu

<figure><img src="image_410.png" /></figure>

<figure><img src="image_411.png" /></figure>

<figure><img src="image_412.png" /></figure>

<figure><img src="image_413.png" /></figure>

<figure><img src="image_414.png" /></figure>

<figure><img src="image_415.png" /></figure>

<figure><img src="image_416.png" /></figure>

<figure><img src="image_417.png" /></figure>

<figure><img src="image_418.png" /></figure>

<figure><img src="image_419.png" /></figure>

<figure><img src="image_420.png" /></figure>

<figure><img src="image_421.png" /></figure>

15,000+
options

<figure><img src="image_422.png" /></figure>

TRISTATE 61.63

BOOL 36.40

INT 1.54

STRING 0.29

HEX 0.14

3<sup>9000</sup>

2<sup>6000</sup>

≈10<sup>6000</sup> variants

(without constraints)

Linux 5.2.8, arm
(% of types' options)
<figure><img src="image_423.png" /></figure>

<figure><img src="image_424.png" /></figure>

android

<figure><img src="image_425.png" /></figure>

debian

GNH: Linux

<figure><img src="image_426.png" /></figure>

ubuntu

## Linux Kernel

≈10^6000 variants

≈10^80 is the estimated number of atoms in the universe

≈10^40 is the estimated number of possible chess positions

<figure><img src="image_427.png" /></figure>
<figure><img src="image_428.png" /></figure>

Florence

# VaMoS'22

Variability Modelling of Software-Intensive Systems

**Abstract** : Biology, medicine, physics, astrophysics, chemistry: all these scientific domains need to process large amount of data with more and more complex software systems. For achieving reproducible science, there are several challenges ahead involving multidisciplinary collaboration and socio-technical innovation with software at the center of the problem. Despite the availability of data and code, several studies report that the same data analyzed with different software can lead to different results. I am seeing this problem as a manifestation of deep software variability: many factors (operating system, third-party libraries, versions, workloads, compile-time options and flags, etc.) themselves subject to variability can alter the results, up to the point it can dramatically change the conclusions of some scientific studies. In this keynote, I argue that deep software variability is a threat and also an opportunity for reproducible science. I first outline some works about (deep) software variability, reporting on preliminary evidence of complex interactions between variability layers. I then link the ongoing works on variability modelling and deep software variability in the quest for reproducible science.

**References** at the ~end of the slides!
## Associative Property of Addition

**You:** Is $(x+y)+z == x+(y+z)$ in informatics?

**ChatGPT:** Yes, in most programming languages, the statement $(x + y) + z == x + (y + z)$ holds true due to the associative property of addition. This property states that the way numbers are grouped doesn't change the sum, which applies both in mathematics and informatics.

However, keep in mind that in the case of floating point numbers, because of the way they are represented internally, the order of operations can sometimes impact the result due to round-off errors. This is more the exception than the rule, and in general, the property will hold true.

**You:** Give me an example with floating point numbers

**ChatGPT:** Sure, I can give you an example in Python where floating point precision may make a difference due to the order of operations. However, please remember that these cases are rare and generally the order won't matter with addition. The issue arises from the fact that computers cannot precisely represent some decimal values.

```python
# Python code
x = 1.0
y = 1e-16  # 0.0000000000000001
z = -1.0

# Add in one order
first_order = (x + y) + z
print(first_order)  # prints: 1e-16
```

```python
# Add in another order
second_order = x + (y + z)
print(second_order)  # prints: 0.0
```

In this example, $(x + y) + z$ is not equal to $x + (y + z)$ due to floating point precision errors. In the first case, x and y are added first, resulting in a value slightly greater than 1.0. Then z is added, yielding a small positive number $(1e-16)$. In the second case, y and z are added first, which exactly cancel each other out because they are precisely opposite in value. Then x is added to zero, yielding 1.0.

It's crucial to note that this discrepancy arises because of the limitations of floating-point precision and not a failure of the associative property.

**You:** Is it true for any programming language?
**ChatGPT:** The behavior regarding floating point precision is consistent across virtually all modern programming languages because they typically use the same underlying standard for floating point numbers: the IEEE 754 standard.

So, yes, the precision issues you might encounter in Python with floating point numbers can also appear in other languages like C++, Java, JavaScript, and so on. This is because the inaccuracies originate from the way computers represent floating point numbers, not from the programming languages themselves.

That said, there might be some differences in how each language handles these precision issues, and some languages provide ways to mitigate precision errors. For instance, some languages like Python have built-in libraries for arbitrary-precision arithmetic, or you might use fixed-point arithmetic in situations where it's crucial to maintain a certain level of precision.

**You:** What about Rust?

**ChatGPT:** Rust, like other modern programming languages, follows the IEEE 754 standard for floating point arithmetic. This means that the precision issues that can arise with floating point numbers in languages like Python, C++, Java, and JavaScript can also occur in Rust.

Here is an example in Rust similar to the Python example above:

```rust
fn main() {
    let x = 1.0f64;
    let y = 1e-16f64;
    let z = -1.0f64;

    let first_order = (x + y) + z;
    println!("{}", first_order); // prints: 1e-16

    let second_order = x + (y + z);
    println!("{}", second_order); // prints: 0.0
}
```

This Rust code will have the same outcome as the Python code. `(x + y) + z` does not equal `x + (y + z)` due to floating point precision errors. Again, it's important to note that this is due to the limitations of floating point representation and not a failure of the associative property.

**You:** what about C?

**ChatGPT:** Yes, the same floating point precision issue would occur in the C programming language as well. Here's the equivalent code to the previous examples:
```c
#include <stdio.h>

int main() {
    double x = 1.0;
    double y = 1e-16;
    double z = -1.0;

    double first_order = (x + y) + z;
    printf("%e\n", first_order); // prints: 1.000000e-16

    double second_order = x + (y + z);
    printf("%e\n", second_order); // prints: 0.000000e+00

    return 0;
}
```

This code will behave just like the Python and Rust examples, with `(x + y) + z` not equalling `x + (y + z)` due to floating point precision errors. Just like in other languages, this issue arises from how computers represent floating point numbers, and not from a failure of the associative property.

**You:** replace double by float?

**ChatGPT:** In C, a `float` has less precision than a `double`. The `double` type has more bits of precision than the `float` type, so the effects of floating-point precision issues can be even more noticeable with `float`. Here's an example:

```c
#include <stdio.h>

int main() {
    float x = 1.0f;
    float y = 1e-16f;
    float z = -1.0f;

    float first_order = (x + y) + z;
    printf("%e\n", first_order); // prints: 0.000000e+00

    float second_order = x + (y + z);
    printf("%e\n", second_order); // prints: 0.000000e+00

    return 0;
}
```

In this case, you'll notice that both `first_order` and `second_order` print `0.000000e+00`. This happens because `1e-16` is too small to be represented accurately using a `float`, so it gets rounded down to `0.0f`. Therefore, both `(x + y) + z` and `x + (y + z)` end up as `0.0f`. So, in this specific case, the two
expressions are equal, but not for the reason you might expect; it's due to lack of precision in the `float` type rather than the associative property of addition.

**You:** using `float` in C, can you devise a counterexample of `(x+y)+z == x+(y+z)`

**ChatGPT:** Certainly! When using `float` in C, the effects of precision issues are more noticeable due to its lower precision compared to `double`. Here's an example where `(x + y) + z != x + (y + z)` due to precision errors:

```c
#include <stdio.h>
```

```c
int main() {
    float x = 1.0e10f;
    float y = 1.0f;
    float z = -1.0e10f;

    float first_order = (x + y) + z;
    printf("%f\n", first_order); // prints: 1.000000

    float second_order = x + (y + z);
    printf("%f\n", second_order); // prints: 0.000000

    return 0;
}
```

In this case, the precision of `float` is not sufficient to hold the result of `x + y` accurately because `y` is significantly smaller than `x`. Therefore, the result of `(x + y) + z` is 1.0 due to the loss of precision in `x + y`. On the other hand, `y + z` equals -1.0e10 exactly because `y` and `z` are of comparable magnitude, and then adding `x` results in 0.0. So, `(x + y) + z` is not equal to `x + (y + z)`.

**You:** using `double` in the C program, any chance to mitigate the error in precision with compiler flags?

**ChatGPT:** While there may be compiler flags to control floating-point behavior, they typically won't solve the issue at hand. The precision problem isn't something a compiler can easily fix; it's a fundamental limitation of the floating-point number system used in most computer hardware.

For instance, the GCC and Clang compilers offer some flags to control floating point behavior (like `-ffloat-store`, `-fexcess-precision`, `-frounding-math`, etc.). However, these flags mainly affect the way intermediate results are stored or how rounding is handled. They are not capable of increasing the precision of `double` beyond its inherent limits according to the IEEE 754 standard.
If you need more precision than `double` can provide and want to avoid these types of errors, you might need to use libraries for arbitrary precision arithmetic, such as GMP or MPFR, but these will likely make your computations slower. Another approach is to rearrange your equations so that you avoid subtracting nearly equal numbers or adding a very small number to a very large one, as these are the types of operations that tend to lead to significant loss of precision.

**You:** what about `-mfpmath=387`?

**ChatGPT:** The `-mfpmath=387` option is specific to the GCC compiler on x86 platforms. It instructs the compiler to use the floating-point unit on the x87 coprocessor for floating-point calculations instead of the SSE/SSE2 units.

The x87's FPU operates at a higher internal precision (80 bits) than the usual `double` (which is 64 bits). So, theoretically, computations done using the x87's FPU can be more accurate because of the extra precision during intermediate calculations.

However, the difference in precision between the x87 FPU and SSE/SSE2 only matters for the intermediate calculations. The final stored result will still be truncated to fit into a `double` or `float`, so it doesn't inherently increase the precision of your final results.

Moreover, the use of `-mfpmath=387` may not guarantee that the rounding errors would be eliminated completely. In complex calculations, the final results may still suffer from floating point precision issues due to the inherent limitations of the floating-point representation.

In summary, while `-mfpmath=387` might provide slightly more accurate results in some specific cases due to higher precision in intermediate steps, it doesn't completely mitigate the limitations of the floating-point number system. If absolute precision is required, consider using libraries that support arbitrary-precision arithmetic, or apply numerical methods that are designed to minimize errors.

**You:** what about `clang`?

**ChatGPT:** The `Clang` compiler, unlike `GCC`, does not currently support the `-mfpmath` option (as of my knowledge cutoff in September 2021).

`Clang` and `GCC` are designed to be mostly command-line compatible, so for many options, if `GCC` has a flag, `Clang` is likely to have the same or a similar one. However, this is not the case for `-mfpmath`. The main reason is that `-mfpmath=387` is a very hardware-specific flag tied to the x87 floating point unit, which is not used by `Clang` for floating point computations by default. `Clang` typically uses `SSE` for floating point arithmetic on x86 and x86_64.
As a result, if you're using Clang and you need to increase the precision of your floating point calculations, you'll need to explore other options. You might want to consider using a library for arbitrary precision arithmetic or a library for multiprecision floating-point computations, such as MPFR, or changing the way you perform your calculations to minimize the impact of rounding errors.

It's always best to check the most recent compiler documentation, as compiler features and flags can change over time.

**You:** can you adapt the float example with GMP?

**ChatGPT:** The GNU Multiple Precision Arithmetic Library (GMP) provides functions for multiple precision arithmetic. However, GMP mainly provides functions for integer and rational number calculations, not floating-point calculations.

If you need more precision for floating-point calculations, the GNU MPFR library (Multiple Precision Floating-Point Reliable) is a good choice. It is built on top of GMP and provides functions for high-precision floating-point arithmetic.

Here is an example of how you might use MPFR to calculate the expression in a way that demonstrates the associative property of addition:

```c
#include <stdio.h>
#include <mpfr.h>

int main() {
    mpfr_t x, y, z, first_order, second_order, temp;

    // Initialize variables
    mpfr_init2(128, x, y, z, first_order, second_order, temp, (mpfr_ptr) 0);

    mpfr_set_d(x, 1.0e10, MPFR_RNDN);
    mpfr_set_d(y, 1.0, MPFR_RNDN);
    mpfr_set_d(z, -1.0e10, MPFR_RNDN);

    // (x + y) + z
    mpfr_add(temp, x, y, MPFR_RNDN);
    mpfr_add(first_order, temp, z, MPFR_RNDN);

    // x + (y + z)
    mpfr_add(temp, y, z, MPFR_RNDN);
    mpfr_add(second_order, x, temp, MPFR_RNDN);

    // Print results
    mpfr_printf("(x + y) + z = %.15Rf\n", first_order);
    mpfr_printf("x + (y + z) = %.15Rf\n", second_order);
```
Do note that ` bigDecimal ` operations can be slower than primitive floating-point operations, due to the overhead of object creation and the more complex algorithms used for high-precision arithmetic. Hence, ` bigDecimal ` should be used when the precision requirement outweighs the need for speed.

**You:** the same Java example but using `float`?

**ChatGPT:** Sure! Here's the example in Java using `float`:

```java
public class Main {
    public static void main(String[] args) {
        float x = 1.0e10f;
        float y = 1.0f;
        float z = -1.0e10f;

        float firstOrder = (x + y) + z;
        System.out.println("(x + y) + z = " + firstOrder);

        float secondOrder = x + (y + z);
        System.out.println("x + (y + z) = " + secondOrder);
    }
}
```

In this example, just like in the C example, `(x + y) + z` is not equal to `x + (y + z)` due to floating point precision errors. Java's `float` has a precision of about 7 decimal digits, so it is not able to represent the result of `x + y` accurately because `y` is significantly smaller than `x`. Therefore, the result of `(x + y) + z` is 1.0 due to the loss of precision in `x + y`. On the other hand, `y + z` equals `-1.0e10` exactly, and then adding `x` results in 0.0.