# Increasing computational reproducibility or how-to share *your* research code

Camille Maumet

► To cite this version:

Camille Maumet. Increasing computational reproducibility or how-to share *your* research code. MRI Together 2024 - Global workshop on open, reproducible, and inclusive MR, Dec 2024, Online, Australia. hal-04814565v2

HAL Id: hal-04814565

https://inria.hal.science/hal-04814565v2

Submitted on 2 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# Increasing computational reproducibility or how-to share *your* research code

Camille Maumet

Inria, Univ Rennes, Inserm, CNRS, IRISA
An MRI study

<figure><img src="image_3.png" /></figure>

Acquisition

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

Pre-processing

<figure><img src="image_6.png" /></figure>

Derived data

<figure><img src="image_7.png" /></figure>

Statistics

<figure><img src="image_8.png" /></figure>
An MRI study

<figure><img src="image_9.png" /></figure>

Acquisition

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

Pre-processing

<figure><img src="image_12.png" /></figure>

Derived data

<figure><img src="image_13.png" /></figure>

Statistics

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>
An MRI study

<figure><img src="image_16.png" /></figure>

Acquisition

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>

Pre-processing

<figure><img src="image_19.png" /></figure>

Derived data

<figure><img src="image_20.png" /></figure>

Statistics

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>

Image credits: MRI by Flowicon <u>Noun Project</u> (CC BY), data process by Konkapp <u>Noun Project</u> (CC-BY), analysis by Siti Solekah <u>Noun Project</u> (CC-BY)
An MRI study

<figure><img src="image_25.png" /></figure>

Acquisition

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>

Pre-processing

<figure><img src="image_28.png" /></figure>

Derived data

<figure><img src="image_29.png" /></figure>

Statistics

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

Image credits: MRI by Flowicon <u>Noun Project</u> (CC-BY), analysis by Siti Solekah <u>Noun Project</u> (CC-BY)
An MRI study

<figure><img src="image_33.png" /></figure>

Acquisition

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

Pre-processing

<figure><img src="image_36.png" /></figure>

Derived data

<figure><img src="image_37.png" /></figure>

Statistics

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>
An MRI study

Acquisition

<figure><img src="image_41.png" /></figure>

Pre-processing

<figure><img src="image_42.png" /></figure>

<figure><img src="image_43.png" /></figure>

Data

<figure><img src="image_44.png" /></figure>

Statistics

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>
An **MRI** study

<figure><img src="image_49.png" /></figure>
An MRI study

<figure><img src="image_50.png" /></figure>

<figure><img src="image_51.png" /></figure>
An MRI study

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

Which version of
the SPM software
was used?

Which registration
algorithm?

How was the brain
mask computed?
An MRI study

<figure><img src="image_54.png" /></figure>

Humm... I can't
reproduce the
published results...

<figure><img src="image_55.png" /></figure>

Did I understand
correctly the
method that was
used?

Which version of
the SPM software
was used?

Which registration
algorithm?

How was the brain
mask computed?

... is this due to my
(different) data?

... or is this due to
my (different)
computing
resources?

... or even
something
else?
An MRI study

<figure><img src="image_56.png" /></figure>

Humm... I can't
reproduce the
published results...

<figure><img src="image_57.png" /></figure>

Which version of
the SPM software
was used?

Which registration
algorithm?

How was the brain
mask computed?

Is this due to my
different) data?

Is this due to
(different)
computing
resources?

... or even
something
else?
# Towards reproducible research

<figure><img src="image_58.png" /></figure>

Acquisition

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>

Pre-processing

<figure><img src="image_61.png" /></figure>

Derived data

<figure><img src="image_62.png" /></figure>

Statistics

<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>

<figure><img src="image_65.png" /></figure>

Image credits: MRI by Flowicon Noun Project (CC BY), data process by Konkapp Noun Project (CC-BY), analysis by Siti Solekah Noun Project (CC-BY)
# Towards reproducible research

<figure><img src="image_66.png" /></figure>

Acquisition

<figure><img src="image_67.png" /></figure>

<figure><img src="image_68.png" /></figure>

Pre-processing

<figure><img src="image_69.png" /></figure>

Derived data

<figure><img src="image_70.png" /></figure>

Statistics

<figure><img src="image_71.png" /></figure>

<figure><img src="image_72.png" /></figure>

**Open access**

(pre-prints, diamond
open access, etc.)

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>
# Towards reproducible research

<figure><img src="image_75.png" /></figure>

Acquisition

<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>

Pre-processing

<figure><img src="image_78.png" /></figure>

<figure><img src="image_79.png" /></figure>

Derived data

<figure><img src="image_80.png" /></figure>

Statistics

<figure><img src="image_81.png" /></figure>

<figure><img src="image_82.png" /></figure>

<figure><img src="image_83.png" /></figure>

<figure><img src="image_84.png" /></figure>

<figure><img src="image_85.png" /></figure>

<figure><img src="image_86.png" /></figure>

**Open data**
(FAIR principles)

Public-nEUro

Radboud Data
Repository

<figure><img src="image_87.png" /></figure>

OpenNEURO

<figure><img src="image_88.png" /></figure>

<figure><img src="image_89.png" /></figure>

LORIS

shaneir

<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /></figure>

**Open access**

(pre-prints, diamond
open access, etc.)

<figure><img src="image_92.png" /></figure>

<figure><img src="image_93.png" /></figure>
# Towards reproducible research

<figure><img src="image_94.png" /></figure>

Image credits: MRI by Flowicon [Noun Project](CC BY), data process by Konkapp [Noun Project](CC-BY), analysis by Siti Solekah [Noun Project](CC-BY)
# Towards **reproducible research**

**Computational reproducibility**

Ability to reproduce **exactly the same results** using the **same data** and **same code**.

**Replicability**

Ability to reach the **same scientific findings** using
**similar data** and a **similar method**.
# Towards **reproducible research**

**Computational reproducibility**

Ability to reproduce **exactly the same results** using the **same data** and **same code**.

**Replicability**

Ability to reach the **same scientific findings** using
**similar data** and a **similar method**.
# Open code

<figure><img src="image_95.png" /></figure>

Open source

Transparency

* With an open **source licence** and **publicly accessible**.

<figure><img src="image_96.png" /></figure>

<figure><img src="image_97.png" /></figure>
<figure><img src="image_98.png" /></figure>

Archived

Sustainability

* Archived for **long-term preservation**
  (e.g. using Software Heritage, Zenodo)

<figure><img src="image_99.png" /></figure>

Software Heritage

zenodo

https://zenodo.org/

<figure><img src="image_100.png" /></figure>

https://www.softwareheritage.org/
# Open code

<figure><img src="image_101.png" /></figure>

## Using coding best practices

### Reutilisability

<figure><img src="image_102.png" /></figure>

## Following software development best practices

* **Versioning**: using a forge (Github, gitlab, etc.).

* **Listing dependencies**.

* **README**, file tree template, etc.

```
|-- CITATION
```

```
|-- README
```

```
|-- LICENSE
```

```
|-- requirements.txt
```

```
|-- data
```

```
|-- birds_count_table.csv
```

```
|-- doc
```

```
|-- notebook.md
```

```
|-- manuscript.md
```

```
|-- changelog.txt
```

```
|-- results
```

```
|-- summarized_results.csv
```

```
|-- src
```

```
|-- sightings_analysis.py
```

```
|-- runall.py
```

### Project/Paper Title

### Table of contents

* How to cite?

* Contents overview

* Reproducing figures and tables

* Table 1

* Fig. 1

* Fig. 2

* Reproducing full analysis

### How to cite?

See CITATION.

### Contents overview

<Summarise what's in this repository>

### Reproducing figures and tables

<Instructions on how to use summary/derived data in the results directory to create figures and tables>

<Specify precise steps, including any datasets that need to be downloaded and path variables that need to be set>

### Table 1

Fig. 1

Fig. 2

### Reproducing full analysis

<Instructions on how to (1) obtain raw data; (2) process it to create summary/derived data in the results>

<Specify precise steps, including any datasets that need to be downloaded and path variables that need to be set>
# How-to **open** your **code**

<figure><img src="image_103.png" /></figure>

**Open source**
*Transparency*

* with an open **source licence** and **publicly accessible**.

**Archived**
*Sustainability*

* in an **archive** for **long-term preservation** (e.g. using Software Heritage, Zenodo)

**Using coding best practices**
*Reutilisability*

* following **software coding best practices**

* **Versioning** using a forge (Github, GitLab, etc.)

* **Listing dependencies**

* **README**, file tree template, etc.
# Are my results (computationally) reproducible now? **Well...**

Beyond source code, **environnements** (OS, libraries, etc.) can also have an impact...

→ *docker, Guix, Nix*

**Computational** reproducibility **in 10 years** is still an open question...

Perkel, Jeffrey M. "Challenge to scientists: does your ten-year-old code still run?." *Nature* 584.7822 (2020): 656-659.
# Towards **reproducible research**

**Computational reproducibility**

Ability to reproduce **exactly the same results** using the **same data** and **same code**.

**Replicability**

Ability to reach the **same scientific findings** using
**similar data** and a **similar method**.
# Towards **reproducible research**

**Computational reproducibility**

Ability to reproduce **exactly the same results** using the **same data** and **same code**.

**Replicability**

Ability to reach the **same scientific findings** using
**similar data** and a **similar method**.
# Towards **reproducible research**

**Computational reproducibility**

Ability to reproduce **exactly the same results** using the **same data** and **same code**.

**Replicability**

Ability to reach the **same scientific findings** using
**similar data** and a **similar method**.

**Reusability**

Ability to reuse some research **artefacts for new purposes**
# Towards **reproducible research**

**Computational reproducibility**

Ability to reproduce **exactly the same results** using the **same data** and **same code**.

**Replicability**

Ability to reach the **same scientific findings** using
**similar data** and a **similar method**.

**Reusability**

Ability to reuse some research **artefacts for new purposes**
# Open software

<figure><img src="image_104.png" /></figure>

<figure><img src="image_105.png" /></figure>

## Open source

* Choose an **open source license**

* Make **code available for download**

* Use an **archive** for sustainability
# Open software

<figure><img src="image_106.png" /></figure>

Who can **access** and **use** the software?

<figure><img src="image_107.png" /></figure>

## Open source

* Choose an **open source license**

* Make **code available for download**

* Use an **archive** for sustainability
<figure><img src="image_108.png" /></figure>

# Open software

Who can **access** and **use** the software?

<figure><img src="image_109.png" /></figure>

## Open source

* Choose an **open source license**

* Make **code available for download**

* Use an **archive** for sustainability

Who can
**contribute**?

<figure><img src="image_110.png" /></figure>

## Open development

* Accept **external contributions** (e.g. Github PR)

* Be transparent in terms of **development choices** (e.g. issues, project board)
# Open software

Who can **access** and **use** the software?

<figure><img src="image_111.png" /></figure>

## Open source

* Choose an **open source license**

* Make **code available for download**

* Use an **archive** for sustainability

Who can
**contribute**?

<figure><img src="image_112.png" /></figure>

## Open development

* Accept **external contributions** (e.g. Github PR)

* Be transparent in terms of **development choices** (e.g. issues, project board)

Who can **decide** for **future directions**?

<figure><img src="image_113.png" /></figure>

## Open governance

* Transparency in how decision are made

* Possibility to join the **leadership of the project**
# Towards a more **collaborative** science

**Beyond technical solutions, open science brings a new culture**

* Sharing resources towards the **scientific community**

* **Reusing existing resources** (code, data etc.) generated by others

**toward a more collaborative research.**

<figure><img src="image_114.png" /></figure>
# Towards a more **collaborative** science

Beyond technical solutions, open science brings a new culture

* Sharing resources towards the **scientific community**

* **Reusing existing resources** (code, data etc.) generated by others

toward a more collaborative research.

<figure><img src="image_115.png" /></figure>

**Open science awards** for
open software / open data /
open PhD thesis

https://www.ouvrirlascience.fr/category/openscience-prices/

<figure><img src="image_116.png" /></figure>
# Increasing computational reproducibility or how-to share *your* research code

Thank you

Camille Maumet

<figure><img src="image_117.png" /></figure>

**Credit:** Presentation template and icons by SlidesCarnival, adapted

<figure><img src="image_118.png" /></figure>

@cmaumet.bsky.social

@cmaumet@fediscience.org