<figure><img src="image_1.png" /></figure>

HAL
open science

# Reproducible neuroimaging: a review on past and emerging questions

Camille Maumet

<figure><img src="image_2.png" /></figure>

## To cite this version:

Camille Maumet. Reproducible neuroimaging: a review on past and emerging questions. 2023 -
Journées Recherche reproductible: état des lieux, Mar 2023, Paris, France., pp.1-27. hal-04020177v3

HAL Id: hal-04020177

https://inria.hal.science/hal-04020177v3

Submitted on 16 Mar 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_4.png" /></figure>

INSTITUT PASTEUR

8 & 9 MARS 2023

<figure><img src="image_5.png" /></figure>

Reproducible neuroimaging: a review
on past and emerging questions

Reproductibilité en neuroimagerie
état des lieux

March 8, 2023

Camille Maumet

Univ Rennes, Inria, CNRS, Inserm
# Reproducible brain imaging

Acquisition

<figure><img src="image_6.png" /></figure>

Data preparation

<figure><img src="image_7.png" /></figure>

Derived data

Statistical analysis

<figure><img src="image_8.png" /></figure>
# Reproducible brain imaging

Acquisition

<figure><img src="image_9.png" /></figure>

Different / Same
Data

Data preparation

<figure><img src="image_10.png" /></figure>

Derived data

Statistical analysis

<figure><img src="image_11.png" /></figure>

Different / Same
Method

Different / Same
Experimenter
# Towards **reproducible** neuroimaging

Irreproducible with...

Different Data

<figure><img src="image_12.png" /></figure>

about 30 participants
per study

<figure><img src="image_13.png" /></figure>

[Poldrack et. al, Nature Neuroscience 2017]
# Towards **reproducible** neuroimaging

Irreproducible with...

Different Data
Irreproducible with...

Different Data

## Lack of power

### Power of neuroscience studies

Power = Prob. to correctly find a significant effect when a the alternative hypothesis is true.

<figure><img src="image_14.png" /></figure>

[Button et. al, Nat Rev Neurosci 2013]

We need bigger datasets
# Towards reproducible neuroimaging

Irreproducible with...

Different Data

## Lack of power

### Power of neuroscience studies

Power = Prob. to correctly find a significant effect when a the alternative hypothesis is true.

<figure><img src="image_15.png" /></figure>

[Button et. al, Nat Rev Neurosci 2013]

We need bigger datasets

## Generalizability and fairness

### X-ray: Lung opacity detection

Model trained on male images, tested on female images

<figure><img src="image_16.png" /></figure>

[Larrazabal et. al, PNAS 2020]

We need representative and diverse datasets
# Open data

<figure><img src="image_17.png" /></figure>

Unique study

30 participants

<figure><img src="image_18.png" /></figure>

OpenNEURO

<figure><img src="image_19.png" /></figure>

NEUROVAULT

studyforrest.org

L I E L N L F T Q K T Q R V
S M Y C O N N E C T O M E Q
G S P K K W A R R G K E H R

<figure><img src="image_20.png" /></figure>

NITRC

<figure><img src="image_21.png" /></figure>

OSF

+ Images

+ Homogenous

- Datasets
# Open data

<figure><img src="image_22.png" /></figure>

Unique study

30 participants

<figure><img src="image_23.png" /></figure>

OpenNEURO

<figure><img src="image_24.png" /></figure>

NEUROVAULT

<figure><img src="image_25.png" /></figure>

NITRC

<figure><img src="image_26.png" /></figure>

OSF

Consortium

1000 participants

<figure><img src="image_27.png" /></figure>

ABIDE

Autism Brain Imaging
Data Exchange

1000 Functional Connectomes Project

L I E L N L F T Q K T Q R V
S M Y C O N N E C T O M E Q
G S P K K W A R R G K E H R

<figure><img src="image_28.png" /></figure>

CORR

CONSORTIUM FOR
RELIABILITY AND
REPRODUCIBILITY

<figure><img src="image_29.png" /></figure>

+ Images

+ Homogenous

- Datasets
# Open data

<figure><img src="image_30.png" /></figure>

Unique study

30 participants

<figure><img src="image_31.png" /></figure>

OpenNEURO

<figure><img src="image_32.png" /></figure>

NEUROVAULT

<figure><img src="image_33.png" /></figure>

NITRC

<figure><img src="image_34.png" /></figure>

OSF

studyforrest.org

L I E L N L F T Q K T Q R V
S M Y C O N N E C T O M E Q
G S P K K W A R R G K E H R

Consortium

1000 participants

<figure><img src="image_35.png" /></figure>

ABIDE

Autism Brain Imaging
Data Exchange

<figure><img src="image_36.png" /></figure>

CORR

CONSORTIUM FOR
RELIABILITY AND
REPRODUCIBILITY

<figure><img src="image_37.png" /></figure>

Cohort

100 000 participants

1000 Functional Connectomes Project

<figure><img src="image_38.png" /></figure>

HUMAN
Connectome
PROJECT

biobank™
Improving the health of future generations

+ Images

+ Homogenous

- Datasets
# On our way to fix irreproducibility

Irreproducible with...

Different Post-processing Methods

<figure><img src="image_39.png" /></figure>

Data preparation

<figure><img src="image_40.png" /></figure>

Derived data

Analysis

<figure><img src="image_41.png" /></figure>
# On our way to fix irreproducibility

Irreproducible with...

Different Post-processing Methods

<figure><img src="image_42.png" /></figure>
# On our way to fix irreproducibility

Irreproducible with...

Different Post-processing Methods

<figure><img src="image_43.png" /></figure>
# On our way to fix irreproducibility

Irreproducible with...

Different Post-processing Methods

<figure><img src="image_44.png" /></figure>

<figure><img src="image_45.png" /></figure>
# On our way to fix irreproducibility

Irreproducible with...

Different Post-processing Methods

<figure><img src="image_46.png" /></figure>
# On our way to fix irreproducibility

Irreproducible with...

Different Post-processing Methods

<figure><img src="image_47.png" /></figure>
# On our way to fix irreproducibility

Irreproducible with...

## Different Post-processing Methods

<figure><img src="image_48.png" /></figure>
# On our way to fix irreproducibility

Irreproducible with...

## Different Post-processing Methods

<figure><img src="image_49.png" /></figure>

<figure><img src="image_50.png" /></figure>

A family of acceptable pipelines
over 10^30 combinations...
**Many analysts project : NARPS**

<figure><img src="image_51.png" /></figure>

Data preparation

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

Derived data

<figure><img src="image_54.png" /></figure>

Analysis

<figure><img src="image_55.png" /></figure>

<figure><img src="image_56.png" /></figure>
# Many analysts project : NARPS

<figure><img src="image_57.png" /></figure>

1 dataset

<figure><img src="image_58.png" /></figure>

Data preparation

<figure><img src="image_59.png" /></figure>

<figure><img src="image_60.png" /></figure>

Derived data

<figure><img src="image_61.png" /></figure>

Analysis

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

9 'yes/no' questions

e.g. Is there a parametric effect of gain:
Positive effect in ventromedial PFC -
for the equal indifference group?
# Many analysts project : NARPS

<figure><img src="image_64.png" /></figure>
**Many analysts project : NARPS**

Results over the **70 post-processing pipelines**:

<figure><img src="image_65.png" /></figure>
# Many analysts project : NARPS

## Results over the 70 post-processing pipelines

<figure><img src="image_66.png" /></figure>

**Significant finding**
(>80% of teams agree)

**Non-significant finding**
(>80% of teams agree)
**Many analysts project : NARPS**

Results over the **70 post-processing pipelines**:

<figure><img src="image_67.png" /></figure>

**Significant finding**
(>80% of teams agree)

Contradictory results
(Teams disagree)

**Non-significant finding**
(>80% of teams agree)
# Different levels of variability in post-processing pipelines

* Different end-to-end pipelines [Carp 2013]

* Different Software Version [Gronenschild 2012]

* Different operating system [Glatard 2015]

* Different post-processing software packages [Bowring 2019, 2021]
# Open questions

## Stratégie adoptée par votre communauté

* sur le plan technique

* Partage de code

* pour les carrières

* Data papers

* en terme d'Animation

* Open science community (OHBM open science SIG, Brainhack)

* Hackathons collaboratifs "Brainhack"

## Exemple de problèmes techniques et scientifiques

* Open data in Europe (GDPR, infrastructures)

* Causes of analytical variability (different methods) not well understood
**Reproductibilité computationnelle en neuroimagerie:**
**état des lieux**

<figure><img src="image_68.png" /></figure>

Camille Maumet

Merci

<figure><img src="image_69.png" /></figure>

<figure><img src="image_70.png" /></figure>