# Towards reproducible neuroimaging across different analysis pipelines

Camille Maumet

<figure><img src="image_1.png" /></figure>

## To cite this version:

Camille Maumet. Towards reproducible neuroimaging across different analysis pipelines. CogBases 2023, Oct 2023, Paris, France. 2023. [hal-04236791](https://hal HAL Id: hal-04236791)

https://inria.hal.science/hal-04236791

Submitted on 11 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>
<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

# Towards reproducible neuroimaging

across different analysis pipelines

CogBases workshop 2023

Camille Maumet

Inria, Univ Rennes, Inserm, CNRS, IRISA
# Reproducible brain imaging

Acquisition

<figure><img src="image_5.png" /></figure>

Data preparation

<figure><img src="image_6.png" /></figure>

Derived data

Statistical analysis

<figure><img src="image_7.png" /></figure>
# Reproducible brain imaging

Acquisition

<figure><img src="image_8.png" /></figure>

Different / Same Data

Data preparation

<figure><img src="image_9.png" /></figure>

Derived data

Statistical analysis

<figure><img src="image_10.png" /></figure>

Different / Same Method

Different / Same Experimenter
<figure><img src="image_11.png" /></figure>

<figure><img src="image_12.png" /></figure>

On our way to fix
irreproducibility...
Towards **reproducible** neuroimaging

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>

Data preparation

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>

Derived data

<figure><img src="image_17.png" /></figure>

Analysis

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>
# Towards **reproducible** neuroimaging

Irreproducible with...

Same Data, Same Methods

<figure><img src="image_20.png" /></figure>

Data preparation

<figure><img src="image_21.png" /></figure>

Derived data

Analysis

<figure><img src="image_22.png" /></figure>

?

<figure><img src="image_23.png" /></figure>

* Sharing code

* Sharing environments: Docker, guix
<figure><img src="image_24.png" /></figure>

Irreproducible with...

Different Data

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

Data preparation

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>

Derived data

<figure><img src="image_29.png" /></figure>

Analysis

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>
<figure><img src="image_32.png" /></figure>

Irreproducible with...

Different Data

<figure><img src="image_33.png" /></figure>

Data preparation

<figure><img src="image_34.png" /></figure>

Derived data

Analysis

<figure><img src="image_35.png" /></figure>
Irreproducible with...

Different Data

## Lack of power

### Power of neuroscience studies

Power = Prob. to correctly find a significant effect when a the alternative hypothesis is true.

<figure><img src="image_36.png" /></figure>

[Button et. al, Nat Rev Neurosci 2013]

We need bigger datasets
Irreproducible with...

Different Data

## Lack of power

**Power of neuroscience studies**

Power = Prob. to correctly find a significant effect when a the alternative hypothesis is true.

<figure><img src="image_37.png" /></figure>

[Button et. al, Nat Rev Neurosci 2013]

We need bigger datasets

## Generalizability and fairness

**X-ray: Lung opacity detection**

Model trained on male images, tested on female images

<figure><img src="image_38.png" /></figure>

[Larrazabal et. al, PNAS 2020]

We need representative
and diverse datasets
<figure><img src="image_39.png" /></figure>

# Different pipelines... different results?
<figure><img src="image_40.png" /></figure>

Irreproducible with...

Different Analysis

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>

Data preparation

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

Derived data

<figure><img src="image_45.png" /></figure>

Analysis

<figure><img src="image_46.png" /></figure>

<figure><img src="image_47.png" /></figure>
<figure><img src="image_48.png" /></figure>

Irreproducible with...

Different Analysis

<figure><img src="image_49.png" /></figure>
Irreproducible with...

Different Analysis

<figure><img src="image_50.png" /></figure>
# Towards **reproducible** neuroimaging

Irreproducible with...

Different Analysis

<figure><img src="image_51.png" /></figure>
# Towards **reproducible** neuroimaging

Irreproducible with...

Different Analysis

<figure><img src="image_52.png" /></figure>

<figure><img src="image_53.png" /></figure>

≠ algorithm

<figure><img src="image_54.png" /></figure>

≠ software

<figure><img src="image_55.png" /></figure>

≠ software version

<figure><img src="image_56.png" /></figure>

≠ parameters
# Towards **reproducible** neuroimaging

Irreproducible with...

Different Analysis

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>

≠ algorithm

<figure><img src="image_59.png" /></figure>

≠ software

<figure><img src="image_60.png" /></figure>

≠ software version

<figure><img src="image_61.png" /></figure>

≠ parameters

A family of acceptable pipelines
over 10^30 combinations...
**Many-analyst project : NARPS**

<figure><img src="image_62.png" /></figure>

Data preparation

<figure><img src="image_63.png" /></figure>

<figure><img src="image_64.png" /></figure>

Derived data

<figure><img src="image_65.png" /></figure>

Analysis

<figure><img src="image_66.png" /></figure>

<figure><img src="image_67.png" /></figure>
**Many-analyst project : NARPS**

<figure><img src="image_68.png" /></figure>

1 dataset

<figure><img src="image_69.png" /></figure>

Data preparation

<figure><img src="image_70.png" /></figure>

<figure><img src="image_71.png" /></figure>

Derived data

<figure><img src="image_72.png" /></figure>

Analysis

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>

9 'yes/no' questions

e.g. Is there a parametric effect of gain:
Positive effect in ventromedial PFC -
for the equal indifference group?
**Many-analyst project : NARPS**

<figure><img src="image_75.png" /></figure>
**Many-analyst project : NARPS**

Results over the **70 post-processing pipelines**:

<figure><img src="image_76.png" /></figure>
**Many-analyst project : NARPS**

Results over the **70 post-processing pipelines**:

<figure><img src="image_77.png" /></figure>

**Significant finding**
(>80% of teams agree)

All 'yes'

**Non-significant finding**
(>80% of teams agree)

All 'no'
**Many-analyst project : NARPS**

Results over the **70 post-processing pipelines**:

<figure><img src="image_78.png" /></figure>

**Significant finding**
(>80% of teams agree)

All 'yes'

Contradictory results
(Teams disagree)

**Non-significant finding**
(>80% of teams agree)

All 'no'
# Different levels of variability in post-processing pipelines

* Different end-to-end pipelines [Carp 2012]

* Different software version [Gronenschild 2012]

* Different operating system [Glatard 2015]

* Different post-processing software packages [Bowring 2019, 2021]
# Variability across software

* Reproduced 3 published functional MRI studies

* Using 3 different software

<figure><img src="image_79.png" /><figcaption>Alex Bowring Tom Nichols</figcaption></figure>

<figure><img src="image_80.png" /><figcaption>Software 1</figcaption></figure>

Pipeline

<figure><img src="image_81.png" /><figcaption>Software 2</figcaption></figure>

<figure><img src="image_82.png" /></figure>

Pipeline

<figure><img src="image_83.png" /></figure>

<figure><img src="image_84.png" /><figcaption>Software 3</figcaption></figure>

<figure><img src="image_85.png" /></figure>

Pipeline

<figure><img src="image_86.png" /></figure>

<figure><img src="image_87.png" /></figure>
Variability across software

<figure><img src="image_88.png" /></figure>

Thresholded maps in AFNI,
FSL, SPM + original
# Variability across software

<figure><img src="image_89.png" /></figure>

<figure><img src="image_90.png" /></figure>

[Bowring et. al, HBM 2021]

Thresholded maps in AFNI, FSL, SPM + original

[Bowring et. al, HBM 2019]
<figure><img src="image_91.png" /></figure>

# Understanding
analytical variability
**Understanding** analytical variability...

?

<figure><img src="image_92.png" /></figure>
**Understanding** analytical variability...

?

<figure><img src="image_93.png" /></figure>

Numerical
uncertainty
(e.g. ≠ OS)

[Glatard 2015,
Kiar 2021]

Different
implen.

Unwanted
variability

[Gronenschild 2012,
Bowring 2019]
# Understanding analytical variability...

?

<figure><img src="image_94.png" /></figure>

Numerical
uncertainty
(e.g. ≠ OS)

Assumption
violations

Unwanted
variability

Different
implen.

Unreliable
results

Data
induced
variability
**Understanding** analytical variability...

?

Better fit to the data

Variability of interest

?

Assumption violations

<figure><img src="image_95.png" /></figure>

Numerical uncertainty (e.g. ≠ OS)

Unwanted variability

Unreliable results

Different implem.

Data induced variability
<figure><img src="image_96.png" /></figure>

# Describing the neuroimaging pipeline space
# NARPS open pipelines

<figure><img src="image_97.png" /><figcaption>Boris
Clenet</figcaption></figure>

<figure><img src="image_98.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_99.png" /><figcaption>Jérémy
Lefort-Besnard</figcaption></figure>

<figure><img src="image_100.png" /></figure>

Data preparation

<figure><img src="image_101.png" /></figure>

Derived data

Analysis

<figure><img src="image_102.png" /></figure>

1 dataset

70 Post-processing
by different teams

<figure><img src="image_103.png" /></figure>

<figure><img src="image_104.png" /></figure>

<figure><img src="image_105.png" /></figure>

<figure><img src="image_106.png" /></figure>

<figure><img src="image_107.png" /></figure>

<figure><img src="image_108.png" /></figure>

9 'yes/no' questions
# NARPS open pipelines

<figure><img src="image_109.png" /><figcaption>Boris
Clenet</figcaption></figure>

<figure><img src="image_110.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_111.png" /><figcaption>Jérémy
Lefort-Besnard</figcaption></figure>

<figure><img src="image_112.png" /></figure>

Data preparation

<figure><img src="image_113.png" /></figure>

Derived data

Analysis

<figure><img src="image_114.png" /></figure>

1 dataset

70 Post-processing
by different teams

<figure><img src="image_115.png" /></figure>

<figure><img src="image_116.png" /></figure>

<figure><img src="image_117.png" /></figure>

<figure><img src="image_118.png" /></figure>

<figure><img src="image_119.png" /></figure>

<figure><img src="image_120.png" /></figure>

9 'yes/no' questions

## NARPS open pipelines

- (Re)-coding the 70 pipelines

- Collaborative effort

<figure><img src="image_121.png" /></figure>

**Project NARPS open pipelines**

Join us at the Brainhack Global!

<figure><img src="image_122.png" /></figure>

https://github.com/Inria-Empenn/narps_open_pipelines/
# Provenance

"Provenance" describe how an artifact or set of artifacts was produced" [Wikipedia](https://.wikipedia.org/)

"Provenance" is information about entities, activities, and people involved in producing a piece of data [...]" [W3C PROV](https://www.w3.org/TR/prov/)

<figure><img src="image_123.png" /></figure>

W3C PROV model

[https://www.w3.org/TR/2013/NOTE-prov-primer-20130430/](https://www.w3.org/TR/2013/NOTE-prov-primer-20130430/)

wasDerivedFrom

wasAttributedTo

Agent

used

wasAssociatedWith

Entity

wasGeneratedBy

Activity
Provenance Use Case 1 **An unambiguous description of the methods**

<figure><img src="image_124.png" /></figure>
Provenance Use Case 1 **An unambiguous description of the methods**

<figure><img src="image_125.png" /></figure>

<figure><img src="image_126.png" /></figure>

Which release of
SPM was used?

Which registration
algorithm was
selected?

How was the brain
mask computed?
Provenance Use Case 1 **An unambiguous description of the methods**

Can we
**reproduce**
**exactly the same**
**results**
(same data, same
methods)?
Provenance Use Case 1 **An unambiguous description of the methods**

<figure><img src="image_127.png" /></figure>

Can we
reproduce
exactly the same
results
(same data, same
methods)?
## Provenance Use Case 2 **Dealing with
unwanted variability**

> A previously
> widely-used
> parameter is no
> longer considered
> as best practice
## Provenance Use Case 2 **Dealing with
unwanted variability**

A previously
widely-used
parameter is no
longer considered
as best practice

A bug was found
in a given tool
version, can we
retrieve all
corresponding
results?
## Provenance Use Case 2 Dealing with unwanted variability

<figure><img src="image_128.png" /></figure>

A previously widely-used parameter is no longer considered as best practice

A bug was found in a given tool version, can we retrieve all corresponding results?
Provenance Use Case 3 **Exploring factors impacting analytical variability**

<figure><img src="image_129.png" /></figure>

Are my conclusions dependant on the **size of the smoothing kernel** used at Step 5?

<figure><img src="image_130.png" /></figure>

**BIDS-Prov** will soon be open for community feedback!

<figure><img src="image_131.png" /></figure>

http://bids.neuroimaging.io/bep028
<figure><img src="image_132.png" /></figure>

# Which solutions for practitioners?
# Data re-use

<figure><img src="image_133.png" /><figcaption>Xavier
Rolland</figcaption></figure>

<figure><img src="image_134.png" /><figcaption>Pierre
Maurel</figcaption></figure>

<figure><img src="image_135.png" /></figure>
# Data re-use

<figure><img src="image_136.png" /><figcaption>Xavier
Rolland</figcaption></figure>

<figure><img src="image_137.png" /><figcaption>Pierre
Maurel</figcaption></figure>

<figure><img src="image_138.png" /></figure>

* Using HCP data under the null

* Comparisons of pipelines 2-by-2

* 3 variations

* Smoothing: 5 or 8

* HRF derivative: yes or no

* Motion regressors: 0, 6, 24

* "Default" : 5mm, 24 regressors, no temp. deriv.

<figure><img src="image_139.png" /></figure>
# Data re-use

<figure><img src="image_140.png" /><figcaption>Xavier Rolland</figcaption></figure>

<figure><img src="image_141.png" /><figcaption>Pierre Maurel</figcaption></figure>

<figure><img src="image_142.png" /></figure>

* Using HCP data under the null

* Comparisons of pipelines 2-by-2

* 3 variations

* Smoothing: 5 or 8

* HRF derivative: yes or no

* Motion regressors: 0, 6, 24

* "Default" : 5mm, 24 regressors, no temp. deriv.

<figure><img src="image_143.png" /></figure>

<figure><img src="image_144.png" /></figure>
<figure><img src="image_145.png" /><figcaption>Elodie
Germani</figcaption></figure>

# Building on large and more diverse samples

* Latent representation of fMRI statistic maps based on NeuroVault

* Apply to a **decoding task**

<figure><img src="image_146.png" /></figure>

NEUROVAULT

neurovault.org

<figure><img src="image_147.png" /></figure>

[Germani et. al, Gigascience 2023]

<figure><img src="image_148.png" /></figure>

https://inria.hal.science/hal-03769993
<figure><img src="image_149.png" /><figcaption>Jérémy</figcaption></figure>

<figure><img src="image_150.png" /><figcaption>Tom</figcaption></figure>

Lefort-Besnard Nichols

# Same data meta-analysis

* A statistical approach to **combine the results of different pipelines** and obtain a **single (consensus) summary**

<figure><img src="image_151.png" /></figure>

**Poster** Systematic review and evaluation of
meta-analysis methods for same data
meta-analyses Jérémy Lefort-Besnard

## Same data meta-analysis

### Enter the multiverse...

Lefort-Besnard Jeremy¹, Nichols Thomas²*, Maumet Camille¹†

¹Inria, Univ Rennes, CNRS, Inserm, IRISA UMR 6074, Empaena ERL U 1225, Rennes, France

²Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, Nuffield Department of Population Health, University of Oxford, Oxford, UK

†Sharing last authorship

### Problem

Researchers using (MRI data have a wide range of analysis tools to model brain activity. This diversity of analytical approaches means there are many possible variations of the same imaging result (Bowing et al., 2019). Analyzing a dataset with a single approach can thus be misleading.

Alternatively, a multiverse analysis can be used, where multiple sets of results are obtained from running different pipelines on the same single dataset. Such a setting produces several test statistic maps. Meta-analysis approaches can then help to effectively extract valid and robust results from these maps.

A required assumption for traditional meta-analysis is the independence among input datasets (Fig 1a). This assumption is no longer true in multiverse setting (Fig 1b), thus treating dependent studies as independent may lead to invalidity and thus decrease the generalizability of the result.

Therefore, we present a variety of same data meta-analysis (SDMA) models. The validity and accuracy of these models were assessed in a set of different simulations as well as on a real world dataset from "NARPS", a multiverse analysis with 70 different statistic maps originating from the same data. We also contrasted the outcomes of these SDMA models with those of a conventional meta-analysis approach.

<figure><img src="image_152.png" /><figcaption>Fig 1: Traditional meta-analyses setting (a) and multiverse setting (b). The traditional approach assumes independence among images (K studies and thus K pipelines computed with N different participants each) when estimating summary effect. This underlying assumption is undermined in the multiverse setting where the outcomes are dependent over pipelines (One unique set of participants data for each pipeline).</figcaption></figure>

<figure><img src="image_153.png" /></figure>
<figure><img src="image_154.png" /></figure>

<figure><img src="image_155.png" /></figure>

# Towards reproducible neuroimaging
across different analysis
pipelines

Camille Maumet

Thank you!

<figure><img src="image_156.png" /></figure>

We are hiring
[https://team.inria.fr/empenn/job-offers](https://team.inria.fr/empenn/job-offers)

Credit: Presentation template and icons by
[SlidesCarnival](https:// SlidesCarnival, adapted

<figure><img src="image_157.png" /></figure>

@cmaumet@fediscience.org

<figure><img src="image_158.png" /></figure>

@cmaumet