# Towards reproducible neuroimaging - across different analysis pipelines

Camille Maumet

► To cite this version:

Camille Maumet. Towards reproducible neuroimaging - across different analysis pipelines. QLS 2024 - McGill Seminar Series in Quantitative Life Sciences and Medicine, Feb 2024, Montreal (Online), Canada. 2024. hal-04479231

HAL Id: hal-04479231

https://inria.hal.science/hal-04479231

Submitted on 27 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Towards reproducible neuroimaging - across different analysis pipelines

Camille Maumet

► To cite this version:

Camille Maumet. Towards reproducible neuroimaging - across different analysis pipelines. QLS 2024 - McGill Seminar Series in Quantitative Life Sciences and Medicine, Feb 2024, Montreal (Online), Canada. 2024. hal-04479231

HAL Id: hal-04479231

https://inria.hal.science/hal-04479231v1

Submitted on 27 Feb 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

# Towards reproducible neuroimaging

across different analysis pipelines

## McGill Seminar Series in Quantitative Life Sciences and Medicine

### Camille Maumet

Inria, Univ Rennes, Inserm, CNRS, IRISA
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

# Towards reproducible neuroimaging

across different analysis pipelines

## McGill Seminar Series in Quantitative Life Sciences and Medicine

### Camille Maumet

Inria, Univ Rennes, Inserm, CNRS, IRISA
# A crisis in experimental research

The reproducibility crisis has led to reduced
confidence in research findings

Low reproduction rates in many fields

Cancer research: <11%

Psychology: 36%

Medicine: 44%

(Begley & Ellis 2012) (Open Science Collab 2016) (Ioannidis 2005)
# A crisis in experimental research

The reproducibility crisis has led to reduced
confidence in research findings

Low reproduction rates in many fields

Cancer research: <11%

Psychology: 36%

Medicine: 44%

(Begley & Ellis 2012) (Open Science Collab 2016) (Ioannidis 2005)
# A crisis in experimental research

The reproducibility crisis has led to reduced confidence in research findings

Low reproduction rates in many fields

Cancer research: <11%

Psychology: 36%

Medicine: 44%

(Begley & Ellis 2012) (Open Science Collab 2016) (Ioannidis 2005)

Wasted money
& effort for
research

<figure><img src="image_4.png" /></figure>

Delayed
translation into
clinical practice

<figure><img src="image_5.png" /></figure>

Reduced trust
in science

<figure><img src="image_6.png" /></figure>
# A crisis in experimental research

The reproducibility crisis has led to reduced confidence in research findings

## Low reproduction rates in many fields

- Cancer research: <11%

- Psychology: 36%

- Medicine: 44%

(Begley & Ellis 2012) (Open Science Collab 2016) (Ioannidis 2005)

## Wasted money & effort for research

<figure><img src="image_4.png" /></figure>

## Delayed translation into clinical practice

<figure><img src="image_5.png" /></figure>

## Reduced trust in science

<figure><img src="image_6.png" /></figure>
# Reproducible research

<figure><img src="image_7.png" /></figure>

Data

<figure><img src="image_8.png" /></figure>

Analysis

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

Results
# Reproducible research

<figure><img src="image_7.png" /></figure>

Data

<figure><img src="image_8.png" /></figure>

Analysis

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

Results
# Reproducible research

## Original study

<figure><img src="image_11.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_11.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_14.png" /></figure>

<figure><img src="image_15.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_16.png" /></figure>

<figure><img src="image_17.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_18.png" /></figure>

Data

Analysis

Results

Which release of SPM was used?

Which registration algorithm was selected?

... or is this due to my use of a different dataset?

How was the brain mask computed?

Did we understand the method correctly?

... a different computing platform?

... something else!?

Hum... **we can't reproduce** the original results...

<figure><img src="image_19.png" /></figure>

≠ Data

≠ Analysis

<figure><img src="image_20.png" /></figure>

Results

<figure><img src="image_21.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_18.png" /></figure>

Data

Analysis

Results

Which release of SPM was used?

Which registration algorithm was selected?

... or is this due to my use of a different dataset?

How was the brain mask computed?

Did we understand the method correctly?

... a different computing platform?

... something else!?

Hum... **we can't reproduce** the original results...

<figure><img src="image_19.png" /></figure>

≠ Data

≠ Analysis

<figure><img src="image_20.png" /></figure>

Results
# Reproducible research

## Original study

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

<figure><img src="image_27.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_24.png" /></figure>

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>
# Reproducible research

<figure><img src="image_28.png" /></figure>
# Reproducible research

Original study

<figure><img src="image_27.png" /></figure>

same data

different data

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>

Replicability

same analysis

<figure><img src="image_30.png" /></figure>

<figure><img src="image_31.png" /></figure>

different analysis
# Reproducible research

Original study

<figure><img src="image_29.png" /></figure>

same data

<figure><img src="image_30.png" /></figure>

different data

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

same analysis different analysis

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

<figure><img src="image_37.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_37.png" /></figure>

## Reproductions

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>
# Reproducible research

<figure><img src="image_38.png" /></figure>
<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>

On our way to fix
irreproducibility...
<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>

On our way to fix
irreproducibility...
# Reproducible research

## Original study

<figure><img src="image_45.png" /></figure>

## Reproducibility

<figure><img src="image_46.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_41.png" /></figure>

## Reproducibility

<figure><img src="image_42.png" /></figure>
# Fixing irreproducibility with different data

Small samples

Low statistical power

Power of neuroscience studies

<figure><img src="image_47.png" /></figure>

(Button et. al, 2013)

High rate of **false positives**
in the literature
# Fixing irreproducibility with different data

Small samples

Low statistical power

Power of neuroscience studies

<figure><img src="image_43.png" /></figure>

(Button et. al, 2013)

High rate of **false positives**
in the literature
# Fixing irreproducibility with different data

## Small samples
## Low statistical power

Power of neuroscience studies

<figure><img src="image_48.png" /></figure>

(Button et. al, 2013)

High rate of **false positives**
in the literature

## Lack of representativity and diversity of the samples

**X-ray: Lung opacity detection**

Model trained on male images, tested on female images

<figure><img src="image_49.png" /></figure>

(Larrazabal et. al, 2020)

Lack of **generalisability**
and **fairness**
# Fixing irreproducibility with different data

## Small samples
## Low statistical power

Power of neuroscience studies

<figure><img src="image_44.png" /></figure>

(Button et. al, 2013)

High rate of **false positives**
in the literature

## Lack of representativity and diversity of the samples

**X-ray: Lung opacity detection**

Model trained on male images, tested on female images

<figure><img src="image_45.png" /></figure>

(Larrazabal et. al, 2020)

Lack of **generalisability**
and **fairness**
# Fixing irreproducibility with different data

Small samples
Low statistical power

Power of neuroscience studies

<figure><img src="image_50.png" /></figure>

(Button et. al, 2013)

Lack of representativity and
diversity of the samples

X-ray: Lung opacity detection

Model trained on male images, tested on female images

<figure><img src="image_51.png" /></figure>

(Larrazabal et. al, 2020)

High rate of **false positives**
in the literature

Lack of **generalisability**
**and fairness**

Solution: Data
Sharing
# Fixing irreproducibility with different data

Small samples

Low statistical power

Power of neuroscience studies

<figure><img src="image_46.png" /></figure>

(Button et. al, 2013)

Lack of representativity and
diversity of the samples

X-ray: Lung opacity detection

Model trained on male images, tested on female images

<figure><img src="image_47.png" /></figure>

(Larrazabal et. al, 2020)

High rate of **false positives**
in the literature

Lack of **generalisability**
**and fairness**

Solution: Data
Sharing
# Neuroimaging data sharing

Single study

30 sub.

<figure><img src="image_52.png" /></figure>

OpenNEURO

<figure><img src="image_53.png" /></figure>

NEUROVAULT

<figure><img src="image_54.png" /></figure>

NITRC

<figure><img src="image_55.png" /></figure>

OSF
# Neuroimaging data sharing

Single study

30 sub.

<figure><img src="image_48.png" /></figure>
# Neuroimaging data sharing

Single study

30 sub.

<figure><img src="image_56.png" /></figure>

OpenNEURO

<figure><img src="image_57.png" /></figure>

NEUROVAULT

<figure><img src="image_58.png" /></figure>

NITRC

<figure><img src="image_59.png" /></figure>

OSF

Consortium

1000 sub.

<figure><img src="image_60.png" /></figure>

ABIDE

Autism Brain Imaging
Data Exchange

1000 Functional

Connectomes Project

<figure><img src="image_61.png" /></figure>

CRR

CONSORTIUM FOR
RELIABILITY AND
REPRODUCIBILITY

<figure><img src="image_62.png" /></figure>
# Neuroimaging data sharing

Single study

30 sub.

<figure><img src="image_49.png" /></figure>

OpenNEURO

<figure><img src="image_50.png" /></figure>

NEUROVAULT

<figure><img src="image_51.png" /></figure>

NITRC

<figure><img src="image_52.png" /></figure>

OSF

Consortium

1000 sub.

<figure><img src="image_53.png" /></figure>

ABIDE

Autism Brain Imaging
Data Exchange

1000 Functional

Connectomes Project

<figure><img src="image_54.png" /></figure>

CRR

CONSORTIUM FOR
RELIABILITY AND
REPRODUCIBILITY

<figure><img src="image_55.png" /></figure>
# Neuroimaging data sharing

Single study
30 sub.

<figure><img src="image_63.png" /></figure>

OpenNEURO

<figure><img src="image_64.png" /></figure>

NEUROVAULT

<figure><img src="image_65.png" /></figure>

NITRC

<figure><img src="image_66.png" /></figure>

OSF

Consortium
1000 sub.

<figure><img src="image_67.png" /></figure>

ABIDE

Autism Brain Imaging
Data Exchange

<figure><img src="image_68.png" /></figure>

CRR

CONSORTIUM FOR
RELIABILITY AND
REPRODUCIBILITY

1000 Functional

Connectomes Project

<figure><img src="image_69.png" /></figure>

Cohort
100 000 sub.

<figure><img src="image_70.png" /></figure>

HUMAN
Connectome
PROJECT

biobank™
# Neuroimaging data sharing

Single study
30 sub.

<figure><img src="image_56.png" /></figure>

OpenNEURO

<figure><img src="image_57.png" /></figure>

NEUROVAULT

<figure><img src="image_58.png" /></figure>

NITRC

<figure><img src="image_59.png" /></figure>

OSF

Consortium
1000 sub.

<figure><img src="image_60.png" /></figure>

ABIDE

Autism Brain Imaging
Data Exchange

<figure><img src="image_61.png" /></figure>

CRR

CONSORTIUM FOR
RELIABILITY AND
REPRODUCIBILITY

1000 Functional

Connectomes Project

<figure><img src="image_62.png" /></figure>

Cohort
100 000 sub.

<figure><img src="image_63.png" /></figure>

HUMAN
Connectome
PROJECT

biobank™
<figure><img src="image_71.png" /></figure>

# Different pipelines... different results?
<figure><img src="image_64.png" /></figure>

# Different pipelines... different results?
# Reproducible research

## Original study

<figure><img src="image_72.png" /></figure>

## Reproducibility

<figure><img src="image_73.png" /></figure>

<figure><img src="image_74.png" /></figure>

<figure><img src="image_75.png" /></figure>

<figure><img src="image_76.png" /></figure>

<figure><img src="image_77.png" /></figure>
# Reproducible research

## Original study

<figure><img src="image_65.png" /></figure>

## Reproducibility

<figure><img src="image_66.png" /></figure>

<figure><img src="image_67.png" /></figure>

<figure><img src="image_68.png" /></figure>

<figure><img src="image_69.png" /></figure>

<figure><img src="image_70.png" /></figure>
# Reproducible research

<figure><img src="image_78.png" /></figure>

**Computational Reproducibility**

<figure><img src="image_79.png" /></figure>

<figure><img src="image_80.png" /></figure>

Data

≠ Analysis

<figure><img src="image_81.png" /></figure>

Results

<figure><img src="image_82.png" /></figure>

<figure><img src="image_83.png" /></figure>

Results

<figure><img src="image_84.png" /></figure>

≠ Data

≠ Analysis

<figure><img src="image_85.png" /></figure>

Results

**Replicability**

different analysis

same analysis
# Reproducible research

<figure><img src="image_71.png" /></figure>
A multiplicity of analysis pipelines

<figure><img src="image_86.png" /></figure>
# A multiplicity of analysis pipelines

<figure><img src="image_72.png" /></figure>
# A multiplicity of analysis pipelines

<figure><img src="image_87.png" /></figure>
# A multiplicity of analysis pipelines

<figure><img src="image_73.png" /></figure>
# A multiplicity of analysis pipelines

<figure><img src="image_88.png" /></figure>

Input data

<figure><img src="image_89.png" /></figure>

<figure><img src="image_90.png" /></figure>

Results
# A multiplicity of analysis pipelines

<figure><img src="image_74.png" /></figure>

Input data

<figure><img src="image_75.png" /></figure>

<figure><img src="image_76.png" /></figure>

Results
# A multiplicity of analysis pipelines

<figure><img src="image_91.png" /></figure>

Input data

<figure><img src="image_92.png" /></figure>

<figure><img src="image_93.png" /></figure>

Results
# A multiplicity of analysis pipelines

<figure><img src="image_77.png" /></figure>

Input data

<figure><img src="image_78.png" /></figure>

<figure><img src="image_79.png" /></figure>

Results
# A multiplicity of analysis pipelines

<figure><img src="image_94.png" /></figure>

Input data

<figure><img src="image_95.png" /></figure>

<figure><img src="image_96.png" /></figure>

Results

<figure><img src="image_97.png" /></figure>

≠ algorithm
# A multiplicity of analysis pipelines

<figure><img src="image_80.png" /></figure>

Input data

<figure><img src="image_81.png" /></figure>

<figure><img src="image_82.png" /></figure>

Results

<figure><img src="image_83.png" /></figure>

≠ algorithm
# A multiplicity of analysis pipelines

<figure><img src="image_98.png" /></figure>

Input data

<figure><img src="image_99.png" /></figure>

<figure><img src="image_100.png" /></figure>

Results

<figure><img src="image_101.png" /></figure>

<figure><img src="image_102.png" /></figure>
# A multiplicity of analysis pipelines

<figure><img src="image_84.png" /></figure>

Input data

<figure><img src="image_85.png" /></figure>

<figure><img src="image_86.png" /></figure>

Results

<figure><img src="image_87.png" /></figure>

≠ algorithm

<figure><img src="image_88.png" /></figure>

≠ software
# A multiplicity of analysis pipelines

<figure><img src="image_103.png" /></figure>

Input data

<figure><img src="image_104.png" /></figure>

<figure><img src="image_105.png" /></figure>

Results

<figure><img src="image_106.png" /></figure>

<figure><img src="image_107.png" /></figure>

<figure><img src="image_108.png" /></figure>
# A multiplicity of analysis pipelines

<figure><img src="image_89.png" /></figure>

Input data

<figure><img src="image_90.png" /></figure>

<figure><img src="image_91.png" /></figure>

Results

<figure><img src="image_92.png" /></figure>

<figure><img src="image_93.png" /></figure>

<figure><img src="image_94.png" /></figure>
# A multiplicity of analysis pipelines

<figure><img src="image_109.png" /></figure>

Input data

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

Results

<figure><img src="image_112.png" /></figure>

≠ algorithm

<figure><img src="image_113.png" /></figure>

<figure><img src="image_114.png" /></figure>

≠ parameters

<figure><img src="image_115.png" /></figure>

<figure><img src="image_116.png" /></figure>

≠ software

<figure><img src="image_117.png" /></figure>

<figure><img src="image_118.png" /></figure>

≠ software version
# A multiplicity of analysis pipelines

<figure><img src="image_95.png" /></figure>

Input data

<figure><img src="image_96.png" /></figure>

<figure><img src="image_97.png" /></figure>

Results

<figure><img src="image_98.png" /></figure>

≠ algorithm

<figure><img src="image_99.png" /></figure>

≠ parameters

<figure><img src="image_100.png" /></figure>

<figure><img src="image_101.png" /></figure>

≠ software

<figure><img src="image_102.png" /></figure>

<figure><img src="image_103.png" /></figure>

≠ software version
# A multiplicity of analysis pipelines

<figure><img src="image_119.png" /></figure>

Input data

<figure><img src="image_120.png" /></figure>

<figure><img src="image_121.png" /></figure>

Results

<figure><img src="image_122.png" /></figure>

≠ algorithm

<figure><img src="image_123.png" /></figure>

<figure><img src="image_124.png" /></figure>

≠ parameters

<figure><img src="image_125.png" /></figure>

<figure><img src="image_126.png" /></figure>

≠ software

<figure><img src="image_127.png" /></figure>

<figure><img src="image_128.png" /></figure>

≠ environment

<figure><img src="image_129.png" /></figure>

<figure><img src="image_130.png" /></figure>

≠ software version
# A multiplicity of analysis pipelines

<figure><img src="image_104.png" /></figure>

Input data

<figure><img src="image_105.png" /></figure>

<figure><img src="image_106.png" /></figure>

Results

<figure><img src="image_107.png" /></figure>

≠ algorithm

<figure><img src="image_108.png" /></figure>

<figure><img src="image_109.png" /></figure>

≠ parameters

<figure><img src="image_110.png" /></figure>

<figure><img src="image_111.png" /></figure>

≠ software

<figure><img src="image_112.png" /></figure>

<figure><img src="image_113.png" /></figure>

≠ environment

<figure><img src="image_114.png" /></figure>

<figure><img src="image_115.png" /></figure>

≠ software version
# A multiplicity of analysis pipelines

<figure><img src="image_131.png" /></figure>

Input data

<figure><img src="image_132.png" /></figure>

Results

... and many more...

<figure><img src="image_133.png" /></figure>

≠ algorithm

<figure><img src="image_134.png" /></figure>

≠ software

<figure><img src="image_135.png" /></figure>

≠ software version

<figure><img src="image_136.png" /></figure>

≠ parameters

<figure><img src="image_137.png" /></figure>

≠ environment
# A multiplicity of analysis pipelines

<figure><img src="image_116.png" /></figure>

Input data

<figure><img src="image_117.png" /></figure>

<figure><img src="image_118.png" /></figure>

Results

... and many more...

<figure><img src="image_119.png" /></figure>

≠ algorithm

<figure><img src="image_120.png" /></figure>

≠ software

<figure><img src="image_121.png" /></figure>

≠ software version

<figure><img src="image_122.png" /></figure>

≠ parameters

<figure><img src="image_123.png" /></figure>

≠ environment
**Many-analyst** project : NARPS

<figure><img src="image_138.png" /></figure>

1 dataset

9 ‘yes/no’
questions

[Botvinik-Nezer et. al, Nature 2020]
**Many-analyst** project : NARPS

<figure><img src="image_124.png" /></figure>

1 dataset

9 ‘yes/no’
questions

[Botvinik-Nezer et. al, Nature 2020]
**Many-analyst project : NARPS**

<figure><img src="image_139.png" /></figure>

1 dataset

<figure><img src="image_140.png" /></figure>

9 'yes/no'
questions

<figure><img src="image_141.png" /></figure>

70 analysis pipelines
by different teams

[Botvinik-Nezer et. al, Nature 2020]
# Many-analyst project : NARPS

<figure><img src="image_125.png" /></figure>

1 dataset

<figure><img src="image_126.png" /></figure>

9 'yes/no'
questions

<figure><img src="image_127.png" /></figure>

70 analysis pipelines
by different teams

[Botvinik-Nezer et. al, Nature 2020]
**Many-analyst project : NARPS**

Results over the **70 post-processing pipelines**:

<figure><img src="image_142.png" /></figure>

9 'yes/no' questions

[Botvinik-Nezer et. al, Nature 2020]
**Many-analyst project : NARPS**

Results over the **70 post-processing pipelines**:

<figure><img src="image_128.png" /></figure>

9 'yes/no' questions

[Botvinik-Nezer et. al, Nature 2020]
**Many-analyst project : NARPS**

Results over the **70 post-processing pipelines**:

<figure><img src="image_143.png" /></figure>

9 'yes/no' questions

**Significant finding**
(>80% of teams agree)

All 'yes'

Contradictory results
(Teams disagree)

**Non-significant finding**
(>80% of teams agree)

All 'no'

[Botvinik-Nezer et. al, Nature 2020]
**Many-analyst project : NARPS**

Results over the **70 post-processing pipelines**:

<figure><img src="image_129.png" /></figure>

**9 'yes/no' questions**

[Botvinik-Nezer et. al, Nature 2020]
# Variability across neuroimaging software

## Principles of functional MRI

<figure><img src="image_144.png" /><figcaption>**Alex Bowring**
Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_145.png" /></figure>

<figure><img src="image_146.png" /></figure>
# Variability across neuroimaging software

## Principles of functional MRI

<figure><img src="image_130.png" /><figcaption>**Alex Bowring**
Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_131.png" /></figure>

<figure><img src="image_132.png" /></figure>
# Variability across neuroimaging software

## Principles of functional MRI

<figure><img src="image_147.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_148.png" /></figure>

<figure><img src="image_149.png" /></figure>
# Variability across neuroimaging software

## Principles of functional MRI

<figure><img src="image_133.png" /><figcaption>Alex Bowring</figcaption></figure>

<figure><img src="image_134.png" /><figcaption>Tom Nichols</figcaption></figure>

Oxford Uni. (UK)

<figure><img src="image_135.png" /></figure>
# Variability across neuroimaging software

<figure><img src="image_150.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_151.png" /></figure>

## Principles of functional MRI

<figure><img src="image_152.png" /></figure>

Data preparation

<figure><img src="image_153.png" /></figure>

Derived
data

Statistical analysis

<figure><img src="image_154.png" /></figure>

Results

## fMRI results: Activations maps

<figure><img src="image_155.png" /></figure>

<figure><img src="image_156.png" /></figure>

Visual

<figure><img src="image_157.png" /></figure>

<figure><img src="image_158.png" /></figure>

Hand motion

<figure><img src="image_159.png" /></figure>

<figure><img src="image_160.png" /></figure>

Music
# Variability across neuroimaging software

## Principles of functional MRI

<figure><img src="image_136.png" /><figcaption>Alex Bowring</figcaption></figure>

<figure><img src="image_137.png" /><figcaption>Tom Nichols</figcaption></figure>

Oxford Uni. (UK)

<figure><img src="image_138.png" /></figure>

Data preparation

<figure><img src="image_139.png" /></figure>

Derived
data

Statistical analysis

<figure><img src="image_140.png" /></figure>

Results

### fMRI results: Activations maps

<figure><img src="image_141.png" /></figure>

<figure><img src="image_142.png" /><figcaption>Visual</figcaption></figure>

<figure><img src="image_143.png" /></figure>

<figure><img src="image_144.png" /><figcaption>Hand motion</figcaption></figure>

<figure><img src="image_145.png" /></figure>

<figure><img src="image_146.png" /><figcaption>Music</figcaption></figure>
# Variability across neuroimaging software

## Principles of functional MRI

<figure><img src="image_161.png" /><figcaption>Alex Bowring
Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_162.png" /></figure>

**Goal 1:** Investigate the variability observed in fMRI results when using different software packages
# Variability across neuroimaging software

## Principles of functional MRI

<figure><img src="image_147.png" /><figcaption>Alex Bowring</figcaption></figure>

<figure><img src="image_148.png" /><figcaption>Tom Nichols</figcaption></figure>

Oxford Uni. (UK)

<figure><img src="image_149.png" /></figure>

**Goal 1:** Investigate the variability observed in fMRI results when using different software packages
# Variability across neuroimaging software

## Principles of functional MRI

<figure><img src="image_163.png" /><figcaption>Alex Bowring
Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_164.png" /></figure>

**Goal 1:** Investigate the variability observed in fMRI results when using different software packages

* Reproduced 3 studies from the literature

* Using 3 fMRI software: AFNI, FSL, SPM
  (used in 80% literature (Carp, 2012))

* Pipelines aligned using available configurations.
# Variability across neuroimaging software

## Principles of functional MRI

<figure><img src="image_150.png" /><figcaption>Alex Bowring
Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_151.png" /></figure>

**Goal 1:** Investigate the variability observed in fMRI results when using different software packages

* Reproduced 3 studies from the literature

* Using 3 fMRI software: AFNI, FSL, SPM
  (used in 80% literature (Carp, 2012))

* Pipelines aligned using available configurations.
# Variability across neuroimaging software

* Variability in the activation maps

<figure><img src="image_165.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<table><thead><tr><th></th><th>study ds001</th><th>study ds109</th><th>study ds120</th></tr></thead><tbody><tr><td>AFNI software</td><td></td><td></td><td></td></tr><tr><td>FSL software</td><td></td><td></td><td></td></tr><tr><td>SPM software</td><td></td><td></td><td></td></tr><tr><td>Original paper</td><td></td><td></td><td></td></tr></tbody></table>

[Bowring Maumet* Nichols*, HBM 2019]
# Variability across neuroimaging software

* Variability in the activation maps

<figure><img src="image_152.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<table><thead><tr><th></th><th>study ds001</th><th>study ds109</th><th>study ds120</th></tr></thead><tbody><tr><td>AFNI software</td><td></td><td></td><td></td></tr><tr><td>FSL software</td><td></td><td></td><td></td></tr><tr><td>SPM software</td><td></td><td></td><td></td></tr><tr><td>Original paper</td><td></td><td></td><td></td></tr></tbody></table>

[Bowring Maumet* Nichols*, HBM 2019]
# Variability across neuroimaging software

* Variability in the activation maps

<figure><img src="image_166.png" /><figcaption>**Alex Bowring** Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<table><thead><tr><th></th><th>study ds001</th><th>study ds109</th><th>study ds120</th></tr></thead><tbody><tr><td>AFNI software</td><td></td><td></td><td></td></tr><tr><td>FSL software</td><td></td><td></td><td></td></tr><tr><td>SPM software</td><td></td><td></td><td></td></tr><tr><td>Original paper</td><td></td><td></td><td></td></tr></tbody></table>

[Bowring Maumet* Nichols*, HBM 2019]
# Variability across neuroimaging software

* Variability in the activation maps

<figure><img src="image_153.png" /><figcaption>**Alex Bowring** Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_154.png" /></figure>

AFNI
software

FSL
software

SPM
software

Original
paper

[Bowring Maumet* Nichols*, HBM 2019]
# Variability across neuroimaging software

* Variability in the activation maps

<figure><img src="image_167.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_168.png" /></figure>

<figure><img src="image_169.png" /></figure>

<figure><img src="image_170.png" /></figure>

[Bowring Maumet* Nichols*, HBM 2019]
# Variability across neuroimaging software

* Variability in the activation maps

<figure><img src="image_155.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_156.png" /></figure>

<figure><img src="image_157.png" /></figure>

<figure><img src="image_158.png" /></figure>

[Bowring Maumet* Nichols*, HBM 2019]
# Variability across neuroimaging software

* Variability in the activation maps

<figure><img src="image_171.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_172.png" /></figure>

study ds001

study ds001

<figure><img src="image_173.png" /></figure>

z=4

study ds109

<figure><img src="image_174.png" /></figure>

z=0

study ds120

<figure><img src="image_175.png" /></figure>

x=0

AFNI
software

<figure><img src="image_176.png" /></figure>

z=4

4.2

2.1

4.2

<figure><img src="image_177.png" /></figure>

x=0

<figure><img src="image_178.png" /></figure>

<figure><img src="image_179.png" /></figure>

<figure><img src="image_180.png" /></figure>

z=4

<figure><img src="image_181.png" /></figure>

z=0

<figure><img src="image_182.png" /></figure>

x=0

FSL
software

AFNI software

<figure><img src="image_183.png" /></figure>

z=-32

<figure><img src="image_184.png" /></figure>

z=-18

<figure><img src="image_185.png" /></figure>

z=0

<figure><img src="image_186.png" /></figure>

z=12

<figure><img src="image_187.png" /></figure>

z=24

FSL software

<figure><img src="image_188.png" /></figure>

z=-32

<figure><img src="image_189.png" /></figure>

z=-18

<figure><img src="image_190.png" /></figure>

z=0

<figure><img src="image_191.png" /></figure>

z=12

<figure><img src="image_192.png" /></figure>

z=24

SPM software

<figure><img src="image_193.png" /></figure>

z=-32

<figure><img src="image_194.png" /></figure>

z=-18

<figure><img src="image_195.png" /></figure>

z=0

<figure><img src="image_196.png" /></figure>

z=12

<figure><img src="image_197.png" /></figure>

z=24

SPM
software

<figure><img src="image_198.png" /></figure>

X=4

<figure><img src="image_199.png" /></figure>

<figure><img src="image_200.png" /></figure>

Original

[Bowring Maumet* Nichols*, HBM 2019]

Dice coefficient

<figure><img src="image_201.png" /></figure>

AFNI

FSL

SPM
# Variability across neuroimaging software

* Variability in the activation maps

<figure><img src="image_159.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_160.png" /></figure>

study ds001

study ds001

<figure><img src="image_161.png" /></figure>

<figure><img src="image_162.png" /></figure>

<figure><img src="image_163.png" /></figure>

<figure><img src="image_164.png" /></figure>

study ds109

<figure><img src="image_165.png" /></figure>

<figure><img src="image_166.png" /></figure>

<figure><img src="image_167.png" /></figure>

<figure><img src="image_168.png" /></figure>

study ds120

<figure><img src="image_169.png" /></figure>

<figure><img src="image_170.png" /></figure>

<figure><img src="image_171.png" /></figure>

AFNI software

<figure><img src="image_172.png" /></figure>

FSL software

<figure><img src="image_173.png" /></figure>

SPM software

<figure><img src="image_174.png" /></figure>

<figure><img src="image_175.png" /></figure>

<figure><img src="image_176.png" /></figure>

<figure><img src="image_177.png" /></figure>

<figure><img src="image_178.png" /></figure>

<figure><img src="image_179.png" /></figure>

<figure><img src="image_180.png" /></figure>

<figure><img src="image_181.png" /></figure>

<figure><img src="image_182.png" /></figure>

<figure><img src="image_183.png" /></figure>

<figure><img src="image_184.png" /></figure>

<figure><img src="image_185.png" /></figure>

<figure><img src="image_186.png" /></figure>

<figure><img src="image_187.png" /></figure>

AFNI
software

FSL
software

SPM
software

Original

[Bowring Maumet* Nichols*, HBM 2019]

Dice coefficient

<figure><img src="image_188.png" /></figure>
# Variability across neuroimaging software

* Variability in the unthresholded statistic maps

<figure><img src="image_202.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<table><thead><tr><th></th><th>study ds001</th><th>study ds109</th><th>study ds120</th></tr></thead><tbody><tr><th>AFNI software</th><td></td><td></td><td></td></tr><tr><th>FSL software</th><td></td><td></td><td></td></tr><tr><th>SPM software</th><td></td><td></td><td></td></tr></tbody></table>

## Correlations

AFNI vs. FSL 0.62

AFNI vs. SPM 0.61

FSL vs. SPM 0.68

**Conclusion:** The choice of software impacted both the final activation and statistic maps.

[Bowring Maumet* Nichols*, HBM 2019]
# Variability across neuroimaging software

* Variability in the unthresholded statistic maps

<figure><img src="image_189.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<table><thead><tr><th></th><th>study ds001</th><th>study ds109</th><th>study ds120</th></tr></thead><tbody><tr><th>AFNI software</th><td></td><td></td><td></td></tr><tr><th>FSL software</th><td></td><td></td><td></td></tr><tr><th>SPM software</th><td></td><td></td><td></td></tr></tbody></table>

## Correlations

AFNI vs. FSL 0.62

AFNI vs. SPM 0.61

FSL vs. SPM 0.68

**Conclusion:** The choice of software impacted both the final activation and statistic maps.
# Variability across neuroimaging software

<figure><img src="image_203.png" /><figcaption>Alex Bowring
Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_204.png" /></figure>

**Goal 2:** Investigate how remaining differences in software implementations can explain the observed differences

## study ds001

1. Preprocessing

2. Subject-level Signal

3. Subject-level Drift

4. Subject-level Noise

5. Group-level Stat

<figure><img src="image_205.png" /></figure>

<figure><img src="image_206.png" /></figure>

<figure><img src="image_207.png" /></figure>

<figure><img src="image_208.png" /></figure>

<figure><img src="image_209.png" /></figure>

AFNI
software

<figure><img src="image_210.png" /></figure>

<figure><img src="image_211.png" /></figure>

<figure><img src="image_212.png" /></figure>

FSL
software

[Bowring Nichols* Maumet*, HBM 2021]
# Variability across neuroimaging software

<figure><img src="image_190.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_191.png" /></figure>

**Goal 2:** Investigate how remaining differences in software implementations can explain the observed differences

## study ds001

1. Preprocessing

2. Subject-level Signal

3. Subject-level Drift

4. Subject-level Noise

5. Group-level Stat

<figure><img src="image_192.png" /></figure>

<figure><img src="image_193.png" /></figure>

<figure><img src="image_194.png" /></figure>

<figure><img src="image_195.png" /></figure>

<figure><img src="image_196.png" /></figure>

AFNI
software

<figure><img src="image_197.png" /></figure>

<figure><img src="image_198.png" /></figure>

<figure><img src="image_199.png" /></figure>

FSL
software

[Bowring Nichols* Maumet*, HBM 2021]
# Variability across neuroimaging software

<figure><img src="image_213.png" /><figcaption>Alex Bowring
Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_214.png" /></figure>

**Goal 2:** Investigate how remaining differences in software implementations can explain the observed differences

**study ds001**

1. Preprocessing

2. Subject-level Signal

3. Subject-level Drift

4. Subject-level Noise

5. Group-level Stat

<figure><img src="image_215.png" /></figure>

<figure><img src="image_216.png" /></figure>

<figure><img src="image_217.png" /></figure>

<figure><img src="image_218.png" /></figure>

AFNI
software

<figure><img src="image_219.png" /></figure>

<figure><img src="image_220.png" /></figure>

<figure><img src="image_221.png" /></figure>

<figure><img src="image_222.png" /></figure>

FSL
software

**Conclusion:** 1 main contributing factor was found but different for each study

[Bowring Nichols* Maumet*, HBM 2021]
# Variability across neuroimaging software

<figure><img src="image_200.png" /><figcaption>Alex Bowring Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_201.png" /></figure>

**Goal 2:** Investigate how remaining differences in software implementations can explain the observed differences

**study ds001**

1. Preprocessing

2. Subject-level Signal

3. Subject-level Drift

4. Subject-level Noise

5. Group-level Stat

<figure><img src="image_202.png" /></figure>

<figure><img src="image_203.png" /></figure>

<figure><img src="image_204.png" /></figure>

<figure><img src="image_205.png" /></figure>

AFNI
software

<figure><img src="image_206.png" /></figure>

<figure><img src="image_207.png" /></figure>

<figure><img src="image_208.png" /></figure>

<figure><img src="image_209.png" /></figure>

FSL
software

**Conclusion:** 1 main contributing factor was found but different for each study

[Bowring Nichols* Maumet*, HBM 2021]
<figure><img src="image_223.png" /></figure>

# Understanding
analytical variability
<figure><img src="image_210.png" /></figure>

# Understanding
analytical variability
# Understanding analytical variability...

?

<figure><img src="image_224.png" /></figure>
# Understanding analytical variability...

?

<figure><img src="image_211.png" /></figure>
Understanding analytical variability...

?

<figure><img src="image_225.png" /></figure>

Numerical
uncertainty
(e.g. ≠ OS)

Different
implen.

<figure><img src="image_226.png" /></figure>

Unwanted
variability

<figure><img src="image_227.png" /></figure>
Understanding analytical variability...

?

<figure><img src="image_212.png" /></figure>

Numerical
uncertainty
(e.g. ≠ OS)

Different
implen.

<figure><img src="image_213.png" /></figure>

Unwanted
variability

<figure><img src="image_214.png" /></figure>
# Understanding analytical variability...

?

<figure><img src="image_228.png" /></figure>

Numerical
uncertainty
(e.g. ≠ OS)

Assumption
violations

Unwanted
variability

Different
implen.

Unreliable
results

Data
induced
variability
# Understanding analytical variability...

?

<figure><img src="image_215.png" /></figure>

Numerical
uncertainty
(e.g. ≠ OS)

Assumption
violations

Unwanted
variability

Different
implen.

Unreliable
results

Data
induced
variability
# Understanding analytical variability...

?

Better fit to the data

Variability of interest

?

Assumption violations

<figure><img src="image_229.png" /></figure>

Numerical uncertainty (e.g. ≠ OS)

Data induced variability

Different implem.

Unreliable results

Unwanted variability
# Understanding analytical variability...

?

Better fit to the data

Variability of interest

?

Assumption violations

<figure><img src="image_216.png" /></figure>

Numerical uncertainty (e.g. ≠ OS)

Data induced variability

Unreliable results

Different implem.

Unwanted variability
<figure><img src="image_230.png" /></figure>

# Describing the neuroimaging pipeline space
<figure><img src="image_217.png" /></figure>

# Describing the neuroimaging pipeline space
# NARPS open pipelines

<figure><img src="image_231.png" /><figcaption>Boris
Clenet</figcaption></figure>

<figure><img src="image_232.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_233.png" /><figcaption>Jérémy
Lefort-Besnard</figcaption></figure>

<figure><img src="image_234.png" /></figure>

1 dataset

<figure><img src="image_235.png" /></figure>

9 'yes/no'
questions

<figure><img src="image_236.png" /></figure>

70 analysis pipelines
by different teams

1- H1: “yes”

2- Activation +
Statistic brain
maps

3- Textual
description of the
methods

<figure><img src="image_237.png" /></figure>
# NARPS open pipelines

<figure><img src="image_218.png" /><figcaption>Boris
Clenet</figcaption></figure>

<figure><img src="image_219.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_220.png" /><figcaption>Jérémy
Lefort-Besnard</figcaption></figure>

<figure><img src="image_221.png" /></figure>

1 dataset

<figure><img src="image_222.png" /></figure>

9 'yes/no'
questions

<figure><img src="image_223.png" /></figure>

70 analysis pipelines
by different teams

1- H1: “yes”

2- Activation +
Statistic brain
maps

3- Textual
description of the
methods

<figure><img src="image_224.png" /></figure>
# NARPS open pipelines

<figure><img src="image_238.png" /><figcaption>Boris
Clenet</figcaption></figure>

<figure><img src="image_239.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_240.png" /><figcaption>Jérémy
Lefort-Besnard</figcaption></figure>

<figure><img src="image_241.png" /></figure>

<figure><img src="image_242.png" /></figure>

9 'yes/no'
questions

<figure><img src="image_243.png" /></figure>

70 analysis pipelines
by different teams

1- H1: “yes”

2- Activation +
Statistic brain
maps

<figure><img src="image_244.png" /></figure>

3- Textual
description of the
methods

+ Code

<figure><img src="image_245.png" /></figure>
# NARPS open pipelines

<figure><img src="image_225.png" /><figcaption>Boris
Clenet</figcaption></figure>

<figure><img src="image_226.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_227.png" /><figcaption>Jérémy
Lefort-Besnard</figcaption></figure>

<figure><img src="image_228.png" /></figure>

1 dataset

<figure><img src="image_229.png" /></figure>

9 'yes/no'
questions

<figure><img src="image_230.png" /></figure>

70 analysis pipelines
by different teams

1- H1: “yes”

2- Activation +
Statistic brain
maps

3- Textual
description of the
methods

<figure><img src="image_231.png" /></figure>

+ Code

<figure><img src="image_232.png" /></figure>
# NARPS open pipelines

<figure><img src="image_246.png" /><figcaption>Boris
Clenet</figcaption></figure>

<figure><img src="image_247.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_248.png" /><figcaption>Jérémy
Lefort-Besnard</figcaption></figure>

brainhack.org

**Credits** - This project was presented and received contributions during the following events:

* Empenn hackathon (February 2024)

* [Brainhack Marseille 2023](December 2023)

* [ORIGAMI lab](September 2023) hackathon

* [OHBM Brainhack 2023](July 2023): Arshitha Basavaraj, Boris Clénet, Rémi Gau, Élodie Germani, Yaroslav Halchenko, Camille Maumet, Paul Taylor.

* [e-ReproNim FENS NENS Cluster Brainhack](June 2023) : Liz Bushby, Boris Clénet, Michael Dayan, Aimee Westbrook.

* [OHBM Brainhack 2022](June 2022): Elodie Germani, Arshitha Basavaraj, Trang Cao, Rémi Gau, Anna Menacher, Camille Maumet.

<figure><img src="image_249.png" /></figure>

https://github.com/Inria-Empenn/narps_open_pipelines/
# NARPS open pipelines

<figure><img src="image_233.png" /><figcaption>Boris
Clenet</figcaption></figure>

<figure><img src="image_234.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_235.png" /><figcaption>Jérémy
Lefort-Besnard</figcaption></figure>

brainhack.org

**Credits** - This project was presented and received contributions during the following events:

* Empenn hackathon (February 2024)

* [Brainhack Marseille 2023](December 2023)

* [ORIGAMI lab](September 2023) hackathon

* [OHBM Brainhack 2023](July 2023): Arshitha Basavaraj, Boris Clénet, Rémi Gau, Élodie Germani, Yaroslav Halchenko, Camille Maumet, Paul Taylor.

* [e-ReproNim FENS NENS Cluster Brainhack](June 2023) : Liz Bushby, Boris Clénet, Michael Dayan, Aimee Westbrook.

* [OHBM Brainhack 2022](June 2022): Elodie Germani, Arshitha Basavaraj, Trang Cao, Rémi Gau, Anna Menacher, Camille Maumet.

<figure><img src="image_236.png" /></figure>

https://github.com/Inria-Empenn/narps_open_pipelines/
# Tracing provenance in neuroimaging

<figure><img src="image_250.png" /><figcaption>Satra Ghosh</figcaption></figure>

<figure><img src="image_251.png" /></figure>

BIDS

BRAIN IMAGING DATA STRUCTURE

<figure><img src="image_252.png" /></figure>

<figure><img src="image_253.png" /></figure>

Massachusetts
Institute of
Technology

> **"Provenance"** describe how an artifact or set of artifacts was produced" [Wikipedia](https://en.wikipedia.org/wiki/Provenance)

* Based on W3C PROV

<figure><img src="image_254.png" /></figure>

https://www.w3.org/TR/2013/NOTE-prov-primer-20130430/

* As part of neuroimaging standardization efforts

<figure><img src="image_255.png" /></figure>

BIDS

BRAIN IMAGING DATA STRUCTURE
# Tracing provenance in neuroimaging

<figure><img src="image_237.png" /></figure>

Satra Ghosh

<figure><img src="image_238.png" /></figure>

BIDS

BRAIN IMAGING DATA STRUCTURE

<figure><img src="image_239.png" /></figure>

<figure><img src="image_240.png" /></figure>

Massachusetts
Institute of
Technology

> **"Provenance"** describe how an artifact or set of artifacts was produced" [Wikipedia](https://en.wikipedia.org/wiki/Provenance)

* Based on W3C PROV

<figure><img src="image_241.png" /></figure>

https://www.w3.org/TR/2013/NOTE-prov-primer-20130430/

* As part of neuroimaging standardization efforts

<figure><img src="image_242.png" /></figure>

BIDS

BRAIN IMAGING DATA STRUCTURE
Provenance Use Case 1 An unambiguous
description of the methods

<figure><img src="image_256.png" /></figure>
Provenance Use Case 1 An unambiguous
description of the methods

<figure><img src="image_243.png" /></figure>
Provenance Use Case 1 An unambiguous description of the methods

<figure><img src="image_257.png" /></figure>

<figure><img src="image_258.png" /></figure>

Which release of
SPM was used?

Which registration
algorithm was
selected?

How was the brain
mask computed?
Provenance Use Case 1 An unambiguous description of the methods

<figure><img src="image_244.png" /></figure>

<figure><img src="image_245.png" /></figure>

Which release of
SPM was used?

Which registration
algorithm was
selected?

How was the brain
mask computed?
Provenance Use Case 1 An unambiguous
description of the methods

Can we
reproduce
exactly the same
results
(same data, same
methods)?
Provenance Use Case 1 An unambiguous
description of the methods

Can we
reproduce
exactly the same
results
(same data, same
methods)?
Provenance Use Case 1 An unambiguous
description of the methods

<figure><img src="image_259.png" /></figure>

Can we
reproduce
exactly the same
results
(same data, same
methods)?
Provenance Use Case 1 An unambiguous
description of the methods

<figure><img src="image_246.png" /></figure>

Can we
reproduce
exactly the same
results
(same data, same
methods)?
## Provenance Use Case 2 Dealing with
unwanted variability

> A previously
> widely-used
> parameter is no
> longer considered
> as best practice
## Provenance Use Case 2 Dealing with
unwanted variability

> A previously
> widely-used
> parameter is no
> longer considered
> as best practice
## Provenance Use Case 2 Dealing with
unwanted variability

A previously
widely-used
parameter is no
longer considered
as best practice

A bug was found
in a given tool
version, can we
retrieve all
corresponding
results?
## Provenance Use Case 2 Dealing with unwanted variability

A previously widely-used parameter is no longer considered as best practice

A bug was found in a given tool version, can we retrieve all corresponding results?
## Provenance Use Case 2 Dealing with unwanted variability

<figure><img src="image_260.png" /></figure>

A previously widely-used parameter is no longer considered as best practice

A bug was found in a given tool version, can we retrieve all corresponding results?
## Provenance Use Case 2 Dealing with unwanted variability

<figure><img src="image_247.png" /></figure>

A previously widely-used parameter is no longer considered as best practice

A bug was found in a given tool version, can we retrieve all corresponding results?
Provenance Use Case 3 **Exploring factors impacting analytical variability**

<figure><img src="image_261.png" /></figure>

Are my conclusions dependant on the **size of the smoothing kernel** used at Step 5?

<figure><img src="image_262.png" /></figure>

**BIDS-Prov** will soon be open for community feedback!

<figure><img src="image_263.png" /></figure>

http://bids.neuroimaging.io/bep028
Provenance Use Case 3 **Exploring factors impacting analytical variability**

<figure><img src="image_248.png" /></figure>

Are my conclusions dependant on the **size of the smoothing kernel** used at Step 5?

<figure><img src="image_249.png" /></figure>

**BIDS-Prov** will soon be open for community feedback!

<figure><img src="image_250.png" /></figure>

http://bids.neuroimaging.io/bep028
<figure><img src="image_264.png" /></figure>

Which solutions for
practitioners?
<figure><img src="image_251.png" /></figure>

Which solutions for
practitioners?
# Same data meta-analysis

<figure><img src="image_265.png" /><figcaption>Jérémy Lefort-Besnard Tom Nichols Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_266.png" /></figure>

**Goal:** Investigate the validity of same-data meta-analysis methods

<figure><img src="image_267.png" /></figure>

[Lefort-Besnard Nichols* Maumet*, OHBM 2024]
# Same data meta-analysis

<figure><img src="image_252.png" /><figcaption>Jérémy
Lefort-Besnard</figcaption></figure>

<figure><img src="image_253.png" /><figcaption>Tom Nichols
Oxford Uni. (UK)</figcaption></figure>

<figure><img src="image_254.png" /></figure>

**Goal:** Investigate the validity of same-data meta-analysis methods

[Lefort-Besnard Nichols* Maumet*, OHBM 2024]
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_268.png" /><figcaption>## Multi-stage functional MRI analysis</figcaption></figure>

<figure><img src="image_269.png" /><figcaption>Xavier Rolland</figcaption></figure>

<figure><img src="image_270.png" /><figcaption>Pierre Maurel</figcaption></figure>
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_255.png" /><figcaption>## Multi-stage functional MRI analysis</figcaption></figure>

<figure><img src="image_256.png" /><figcaption>Xavier
Rolland</figcaption></figure>

<figure><img src="image_257.png" /><figcaption>Pierre
Maurel</figcaption></figure>
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_271.png" /><figcaption>Xavier Rolland</figcaption></figure>

<figure><img src="image_272.png" /><figcaption>Pierre Maurel</figcaption></figure>

## Multi-stage functional MRI analysis

<figure><img src="image_273.png" /></figure>

Subject-Level

Group-Level
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_258.png" /><figcaption>Xavier Rolland</figcaption></figure>

<figure><img src="image_259.png" /><figcaption>Pierre Maurel</figcaption></figure>

## Multi-stage functional MRI analysis

<figure><img src="image_260.png" /></figure>

Subject-Level

Group-Level
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_274.png" /><figcaption>## Multi-stage functional MRI analysis</figcaption></figure>

<figure><img src="image_275.png" /><figcaption>Xavier Rolland</figcaption></figure>

<figure><img src="image_276.png" /><figcaption>Pierre Maurel</figcaption></figure>

Subject-Level

Group-Level
# Validity of mega-analyses
in neuroimaging

## Multi-stage functional MRI analysis

<figure><img src="image_261.png" /><figcaption>Xavier Rolland</figcaption></figure>

<figure><img src="image_262.png" /><figcaption>Pierre Maurel</figcaption></figure>

<figure><img src="image_263.png" /></figure>

Subject-Level

Group-Level
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_277.png" /><figcaption>Xavier Rolland</figcaption></figure>

<figure><img src="image_278.png" /><figcaption>Pierre Maurel</figcaption></figure>

## Multi-stage functional MRI analysis

<figure><img src="image_279.png" /></figure>

Subject-Level

Group-Level
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_264.png" /><figcaption>Xavier Rolland</figcaption></figure>

<figure><img src="image_265.png" /><figcaption>Pierre Maurel</figcaption></figure>

## Multi-stage functional MRI analysis

<figure><img src="image_266.png" /></figure>

Subject-Level

Group-Level
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_280.png" /><figcaption>Xavier Rolland</figcaption></figure>

<figure><img src="image_281.png" /><figcaption>Pierre Maurel</figcaption></figure>

## Multi-stage functional MRI analysis

<figure><img src="image_282.png" /></figure>

**Goal:** Investigate the validity of group statistics using subject data prepared with different pipelines

Subject-Level

Group-Level
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_267.png" /><figcaption>Xavier Rolland</figcaption></figure>

<figure><img src="image_268.png" /><figcaption>Pierre Maurel</figcaption></figure>

## Multi-stage functional MRI analysis

<figure><img src="image_269.png" /></figure>

**Goal:** Investigate the validity of group statistics using subject data prepared with different pipelines

Subject-Level

Group-Level
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_283.png" /><figcaption>Xavier
Rolland</figcaption></figure>

<figure><img src="image_284.png" /><figcaption>Pierre
Maurel</figcaption></figure>

**Goal:** Investigate the validity of group statistics using
subject data prepared with different pipelines

* HCP Young-Adult (1000 subjects)

* 3 pipeline variations

* Spatial smoothing: 5 or 8 mm

* Modelling of the brain response: with/out derivative

* Correction for subject motion: 0,6, 24 covariates

<figure><img src="image_285.png" /></figure>
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_270.png" /><figcaption>Xavier
Rolland</figcaption></figure>

<figure><img src="image_271.png" /><figcaption>Pierre
Maurel</figcaption></figure>

**Goal:** Investigate the validity of group statistics using
subject data prepared with different pipelines

* HCP Young-Adult (1000 subjects)

* 3 pipeline variations

* Spatial smoothing: 5 or 8 mm

* Modelling of the brain response: with/out derivative

* Correction for subject motion: 0,6, 24 covariates

<figure><img src="image_272.png" /></figure>
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_286.png" /><figcaption>Xavier
Rolland</figcaption></figure>

<figure><img src="image_287.png" /><figcaption>Pierre
Maurel</figcaption></figure>

"Reference" pipeline :

o canonical HRF

o smoothing 5mm

o 24 motion regressors

* P-p plots (Bland-Altman)

<figure><img src="image_288.png" /></figure>

**Conclusion:** Invalidity of the mega-analyses for multiple variations studied (e.g. different smoothing, 1+ variation)
# Validity of mega-analyses
in neuroimaging

<figure><img src="image_273.png" /><figcaption>Xavier
Rolland</figcaption></figure>

<figure><img src="image_274.png" /><figcaption>Pierre
Maurel</figcaption></figure>

"Reference" pipeline :

o canonical HRF

o smoothing 5mm

o 24 motion regressors

* P-p plots (Bland-Altman)

<figure><img src="image_275.png" /></figure>

**Conclusion:** Invalidity of the mega-analyses for multiple variations studied (e.g. different smoothing, 1+ variation)
# Leveraging large and open datasets

## Shared data with limited metadata

<figure><img src="image_289.png" /></figure>

500 000+ images

A variety of fMRI tasks, preprocessing and analysis
Minimal metadata required.

<figure><img src="image_290.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_291.png" /><figcaption>Elisa
Fromont</figcaption></figure>
# Leveraging large and open datasets

## Shared data with limited metadata

<figure><img src="image_276.png" /></figure>

**NEUROVAULT**

500 000+ images

A variety of fMRI tasks, preprocessing and analysis

Minimal metadata required.

<figure><img src="image_277.png" /><figcaption>**Elodie Germani**</figcaption></figure>

<figure><img src="image_278.png" /><figcaption>**Elisa Fromont**</figcaption></figure>
# Leveraging large and open datasets

## Shared data with limited metadata

<figure><img src="image_292.png" /></figure>

**NEUROVAULT**

500 000+ images

A variety of fMRI tasks, preprocessing and analysis

Minimal metadata required.

**Goal:** Can we take advantage of a large pool of unlabelled data to help answer new research questions?

<figure><img src="image_293.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_294.png" /><figcaption>Elisa
Fromont</figcaption></figure>
# Leveraging large and open datasets

<figure><img src="image_279.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_280.png" /><figcaption>Elisa
Fromont</figcaption></figure>

## Shared data with limited metadata

<figure><img src="image_281.png" /></figure>

**NEUROVAULT**

500 000+ images

A variety of fMRI tasks, preprocessing and analysis
Minimal metadata required.

**Goal:** Can we take advantage of a large pool of unlabelled data to help answer new research questions?
# Leveraging large and open datasets

<figure><img src="image_295.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_296.png" /><figcaption>Elisa
Fromont</figcaption></figure>

## Shared data with limited metadata

<figure><img src="image_297.png" /></figure>

**NEUROVAULT**

500 000+ images

A variety of fMRI tasks, preprocessing and analysis
Minimal metadata required.

**Goal:** Can we take advantage of a large pool of unlabelled data to help answer new research questions?

* Step 1: Build a model-agnostic representation of fMRI statistic maps

<figure><img src="image_298.png" /></figure>

* Step 2: Fine-tune a classifier (predicting cognitive process:

<figure><img src="image_299.png" /></figure>

<figure><img src="image_300.png" /></figure>
# Leveraging large and open datasets

<figure><img src="image_282.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_283.png" /><figcaption>Elisa
Fromont</figcaption></figure>

## Shared data with limited metadata

<figure><img src="image_284.png" /></figure>

**NEUROVAULT**

500 000+ images

A variety of fMRI tasks, preprocessing and analysis
Minimal metadata required.

**Goal:** Can we take advantage of a large pool of unlabelled data to help answer new research questions?

* Step 1: Build a model-agnostic representation of fMRI statistic maps

<figure><img src="image_285.png" /></figure>

* Step 2: Fine-tune a classifier (predicting cognitive process: )

<figure><img src="image_286.png" /></figure>

<figure><img src="image_287.png" /></figure>
# Leveraging large and open datasets

<figure><img src="image_301.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_302.png" /><figcaption>Elisa
Fromont</figcaption></figure>

* Tested on two datasets:

* Single large study (HCP Young adult) 23 tasks x 787 subjects

* Collection of studies (Brainpedia) 29 studies, 36 tasks, 826 subjects

Single study

<figure><img src="image_303.png" /></figure>

Collection of studies

<figure><img src="image_304.png" /></figure>

**Conclusion:** The pre-trained classifier always performed better especially when the problem was complex (small samples, heterogeneity).
# Leveraging large and open datasets

<figure><img src="image_288.png" /><figcaption>Elodie
Germani</figcaption></figure>

<figure><img src="image_289.png" /><figcaption>Elisa
Fromont</figcaption></figure>

* Tested on two datasets:

* Single large study (HCP Young adult) 23 tasks x 787 subjects

* Collection of studies (Brainpedia) 29 studies, 36 tasks, 826 subjects

Single study

<figure><img src="image_290.png" /></figure>

Collection of studies

<figure><img src="image_291.png" /></figure>

**Conclusion:** The pre-trained classifier always performed better especially when the problem was complex (small samples, heterogeneity).
<figure><img src="image_305.png" /></figure>

<figure><img src="image_306.png" /></figure>

# Towards reproducible neuroimaging
across different analysis
pipelines

Camille Maumet

Thank you!

Credit: Presentation template and icons by
[SlidesCarnival](https:// slidescarnival.com ), adapted

<figure><img src="image_307.png" /></figure>

@cmaumet.bsky.social

<figure><img src="image_308.png" /></figure>

@cmaumet@fediscience.org
<figure><img src="image_292.png" /></figure>

<figure><img src="image_293.png" /></figure>

# Towards reproducible neuroimaging
across different analysis
pipelines

Camille Maumet

Thank you!