# Different analysis pipelines, different results... How to deal with analytical variability?

Camille Maumet

► To cite this version:

Camille Maumet. Different analysis pipelines, different results... How to deal with analytical variability?. 20th IEEE International Symposium on Biomedical Imaging (ISBI 2023), Apr 2023, Cartagena (Colombia), Colombia. [https://inria.hal.science/hal-04073419](https://inria.hal.science/hal-04073419)

HAL Id: hal-04073419

https://inria.hal.science/hal-04073419

Submitted on 18 Apr 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# Different analysis pipelines, different results...

## How to deal with analytical variability?

Camille Maumet

Inria, Univ Rennes, Inserm, CNRS, IRISA
# Analytical pipelines

<figure><img src="image_3.png" /></figure>

Data preparation

<figure><img src="image_4.png" /></figure>

Derived data

Analysis

<figure><img src="image_5.png" /></figure>
# Analytical pipelines

<figure><img src="image_6.png" /></figure>
# Analytical pipelines

<figure><img src="image_7.png" /></figure>
# Analytical pipelines

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>
# Analytical pipelines

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

≠ algorithm

<figure><img src="image_12.png" /></figure>

≠ software
# Analytical pipelines

<figure><img src="image_13.png" /></figure>

≠ software version

<figure><img src="image_14.png" /></figure>

≠ algorithm

<figure><img src="image_15.png" /></figure>

≠ software

<figure><img src="image_16.png" /></figure>

≠ software version
# Analytical pipelines

<figure><img src="image_17.png" /></figure>

≠ parameters

<figure><img src="image_18.png" /></figure>

≠ algorithm

<figure><img src="image_19.png" /></figure>

≠ software

<figure><img src="image_20.png" /></figure>

≠ software version

<figure><img src="image_21.png" /></figure>

≠ parameters
# Analytical pipelines

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /></figure>

A family of "acceptable" pipelines
over 10^30 combinations...
**Many analysts project : NARPS**

<figure><img src="image_24.png" /></figure>

Data preparation

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

Derived data

<figure><img src="image_27.png" /></figure>

Analysis

<figure><img src="image_28.png" /></figure>

<figure><img src="image_29.png" /></figure>
# Many analysts project : NARPS

<figure><img src="image_30.png" /></figure>

1 dataset

<figure><img src="image_31.png" /></figure>

Data preparation

<figure><img src="image_32.png" /></figure>

<figure><img src="image_33.png" /></figure>

Derived data

<figure><img src="image_34.png" /></figure>

Analysis

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>

9 'yes/no' questions

e.g. Is there a parametric effect of gain:
Positive effect in ventromedial PFC -
for the equal indifference group?
# Many analysts project : NARPS

<figure><img src="image_37.png" /></figure>
**Many analysts project : NARPS**

Results over the **70 post-processing pipelines**:

<figure><img src="image_38.png" /></figure>
# Many analysts project : NARPS

## Results over the 70 post-processing pipelines

<figure><img src="image_39.png" /></figure>
**Many analysts project : NARPS**

Results over the **70 post-processing pipelines**:

<figure><img src="image_40.png" /></figure>

**Significant finding**
(>80% of teams agree)

Contradictory results
(Teams disagree)

**Non-significant finding**
(>80% of teams agree)
# Different levels of variability in analytical pipelines

* Different end-to-end pipelines using different algorithms+parameters [Churchill 2012, Carp 2013, Li 2021, Dafflon 2022]

* Different Software Version [Gronenschild 2012]

* Different operating system [Glatard 2015, Kiar 2021]

* Different software packages [Bowring 2019,

* and more...
# Variability across software

* Reproduced 3 published functional MRI studies

* ds001, ds109, ds120 (from [Openneuro](https://openneuro.org/))

* Using 3 different software packages

* AFNI, FSL, SPM

<figure><img src="image_41.png" /><figcaption>Alex Bowring Tom Nichols</figcaption></figure>

**Software 1**

<figure><img src="image_42.png" /></figure>

Pipeline

<figure><img src="image_43.png" /></figure>

**Software 2**

<figure><img src="image_44.png" /></figure>

Pipeline

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>

**Software 3**

<figure><img src="image_47.png" /></figure>

Pipeline

<figure><img src="image_48.png" /></figure>

<figure><img src="image_49.png" /></figure>
# Software Comparison Project

<figure><img src="image_50.png" /><figcaption>Comparison of the final results</figcaption></figure>

<figure><img src="image_51.png" /><figcaption>Comparison of the statistic maps</figcaption></figure>
# Understanding
analytical variability
# Explaining analytical variability...

?

<figure><img src="image_52.png" /></figure>
# Explaining analytical variability...

?

<figure><img src="image_53.png" /></figure>

Numerical
uncertainty
(e.g. ≠ OS)

[Glatard 2015,
Kiar 2021]

Different
implen.

Unwanted
variability

[Gronenschild 2012,
Bowring 2019]
# Explaining analytical variability...

?

<figure><img src="image_54.png" /></figure>

Numerical
uncertainty
(e.g. ≠ OS)

Assumption
violations

Unwanted
variability

Different
implen.

Unreliable
results

Data
induced
variability
# Explaining analytical variability...

?

Better fit to the data

Variability of interest

Assumption violations

<figure><img src="image_55.png" /></figure>

Numerical uncertainty (e.g. ≠ OS)

Unreliable results

Data induced variability

Different implem.

Unwanted variability
On our way...

* Sharing a **variety of research artefacts** and in particular **code** along with paper

* Storing and **sharing provenance** (BIDS-Prov)

* **Benchmarking** different methods
<figure><img src="image_56.png" /></figure>

Different analysis pipelines,
different results...

How to deal with analytical variability?

Camille Maumet

Thank you!

<figure><img src="image_57.png" /></figure>

<figure><img src="image_58.png" /></figure>