# Introduction to the special session "10 years of reproducibility in biomedical research: how can we achieve generalizability and fairness?"

Camille Maumet

► To cite this version:

Camille Maumet. Introduction to the special session "10 years of reproducibility in biomedical research: how can we achieve generalizability and fairness?". ISBI 2023 - 20th IEEE International Symposium on Biomedical Imaging, Apr 2023, Cartagenas de Indias, Colombia. pp.1-12. hal-04073425

HAL Id: hal-04073425

https://inria.hal.science/hal-04073425

Submitted on 18 Apr 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /></figure>

10 years of reproducibility in biomedical research

how can we achieve

generalizability and fairness?

Camille Maumet

Inria, Univ Rennes, Inserm, CNRS, IRISA, France

<figure><img src="image_4.png" /></figure>

Université
de Rennes

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

Inserm

Institut national
de la santé et de la recherche médicale

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

UMR

IRISA
On our way to fix irreproducibility
On our way to fix irreproducibility

Irreproducible with...

Same Data
# On our way to fix irreproducibility

Irreproducible with...

Same Data

Lack of experimental details

**Proposed solutions**
Sharing code,
Programming best
practices, etc.
# On our way to fix irreproducibility

Irreproducible with...

Same Data

Lack of experimental
details

**Proposed solutions**
Sharing code,
Programming best
practices, etc.

<figure><img src="image_9.png" /></figure>

Open question

<figure><img src="image_10.png" /></figure>

How to ensure computational reproducibility with increasingly big sample sizes?

<figure><img src="image_11.png" /></figure>

Talk 1 by Adina Wagner
On our way to fix irreproducibility

Irreproducible with...

Different Data
# On our way to fix irreproducibility

Irreproducible with...

Different Data

False positive finding

Low statistical power

SCIENCE

A Waste of 1,000 Research Papers

Decades of early research on the genetics of depression were built on nonexistent foundations. How did that happen?

ED YONG MAY 11, 2019

<figure><img src="image_12.png" /></figure>

SEAN NEL / SHUTTERSTOCK

In 1996, a group of European researchers found that a certain gene SLC6A4, might influence a person's risk of depression.

It was a blockbuster discovery at the time. The team found that a version of the gene was more common among 454 people who had disorders than in 570 who did not. In theory, anyone who had this gene variant could be at higher risk for depression, and that finding might help in diagnosing such disorders, assessing suicidal behavior.

Power of neuroscience studies

<figure><img src="image_13.png" /></figure>

[Button, Nat Rev Neurosci 2013]
# On our way to fix irreproducibility

Irreproducible with...

Different Data

False positive finding

Lack of generalisability

SCIENCE

## A Waste of 1,000 Research Papers

Decades of early research on the genetics of depression were built on nonexistent foundations. How did that happen?

ED YONG MAY 17, 2019

<figure><img src="image_14.png" /></figure>

SEAN NEL | SHUTTERSTOCK

In 1996, a group of European researchers found that a certain gene, called *SLC6A4*, might influence a person's risk of depression.

It was a blockbuster discovery at the time. The team found that a less active version of the gene was more common among 454 people who had mood disorders than in 570 who did not. In theory, anyone who had this particular gene variant could be at higher risk for depression, and that finding, they say, might help in diagnosing such disorders, assessing suicidal behavior, or even

[Border, Am J Psy. 2019]

Power of neuroscience studies

<figure><img src="image_15.png" /></figure>

[Button, Nat Rev Neurosci 2013]

The New York Times

## Many Facial-Recognition Systems Are Biased, Says U.S. Study

Algorithms falsely identified African-American and Asian faces 10 to 100 times more than Caucasian faces, researchers for the National Institute of Standards and Technology found.

<figure><img src="image_16.png" /></figure>

Morning at Grand Central Terminal. Technology for facial recognition is frequently biased, a new study confirmed. Timothy A. Clary/Agence France-Presse — Getty Images

[Buolamwini, MIT 2017]

X-ray opacity detections

<figure><img src="image_17.png" /></figure>

[Larrazabal, PNAS 2020]
# On our way to fix irreproducibility

Irreproducible with...

Different Data

False positive finding

Low statistical power

Lack of generalisability

Unrepresentative sample

<figure><img src="image_18.png" /></figure>

**Proposed solution**
Data Sharing
# On our way to fix irreproducibility

Irreproducible with...

Different Data

False positive finding

Low statistical power

Lack of generalisability

Unrepresentative sample

<figure><img src="image_19.png" /></figure>

**Proposed solution**
Data Sharing

<figure><img src="image_20.png" /></figure>

**Open question**

How do scientific findings apply to subgroups of the population?

<figure><img src="image_21.png" /></figure>

**Talk 3** by
Melanie Ganz

<figure><img src="image_22.png" /></figure>

**Open question**

How can inclusivity support fairness?

<figure><img src="image_23.png" /></figure>

**Talk 4** by
Cassandra Gould
van Praag

<figure><img src="image_24.png" /></figure>
# Fixing the reproducibility issue

Irreproducible with...

Different methods

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>

**Open question**

<figure><img src="image_27.png" /></figure>

How does a finding obtained with a given analysis vary when computed using another tool or algorithm?

<figure><img src="image_28.png" /></figure>

**Talk 2** by Camille Maumet
<figure><img src="image_29.png" /></figure>

**Talk 1 Towards computational reproducibility** when working with very large datasets

<figure><img src="image_30.png" /><figcaption>**Adina Wagner**
Psychoinformatics Lab
INM-7
Juelich Research Center
Germany</figcaption></figure>

**Talk 2 Different analysis pipelines, different results... How to deal with analytical variability?**

<figure><img src="image_31.png" /><figcaption>**Camille Maumet**
Empenn team
Inria, Univ Rennes, CNRS, Inserm
France</figcaption></figure>

**Talk 3 Fairness in biomedical imaging:** overview and a case study in neuroimaging

<figure><img src="image_32.png" /><figcaption>**Melanie Ganz**
Neurobiology Research
Unit at Rigshospitalet
Dpt. for Computer
Science
Uni. of Copenhagen,
Denmark</figcaption></figure>

**Talk 4 How does inclusivity support fairness in biomedical imaging?**

<figure><img src="image_33.png" /><figcaption>**Cassandra Gould Praag**
Dpt. of Psychiatry
Uni. of Oxford
United Kingdom</figcaption></figure>