<figure><img src="image_1.png" /></figure>

HAL
open science

# OpenBHB: a Large-Scale Multi-Site Brain MRI Data-set for Age Prediction and Debiasing

Benoit Dufumier, Antoine Grigis, Julie Victor, Corentin Ambroise, Vincent

Frouin, Edouard Duchesnay

► To cite this version:

Benoit Dufumier, Antoine Grigis, Julie Victor, Corentin Ambroise, Vincent Frouin, et al.. OpenBHB: a Large-Scale Multi-Site Brain MRI Data-set for Age Prediction and Debiasing. NeuroImage, 2022, 263, pp.119637. 10.1016/j.neuroimage.2022.119637. hal-04482833

HAL Id: hal-04482833

https://hal.science/hal-04482833v1

Submitted on 18 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

# OpenBHB: a Large-Scale Multi-Site Brain MRI Data-set for Age Prediction and Debiasing

<figure><img src="image_4.png" /></figure>

Benoit Dufumier*, Antoine Grigis, Julie Victor, Corentin Ambroise, Vincent Frouin, Edouard Duchesnay

NeuroSpin, CEA Saclay, Université Paris-Saclay, France

## ABSTRACT

Prediction of chronological age from neuroimaging in the healthy population is an important issue because the deviations from normal brain age may highlight abnormal trajectories towards brain disorders. As a first step, ML models have emerged to predict chronological age from brain MRI, as a proxy measure of biological age. However, there is currently no consensus w.r.t which Machine Learning (ML) model is best suited for this task, largely because of a lack of public benchmark. Furthermore, new large emerging population neuroimaging datasets are often biased by the acquisition center images are coming from. This bias heavily deteriorates models generalization capacities, especially for Deep Learning (DL) algorithms that are known to overfit rapidly on the simplest features (known as simplicity bias). Here we propose a new public benchmarking resource, namely Open Big Healthy Brains (OpenBHB), along with a challenge for both brain age prediction and site-effect removal through a representation learning framework. OpenBHB is large-scale, gathering >5K 3D T1 brain MRI from Healthy Controls (HC) and highly multi-sites, aggregating >60 centers worldwide and 10 studies. OpenBHB is expected to grow both in terms of available modalities and number of subjects. All OpenBHB datasets are uniformly preprocessed, including quality check, with container technologies that consist in: 3D Voxel-Based Morphometry maps (VBM from CAT12), quasi-raw (simple linear alignment of images), and Surface-Based Morphometry indices (SBM, from FreeSurfer). The OpenBHB challenge is permanent and we provide all tools, materials and tutorials for participants to easily submit and benchmark their model against each other on a public leaderboard.

## 1. Introduction

Brain development during aging implies several complex processes (e.g., cortical thinning or synaptic pruning) that vary drastically across individuals Østby et al. (2009). In particular, this maturation affects several functional and structural networks involved in cognition (e.g., working memory), motor functions, or emotion (e.g., Default Mode Network). It has been shown Bashyam et al. (2020b); Cole and Franke (2017); Cole et al. (2017); Jonsson et al. (2019); Peng et al. (2021) that machine learning (ML) models can learn from neuroimaging data to accurately estimate chronological age from the healthy population, taking into account the general variability (both environmental and genetic Jonsson et al. (2019)) to build strong predictors of brain development. The Brain Age Gap (BAG, defined as the absolute difference between chronological and predicted age) has been used as a proxy measure to detect both neurodegenerative and neurodevelopmental disorders Gaser et al. (2013); Kaufmann et al. (2019); Koutsouleris et al. (2014); Schnack et al. (2016) (e.g., Alzheimer or schizophrenia). It has also been described as a predictor of mortality Cole et al. (2018) and other brain disorders (such as major depressive disorder Han et al. (2020), bipolar disorder Kaufmann et al. (2019) or traumatic brain injuries Cole et al. (2015)).

Nevertheless, there are currently several shortcomings in the neuroimaging literature that heavily limit the clinical impact of such algo-

rithms. First, the lack of public benchmarks necessarily limits the comparison and reproducibility of competing works on brain age prediction. Recent studies He et al. (2021a,b) show that the choice of age range, number of samples, and pre-processing strategies - e.g. Region-Of-Interest (ROI), Voxel-Based Morphometry (VBM) or Surface-Based Morphometry (SBM) - are drastically different across studies, making the comparison difficult. In this regard, the Predictive Analytic Challenge Fisch et al. (2021) (PAC) in 2019 catalyzed the development of new ML algorithms and Deep Learning (DL) networks specially engineered for relatively large-scale (N = 2636) brain MRI data. Nevertheless, the development of DL architectures for neuroimaging data is still lagging behind the ones developed for natural images (e.g., there is still no consensus whether DL models are more efficient than simple regularized linear models Abrol et al. (2021); Schulz et al. (2020) on phenotype prediction even if more and more evidence is accumulating for the former He et al. (2021a,b); Peng et al. (2021)).

Second, most large emerging datasets are multi-sites (e.g., ABIDE, ABCD, ADNI, ENIGMA, SCHIZCONNECT), partly because of the high acquisition cost per patient in each study. Several recent works Glocker et al. (2019); Wachinger et al. (2021) have shown that ML models are heavily biased by the acquisition site, and they generalize poorly to MRI images coming from never-seen sites. This issue can be attributed to the difference between scanners manufacturers, specifications, settings, and hardware. This is an important limitation for

<figure><img src="image_5.png" /></figure>

<footnote>* Corresponding author.
E-mail address: benoit.dufumier@cea.fr (B. Dufumier).</footnote>
applying these models to neuroimaging data, especially for personal-
ized medicine in psychiatry. In this context, harmonization methods
Fortin et al. (2018); Garcia-Dias et al. (2020); Kia et al. (2020) have
emerged to remove this undesired variability from the data. However,
such harmonization models estimate their parameters on the entire
dataset, or at least, on a great portion of it containing all sites. It is also
a limitation in the context of personalized medicine, where MRI data
coming from new hospitals would mean to re-train the whole model
before making a new prediction. Besides, these methods are also sensi-
tive to the number of samples per site as some statistics (mean or vari-
ance) are estimated for each site separately. Other recent approaches
Bashyam et al. (2020a); Dewey et al. (2020); Liu et al. (2021); Liu and
Yap (2021); Robinson et al. (2020) are integrating DL to perform image-
to-image translation (e.g style transfer) in order to bring all images in
a common debiased space. Validating such approaches is often difficult
and it either relies on travelling patients (scanned at multiple sites),
which is very costly, or on statistical analysis on the generated images
(e.g using Fréchet Inception Distance Liu and Yap (2021)) or directly
by demonstrating that biological variables are well preserved (e.g age
or sex Bashyam et al. (2020a); Robinson et al. (2020)). Other line of
work Dinsdale et al. (2021) directly tries to remove site information via
adversarial attack while training an encoder to predict the biological
variable of interest (e.g age or sex). In that case, the validation pro-
cedure simply consists in evaluating the encoder's capacity to retain
biological and site information. All these approaches use different val-
idation procedure and they are hardly comparable to one another (as
they generally do not even use the same datasets and modalities).

As a result, we propose the OpenBHB Challenge on brain age pre-
diction with site-effect removal. This challenge is based on the large-
scale (N > 5000) multi-site brain MRI dataset OpenBHB that contains
both minimally preprocessed data along with VBM and SBM measures
derived from raw T1w MRI. All images in OpenBHB have passed a semi-
automatic visual quality check, and the data are publicly available on
the online IEEE Dataport platform. The challenge consists in learning
a representation of the data such that i) brain age variability is pre-
served and ii) site-related information is removed. The submitted mod-
els should output a vector representing input data such that brain age
can be easily predicted (i.e. through linear evaluation) and acquisition
site signal is absent (i.e. random chance for predicting site with linear
evaluation). Thus, this challenge is closely related to several hot topics
in ML/DL, such as representation learning driven by a supervised signal
Dufumier et al. (2021b); Khosla et al. (2020), debiasing and trustwor-
thy Al Bahng et al. (2020); Barbano et al. (2021); Cadene et al. (2019);
Clark et al. (2019); Tartaglione et al. (2021). To evaluate the submit-
ted models, we propose a novel metric computed on two test sets: an
internal test that contains images from the same sites as training and
external test including images from distinct sites. We hope this challenge
will facilitate the benchmarking of ML and DL models for both brain age
prediction and site-effect removal through a representation learning ap-
proach.

We plan to extend OpenBHB with additional subjects, longitudinal
data and other modalities (e.g., resting-state functional MRI and diffu-
sion MRI) that bring complementary structural and functional informa-
tion to the current T1w images.

In summary, in this paper our main contributions are:

* OpenBHB, a new large-scale (N > 5000) brain MRI dataset publicly available that includes:

* preprocessed quasi-raw, VBM and SBM T1w data;

* a visual quality check;

* a training, validation and test splits used for the OpenBHB chal-
lenge;

* a new challenge for brain age prediction with site-effect removal.

* a leader-board for the comparison of submitted models with a new metric.

* an online platform to submit the trained models.

2. OpenBHB dataset

2.1. Public datasets aggregated in OpenBHB

OpenBHB aggregates 10 publicly available datasets, namely IXI<fnref n="1" />,
ABIDE 1², ABIDE 2³, CoRR Zuo et al. (2014), GSP Buckner et al. (2014),
LOCALIZER Orfanos et al. (2017), MPI-Leipzig Babayan et al. (2019),
NAR Nastase et al. (2021), NPC Sunavsky and Poppenk (2020), and RBP
Follmer et al. (2018); Li and Clariana (2019). Currently, OpenBHB is fo-
cused only on Healthy Controls (HC) since the main challenge consists
in modeling the (normal) brain development by building a robust brain
age predictor. As a result, we only included HC from ABIDE 1 and 2,
and we left out the subjects with Autism Spectrum Disorders in the cur-
rent release. OpenBHB contains N = 5330 3D T1 brain MRI scans from
HC acquired on 71 different acquisition sites with eventually multiple
acquisition protocols per site (see Table 1). As highlighted in the map
accompanying Table 1, the subjects included in OpenBHB come from
European-American, European, and Asian genetic backgrounds, promot-
ing more diversity in OpenBHB. To manage redundant images, one ses-
sion per participant has been retained along with its best-associated run,
selected according to image quality. We also provide the participants
phenotype as well as site and scanner information associated with each
image, which essentially includes age, sex, acquisition site, diagnosis
(in our case only HC), MRI scanner magnetic field, and MRI scanner set-
tings identifier (a combination of multiple information composed of a
subset of the repetition time, echo time, sequence name, flip angle, and
acquisition coil). Some widespread confounds are also proposed, such
as the Total Intracranial Volume (TIV), the CerebroSpinal Fluid Volume
(CSFV), the Gray Matter Volume (GMV), and the White Matter Volume
(WMV).

The overall age and sex distributions for OpenBHB are plotted Fig. 2.
It should be noticed that sex distribution is globally well balanced for
all age bins. Age distribution contains 2 main modes centered around
10 years old (during synaptic pruning) and 25 years old with a long tail
above 40 until 88 years (and fewer samples in this range).

Additionally, we performed appearance analysis from VBM data, as
it preserves cortical and sub-cortical information from raw images (as
opposed to SBM) and ROI measures can be derived easily to reduce data
dimensionnality. Specifically, we represented the t-SNE visualization of
ROI extracted from VBM data in Fig. 3 per study. This plot clearly sug-
gests that age and site effect are driving the representation, thus justify-
ing the objective of the OpenBHB challenge. Datasets with *both* a large
number of sites and a large age range cover wider regions in t-SNE space
than others (e.g CoRR covers almost all regions while GSP and IXI cover
only middle and upper regions; even if they are lifespan, they only in-
clude 6 sites together vs 18 sites for CoRR). This is even more obvious
with MPI-Leipzig that covers mostly small left and upper regions while
it is also lifespan.

2.2. Preprocessing and derived anatomical features

All data are preprocessed uniformly with container technolo-
gies comprising quasi-raw, CAT12 VBM, and FreeSurfer (see
Fig. 2.2), which allows us to control the different software ver-
sions over time. The project hosting the codes is freely accessible
at https://brainprep.readthedocs.io. We conducted a semi-automatic
quality control (QC) guided with quality metrics leading to a selection
of images that meet the quality criteria for all three pre-processing
pipelines (see Fig. 5 and the detailed QC per pre-processing below).

<footnote n="1">¹ https://brain-development.org/ixi-dataset.</footnote>

<footnote n="2">² http://fcon_1000 projects.nitrc.org/indi/abide/abide_I.html.</footnote>

<footnote n="3">³ http://fcon_1000 projects.nitrc.org/indi/abide/abide_II.html.</footnote>
<caption>**Table 1**</caption>

OpenBHB demographic information. Acquisition settings include mainly the magnetic field strength and acquisition protocol used for MRI acquisition (see Section 2.1 for more details). Only images with available acquisition settings are included in the OpenBHB challenge (the number of sites excluded for the challenge are indicated in parentheses). Six sites are shared between ABIDE 1 and 2, and only healthy subjects are considered in the current release.

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /><figcaption>**Fig. 1.** Illustration of the OpenBHB dataset along with the proposed challenge. OpenBHB is a large-scale (N > 5K subjects), international (covers Europe, North America, and China), lifespan (5–88 years old) brain MRI dataset including images preprocessed with three pipelines (quasi-raw, VBM with CAT12, and SBM with FreeSurfer). It is openly accessible on IEEE Dataport. It comes with a new challenge on representation learning for brain age prediction with site debiasing. Challenge information and dataset accessibility procedure are described in our website.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>(a) Overall age and sex distributions.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(b) Age and sex distributions by study</figcaption></figure>

**Fig. 2.** Demographic description of OpenBHB (a) overall and (b) by study. Age histograms show a peak distribution for young adults (20–30 years old) with a long tail distribution for older adults (60–80 years old). While age disparities are observed between studies, the age remains a poor site predictor (see Table 3). All data-sets are well-balanced between males and females.

### 2.2.1. Quasi-raw

**Steps:** Minimally preprocessed data were generated using ANTS Avants et al. (2009) bias field correction, FSL FLIRT Jenkinson and Smith (2001) with 9 degrees of freedom (no shearing) followed by affine registration to the 1mm³ MNI template, and the application of a brain mask to remove non-brain tissues in the final images.

**Quality control:** First, we computed the correlation between each image and the mean of every other image in order to sort them by increasing correlation score. Then, images were manually inspected in-house following this sorting and a first threshold was set to remove the first *k* images. Additionally, we used the average correlation (using Fisher's z transform) between registered images as a metric of
<caption>**Table 2**</caption>

Training, Validation, Internal (stratified) and External test splits for the OpenBHB Challenge. 679 images from OpenBHB have missing label information (e.g acquisition setting) and are not considered in this challenge. External test is fully independent from OpenBHB and it is only used to assess the generalization capacity of submitted algorithms in the OpenBHB Challenge.

<table><caption>**Table 3**</caption><thead><tr><th>Split</th><th>Characteristic</th><th># Subjects</th><th>Age</th><th>Sex (%M)</th><th># Sites×Acq</th></tr></thead><tbody><tr><td>Train</td><td>Public, from OpenBHB</td><td>3227</td><td>25.2 ± 14.6</td><td>52</td><td>58</td></tr><tr><td>Validation</td><td>Public, from OpenBHB</td><td>757</td><td>23.8 ± 12.8</td><td>55</td><td>64</td></tr><tr><td>Internal Test</td><td>Private, from OpenBHB</td><td>664</td><td>25.3 ± 14.2</td><td>52</td><td>64</td></tr><tr><td>External Test</td><td>Private, new sites</td><td>720</td><td>22.3 ± 11.1</td><td>53.3</td><td>8</td></tr></tbody></table>

Performance for predicting one of the 64 pairs (site, acquisition setting) included in the OpenBHB Challenge. All CNN are 3D adaptation of the original architectures and AlexNet corresponds to the 3D version proposed in Abrol et al. (2021).

<table><caption>**Table 4**</caption><thead><tr><th>Input</th><th>Model</th><th>Balanced Accuracy (%)</th><th>Accuracy (%)</th></tr></thead><tbody><tr><td rowspan="3">Quasi-Raw</td><td>AlexNet Krizhevsky et al. (2012)</td><td>83.1</td><td>86.8</td></tr><tr><td>DenseNet Huang et al. (2017)</td><td>87.8</td><td>91.4</td></tr><tr><td>ResNet He et al. (2016)</td><td>79.1</td><td>88.7</td></tr><tr><td rowspan="3">VBM</td><td>AlexNet Krizhevsky et al. (2012)</td><td>42.3</td><td>47.8</td></tr><tr><td>DenseNet Huang et al. (2017)</td><td>62.2</td><td>72.4</td></tr><tr><td>ResNet He et al. (2016)</td><td>60.5</td><td>71.4</td></tr><tr><td>SBM(Xhemi)</td><td>Linear</td><td>38.0</td><td>57.5</td></tr><tr><td>SBM(ROI-Destrieux)</td><td>Linear</td><td>70.5</td><td>74.3</td></tr><tr><td>SBM(ROI-Desikan)</td><td>Linear</td><td>67.6</td><td>73.9</td></tr><tr><td>Age</td><td>Linear</td><td>2.86</td><td>26.7</td></tr><tr><td>Random Level</td><td>-</td><td>1.56</td><td>7.35</td></tr></tbody></table>

Baselines obtained with 1) no de-biasing strategy (first 3 rows) and 2) ComBat residualization on training data (last 3 rows) with VBM and Quasi-Raw data for 3 representative CNN families. MAE=Mean Absolute Error and BAcc=Balanced Accuracy (in %). Balanced Accuracy on internal test should be compared to baseline BAcc = 2.86% which is the prediction power of true age for predicting site in the internal test set.

<table><thead><tr><th rowspan="3">Method</th><th rowspan="3">Model (Latent Dim. # params)</th><th colspan="4">VBM</th><th colspan="4">Quasi-Raw</th></tr><tr><th colspan="2">Int. Test</th><th colspan="2">Ext. Test</th><th colspan="2">Int. Test</th><th colspan="2">Ext. Test</th></tr><tr><th>MAE↓</th><th>BAcc↓</th><th>MAE↓</th><th>LC↓</th><th>MAE↓</th><th>BAcc↓</th><th>MAE↓</th><th>LC↓</th></tr></thead><tbody><tr><td rowspan="3">Baseline</td><td>DenseNet (1024, 11.2M)</td><td>2.55<sup>±0.009</sup></td><td>8.0<sup>±0.9</sup></td><td>7.13<sup>±0.05</sup></td><td>3.34</td><td>2.48<sup>±0.03</sup></td><td>15.2<sup>±0.6</sup></td><td>2.92<sup>±0.07</sup></td><td>1.66</td></tr><tr><td>ResNet (512, 33.2M)</td><td>2.67<sup>±0.05</sup></td><td>6.7<sup>±0.1</sup></td><td>4.18<sup>±0.01</sup></td><td>1.86</td><td>2.60<sup>±0.003</sup></td><td>7.6<sup>±0.1</sup></td><td>2.85<sup>±0.004</sup></td><td>1.31</td></tr><tr><td>AlexNet (128, 2.5M)</td><td>2.72<sup>±0.01</sup></td><td>8.3<sup>±0.2</sup></td><td>4.66<sup>±0.05</sup></td><td>2.21</td><td>2.96<sup>±0.005</sup></td><td>16.2<sup>±0.5</sup></td><td>3.65<sup>±0.009</sup></td><td>2.11</td></tr><tr><td rowspan="3">ComBat Fortin et al. (2018)</td><td>DenseNet (1024, 11.2M)</td><td>5.92<sup>±0.01</sup></td><td>2.23<sup>±0.06</sup></td><td>10.48<sup>±0.17</sup></td><td>3.38</td><td>N.A.</td><td>N.A.</td><td>N.A.</td><td>N.A</td></tr><tr><td>ResNet (512, 33.2M)</td><td>4.15<sup>±0.009</sup></td><td>4.5<sup>±0.0</sup></td><td>4.76<sup>±0.03</sup></td><td>1.88</td><td>N.A.</td><td>N.A.</td><td>N.A.</td><td>N.A</td></tr><tr><td>AlexNet (128, 2.5M)</td><td>3.37<sup>±0.01</sup></td><td>6.8<sup>±0.3</sup></td><td>5.23<sup>±0.12</sup></td><td>2.33</td><td>N.A.</td><td>N.A.</td><td>N.A.</td><td>N.A</td></tr></tbody></table>

<figure><img src="image_10.png" /><figcaption>**Fig. 3.** t-SNE representation of VBM ROI normalized by TIV for each sample in OpenBHB Challenge. Age and sites dominate the representation, and studies with multiple sites have a broader variety of images (in terms of surface covered in the t-SNE space). For instance, MPI-Leipzig covers mostly a single region on the left while NAR is more varied. CoRR is the most varied study as it covers almost all representation space.</figcaption></figure>
<figure><img src="image_11.png" /><figcaption>Fig. 4. Illustration of the OpenBHB available preprocessed data: quasi-raw, Gray Matter (GM) CAT12 VBM, and FreeSurfer "recon-all" from left to right.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 5. OpenBHB quality assessment and image selection. Several metrics of quality have been used to perform the quality check (QC) on OpenBHB (train+internal test) and the external test. IQR: Image Quality Rating; NCR: Noise Contrast Ratio; Correlation: Average correlation between each registered images and all the other ones (Z-transformed). The manually determined cutting threshold used as inclusion criteria in QC is indicated by the red vertical line. A clear domain gap is observed on VBM and SBM (FreeSurfer) data between train and external test. However we have similar image quality between OpenBHB train and internal test for all modalities (VBM, SBM, Quasi-raw).. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)</figcaption></figure>

quality and we retained only images at a threshold higher than 0.5
(see Fig. 5).

### 2.2.2. CAT12 VBM

**Steps:** Voxel-Based Morphometry (VBM) was performed with CAT12 Gaser and Dahnke (2016) (http://www.neuro.uni-jena.de/cat). The analysis stream includes non-linear spatial registration to the 1.5mm³ MNI template, Gray Matter (GM), White Matter (WM), and CerebroSpinal Fluid (CSF) tissues segmentation, bias correction of intensity non-uniformities, and segmentations modulation by scaling with the amount of volume changes due to spatial registration. VBM is most often applied to investigate the GM. The sensitivity of VBM in the WM is low, and usually, diffusion-weighted imaging is preferred for that purpose. For this reason, only the modulated GM images are shared. Moreover, CAT12 computes GM volumes averaged on the Neuromorphometrics atas that includes 284 brain cortical and sub-cortical ROI.

**Quality control:** We performed the same in-house QC visual analysis as for quasi-raw images (see Section 2.2.1). Additionally, we also monitored the Noise Contrast Ratio (NCR) and Image Quality Rating (IQR) as two metrics of quality and we retained only images at a threshold below 4 (see Fig. 5).

### 2.2.3. FreeSurfer

**Steps:** Cortical analysis was performed with FreeSurfer “recon-all” (https://surfer.nmr.mgh.harvard.edu). The analysis stream includes intensity normalization, skull stripping, segmentation of GM (pial) and WM, hemispheric-based tessellations, topology corrections and inflation, and registration to the "fsaverage" template. From the available morphological measures, the Desikan Desikan et al. (2006) and Destrieux Fischl et al. (2004) ROI-based cortical thickness (CT), surface area (SA), and curvature (CR) are shared. Specifically, 7 ROI-based features computed both on Desikan and Destrieux atlases are shared including: the cortical thickness (mean and standard deviation), GM volume, surface area, integrated mean and Gaussian curvatures and intrinsic curvature index. Moreover, vertex-wise cortical thickness, curvature and average convexity features Fischl et al. (1999) (measuring the depth/height of a vertex above the average surface) are also accessible on the high-resolution seven order icosahedron. To allow inter-hemispheric cortical surface-based analysis, we further transform the right hemisphere features into the left one, using the symmetric "fsaverage_sym" Freesurfer template and the "xhemi" routines Greve et al. (2013). The final vertex-wise cortical features comprise 163,842 nodes per hemisphere.
OpenBHB Splits

<figure><img src="image_13.png" /><figcaption>**Fig. 6.** Splitting strategy used for OpenBHB. The public data available are split into a training and validation set (useful for cross-validation and to derive comparable public results). Private data (used to score the models submitted to the OpenBHB challenge) are composed of 2 subsets: an internal test (stratified on age, sex and site from the OpenBHB dataset) and an external test (independent from OpenBHB and with acquisition centers distinct from the public data). Importantly, the validation set is built in a similar fashion from public OpenBHB data to allow participants to derive all the challenge's metrics.</figcaption></figure>

**Quality control:** Similarly with quasi-raw and VBM, we first performed a visual analysis on images ranked by the correlation score. In addition we used the Euler number as a metric of quality and we retained images at a threshold greater than -217, as specified in Rosen et al. (2018) (see Fig. 5).

## 2.3. Train-Validation-Test splits of OpenBHB with external test for the OpenBHB challenge

For the proposed OpenBHB Challenge (see hereafter Section 3), we have carefully designed a train-validation-test split (Table 2) such that the public *training* and *validation* sets and the so-called *internal test* set, that are all issued from OpenBHB, share the essential statistical properties needed for the challenge, i.e., similar age, sex, and site distributions. Additionally, to assess the generalization powers of submitted models in the challenge, we have built an independent *external test* set (issued from other sites) described hereafter. Fig. 6 gives a quick visual overview of the splitting strategy. Notably, both internal and external tests are currently kept private to the participants in the context of the OpenBHB Challenge.

**Training, Validation and Internal test splits of OpenBHB.** We used a stratified sampling of OpenBHB similar distributions of essential variables, i.e., age, sex, and site. Considering the large number of sites (> 60) and age bins in OpenBHB - for a reasonable binarization scheme (e.g., 5-years bins) - it is prohibitive to use the naïve stratification approach based on the cartesian product between sites, binarized age, and sex (there would not be enough samples per bin). To properly define such a split, we used the iterative stratification algorithm Sechidis et al. (2011) that tries to optimally preserve the different age, sex, and site histograms between train and test. Using this method, we obtained a training/internal test split that holds well age, sex, and site statistics (see Fig. 7 and Table 2).

**External test.** In the context of the OpenBHB Challenge, we also built an external test, fully independent from OpenBHB that also preserves age and sex statistics (see Fig. 7 and Table 2) but that contains images acquired on independent sites (i.e., with no site overlap with OpenBHB). These sites are kept private to avoid any bias or data leakage during the challenge. The external test set is used to evaluate the generalization capacity of ML models on never-seen sites for brain age prediction (see Section 3 for more details). As for the OpenBHB challenge, MRI scans from this external test have been acquired on both 1.5T and 3T in geographically distinct locations, across 8 acquisition centers.

**Region coverage analysis between training, validation and tests** Previously we showed that training, validation and the 2 test sets share the same age and sex distributions (along with the same site distribution between training and internal test). As we saw in Fig. 3, we can use

t-SNE visualization tool on VBM data to analyze the appearance similarity between training and internal or external test sets (similarly with Castillo-Navarro et al. (2021) in the context of semi-supervised learning for labelled/unlabelled splits). Specifically, we performed dimensionality reduction with t-SNE to project ROI VBM data in a 2D space. Then we analyzed the regions covered by training, internal test and external test in this space. In Fig. 8, we observe that most regions are covered both by the training and internal test in the t-SNE projected space while some regions are only covered by the external test (e.g bottom and right regions, usually for younger participants). This suggests that the training set covers well the internal test but there is a domain gap between training and external test. To quantitatively assess the coverage of both internal and external tests over training, we took inspiration from Castillo-Navarro et al. (2021) by defining the Intersection Over Union (IoU) metric between train and test. We have defined the regions covered by each split by performing a one-class SVM algorithm on data reduced by t-SNE and we have computed IoU between the regions covered by 2 splits (see Castillo-Navarro et al. (2021) for more details). In our case, we have IoU=0.94 between {training,validation} and internal test and IoU=0.64 between {training,validation} and external test. This further supports our claim that our training and validation sets represent well our internal test set while the external test covers new regions, distinct from training and validation. This could be directly related to site effect that is present in our dataset (see Section 3.4) and, thus, we favor algorithms insensitive to domain gap in the OpenBHB challenge.

## 2.4. Data organization and accessibility

**Data sharing.** A data sharing platform distributes the prepared OpenBHB dataset. All up-to-date information are centralized at this location https://baobablab.github.io/bhb.

**Data organization.** Currently, we are only sharing the training and validation splits for the OpenBHB challenge, in order to avoid data leakage. All modalities (quasi-raw, VBM, SBM) are stored in NumPy (.npy) format to allow easy cross-platform implementation. A resource folder contains information about the actual geometry of all manipulated data (i.e VBM and quasi-raw MNI templates as well as ROI labels for the Desikan Desikan et al. (2006) and Destrieux Fischl et al. (2004) atlases used by FreeSurfer and for the Neuromorphometric atlas used by CAT12). The data are stored in N-dimensional arrays, where the first two dimensions are sample size and number of modalities (or channels, equal to 1 here), followed by the data dimension. All metadata are stored in the participants TSV file and quality control metrics in the *qc* TSV file. Finally, the 64 pair (site, acquisition setting) labels used in the OpenBHB challenge are available in the TSV file *official_site_class labels*. In more details, the directory contains:
<figure><img src="image_14.png" /><figcaption>**Fig. 7.** Age, sex and study distribution between the training + validation and internal testing splits defined for the OpenBHB Challenge. All statistics are well preserved between both splits, thus avoiding any obvious bias in ML predictions.</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>**Fig. 8.** t-SNE representation of VBM ROI normalized by TIV for each sample in the challenge splits. Internal test is representative of the training set in terms of covered regions while external test has regions not represented in train (especially for younger participants, bottom and right regions).</figcaption></figure>

* **participants.tsv**: metadata table with columns participant identifier, study, sex, age, site, acquisition settings, TIV, CSFV, GMV, and WMV (cf. Section 2.1).

* **qc.tsv**: quality control table with columns participant identifier, recon-all Euler, CAT12 VBM NCR, CAT12 VBM IQR, and quasi-raw correlation (cf. Section 2.2).

* **official_site_class labels.tsv**: MRI images are biased according to the specific scanner used and the acquisition protocol set (e.g., repetition time, echo time, etc.). Consequently, the correct confounding variable to remove is the *pair* (site, acquisition setting). As a result, we discretized all these pairs to create the confounding variable "siteXacq" that is defined in this file. All participants in the OpenBHB Challenge should use this variable to remove "site effect". Thus, when referring to "site removal", we implicitly englobe both the scanner and acquisition protocol used to generate the final MRI image.

* **sub\*-desc-gm_T1w.npy**: the GM VBM image with shape `[1 × 1 × 121 × 145 × 121]` (see Section 2.2.2). The first two dimensions represent the number of sample and channel (only one sample and GM tissue here) while the last three dimensions are the spatial image dimension (`1.5mm³` spatial resolution registered on MNI template).

The corresponding MNI template is stored as NIfTI in `resource/cat12vbm_space-MNI152_desc-gm_TPM.nii.gz`.

* **sub\*-preproc-cat12vbm_desc-gm_ROI.npy**: the GM volumes averaged on the anatomical Neuromorphometrics template with shape `[1 × 1 × 284]` (see Section 2.2.2). This template includes 142 cortical and sub-cortical regions for both GM and CSF volumes, thus totaling 284 GM volumes that are stored in the last dimension. The template can be found as NIfTI file in `resource/neuromorphometrics.nii`.

* **sub\*-preproc-quasiraw_T1w.npy**: the T1w quasi-raw (minimally preprocessed) image with shape `[1 × 1 × 182 × 218 × 182]` (see Section 2.2.1). The resulting image contains all brain tissues and it is registered to the MNI template with `1mm³` spatial resolution. The last three dimensions are the spatial image dimension. The template can be found as NIfTI in `resource/quasiraw_space-MNI152_desc-brain_T1w.nii.gz`.

* **sub\*-preproc-freesurfer_desc-desikan_ROI.npy**: cortical thickness (with standard deviation), GM volume, surface area, integrated mean (and Gaussian) curvature and intrinsic curvature index averaged on the Desikan cortical template Desikan et al. (2006) with shape `[1 × 7 × 68]` (see Section 2.2.3). These 7 features are stored in the second (channel) dimension for all 68 brain regions de-
fined by Desikan template (34 by hemisphere), stored in last dimension. Brain region labels and channels order can be found in resource/freesurfer_atlas-desikan labels.txt and freesurfer_channels.txt respectively.

* sub-* _preproc-freesurfer_desc-destrieux_ROI.npy: same cortical thickness, GM volume, curvature and surface area measures as previously, averaged on the Destrieux cortical template Fischl et al. (2004) with shape [1 × 7 × 148] (see Section 2.2.3). This template includes 148 brain regions (74 by hemisphere), on which the same 7 features are computed. Brain region labels and channels order can be found in resource/freesurfer_atlas-destrieux labels.txt and resource/freesurfer_channel s.txt respectively.

* sub-* _preproc-freesurfer_desc-xhemi_T1w.npy: cortical thickness, curvature, average convexity features Fischl et al. (1999) and Desikan cortical parcellation measures computed on the high-quality "fsaverage" mesh with shape [1 × 8 × 163842] (see Section 2.2.3). Both right and left hemisphere measures are provided on the "fsaverage_sym" template (163842 vertices), providing 4 × 2 = 8 features on each vertex that are stored in second (channel) dimension, the last dimension being the mesh. Channels order can be found in resource/freesurfer_xhemi_channel s.txt

### 3. OpenBHB challenge: Representation learning for age prediction with site effect removal

We propose a challenge to compare the models capacity to encode a relevant representation of the data that preserves the biological variability associated with age while removing the site-specific information. All required information, data loading, models submission, etc. is described on the web page: https://baobablab.github.io/bhb/challenges/ageentectionwithsite removal.

#### 3.1. Background

Predicting phenotype (such as age and sex) from brain imaging data is a key challenge to answer several exciting questions (e.g., biomarker discoveries for psychiatric disorder or neurocognitive impairment with brain age, personalized medicine with normative modeling, etc.). Recent efforts in ML/DL for neuroimaging along with the availability of large-scale datasets have led to the development of models increasingly accurate, capable of predicting chronological age within 2–3 years, and sex with 99% accuracy from brain MRI. However, these models are severely biased by non-biological variability, particularly associated with acquisition sites (different acquisition protocols, manufacturers, magnetic fields, etc.). These sources of variability inevitably limit the performance of such models and can even bias the neuroimaging community towards over-optimistic results, mainly when using small samples size.

#### 3.2. Challenge description

Consequently, we propose a representation learning challenge using the OpenBHB dataset. The aim is to learn a representation of the 3D T1 anatomical MRI pre-processed with the three pipelines described previously that full-fills 2 key properties: 1) the representation should be predictive of biological age; 2) site information should be removed from the representation. Thus, we aim to compare the capacity of the proposed models to encode a relevant representation of the data (feature extraction and dimensionality reduction) that preserves the biological variability associated with age while removing the site-specific information. The algorithms submitted must output a vector of features with $p < 10^4$ dimensions for each input data. Both quasi-raw, VBM, and SBM features (including ROI-based and voxel-wise features for VBM and SBM data, see Section 2.4) will be given as input data and the participants are free to use only a subset of these modalities (for instance only voxel-wise VBM and ROI-based SBM).

Models submitted will be evaluated with the standard linear evaluation protocol (detailed hereafter) to predict age and site from the embedded features. A general overview of the model evaluation workflow for this challenge is depicted in Fig. 9.

**Linear evaluation details.** A logistic regression is used to evaluate the representation quality on site prediction (as an innovative way to measure if site information has been removed). It is trained on the public data encoded by the model submitted and tested on the private data (only internal test) also encoded. As for age prediction, a Ridge regression is trained on the public data and tested on both internal and external test sets (see Section 2.3). To generate more robust metrics for this challenge, a 3-fold CV scheme has been implemented for the training phase of the linear probe (see Fig. 9). This 3-CV scheme is implemented on the whole public data-set (training + validation). As a result, for each metric, a mean and standard deviation are computed and reported in the final official leaderboard.

**Metric.** We have developed a novel metric that jointly evaluates two critical properties of the learned representation: its robustness w.r.t sites and the quantity of information preserved w.r.t chronological age. This metric combines two reference metrics: Mean Absolute Error for age prediction (MAE, to be minimized), and Balanced Accuracy for site prediction (BAcc, it should be equal to random chance). BAcc is the preferred metric for classification since sites distribution is heavily imbalanced in OpenBHB (see Fig. 7).

To compute the challenge's metric described above, two distinct test sets has been derived (see Section 2.3): (i) an Internal test set containing images from the same sites as the training set (both in OpenBHB); (ii) an External test completely independent of OpenBHB, with no site overlap. Only external set is used to compute MAE for age prediction while internal test is used to derive BAcc for site prediction. All metrics will be displayed during the challenge, and the overall ranking will be based on the following metric (the lower, the better):

$$L_c = \text{BAcc}(\text{sites})^\alpha \cdot \text{MAE}_{\text{external test}}(\text{age}) \quad (1)$$

If the representation learnt retains all site information, then the BAcc is equal to 1. The other way around, if the representation learnt is independent of the site, then the BAcc is no more than random chance which is $1/N_{sites}$ where $N_{sites}$ is the number of sites ($N_{sites} = 64$ for this challenge). As a result, BAcc and MAE give complementary information to quantitatively assess whether site information is completely removed from the representation (low BAcc) and biological variability is preserved (low MAE). One remaining question is: what weight $\alpha$ do we chose as a trade-off for ranking algorithms ? In Fig 10, we have performed an analysis of several baseline algorithms (CNN and MLP) trained with several modalities (VBM, SBM, Quasi-Raw) and we have represented their performance in the 2D plane (MAE(ext), Bacc(sites)). Based on this analysis, we have selected the optimal weight $\alpha = 0.3$ such that i) algorithms are ranked according to MAE in priority ($\alpha \ll 1$) but ii) models with similar MAE are ranked through BAcc ($\alpha > 0$). In particular, it ensures that perfectly debiased algorithms that poorly predict age have poor ranking, as well as good age predictors with very strong biased representations towards sites.

The two test sets are hidden to the participants, and they only have access to the training and validation sets with all meta-data information described in Section 2.1 (in particular, participant's age and sex and acquisition site).

#### 3.3. Leaderboard and submission

The leaderboard (available here) contains the 3 metrics defined in the current challenge: MAE for age prediction (computed on external test but also internal test for completeness), BAcc for site prediction (computed only on internal test) and $L_c$ (as defined in Eq. 1). The algorithms are ranked according to this last metric. The full submission process (including the expected code) is described in the challenge web page.
Model Evaluation Workflow

<figure><img src="image_16.png" /><figcaption>**Fig. 9.** Model evaluation workflow of a new submission. When a new trained model is submitted to our, a linear probe (regressor for age prediction and classifier for site classification) is trained on top of the public embedded data (i.e. public data encoded by the submitted model). Once trained, this linear probe predicts the downstream targets (age and site) on the private embedded data (age is predicted from both private internal and external tests while site is predicted from private internal test only). 3 metrics are then derived: Mean Absolute Error (MAE) for age prediction on internal and external test; Balanced Accuracy (BAcc) for site prediction on internal test. These 3 metrics are combined to derive the final challenge metric $L_c$ (see eq. 1).</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>**Fig. 10.** Optimal challenge metric for ranking submitted algorithms during the OpenBHB challenge. Each point represents an algorithm performance (CNN or MLP) ran on either VBM, Quasi-Raw or SBM data with a specific architecture in the (MAE(age), Bacc(sites)) plane. Color represents the ranking. Perfect algorithms should be in the bottom left corner. Isoline with constant $L_c$ (i.e same ranking) is represented with solid colored lines. The following metric is tested: $L_c = Bacc^α · MAE(ext)$ with α a hyper-parameter. Black arrows represent the decreasing ranking trend of the submitted algorithms (e.g. for α = 1, algorithms with low Bacc(sites) have good ranking, no matter their MAE(age) while for α = 0.2, algorithms with low MAE(age) have good ranking, no matter their Bacc(sites)). α = 0.3 is a good empirical trade-off to have good ranking of algorithms that both i) preserve age variability (low MAE) and ii) remove site information (low Bacc). The ranking metric $L_c = BAcc^{0.3} · MAE(ext)$ is the final choice retained for the OpenBHB challenge.</figcaption></figure>
**Impact in neuroimaging.** This challenge tackles several key problems encountered by the neuroimaging community that require the development of new innovative algorithms that might be borrowed from the computer vision field. Our three preprocessing pipelines allow to use both DL models and standard linear or kernel methods (that latter working on VBM and SBM data while the former works on all modalities). Furthermore, site can be viewed as a confounding variable to remove so this challenge encourages the use of debiasing algorithms Bahng et al. (2020); Barbano et al. (2021); Cadene et al. (2019); Clark et al. (2019); Tartaglione et al. (2021), a hot topic in computer vision often related to Trustworthy AI and Fairness. We envisage an increasing interest by the vision community in this field (e.g., aiming at building racially or gender debiased models). This challenge also tries to build upon new evaluation strategies that have emerged in representation learning with DL, in particular the linear evaluation setting Alain and Bengio (2016); Caron et al. (2020); Chen et al. (2020); He et al. (2020). We hope it urges researchers to develop new methods that can be translated into an unsupervised setting (such as new self-supervised regularization terms), and thus enhance the DL capacity to generalize well to different tasks (such as brain tumor segmentation or computer-aided diagnosis). Finally, generalizing well on data from never-seen sites implies a good robustness of the algorithm developed to out-of-domain images (related to Domain Adaptation, also a hot topic in the computer vision field-see for instance Wang and Deng (2018) for a comprehensive survey). Consequently, this challenge intends to bring together both neuroimaging and computer vision communities on a new large-scale 3D biomedical dataset.

## 3.4. Name-That-Site performance

In order to assess the current bias in OpenBHB, we played at the game "Name-That-Site", inspired by Torralba and Efros (2011) on natural images and Wachinger et al. (2021) on brain MRI (originally created as "Name-That-Dataset"). We train a classifier on different input data x to classify between the 64 pairs (site, acquisition setting) in the OpenBHB Challenge and we test it by following the challenge training + validation/test splits defined Section 2.3. We used a CNN for VBM and Quasi-Raw inputs and linear logistic regressions for ROI surface-based measures and age.

### 3.4.1. Training details

We considered 3D adaptation of three representative CNN architectures: AlexNet inspired from Abrol et al. (2021) (5 convolutional layers and 2.5M parameters), ResNet18 He et al. (2016) (18 convolutional layers and 33.2M parameters) and DenseNet121 Huang et al. (2017) (121 convolutional layers and 11.3M parameters) for whole-brain imaging data (VBM and quasi-raw). Their implementation is available on challenge web page : https://baobablab.github.io/bhb/challenges/age prediction_with_site_removal. We trained them for 300 epochs with an initial learning rate $\alpha = 10^{-4}$ reduced by factor 0.9 every 10 epochs. We used the standard cross-entropy loss optimized with Adam Kingma and Ba (2015) optimizer ($\beta_1, \beta_2$) = (0.9, 0.999). As for SBM data (both ROI-based and mesh-based features), we simply optimized a logistic regression using scikit-learn Pedregosa et al. (2011) and we cross-validated the regularization term $C \in \{0.01, 0.1, 1, 10, 100\}$ on the validation set. All SBM features are flattened (i.e we merge channels and spatial dimensions) to perform logistic regression.

### 3.4.2. Results

Results in Table 3 first indicate that all modalities preserve site information, especially input data in their rawest form (Quasi-Raw) with 83% BAcc. Second, age give small hints about site (2.86% BAcc) indicating that participants inclusion is biased for some datasets (e.g ABIDE) but it remains negligible compared to other modalities (always > 42% BAcc). These results are in line with recent literature

Wachinger et al. (2021) and it highlights the high non-biological heterogeneity that remains in MRI images even after non-linear registration and normalization (e.g VBM). Finally, these results suggest that Bacc is an appropriate metric to quantify the bias in models representation.

As a final remark, we notice that even if BAcc using age is low (< 3%), it is still not equal to the theoretical random level ($\frac{1}{N_{sites}} = 1/64$). Still, in our final metric $\mathcal{L}_{challenge}$ for the challenge (see Eq. 1), we used this theoretical bound to weight the two MAE computed on internal and external test. It means that we always give more importance to brain age predictions quality on the external test, projecting this challenge in a more realistic clinical scenario.

## 3.5. Baselines for the OpenBHB challenge

Next, we performed baseline experiments on different data modalities (Quasi-Raw, VBM, SBM) for the OpenBHB Challenge. Specifically, we trained CNNs with various architectures on 3D whole-brain imaging data (voxel-wise VBM and Quasi-Raw). The objective function is a simple $\ell_1$ loss on age prediction for all models.

We tested data-based debiasing model with the popular ComBat Fortin et al. (2018) residualization method, applied only on training set (both internal and external test sets are left non-harmonized since age and site labels are not available). This approach is compared to the standard brain age prediction DL-based model which does not take site information into account.

### 3.5.1. Training and evaluation details

We used the same CNN architectures as described Section 3.4.1 for VBM and quasi-raw data. We optimized the $\ell_1$ loss between true and predicted age with Adam optimizer and initial learning rate $\alpha = 10^{-4}$ decreased by a factor 0.9 every 10 epochs. Then, the last fully-connected (FC) layer is removed and the representation is evaluated using 1) Logistic Regression for site prediction with cross-validation of $\ell_2$-regularization parameter $C \in \{0.01, 0.1, 1, 10, 100\}$ and 2) Ridge Regression for age prediction on internal and external test with cross-validation of regularization in $\{0.01, 0.1, 1, 10, 100\}$. Evaluation procedure is detailed in Section 3.2 and it is executed in the same manner for all submitted models on server side. These models have been submitted to the official challenge and the results can also be found in the official leaderboard.

ComBat residualization is performed only on training data using age, sex (biological variables to keep) and site (confounding variable to remove). We used the official GitHub implementation of ComBat<fnref n="4" />.

For SBM ROI-based and mesh-based data, we used vanilla Multi-Layer Perceptrons (MLP) with varying depth but constant latent space dimension (128) and number of neurons per hidden layer (128). FC layer is added on top of MLP encoder to optimize $\ell_1$ loss on age prediction with a cross-validation on learning rate $\alpha \in \{10^{-4}, 10^{-3}, 10^{-2}\}$. The evaluation procedure is the same as previously. Importantly, for MLP training on SBM Xhemi (mesh-based measures) we did not include the channels associated to Desikan cortical parcellation (since it does not bring more anatomical information).

### 3.5.2. Results

We reported the 3 metrics used in the OpenBHB Challenge: MAE on internal and external test and Bacc on site prediction (see Section 3). The latent space dimension varying across CNN architectures is systematically reported.

First, we notice that all models retain site information without any debiasing strategy. Overall, Quasi-Raw data are more biased than VBM, as we saw previously (see Table 3), and CNN preserve this bias to some extent. This is especially true for DenseNet, the best performing network on both VBM and Quasi-Raw on the internal test, one of the models that retains the most site information (8.0% Bacc and 15.2% Bacc

<footnote n="4">⁴ https://github.com/Jfortin1/neurocombat sklearn.</footnote>
<caption>**Table 5**</caption>

Baseline results on SBM data using simple Multi-Layers Perceptron (MLP) with increasing depth (from 1 to 4) trained to predict age. The latent space dimension is fixed to 128 for all models. ComBat residualization is used only during training of the MLP models as a debiasing method. Linear models are also evaluated for comparison purposes (even if they are not accepted as a valid model in this challenge). Overall, deeper models lead to better data representation but they are still very biased by site without any debiasing strategy. ComBat residualization degrades performance for age prediction but it removes efficiently site bias (especially for the deepest models).

<table><thead><tr><th rowspan="3">Modality</th><th rowspan="3">Model (Hidden Layers)</th><th colspan="4">Baseline</th><th colspan="4">ComBat Fortin et al. (2018)</th></tr><tr><th colspan="2">Int. Test</th><th colspan="2">Ext. Test</th><th colspan="2">Int. Test</th><th colspan="2">Ext. Test</th></tr><tr><th>MAE↓</th><th>BAcc↓</th><th>MAE↓</th><th>L<sub>c</sub>↓</th><th>MAE↓</th><th>BAcc↓</th><th>MAE↓</th><th>L<sub>c</sub>↓</th></tr></thead><tbody><tr><td rowspan="5">FSL-Xhemi</td><td>Linear(Original)</td><td>4.69</td><td>38.0</td><td>5.96</td><td>4.46</td><td>5.08</td><td>30.8</td><td>6.19</td><td>4.35</td></tr><tr><td>MLP(128)</td><td>3.72<sub>±0.04</sub></td><td><b>8.6</b><sub>±1.0</sub></td><td>5.31<sub>±0.07</sub></td><td><b>2.54</b></td><td>4.06<sub>±0.05</sub></td><td>6.00<sub>±0.05</sub></td><td>5.0<sub>±0.2</sub></td><td>2.15</td></tr><tr><td>MLP(128, 128)</td><td>3.90<sub>±0.03</sub></td><td>16.4<sub>±1.14</sub></td><td>5.06<sub>±0.06</sub></td><td>2.94</td><td>4.49<sub>±0.04</sub></td><td>6.6<sub>±0.4</sub></td><td>5.34<sub>±0.03</sub></td><td>2.36</td></tr><tr><td>MLP(128, 128, 128)</td><td>3.75<sub>±0.008</sub></td><td>16.7<sub>±1.28</sub></td><td>5.07<sub>±0.07</sub></td><td>2.96</td><td><b>4.27</b><sub>±0.02</sub></td><td>4.2<sub>±0.1</sub></td><td>5.43<sub>±0.04</sub></td><td>2.1</td></tr><tr><td>MLP(128, 128, 128, 128)</td><td><b>3.49</b><sub>±0.01</sub></td><td>14.4<sub>±1.05</sub></td><td><b>4.85</b><sub>±0.02</sub></td><td>2.71</td><td>4.28<sub>±0.01</sub></td><td><b>4.0</b><sub>±0.04</sub></td><td><b>5.25</b><sub>±0.01</sub></td><td><b>2.0</b></td></tr><tr><td rowspan="5">FSL-ROI Destrieux</td><td>Linear(Original)</td><td>4.96</td><td>70.5</td><td>7.21</td><td>6.49</td><td>6.44</td><td>36.7</td><td>9.30</td><td>6.88</td></tr><tr><td>MLP(128)</td><td>3.96<sub>±0.01</sub></td><td>24.9<sub>±0.9</sub></td><td>6.72<sub>±0.12</sub></td><td>4.43</td><td>5.65<sub>±0.08</sub></td><td>15.0<sub>±1.19</sub></td><td>11.79<sub>±0.98</sub></td><td>6.67</td></tr><tr><td>MLP(128, 128)</td><td>3.36<sub>±0.01</sub></td><td>32.4<sub>±0.94</sub></td><td>4.89<sub>±0.10</sub></td><td>3.49</td><td>4.52<sub>±0.04</sub></td><td>19.8<sub>±0.9</sub></td><td>6.27<sub>±0.19</sub></td><td>3.86</td></tr><tr><td>MLP(128, 128, 128)</td><td><b>3.07</b><sub>±0.02</sub></td><td>21.1<sub>±1.13</sub></td><td>4.69<sub>±0.05</sub></td><td>2.94</td><td>4.39<sub>±0.02</sub></td><td>7.1<sub>±0.2</sub></td><td>5.81<sub>±0.09</sub></td><td>2.63</td></tr><tr><td>MLP(128, 128, 128, 128)</td><td>3.12<sub>±0.005</sub></td><td><b>12.8</b><sub>±0.5</sub></td><td><b>4.48</b><sub>±0.02</sub></td><td><b>2.42</b></td><td><b>4.37</b><sub>±0.03</sub></td><td><b>6.5</b><sub>±0.06</sub></td><td><b>5.39</b><sub>±0.15</sub></td><td><b>2.37</b></td></tr><tr><td rowspan="5">FSL-ROI Desikan</td><td>Linear(Original)</td><td>5.27</td><td>67.6</td><td>6.58</td><td>5.85</td><td>7.97</td><td>7.06</td><td>13.5</td><td>6.09</td></tr><tr><td>MLP(128)</td><td>4.80<sub>±0.06</sub></td><td>19.8<sub>±2.3</sub></td><td>14.7<sub>±1.16</sub></td><td>9.04</td><td>6.01<sub>±0.02</sub></td><td>17.3<sub>±1.91</sub></td><td>25.0<sub>±3.0</sub></td><td>14.77</td></tr><tr><td>MLP(128, 128)</td><td>3.46<sub>±0.02</sub></td><td>32.6<sub>±1.43</sub></td><td>5.96<sub>±0.15</sub></td><td>4.26</td><td>5.21<sub>±0.11</sub></td><td>21.7<sub>±2.44</sub></td><td>7.00<sub>±0.44</sub></td><td>4.43</td></tr><tr><td>MLP(128, 128, 128)</td><td>3.51<sub>±0.01</sub></td><td>22.7<sub>±1.31</sub></td><td><b>4.95</b><sub>±0.01</sub></td><td>3.17</td><td><b>4.58</b><sub>±0.02</sub></td><td>9.2<sub>±0.7</sub></td><td><b>5.19</b><sub>±0.03</sub></td><td><b>2.54</b></td></tr><tr><td>MLP(128, 128, 128, 128)</td><td><b>3.37</b><sub>±0.003</sub></td><td><b>11.2</b><sub>±0.5</sub></td><td>5.27<sub>±0.006</sub></td><td><b>2.73</b></td><td>5.23<sub>±0.006</sub></td><td><b>5.9</b><sub>±0.16</sub></td><td>6.02<sub>±0.13</sub></td><td>2.57</td></tr></tbody></table>

on VBM and Quasi-Raw respectively). This would explain the drop in performance on the external test (+4.58 MAE on VBM). In this regard, ResNet is the best trade-off (with also the best ranking for the challenge) as it is robust to site and it generalizes well on the external test. These results also suggest that having a deeper network (e.g., DenseNet and 121 layers) does not translate necessarily in better generalization performance (e.g., compared to ResNet with 18 layers), in line with Dufumier et al. (2021a).

Interestingly, ComBat harmonization does remove most of site bias in CNN representation space (with site prediction Bacc decreased by 2% but not still matching the one obtained using only age as input, see Table 3). However, it also heavily degrades CNN performance on age prediction for all testing sets (in particular for DenseNet). ComBat is not fitted for Quasi-Raw data as it mainly relies on voxel-wise statistics, and raw data are not properly registrated voxel-wise across images. Consequently, we did not evaluate this approach on Quasi-Raw images.

For completeness, we finally evaluated several vanilla Multi-Layer Perceptrons (MLP) with varying depth on brain age prediction using surface-based mesh (Xhemi) and ROI features on 2 atlases, namely Destrieux and Desikan (see Section 2.2.3 for more details). As opposed to VBM, SBM also includes geometrical properties of brain sulci and gyri (e.g local curvature). Thus, it conveys complementary information that may have been lost in volume-based VBM data.

For comparison purposes, we also added the performance of a linear model trained directly on input data. We emphasize this model is not authorized in the official challenge (since we expect a model that outputs a low-dimensional representation of the input data). Nevertheless, it gives a general baseline for all submitted models. This linear model is trained on the whole training + validation set with a cross-validation procedure details in Section 3.5.1. We did not perform a 3-fold CV for training the linear probe (corresponding here directly to the linear model) so we do not report a standard deviation.

In Table 5, we observe that a finer atlas with 148 regions (Destrieux) leads to better brain age estimation as opposed to coarse atlas (Desikan with only 68 regions), but it also preserves more site information. Furthermore, the deeper the MLP is, the better in terms of MAE for both atlases. Overall, whole-brain approach seems better suited as it enables CNN to extract fine-grained information for brain age, making more accurate predictions while removing more non-biological site information. Nevertheless, merging VBM with SBM data may result in a better representation as both modalities can bring complementary information to model brain development.

These results further justify the necessity of the OpenBHB Challenge since current standard neuroimaging debiasing methods all have their limitations and it can provide an innovative way to develop and benchmark new ML algorithms on brain age prediction, multi-site harmonization and debiasing.

# 4. Future works with OpenBHB

## 4.1. Towards transfer learning for computer-aided diagnosis

This challenge is a first step towards building new algorithms for phenotype prediction robust across sites. However, as recently proven Dufumier et al. (2021b), it is also possible to leverage such multi-center large-scale dataset to significantly improve classification performance on other hard computer-aided diagnosis (CAD) tasks such as Alzheimer's detection or schizophrenia diagnosis, with Transfer Learning (TL). First, pre-training algorithms to remove site effect is critical for computer-aided diagnosis since most clinical datasets with moderate size ($N > 100$) are multi-site (e.g ADNI Jack Jr et al. (2008), ABIDE, SCHIZCONNECT, etc.) and it is known Koppe et al. (2021); Wachinger et al. (2021) that acquisition site can heavily bias ML models. Second, self-supervised algorithms -a class of unsupervised methods- are attracting more and more attention from the medical imaging community Chen et al. (2019); Dufumier et al. (2021b); Taleb et al. (2020); Tao et al. (2020); Zhou et al. (2021); Zhuang et al. (2019) since they are able to leverage large un-annotated dataset to improve performance on several downstream tasks with TL. As a result, OpenBHB provides a way to benchmark algorithms on TL, and potentially catalyze research to find new innovative self-supervised algorithms on brain MRI for CAD.

## 4.2. Towards multi-modal integration for new bio-markers discovery

In the context of brain age prediction as a tool for CAD, sMRI data provide meaningful information about subtle anatomical modifications in cortical and sub-cortical structures (e.g atrophies or hypertrophies). On the other hand, resting-state fMRI and Diffusion Weighted Images (DWI) give hints about functional and structural brain connectivity, that can be altered for patients with psychiatric disorders (e.g autism spectrum disorder, schizophrenia or bipolar disorder Baker et al. (2014)).
However, little is known about the predictive power of a multi-modal approach combining both sMRI, DWI and resting-state fMRI for brain age modelling or CAD. A future release of the OpenBHB dataset might bring some answers. Almost all datasets included in the current OpenBHB release - IXI and NAR except - have rfMRI data available for all participants and only a few of them (IXI, NPC, RBP) also have DWI (see Table 1). Consequently, we envision to integrate first the rfMRI data as an additional modality in OpenBHB in order to encourage the development of new ML models for multi-modal integration.

## 5. Author contributions statement

B.D. and C.A. conceived the experiments, B.D. conducted the experiments. A.G. and B.D. implemented the challenge. E.D. supervised the project. V.F., C.A. and A.G. provided critical feedbacks. J.V. and A.G. pre-processed all data. All authors reviewed the manuscript.

## Acknowledgments

We are exceedingly grateful to Alexandre Gramfort, François Caud, Guillaume Lemaître, Thomas Moreau (Université Paris-Saclay, Inria, CEA) and Roman Yurchak (Symerio SME, Paris area) to provide access and active support to the challenge platform: https://ramp.studio. This work is supported by Big2Small, a teaching and research chair in AI of French National Research Agency and R-LiNK and Horizon 2020 EU grant (H2020-SC1-2017, 754907).

## References

Abrol, A., Fu, Z., Salman, M., Silva, R., Du, Y., Plis, S., Calhoun, V., 2021. Deep learning encodes robust discriminative neuroimaging representations to outperform standard machine learning. Nat. Commun. 12 (1), 1–17.

Alain, G., Bengio, Y., 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644.

Avants, B.B., Tustison, N., Song, G., 2009. Advanced normalization tools (ants). Insight J. 2 (365), 1–35.

Babayana, A., Erbey, M., Kumral, D., Reinelt, J.D., Reiter, A.M., Röbbig, J., Schaare, H.L., Uhlig, M., Anwander, A., Bazin, P.-L., et al., 2019. A mind-brain-body dataset of mri, eeg, cognition, emotion, and peripheral physiology in young and old adults. Sci. Data 6 (1), 1–21.

Bahng, H., Chun, S., Yun, S., Choo, J., Oh, S.J., 2020. Learning de-biased representations with biased representations. In: International Conference on Machine Learning. PMLR, pp. 528–539.

Baker, J.T., Holmes, A.J., Masters, G.A., Yeo, B.T., Krienen, F., Buckner, R.L., Öngür, D., 2014. Disruption of cortical association networks in schizophrenia and psychotic bipolar disorder. JAMA Psychiatry 71 (2), 109–118.

Barbano, C.A., Tartaglione, E., Grangetto, M., 2021. Bridging the gap between debiasing and privacy for deep learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3806–3815.

Bashyam, V. M., Doshi, J., Erus, G., Srinivasan, D., Abdulkadir, A., Habes, M., Fan, Y., Masters, C. L., Maruff, P., Zhuo, C., Völzke, H., Johnson, S. C., Fripp, J., Koutsouleris, N., Satterthwaite, T. D., Wolf, D. H., Gur, R. E., Gur, R. C., Morris, J. C., Albert, M. S., Grabe, H. J., Resnick, S. M., Bryan, R. N., Wolk, D. A., Shou, H., Nasrallah, I. M., Davatzikos, C., 2020a. Medical image harmonization using deep learning based canonical mapping: Toward robust and generalizable learning in imaging. 2010.05355.

Bashyam, V.M., Erus, G., Doshi, J., Habes, M., Nasrallah, I., Truelove-Hill, M., Srinivasan, D., Mamourian, L., Pomponio, R., Fan, Y., et al., 2020. Mri signatures of brain age and disease over the lifespan based on a deep brain network and 14 468 individuals worldwide. Brain 143 (7), 2312–2324.

Buckner, R. L., Roffman, J. L., Smoller, J. W., 2014. Brain Genomics Superstruct Project (GSP). doi:10.7910/DVN/25833.

Cadene, R., Dancette, C., Cord, M., Parikh, D., et al., 2019. Rubi: reducing unimodal biases for visual question answering. Adv. Neural Inf. Process. Syst. 32, 841–852.

Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A., 2020. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, Vol. 33.

Castillo-Navarro, J., Le Saux, B., Boulch, A., Audebert, N., Lefèvre, S., 2021. Semi-supervised semantic segmentation in earth observation: the minifrance suite, dataset analysis and multi-task network study. Mach. Learn. 1–36.

Chen, L., Bentley, P., Mori, K., Misawa, K., Fujiwara, M., Rueckert, D., 2019. Self-supervised learning for medical image analysis using image context restoration. Med. Image Anal. 58, 101539.

Chen, T., Kornblith, S., Norouzi, M., Hinton, G., 2020. A simple framework for contrastive learning of visual representations. In: International conference on machine learning. PMLR, pp. 1597–1607.

Clark, C., Yatskar, M., Zettlemoyer, L., 2019. Don't take the easy way out: ensemble based methods for avoiding known dataset biases. arXiv preprint arXiv:1909.03683.

Cole, J.H., Franke, K., 2017. Predicting age using neuroimaging: innovative brain ageing biomarkers. Trends Neurosci. 40 (12), 681–690.

Cole, J.H., Leech, R., Sharp, D.J., Initiative, A.D.N., 2015. Prediction of brain age suggests accelerated atrophy after traumatic brain injury. Ann. Neurol. 77 (4), 571–581.

Cole, J.H., Poudel, R.P., Tsigkrasoulis, D., Caan, M.W., Steves, C., Spector, T.D., Montana, G., 2017. Predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker. Neuroimage 163, 115–124.

Cole, J.H., Ritchie, S.J., Bastin, M.E., Hernández, M.V., Maniega, S.M., Royle, N., Corley, J., Pattie, A., Harris, S.E., Zhang, Q., et al., 2018. Brain age predicts mortality. Mol. Psychiatry 23 (5), 1385–1392.

Desikan, R.S., Ségonne, F., Fischl, B., Quinn, B.T., Dickerson, B.C., Blacker, D., Buckner, R.L., Dale, A.M., Maguire, R.P., Hyman, B.T., et al., 2006. An automated labeling system for subdividing the human cerebral cortex on mri scans into gyral based regions of interest. Neuroimage 31 (3), 968–980.

Dewey, B.E., Zuo, L., Carass, A., He, Y., Liu, Y., Mowry, E.M., Newsome, S., Oh, J., Calabresi, P.A., Prince, J.L., 2020. A disentangled latent space for cross-site mri harmonization. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. Springer International Publishing, Cham, pp. 720–729. doi:10.1007/978-3-030-59728-3-70.

Dinsdale, N.K., Jenkinson, M., Namburete, A.I., 2021. Deep learning-based unlearning of dataset bias for mri harmonisation and confound removal. Neuroimage 228, 117689.

Dufumier, B., Gori, P., Battaglia, I., Victor, J., Grigis, A., Duchesnay, E., 2021. Benchmarking cnn on 3d anatomical brain mri: architectures, data augmentation and deep ensemble learning. arXiv preprint arXiv:2106.01132.

Dufumier, B., Gori, P., Victor, J., Grigis, A., Wessa, M., Brambilla, P., Favre, P., Polosan, M., McDonald, C., Piguet, C.M., et al., 2021. Contrastive learning with continuous proxy meta-data for 3d mri classification. International conference on medical image computing and computer-assisted intervention. Springer.

Fisch, L., Leenings, R., Winter, N.R., Dannlowski, U., Gaser, C., Cole, J.H., Hahn, T., 2021. Predicting chronological age from structural neuroimaging: the predictive analytics competition 2019. Front. Psychiatry 12.

Fisch, B., Sereno, M.I., Dale, A.M., 1999. Cortical surface-based analysis: ii: inflation, flattening, and a surface-based coordinate system. Neuroimage 9 (2), 195–207.

Fisch, B., Van Der Kouwe, A., Destrieux, C., Halgren, E., Ségonne, F., Salat, D.H., Busa, E., Seidman, L.J., Goldstein, J., Kennedy, D., et al., 2004. Automatically parcellating the human cerebral cortex. Cerebral Cortex 14 (1), 11–22.

Follmer, D.J., Fang, S.-Y., Clariana, R.B., Meyer, B.J., Li, P., 2018. What predicts adult readers' understanding of stem texts? Read. Writ. 31 (1), 185–214.

Fortin, J.-P., Cullen, N., Sheline, Y.I., Taylor, W.D., Aselcioglu, I., Cook, P.A., Adams, P., Cooper, C., Fava, M., McGrath, P.J., et al., 2018. Harmonization of cortical thickness measurements across scanners and sites. Neuroimage 167, 104–120.

Garcia-Dias, R., Scarpazza, C., Baecker, L., Vieira, S., Pinaya, W.H., Corvin, A., Redolfi, A., Nelson, B., Crespo-Facorro, B., McDonald, C., Tordesillas-Gutiérrez, D., Cannon, D., Mothersill, D., Hernaúz, D., Morris, D., Setien-Suero, E., Donohoe, G., Frisoni, G., Tronchin, G., Sato, J., Marcelis, M., Kempton, M., van Haren, N.E., Gruber, O., McGorry, P., Amminger, P., McGuire, P., Gong, Q., Kahn, R.S., Ayesa-Arriola, R., van Amelsvoort, T., Ortiz-García de la Foz, V., Calhoun, V., Cahn, W., Mechelli, A., 2020. Neuroharmony: a new tool for harmonizing volumetric mri data from unseen scanners. Neuroimage 220, 117127. doi:10.1016/j.neuroimage.2020.117127.

Gaser, C., Dahnke, R., 2016. Cat-a computational anatomy toolbox for the analysis of structural mri data. HBM 2016, 336–348.

Gaser, C., Franke, K., Klöppel, S., Koutsouleris, N., Sauer, H., Initiative, A.D.N., 2013. Brainage in mild cognitive impaired patients: predicting the conversion to alzheimer's disease. PLoS ONE 8 (6), e67346.

Glocker, B., Robinson, R., Castro, D.C., Dou, Q., Konukoglu, E., 2019. Machine learning with multi-site imaging data: an empirical study on the impact of scanner effects. arXiv preprint arXiv:1910.04597.

Greve, D.N., Van der Haegen, L., Cai, Q., Stufflebeam, S., Sabuncu, M.R., Fischl, B., Brysbaert, M., 2013. A surface-based analysis of language lateralization and cortical asymmetry. J. Cogn. Neurosci. 25 (9), 1477–1492.

Han, L.K., Dinga, R., Hahn, T., Ching, C.R., Eyler, L.T., Aftanas, L., Aghajani, M., Aleman, A., Baune, B.T., Berger, K., et al., 2020. Brain aging in major depressive disorder: results from the enigma major depressive disorder working group. Mol. Psychiatry 1–16.

He, K., Fan, H., Wu, Y., Xie, S., Girshick, R., 2020. Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738.

He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.

He, S., Grant, P.E., Ou, Y., 2021. Global-local transformer for brain age estimation. IEEE Trans. Med. Imaging.

He, S., Pereira, D., Perez, J.D., Gollub, R.L., Murphy, S.N., Prabhu, S., Pienaar, R., Robertson, R.L., Grant, P.E., Ou, Y., 2021. Multi-channel attention-fusion neural network for brain age estimation: accuracy, generality, and interpretation with 16,705 healthy mris across lifespan. Med. Image Anal. 72, 102091.

Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q., 2017. Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700–4708.

Jack Jr, C.R., Bernstein, M.A., Fox, N.C., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britton, P.J., L. Whitwell, J., Ward, C., et al., 2008. The alzheimer's disease neuroimaging initiative (adni): mri methods. J. Magn. Resonance Imaging 27 (4), 685–691.

Jenkinson, M., Smith, S., 2001. A global optimisation method for robust affine registration of brain images. Med. Image Anal. 5 (2), 143–156.
Jonsson, B.A., Bjornsdottir, G., Thorgeirsson, T.E., Ellingsen, L.M., Walters, G.B., Gudbjartsson, D.F., Stefansson, H., Stefansson, K., Ulfarsson, M.O., 2019. Brain age prediction using deep learning uncovers associated sequence variants. Nat. Commun. 10 (1). doi:10.1038/s41467-019-13163-9.

Kaufmann, T., van der Meer, D., Doan, N.T., Schwarz, E., Lund, M.J., Agartz, I., Alnæs, D., Barch, D.M., Baur-Streubel, R., Bertolino, A., et al., 2019. Common brain disorders are associated with heritable patterns of apparent aging of the brain. Nat. Neurosci. 22 (10), 1617–1623.

Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D., 2020. Supervised contrastive learning. Adv. Neural Inf. Process. Syst. 33.

Kia, S. M., Huijsdens, H., Dinga, R., Wolfers, T., Mennes, M., Andreassen, O. A., Westlye, L. T., Beckmann, C. F., Marquand, A. F., 2020. Hierarchical bayesian regression for multi-site normative modeling of neuroimaging data. 2005.12055.

Kingma, D.P., Ba, J., 2015. Adam: a method for stochastic optimization. ICLR.

Koppe, G., Meyer-Lindenberg, A., Durstewitz, D., 2021. Deep learning for small and big data in psychiatry. Neuropsychopharmacology 46 (1), 176–190.

Koutsouleris, N., Davatzikos, C., Borgwardt, S., Gaser, C., Bottlender, R., Frodl, T., Falkai, P., Riecher-Rössler, A., Möller, H.-J., Reiser, M., et al., 2014. Accelerated brain aging in schizophrenia and beyond: a neuroanatomical marker of psychiatric disorders. Schizophr. Bull. 40 (5), 1140–1153.

Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp. 1097–1105.

Li, P., Clariana, R.B., 2019. Reading comprehension in l1 and l2: an integrative approach. J. Neurolinguistic. 50, 94–105.

Liu, M., Maiti, P., Thomopoulos, S., Zhu, A., Chai, Y., Kim, H., Jahanshad, N., 2021. Style transfer using generative adversarial networks for multi-site mri harmonization. In: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. Springer International Publishing, Cham, pp. 313–322.

Liu, S., Yap, P.-T., 2021. Learning multi-site harmonization of magnetic resonance images without traveling human phantoms. 2110.00041.

Nastase, S.A., Liu, Y.-F., Hillman, H., Zadbood, A., Hasenfratz, L., Keshavarzian, N., Chen, J., Honey, C.J., Yeshurun, Y., Regev, M., et al., 2021. Narratives: fmri data for evaluating models of naturalistic language comprehension. bioRxiv. 2020.12

Orfanos, D.P., Michel, V., Schwartz, Y., Pinel, P., Moreno, A., Le Bihan, D., Frouin, V., 2017. The brainomics/localizer database. NeuroImage 144, 309–314.

Østby, Y., Tarnnes, C.K., Fjell, A.M., Westlye, L.T., Due-Tønnessen, P., Walhovd, K.B., 2009. Heterogeneity in subcortical brain development: a structural magnetic resonance imaging study of brain maturation from 8 to 30 years. J. Neurosci. 29 (38), 11772–11782.

Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al., 2011. Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12, 2825–2830.

Peng, H., Gong, W., Beckmann, C.F., Vedaldi, A., Smith, S.M., 2021. Accurate brain age prediction with lightweight deep neural networks. Med. Image Anal. 68, 101871.

Robinson, R., Dou, Q., Coelho de Castro, D., Kamnitsas, K., de Groot, M., Summers, R.M., Rueckert, D., Glocker, B., 2020. Image-level harmonization of multi-site data using image-and-spatial transformer networks. Lect. Notes Comput. Sci. 710–719. doi:10.1007/978-3-030-59728-3_69.

Rosen, A.F., Roalf, D.R., Ruparel, K., Blake, J., Seelaus, K., Villa, L.P., Ciric, R., Cook, P.A., Davatzikos, C., Elliott, M.A., Garcia de La Garza, A., Gennatas, E.D., Quarmley, M., Schmitt, J.E., Shinohara, R.T., Tisdall, M.D., Craddock, R.C., Gur, R.E., Gur, R.C., Satterthwaite, T.D., 2018. Quantitative assessment of structural image quality. Neuroimage 169, 407–418. doi:10.1016/j.neuroimage.2017.12.059.

Schnack, H.G., Van Haren, N.E., Nieuwenhuis, M., Hulshoff Pol, H.E., Cahn, W., Kahn, R.S. 2016. Accelerated brain aging in schizophrenia: a longitudinal pattern recognition study. Am. J. Psychiatry 173 (6), 607–616.

Schulz, M.-A., Yeo, B.T., Vogelstein, J.T., Mourao-Miranada, J., Kather, J.N., Kording, K., Richards, B., Bzdok, D., 2020. Different scaling of linear models and deep learning in ukbiobank brain images versus machine-learning datasets. Nat. Commun. 11 (1), 1–15.

Sechidis, K., Tsoumakas, G., Vlahavas, I., 2011. On the stratification of multi-label data. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, pp. 145–158.

Sunavsky, A., Poppenk, J., 2020. Neuroimaging predictors of creativity in healthy adults. Neuroimage 206, 116292.

Taleb, A., Loetzsch, W., Danz, N., Severin, J., Gaertner, T., Bergner, B., Lippert, C., 2020. 3d self-supervised methods for medical imaging. In: Advances in Neural Information Processing Systems, Vol. 33, pp. 18158–18172.

Tao, X., Li, Y., Zhou, W., Ma, K., Zheng, Y., 2020. Revisiting rubik's cube: self-supervised learning with volume-wise transformation for 3d medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 238–248.

Tartaglione, E., Barbano, C.A., Grangetto, M., 2021. End: entangling and disentangling deep representations for bias correction. arXiv preprint arXiv:2103.02023.

Torralba, A., Efros, A.A., 2011. Unbiased look at dataset bias. In: CVPR 2011. IEEE, pp. 1521–1528.

Wachinger, C., Rieckmann, A., Pölsterl, S., Initiative, A.D.N., et al., 2021. Detect and correct bias in multi-site neuroimaging datasets. Med. Image Anal. 67, 101879.

Wang, M., Deng, W., 2018. Deep visual domain adaptation: a survey. Neurocomputing 312, 135–153.

Zhou, Z., Sodha, V., Pang, J., Gotway, M.B., Liang, J., 2021. Models genesis. Med. Image Anal. 67, 101840.

Zhuang, X., Li, Y., Hu, Y., Ma, K., Yang, Y., Zheng, Y., 2019. Self-supervised feature learning for 3d medical images by playing a rubik's cube. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, pp. 420–428.

Zuo, X.-N., Anderson, J.S., Bellec, P., Birn, R.M., Biswal, B.B., Blautzik, J., Breitner, J.C., Buckner, R.L., Calhoun, V.D., Castellanos, F.X., et al., 2014. An open science resource for establishing reliability and reproducibility in functional connectomics. Sci. Data 1 (1), 1–13.