Robust Alzheimer's Progression Modeling using Cross-Domain Self-Supervised Deep Learning
Authors: Saba Dadsetan
Mohsen Hejrati
Shandong Wu
Somaye Hashemifar
Venue: Accepted by TMLR ()

Abstract: Developing successful artificial intelligence systems in practice depends on both robust deep learning models and large, high-quality data. However, acquiring and labeling data can be prohibitively expensive and time-consuming in many real-world applications, such as clinical disease models. Self-supervised learning has demonstrated great potential in increasing model accuracy and robustness in small data regimes. In addition, many clinical imaging and disease modeling applications rely heavily on regression of continuous quantities. However, the applicability of self-supervised learning for these medical-imaging regression tasks has not been extensively studied. In this study, we develop a cross-domain self-supervised learning approach for disease prognostic modeling as a regression problem using medical images as input. We demonstrate that self-supervised pretraining can improve the prediction of Alzheimer's Disease progression from brain MRI. We also show that pretraining on extended (but not labeled) brain MRI data outperforms pretraining on natural images. We further observe that the highest performance is achieved when both natural images and extended brain-MRI data are used for pretraining.

Published in Transactions on Machine Learning Research (07/2023)
Robust Alzheimer's Progression Modeling using Cross- Domain Self-Supervised Deep Learning
Saba Dadsetan Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA Department of Artificial Intellignnce, Genentech Inc., South San Froncisco, CA, USA
dadsetas@ggene.com
Mohsen Hejrati
hejratis@gene.ccm com
Department Artificial Intelligence, Cenentech Inc., South San Prancisco, CA,, USA
Shandong Wu
shw83@ypitt.cmm edu
Intelligent Systems Program, University of Piltsburgh, Piltsburgh, PA, USA
Somayc Hashemifar
Department of Artificial Intelligence, Genendech Inc., South San Prancisco, CA,, USA
for the Swedish BioFINDER study group"
The Alzheimer's Disease Neuroimaging Initiative
for the BIOCARD Study team?
Reviewed OpenReview:
Abstract
Developing successful artificial intelligence systems in practice depends on both robust deep learning models and large, high-quality data. However, acquiring and labeling data be prohibitively expensive and time-consuming in many real-world applications, such as clini- cal disease models. Self-supervised learning has demonstrated great potential in increasing model accuracy and robustness in small data regimes. In addition, many clinical imaging and disease modeling applications rely heavily on regression of continuous quantities. How- ever, the applicability self-supervised learning for these medical-ine lanlliin-i regression tasks has been extensively studied. In this study, we develop cross- coos-domain self-supervised learning approach for discase prognostic modeling images input. We demonstrate that self-supervised pretraining can improve the predic- tion of Alzheimer's Disease progression from brain MRI. We also show that pretraining extended (but not labeled) brain MRI data outperforms pretraining on natural images. We further observe that the highest performance is achieved when both natural images and extended brain-MRI data are used for pretraining.
regression problem using modical
"A complete list of the BioFINDER study group members can be found at www. www.www................ www.ffiiee.ee........
Data used in preparation of this article were obtained from the Alzheimer's Disease Neurolmaging Initiative (ADNT) database fadii. lon..us..eell.. As such, the investigators within the ADNT contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis writing of this report. A complete listing of ADNI investigators can be found at http://adni. loni, use. edw/ wwwpontenttupllooooo hhww http://AAA_.ccooooeeeeeeeeeeeeee Acknooledgement, Lis.t.tttppp pdf
+Data used preparation of this article National Institute Aging. The BIOCARD study team did participate the analysis writing of this report, however, they contributed the design and implementation the study. listing BIOCARD investigators can be found the RIOCARD website (on 'BIOCARD Data. Access Procedures' page, 'Acknowledggement Agreement' document).
derived from BIOCARD study, supported grant U19 AC033655 from the

---

Published in Transactions on Machine Learning Research (07/ (07/202) 2023)
1 Introduction
Developing reliable and robust artificial intelligence systems requires elficient deep learning techniques, well as the creation and annotation of large volumes of data for training. However, the construction of a labeled datasct is often timo-consuming and expensive, such as in the medical imaging domain, duc to the complexity of annotation tasks and the high expertise required for the manual interpretation. To alleviate the lack of annotations in medical imaging, transfer learning from natural images is becoming increasingly popular (Liu et al.) 2020b] McKinney et al.] 2020: Menegola et al.) 2017 Xie et al. 2019).
Although numerous experimental studies indicate the effectiveness of fine-tuning from either supervised
self=supervised ImageNet models( (Aluubaidi et. al. 2020 Graziani et al.
2019 Heker & GGeenspan 2020:
Zhou et al. 2021; Hosseinzadeh Taher et al.) 2021 Azizi et al. 2021 it does not always improve the performance due to domain mismatch problem [Rghu et al. 2019).
In the meantime, self-supervised learning (SSL.) has demonstrated great success in many downstream appli- cations in computer vision, where the labeling process quite expensive and timo-consuming (Doersch ct al.
2015 Gidaris et al. 2018 Noroozi & Favaro 2016 Zhang et al.) 2016: Ye al. 2019 Bachman et al.) 2019:
Tian et al.) 2020a. Henaff 2020 Oord et al. 2018). Medical research and healthcare are especially well-
poised to benefit from SSL learning approaches, given the prevalence of unprecedented amounts of medical images generated by hospital and non-hospital settings. Despite this demand, the use of SSL approaches in the medical image domain has received limited attention. Only a few studes have investigated the impact of SSL in the medical image analysis domain for limited applications including classification (Liu et al.
2019 Sowrirajan et al.] 2021 He al. 2020b Azizi ct al.] [2022] Zhu ct al. 2020 Liu ct al. 2020a) and segmentation (Ronneberger et al.] 2015
Bai el. al. 2019 Chaitanya et. al., 2020 Spilzer et al., 2018
In this study, we focus developing a self-supervised deep learning model for predicting the progression of Alzheimer's discase (AD) by using high dimensional magnetic resonance imaging (MRI). AD is slowly progressing disease caused by the degeneration of brain cells, with patients showing clinical symptoms years after the onset of the discase. Therefore, accurate prognosis and treatment of AD in its early stage is critical to prevent non-reversible and fatal brain damage. By accurately predicting the progression of AD, clinicians can start treatment earlier and provide more personalized care.
Many existing methods for predicting AD progrossion have focused classifying patients into coarse cat- egories, such as Mild Cognitive Impairment (MCI) or dementia, as well as predicting conversion from one category to another (e.g. MCI to dementia) (Risacher et al. 2009 Venugopalan et al.] 2021 Oh et al. 2019). However, applications in clinical trials require a more fine- aueeggrined measurement scale, because clin- ical trial populations are typically narrowly defined (cg. only MCT). An alternative approach is to predict the outcome of cognitive and functional tests, which are represented by continuous numerical values. By framing the prediction task as a regression rather than a classification, prognostic models offer more granular estimates of discase progression. Recently, a few deep lcarning based approaches, including the recurrent neural network (RNN) and convolutional neural networks (CNN) have been proposed for predicting disease progression of AD patients based on MRTs. Nguyen et al.) (2020) adapted MinimalRNN to integrate longitu- dinal clinical information and cross-sectional tabular imaging features for regressing endpoints. EI-Sappaghi ct al. (2020) utilized an ensemble model based on stacked CNN and a bidirectional long short-term memory (BiL.STM) to predict the endpoints on the fusion of time series clinical features and derived imaging features. Recently, several mothods have started to employ CNN based models to extract features from raw medical imaging. Tian al. (2022) applied CNN with a multi-task interaction layers composed of feature decoupling modules and feature interaction module to predict the disease progression.
Despite demand, little progress has been made because of the difficult design requirements, lack of large-scale, homogeneous datasets that contain early stage AD patients, and noisy endpoints that are potentially hard to predict. Much of the prior work has focused on using image-derived features to overcome the complexity and high variability raw MRTs, and small datasets. Most current prognosis models are trained on a single dataset (i.c. cohort), which limits their generalizability to other cohorts. They also use a limited number of annotated images, which can lead to problems such as domain shift, and heterogeneity.
2

---

Published in Transactions on Machine Learning Research (07/2023)
We established a cross-domain self-supervised transfer learning approach that learns transferable and gen- eralizable representations for medical images. Our approach leverages SSL on both unlabeled large-scale natural images and an in-domain medical image dataset comprised from 11 different internal and external studics. These representations can be further fine-tuned for downstream tasks such as discase progression prediction, with using limited labeled data from the clinical setting. We evaluated the performance of different supervised and self-supervised models pretrained on either natural images or medical images, or both. Our extensive experiments reveal that (1) Self-supervised pretraining on natural images followed by self-supervised learning on unlabeled medical images outperforms alternative transfer learning methods, indi- cating the potential of SSL in reducing the reliance on data annotation compared to supervised approaches (2) Self-supervised models pretrained on medical images outperform those pretrained on natural images, denoting that SSL on medical images yield discriminative feature represenlations for regression task.
2 Related works
The recent development and success of self-supervised learning techniques, including contrastive learning (Wu
2020 Misra & Maaten 2020 mutual et al.] 2018; He et al.) 2020a Chen et al.] 2020c bla. Grill et al.
information reduction (Tan et al. 2020b), clustering CCaron et al. 2020 Li et al. 2020), and redundancy-
reduction methods (Zbontar et al. 2021 Bardes et al. 2021] in computer vision indicate their effectiveness in improving the performance of AT systems. These methods train models on different, pretext tasks to enable the network to learn high-quality representations without label information. SimCLR (Chen et al. 2020cb maximizes agreement between representations of different augmentations of the same image by using a contrastive loss in the latent space. Barlow Twins (Zbontar et al., 2021) measure the corss-corneattion matrix between the embedding of two identical networks and its goal is to make this cross-correlation close to the identity matrix. SwAV (Caron et al. 2020) simultaneously clusters the images while enforcing consistency between cluster assignments produced for differently augmented views of the same image, instead of comparing features directly as in contrastive learning.
Subsequently, self-supervised learning has been employed for medical imaging applications including classifi- cation and segmentation to learn visual representations of medical images by incorporating unlabeled medical images. While some approaches have designed domain-specifie pretext tasks (Bai et al. 2019 Spitzer et al,, 2018 Zhuang et al. 2019 Zhu et al. 2020), others have adjusted well-known self-supervised learning meth- ods to medical data He et al. 2020b Li et al. 2021 Zhou el. al.] 2020 Sowrirajan et al. 2021). Very
recently Azziz et al. 2022) has applied SimCLR on a combination of unlabeled ImageNet dataset and task specific medical images for medical image classification; their experiments and improved performance suggest that pretraining on ImageNet is complementary to pretraining on unlabeled medical images.
Although aforementioned approaches demonstrate improvement of the performance on challenging medical datasets, all of them are limited to classification and segmentation tasks and their benefits and potential effects for the prognosis prediction tasks, regression tasks, have not been studied. Formulating progress prediction a regression rather than a traditional classification problem leads to more fine-grained mea- surement scale which is crucial for real-world applications. Therefore, the development of self-supervised networks is in great demand for efficient dala-utilization in medical imaging for disease prognosis. To the best of our knowledge, this is the first study of developing a supervied deep convolution neural network on modical data images from various cross-domain datascts to predict granular understanding of discase progression.
3 Methodology
3.1 Task and Data
In Alzheimer's Disease clinical trials, the most important cognilive test that is performed to assess current patient function and the likelihood of AD progression is Clinical Dementia Rating Scale Sum of Boxes (CDR- SB). CDR-SB is a score provided by clinicians based on clinical evaluations and its ranges from to 18, with higher scores indicating greater severity of symptoms. CDR-SB score is then used to assign Alzheimer's status of patient.
3

---

Published in Transactions on Machine Learning Research (07/2023)
Our goal is to use a regression approach to predict the future status of patients with AD based on their initial visit. Specifically, our model takes as input 2D slice stacks of an MRI volume that are collected at the first visit and predicts the CDR-SB value at month 12 (i.e. after one year). By accurately predicting the progression of AD in patients, clinicians can initiate treatment at an carlier stage and tailor the most suitable and effective treatment. for each individual patient.
All individuals included in our analysis are around 10k from eight external studies, including ADNI Petersen et al. 2010 BIOFINDER Mlattsson-Carlgeee et al., 2020), FACEHBI Saoreno-Gran et al. 2018), AIBL (El-
lis et al. 22009) HABS (Dgleyy et al. 2017). BIOCARD [Mohhekar et al 2013), WRAP (Langhough Koscik
et al. 2021), and OASTS-3 (LaMontagne et al. 2019 and five internal studies including Scarlet RoAD
(NCT01224106), Marguerite RoAD (NCT02051608), CREAD 1 & 2 (NCT02670083, NCT03114657), BLAZE (NCT01397578), and ABBY (NCT0L343966).
MR volumes were standardized using the following preprocessing steps. First, a brain mask was inferred for each volume using SyuthSeg (Billot et al.) 202T), a deep learning segmentation package. During training, the volumes and segmentations were resampled isotropically to 1 mm voxel size, standardized to canonical (RAS+) orientation, intensity rescaled to 0,1 and Z-score normalized. Finally, volumes are cropped or padded to (224,224).
For training self-supervised models, we offer two training sets of medical images called the center-slice and 5-slice datasets. The 5-slice dataset is generated by extracting a stack of five slices from the 3D MRI volumes. This stack comprises the center slice of the brain sub-volumes and four adjacent slices, with intervals of 5 (two to the right, two to the left). In contrast, the center-slice dataset only includes the center slice. Both training sets include subjects from all datasets except OASIS-3 and datasct ABBY, which are reserved out-study lest sets (see Table D. The 5-slice dataset contains a total of 218, 700 unlabeled images, while the center-slice dataset comprises 43, 740 unlabeled images. Around 90% of the development set is used for training, while the remaining 10% is set aside for validation. We select the best model based on the minimum self-supervised validation loss and transfer its backbone weights to the supervised model.
To create a labeled datasets for fine tuning, the participants are selected from five studies, including ADNI, Scarlet RoAD, Marguerite RoAD, CREAD & 2, and BLAZE. All studies were filtered to include patients who are positive for amyloid pathology, have MMSE larger than 20 and are diagnosed with early stages of AD i.e. prodromal or mild at baseline. Amyloid positivity is detected by either cerebrospinal fluid (CSF) or amyloid positron emission tomography (PET). The fine-tuning dataset consists of approximately 1000 images, none of which are used for self-supervised learning training. Roughly 30% of the fine-tuning dataset is reserved as an in-study test set, while the remaining data is divided into training and validation sets (see Table 1
We have three test sets: in-study test set and two out-study test sets. The in-study test set consists of one-third of the patients from ADNI, Scarlet RoAD, Marguerite RoAD, CREAD 1 & 2, and BLAZE. The two out-study test sets are OASIS-3 and study ABBY, that are not utilized for either self-supervised pretraining or fine-tuning, The primary purpose of these out-study test sets is to assess the generalizability of our model on external datasets.
3.2 self-supervised learning platform for progression prediction task
We evaluated the performance of three SSL pretraining approaches in predicting disease progression. The first approach was to explore the pretrained models on unlabeled natural images to see if they could be transferred to medical images. The second approach was to use pretrained models on unlabeled in-domain medical images to assess their performance disease progression. The third approach domain SSL (referred to CDSSL) to leverage unlabeled data from multiple domains, including natural images and medical images. To establish a reference point, the target model is trained using random initialization, serving as a baseline for comparison.
to apply cross-
Exploring pretrained models on unlabeled natural images
SSL models trained on large datasets of natural images, such as ImageNet, have been shown to outperform supervised TmageNet models on several computer vision tasks. In this experiment, we hypothesized that
4

---

Published in Transactions on Machine Learning Research (07/2023)
Study
Number of patients
SSL
Fine-tuning
in-study test.
out-study test
HABS
289
BIOFINDER
752
BIOCARD
434
AIBL
1112
WRAP
578
FACEHBI
236
ADNI
2332
Scarlet RoAD
797
Marguerite RoAD
470
CREAD 1 &
2417
BLAZE
91
ABBY
445
OASIS-3
46
Table 1: Summary of datasets. The V indicates whether a study is utilized for a split. OASIS-3 and study
ABBY are designated as out-study test sets, meaning they have been utilized for either SSL pretraining or fino-tuning. The in-study test set includes patients from ADNT, Scarlet RoAD, Marguerite RoAD, CREAD & 2, and BLAZE; but there is no overlap between the splits. We were unable to find any labels for the first 6 rows of datasets
these models could be transferred to medical images and used to predict disease progression. We initialized the backbone encoder with weights from SSL models trained on ImageNet to exploit these benefits. In our study, we specifically concentrate on three prominent SSL methods: SimCLR, a contrastive approach: BarLow Twins (BLT), a redundancy reduction approach; and SwAV, a chustering- based contrastive learning approach. These methods have demonstrated remarkable performance on benchmarks designed for natural images. Although there are other SSL strategies available, their performance ImageNet is comparable to the ones have selected.
Exploring pretrained models on unlabeled in-domain medical images
As shown in Figure 1 a), we used SimCLR, Barlow Twins, and SwAV to learn distinctive representations of unlabeled medical images. These methods have all been shown to effective the classification and segmentation of medical images. We hypothesized that these models would be able to learn the features that are specific to medical images and be more effective predicting discase progression than the models trained unlabeled natural images.
Exploring CDSSL pretrained models both domains
representations learned from natural images may not be optimal for the medical imaging domain because of the large distribution shift between natural and medical images. Medical images are typically monochromatic and have similar anatomical structures, while natural images are typically colorful and have a wider variety of objects and scenes. We hypothesized that this discrepancy could be minimized by further pretraining on medical data. As shown in Figure bb-c, we used SimCLR, Barlow Twins, and SwAV to learn distinctive representations of unlabeled medical images top of pretraining ImageNet.
Fine Tuning
The progression prediction task utilizes a ResNet50 backbone (He et al. 2016) followed by a lincar layer, with the backbone being initialized randomly or with pretrained models. The model loss calculated using the mean square error (MSE) criterion (see Figure [ dd).
4 Results
We proposed the first benchmarking study evaluate the effectiveness of different pretraining models for disease progression prediction regression problem. Our main objective is to investigate the transferability
5

---

Published in Transactions on Machine Learning Research (07 (07/2023)
of features learned by pretraining natural medical images, or both, to the medical task of discase progression prediction. To evaluate our model's performance, used the Pearson correlation coefficient (r) and the coefficient of determination (1²2). R² is calculated as bellow:
Σ [tt 0)2 Σ
R²2 =
(1)
Vii 7)²
where Σ (e:) and (y: -)² respectively indicate the sum of squared residuals and total sum squared. In clinical studies, R² is widely employed to evaluate the ability a model to predict future out- comes Franzmeier et al. 2020).
4.1 Self-Supervised Pretraining
Experiments On Natural Images. This experiment examined the transferability of standard ImageNet models through the utilization of three widely recognized SSL methods: SimCLR, Barlow Twins, and Sw AV. All SSL models undergo pretraining on the ImageNet dataset and employ a ResNet-50 backbone.
SSt-medical images
CDSSI-Laibeld ImaggeNet
CDSSI-UUnlbelldd ImageNNtt
Uslabeted Medical
Labeled
Uniabeled InageNet
Selfooparsiel Model
Supervised Model
CNN
CNN Buidbune
Contostive Low
Hackhonn
Uslabeled Mediaal Image
Sclfeuperiisal
Images
Self-suprrised Model
A
CNN
Backtrons
Contoutive Low
Backhone
Backhene
Backbore
(u)
(b)
Transferring Pre-trained Backbone
Fine-tuning
Labeled Medical
Ussea Test Set
CNN
Downsttram Taok
Hackbone
(d)
(c)
Figure Different approaches for self-supervised pretraining on in-domaín medical imaging, including (a) random initialization, (b) supervised ImageNet initialization, and (c) self-supervised ImageNet initialization. (d) Performing fine-tuning by transferring the backbone from one of the scenarios (e) utilization of the trained model on unseen test sets.

---

Published in Transactions on Machine Learning Research (07 2202)
pretraining
Initialization
R
pretraining
Initialization
R
Random
0.07
Random
0.07
Supervised
ImageNet
0.06
SimCLR
0.17
SimCLR. 0.10
Self-Sueervied
SwAV
0.10
Self-Supervise
SwAV
0.08
Barlow Twins
0.13
Barlow Twins
0.14
(b) pretraining on Medical Images.
(a) pretraining on Natural Images.
Table 2: Comparison of different pretraining schemes on downstream task.
Results. The results in Table 2a show that transfer learning from the supervised ImageNet model does not improve over random initialization. This is likely due to the significant domain shift between the pretraining and regression target task. In particular, supervised ImageNet models tend to capture domain- specific semantic features, which can be inefficient if the pretraining and target data distributions differ. This finding is consistent with recent studies on other medical tasks that demonstrate that transfer learning from supervised ImageNet pretraining does not always correlate with performance on either classification or segmentation of medical images (Dippel et al. 2021 Vendrow & &Scooneeld 2022 Hosseinzadeh Taher et al. 2021
In contrast, transfer learning from self-supervised ImageNet models provides superior performance compared with both random initialization and transfer learning from the supervised ImageNet. model. The best self- supervised model (i.e., Barlow Twins) achieves a performance improvement of 7% and 8% over random initialization and the supervised ImageNet model, respoctively. This is likely due to the fact that self- supervised Tet models are trained to learn general-purpose features that are not biased toward any particular task. As result, they are better able to generalize to new domains.
Experiments On Medical Images. To investigate the effect of using in-domain medical images for self- supervised pretraining, we trained three SSL methods, SimCLR, Barlow Twins, and SwAV, on 11 unlabeled medical imaging datasets, which we call in-domain datasets. All SSL models were randomly initialized and then fine-tuned on our labeled dataset.
Results. Table 2b shows the performance of SSL models pretrained on the center slice dataset, measured by the R² score. We observe that SimCLR pretraining on the in-domain dataset achieves the highost performance, providing 7% and 4% boost over SwAV and Barlow Twins, respectively. This may be due to the superiority of contrastive learning for identifying significant MRI features for predicting progression of Alzheimer's disease in termss of CDR-SB. Moreover, the performance of SimCLR pretraining on the in- domain dataset exceeds that of both supervised and self-supervised pretraining on the ImageNet dataset (as seen in Table 2ab. This suggests that pretraining on the in-domain dataset encodes domain-specifie features that reflect the distinctive characteristics of medical images.
In contrast, pretraining Barlow Twins on in-domain data does not yield performance improvement compared to Barlow Twins pretrained on ImageNet. This result indicates that the features learned by Barlow Twins through pretraining on ImageNet demonstrate sufficient generalizability to medical images. Thus, the limited number of unlabeled medical images in the in-domain dataset (40k compared to 1.3M in ImageNet) may only provide marginal performance gains for the redundancy reductions- based Barlow Twins method.
Scaling Ablations. We conducted further experiments to evaluate the benefits of using more unlabeled images for self-supervised pretraining. For each SSL method, we trained two separate models, one pretrained center-slice dataset and the other pretrained on 5-slice dataset.
Results. As shown in Table 3, all three SSL models that were pretrained on the 5-slice datasct achieved belter results than those that were pretrained on the center-slice dataset. This suggests that SSL models can benefit from larger in-domain unlabeled datasets for pretraining. Specifically, the performance of Barlow Twins improved by 1% when was pretrained on the 5-slice datasct. This may confirm our previous as- sumption that Barlow Twins, in contrast to SimCLR, requires more unlabeled in-domain images to constrain the ImageNet-based features for medical tasks.

---

Published in Transactions on Machine Learning Research (07/2023)
Self-Supervised Model
Dataset
R²
center-slice
0.17
SimCLR
5-slice 0.19
center-slice 0.10
SwAV
5-slice 0.12
center-slice
0.13
Barlow Twins
5-slice
0.14
Table 3: Performance comparison of self-supervised models pretrained on datasel of different sizes
Cross-Domale Experiments. In this experiment, we examine the effects of self-supervised pretraining both natural images and medical images. To achieve this, we pretrained SimCLR on the 5-slice dataset with two distinet initialization schemes: Supervised ImageNet. (referred to as Labeled AmnageNet=<nndommae and Barlow Twins on the ImageNet dataset (referred to as Unlabeled ImageNet--ln-dommann We selected SimCLR and Barlow Twins for our experiments because they achioved the highest performance when pre- trained on either natural medical images, respectively, as shown in Tables 2al and 2b Figure bb,c) shows both cross-domain self-supervised pretraining schemes. In this section, also include the Pearson correlation coefficients to provide insights into the strength and direction of the relationship between the predicted values and the target values of CDR-SB.
Results. The results are displayed in Table 4a It is evident that the highest performance is achieved when both the unlabeled ImageNet and in-domain datasets are utilized for pretraining. Specifically, the Unlabeled ImageNet-<In-oomal pretraining surpasses the performance of models pretrained solely on the in-domain dataset or ImageNet. It yields significant performance improvements of 14%, 7%, and 2% com- pared to random initialization, ImageNet pretraining alone, and in-domain dataset pretraining alone, re- spectively. Consistent with earlier research (Hosseinzadeh Taher et al.) 2021 Azizi el. al. 2021), these findings suggest that combining pretraining on ImageNet with pretraining on in-domain datasets leads to more robust representations for medical applications. Additionally, it is worth mentioning that the Labeled ImageNet-<In-oonee Unlabeled ImageNet--In-oomin approach. These observations restate the effectiveness of self-supervised models in generating more generic representations that can be applied to target tasks with limited data, thereby reducing the need for extensive annotations and associated costs.
pretraining approach demonstrates inferior performance compared to the
we used Steiger's 71 method (Steiger) 1980). This method is utilized to compute the two-tailed p-value at a 95% confidence interval, and it's a statistical approach uniquely suited for assessing the significance of variances bet ween de- pendent correlation coefficients. The results are displayed in Table Notably, the Unlabeled ImageNet domain pretraining model demonstrates a statistically significant difference compared to othor modcls.
To compare the Pearson correlation coefficients of the models in Table 4a.
Figure illustrates the average frequency distribution of residuals across all three folds for the lop-performing models. Mean and standard deviation values are provided for each plot. Notably, the cross-domain model, specifically Barlow Twins TWins-+SimCIR, exhibits a higher concentration of residuals near zero and demonstrates smaller mean and standard deviation values.
4.2 In- and out-of- Domaan Generalization
To assess the robustness of our nop-performing model, Barlow TWins=SSimLRR, the 5-slice dataset, we evaluated its performance using three distinct test sets: an in-study test set and two out-study test sets. Furthermore, we compare this model's performance with the best-performing models initialized either by ImageNet or an in-domain dataset.
According to the results presented in Table 4b and Table G the highest performance achieved when both the unlabeled ImageNet and in-domain dataset are utilized for pretraining. Specifically, the Unlabeled Inn-eNet--nn-ommin pretraining approach exhibits significant improvement of 10% and 6%
8

---

Published in Transactions on Machine Learning Research (07/2023)
over the in-domain and ImageNet pretrained models, respectively, for the in-study test set. Similarly, out-study test sets ABBY and OASIS-3, the same model demonstrates an improvement up to 6% com- pared to other models. These results indicate that Unlabeled ImageNet--In-domain pretraining effec- tively encodes semantic features that are generalizable to other studies. Furthermore, the performance of Labeled ImageNed--lnndommin pretraining is inferior that of Unlabeled Imageeel- pretrain- ing. This indicates that supervised ImageNet models encode domain-specific semantic features, which may not be elficient when the pretraining and target data distributions significantly differ.
4.3 Visualizing Model Saliency Maps
Attribution methods are tool for investigating and validating machine learning models. Using the inter- protability of the ML modcls, can significantly help obtaining a bigger picture about risk factors inffuences on short-term prognosis. We used the GradCAM (Selvaraju et al.) 2017) method to extract and evaluate the varying importance of each part of brain MRIs using a gradient of final score. In this method, regions of an image marked with different colors ranging from red blue. Generally, areas that are closer to the red color contribute more significantly to the final result, based on the imput data (i.c., MRI slice).
Pretraining Method
Pretraining Dataset
R²
MSE
Random
0.07 (0.03)
0.33 (0.04)
5.31 (0.68)
Barlow Twins
ImageNet
0.16 (0.01)
0.42 (0.01)
4.81 (0.65)
SimCLR
In-domain
0.19 (0.02)
0.44 (0.01)
4.61 (0.62)
Supervised ImageNet - SimCLR Labeled ImageNet - In-domain 0.17 (0.04)
0.43 (0.05)
4.74 (0.82)
Barlow Twins + SimCLR
Unlabeled UnlabeleddImmgeNNee + In-domain 0.21 (0.02) 0.46 (0.01) 4.52 (0.56)
(a) Results of best performing models in each domain and their combination in cross-domain SSL setting validation set. All values are the mean over 3-fold stratified cross- validation, reported with standard
deviation in brackets
Pretraining Method
Pretraining Dataset
/f/rr on in-study test
18/r on ABBY
11/r on OASIS-3
Random
0.04/0.30
0.02/0.29
-0.01/0.10
Barlow Twins
ImageNet
0.12/00.35
0.07 1.07/0.34
-0.06/0.15
SimCLR
In-domain
0,14/00.38
0,09/0.35
0,11/0.36
Supervised ImageNet > SinCLR
Labeled InngeNet + In-domain
0.11/0.33
0.11/0..3
0.10/0.35
Harlow Twins + SimCLR
Unlabeled Iaaeoeeddd_mmaeeeeeeeee - In-domain
0.18/00.2
0.14/0.38
0.17/0.42
(b) Results on independent dataset
Table 4: Results different pretraining schemes on both (a) validation and (b) test sets in terms R² and r. OASTS-3 and ABBY are out-study test sets, meaning they have not been utilized for either SST. pretraining or fine-tuning.
Pretraining Method
Random
Barlow Twins
SimCLR
Supervised NeageeNt SimCLR
Barlow Twins
SimCLR
Random
0.003**
0.0001+4*
0.0001 **+
0.0001
Barlow Twins
0.003**
0.04*
0.27
0.0004***
SimCLR
0.0001
0.01*
0.35
0.012*
Supervised ImageNet - Sim CLR
0.0001**< ***
0.27
0.004**
Barlow Twins - SimCLR
0.0001
0.0
0.012
0.004
Table 5: Statistical significance of the resulting Pearson correlation coefficients. The p-values indicate the statistical significance of difference between the Pearson correlation coefficients of different models in Table Significance levels indicated by * ** and *** for p-values < 0.05, < 0.01, and < 0.001, respectively.

---

Published in Transactions on Machine Learning Research (07 (022))
0.25
Barken Tndax SSaCCR
Maa---Ai Sal 5al-215
Saperitidd laugr StoCCER
Manimbok Sal-311
Randoce
Mare- 0.00 5xl-2:33
0.15
0.05
0.00
-3.0
-2.
2.5
Residuals
Figure 2: Average frequency distribution. Each color represents the frequency distribution of residuals for a specific experiment.
Figure presents various examples of saliency maps, which depict the significance of different regions in the MRIs at pixel level. Notably, our rop-performing model, Barlow Twins->SimCLR, consistently highlights the subcortical areas of the brain. This observation aligns with prior research indicating that the initial stages of AD exhibit abnormal tau accumulation in the entorhinal cortex and subcortical brain regions (Rueb) eet al. 2017; Liu et. al. 2012). Therefore, it is reasonable that our model, i.e. Barlow exhibits higher attention to those specific brain regions in a population individuals with prodromal to mild Alzheimer's discase at bascline.
The randomly initialized model highlights random features all around MRTs, including background areas. In contrast, the model initialized with unsupervised ImageNet is more focused on brain regions, rather than irrelevant areas such as the background.
5 Discussion
We present cross-domain self-supervised learning framework for predicting the progression of Alzheimer's disease from MRIs, which is formulated as a regression task. Our study is pioneering effort aggregate a comprehensive set of internal and external cohorts/studies to create substantial dataset for model training. we show that using CL. pretraining on natural images followed by pretraining on medical images achieved the highest accuracy. This approach effectively alleviates domain shift challenge and greatly improves the generalization of the pretrained features.
Our extensive experiments demonstrate the effectiveness of our approach to combat. the lack of large-scale annotated data for training deep models for progression prediction. Our best performing model exhibits a substantial improvement over the fully supervised model, demonstrating that the appropriate utilization of unlabeled images, including both natural and medical images, indeed provides additional useful information that the model successfully learns from. Moreover, it indicates that the proposed Cross-Domale self super- vised learning approach can learn domain-invariant features, which enhances model generalization ability and robustnesss Therefore, it has the potential to identify patients at higher risk of progressing to AD and help develop better therapies at lower cost to society.
Our model has limitations related to the heterogeneity of the datasets used to train it, including differences in criteria for amyloid positivity across studies. Because of limited resources, our dataset analysis is focused
10

---

Published in Transactions on Machine Learning Research (07 (07/2023)
Original Images
Random Pre-Truining Model
Barlow Twins (ImageNet) Tre-iinnn Model
Barlow Twins SimCLR Model (Unlabeled Imagenet In-domuin)) Pre- Training Model
Figure 3: Illustrating the interpretation of three pretraining models using GradCam technique. The top row showcases the original MRI slices, while the subsequent rows, from top to bottom, illustrate the saliency maps generated by the following models: a randomly-initializen pretrained model, a pretrained model on natural images, and our best model, Barlow Twins - SimCLR.
subset of five slices per MRI scan. As result, we are only able to utilize small portion of the MRI volumes information in each scan. It is important to note that this constraint may hinder our model's ability to effectively learn from the complete set of brain regions. These limitations should be taken into consideration when interpreting the results from our model.
Our research only explores the benefits of 2D MRIs for prognostic prediction. This reduces the number of parameters and simplifics the model. 3D deep neural networks require many more learnable parameters to solve such complex problem, and they do not always produce accurate results (Wang et al. 2019). In future work, we will expand our dataset to include more than 5 slices from each image, as well as other views such as coronals and sagittals, allowing us to introduce more in-domain information to our pretraining model. We also will incorporate 3D MRIs into our analysis to compare their performance against 2D slices.
Acknowledgements
We would like to thank all of the study participants and their families, and all of the site investigators, study coordinators, and staff. Assistance in preparing this article for publication was provided by Genentech, Inc.
Alzheimer's Disease Neuroimaging Initiative (ADNT) is funded by the National Institute on Aging, the Na- tional Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following organizations: AbbVic; Alzheimer's Association; Alzheimer's Drug Discovery Foundation; Araclon Biotoch; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company: CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; FaroImmun; F. Hoffmann-La Roche Ltd and its affiliated
11

---

Published in Transactions on Machine Learning Research (07/2023)
company Genentech, Inc.; Fujirebio; GE Healtheare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Re- search & Development, LLC.;; Johnson & Johnson Pharmaceutical Research & Development LLC.,; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Com- pany; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www. www.fnih.org).... The grantee organization the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer's Therapeutic Research Institute at the University of Southern California. ADNT data are disseminated by the Laboratory for Neurolmaging at the University of Southern California.
The Australian Imaging Biomarkers and Lifestyle (AIBL) study (ww..AIB..ciroo.m is a consortium be- tween Austin Health, CSTRO, Edith Cowan University, the Florey Institute (The University of Melbourne), and the National Aging Rescarch Institute. Partial financial support was provided by the Alzheimer's As- sociation (US), the Alzheimer's Drug Discovery Foundation, an anonymous foundation, the Science and Industry Endowment Fund, the Dementia Collaborative Research Centres, the Victorian Government's Op- crational Infrastructure Support program, the McCusker Alzheimer's Rescarch Foundation, the National Health and Medical Research Council, and the Yulgilbar Foundation. Numerous commercial interactions have supported data collection and analysis. In-kind support has also been provided by Sir Charles Gaird- ner Hospital, Cogstate Ltd., Hollywood Private Hospital, the University of Melbourne, and St. Vincent's Hospital.
The BIOCARD study is supported by grant from the National Institute on Aging: : U19-AG03365. The BIOCARD Study consists of 7 Cores with the following members: (1) the Administrative Core (Marilyn Albert, Barbara Rodzon, Corinne Pettigrew), (2) the Clinical Core (Marilyn Albert, Anja Soldan, Rebecca Gottesman, Ned Sacktor, Scott Turner, Leonie Farrington, Maura Grega, Gay Rudow, Scott Rudow, Ros- tislav Brichko), (3) the Imaging Core (Michael Miller, Susumu Mori, Tilak Ratnanather, Timothy Brown, Hayan Chi, Authony Kolasny, Kenichi Oishi, Laurent Younes), (4) the Biospecimen Core (Richard O'Brien, Abhay Moghekar, Jacqueline Darrow, Alexandria Lewis), (5) the Informatics Core (Roberta Scherer, David Shade, Ann Ervin, Jennifer Jones, Hamadou Coulibaly, April Broadnax, Lisa Lassiter), (6) the Biostatistics Core (Mei-Cheng Wang, Jiangxia Wang, Yuxin Zhu), and (7) the Neuropathology Core (Juan Troncoso, Olga Pletnikova, Gay Rudow, Karen Fisher). The authors would like to acknowledge the contributions to BIOCARD of the Geriatric Psychiatry Branch (GPB) of the intramural program of the National Institute of Mental Health who initiated the study (PI Dr. Trey Sunderland).
The BioFINDER study was supported by the Furopean Research Council, the Swedish Research Council, the Strategic Research Programme in Neuroscience at Lund University (MultiPark), the Crafoord Foundation, the Swedish Brain Foundation, The Swedish Alzheimer foundation, the Torsten Söderberg Foundation at the Royal Swedish Academy of Sciences, and the regional agreement on medical training and clinical research (ALF) between Region Skåne and Lund University.
Data were provided in part by OASIS-3: Longitudinal Multimodal Neuroimaging: Principal Investigaatos: T. Benzinger, D. Marcus, J. Morris; NIH P30 AG066444, P50 AG00561, P30 NS09857781, P01 AG026276, P01 AG003991, R01 AG043434, UL1 TR000448, R01 EB009352. AV-45 doses were provided by Avid Ra- diopharmaceuticals, a wholly owned subsidiary of Eli Lilly.
The Harvard Aging Brain Study (HABS- POLAG036694; https://habs.nng www.:s............mmmmoo was launched in 2010, funded by the National Institute on Aging. and is led by principal investigators Reisa A. Sperling MD and Keith A. Johnson MD at Massachusetts General Hospital/Harvard Medical School in Boston, MA.
The Wisconsin Registry for Alzheimer's Prevention (WRAP) dataset were supported with grants from the CS National Institutes of Health (grant Nos. AG027161 and AG021155).
The FACEHBI study was supported by funds from Fundació ACE Institut Català de Neurociències Aplicades, Grifols, Life Molccular Imaging, Araclon Biotech, Alkahest, Laboratorio de Análisis Echevarne and IrsiCaixa.
12

---

Published in Transactions on Machine Learning Research (07 2223)
References
Laith Alzubaidi, Mohammed A Fadhel, Omran Al-Shamma, Jinglan Zhang, J Santamaria, Ye Duan, and Samcer R. Olciwi. Towards a better understanding of transfer learning for medical imaging: a case study, Ayplied Sciences, :0(13):4523, 2020.
Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, et al. Big self-supervised models advance medical image classification. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 3478 3488, 2021.
Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Kornblith, Ting Chen, Patricia MacWilliams, Sara Mahdavi, Ellery Wulczyn, al. Robust and efficient medical imaging with self-supervision. arXiv preprint ar Xiv:2200 09723, 2022.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in nearal information processing systems, 32, 2019.
Wenjia Bai, Chen Chen, Giacomo Tarroni, Jinmiing Duan, Florian Guitton, Steffen E Petersen, Yike Guo, Paul M Matthews, and Daniel Rueckert. Self-supervised learning for cardiac mr image segmentation by anatomical position prediction. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 541 549. Springer, 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varannc-invrianccccovaaaaa regularization for self-supervised learning. Xixiv preprinl ar Xix::22105. 04906, 2021.
Benjamin Billot, Douglas N Greve, Oula Puonti, Axel Thielscher, Koen Van Leemput, Bruce Fischl, Adrian V Dalca, and Juan Eugenio Tglesias. Synthseg: Domain randomisation for segmentation of brain mri scans any contrast and resolution. arXie preprint arXiv:2107.0955
2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in Newral Information Processing Systems, 13:9912-9924, 2020.
Krishna Chaitanya, Ertunc Erdil, Necrav Karani, and Ender Konukoglu. Contrastive learning of global and local features for medical image segmentation with limited annotations. Advonces in Neurol Information Processing Systems, 33:12546-12558, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive Icarning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self- supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33:22243 22255, 2020b.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0429 2020c.
Alexander Dagley, Molly LaPoint, Willem Hujjbers, Trey Hedden, Donald G McLaren, Jasmeer P Chatwal, Kathryn Papp, Rebecca E Amariglio, Deborah Blacker, Dorene M Rentz, et al. Harvard aging brain study: dataset and accessibility. Neuroimage, 144:255-255 2017.
Jonas Dippel, Steffen Vogler, and Johannes Höhne. Towards fine- iineggrined visual representations by com- bining contrastive learning with image reconstruction and iitention-weighter pooling. arXie preprint arXiv:2104.04323 2021.
Carl Doersch, Abhinav Gupta, and Alexci Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEFE international conference on computer vision, pp. 1422-1430, 1430, 2015.
13

---

Published in Transactions on Machine Learning Research (07/2023)
Shaker El-Sappagh, Tamer Abuhmed, SM Riazul Islam, and Kyung Sup Kwak. Multimodal multitask deep learning model for alzheimer's 's disease progression detection based on time series data. Newrocompaling, 412:197 215, 2020.
JR Ellis, Pradeep Jonathan Nathan, Victor L Villemagne, RS Mulligan, Timothy Saunder, K Young, Cather- ine L Smith, J Welch, Michael Woodward, Keith A Wesnes, et al. Galantamine iolantamine-induoed induced improvements in cognitive function are not related to alterations in a432 nicotinic receptors in early alzheimer's s disease as measured in vivo by [- fluoro-a-85380 pet. Psychopharmacology, 202(1):79-91 2009.
Nicolai Franzmeier, Nikolaos Koutsouleris, Tammie Benzinger, Alison Goate, Celeste M Karch, Anne M Fagan, Eric McDade, Marco Duering, Martin Dichgans, Johannes Levin, et al. Predicting sporadic alzhcimer's discase progression via inherited alzheimer's ciscase-informed machine-loarning. Alzheimer's Dementia, 16(3):501-55) 511, 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, arXiv preprint arXiv:1803.07728, 2018
Mara Graziani, Vincent Andrearczyk, and Henning Müller. Visualizing and interpreting feature reuse of pretrained cmns for histopathology. In Irish Machine Vision and Image Processing Conference (IMVIP 2019), Dublin, Ireland, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchalskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21284 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9773 2020a.
Xuchai Hc, Xingyi Yang, Shanghang Zhang, Jinyu Zhao, Yichen Zhang, Eric Xing, and Pengtao Xic. Sample- efficient deep learning for covid-19 diagnosis based on et scans. medroie, 2020b.
Michal Heker and Hayit Greenspan. Joint liver lesion segmentation and classification via transfer learning. arXiv preprint Xiv:20004 12352, 2020.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International confer- ence on machine learning, pp. 4182-4192. PMLR, 2020.
Mohammad Reza Hosseinzadeh Taher, Fatemeh Haghighi, Ruibin Feng, Michael B Gotway, and Jianming Liang. A systematic benchmarking analysis of transfer learning for medical image analysis. In Domain Adaptation and Representation Transfer, and Affordable Healthoare and AT for Resource Diverse Global Health, pp. 3 13. Springer, 2021.
Pamela J LaMontagne, Tammie LS Benzinger, John C Morris, Sarah Keefe, Russ Hornbeck, Chengjie Xiong, Elizabeth Grant, Jason Hassenstab, Krista Moulder, Andrei G Vlassenko, et al. Oasis-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and alzheimer disease. MedRaie, pp. 2019 12, 2019.
Rebecca Langhough Koscik, Bruce P Hermann, Samantha Allison, Lindsay R Clark, Erin M Jonaitis, Kim- berly D Mucller, Tobey Betthauser, Bradley Christian, Lianlian Du, Ozioma Okonkwo, et al. Validity evidence for the research category, category,"coguiiivell impairment and alzheimer's disease. Frontiers in Aging Neuroscience, 13:688478, 2021.
haimpaired-dellnnnge " as risk marker for mild cognitive
14

---

Published in Transactions on Machine Learning Research (07/2023)
Hongwei Li, Fei-Fei Xue, Krishna Chaitanya, Shengda Luo, Ivan Ezhov, Benedikt Wiestler, Jianguo Zhang, and Bjoern Menze. Imbalance-aware self-supervised learning for 3d radiomic representations. In Interna- tional Conference on Medical Image Computing and Computer-Assisted Assisted Intervention, pp. 36 46. Springer, 2021.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. Prototypical contrastive learning of unsupervised inpresentaaiios. Xrxiv preprint urXiv:2005.04 04966, 2020.
Jingyu Liu, Gangming Zhao, Yu Fei, Ming Zhang, Yizhou Wang, and Yizhou Yu. Align, attend and locate: Chest. x-ray diagnosis via contrast. induced altention network with limited supervision. In Proceedings of the IEEE/CVT International Conference on Computer Vision, pp. 10632-10641, 2019.
Li Liu, Valerie Drouet, Jessica W Wu, Menno Witter, Scott A Small, Catherine Clelland, and Karen Duff. Trans-synaptic spread of tau pathology in vivo. PloS one, :(2):23130, 2012.
Quande Liu, Lequan Yu, Luyang Luo, Qi Dou, and Pheng Ann Heng. Semi-supervised medical image classification with relation-driven self-ensembling model. IEEE transactions on medical imaging, 3911)): 2020a.
Yuan Liu, Ayush Jain, Clara Eng, David H. Way, Kang Lee, Peggy Bui, Kimberly Kanada, Guilherme de Oliveira Marinho, Jessica Gallegos, Sara Gabriele, et al. A deep learning system for differential diagnosis of skin diseases. Nature medicine, 26(6):900 908, 2020b.
Niklas Carlssnn,Carlgren, Antoine Leuay, Shorena Janelidze, Sebastian Palmqvist, Erik Stomrud, Olof Strandberg, Ruben Smith, and Oskar Hansson. The implications of different approaches to define at (n) in alzheimer disease. Newrology, 94(21):e2233-e22) e2244, 2020.
Scott Mayer McKinney, Marcin Sienick, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg Corrado, Ara Darzi, et al. International evaluation of an ai system for breast cancer screening. Nature, 577(7788):89-944 2020.
Afonso Menegola, Michel Fornaciali, Ramon Pires, Flávia Vasques Bittencourt, Sandra Avila, and Eduardo Valle. Knowlcdge transfer for melanoma screening with deep lcarning. In 2017 IEEE 14th international symposium on biomedical imaging (ISBI 2017), pp. 297-300. IEEE, 2017.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarianl representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 6707-6717, 2020.
Abhay Moghekar, Shanshan Li, Yi Lu, Ming Li, Mei-Cheng Wang, Marilyn Albert, Richard O'Brien, BIO- CARD Research Team, et al. Csf biomarker changes precede symptom onset of mild cognitive impairment. Newrology, (120)) 2013.
Sonia Moreno-Grand Octavio Rodríguez-Gómer Angela Sanabria, Alba Pérez- Cordón, Domingo Sánchez- Ruiz, Carla Abdelnour, Sergi Valero, Isabel Hernandez, Maitée Rosende- Roca,, Ana Mauleon, et al. Explor- ing apoe genotype effects on alzheimer's 's disease risk and amyloid 8 burden in individuals with subjective cognitive decline: The fundacioace healthy brain initiative (facchbi) study bascline results. Alzheimer's 's B Dementia, 14(5):634-64) 2018
Minh Nguyen, Tong He, Lijun An, Daniel C Alexander, Jiashi Feng, BT Thomas Yeo, Alzheimer's Dis- ease Neuroimaging Initiative, et al. Predicting alzheimer's 's disease progression using deep recurrent neural networks. Newrolmage, 222:117203, 2020.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69 84. Springer, 2016.
Kanghan Oh, Young- Chul Chung, Ko Woon Kim, Woo-Sung Kim, and II-Scok Oh. Classification and visualization of alzheimer's discase using volumetric convolutional neural network and transfer learning. Scientifie Reports, 9(1):1-16, 2019.
15

---

Published in Transactions on Machine Learning Research (07/2023)
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint Xive 1tm1187..00 007.03748, 2018.
Ronald Carl Petersen, PS Aisen, Laurel Beckett, MC Donohue, AC Gamst, Danielle Harvey, CR Jack, WJ Jagust, LM Shaw, AW Toga, et al. Alzheimer's disease neuroimaging initiative (adni): clinical characterization. Newrology, 74(3):201-209, 2010.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems, 32, 2019.
Shannon L Risacher, Andrew J Saykin, John D Wes, Li Shen, Hiram A Firpi, and Brenna C McDonald. Baseline mri predictors of conversion from mci to probable ad in the adni cohort. Curvent Alzheimer Research, 6(4):347-33 361, 2009.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and compuler-assisted interven- tion, pp. 234-241. Springer, 2015.
Udo Rueb, Katharina Stratmann, Helmut Hcinsen, Kay Seidel, Mohamed Bouzrou, and Horst- Werrer Korf. Alzheimer's disease: characterization of the brain sites of the initial tau cytoskeletal pathology will improve the success of novel immunological anti-tau treatment approaches. Journal of Alzheimer's Disease, 57(3):
683 696, 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE international conference on computer vision, pp. 118-626,44 2017.
Hari Sowrirajan, Jingbo Yang, Andrew Y Ng, and Pranav Rajpurkar. Moco pretraining improves represen- tation and transferability of chest x-ray models. In Medical Imaging with Deep Learning, pp. 728-744.
PMLR, 2021.
Hannah Spitzer, Kai Kiwitz, Katrin Amunts, Stefan Harmeling, and Timo Dickscheid. Improving cytoar- chitectonic segmentation of human brain areas with supervisd stamese networks. In International Conference on Medical Image Computing and Computer- computer-Assssted Assisted Intervention, pp. 663-671, Springer, 2018.
James H Steiger. Tests for comparing elements of a correlation matrix. Psychological bulletin, 87(2):245, 1980.
Xu Tian, Jin Liu, Hulin Kuang, Yu Sheng, Jianxin Wang, and The Alzheimer's Disease Neuroimaging Initiative. baied multi-task decoupling learning for alzheimer's disease detection and mmse score prediction: A multi-site validation. arXiv preprint arxiv:2204.01 017708, 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, In European conference computer vision, pp. 776-794. Springer, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827 (06667777799 2020b.
Edward Vendrow and Ethan Schonfeld. Understanding transfer learning for chest radiograph clinical report generation with modified transformer architectures. arXiv preprint arXiv:2205.022 02844, 2022.
Janani Venugopalan, Li Tong, Hamid Reza Hassanzadeh,, and May D Wang. Multimodal deep learning models for carly detection of alzhcimer's discase stage. Scientific reports, 11(1):1-13, 2021.
Hongfei Wang. Yanyan Shen, Shuqiang Wang, Tengfei Xiao, Liming Deng, Xiangyu Wang, and Xinyan Zhao. Ensemble of 3d densely connected convolutional network for diagnosis of mild cognitive impairment and alzheimer's 's disease. Neurocomputing, 333:145, 10:1145-155 156, 2019.
16

---

Published in Transactions on Machine Learning Research (07/2023)
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pallern recognilion. pp. 3733 3742, 2018.
Huidong Xie, Hongming Shan, Wenxiang Cong, Xiaohua Zhang, Shaohua Liu, Ruola Ning, and Ge Wang. Dual network architecture for few-view ct-trained imagenct data and transferred for medical imaging. In Developments in X-ruy Tomography XII, volume 11113, pp. 184 184-194. SPIE, 2019.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embodding learning via invariant and spreading instance leature. In Proceedings of the IBEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6210 0210-6219, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310-12320. PMLR,
2021.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pp. 149.666. Springer, 2016.
Hong-Yo Yu Zhou, Shuang Yu, Cheng Bian, Yifan Hu, Kai Ma, and Yefeng Zheng. Comparing to learm: Surpassing imagenet pretraining on radiographs by comparing image representations. In International Conference on Medical Image Computing and Computer-Assiset Assisted Intervention, pp. 398 407. Springer, 2020.
Zongwei Zhou, Vatsal Sodha, Jiaxuan Pang, Michael B Gotway, and Jianming Liang, Models genesis. Medical image analysis, 67:101840, 2021.
Jiuwen Zhu, Yuexiang Li, Yifan Hu, Kai Ma, Kevin Zhou, and Yefeng Zheng. Rubik's cube+: A self- supervised feature learning framework for 3d medical image analysis. Medical image analysis, 64:101746, 2020.
Xinrui Zhuang, Yuexiang Li, Yifan Hu, Kai Ma, Yujiu Yang, and Yefeng Zheng. Self-supervised feature learning for 3d medical images by playing a rubik's cube. In International Conference on Medical Image Compuling and Compuler-Assislet Intervention, pp. 420 428. Springer, 2019.
A Appendix
A.1
Self-supervised Models
SimCLR
In this method, the learning rate is 1e 1e-4, 4, Adam is used as an optimizer and NT-Xent loss is used as a loss function. We use an implementation of SimCLR in Pytorch-lightning repository for creating our framework.
Barlow Twins
This method utilizes LARS as an optimizer and 1e as learning rate scheduler, with CosineWarmup Warmup serving as the learning rale scheduler. This repository is used as basis https: http:///ggittabbcooSeeaaarar Com//eanWaren/ Lightming-barrlwttiine git]] SwAV
For this method, we use Adam as the optimizer, with a learning rate of 1e-4. Like SimCLR, we use the SwAV base model implementation from the Pytorch-lightning repository https://githui: com heoom/ightnnigg-i AI/
Lightning-bolts .git
A.2 MSE values corresponding to Table 4b
17

---

Published in Transactions on Machine Learning Research (07/202) 2023)
Pretraining Method
Pretraining Datases
MSE on in-study test
MSE on ABBY
MSE on OASIS-3
Random
5.88
6.11
4.03
Barlow Twins
ImageNet
5.49
5.89
4.11
SimCLR
In-domain
5.22
5.42
3.45
Supervised ImageNet + SimCLR Labeled ImageNet In-domsin
5.56
5.65
3.19
Barlow Twins + SinCLR
Unlabeled InngeNet In-domaiu
5.11
5.33
3.20
Table 6: Results of different pretraining schemes test sets in terms of MSE.
18


Reviews:
[{"review_id": "VZ67EW4y3p", "reviewer": "Reviewer_XPWA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1) Trained and evaluated on a large, multi-institutional dataset.\n2) Investigated multiple settings for self-supervised learning and the schemes for utilizing natural image pre-trained model.\n\nWeakness:\nThe \"Methodology\" section and Fig. 1 need substantial improvement to understand the experiment setting, specifically:\n1) What is the term \"defined comparison\" in Fig. 1? The term has never been described in the manuscript. This is critical as self-supervised learning relies on a certain invariance or intrinsic data structure. But the reviewer failed to understand what pairs of MR images are defined as \"similar\" and not.\n2) It is unclear what exact task has been performed in the experiment, although it is likely a classification task. In addition, while the author claimed that R2 is a better metric, it is suggested that accuracy be reported so that the reader can compare the model's performance with other methods. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a self-supervised learning scheme for brain MR image analysis in AD patients with the help of networks trained on natural images. The model was evaluated with ablation studies on the task of AD classification. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See my comments above.", "broader_impact_concerns": "This work involves little if any ethical concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nrHxa8QjQy", "reviewer": "Reviewer_cwKK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n-- large-scale evaluation using three main SSL methods \n-- empirical results on utilising ImageNet and other Medical Imaging Datasets to pre-train SSL for the main downstream task\n-- promising results that could be transferred to other medical conditions.\n\nWeaknesses\n\n-- overly non-technical; the technical bit reads more like a blog or guidelines\n-- R^2 is uninformative as a means to provide a quantitative evaluation of deep learning model fit. It is a metric suitable for linear models mostly to explain variance when manipulating various independent variables. \n-- Not very well written, e.g. what is the purpose of the sub-section 4.2 heading? \n-- What is the takeaway message of figure 2? For instance, are you trying to fit a linear model on the predicted vs ground truth data?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This research paper explores the potential of using self-supervised learning (SSL) techniques to assess the progression of Alzheimer's disease. Specifically, it utilises various SSL approaches such as SimCLR, BarlowTwins, and SWaV to evaluate the effectiveness of SSL methods in a medical setting. \n\nThe paper is noteworthy as it is the first to consider multiple SSL methods and evaluate how different data combinations, such as pretraining on imagenet/other medical datasets and then using Alzheimer's as the downstream task, can be used to assess Alzheimer's progression. Additionally, the paper highlights SSL's usefulness in predicting a score related to Alzheimer's progression more accurately. \n\nIt should be noted that there is no technical contribution in terms of self-supervised learning, but it might appeal to the medical community as a means to apply SSL in other settings.\n\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "--The results do not have to be scattered like that but instead, a table should summarise the findings, across SSL methods and combinations of data.\n\n-- Rather than having speculative headings for the sub sections in section 4, please use informative headings and then discuss the findings and limitations when showing the results.\n\n", "broader_impact_concerns": "there is no reflection regarding clinical evaluation of any sort, or how such a system could be used in a more realistic setting. The ML community will not know how the clinically relevant score to evaluate Alzheimer's progression is used in practice.\n\nCould you please confirm that the internal datasets used correspond to a clinical study that has received the relevant ethics approvals?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yQxPv3Pa1K", "reviewer": "Reviewer_6d7X", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Self-supervised pre-training is quite important in many applications, especially in the medical domain, where the number of labeled examples is limited. There are numerous self-supervised techniques and combinations of different pre-training strategies. Therefore, it is quite useful to investigate a subset of these methods and report the results for a certain application. In that sense, I think such an analysis would be of interest to many researchers in the field.\n- Description of the analysis reported in the paper is explained clearly except for a few convoluted pieces of information that are crucial. \n\nWeaknesses:\n- The results reported in the paper after a single training run. Sometimes, the quantitative results are very close to each other and it might be difficult to conclude that one is better than the other without seeing the average+-std results of multiple runs with different seeds. Also, a statistical significance tests between these results would be useful to really conclude that the one method is better than the other.\n\n- I didn't quite understand the medical datasets that are used in self-supervised pre-training, fine-tuning and the final evaluation. Are there any datasets that are used in both pre-training and final evaluation (I assume images do not overlap but I am asking if samples from one dataset can appear in different splits?) If so, this doesn't really show the generalization performance of the models. These splits should be indicated clearly.\n\n- The results are obtained on a single test dataset. How would the results change if different test sets were used? I would like to see an experiment designed as follows to evaluate the results better: 1) medical datasets to be used for pre-training are fixed (let's say datasets A, B, C) and multiple datasets reserved for fine-tuning and final testing (let's call them dataset D, E), 2) model is pre-trained using A, B, C, 3) in one experiment, the pre-trained model is fine tuned on D and tested on E, and in the other vice versa. And, make sure that D and E are not very close to each other. I think this would be a more realistic setup to evaluate the performance. There is an experiment with an independent test set, but which dataset is not specified. And also, what are the test datasets used in the other experiments?\n\n- What would be the upper bound when a model is trained on the target dataset with full supervision using sufficient labels? Can such an experiment be designed? Such an analysis would help to understand the gap between the upper bound and the self-supervised pre-training.\n\n- Most of the results are reported in terms of R^2, but only a single experiment is reported with R^2 and Pearson's correlation. I think all results should be given with both metrics as it is difficult to interpret R^2.\n\nMinor points:\n- In Fig. 2, the results are given with different ranges in the axes of Figures a, b, and c. Currently, the difference seems very small by looking at the plots. Fixing the axes' ranges might help.\n- On page 9 of the results section: I think Table 3a should be Table 3b.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates different pre-training strategies for the prognosis of Alzheimer's disease progression using neural networks.\nThe investigated strategies include supervised/self-supervised pre-training with ImageNet followed by self-supervised pre-training on a large medical dataset and self-supervised pre-training on a large medical dataset w/ random initialization.\nThe results show that self-supervised pre-training with ImageNet followed by self-supervised pre-training on a large medical dataset led to the best scores on an unseen dataset.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Reporting results of multiple runs is crucial to understanding which method is better (please also see my first comment in the weaknesses section). Currently, it is difficult to reach conclusions reported in the paper.\n\n- The information about the medical dataset split that are used during pre-training, fine-tuning, and test should be clarified. There should also be experiments with different test sets used; e.g. by rotating the test dataset with the ones used during pre-trainined and/or fine-tuning.\n\n- I would be interested in seeing at least one experiment where the result of the fully supervised setting on the final test set is reported. This would give the idea about the gap between self-supervised pre-training and the upper bound.", "broader_impact_concerns": "I don't have any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyzes the impact of pre-training on the task of Alzheimer's disease progression. The authors investigate the use of supervised and self-supervised learning strategies, especially SimCLR, BarlowTwins, and SWaV.

In their initial reviews, the reviewers found the experimental study interesting but pointed out the lack of methodological contributions in the work. They also raised concerns about the evaluation protocol (dataset splits), the absence of statistical significance testing, and the need to use more relevant metrics beyond $R^2$. RcwKK and RXPWA additionally highlighted some limitations in the paper presentation. The revision overall successfully improved the experiments, clarified the results, provided consolidated findings, and highlighted the link between the results and the clinical outcome. After the discussion period, R6d7X and RcwKK recommended weak acceptance due to the limited novelty and the interesting experiments, while RXPWA was more inclined to reject the paper.

The AE carefully reviewed the submission and discussions. The AC agrees that there is no methodological contribution, but also considers that the experimental results are relevant. The main finding of this work is that self-supervised pre-training on ImageNet, followed by self-supervised pre-training on a large medical dataset, yielded the best results for estimating Alzheimer's disease progression. While it would have been beneficial to consider other recent baselines, such as foundation models (e.g. CLIP), the paper's claims are supported by the experiments and would certainly be of interest to a non-negligible subset of the TMLR audience. Therefore, the AE recommends accepting the paper.

Decision: Accept (poster)