From Barlow Twins to Triplet Training: Differentiating Dementia with Limited Data

Authors: yitong  li
tom nuno wolf
sebastian  pölsterl
igor  yakushev
dennis m hedderich
christian  wachinger
Venue: MIDL_io_2024_Conference
Published: 2024-01-17 15:21:05

Abstract: Differential diagnosis of dementia is challenging due to overlapping symptoms, with structural magnetic resonance imaging (MRI) being the primary method for diagnosis. Despite the clinical value of computer-aided differential diagnosis, research has been limited, mainly due to the absence of public datasets that contain diverse types of dementia. This leaves researchers with small in-house datasets that are insufficient for training deep neural networks (DNNs). Self-supervised learning shows promise for utilizing unlabeled MRI scans in training, but small batch sizes for volumetric brain scans make its application challenging. To address these issues, we propose Triplet Training for differential diagnosis with limited target data. It consists of three key stages: (i) self-supervised pre-training on unlabeled data with Barlow Twins, (ii) self-distillation on task-related data, and (iii) fine-tuning on the target dataset. Our approach significantly outperforms traditional training strategies, achieving a balanced accuracy of 75.6%. We further provide insights into the training process by visualizing changes in the latent space after each step. Finally, we validate the robustness of Triplet Training in terms of its individual components in a comprehensive ablation study. Our code is available at https://github.com/ai-med/TripletTraining.

Main paper: 
 introduction: 
 The number of patients suffering from dementia is expected to increase to 152.8 million by 2050 (Nichols et al., 2022) , with Alzheimer's Disease (AD) accounting for 60-80% of affected patients. Frontotemporal dementia (FTD) is the second most common type of dementia in the younger-elderly population (aged < 65 years) (Young et al., 2018) . Accurately diagnosing different dementia types is challenging as symptoms overlap, but is crucial for patient management, therapy, and prognosis. In the clinical routine, differential diagnosis incorporates structural magnetic resonance imaging (sMRI) to evaluate distinct atrophy patterns. Despite the clinical importance of differential diagnosis, there is limited research in computer-aided diagnosis for this task compared to classifying AD and cognitively normal (CN) subjects, largely rooted in the lack of related public MRI datasets. Accessing in-house data from hospitals is an alternative; however, even if available, such data is typically too small to train DNNs successfully. At the same time, public datasets exist that focus on single types of dementia. For AD, the Alzheimer's disease neuroimaging initiative (ADNI, adni.loni.usc.edu) provides an extensive resource (Jack et al., 2008) . Similarly, the initiative on Neuroimaging in Frontotemporal Dementia (NIFD, 4rtni-ftldni.ini.usc.edu) collected data for FTD. As a result, previous research on the differential diagnosis of AD and FTD combined the two datasets (Ma et al., 2020; Hu et al., 2021; Nguyen et al., 2022) . An inherent limitation of such a combination is the confounding of dataset and diagnosis, potentially yielding shortcut learning that differentiates datasets instead of diagnosis (Geirhos et al., 2020) . While the evaluation of such a merged dataset easily leads to inflated estimates of classification accuracy, it can instead provide a valuable resource in the training process.Population imaging studies, e.g., UK Biobank (Miller et al., 2016) , establish an even larger resource of MRI data for training, but they do not contain task-related labels. Recent advances in self-supervised learning (SSL) can provide means to benefit from such data in an unsupervised fashion, which have not yet been incorporated for differential diagnosis. A challenge for applying common SSL methods like SimCLR (Chen et al., 2020) or SwAV (Caron et al., 2020) to 3D brain MRI data is the need for large batch sizes and hence GPU memory, as they rely on hard negative samples to avoid collapse. Barlow Twins (Zbontar et al., 2021) is an alternative that eliminates the need for negative samples and naturally avoids collapse by redundancy reduction. As a result, it demonstrates better robustness to small batch sizes, which makes it well-suited for SSL in neuroimaging.We introduce Triplet Training for differential diagnosis with limited target data. Triplet Training, see Figure 1 , combines three learning strategies to include all relevant MRI data in training. First, self-supervision trains the network on task un-related data without target labels (UK Biobank). Second, we apply self-distillation on a task-related dataset that is created by merging data from ADNI and NIFD. Third, we fine-tune the model on a training set of the small in-house clinical data. Our results demonstrate that Triplet Training outperforms competing methods while being robust to a variety of properties.To summarize, our key contributions are:• Triplet Training for learning DNNs with limited target data.• Adapting Barlow Twins as an efficient SSL algorithm on volumetric brain MRI data.• Self-distillation to distill knowledge from the SSL-trained teacher network in combination with task-related labels. • Reporting of test accuracy for differential diagnosis of AD and FTD on a wellcharacterized single-site clinical dataset.Differential Diagnosis of AD and FTD with DNNs. One line of research for differential diagnosis performs brain segmentation (Ma et al., 2020; Nguyen et al., 2022) and uses volume and thickness measurements for the classification. Such an approach may restrict learning general dementia-specific features across the entire brain. Motivated by the success of using a 3D-ResNet50 encoder-decoder on MRI (Hu et al., 2021) to extract latent representations for classification, we selected a 3D-ResNet as the backbone for our work.As no public dataset exists comprising both AD and FTD patients, these methods combined ADNI and NIFD. The fundamental problem of such an approach is that datasets coincide with diagnosis; hence, it cannot be determined whether the network inadvertently learns to differentiate datasets instead of pathology (Geirhos et al., 2020) . Thus, we incorporate ADNI and NIFD in Triplet Training for pretraining and evaluate on the in-house single-site dataset to allow for a reliable performance assessment.Self-Supervised Learning and Self-Distillation in Medical Image Analysis. A variety of research (Azizi et al., 2021; Chaitanya et al., 2020; Chen et al., 2019; Taleb et al., 2020; Zhou et al., 2020; Hosseinzadeh Taher et al., 2021; Li et al., 2021; Tran et al., 2022; Zhou et al., 2019) concluded that self-supervised pre-training on domain-related datasets (i.e., unlabeled (3D) medical images) improves performance on medical downstream tasks. Haghighi et al. (2022) added restorative and adversarial branches to the SSL pipeline for medical downstream tasks. Additionally, Jiang and Miao (2022) and Ye et al. (2022) showed how SSL trained on task-unrelated medical images improves generalization on low-data regimes. This problem has also been tackled with self-distillation in Paluru et al. (2023) , Li et al. (2022), and Sun et al. (2021) .In summary, self-supervised pre-training and self-distillation on medical images improve the performance of the downstream task, with domain-related datasets adding additional benefits. Such approaches have not yet been explored for differential diagnosis and have not yet been extended to Triplet Training. Moreover, research on Barlow Twins has been limited despite its attractive properties for volumetric medical images. 
methods: 
 In this section, we present the details of Triplet Training to tackle the limited data availability for the target task. We utilize SSL with Barlow Twins to integrate task un-related data in the initial step. In the second step, we propose to include task-related data via self-distillation. Self-distillation fully utilizes the previous SSL step by aligning the distribution of latent features extracted by the student network with those learned from SSL, using the Kullback-Leibler (KL) divergence. This method not only builds on prior learning but also reduces the risk of overfitting on the task-related dataset. Finally, we fine-tune the We define a 3D image as I ∈ R H×W ×D , with H, W , D as height, width and depth, respectively. A dataset consists of N 3D images I i , i = 1, . . . , N , and class labels y i if available. Our model consists of a feature extractor f : R H×W ×D → R Z , with Z the latent space dimension, and a projection head g : R Z → R C , which maps the latent vectors to outputs of dimension C. We select a 3D-ResNet backbone for the feature extractor f and a two-layer MLP for the projection head g (implementation details in Section A). We utilize three datasets:1. The unlabeled dataset U comprises N = 39, 560 samples X U i = (I U i ) extracted from the UK Biobank (Miller et al., 2016) . 3. The labeled target in-house dataset T consists of N = 329 samples X T i = (I T i , y T i ), y T i ∈ {CN, AD, F T D} from hospital Klinikum rechts der Isar, Munich, Germany. Table 1 reports demographic statistics for all three datasets.1. Self-Supervised Learning. The self-supervision task proposed in Barlow Twins (BT) de-correlates features in latent space and has shown to be relatively robust with respect to the batch size (Zbontar et al., 2021) . This benefits training with 3D medical images because their large size limits batch sizes. Hence, BT presents a promising approach for the initial step of Triplet Training.To pre-train the feature extractor f θ with trainable parameters θ on the unlabeled dataset U, two different augmentations A and B of an input image I U i are required. These augmented images A(I U i ) and B(I U i ) are fed into a neural network consisting of the feature extractor f θ and a projection head g θ , yielding two output latent vectorsz A i = g θ (f θ (A(I U i ))) and z B i = g θ (f θ (B(I U i ))), z A i , z B i ∈ R C .The model is optimized by maximizing the cross-correlation between corresponding features of different augmentations L BT = c (1 -C cc ) 2 + λ 1 c j̸ =c C cj 2 , with C cj = i z A i,c z B i,j i (z A i,c ) 2 i (z B i,j ) 2with c = 1, . . . , C indices across the latent space dimension C, i the index of a sample within the dataset U, and λ 1 a constant hyper-parameter. This loss makes embeddings invariant to distortions while also reducing redundant information. We denote the resulting weights after this self-supervised pre-training step as θ ′ . 2. Self-Distillation. This step requires the feature extractor f θ , with pre-trained weights θ = θ ′ from the previous step, as a teacher. We freeze the teacher network f θ during training to reduce the risk of over-fitting towards the task-related dataset D. We randomly initialize a student network f ψ with trainable parameters ψ of the same architecture as the teacher, and an additional projection head g ψ . Inspired by Tian et al. (2020) , the student is trained on the task-related dataset D by minimizing the KL divergence L KL between the outputs of the feature extractors f θ (I D i ) and f ψ (I D i ), and minimizing the cross-entropy L CE between the predictions of the student g ψ (f ψ (I D i )) and the related class labels y D i :L SD = λ 2 L KL (Z ψ ∼ f ψ (I D i ), Z θ ∼ f θ (I D i )) + (1 -λ 2 ) i L CE (g ψ (f ψ (I D i )), y D i ),with Z θ and Z ψ random variables sampled via forward passes of the samples from the dataset D, and λ 2 a constant hyper-parameter trading off the importance of the first and second terms of L SD . The resulting weights of the student network are denoted as ψ ′ .3. Fine-Tuning. In the final step, we optimize the student network f ψ , g ψ initialized with pre-trained weights ψ = ψ ′ from the previous step, by fine-tuning it on the in-house dataset T for the target task using cross-entropy loss:L FT = i L CE (g ψ (f ψ (I T i )), y T i ). 
experiments: 
 Pre-processing and Data Augmentation: Each T1-weighted MRI scan is pre-processed using SPM1 and the VBM pipeline of CAT12 (Gaser et al., 2022) . The results are graymatter density volumes (samples with a quality control score lower than B-are discarded), Evaluation: As the target dataset T is relatively small, we perform 5-fold cross-validation with ratios of 65%, 15%, and 20% for train, validation, and test sets, respectively, stratified by age, gender, and diagnostic labels to prevent biased results (Barnes et al., 2010) . Additionally, we split a balanced 20%-portion of the task-related dataset D to perform further evaluations for the task at hand. Miscellaneous: Hyper-parameters for the individual training steps and search spaces of baseline methods are reported in Section B.1. We implement models with PyTorch (Paszke et al., 2019 ) and train on one NVIDIA GeForce 3090 with 24 GByte memory. We train the model for 29,300 self-supervised iterations (24 hours), followed by 600 self-distillation iterations (2.5 hours) and 150 fine-tuning iterations with early stopping (40 minutes). 
results: 
 As a baseline, we implement a non-deep learning approach for the differential diagnosis on T , by extracting FreeSurfer (Fischl, 2012) volume and thickness features from MRI scans to train an XGBoost classifier, which achieves a balanced accuracy (BAcc) of 66.46 ± 3.45%. As seen in Table 2 , training a DNN on the target dataset T alone results in a BAcc of 67.15 ± 4.78%, which is likely due to the overfitting on the small task-specific data. Pretraining the model on the task-related dataset D improves the performance only marginally by 1.29%. Pre-training with unlabeled U with established SSL methods (SimCLR (Chen et al., 2020) , VICReg (Bardes et al., 2022) , DiRA (Haghighi et al., 2022) , and Barlow Twins (Zbontar et al., 2021) ) and then fine-tuning on T outperforms supervised pre-training on D by 2.92% (with Barlow Twins). Triplet Training, which adds a self-distillation step on D after self-supervised pre-training, significantly outperforms all competing approaches on the target dataset, achieving a BAcc of 75.57 ± 3.62% with the highest true positive rates for both types of dementia (see Table 2 ).Additionally, we evaluate Triplet Training on the hold-out test set of D after selfdistillation on D, which clearly outperforms (+7.4%) supervised training on D alone (de- Ablation Study 2: Benchmark Self-Supervised Approaches. We replace the SSL algorithm (Barlow Twins) in the initial step of Triplet Training with three SOTA algorithms. Table 3 reports that Triplet Training showcases high and consistent accuracy across all SSL methods, highlighting its robustness and generalizability. Among them, Barlow Twins and SimCLR demonstrate the best performance, and introduce few additional hyper-parameters compared to the other methods. We argue that Barlow Twins is the optimal choice, as it has shown to be robust in terms of the batch sizes. https://www.fil.ion.ucl.ac.uk/spm/software/spm12 
 
 Appendix:  


Decision: All reviewers found the proposed method to be novel and the results promising.
2 weak accept and 2 strong accept recommendations from reviewers.

Reviews: review: paper_summary: paper_summary: The authors propose a three-stage training approach for the case of limited datasets, and present its performance for differentiating dementia from 3D MRI volumes. After a thorough description of their approach, they evaluate their model and compare it to other methods in the field, showing robustness, and the usefulness of all three stages.
main_review: main_review: I would like to thank the authors for adequately addressing all my questions, and the questions of the other reviewers. I believe the review process even further improved the quality of the manuscript, and it is suitable for acceptance.
strength_weakness: strength_weakness: The description of the performed experiments, and the statistical analysis of the results could be improved to provide an even more confident argument about the performance of the proposed training approach.
questions: questions: - Could the authors address how the very specific number of iterations have been assessed for each training stage? Was it empirical based on the validation dataset?
- Could the authors describe UMAP which was used for visualizing the latent space? Or provide a reference for it?
- The standard deviations of the results (Table 3) suggest that the performance of the four methods are similar. Did the authors perform a statistical test of significance to find BT the best model for BAcc?
paper_summary: paper_summary: This article primarily introduces Triplet Training as a solution for the limited dataset challenge in differential diagnosis of Alzheimer's Disease (AD) and Frontotemporal Dementia (FTD). By incorporating Barlow Twins for self-supervised pre-training on unlabeled data and utilizing self-distillation on task-related data with a teacher-student network, useful knowledge is obtained. This allows leveraging abundant unlabeled data to assist model training. The method is logically rigorous, theoretically sound, and excellently written.
main_review: main_review: The overall operational procedure is clearly described, with rich novelty in the ideas presented. The illustrations are clear and easy to understand. In terms of experimentation, there is a significant improvement compared to previous baseline models. Overall, from theoretical innovation to experimental results and writing, the quality of the work is commendable. 

The theoretical richness of this study is valuable, particularly in the current context of insufficient diagnostic data for dementia in medical datasets.
strength_weakness: strength_weakness: It is recommended to open-source the network and provide reproducible code. Additionally, can this network be utilized for clinical diagnosis? Deployment and implementation should be considered in the future.  
In the manuscript, you mentioned that the evaluation method is: "As the target dataset T is relatively small, we perform 5-fold cross-validation with ratios of 65%, 15%, and 20% for train, validation, and test sets, respectively, stratified by age, gender, and diagnostic labels to prevent biased results." You need to ensure that this ratio is applied consistently across all datasets when comparing the network with other baseline models. 
Additionally, using light grey color might not be easily distinguishable; changing it to green could be a viable alternative.
questions: questions: I recommend acceptance.
paper_summary: paper_summary: This work proposed a three-stage training architecture to address the limited data available for AI diagnosis of dementia. The first stage uses self-supervision to learn the stable features invariant through different augmentations. Then, a self-distillation network is trained to regress the task-related label from the extracted features in Stage 1. Lastly, pre-train the second regression network on a limited private dataset. From the experiments, the newly proposed architecture performs well on the private clinical dataset and shows strong generalizability.
main_review: main_review: The methodology proposed not only serves the objectives of this study but also shows potential applicability in broad, therefor it could bring interesting discussions to the conference. The experiments align with current research trends, are carefully thought out and the manuscript is well written.
strength_weakness: strength_weakness: The Ablation Experiments seem to over-focus on details like hyper-parameters and benchmarks rather than the architecture design. The ablation should contain more essential compressions, like Why self-distillation is better than the directly fine-tuning or frozen feature extractor followed by a tail regressor in stage 2.
questions: questions: More relevant ablation with analysis in detail to prove the key advantage of the design.
paper_summary: paper_summary: The authors propose Triplet Training, a combination of representation learning approaches that has not yet been explored for differential diagnosis. The framework involves three stages: self-supervised pre-training using Barlow Twins, self-distillation on task relevant data, and fine-tuning on targeted small datasets, leading to improvements in diagnostic accuracy in classifying two types of dementia and control cases.
main_review: main_review: I would like to thank the authors for adequately addressing all my questions, and the questions of the other reviewers. I've decided to increase my initial score, believe it accurately reflects the paper's quality and merits within the context of its field.
strength_weakness: strength_weakness: The authors provide background by emphasizing the challenges in distinguishing between different dementia types. Although their study only includes two subtypes. It is unclear if the framework would perform on differentiating more than two subtypes given diagnosis is challenging as symptoms overlap.
questions: questions: Could you please clarify the nature of the input provided to the neural networks? Specifically, are you utilizing volumetric gray matter density maps, and is white matter excluded from consideration? 

Will the code associated with this study will be made publicly accessible upon publication?
paper_summary: paper_summary: The authors are proposing a training strategy for classification of neurodegeneration (AD vs CN vs FTD) using a DL model. They propose 3 steps. 1) train a backbone encoder on a very large dataset using a barlow twin approach. 2) train a second encoder using distillation from the first model on a smaller, but more relevant dataset. 3) fine tune the training on a small target dataset. 

Results are better than straightforward training designs (although still not good enough to be clinically useful). The experiments are convincing and are well explained given the limited space available.
main_review: main_review: This is a good paper and the authors have added more information about the double Autoencoder scheme which was a question raised by most of the reviewers. The results are very encouraging, although lacking some quantitative validation.
strength_weakness: strength_weakness: The premise of the paper is not correct. Structural MRI is not the primary method for the diagnosis of dementia. Dementia is commonly clinically diagnosed with neuropsy evaluation. More specific diagnosis is achieved in most cases with CSF or PET markers for pathological proteins (e.g. Tau, Amyloid,…). MRI is used mostly for screening out non degenerative causes (e.g. cancer) or to refine/confirm the diagnosis in some disorders (e.g FTD). 

The rational for the paper is still correct though: that MRI can be useful for differential diagnosis (e.g. AD Vs FTD), that disorders often overlap, and that there is a lack of large databases covering most disorders. 

The authors should give more consideration to the label ground truth used. Are they using the clinical ADNI label (mostly based on MMSE) or do they use a more recent label using other biomarkers (e.g. Amyloid)? This is important because the clinical diagnosis of AD for example is only 80% sensitive and 70% specific. More detail about this should be provided. 

3/ a few critical details are missing (e.g. above), including what are the methods used for data augmentation of the first step for the BT strategy?
questions: questions: Are all the datasets using similar MRI sequences? 

Are the datasets different in terms of main known risk factors (age, gender, ApoE4,…)

How is data augmentation implemented for the BT learning?

Comments: comment: We greatly appreciate your very in-depth review and insightful suggestions for the improvement of the paper. We hope the following clarifications can address your questions and concerns, and we are open to further questions:
1. Thanks for pointing out the detailed medical premises of structural MRI. We took the chance to improve the manuscript and rephrased the corresponding paragraphs. We target using MRI for the differential diagnosis of dementia.

2. All datasets acquire MPRAGE images (with some ADNI data for MPRAGE/IR-SPGR). We extracted volumetric gray matter density maps from T1-weighted MRI using the VBM pipeline of CAT12 [3] as the input.

3. We show the statistics of all the datasets in Table 1 of the manuscript, which includes the information of age and gender across datasets. Regrettably, we don’t have the complete information for ApoE4 for the in-house data available.

4. As harmonizing high-dimensional images across sites is still ongoing research with the risk of removing disease-related information, in our study, we use gray matter density maps as the input, an established approach for standardizing the input with respect to scanner effects [4].

5. We performed random cropping with resizing, random flipping, and random affine transformation for data augmentation in SSL training. Detailed information can be found in the Supplementary Table 5.

[3] Gaser C, Dahnke R, Thompson PM, Kurth F, Luders E, Alzheimer"s Disease Neuroimaging Initiative. CAT - A Computational Anatomy Toolbox for the Analysis of Structural MRI Data. bioRxiv 2022.06.11.495736; doi: https://doi.org/10.1101/2022.06.11.495736

[4] Dhinagar, N.J., Thomopoulos, S.I., Owens-Walton, C., Stripelis, D., Ambite, J.L., Steeg, G.V. and Thompson, P.M. (2022), Alzheimer’s Disease Detection with a 3D Convolutional Neural Network using Gray Matter Maps from T1-weighted Brain MRI. Alzheimer's Dement., 18: e066446. https://doi.org/10.1002/alz.066446
We greatly appreciate your positive review and detailed comments, we hope the following clarifications can address your questions and we are open to further questions:
1. We would gladly validate our framework in distinguishing more than two dementia subtypes once we collect enough data for the rare dementia types. Unfortunately, we are limited by the availability of publicly available datasets including more than these two subtypes.

2. The input of the neural networks is the volumetric gray matter density maps extracted via the VBM pipeline of CAT12 [3], excluding the white matter. Using gray matter density maps as the input is an established approach for standardizing the input with respect to scanner effects [4].

3. We have added the link of the code to the manuscript and we are eager to make our code public upon acceptance to facilitate reproducibility.

[3] Gaser C, Dahnke R, Thompson PM, Kurth F, Luders E, Alzheimer"s Disease Neuroimaging Initiative. CAT - A Computational Anatomy Toolbox for the Analysis of Structural MRI Data. bioRxiv 2022.06.11.495736; doi: https://doi.org/10.1101/2022.06.11.495736

[4] Dhinagar, N.J., Thomopoulos, S.I., Owens-Walton, C., Stripelis, D., Ambite, J.L., Steeg, G.V. and Thompson, P.M. (2022), Alzheimer’s Disease Detection with a 3D Convolutional Neural Network using Gray Matter Maps from T1-weighted Brain MRI. Alzheimer's Dement., 18: e066446. https://doi.org/10.1002/alz.066446
We greatly appreciate your very positive review and constructive suggestions. We plan to explore the same strategy for a multi-modality dataset in future work. In this study, we focused our ablation study on validating the robustness of Triplet Training across hyper-parameters and SSL algorithms in the initial stage. We evaluated direct fine-tuning and frozen feature extractors during the early design of our pipeline, which all failed to surpass the current one (BAcc of 69.81 +- 4.52% with frozen feature extractors during fine-tuning and 72.40 +- 4.36% with direct fine-tuning on D). We argue that self-distillation marginalizes the risk of overfitting on the task-related dataset compared to the other approaches.
We greatly appreciate your positive and detailed review, we have added the link of our code to the manuscript and we are eager to make our code public upon acceptance. We hope the following clarifications can address your questions:
1. The same splits of target dataset T are used to evaluate all baseline models as well as our pipeline. We adapt the data split method from ClinicaDL [2] to evenly distribute age, gender, and diagnostic labels across training, validation, and test data splits. 

2. Thank you for your valuable feedback regarding the color choices in our plots. We aimed to design our plots with accessibility and clarity in mind, ensuring they are colorblind-friendly and easily interpretable. To achieve this, we carefully tested various color palettes, ultimately selecting the current scheme as it best met these inclusivity and readability standards. 

[2] Thibeau-Sutre, E., Díaz, M., Hassanaly, R., Routier, A., Dormont, D., Colliot, O., Burgos, N.: ‘ClinicaDL: an open-source deep learning software for reproducible neuroimaging processing‘, 2021. hal-03351976
We greatly appreciate your very positive review and suggestions to improve the manuscript, we have added the link of our code to the manuscript and we are eager to make the code public upon acceptance. We hope the following clarifications can address your questions and are open to further questions:
1. We determined the number of iterations for each step based on the convergence of the validation set. In addition, we also used early stopping in the last training stage to prevent overfitting. We have updated these experimental details in the manuscript.

2. Uniform Manifold Approximation and Projection (UMAP) is a dimension reduction technique that can be used for high-dimensional data visualization, similarly to t-SNE, but also for general non-linear dimension reduction. The details for the underlying mathematics can be found in reference [1], which we have also added to our manuscript.

3. The performance of the four SSL algorithms used in the initial stage of Triplet Training is consistent, suggesting that the high performance was gained through the whole designated pipeline of Triplet Training rather than the choice of SSL algorithms. This highlights the robustness and generalizability of Triplet Training. We argue that Barlow Twins is the optimal choice, as it introduces fewer additional hyper-parameters and has shown to be robust in terms of batch sizes.

[1] McInnes, L, Healy, J, UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, ArXiv e-prints 1802.03426, 2018