scSSL-Bench: Benchmarking Self-Supervised Learning for Single-Cell Data
Authors: Olga Ovcharenko
Florian Barkmann
Philip Toma
Imant Daunhawer
Julia E Vogt
Sebastian Schelter
Valentina Boeva
Venue: ICML 2025 spotlightposter (2025)
Keywords: Benchmark
self-supervised learning
single-cell data
single-cell RNA sequencing
batch integration
augmentations
contrastive learning
Primary area: applications->health_medicine
TL;DR: scSSL-Bench ia a benchmark that provides an evaluation platform and concrete recommendations for applying self-supervised learning to single-cell analysis.

Abstract: Self-supervised learning (SSL) has proven to be a powerful approach for extracting biologically meaningful representations from single-cell data. To advance our understanding of SSL methods applied to single-cell data, we present scSSL-Bench, a comprehensive benchmark that evaluates nineteen SSL methods. Our evaluation spans nine datasets and focuses on three common downstream tasks: batch correction, cell type annotation, and missing modality prediction. Furthermore, we systematically assess various data augmentation strategies. Our analysis reveals task-specific trade-offs: the specialized single-cell frameworks, scVI, CLAIRE, and the finetuned scGPT excel at uni-modal batch correction, while generic SSL methods, such as VICReg and SimCLR, demonstrate superior performance in cell typing and multi-modal data integration. Random masking emerges as the most effective augmentation technique across all tasks, surpassing domain-specific augmentations. Notably, our results indicate the need for a specialized single-cell multi-modal data integration framework. scSSL-Bench provides a standardized evaluation platform and concrete recommendations for applying SSL to single-cell analysis, advancing the convergence of deep learning and single-cell genomics.

ScSSL-Bench:
Benchmarking Self-Supervised Learning for Single-Cell Data
Olga Ovcharenko
Florian Barkmann" *2 Philip Toma Imant Daunhawer? Julia E. Vogt 2
Sebastian Schelter Valentina Boeva 233
Abstract
abling cell molecular profiling at unprecedented resolu- tion (Sikkema al., 2023; Eraslan al., 2022). This breakthrough has revolutionized our ability to understand diseases, develop personalized treatments, and trace the ori- gins of complex conditions like cancer and autoimmune disorders. scRNA-seq (Tang et al., 2009) captures gene expression levels individual cells and generates a high- dimensional matrix where each row represents a cell and each column represents a gene's expression level. Multi- omies approaches simultaneously measure additional molec- ular features, including chromatin accessibility through ATAC-seq (Grandi et al., 2022) protein levels via CITE- seq (Stoeckius et al., 2017). Modern multi-omics experi- ments generate massive datasets encompassing hundreds of thousands of cells, with each cell characterized by diverse measurements: the expression of tens of thousands of genes, the accessibility of hundreds of thousands of chromatin re- gions, and the abundance of hundreds of surface proteins. Multi-modal profiling provides an unprecedented view of cellular state and function. However, the resulting datasets are susceptible to batch effects - systematic technical vari- ations introduced during sample preparation, sequencing, or processing (Lähnemann et al., 2020). If left uncorrected, batch effects mask genuine biological signals and compro- mise downstream analyses (Heumos et al., 2023). For in- stance, when comparing blood samples from cancer patients processed in different laboratories, batch effects can make immune cells from the same patient appear more different from each other than from cells of other patients, masking crucial patterns in how the immune system responds to the tumor (Slyper et al., 2020).
Self-supervised learning (SSL) has proven to be a powerful approach for extracting biologically meaningful representations from single-cell data. To advance our understanding SSL methods applied to single-cell data, we present scSSL- Bench, a comprehensive benchmark that evalu- ates nineteen SSL methods. Our evaluation spans nine datasets and focuses on three common down- stream tasks: batch correction, cell type anno- tation, and missing modality prediction. Fur- thermore, we systematically assess various data augmentation strategies. Our analysis reveals task-specific trade-offs: the specialized single-cell frameworks, VI. CLAIRE, and the finetuned scGPT excel at uni-modal batch correction, while generic SSL methods, such as VICReg and Sim- CLR, demonstrate superior performance in cell typing and multimmoal data integration. Random masking emerges as the most effective augmenta- tion technique across all tasks, surpassing domain- specific augmentations. Notably, our results indi- cate the need for specialized single-cell multi- modal data integration framework. scSSL- Bench provides a standardized evaluation platform and concrete recommendations for applying SSL to single-cell analysis, advancing the convergence of deep learning and single-cell genomics.
1. Introduction
Recent progress in single-cell RNA sequencing (scRNA- seq) and multi-omics sequencing technologies has trans- formed our understanding of cellular heterogeneity by
The success of self- self-supervised learning (SSL) methods in computer vision (He et al., 2020; Chen et al., 2020), video processing (Schiappa et al., 2023), and natural language processing (Min et al., 2023) has inspired their application to single-cell data. Several models have been adapted for analyzing single-cell data (Han et al., 2022; Liu et al., 2024; Li et al., 2023; Tang al., 2024), showing promising re- sults in mitigating batch effects and improving downstream analyses. There is an interest in the genomics community in finding standardized approaches for applying SSL to single- cell analysis. A recent work (Richter et al., 2024a) discusses scenarios in which SSL is applicable to single-cell genomics.
Equal contribution 'BIFOLD & TU Berlin, Berlin, Ger- many 'Department of Computer Science, ETH Zurich, Zurich, Switzerland "Swiss Institute of Bioinformatics, Lausanne, Switzerland *Paris Cité University, Cochin Institute, INSERM Olga Oveharenko
U1016, Paris, France. Correspondence <ovcharenko@@n cettuu-bblindee det, Sebastian Schelter <schelter@tu- berlin.de>, Valentina Boeva iaaleentinaaboevar eth.cthz.ch>..
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Benchmarking Self-Supervised Learning for Single-Cell Data
The authors compare the performance of masked autoen- coders and two SSL methods (BYOL (Grill et al., 2020) and Barlow Twins (Zbontar et al., 2021)), discuss the effects of pre-training on auxiliary data, and empirically study the efficacy of zero-shot and fine-tuned SSL.. However, their work lacks a comparison to specialized single-cell mod- els and does not explore individual hyperparameters and regularization techniques. Furthermore, building upon in- novations in natural language processing (Vaswani et al., 2017; Devlin et al., 2019b), single-cell foundation mod- els (Cui et al., 2024; Yang et al., 2022a; Theodoris et al., 2023) have recently emerged as powerful tools to under- stand cellular heterogeneity and gene-gene interactions, and require a comparison to contrastive methods.
the most beneficial augmentation technique that surpasses biology-specific augmentations. (3) An assessment of de- sign decisions suggested in the related work, e.g., retaining projector and domain-specific batch normalization, helps to answer RQ3 and to find best practices that can be adopted by the single-cell genomics community (Subsection 4.3). We find that neither domain-specific batch normalization nor retaining the projector during inference improves results.
We provide our benchmark code under open license https: //g/t//tttt thub. com/ Boova /as/s-sSLLLBBB Bench for reproducibility and for fostering further research on SSL for single-cell data.
2. Background
Since the majority of SSL methods were originally devel- oped for image and text data, there is lack of a systematic comparison of models, hyperparameters, training regimes, regularization techniques, and augmentations for single-cell genomics data (Toma et al., 2024). This knowledge gap limits our understanding of how to effectively adapt and op- timize SSL methods for single-cell data. Our work seeks to fill this gap and focuses on the following research questions:
In this Section, we discuss SSL on single-cell data, the corresponding downstream tasks, and specialized methods.
2.1. Machine Learning on Single-Cell Data
Data: There exist multiple technologies that measure dif- ferent aspects of the cellular state. scRNA-seq (Tang et al., 2009) measures which genes are expressed in each cell and produces a high-dimensional sparse count matrix represent- ing individual cells as rows and genes as columns. CITE- seq (Stoeckius et al., 2017), 10x multiome (Baysoy et al., 2023), and TEA-seq (Swanson et al., 2020) profile com- plementary to gene expression aspects such as chromatin accessibility and protein abundance within cell.
RQ1 - Do specialized single-cell SSL methods outper- form generic SSL methods? How does the performance of SSL models differ for uni-omics and multi-omics data?
R@2 How do hyperparameters and augmentation tech- niques impact the performance of generic SSL methods for single-cell data?
Downstream Tasks: Learned cell representations are com- monly used for multiple downstream tasks.
RQ3 - Are batch normalization and multi-modal integra- tion techniques proposed for image data beneficial for single-cell genomics data as well?
Batch correction - Single-cell data can be affected by batch effects, which challenges the ability to measure true bio- logical variation (Yu et al., 2023; Polański et al., 2019). Batch effects are technical biases introduced while sequenc- ing because of differences in sequencing platforms, tim- ing, reagents, or experimental conditions across laborato- ries (Zhang et al., 2024). To address the presence of hatch effects, a common approach is learning a batch-corrected lower-dimensional embedding, where cells cluster based on their cell type and cell state rather than their experimental batch of origin (Hao et al., 2024) (Figure G2 in the Appendix illustrates cells before and after batch correction).
Our main contribution is open-source benchmark, scSSL- Bench, which compares the performance of several self- supervised learning methods for single-cell data. (1) To address RQI, we evaluate nineteen generic and specialized single-cell SSL methods across seven different single-cell uni-modal and two molti-modalloeo datasets, assessing their per- formance three common downstream tasks: batch correc- tion, cell type annotation, and missing modality prediction (Subsection 4. 1). Our results reveal that specialized frame- works, VI and CLAIRE, together with the foundation model, scGPT, are the best for uni-modal batch correction, while generic SSL techniques such as VICReg and SimCLR outperform domain-specific methods for multi-modal batch correction and the other two tasks on single-modal data. (2) For RQ2, we evaluate various model architectures and hyperparameters, including representation and projection dimensionality, augmentation strategies, and multi-modal integration methods. (Subsection 4.2). Overall, we find that a moderate to larger embedding dimensionality con- sistently leads to improved results and identify masking as
Cell type annotation - This task (also called as query-to- reference mapping) revolves around unsupervised transfer learning (Yang et al., 2022b), where the primary objective is to annotate cells of a hold-out dataset (query) by mapping them a joint latent space of pre-annotated train dataset (reference), whose cell types are known (Lotfollahi et al., 2022). Once test and train data are aligned, held-out cells are annotated using a classifier trained on embeddings of the reference dataset. Figure G3 in the Appendix visualizes how
2

---

Benchmarking Self-Supervieed Learning for Single-Cell Data
Representation Learning
Downstream Tasks
supervised learning methods
Nineteen generic and specialized selt-
SSt. Methods
Generie
SSmmCL
BYOL
Bio Contrast
SoGPT ScBERT
Baseline
Evaluation of leareed embeddings on three downstream
Batch Correction
MoCo
ViGReg
CLAIRE
Geneformer
SCDC
SimSiam Barlow
Concarto
scHuttartly
Input Data
NNNCLR
Twins
-SCUUP
scTEL
ScVi/totaM
Goussian Noice
Masking
BSKNN
Cell Type Annotation
Call Unimect
Cells
Augmenntttinns
Batch
Misging Modality Prediction
Genes
InnerSwap
GrossOver
MNN
Six augm- entations create two views of each cell
Nine single-cell genomics datasets (seven uni-modal and two multi-modall)
Figure 1. Outline of scSSL-Bench: As input, ScSSL-Bench takes scRNA-sec -equassequaat data (cell-by-gent count matrix), where each value in the matrix represents the number of reads in cell for the corresponding gene. scSSL-Bench trains one of nineteen methods: Generie, specialized contrastive (Bio Contrast), specialized generative (Bio Gen), and baselines. For self-supervised generic methods, scSSL-Bench uses augmentations
create two views of cell. The learned embeddings are evaluated on three downstream tasks.
the learned representations of train and hold-out sets and train cell types are used to predict the cell types of hold-out data (query, blue result) during subsequent inference.
is a contrastive self-supervised distillation framework that uses an asymmetric teacher-student network structure (Hu et al., 2023) and dropout to create two augmented views of cell (a positive pair). Positive/negative pairs are con- trasted using NTXent loss (Sohn, 2016). Concerto also supports single-cell multi-modal data, e.g. pairs of RNA and protein. The scCLIP (Xiong et al., 2023) method is a generalized multi-modal transformer model that applies contrastive learning to single-cell multi-omics data, which adopts ideas from CLIP (Radford et al., 2021) by defining podality-speciffie encoders, constructing positive pairs from two modalities of the same cell, and contrasting them using InfoNCE loss (van den Oord et al., 2019).
Missing modality prediction For multi-modaa datasets, missing modality prediction enables the inference of unmea- sured (missing) modalities in held-out (query) cells (Yang et al., 2022b). Given multi-modal train data (reference) with RNA and protein expressions and hold-out data containing only RNA, the goal is to predicts the hold-out dataset's 's orig- inal protein values by averaging the proteins of the nearest neighbors from the train set (referred to kNN probing).
2.2. Self-Supervised Learning (SSL) Methods
Single-Cel Generative Methods: Generative approaches, from variational autoencoders (VAE) to transformer- based foundation models, are used to learn single-cell representa- tions and exploit the biological batch/cell type annotations during training, which can be seen as leaking information compared to contrastive SSL methods. The state-of-the-at single-cell method se VI (Lopez et al., 2018) is a widely- used VAE that leverages zero-inflated negative binomial distribution as reconstruction loss. A multi-modal version of VI, totaIVI (Gayoso et al., 2021), allows joint analysis of RNA and protein expressions. SCDC (Li et al., 2024) is uni-modal method that employs biological batch and cell type encoders create concatenated representation that is reconstructed using the decoder. To improve the discrimi- nation of the batch information, SCDC uses a specialized batch discriminator. ScTEL (Chen et al., 2025) leverages transformer and LSTM layers to establish mapping from RNA expression to unobserved protein expression in the same cells. scButterfly (Cao et al., 2024b) supports CITE- seq (Stoeckius et al., 2017) data and employs a dual-VAF architecture with modality-specine pretraining and dis-
SSL aims discover useful data representations without relying on annotations (Geiping al., 2023) by leveraging the dis-/similarity of data samples. We refer to Appendix A for details on generic SSL methods (and their contrastive and non-contrastive variants).
Single-Cell Contrastive Methods: There are several SSL frameworks tailored for single-cell data, CLEAR (Han et al.,, 2022) employs contrastive SSL and leverages InfoNCE loss (van den Oord et al., 2019). Positive/negative pairs are created by adding Gaussian noise, random masking, cross- ing over genes between two cells. CLAIRE (Yan et al., 2023) suggests a novel augmentation strategy by finding mutual nearest neighbors (MNN) between and nearest neighbors (KNN) within experimental batches in a dataset. CLAIRE uses biolo-biological batch MNN pairs as initial positive pair seeds, which are then "mixed" with intra-biological- batch neighbors to generate positive pairs CLAIRE ex- tends MoCo's (He et al., 2020) architecture with online and momentum encoders. Concerto (Yang et al., 2022b)
3

---

Benchmarking Self-Supervised Learning for Single-Cell Data
criminator to encourage the mixing of different modalities. Recently, ransformer-based single-cell foundation models (scFMs) have emerged. For generalizability, scFMs are pre- trained on tens of millions of cells. scBERT (Yang et al., 2022a) adapts BERT's (Devlin et al., 2019a) masked lan- guage modeling to learn contextual gene representations, scGPT (Cui et al., 2024) uses GPT-style pretraining to create transferable representations across cell types and experimen- tal conditions, and Geneformer (Theodoris et al., 2023) em- ploys a transformer architecture pre-trained on large-scale gene expression datasets to capture gene-gene interactions.
models scGPT (Cui et al., 2024), Geneformer (Theodoris et al., 2023), and scBERT (Yang al., 2022a). Additionally, for multi-omics integration, we evaluate scButterfly (Cao et al., 2024b) and scTEL (Chen et al., 2025), which leverage variational autnencoders. SCDC (Li et al., 2024) and princi- pal component analysis (PCA) (Pearson, 1901) are used as baselines in scSSL-Bench. We include PCA as baseline to assess whether more complex SSL methods offer substantial improvements over simple linear dimensionality reduction technique that does not correct for batch effects. We refer to Subsection 2.2 for detailed descriptions of each method.
First, in all contrastive methods except Concerto, two views are created by augmenting a single sample. Second, both views are encoded by a network with shared weights, pro- ducing data representations. Concerto removes the necessity for transforming samples by placing dropout layer behind the encoder backbone. Finally, while training, all represen- tations produced by the encoder are passed into projector to improve robustness (Xue et al., 2024). In all contrastive approaches but Concerto and scCLIP, the projector is dis- carded during inference, keeping only the encoder's output.
3. Benchmark Design
Figure illustrates our design of seSSL-Bench. The input of the benchmark are cell-by-gene count matrices contain- ing scRNA-seq CITE-seq data I Depending on the data and SSL method, seSSh-BBench trains one of nineteen representation learning frameworks using augmentations to create positive/negative pairs for self-supervised ap- proaches. Finally, the Iearned representations are evaluated on three downstream tasks
Downstream Tasks and Evaluation: To address RQI, our benchmark evaluates multiple single-cell datasets three tasks: batch correction, cell type annotation, and modality prediction (see Subsection 2.1 for details).
3.1. Datasets, Models, and Tasks
We consider nine single-cell genomics datasets that represent common established benchmarks (Richter et al., 2024a). Peripheral Blood Mononuclear Cells (PBMC), Pancreas, Immune Cell Atlas, Mouse Cell Atlas (MCA), Human Immune Cells (HIC), Lung, and Tabula Sapiens are seven single-modal datasets collected using ScRNA- seq (Tang et al., 2009) technology. Multi-modal Periph- eral Blood Mononuclear Cells (PBMC-M) and Multi-modal Bone Marrow Mononuclear Cells (BMMC) are multi-modal datasets collected using CITE-seq technology that contain RNA and protein or gene expression and protein abundance (ADT) respectively. Further details in Appendix B.
Datasets:
Batch correction - the quality of patch-corrected embed- dings is measured by biological conservation and batch correction metrics. These metrics were introduced in single- cell integration benchmarking (scIB) (Büttner al., 2019; Luecken et al., 2022; Tran et al., 2020), tool that is widely used in the single-cell community, see Appendix C for de- tails. Analogous to Luecken et al. 2022, we combine bio conservation (Bio), measuring the similarity between cell embeddings and ground-truth cell types or states, and batch correction (Batch), measuring how well the batch effect is removed, by aggregating these scores into total score by Total = 0.6 Bio+ 0.4 x Batch. All tables showing batch correction results are min-max scaled inside each dataset.
SSL Methods: To investigate RQI, we benchmark nine- teen existing SSL methods and divide them into four cate-
+5.4+0.4
gories: generic, domain-specifit specialized contrastive and generative methods, and baselines. SimCLR (Chen et al., 2020), MoCo (He al., 2020), SimSiam (Chen & He, 2020), NNCLR (Dwibedi al., 2021), BYOL (Grill et al., 2020), VICReg (Bardes et al., 2022), and BarlowTwins (Zbontar et al., 2021) are generic SSL architectures that we adopt to single-cell (multi-omics) data (see Figure GI for archi- tecture details). Contrastive domain-specific methods that are tailored for the single-cell data include Concerto (Yang et al.. 2022b), CLEAR (Han al., 2022), CLAIRE (Yan et al., 2023), and scCLIP (Xiong et al., 2023). Genera- tive methods include commonly used for single-cell data integration, SCVI (Lopez et al., 2018) and totalVI (Gayoso et al., 2021), which are single-cell specialized variational autoencoder- basedd methods, and single-cell foundation
Cell type annotation each dataset is divided into train (ref- erence) and test (query) data that consists of up to three held- out (experimental) batches with unseen cells (details in Ap- pendix C). We train a k-nearest neighbors (KNN) classifier with train (reference) embeddings and cell types to annotate test (query) data representation. Next, k-nearest neighbor probing (Marks et al., 2025) is used to predict cell types, and performance is evaluated using the macro-average Fl Fl-score and classification accuracy (Heryanto et al., 2024).
Missing modality prediction on multi-modal datasets - we evaluate the quality of the inferred modality by measuring the Pearson correlation between the original and predicted values, see Appendix C for more details.
4

---

Benchmarking Self-Supervised Learning for Single-Cell Data
3.2. Augmentation, Batch Normalization, and Multi-Modal Integration
4. Experiments
As detailed in Section 3, we benchmark nineteen SSL meth- ods on nine single-cell datasets derived from different tis- sues with considerable variation in data size and complexity. See Appendix B for details about the datasets. All models are trained with five unique random seeds and we report their mean performance and standard deviation.
Augmentations: We evaluate augmentations for single- cell data proposed in CLEAR (Han al., 2022) and CLAIRE (Yan et al., 2023) to investigate RQ2. The pur- pose of augmentations in contrastive SSL is to transform the original sample into two distinct views that are contrasted during training (Zhang & Ma, 2022). Multiple augmenta- tions can be applied to a data sample to improve the gener- alization and robustness of representations. The authors of CLEAR (Han et al., 2022) introduce four augmentations for scRNA-seq data, each of which we apply with 50% proba- bility: Masking, Gaussian noise, InnerSwap. and CrossOver. First, a random mask sets 20% of a cell's genes to zero, followed by additive Gaussian noise (with mean and stan- dard deviation 0.2) to 80% of genes in the cell. Then, 10% of genes are swapped within the cell (InnerSwap), before mutating 25% of gene expression values with another ran- dom cell (CrossOver). CLAIRE uses a neighborhood-baan approach: mutual nearest neighbors (MNN) in the uninte- grated space are computed for each cell across all batches. During augmentation, an inter- and an intra-batch views are computed by mutating between neighboring cells (Yan et al., 2023). We also evaluate sampling positive pairs from batch-balanced KNN (BBKNN) graph. We investigate the impact of the MNN and BBKNN augmentations on the batch correction performance.
4.1. Generic versus Specialized SSL Methods
RQI focuses on the comparison of specialized single-cell SSL frameworks and generic SSL methods. For that, we evaluate several models and two baselines on three impor- tant downstream tasks for uni- and multi-omics datasets.
Batch Correction:
The batch correction performance of all methods across five datasets is presented in Table 1. Our analysis includes two multi-modal (CITE-seq) datasets (PBMC-M and BMMC) and three single-modality (scRNA- seq) datasets (PBMC, Pancreas, and Immune Cell Atlas).
For scRNA-seg datasets, our results show that Vc is the persperffoming method that balances both batch correction and bio conservation (Table 1). seVI performance drops for the MCA and Lung datasets (Table HI) but excels for the Tabula Sapiens dataset (Table HI1). CLAIRE ranks second-best overall but tends to overcorrect batch effects at the expense of biological variance. Overall and compar- ing to other single-cell generative models, finetuned scGPT performs well for the second- largest evaluated dataset, Im- mune Cell Atlas, by scoring high in bio conservation and total, but for smaller datasets batch score is significantly lower than other methods. Zero-shot scGPT and finetuned Geneformer show unsatisfactory performance. Across all common benchmarked SSL methods, VICReg, SimCLR, and MoCo perform satisfactorily for the Pancreas dataset. However, for the PBMC and Immune Cell Atlas datasets, these SSL methods prioritize batch correction over bio con- servation as indicated by their high batch and low bio score. In comparison to the other methods, in all cases, Concerto significantly underperforms and achieves a total score lower than the baselines, PCA and SCDC. As expected, PCA shows an adequate bio conservation score since it uses raw data and captures the true biological signal.
Domain-Specific Batch Normalization (DSBN): Con- certo (Yang et al., 2022b) adapts the idea of DSBN (Chang et al., 2019), technique originally suggested for image data. DSBN helps to learn domain-specific information to produce
representations by applying separate batch normalization layers for each domain. To in- vestigate RQ3, we replace the common batch normalization with DSBN where each experimental batch (different lab- oratory experiment in the same dataset) gets its own batch normalization layer, similar to Concerto.
Multi-Modal Integration: For the multi-omics datasets PBMC-M and BMMC, we evaluate three integration meth- ods as part of RQ3. First, addition takes two embeddings of the same dimensionality (one per modality) and adds them together to get a joint representation, similar to the Con- certo (Yang et al., 2022b). Second, concatenation appends two embeddings. Third, instead of contrasting joint views of cell, two modalities of the same cell are contrasted using symmetric cross-entropy loss (Wang et al., 2019) and the CLIP approach. After training with the CLIP approach (Rad- ford et al., 2021; Xiong et al.. 2023), we concatenate two embeddings during inference.
datasets, PBMC-M and BMMC, we ob- serve that generic methods such as SimCLR, BYOL, MoCo, and VICReg are the best- perforrrfoning methods, within their category and overall (Table 1). Interestingly, MoCo over- corrects for all single- and multi-modal datasets. The re- sults show that there is room for improvement in specialized methods. Concerto, scCLIP, and scButterfly reach low batch correction results compared to general methods as MoCo or SimCLR. For PBMC-M, Concerto preserves biological variance and shows high bio score. scTEL succeeds at
For
5

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Table 1. Batch integration performance across five datasets. We show each method's hiological conservation score (Bio), hatch correction score (Batch), and total score (Total), with values computed across five runs with different random seeds. We group the methods by category (generic SSL, single-cell contrastive SSL frameworks, generative methods, and baselines). For uni-modal data (PBMC, Pancreas, and Immune Cell Atlas), the specialized encoder-decoder method scVI, the domain-specific SSL method CLAIRE, and foundation model scGPT outperform other methods, For the multi-modal datasets PBMC-M and BMMC, generic methods achieve higher scores.
PBMC-M
BMMC
PBMC
Pancreas
Immune Cell Atlas Total
Method
Bio
Batch
Total
Bio
Batch
Total
Bio
Batch
Total
Bio
Batch
Total
Bio
Batch
0.877
0.434
0.700
0.877
0.601
0.767
0.370
0.563
0.447
0.791
0.615
0.721
0.555
0.753
0.635
SimCLR
0.020
+: 0.001
+: 0012
+ 0.025
+: 0.002
0.016
0.002
0.005
+ 0.000
+ 0.002
+ 0.019
= 0.009
+ 0.020
+: 0.16
+ 0.017
0.786
0.581
0.704
0.647
0.819
0.716
0.336
0.594
0.439
0.754
0.638
0.707
0.404
0.882
0.595
MoCo
+ 0.005
= 0.016
+ 0.003
+ 0.048
+ 0.024
* 0.038
0.007
+ 0.014
+ 0.010
+ 0.008
+ 0.017
0001
+ 0.024
+ 0.022
# 0.016
0.903
0.455
0.724
0.753
0.571
0.680
0.271
0.512
0.368
0.531
0.635
0.572
0.358
0.640
0.470
SimSiam
d 0.057
1 0.029
1. 0.046
J: 0.007
d. 0.002
A 0.005
1 0.016
1. 0.002
d. 0.011
J 0.113
d 0.017
0.061
A. 0.040
1. 0.020
di 0.030
0.877
0.534
0.740
0.819
0.580
0.723
0.345
0.544
0.424
0.701
0.579
0.652
0.430
0.665
0.524
NNCLR
0.033
1: 0.004
+: 0.018
+ 0.021
+ 0.008
+: 0.016
1 0.011
+ 0.009
d: 0.010
+ 0.052
+ 0.015
+ 0.037
+: 0,028
+: 0.007
+ 0.017
0.928
0.493
0.754
0.742
0.693
0.722
0.134
0.748
0.379
0.578
0.659
0.610
0.222
0.864
0.479
BYOL
0.065
- 0.016
+ 0.033
0.043
0.016
0.019
0.017
0.076
+ 0.020
+ 0.029
+ 0.012
0.013
0.031
+ 0.009
+ 0.021
0,814
0.405
0.651
0.832
0.656
0.761
0.412
0.607
0.490
0.811
0.617
0.733
0.529
0.816
0.644
VICReg
1: 0.013
1: 0.051
0.009
0.010
0.000
di 0.006
1. 0.003
0.001
1: 0.022
di 0.012
Barlow
0,902
0.430
0.713
0.859
0.612
0.760
0.341
0.523
0.414
0.694
0.580
0.648
0.535
0.734
0.614
Twins
0.048
+: 0.014
1: 0.034
+: 0.018
+: 0.011
0.006
0.010
0.005
d: 0.004
+ 0.011
+ 0,010
0.034
+: 0,012
1 0.020
0.785
0.422
0.64
0.524
0.661
0.579
0.055
0.566
0.260
0.102
0.367
0.208
0.426
0.810
0,580
Concerto
0.002
= 0.003
+ 0.002
+ 0.019
0.008
0.015
0.000
+ 0.000
+ 0.000
+ 0.003
+ 0.000
0.002
0.014
+ 0.025
+ 0.017
0.580
0.209
0.432
0.698
0.249
0.518
0.775
0.327
0.596
CLEAR
0.000
0.002
# 0.001
+ 0.011
0.002
0.006
0.037
+ 0.008
+ 0.022
0.714
0.866
0.774
0.582
0.959
0.732
0.548
0.527
0.539
CLAIRE
= 0.009
+ 0.005
+ 0.008
+ 0.003
+ 0.014
+ 0.004
+ 0.018
+ 0.011
+ 0.011
0.643
0.402
0.546
0.638
0.194
0.460
scCLIP
0.002
0.004 + 0.001
+ 0.006
0.005
0.005
scGPT
0.440
0.469
0.451
0.473
0.168
0.351
0.380
0.516
0.435
(zero-shot)
# 0.010
+ 0.017
# 0.013
+ 0.001
= 0.003
= 0.000
+ 0.012
+ 0.014
+ 0.008
scGPT
0.940
0.514
0.770
0.873
0.345
0.662
0.979
0.485
0.781
(finetuned)
0.012
+ 0.011
+ 0.011
+ 0.003
+ 0.047
0.021
= 0.017
+ 0.019
# 0.017
Geneformer
0.024
0.462
0.199
0.004
0.437
0.177
0.013
0.265
0.114
(finctuned)
0.000
+ 0.000
+ 0.000
+ 0.000
+ 0.000
0.000
+ 0,000
+ 0.000
+ 0.000
0.702
0.391
0.577
0.781
0.297
0.587
scButterfly
0.002
+
+: 0.002
1 0.000
0.004
0.002
0.089
0.800
0.373
0.000
0.706
0.282
scTEL
+ 0.002
# 0.004
+ 0:001
+ 0.006
+ 0.005
+ 0.005
/
0.702
0.305
0.543
0.755
0.272
0.562
0.918
0.871
0.899
0.805
0.511
0.688
0.862
0.593
0.754
scVI
+ 0.002
+ 0.002
+ 0.001
+ 0.002
+ 0.001
0.002
0.015
+ 0.000
+ 0.009
+ 0.002
+ 0.007
- 0.001
+ 0.033
+ 0.013
4 0.024
0.679
0.605
0.649
0.648
0.350
0.529
0.698
0.565
0.645
SCDC
1. 0.028
1. 0018
0.023
0.002
0.024
1. 0.019
1 0.020
0.448
0.369
0.417
0.538 + 0.002
0.282
0.436
0.558
0.303
0.456
0.683
0.266
0.516
0.677
0.276
0.517
PCA
# 0.000
# 0.001
+ 0.000
+ 0.007
+ 0.002
* 0.003
+ 0.000
+: 0.002
+ 0.001
+ 0.000
+ 0.001
#: 0.000
4 0.000
+ 0.000
batch correction while failing in bio conservation and scor- ing almost zero for both multi-modal datasets. Despite the success of seVI for uni-modal data, totalVI's results are unsatisfactory compared to the generic contrastive methods. For BMMC dataset, totalVI total score (0.562) is almost twice lower than then score of best method SimCLR (0.767).
Cell Type Annotation: We assess the cell-typing perfor- mance on the single-modal scRNA-seq datasets and CLEAR augmentations for the hold-out batch of the Immune Cell Atlas dataset (Figure 2) and all study data (Table H3). For all experiments, we do not use a projector during infer- ence. Table H3 evaluates cell-typing for the Pancreas dataset
6

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Macro F1
Accuracy
Of note, Concerto and Votal do not support uni-modal
inference for multi-modal data.
Immune Cell Atlas
1.0
Missing Modality Prediction: Figure 3 shows the ability to predict missing protein values while given only RNA or gene expression (GEX) during inference. The model is trained on multi-omics data using CLEAR augmentations and concatenation to combine modalities. The standard de- viation is close to zero, see Table H5. VICReg and SimCLR outperform other methods, including specialized single-cell frameworks. We assume that Concerto, ScCLIP, and sc- TEL do not learn enough information about the secondary modality (protein) and its connection to the main modality (RNA) and, therefore, are not able to predict the missing modality. We evaluate scButterfly in two modes: averaging KNN, as done for other methods, and generating proteins directly from gene expression data. While the performance differences between the two approaches are insignificant, the scButterfly's 's Pearson correlation is unsatisfactory low compared to generic contrastive methods. High values of Pearson's 's correlations show that models effectively infer protein values from gene expression data.
Metrics
0.8
0.6
0.4
0.2
0.0
SCGPT (zero)
SCDC
SCGPT
Geneformer.
BERT
(Pt
C
(R)
LAIRE CLEAR
BYOL
Barlow Twins
SimSiar
NN
SimCLP
MoCo
ViCReg
Figure 2. Uni-modal cell-typing with one sequencing technology (10X 5' v2) of the Immune Cell Atlas as hold-out set. We train the encoder and classifier. The finetuned scGPT and Geneformer perform the best, while the generic VICReg method is close third. The methods are grouped by category (baselines, specialized generative, specialized contrastive, and generic).
where unique batches were used as hold-out data. The best-performing methods are VICReg, CLEAR, and in rare cases finetuned single-cell foundation models (scFMs). All generic SSL methods perform well, together with VI, which takes additional information as input. Although fine- tuned scFMs achieve adequate accuracy in most cases, they perform the best only for two datasets, the Immune Cell Atlas and the Xin study from the Pancreas, which are larger than the other three evaluated datasets. CLAIRE shows competitive results for the Pancreas dataset but falters in the second-biggess benchmarked dataset Immune Cell At- las. Additionally, CLAIRE and scFMs have a significantly higher computational load than the other methods. PCA and zero-shot scGPT perform unsatisfactorily.
Summary and Findings: Overall, specialized SSL meth- ods designed for single-cell analysis have not demonstrated clear advantages over general- purpose approaches, except for VI, finetuned scGPT, and CLAIRE. We attribute the su- perior performance of scVI, finetuned scGPT, and CLAIRE to the fact that these methods leverage experimental batch information during training. For multi-omics data, the best models are SimCLR and VICReg. Our findings indicate that current single-cell SSL methods such as scCLIP totalVI need improvement for multi-modal downstream tasks, as they do not yet surpass or compete with generic architec- tures in performance.
Table H4 shows cell-typing performance for multi-modal embeddings, where we integrate modalities through concate- nation and train with CLEAR augmentations. We evaluate models using either both modalities (c.g., RNA and protein) or just the main modality (e.g., RNA) to assess whether representations capture information about the second modal- ity and if single main modality is sufficient during infer- ence when the second modality is unavailable. scButterfly slightly outperforms VICReg and SimCLR, that show com- petitive results, with scButterfly leading overall performance (Table H4). All generic contrastive methods achieve good accuracy and outperform specialized contrastive methods like Concerto and scCLIP. While totalVI struggles with batch correction, performs well in cell-typing. All models except scCLIP show better performance with multi-omics data than single modality, though the performance drop for single-modality inference is minimal. Notably, scCLIP ap- pears to treat the second modality as noise. We conclude that scButterfly and generic contrastive models can be used for a single modality inference if the second modality is missing.
PBMC-M
BMMC
Pearson Mean
1.0
0.8-
0.6
0.4
SCCLIP SCTEL
Concerto
scButterfly (gen)
scButterfly (KNN)
MoCo
BYOL SimSiam
Barlow Twins SimCLR, vicReg
NNCLR
Figure 3. Missing modality prediction for models trained on the multi-moda datasets, PBMC and BMMC. We show the average Pearson correlation between the original and inferred missing modality: protein for PBMC-M and ADT (protein abundance) for BMMC. The methods are sorted from worst (left) to best (right) within group (specialized contrastive, generative, and generic).
7

---

Benchmarking Self-Supervised Learning for Single-Cell Data
4.2. Ablation Study
SimCLR
MoCo
NNCLR
RQ2 investigates how hyperparameters and augmentations impact the performance of single-cell SSL. We conduct hyperparameter tuning for all generic methods using two datasets: HIC and MCA. For the mentioned frameworks, we use the hyperparameters proposed in the respective original papers. For the generic methods, we focus on augmen- tations, the representation dimensionality, the projection dimensionality, and the temperature parameter.
MCA
HIC
PBMC-M
BMMC
0.8
0.7
0.6
0.5
Total Batch
0.7
0.6-
80
88
0.5
0.7-
Representation Dimensionality: We perform grid search over the representation dimensionality for the HIC and MCA datasets, evaluating the batch correction performance (de- tails in Appendix D). We train all models with embedding dimensions {8. 16, 32, 64, 128, 256, 512, 1024]. Models are ranked according to the SCIB (Luecken et al., 2022) to- tal score, which is min-max scaled across all models. Our findings indicate that lower dimensionalities of 64 and 128 consistently lead to the best performance across all con- sidered methods, while the larger dimensionality of 1024 achieves similar but requires more training time and mem-
0.6
0.1
0.1
Temperature 0.1
0.1
Figure 4. Temperature impact on the loss of three contrastive meth- ods on four datasets (columns). Bio conservation, batch correction, and total scores represented the y-axis. The results are not min-max scaled for easier comparison. Overall, smaller tempera- ture leads to better data integration.
ory (Figure G4 in Appendix). Given these observations, we
HIC
MCA
PBMC
adopt the embedding size of 64 for subsequent experiments.
Augmentation
GaussianN. InnerSwap Masking
67
64
9'0
Projector Dimensionality: To learn more robust represen- tations, sule--sue models may benefit from projection heads (Xue et al., 2024). We investigate the impact of pro-
B8KNN
62
57
67
64
C6
41
MNN
CrossOver
64666 66
69
04
jection dimensionality during training by introducing scale factor. At inference time, the projection head is discarded, and only the encoder is used. For contrastive methods, the projection size scaled down by this factor, while for non-contrastive methods, it is scaled up by the same fac- tor (see Appendix D). Our results reveal that the projector effect is ambiguous for most models (Figure G5). How- ever, BarlowTwiin BYO,, and VICReg show an improved performance with larger scaling factors.
cussirss Sapp
Ma Masking
BBKNN
CrassOver
MNN
ver
Gausslant
Masking
B8KNN
CrossŌver ver
MNN
Gaussiant
Innerswa wap
MNN B8X
CrassOver
Mast
Augmentation 2
Figure 5. Evaluation of individual and combined data augmenta- tions for the VICReg method based on total score for hatch correc- tion. Diagonal entries correspond to single augmentation, and off- entries correspond to the two sequentially applicd augmentations. Hyperparameters are based on ablation results (Ta- ble 1110), evaluation for SimCLR and MoCo are in Figure G7.
Figure 4 shows the temperature to effect for SimCLR, MoCo, and NNCLR models. We eval- uate 1 E (0.1, 1, 10) using SCIB-METRICE scores. Overall, lower 1 values lead better scores. The Py default value (t" = 0.5) performs well across all models and datasets (used for subsequent experiments). For ScRNA- seq datasets, MCA and HIC, batch correction metric de- creases with increasing tº, except for NNCLR model and HIC dataset. For HIC data, higher 1 leads to uncorrected batch effects, while for MCA data, it results in clustering of similar cell types that expected to be near but not fully mixed together (e.g.. CD4 T and CD8 T cells). For multi-modal datasets. PBMC-M and BMMC, to effects are inconsistent. For PBMC-M, MoCo produces extremely similar results with all evaluated to, while SimCLR and NNCLR mix batches better with higher 19, e.g., with 1 of 10 cell types are separated with mixed batches. For BMMC, lower o achieves better batch correction.
Temperature Impact:
Augmentation Ablation: The space of augmentations in the single-cell domain can be split into: random trans- formations (Han et al., 2022; Richter et al., 2024b) and hawbbborood-bassedd transformations (Yan et al., 2023; Liu et al., 2024). We perform an ablation for all studied aug- mentations and optimize hyperparameters for each (see Ap- pendix E). To study how augmentations affect each other, we train VICReg, SimCLR, and MoCo models with combi- nations of two augmentations. We choose these models due to their consistently good performance. Random masking the best rest-performing augmentation alone and combined with others (Figure 5). Additionally, CrossOver performs com- petitively, especially for the SimCLR model (Figure G7).
Summary and Findings: Higher- digeer-di embeddings and lower temperatures enhance consistency and perfor- mance. Larger representations have better bio conservation,
8

---

Benchmarking Self-Supervised Learning for Single-Cell Data
smaller- batch correction. However, the optimal embedding size is 64 or 128. Among various data augmentation tech- niques, masking proves most effective and surpasses even sophisticated hoolgyy-spccife approaches that incorporate batch information, such as MNN and BBKNN.
et al., 2021; Xiong al., 2023). For each modality, we train model with CLEAR (Han et al., 2022) augmentationsand discard the projector during inference. See Appendix F for details. Table H9 shows that concatenation is the best in- tegration method. Addition and concatenation show high results in bio conservation, while the CLIP-based approach performs better in batch correction.
4.3. Impact of Batch Normalization and Multi-Modal Integration Proposed for Image Data
Previously recommended tech- niques, such keeping the projector using DSBN, fail to enhance performance. For combining multiple modalities, concatenation turns out to be the most effective approach.
Summary and Findings:
In the following, we address RQ3 and study the impact of common SSL techniques, such as retaining the projector and domain-specific batch normalization (Chang et al., 2019).
Retaining Projector: To evaluate the impact of the pro- jector during inference, we train model consisting of encoder and projector with CLEAR augmentations, and evaluate it with/without projection. Although it is com- mon to only use the encoder during inference to create an embedding (Chen et al., 2020; Chen & He, 2020), the pro- jector is also leveraged in the single-cell community and we compare the two approaches. First, we analyze how retaining the projection layer affects batch integration for single-modal data (Table H6). While using the projection layer slightly improves batch correction, it generally reduces biological conservation and total (which weights bio conser- vation more) scores. For example, the total score decreases from 0.625 to 0.608 for SimCLR on the MCA dataset. These findings suggest using only the encoder during inference, rather than the combined encoder and projector. The former better preserves biological signals despite slightly worse batch correction performance. Second, we observe sim- ilar trend on multi-modal datasets as on ScRNA-seq data (Table H7). The effects are less consistent and conclusive (scores change among models/datasets), and only MoCo's batch correction benefits from projection. Remarkably, Con- certo uses encoder and projector during inference but has comparably unsatisfactory batch correction performance.
5. Conclusions
We introduced a comprehensive benchmark, scSSL-Bench, for self-supervised learning on uni- and multi-modal single- cell data. First, we observe that specialized single-cell SSL methods perform better than generic methods for uni-modal data and underperform for multi-modal data (RQ1). The best scRNA-seq single-modal modal data integration methods are sc VI, CLAIRE, and the finetuned scGPT, all specialized for single-cell data. Generative VI and the single-cell founda- tion model scGPT significantly outperform all SSL methods, while CLAIRE shows good scores only for subset of the datasets. For multi-omics data, the generic methods Sim- CLR and VICReg perform the best and even outperform all other methods in the cell type annotation and missing modality prediction tasks for single-modal data. According to our findings, there is a need for improving existing and developing new multi-modal specialized SSL methods since current existing frameworks do not outperform generic ar- chitectures, and multi-modal integration turned out to be more difficult task than uni-modal (RQ1). Second, we conclude that masking augmentation leads to the biggest improvements alone and in combination with other types of augmentations, and moderately-size to large embedding sizes lead to better results (RQ2). Third, retaining the projec- tion head or applying domain-speciff batch normalization do not significantly influence the scores and rather degrade the total data integration score by achieving a lower bio conservation and higher batch correction indicating primar- ily regularization potential (RQ3). Finally, our benchmark offers a standardized framework for assessing new SSL. methods, enabling researchers systematically evaluate and compare their approaches against established baselines.
Domain-Specific Batch Normalization (DSBN): Inspired by Concerto (Yang et al., 2022b) and common practices from computer vision applications, we evaluate whether models benefit from DSBN (Chang et al., 2019). Although the Concerto (Yang et al., 2022b) manuscript discusses the usage of DSBN, the publicly available code does not ap- ply DSBN. Therefore, we evaluate DSBN only for generic methods. Table H8 shows reduced total performance when leveraging DSBN compared to standard batch normaliza- tion. For HIC dataset, DSBN leads to slightly better batch correction but worse bio conservation. However, it is not the case for the MCA dataset.
A Acknovledgements
The authors thank Sebastian Baunsgaard, Dmitry Kobak, and Thomas Sutter for their insightful comments and con- structive feedback on the manuscript. We thank the three anonymous reviewers for their thorough reviews, which significantly improved the quality of this paper.
Multi-Modal Integration Methods: In Table H9, com- pare three ways to combine multiple modalities of a cell: wlement-wise addition of uni-modal embeddings (Yang et al., 2022b), concatenation of uni-modal embeddings, and multi- modal contrastive learning with the CLIP objective (Radford
9

---

Benchmarking Self-Supervieed Learning for Single-Cell Data
Computational data analysis was performed Leonhard Med secure trusted research environment at ETH Zurich and at the BIFOLD Hydra cluster.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In ICML, pp. 1597-1607, 2020.
FB is supported by the Swiss National Science Foundation (SNSF) (grant number 205321-207931).
Chen, X. and He, K. Exploring Simple Siamese Represen- tation Learning, 2020. URL abs / 2011 10566.
Impact Statement
Chen, Y., Fan, X., Shi, C., Shi, Z., and Wang, C. A joint analysis of single cell transcriptomies and proteomics using transformer. npj Systems Biology and Applications, 11(1):1, Jan 2025. ISSN 2056-7189. doi: 10. 10.1058/4415 40-024-00484-9, URL https: // doi. 103
This work provides systematic benchmark of self- supervised learning (SSL) methods in single-cell genomies; we evaluate nineteen approaches across batch correction, cell typing, and missing modality prediction tasks. Our results offer practical guidelines for applying SSL to bio- logical data, advancing computational tools for single-cell analysis. The implications span from biomedical research to personalized medicine, where improved data integration enables better diagnostic and therapeutic strategies. By stan- dardizing evaluation protocols, this benchmark promotes reproducibility and cross-disciplinary collaboration.
Conde, D., Xu, C.,, Jarvis, L. B., Rainbow, D. B., Wells, S. B., et al. Cross-tissue immune cell analysis reveals dissue-specfii features in humans. Science, 376(6594): eab15197, 2022. doi: 10.1 12//science URL https: http://wwwwwnn..cc///../..c.wete www.aciienneiaieeeee
References
Cui, H., Wang. C., Maan, H.,, Pang, K., Luo, F., Duan, N., and Wang, B. scGPT: toward building a foundation model for single-cell multi-omics using generative AI. Nature 2024.
Bardes, A., Ponce, J., and LeCun, Y. VICReg: Beriane-IIvvariace-CCoe Covariance Regularization for Self- Supervised Learning. In ICLR, 2022.
Methods,
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Lan- guage Understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, Min- neapolis, Minnesota, June 2019a. Association for Compu- tational Linguistics. doi: 10.18653/w1/N1199442 URL wtt.lloggoorggg19111//
Baysoy, A., Bai, Z., Satija, R., and Fan, R. The technological landscape and applications of single-cell multi-omics. Nature Reviews Molecular Cell Biology, 24(10):695-713, Oct 2023. ISSN 1471-0080. doi: 10. 10.1038//41580-023 -00615-w. URL https://t/t./..... /tt.dooorgggooooooooooooooooooo. 103////3/// 0.:000022000111
Büttner, M., Miao, Z., Wolf, F. A., Teichmann, S. A., and Theis, J. A test metric for assessing single-cell RNA- seq batch correction. Nature Methods, 16(1):43 Jan 2019. ISSN 1548-7105. doi: 00.1038/441522-08888
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 confer- ence of the North American chapter of the association for computational linguistics: human language technologies, volume I (long and short papers), pp. 4171-4186 2019b.
4-1. URL https: /tw.//oiio//// /10 1038/ 1tt.:/11..11///1111/ -01 8- 0254-1.
Cao, J., Nai, R., Yang, Q. Huang, J., and Gao, Y. An empir- ical study on disentanglement of negative-free contrastive learning. In NeurlPS, 2024a. ISBN 9781713871088
Cao. Y., Zhao, X., Tang, S., Jiang, Q. Li, S., Li, S., and Chen, S. scButterfly: versatile single-cell cross- modality translation method via dual-aligned variational autoencoders. Nature Communications, 15(1):2973, Apr 2024b. ISSN 2041-1723. doi: 10. 1038/s41467-0244 7418-x. URL https: /doi /tt.//ii..ooo..////// 10. 1038/s414 02--2-2-471---
Ding, J., Adiconis, X., Simmons, S. K., Kowalczyk, M. S., Hession, C. C., et al. Systematic comparison of single- cell and single-nucleus ma-sequeniing methods. Nature Biotechnology, 38(6):33-7746, Jun 2020. ISSN 1546- 1696. doi: 10. 100100088/4415577-000006666 URL https: //doi org/ 10. 00w..01100//11000000-00--0-
8.
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., and Zis- serman, A. With a Little Help from My Friends: Nearest- Neighbor Contrastive Learning of Visual Representations,
Chang, W.-G., You, T., Seo, S., Kwak, S., and Han, B. Domain-specific batch normalization for unsupervised domain adaptation. In Proceedings of the IEEF/CVF conference on Computer Vision and Pattern Recognition, pp. 7354-7362, 2019.
10

---

Benchmarking Self-Supervised Learning for Single-Cell Data
2021. URL www.h:tst//.ai..ooooo 4548.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. Mo- mentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9738-9738, 2020.
Eraslan, G., Drokhlyansky, E., Anand, S., Fiskin, E., Sub- ramanian, A., Slyper, M., Wang, J., Van Wittenberghe, N., Rouhana, J. M., Waldman, J., et al. Single-nucleus cross-tissue molecular reference maps toward understand- ing disease gene function. Science, 37666594) 2022.
Heryanto, Y. D., Zhang, Y.-z., and Imoto, S. Predicting cell types with supervised contrastive learning cells and their types. Scientific Reports, 14(1):430, Jan 2024. ISSN 2045-2322. doi: 10.10888/41598--23355000035 https:/ /tt./oo..gg//////.oooo//oo//...t 10. 10388 s 023-5 0185-2 2.
URL
Gayoso, A., Steier, Z., Lopez, R., Regier, J., Nazor, K. L., Streets, A., and Yosef, N. Joint probabilistic modeling of single-cell multi-omic data with totalVI. Nature Methods, 18(3):272-288) Mar 2021. ISSN 1548-7105. doi: 10.1 038//41199-0000--00050xxx URL https:/ttttto /ttp://oo..r//// 01050-x.
Heumos, L., Schaar, A. C., Lance, C., Litinetskaya, A., Drost, F., Zappia, L., Lücken, M. D., Strobl, D. C., Henao, J., Curion, F., et al. Best practices for single-cell amalysis across modalities. Nature Reviews Genetics, 24(5):500- 572, 2023.
Geiping, J., Garrido, Q., Fernandez, P., Bar, A., Pirsiavash, H., LeCun, Y., and Goldblum, M. A Cookbook of Self- Supervised Learning, 2023. URL https://arxiv: org/abs hts/abb//3044. 2304. 12210.
Hu, C., Li, X., Liu, D., Wu, H., Chen, X., Wang, J., and Liu, X. Seacher-Student Architecture for Knowledge Distillation: A Survey, 2023. URL https: //arxiv. org/ 2abs/2308 64268.
Grandi, F. C., Modi, H., Kampman, L., and Corces, M. R. Chromatin accessibility profiling by ATAC-seq. Nature Protocols, 1766):111-11552 Jun 2022. ISSN 1750-2799. doi: 10. 0011033//41199--22-000222 //doi welli.oog//e owg. 10t....00//1111 / 0a01://4411590022-00606- 022-0062222222222 9.
Jing, L., Vincent, P., LeCun, Y., and Tian, Y. Understand- ing Dimensional Collapse in Contrastive Self-supervised Leaming. In International Conference on Learning Rep- resentations, 2022. URL https: wpw.//ppe.........eeeeee ?
URL https:
Grill, J..B., Strub, F., Altché, F., Tallec, C., Richemond, P.,
Buchatskaya. E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to seff-super learning. NeurlPS, 33: 21271-21284, 2020.
Jones, R. C., Karkanias, J., Krasnow, M. A., Pisco, A. O., Quake, S. R., et al. The Tabula Sapiens: A multiple-organ, single-cell transcriptomic atlas of humans. Science, 376 (6594))cabl4896, 2022. doi: 10. 10.1 10.1126/ccieec URL httph:tt/////. www. science org arg/doi/bb// 10. 1t.1126/scienn ab14896.
Han, W., Cheng, Y., Chen, J., Zhong, H., Hu, Z., Chen, S., Zong, I.., Hong, I.,, Chan, T.-F., King, I., Gao, X., and Li, Y. Self-supervised contrastive learning for integrative single cell RNA-seq data analysis. Briefings in Bioinfor- matics, bbb5)3b 09 2022. ISSN 1477-4054. doi: 10.1093/bib/bbacc77.
Kingma, D. P. and Ba, J. Adam: Method for Stochastic Optimization, 2017. URL https: ptttpp:://axiiii //tx/arxiii.///////////////aaav/ org/ab s/1412.699 6980.
Hao, Y. Hao, S., Andersen-Nissen, E., Mauck, W. M., Zheng, S., Butler, A., Lee, M. J., et al. Integrated analysis of multimodal single-cell data. Cell, 184(13): 3573-3587.e22 2021. ISSN 0092-8674. doi: https: //00..og/10..0011..cel....2.2..0
Lähnemann, D., Köster, J., Szezurek, E., McCarthy, D. J., Hicks, S. C., Robinson, M. D., Vallejos, C. A., Campbell, R., Beerenwinkel, N., Mahfouz, A., et Eleven grand challenges in single-cell data science. Genome biology, 21(1):1-35, 2020.
URL https:
//tw: www.:ssciicceciic....ocm.c com/ cwamm//cienneie scieecceerrerriiee cle/pp11/S00228544100000
Lance, C., Luecken, M. D., Burkhardt, D. B., Cannoodt, R., Rautenstrauch, P., Laddach, A., Ubingazhibov, A., Cao, Z.-J., Deng, K., Khan, S., Liu, Q., Russkikh, N., et al. Multimodal single cell data integration challenge: Results and lessons learned. In NeurIPS Competitions and Demonstrations Track, volume 176, pp. 176. 2022. URL https: http///oceeddeee proceedings.... s s/v176/1ance222 html.
Hao, Y., Stuart, T., Kowalski, M. H., Choudhary, S., Hoff- man, P., et al. Dictionary learning for integrative, multi- modal and scalable single-cell analysis. Nature Biotech- nology, 42(2):293-304 Feb 2024. ISSN 1546-1696. URL https: //doi. org /10 /ttp///l....///...0////////111//
doi: 10.
ngs mmlg..ml... m1
pres

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Li, Y., Zhang, D., Yang, M., Peng. D., Yu, J., Liu, Y., Lv, J. Chen, I.., and Peng. X. scBridge embraces cell hetero- geneity in single-cell RNA-seq and ATAC-seq data inte- gration. Nature Communications, 14(1):6045, Sep 2023. ISSN 2041-1723. doi:
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., and Roth, D. Recent advances in natural language processing via large pre-trained language models: A survey, ACM Computing Surveys, 56(2):1-40, 2023. doi: 10.1145/3605943, URL
URL https::///.o perdiiiorre / 10. 1008000888000 23-41795-5.
https: //to:://dd... org/ 11 11111...1115/355933
Pearson, K. LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin philosophical magazine and journal of science, 2(11): 559-572, 1901.
Li, Y. Lin, Y., Hu, P., Peng, D., Luo, H., and Peng, X. Single-Cell RNA-Seq Debiased Clustering via Batch Ef- feet Disentanglement. IEEE Transactions on Neural Ner- works and Learning Systems, (33(8)111771-111 doi: 10.1109/TN 100//NNSS22202222200000 2023.3260003
2024.
Polański, K., Young, M. D., Miao, Z., Meyer, K. B., Te- ichmann, S. A., and Park, J.-E. BBKNN: Fast Batch Alignment of Single Cell Transcriptomes. Bioinformat- ics, 2019. doi: 10.0093bioinfommacccs/o222...
Liu, J., Zeng, W., Kan, S., Li, M., and Zheng, R. CAKE: a flexible self-supervised framework for enhancing cell visualization, clustering and rare cell identification. Brief- ings in Bioinformatics, 3bbad455,, 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In JCML, pp. 8748-88 8863,,873,480 2021.
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., and Yosef, N. Deep generative modeling for single-cell transcrip- tomics. Nature Methods, 15(12):1033-100 Dec 2018. ISSN 1548-7105, doi:
Richter, T., Bahrami, M., Xia, Y., Fischer, D. S., and Theis, F. J. Delineating the effective use of self-supervised learn- ing in single-cell genomics. Nature Machine Intelligence, Dec 2024a. ISSN 2522-5839. doi: 10. 10.1038/<42256-222 -0994-3. URL https: /ww.//oioo//// org wtt.o/////...//////.... 1038/0088800 2256 024 024-0024--004-33
Lotfollahi, M., Naghipourfar, M., Luecken, M. D., Khajavi, M., Büttner, M., Wagenstetter, M., Avsec, Ž., Gayoso, A., Yosef, N., Interlandi, M., Rybakov, S., Misharin, A. V., and Theis, F. J. Mapping single-cell data reference atlases by transfer learning. Nature Biotechnology, 40(1): 121-130, Jan 2022. ISSN 1546-1696. doi: 10.1038/s4 1587-021-01001-7, URL https: / otg/ 1188/444587-021-100001
Richter, T., Bahrami, M., Xia, Y., Fischer, D. S., and Theis, F. J. Delineating the Effective Use of Self-Supervised Learning in Single-Cell Genomics. bioRxiv, 2024b. doi: 10. 10.100//20224022.66556644 URL https: orxiv. org http://///..oteeeeaaa/////////// 024.02. 12..0..16.88662
Luecken, M., Burkhardt, D., Cannoodt, R., Lance, C.,, Agrawal, A., Alice, H., Chen, A., Deconinck, L., De- tweiler, A., Granados, A., et al. A sandbox for prediction and integration of DNA, RNA, and proteins in single cells. In Vanschoren, and Yeung, S. (eds.), NeurIPS Track on Dutasets and Benchmarks, volume 2021. URL httpe///aattettt-eeeccaarreepppp
Schiappa, M. C., Rawat, Y. S., and Shah, M. Self-supervised learning for videos: A survey. ACM Computing Surveys, 55(13s):1-37, 2023. doi: 10.1145/3577925. URL https://doi.rrs 1ttp://1..14//3577900
gs os..cerrpp..cc/pppeeefffll//ep/p /fL1e/1188006644333114448bbb88 22 22-Paper-roundd. .df.
Sikkema, I.., Ramirez-Suústeguu C., Strobl, D. C., Gillett, T. E., Zappia, L., Madissoon, E., et al. An integrated cell atlas of the lung in health and disease. Nature Medicine, 2966) Jun 2023. ISSN 1554-170X. doi: 10. 10.1038/441511-022-0223177 URL wtttt:::////.. /oii.
Luecken, M. D., Büttner, M., Chaichoompu, K., Danese, A., Interlandi, M., Müller, M. F., Strobl, D. C., Zappia, L., Dugas, M., Colomé-Tatche M., et al. Benchmark- ing level data integration in single-cell genomics. Nature Methods, 19(1):11-50, 2022.
Slyper, M., Porter, C. B., Ashenberg. O., Waldman, J., Drokhlyansky, E.,, Wakiro, I., Smillie, C., Smith-Rosarioo G., Wu, J., Dionne, D., et al. A single-cell and single- nucleus RNA-Seq toolbox for fresh and frozen human tumors. Nature Medicine, 26(5):792 802, 2020.
Marks, M., Knott, M., Kondapaneni, N., Cole, E., Defra- eye, T., Perez-Cruz, F., and Perona, P. A closer look at benchmarking self-supervised pre-training with image classification. International Journal of Computer Vision Apr 2025. ISSN 1573-1405. doi: 10. 10.1007//112633-05 -02402-w. w. URL https:////........t. : /tt.doooorooooooo.oooooooooooeo. 1263- 0263-025-02402-11
Sohn, K. Improved deep metric learning with multi-class N- N-pair loss objective. In NewrIPS, pp. 1857-1865, 2016. ISBN 97815 108338819
12

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Stoeckius, M., Hafemeister, C., Stephenson, W., Houck- Loomis, B., Chattopadhyay, P. K., Swerdlow, H., Satija, R., and Smibert, P. Simultaneous epitope and tran- scriptome measurement in single cells. Nature Meth- ods, 14(9):865-88))) Sep 2017. ISSN 1548-7105. doi: 10.1033/mmeth URL https::tp//.mm 4380.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, t.., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Wang, Y., Ma, X., Chen, Z., Luo, Y., Yi, J., and Bailey, J. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF international conference computer vision, pp. 322-330, 2019.
Susmelj, I., Heller, M., Wirth, P. Prescott, J., Ebner, M., and et al. Lightly, 2023. URL https: htttts://githbbs om/ om/ httpllighhlllaaaaa lli
Wolf, F. A., Angerer, P., and Theis, F. J. SCANPY: large- scale single-cell gene expression data analysis. Genome Biology, 19(1):15, Feb 2018. ISSN 1474-760X. doi: 10. 10.118666130099-077---133833 g/10. 1110.11800 11w./11665500 s13059 01 1382-0.
Swanson, E., Lord, C., Reading, J., Heubeck, A. T., Savage, A. K., Green, R., Li, X.-j.. Torgerson, T. R., Bumol, T. F. Graybuck, L. T., and Skene, P. TEA-seq: a trimodal assay for integrated single cell measurement of transcription, epitopes, and chromatin accessibility. URL https: www http://wwww.ooooooooooooootttt/e 2//220/11/16/22022 09w.://2222........222. 2483887
URL https: : /dwid.i.. or
Xiong, L., Chen, T., and Kellis, M. scCLIP: Multi-modal Single-cell Contrastive Learning Integration Pre-training, In NeurlPS 2023 Al for Science Workshop, 2023. URL https://openzer: ope/review net /ateefforee forum?rrmmmmmmmm hetet/Eorrmmmdddkkem K 52Hxct.
bioRxiv, 2020. doi:
Tang, F., Barbacioru, C., Wang. Y. Nordman, E., Lee, C., Xu, N., Wang, X., Bodeau, J., Tuch, B. B., Siddiqui, A., Lao, K., and Surani, M. A. mRNA-Seq whole- transcriptome analysis of single cell. Nature Meth- ods, 675):377-382, May 2009. ISSN 1548-7105. doi: 10. 10.1008//meth. 1315. URL https::////mm. http://ooooooooooo org/10 1
Xu, C.,, Lopez, R., Mehlman, E., Regier, J., Jordan, M. I., and Yosef, N. Probabilistic harmonization and annotation of single-cell transcriptomics data with deep generative models. Molecular Systems Biology, 17(1):e9620, 2021. doi: http:://bb..orr///.55553222 20200990. URL https: //wp:///wwwwwt
httpps///ww.eembopresssss ewwopress.eeeeeeeeoooooooo wwt./org///oo/ooooooo .15552 11:52//mbb.. 20209620.
Tang. Z., Chen, G., Chen, S., Yao, J., You, L.., and Chen, C. Y.CC. Modal-nexus auto-encoder for multi-modality cellular data integration and imputation. Nature Com- munications, 15(1),0021, Oct 2024. ISSN 2041-1723. URL https: ///ooi.or///// 1038 /0t////144447000 022-11667-22243555--- 6.
Xue, Y., Gan, E., Ni, J., Joshi, S., and Mirzasoleiman, B. Investigating the Benefits of Projection Head for Rep- resentation Learning. In ICLR, 2024. URL https:
doi:
/ //arxiv....////////// www. abs 240.344333.00 11391.
Yan, X., Zheng, R., Wu, F., and Li, M. CLAIRE: contrastive learning-batt batch correction framework for better bal- ance between batch mixing and preservation of cellular heterogeneity. Bioinformatics (Oxford, England), 39, 02 2023. doi: 10.:09//bioinfomaaiiss/baa
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., Al Sayed, Z. R., Hill, M. C., Mantineo, H., Brydon, E. M., Zeng, Z., Liu, X. S., et al. Transfer learning enables predictions in network biology. Nature, 618(7965):616- 624, 2023.
Yang, F., Wang, W., Wang, F., Fang, Y., Tang, D., Huang, J., Lu, H., and Yao, J. ScBERT as a large-scale pretrained deep language model for cell type annotation of single- cell RNA-seq data. Nature Machine Intelligence, 4(10): 852-866, 2022a.
Toma, P., Ovcharenko, O., Daunhawer, I., Vogt, J. E., Bark- mann, F., and Boeva, Benchmarking self-supervised learning for single-cell data. In NeurIPS 2024 Workshop: Selp-Super Searninn-TThon and Practice, 2024.
Tran, H. T. N., Ang, K. S., Chevrier, M., Zhang, X., Lee, N. Y. S., Goh, M., and Chen, J. A benchmark of batch- effect correction methods for single-cell RNA sequencing data. Genome Biology, 21(1):12, Jan 2020. ISSN 1474- 760X. doi: 11800.11166110 0000110011000001111119900000 //doi. ///d/i.org/1// /00
Yang. M., Yang. Y., Xie, C., Ni, M., Liu, J., Yang, H., Mu, F., and Wang. J. Contrastive learning enables rapid mapping multimodal single-cell atlas of multimillion scale. Nature Machine Intelligence, 4(8):696-709 2022b.
URL https:
Yu, X.. Xu, X., Zhang, J., and Li, X. Batch alignment of single-cell transcriptomies data using deep metric learn- ing. Nature Communications, 14(1):960, 2023. doi:
van den Oord, A., Li, Y., and Vinyals, O. Representation Learning with Contrastive Predictive Coding, 2019. URL https: / rrxiv. www. abs// 807 03748.
10.
13

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Zbontar, J., Jing, L.,, Misra, LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In International conference machine learn- ing. Pp. 12310-18 121122000. PMLR, 2021.
Pattern Recognition, pp. 16650-16659, 2022.
Zhang, Z., Mathew, D., Lim, L.,., Mason, K., Martinez, M., Huang, S., Wherry, E. J., Susztak, K., Minn, A. J., Ma, Z., and Zhang, N. R. Recovery of biological signals lost single-cell batch integration with cellanova. Nature Biotechnology, Nov 2024. ISSN 1546-1696 doi: 10.103
Zhang, J. and Ma, K. Rethinking the augmentation module in contrastive learning: Learning hierarchical augmen- tation invariance with expanded views. In Proceedings of the IEEE/CVF Conference on Computer Vision and
8/$41587700240024633-11 URL https: / adoi.org 10. 0001108//1411177-222--0423
14

---

Benchmarking Self-Supervised Learning for Single-Cell Data
A. Generic Self-Supervised Methods
In contrastive SSL, different augmentations/modlliiis of the same instance are used to create positive pairs (i.e., similar examples), while pairs of distinct instances represent negative pairs (i.e., dissimilar examples) (Chen et al., 2020). Non- contrastive methods, also called negative- contrastive learning, leverage only positive pairs (Cao et al., 2024a).
Contrastive Methods: A common framework for contrastive learning is SimCLR (Chen et al., 2020). Originally, SimCLR applies three image-data-specific augmentations to create positive/negative pairs and maximizes agreement between different augmented views via a lemperature-scaled cross-entropy loss (NTXent) (Sohn, 2016) loss. MoCo (He et al., 2020) uses an additional momentum encoder for the augmented views. The key advantage of Moco is improved memory efficiency. NNCLR (Dwibedi et al., 2021) samples nearest neighbors of an instance to define positive pairs. Both MoCo and NNCLR contrast via InfoNCE (van den Oord et al., 2019) loss. SimCLR (Chen et al., 2020), MoCo (He et al., 2020), and NNCLR (Dwibedi et al., 2021) rely on positive and negative samples.
The emergence of non-contrastive methods was facilitated by an improved understanding of instabiliitess during model training. BYOL (Grill et al., 2020) uses online (trainable) and target (fixed) networks to train a representation. Online and target networks receive actual samples and augmented views respectively. The target network is updated using exponential moving averages of weights of the previous online networks. BYOL (Grill et al., 2020) minimizes the similarity of two representations produced by both networks. Additionally, using momentum encoders in MoCo (He et al., 2020) and BYOL (Grill et al., 2020) helps against dimensionality collapse from, for instance, the lack of negative pairs. Dimensionality collapse appears when embeddings span in a lower-dimensional subspace instead of the entire space (Jing et al., 2022). SimSiam, like SimCLR, uses encoder with shared weights to process two augmented views. One encoder output (left) is fed to an additional predictor network before maximizing similarity, while the other output (right) is used directly. The encoder and predictor are updated with only left path gradients. SimSiam (Chen & He, 2020) and B YOL (Grill et al., 2020) use a predictor network to achieve better performance and avoid representation collapse without leveraging negative pairs. Barlow Twins (Zbontar et al., 2021) uses augmentations to train two identical networks and calculates a cross-correlation matrix (Zbontar et al., 2021) 1) between the trained representations to reduce redundancy of the embeddings. VICReg (Bardes et al., 2022) is a joint embedding architecture with variance, invariance, and covariance regularization. The authors introduce regularization terms to the loss to control the variance of embeddings and decorrelate latent variables.
Methods:
B. Datasets
All datasets used in our benchmark are publicly available.
Human Immune Cells (HIC): This dataset comprises 33,506 cells and includes 12,303 genes from ten different donors assembled by Luecken et al.(2022) from five studies. One study derived cells from the human bone marrow and the other four from the human peripheral blood. There are 16 cell types annotated in the dataset. Availability: https: / //ooi... woro/r///// C 6084 4 /m9. f gshare. 12420968. 8
Mouse Cell Atlas (MCA): This dataset comprises 6,954 cells collected across two studies (Tran et al., 2020) with the first study consisting of 4,239 cells and the second batch containing 2,715 cells. Three different sequencing protocols were used. The harmonized dataset contains 51,817 genes and eleven cell types. Availability: https 10760158
figsha re. cwm/ files/ wwwle/eleeeellll1... 10351110 and https: /esttp::/nddwniii http::///dowwonn/eeeeee figshare. com wtl/fflls////// fillee//eeee/5e
Peripheral Blood Mononuclear Cells (PBMC): Collected by Ding et (2020),) this dataset contains 30,449 cells from two patients and includes 33,694 genes. Cells were sequenced with seven different protocols (10x Chromium (v2). 10x Chromium (v3), Drop-seq, inDrops, Chromium (v3), Seq- Well, CEL-Seq2). We have made use of the annotations of nine unique cell types (D4+ T cell, Cytotoxic T cell, Natural killer cell, CD16+ monocyte, CD14+ monocyte, Megakaryocyte, B cell, Dendritic cell, Plasmacytoid dendritic cell) provided in the original study. Also, we removed the unassigned cells. Availability: httpps: wtwp:leeeeeeeeeeeeeeeeeeeeeeeeee www.obbosrittuu...o....... al--comparisoon-pbeeee
wwwuuueerrgooo org/ s weiiiggleeellee study SCP424/ / s in le-c c
Pancreas: This dataset was collected by Tran et al.(2020) combining five studies of the human pancreas. It comprises 14,767 cells, with 5,975 genes shared across all studies, sequenced by four ScRNA-seq technologies (inDrop, CEL-Seq2, Smart-Seq2. SMARTer). The harmonized dataset contains 13 cell types (alpha, beta, ductal, acinar, delta, pancreatic stellate, pancreatic polypeptide, endothelial, macrophage, mast, epsilon, Schwann and T cell). Availability: https: // //figshare. com
15

---

Benchmarking Self-Supervieed Learning for Single-Cell Data
Lung: This dataset contains 32,426 cells across 16 batches and two technologies (Drop-seq and 10x Chromium), assembled by Luecken al.(2022) from three labs. The harmonized dataset includes 15,148 genes. The cells are derived from transplant patients and lung biopsies and are annotated as 17 cell types. Availability; https: https:s//fighharee com/ndo wnloader/ wwweee/eii. /
This dataset contains 329,762 cells and includes 36,398 genes across twelve batches and three different sequencing technologies (10x 5' vl, 10x 5' v2, 10x 3' v3), collected by Conde et al.(2022). The cells originate from 16 different tissues. The annotations include 35 fine-grain cell types. Availability: https: https/:://aaaasssscoom wewp:aasttss..lleeeeeeeeeeee celxxgene zisciennee com/08f58b33 a01s 4300- 8ebo- 2b93c18f261) .h5ad
Immune Cell Atlas:
Tabula Sapiens: This dataset was collected by Jones et al.(2) and contains 1,136,218 cells from 24 tissues and organs, sequenced with 10x 3' v3, 10x 5' v2, Smart-seq, and Smart-seq3 protocols. Tabula Sapiens is a molecular reference atlas for more than 400 cell types of the human body. Availability: https: :tteps:/cceeee htt////llllleeeeeeeeeeeee cwwscience. com/ collectio
ns/e5f5882 9- 1a66- 4005- - a624 www.ee222-99000000 9046778e74FF
Multi-modal Peripheral Blood Mononuclear Cells (PBMC-M): This dataset was collected by Hao et a2.(2) with 161,764 cells across eight batches. For each cell, two modalities are available: RNA and protein. RNA has 18,702 genes, while the dimension of protein is 224. As a pre-processing step, we merge different T cell granularities, similar to the Concerto framework (Yang et al., 2022b). Availability: https:
fww.httcccccccccooo wwwp//tt////////ooo/////../..aat dataaaageeaa mu momtmli im
odal/pbmc, mulnnn___utt t mimmmdal.... h5seurat
Multi-modal Bone Marrow Mononuclear Cells (BMMC). This dataset was collected by Luecken et al.(2021) and contains 90,261 cells across thirteen batches and twelwe healthy human donors (Lance et al., 2022). Each cell has two modalities: Gene expression (GEX) and protein abundance (ADT). While GEX has 13,953 genes, the protein abundance dimension 134. Pre-processing is the same as PBMC-M. Availability; https: htpp://wwwwr /wwww/wwww.///.//////// ncbi nlm, nih. c.i Cggi7acc=GSE19410
C. Evaluation Details
Preprocessing: All datasets are preprocessed using SCANPY (Wolf et al., 2018) normalize-total function, which scales the total counts per cell to 10,000, followed by log-transformation. We subsequently perform batch-aware feature selection to choose the 4,000 most vaghle-variable genes (HVGs) for further processing. For multi-modal PBMC-M and BMMC datasets, we select 2,000 HVGs contrary to 4,000 HVGs for the single modality datasets.
The evaluated metrics are divided into two categories: those that measure the conservation of biological variance and that that measure the batch correction (Tran et al., 2020; Luecken et al., 2022). To evaluate conservation of biological variation, we calculate the isolated labels score, the Leiden NMI and ARI, the silhouette label score, and the cLISI metric. To evaluate batch correction, we calculate the graph connectivity, kBET per label, iLISI for each cell, the PCR comparison score, and the silhouette coefficient per batch. For details and definitions of the used evaluation metrics, as well as their implementation, we refer to (Luecken et al., 2022).
Batch Correction:
Cype-Type Annotation and Missing Modality Prediction: In the PBMC-M dataset, for cell-type annotation mapping and missing modality inference, we hold out batches P3, P5, and P8. In the BMMC dataset, for cell-type annotation mapping and missing modality inference, we hold out batches s4d1, s4d8, and s4d9, Similar to the approach of (Xu et al., 2021), we perform cell-type annotation by fitting a non- non- non-parametric supervised classifier (k-nearest neighbors (KNN) classifier with 11). For missing modality prediction, we fit a KNN classifier with = 5, as in (Yang et al., 2022b).
D. Hyperparameter Tuning
In all experiments, we use the augmentation pipeline proposed by CLEAR (Han et al., 2022) foundation, unless stated differently. Experiments described in this section were computed for all methods except Concerto. For the latter, we use the original model from (Yang et al., 2022b).
Optimization: All models in this benchmark, except Concerto, were trained with the Adam optimizer (Kingma & Ba, 2017). We use a stepwise learning rate schedule with base learning rate 1e-4 and fix the batch size 256. When applicable, the memory bank size was set to 2048
Encoder Architecture: We fix the encoder across all architectures and only perform hyper-parameter search on the
16

---

Benchmarking Self-Supervised Learning for Single-Cell Data
dimensionality of the encoder output, i.e., the representation dimensionality. The encoder consists of fully connected layer reducing the dimensionality to 128, followed by a ReLU activation and batch normalization. A further fully connected layer encodes the hidden representation to the representation dimension, followed by batch normalization.
Projector Dimensionality: Projection heads benefit self-supervised models in learning robust representations (Xue et al., 2024). At inference, the projection head is discarded, and only the (backhone) encoder is used for inference. All evaluated architectures subject to our evaluation include a projection head. We perform a hyperparameter search to find the best output dimension of the projector.
All projection heads were implemented noted in the respective works. In their respective works, SimCLR, MoCo, SimSiam, and NNCLR are evaluated with projectors that retain or scale down the dimensionality of the representation. Twin,,,wins,, BYOL, and VICReg are evaluated with projectors that retain or scale up the dimensionality. We follow this rationale and search grid of scaling factors {1,2,4] 4]. To compute the projection dimensionality, the scaling factor is either divided (scale-down models) or multiplied (scale-up models) with the representation's 's dimension.
porianeeiivvariane--ovaari the original work. We evaluate a grid of parameters, where the invariance term and the variance term X, a {5, 10, 25, 503, while the invariance term 3 is fixed to 1. We find that À and a fixed to 5 perform well across both ablation datasets.
Regularization Hyperparameters:
regularization hyperparameters are used as is done in
Augmentation Strength: Augmentations are known to benefit SSL models in finding robust representations. Details of the evaluated augmentations are listed in Appendix E. We perform grid search to optimize the hyperparameters for all augmentations. This includes a for all models, a for the Gaussian Noise augmentation, and the KNN-size for the neareet-neighhor-buseee transforms MNN and BBKNN. For each augmentation, the original CLEAR hyperparameters are fixed, and only the hyperparameters of the evaluated augmentation are adapted. For the ablation of BBKNN, we remove CrossOver, and replace it by BBKNN. Due the implementation of MNN, we remove CrossOver and insert MNN the front of the augmentation pipeline. Results of the ablation are recorded in Figure G6.
E. Augmentations
We evaluate six augmentations in this work. For all, the parameter a defines the proportion of values affected by the transform. Augmentations are applied sequentially. Masking is performed by setting gene expressions to zero. Gaussian noise computes noise vector computed from the normal distribution (with zero mean and standard deviation o)) and adds it to the input. InnerSwap switches expressions between genes within cell, while CrossOver switches expressions of the same gene between two random cells.
The MNN augmentation refers to our implementation of CLAIRE's augmentation (Yan et al., 2023). For each cell, it computes an intra- and inter-batch neighborhood based on its mutual nearest neighbors. Then, views are computed by interpolating between neighbors. We do not filter cell-neighoorhoods based on representation similarities during early stages of model training, as is done in the original work. This work introduces the BBKNN augmentation. uses a non-trimmed batch-balanced KNN graph (Polański et al., 2019) to define a set of #batches KNN neighbors for each cell. Views are computed by interpolating between neighbors. It differs from CLAIRE's s concept in that it does not distinguish between intra- and bnter-batch neighbors. While MNN always produces a view based on neighbors within and a view based on neighbors from outside the batch, this is not the case for BBKNN. Due to its implementation, the MNN augmentation is limited to be applied first in any augmentation pipeline. We refer to (Yan et al., 2023) for further detail on the interpolation process.
F. Multi-Modal Setting
Recent developments in the single-cell analysis allow the measurement of multiple aspects of cellular state. Data containing multiple modalities of cell, e.g., RNA and protein, is called multi-omics. Existing self-supervised methods for single-cell data integration can be extended to the multimodal setting by combining views produced by specialized models for different modalities. We train two models for each modality; each model consists of an encoder and projector. As is common (He et al., 2020; Chen et al., 2020; Geiping et al., 2023), only the encoder is used to infer the integrated representation. However, in the single-cell community, the projector is also used during inference, and, therefore, we also evaluate whether projection during prediction improves performance. Additionally, there are various techniques to combine representations (Radford al., 2021; Xiong al., 2023; Yang et al., 2022b). We evaluate three approaches: Addition, concatenation, and CLIP.
Encoder& & Projector Embedding Evaluation: Using CLEAR augmentations, we train two models for each modality, each
17

---

Benchmarking Self-Supervised Learning for Single-Cell Data
consisting of encoder and projector. In Table H7, compare data integration performance with and without projector during inference. Interestingly, SimCLR benefits from projection, while VICReg performance degrades. We conclude that the effect of projection is inconsistent across models.
G. Supplementary Figures
SimCLR
SimSiam
NNCLR
MoCo
Negaitva
NTXond
InfoNCE
Negnisss
NTXent
MomBBank
Predictor
Predictor
NN-Bask
strg-
Projector--*
Projector
EMA
Projector-
Projector
Projector- ...
-Projector
Projector- Projectee
Eoroder
Encader
2M14
Encadre
Raceder
Faceder
Receder
Racedor
Rocoddr
VICReg
BYOL
Barlow Twins
Encoder:
Bediundanry- Reduction
Aarisee--nwarianne
Term
Covoriance Beg.
Predictor
Projector
Projector
ZMA
Projector
Projector
Linear
Projector
Projecter
ReL.T
Eneeder
Eneoder
Encoder--..
Encader
Facalre
Eacader
HatckNorm
Linear
Figure GI. Overview considered methods, Dotted lines between the encoder and projector blocks represent weight sharing. Exponential Moving Average (EMA) denotes the updating of weights with momentum. This figure inspired by (Bardes al., 2022; He al., 2020; Dwibedi al., 2021) based on our implementation of models with LightlySSL (Susmelj et al., 2023).
bpp
Referonce
Uncorrected
UNAP22
UNAPP
UIMAP2
UNAP2
Quary
Input
DC NK
Mano
UMAP1
UMAP!
UMAP1
UMAP!
cell lype
baich
Refseence (oell type)
Query foell ype)
UMAP2
B CD4
CDB
Conrected
UMAP2
CD4 CDO T DC
B CDIT
CDR
Result
Mona NK
DC
Mora
NK
Mono
Figure G2. Batch Correction. The uncorrected (red) figure shows that cells cluster rather by batch (technical noise) than cell type (true biological signal) before the hatch correction. After training a model and learning corrected representation (green), cells are grouped by cell type, and batches are mixed.
Reference. Model gets annotated train dataset (reference, pink input) as input and learns the correspond- ing latent space. During inference, representations of train (refer- ence) and hold-out sets (query). train cell types, and classifier are used predict cell types of hold-out data (query, blue result).
Figure G3. Duere-to-Referencee.
18

---

Benchmarking Self-Supervised Learning for Single-Cell Data
MCA- CCA-Dataset
MiC-Dataset
0.7
Total Score
BarlowTwins
0.5
MoCo
NNCLR
0.4
SimCLR
SimSiam
0.3
VICReg
0.3
0.2
8
16
32
64
128
256
512
1024
16
32
64
128
256
512
1024
Representation
Regresentation
Figure G4. Tuning of the encoder based the representation dimensionality. The encoder architecture is defined in Appendix D. Lines correspond to the mean total score across five runs with unique seeds.
MCA-Datasse
HIC-Datased
0.7-
Total Score
0.6
BYOL
0.6-
BarlowTwin:
0.5-
0.5
MoCo
NNCLR
0.4
SimCLR
0.31
0.3
SimSiam
0.2
0.2
VICReg
Scale Factor
Figure GS. Tuning of the projector. The scale factor is defined in Appendix D: contrastive methods, the projected size decreased according to the scale factor, while for non-contrastive methods the projection size increases in accordance with the scale factor. Lines correspond to the mean total score across five runs with unique seeds.
0.50
0.66
Total Score
0.45
0.65
mask
0.40-
0.64
swap
gauss-0.2
0.35-
0.63
gauss-0.4
gauss-0.6
0.25
0.3
0.5
0.7
0.9
0.1
0.3
0.5
0.7
0.9
Alpha
0.64
Total Score
0.62
bbknn-interpolate-3
3bkon-interppolte-5
0.60
crossover
mnn-interpolate-3
0.58
mnn-Interpolate-5
0.56
0.3
0.7
Alpha
Figure G6. Ablation on the augmentation hyperparameters. The figure aggregates results for all methods, trained on the HIC datasct.
19

---

Benchmarking Self-Supervised Learning for Single-Cell Data
HIC
MCA
PBMC
HIC
MCA
PBMC
Augmentation
GaussianN.
57
65
Augmentation
GaussianN,
InnerSwap
86
InnerSwap
64
35
Masking
Masking
BBKNN
58
05
65-62
65
43
BBKNN
32
01
59
33
69
50
23
MNN
MNN
0.4
CrossOver
57
66
62
66
60
70
70
49
64
CrossOver
37
65
73
35
57
Innerswwn
Inners Mask
BBKN
Cross
MNN
Inners wap
Mask
BB
CrossOvel
MNN
Caussiant
Innersy SnnerSwap wwap
BBK MNN CrossOver
Inmerrww Mack BBKAN
wap
CrossOve
MNN
Inners
Mask
BOKNN
Crossōver
MNN
Gaussland
Inners Swap Masking
MNN BBKNI
Cross
Augmentation 2
Augmentation 2
(a) SimCLR
(b) MoCo
Figure G7. Evaluation of individual and combined data augmentations based on total score for batch correction for SimCLR and MoCo
method. Diagonal entries correspond to a single augmentation, off-diagonal entries correspond to two sequentially applied augmentations.
H. Supplementary Tables
Table Batch correction benehmark methods trained using the CLEAR pipeline. This table an extension to Table containing datasets that were used during hyperparameter tuning.
MCA
HIC Batch
Lung Batch
Method
Bio
Batch
Total
Bio
Total
Bio
Total
0.519
0.666
0.578
0.753
0.536
0.666
0.184
0.628
0.362
SimCLR
1: 0.032
0.026
1 0.021
1 0.027
0.013
1 0.018
1 0.005
1 0.025
1 0.007
0.280
0.791
0.484
0.791
0.603
0.716
0.148
0.678
0.360
MoCo
+ 0.050
0.020
0.084
0.014
0.014
0.008
0.000
= 0.004
+ 0.001
0.154
0.673
0.362
0.585
0.487
0.546
0.100
0.654
0.322
SimSiam
0.043
0.033
0.036
0.043
0.055
+ 0.011
0.332
0.665
0.465
0.695
0,500
0.617
0.147
0.662
0.353
NNCLR
1 0.084
0.035
1 0.060
1 0.019
0.008
1 0.010
+: 0.002
+ 0.019
1 0.008
0.000
0.694
0.277
0.582
0.707
0.632
0.028
0.705
0.299
BYOL
1 0.009
0.037
0.016
0.021
0.027
0.022
0.003
1 0.003
1 0.001
0.515
0.709
0.592
0.830
0.581
0.730
0.208
0.622
0.374
VICReg
0.018
Barlow
0.471
0.627
0.533
0.784
0.533
0.684
0.173
0.651
0.364
Twins
+ 0.032
0.029
+ 0.026
+ 0.011
0.004
+ 0.008
0.007
+ 0.010
+ 0.000
0.497
0.559
0.522
0.000
0.627
0.251
0.152
0.711
0.376
Concerto
1 0.036
0.013
0.027
0.000
0.004
1 0.001
+ 0.008
+ 0.008
+ 0.008
0.684
0.385
0.565
0.615
0.262
0.474
0.876
0.326
0.656
CLEAR
+ 0.042
0.008
+ 0.027
4 0.004
0.007
+: 0.005
+ 0.088
+ 0.049
+ 0.050
0.467
0.978
0.672
0.511
0.974
0.696
0.315
0.907
0.552
CLAIRE
+ 0.015
0.007
0.024
0.014
0.324
+ +00.58
0.176
0.723
0.254
0.536
0.768
0.659
0.725
0.716
0.659
0.693
ScVI
+: 0.021
0.017
0.019
0.007
0.003
0.006
0.176
+ 0.017
+ 0.111
0.538
0.297
0.442
0.635
0.164
0.446
0.748
0.245
0.547
PCA
+ 0.027
0.000
0.016
+ 0.014
0.003
+ 0.009
0.114
+ 0.062
+ 0.093
20

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Table H2. Uni-modal cell-typing with CLEAR augmentations. We define one technology (10X 5' v2) of the Immune Cell Atlas as hold-out set, train the encoder and knn-classifier. The generic model VICReg outperforms all other methods.
Immune Cell Atlas (10X 5' v2)
Method
Macro F1
Acc
SimCLR
0.788
0.830
MoCo
0.794 0.006
0.835
SimSiam
0.711 0.015
0.768
NNCLR
0.740 0.010
0.804
BYOL
0.680 0.002
0.724
VICReg
0.820 = 0.012
0.866 0.003
Barlow Twins
0.727 0.004
0.752
CLEAR
0.806 + 0.000
0.855 + 0.000
CLAIRE
0.436 +: 0.000
0.492 1: 0.002
ScGPT (zero-shot)
0.358
0.439 + 0.001
scGPT (finetuned)
0.835 = 0.008
0.850 + 0.008
Geneformer (finetuned)
0.831 0.000
0,878
ScBERT
0.818
0.873
scVI
0.750 0.001
0.804 0.000
SCDC
0.595
0.009
0.642 0.000
PCA
0.071 1
+ 0.007
0.124 0.000
21

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Table H3. Cell-type annotation with CLEAR augmentations on the Pancreas dataset. We define individual studies as holdout sets during
training. Accuracy and Macro are computed on the holdout set.
Mutaro et al.
Segerstolpe et al.
Wang et al.
Xin et al.
Method
Macro Fl
Acc
Macro Fl
Acc
Macro Fl
Acc
Macro F1
Acc
0.796
0.936
0.781
0.946
0.894
0.945
0.778
0.819
SimCLR
+ 0.014
+ 0.012
+ 0.022
+ 0.011
+ 0.005
+ 0.010
# 0.050
= 0.019
0.825
0.939
0.799
0.936
0.844
0.919
0.722
0.798
MoCo
+: 0.045
t 0.013
+: 0.033
+: 0.006
+ 0.020
+: 0.021
+ 0.045
= 0.010
0.679
0.875
0.595
0.803
0.586
0.798
0,464
0,718
SimSiam
+ 0.042
+: 0.026
+ 0.022
+ 0.014
+ 0.036
+ 0.022
# 0.071
0.037
0.707
0.897
0.619
0.854
0.729
0.868
0.472
0.721
NNCLR
0.043
0.024
+ 0.013
1 0.012
+ 0.019
f 0.012
# 0.033
0.025
0.720
0.884
0.670
0.854
0.656
0.813
0.524
0.724
BYOL
1: 0.037
1: 0.021
+ 0.005
1: 0.006
1 0.048
1 0.008
# 0.062
0.013
0.853
0.947
0.855
0.976
0.877
0.937
0.830
0.839
VICReg
+ 0.039
+ 0.002
+ 0.008
+ 0.006
+ 0.008
+ 0.004
+ 0.023
0.010
Barlow
0.700
0.868
0.673
0.870
0.737
0.878
0.487
0.720
Twins
+ 0.010
+ 0.005
+ 0.017
+ 0.007
+ 0.012
+ 0.010
= 0.003
0.009
0.106
0.431
0.113
0.419
0.105
0.435
0.112
0,406
Concerto
1: 0.000
1: 0.000
d: 0.000
+: 0.000
+ 0.000
4 0.000
d: 0.000
# 0.000
0.950
0.961
0.898
0.967
0.891
0.941
0.987
0.994
CLEAR
1: 0.000
1: 0.001
+ 0.000
1: 0.002
+ 0.000
+: 0.007
+: 0.000
0.001
0.941
0.937
0.919
0.955
0.893
0.945
0.965
0.989
CLAIRE
+ 0.000
+ 0.002
+ 0.000
+ 0.002
+ 0.000
+ 0.001
# 0.000
0.002
scGPT
0.502
0.765
0.549
0.826
0.327
0.519
0.581
0.786
(zero-shot)
0.00
0.00
0.02
0.01
0.15
0.21
0.00
0.00
scGPT
0.850
0.917
0.803
0.955
0.466
0.560
0.989
0.995
(finetuned)
+
0.02
+
+ 0.154
+ 0.217
+ 0.006
0.002
Geneformer
0.622
0.916
0.630
0.944
0.673
0.891
0.564
0.993
(finetuned)
+: 0.000
+ 0.000
+ 0.000
+ 0.000
+ 0.000
+ 0.000
+ 0.000
= 0.000
0.642
0.919
0.715
0.953
0.710
0.902
0.975
0.990
ScBERT
+ 0.004
+ 0.012
+ 0.002
+ 0.002
+ 0.0830
+ 0.001
+ 0.007
0.000
0.616
0.889
0.635
0.903
0.680
0.897
0.741
0.916
ScVI
+ 0.015
+ 0.012
+ 0.127
+ 0.023
# 0.13
+ 0.023
+ 0.171
0.023
0.571
0.860
0.581
0.875
0.689
0.908
0.442
0.800
SCDC
+ 0.004
+ 0.021
+ 0.001
+ 0.015
+ 0.003
+ 0.002
+ 0.002
0.009
0.071
0.082
0.171
0.236
0.195
0.371
0.054
0.427
PCA
+ 0.000
0.00
+ 0.000
+ 0.000
+ 0.000
+ 0.000
0.000
0.000
22

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Table H4. Cell-type annotation for multi-modal datasets with CLEAR pipeline. On the left, modalities (RNA Protein GEX (gene expression) + ADT (protein abundance)) were used during inference. On the right, we show inference performance with single modality (RNA GEX). models were trained with two modalities.
RNA + Protein
GEX + ADT
RNA
GEX
Method
PBMC-M
BMMC
PBMC-M
BMMC
Macro Fl
Acc
Macro Fl
Acc
Macro F1
Acc
Macro FI
Acc
0.950
0.977
0.770
0.876
0.906
0.946
0.749
0.848
SimCLR
0,002
0.001
0.028
0.022
+ 0.002
0.001
0.050
: 0.035
0.930
0.969
0.609
0.771
0.778
0.835
0.630
0.717
MoCo
0.007
+ 0.004
+ 0.001
+ 0.041
+ 0.016
+ 0.010
+ 0.065
+ 0.073
0.933
0.968
0.666
0.820
0.846
0.884
0.670
0.792
SimSiam
0.002
+ 0.001
0.069
+ 0.036
- 0.016
+ 0.019
0.082
+ 0.056
0.941
0.971
0.734
0.856
0.860
0.901
0.703
0.806
NNCLR
0.004
0.002
0.055
0.043
+ 0.008
0.003
0.085
+ 0.070
0.933
0.968
0.737
0.847
0.857
0.898
0.704
0.795
BYOL
0.037
+ 0.047
0.034
0.048
+ 0.054
0.950
0.977
0.808
0.899
0.923
0.957
0.785
0.887
VICReg
0.001
0.000
+ 0.021
+ 0.017
+ 0.005
- 0.004
+ 0.014
+ 0.015
Barlow
0.949
0.976
0.766
0.863
0.890
0.919
0.733
0.816
Twins
0.001
0.001
0.012
0.003
+ 0.014
0.008
+ 0.013
+ 0.017
0.892
0.947
0.673
0.825
Concerto
0.000
1 0.001
1 0.000
1: 0.000
0.699
0.851
0.557
0.797
0.728
0.857
0.635
0.818
scCLIP
0.009
0.025
+ 0.015
+ 0.008
+ 0.019
+ 0.004
+ 0.071
+ 0.051
0.949
0.976
0.844
0.924
0.946
0.973
0.831
0.920
scButterfly
0.000
+ 0.000
+ 0.002
+ 0.000
+ 0.000
+ 0.000
+ 0.000
+ 0.001
0.173
0.211
0.039
0.149
scTEL
0.001
+ 0.031
+ 0.003
+ 0.004
0.829
0.977
0.829
0.911
total VI
23

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Table H5. Missing modality prediction for methods trained with the CLEAR pipeline on multi-modal datasets. We show the average
Pearson correlation between the original and inferred missing modality: protein for PBMC-M, and ADT (protein abundance) for BMMC.
PBMC-M
BMMC
Method
Pearson Mean
Pearson Mean
SimCLR
0.866
0.757 + 0.002
MoCo
0.856 0.001
0.721
+ 0.004
SimSiam
0.859 0.002
0.748
1: 0.002
NNCLR
0.861 0,002
0.751
+ 0.001
BYOL
0.860 0.000
0.738 + 0.002
VICReg
0.865
0.759 + 0.001
Barlow Twins
0.864 0.001
0.755 + 0.001
Concerto
0.742 d: 0.006
0.542 1 0.001
seCLIP
0.614 + 0.003
0.175 + 0.005
scButterfly (kNN)
0.856 0.000
0,651 + 0.001
scButterfly (generated)
0.840 0.000
0.624
+ 0.002
scTEL (100 epochs)
0.022 0.005
0.047
+: 0.002
Table H6. Batch correction results for methods with and without projection layer during inference. All methods are trained using the CLEAR pipeline. Results are not min-max scaled easier comparison.
Encoder
Encoder + Projection
MCA
HIC Batch
MCA
HIC
Method
Bio
Batch
Total
Bio
Total
Bio
Batch
Total
Bio
Batch
Total
0.620
0.633
0.625
0.683
0.567
0.637
0.575
0.658
0.608
0.674
0.575
0.635
SimCLR
0.007
+ 0.014
0.000
+ 0.004
+ 0.000
+ 0.002
0.561
0.706
0.619
0.721
0.585
0.667
0.536
0.726
0.612
0.705
0.602
0.664
MoCo
0.013
+ 0.002
+ 0.007
+ 0.000
+ 0.003
+ 0.001
0.000
= 0.015
+ 0.006
= 0.019
+ 0.001
+ 0.012
0.519
0.656
0.574
0.623
0.545
0.592
0.435
0.681
0.533
0.598
0.540
0.575
SimSiam
0.008
0.025
+ 0.006
0.040
0.012
+ 0.029
0.009
0.027
+ 0.006
0.028
0.017
+ 0.030
0.570
0.646
0.600
0.680
0.550
0.628
0.493
0.675
0.566
0.657
0.557
0.617
NNCLR
0.012
+ 0.003
+ 0.008
+ 0.023
+ 0.003
+ 0.013
0.001
0.013
+ 0.004
0.026
+ 0.005
+ 0.018
0.486
0.670
0.560
0.637
0.602
0.623
0.398
0.667
0.506
0.552
0.607
0.574
BYOL
0.017
0.003
+ 0.012
0.007
0.000
+ 0.004
0.004
0.028
+ 0.013
0.010
0.007
+ 0.009
0.624
0.653
0.636
0.716
0.584
0.663
0.587
0.699
0.632
0.726
0.588
0.671
VICReg
0.005
+ 0.009
+ 0.006
+ 0.007
+ 0.002
+ 0.003
0.002
= 0.009
+ 0.005
0.005
+ 0.005
+ 0.005
Barlow
0.606
0.623
0.613
0.716
0.566
0.656
0.544
0.664
0.592
0.675
0.562
0.630
Twins
0.009
0.013
+ 0.000
0.006
0.006
+ 0.001
0.018
0.004
+ 0.012
0.029
0.006
+ 0.020
0.634
0.529
0.592
0.357
0.470
0.402
Concerto
+: 0.001
0.001
+ 0.001
0.696
0.438
0.593
0.642
0.408
0.549
CLEAR
0.010
0.001
0.006
0.003
+ 0.001
0.689
0.763
0.718
0.699
0.700
0.699
CLAIRE
0.018
1 0.008
d: 0.014
d: 0.029
0.003
1 0.018
0.651
0.348
0.530
0.654
0.320
0.521
0.651
0.348
0.530
0.654
0.302
0.521
PCA
0.000
+ 0.000
+ 0.005
0.000
+ 0.000
+ 0.005
24

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Table H7. Data integration for methods using the CLEAR pipeline on multimodal datasets. We compare the effect of retaining the projection head during inference to the representation quality when using only the encoder. This table is not min-max scaled.
Encoder
Encoder + Projection
PBMC-M
BMMC Batch
PBMC-M
BMMC
Method
Bio
Batch
Total
Bio
Total
Bio
Batch
Total
Bio
Batch
Total
0.738
0.518
0.650
0.718
0.574
0.660
0.741
0.513
0.650
0.712
0.567
0.654
SimCLR
0.000
1 0.010
1 0.004
1 0.016
1 0.001
+ 0.010
0.021
# 0.005
1 0.015
0.004
1 0.001
1 0.002
0.776
0.570
0.694
0.677
0.582
0.639
0.762
0.606
0.700
0.592
0.637
0.610
MoCo
0.001
0.007
0.002
0.004
1 0.014
0.004
4 0.012
0.766
0.563
0.685
0.677
0.559
0.629
0.767
0.565
0.686
0.635
0.553
0.602
SimSiam
0.051
+ 0.012
+ 0.036
1 0.011
+ 0.000
+: 0.006
0.024
0.006
+: 0.017
0.021
4 0.002
+ 0.014
0.750
0.535
0.664
0.706
0.565
0.650
0.743
0.533
0.659
0.695
0.575
0.647
NNCLR
0.010
+ 0.000
0.001
0.002
0.024
+ 0.029
0.006
1 0.011
0.789
0.557
0.696
0.701
0.574
0.651
0.783
0.543
0.687
0.691
0.584
0.648
BYOL
0.015
+ 0.000
+ 0.009
+ 0.005
+ 0.007
+ 0.006
0.007
= 0.009
+ 0.008
0.006
+ 0.002
t 0.004
0.749
0.478
0.641
0.722
0.582
0.666
0.763
0.491
0.654
0.714
0.578
0.660
VICReg
0.017
+ 0.002
+ 0.000
+ 0.002
0.001
+ 0.001
Barlow
0.755
0.509
0.657
0.704
0.577
0.653
0.712
0.506
0.630
0.704
0.581
0.655
Twins
0.018
+ 0.015
+ 0.017
+ 0.001
+ 0.005
+ 0.003
0.006
0.003
+ 0.005
0.002
+ 0.009
+ 0.002
0.773
0.436
0.638
0.604
0.525
0.573
Concerto
+ 0.117
0.006
+ 0.072
0.089
+ 0.01
+ 0.054
0.602
0.504
0.563
0.558
0.322
0.464
0.602
0.504
0.563
0.558
0.322
0.464
PCA
0.000
+ 0.000
+ 0.000
+ 0.000
+ 0.000
+ 0.000
0.000
= 0.000
+ 0.000
= 0.000
+ 0.000
+ 0.000
25

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Table H8. Batch correction benchmark for methods trained using the CLEAR pipeline with domain specific batch normalization (DSBN).
Results are not min- max scaled for easier comparison.
Batch Normalization
DSBN
HIC
MCA
HIC Batch
MCA
Method
Bio
Batch
Total
Bio
Batch
Total
Bio
Total
Bio
Batch
Total
0.703
0.573
0.651
0.627
0.644
0.634
0.680
0.583
0.641
0.624
0.636
0.629
SimCLR
0.022
1 0.014
1 0.009
1 0.009
1 0.026
1. 0.014
0.020
0.015
1 0.008
0.008
0.009
1 0.006
0.707
0.582
0.657
0.518
0.731
0.603
0.648
0.612
0.633
0.549
0.697
0.608
MoCo
0.010
1 0.020
d: 0.011
1 0.060
d 0.032
1 0.047
0.040
0.004
1. 0.022
0.013
1. 0.003
1 0.008
0.619
0.544
0.589
0.523
0.668
0.581
0.603
0,595
0.600
0.502
0.635
0.555
SimSiam
0.053
1 0.043
d: 0.049
1 0.069
d 0.072
+ 0.069
0.074
0.025
+: 0.034
0.018
0.013
4 0.015
0.658
0.546
0.613
0.574
0.637
0.599
0.659
0,590
0.632
0.543
0.651
0.587
NNCLR
0.015
+ 0.011
+ 0.010
+ 0.120
+ 0,056
+ 0.087
0.011
0.006
+ 0.009
0.019
+ 0.001
+ 0.011
0.607
0.624
0.614
0.483
0.679
0,561
0.576
0.600
0.586
0.473
0.673
0.553
BYOL
0.024
+ 0.016
+ 0.020
0.005
0,042
+ 0.019
0.050
0.018
+: 0.023
0.021
0,014
+ 0.017
0.706
0.592
0.661
0.615
0.665
0.635
0.674
0.591
0.641
0.619
0.649
0.631
VICReg
0.034
+ 0.019
+: 0.017
+ 0.016
+ 0.020
+ 0.017
0.056
0.012
+ 0.033
0.004
+ 0,002
+ 0.003
Barlow
0.713
0.572
0.656
0.603
0.636
0.617
0.707
0.577
0.655
0.603
0.634
0.615
Twins
0.014
0.008
+ 0.005
0.010
0,064
+ 0.027
0.004
0.003
+ 0.002
0.020
0.011
+ 0.008
0.357
0.470
0.402
0.635
0.529
0.593
Concerto
0.000
+ 0.029
+ 0.012
+ 0.016
+ 0.028
+ 0.021
0.656
0.320
0.522
0.651
0.348
0.530
0.656
0.320
0.522
0.651
0.348
0.530
PCA
0.005
+: 0.005
#: 0.012
+: 0.001
0.001
4: 0.000
Table H9. Comparison of different multi-omics integration methods using the CLEAR pipeline. Data integration metrics were computed for the BMMC dataset.
Add Batch
Concat Batch
CLIP + Concat
Method
Bio
Total
Bio
Total
Bio
Batch
Total
0.827
0.3
0.616
0.84
0.273
0.613
0.511
0,504
0.508
SimCLR
- 0.078
+ 0.057
+ 0.05
+ 0.093
+ 0.058
+ 0.065
+ 0.223
+ 0.094
+ 0.166
0.935
0.407
0.724
0.056
0.8
0.354
0.566
0.464
0.525
MoCo
+ 0.07
+ 0.019
+ 0.045
+ 0.065
+ 0.000
+ 0.039
+ 0.157
0.161
0.049
0.453
0.174
0.341
0.506
0.21
0.387
0.197
0.364
0.264
SimSiam
+ 0.175
- 0.025
+ 0.107
+ 0.146
+ 0.041
+ 0.083
+ 0.177
0.011
0.679
0.225
0.498
0.768
0.231
0.553
0.584
0.5
0.551
NNCLR
1: 0.171
+ 0.041
+ 0.113
+ 0.105
+ 0.034
+ 0.06
+ 0.088
+ 0.079
+ 0.066
0.117
0.8
0.39
0.527
0.673
0.586
0.46
0.403
0.437
BYOL
+ 0.19
+ 0.000
+ 0.066
+: 0.029
+ 0.074
+ 0.03
+ 0.123
+ 0.152
+ 0.1
0.791
0.449
0.654
0.887
0.484
0.726
0.72
0.38
0.584
VICReg
0.089
- 0.014
+ 0.052
+ 0.035
+ 0.009
0.022
+ 0.079
0.055
0.056
Barlow
0.717
0.262
0.535
0.852
0.27
0.62
0.706
0.352
0.565
Twins
+ 0.055
+ 0.026
4 0.039
+ 0.096
+ 0.012
+ 0.059
+ 0.115
0.085
0.055
26

---

Benchmarking Self-Supervised Learning for Single-Cell Data
Table HI0. Augmentation Parameters for the CLEAR (Han al., 2022) augmentations the left. Results for the ablation of all augmentations on the right, including the CLAIRE (Yan et al., 2023) augmentation denoted as MNN, and our BBKNN augmentation. Results stem from our ablation detailed in Appendix C.
CLEAR
Ablation Result
Augmentation
Cx
0
knn
knn
Masking
0.2
0.5
Gaussian Noise
0.8
0.2
0.3
0.2
InnerSwap
0.1
0.3
CrossOver
0.25
0.1
BBKNN
0.9
MNN
0.5
Table H11. I. Batch correction benchmark for the Tabula Sapiens dataset (1.1 million cells) trained using the CLEAR pipeline (only rum) show the scalability of our benchmark. The previously drawn conclusion that haselines outperform SSL methods in uni-modal batch correction holds. Generic SSL methods are good at batch correction but not preservation of the true biological variance.
Tabula Sapiens
Method
Bio
Batch
Total
SimCLR
0.374
0.567
0.451
MoCo
0.342
0.541
0.421
SimSiam
0.237
0.430
0.314
NNCLR
0.312
0.454
0.368
BYOL
0.085
0.800
0.371
VICReg
0.327
0.657
0.459
BarlowTwins
0.376
0.427
0.396
SCVI
0.723
0.254
0.536
PCA
0.538
0.297
0.442
27


Reviews:
[{"review_id": "pUmYpN7OCG", "reviewer": "Reviewer_X2so", "summary": "The authors present scSSL-Bench, a single-cell data benchmark that integrates 12 different approaches and 8 different datasets. The authors run extensive experimentations to answer three critical questions and provide invaluable insights and takeaways - This is no easy feat considering there are many moving parts due to different data augmentation, normalization, and training strategies.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and highlighting the relevance of our benchmark and the quality of our experiments. In the following, we address your questions and suggestions, which have improved the quality of the paper and the benchmark.\n\n**Single-cell FM:** We acknowledge that Foundation Models (scFMs) have recently gained significant attention in the single-cell genomics community. Although scSSL-Bench evaluates a diverse set of SSL approaches, we added scGPT, Geneformer, and scBERT to the comparison, see https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/table_1.pdf and https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/table_H3.pdf.\n\nAs suggested, we will concisely discuss and cite the scFMs in the Introduction to provide the necessary context for readers and will update the discussion according to our new results, which showcase that scFMs demonstrate strong performance for bio conservation in batch integration and good performance in cell typing. Our analysis reveals a substantial performance improvement in scGPT after fine-tuning compared to its zero-shot performance, underscoring the importance of fine-tuning scFMs. We will also add a section to the appendix detailing hyperparameters and fine-tuning.\n\nWe also compare to one more single-modality method SCDC (https://ieeexplore.ieee.org/document/10086540) and two more multi-modal methods, scTEL (www.nature.com/articles/s41540-024-00484-9) and scButterfly (www.nature.com/articles/s41467-024-47418-x), as suggested by Reviewer m4qB, see https://anonymous.4open.science/r/scSSL-Bench-Rebuttal for new results.\n\n**Datasets Overview**: We have extended the dataset overview provided in Appendix B to include the sequencing technologies for each dataset, the number of features (genes, proteins), and the number of cells of a specific cell type, see details in https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/datasets.pdf. For space consideration, we could not add this additional information to the main text.\n\n**Why These Datasets:** The selected datasets represent commonly used and established benchmarks in the single-cell literature, enabling direct comparisons with previous studies. For instance, these datasets have been employed in widely referenced studies and benchmarkings (e.g., www.nature.com/articles/s42256-024-00934-3, www.nature.com/articles/s42256-022-00518-z, www.nature.com/articles/s41540-024-00484-9, www.nature.com/articles/s41467-024-47418-x, https://academic.oup.com/bioinformatics/article/39/3/btad099/7055295, https://academic.oup.com/bib/article/23/5/bbac377/6695268), providing comparability across existing work. We now highlight this aspect in the manuscript. Furthermore, scSSL-Bench is engineered for scalability and easy extension. Our implementation with Hydra, a configuration management framework that enables flexible experiment configuration, parameter sweeping, and support for HPC environments, makes adding new datasets straightforward with minimal adjustments. This allows researchers to extend the benchmark with additional datasets. To this point, we also added the Tabula Sapiens dataset with 1.1 million cells to our evaluation, see https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/sapiens_bc.pdf.\n\n**Augmentation Analysis**: Initially, we demonstrated augmentation ablations using VICReg due to its consistently strong performance. Following the reviewer's recommendation, we have extended our augmentation ablations to include SimCLR https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/augmentations_simclr.pdf and MoCo https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/augmentations_moco.pdf. This additional analysis confirms our original finding that masking is the most effective augmentation technique across all three SSL methods. Additionally, CrossOver shows competitive performance, especially for SimCLR.\n\n**Representation Dimensions:** Our original rationale for selecting relatively small embeddings was guided by prior work in the literature, e.g., scVI www.nature.com/articles/s41592-018-0229-2, that demonstrated effective performance with lower-dimensional embeddings for single-cell analyses. We have extended the evaluation up to 1024 and come to the same conclusion. The representation dimensions of 64 or 128 reach a similar performance as 1024 while requiring less training time and memory for training and downstream tasks. We have updated Figure G4 with the following plots https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/projection_HIC.pdf and https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/projection_MCA.pdf.\n\nWe appreciate the reviewer's thoughtful feedback and have made substantial improvements to the manuscript accordingly. These changes have significantly strengthened our work and we hope scSSL-Bench will serve as a valuable resource for the single-cell and machine learning communities. We are happy to answer any additional questions that the reviewers might have.", "claims_and_evidence": "Yes the authors pose 3 research questions which are extensively supported by detailed experimentations.", "methods_and_evaluation_criteria": "Yes, it does.", "theoretical_claims": "No theoretical claims were made in the paper.", "experimental_designs_or_analyses": "I checked the experimental designs and analyses - Couldn't have been phrased better.", "relation_to_broader_scientific_literature": "I think this is a timely addition that reflects the current single-cell literature mostly.", "essential_references_not_discussed": "While I like how the authors extensively cited existing literature, I find the lack of any references to single-cell foundation models somewhat puzzling. Yes, there are lots of hype around single-cell FM, but still, I think a reader who doesn't understand nuances might be confused by lack of single-cell FM.", "other_strengths_and_weaknesses": "I really appreciate the gargantuan effort that the authors had put into curating the benchmark. Having carried out effort myself, I know it's no easy feat, trying to draw overall takeaways/insights with so many moving parts. I think this will be a great contribution to the field where anyone not experienced enough in single-cell field can use it to jumpstart their research. I haven't tried out the github myself, but hopefully it is very straightforward to use for any newcomers. I hope the authors keep contributing to this benchmark, so that it can stand test of time.\n\nDespite its strengths, I have few minor comments that are holding me back in giving higher scores.\n- I think the authors needs to provide more detail on the 8 datasets used, e.g., sequencing technologies, how large is the gene panel for each dataset, since the readers would want to pick up the detail right away.\n- My understanding is that there are way more publicly-available single-cell dataset than these 8. Why were these 8 chosen? Are there future plans to add much more datasets to this?\n- As mentioned above, I find the lack of any discussion on single-cell FM quite puzzling. Yes, there is lots of hype, but I think they need to be either mentioned or benchmarked, since this will eventually be asked by end-users.\n- Since augmentation in training is really crucial and important, ideally authors should provide one or two other SSL methods with augmentation ablations (only VICReg shown in the paper so far), to demonstrate the trend holds.", "other_comments_or_suggestions": "- The dimension ablation seem intriguing, I am more used to much higher latent dimension (e.g., 512 in scGPT). So what is the rationale for only staying in the low regime (8,16,32,...)?", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DqIV02dO4W", "reviewer": "Reviewer_KvUM", "summary": "This paper proposes a self-supervised learning (SSL) benchmark for single-cell data. The authors tried twelve representative SSL methods and conducted comprehensive evaluations on eight datasets across three downstream tasks. The experimental designs are technically sound, and the paper is well-organized and written.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. We appreciate your recognition that our paper could be a helpful reference for researchers interested in single-cell representation learning and that we conducted comprehensive evaluations. We address your questions and suggestions, which we have integrated into the updated manuscript to further improve the paper.\n\n**Additional Methods:** Following your’s and reviewer X2so’s suggestion, we extended the benchmark to also include single-cell Foundation Models (scFMs). As requested, we included scBERT for cell type annotation of single-cell RNA-seq data into our benchmark. In addition, we added Geneformer and scGPT for batch integration and cell type annotation, see https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/table_1.pdf, https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/table_H2.pdf and https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/table_H3.pdf.\n\nWe added scBridges to the introduction. Although scSSL-Bench currently leverages only RNA-seq and CITE-seq data, we cite scBridge and think it is valuable to extend the benchmark with ATAC-seq integration later. Moreover, we included results for SCDC (https://ieeexplore.ieee.org/document/10086540) for uni-modal tasks, and scButterfly and scTEL for multi-omics tasks. We accordingly updated the tables, see details in https://anonymous.4open.science/r/scSSL-Bench-Rebuttal.\n\n**Dataset Scale and Diversity**: We agree that incorporating very large datasets like the Human Fetal Atlas (\\~4 million cells) and Mouse Atlas (\\~300 thousand cells) would further validate our benchmark's scalability. While time constraints prevented their inclusion in this revision, we explicitly note this as a direction for future work in the manuscript.\n\nTo demonstrate scalability, we have evaluated SSL methods and baselines on Tabula Sapiens (\\~1.1 million cells, https://cellxgene.cziscience.com/collections/e5f58829-1a66-40b5-a624-9046778e74f5), see new table https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/sapiens_bc.pdf. We'd also like to clarify that many of our existing datasets are substantial in size and widely used in the community, including Immune Cell Atlas (Conde et al., \\~330,000 cells), multi-modal PBMC (Hao et al., \\~160,000 cells), and BMMC (Lucken et al., \\~90,000 cells), see https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/datasets.pdf.\n\n**Results Discussion:** In the updated paper, we explain, wherever possible, the trade-offs between batch correction and bio conservation. Specifically, when describing how certain SSL methods prioritize batch correction over bio conservation, we discussed potential reasons, such as the nature of the SSL methods, the impact of specific augmentations, and the methods' underlying loss functions. For instance, in line 300, batch correction is prioritized since PBMC and ImunneCellAtlas datasets contain “harder” batch effects, SSL models try to correct the batch effects, neglecting bio conservation. \n\n**Representation Dimensions:** Our original rationale for selecting relatively small embeddings was guided by prior work in the literature, e.g., scVI www.nature.com/articles/s41592-018-0229-2, that demonstrated effective performance with lower-dimensional embeddings for single-cell analyses. We have extended the evaluation up to 1024 and come to the same conclusion. The representation dimensions of 64 or 128 reach a similar performance as 1024 while requiring less training time and memory for training and downstream tasks. We have updated Figure G4 with the following plots https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/projection_HIC.pdf and https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/projection_MCA.pdf.\n\n**Averaging for Missing Modality Prediction**: We agree that the characteristics and neighborhood structures of two modalities may differ, potentially affecting prediction accuracy. However, our choice of averaging nearest neighbors follows a common practice in the literature, e.g., Concerto www.nature.com/articles/s42256-022-00518-z.  Moreover, after adding scButterfly as suggested by reviewer m4qB that can generate missing modalities, we get a slightly better Person correlation with the nearest neighbor averaging using scButterfly embeddings (0.856) than generating proteins directly using scButterfly (0.84), see https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/table_H5.pdf. We acknowledge the concern regarding potential discrepancies between modalities and now add this point to the discussion, highlighting that exploring more sophisticated prediction paradigms is an interesting avenue for future research.\n\nWe sincerely thank you for your thoughtful review and constructive suggestions, which have significantly improved the quality and clarity of our paper. We welcome any additional questions that the reviewer might have.", "claims_and_evidence": "This work benchmarks the performance of representative SSL methods on single-cell data. The comparisons and ablation studies are well-designed, and the results could be a valuable reference for researchers interested in this area.", "methods_and_evaluation_criteria": "The authors evaluated twelve representative SSL methods on eight datasets across three downstream tasks, including batch correction, cell type annotation, and missing modality prediction. The evaluations are comprehensive enough to help understand the effectiveness of different SSL methods on single-cell data. Ablation studies are also conducted to help interpret the importance of each component in the SSL methods.", "theoretical_claims": "This work has no theoretical claims.", "experimental_designs_or_analyses": "The chosen SSL methods and downstream tasks for benchmarking are representative. The experiments are well-designed to reflect the performance of different SSL methods and the effectiveness of different components in method design.", "relation_to_broader_scientific_literature": "This work could be a helpful reference for researchers interested in single-cell representation learning.", "essential_references_not_discussed": "The current benchmarked methods are mostly discriminative ones. Currently, there are some generative single-cell SSL methods such as scBert (scBERT as a large-scale pretrained deep language model for cell type annotation of single-cell RNA-seq data, Nature Machine Intelligence 2022) and Geneformer (Transfer learning enables predictions in network biology, Nature 2023). The authors need to include these generative SSL methods in the benchmark as well.\n\nBesides, the authors are also encouraged to include recent batch correction and data integration methods for single-cell data, such as Single-Cell RNA-Seq Debiased Clustering via Batch Effect Disentanglement (TNNLS 2024), scBridge embraces cell heterogeneity in single-cell RNA-seq and ATAC-seq data integration (Nature Communications 2023), etc.", "other_strengths_and_weaknesses": "The current datasets used for evaluation are all relatively small. It is interesting to see whether the same conclusions could be arrived at on much larger datasets such as the Human Fetal Atlas and Mouse Atlas.\n\nWhen discussing the experimental results, the authors are encouraged to provide more in-depth explanations instead of plain descriptions. For example, in line 300, the authors write, ''these SSL methods prioritize batch correction over bio conservation as indicated by their high batch and low bio score.'' But why? More explanations are expected.", "other_comments_or_suggestions": "When evaluating different representation dimensions, more choices like 128 and 256 are expected.", "questions_for_authors": "I expect the authors to respond to my previous concerns. In addition, for missing modality prediction, the authors use the average of the nearest neighbors in the observed modality as the prediction results. However, the characteristics and neighbors in the two modalities could usually differ. In this case, is such a prediction paradigm reasonable?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uppJXb9Rlb", "reviewer": "Reviewer_m4qB", "summary": "This paper proposed a benchmarking analysis for SSL method's application in single-cell data analysis.\n\n## update after rebuttal\n\nI raised my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express sincere gratitude to the reviewer for providing feedback and raising several points about the validity of the work, which we address below and extend our evaluation accordingly. We hope that the reviewer will consider updating their review score if they find our comments and new results satisfying. \n\n**Motivation:** SSL is recognized for effectively learning robust representations from unlabeled data, enhancing downstream tasks. Many recent studies show the substantial advantages of SSL in single-cell analyses. Richter et al. (www.nature.com/articles/s42256-024-00934-3) demonstrates SSL methods excel in transfer learning, zero-shot, and cross-modality predictions, outperforming supervised approaches. The widely cited methods scVI , totalVI, CLEAR, CLAIRE, and Concerto show SSL's ability to capture biological heterogeneity and manage batch effects, better than supervised methods.\n\n**Additional Baselines:** Based on the reviewers’ suggestions, we added scButterfly and scTEL (www.nature.com/articles/s41540-024-00484-9) for multi-omics. The suggestions significantly improved the benchmark. scButterfly excels in cell typing, but scTEL underperforms in all tasks. However, our findings still indicate that SSL methods outperform other for multi-omics. New results in https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/table_1.pdf, https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/table_H4.pdf, https://anonymous.4open.science/r/scSSL-Bench-Rebuttal/table_H5.pdf. For uni-modal datasets, we added scGPT, scBERT and SCDC (https://ieeexplore.ieee.org/document/10086540), see our responses to X2so and KvUM.\n\nTo clarify, our benchmark focuses on RNA-seq and CITE-seq data (RNA+protein) and is not directly comparable to scGLUE and Monae, which use RNA+ATAC. ATAC is a completely different modality that uses different methods. scGlue/Monae’s documentation covers only RNA+ATAC, without CITE-seq guidance.\n\n**Selected Tasks:** The selected tasks are integral to ongoing single-cell challenges (https://proceedings.mlr.press/v176/lance22a, https://openproblems.bio/results), supported by existing literature (scVI, totalVI, CLEAR, CLAIRE, Concerto), and relevant to a broader community (https://genomebiology.biomedcentral.com/articles/10.1186/s13059-020-1926-6). \n- *Batch Correction* is crucial for identifying true biological variation from experiment variability, e.g., platforms constantly change chemistry settings. Often, one cannot access a previous technology version and instead performs batch correction to unmask the biological signal, see prior studies www.nature.com/articles/s41592-018-0254-1, https://doi.org/10.1093/bioinformatics/btz625, www.nature.com/articles/s41592-021-01336-8, https://pubmed.ncbi.nlm.nih.gov/34062119.   \n- *Cell Typing* maps new cells onto existing reference atlases and is vital for biological discoveries (www.nature.com/articles/s41587-021-01001-7, Concerto, scButterfly). Although the reviewer identifies cell typing as supervised, in the SSL context it is about unsupervised representation learning followed by minimal supervised inference.  \n- *Missing Modality Prediction* infers unseen modalities, tests SSL method's ability to generalize across data types,  and enhances the utility of existing single-modality datasets. The inferred modalities can be used for the improved analysis (https://pubmed.ncbi.nlm.nih.gov/34062119, Concerto, scButterfly).\n\n**Contributions:** We cited Richter et al.from December 2024 in our initial paper and described the distinctions on line 36-55, col.2. To clarify our contributions and differences:\n\n- *Scope:* Richter et al. evaluate masked autoencoders (MAE), BYOL, and Barlow Twins on transfer learning, zero-shot, and fine-tuned SSL scenarios. Our scSSL-Bench assesses a broader range of several SSL methods, specialized and generic.  \n- *Tasks:* Richter et al. focus on transfer learning. We evaluate batch correction, cell typing, and modality prediction for single- and multi-omics, addressing practical challenges from the community.  \n- *Hyperparameters/Augmentations:* We address the impact of hyperparameters and augmentations for single-cell data, contributing valuable insights into optimal SSL configurations.  \n- *Findings:* Richter et al. conclude in favor of MAE over contrastive methods at scale. We highlight different SSL methods perform optimally for different tasks and modalities, noting specialized SSL methods (scVI, CLAIRE) and scFMs excel at uni-modal batch correction, while generic SSL methods (VICReg, SimCLR) dominate for multi-modal data.\n\n**Figure G1** is an overview of benchmarked models showing structural differences. The visualized models can be applied to any input type. As we do not reference modalities in this figure, we're uncertain about the specific concern. We welcome suggestions to improve clarity.\n\nWe appreciate the time you've taken to review our manuscript and provide your comments. We welcome any additional questions you might have.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No, they do not have theoretical claims in the manuscript.", "experimental_designs_or_analyses": "Yes. I have questions about this section, which is provided in my comments.", "relation_to_broader_scientific_literature": "I think the readers in single-cell analysis will be interested in reading this paper, but only limited to these people.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA.", "other_comments_or_suggestions": "Please see my comments.", "questions_for_authors": "The authors proposed a benchmarking analysis for selecting the most suitable approach in performing self supervised learning (SSL) for single-cell data analysis. I have a couple of questions about the novelty and the meaning of this work. If the authors can address them, I may consider increasing the score. Please see my comments below\n\n1. The motivation is not very clear to me. How to justify that the self supers learning approach is very powerful in analyzing single-cell data? Some cited methods such as scCLIP are not published. Could the authors provide more support to strengthen their position of self-supervised learning?\n\n2. The baseline models should include more current research to support their conclusions. For example, there exists a couple of multi-omic data integration approaches which could be treated as potential better baselines (scGLUE, https://github.com/gao-lab/GLUE; scButterfly, https://www.nature.com/articles/s41467-024-47418-x; Monae, https://www.nature.com/articles/s41467-024-53355-6). \n\n3. The selected tasks are also not interesting, especially cell-type annotation and modality prediction, which are actually two supervised learning tasks. We can not directly predict expression profiles or shall have levels without references. If the others can demonstrate the emergent abilities of SSL-based approaches in handling this task, I think it would be very interesting. Otherwise, the authors may focus on more SSL-related tasks, for example, only SSL-based methods can address and other baselines cannot perform, but it will be superbly hard, as simple tasks such as clustering are also well-explored.\n\n4. Could the authors distinguish their contributions and the contributions of this paper (https://www.nature.com/articles/s42256-024-00934-3)? They are all talking about SSL for single-cell data analysis and the conclusions are similar.\n\n5. Figure G1 is also not clear to me. From my understanding, the authors are working on single-cell data. Therefore, why do they include other modalities are examples? I suggest others to revise the figure using single-cell data as an example", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}]

Author rebuttal:

Meta-review: This paper presents scSSL-Bench, a comprehensive and timely benchmarking platform that evaluates 12 SSL methods across 8 single-cell datasets and three downstream tasks. The authors offer detailed methodological comparisons, ablation studies on data augmentations and embedding dimensions, and practical recommendations tailored for batch correction, cell-type annotation, and modality prediction. All reviewers ultimately supported acceptance, with two strong supports acknowledging the paper’s rigor despite concerns of overlap with existing work. Importantly, the authors thoughtfully and thoroughly addressed reviewer concerns, significantly enhancing the benchmark's scope and reproducibility. Given the clarity and comprehensiveness of this benchmark, I recommend acceptance.
Decision: Accept (spotlight poster)