Title: Cell ontology guided transcriptome foundation model
Venue: NeurIPS 2024
Accepted: yes
Keywords: Cell ontology graph+transcriptome foundation model+large-scale pre-training+cell representation learning+single cell RNA sequencing data
Abstract: Transcriptome foundation models (TFMs) hold great promises of deciphering the transcriptomic language that dictate diverse cell functions by self-supervised learning on large-scale single-cell gene expression data, and ultimately unraveling the complex mechanisms of human diseases. However, current TFMs treat cells as independent samples and ignore the taxonomic relationships between cell types, which are available in cell ontology graphs. We argue that effectively leveraging this ontology information during the TFM pre-training can improve learning biologically meaningful gene co-expression patterns while preserving TFM as a general purpose foundation model for downstream zero-shot and fine-tuning tasks. To this end, we present **s**ingle **c**ell, **Cell**-**o**ntology guided TFM (scCello). We introduce cell-type coherence loss and ontology alignment loss, which are minimized along with the masked gene expression prediction loss during the pre-training. The novel loss component guide scCello to learn the cell-type-specific representation and the structural relation between cell types from the cell ontology graph, respectively. We pre-trained scCello on 22 million cells from CellxGene database leveraging their cell-type labels mapped to the cell ontology graph from Open Biological and Biomedical Ontology Foundry. Our TFM demonstrates competitive generalization and transferability performance over the existing TFMs on biologically important tasks including identifying novel cell types of unseen cells, prediction of cell-type-specific marker genes, and cancer drug responses. Source code and model
weights are available at https://github.com/DeepGraphLearning/scCello.

Cell ontology guided transcriptome foundation model

Abstract

Transcriptome foundation models (TFMs) hold great promises of deciphering the transcriptomic language that dictate diverse cell functions by self-supervised learning on large-scale single-cell gene expression data, and ultimately unraveling the complex mechanisms of human diseases. However, current TFMs treat cells as independent samples and ignore the taxonomic relationships between cell types, which are available in cell ontology graphs. We argue that effectively leveraging this ontology information during the TFM pre-training can improve learning biologically meaningful gene co-expression patterns while preserving TFM as a general purpose foundation model for downstream zero-shot and fine-tuning tasks. To this end, we present s ingle c ell, Cell -o ntology guided TFM (scCello). We introduce cell-type coherence loss and ontology alignment loss, which are minimized along with the masked gene expression prediction loss during the pre-training. The novel loss component guide scCello to learn the cell-type-specific representation and the structural relation between cell types from the cell ontology graph, respectively. We pre-trained scCello on 22 million cells from CellxGene database leveraging their cell-type labels mapped to the cell ontology graph from Open Biological and Biomedical Ontology Foundry. Our TFM demonstrates competitive generalization and transferability performance over the existing TFMs on biologically important tasks including identifying novel cell types of unseen cells, prediction of cell-typespecific marker genes, and cancer drug responses. Source code and model weights are available at https://github.com/DeepGraphLearning/scCello.

1 Introduction

Cells are basic units of all living organisms. Deciphering diverse cell functions through gene expression is a long-standing challenge in life science and yet the essential path towards precision and personalized medicine. In this context, single-cell RNA sequencing (scRNA-seq) has emerged as a pivotal technique to measure the gene expression in individual cells. The vast amount of publicly available scRNA-seq data offers a rich transcriptomic data source [48] for learning cell representations towards various research applications, such as cancer therapy [62] and drug discovery [5].

Recently, several Transcriptome Foundation Models (TFMs) were developed to improve cell representation learning. They mainly utilize pre-training methods analogous to natural language processing like masked token prediction, treating genes as 'tokens' and cells as 'sentences' [14, 61, 70, 55]. However, the existing TFMs treat cells as independent samples and ignore their cell-type lineages. On the other hand, prior knowledge of the taxonomic relationships of cell types has been made available through the cell ontology graph by Open Biological and Biomedical Ontology Foundry [4]. Effectively leveraging the ontology knowledge can improve the quality of the pre-training on large-scale scRNA-seq atlases, which are heterogeneous and encompass hundreds of cell types. This can be done by training the TFM to recognize the inherent ontology relationships among cell types, thereby

refining the cell representations. For instance, 'mature α -β T cell' should be closer to 'mature T cells' compared to more general term 'T cells' and farther from neurons and astrocytes from the brain (e.g., Tab. 7).

To capture this intuition, we propose scCello, a s ingle c ell, Cell -o ntology guided TFM. scCello learns cell representation by integrating cell type information and cellular ontology relationships into its pre-training framework. scCello's pre-training framework is structured with three levels of objectives: (1) gene level : a masked token prediction loss to learn gene co-expression patterns, enriching the understanding of gene interactions (Sec. 2.2); (2) intra-cellular level : an ontologybased cell-type coherence loss to encourage cell representations of the same cell type to aggregate, prompting consistency between cells and their types (Sec. 2.3); and (3) inter-cellular level : a relational alignment loss to guide the cell representation learning by consulting the cell-type lineage from the cell ontology graph (Sec. 2.4)..

We demonstrate the generalizability and transferability of scCello on 22 million cells from CellxGene. For model generalization, we observe that scCello excels on cell type identification across all datasets in both zero-shot setting (i.e., directly using the pre-trained model) (Sec. 4.2.1) and fine-tuning setting (Sec. 4.2.2). In particular, scCello accurately classifies novel cell types by leveraging the ontology graph structure (Sec. 4.3). For transferability, scCello demonstrates competitive performances in predicting cell-type-specific marker genes (Sec. 4.4) and cancer drug responses (Sec. 4.5). Additionally, scCello is robust against batch effects (Sec. 4.6). Finally, we validate our contribution via ablation study (Sec. 4.7).

2 Method

Fig. 1 illustrates an overview of scCello. We present the details of individual components below.

2.1 Data Preprocessing

Cell ontology graph. Cell ontology is a widely used metadata schema for standard cell type annotations [16]. We downloaded the ontology from Open Biological and Biomedical Ontology Foundry ( https://obofoundry.org/ ). It is structured as an unweighted directed acyclic graph G = ( V , E ) , where each node v ∈ V corresponds to a distinct cell type and each directed edge ( u, v ) ∈ E denotes a hierarchical lineage relationship of the form "is a subtype of" between cell types (Fig. 1a). To accurately represent the inherently symmetric "being biologically similar" relationship between cell types, the directed graph was transformed into an undirected one for subsequent calculation of cellular ontology relationships in Sec. 2.4.

scRNA-seq data. The scRNA-seq data were downloaded from CellxGene. After the preprocessing (App. B), we obtained 22 million cells. Each single-cell transcriptome is represented by a sequence of tuples, each containing genes and their expression counts. 1 Each sequence was then ordered by the rank of the gene expression values [61] , akin to the sequential ordering of natural languages. Given a batch of B cells, each cell i ∈ { 1 , . . . , B } was assigned a cell type ontology identifier c i ∈ V from the CellxGene database, to enable mapping between cell and cell ontology.

2.2 Masked Gene Prediction

Same as BERT [15], scCello predicts a randomly masked gene token in each cell based on its surrounding context in the sequence. This objective L MGP aims to learn the dynamic gene coexpression network.

2.3 Intra-Cellular Ontology Coherence

A straightforward approach to encourage learning the cell representations that are coherent to the cell type labels is to apply cross-entropy loss for supervised cell type classification. However, this approach is limited in learning cell representation for the foundation model. Instead, we employed a supervised contrastive loss as our objective L Intra , which directly optimizes the TFM rather than

B

D

Ontology Relationship

…

Cells

B

C

𝒉

!

𝒉

#

Regulatory T cell

Genes

C

…

Neuron

1

1

2

2

3

3

𝒛

$

%

𝒛

$

Cell Reprs. Distance

>

>

𝒛

$

%

𝒛

&

𝒛

$

%

𝒛

'

3

2

1

Linear

…

Cell Type

A

A

B

B

C

C

>

𝒛

$

%

𝒛

$

𝒛

$

%

𝒛

&

𝒛

$

%

𝒛

'

Native cell

…

A

Nucleate cell

A

E

Similarity

?

"Cell Type E"

…

…

Distance

Genes

Knockout

Cell Reprs.

Marker

Gene

Logits

(b) scRNA-seq data

(a) Cell Ontology Graph

Cell Reprs.

IC

50

?

+

Cell Reprs.

Cell Type Classification

Novel Cell Type Classification

𝒉

(

𝒛

'

𝒛

$

%

𝒛

𝟑

𝒛

$

%

𝒛

&

?

?

(d) Downstream Tasks

A

(c) scCello Pre-training Framework

𝒛

$

𝒛

&

Relational Alignment

Cell-Type Coherence

>

sim(𝐴, 𝐴)

sim(𝐴, 𝐵)

sim(𝐴, 𝐶)

Guide

Cancer Drug Prediction

Cell Type

Clustering

Batch

Integration

！

merely learning through the linear classifier:

̸

where z i and h c i denote the latent representation of cell i and cell type c i , respectively.

This supervised contrastive loss pulls representations of the same class (positives) and repels representations of different classes (negatives). It often leads to representations that are at least as discriminative as the cross-entropy loss [22]. To reduce the degrees of freedom available for TFM optimization, we introduce a regularization term L Reg :

where the linear layer is shared across all cells and cell types. Thereby, it constrains the cell type representation space to be an affine transformation of the cell representation space.

2.4 Inter-Cellular Relational Alignment

To encourage TFMs to learn inter-cellular ontology relationships, scCello forces cell representations to truthfully reflect the pairwise node structural similarity derived from the cell ontology graph, using a relational alignment objective. This objective constitutes the most important part of scCello.

Ontology relationships. To effectively quantify ontology relationships between cell types from the ontology graph, scCello estimates pairwise node structural similarities as proxies using Personalized PageRank (PPR) [20]. PPR is a graph learning algorithm. The PPR score PPR( u, v ) estimates the probability for a random walk. It starts from a given target node u ∈ V and terminates at another node v ∈ V . Importantly, this is a context-sensitive structural similarity measure that accounts both direct connections and broader subgraph patterns [67]. It also provides robustness against variations in global network structures, such as variable node degrees and clustering coefficients [11]. To improve robustness (as justified in App. A), we transform PPR( · ) through a non-linear function to derive the structural similarities sim( · ) as ontology relationships tunable by a hyper-parameter threshold s :

̸

Relational alignment. Cells with closely related cell types tend to be more similar than those with distinct cell types. This observation guides scCello to align the distances between cell representations w.r.t. a target cell, with their structural similarities sim( · ) (as shown in Fig. 1c). Specifically, given a batch of B cells, if we consider a target cell i and another cell in the batch j = i , the representation distance z T i z j should reflect their structural similarity sim( c i , c j ) . Accordingly, a negative sample set Ω i,j = { k | sim( c i , c j ) > sim( c i , c k ) , 1 ≤ k ≤ B } can be produced, where cell pair ( i, k ) are considered less similar to the cell pair ( i, j ) and should be contrasted against in the representation space using the objective L Inter :

̸

Notably, ancestor cell types, which can reach the target cell type via the directed "is a subtype of" edge on the ontology graph, are structurally distant from the target cell type. Despite being distant, they fall into the same, broader cell type category. Contrasting cells associated with these distant ancestor cell types with the target cell is counter-intuitive. Therefore, scCello explicitly excludes such cells from the negative sample set, avoiding inappropriately pushing away biologically similar cells. This enhances scCello's capability to discern subtle similarities and differences within the cell types.

2.5 Overall Pre-training Objective

During pre-training, we seek to minimize the loss functions of all pre-training tasks simultaneously:

where θ denotes all learnable parameters in scCello, which adopts transformer stacks as model backbones. We state the detailed information of model architectures in App. D.

3 Related Work

The rapid growth of scRNA-seq datasets has opened new avenues for constructing TFMs, enabling transfer learning across various biological downstream tasks. Initial efforts, such as scBERT [73], Exceiver [13] and Geneformer [61], borrows the concept of masked language modeling [15] from natural language processing (NLP) domain for pre-training, by treating cells as sentences and genes as tokens. Concurrently, tGPT [59] and scGPT [14] explored generative modeling [53], and CellLM [76]

adapted the idea of contrastive learning [42]. Following the concept of 'scaling' towards emergent ability [70] in NLP, scFoundation [25] proposes the largest foundation model at the time in terms of model size and pre-training data size; scHyena [49] scales modeling context window size to the full length of scRNA-seq data with Hyena operator [51] instead of conventionally used transformers. scTab [18] is the first to explore large-scale supervised learning mechanism for scRNA-seq pretraining, and is capable of annotating unseen tissue cells for real-world applications. Moreover, SCimilarity [28] and UCE [55] focus on developing a unified latent space as a large-scale reference atlas for querying new cells. Yet, these TFMs mainly treat cells as independent samples during training and ignore their biological ontology relationships. scCello bridges this gap by incorporating cell type relationships derived from the cell ontology graph into TFM pre-training. This strengthens TFMs' model generalization and transferability capability, as shown in Sec. 4.

4 Experiments

As an overview, the following experiments show that, (1) scCello can generalize to unseen cells, and to more difficult settings, such as cells of unseen cell types, tissues, and donors (Sec. 4.2.1); (2) scCello can benefit from fine-tuning on target datasets (Sec. 4.2.2); (3) the structural similarity embedded in scCello helps to classify novel cell types in a zero-shot manner (Sec. 4.3); (4) scCello effectively transfers to different downstream tasks (Sec. 4.4 and Sec. 4.5); (5) scCello is robust to batch effects that arise from different experimental conditions (Sec. 4.6); (6) Each loss component in Eqn. 5 is beneficial to scCello (Sec. 4.7). For every table reported, we used bold to highlight the best performance and results within 0.005 difference from the best. We used underlining to denote the second-best performances. For all metrics, ↑ indicates the higher the better.

4.1 Setups

Pre-training and downstream datasets. We collected a large pre-training dataset consisting of 22 million cells along with downstream datasets. In particular, we generated one in-distribution (ID) and six out-of-distribution (OOD) datasets (App. B). The ID dataset is denoted as D id . For the OOD setting, we introduced three scenarios: unseen c ell t ypes ( { D ct i } 2 i =1 ), unseen cell t i s sues ( { D ts i } 2 i =1 ), and unseen d o n ors ( { D dn i } 2 i =1 ). Each scenario has two datasets. Notably, the OOD donor setting presents more realistic challenges than ID and other OOD settings because of the potential batch effects in the test donors.

Pre-training configurations. An Adam optimizer [38] (learning rate: 0 . 001 , weight decay: 0 . 001 , warm-up steps: 3 , 333 ) was used to train the scCello for 40 , 000 steps on 4 NVIDIA A100 GPUs on Compute Canada. We used 192 for batch size. More details are introduced in App. D.

Baselines. Across all downstream tasks, scCello is benchmarked with leading open-source largescale TFMs: Geneformer [61], scGPT [14], scTab [18], UCE [55], and three TFM ablations. We also implemented ablated versions of scCello that only differ in the pre-training objectives from scCello: scCello using only the masked gene prediction loss (denoted as MGP), scCello using only the cell type supervised classification (denoted as Sup), and scCello using only the two losses (denoted as MGP+Sup). The three ablated TFMs provide a reference to isolate the effect of implementation details and training configurations. For each task, we also selected state-of-the-art non-TFM methods for fair comparison.

Downstream metrics. We evaluated the 3 tasks by the following metrics. (1) Clustering metrics include normalized mutual information (NMI), adjusted rand index (ARI), average silhouette width (ASW), and the average of the 3 scores (AvgBio) to assess both between-cluster separation and within-cluster closeness [14]. The batch integration task (Sec. 4.6) is evaluated by ASW b , graph connectivity (GraphConn) and their average (AvgBatch), along with an overall score (Overall = 0 . 6 × AvgBio + 0 . 4 × AvgBatch) to balance biological relevance and batch consistency following [14]. (2) Classification metrics include accuracy (Acc), Macro F1 and area under the ROC curve (AUROC) [50]. (3) Regression task metrics include Pearson correlation coefficient score (PCC) [50]. Details for each metric were provided in App. E.1.

1 Harmony could be over-corrected w.r.t. batch labels for datasets with many batches [10].

4.2 Cell Type Identification

4.2.1 Zero-shot Cell Clustering Results

Setup. For the cell type clustering task, TFM baselines and four non-TFM methods were evaluated: (1) raw data expressions of highly variable genes ( abbr. , Raw Data) [34]; (2) Seurat [27]; (3) Harmony [40] (4) scVI [44]. Cell representations were extracted from the baselines and clustered by Louvain algorithm [6]. We evaluated the clustering performance of each method on both ID dataset D id and OOD datasets D cond i ( cond ∈{ ct, ts, dn } , i ∈{ 1 , 2 } ).

ID and OOD generalization. We reported zero-shot cell type clustering performance in Tab. 1, and included all the metrics for all datasets in App. E.2.1 due to space constraint. For both the ID and OOD settings, scCello consistently outperforms all baselines, achieving a 16.1% improvement in AvgBio on the ID dataset and a 12.1% improvement in average AvgBio across the six OOD datasets. Interestingly, while scCello outperforms non-TFM methods by a large margin, Geneformers and scGPT barely surpass these methods. The latter is consistent with previous observations [75].

In the OOD experiments, scCello confers strong generalization capability across unseen cell types tissue, and donors. In cell type clustering, scCello is the second best only trailing UCE by 0.03 and the best method for dataset 1 and 2. The OOD tissue setting highlights scCello's ability to transfer its learned knowledge to different unseen tissues. Specifically, scCello achieve 0.6 and 0.7 while most methods conferred below 0.6 and 0.7 for the two datasets, respectively. For the unseen OOD donor scenario, most methods perform poorly with AvgBio ranging between 0.45 and 0.55. scCello led the chart achieving AvgBio above 0.6 in both datasets. Overall, scCello showcases strong model generalization capabilities across a range of biological conditions, which is attributable to the integration of cell ontology priors during its TFM pre-training. Indeed, the ablated models namely MGP, Sup, and MGP+Sup conferred lower scores compared to the full model.

4.2.2 Fine-tuning Results

Setup. We benchmarked all TFM baselines except UCE for its lack of fine-tuning support. These TFMs were fine-tuned on a subset of our pre-training data with supervised classification loss (details in App. E.2.2). We assessed both classification and clustering performance on the ID dataset D id . We also compared with a non-TFM method, scANVI [72].

Improvement with fine-tuning. In Tab. 2, The fine-tuned scCello outperforms other TFMs and non-TFM methods on both classification and clustering metrics, achieving up to 25.9% improvement in Macro F1 over the best baseline. Moreover, scCello without fine-tuning still surpasses the performance of the other fine-tuned methods, further highlighting its superior transferability.

4.3 Novel Cell Type Classification

Novel cell type classification aims to label cells of unseen cell types without further fine-tuning. This task is useful for annotating completely new scRNA-seq datasets but infeasible for most of the supervised methods that solely rely on the labels observed in the training data [8, 31, 68]. Leveraging the cell ontology graph that comprises the lineage relations among all of the known cell types, scCello makes this task feasible.

Setup. Our goal is to classify new query cells into "novel cell types" not seen during pre-training. To do this, we generate representations for both query cells and novel cell types, using similarity measures for classification. This process involves utilizing similarities between TFM-derived representations for the former and biological relationships from the cell ontology graph for the later. Details were described in App. E.3.

We benchmarked all TFMs and evaluated them on OOD cell type datasets D ct 1 and D ct 2 . We increased the difficulty of this task by the number of novel cell types (#Cell Types) that exist among the query cells. Specifically, we simulated five difficulty levels, with the number of novel cell types ranging from 10% to 100% of the total cell types. To assess the variance of the performance, we randomly sampled cell type combinations 20 times at each level.

OOD generalization. In Fig. 2, scCello led other TFMs by a large margin, achieving up to 76.8% Acc to classify 9 novel cell types (i.e., 10% of the total heldout cell types) and 33.5% Acc to classify up to 87 novel cell types (i.e., 100% of the total heldout cell types) (Tab. 16 and Tab. 17). These results show a significant leap from the existing TFMs, which either do not work or only work for annotating a handful of novel types [68, 45, 66].

4.4 Marker Gene Prediction

Cell-type-specific genes, or marker genes, are highly expressed in a specific cell type but exhibit low expression in others. These genes play a crucial role in delineating cell functions in diverse tissue contexts. Identifying marker genes in less characterized cell types is an ongoing challenge [52].

Setup. We sought to assess whether the pre-trained TFMs can discriminate marker from non-marker genes for any cell type without any supervised fine-tuning. This zero-shot experiment evaluates whether the TFM is able to learn biologically meaningful gene co-expression patterns without supervision. For each cell, we quantified the marker gene potential of each gene by the changes in TFM-generated cell representations after in-silico knockout of the target gene (details in App. E.4). Here we assume that the larger the change the higher the marker gene potential. We discussed the

1.0

Geneformer

MGP

Sup

0.8

scTab

MGP+Sup

cc

UCE

scCello

0.6

0.4

0.2

0.0

10%

25%

50%

75%

100%

Portion of the Total #Cell Types

Raw Data

scVI

scTab

Sup

Seurat

Geneformer

UCE

MGP+Sup

Harmony

MGP

+

scCello

Datasets

Did

D

D

Dts

O

O

Dts

an

Den

0.55

0.60

0.65

0.70

0.75

0.80

OverallScore

caveat of this approach in Sec. 5. As test data, we used GSE96583 [32] ( D mk 1 ) and GSE130148 [65] ( D mk 2 ) . We obtained the marker gene labels from CellMarker2 [29] and PanglaoDB [21]. We also compared with a non-TFM method, Differential Expression Tests (DET) [60].

Zero-shot transferability. In Tab. 3, scCello outperforms other TFMs, improving upon the secondbest method by 1.8% in average AUROC. The inclusion of cell label information during pre-training boosts TFM performance, as evidenced by the strong results of scTab, Sup, MGP+Sup and scCello. This is due to the biological correlation between marker genes and cell types. Furthermore, employing cell ontology graphs further improves the prediction accuracy over MGP+Sup.

4.5 Cancer Drug Response Prediction

Developing effective drugs for cancer treatment is challenging due to individual variability in drug responses. Accurately predicting cancer drug responses (CDR) can greatly aid anti-cancer drug development and improve our understanding of cancer biology [43].

Setup. Following the approach of scFoundation [26], cell representations were extracted from fixed TFMs and integrated into the DeepCDR [43] pipeline to estimate the half-maximal inhibitory concentration ( IC 50 ) values of drugs (details in App. E.6). We benchmarked our method against DeepCDR, scFoundation, and other TFM baselines, using the same pre-processed data as DeepCDR.

Zero-shot transferability. In Tab. 4, scCello is among the top 3 along with scGPT and UCE, achieving 7.4% improvement in PCC over the base method DeepCDR. This highlights scCello's transferability in enhancing specialized task-oriented methods. In particular, it can be used as an powerful feature extractor for diverse downstream tasks.

4.6 Batch Integration

The scRNA-seq atlases, assembled from datasets across various labs and conditions, are prone to unwanted technical variations known as batch effects [46]. These effects can significantly affect the generalization ability of TFMs especially because they require pre-training on a massive amount of heterogeneous scRNA-seq data pooled from many studies. Here we sought to evaluate scCello's robustness to batch effects without fine-tuning.

Setup. We adopted the same baselines as in zero-shot cell type clustering (Sec. 4.2.1), and followed the evaluation protocol of scGPT [14]. We evaluated on one ID dataset D id and six OOD datasets D cond i ( cond ∈ { ct, ts, dn } , i ∈ { 1 , 2 } ) (see complete results of all metrics in App. E.7).

Robustness to data noise. Fig. 3 shows that scCello excels in 3 out of 7 datasets, and achieves comparable performance on another 3 datasets. The performance is attributable to the use of cell type information as the ablated baseline MGP conferred much lower batch integration score compared to Sup and scCello.

4.7 Ablation Study

Ablation of pre-training losses. Tab. 5 reports the cell type clustering (Sec. 4.2.1) and novel cell type classification (Sec. 4.3) performance of scCello by using full or partial pre-training losses. Removing any of the four losses in Eqn. 5 resulted in decreased performance, corroborating the benefits of the proposed pre-training losses. Notably, removing the inter-cellular ontology relation loss L Inter led to 56.1% and 65.3% decrease in terms of Acc. and Macro F1 on novel cell type classification task, respectively. This shows the upmost importance of the structurally induced loss and ultimately the use of cell ontology graph information.

Parameter efficiency. Tab. 6 demonstrates that scCello is highly parameter-efficient, utilizing up to 60 times fewer parameters than the largest existing TFM, UCE, while still achieving the best average performance rankings across all downstream tasks. With an average performance rank of 1.3, scCello consistently ranks first or near the top in nearly every task.

Visualization. Visualization and analysis of scCello's learned cell representations were presented in App. E.8. In short, biologically similar cell types are closer to each other and farther from those dissimilar ones in the t-SNE 2D space (Fig. 11).

5 Discussion and Conclusion

Limitation and future work. The cell ontology is constantly revised and expanded. In the future, we plan to investigate more efficient methods for fine-tuning scCello to enable continual learning of updated ontology, rather than retraining the entire model. Additionally, we aim to scale up the model size of scCello to increase its expressiveness and capacity. For the zero-shot marker gene prediction experiments (Sec. 4.4), one caveat is that our in-silico gene knockout approach also detects essential genes such as housekeeping genes [17] and transcription factors that are master regulators [9], which may not necessarily be marker genes. Nonetheless, deletion of these influential genes will also lead to large change of the transcriptome landscape of the cell. We will explore this in future study.

Societal impact. This work proposes a novel cell ontology-guided TFM, scCello, to enhance cell representation learning. On the positive side, once pre-trained, scCello can serve as a foundational model capable of facilitating scientific discoveries across various downstream tasks related to cells and cellular processes. However, on the negative side, the pre-training of scCello requires significant computational resources, potentially resulting in substantial carbon dioxide emissions that could contribute to environmental harm.

Conclusion. The proposed scCello incorporates cell ontology knowledge into its pre-training process by simultaneously modeling at the gene level, intra-cellular level, and inter-cellular level. We constructed a large-scale cell type identification benchmark to evaluate the model's generalization capabilities, both in-distribution and out-of-distribution. Our evaluation demonstrates that scCello also exhibits strong transferability, as evidenced by its performance on other biologically meaningful downstream tasks such as zero-shot novel cell type classification and cell-type-specific marker gene prediction. Foundational models are typically heavy on the parameters for them to have sufficient capacity to learn from unlabeled data from scratch. This limits their usage to only fine-tuning tasks as pre-training them is prohibitive without large compute. Our proposed approach provides an efficient way of leveraging the prior knowledge at the pre-training, which led to much smaller parameter size while achieving performance comparable of the TFMs that are 5-60 times bigger. Together, scCello is a knowledge-informed and general purpose deep learning model that can be fine-tuned

for a wide array of downstream applications, aiding in the rapid identification of novel cell types, disease-associated genes, and effective cancer drugs.

Acknowledgments and Disclosure of Funding

The authors would like to thank Chence Shi, Meng Qu, Zhaocheng Zhu, and Sophie Xhonneux for their helpful discussions and comments. We also appreciate all anonymous reviewers for their constructive suggestions. This project is supported by Intel-MILA partnership program, the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronics Co., Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bird Gift Fund and a NRC Collaborative R&D Project (AI4D-CORE-06). This project was also partially funded by IVADO Fundamental Research Project grant PRF-2019-3583139727. Y.L. is supported by Canada Research Chair (Tier 2) in Machine Learning for Genomics and Healthcare (CRC-2021-00547) and Natural Sciences and Engineering Research Council(NSERC) Discovery Grant (RGPIN-2016-05174). The computation resource of this project is supported by Mila, Calcul Québec and the Digital Research Alliance of Canada.

References

Shibla Abdulla, Brian D. Aevermann, Pedro Assis, Seve Badajoz, Sidney M. Bell, Emanuele Bezzi, Batuhan Cakir, Jim Chaffer, Signe Chambers, J. Michael Cherry, Tiffany Chi, Jennifer Chien, Leah Dorman, Pablo Garcia-Nieto, Nayib Gloria, Mim Hastie, Daniel Hegeman, Jason Hilton, Timmy Huang, Amanda Infeld, Ana-Maria Istrate, Ivana Jelic, Kuni Katsuya, Yang-Joon Kim, Karen Liang, Mike Lin, Maximilian Lombardo, Bailey Marshall, Bruce Martin, Fran McDade, Colin Megill, Nikhil Patel, Alexander V. Predeus, Brian Raymor, Behnam Robatmili, Dave Rogers, Erica Rutherford, Dana Sadgat, Andrew Shin, Corinn Small, Trent Smith, Prathap Sridharan, Alexander Tarashansky, Norbert Tavares, Harley Thomas, Andrew Tolopko, Meg Urisko, Joyce Yan, Garabet Yeretssian, Jennifer Zamanian, Arathi Mani, Jonah Cool, and Ambrose J. Carr. Cz cell×gene discover: A single-cell data platform for scalable exploration, analysis and modeling of aggregated data. bioRxiv , 2023.

Peter Angel and Michael Karin. The role of jun, fos and the ap-1 complex in cell-proliferation and transformation. Biochimica et Biophysica Acta (BBA)-Reviews on Cancer , 1072(2-3):129157, 1991.

Sercan Ö Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 6679-6687, 2021.

Jonathan Bard, Seung Y Rhee, and Michael Ashburner. An ontology for cell types. Genome biology , 6:1-5, 2005.

Nurken Berdigaliyev and Mohamad Aljofan. An overview of drug discovery and development. Future medicinal chemistry , 12(10):939-947, 2020.

Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment , 2008(10):P10008, 2008.

Felipe A. Vieira Braga, Gozde Kar, Marijn Berg, Orestes A. Carpaij, Krzysztof Pola´ nski, Lukas M. Simon, Sharon Brouwer, Tomás Gomes, Laura Hesse, Jian Jiang, Eirini Sofia Fasouli, Mirjana Efremova, Roser Vento-Tormo, Carlos Talavera-López, Marnix R. Jonker, Karen Affleck, Subarna Palit, Paulina M. Strzelecka, Helen V. Firth, Krishnaa T. Mahbubani, Ana Cvejic, Kerstin B. Meyer, Kourosh Saeb-Parsy, Marjan A. Luinge, Corry-Anke Brandsma, Wim Timens, Ilias Angelidis, Maximilian Strunz, Gerard H. Koppelman, Antoon J. M. van Oosterhout, Herbert B. Schiller, Fabian J Theis, Maarten van den Berge, Martijn C. Nawijn, and Sarah A. Teichmann. A cellular census of human lungs identifies novel cell states in health and in asthma. Nature Medicine , 25:1153 - 1163, 2019.

Maria Brbic, Marinka Zitnik, Sheng Wang, Angela Oliveira Pisco, Russ B. Altman, Spyros Darmanis, and Jure Leskovec. Mars: discovering novel cell types across heterogeneous singlecell experiments. Nature Methods , 17:1200 - 1206, 2020.

Sunny Sun-Kin Chan and Michael Kyba. What is a master regulator? Journal of stem cell research & therapy , 3, 2013.

Ruben Chazarra-Gil, Stijn van Dongen, Vladimir Yu Kiselev, and Martin Hemberg. Flexible comparison of batch correction methods for single-cell rna-seq using batchbench. Nucleic acids research , 49(7):e42-e42, 2021.

Fan Chen, Yini Zhang, and Karl Rohe. Targeted sampling from massive block model graphs with personalized pagerank. Journal of the Royal Statistical Society Series B: Statistical Methodology , 82(1):99-126, 2020.

David Combe, Christine Largeron, Mathias Géry, and El˝ od Egyed-Zsigmond. I-louvain: An attributed graph clustering method. In Advances in Intelligent Data Analysis XIV: 14th International Symposium, IDA 2015, Saint Etienne. France, October 22-24, 2015. Proceedings 14 , pages 181-192. Springer, 2015.

William Connell, Umair Khan, and Michael J Keiser. A single-cell gene expression language model. arXiv preprint arXiv:2210.14330 , 2022.

Haotian Cui, Chloe X. Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. scgpt: toward building a foundation model for single-cell multi-omics using generative ai. Nature methods , 2024.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Alexander D. Diehl, Terrence F. Meehan, Yvonne M. Bradford, Matthew H. Brush, Wasila M. Dahdul, David S. Dougall, Yongqun He, David Osumi-Sutherland, Alan Ruttenberg, Sirarat Sarntivijai, Ceri E. Van Slyke, Nicole A. Vasilevsky, Melissa A. Haendel, Judith A. Blake, and Christopher J. Mungall. The cell ontology 2016: enhanced content, modularization, and ontology interoperability. Journal of biomedical semantics , 7(44), 2016.

Eli Eisenberg and Erez Y Levanon. Human housekeeping genes, revisited. TRENDS in Genetics , 29(10):569-574, 2013.

Felix Fischer, David S Fischer, Evan Biederstedt, Alexandra-Chloé Villani, and Fabian J Theis. Scaling cross-tissue single-cell annotation models. bioRxiv , 2023.

Felix Fischer, David S. Fischer, Evan Biederstedt, Alexandra-Chloé Villani, and Fabian J. Theis. Scaling cross-tissue single-cell annotation models. bioRxiv , 2023.

Dániel Fogaras, Balázs Rácz, Károly Csalogány, and Tamás Sarlós. Towards scaling fully personalized pagerank: Algorithms, lower bounds, and experiments. Internet Mathematics , 2(3):333-358, 2005.

Oscar Franzén, Li-Ming Gan, and Johan LM Björkegren. Panglaodb: a web server for exploration of mouse and human single-cell rna sequencing data. Database , 2019:baz046, 2019.

Florian Graf, Christoph Hofer, Marc Niethammer, and Roland Kwitt. Dissecting supervised contrastive learning. In International Conference on Machine Learning , pages 3821-3830. PMLR, 2021.

Aric Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANL), Los Alamos, NM(United States), 2008.

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Le Song, and Xuegong Zhang. Large scale foundation model on single-cell transcriptomics. bioRxiv , pages 2023-05, 2023.

Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Le Song, and Xuegong Zhang. Large scale foundation model on single-cell transcriptomics. bioRxiv , 2023.

Yuhan Hao, Tim Stuart, Madeline H Kowalski, Saket Choudhary, Paul Hoffman, Austin Hartman, Avi Srivastava, Gesmira Molla, Shaista Madad, Carlos Fernandez-Granda, et al. Dictionary learning for integrative, multimodal and scalable single-cell analysis. Nature biotechnology , 42(2):293-304, 2024.

Graham Heimberg, Tony Kuo, Daryle DePianto, Tobias Heigl, Nathaniel Diamant, Omar Salem, Gabriele Scalia, Tommaso Biancalani, Shannon Turley, Jason Rock, et al. Scalable querying of human cell atlases via a foundational model reveals commonalities across fibrosis-associated macrophages. bioRxiv , pages 2023-07, 2023.

Congxue Hu, Tengyue Li, Yingqi Xu, Xinxin Zhang, Feng Li, Jing Bai, Jing Chen, Wenqi Jiang, Kaiyue Yang, Qi Ou, et al. Cellmarker 2.0: an updated database of manually curated cell markers in human/mouse and web tools based on scrna-seq data. Nucleic Acids Research , 51(D1):D870-D876, 2023.

Congxue Hu, Tengyue Li, Yingqi Xu, Xinxin Zhang, Feng Li, Jing Bai, Jingrun Chen, Wenqi Jiang, Kaiyue Yang, Qi Ou, Xia Li, Peng Wang, and Yunpeng Zhang. Cellmarker 2.0: an updated database of manually curated cell markers in human/mouse and web tools based on scrna-seq data. Nucleic Acids Research , 51:D870 - D876, 2022.

Aleksandr Ianevski, Anil K. Giri, and Tero Aittokallio. Fully-automated and ultra-fast cell-type identification using specific marker combinations from single-cell transcriptomic data. Nature Communications , 13, 2022.

Hyun Min Kang, Meena Subramaniam, Sasha Targ, Michelle Nguyen, Lenka Maliskova, Elizabeth McCarthy, Eunice Wan, Simon Wong, Lauren Byrnes, Cristina M Lanata, et al. Multiplexed droplet single-cell rna-sequencing using natural genetic variation. Nature biotechnology , 36(1):89-94, 2018.

Hyun Min Kang, Meena Subramaniam, Sasha Targ, Michelle Ly Thai Nguyen, Lenka Maliskova, Elizabeth E. McCarthy, Eunice Wan, Simon Wong, Lauren E. Byrnes, Cristina M. Lanata, Rachel E. Gate, Sara Mostafavi, Alexander Marson, Noah A. Zaitlen, Lindsey A. Criswell, and Chun Jimmie Ye. Multiplexed droplet single-cell rna-sequencing using natural genetic variation. Nature biotechnology , 36:89 - 94, 2017.

Kasia Zofia Kedzierska, Lorin Crawford, Ava Pardis Amini, and Alex X Lu. Assessing the limits of zero-shot foundation models in single-cell biology. bioRxiv , pages 2023-10, 2023.

Hadas Keren-Shaul, Ephraim Kenigsberg, Diego Adhemar Jaitin, Eyal David, Franziska Paul, Amos Tanay, and Ido Amit. Mars-seq2. 0: an experimental and analytical pipeline for indexed sorting combined with single-cell rna sequencing. Nature protocols , 14(6):1841-1862, 2019.

Shima Khoshraftar and Aijun An. A survey on graph representation learning methods. ACM Transactions on Intelligent Systems and Technology , 15(1):1-55, 2024.

Cheorl-Ho Kim. Glycobiology of innate immunology . Springer, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

Ilya Korsunsky, Nghia Millard, Jean Fan, Kamil Slowikowski, Fan Zhang, Kevin Wei, Yuriy Baglaenko, Michael Brenner, Po-ru Loh, and Soumya Raychaudhuri. Fast, sensitive and accurate integration of single-cell data with harmony. Nature methods , 16(12):1289-1296, 2019.

Samuel A Lambert, Arttu Jolma, Laura F Campitelli, Pratyush K Das, Yimeng Yin, Mihai Albu, Xiaoting Chen, Jussi Taipale, Timothy R Hughes, and Matthew T Weirauch. The human transcription factors. Cell , 172(4):650-665, 2018.

Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. Ieee Access , 8:193907-193934, 2020.

Qiao Liu, Zhiqiang Hu, Rui Jiang, and Mu Zhou. Deepcdr: a hybrid graph convolutional network for predicting cancer drug response. bioRxiv , 2020.

Romain Lopez, Jeffrey Regier, Michael B Cole, Michael I Jordan, and Nir Yosef. Deep generative modeling for single-cell transcriptomics. Nature methods , 15(12):1053-1058, 2018.

Mohammad Lotfollahi, Mohsen Naghipourfar, Malte D. Luecken, Matin Khajavi, Maren Büttner, Marco Wagenstetter, Žiga Avsec, Adam Gayoso, Nir Yosef, Marta Interlandi, Sergei Rybakov, Alexander V. Misharin, and Fabian J Theis. Mapping single-cell data to reference atlases by transfer learning. Nature Biotechnology , 40:121 - 130, 2021.

Malte D. Luecken, Maren Büttner, Kridsadakorn Chaichoompu, Anna Danese, Marta Interlandi, MF Mueller, D Strobl, Luke Zappia, Martin Dugas, Maria Colomé-Tatché, and F Theis. Benchmarking atlas-level data integration in single-cell genomics. Nature Methods , 19:41 - 50, 2020.

Malte D Luecken, Maren Büttner, Kridsadakorn Chaichoompu, Anna Danese, Marta Interlandi, Michaela F Müller, Daniel C Strobl, Luke Zappia, Martin Dugas, Maria Colomé-Tatché, et al. Benchmarking atlas-level data integration in single-cell genomics. Nature methods , 19(1):41-50, 2022.

Colin Megill, Bruce Martin, Charlotte Weaver, Sidney Bell, Lia Prins, Seve Badajoz, Brian McCandless, Angela Oliveira Pisco, Marcus Kinsella, Fiona Griffin, et al. Cellxgene: a performant, scalable exploration platform for high dimensional sparse matrices. bioRxiv , pages 2021-04, 2021.

Gyutaek Oh, Baekgyu Choi, Inkyung Jung, and Jong Chul Ye. schyena: Foundation model for full-length single-cell rna-seq analysis in brain. arXiv preprint arXiv:2310.02713 , 2023.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikitlearn: Machine learning in python. the Journal of machine Learning research , 12:2825-2830, 2011.

Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning , pages 28043-28078. PMLR, 2023.

Yixuan Qiu, Jiebiao Wang, Jing Lei, and Kathryn Roeder. Identification of cell-type-specific marker genes from co-expression patterns in tissue samples. Bioinformatics , 37(19):3228-3234, 2021.

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.

William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association , 66(336):846-850, 1971.

Yanay Rosen, Yusuf Roohani, Ayush Agrawal, Leon Samotorcan, Tabula Sapiens Consortium, Stephen R Quake, and Jure Leskovec. Universal cell embeddings: A foundation model for cell biology. bioRxiv , pages 2023-11, 2023.

Peter J Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics , 20:53-65, 1987.

T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. arXiv preprint arXiv:2303.10993 , 2023.

Robert Salomon, Dominik Kaczorowski, Fatima Valdes-Mora, Robert E Nordon, Adrian Neild, Nona Farbehi, Nenad Bartonicek, and David Gallego-Ortega. Droplet-based single cell rnaseq tools: a practical guide. Lab on a Chip , 19(10):1706-1727, 2019.

Hongru Shen, Jilei Liu, Jiani Hu, Xilin Shen, Chao Zhang, Dan Wu, Mengyao Feng, Meng Yang, Yang Li, Yichen Yang, et al. Generative pretraining from large-scale transcriptomes for single-cell deciphering. Iscience , 26(5), 2023.

Charlotte Soneson and Mark D Robinson. Bias, robustness and scalability in single-cell differential expression analysis. Nature methods , 15(4):255-261, 2018.

Christina V. Theodoris, Ling Xiao, Anant Chopra, Mark D. Chaffin, Zeina R Al Sayed, Matthew C. Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng, X. Shirley Liu, and Patrick T. Ellinor. Transfer learning enables predictions in network biology. Nature , 618:616624, 2023.

Ander Urruticoechea, Ramon Alemany, J Balart, Alberto Villanueva, Francesc Vinals, and Gabriel Capella. Recent advances in cancer therapy: an overview. Current pharmaceutical design , 16(1):3-10, 2010.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Petar Veliˇ ckovi´ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

Felipe A Vieira Braga, Gozde Kar, Marijn Berg, Orestes A Carpaij, Krzysztof Polanski, Lukas M Simon, Sharon Brouwer, Tomás Gomes, Laura Hesse, Jian Jiang, et al. A cellular census of human lungs identifies novel cell states in health and in asthma. Nature medicine , 25(7):11531163, 2019.

Hui Wan, Liang Chen, and Min Deng. scemail: Universal and source-free annotation method for scrna-seq data with novel cell-type perception. Genomics, Proteomics & Bioinformatics , 20:939 - 958, 2022.

Hanzhi Wang, Zhewei Wei, Junhao Gan, Sibo Wang, and Zengfeng Huang. Personalized pagerank to a target node, revisited. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 657-667, 2020.

Wenchuan Wang, Fan Yang, Yuejing Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data. Nature Machine Intelligence , 4:852 - 866, 2022.

Xiliang Wang, Yao He, Qiming Zhang, Xianwen Ren, and Zemin Zhang. Direct comparative analyses of 10x genomics chromium and smart-seq2. Genomics, Proteomics and Bioinformatics , 19(2):253-266, 2021.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 , 2022.

F Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology , 19:1-5, 2018.

Chenling Xu, Romain Lopez, Edouard Mehlman, Jeffrey Regier, Michael I Jordan, and Nir Yosef. Probabilistic harmonization and annotation of single-cell transcriptomics data with deep generative models. Molecular systems biology , 17(1):e9620, 2021.

Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell rna-seq data. Nature Machine Intelligence , 4(10):852-866, 2022.

Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems , 34:2879828810, 2021.

Hongyu Zhao, Tianyu Liu, Kexing Li, Yuge Wang, and Hongyu Li. Evaluating the utilities of large language models in single-cell data analysis. 2023.

Suyuan Zhao, Jiahuan Zhang, and Zaiqing Nie. Large-scale cell representation learning via divide-and-conquer contrastive learning. arXiv preprint arXiv:2306.04371 , 2023.

random walk path from

u

to

k

random walk path from

u

to

v

v

k

u

…

v

k

u

…

…

PPR

𝑢, .

Native cell

Nucleate cell

Regulatory T cell

PPR

𝑢, 𝑘

PPR

𝑢, 𝑣

…

v

k

u

…

Cell Ontology Graph

Personalized PageRank (PPR)

d

A PPR Transformation

Personalized PageRank (PPR). Personalized PageRank (PPR) extends the classic PageRank algorithm, which Google originally developed to rank web pages in search engines. PageRank conducts this by analyzing large-scale hyperlinked graphs on the web using random walker simulations. Unlike traditional PageRank that assigns a universal score to each web page, PPR customizes these scores. Specifically, individual user preferences during searches are incorporated, so that PPR can focus on web pages particularly relevant to each user. Due to its flexibility and effectiveness, PPR has been widely applied in graph learning across various fields, such as social networks, recommendation systems, and biological data analysis.

As illustrated in Fig. 4, this algorithm starts with a predefined preference node (or target node), which is emphasized according to the user's interests. Subsequently, a random walk is conducted on the graph to facilitate graph traversal. At each step of the walk, there is a fixed probability α that the walker will jump back to the target node from the current node instead of moving to an adjacent node chosen at random. This process of jumping, commonly referred to as "teleportation", biases the walk towards subgraphs that are of particular importance to the target node, thus personalizing the results according to user preferences. The walk continues until it reaches a steady state, at which point the likelihood of being on each node stabilizes into a steady-state distribution. These stabilized

(a) Distribution of PPR scores.

(b) Distribution of structural similarity.

10

3

Count

10²

Count

10

4

10'

10°

0

10

-6

10

5

10

10

10

2

10

2

4

8

10

PPR Score PPR()

Structural Similarity sim()

10

PPR Score PPR()

10

10

-3

10

4

10

-5

10

6

1

2

3

4

5

6

7

8

9

10

11

Structural Similarity sim()

Count Per Cell Type

10

10

10°

0

2

3

4

5

6

7

8

9

10

11

Structural Similarity sim()

probabilities, reflecting both the graph's structure and the user's preferences, determine the PPR scores. These scores effectively evaluate each node's structural similarities and rank them according to their relevance and importance from a personalized perspective.

PPR transformation. In scCello, the PPR algorithm is applied to the cell ontology graph to assess the structural similarities among cell types, or to measure their importance relative to a specified target cell type. We implemented PPR using the "pagerank" function in NetworkX [23] with "personalization" as arguments.

However, modification is needed to integrate PPR into TFM pre-training. The PPR scores are in real-number format and susceptible to numerical noise. Also, as shown in Fig. 5a, these scores typically exhibit a skewed distribution, concentrated around lower magnitudes. Consequently, setting precise thresholds to differentiate between node similarity and dissimilarity is challenging. Moreover, the vast amount small PPR values may be indistinguishable from noise.

To mitigate the effects of numerical noise and skewed magnitudes for the PPR scores, we employ truncation, logarithmic scaling, and discretization as outlined in Eqn. 3. Note that Eqn. 3 defines a monotonic, non-decreasing function that preserves the relative order between nodes. Its minimum value is set to 1 for the least similar cell types.

This equation transforms the raw PPR score, PPR( · ) , into the final structural similarity, sim( · ) . This transformation ensures that sim( · ) accurately reflects pronounced similarities as defined by the cell ontology and avoids emphasizing minor dissimilarities that could mislead during TFM pre-training.

Analyses. In Fig. 5, we present a comparison of the distributions for the PPR score, PPR( · ) , and the transformed structural similarity, sim( · ) . After transformation, the distribution of sim( · ) is less skewed and exhibits clear discretization. This facilitates the setting of definitive thresholds for distinguishing between similarity and dissimilarity among cell types, thereby enabling the effective incorporation of the cell ontology graph in scCello's pre-training.

In addition, we provide detailed insights into the scale of structural similarity, the distribution of these similarities for each cell type, and examples of cell types associated with various levels of structural similarity:

Fig. 6 illustrates the correspondence between the structural similarity after PPR transformation and the original PPR scores, showcasing a log-linear relationship as expected. This helps clarify the scaling of structural similarity, which is discretized into integer levels ranging from 1 to 11.

Fig. 7 demonstrates how frequently each target cell type is associated with other cell types at specific levels of structural similarity. Consequently, during scCello's pre-training, a substantial number of negative samples are expected to be utilized in the inter-cellular relational alignment objective, as outlined in Sec. 2.4.

Tab. 7 displays examples of highly similar and dissimilar cell types categorized into various levels of structural similarity, specifically targeting "T cell" and "neuron" types.

B Data Preprocessing Details

Download and Preprocessing. We downloaded from CellxGene [1] census version 2023-7-25. We focused on 291 datasets for human scRNA-seq. We preprocessed the dataset by the following steps:

Remove non-primary cells. Some data on CellxGene was duplicated due to multiple submissions of the same dataset from different research groups, therefore cells marked as "non-primary" were filtered out to prevent label leakage between pre-training and downstream.

Filter out cells not produced by 10x-based [69] sequencing protocols. There are numerous sequencing protocols in CellxGene database besides 10x-based sequencing [69], such as Drop-seq [58] and MARS-seq [35]. Only sequencing data from 10x-based sequencing protocols was kept to avoid large variation of data signals [46].

Exclude cancer cells. Cancer cells were highly dissimilar to normal cells and even occupied a large amount in the CellxGene database (nearly 12%). These cells could bring unexpected signals and skew the data, therefore we excluded these cancer cells.

To build downstream datasets for out-of-distribution (OOD) generalization evaluation, we first held out two category sets for each of the three settings: unseen cell types, unseen tissues and unseen

donors. Each category set were randomly selected with selection ratios 15%, 15% and 10% for the three OOD settings respectively. During the selection, we prohibited any category associated with more than 0.1% of the total pre-processed cells from being selected. This avoids losing too much data for pre-training. After the selection, cells associated with each held category set are collected, resulting in two OOD downstream datasets for each of the three OOD settings. These datasets are denoted as { D ct i } 2 i =1 for the OOD c ell t ype setting, { D ts i } 2 i =1 for the OOD t i s sue setting, and { D dn i } 2 i =1 for the OOD d o n or setting.

By excluding cells with at least one property belong to any of the six held category sets, the remaining data is further split into 99.9% as our pre-training data and 0.1% as the in-distribution (ID) downstream dataset D id . This way, our pre-training data and the ID dataset D id share similar data distributions.

Data Statistics. We summarize the data statistics for our curated pre-training dataset, one ID dataset and six OOD datasets in Tab. 8.

C Discussion on Ontology Graph Modeling

Graph Neural Networks (GNNs). One essential component of scCello is to model the cell ontology prior graph. GNNs are essential for managing graph-structured data by using a process called message passing [36]. This process lets nodes gather information from their neighbors, capturing their local connections and features, as seen in technologies like GCN [39]. Further advancements like GraphSAGE [24] and GAT [64] allow GNNs to handle larger areas of the graph and more complex relationships, useful in fields from social networks to drug discovery.

Recently, combining GNNs with transformer models, like GraphFormer [74], has proven effective. This integration allows the models to process both textual and graph data simultaneously, enhancing the understanding of the graph's structure without needing external measures.

Why we left GNNs for future work. While using GNNs in TFM modeling can internalize ontology graph knowledge in TFM modeling and no custom metric like that for PPR transformation is necessary, the cellular ontology graph presents specific challenges that hinder effective direct modeling by GNNs. The key issues is that the ontology graph is extremely sparse with about 2.7k nodes and 3.9k edges, and faces long-distance issues. For example, the average pairwise distance of 398 nodes (i.e., cell types) associated with our pre-training scRNA-seq data is 7.39, and the maximum distance is 18. Therefore, it requires multiple layers of graph propagations (around 7 layers), risking over-smoothing [57] of cell type representations.

D Implementation Details

scRNA-seq Data. scRNA-seq can enable the quantification of gene expression profiles of individual cells. Each cell's gene expression profile can be described by the set ˆ X = { ( e 1 , g 1 ) , ( e 2 , g 2 ) , . . . , ( e M , g M ) } , where e k denotes the expression count of gene g k , with e k ≥ 0 . A value of e k = 0 indicates that the gene g k is not expressed or not detected by the sequencing experiment. We use the same gene vocabulary set as [61], with the number of genes M =25 , 424 .

Gene Token Vocabulary. The gene vocabulary set contains both protein-coding genes and miRNA genes. M , the number of genes, is not the same as the number of all tokens in the model vocabulary. scCello has M gene tokens plus three more special tokens [MASK] for masking, [CLS] for the start of a sentence and [PAD] for padding.

Rank Value Encoding. Unlike natural languages, which inherently follow a sequential order, scRNA-seq data presents a unique challenge due to the lack of intrinsic order among gene tokens. Therefore, we employ Rank Value Encoding [61] approach to rank genes based on their normalized expression set { (˜ e i , g i ) } i M =1 . Specifically, gene expressions are first normalized by the total count within a cell [71] in a cell-wise manner, and then normalized through gene-specific weighting factors in a gene-wise manner. These factors are adopted from [61], which calculates the non-zero median value of expression of each detected gene across all cells. By design, these factors are assigned to emphasize lowly-expressed but essential genes, such as transcription factors [41], while deprioritizing ubiquitously expressed housekeeping genes [17].

After the normalization and ranking, it results in an ordered sequence of gene identities X = [ g π (1) , g π (2) , . . . , g π ( M ) ] with an index permutation π ( · ) , satisfying ˜ e π (1) ≥ ˜ e π (2) ≥ · · · ≥ ˜ e π ( M ) . To mitigate memory consumption, zero-expressed genes are removed and the gene sequence is

further truncated with a context length L =2 , 048 in practice. This rank-based approach offers better robustness against technical artifacts than directly using the original numerical expressions, which can vary significantly in magnitude across different experimental assays [46].

Cell and Cell Type Representations. Given a pre-training dataset with N cells X = { X 1 , X 2 , . . . , X N } , each cell X i can be mapped to a specific cell type ontology identifier c i ∈ V . For analyzing, scCello denotes cell X i 's representation as z i and cell type c i 's representation as h c i .

Masked Gene Prediction. Given a batch of cells { X i } B i =1 , scCello predicts a gene token g k based on the ordered gene sequence context X i, \ k =[ g 1 , . . . , g k -1 , [MASK] , g k +1 . . . , g M ] after replacing the token with a special [MASK]. This objective (term as L MGP ) aims to capture complex but important gene-gene interactions within one cell, like regulatory mechanisms between transcription factors and other genes:

where tokens are masked by a pre-defined distribution Ψ , same as that in BERT [15]. Specifically, 80% selected genes are replaced with [MASK], 10% selected genes are kept the same as its original, and 10% selected genes are replaced with random gene tokens.

Model Architecture. scCello utilizes a stack of self-attention transformer encoder layers [63], eacg composed of a self-attention and feedforward neural networks. The self-attention mechanism processes the input sequence, effectively capturing interactions between gene tokens.

Configuration Hyper-parameters. Besides scCello, we also summarize essential hyper-parameters for TFM baselines in Tab. 9 for comparison. It includes pre-training configurations like batch size, sequence length, and training time consumed. It also includes architecture configurations for the transformer model backbone, such as the number of transformer layers and the embedding size of transformer layers. Note that scTab uses TabNet [3] instead of transformer layers as model backbone, therefore its architecture configurations are not recorded in the table.

E Downstream Experiment Details

E.1 Evaluation Metrics

All metrics used in downstream tasks are summarized in Tab. 10 and introduced below.

Normalized Mutual Info Score (NMI). The NMI is a metric that quantifies the similarity between two differen clustering assignments or labelings of the same set of samples. We use NMI to compare the cell-type labels, with the cluster indices obtained from applying the Louvain clustering algorithm [12] on the target dataset.

We denote the two label assignments of the same N cell samples as C and K , representig the cell-type labels and the Louvain cluster indices, respectively. The entropy of a label assignment, say C , is a

measure of the uncertainty associated with that assignment set. It's calculated as:

where | C | is the number of unique cell types and P ( i ) = | C i | N is the probability that a randomly selected sample belongs to the class C i . The entropy H ( K ) for the cluster indices K is computed similarly, with Q ( j ) = | K j | N being the probability of a sample belonging to the cluster K j :

The mutual information (MI) between C and K qunatifies the amount of information shared between the two label assignments. It is calculated by:

where R ( i, j ) = | C i ∩ K j | N is the probability that a randomly selected sample belongs to both the class C i and the cluster K j .

The normalized mutual information (NMI) is defined as:

NMI is a normalized version of MI, scaled by the mean of the entropy terms for cell-type labels and cluster indices. This normalization ensures that NMI values range from 0 to 1, where 0 indicates no correlation between the two label assignments, and 1 represents a perfect match.

To obtain the best match between the clusters and the cell-type labels, we performed optimized Louvain clustering over a range of resolutions from 0.1 to 2, in steps of 0.1. The clustering output with the highest NMI score, when compared to the cell-type label set, was selected as the optimal clustering result. The implementation of NMI used in this study was from the scib python library [47].

Adjusted Rand Index Score (ARI). The ARI is another metric used to evaluate the similarity between the clustering assignment and the cell type labels of the same set of samples, similar to the NMI metric. In this context, we similarly denote the cell-type labels as C and the Louvain [12] cluster indices computed on the target dataset as K .

The Rand Index (RI) is a measure of the overlap between the two clusterings, C and K . It considers both the correct clustering overlaps and the correct disagreements between the two clusterings [54]. Formally, if we define a as the number of pairs of elements that belong to the same set in both C and K , and b as the number of pairs of elements that are in different sets in C and in different sets in K , the unadjusted RI is given by:

where N is the total number of cell samples and C N 2 represents the total number of possible pairs in the dataset.

However, the unadjusted RI does not account for the possibility of random label assignments leading to correct overlaps by chance. To address this issue, the adjusted RI (ARI) is introduced, which corrects for randomly correct labels by discounting the expected RI of random labelings:

The ARI ranges from 0 to 1, where 0 corresponds to a random labeling, and 1 indicates a perfect match between the two clustering assignments.

Similar to NMI, we performed NMI-optimized Louvain clustering to obtain the best match between the clusters and the cell-type labels. Specifically, we executed Louvain clustering over a range of resolutions and selected the clustering output with the highest NMI score when compared to the cell type label set. The implementation of ARI used in this study was from the scib python library [47].

Average Silhouette Width Score (ASW). The silhouette width [56] is a metric that evaluates the quality of a clustering solution by quantifying the relationship between the within-clustering distances and the between-cluster distances for each data point. Like the NMI and the ARI, the silouette calculates the similarity between the clustering assignment and the cell type labels of the same set of samples.

For each cell sample, the silhouette width is computed based on two scores: (1) a : the mean distance between a sample and all other samples in the same cluster; and (2) b the mean distance between a sample and all samples in the nearest neighboring cluster. The silhouette score s i for each sample i is defined as

The silhouette score ranges from -1 to 1, with higher values indicating that the sample is well-matched to its own cluster and dissimilar to the nearest neighboring cluster.

To obtain an overall assessment of the clustering quality, the average silhouette width (ASW) is calculated by averaging the silhouette scores s i across all samples. This overall ASW, denoted as ASW o , ranges between -1 and 1, with the following interpretations:

ASW o close to 1: The clusters are dense and well-separated.

ASW o around 0: The clusters overlap, and the between-cluster and within-cluster variability are approximately equal.

ASW o near -1: Strong misclassification has occurred, where the within-cluster variability is greater than the between-cluster variability.

To ensure that the final ASW metric falls within the range of 0 to 1, a scaling operation is often applied:

This scaled ASW value, ranging from 0 to 1, provides a convenient measure for evaluating the quality of the clustering solution, with higher values indicating better separation and cohesion of the clusters.

AvgBio. This score combines the three clustering metrics: NMI, ARI and ASW.

Silhouette Variant Score (ASW b ). To evaluate the effectiveness of the batch integration task (Sec. 4.6), a variant of the average silhouette width score (ASW) is employed, referred to as the ASW b . Unlike ASW based on cell type labels, ASW b considers batch labels. This score is designed to assess the degree of batch mixing, where a score of 0 indicates well-mixed batches, and deviations from 0 suggest the presence of a batch effect.

We take the absolute value of the original silhouette width score ˜ s i for sample i based on batch labels:

To ensure higher scores indicate better batch mixing, these scores are scaled by subtracting them from 1. As we expect batches to integrate within cell identity clusters, we compute the ASW b,j score for each cell label j separately, using the following equation:

where C j = { i | c i = j } N i =1 is the set of cell indices whose cell type label is exactly j .

To obtain the final ASW b score, the label-specific ASW b,j scores are averaged across the set of unique cell type labels:

where V represents the set of unique cell type labels.

Graph Connectivity (GraphConn). The GraphConn metric is designed to assess whether the k -nearest neighbor ( k NN) graph representation of the integrated data directly connects all cells with the same cell type label. This metric operates on the k NN graph, denoted as G k NN , which is pre-processed by the Scanpy library using the "scanpy.pp.neighbors" function.

For each cell type label v ∈ V , where V represents the set of cell type labels (Sec. 2), a subset k NN graph G k NN ( V v ; E v ) is created. This subset graph contains only cells from the given label v .

Using these subset kNN graphs, the GraphConn score is computed as follows:

Here, | LCC( · ) | is the number of nodes in the largest connected component of the graph and |V v | is the number of nodes with cell type v .

The resultant GraphConn score has a range of (0; 1] , where a score of 1 indicates that all cells with the same cell type are connected in the integrated k NN graph. The lowest possible score indicates a graph where no cell is connected to any other cell.

It's important to note that the GraphConn score is computed directly on the kNN graph representation of the integrated data. As a result, this metric can be used to evaluate the quality of any integration output, regardless of the specific integration method used.

AvgBatch. This score combines two metrics: ASW b and GraphConn.

Overall. We follow scGPT [14]to calculate a weighted average score of both the batch removal score ASW b and the bio-conservation score AvgBio to balance biological relevance and batch consistency, following the equation:

Accuracy (Acc). In classification tasks like cell type classification (Sec. 4.2.2) and novel cell type classification (Sec. 4.3), we denote the predicted values of the i -th sample as ˆ y i and the corresponding true label as y i . Then the accuracy metric is defined as

where the ✶ [ · ] is the indicator function. Macro F1 Score (Macro F1). The F1 Score is essentially defined for binary classification tasks.

where TP is the number of true positives, FN the number of false negatives, and TP the number of false positives. The recall is intuitively the ability of the classifier to find all the positive samples; The precision is intuitively the ability of the classifier not to label as positive a sample that is negative. For multi-class classification, macro F1 is defined as the average F1 taken over all different classes.

ROC AUC Score (AUROC). The Area Under the Receiver Operating Characteristic (AUROC) curve is a metric commonly used to evaluate the performance of binary classification models. It provides a comprehensive measure of the trade-off between the true positive rate (sensitivity) and the false positive rate (1 - specificity) across different classification thresholds.

In a binary classification task, the model's output is typically a probability or score that represents the likelihood of a sample belonging to the positive class. By varying the classification threshold, different operating points on the ROC curve can be obtained, where each point represents a specific combination of true positive rate (TPR) and false positive rate (FPR).

The ROC curve is created by plotting the TPR (y-axis) against the FPR (x-axis) for different classification thresholds. The AUROC is then calculated as the area under this ROC curve, providing a single scalar value that summarizes the overall performance of the binary classifier. The AUROC ranges from 0 to 1, with the following interpretations: (1) AUROC=1 indicates perfect classification, where the classifier can perfectly distinguish between the positive and negative classes; (2) AUROC=0.5 indicates random guessing, indicating that the classifier performs no better than a random prediction.

The AUROC is a widely used metric because it provides a comprehensive evaluation of the classifier's performance across all possible classification thresholds. It is invariant to class imbalance and does not require choosing a specific threshold, making it a robust and threshold-agnostic measure.

Furthermore, the AUROC has a statistical interpretation as the probability that a randomly chosen positive instance will have a higher predicted probability than a randomly chosen negative instance, which provides a clear interpretation of the metric's value.

Pearson correlation coefficient score (PCC). The PCCis a widely used measure of the linear relationship between two variables. It quantifies the strength and direction of the linear association between the variables, ranging from -1 to 1. The formula for the PCC between two variables, A and B, is given by:

where A i and B i are the individual observations of variables A and B, respectively. A and B are the sample means of A and B, respectively. n is the number of observations.

The numerator represents the covariance between A and B, which measures how much A and B vary together from their respective means. The denominator normalizes the covariance by the product of the standard deviations of A and B, ensuring that the correlation coefficient falls within the range of -1 to 1. The interpretation of this PPC metric is as follows: (1) r AB =1 indicates perfect positive linear correlation (as A increases, B increases proportionally); (2) r AB = -1 indicates perfect negative linear correlation (as A increases, B decreases proportionally); (3) r AB =0 indicates no linear correlation between A and B; (4) 0 < | r AB | < 1 indicates that the strength of the linear correlation increases as the value approaches 1 (either positive or negative).

In the context of regression analysis, computing the PCC between each regressor (independent variable) and the target variable can provide insights into the linear relationships between the predictors and the response variable.

E.2 Cell Type Identification

E.2.1 Zero-shot Identification ( i.e. , Cell Type Clustering)

Method. We here discuss the experimental details for Sec. 4.2.1. Cell representations extracted from each baseline model are used to compute the k nearest neighbor ( k NN) graph using Scanpy's standard protocols [71]. These representations and the k NN graph are then processed with Louvain clustering algorithms at various resolutions, ranging from 0.1 to 2 in steps of 0.1. The optimized clustering result is determined by the highest gained NMI score achieved across all the resolutions.

For implementation, we accelerated Louvain clustering by adopting RAPIDS, a software library that enhances data science pipelines by entirely utilizing NVIDIA GPUs instead of traditional CPUs. Additionally, we conducted ten iterations of dataset down-sampling and reported the averaged NMI,

ARI, ASW, and AvgBio scores. This approach significantly reduced the time required to evaluate a dataset, such as D id , from days to just a few minutes.

Datasets. As introduced in Sec. 4.2.1, we evaluate one ID dataset ( D id ) and six OOD datasets ( D cond i with cond ∈ { ct, ts, dn } and i ∈ { 1 , 2 } ) to demonstrate our model's generalization capabilities. These evaluations address various scenarios involving unseen cells for comprehensive testing, including cells with distributions similar to our pre-training dataset, as well as those associated with unseen cell types, tissues, and donors.

Hyper-parameters. We used k = 15 neighbors to compute the k NN graph, with node distances calculated using the euclidean distance between cell representations. The Louvain clustering used seed 0 as the random state and treated the k NN graph as unweighted and directed.

Performance. In Sec. 4.2.1, Tab. 1 reports only the AvgBio metric for six OOD datasets due to space constraints. Full metrics, including NMI, ARI, and ASW, are detailed in: (1) Tab. 11 for the two OOD cell type datasets ( D ct 1 and D ct 2 ); (2) Tab. 12 for the two OOD tissue datasets ( D ts 1 and D ts 2 ); and (3) Tab. 13 for the two OOD donor datasets ( D dn 1 and D dn 2 ).

E.2.2 Identification with Fine-tuning ( i.e. , Cell Type Classification)

Method. In this setting, the TFMs are further fine-tuned by adding a simple linear layer atop their model backbones, which transforms the hidden representations into prediction logits. The dimensions of these logits correspond to the number of cell type classes predicted. Importantly, all model parameters, including those of the TFM backbone and the newly added linear layer, are trainable during fine-tuning. The model checkpoint that achieves the highest Macro F1 score on the validation data is then selected for final testing.

Datasets. We fine-tuned TFMs on a subset of our curated pre-training data, randomly selecting 90% for training and using the remaining 10% for validation. The final performance was tested on the ID dataset D id , which consists of cell samples never seen during scCello 's pre-training. We explored two subset sizes, 0.1% and 1% of the pre-training data, to simulate scenarios where 10 × more annotated data becomes available. This exploration is meaningful for real-world applications, where annotating data is both costly and time-consuming.

Hyper-parameters. For scCello, we set the following hyper-parameters for fine-tuning: a learning rate of 5 . 0 × 10 -5 , a linear learning rate scheduler with 500 warmup steps, a weight decay of 0.001, and a batch size of 24. The same fine-tuning configuration was applied to the three ablation TFMs pre-trained using scCello's codebase (MGP, Sup, and MGP+Sup). For other TFM baselines, we searched for the optimal learning rate to report the final performance.

Performance. In Sec. 4.2.2, we reported classification and clustering metrics for TFMs fine-tuned with the 0.1$ subset of the pre-training data. Here, we extend our reporting to TFMs fine-tuned with 1% of a pre-training subset that is 10 × larger. We compare performances at these two subset selection ratios in Tab. 14. We observe that,

Known cell

types

u

∈ 𝒱

kn

Unknown cell

types

𝑣 ∈ 𝒱

Query cell

q

…

…

similarity

cell type repr.

cell repr.

𝓏

q

u

2

similarity

…

u

4

v

PPR 𝑢

#

, 𝑣

u

2

𝒉

$

#

𝑠 𝑞, 𝑣

𝑠 𝑞, 𝑢

%

𝑠 𝑞, 𝑢

#

𝑠 𝑞, 𝑢

&

= 𝒛

'

(

𝒉

$

#

𝑠 𝑞, 𝑣

Step 1: Known and

unknown cell types

Step 2: Similarity of query

to known cell types

Step 3: Similarity of query

to unknown cell types

Step 4: Decide

query cell type

𝒔 𝑞, 𝒖

= 𝑠 𝑞, 𝑢

&

, 𝑠

𝑞, 𝑢

#

, … , 𝑠

𝑞, 𝑢

𝒱

$%

𝒔 𝑣, 𝒖

= PPR 𝑢

&

, 𝑣

, … , PPR 𝑢

𝒱

$%

, 𝑣

𝑠 𝑞, 𝑣

= SpearmanR(𝒔 𝑞, 𝒖 , 𝒔(𝑣, 𝒖))

𝑣

∗

+∈𝒱

!"#"

𝑠(𝑞, 𝑣)

PPR 𝑢

-

, 𝑣

PPR 𝑢

&

, 𝑣

PPR 𝑢

%

, 𝑣

…

u

1

u

3

u

3

u

1

As the size of fine-tuning data increases, all TFMs except scTab show benefits and scCello achieves 48.9% improvement in Macro F1 when the data size gets 10 × larger. scTab's underperformance may be related to its model capacity, as it employs a TabNet architecture [3]-unlike others that use the powerful standard Transformers [63].

Across both the classification and clustering metrics, scCello's prevails other TFM baselines by a large margin. Remarkably, even when fine-tuned with a smaller 0.1% pre-training subset, scCello surpasses TFMs fine-tuned with a much larger 1% subset, achieving a 3.9% improvement over the best baseline. This underscores scCello 's superiority, attributed to its cell ontology-guided pre-training.

Interestingly, clustering performance does not necessarily correlate directly with classification performance. For instance, while MGP+Sup outperforms Sup in classification metrics, it does not do so in clustering metrics. This observation underscores the importance to evaluate both the clustering and classification performances for cell type identification with model fine-tuning, which can make the evaluation setting more comprehensive and rigorous.

E.3 Novel Cell Type Classification

Method. In this task, we define "known cell types" V kn ⊆ V as the 398 cell types from our labeled pre-training dataset (see dataset statistics Tab. 8). "Novel cell types", or "unknown cell types" V unkn ⊆ V , are those present only in the target downstream dataset and not observed during TFM pre-training ( V unkn = V \ V kn ).

Given a new query cell q , we aim to classify it to one of the unknown cell types V unkn . To solve this problem, we choose to first calculate representations for both the query cell sample and the unknown

cell types. And then we measure the similarity between the two representations to determine the prediction results v q ∈ V unkn .

Since unknown cell types are absent from the pre-training dataset, their representations cannot be directly obtained from any TFM baselines or our model, despite its ability to learn representations for known cell types. To address this problem, we leverage the known cell types V kn as a bridge to represent the query cells through the similarity between the cell and cell type representations produced by TFMs, and also represent the unknown cell types using the structural similarity relationships between the known and unknown ones derived from the cell ontology graph.

Specifically, our approach is illustrated in Fig. 8 and involves the following steps:

Representations for known cell types. Although scCello inherently learns cell type representations during pre-training, most existing TFMs do not output cell type representations directly. For benchmarking, we propose a protocol to calculate known cell type representations for general TFMs. Specifically, the representation for each known cell type is calculated by averaging cell representations derived from TFMs across cells belonging to this cell type. We used cell samples from a subset (10%) of our curated pre-training dataset, because the whole 22 million dataset is too large to fit.

We denote the known cell type representations as { h u } u ∈V kn , to differentiate with the notation of scCello's learned cell type representations { h u } u ∈V kn introduced in Sec. 2. For fair comparison, scCello also follows this protocol to generate known cell type representations, instead of using its learned ones. Nevertheless, we emphasize scCello's capability to conduct this task alone without further accessing reference databases like our pre-training dataset.

Similarity vector for a query cell to known cell types. We first derive the cell representations for the query cell q from TFMs. Then, we estimate the similarity between the query cell q and any known cell type u ∈ V kn using the cosine similarity between their representations s ( q, u ) = z T q h u . For all known cell types, this results in a similarity vector:

where we define the order of vector indices as u = [ u 1 , u 2 , . . . , u |V kn | ] satisfying u 1 < u 2 < · · · < u |V kn | .

Similarity vector for unknown cell types to known cell types. For each unknown cell type v ∈ V unkn , we estimate the similarity s ( v, u ) between the unknown v and the known cell types u . To achieve this, we leverage the cell ontology graph to calculate structural proximities as proxies. The proximities are measured using the raw PPR score PPR( u, v ) , u ∈ V kn , v ∈ V unkn , which is introduced in Sec. 2.4. Therefore, the similarity vector can be represented as:

Align the similarity vectors for the query cell and the unknown cell types. Intuitively, the similarity vector s ( q, u ) indicates a profiling for the query cell q , with known cell types u as a frame of reference; and the similarity vector s ( v, u ) conveys similar profiling for an unknown cell type v . Therefore, the more similar the two similarity vectors s ( q, u ) and s ( v, u ) is, the higher possibility for the query cell to be alike this unknown cell type. We derive it using Spearman Ratio [50] SpearmanR( · ) as the similarity measure:

Other formulas for the vector similarity function are available, like the commonly used cosine similarity (i.e., d ( q, v ) = d ( q, u ) T s ( q, u ) ). Our approach is not sensitive to the choice of the similarity metric. As shown in Fig. 10, using the dot product as the similarity score led to similar relative performance as in Fig. 2, where scCello generally performs better or on par with other TFMs. Therefore, we used Spearman Ratio throughout the experiments.

Select the final answer. The unknown cell type v ∗ with the largest distance is selected as the prediction for novel cell type classification:

(a) Acc for D ct 1 (same as Fig. 2, put for comparison).

(b) Macro F1 for

D

ct

1

.

(c) Acc for

D

ct

2

.

(d) Macro F1 for

D

ct

2

.

1.0

Geneformer

MGP

Geneformer

MGP

0.7

Sup

Sup

0.8

scTab

MGP+Sup

0.6

scTab

MGP+Sup

cc)

UCE

scCello

UCE

scCello

0.5

0.6

1

Macro

0.4

0.4

8

0.3

A

0.2

0.2

0.1

0.0

0.0

10%

25%

50%

75%

100%

10%

25%

50%

75%

100%

Portion of the Total #Cell Types

Portion of the Total #Cell Types

0.7

Geneformer

MGP

Geneformer

MGP

Sup

Sup

0.8

0.6

scTab

MGP+Sup

scTab

MGP+Sup

UCE

scCello

0.5

UCE

scCello

A

0.6

E

00

Macrol

0.4

0.4

0.3

0.2

A

0.2

0.1

0.0

0.0

10%

25%

50%

75%

100%

10%

25%

50%

75%

100%

Portion of the Total #Cell Types

Portion of the Total #Cell Types

In real-world applications, our approach is still applicable since almost all cell types are included in the cell ontology graph. But we won't be able to know whether the newly coming query cells are from unknown cell types V unkn or known cell types V kn . Therefore, we can expand the unknown cell type set V unkn to all the cell type defined in the ontology graph V , and conduct similar processes in our approach.

Datasets. We evaluate on OOD cell type datasets D ct 1 and D ct 2 . The cell types in D ct 1 and D ct 2 are already aligned to the cell ontology graph using the ontology identifiers provided by CellxGene database, and are a subset of all the unknown cell types V unkn . We recognize that the prediction task becomes more challenging as the number of novel cell types increases. Therefore, we constrain the complete unknown cell type set to the cell types occurred in the datasets we used.

To further reflect the challenge, we created five difficulty levels, where the number of cell types spanned from 10%, 25%, 50%, 75% to 100% of the total cell type count. For example, if we use 25% cell types in the OOD cell type dataset D ct 1 with a total 87 cell types, the unknown cell types include ( 87 × 25% ≈ 22 ) randomly selected cell types from the complete set { c i | X i ∈ D ct 1 } . To account for potential biases, we randomly sampled 20 distinct combinations of cell types for each difficulty level.

Hyper-parameters. The PPR( · ) score is calculated using the "nx.pagerank" function with alpha hyper-parameter set to 0.9.

Performance. The full metrics for both accuracy and macro f1 score on the two OOD cell type datasets D ct 1 and D ct 2 are reported in Fig. 9. Besides plots, the numerical results are also summarized in Tab. 16 and Tab. 17 for reference.

E.4 Marker Gene Prediction

Method. Wehere explain our approach for this task in details. Given a cell's gene expression profile, we enumerate each gene and attempt to knock it out, either by replacing it with a special [MASK] token or by reducing its expression to zero. The former method is used for Geneformer, MGP, Sup, MGP+Sup, and scCello, while the latter is applied to scGPT, scTab, and UCE. By comparing the cell representations of the mutated expression and those of the original expression, we assess the impact of each gene's knockout. A greater impact suggests a higher likelihood of the gene being a marker gene. This zero-shot approach requires no further fine-tuning and is particularly useful when additional computational resources or annotated datasets for fine-tuning are unavailable.

Notably, we acknowledge the shortage of our method: for house keeping genes ( i.e. , non-marker genes), knocking out these genes will also have large impact on the cell because the cell would die [17]. Therefore, a high impact from gene knockout does not necessarily indicate a marker gene, but rather an "important" gene. However, this issue is not critical empirically, as the number of well-documented housekeeping genes is about 400, which is small compared to the extensive gene token vocabulary of M = 25 , 424 .

Datasets. As introduced in Sec. 4.4, we used the datasets from GSE96583 [33] and GSE130148 [7]. One the one hand, the GSE96583 dataset D mk 1 inherently contains five cell subsets associated with 9 cell type classes. The five cell subsets are denoted as "GSE96583_1", "GSE96583_2", "GSE96583_3", "GSE96583_4", "GSE96583_5", respectively. On the other hand, the GSE130148 dataset D mk 2 contains 13 cell type classes. The size of these two datasets are summarized in Tab. 21, and their associated cell types are recorded in Tab. 20 for demonstration. Additionally, the ground truth cell-type-specific marker genes are originally sourced from two databases: CellMarker2 [29] and PanglaoDB [21].

(a) Acc for

D

ct

1

.

(b) Macro F1 for

D

ct

1

.

(c) Acc for

D

ct

.

(d) Macro F1 for

D

ct

.

1.0

Geneformer

MGP

0.9

Geneformer

MGP

Sup

0.8

Sup

scTab

MGP+Sup

scTab

MGP+Sup

0.8

C

UCE

scCello

0.7

UCE

scCello

0.6

0.6

Macro

0.5

0.4

0.4

0.3

0.2

0.2

0.1

10%

25%

50%

75%

100%

10%

25%

50%

75%

100%

Portion of the Total #Cell Types

Portion of the Total #Cell Types

1.0

Geneformer

MGP

0.9

Geneformer

MGP

0.9

Sup

Sup

scTab

MGP+Sup

0.8

scTab

MGP+Sup

C

0.8

UCE

scCello

0.7

UCE

scCello

C

0.7

0.6

iccuracy

0.6

Macro

0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

10%

25%

50%

75%

100%

10%

25%

50%

75%

100%

Portion of the Total #Cell Types

Portion of the Total #Cell Types

2

2

Performance. In Sec. 4.4, we only reported the average performance across the 5 subsets of GSE96583 ( D mk 1 ) and the individual performance of GSE130148 ( D mk 2 ) in Tab 3. Here, we provide complete results for all five subsets in Tab. 19.

E.5 Novel Marker Gene Prediction

Method. We adopted a methodology similar to our zero-shot marker gene prediction protocol (see App. E.4) for calculating differences in cell representations through in-silico gene perturbation:

For each cell, we calculated the change in cell representations after removing each gene in-silico and selected the top 10% genes with the largest changes, excluding the known marker genes.

For each cell type, we identified the 10 most frequent genes among the top 10% across all cells, to obtain 10 candidate novel marker genes.

To ensure specificity, we removed genes present in more than one cell type.

Datasets. We used the same two datasets (GSE96583 and GSE130148) in Sec. 4.4 for marker gene prediction, where marker gene labels were retrieved from CellMarker2 [30] and PanglaoDB [21].

Performance. As a case study, we followed the above steps to find novel marker genes for two cell types:

For cell type 'CD14+ Monocytes', two genes were found: FOS and LGALS2. FOS is typically expressed in response to stress signals, cytokines, and growth factors [2]; LGALS2

is involved in modulating immune responses and inflammatory processes in monocytes [37]. Both are plausible marker gene candidates.

For cell type 'Megakaryocytes', five genes were found: GNG11, TUBB1, H2AC6, CAVIN2, CLU. GNG11 is confirmed as marker genes in literature, TUBB1 is a likely marker, while H2AC6, CAVIN2, and CLU require further investigation.

E.6 Cancer Drug Response Prediction

Method. In this task, we first compute cell line level representations from scRNA-seq data and drug representations for associated drugs. Both these two representations are then input into the DeepCDR framework for training. Finally, we calculate the PCC between the predicted and actual IC50 values for each drug across all cell lines and report the average performance across all tested drugs.

Specifically, for TFMs, single-cell gene expression data are inputted into each model to generate cellspecific representations for each gene. These are then aggregated into cell line-level representations through max-pooling across all genes for each dimension. Conversely, the DeepCDR method uses raw gene expressions, aggregating them directly before max-pooling. Additionally, drugs are represented as graphs and encoded using graph neural networks to obtain drug representations.

Datasets. In our experiments, we utilized cell line and drug-paired data pre-processed by DeepCDR [43], including 223 drugs and 561 cell line bulk gene expression profiles for 697 genes from 31 different cancer types. Among the dataset, 89,585 cell line-drug samples were used for training and 4,729 for testing [26].

Hyper-parameters. We following scFoundation's implementation to set the parameters in the DeepCDR framework, like '-use_gexp' as True, and both '-use_mut' and '-use_methy' as False.

Performance. Results are already reported in Tab. 4 in Sec. 4.5.

E.7 Batch Integration

Method. This batch integration task aims to seamlessly integrate scRNA-seq data from different batches, which can be conducted using the same protocol as cell type clustering. After clustering, model performance is evaluated. Besides using cell type labels and clustering indices from the optimized Louvain algorithm to calculate the preservation of biological signals (NMI, ARI, ASW and AvgBio), this task also use batch labels to measure the removal of batch effects (ASW b and AvgBatch). See App. E.1 for metric calculation details.

Datasets. As introduced in Sec. 4.6, all datasets used in the cell type clustering task (Sec. 4.2.1) are evaluated, including one ID dataset D id and six OOD datasets D cond i ( cond ∈ { ct, ts, dn } , i ∈ { 1 , 2 } ) .

Hyper-parameters. We use the same hyper-parameters as that in cell type clustering.

Performance. In Sec. 4.6, the Overall score, a weighted average of AvgBio and AvgBatch, is already reported in Fig. 3. Complete results for all metrics are included in Tab. 23 for the ID dataset D id , Tab. 24 for the OOD cell type datasets D ct 1 and D ct 2 , Tab. 25 for the OOD tissue datasets D ts 1 and D ts 2 , and Tab. 26 for the OOD donor datasets D dn 1 and D dn 2 .

E.8 Visualization for Learned Cell Representations

We calculate known cell type representation as introduced in Sec. E.3, by averaging cell representations for each type on 10% of the pre-training data. Then we apply tSNE to project the known cell type representations to 2D space and visualize in Fig. 11. Highly correlated cell types are clustered together as expected, and dissimilar cell types are distant.

fibroblast

epithelial cell

animal cell

CD4-positive,alpha-betaTcell

myeloid cell

myeloidleukocyte

connectivetissuecell

somatic cell

progenitor cell

We also calculate the Spearman R correlation of the pairwise similarity of known cell type representations and the ontology structure (1 for an edge between two cell types and 0 for no edge between them) in Tab. 22. As expected, scCello learned a biologically informative representation space that is much more correlated to the true ontology structure than other methods. This implies scCello's potential generalization ability to other cell-type-related downstream tasks.

We also compare the cell representation distribution for scCello with its ablated version excluding the relational alignment objective (i.e., Eqn. 4), to analyze how the ontology relational alignment loss benefits clustering performance. Following the same visualization method for Fig. 11, visualization results for the ablated model is shown in Fig. 12. Adding the relational alignment loss makes similar cell types clustered more closely and pushes dissimilar types farther apart. Therefore, relational alignment enables scCello to align better with biological intuitions, and produce more effective cell representations as evaluated by broad downstreams.

fibroblast

epithelial cell

animal cell

CD4-positive,alpha-betaTcell

myeloid cell

myeloidleukocyte

connectivetissuecell

somatic cell

progenitor cell

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In this work, we propose scCello, a novel TFM designed to leverage cell ontology priors for enhancing cell representation and understanding. This main claim is accurately and clearly stated and emphasized in the abstract and introduction. It reflects our contribution and scopes.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We state several limitations of our work in Sec. 5, such as the lack of continue learning capability for our proposed model scCello, the relative small model scale for scCello, and our downstream approach for the zero-shot marker gene experiment.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: We propose a novel cell-ontology guided transcriptomic foundation model scCello in this work. It's a biological insight driven method and focuses on strong downstream applications. We do not establish theoretical results in this paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide all the details needed to reproduce our work, such as all experimental results for every metric and every dataset, pre-training setups in Sec. 4.1 and App. D, and downstream task settings in App. E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Our code and datasets will be released upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the training and test details are justified in App. D for pre-training, and in App. E for downstreams.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In this work, we provide error bars whenever random sampling on key experimental factors is involved. For example, in the novel cell type prediction task, we have random sampling procedures for cell type combinations and we report box plots with error bars to demonstrate the statistical significance of this experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: As indicated in Tab. 9, the pre-training our proposed model requires training for 2 days on 4 × A100 NVIDIA GPUs, each with 40G GPU memory.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the NeurIPS Code of Ethics throughout the entire project.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the social impact in Sec. 5 for both positive and negative influences. Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: All the datasets we utilized for both pre-training and downstream tasks are publicly available, and we did not implement explicit safeguards for these datasets. While our proposed model, scCello, aims to advance our understanding of cell representation learning for scientific discovery purposes and carries relatively little inherent risk for misuse, we acknowledge that as an open-source model, we cannot guarantee zero potential for misuse if the methods were to fall into malicious hands. Despite our intentions for beneficial applications, the open availability of scCello introduces a degree of uncertainty regarding potential mishandling or nefarious exploitation that we cannot definitively preclude.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All baseline methods and datasets are properly credited. Their license and terms of use are properly respected.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [No]

Justification: Code and datasets will be released upon acceptance. Clear documentations will be provided along.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We did not perform crowdsourcing experiments and research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces scCello, a Transcriptome Foundation Model (TFM) which incorporates prior information from cell ontology graphs denoting the relationships between different cell types in order to guide cell representations. The pretraining objective in the scCello framework encapsulates masked gene expression prediction, a supervised contrastive objective for aligning cell representations to cell ontology terms, and relational alignment of similar cell types, which helps guide representation learning during pretraining to reflect  taxonomic relationships between cells. The foundation model, scCello, is pretrained on 22 million cells from the CellxGene public data repository, and demonstrates improved performance over other single-cell foundation models in downstream tasks, including cell type clustering, cell type identification, marker gene prediction, and batch integration.\n\nSTRENGTHS:\nThis work presents a novel foundation model for single-cell transcriptomics, scCello, that incorporates prior information about cell ontology in order to guide representation learning over a large pretraining corpus. The methodology and motivation for the paper is clear, and the work represents a significant step towards training foundational models that incorporate prior information, which subsequent works may build and improve upon.\n\nFor pretraining, the authors introduce a multi-component loss function, encapsulating: (i) masked gene expression prediction, (ii) a supervised contrastive objective to align representations to of cells with similar cell ontology terms, and (iii) a relational alignment objective which enforces cell representations to follow the structural similarity of their cell types in the cell ontology graph. For quantifying cell type similarity, the Personalized PageRank (PPR) algorithm is used to derive structural similarity of pairwise cell types in the cell ontology graph, which is then used to guide representation learning.\n\nThe single-cell RNA seuqencing data preprocessing methodology seems sound, and follows common techniques for data preprocessing used by other foundation models. The authors also include a hyperparameter and compute resource comparison of scCello against other established single-cell foundational models - Geneformer, scGPT, scTAB, UCE.\n\nOverall, the paper is well-written, motivated well, and clearly explained. Adequate details are provided for the loss objectives and methodology, and additional dataset acquisition and preprocessing details are provided in the appendix.\n\nWEAKNESSES:\nThe authors mention that cell type ontology identifiers were obtained for the 22 million cell pretraining dataset from the CellxGene database, to enable mapping between individual cells and cell ontology. While this allows for additional priors from cell ontology graphs to be used during pretraining of scCello, it also necessitates labeled data during pretraining, which may limit the scalability of scCello in terms of available pretraining data compared to other methods which do not require annotated pretraining single-cell data.\n\nQUESTIONS:\nFor the overall pretraining objective in equation 5, more details could be added about how the four loss terms are weighted relative to one another. Are their coefficients to weigh the different loss terms? If not, do the magnitudes of the different loss terms differ significantly?\n\nThe authors quantify ontology relationships in the cell ontology graph by using structural similarity, and use the Personalized PageRank (PPR) algorithm to estimate pairwise node similarities for cell types. Examples of cell types associated with varying levels of structural similarity are also presented in Table 7. It would be helpful if the authors provided a motivation for going with a structural similarity measure such as PPR score, rather than a simpler metric such as the distance of two nodes in the cell ontology graph.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative impacts of their work, and agree to do a code implementation release upon acceptance of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduced a transcriptome foundation model (TFM)  named scCello to resolve the current problem of most of the TFMs they treat cells as independent samples and ignore the taxonomic relationships between cell types. By integrating cell ontology information as well as incorporating three key objectives in the pretraining framework: masked gene prediction, cell type coherence loss, and ontology alignment loss, the model can learn gene coexpression patterns, cell type-specific representations, and structural relationships between cell types. The authors performed a series of experiments to show that the model can predict the unseen cell types, integrate datasets across diverse batches, cluster cell types, predict the cancer response to the drug, and predict the marker genes.\n\nSTRENGTHS:\n1. The manuscript is well-written and well-structured, and it is easy to read. \n2. The authors effectively incorporate cell type ontology information at the pertaining stage, demonstrating in the comprehensive experiment results that this ontology information enhances the model's ability to learn cell type taxonomic relationships, thereby producing more biologically meaningful cell representations.\n\nWEAKNESSES:\n1. In the zero-shot cell type clustering experiment, it is not clear that the authors run the Louvain algorithm with default settings or the optimal setting on Seurat, Harmony, and scVI.\n2. In the marker gene prediction part, what is the difference between predicted marker genes based on scCello and the traditional marker genes based on the differential expression level? \n3. There is a paper (A Deep Dive into Single-Cell RNA Sequencing Foundation Models) indicating that the L1 logistic regression model achieves decent performance in cell type annotation task. Could the authors add the L1 logistic regression model for comparison in Table 2?\n\nQUESTIONS:\n1. How did the method achieve a high score in the batch integration? In the loss function, there is no specific loss for batch correction. Could the authors give some explanations for this question?\n2. Can the model predict the novel marker genes?\n\nLIMITATIONS:\nThe authors have addressed all limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents scCello, a single-cell, Cell-ontology guided Transcriptome Foundation Model (TFM) that leverages cell-type relationships from cell ontology graphs to enhance cell representation learning. scCello incorporates three levels of objectives during pre-training: masked gene prediction, cell-type coherence, and ontology alignment. This approach improves the model's generalization and transferability capabilities, leading to superior performance in tasks such as cell type identification, novel cell type classification, marker gene prediction, and cancer drug response prediction. scCello is also robust to batch effects and demonstrates high parameter efficiency compared to existing TFMs.\n\nSTRENGTHS:\n1.Integrating cell ontology graphs into TFM pre-training: This is the most innovative aspect of the paper. scCello improves model understanding of biological relationships between cells by incorporating cell-type relationships from cell ontology graphs into TFM pre-training, enhancing the model's generalization capability and transferability.\n2.Introducing a multi-level objective function: scCello employs a multi-level objective function that encompasses gene-level masked gene prediction, intra-cellular cell-type coherence, and inter-cellular ontology alignment. This multi-level approach enables the model to learn complex relationships between genes, cells, and cell types, leading to more precise and robust cell representations.\n3.Validating the model's effectiveness and advantages: The paper presents a comprehensive set of experiments demonstrating the effectiveness and superiority of the scCello model in tasks such as cell type identification, novel cell type classification, marker gene prediction, and cancer drug response prediction. Additionally, the paper highlights scCello's robustness in handling batch effects and its parameter efficiency, suggesting the model's strong potential for real-world applications.\n\nWEAKNESSES:\n1.Lack of Theoretical Justification: The paper heavily relies on empirical results to showcase scCello's effectiveness. However, it lacks a rigorous theoretical analysis of the proposed approach. A deeper theoretical understanding of the objective functions, their impact on learning, and how they contribute to generalization capability would significantly strengthen the paper.\n2.Limited Comparative Analysis: The paper primarily focuses on comparing scCello with existing TFMs but doesn't thoroughly analyze its performance against methods specifically designed for downstream tasks like cell type identification, marker gene prediction, and cancer drug response prediction. A more robust comparison with specialized methods would better illustrate scCello's strengths and highlight its relative advantages.\n3.The paper's current model size might be insufficient to fully capture the complexities of single-cell transcriptomic data. This limitation could hinder the model's ability to achieve optimal performance on tasks requiring deep understanding of complex biological processes, especially when dealing with large and diverse datasets.\n\nQUESTIONS:\n1.Can the authors elaborate on the theoretical foundations underlying the effectiveness of integrating cell ontology into TFM pre-training? Specifically, how do the proposed objective functions contribute to learning more informative and robust cell representations?\n2.Can the authors provide a more comprehensive comparison of scCello's performance against specialized methods designed for downstream tasks like cell type identification, marker gene prediction, and cancer drug response prediction?\n\nLIMITATIONS:\n1.The cell ontology is constantly evolving, and the paper acknowledges that scCello currently requires retraining the entire model for any updates. This limitation hinders the model's adaptability and its ability to keep up with the latest knowledge in cell biology.\n2.The paper states that they aim to scale up the model size in future work. This indicates that the current model size might not be sufficient to fully capture the complexities of single-cell transcriptomic data, potentially limiting its performance on certain tasks or datasets.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new transcriptome foundation model (scCello) to generate cellular representations from single-cell RNA-seq data. The key contribution is the integration of known cell type labels (previously annotated by CellxGene submitters) within two novel objectives. First, the authors introduce a cell-type coherence loss to minimize the distance between a cell's representation and the learned representation of its associated cell type. Second, the authors introduce an ontology relational alignment loss to ensure that the similarity between the representations of two cells matches the similarity of their corresponding cell types in an ontology graph. The authors benchmark their model against previously pre-trained transcriptome foundation models and traditional (non-foundation) models. They evaluate performance on tasks including zero-shot cell clustering, fine-tuned cell type classification, and marker gene discovery and find that scCello is state of the art on almost all tasks.\n\nSTRENGTHS:\n- *Conceptually simple but effective*: Utilizing cell type annotations, which are almost always available in published scRNA-seq datasets, to enhance the model's representations is smart and relatively straightforward. Other methods have proposed incorporating cell type labels before, but a cell type classification loss that treats each cell type as independent has shortcomings. The ontology relational alignment loss presented here which uses pre-existing ontology graphs to determine the similarity between cell types seems intuitively effective and empirically turns out to be too. \n\n- *Meaningful task selection*: The authors do a good job benchmarking their model on meaningful tasks. For example, they evaluate clustering performance not just on heldout IID data, but also in unobserved cell types, tissues, and donors. The model's improved performance on novel cell type classification and marker gene prediction suggests that it has captured biologically meaningful information. \n\n- *Appropriate baselines*: The authors do not just compare to existing transcriptome foundation models, which have been heavily criticized for not being competitive with traditional methods. \n\n- *Algorithm for novel cell type classification*: Their method of comparing the similarity vector between a cell and prototype representations of each cell type to the similarity vector between a cell type and all other cell types, as derived from the ontology graph, is clever.\n\nWEAKNESSES:\n- *Lacking analysis on ontology relational alignment loss*: While it seems intuitive that the ontology relational alignment loss is beneficial, the significant performance improvements are unexpected and probably warrant further analysis. For example, understanding which cell types show improved clustering performance could help provide some intuition. \n\n- *Missing some baselines*: Since the model assumes that cell type annotations are available, it should be compared against traditional methods that utilize these annotations (e.g. scANVI).\n\n- *Limited to 10x data*: The batch correction results, as is, are impressive, but it would be interesting to see if this method could handle data from different sequencing technologies.\n\nQUESTIONS:\n1. The regularization loss (Eq. 2) was proposed to prevent class collapse. But it's unclear why this would help. In fact, it seems to be encouraging all samples of cell type c_{i} to have the same embedding: Linear(h_{c_{i}}).\n2. How did you train scVI for Table 1? Did you train it on your pre-training dataset, in-distribution dataset, and out-of-distribution datasets jointly? If not, for a fair comparison, it seems like you should. \n3. To avoid picking up housekeeping genes in the marker gene analysis, could you see if knocking out housekeeping genes leads to a representation change that differs from that induced by knocking out marker genes. I imagine knocking out housekeeping genes might shift the cells to a previously uncharted part of the latent space (dead cells), whereas knocking out marker genes might shift the cells to a different cell type.\n\nLIMITATIONS:\nThe authors are forthcoming about the limitations of their model.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces scCello, a new transcriptome foundation model (TFM), which learns cell representations over RNA gene expressions. Apart from using Masked language modeling (MLM), masking random gene expressions in cells, the work leverages structural knowledge from ontologies to improve the learned representations. It enforces cell type coherence, where cells of the same type should be close in representation space. Similarly, the representation of structurally close cells, measured over a PageRank measure, should be close. These structurally inferred objectives are encoded as separate contrastive losses. The pretaining objective tries to minimise the sum over the MLM loss and the two contrastive losses together with a regularisation term, which tackles class collapse. scCello is backed by a transformer-based encoder-only model with roughly 10.5 min parameters, which is rather small compared to competitors. The evaluation covers multiple downstream tasks and an ablation study over the optimisation objectives. The results show that scCello is either competitive with state of the art performance or improves it and that the aggregation of the losses is beneficial.\n\nSTRENGTHS:\n- Data: It's great that the work leverages diverse data sources (structural and token-based) and tries to incorporate the strengths of both of them by using custom losses. \n- Experiments: The evaluation is extensive comparing the performance of the proposed model with multiple competitors and across multiple tasks. I also appreciated the ablation study and the analysis of overall performance wrt. #parameters. \n- Model: With a reasonable size the produced model can surpass many larger-sized competitors.\n\nWEAKNESSES:\n**Incorporating structural knowledge**\n\nGNN have been used for incorporating structural knowledge into pretraining of transformer-based models. For instance, take a look at GraphFormers (https://arxiv.org/abs/2105.02605). Such methods can be more elegant than the proposed PPR metric and the contrastive loss, since everything can be captured inside the model and no custom metric is necessary. While the author show that their proposed approach is beneficial for overall performance, such methods should be discussed or compared to, since they overlap in their goal of fusing structural and textual data with a major contribution of the paper.\n\n**Limited Novelty**\nThe paper does not advance any method or approach. It is an application of existing methods for creating a TFM model.\n\nQUESTIONS:\nWhile you analysed the impact of the single training objectives, you didn't present numbers on different model sizes for scCello. Do you have any evidence how larger or smaller model impact downstream performances?\n\nIf ontologies and knowledge base grow, how does the training approach scale with them? Since you use contrastive loss, you could run into bottlenecks, if the number of negatives grows?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}