Cell ontology guided transcriptome foundation model
Authors: Xinyu Yuan
Zhihao Zhan
Zuobai Zhang
Manqi Zhou
Jianan Zhao
Boyu Han
Yue Li
Jian Tang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Cell ontology graph+transcriptome foundation model+large-scale pre-training+cell representation learning+single cell RNA sequencing data
Primary area: machine_learning_for_healthcare
TL;DR: A cell ontology guided transcriptome foundation model for enhanced cell representation learning, which is pre-trained on 20 million cells from CellxGene database leveraging their cell-type labels aligned with cell ontology graph

Abstract: Transcriptome foundation models (TFMs) hold great promises of deciphering the transcriptomic language that dictate diverse cell functions by self-supervised learning on large-scale single-cell gene expression data, and ultimately unraveling the complex mechanisms of human diseases. However, current TFMs treat cells as independent samples and ignore the taxonomic relationships between cell types, which are available in cell ontology graphs. We argue that effectively leveraging this ontology information during the TFM pre-training can improve learning biologically meaningful gene co-expression patterns while preserving TFM as a general purpose foundation model for downstream zero-shot and fine-tuning tasks. To this end, we present **s**ingle **c**ell, **Cell**-**o**ntology guided TFM (scCello). We introduce cell-type coherence loss and ontology alignment loss, which are minimized along with the masked gene expression prediction loss during the pre-training. The novel loss component guide scCello to learn the cell-type-specific representation and the structural relation between cell types from the cell ontology graph, respectively. We pre-trained scCello on 22 million cells from CellxGene database leveraging their cell-type labels mapped to the cell ontology graph from Open Biological and Biomedical Ontology Foundry. Our TFM demonstrates competitive generalization and transferability performance over the existing TFMs on biologically important tasks including identifying novel cell types of unseen cells, prediction of cell-type-specific marker genes, and cancer drug responses. Source code and model
weights are available at https://github.com/DeepGraphLearning/scCello.

Cell ontology guided transcriptome foundation model
Xinyu Yuan!:2 Zhihao Zh Zuobai Zhang" Manqi Zhou' Jianan Zhao!?, Boyu Han', Yue Lit'*, Jian Tang 1.566'
Mila Quéhec Al Institute, Uniiversity of Montréal McGill University, *Cornell University, SHEC Montréal, 'CIFAR AI Chair *Correspondence: liyue@ mila.quebec; tangjian @ mila.quebec
Abstract
Transcriptome foundation models (TFMs) hold great promises of deciphering the transcriptomic language that dictate diverse cell functions by self-supervieed learn- ing on large-scale single-cell gene expression data, and ultimately unraveling the complex mechanisms of human diseases. However, current TFMs treat cells as independent samples and ignore the taxonomic relationships between cell types, which are available in cell ontology graphs. We argue that effectively leveraging this ontology information during the TFM pre-training can improve learning biolog- ically meaningful gene co-expression patterns while preserving TFM as a general purpose foundation model for downstream zero-shot and fine-tuning tasks. To this end, we present single cell, Cell-ontology guided TFM (scCello). We introduce cell-type coherence loss and ontology aligmment loss, which minimized along with the masked gene expression prediction loss during the pre-training. The novel loss component guide scCello to learn the cell-type-seecific representation and the structural relation between cell types from the cell ontology graph, respectively. We pre-trained scCello 22 million cells from CellxGene database leveraging their cell-type labels mapped to the cell ontology graph from Open Biological and Biomedical Ontology Foundry. Our TFM demonstrates competitive generalization and transferability performance over the existing TFMs on biologically important tasks including identifying novel cell types of unseen cells, prediction of cell-type- specific marker genes, and cancer drug responses. Source code and model weights are available at
Introduction
Cells are basic units of all living organisms. Deciphering diverse cell functions through gene expression is a long-standing challenge in life science and yet the essential path towards precision and personalized medicine. In this context, single-cell RNA sequencing (scRNA-seq) has emerged as pivotal technique to measure the gene expression in individual cells. The vast amount of publicly available scRNA-seq data offers a rich transcriptomic data source (48) for learning cell representations
towards various research applications, such as cancer therapy 62 and drug discovery [5)
Recently, several Transcriptome Foundation Models (TFMs) were developed to improve cell represen- tation learning. They mainly utilize pre- pre-training methods analogous to natural language processing like masked token prediction, treating genes as "tokens" and cells as "sentences" [14] 61 70 55].
However, the existing TFMs treat cells as independent samples and ignore their cell-type lineages. On the other hand, prior knowledge of the taxonomic relationships of cell types has been made available through the cell ontology graph by Open Biological and Biomedical Ontology Foundry [4]. Effec- tively leveraging the ontology knowledge can improve the quality of the pre-training on large-scale scRNA-seq atlases, which are heterogeneous and encompass hundreds of cell types. This can be done by training the TFM to recognize the inherent ontology relationships among cell types, thereby
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

refining the cell representations. For instance, "mature a- T cell" should be closer to "mature T cells" compared to more general term "T cells" and farther from neurons and astrocytes from the brain (e.g., Tab. 7
To capture this intuition, propose scCello, single cell, Cell-ontology guided TFM. scCello learns cell representation by integrating cell type information and cellular ontology relationships into its pre-training framework. scCello's pre-training framework is structured with three levels of objectives: (1) gene level: a masked token prediction loss to learn gene co-expression patterns,
enriching the understanding of gene interactions (Sec. 2.2): : (2) intra-cellular level: an ontology- based cell-type coherence loss to encourage cell representations of the same cell type to aggregate, prompting consistency between cells and their types (Sec. (.3); and (3) inter-cellular level:
relational alignment loss to guide the cell representation learning by consulting the cell-type lineage from the cell ontology graph (Sec. 2.4).
We demonstrate the generalizability and transferability of scCello on 22 million cells from CellxGene. For model generalization, we observe that scCello excels cell type identification across all datasets in both zero-shot setting (i.e., directly using the pre-trained model) (Sec. 4.2.1) and fine-tuning setting (Sec. 4.2.2). p. In particular, scCello accurately classifies novel cell types by leveraging the ontology graph structure (Sec. 4.3). For transferability, scCello demonstrates competitive performances in pre- dicting cell-ty cell-tye--peeifee marker genes (Sec. 4.4) and cancer drug responses (Sec. 4.5) Additionally, scCello is robust against batch effects (Sec. 4.6). Finally, we validate our contribution via ablation
study (Sec. 4.7)
2
Method
Fig. 1
fillustrates overview of scCello. We present the details of individual components below.
2.1 Data Preprocessing
Cell ontology graph. Cell ontology is widely used metadata schema for standard cell type annotations [16]. We downloaded the ontology from Open Biological and Biomedical Ontology Foundry (https:/ http:::/oooofoouore/ wogg/ry.org//.............
structured as an unweighted directed acyclic graph g = (V,E), where each node e E V corresponds to a distinct cell type and each directed edge (u, & denotes hierarchical lineage relationship of the form "is subtype of" between cell types (Fig. [.. To accurately represent the inherently symmetric "being biologically similar" relationship between cell types, the directed graph was transformed into an undirected one for subsequent calculation of cellular ontology relationships in Sec. 2.4
scRNA-see data. The scRNA-seq data were downloaded from CellxGene. After the preprocessing (App. B). we obtained 22 million cells. Each single-cell transcriptome is represented by a sequence of tuples, each containing genes and their expression counts Each sequence was then ordered by the rank of the gene expression values [6]] akin the sequential ordering of natural languages. Given batch of B cells, each cell i E (1..... ... B} } was assigned a cell type ontology identifier c; E V from the CellxGene database, to enable mapping between cell and cell ontology.
2.2 Masked Gene Prediction
Same as BERT (5), scCello predicts a randomly masked gene token in each cell based on its surrounding context in the sequence. This objective LMGP aims to learn the dynamic gene co- expression network.
2.3 Intra-Cellular Ontology Coherence
straightforward approach to encourage learning the cell representations that are coherent to the cell type labels is to apply cross-entropy loss for supervised cell type classification. However, this approach is limited learning cell representation for the foundation model. Instead, we employed supervised contrastive loss our objective Linua. which directly optimizes the TFM rather than
ssRRNA-seq data from CellxGene database https://celllgeett htt.://lwc..eee.cccsscccco....o. cziscience.com/

---

(a) Cell Ontology Graph
(c) scCello Pre-training Framework
Ontology Relationship
A
B
C
Native cell
Relational Alignment
A
Neuron
Nucleate cell
ha
B
hB
hc
Regulatory T cell
(A, (A, sim(A, sim(,, C)
Linear
Guide
(b) scRNA-seq data
2
z Zz > z
Cell Reprs. Distance 2
Cells
Cell Type
z 2
N 2
o 23
22
2
Cell-Type Coherence
3
Genes
(d) Downstream Tasks
Cell Type
Batch
Cell Type Classification
Cancer Drug Prediction
Clustering
Integration
Cell Reprs.
Cell Reprs.
IC
Novel Cell Type Classification
Marker Gene Prediction
Similarity
Genes
Cell Reprs.
Distanee
Logits
Marker
Gene
"Cell Type E!
Knockout
Figure 1: (a) Cell ontology graph describes taxonomic relationships between cell types. (b) Each cell in scRNA-seq data represented by gene sequences, and associated with cell type ontology identifier. (c) The pre-training framework of scCello is structured with three levels of objectives: gene-level masked gene prediction, intra-cellular level cell type coherence and inter-cellular level ontology alignment. For example, as shown in panel b, cells 1, 2, and 3 are labelled with cell type A, and C. The intra-cellular cell type coherence loss encourages alignment embedding z with h with ha, and Z3 with he. The inter-cellular level ontology alignment loss encourages representational learning of cell similarities z, ny between cell i and j to be consistent to the similarity of their corresponding cell types sim(c,, cy) based on the ontology relationships. (d) Downstream tasks enabled by scCello and demonstrated in the study.
merely learning through the linear classifier:
exp x he, hT,/T)
Lintra
Σ log
(1)
exp(22Th (ooz2/ha/// /T)
Σ Σj=1 expl 2. h ht,/
where and he, denote the latent representation of cell i and cell type q, respectively.
This supervised contrastive loss pulls representations of the same class (positives) and repels rep- resentations of different classes (negatives). often leads to representations that are at least as
discriminative the cross-entrope loss [22). To reduce the degrees of freedom available for TFM optimization, we introduce regularization term LRegi
Lieg LB Linear(h..)) 2:113
(2)

---

where the linear layer is shared across all cells and cell types. Thereby, it constrains the cell type representation space be an affine transformation of the cell representation space.
2.4 Inter-Cellular Relational Alignment
To encourage TFMs to learn inter-cellular ontology relationships, scCello forces cell representations truthfully reflect the pairwise node structural similarity derived from the cell ontology graph, using relational alignment objective. This objective constitutes the most important part of scCello.
Ontology relationships. To effectively quantify ontology relationships between cell types from the ontology graph, scCello estimates pairwise node structural similarities as proxies using Personalized PageRank (PPR) [20]. PPR is a graph learning algorithm. The PPR score PPR(u, v) estimates the probability for random walk. It starts from given target node u. E V and terminates at another node E V. Importantly, this is a context-sensitice structural similarity measure that accounts both direct connections and broader subgraph patterns [67]. It also provides robustness against variations in global network structures, such as variable node degrees and clustering coefficients [1]] To improve robustness (as justified in App. [A), transform PPR(-) through non-linear function to derive the structural similarities sim(.) as ontology relationships tunable by hyper-paramett threshold s:
[log2( pP
+1)1, if PPR(u, v) 2
(um, u. v)
otherwise
Relational alignment. Cells with closely related cell types tend to be more similar than those with distinct cell types. This observation guides scCello align the distances between cell representations w.rt. a target cell, with their structural similarities sim(-) (as shown in Fig. D Specifically, given a batch of B cells, if we consider target cell i and another cell in the batch + the representation distance z!¹ z; should reflect their structural similarity sim (c,, oj). Accordingly, a negative sample set Di =
Cy) > sim(c, ck), 1 < k S B} be produced, where cell pair (i, k) are considered less similar to the cell pair (1, and should be contrasted against in the representation space using the objective Lintor:
B Linter Σ
exp(2 exp(2[²2//T) exp(<f'zz)f + EkEn, exp(2 2x/T)
Σ 1=11iei
log
Notably, ancestor cell types, which can reach the target cell type via the directed "is a subtype of" edge on the ontology graph, structurally distant from the target cell type. Despite being distant, they fall into the same, broader cell type category. Contrasting cells associated with these distant ancestor cell types with the target cell is counter-intuitive. Therefore, scCello explicitly excludes such cells from the negative sample set, avoiding inappropriately pushing away biologically similar cells. This enhances scCello's capability discern subtle similarities and differences within the cell types.
2.5 Overall Pre-training Objective
During pre-training, we seek to minimize the loss functions of all pre-training tasks simultancously:
8* - arg min LMGP + Linter + Lintra + LReg
(5)
where 9 denotes all learnable parameters in scCello, which adopts transformer stacks as model backbones. We state the detailed information of model architectures in App. D
3 Related Work
The rapid growth of scRNA-seq datasets has opened new avenues for constructing TFMs, enabling transfer learning across various biological downstream tasks. Initial efforts, such as scBERT [73], Exceiver [3] and Geneformer [61, borrows the concept of masked language modeling [15] from
natural language processing (NIP) domain for pre-training. by treating cells as sentences and genes as tokens. Concurrently, (GPT 55) and scGPT [4] explored generative modeling (53), and CellLM (76

---

adapted the idea of contrastive learning 42
Following the concept of "scaling" towards emergent ability 770 in NLP. scFoundation 25 proposes the largest foundation model at the time in terms of model size and pre-training data size; scHyena
49 scales modeling context window size to the full length of scRNA-seq data with Hyena operator [51 instead of conventionally used transformers.
scaab [88 is the first to explore large-scale supervised learning mechanism for scRNA-seq pre- training, and is capable of annotating unseen tissue cells for real-world applications. Moreover, SCimilarity (28) and UCE [55] focus developing unified latent space a large-scale reference atlas for querying new cells. Yet, these TFMs mainly treat cells as independent samples during training and ignore their biological ontology relationships. scCello bridges this gap by incorporating cell type relationships derived from the cell ontology graph into TFM tre-training. This strengthens TFMs' model generalization and transferability capability, shown in Sec.
4 Experiments
As an overview, the following experiments show that, (1) scCello can generalize to unseen cells, and to more difficult settings, such as cells of unseen cell types. tissues, and donors (Sec. 4.2. )
(2) seCello can benefit from fine-ture tuning on target datasets (Sec. [42.2]; (3) the structural similarity
embedded in scCello helps classify novel cell types in a zero-shot manner (Sec. 4.3); ; (4) scCello
effectively transfers to different downstream tasks (Sec. and Sec. 4.5): (5) scCello is robust to batch effects that arise from different experimental conditions (Sec. (Sec.4..6. (6) Each loss component in
Eqn. is heneficial to scCello (Sec. 4.7). For every table reported, we used bold to highlight the best performance and results within 0.005 difference from the best. We used underlining to denote the second-best performances. For all metrics, 1 indicates the higher the better.
4.1 Setups
Pre-training and downstream datasets. We collected a large pre- pre-training dataset consisting of 22 million cells along with downstream datasets. In particular, and six oft-o--distribution (OOD) datasets (App. Bj. The ID dataset is denoted as D For the OOD
generated one in-distribution (ID)
setting, introduced three scenarios: unseen cell types ((DF) 2 unseen cell tissues ({ ) and unseen donors ({ ((Diin ),=1). Each scenario has two datasets. Notably, the OOD donor setting presents more realistic challenges than ID and other OOD settings because of the potential batch effects in the test donors.
Pre-training configurations. An Adam optimizer 38 (learning rate: 0.001, weight decay: 0.001, warm-up steps: 3, 333) was used to train the scCello for 40, 000 steps on 4 NVIDIA A100 GPUs on Compute Canada. We used 192 for batch size. More details are introduced App. D
Baselines. Across all downstream tasks, scCello is benchmarked with leading open-source large- scale TFMs: Geneformer [61], scGPT [4, scTab [18. UCE (55), and three TFM ablations. We also implemented ablated versions of scCello that only differ in the pre- pre-training objectives from scCello: scCello using only the masked gene prediction loss (denoted as MGP), scCello using only the cell type supervised classification (denoted as Sup), and scCello using only the two losses (denoted as MGP+Sup). The three ablated TFMs provide a reference to isolate the effect of implementation details and training configurations. For each task, we also selected state-of-the-an non-TEM methods for fair comparison.
Downstream metrics. We evaluated the 3 tasks by the following metrics. (1) Clustering metrics include normalized mutual information (NMI), adjusted rand index (ARI), average silhouette width (ASW), and the average of the 3 scores (AvgBio) to assess both between-cluster separation and within-cluster closeness [1]. The batch integration task (Sec. [4.6) is evaluated by ASW,, b, graph connectivity (GraphConn) and their average (AvgBatch), along with an overall score (Overall 0.6x AvgBio 0.4 AvgBatch) to balance biological relevance and batch consistency following [14] (2) Classification metrics include accuracy (Acc), Macro Fl and area under the ROC curve (AU-
ROC) (50). (3) Regression task metries include Pearson correlation coefficient score (PCC) 50
Details for each metric were provided in App. E.I
5

---

Table 1: Zero cell type clustering on the curated ID and OOD datasets.
In-Distribution (ID)
Out-of-Distriution (ODD)
Method
D
D AvgBiott
D2 AugBio" Non-TFM Methods
D AugBiot
Dg Augliot
1
D AvgBiot
NMI
ARI1
ASW1
Augliiat
AvgBiot
OOD Avg.T
Raw
0.566
0.237
0.453
0,703
0.540
0.631
0458
0.460
Seurat
0.648
0.270
0.407
0.442
0,752
0.737
0,587
0.636
0.466
0.489
0.611
Harmony'
0.621
0.382
0,421
0.452
0.417
0.462
0.515
0456
0.474
scV1
0.660
0.297
0.464
0.474
0.760
0.725
0.577
0.634
0.478
0.502
0.613
Outology-Agnoste TFMs
Geneformer
0.616
0.261
0.418
0.432
0.689
0.668
0.539
0.597
0,468
0.482
0.574
scGPT
0.615
0.707
0.544
0.466
0.477
scTab
0.707
0.479
0.544
0.577
0.759
0.726
0.515
0.657
0OM
0OM
UCE
MGP
0.662
0.306
0.451
0.473
0,714
0.740
0,576
0.628
0.488
0,518
0.611
0.703
0.393
0.569
0.555
0.767
0.775
0.605
0.680
0.552
0.573
0.659
MGP+Sup
0.661
0.337
0.550
0.516
0.758
0.764
0.610
0.672
0.553
0.570
0.655
Gallology-Enhannee TFMs
scCello
0.785
0.558
0.667
0.670
0.769
0.786
0.612
0.705
0.608
0.643
0.687
Harmony could be over-corrected w.r.f. batch labels for datasets with many batches
4.2
Cell Type Identification
4.2.1
Zero-shot Cell Clustering Results
Setup. For the cell type clustering task, TFM baselines and four non-TFM methods were evaluated:
(I) raw data expressions of highly variable genes (abbr, Raw Data) [34]; (2) Seurat [27] (3) Harmony (40] (4) VC (4). Cell representations were extracted from the baselines and clustered by
Louvain algorithm 6 We evaluated the clustering performance of each method on both ID dataset
Díd and OOD datasets Drond (conde ts, dn3, (E,22)
ID and OOD generalization. We reported zero-shot cell type clustering performance in Tab. and included all the metrics for all datasets in App. |..2T due space constraint. For both the ID and OOD settings, scCello consistently outperforms all baselines, achieving 16. 1% improvement in AvgBio on the ID dataset and 12.1% improvement in average AvgBio across the six OOD datasets. Interestingly, while scCello outperforms non-TFM methods by large margin, Geneformers and ScGPT barely surpass these methods. The latter consistent with previous observations 75
In the OOD experiments, scCello confers strong generalization capability across unseen cell types tissue, and donors. In cell type clustering, scCello is the second best only trailing UCE by 0.03 and the best method for dataset and 2. The OOD tissue setting highlights scCello's ability to transfer its learned knowledge to different unseen tissues. Specifically, scCello achieve 0.6 and 0.7 while most methods conferred below 0.6 and 0.7 for the two datasets, respectively. For the unseen OOD donor scenario, most methods perform poorly with AvgBio ranging between 0.45 and 0.55. scCello led the chart achieving AvgBio above 0.6 in both datasets. Overall, scCello showcases strong model generalization capabilities across a range of biological conditions, which is attributable the integration of cell ontology priors during its TFM pre-training. Indeed, the ablated models namely MGP, Sup. and MGP+Sup conferred lower scores compared to the full model.
4.2.2 Fine-tuning Results
Setup. We benchmarked all TFM baselines except UCE for its lack of fine-tuning support. These TFMs were fine-tuned on subset of our pre-training data with supervised classification loss (details in App. E.2.2). We assessed both classification and clustering performance on the ID dataset Did We also compared with a non-TFM method, scANVI (72).
Improvement with fine-tuning. In Tab. 2 The fine-tuned scCello outperforms other TFMs and non-TFM methods on both classification and clustering metrics, achieving up to 25.9% improvement in Macro Fl over the best baseline. Moreover, scCello without fine-tuning still surpasses the performance of the other fine-tuned methods, further highlighting its superior transfcrability.

---

Table 2: Cell type identification using fine-tuned TFMs. Both the classification and clustering performances on the ID dataset Ddd are reported.
Table 3: Marker gene prediction a
binary classification task to identify cell- type-specifie marker genes.
Classification
Clustering
Dusk
D
Method
Method
Avg.
Acc 1
Macro Fl
AvgBio
AUROC
AUROCT
Non-TFM Methods
Non-TFM Methods
SCANVI
0,763
0.490
0,568
DET
0.721
0.683
0.702
Scratch
0.621
0.223
0,544
Ontology-Agnostic TFMs
Matology-Agnoste TFMs
Geneformer
0.452
0.470
0,461
Geneformer
0.747
0.440
0,439
scGPT
0.385
0.387
0.386
scGPT
0.712
0.344
0.477
scTab
0.672
0.727
0.700
scTab
0,778
0.373
0.606
UCE
0.500
0,500
0,500
MGP
0.722
0.287
MGP
0.579
0,629
0.604
Sup
0.812
0.363
0.659
Sup
0.689
0.693
0.696
MGP+Sup
0.820
0.406
0.607
MGP+Sup
0.730
0.730
0,730
Ontology-Enhen Enhanced TFMs
Ontology-Eniln -Enhunced TFMs
scCello
0.511
0.694
scCello
0.756
0.729
0.743
Table 4: Cancer drug response prediction: regression task to predict the ICso values of drugs.
Method
Non-TFM Methods
Ontology-Agnodie TFMs
Ontolog--
DeepCDR
scFondaaioon
Geneformer
scPPT
w<Tbb
UCE
MGP
Sup
MGP4Sup
scCello
1
0.854
0.882
0
0.919
0.913
0,922
0.872
0.915
0.916
0.917
4.3
Novel Cell Type Classification
Novel cell type classification aims to label cells of unseen cell types without further fine-tuning. This task is useful for annotating completely new scRNA-seq datasets but infeasible for most of the supervised methods that solely rely on the labels observed in the training data [8] [811]688 68). Leveraging the cell ontology graph that comprises the lineage relations among all of the known cell types, scCello makes this task feasible.
Setup. Our goal is to classify new query cells into "novel cell types" not seen during pre-training. To do this, we generate representations for both query cells and novel cell types, using similarity measures for classification. This process involves utilizing similarities between TFM-derived representations for the former and biological relationships from the cell ontology graph for the later. Details were described in App. B.3
We benchmarked all TFMs and evaluated them on OOD cell type datasets Dj' and D We increased the difficulty of this task by the number of novel cell types (#Cell Types) that exist among the query cells. Specifically, we simulated five difficulty levels, with the number of novel cell types ranging from 10% to 100% of the total cell types. To assess the variance of the performance, we randomly sampled cell type combinations times at each level.
OOD generalization. In Fig. 2 scCello led other TFMs by a large margin, achieving up to 76.8% Acc classify 9 novel cell types (i.e., 10% of the total heldout cell types) and 33.5% Acc to classify up to 87 novel cell types (i.e., 100% of the total heldout cell types) (Tab. [6] and Tab. [7]. These results show significant leap from the existing TFMs, which either do not work only work for
annotating a handful of novel types 668 45 66)
4.4 Marker Gene Prediction
Cell-type-specific genes, or marker genes, are highly expressed in a specific cell type but exhibit low expression in others. These genes play a crucial role in delineating cell functions in diverse tissue contexts. Identifying marker genes in less characterized cell types is an ongoing challenge [52]
Setup. We sought to assess whether the pre-trained TFMs can discriminate marker from non- non-marker genes for any cell type without any supervised fine-tuning. This zero-shot experiment evaluates whether the TFM able to learn biologically meaningful gene co-expression patterns without supervision. For each cell, we quantified the marker gene potential of each gene by the changes in - cell representations after in-silico knockout of the target gene (details App. App. Here assume that the larger the change the higher the marker gene potential. We discussed the
7

---

1.0
Geneformer
MGP
xGGPT
Sup
Scurat Harmozy
Genelormer
UCE
MGP Sup
sctaab
MGGP-SSe
Accuracy (Ace)
ID and OOD Datasets
Die
D1
D5
0.4
D
D
0.2
Df
0.0
Dg
10%
25%
50%
75%
100%
0.55
0.60
0.65
0.70
0.75
0.80
Portion of the Total #Cell Types
Overall Score
Figure 2: Novel cell type classification OOD cell type dataset D for increasing difficulties.
Figure 3: Batch integration the curated ID and OOD datasets.
caveat of this approach in Sec. As test data, we used GSE96583 32 (Diuk) and GSE130148 65 (Dyak). We obtained the marker gene labels from CellMarker2 [29] and PanglaoDB 21 We also compared with non-TFM method, Differential Expression Tests (DET) 66)
Zero-shot transferability. In Tab. 3 scCello outperforms other TFMs, improving upon the second- best method by 1.8% in average AUROC. The inclusion of cell label information during pre-tres pre-training boosts TFM performance, as evidenced by the strong results of sc'aab, Sup. MGP++up and scCello. This is due to the biological correlation between marker genes and cell types. Furthermore, employing cell ontology graphs further improves the prediction accuracy over MGP+Sup.
4.5 Cancer Drug Response Prediction
Developing effective drugs for cancer treatment is challenging due to individual variability in drug responses. Accurately predicting cancer drug responses (CDR) can greatly aid anti-cancer drug development and improve our understanding of cancer biology 4
Setup. Following the approach of scFoundation [26] cell representations were extracted from fixed TFMs and integrated into the DeepCDR [3] pipeline to estimate the half-maximal inhibitory concentration (/C'o) values of drugs (details in App. E.6). We benchmarked our method against DeepCDR, scFoundation, and other TFM baselines, using the same pre-processed data as DeepCDR.
Zero-shot transferability. In Tab, scCello is among the top 3 along with scGPT and UCE, achieving 7.4% improvement PCC over the base method DeepCDR, This highlights scCello's transferability in enhancing specialized task-oriented methods. In particular, it can be used as an powerful feature extractor for diverse downstream tasks.
4.6 Batch Integration
The scRNA-see atlases, assembled from datasets across various labs and conditions, are prone to unwanted technical variations known as batch effects 46). These effects can significantly affect the generalization ability of TFMs especially because they require pre-training massive amount of heterogeneous scRNA-seg data pooled from many studies. Here we sought evaluate scCello's robustness to batch effects without fine-tuning.
Setup. We adopted the same baselines in zero-shot cell type clustering (Sec. 4.2.1 1, and followed the evaluation protocol of scGPT [14. We evaluated on one ID dataset id and six OOD datasets Doond (cond E {ct, ts, dn }. i (1,2)) (see complete results of all metrics in App. E.7)
Robustness data noise. Fig. shows that scCello excels out of 7 datasets, and achieves comparable performance on another 3 datasets. The performance is attributable to the use of cell type information as the ablated baseline MGP conferred much lower batch integration score compared to Sup and scCello.
8

---

Table Pre-training loss ablation on the cell type clustering and novel cell type classification (abbr., "clf." ") tasks.
Table 6: Overall performance v.s.
the number of parameters.
Cell Type Clustering
Novel Cell Type CIl
Method
Perf. Rank
#Params (M)
Config
D5
D
Df
Geneformer
6.3
10.3
AvgBiot
AvgBiot
Acct
Macro F11
scGPT
6.2
51.3
scTab
4,2
9.7
Full Loss
0.786
0.643
0.335
0,150
UCE
4.8
674.7
w/o CarP
0.774 (11.5%)
0.640 (10,5%5
0.287 (134.5%)
0.131 (.12.79)
MGP
6.7
10.3
w/o Cintrs
wfo Cinter
0.730 (17.158)
0.778 (11.08)
0.620 (1a.6%)
0.626 (12.6%)
0.147 (188.155)
0.280 (110.4%)
0.052
0.118 (-21,2%5)
(106. 8%5)
MGP+Sup
Sup
3.3
3.2
10.4
10.9
w/ Citg
0.764 (12.8)
0.638 (10.8%)
0.296 (111 696)
0.134 (L10.795)
scCello
1.3
10.7
4.7 Ablation Study
Ablation of pre-training losses. Tab. 5 reports the cell type clustering (Sec. 4.2.1 and novel cell type classification (Sec. 4.3 performance of scCello by using full or partial pre- pre-training losses. Removing any of the four losses in Eqn. Jj in decreased performance, corroborating the benefits of the proposed pre-training losses. Notably, removing the inter-cellulat ontology relation loss Linter led to 56.1 % and 65.3% decrease in terms of Acc. and Macro F1 on novel cell type classification task, respectively. This shows the upmost importance of the structurally induced loss and ultimately the use of cell ontology graph information.
Parameter efficiency. Tab. demoosttattss that scCello is highly parameter-efficient, utilizing up to 60 times fewer parameters than the largest existing TFM, UCE, while still achieving the best average performance rankings across all downstream tasks. With an average performance rank of 1.3, scCello consistently ranks first near the top in nearly every task.
Visualization. Visualization and analysis of scCello's learned cell representations were presented in App. E. In short, biologically similar cell types are closer to each other and farther from those dissimilar ones in the t-SNE 2D space (Fig.
5 Discussion and Conclusion
Limitation and future work. The cell ontology is constantly revised and expanded. In the future. we plan investigate more efficient methods for fine-tuning scCello enable continual learning of updated ontology, rather than retraining the entire model. Additionally, we aim to scale up the model size of scCello increase its expressiveness and capacity. For the zero-shot marker gene prediction experiments (Sec. 4.4), one caveat is that our in-silico gene knockout approach also detects essential genes such as housekeeping genes (17) and transcription factors that are master regulators (9. which may not necessarily be marker genes. Nonetheless, deletion of these influential genes will also lead to large change of the transcriptome landscape the cell. We will explore this in future study.
Societal impact. This work proposes novel cell antology-guided TFM, scCello, to enhance cell representation learning. On the positive side, once pre- pre-trained, scCello can serve foundational model capable facilitating scientific discoveries across various downstream tasks related to cells and cellular processes. However, the negative side, the pre-training of seCello requires significant computational resources, potentially resulting in substantial carbon dioxide emissions that could contribute environmental harm.
Conclusion. The proposed scCello incorporates cell ontology knowledge into its pre-training process by simultaneously modeling at the gene level, intra-cellular level, and inter-cellular level. We constructed a large-scale cell type identification benchmark evaluate the model's generalization capabilities, both in-distribution and out-of of-distribution. Our evaluation demonstrates that scCello also exhibits strong transferability, as evidenced by its performance on other biologically meaningful downstream tasks such as zero-shot novel cell type classification and prediction. Foundational models are typically heavy on the parameters for them to have sufficient capacity to learn from unlabeled data from scratch. This limits their usage to only fine-tuning tasks as pre-training them prohibitive without large compute. Our proposed approach provides an efficient way of leveraging the prior knowledge at the pre-training, which led to much smaller parameter size while achieving performance comparable of the TFMs that are 5-60 times bigger. Together, scCello is a knowledge-informed and general purpose deep learning model that can be fine-tuned
marker gene

---

for a wide array of downstream applications, aiding in the rapid identification of novel cell types, genes, and effective cancer drugs.
Acknowledgments and Disclosure of Funding
The authors would like to thank Chence Shi, Meng Qu, Zhaocheng Zhu, and Sophie Xhonneux for their helpful discussions and comments. We also appreciate all anonymous reviewers for their constructive suggestions. This project is supported by Intel-MILA partnership program, the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR Al Chair Program, collaboration grants between Microsoft Research and Mila, Samsung Electronics Co., Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino- Bird Gift Fund and a NRC Collaborative R&D Project (AIAD-CORE-0)). This project was also partially funded by IVADO Fundamental Research Project grant PRF-2011-35833 0000019333119922555 139727. Y.L. is supported by Canada Research Chair (Tier 2) in Machine Learning for Genomics and Healthcare (CRC-2021 1-0054)) and Natural Sciences and Engineering Research CoURCi((NSERC) Discovery Grant RRGPIN-2016-055 74). The computation resource of this project is supported by Mila, Caleul Québec and the Digital Research Alliance of Canada.
References
[1] Shibla Abdulla, Brian D. Aevermann, Pedro Assis, Seve Badajoz, Sidney M. Bell, Emanuele Bezzi, Batuhan Cakir, Jim Chaffer, Signe Chambers, J. Michael Cherry, Tiffany Chi, Jennifer Chien, Leah Dorman, Pablo Garcia-Nieto, Nayib Gloria, Mim Hastie, Daniel Hegeman, Jason Hilton, Timmy Huang, Amanda Infeld, Ana-Maria Istrate, Ivana Jelic, Kuni Katsuya, Yang-Joon Kim, Karen Liang. Mike Lin, Maximilian Lombardo, Bailey Marshall, Bruce Martin, Fran McDade, Colin Megill, Nikhil Patel, Alexander V. Predeus, Brian Raymor, Behnam Robatmili, Dave Rogers, Erica Rutherford, Dana Sadgat, Andrew Shin, Corinn Small, Trent Smith, Prathap Sridharan, Alexander Tarashansky, Norbert Tavares, Harley Thomas, Andrew Tolopko, Meg Urisko, Joyce Yan, Garabet Yeretssian, Jennifer Zamanian, Arathi Mani, Jonah Cool, and Ambrose J. Carr. Cz cellxgene discover: single-cell data platform for scalable exploration, analysis and modeling of aggregated data. bioRxiv, 2023.
[2] Peter Angel and Michael Karin. The role of jun, fos and the ap-l complex in cell-proliferation and transformation. Biochimica et Biophysica Acta (BBA)-Reviews on Cancer, 1072(2-3):129- 157, 1991.
[3] Sercan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In Proceed- ings of the AAAI conference on artificial intelligence, volume 35, pages 6679-6687, 2021.
[4] Jonathan Bard, Seung Y Rhee, and Michael Ashburner. An ontology for cell types. Genome biology, 6:1-5,2000
[5] Nurken Berdigaliyev and Mohamad Aljofan. An overview of drug discovery and development. Future medicinal chemistry, 12(10)993999) 2020.
[6] Vincent D Blondel, Lean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment, 2008(10):P10008, 2008
[7] Felipe A. Vieira Braga, Gozde Kar, Marijn Berg, Orestes A. Carpaij. Krzysztof Polański, Lukas M. Simon, Sharon Brouwer, Tomás Gomes, Laura Hesse, Jian Jiang, Eirini Sofia Fasouli, Mirjana Efremova, Roser Vento-Tormo, Carlos Talavera-López, Marnix R. Jonker, Karen Affleck, Subarna Palit, Paulina M. Strzelecka. Helen V. Firth, Krishmaa T. Mahbubani, Ana Cvejic, Kerstin B. Meyer, Kourosh Saeb-Parsy. Marjan A. Luinge, Corry-Anke Brandsma. Wim Timens, Ilias Angelidis, Maximilian Strunz, Gerard H. Koppelman, Antoon J. M. van Oosterhout, Herbert B. Schiller, Fabian J Theis, Maarten van den Berge, Martijn C. Nawijn, and Sarah A. Teichmann. cellular census human lungs identifies novel cell states in health and in asthma. Nature Medicine, 25:1153
1 1163, 2019.
10

---

[8] Maria Brbic. Marinka Zitmik, Sheng Wang, Angela Oliveira Pisco, Russ B. Altman, Spyros Darmanis, and Jure Leskovec. Mars: discovering novel cell types across heterogeneous single- cell experiments. Nature Methods, 17:1200 1206, 2020.
[9] Sunny Sun-Kin Chan and Michael Kyba. What master regulator? Journal of stem cell research & therapy, 3, 2013.
[10] Ruben Chazarra- Gill Stijn van Dongen, Vladimir Yu Kiselev, and Martin Hemberg. Flexible comparison of batch correction methods for single-cell ma-seq using batchbench. Nucleic acids research, 49(7):c42-cd) c42, 2021.
[11] Fan Chen, Yini Zhang, and Karl Rohe, Targeted sampling from massive block model graphs with personalized pagerank. Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1):99-126, 2020.
[12] David Combe, Christine Largeron, Mathias Géry, and Elöd Sriyed-Zs I-louvain: An attributed graph clustering method. In Advances in Intelligent Data Analysis XIV: 14th International Symposium, IDA 2015, Saint Etienne. France, October 22-24, 2015. Proceedings
14, pages 181-192. Springer, 2015.
[13] William Connell, Umair Khan, and Michael J Keiser. A single-cell gene expression language model. arXiv preprini arXiv:2210. 14330, 2022.
[14] Haotian Cui, Chloe X. Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. segpt: toward building foundation model for single-cell multi-omics using genera- tive Nature methods, 2024.
[15] Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 10tv00480048 2018.
[16] Alexander D. Diehl, Terrence F. Meehan, Yvonne M. Bradford, Matthew H. Brush, Wasila M. Dahdul, David S. Dougall, Yongqun He, David Osumi-Sutherland, Alan Ruttenberg, Sirarat Sarntivijai, Ceri E. Van Slyke. Nicole A. Vasilevsky, Melissa A. Haendel, Judith A. Blake, and Christopher J. Mungall. The cell ontology 2016: enhanced content, modularization, and ontology interoperability. Journal of biomedical semantics, 7(44), 2016.
[17] Eli Eisenberg and Erez Y Levanon. Human housekeeping genes, revisited. TRENDS in Genetics, 29(10):569-553 574, 2013.
[18] Felix Fischer, David S Fischer, Evan Biederstedt, Alexandra Chhlod Villani, and Fabian Theis. Scaling cross-tissue single-cell annotation models. bioRxiv, 2023.
[19] Felix Fischer, David S. Fischer, Evan Biederstedt, Alexandra-Chie Villani, and Fabian J. Theis. Scaling cross-tissue single-cell annotation models. bioRxiv, 2023.
[20] Dániel Fogaras, Balázs Rácz, Károly Csalogány, and Tamás Sarlós. Towards scaling fully personalized pagerank: Algorithms, lower bounds, and experiments. Internet Mathematics, 2(3):333-358, 2005.
[21] Oscar Franzén, Li-Ming Gan, and Johan LM Björkegren. Panglaodb: web server for explo- ration of mouse and human single-cell ma sequencing data. Database, 2019:baz046, 2019.
[22] Florian Graf, Christoph Hofer. Mare Niethammer, and Roland Kwitt. Dissecting supervised contrastive learning. In International Conference on Machine Learning, pages 3821-3830.
PMLR, 2021.
[23] Arie Hagberg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(LANN.), Los Alamos, NM (United States), 2008
[24] Will Hamilton, Zhitao Ying, and Jure Leskovee, Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
11

---

[25] Minsheng Hao. Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Le Song, and Xuegong Zhang. Large scale foundation model on single-cell transcriptomics. bioRxiv, pages 2023-05, 2023.
[26] Minsheng Hao, Jing Gong, Xin Zeng. Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Le Song, and Xuegong Zhang. Large scale foundation model on single-cell transcriptomics. bioRxiv, 2023.
[27] Yuhan Hao, Tim Stuart, Madeline Kowalski, Saket Choudhary, Paul Hoffman, Austin Hartman, Avi Srivastava, Gesmira Molla, Shaista Madad, Carlos Fernandez-Grande et al. Dictionary learning for integrative, multimodal and scalable single-cell analysis. Nature biotechnology, 42(2))293-304, 2024.
[28] Graham Heimberg, Tony Kuo, Daryle DePianto, Tobias Heigl, Nathaniel Diamant, Omar Salem, Gabriele Scalia, Tommaso Biancalani, Shannon Turley. Jason Rock, et al. Scalable querying of human cell atlases via foundational model reveals commonalities across macrophages. bioRxiv, pages 2023 07, 2023.
[29] Congxue Hu, Tengyue Li, Yinggi Xu, Xinxin Zhang, Feng Li, Jing Bai, Jing Chen, Wenqi Jiang, Kaiyue Yang. Qi Ou, et al. Cellmarker 2.0: an updated database of manually curated cell markers in human/mouse and web tools based on scrna-seq data. Nucleic Acids Research, 51 (1(D1):D870-DD7 2023.
[30] Congxue Hu, Tengyue Li, Yingqi Xu, Xinxin Zhang, Feng Li, Jing Bai, Jingrun Chen, Wenqi Jiang, Kaiyue Yang. Qi Ou, Xia Li, Peng Wang, and Yunpeng Zhang. Cellmarker 2.0: an updated database of manually curated cell markers in human/mouse and web tools based on scrna-seq data. Nucleic Acids Research, 51:D870 D876, 2022.
[31] Aleksandr lanevski, Anil K. Giri, and Tero Aittokallio. aully-automated and ultra-fast cell-type identification using specific marker combinations from single-cell transcriptomic data. Nature Communications, 13, 2022.
[32] Hyun Min Kang, Meena Subramaniam, Sasha Targ. Michelle Nguyen, Lenka Maliskova, Elizabeth McCarthy, Eunice Wan, Simon Wong. Lauren Bymes, Cristina M Lanata, et al. Multi- plexed droplet single-cell rna-sequencing using natural genetic variation, Nature biotechnology. 36(1):89-99) 94, 2018
[33] Hyun Min Kang, Meena Subramaniam, Sasha Targ, Michelle Ly Thai Nguyen, Lenka Maliskova, Elizabeth E. McCarthy, Eunice Wan, Simon Wong, Lauren E. Byrnes, Cristina M. Lanata, Rachel E. Gate, Sara Mostafavi, Alexander Marson, Noah A. Zaitlen, Lindsey A. Criswell, and Chun Jimmie Ye. Multiplexed droplet single-cell rna-sequencing using natural genetic variation. Nature biotechnology, 36:89 94, 2017.
[34] Kasia Zofia Kedzierska, Lorin Crawford, Ava Pardis Amini, and Alex X Lu. Assessing the limits of zero-shot foundation models in single-cell biology. bioRxiv, pages 2023-10, 2023.
[35] Hadas Keren-Shur Ephraim Kenigsberg, Diego Adhemar Jaitin, Eyal David, Franziska Paul, Amos Tanay, and Ido Amit. Mars-seq2. 0: an experimental and analytical pipcline for indexed sorting combined with single-cell ma sequencing. Nature protocols, 14(6): 100118841-1866 2019.
[36] Shima Khoshraftar and Aijun An. A survey on graph representation learning methods. ACM Transactions on Intelligent Systems and Technology. 15(1):1-55, 2024.
[37] Cheorl-Ho Kim. Glycobiology of innate immunology. Springer, 2022.
[38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6966 2014.
[39] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint 160i: 2016.
[40] Ilya Korsunsky, Nghia Millard, Jean Fan, Kamil Slowikowski, Fan Zhang, Kevin Wei, Yuriy Baglaenko, Michael Brenner, Po-ru Loh, and Soumya Raychaudhuri. Fast, sensitive and accurate integration of single-cell data with harmony. Nature methods, 16(12):1299-1226 2019.
12

---

[41] Samuel A Lambert, Arttu Jolma, Laura F Campitelli, Pratyush K Das, Yimeng Yin, Mihai Albu, Xiaoting Chen. Jussi Taipale, Timothy R Hughes, and Matthew T Weirauch. The human transcription factors. Cell, 172(4):650-6653 2018.
[42] Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: framework and review. leee Access, 8: 8:193907-193934, 2020.
[43] Qiao Liu, Zhiqiang Hu, Rui Jiang, and Mu Zhou. Deepedr: hybrid graph convolutional network for predicting cancer drug response. bioRxiv, 2020.
[44] Romain Lopez, Jeffrey Regier, Michael Cole, Michael Jordan, and Nir Yosef. Deep generative modeling for single-cell transcriptomics. Nature methods, 15(12):1053-1055 2018.
[45] Mohammad Lotfollahi, Mohsen Naghipourfar, Malte D. Luecken, Matin Khajavi, Maren Büttner, Marco Wagenstetter, Ziga Avsec, Adam Gayoso, Nir Yosef, Marta Interlandi, Sergei Rybakov, Alexander V. Misharin, and Fabian Theis. Mapping single-cell data to reference atlases by transfer learning. Nature Biotechnology. 40:121 130, 2021.
[46] Malte D. Luecken, Maren Büttner, Kridsadakom Chaichoompu, Anna Danese, Marta Inter- landi, MF Mueller, D Strobl, Luke Zappia, Martin Dugas, Maria Colomé and F Theis. Benchmarking atlas-level data integration in single-cell genomics. Nature Methods, 19:41 50, 2020.
[47] Malte D Luecken, Maren Büttner, Kridsadakom Chaichoompu, Anna Danese, Marta Interlandi, Michaela F Müller, Daniel C Benchmarking atlas-level data integration single-cell genomics. Nature methods, 19(1):41-500 2022.
Strobl, Luke Zappia, Martin Dugas, Maria Colomé-Tatché et al.
[48] Colin Megill, Bruce Martin, Charlotte Weaver, Sidney Bell, Lia Prins, Seve Badajoz, Brian McCandless, Angela Oliveira Pisco, Marcus Kinsella, Fiona Griffin, et al. Cellxgene: performant, scalable exploration platform for high dimensional sparse matrices. bioRxiv, pages 2021-04, 2021.
[49] Gyutaek Oh, Baekgyu Choi, Inkyung Jung, and Jong Chul Ye. schyena: Foundation model for full-length single-cell ma-seq analysis in brain. arXiv preprint arXiv:2310.02713, 2023.
[50] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit- learn: Machine learning in python. the Journal of machine Learning research, 12:2825-288 28300, 2011.
[51] Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pages 28043-28078. PMLR, 2023.
[52] Yixuan Qiu, Jiebiao Wang, Jing Lei, and Kathryn Roeder. Identification of cell-type-specific marker genes from co-expression patterns in tissue samples. Bioinformatics, 32(19):3228333) -2344, 2021.
[53] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018
[54] William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846-88) 850, 1971.
[55] Yanay Rosen, Yusuf Roohani, Ayush Agrawal, Leon Samotorcan, Tabula Sapiens Consortium, Stephen Quake, and Jure Leskovee. Universal cell embeddings: A foundation model for cell biology. bioRxiv, pages 2023-11, 11, 2023.
(56) Peter Rousseeuw. Silhouettes: a graphical aid the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
13

---

[57] T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. arXiv preprint arXiv:2303. 2023.
[58] Robert Salomon, Dominik Kaczorowski, Fatima Valdes-Mora, Robert E Nordon, Adrian Neild, Nona Farbehi, Nenad Bartonicek, and David Gallego-Ortega. Droplet-based single cell rnase tools: a practical guide. Lab on Chip, 19(10)1706-1127, 2019.
[59] Hongru Shen, Jilei Liu, Jiani Hu, Xilin Shen, Chao Zhang, Dan Wu, Mengyao Feng, Meng Yang, Yang Li, Yichen Yang, et al. Generative pretraining from large-scale transcriptomes for single-cell deciphering. Iscience, 26(5), 2023.
[60] Charlotte Soneson and Mark D Robinson. Bias, robustness and scalability in single-cell differential expression analysis. Nature methods, 15(4):255-261, 2018.
[61] Christina V. Theodoris, Ling Xiao, Anant Chopra, Mark D. Chaffin, Zeina R Al Sayed, Matthew C. Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng. X. Shirley Liu, and Patrick T. Ellinor. Transfer learning enables predictions in network biology. Nature, 618:616- 624, 2023.
[62] Ander Urruticoechea, Ramon Alemany, J Balart, Alberto Villanueva, Francesc Vinals, and Gabriel Capella. Recent advances in cancer therapy: an overview. Current pharmaceutical design, 16(1):3-10, 2010.
[63] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[64] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.109900 2017.
[65] Felipe A Vieira Braga, Gozde Kar, Marijn Berg, Orestes A Carpaij, Krzysztof Polanski, Lukas M Simon, Sharon Brouwer, Tomás Gomes, Laura Hesse, Jian Jiang, al. A cellular census of human lungs identifies novel cell states in health and in asthma. Nature medicine, 25(7):1153- 1 1163, 2019.
[66] Hui Wan, Liang Chen, and Min Deng. scemail: Universal and source-free annotation method for serna-seq data with novel cell-type perception. Genomics, Proteomics & Bioinformatics, 20:939 958, 2022.
[67] Hanzhi Wang, Zhewei Wei, Junhao Gan, Sibo Wang, and Zengfeng Huang. Personalized pagerank to a target node, revisited. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 657-667, 2020.
[68] Wenchuan Wang, Fan Yang, Yuejing Fang, Duyu Tang. Junzhou Huang, Hui Lu, and Jianhua Yao. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell ma-seq data. Nature Machine Intelligence, 4:852 866, 2022.
[69] Xiliang Wang, Yao He, Qiming Zhang, Xianwen Ren, and Zemin Zhang. Direct comparative analyses of 10x genomics chromium and sea.t-seq2. Genomics, Proteomics and Bioinformatics, (9(2):253-266 2021.
[70] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. Emergent abilities of large language models. arXiv preprint wxXiv2220.07688/ 2022.
[71] F Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology, 19:1-5, 2018
[72] Chenling Xu, Romain Lopez, Edouard Mchlman, Jeffrey Regier, Michael Jordan, and Nir Yosef. Probabilistic harmonization and annotation of single-cell transcriptomics data with deep generative models. Molecular systems biology, 17())e9620, 2021.
[73] Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang. Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. schert as large-scale pretrained deep language model for cell type annotation of single-cell ma-seq data. Nature Machine Intelligence, 4(10):852-866 2022.
14

---

[74] Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems, 34:28798- 28810, 2021.
[75] Hongyu Zhao, Tianyu Liu, Kexing Li, Yuge Wang, and Hongyu Li. Evaluating the utilities of large language models in single-cell data analysis. 2023.
[76] Suyuan Zhao, Jiahuan Zhang, and Zaiqing Nie. Large-scale cell representation learning via divide-and-conquer contrastive learning, arXiv preprinl arXiv:2306. 230,073,0,,37 2023.
15

---

Cell Ontology Graph
Random Walker
Personalized PageRank (PPR)
Native cell
PPR(U.))
PPR(u.))
Nucleate cell
PPR(
Regulatory T cell
random walk path from to rundom walk puth from to k
Figure 4: Graphical illustration of applying the Personalized PageRank (PPR) algorithm to cell ontology graph. As explained App. A PPR conducts random walks over the ontology graph with respect to target cell type u, and converges each node stabilizes into steady distribution. This likelihood distribution determines the final PPR score PPR(-) and reflects the structural similarity between cell types.
steady state when the likelihood of terminating on
A PPR Transformation
Personalized PageRank (PPR). Personalized PageRank (PPR) extends the classic PageRank algo- rithm, which Google originally developed to rank web pages in search engines. PageRank conducts this by analyzing large-scale hyperlinked graphs the web using random walker simulations. Unlike traditional PageRank that assigns a universal score to each web page, PPR customizes these scores. Specifically, individual user preferences during searches are incorporated, that PPR can focus on web pages particularly relevant to each user. Due to its flexibility and effectivvnesss, PPR has been widely applied in graph learning across various fields, such social networks, recommendation systems, and biological data analysis.
As illustrated in Fig. 4 this algorithm starts with predefined preference node (or target node), which emphasized according to the user's s interests. Subsequently, a random walk is conducted on the graph to facilitate graph traversal. At each step of the walk, there fixed probability that the walker will jump back to the target node from the current node instead of moving to an adjacent node chosen at random. This process of jumping, commonly referred walk towards subgraphs that are particular importance to the target node, thus personalizing the results according to user preferences. The walk continues until it reaches steady state, which point the likelihood of being on each node stabilizes into a steady-state distribution. These stabilized
"teleportation", biases the
10
10¹
Count
10
Count
10
10²
10
10
10
10
10"
PPR Score PPR(.)
Structural Similarity sim(-)
(a) Distribution of PPR scores.
(b) Distribution of structural similarity.
Figure 5: Comparison of the distributions for the PPR scores PPR(-) and the structural similarity sim(-) after the transformation.
16

---

10
Count Per Cell Type
PPR Score PPR())
10
10
10
10'
10
10
10"
10"
2
3
10
II
2
6 Structural Similarity simm()
10
11
Structural Similarity sim((
Figure 6: Relationships between the structural similariity sim(-) ()) after PPR transformation and the original PPR scores PPR.
Figure 7: Frequency for each target cell type to be associated with other cell types that is at specific levels of structural similarity.
probabilities, reflecting both the graph's s structure and the user's preferences, determine the PPR scores. These scores effectively evaluate each node's structural similarities and rank them according to their relevance and importance from a personalized perspective.
PPR transformation. In scCello, the PPR algorithm is applied to the cell ontology graph to the structural similarities among cell types, measure their importance relative specified target cell type. We implemented PPR using the "pagerank" function in NetworkX [23 with "personalization" arguments.
However, modification needed to integrate PPR into TFM pre- pre-training. The PPR scores are in real-number format and susceptible to numerical noise. Also, shown in Fig. Sa these scores typically exhibit a skewed distribution, concentrated around lower magnitudes. Consequently, setting precise thresholds to differentiate between node similarity and dissimilarity is challenging. Moreover, the vast amount small PPR values may be indistinguishable from noise.
To mitigate the effects of numerical noise and skewed magnitudes for the PPR scores, we employ truncation, logarithmic scaling, and discretization as outlined in Eqn. 3 Note that Eqn. 3 defines monotonic, non-decreasing function that preserves the relative order between nodes. Its minimum value is set for the least similar cell types.
This equation transforms the raw PPR score, PPR(-), into the final structural similarity, sim(-). This transformation ensures that sim(-) accurately reflects pronounced similarities defined by the cell ontology and avoids emphasizing minor dissimilarities that could mislead during TFM pre-training.
present comparison of the distributions for the PPR score, PPR(-). and the transformed structural similarity, sim(-). After transformation, the distribution of sim(.) is less skewed and exhibits clear discretization. This facilitates the setting of definitive thresholds for distinguishing between similarity and dissimilarity among cell types, thereby enabling the effective incorporation of the cell ontology graph in scCello's 's pre-training.
Analyses. In Fig.
In addition, we provide detailed insights into the scale of structural similarity, the distribution of these similarities for each cell type, and examples of cell types associated with various levels of structural similarity:
Fig. 6 iillustrates the correspondence between the struetural similarity after PPR transfor- mation and the original PPR scores, showcasing log-linear relationship expected. This helps clarify the scaling of structural similarity, which is discretized into integer levels ranging from to 11.
(2) Fig. demonstrates how frequently each target cell type is associated with other cell types at specific levels of structural similarity. Consequently, during scCello's pre-training, substantial number of negative samples are expected to be utilized in the inter-cellular relational alignment objective, outlined in Sec. 2.4
17

---

Table 7: Examples of cell types associated with various levels of structural similarity, sim(.), for specified target cell types. Cell types demonstrated in the cell ontology graph in Fig. I are underlined.
Target Type sim()
Corresponding Cell Types
Cell
"gamma-delta T' cell"", "mature cell", "lymphocyte"
"maurre gamma-deles cell", "a-3 cell"" "mature a-3 T cell", "thymocyte"
"B cell". doouble-osittve,, 0-8 thymocyte". "CD&-positive, a-8 thymocyte". "CD4-positive, o-8 cell", "CD8-positive, o-33 T cell", "double negative thymocyte"
"dendritic cell", "innate lymphoid cell", "plasmahlast", mononuulear cell", "regulatory cell", "memory cell", "mycloid leukoeyte", ", "naive T cell", "mature B cell", "CD4-positive, CD25-positive, a-8 regulatory T cell"
5
"renal intercalated cell", "smooth musele fiher ileum", "type If pneumncyte", "hemutopoictic cell", "neuron". "common lymphoid progenitor".
Neuron
7
"secretory cell"
glutamatergie neuron", "GABAergic neuron", "motor neuron", "neural cell", "peripheral nervous system neuron"
glycinergic neuron", " "retinal bipolar neuron", , "native cell",
5
"enteric neuron", "retina horizontal cell", "amacrine cell",, "ncuronal receptor cell"
"retinal ganglion cell",, "endocrine cell", "neuroendocrine cell", "cerebral cortex GABAergic intemeuron", "muscle cell", "somatic cell"
"germ cell". cell", "tracheal goblet cell". "DN3 thymocyte", "promonocyte", "cerehral cortex endothelial cell",
(3) Tab. 7 displays examples of highly similar and dissimilar cell types categorized into various levels of structural similarity, specifically targeting "T cell" and "neuron" types.
B Data Preprocessing Details
Download and Preprocessing. We downloaded from CellxGene census version 2023-7-25. We focused on 291 datasets for human scRNA-seq. We preprocessed the dataset by the following steps:
(1) Remove non-primary cells. Some data on CellxGene was duplicated due to multiple submissions of the same dataset from different research groups, therefore cells marked as "non-primary" were filtered out to prevent label leakage between pre-training and down- stream.
(2) Filter out cells not produced by 10x-based 69 sequencing protocols. There numerous sequencing protocols in CellxGene database besides 10x-based based sequencing [69] such as Drop-seq (58) and MARS-seq 35
Only sequencing data from 10x-based sequencing protocols was kept to avoid large variation of data signals [46]
(3) Exclude cancer cells. Cancer cells were highly dissimilar to normal cells and even occupied large amount in the CellxGene database (nearly 12%). These cells could bring unexpected signals and skew the data, therefore we excluded these cancer cells.
To build downstream datasets for out-of-distribution (OOD) generalization evaluation, we first held out two category sets for each of the three settings: unseen cell types, unseen tissues and unseen
18

---

Table Data statistics for our curated pre-training and downstream datasets, where the downstream datasets encompass one ID dataset and six OOD datasets under three different OOD scenarios, including unseen cell types, unseen tissues and unseen donors (Sec. (.11 The blue colored numbers represent disjoint categories of that column. For example, in the "cell type" column, the cell type set in the pre-training data, and the cell type set in the OOD cell type dataset D and D are disjoint.
Dataset
#Total Cells
#Cell Types
#Tissues
#Donors
#Conditions
#Batches
Pre-training data
22,293,755
398
140
4,103
55
267
ID dataset D
22.317
318
132
3,447
54
261
OOD cell type dataset D
486,810
87
125
122
35
90
OOD cell type dataset D
435,791
87
106
0OD tissue dataset D
335,675
186
32
1,801
10
28
OOD tissue dataset D
341,681
205
32
2,052
7
25
OOD donor dataset D
2,528, 134
439
91
525
OOD donor dataset n
2,521,868
404
101
525
33
123
Total
572
204
5153
donors. Each category set were randomly selected with selection ratios 15%, 15% and 10% for the three OOD settings respectively. During the selection, we prohibited any category associated with more than 0.1% of the total pre-processed cells from being selected. This avoids losing too much data for pre-training, After the selection, cells associated with each held category set are collected, resulting in two OOD downstream datasets for each of the three OOD settings. These datasets are denoted as (D Ji=1 for the OOD cell type setting, { for the OOD tissue setting, and ( (D for the OOD donor setting.
Ti=1
By excluding cells with at least one property belong any of the six held category sets, the remaining data is further split into 99.9% as our pre-training data and 0.1% as the in-distribution (ID) downstream dataset Did This way, our pre-training data and the ID dataset D'd share similar data distributions.
Data Statistics. We summarize the data statistics for our curated pre-training dataset, one ID dataset and six OOD datasets Tab.
C Discussion on Ontology Graph Modeling
Graph Neural Networks (GNNs). One essential component of scCello is to model the cell ontology prior graph. GNNs are essential for managing graph-structured data by using a process called message passing (36]. This process lets nodes gather information from their neighbors, capturing their local connections and features, as seen in technologies like GCN [39]. Further advancements like GraphSAGE (24) and GAT [64] allow GNNs to handle larger areas of the graph and more complex relationships, useful in fields from social networks to drug discovery.
Recently, combining GNNs with transformer models, like GraphFormer (74), has proven effective. This integration allows the models process both textual and graph data simultaneously, enhancing the understanding of the graph's 's structure without needing external measures.
Why left GNNs for future work. While using GNNs in TFM modeling can internalize ontology graph knowledge in TFM modeling and no custom metric like that for PPR transformation necessary, the cellular ontology graph presents specific challenges that hinder effective direct modeling by GNNs. The key issues is that the ontology graph is extremely sparse with about 2.7k nodes and 3.9k edges, and faces long-distance issues. For example, the average pairwise distance of 398 nodes (i.e., cell types) associated with our pre-training scRNA-seq data is 7.39, and the maximum distance is 18. Therefore, requires multiple layers of graph propagations (around layers), risking smortshog [57] of cell type representations.
19

---

Table 9: Syper-parameters comparison between TFM baselines (introduced in Sec. 4.1 and our TFM
scCello. "The number of" is denoted with the symbol #.
Configuration
Geneformer [61]
scGPT [4
seTab 19
UCE [55
scCello
#Parameters
10,316,196
51,330,049
9,655.628
674,745,857
10,683,654
Total GPUs
12 * V100 (32G)
4 * A100
1 * A100
24 # A100 (80G)
4 * A100 (40G)
Training Time
days
3 days
/
43.5 days
days
Sequence Length
2,048
1,200
19,331 /
1,024 [N]
2,048
Gene Mask Ratio
15%
20%
15%
Batch Size Per GPU
12
32
2,048
6
12
Gradient Accumula- tion Steps
1
1
1
4
Effective Batch Size
144
128
2048
576
192
Cell Reprs.
Avg. pooling
CLS
CLS
CLS
#Genes in Token Vocabulary
Any protein- coding genes
25,424
48,292
19,331
25,424
#Transformer Layers
6
12
/
33
6
Transformer Layer
512
512
/
5,120
512
Hidden Dimension
Transformer Layer
256
512
1,280
256
Embedding Size
#Transoommrr Heads
4
8
1
20
Transformer Layer
GeLU
ReLU
/
ReLU
ReLU
Activation Function
MIP Layer Acti-
ReLU
ReLU
GeLU
ReLU
vation Function
Dropout
0.02
0.2
0.05
0.02
D Implementation Details
scRNA-seq Data. scRNA-seq can enable the quantification of gene expression profiles of individual cells. Each cell's gene expression profile can be described by the set X (1). (02, 92)..................... (CM, 9ax) ]. where ég denotes the expression count of gene gk. with 6 2 0. A value of eg = indicates that the gene 9b is not expressed or not detected by the sequencing experiment. We use the same gene vocabulary set as (61, with the number of genes M-25, 424.
Gene Token Vocabulary. The gene vocabulary set contains both protein-coding genes and miRNA genes. M, the number of genes, is not the same as the number of all tokens in the model vocabulary. scCello has M gene tokens plus three more special tokens [MASK] for masking, [CLS] for the start of a sentence and [PAD] for padding.
Rank Value Encoding. Unlike natural languages, which inherently follow a sequential order, scRNA-seq data presents a unique challenge due to the lack of intrinsic order among gene tokens. Therefore, we employ Rank Value Encoding [61] approach to rank genes based on their normalized expression set ((e, 9:) g.) Specifically, gene expressions are first normalized by the total count within a cell [7T] in a cell-wise manner, and then normalized through gene-specific weighting factors in gene-wise manner. These factors are adopted from [61], which calculates the non-zero median value of expression of each detected gene across all cells. By design, these factors are assigned to emphasize wwly-expressedd but essential genes, such transcription factors 41. while deprioritizing ubiquitously expressed housekeeping genes [7].
After the normalization and ranking, it results in ordered sequence of gene identities X = (1); (2)............................. ..........9333 (M)) with an index permutation x()), satisfying εn(1) 2 (x(() 2 ... 2 (M))
To mitigate memory consumption, zero-expressed genes are removed and the gene sequence is
20

---

Table 10: Metrics used in downstream tasks.
Task
Metrics
Cell Type Clustering (Sec. 4.2.
NMI, ARI, ASW, AvgBio
Cell Type Classification (Sec. 4.2.2
Acc, Macro F1, AvgBio, AvgBBi
Novel Cell Type Classification (Sec. 4.3)
Acc, Macro F1
Marker Gene Prediction (Sec. ).4)
AUROC
Cancer Drug Response Prediction (Sec.
4.5
PCC
NMI, ARI, ASW, AvgBio, as GraphConn, AvgBatch, Overall
Batch Integration (Sec. 4.6
further truncated with a context length I==2, 048 in practice. This rank-based approach offers better robustness against technical artifacts than directly using the original numerical expressions, which can vary significantly in magnitude across different experimental assays [46].
Cell and Cell Type Representations. Given pre-training dataset with N cells X {X1, XX1,X2..... Xe...., XN), ). each cell X2 can be mapped to specific cell type ontology identifier E V. For analyzing, scCello denotes cell X, 's representation as 2, and cell type oj representation he.-
Masked Gene Prediction. Given a batch of cells {X.] scCello predicts gene token 9h based on the ordered gene sequence context (z)x-191.--- 1, [MASK], 9x41 .9r] after replacing the token with special [MASK]. This objective (term (MGP) aims capture complex but important gene-gene interactions within one cell, like regulatory mechanisms between transcription factors and other genes:
MMPP - E log
(6)
where tokens are masked by a pre-defined distribution V, same as that in BERT [5]. 80% selected genes are replaced with [MASK], 10% selected genes are kept the same as its original, and 10% selected genes are replaced with random gene tokens.
Specifically,
Model Architecture. scCello utilizes stack of self-attention transformer encoder layers [63] eacg composed self-attention and feedforward neural networks. The self-attention mechanism processes the input sequence, effectively capturing interactions between gene tokens.
Besides scCello, we also summarize essential hyper- parammterseserss for TFM baselines Tab. Llb.9fee for comparison. It includes pre-training configurations like batch size, sequence length, and training time consumed. It also includes architecture configurations for the transformer model backbone, such as the number of transformer layers and the embedding size of transformer layers. Note that scaaab uses TabNet [3] instead of transformer layers as model backbone, therefore architecture configurations are not recorded in the table.
Configuration Slyper-paramette
E Downstream Experiment Details
F.1
Evaluation Metrics
All metrics used in downstream tasks are summarized in Tab. 10 and introduced below.
Normalized Mutual Info Score (NMI). The NMI is a metric that quantifies the similarity between two differen clustering assignments or labelings of the same set of samples. We use NMI to compare the cell-type labels, with the cluster indices obtained from applying the Louvain clustering algorithm [12] the target dataset.
We denote the two label assignments the same cell samples as and K, representig the cell-type labels and the Louvain cluster indices, respectively. The entropy of label assignment, say C, is
21

---

measure of the uncertainty associated with that assignment set. It's calculated as:
|C| H[C)= - Σ P(t) log P(i)
where |C'| is the number of unique cell types and P(t) |C,l is the probability that a randomly
selected sample belongs to the class C. The entropy H(K) for the cluster indices K is computed
similarly, with Q(1) |Kyl N being the probability of a sample belonging to the cluster Ky:
|K|
H(K) - - log
5=1
The mutual information (MI) between and qunatifies the amount of information shared between the two label assignments. It is calculated by:
ICI |K|
R(i.j) P(i)Q()
MI(C, = Σ
Σ R(i.)) log
(9)
where R(i,j) k is the probability that randomly selected sample belongs to both the class C2 and the cluster Ky. The normalized mutual information (NMI) is defined as:
MI(C, K)
NMI(C, = mean HC)), (K))
(10)
NMI is normalized version of MI, scaled by the mean of the entropy terms for cell-type labels and cluster indices. This normalization ensures that NMI values range from 0 to I, where O indicates no correlation between the two label assignments, and represents perfect match.
To obtain the best match between the clusters and the cell-type labels, we performed optimized Louvain clustering over range of resolutions from 0.1 to 2, in steps of 0. The clustering output with the highest NMI score, when compared to the cell-type label set, was selected as the optimal clustering result. The implementation of NMI used in this study was from the scib python library [47]
Adjusted Rand Index Score (ARI). The ARI another metric used to evaluate the similarity between the clustering assignment and the cell type labels of the same set of samples, similar to the NMI metric. In this context, we similarly denote the cell-type labels as C and the Louvain [2 cluster indices computed on the target dataset K.
The Rand Index (RT) is measure of the overlap between the two clusterings, C and K. considers both the correct clustering overlaps and the correct disagreements between the two clusterings [54]] Formally, we define a as the number of pairs of elements that belong to the same set in both C' and K, and b as the number of pairs of elements that are in different sets in C' and in different sets in K, the unadjusted is given by:
RI b
(11)
C.
where is the total number of cell samples and C. represents the total number of possible pairs in the dataset.
However, the unadjusted RI does not account for the possibility of random label assignments leading to correct overlaps by chance. To address this issue, the adjusted RI (ARI) is introduced, which corrects for randomly correct labels by discounting the expected random labelings:
RI
R E
ARI
(12)
max MaI
22

---

The ARI ranges from O to 1, where O corresponds to random labeling, and indicates perfect match between the two clustering assignments.
Similar to NMI, we performed NMI-optimized I Louvain clustering to obtain the best match between the clusters and the cell-type labels. Specifically, we executed Louvain clustering a range of resolutions and selected the clustering output with the highest NMI score when compared to the cell type label set. The implementation of ARI used in this study was from the scib python library [47).
Average Silhouette Width Score (ASW). The silhouette width [56] is a metric that evaluates the quality of a clustering solution by quantifying the relationship between the within-clustering distances and the between-cluster distances for each data point. Like the NMI and the ARI, the silouette calculates the similarity between the clustering assignment and the cell type labels of the same set of samples.
For each cell sample, the silhouette width is computed based on two scores: (1) a: the mean distance between a sample and all other samples in the same cluster; and (2) the mean distance between a sample and all samples in the nearest neighboring cluster. The silhouette score for each sample i is defined as
b
8;
(13)
max b)
The silhouette score ranges from -1 to I, with higher values indicating that the sample is well-matched to its own cluster and dissimilar to the nearest neighboring cluster.
To obtain an overall assessment of the clustering quality, the average silhouette width (ASW) is calculated by averaging the silhouette scores s; across all samples. This overall ASW, denoted as ASW ranges between -1 and 1, with the following interpretations:
ASW, close to I: The clusters are dense and well-separated.
ASW around 0: The clusters overlap, and the between-cluster and within-cluster variability are approximately equal.
ASW near -1: Strong misclassification has occurred, where the within-cluster variability is greater than the between-cluster variability.
To ensure that the final ASW metric falls within the range of 0 to 1, scaling operation is often applied:
ASW. +1
ASW
(14)
2
This scaled ASW value, ranging from O to 1, provides a convenient measure for evaluating the quality of the clustering solution, with higher values indicating better separation and cohesion of the clusters.
AvgBio. This score combines the three clustering metrics: NMI, ARI and ASW.
AvgBio (NMI + ARI ASW)
(15)
Silhouette Variant Score (ASW,). To evaluate the effectiveness of the batch integration task (Sec. (.6)), a variant of the average silhouette width score (ASW) is employed, referred to as the ASW & Unlike ASW based cell type labels, ASW, considers batch labels. This score is designed to assess the degree of batch mixing, where a score of 0 indicates well-mixed batches, and deviations from 0 suggest the presence of batch effect.
We take the absolute value of the original silhouette width score š, for sample i based on batch labels:
(16)
To ensure higher scores indicate better batch mixing, these scores are scaled by subtracting them from 1. As we expect batches to integrate within cell identity clusters, we compute the ASWa.j score for each cell label j separately, using the following equation:
ASWa,
1
Σ 1-s(1i) s(i) IEC
(17)
[c
23

---

where C = {ālc = ji is the set of cell indices whose cell type label is exactly j.
To obtain the final ASW, score, the label-specific ASW scores averaged across the set of unique cell type labels:
ASW = Σ
ASW
(18)
DV
JEV
where V represents the set of unique cell type labels.
Graph Connectivity (GraphConn). The GraphConn metric is designed to assess whether the k-nearest neighbor (kNN) graph representation of the integrated data directly connects cells with the same cell type label. This metric operates on the ANN graph, denoted GRNN, which is pre-processed by the Scanpy library using the caappy-p..ncihbors. function.
For each cell type label v E V. where V represents the set of cell type labels (Sec. 2). subset ANN graph GANN & is created. This subset graph contains only cells from the given label v.
Using these subset kNN graphs, the GraphConn score is computed as follows:
[L.CC(GRNN Ve. Ev))
GraphConn
Σ
(19)
[V]
Pvl
NEV
Here, [LOC(.) is the number of nodes in the largest connected component of the graph and [Vl is the number of nodes with cell type v.
The resultant GraphConn score has a range of (0; 1], where a score of 1 indicates that all cells with the same cell type connected in the integrated ANN graph. The lowest possible score indicates graph where no cell is connected to any other cell.
It's important to note that the GraphConn score is computed directly the kNN graph representation of the integrated data. As a result, this metric can be used to evaluate the quality of any integration output, regardless of the specific integration method used.
AvgBatch. This score combines metrics: ASW, and GraphConn.
AvgBatch (ASW, + GraphConn)
(20)
Overall. We follow scGPT 14 to calculate a weighted average score of both the batch removal score ASW, and the bio-conservation score AvgBio balance biological relevance and batch consistency, following the equation:
Overall 0.6 * AvgBio + 0.4 AvgBatch
(21)
Accuracy (Acc). In classficcation tasks like cell type classification (Sec. 4.2.2) and novel cell type
classification (Sec. 4.3), we denote the predicted values of the i-th sample as y/s and the corresponding true label y/- Then the accuracy metric is defined as
1 N
Acc(y, 5)
Σ
1
10s
yil
(22)
where the 1|:] is the indicator function.
Macro F1 Score (Macro F1).
The F1 Score is essentially defined for binary classification tasks.
2
Fi Recall + Precision
(23)
TP
Recall
(24)
TP + FN
TP
Precision TP + FP
(25)
where TP the number of true positives, FN the number of false negatives, and TP the number of false positives. The recall is intuitively the ability of the classifier to find all the positive samples; The precision intuitively the ability of the classifier not to label as positive sample that negative. For multi-class classification, macro Fl is defined the average Fl taken over all different classes.
24

---

ROC AUC Score (AUROC). The Area Under the Receiver Operating Characteristic (AUROC) curve is metric commonly used to evaluate the performance of binary classification models. provides a comprehensive measure of the trade-off between the true positive rate (sensitivity) and the false positive rate (1 specificity) across different classification thresholds.
In a binary classification task, the model's output is typically probability or score that represents the likelihood of a sample belonging to the positive class. By varying the classification threshold, different operating points on the ROC curve can be obtained, where each point represents a specific combination of true positive rate (TPR) and false positive rate (FPR).
The ROC curve is created by plotting the TPR (y-axis) against the FPR (x-axis) for different classifi- cation thresholds. The AUROC is then calculated the area under this ROC curve, providing single scalar value that summarizes the overall performance of the binary classifier. The AUROC ranges from O to I, with the following interpretations: (1) AUROC=1 indicates perfect classification, where the classifier can perfectly distinguish between the positive and negative classes; (2) AUROC=0.5 indicates random guessing, indicating that the classifier performs no better than a random prediction.
The AUROC a widely used metric because provides a comprehensive evaluation the classifier's performance across all possible classification thresholds. It is invariant to class imbalance and does not require choosing a specific threshold, making a robust and barehhld-agnostie measure.
Furthermore, the AUROC has statistical interpretation as the probability that randomly chosen positive instance will have a higher predicted probability than randomly chosen negative instance, which provides clear interpretation of the metric's value.
Pearson correlation coefficient score (PCC). The PCCis widely used measure of the linear relationship between two variables. It quantifies the strength and direction of the linear association between the variables, ranging from to The formula for the PCC between two variables, A and B, is given by:
E (A, B) B
TAB
VE (A, -A) Ei=1 Ei -B
B)²
where A; and B, are the individual observations of variables A and B, respectively. A and B are the sample means of A and B, respectively. n is the number of observations.
The numerator represents the covariance between and B, which measures how much and B vary together from their respective means. The denominator normalizes the covariance by the product of the standard deviations of and B, ensuring that the correlation coefficient falls within the range of -1 to 1. The interpretation of this PPC metric is follows: (1) rAB=1 indicates perfect positive linear correlation (as A increases, B increases proportionally); TA) -1 indicates perfect negative linear correlation (as A increases, B decreases proportionally)) (3) TAB= indicates no linear correlation between A and B; (4) 0 < |TB < 1 indicates that the strength of the linear correlation increases as the value approaches 1 (either positive or negative).
In the context of regression analysis, computing the PCC between each regressor (independent variable) and the target variable can provide insights into the linear relationships between the predictors and the response variable.
E.2
Cell Type Identification
E.2.1
Zero-shot Identification (i.e., Cell Type Clustering)
We here discuss the experimental details for Sec. 4.2.1 Cell representations extracted from each baseline model are used to compute the k nearest neighbor (kNN) graph using Scanpy's standard protocols [7 These representations and the kNN graph are then processed with Louvain clustering algorithms various resolutions. ranging from 0.1 to 2 in steps of 0.1. The optimized clustering result is determined by the highest gained NMI score achieved across all the resolutions.
Method.
For implementation, we accelerated Louvain clustering by adopting RAPIDS, a software library that enhances data science pipelines by entirely utilizing NVIDIA GPUs instead of traditional CPUs. Additionally, we conducted ten iterations of dataset down-sampling and reported the averaged NMI,
25

---

Table 1I: Full results for the OOD unseen cell type datasets Det and D in the type clustering.
OOD Cel Data (Dft)
OOD CellType Data (Det
Method
NMI
ARIT
ASW1
AvgBiot
NMI
ARIT
ASW1
AvgBiot
Non- TFM-TFMMEE Methods
Raw Data
0.864
0.718
0.529
0.703
0.823
0.557
0.505
0,629
Scurat
0.893
0.773
0.590
0.752
0.884
0.723
0.605
0.737
Harmony
0.553
0.241
0.432
0.432
0.594
0.248
0.411
0.417
SC
0.905
0.797
0.577
0.760
0.889
0.709
0.577
0.725
Ontology-Aee -oooogyy-gnostti TFMs
Geneformer
0,846
0.697
0.525
0.689
0,846
0.629
0.530
0,668
scGPT
0.866
0.705
0.551
0.707
0.873
0.724
0.564
0.720
scCab
0.886
0.807
0.584
0.759
0.867
0.754
0.557
0.726
UCE
0.902
0.802
0.612
0.772
0.892
0.695
0.635
0.741
MGP
0.860
0.710
0.573
0.714
0.881
0.745
0.595
0.740
Sup
0.892
0.787
0.621
0.767
0.910
0.793
0.622
0.775
MGP+Sup
0.888
0.775
0.611
0.758
0.901
0.779
0.611
0.764
Ontology-Enhancel TFMs
scCello
0.887
0.781
0.640
0.769
0.909
0.817
0.632
0.786
ARI, ASW, and AvgBio scores. This approach significantly reduced the time required to evaluate dataset, such as Díd from days to just few minutes.
Datasets. As introduced in Sec. 4.2.1 we evaluate one ID dataset (Díd) and six OOD datasets (Dcond with cond {dl, (s, dn) and {1,2]) demonstrate model's generalization capabili- ties. These evaluations address various scenarios involving unseen cells for comprehensive testing, including cells with distributions similar our pre-training dataset, as well as those associated with unseen cell types, tissues, and donors.
We used & 15 neighbors compute the kNN graph, with node distances calculated using the euclidean distance between cell representations. The Louvain clustering used seed 0 the random state and treated the &NN graph unweighted and directed.
Stapee-parameters
Performance. In Sec. 4.2.1 Tab. I reports only the AvgBio metric for six OOD datasets due to space constraints. Full metrics, including NMI, ARI, and ASW, detailed in: (1) Tab. for the two OOD cell type datasets (Df* and D (2) Tab. 12 for the two OOD tissue datasets (D)s and D)"); and (3) Tab. 13 for the two OOD donor datasets (Ddn and D
E.2.2 Identification with Fine-tuning (i..,, Cell Type Classification)
Method. In this setting, the TFMs are further fine-tuned by adding a simple linear layer atop their model backbones, which transforms the hidden representations into prediction logits. The dimensions of these logits correspond to the number of cell type classes predicted. Importantly, all model parameters, including those of the TFM backbone and the newly added linear layer, are trainable during fine-tuning. The model checkpoint that achieves the highest Macro F1 score on the validation data is then selected for final testing.
Datasets. We fine-tuned TFMs on subset of our curated pre-training data, randomly selecting 90% for training and using the remaining 10% for validation. The final performance was tested on the ID dataset Did which consists cell samples never seen during scCello pre-training. We explored two subset sizes, 0. % and 1% of the pre-training data, to simulate scenarios where 10x more annotated data becomes available. This exploration is meaningful for real-world applications, where annotating data is both costly and time-consuming.
26

---

Table 12: Full results for the OOD unseen tissue datasets Dj and D the cell type clustering.
OOD Tissue Data (Df-)
OOD Tissue Data (D22)
Method
NMI
ARIT
ASW1
AvgBiot
NMI
ARIT
ASW1
AvgBiot
Non- TFM-TFMMTT Methods
Raw Data
0.733
0.405
0.481
0.540
0.800
0.585
0.508
0,631
Scurat
0.777
0.497
0.488
0.587
0.813
0.560
0.535
0.636
Harmony
0.649
0.302
0.436
0,462
0.684
0,400
0.460
0.515
SC
0.774
0.443
0.516
0.577
0.816
0.550
0.537
0.634
Ontology Baaooogy-Agnnnnne Agnostic TFMs
Geneformer
0.736
0.412
0,468
0.539
0.787
0,499
0.505
0.597
scGPT
0.739
0.407
0.486
0.544
0.794
0.556
0.531
0.627
scCabb
0.754
0.492
0.515
0.515
0.815
0.616
0.541
0.657
UCE
0.787
0.476
0.531
0.598
0.836
0.610
0.562
0.670
MGP
0.766
0.472
0.491
0.576
0.802
0.544
0.537
0.628
Sup
0.788
0.502
0.527
0.605
0.838
0.621
0.580
0.680
MGP+Sup
0.789
0.518
0.524
0.610
0.833
0.612
0.573
0.672
Intology-Enhaneee TFMs
scCello
0.784
0.519
0.534
0.612
0.839
0.675
0.601
0.705
Table 13: Full results for the OOD unseen donor datasets D and D the Cell Type Clustering. Note that scTab is OOM on these two datasets.
OOD Donor Data (Dim)
OOD Donor Data (Dám)
Method
NMI
ARIf
ASW1
AvgBiot
NMI
ARI
ASW1
AvgBiot
Non-TFM Methods
Raw Data
0.665
0.247
0.462
0.458
0.665
0.251
0.462
0.460
Seurat
0.691
0.294
0.413
0,466
0.711
0.335
0.420
0.489
Harmony
0.679
0.286
0.405
0.456
0.690
0.324
0.408
0.474
ScVI
0.699
0.269
0.466
0.478
0.722
0.311
0.471
0.502
Ontology-
Iatology-Agnostic Agnostic TFMs
Geneformer
0.666
0.303
0.434
0.468
0.686
0.327
0.433
0.482
scGPT
0.656
0.259
0.452
0,456
0.677
0.298
0.456
0,477
scTab
OOM
/
0OM
UCE
0.718
0.245
0.491
0,485
0.737
0.284
0.496
0.506
MGP
0.713
0.294
0.457
0.488
0.734
0.359
0.462
0.518
Sup
0.754
0.357
0.545
0.552
0.768
0.395
0.556
0.573
MGP+Sup
0.754
0.373
0.532
0.553
0.768
0.398
0.544
0.570
Ontology-Enhon Enhaaced TFMs
scCello
0.774
0.426
0.625
0.608
0.794
0.486
0.649
0.643
Hyper-parameters. For scCello, we set the following rate of 5.0 10-5 linear learning rate scheduler with 500 warmup steps, weight decay of 0.001, and a batch size of 24. The same fine-tuning configuration was applied to the three ablation TFMs pre-trained using scCello's codebase (MGP, Sup, and MGP+Sup). For other TFM baselines, searched for the optimal learning rate to report the final performance.
for fine-tuning: learning
Performance. Sec. 4.2.2, we reported classification clustering metrics for TFMs fine-tuned with the 0.1$ subset of the pre-training data. Here, we extend our reporting to TFMs fine-tuned with 1% pre-training subset that is x larger. We compare performances at these two subset selection ratios in Tab. We observe that,
27

---

Table 14: Cell type identification with fine-tuning evaluated on the ID dataset D as the pre-training subset data size for fine-tuning increases from 0. % to I % for the subset selection ratio.
Cell Type Classification
Cell Type Clustering
Methods
Acct (0.1% + 1 1%)
Macro FI (0.1% 1%)
AvgBiot (0. % 1%)
Ontology-Agnostic TFMs
Geneformer
0.747 - 0.872
0.440 - 0.664
0.439 - 0.469
scGPT
0.712 > 0.862
0.344 - 0.636
0.477 - 0.481
scTab
0.778 + 0.773
0.373 - 0.455
0.606 + 0.589
MGP
0.722 + 0.861
0.287 - 0.639
0.607 + 0.631
Sup
0.812 + 0.902
0.363 - 0..718
0.659 + 0.668
MGP+Sup
0.820 0.902
0.406 0.735
0.607 0.667
Ontology-Enhancee TFMs
scCello
0.867 - 0.910
0.511 - 0.761
0.694
0.699
Step 1: Known and unknown cell types
Step Similarity of query to known cell types type repr.
Step 3: Similarity of query unknown cell types
Step 4: Decide query cell type
A(g.0)
PPR
Known cell types # E kn
cell repr.
similarity s(e.)) vo Tu
PERIU P1
similarity s(u.))
Unknown cell lypes v E Vunka
2(0.us) ut)
PPRcis Dj
PPREM
U2
ate Nz)
Query cell
* M) (PR((L P)... (P.( (P.RG,.)...PPP(R
s(p.0) (cQeaa).s.... us). (s.
4(g. spearman(erq..))ccooooo s).s(v.())
arg max
Figure 8: Graphical illustration of our approach for classifying novel cell types (i.e., unknown cell types) (introduced in App.
(1) As the size of fine-tuning data increases, all TFMs except sc'Tab show benefits and scCello achieves 48.9% improvement Macro Fl when the data size gets 10x larger. scTab's underperformance may be related its model capacity, it employs TabNet architec- ture 3 unlike others that use the powerful standard Transformers [63]
Across both the classification and clustering metries, scCello's prevails other TFM baselines by a large margin. Remarkably, even when fine-tuned with a smaller 0.1% pre-training subset, scCello surpasses TFMs fine-tuned with much larger 1% subset, achieving 3.9% improvement over the best baseline. This underscores scCello 's superiority, attributed to its cell ontology- guided pre-training.
(3) Interestingly, clustering performance does not necessarily correlate directly with classifica- tion performance. For instance, while MGP+Sup outperforms Sup in classification metrics, it does not do so in clustering metrics. This observation underscores the importance to evaluate both the clustering and classification performances for cell type identification with model fin-tuning, which can make the evaluation setting more comprehensive and rigorous.
E.3 Novel Cell Type Classification
Method. In this task, we define "known cell types" Vkn C V as the 398 cell types from our labeled pre- pre-training dataset (see dataset statistics Tab. g "Novel cell types", "unknown cell types" Vunten C V, are those present only in the target downstream dataset and not observed during TFM pre-training (Vunkn V V..)
Given new query cell y, we aim to classify one the unknown cell types Vunkn- To solve this problem, choose to first calculate representations for both the query cell sample and the unknown
28

---

cell types. And then we measure the similarity between the two representations to determine the prediction results Vy Vunkn-
Since unknown cell types are absent from the pre-training dataset, their representations cannot be directly obtained from any TFM baselines our model, despite its ability to learn representations for known cell types. To address this problem. we leverage the known cell types Vkn as a bridge to represent the query cells through the similarity between the cell and cell type representations produced by TFMs, and also represent the unknown cell types using the structural similarity relationships between the known and unknown ones derived from the cell ontology graph.
Specifically, our approach is illustrated in Fig. and involves the following steps:
(1) Representations for known cell types. Although scCello inherently learns cell type repre- sentations during pre-training, most existing TFMs do not output cell type representations directly. For benchmarking, we propose a protocol to calculate known cell type repre- sentations for general TFMs. Specifically, the representation for each known cell type is calculated by averaging cell representations derived from TFMs across cells belonging to this cell type. We used cell samples from subset (10%) of our curated pre-training dataset, because the whole 22 million dataset is too large to fit.
We denote the known cell type representations (hu uev... to differentiate with the nota- tion of scCello's 's learned cell type representations (h I ueve. introduced in Sec. 2 For fair comparison, scCello also follows this protocol to generate known cell type representations, instead of using its learned ones. Nevertheless, emphasize seCello's capability conduct this task alone without further accessing reference databases like our pre-training dataset.
(2) Similarity vector for a query cell to known cell types. We first derive the cell repre- sentations for the query cell from TFMs. Then, estimate the similarity between the query cell q and any known cell type u E Vkn using the cosine similarity between their representations s(g, u) = zg hu. For all known cell types, this results in a similarity vector:
sqg u) = [d(g, u1), d(q, ......., d(g, viml)
(26)
where we define the order of vector indices as u [u], u2, u....... ................................ Waa satisfying in
U2 <... < V
(3) Similarity vector for unknown cell types to known cell types. For each unknown cell type v E Vunkn, we estimate the similarity s(v, u) between the unknown v and the known cell types u. To achieve this, we leverage the cell ontology graph to calculate structural proximities as proxies. The proximities are measured using the raw PPR score PPR(u, v), V Vankn, which introduced in Sec. 2.4 Therefore, the similarity vector can be represented as:
s(v, u) = [PPR(U, v), PPR(u2, P.....,,PP u]],
(27)
(4) Align the similarity vectors for the query cell and the unknown cell types. Intuitively, the similarity vector s(g, u) indicates profiling for the query cell g. with known cell types frame of reference; and the similarity vector s(v, u) conveys similar profiling for an unknown cell type v. Therefore, the more similar the two similarity vectors 8(g, u) and s(u, u.) is, the higher possibility for the query cell to be alike this unknown cell type. We derive it using Spearman Ratio 50 Spearman as the similarity measure:
s(g, v) Spearman (g u), s(v, u)).
(28)
Other formulas for the vector similarity function are available, like the commonly used cosine similarity (i.c., d(g, v) d(q. u s(g, u)). Our approach is not sensitive to the choice of the similarity metric. As shown in Fig. 10, using the dot product as the similarity score led to similar relative performance as in Fig. 2 where scCello generally performs better or on par with other TFMs. Therefore, we used Spearman Ratio throughout the experiments.
(5) Select the final answer. The unknown cell type o" with the largest distance is selected as the prediction for novel cell type classification:
e" arg max s(g
(29)
29

---

1.0
Genelommer
MGP
Geneloeerr
MGP
waTash
MGP Sup
0.6
MGP Sup
Accuracy (Acc)
UCEE
soC cello
UCE
soC clla
0.6
Macro F1
0.4
0.3
0.1-
10%
25%
50%
75%
100%
10%
25%
50%
75%
100%
Portion of the Total #Cell Types
Portion of the Total #Cell Types
(a) Acc for Dj (same as Fig. put for comparison).
(h) Macro F1 for DT
0.7
ScciP T
Sup MGP-Sop
0.6-
sctiP Tc UCE
wcTbb
MGP-Sop
Accuracy (Ace)
0.6
Macro F1
0.4
0.2
galoe
0.0
10%
25%
50%
75%
100%
10%
25%
50%
75%
100%
Portion of the Total #Cell Types
Portion the Total #Cell Types
(c) Acc for D
Macro for D
Figure 9: Novel cell type classification on two OOD cell type datasets Dj and Dj using the Spearman Ratio similarity measure to compare the representations of the query cells and the novel cell types (App. E.3). Two metrics Acc and Macro Fl are reported.
In real-world applications, our approach is still applicable since almost all cell types are included in the cell ontology graph. But we won't be able to know whether the newly coming query cells are from unknown cell types Vunkn or known cell types Vkn- Therefore, we can expand the unknown cell type set Vunkn to all the cell type defined in the ontology graph V. and conduct similar processes in our approach.
Datasets. We evaluate on OOD cell type datasets D and D The cell types in Det and D are already aligned to the cell ontology graph using the ontology identifiers provided by CellxGene database, and subset all the unknown cell types V Vunkn. We recognize that the prediction task becomes more challenging as the number of novel cell types increases. Therefore, we constrain the complete unknown cell type set to the cell types occurred the datasets we used.
To further reflect the challenge, we created five difficulty levels, where the number of cell types spanned from 10%, 25%, 50%, 75% 100% of the total cell type count. For example, if we use 25% cell types the OOD cell type dataset D with a total 87 cell types, the unknown cell types include (87 25% 22) randomly selected cell types from the complete [c;]X; Dj To account for potential biases, we randomly sampled 20 distinct combinations of cell types for each difficulty level.
The PPR(-) score calculated using the "nx. "nx.pagerank" function with alpha hyper-parameter set to 0.9.
The full metrics for both accuracy and macro fl score on the two OOD cell type datasets D and Dę are reported Fig. Besides plots, the numerical results are also summarized in Tab. [6 and Tab. for reference.
Performance.
30

---

Table 16: Novel cell type classification results on OOD cell type dataset Drt.
10% cell types
25% cell types
50% cell types
75% cell types
100% cell types
Method
Acct
Fir
Acct
Fit
Acct
F1
Acet
FIT
Acef
Fif
Ontology-Agnostic TFMs
Geneformer
0.392
0.207
0.226
0.095
0.157
0.050
0.135
0.036
0.123
0.027
scGPT
0.291
0.178
0.148
0.072
0.041
0.062
0.024
0.052
0.020
scTab
0.380
0.248
0.191
0.096
0.114
0.058
0.088
0.042
0.077
0.035
UCE
0.399
0.253
0.289
0.120
0.205
0.064
0.149
0.040
0.131
0,030
MGP
0.361
0.243
0.233
0.119
0.125
0.048
0.089
0.032
0.076
0.022
Sup
0.464
0.389
0.329
0.200
0.187
0.109
0.139
0.075
0.111
0.055
MGP+Sup
0.556
0.358
0.341
0.172
0.217
0.089
0.193
0.069
0.172
0.056
TFMs
scCello
0.768
0.559
0.547
0.365
0.442
0.246
0.364
0.177
0.335
Table 17: Novel cell type classification results on OOD cell type dataset D5
10% cell types
25% cell types
50% cell types
75% cell types
100% cell types
Method
Acct
Fif
Acct
FIf
Acct
F1
Acct
FI1
Accf
FIf
Ontology-Agnost TFMs
Geneformer
0.367
0.213
0.310
0.125
0.107
0.048
0.069
0.027
0.047
0.020
scGPT
0.411
0.223
0.217
0.085
0.108
0.041
0.077
0.031
0.061
0.025
scTab
0.338
0.245
0.175
0.102
0.113
0.053
0.074
0.038
0.049
0.027
UCE
0.339
0.227
0.244
0.119
0.119
0.064
0.056
0.035
0.037
0,027
MGP
0.411
0.270
0.225
0.114
0.057
0,066
0.036
0.038
Sup
0.581
0.372
0.325
0.204
0.199
0.112
0.140
0.081
0.108
0.063
MGP+Sup
0.428
0.315
0.228
0.143
0.131
0.082
0.069
0.047
0.061
0.041
Ontology-Enhance TFMs
scCello
0.763
0.5(0)
0.498
0.304
0.364
0.196
0.297
0.149
0.247
0.124
E.4 Marker Gene Prediction
We here explain our approach for this task in details. Given cell's gene expression profile, we enumerate each gene and attempt to knock it out, either by replacing with special [MASK] token or by reducing its expression to zero. The former method is used for Geneformer, MGP, Sup, MGP+Sup, and scCello, while the latter is applied to scGPT, scTab, and UCE. By comparing the cell representations of the mutated expression and those of the original expression, we assess the impact each gene's 's knockout. greater impact suggests a higher likelihood of the gene being a marker gene. This zero-shot approach requires no further fine-tuning and is particularly useful when additional computational resources or annotated datasets for fine-tuning are unavailable.
Method.
Notably, we acknowledge the shortage of our method: for house keeping genes (i.e., non-marker genes), knocking out these genes will also have large impact on the cell because the cell would die [7] Therefore, high impact from gene knockout does not necessarily indicate marker gene, but rather an "important" gene. However, this issue not critical empirically, as the number of well-documented housekeeping genes is about 400, which is small compared to the extensive gene token vocabulary of M 25, 424.
Datasets. As introduced Sec. 4.4 we used the datasets from GSE96583 33] and GSE130148 7 One the one hand, the GSE96583 dataset Dyak inherently contains five cell subsets associated with cell type classes. The five cell subsets are denoted as "GSE96583 1". "GSE96583 2". "GSE96583 3", "GSE96583 4", "GSE96583_5 _5" respectively. On the other hand, the GSE130148 dataset Dzuh contains 13 cell type classes. The size of these two datasets are summarized Tab. 21 and their associated cell types are recorded in Tab. 20 for demonstration. Additionally, the ground truth cell-type-speciff marker genes are originally sourced from two databases: CellMarker2 and PanglaoDB 21
31

---

1.0
Genetremer
MCPP
0,9
Genetoenrr
MOP
xailab
MCP Sup
szilaa
MGP
Accuracy (Acc)
UCE
soC cill
0.7
UACE
soC ella
Macro F1
0.6
0,5
0.4
0.3
0.2
0.2
10%
25%
50%
75%
100%
10%
25%
50%
75%
100%
Portion of the Total #Cell Types
Portion of the Total #Cell Types
(a) Acc for D
(h) Macro F1 for D
1.0
0.9
ScCiPT scTab
Sup
SECPPT soTab
Sup
MGP+-Spp
0.8
MGP-Sup
Accuracy (Ace)
0.8
UCT
0.7
Maero F1
0.6
0.6
0.5
0.4
0.4
0.3
0.2
0.2
10%
25%
50%
75%
100%
10%
25%
50%
75%
100%
Portion of the Total #Cell Types
Portion the Total #Cell Types
(c) Acc for D
Macro for D
Figure 10: Novel cell type classification on two OOD cell type datasets DJ and Dff using the cosine similarity measure to compare the representations of the query cells and the novel cell types (App. E.3 Two metrics Acc and Macro Fl are reported.
Performance. In Sec. 4.4 we only reported average performance across the 5 subsets of GSE96583 (Dyak) and the individual performance of GSE130148 (Dgu)) in Tab Here, we provide complete results for five subsets Tab.
E.5 Novel Marker Gene Prediction
Method.
We adopted methodology similar to our zero-shot marker gene prediction protocol (see for calculating differences cell representations through in-silico gene perturbation:
App. E.4)
(1) For each cell, we calculated the change in cell representations after removing each gene in-silico and selected the top 10% genes with the largest changes, excluding the known marker genes.
For each cell type, identified the 10 most frequent genes among the top 10% across all cells, to obtain 10 candidate novel marker genes.
ensure specificity, we removed genes present in more than one cell type.
We used the same two datasets (GSE96583 and (SE110144) in Sec. 4.4 for marker gene prediction, where marker gene labels were retrieved from CellMarker2 30 and PanglaoDB (21]
Datasets.
Performance.
As a case study, we followed the above steps to find novel marker genes for two cell
types:
For cell type "CD14+ Monocytes", two genes were found: FOS and LGALS2. FOS typically expressed in response to stress signals, cytokines, and growth factors 2 LGALS2
32

---

Table 19: Full results for the five data subsets from GSE96583 (Dirk) and one dataset from GSE130148 (Dith) ) in the marker gene prediction task (Sec. 4.4).
GSE96583 (D)
GSE130148 (D!T))
Method
GS895583 AUROCT
GSE96583 2
AUROCT
GSE96583
AUROCT
GSE96583 AUROC
GSE96583 AUROC1
Avg.
AUROCT
D und D
Mon-TFM Methods
DET
0.725
0.710
0.737
0,715
0,717
0.721
0
0.702
Ontology-Agnenn
Genefomer
0.445
0.447
0.478
0.484
0.408
0.452
0.470
01,661
scTab
0.666
0.654
0.589
0.693
0.660
0.672
0.727
0,700
UCE
0.502
0.499
0.500
0,499
0,500
0.500
0.500
0.500
MGP
Sup
0.707
0.697
0.694
0.699
0.700
0.699
0.693
0.696
MGP+Sup
0.734
0.720
0.739
0.734
0.724
0.730
0.730
0.730
Ontology-
scCello
0.767
0.753
0.754
0.748
0.760
0.756
0.729
0.743
Table 20: Cell types for the two marker gene prediction datasets GSE96583 (Diuk) and GSE130148
(Drr)) )
Dataset
Cell Types
"Dendritic cells", "CD8 T cells", "NK cells", "B cells", "Megakaryocytes" "FCGR3A+ Monocytes", "CD14+ Monocytes", "CD4 T cells", "Not Known"
GSE96583
"Macrophages", "T cell", "NK cell", "Mast cell", "Endothelium", "Lymphatic", "Pulmonary Alveolar Type II", "Transformed epithelium', "Ciliated", "Pulmonary Alveolar Type I", "B cell", "Fibroblast", "Secretory"
GSE130148
is involved in modulating immune responses and inflammatory processes in monocytes 37 Both are plausible marker gene candidates.
For cell type "Megakaryocytes" five genes were found: GNG1I, TUBBI, H2AC6, CAVIN2, CLU. GNGI is confirmed as marker genes in literature, TUBB1 is a likely marker, while H2AC6, CAVIN2, and CLU require further investigation.
E.6 Cancer Drug Response Prediction
Method. In this task, first compute cell line level representations from scRNA-seq data and drug representations for associated drugs. Both these two representations are then input into the DeepCDR framework for training. Finally, we calculate the PCC between the predicted and actual IC50 values for each drug across all cell lines and report the average performance across all tested drugs.
Specifically, for TFMs, single-cell gene expression data inputted into each model to generate cell- specific representations for each gene. These are then aggregated into cell line-level representations through max- max-pooling across all genes for each dimension. Conversely, the DeepCDR method uses raw gene expressions, aggregating them directly before max-pooling. Additionally, drugs are represented graphs and encoded using graph neural networks to obtain drug representations.
Datasets. In experiments, utilized cell line and mrug-paired data pre-proce processed by Deep- CDR [43), including 223 drugs and 561 cell line bulk gene expression profiles for 697 genes from 31 different cancer types. Among the dataset, 89,585 cell line-drug samples were used for training and 4,729 for testing (26].
We following scFoundation's implementation to set the parameters in the DeepCDR framework, like "-use_gexx" as True, and both "-use mut" and "-use "-use_mettee methy" as False.
Hyper-parammeers.
33

---

Table 21: The number of cell samples (#Cells) for the marker gene prediction datasets GSE96583
(DT"F) and GSE130148 (Diuh).
Dataset
GSE96583_
GSE96583_2
GSE96583_3
GSE96583_4
GSE96583_S
GSE130148
#Cells
4,246
3,639
14,619
14,446
6,145
10,360
Table 22: The correlation of the ontology structure and the pairwise similarity of known cell type representations
Method
Spearman R
Non- TFM Methods
Raw Data
0.212
Seurat
0.316
Harmony
0.262
Ontology-Agnostic TFMs
Geneformer
0.284
seGPT
0.037
scTab
0.209
UCE
0.285
MGP
0.275
Sup
0.229
MGP+Sup
0.238
Ontology-Enhanced TFMs
scCello
0.506
Performance.
Results are already reported in Tab. in Sec. 4.5
E.7
Batch Integration
Method. This batch integration task aims to seamlessly integrate scRNA-seq data from different batches, which can be conducted using the same protocol as cell type clustering. After clustering. model performance evaluated. Besides using cell type labels and clustering indices from the optimized Louvain algorithm to calculate the preservation of biological signals (NMI, ARI, ASW and AvgBio), this task also batch labels to measure the removal of batch effects (ASW, and AvgBatch). See App. E.I for metric calculation details.
Datasets. As introduced in Sec. 4.6 all datasets used in the cell type clustering task (Sec. 4.2.1 are evaluated, including one ID dataset Ddd and six OOD datasets Doond (cond {et, 18, dn). i (1,2)).
Hyper-parameters.
We use the same hyper- parametrasssssss that in cell type clustering.
Performance. In Sec. 4.6, the Overall score, weighted average of AvgBio and AvgBatch, is
already reported in Fig. 3 Complete results for all metrics are included in Tab. [20 for the ID dataset Did Tab. for the OOD cell type datasets Dr and Dşt Tab, for the OOD tissue datasets D and D and Tab. 26 for the OOD donor datasets D and Dd"
E.8 Visualization for Learned Cell Representations
calculate known cell type representation as introduced Sec. E.3
by averaging cell representa-
tions for each type on 10% of the pre-training data. Then apply 1SNE to project the known cell type representations
space and visualize in Fig. [T Highly correlated cell types are clustered together as expected, and dissimilar cell types are distant.
34

---

fibroblast epithelial cell
animal cell
CD4 (D4-positive, alpha-beta T cell myeloid cell myeloid leukocyte
connective tissue
progenitor cell
Figure 11: Visualization for learned cell representations of scCello (introduced App. E.8 The nodes are different cell types in the pre-training dataset and the edges denote "is a subtype of" relationships in cell ontology G. The coordinates of nodes are calculated using tSNE dimensional reduction for cell type representations derived from scCello. As expected, highly intoloyy-correlate cell type pairs are very close in the latent space, such as myeloid leukocyte and mycloid cell, as well fibroblast and connective tissue cell. Meanwhile, dissimilar cell type pairs remain distant, such as CD4-positive, alpha-beta cell and epithelial cell. The highly biologically informative representation space implies scCello's potential generalization ability other cell-type-related downstream tasks.
We also calculate the Spearman R correlation of the pairwise similarity of known cell type representa- tions and the ontology structure (1 for an edge between two cell types and O for no edge between them) in Tab. 22 As expected, scCello learned biologically informative representation space that much more correlated to the true ontology structure than other methods. This implies scCello's potential generalization ability other cell-type-related downstream tasks.
We also compare the cell representation distribution scCello with its ablated version excluding relational alignment objective (i.e., Eqn. 4. analyze how the ontology relational alignment loss benefits clustering performance. Following the same visualization method for Fig. [T] visualization results for the ablated model is shown in Fig. [2] Adding the relational alignment loss makes similar cell types clustered more closely and pushes dissimilar types farther apart. Therefore, relational alignment enables scCello align better with biological intuitions, and produce more effective cell representations as evaluated by broad downstreams.
35

---

fibroblast epithelial cell
animal cell
CD4 (D4-positive, alpha-beta T cell myeloid cell myeloid leukocyte
connective tissue
progenitor cell
Figure 12: Visualization for learned cell representations of scCello ablation without the relational alignment objective. Compared with Fig [1 without relational alignment objective, some highly correlated cell types are not clustered well together.
Table 23: Batch integration on ID dataset Did
ID Unseen Data (D"n)
Method
ASW.t
GraphConnt AvgBatcht
AvgBiot
Overall
Non-TFM Methods
Raw Data
0.951
0.806
0.878
0.419
0.603
Seurat
0.829
0.686
0.757
0.442
0.568
Harmony
0.824
0.688
0.756
0.421
0.555
ScVI
0.880
0.738
0.809
0.474
0.608
Ontology-Agnostic TFMs
Geneformer
0.875
0.676
0.775
0.432
0.569
ScGPT
0.887
0.691
0.789
0.438
0.578
scTab
0.917
0.925
0.921
0.577
0.715
UCE
0.906
0.788
0.847
0.489
0.632
MGP
0.870
0.728
0,799
0,473
0,603
Sup
0.885
0.809
0.847
0.555
0.672
MGP+Sup
0.892
0.829
0.860
0.516
0.654
Ontology-Enhaneee TFMs
scCello
0.834
0.697
0.766
0.670
0.708
36

---

Table 24: Batch integration on OOD cell type datasets Dr and D
OOD CeliType (LF')
OOD Celrtype (DT)
Method
ASW
GraphCoont
AvgHatcht
AvgBiof
Overall
ASW
GraphCount
AvgBatcht
AsgBiof
Overall?
Non-TFM Methods
Raw Dara
0.934
0.940
0.9.7
0.703
0.797
0,939
0.895
0.917
0.629
0.744
Scurat
0.831
0.928
0,880
0.752
0.803
0 1.44
0.932
0.888
0.737
0.797
Hammony
0.909
0.422
0.593
seVI
0.875
0,959
0.917
0.760
0.823
0,880
0,952
0.916
0.725
0.801
Outology-Ageee Aggnsstic TFMs
Geneformer
0.915
0.907
0.911
0.689
0.778
0.915
0.917
0.916
0.668
0.767
scGPT
0.903
0,913
0,908
0.707
0.787
0.896
0.927
0.912
0.720
0.797
scTabb
0.908
0.759
0.910
0.799
UCE
0.867
0.947
0.907
0,772
0.826
0.854
0.946
0.900
0.741
0.805
MGP
0.894
0.903
0.714
0.788
0.925
0.580
0.753
0.740
0.745
Sup
0.879
0.944
0.122
0.767
0.825
0.879
0.914
0.897
0.775
0.824
MGP+Sup
0.885
0.946
0.16
0.758
0.821
0 0.885
0.925
0.905
0.764
0.820
Ontology Enhanced TFMs
scCello
0.877
0.911
0.894
0.769
0.819
0.858
0.884
0.871
0.786
0.820
Table 25: Batch integration on OOD tissue datasets D and D
Method
(DF)
(D
AS
GraphCoent
AvgBatcht
AvgDiot
Overall
ASW
GraphCoen
AvgDatcht AvgBiot
Overall!
-TMMTFM Methods
Raw Dara
0.941
0.792
0.867
0.540
0.671
0.946
0.862
0.904
0.631
0.240
Seurat
0.865
0.830
0.847
0.587
0.691
0.867
0.841
0.854
0.636
0.723
Harmony
0.905
0.755
0,800
0,462
0.609
0.908
0.744
0.826
0.515
0.639
seVI
0.901
0.861
0.881
0.577
0.699
0.910
0.881
0.896
0.634
0.739
Outology Matology-Aguostcc TFMs
Gencformer
0.925
0
0.865
0.539
0.669
0.924
0.835
0,880
0.597
0.710
seGPT
0:916
0.776
0.846
0.544
0.665
0.920
0.826
0.627
0.725
scTab
0.916
0.872
0.894
0,515
0.667
0,917
0.874
0.896
0.657
0.753
UCE
0.905
0.864
0.885
0.598
0.713
0.911
0.879
0.895
0.670
0.760
MGP
0.887
0.576
0.700
0.901
0.628
0.720
Sup
0.903
0.932
0.918
0.605
0.730
0.899
0.911
0.905
0.680
0.770
MGP+Sup
0.900
0.941
0.921
0.610
0.734
0. 0.898
0.922
0.910
0.672
0.767
Ontology-Enthe Enhanced TFMs
scCello
0.868
0.841
0.855
0.612
0.709
0 0.884
0.819
0.852
0.705
0.764
Table 26: Batch integration on OOD donor datasets Drt and Ddn
OOD Donor Data (D)))
OOD Donor Data (D
Method
ASWat
GraphCoont
AvgBatcht
ArgBiof
Overall:
ASWat
GraphCoon?
ArgBatcht
AvgBior
Overall
Non-TFM Methods
Raw Data
0.945
0.785
0.865
0.458
0.621
0.946
0.787
0.867
0.460
0.623
Scurat
0.875
0.759
0.817
0.466
0.606
0.876
0.771
0.824
0.489
0.623
Harmony
0.893
0.618
0.756
0.456
0.576
0.89]
0.655
0.773
0.474
0.594
scVI
0.914
0.831
0.872
0.478
0.636
0.909
0.837
0.873
0.502
0.650
Ontolegy-Agune Aggastic TFMs
Geneformer
0.921
0.763
0.842
0.468
0.618
0.919
0.768
0.844
01,482
0.627
xPGPT
0.920
0.757
0.839
0,456
0.609
0.920
0.763
0.842
0.477
0.623
scTab
00M
OOM
0OM
0OM
UCE
0.904
0.605
0.907
0.558
0.597
MGP
0.910
0.824
0.867
0,488
0.640
0.906
0.814
0.860
0.518
0.655
Sup
0.909
0.877
0.893
0.552
0.688
0.902
0.857
0.880
0.573
0.696
MGP+Sup
0.553
Ontolugy Enhunced TFMs
scCello
0.845
0,608
37

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In this work, we propose scCello, novel TFM designed to leverage cell ontology priors for enhancing cell representation and understanding. This main claim is accurately and clearly stated and emphasized in the abstract and introduction. It reflects our contribution and scopes.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We state several limitations of our work in Sec. 5 such as the lack continue learning capability for our proposed model scCello, the relative small model scale for scCello, and our downstream approach for the zero-shot marker gene experiment.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
38

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We propose a novel cell-ontology guided transcriptomic foundation model sc- Cello in this work. It's a biological insight driven method and focuses on strong downstream applications. We do not establish theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the details needed reproduce our work. such as all experi- mental results for every metric and every dataset, pre-training setups in Sec. 1..laa and App. and downstream task settings in App.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
39

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our code and datasets will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.co... ctailss.cc/uublicguuiice httpai//ggiideee/eeeubbbbiiii wwt.Submmssiooooiiooooooooooioii for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the training and test details are justified in App. D for pre-training, and in App. [ for downstreams.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In this work, we provide error bars whenever random sampling on key exper- imental factors is involved. For example, in the novel cell type prediction task, we have random sampling procedures for cell type combinations and we report box plots with error bars to demonstrate the statistical significance of this experiments.
40

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: As indicated in Tab, 9 the pre-training proposed model requires training for days on A100 NVIDIA GPUs, each with 40G GPÚ memory.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics throughout the entire project.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
41

---

Answer:
[Yes]
Justification:
We discuss the social impact in Sec. for both positive and negative influences.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: All the datasets utilized for both pre-training and downstream tasks publicly available, and we did not implement explicit safeguards for these datasets. While proposed model, scCello, aims to advance understanding of cell representation learning for scientific discovery purposes and carries relatively little inherent risk for misuse, we acknowledge that misuse the methods were to fall into malicious hands. Despite our intentions for beneficial applications, the open availability of scCello introduces a degree of uncertainty regarding potential mishandling or nefarious exploitation that we cannot definitively preclude.
open-source model, cannot guarantee zero potential for
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-uss should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
42

---

Justification: All baseline methods and datasets are properly credited. Their license and terms of use are properly respected.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, papprssiitheeeeocc thoode com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: Code and datasets will be released upon acceptance. Clear documentations will be provided along.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: We did not perform crowdsourcing experiments and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
43

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
44


Reviews:
[{"review_id": "PLn2Szh8Qd", "reviewer": "Reviewer_yzGS", "summary": "This paper introduces scCello, a new transcriptome foundation model (TFM), which learns cell representations over RNA gene expressions. Apart from using Masked language modeling (MLM), masking random gene expressions in cells, the work leverages structural knowledge from ontologies to improve the learned representations. It enforces cell type coherence, where cells of the same type should be close in representation space. Similarly, the representation of structurally close cells, measured over a PageRank measure, should be close. These structurally inferred objectives are encoded as separate contrastive losses. The pretaining objective tries to minimise the sum over the MLM loss and the two contrastive losses together with a regularisation term, which tackles class collapse. scCello is backed by a transformer-based encoder-only model with roughly 10.5 min parameters, which is rather small compared to competitors. The evaluation covers multiple downstream tasks and an ablation study over the optimisation objectives. The results show that scCello is either competitive with state of the art performance or improves it and that the aggregation of the losses is beneficial.", "questions": "While you analysed the impact of the single training objectives, you didn't present numbers on different model sizes for scCello. Do you have any evidence how larger or smaller model impact downstream performances?\n\nIf ontologies and knowledge base grow, how does the training approach scale with them? Since you use contrastive loss, you could run into bottlenecks, if the number of negatives grows?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Data: It's great that the work leverages diverse data sources (structural and token-based) and tries to incorporate the strengths of both of them by using custom losses. \n- Experiments: The evaluation is extensive comparing the performance of the proposed model with multiple competitors and across multiple tasks. I also appreciated the ablation study and the analysis of overall performance wrt. #parameters. \n- Model: With a reasonable size the produced model can surpass many larger-sized competitors.", "weaknesses": "**Incorporating structural knowledge**\n\nGNN have been used for incorporating structural knowledge into pretraining of transformer-based models. For instance, take a look at GraphFormers (https://arxiv.org/abs/2105.02605). Such methods can be more elegant than the proposed PPR metric and the contrastive loss, since everything can be captured inside the model and no custom metric is necessary. While the author show that their proposed approach is beneficial for overall performance, such methods should be discussed or compared to, since they overlap in their goal of fusing structural and textual data with a major contribution of the paper.\n\n**Limited Novelty**\nThe paper does not advance any method or approach. It is an application of existing methods for creating a TFM model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! We respond to your concerns as below:\n\n&nbsp;\n> **W1: GNNs could offer a more elegant way to fuse structural knowledge into transformers than PPR metrics and contrastive objectives from scCello. GNN-fusion methods should be included as baselines.**\n\nWe would like to argue that the **cellular ontology graph has unique challenges to be directly modeled by GNNs effectively**, supported by following evidence:\n1. The ontology graph is **extremely sparse** with \\~2.7k nodes and \\~3.9k edges, and **faces long-distance issues**. For example, the average pairwise distance of \\~400 nodes (i.e., cell types) associated with our pre-training scRNA-seq data is 7.39, and the maximum distance is 18. \n2. Therefore, **it requires many layers (~7) of graph propagations of GNNs**, which may suffer from **over-smoothing** problems[a] for cell type representations. Even using a method like GraphFormer [b], which has only one layer of propagation in-between the transformers, the requirement of many layers of propagation leads to many layers of transformers. This would be **unaffordable given our computation resources**.\n\nWe also would like to emphasize our claim on scCello’s contribution: previous TFMs ignored cell type lineage relationships between cells and treated each cell as independent training samples. To tackle this, scCello proposes to fuse ontology structure priors into TFM pre-training. **We acknowledge that a careful design of the GNN fusion method may further improve performance.**  We would like to leave it for future work.\n\n[a] Rusch, et al. \"A survey on over-smoothing in graph neural networks.\" arXiv preprint arXiv:2303.10993 (2023).\n\n[b] Yang, et al. \"Graphformers: Gnn-nested transformers for representation learning on textual graph.\" Advances in Neural Information Processing Systems 34 (2021): 28798-28810.\n\n&nbsp;\n> **W2: The paper lacks novelty because it is an application of existing methods for creating a TFM model.**\n\nPrevious TFMs do not consider cell type lineage relationships between cells. **The key novelty of scCello is incorporating the cellular ontology graph into TFM pre-training** for better cell representation learning. **Incorporating an ontology graph is a non-trivial effort, since the ontology graph is sparse and faces long-distance issues** (see answer for W1). And in our method, the PPR metric and two inter- and intra- cellular contrastive objectives are simple and effective.\n\n\nscCello enjoys other novelties like:\n- **Simple but effective way to use cell type information**: Previous methods have proposed using cell type labels before, but treating each cell type independently with cell type classification loss has shortcomings. scCello uses pre-existing ontology graphs to determine the similarity between cell types, and this seems intuitively effective and empirically turns out to be true. \n- **New comprehensive evaluation for cell type clustering**: scCello evaluated not just on heldout in-distribution data, but also on unobserved cell types, tissues and donors. \n- **Novel algorithm for novel cell type classification**: scCello compares the similarity vector between a cell and prototype representations of each cell type to the similarity vector between a cell type and all other cell types, as derived from the ontology graph.\n\n&nbsp;\n> **Q1: How does the scaling of model sizes influence downstream performance?**\n\n**As discussed in Sec. 5, we leave scaling study as future work** because we lack the computational resources and scaling study is also not our focus in this paper. We focus on developing new algorithms for improving TFM pre-training strategy aiding various cell-related and gene-related downstream tasks.\n\n&nbsp;\n> **Q2: What’s the scaling complexity for training scCello with respect to the size of the ontology graph? When the number of negatives increases, the contrastive term in scCello faces a time bottleneck.**\n\n**The time complexity** of scCello includes that for PPR pre-calculation and TFM pre-training:\n\n- The pre-calculation needs $O(N*(I * (N + E)))$, where $N$ is the number of graph nodes, $E$ is the number of graph edges, and $I$ is the number of interactions. Specifically, for each target node, numerous iterations of graph message passing were run till convergence. Empirically, it was fast to accomplish and took less than several minutes to finish the ontology graph with 2.7k nodes. Since the Cell Ontology updates stay roughly on the scale of thousands of nodes [c], PPR calculation would still be fast in the coming future.\n\n- The TFM pre-training contains pairwise calculation for cells and their associated cell types within the batch, for the intra- and inter- cellular contrastive objectives. Its time complexity per batch for contrastive terms is roughly $O(B^2 * D)$, instead of $O(N^2 * D)$, where $B$ is the fixed batch size per GPU ($B=12$) and $D$ is the feature dimension. Therefore, the complexity is independent from N.\n\n**Therefore, the time complexity of scCello does not increase with the ontology graph size**.\n\n\n**The number of negatives** in scCello is fixed and equals the batch size minus 1 (i.e., $B-1$), following the common setting in contrastive learning papers [d,e,f]. Because the batch size $B$ in scCello is already maximized to fully utilize GPU memories, the number of negatives **cannot increase**.\n\n[c] Diehl, et al. \"The Cell Ontology 2016: enhanced content, modularization, and ontology interoperability.\" Journal of biomedical semantics 7 (2016): 1-10.\n\n[d] Chen, et al. \"A simple framework for contrastive learning of visual representations.\" International conference on machine learning. PMLR, 2020.\n\n[e] He, et al. \"Momentum contrast for unsupervised visual representation learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[f] Jaiswal, Ashish, et al. \"A survey on contrastive self-supervised learning.\" Technologies 9.1 (2020): 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9OyWJSIKLn", "reviewer": "Reviewer_BhVy", "summary": "This paper introduces a new transcriptome foundation model (scCello) to generate cellular representations from single-cell RNA-seq data. The key contribution is the integration of known cell type labels (previously annotated by CellxGene submitters) within two novel objectives. First, the authors introduce a cell-type coherence loss to minimize the distance between a cell's representation and the learned representation of its associated cell type. Second, the authors introduce an ontology relational alignment loss to ensure that the similarity between the representations of two cells matches the similarity of their corresponding cell types in an ontology graph. The authors benchmark their model against previously pre-trained transcriptome foundation models and traditional (non-foundation) models. They evaluate performance on tasks including zero-shot cell clustering, fine-tuned cell type classification, and marker gene discovery and find that scCello is state of the art on almost all tasks.", "questions": "1. The regularization loss (Eq. 2) was proposed to prevent class collapse. But it's unclear why this would help. In fact, it seems to be encouraging all samples of cell type c_{i} to have the same embedding: Linear(h_{c_{i}}).\n2. How did you train scVI for Table 1? Did you train it on your pre-training dataset, in-distribution dataset, and out-of-distribution datasets jointly? If not, for a fair comparison, it seems like you should. \n3. To avoid picking up housekeeping genes in the marker gene analysis, could you see if knocking out housekeeping genes leads to a representation change that differs from that induced by knocking out marker genes. I imagine knocking out housekeeping genes might shift the cells to a previously uncharted part of the latent space (dead cells), whereas knocking out marker genes might shift the cells to a different cell type.", "limitations": "The authors are forthcoming about the limitations of their model.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- *Conceptually simple but effective*: Utilizing cell type annotations, which are almost always available in published scRNA-seq datasets, to enhance the model's representations is smart and relatively straightforward. Other methods have proposed incorporating cell type labels before, but a cell type classification loss that treats each cell type as independent has shortcomings. The ontology relational alignment loss presented here which uses pre-existing ontology graphs to determine the similarity between cell types seems intuitively effective and empirically turns out to be too. \n\n- *Meaningful task selection*: The authors do a good job benchmarking their model on meaningful tasks. For example, they evaluate clustering performance not just on heldout IID data, but also in unobserved cell types, tissues, and donors. The model's improved performance on novel cell type classification and marker gene prediction suggests that it has captured biologically meaningful information. \n\n- *Appropriate baselines*: The authors do not just compare to existing transcriptome foundation models, which have been heavily criticized for not being competitive with traditional methods. \n\n- *Algorithm for novel cell type classification*: Their method of comparing the similarity vector between a cell and prototype representations of each cell type to the similarity vector between a cell type and all other cell types, as derived from the ontology graph, is clever.", "weaknesses": "- *Lacking analysis on ontology relational alignment loss*: While it seems intuitive that the ontology relational alignment loss is beneficial, the significant performance improvements are unexpected and probably warrant further analysis. For example, understanding which cell types show improved clustering performance could help provide some intuition. \n\n- *Missing some baselines*: Since the model assumes that cell type annotations are available, it should be compared against traditional methods that utilize these annotations (e.g. scANVI).\n\n- *Limited to 10x data*: The batch correction results, as is, are impressive, but it would be interesting to see if this method could handle data from different sequencing technologies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive review and constructive comments, which we carefully address below:\n\n&nbsp;\n\n> **W1: Provide example for cell types to analyze how the ontology relational alignment loss benefits clustering performance**\n\nTo compare scCello with its ablation excluding the relational alignment term, we visualize their cell representation distributions. Following App. D.7, we calculate prototype representations by averaging cell representations for each type on 10% of the pre-training data, and visualize them via tSNE. **The tSNE figures are in Figure R2 in the PDF file attached to the global rebuttal response.**\n\nAs shown in Figure R2, adding the relational alignment loss makes similar cell types clustered more closely and pushes dissimilar types farther apart. Therefore, relational alignment enables scCello to align better with biological intuitions, and produce more effective cell representations as evaluated by broad downstreams.\n\n&nbsp;\n\n> **W2: Add scANVI baseline for the fine-tuning setting of cell type identification**\n\nTable A: Extended results for scANVI for cell type identification under fine-tuning setting. The classification (Clf.) and clustering (Clst.) performances on the ID dataset $D^{id}$ are reported. \n\n| **Method** | **Clf. Acc** | **Clf. Macro F1** | **Clst. AvgBio** | \n|--- |---  |--- |--- |\n| scCello | 0.867 | 0.511 | 0.694|\n| scANVI |0.382 | 0.024 | 0.472|\n\n&nbsp;\n\nAs suggested, we add **scANVI** as baselines, implemented with scvi library [a]. As shown in Table A above, **scCello outperforms scANVI.**\n\n[a] Gayoso, et al. \"A Python library for probabilistic analysis of single-cell omics data.\" Nature biotechnology 40.2 (2022): 163-166.\n\n&nbsp;\n\n> **W3: Whether the impressive batch correction results can extend from 10x assay to different sequencing technologies?**\n\nIn principle, yes. Because scCello follows Geneformer’s **Rank Value Encoding (see App. C) to tokenize scRNA-seq data** into token sequences. As explained in Geneformer, this rank-based approach is **more robust against technical artifacts** than using raw numerical expressions, which can vary significantly across different assays.\n\nWe acknowledge that generalizability to heterogeneous platforms is an important aspect of a TFM and may require additional consideration to address technical biases from the various sequencing protocols. However, we believe that our cell-ontology-informed TFM would be less affected by batch effect. We leave this as future work. \n\n&nbsp;\n\n> **Q1: The regularization loss (Eq. 2) could cause class collapse**\n\nThanks for the constructive comment! We would like to explain that the regularization term would not lead to collapse empirically, primarily due to the masked gene prediction loss. This loss relies solely on the gene expression patterns in cells, making it hard for all cell representations $z_i$ to collapse into the linear transformation of their cell type representations $\\textrm{Linear}(h_{c_i})$. In addition, the linear layer and the cell type representations were not fixed and were updated every batch\n\nWe notice that this empirical justification was not mentioned in our paper, where our intuition for designing the regularization loss was to constrain the freedom of the optimization space.\nIn our next version, we’ll add this in Sec. 2.3 for better explanation to avoid confusions.\n\n&nbsp;\n\n> **Q2: Which dataset did we use to train scVI for Tab. 1?**\n\nFor all datasets in Tab. 1, including one ID and six OOD datasets, **scVI was trained on each of them individually.** We didn’t incorporate scCello’s pre-training dataset, because (1) scVI is not a foundation model and lacks a “pre-training stage”, and (2) even if we pre-train scVI, it does not have the capacity (in terms of both architectural design and parameter size) to be benefit from the pre-training on 22 millions cells.\n\n&nbsp;\n\n> **Q3: Whether the cell representation changes differ when knocking out housekeeping genes and marker genes?**\n\n\nFor a case study, we identified six common housekeeping genes from literature [b]: ACTB, GAPDH, HPRT1, SDHA, UBC, and YWHAZ, and used known marker genes for “B cell” from CellMarker2 and PanglaoDB.\n\nWe follow our approach in Sec. 4.4 and App. D.4 to perturb genes in-silico. For each cell of “B cell”, we calculate cell representations for (1) cell with no genes perturbed (2) cell with each of the six housekeeping genes perturbed (3) and cell with each of the B cell marker genes perturbed. To mitigate sample variance, we averaged these representations across all B cells for each gene. We visualize them via tSNE to compare the effects of perturbing housekeeping versus marker genes.\n\n\n**The tSNE figures are in Figure R3 in the PDF file attached to the global rebuttal response.** As shown in Figure R3, **knocking out housekeeping genes and marker genes leads to similar distribution shifts in cell representations.**\n\nAs discussed in Sec. 5 and App. D.4, housekeeping genes could be predicted as positives. This is an issue for all TFMs in our evaluation, which does not influence model benchmarking. Given that housekeeping genes are well-documented and relatively few (\\~400) compared to the extensive gene vocabulary (M = 25,424) in scCello, we can exclude them using external reference in our downstream pipeline. We leave this as future work.\nAdditionally, knocking out marker genes is unlikely to shift cells to a different cell type unless two cell types differ by only one gene. In such cases, masking that gene could cause a shift to the other cell type.\n\n[b] Silver et al. \"Selection of housekeeping genes for gene expression studies in the adult rat submandibular gland under normal, inflamed, atrophic and regenerative states.\" BMC molecular biology 9 (2008): 1-15.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RW7kCwrUdf", "reviewer": "Reviewer_bUQe", "summary": "This paper presents scCello, a single-cell, Cell-ontology guided Transcriptome Foundation Model (TFM) that leverages cell-type relationships from cell ontology graphs to enhance cell representation learning. scCello incorporates three levels of objectives during pre-training: masked gene prediction, cell-type coherence, and ontology alignment. This approach improves the model's generalization and transferability capabilities, leading to superior performance in tasks such as cell type identification, novel cell type classification, marker gene prediction, and cancer drug response prediction. scCello is also robust to batch effects and demonstrates high parameter efficiency compared to existing TFMs.", "questions": "1.Can the authors elaborate on the theoretical foundations underlying the effectiveness of integrating cell ontology into TFM pre-training? Specifically, how do the proposed objective functions contribute to learning more informative and robust cell representations?\n2.Can the authors provide a more comprehensive comparison of scCello's performance against specialized methods designed for downstream tasks like cell type identification, marker gene prediction, and cancer drug response prediction?", "limitations": "1.The cell ontology is constantly evolving, and the paper acknowledges that scCello currently requires retraining the entire model for any updates. This limitation hinders the model's adaptability and its ability to keep up with the latest knowledge in cell biology.\n2.The paper states that they aim to scale up the model size in future work. This indicates that the current model size might not be sufficient to fully capture the complexities of single-cell transcriptomic data, potentially limiting its performance on certain tasks or datasets.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.Integrating cell ontology graphs into TFM pre-training: This is the most innovative aspect of the paper. scCello improves model understanding of biological relationships between cells by incorporating cell-type relationships from cell ontology graphs into TFM pre-training, enhancing the model's generalization capability and transferability.\n2.Introducing a multi-level objective function: scCello employs a multi-level objective function that encompasses gene-level masked gene prediction, intra-cellular cell-type coherence, and inter-cellular ontology alignment. This multi-level approach enables the model to learn complex relationships between genes, cells, and cell types, leading to more precise and robust cell representations.\n3.Validating the model's effectiveness and advantages: The paper presents a comprehensive set of experiments demonstrating the effectiveness and superiority of the scCello model in tasks such as cell type identification, novel cell type classification, marker gene prediction, and cancer drug response prediction. Additionally, the paper highlights scCello's robustness in handling batch effects and its parameter efficiency, suggesting the model's strong potential for real-world applications.", "weaknesses": "1.Lack of Theoretical Justification: The paper heavily relies on empirical results to showcase scCello's effectiveness. However, it lacks a rigorous theoretical analysis of the proposed approach. A deeper theoretical understanding of the objective functions, their impact on learning, and how they contribute to generalization capability would significantly strengthen the paper.\n2.Limited Comparative Analysis: The paper primarily focuses on comparing scCello with existing TFMs but doesn't thoroughly analyze its performance against methods specifically designed for downstream tasks like cell type identification, marker gene prediction, and cancer drug response prediction. A more robust comparison with specialized methods would better illustrate scCello's strengths and highlight its relative advantages.\n3.The paper's current model size might be insufficient to fully capture the complexities of single-cell transcriptomic data. This limitation could hinder the model's ability to achieve optimal performance on tasks requiring deep understanding of complex biological processes, especially when dealing with large and diverse datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments! We respond to your concerns as below:\n\n&nbsp;\n> **W1: This paper lacks of theoretical justification for deeper understanding of objective’s impact on learning and model’s generalization capability**.\n\nOur paper focuses on developing a new algorithm to improve TFM pre-training, which facilitates practical cell-related and gene-related downstream tasks. To show the effectiveness of scCello, we focus on empirical evidence on real-world data. While we appreciate the **theoretical justification for model generalization capability like analysis of error bound, it is outside the scope of our current study.** We would like to leave that as future work.\n\n&nbsp;\n> **W2 & Q2: This paper lacks of non-TFM baselines specialized for downstream tasks like cell type identification, marker gene prediction, and cancer drug response prediction**.\n\nTable B: Extended results for scANVI and L1-LR for cell type identification under the fine-tuning setting. The classification (Clf.) and clustering (Clst.) performances on the ID dataset $D^{id}$ are reported. The clustering performance for L1-LR is not reported because L1-LR is a classification method and there are no internal cell representations for clustering.\n| **Method** | **Clf. Acc** | **Clf. Macro F1** | **Clst. AvgBio** | \n|--- |---  |--- |--- |\n| scCello | 0.867 | 0.511 | 0.694|\n| scANVI |0.382 | 0.024 | 0.472|\n| L1-LR| 0.747 | 0.491 | / |\n&nbsp;\n\nTable C: Extended results for DET for marker gene prediction on the two datasets $D^{mk}_1$ and $D^{mk}_2$ used in Sec. 4.4. AUROC is reported\n| **Method** | **$D^{mk}_1$** | **$D^{mk}_2$** | **Avg.** |\n|--- |---  |--- |--- |\n| scCello |0.756 | 0.729 | 0.743|\n| DET | 0.721 | 0.683 | 0.702 |\n\n&nbsp;\n\nFollowing the reviewer’s suggestion, we specify existing specialized baselines and **add three more baselines**:\n\n- **For the zero-shot setting of cell type identification**, we included non-TFM specialized methods like “Raw Data”, “Seurat”, “Harmony” and “scVI” in Sec. 4.2.1\n- **For the fine-tuning setting of cell type identification**, we add two more baselines: scANVI[a] implemented with scvi library, and L1-regularized Logistic Regression (L1-LR)[b] implemented with Scikit-Learn library. **As shown in Table B above, scCello outperforms both scANVI and L1-LR**.\n- **For marker gene prediction**, we add one traditional method Differential Expression Tests (DET)[c] to identify cell-type-specific marker genes, following Seurat’s [d] implementation. DET determines marker genes by comparing gene expression among cells of one cell type against cells from other cell types. **As shown in Table C above, scCello outperforms DET**.\n- **For cancer drug response prediction**, we included a specialized non-TFM method DeepCDR[e] in Sec. 4.5, which was the state-of-the-art traditional method.\n\n[a] Xu, et al. \"Probabilistic harmonization and annotation of single‐cell transcriptomics data with deep generative models.\" Molecular systems biology 2021\n\n[b] Vidaurre et al. \"A survey of L1 regression.\" International Statistical Review 2013\n\n[c] Soneson et al. \"Bias, robustness and scalability in single-cell differential expression analysis.\" Nature methods 2018\n\n[d] Hao et al. \"Dictionary learning for integrative, multimodal and scalable single-cell analysis.\" Nature biotechnology 2024\n\n[e] Liu, Qiao et al. \"DeepCDR: a hybrid graph convolutional network for predicting cancer drug response.\" Bioinformatics 2020\n\n&nbsp;\n> **W3 & L2: scCello’s current model size could be insufficient to handle the large and complex scRNA-seq, and achieve the optimal performance on downstreams**\n\n\nAs discussed in Sec. 5, we leave the scaling law study of scCello as future work due to limited computational resources and because scaling is not the focus of this paper. Our primary focus is on developing new algorithms to improve TFM pre-training strategies.\n\n&nbsp;\n> **Q1: Analyze the effectiveness of each component of scCello’s objectives to integrate cell ontology into TFM pre-training for better cell representation learning**\n\n**The general effectiveness** arises in **injecting prior structural knowledge** (i.e., the ontology graph) into the representation learning process. Knowledge injection can advance model generalization, as shown in previous studies[f,g].\n\n**For each loss component**, (1) the marker gene prediction loss captures dynamic gene co-expression patterns, to enrich the understanding of gene interactions; (2) the intra-cellular ontology coherence loss encourages cell representations of the same cell type to aggregate, prompting consistency between cells and their types; (3) the inter-cellular relational alignment loss guides the cell representation learning by injecting the cell-type lineage relationships derived from the cell ontology graph.\n\n[f] Martino et al. \"Knowledge injection to counter large language model (LLM) hallucination.\" European Semantic Web Conference. Cham: Springer Nature Switzerland 2023\n\n[g] Ovadia et al. \"Fine-tuning or retrieval? comparing knowledge injection in llms.\" arXiv preprint arXiv:2312.05934,2023\n\n&nbsp;\n> **L1: scCello currently requires retraining the entire model for any updates from the constantly evolving cell ontology, which hinders its adaptability**\n\nAs discussed in Sec. 5, we leave the support of dynamic growing ontology as future work. Notably, while Cell Ontology updates a few times a year[h], most of the ontology remains stable. Therefore, **we can fine-tune scCello with relatively much less computation costs**, by using recent advances in dynamic graph representation learning[i]. These methods, which handle evolving nodes, attributes, and edges over time, may effectively incorporate new changes in ontology graphs.\n\n[h] Diehl et al. \"The Cell Ontology 2016: enhanced content, modularization, and ontology interoperability.\" Journal of biomedical semantics 2016\n\n[i] Kazemi et al. \"Representation learning for dynamic graphs: A survey.\" Journal of Machine Learning Research 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dPvxCgJq0y", "reviewer": "Reviewer_E5ju", "summary": "The authors introduced a transcriptome foundation model (TFM)  named scCello to resolve the current problem of most of the TFMs they treat cells as independent samples and ignore the taxonomic relationships between cell types. By integrating cell ontology information as well as incorporating three key objectives in the pretraining framework: masked gene prediction, cell type coherence loss, and ontology alignment loss, the model can learn gene coexpression patterns, cell type-specific representations, and structural relationships between cell types. The authors performed a series of experiments to show that the model can predict the unseen cell types, integrate datasets across diverse batches, cluster cell types, predict the cancer response to the drug, and predict the marker genes.", "questions": "1. How did the method achieve a high score in the batch integration? In the loss function, there is no specific loss for batch correction. Could the authors give some explanations for this question?\n2. Can the model predict the novel marker genes?", "limitations": "The authors have addressed all limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The manuscript is well-written and well-structured, and it is easy to read. \n2. The authors effectively incorporate cell type ontology information at the pertaining stage, demonstrating in the comprehensive experiment results that this ontology information enhances the model's ability to learn cell type taxonomic relationships, thereby producing more biologically meaningful cell representations.", "weaknesses": "1. In the zero-shot cell type clustering experiment, it is not clear that the authors run the Louvain algorithm with default settings or the optimal setting on Seurat, Harmony, and scVI.\n2. In the marker gene prediction part, what is the difference between predicted marker genes based on scCello and the traditional marker genes based on the differential expression level? \n3. There is a paper (A Deep Dive into Single-Cell RNA Sequencing Foundation Models) indicating that the L1 logistic regression model achieves decent performance in cell type annotation task. Could the authors add the L1 logistic regression model for comparison in Table 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive review and insightful comments! We respond to your concerns as below:\n\n&nbsp;\n> **W1: In zero-shot cell type clustering, do authors use the optimal setting or the default setting for the Louvain Algorithm for Seurat, Harmony and scVI?**\n\nIn zero-shot cell type clustering, we **used the optimal setting for the resolution** hyper-parameter in the Louvain algorithm across both TFM and non-TFM baselines, including Seurat, Harmony, and scVI. Notably, for all tasks involving the Louvain algorithm to calculate AvgBio or AvgBatch, we used the optimal resolution. \n\n&nbsp;\n> **W2: In marker gene prediction, how does the performance of scCello compare to the traditional marker genes prediction method based on the differential expression level?**\n\nTable D: Extended results for Differential Expression Tests (DET) for marker gene prediction on the two datasets $D^{mk}_1$ and $D^{mk}_2$ used in Sec. 4.4. AUROC is reported.\n\n| **Method** | **$D^{mk}_1$** | **$D^{mk}_2$** | **Avg.** |\n|--- |---  |--- |--- |\n| scCello |0.756 | 0.729 | 0.743|\n| DET | 0.721 | 0.683 | 0.702 |\n\n&nbsp;\n\nAs suggested, we add the Differential Expression Tests (DET)[b] as a traditional baseline, implemented with Seurat [c]. DET determines marker genes by comparing gene expression among cells of one cell type against cells from other cell types. \n\nFor the metric threshold, we follow conventions [c] to set `p_val_adj` < 1 and `avg_log2FC` > 0, where `p_val_adj` refers to the adjusted p-value, and `avg_log2FC` refers to the log fold-change of the average expression between the two groups. \n\n**As shown in the Table D above, scCello outperforms DET.**\n\n[b] Soneson et al. \"Bias, robustness and scalability in single-cell differential expression analysis.\" Nature methods 2018\n\n[c] Hao et al. \"Dictionary learning for integrative, multimodal and scalable single-cell analysis.\" Nature biotechnology 2024\n\n&nbsp;\n> **W3: For the fine-tuning setting of cell type identification, could authors add the L1 Logistic Regression (L1-LR) model for baseline comparison?**\n\nTable E: Extended results for L1-LR for cell type identification under fine-tuning setting. The classification (Clf.) and clustering (Clst.) performances on the ID dataset $D^{id}$ are reported. The clustering performance for the L1-LR model is not reported because L1-LR is a classification method and there are no internal cell representations for clustering.\n\n| **Method** | **Clf. Acc** | **Clf. Macro F1** | **Clst. AvgBio** | \n|--- |---  |--- |--- |\n| scCello | 0.867 | 0.511 | 0.694|\n| L1-LR| 0.747 | 0.491 | / |\n\n&nbsp;\n\n**Yes. We add the L1-LR model as a baseline**, following the implementation from the above suggested paper [d]. **As shown in the Table E above, scCello outperforms L1-LR.** \n\nInterestingly, compared to Tab. 2 in paper, L1-LR indeed outperforms all other TFM baselines on Macro-F1, which agrees with the previous finding in [d].\n\n[d] Boiarsky et al. \"A deep dive into single-cell RNA sequencing foundation models.\" bioRxiv 2023\n\n&nbsp;\n> **Q1: How did scCello achieve a high score in the batch integration, without explicitly considering batch correction objectives?**\n\nThe primary reason is that **scCello injects cell type lineage structural priors into its TFM pre-training, enhancing model generalization and robustness.** This allows scCello to better capture biological signals in scRNA-seq data, improving cell representation learning and mitigating batch biases.\n\nAdditionally, scCello employs Geneformer’s [e] **Rank Value Encoding** approach (see App. C) to tokenize scRNA-seq data into ordered tokens. This rank-based method offers better robustness against technical artifacts compared to using raw numerical expressions, which can vary significantly across different assays.\nLastly, we focused on scRNA-seq data from 10x assays, following scTab’s [f] preprocessing steps. **This minimized technical biases in pre-training.** We plan to explore training on more heterogeneous data from diverse scRNA-seq platforms in future work.\n\n[e] Theodoris et al. \"Transfer learning enables predictions in network biology.\" Nature 618.7965,2023\n\n[f] Fischer et al. \"Scaling cross-tissue single-cell annotation models.\" bioRxiv 2023\n\n&nbsp;\n> **Q2: Can the model predict the novel marker genes?**\n\n**Yes, we can use the following approach**, where the calculation of cell representation difference via in-silico gene perturbation is similar to the protocol used in our zero-shot marker gene prediction (see App. D.4). \n\n1. We used the same two datasets (GSE96583 and GSE130148) in Sec. 4.4 for marker gene prediction, where marker gene labels were retrieved from CellMarker2 and PanglaoDB.\n2. For each cell, we calculated the change in cell representations after removing each gene in-silico and selected the top 10% genes with the largest changes, excluding the known marker genes.\n3. For each cell type, we identified the 10 most frequent genes among the top 10% across all cells, to obtain 10 candidate novel marker genes.\n4. To ensure specificity, we removed genes present in more than one cell type.\n\n\nAs a case study, we followed the above steps to find novel marker genes for two cell types:\n- For cell type “CD14+ Monocytes”, two genes were found: FOS and LGALS2. FOS is typically expressed in response to stress signals, cytokines, and growth factors [g]; LGALS2 is involved in modulating immune responses and inflammatory processes in monocytes [h]. Both are plausible marker gene candidates.\n- For cell type “Megakaryocytes”, five genes were found: GNG11, TUBB1, H2AC6, CAVIN2, CLU. GNG11 is confirmed as marker genes in literature, TUBB1 is a likely marker, while H2AC6, CAVIN2, and CLU require further investigation.\n\n[g] Angel, et al. The role of Jun, Fos and the AP-1 complex in cell-proliferation and transformation. Biochimica et Biophysica Acta (BBA)-Reviews on Cancer, 1072(2-3), 129-157.\n\n[h] Kim, et al. Glycobiology of innate immunology. Berlin/Heidelberg, Germany: Springer, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ApD1kmK9Au", "reviewer": "Reviewer_hRwa", "summary": "This paper introduces scCello, a Transcriptome Foundation Model (TFM) which incorporates prior information from cell ontology graphs denoting the relationships between different cell types in order to guide cell representations. The pretraining objective in the scCello framework encapsulates masked gene expression prediction, a supervised contrastive objective for aligning cell representations to cell ontology terms, and relational alignment of similar cell types, which helps guide representation learning during pretraining to reflect  taxonomic relationships between cells. The foundation model, scCello, is pretrained on 22 million cells from the CellxGene public data repository, and demonstrates improved performance over other single-cell foundation models in downstream tasks, including cell type clustering, cell type identification, marker gene prediction, and batch integration.", "questions": "For the overall pretraining objective in equation 5, more details could be added about how the four loss terms are weighted relative to one another. Are their coefficients to weigh the different loss terms? If not, do the magnitudes of the different loss terms differ significantly?\n\nThe authors quantify ontology relationships in the cell ontology graph by using structural similarity, and use the Personalized PageRank (PPR) algorithm to estimate pairwise node similarities for cell types. Examples of cell types associated with varying levels of structural similarity are also presented in Table 7. It would be helpful if the authors provided a motivation for going with a structural similarity measure such as PPR score, rather than a simpler metric such as the distance of two nodes in the cell ontology graph.", "limitations": "The authors have adequately addressed the limitations and potential negative impacts of their work, and agree to do a code implementation release upon acceptance of the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This work presents a novel foundation model for single-cell transcriptomics, scCello, that incorporates prior information about cell ontology in order to guide representation learning over a large pretraining corpus. The methodology and motivation for the paper is clear, and the work represents a significant step towards training foundational models that incorporate prior information, which subsequent works may build and improve upon.\n\nFor pretraining, the authors introduce a multi-component loss function, encapsulating: (i) masked gene expression prediction, (ii) a supervised contrastive objective to align representations to of cells with similar cell ontology terms, and (iii) a relational alignment objective which enforces cell representations to follow the structural similarity of their cell types in the cell ontology graph. For quantifying cell type similarity, the Personalized PageRank (PPR) algorithm is used to derive structural similarity of pairwise cell types in the cell ontology graph, which is then used to guide representation learning.\n\nThe single-cell RNA seuqencing data preprocessing methodology seems sound, and follows common techniques for data preprocessing used by other foundation models. The authors also include a hyperparameter and compute resource comparison of scCello against other established single-cell foundational models - Geneformer, scGPT, scTAB, UCE.\n\nOverall, the paper is well-written, motivated well, and clearly explained. Adequate details are provided for the loss objectives and methodology, and additional dataset acquisition and preprocessing details are provided in the appendix.", "weaknesses": "The authors mention that cell type ontology identifiers were obtained for the 22 million cell pretraining dataset from the CellxGene database, to enable mapping between individual cells and cell ontology. While this allows for additional priors from cell ontology graphs to be used during pretraining of scCello, it also necessitates labeled data during pretraining, which may limit the scalability of scCello in terms of available pretraining data compared to other methods which do not require annotated pretraining single-cell data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for positive reviews and invaluable suggestions  We respond to your questions as below:\n\n&nbsp;\n> **W1: Although cell type ontology labels allow the use of cellular ontology structural priors to guide TFM pre-training, the need for labeled data may limit scCello's scalability in fully utilizing unannotated scRNA-seq data.**\n\nThanks for bringing this up. We would like to argue that **scCello can also leverage unannotated scRNA-seq data**, by only using its masked gene prediction objective, and neglecting the intra- and inter- cell-level losses that require cell type labels. This reduces scCello to Geneformer-like TFMs. Therefore, scCello can be trained on partially labeled scRNA-seq data. We will explore this scenario in future work.\n\n&nbsp;\n> **Q1: How are the four loss terms in Eqn. 5 weighted relative to one another? And do the magnitude of these loss terms differ significantly?**\n\nThe weights between the four loss terms in Eqn. 5 **are simply all ones.** And the magnitude of these loss terms **do not differ significantly.**\n\nFor visualization, **four training loss figures are included in Figure R1 in the PDF file attached to the global rebuttal response.** Specifically, the training process takes 40k steps as detailed in Sec. 4.1, and the magnitudes of losses are roughly as follows:\n- The masked gene prediction loss $\\mathcal{L}_{\\textrm{MGP}}$ starts from 19.55 (1 step), achieves 8.09 (2k steps), and ends at 3.57 (40k steps)\n- The intra-cellular ontology coherence loss $\\mathcal{L}_{\\textrm{Intra}}$ starts from 13.14 (1 step), achieves 1.29 (2k steps), and ends at 0.39 (40k steps).\n- The inter-cellular relational alignment loss $\\mathcal{L}_{\\textrm{Inter}}$ starts from\n1.99 (1 step), achieves 0.57 (2k steps), and ends at 0.26 (40k steps).\n- The regularization term starts from 0.51 (1 step), achieves 0.03 (2k steps), and ends at 0.02 (40k steps).\n\nIn summary, all four losses weighted with all ones do not differ significantly in magnitudes, and can be optimized effectively throughout the learning process as shown in Figure R1. \n\n&nbsp;\n> **Q2: Can authors provide a motivation for going with a structural similarity measure such as PPR score, rather than a simpler metric such as node distance.**\n\nThe PPR metric considers structural topology between node pairs. Intuitively, for a target node and a query node, **PPR performs graph propagations to gather information from multiple paths between this pair of nodes via random walking** (see App. A for algorithm details). In contrast,  simple metrics like **pairwise distance only consider the shortest paths.** Therefore, PPR inherently is more informative with a stronger capability to capture subgraph structures, and may be more generalizable compared to simpler metric like node distance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to appreciate all reviewers for your constructive suggestions and valuable comments on our paper!

Here is a brief summary of important points from all reviewers:

- **Performance comparison with more non-TFM traditional methods specialized for downstreams (Review BhVy, bUQe, E5ju)**: for the fine-tuning setting of cell type identification, we evaluate two more baselines scANVI and L1 Logistic Regression (L1-LR), and scCello outperforms both of them. For the marker gene prediction task, we incorporate Differential Expression Tests (DET) for comparison and scCello also outperforms it.

We also would like to highlight our contributions:

- **Innovation in integrating cell ontology graphs as priors**: scCello enhances the model’s understanding of biologically important cell type lineage relationships between cells using cell ontology graphs, improving its generalization and transferability
- **Introduction of a multi-level objective function**: scCello employs a simple yet effective multi-level objective pre-training strategy. It encompasses gene-level prediction, intra-cellular ontology coherence, and inter-cellular relational alignment, enabling the model to learn complex relationships between genes, cells, and cell types. This leads to more precise and robust cell representations for scCello.
- **Model effectiveness and comprehensive evaluation**: We conducted  a comprehensive set of experiments, and evaluated scCello’s effectiveness and superiority against various non-TFM and TFM baselines across broad downstream tasks. scCello is also robust to batch effects and parameter efficient, highlighting its strong potential to be applied in real-world applications. 

**More detailed per-question feedback is presented below**. We use prefixes **“W”, “Q”, and “L” to indicate Weakness, Question, and Limitation** raised in reviewers’ comments.  We hope these feedbacks can address your concerns, and we welcome any further questions.

Meta-review: This paper presents an algorithm, scCello, designed to enhance the pre-training of Transcriptome Foundation Models (TFM) by incorporating taxonomic relationships between cell types within cell ontology graphs. This approach aims to improve performance in cell-related and gene-related downstream tasks. Extensive experiments have been conducted to evaluate the proposed scCello in comparison with other TFM models across several real-world biological and drug response tasks.

All reviewers agree that the proposed method is practically useful, although some do not consider it novel theoretically. Nevertheless, all reviewers are convinced that the experiments are extensive and adequately support the narrative of the paper. Most of the questions raised concerned technical and experimental details, and reviewers were satisfied with the authors' responses; no reviewer lowered their rating, and one even raised theirs after the rebuttal.

This work is significant in applications and is worthy of being reported.
Decision: Accept (spotlight)