sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
Authors: Hongru Hu
Shuwen Zhang
Yongin Choi
Venkat S. Malladi
Gerald Quon
Venue: ICML 2025 poster (2025)
Keywords: Representation Learning
Large Language Model
Batch Effect Correction
Gene Module Discovery
Primary area: applications->health_medicine

Abstract: Single-cell RNA sequencing (scRNA-seq) enables high-resolution exploration of cellular diversity and gene regulation, yet analyzing such data remains challenging due to technical and methodological limitations. Existing task-specific deep generative models like Variational Auto-Encoder (VAE) and its variants struggle to incorporate external biological knowledge, while transformer-based foundational large Language Models (LLMs or large LaMs) face limitations in computational cost and applicability to tabular gene expression data. Here, we introduce sciLaMA (single-cell interpretable Language Model Adapter), a novel representation learning framework that bridges these gaps by integrating static gene embeddings from multimodal LaMs with scRNA-seq tabular data through a paired-VAE architecture. Our approach generates context-aware representations for both cells and genes and outperforms state-of-the-art methods in key single-cell downstream tasks, including batch effect correction, cell clustering, and cell-state-specific gene marker and module identification, while maintaining computational efficiency. sciLaMA offers a computationally efficient, unified framework for comprehensive single-cell data analysis and biologically interpretable gene module discovery.

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
Hongru Hu
Shuwen Zhang? Yongin Choi 12
Venkat S. Malladi
Gerald Quon" 12
Abstract
1. Introduction
Single-cell RNA sequencing (scRNA-seq) has revolution- ized studies of cellular heterogeneity and transcriptome dy- namics by providing gene expression profiles at single-cell resolution. Deep generative models, particularly Variational Autoencoders (VAE) (Kingma & Welling, 2014) and its vari- ants, have become widely used for analyzing scRNA-seq data, which enable dimensionality reduction and representa- tion learning by projecting cells from high-dimensional gene spaces to lower-dimensional embedding spaces (Grønbech et al., 2020; Lopez et al., 2018). These cell embeddings facilitate downstream tasks such as cell clustering, trajec- tory inference, and differential expression analysis (Chen et al., 2021; Kana et al., 2023; Yan et al., 2023). VAE's nonlinear representation capabilities allow them to effec- tively model complex cellular landscapes, making them well-suited for tabular data like gene expression matrices. However, scRNA-seq analysis remains challenging due to technical noise, sparse measurements, and batch effects, which often obscure true biological signals (Lähnemann et al., 2020). Incorporating external prior knowledge of genes, such as their functional annotations or molecular se- quence data, has the potential mitigate these challenges. However, the representation of input gene expression data as fixed-length vectors in traditional VAEs such as SC tools (Lopez et al., 2018) is not directly compatible with the different representations of prior gene knowledge, such as variable-length molecular sequences or text descriptions. This prevents prior gene knowledge from being directly incorporated into traditional VAE architectures.
Single-cell RNA sequencing (scRNA-seq) en- ables high-resolution exploration of cellular di- versity and gene regulation, yet analyzing such data remains challenging due to technical and methodological limitations. Existing task-specific deep generative models like Variational Auto- Encoder (VAE) and its variants struggle to in- corporate external biological knowledge, while cansformer-based foundational large Language Models (LLMs or large LaMs) face limitations in computational cost and applicability to tabular gene expression data. Here, we introduce seiL- aMA (single-cell interpretable Language Model Adapter), a novel representation learning frame- work that bridges these gaps by integrating static gene embeddings from multimodal LLMs with scRNA-seq tabular data through a paired-VAN VAE architecture. Our approach generates context- aware representations for both cells and genes and outperforms state-of-the-art methods in key single-cell downstream tasks, including batch effect correction, cell clustering, and cell-state- specific gene marker and module identification, while maintaining computational efficiency. scil.- aMA offers a computationally efficient, unified framework for comprehensive single-cell data analysis and biologically interpretable gene mod- ule discovery. Source code available at https: //git tuittuub..... www.cicmccc..c/eeeeee..c. croooftt /ssiLaMM
Large Language Models (LLMs), the other hand, have emerged as powerful tools for learning gene representations from various sources, including literature- scriptions (Chen & Zou, 2024; Liu et al., 2023), molecular sequences (Elnaggar et al., 2022; Lin et al., 2023), and large- scale expression atlases (Cui al., 2024: Theodoris et al., 2023). These models encode sequential data through tok- enization and transformer architectures create static gene embeddings that capture rich biological information. How- ever, LLMs also face challenges: they are computationally expensive to train and inherently less suited for processing tabular data such as gene expression matrices, where VAEs
textual de-
"Equal contribution 'Deparmeent of Molecular and Cellular Biology, University of California, Davis, CA USA 'Genome Center, University California, Davis, CA USA 'Department of Quantitative Health Sciences, Mayo Clinic, Rochester, MN USA 'Health Futures, Microsoft Research, Redmond, WA USA. Correspondence Hongru Hu <hrhu@ <hrhu@ucdavis.er saedavis.edu>.. Venkat S. Malladi <vmalladi microsoft.com>. Gerald Quon <gquon@ucdavis.ddus udu'.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
demonstrate superior performance (Kedzierska et al., 2023). To bridge the complementary strengths of VAEs and LLMs, we propose sciLaMA (single cell interpretable Language Model Adapter), a novel representation learning framework that extends the siVAE (Choi et al., 2023) architecture to integrate precomputed static gene embeddings from pre- trained multimodal LLMs with scRNA-seq tabular data. By combining the representation power of VAEs with the adaptable and knowledge-rich embeddings of LLMs, our ap- proach projects static gene information into context-aware representations by aligning each dimension of gene and cell latent space within the unified paired- -AE framework (Sec- tion 3). This approach presents a unified framework that improves over state of the art methods in single-cell analysis in three tasks: (I) cell representation learning and batch effect correction, (2) gene expression imputation, and (3) discovery of biologically meaningful gene modules and cell- state-specific regulatory dynamics, all while maintaining computational efficiency.
to the representational challenges discussed above. Meth- ods such as GLUE (Cao & Gao, 2022) incorporate external information about regulatory interactions among features in the form of feature variables, however, such approaches struggle to utilize information such as molecular sequences or natural language descriptions of genes.
Single-cell foundation language models. Transformer- based large language models (LLMs) have recently been applied for single-cell data analysis. Unlike VAE-based methods, which treat scRNA-seq data as a cell-by-gene matrix, models such as scGPT (Cui et al., 2024) represent expression profiles as sequences of tokens, drawing similar- ities to natural language and demonstrating a novel way to represent single-cell data. However, despite their promise, single-cell LLMs exhibit certain limitations. Their perfor- mance in zero-shot scenarios is often unreliable, and fine- tuning them requires extensive computational resources and technical expertise compared to task- specific small models (Kedzierska et al., 2023). These drawbacks emphasize the need for approaches that are computationally efficient and capable of bridging foundational knowledge with real-world single-cell tasks.
Contributions: (1) We introduce a novel framework that in- corporates external gene knowledge from pretrained LLMs with scRNA-seq data, facilitating context-aware cell and gene representation learning. (2) We demonstrate that our approach reduces computational requirements while im- proving performance compared to existing tfat-off-the-ar methods across various single-cell tasks.
Applications of static gene embeddings in single-cell anal- ysis. Gene embeddings derived from non-single-cell biolog- ical data modalities can complement information derived from single-cell data analysis. For instance, precomputed gene embeddings from protein language models (PLMs), such as ESM and ProfTrans (EInaggar et al., 2022: Lin et al., 2023), capture gene molecular properties and have been applied in frameworks like SATURN (Rosen al., 2024) to identify conserved master regulatory genes across species. Similarly, models such as GenePT (Chen & Zou, 2024) and scELMo (Liu et al., 2023) use embeddings derived from textual descriptions of gene functions and biologi- cal pathways via natural language models such as OpenAl text-embedding model (OpenAI, 2022). These applications demonstrate the feasibility of incorporating external static gene embeddings from various modalities into single-cell analysis frameworks. By integrating such embeddings, re- searchers are able to improve the robustness of single-cell analysis. facilitate gene module characterization, and un- cover regulatory dynamics.
2. Related work
Deep generative approaches for single cell analysis. Deep generative models, particularly those based on variational autoencoders (VAEs), have advanced single-cell RNA se- quencing (scRNA-seq) analysis. Methods such as scVI- tools learn low-dimensional cell embeddings for cell-centric tasks such as visualization, clustering, and batch correction. Researchers have also further utilized feature attribution techniques identify important genes in specific cell pop- ulations and infer gene modules (Janizek et al., 2023) by leveraging the learned cell embeddings. However, these ap- proaches primarily focus on cell representations without in- ferring gene representations, and pipelines leveraging other tools are needed for gene-centric analyses such as marker identification and gene network discovery. To address this limitation, siVAE (Choi et al., 2023) introduced unified framework for learning both cell and gene representations, enabling direct gene-centric analyses using the gene repre- sentations and therefore eliminating the need for explicit gene module inference via external tools. However, VAB gene representation learning involves training an encoder whose number of input nodes scales with the number of cells, thus limiting its applications to large datasets. More- over, SI, siVAE, and most other VAE methods do not integrate external knowledge into scRNA-seq analysis due
3. Methods
Conceptually, scil.aMA is an adapter framework that inte- grates pretrained gene embeddings from LLMs of different modalities, and tailors them for downstream single-cell anal- yses. Instead of learning gene representations de novo, seiL- aMA adapts and contextualizes these precomputed static gene embeddings by incorporating context specific cell- data (cell-by-gene expression matrix). In this section, we detail the technical components of the sciLaMA framework
2

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
and its application to single-cell analysis.
output h h
gene gene
G (gj)
3.1. Input data processing and notation
The sciLaMM framework requires two inputs: (1) A set of gene expression inputs {c:]<= representing scRNA-seq data for N cells (scaled log- normallaed expression) drawn from a specific cell population. Each of the N cell vectors e₁ has M measurements corresponding to individual genes. (2) Static gene embeddings {g, 1 derived from a single pre- trained language model (LaM). These embeddings provide D-dimensional representations of M genes, capturing their properties derived from external prior knowledge, where the number D depends on the embedding dimensionality of the specific LLM.
]
gene
+
exp (0.5 log (o gene )2)
(
(3)
h gome
(4)
3.2.3. SCILAMA RECONSTRUCTION OUTPUT
Similar to the siVAE framework (Choi et al., 2023), the output of sciLaMA is the reconstruction of the single cell expression data for gene j in cell i, denoted Cı,j. via combining the respective cell and gene decoder outputs h and h gene.
25 = (n.aal)² 2'
x h + by
(5)
3.2. sciLaMA architecture
3.3. Optimization
sciLaaMA is based paired encoder-decoder design, in- spired by siVAE (Choi et al., 2023), interpretable deep generative model that jointly learns sample (cell) and fea- ture (gene) embeddings using a paired VAE design. si only uses scRNA-seq data to learn both sets of embeddings, whereas sciLaMA uses external data to inform gene embed- dings. sciLaMA consists of two encoder-decoders: one for cells and one for genes (Figure la).
The optimization of the sciLaMA framework involves step- wise training procedure designed for representation learning of both cells and genes (Appendix B), and the training objec- tives follow the evidence lower bound (ELBO) framework, combining reconstruction accuracy and regularization via Kullback-Leibler (KL) divergence.
Step 1: Pretraining the Cell Encoder and Decoder: We first pretrain the weights of the cell encoder and decoder (crell and الsel respectively) by treating the encoder-decoder as a VAE, where the objective function focuses on matching cell decoder outputs hç to the original expression vectors e; via a linear transformation with parameters W kl and b. The loss function Leell for this step is defined
3.2.1. CELL ENCODER AND DECODER
The cell encoder f (+) projects each cell i's expression
profile C, represented as an M -ddimensional gene expres- sion vector, to parameters of a K --dimensional variational posterior distribution with mean fu E æ* and variance (2 E IR*, A latent embedding 2, sampled via the reparameterization trick:
x W +b goeltroonn = (c: acellrocon)
apellrecon = (h. T
(6)
)" T (c: - -2 2gellrocon)
(7)
(H
cell) + soell (c:)
Leell Σ Cpellrecon + 3 KL (N(z uل:") off""l
(0)
20001 uusal + t exp (0.5 log(o,(")))
(1)
cell = guial (2 (2çe!l)
(2)
(8)
where 33 represents the weight of the KL divergence term in VAEs, and is tuned to prioritize accurate reconstruction during the early stages of training.
where ~ NNO I) and @ denotes wism multi- plication. Iilll (.) represents the cell decoder without a conventional final linear transformation layer, and outputs hgell E R for cell i.
Pretraining the Gene Encoder and Decoder: Once the cell encoder and decoder are pretrained, its pa- rameters (g woell Weell, and b) are frozen, and we then pretrain the parameters (ōæeme, eigene) of the gene encoder f (.) and decoder g (-), respectively. The loss func- tion Lgene for this step is defined as:
Step 2:
3.2.2. GENE ENCODER AND DECODER
Similarly, the gene encoder fgene maps each gene j's ex- ternal static embedding 9j E DR derived from a pretrained LLM,, into the contextual embedding space by predicting the parameters of K K-dimensional variational posterior distri- bution with mean 1 505000 E R and variance (a (0 )² E R$
Crecon (c. -2,)² (c. - -a,)
(9)
Egene
rrecon + KLKK (N(z) gene 11 1 o0
gene (I, I))
The leve- decoder g geaene (.) is then used to produce
(10)
3

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
scILaMA framework overview
b
Nuttmodal Languagge Model based pens lg
externll LaM deried gare
Kcftk eeq
ground tuthh expression
faried
Singgle call expression
5 grow IIll Mil II D (LaM untoudding cirn)
Ow! I 1 N (gores))
C alggmment
MARCEEH DELDDY IREHHURL COEMERRG
Moledular sequence
(ENISAALSALSOMMNN LUSSMIN MPVUIT
Deea 94
L
Protein
K Lanantt cITO
K CITY 1
Larguage
510044 is caldum-chding
Text dessrrppaa
pronatt invotved oel mot
Downstream applications
Wnitp- Unstp-omweir and NF--3
pathways, promroting call micrit on and trusston...
/ood hotden dirj
C. [Jee hrsdden cim)
geneVAE
ColIVAE
betzh offic
geew repoession imputation
matker idteritooiionn
implict gane diacovery
BIBI
reconstrutedd gene espressioon in oell
Figure 1. sc overview. (a) Diagram of the scil LMA framework, which utilizes static gene emheddings generated from multimodal language models and employs paired encoder-decoders for both genes and cells. (b) Visualizations of cell and gene latent and last- hast-hidden spaces and their operations for different components of the loss functions. (c) Illustrations of downstream applications using SciLaMA.
Note that unlike the reconstruction term 2 Coellreeon from pre- vious step (Equation (7)), this loss function operates the outputs of the last hidden layers of both cell and gene decoders (Equation (5)). Because the inputs to the gene encoder are the prior L.LM-defined gene embeddings Пj- and the output is reconstruction of the gene expression mea- surements c,, this pretraining serves to help adapt the LLM embeddings to the current (gene expression) context.
3.4. Inference and Embedding Extraction
After training the sciLaMA framework, the learned cell and gene embeddings can be extracted for downstream analyses. Given the trained encoders f (+) and faee (-), they can
be used to project cell expression profile e(l)) gene embedding g0 into the cell (z(1)) gene (2(2)) latent space for downstream visualization or analysis.
Step 3: Joint Optimization of sciLaMA: In the final step, all parameters of the sciLaMA framework are optimized to improve the reconstruction quality of the expression matrix. The loss function LAiLLMMA for this step is:
(4"))00)) + fgall (c
(14)
20 ~ NN(( (1L")
(15)
(n )(2) seeme (0)
(16)
aiggmment
t.j
(2) T
x
2
gene + by
(11)
N ( 201)
(17)
aaligmment
Ci
caligument) C
T
ci 2
alligament
(12)
4. Experiments
The experiments evaluating sciLaMA are designed to assess its performance in single-cell analysis at both cell- and gene- level tasks. For cell-level tasks, sciLaMA is assessed by evaluating its capacity to generate cell embeddings that si- multaneously preserve biological signals and remove batch effects, with performance measured by (1) cell clustering an- notation accuracy, (2) cell type separation precision, and (3) the effectiveness of batch mixing. For gene-level tasks, seiL- aMA is evaluated on its ability to impute gene expression, identify gene markers, infer developmental trajectories and discover temporal dynamic gene modules (Figure Ic). De- tailed methodologies are listed in the Appendices C and D.
LuCLLMA
= Σ Crooon + 7. 2. allggmment
+B- KL (N((2²0)) cell
cell [N I))
+B- KL-KL (2( (000 gene gone [l, 1)
(13)
where C. Calignment is a reconstruction based regularization term that encourages alignment between the latent spaces of cells (2şell) and genes (2)"h( by enforeing that the linear product of the embeddings approximates the original ex- pression value of gene j in cell i (ct,). This term, inspired by siV, serves the interpretability term, ensuring that individual dimensions of the cell and gene embeddings eellll and = correspond meaningfully to each other. is a scalar weight (default- = 0.05) that determines the influence of C. calignment term on the overall loss function. A small value prevents it from dominating the optimization process.
4.1. Prior Knowledge Improves Cell Representation Learning
We first evaluated cell-level tasks because gene-level analy- sis tasks are largely cell state-specific, and therefore rely on
4

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
Finbadd Becttakk
Table 1. Cell representation learning and integration performance on human pancreatic datasets. Adjusted Rand Index (ARI) and Normalized Mutual Information (NMI for cluster annotation ac- curacy; Average Silhouette Width (ASW) for cell type separation; batchASW and graph integration local inverse Simpson's 's Index
sc
An
000-
(iLISI) for batch mixing quality.
Cell Tyok Duutenng-nnddre
Methods
ARI 1
NMI 1
ASW
batchASW 1
ILISI
Sezaration
sciLaMA (avg.)
0.522
0.745
0.535
0.865
0.238
sciLaMA (s...)
0.436
0.698
0.539
0.832
0.210
scGPT fine-tuned
0.483
0.704
0.650
0.736
0.074
SCOPT
scVI-batch
0.447
0.718
0.499
0.744
0.115
Co Type ACTST
scVI-raw
0.297
0.570
0.453
0.621
0.030
Bulc
scGPT zero-shot
0.321
0.487
0.442
0,588
0,005
CellPLM rert
0.330
0.516
0.421
0.492
1.11c 16
GenePT-w
0.022
0.079
0.192
0.553
0.121
work is a general, powerful framework for tackling cell-level tasks. For cell type clustering and annotation, sciLaMA achieved an average adjusted Rand index (ARI) of 0.522 and normalized mutual information (NMI) of 0.745, outper- forming VI (with batch variable consideration) by 16.78% and 3. 3.76%, respectively, and fine-tuned scGPT by 8.07% and 5.82%. Additionally, its ARI and NMI values were approximately 1.5 times higher than those of the best zero- shot foundation models, showcasing its ability to generate well-defined cell clusters aligned with cell type annotations from the original studies. In cell type separation, sciLaMA achieved an average silhouette width (ASW) of 0.535 and graph cell type local inverse Simpson's 's index (cI.ISI) of 0.9935, indicating precise separation of cell types with pre- served biological variation. Furthermore, for batch effect correction, sciLaMA achieved the highest batch-AST ASW of 0.865 and a graph integration- LISI (iLISI) of 0.238, surpass- ing the next-best models by 16.26% and 96.69%, respec- tively. These results collectively highlight scil LaM's robust ability to integrate cells across batches while maintaining accurate cell type representations and biological relevance.
Figure 2. Robust cell representation learning and integration with sciLaMA. (a) Quantitative performance comparison of mod- els based on sciLaMA against other methods in preserving biolog- ical variance (blue and orange metrics) and removing batch effects (green metrics). (b-c) Scatter plot directly comparing sciLaMA- GPT (y-axis) to sciLaMA (s.i.) (x-axis, b) and fine-tuned scGPT (x-axis, c). (d) UMAP (Meinnes et al., 2020) visualizations of cell embeddings with colors indicating cell types (top) and batch origins (bottom).
cell-level tasks such as accurate cell clustering and robust cell representations. To evaluate sciLaMA's performance and assess the impact of incorporating prior knowledge en- coded as gene embeddings on cell-level tasks (Section 4), we benchmarked sciLaMA against the state-of-the-ard (SOTA) model VI (Lopez et al., 2018) and foundation models such scGPT, CellPLM, and GenePT (Chen & Zou, 2024; Cui et al., 2024; Wen et al., 2023). Multiple variants of sciLaMA were created, each using a different set of gene embeddings precomputed using different prior knowledge databases to determine which prior knowledge is most rel- evant to single cell tasks: MLLaMA-GenePP sciLaMA- ProtTrans, CcLLMMA-CllPL...
Interestingly, the performance of sciLaMMA (s.i.) without any external prior knowledge from LLMs is worse than all variants of sciLaMA with prior knowledge despite the diver- sity of prior knowledgebases, suggesting that incorporating prior knowledge of gene function is broadly acting to regu- larize sciLaMA and prevent overfitting (Figure 2b). These results are consistent with the observation across all tasks. LciL outperformed scVI,, another SOTA VAE-based model without external knowledge, again supporting that incorporating prior gene knowledge is beneficial to single cell analysis.
seil.MM--ChaGGP SciLaMA-ESM. To determine the extent which the scil.- aMA framework itself is superior to other models, we cre- ated the inforeinformed" version of sciLaMA, sciLaMA (s.i.), to represent the framework when learning gene em- beddings from the transposed single cell expression data itself solely (without prior -.d-ivereeedd knowledge). Cell- level tasks were evaluated using five pancreatic seRNA-seq datasets from different labs and sequencing platforms (Tran et al., 2020).
and
While experiment above confirmed incorporating prior knowledge is helpful for single cell analysis, we also won- dered whether with framework inspired by paired VAEs, is sciLaMA the best framework for integrating prior knowl- edge? To explore this, we directly compared the transformer- based foundation model scGPT that was subsequently fine-
Across multiple standard integration metrics (Luecken et al., 2022), all sciLaMA variants robustly outperformed other models both individually (Figure 2a,d, Appendix E) as well as on average (Table 1), suggesting that the sciL frame-
5

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
PCC
T-J5D
Table 2. Comparison of runtime (in seconds) for modeling 14,767 human pancreatic cells sourced from five different origins on single NVIDIA A100 80GB GPU. Due to memory limitations, the batch size for scGPT was set to 10, while siVAE and the various sciLaMA configurations utilized batch size 128.
BoILaWA ESM
Cel
#allaki ChonGPT
aclaM Profhens
wouatstassarr
SCC
MRMSE
gnvi
Method
ScGPT fine-tune
SIVAE
sciLaMA (avg.)
ecProjecion
Runtime (s) 1
19,474
2,265
759
Tangram
PCC
300
1.J5D
1RMME
Ranking
1
+0. Scoor 1
tuned on our training single cell data (scGPT-fnnetune) with seiL.aAA-sGPPT (sciLaMA using pretrained scGPT gene embeddings). Both models are based on the same pretrained scGPT-whole-humee as prior knowledge, but differ in how the pretrained embeddings are updated further. sciLaMA- scGPT outperformed fine-tuned scGPT by 6.82% in cell type clustering and annotation task (ARI and NMI) (Fig- ure 2c). Although the fine-tuned scGPT achieved marginally better results in silhouette width (ASW), its lower batch- ASW and integration-LISI (ILISI) scores (by 34.57% on average) indicate poor batch integration. This comparison underscores the lightweight and well-designed nature of sciLaMA, which improves performance while being more computationally efficient, reducing runtime by 25-fold com- pared to fine-tuned scGPT (Table 2).
woL.tt-Puuttmm net elti-GeoffT BOLA
Measured
Imouted nel aldk Chantery
Measured
oct aloh-crsttrt
Games
Figure 3. Accurate imputation of unseen gene expression with acA. (a) Quantitative performance comparison of models based sciLaMA against other methods for gene imputation task using leave-one- leave-one-gene-owe gene-out strategy. (b) Metric values for 30 genes from the spatial dataset across methods (color-coded), (e) Example visualizations of measured (left) and imputed (right) spatial gene expression patterns.
4.2. sciLaMA Reconstructs Gene Expression with High Accuracy
We next benchmarked sciLaMA accuracy on gene-level tasks, starting with the imputation of gene expression pat- terns. Gene imputation, the prediction of missing or masked gene expression levels based on other genes' profiles, is particularly beneficial for sparsely measured datasets, such as Multiplexed Error-Robust Fluorescence in situ Hybridiza- tion (MERFISH) or Setibbddy-Derived Tags (ADTs), where only a subset of genes is typically quantified in an exper- iment. We benchmarked sciLaMA against leading mod- els for gene imputation accuracy, including scProjection, VI, uniPort and Tangram (Johansen et al., 2023; Lopez et al., 2019; Cao et al., 2022: Biancalani et al., 2021). The experimental setup employed leave-one-gene-out strategy, where the expression of single gene was masked across all cells, and the models were tasked with predicting its expression pattern based on the remaining genes.
Jensen-Shannon Divergence (1-JSD), and 3.32% in 1/Root Mean Squared Error (1/RMSE) on average. These met- rics indicate that its predictions were more aligned with true gene expression patterns compared other models (Figure 3a). Notably, the results demonstrate the signifi- cance of incorporating external gene information gain, as evidenced by sciLaMA's performance superiority over the baseline SciLaMM (s.i.) model, as well as additional baseline sciLaMA (random) and sciLaMA (shuffled) (Appendix E). Unlike sciLaMA, these baseline models do not leverage meaningful prior knowledge derived from LLMs. Specif- ically, scil.aMA (s.i.) utilizes transposed single-cell ex- pression matrix, sciLaMA (random) employs a randomized input matrix for the gene encoder, and sciLaMA (shuffled) uses shuffled external gene embeddings to intentionally dis- rupt dimension alignment. Collectively, these comparisons emphasize the significance of leveraging structured, seman- tically meaningful gene embeddings derived from LLMs to enhance generalizability.
Our results show that sciLaMA models consistently out- performed competing models in imputation accuracy (Fig- ure ba,b, and Table 3) on the spatial transcriptomics data (Codeluppi et al., 2018). sciLaMA achieved the high- est scores across established metrics (Appendix D) (Li
et al., 2022), outperforming the average performance of other benchmarked methods (Johansen et al., 2023; Lopez et al., 2019; Cao et al., 2022; Biancalani et al., 2021) 1) by 27.39% in Pearson Correlation Coefficient (PCC), 15.58% in Spearman Correlation Coefficient (SCC), 32.86% in
Figure 3c illustrates examples of measured versus imputed spatial patterns for genes such as Cpne5 and Sox10 and show sciLaMA accurately predicts expression while preserving spatial organization and region-specine heterogeneity of expression, which crucial for understanding tissue spatial
6

---

sciLaMA: A Single-Cel Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
PEMCs saLaMA embeddlnng
PBMCSs gene embodding
GO onr chmment analysis on module
ScLaMA contestual gene embedding (multi-source humen pancrees datssats)
gronie peosle
0 alu
Shate gene enibediing
LA contcrlad #moodsing
EMIT
UMAF
UMAP
UMAP
Figure 4. Marker gene identification and validation using sciLaMA. (a) UMAP of human PBMC 3K dataset cell embedding using sciLaMA, with points representing cells colored by cell type and outlined by coarse cell classes using dashed circles. (b) Comparison of LLM-derived static gene embedding (top) and
contextual gene embedding (bottom) with points representing genes. Marker genes are colored by cell type specificity, and those from the same circle are relevant the same broader cell classes. Color codes are consistent between (a) and (b). (c) A graph of gene module identified through sciLaMA-base gene clustering, with Gene Ontology (GO) terms enriched for mosule-associated genes. The module includes PPBP gene, a known marker for Megakaryocytes. (d) Bar chat of the top six GO terms and significance (adjusted p-values). (e) UMAP visualization of sciLaMA contextual gene embedding on colti-souure human pancreas datasets. Marker gene modules associated with different cell types are highlighted.
with particular cellular states or types. aaiL not only re- duces the manual labor involved in marker identification but also opens up possibilities for discovering new biological insights by detecting subtle, coordinated gene expression patterns that expert-driven methods might overlook.
Table Evaluation of gene expression imputation performance on spatial transcriptomics data across multiple methods using Pearson Correlation Coefficient (PCC), Spearman Correlation Coefficient (SCC), Jensen-Shannon Divergence (JSD), and Root Mean Square Error (RMSE). A leave-one-gene-our strategy was applied on 30
measured genes.
To assess sciLaMA's efficacy in marker gene identification, we compared its contextual gene embeddings to static em- beddings from the LLMs. sciLaMA's contextualization significantly improved the clustering of markers associated with the same cell states (Figure 4a-b). For example, in the static embeddings (Figure 4b, top), marker genes for the same cell type do not cluster as expected, while in the sciL contextual embeddings (Figure 4b, bottom), mark- ers for the same cell states group together, as indicated by the circles. Moreover, PPBP is a well-established marker for Megakaryocytes (platelet precursor cells) in human pe- ripheral blood mononuclear cells (PBMCs) (Butler et al., 2018), and sciLaMA's contextual gene embedding presents cluster that includes it. Neighboring genes within this cluster were linked to platelet-related biological processes, cellular components, and molecular functions, confirmed via Gene Ontology (GO) enrichment analysis (Figure 4c-d) (Subramanian et al., 2005). Many of these genes, though not previously annotated as Megakaryocyte markers from the original study, exhibit strong ro-expression with PPBP and functional links to platelet biology. Their coordinated clus- tering in biologically meaningful modules indicates their relevance to Megakaryocyte identity.
Methods
PCC (t)
(CC
JSD (L)
RMSE (1)
sciLaMA (avg.)
0,222 + 0.027
0,217 + 0,028
0.283 + 0.008
1.242 0.022
saProjection
0.177+0..00
0.207 + 0.029
0.352 + 0.032
1,277 + 0.023
gimVI
0.224 0.021
0,207 0.024
0,580 0.014
1,243 0.017
uniPort
0,166 + 0,027
0.027
0.451 + 0.017
1,287 + 0.022
Tangram
0.154 0.019
0.458 + 0.017
0.016
structure.
4.3. sciLaMA Enables Marker Gene Identification
In single-cell studies, identifying and validating marker genes characteristic of individual cell types is another essen- tial process for cell type annotation traditionally dependent on expert domain knowledge. Conventionally, bioinfor- maticians preprocess and integrate data, cluster cells, and then experts annotate these clusters using known biomark- ers or gene signatures relevant to specific cell types (Butler et al., 2018; Wolf et al., 2018). Such division of labor is time-consuming and demands extensive collaboration. scil.- aMA streamlines this process by simultaneously integrating cells and implicitly organizing genes into biologically mean- ingful modules within its contextual gene representation space. By analyzing gene embeddings, sciLaMA can iden- tify groups of genes that are consistently co-expressed or show coordinated patterns within specific cell types. This goes beyond simply checking the expression levels of pre- defined markers such as CD4 for T-cells. Instead, it reveals potentially unknown gene modules that strongly correlate
Furthermore, seil.aMA robustly identified marker modules across multiple datasets, demonstrating its effectiveness even in the presence of batch effects (Figure 4e). Im- portantly, sciLaMA integrating LLM-derived priors gene knowledge outperformed the self-informed version scil.-
7

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
PO mouse neurodeveloppent
Call arrbseddnng
Table 4. Cell representation learning performance on PO mouse neurodevelopment dataset, with ARI and NMI quantifying cluster annotation accuracy, and ASW and eLISI quantifying cell type
ARd
IP Guaddebu co02, cranc EX23 Clid
MM
separation.
Methods
ARI
NMI 1
ASW t
cLISI 1
sciLaMM
0.316
0.351
0.518
0.738
Exceaatoy nowrcn on
progenttor
scVI
0.284
0.291
0.501
0.501
d
MclLaMA gene embedding
wit
alignment
aign
clearer transitions between developmental stages, such as the progression from intermediate progenitors (IPs) to layer- 2-3 excitatory neurons (ExNs). sciL.aMA outperformed sc VI in trajectory clarity by 20.65% overall (Table 4).
gete
Pseudotime Eseudotime-aligned heatmaps of gene expression (Figure 5e, and Appendix E) highlighted camporl--pecciti genes with coordinated expression shifts corresponding to distinct stages of cell differentiation. Additionally, sciLaMA's con- textual gene embeddings further illuminated temporal re- lationships between genes, offering insights into the se- quential activation of developmental markers (Figure 5d). This analysis provides a comprehensive perspective on the dynamic interplay of genes during cell differentiation and development.
Figure 5. Enhanced developmental gene trajectory analysis with sciLaMA. (a) Overview of PO mouse neurodevelopment data, with five cell types from early progenitors to mature excita- tory neurons. (b) UMAP visualizations of cell embeddings using sciLaMA (top) and scVI (bottom) with bar plot comparing cell type annotation and separation performance. (c) Pseudotime (x- axis) heatmap displaying the dynamic changes in gene expression across developmental stages. Rows represent ordered temporal specific genes. (d) UMAP visualizations of gene embeddings with- out (left) and with (right) embedding alignment using sciLaMA. Temporal specific genes (from (c)) are highlighted with color gra- dient.
By accurately mapping cell lineages and identifying stage- specific gene modules, sciLaMA provides researchers with a powerful tool for understanding cell differentiation and de- velopmental processes. When applied to organoid datasets, sciLaMA can also compare developmental trajectories of organoids with those of real tissues. For example, it can identify which gene modules from real tissues correspond to specific stages in organoid development, aiding in the as- sessment of organoid fidelity. This capability has significant implications for therapeutic strategies, enabling researchers to evaluate how organoids can model human diseases and inform potential treatment designs.
aMA (s.i.) across clustering metrics (Table S7), which indicates the value of leveraging pretrained static gene em- beddings. These findings highlight sciLaMA's potential to streamline single-cell studies by reducing reliance on manual annotation and revealing novel biological insights, which advances gene module discovery.
4.4. sciLaMA Enhances Trajectory Analysis by Unveiling Temporal Dynamics of Genes
Building upon its strength in identifying gene markers and modules across discrete cell types, sciLaMA also excels at capturing temporal dynamics in developmental processes. This capability enables the study of continuous gene expres- sion changes across time and facilitates the analysis of cell differentiation and developmental trajectories.
5. Conclusion
This study introduces sciLaMA, novel framework that inte- grates external gene knowledge from language models with single-cell expression data to address critical challenges in single-cell analysis and enable comprehensive downstream tasks spanning both cell-level and gene-level analyses. Our experiments demonstrate the framework's 's effectiveness and performance superiority, and highlight the value of incor- porating external gene knowledge through an innovative design. These findings establish sciLaMA as a powerful tool for advancing our understanding of cellular heterogene- ity and gene regulation, and showcase how language models can be leveraged through a lightweight adapter framework.
To investigate sciLaMA's capability in this context, we con- ducted pseudotime trajectory analysis using cell embeddings learned by LaMA and compared them with those from VI, a SOTA single-cell model. The analysis was applied to a dataset capturing PO mouse cortex development (Fig- ure 5a) (Chen et al., 2019). Pseudotime visualizations (Fig- ure 5b, and Appendix E) illustrated that sciLaMA provided
8

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
Acknowledgements
This work began during H.H.'s research internship at Mi- crosoft Research and was supported by an NSF CAREER award (1846559, G.Q.). Additional funding came from the National Institutes of Health, including the Office of the Di- rector/National Institute of Mental Health (DP2 MH129987, G.Q.) and the National Institute of Child Health and Human Development (P50 HD103526). We thank Erdal Cosgun, Shuangjia Lu, and the Microsoft Research Health Futures team for their guidance and technical insights.
Impact Statement
This paper aims to integrate contextual and general knowl- edge about cells and genes to enhance single-cell data anal- ysis. By leveraging large language models from different modalities, our methodology supports critical tasks with improved performance and computational efficiency. This approach has the potential advance the field of Machine Learning in scientific research by facilitating the study of cellular heterogeneity, gene regulation, and developmental processes across various biological contexts. Furthermore, its ability to streamline analysis reduces reliance on exten- sive manual annotation and switching between disparate tools on various tasks. The implications of this approach include its contribution to biomedical research, where it may support therapeutic discovery and precision medicine. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
References
Cui, H., Wang, C., Maan, H., Pang, K., Luo, F,, Duan, N., and Wang, B. segpt: Toward building foundation model for single-cell multi-omics using generative ai. Nature Methods, 21(8):1470-148) 2024. doi: 10.1038/ s4 $41592-024-02201-1
Baron, M., Veres, A., Wolock, S. L., Faust, A. L.,, Gaujoux, R., Vetere, A., Ryu, J. H., Wagner, B. K., Shen-Orr, S. S., Klein. A. M., Melton, D. A., and Yanai, I. A single-cell transcriptomic map of the human and mouse pancreas reveals inter- and intra-cell population structure. Cell Systems, 3(4):346-36) 2016. doi: 10.1016fj.cels.2016. 08.011.
Elnaggar, A.. Heinzinger, M., Dallago, C.. Rehawi, G., Wang, Y., Jones, L.., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., Bhowmik, D., and Rost, B. Prottrans: Toward understanding the language of life through self- supervised learning. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 44(10):7112-712) 2022. doi: 10. 109/TPAMI.201.3300 330355931.
Biancalani, T., Scalia, G., Buffoni, L., Avasthi, R., Lu, Z., Sanger, A., Tokcan, N., Vanderburg, C. R., Segerstolpe, A., Zhang, M., et al. Deep learning and alignment of spatially resolved single-cell transcriptomes with tangram. Nature Methods, 18(1 1):352--133, 2021. doi: 10.1038/
Grønbech, C. H., Vording, M. F., Timshel, P. N., Sønderby, C. K., Pers, T. H., and Winther, O. scvae: Vari- ational auto-encoders for single-cell gene expression data. Bioinformatics, (6(16):441544422 2020. doi: 10. No..0///iionfrrmatis/bbaaa
s41592-021-01264--
Butler, A., Hoffman, P., Smibert, P., Papalexi, E., and Satija, R. Integrating single-cell transcriptomic data across differ- ent conditions, technologies, and species. Nature Biotech- nology, 36(5):411-422 4200, 2018. doi: 10.1038/nb..40%
Janizek, J. D., Spiro, A., Celik, S., Blue, B. W., Russell, J. C., Lee, T.-I,, Kaeberlin, M., and Lee, S.-L Pause: Principled feature attribution for unsupervised gene ex- pression analysis. Genome Biology, 24(1):81, 2023. doi: 000-66//3359-02---00000
Cao, K., Gong, Q.. Hong. Y., and Wan, L. unified compu- tational framework for single-cell data integration with optimal transport. Nature Communicationss 13(1):7419, 2022. doi: 10. 10.1038//41467-022-3333 35094-8.
Johansen, N., Hu, H., and Quon, G. Projecting mea- surements onto single cell atlases to extract cell type- specific expression profiles using seprojection. Na- ture Communications, 14(1):5192, 2023. doi: 10.1038/ 111467023-4074444
Cao, Z.-J. and Gao, G. Multi-omics single-cell data integra- tion and regulatory inference with graph-linked embed- ding. Nature Biotechnology, 40(10): 2022. doi: 10. 10.10088/415587-02---1144444
Chen, S., Lake, B. B., and Zhang, K. High-throughput sequencing of the transcriptome and chromatin accessi- bility in the same cell. Nature Biotechnology, 37(12): 1452-1457, 2019. doi: 10. 10.10388/41157--019000900
Kana, O., Nault, R., Filipovic, D., Marri, D., Zacharewski, T., and Bhattacharya, S. Generative modeling of single- cell gene expression for dose-dependent chemical per- turbations. Patterns, 4(8):100817, 2023. doi: 10.1016/j. patter..022.100811
Chen, X., Xu, J., Zhou, R., Chen, W., Fang, J., and Liu, C. Trajvae: variational autoencoder model for trajectory generation. Neurocomputing, 428:332-339, 2021. doi: ne.0016j.neccom.2200...111
Kedzierska, K. Z., Crawford, L., Amini, A. P., and Lu, A. X. Assessing the limits of zero-shot foundation models in single-cell biology. 2023. doi: 10.1101/2023.1..1 561085.
Chen, Y. and Zou, J. Simple and effective embedding model for single-cell biology built from chatgpt. Na- ture Biomedical Engineering, 2024. doi: 10.1038/ s41551-024-01284--
Kingma, P. and Welling, M. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2014. doi: 10. 48550/arXiv. 1312.6114.
Choi, Y., Li, R., and Quon, G. sivae: Interpretable deep generative models for single-cell transcriptomes. Genome Biology, 24(1):29, 2023. doi: 10.1 $13059-023-02850.
Lähnemann, D., Köster, J., Szczurek, E., McCarthy, D. J., Hicks, S. C., Robinson, M. D., Vallejos, C. A., Campbell, K. R., Beerenwinkel, N., Mahfouz, A., et Eleven grand challenges in single-cell data science. Genome Biology, 21(1)331, 2020. doi: 10.1 (0.1186/s13 1006//1339--200 1926-6.
Codeluppi, S., Borm, L. E., Zeisel, A., La Manno, G., van Lunteren, J. A., Svensson, C. I., and Linnarsson, S. Spa- tial organization the somatosensory cortex revealed by osmfish. Nature Methods, 15(11):932- 935, 2018. doi: 10. 1.110///41592-08--1772
Li, B., Zhang, W., Guo, C.,, Xu, H., Li, L., Fang, M., Hu, Y., Zhang, X., Yao, X., Tang, M., et al. Benchmarking spatial and single-cell transcriptomics integration meth- ods for transcript distribution prediction and cell type
10

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
deconvolution. Nature Methods, 19(6):62--670, 2022. doi: 10. 1000008//41122-002---49----
Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee,
S., Ebert, B. L.., Gillette, M. A., Paulovich, A., Pomeroy, S. L., Golub, T. R., Lander, E. S., and Mesirov, J. P. Gene set enrichment analysis: A knowledge-basee approach for interpreting genome-wide expression profiles. Pro- ceedings of the National Academy of Sciences, 102(43); 15345-15550 2005. doi: 00.1073/paas.05065880000
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. 2023.
Liu, T., Chen, T., Zheng, W., Luo, X., and Zhao, H. scelmo: Embeddings from language models are good learners for single-cell data analysis. 2023.
Theodoris, C. V., Xiao, L., Chopra, A., Chaffin, M. D., AI Sayed, Z. R., Hill, M. C., Mantineo, H., Bry- don, E. M., Zeng, Z., Liu, X. S., and Ellinor, P. T. Transfer learning enables predictions network biol- ogy. Nature, 618(7965):616666 624, 2023. doi: 10.1038/ s4 $41586-023-061139999
Lopez, R., Regier, J., Cole, M. B., Jordan, M. I., and Yosef, N. Deep generative modeling for single-cell transcrip- tomics. Nature Methods, 15(1 1553310003- 2018. doi: 12110888411592-01---22229
Tran, H. T. N., Ang, K. S., Chevrier, M., Zhang, X., Lee, N. Y. S., Goh, M., and Chen, J. A benchmark of batch- effect correction methods for single-cell ma sequencing data. Genome Biology, 21(1):12, 2020. doi: 10.1186 s $13059-019- 1850-9.
Lopez, R., Nazaret, A., Langevin, M., Samaran, J., Regier, J., Jordan, M. I., and Yosef, N. A joint model of unpaired data from scrna-seq and spatial transcriptomics for im- puting missing gene expression measurements. arXiv preprint arviv: 1.tv:/905.00 022602 2019.
Wang. J., Schug. J., Won, K.-., Liu, C., Naji, A., Avra- hami, D., Golson, M. L.,, and Kaestner, K. H. Single-cell transcriptomies of the human endocrine pancreas. Dia- betes, 65(10)33028 80388,008800000 2016. doi: 10.2337/ab16-0005.
Luecken, M. D., Büttner, M., Chaichoompu, K., Danese, A., Interlandi, M., Mueller, M. F., Strobl, D. C., Zap- pia, L.., Dugas, M., Colomé Tatché, M., and Theis, J. Benchmarking atlas-level data integration in single-cell genomics. Nature Methods, 19(1):41-50, 2022. doi: 10.10088441592-021-0011
Wen, H., Tang, W., Dai, X., Ding, J., Jin, W., Xie, Y., and Tang. Cellplm: Pre-training of cell language model beyond single cells. 2023. doi: 10.1101/2023.00.03. 560734.
McInnes, L., Healy, J., and Melville, Umap: Uniform manifold approximation and projection for dimension 2020. doi:
reduction. arXiv preprint
Wolf, F. A., Angerer, P., and Theis, F. Scanpy: Large-scale single-cell gene expression data analysis. Genome Biol- ogy, 19(1):15, 2018. doi: 10.
10.48550/arXiv.1188 1802..0346
Muraro, M. J., Dharmadhikari, G., Grün, D., Groen, N., Dielen, T., Jansen, E., van Gurp, L., Engelse, M. A., Carlotti, F., de Koning, E. J. P., and van Oudenaarden, A.
Xin, Y., Kim, J., Okamoto, H., Ni, M., Wei, Y., Adler, C., Murphy, A. J., Yancopoulos, G. D., Lin, C., and Gromada, J. Rna sequencing of single human islet cells reveals type 2 diabetes genes. Cell Metabolism, 24(4):608-61) 615, 2016. doi: 10. 00.0016j.mme.2216...8 6.1.0000..
A single-cell transcriptome atlas of the human pancreas. Cell Systems, 3(4):3(3 394,, 2016. doi: 10.1016/j cels. 2016.09.002.
OpenAL.
New and improved embedding model: Text-embbeding--ad--000. URL
Yan, J., Ma, M., and Yu, Z. bmvae: A variational autoen- coder method for clustering single-cell mutation data. Bioinformatics, 39taa,bt 2023. doi: 10.1093/ bioinformaticc/bbacc99..
2022.
https::/ com/ //de////////// and-and-Bo seaanddiprooee-eee wew.reed--eeedddd---dnddeeee m/
Rosen, Y., Brbić, M., Roohani, Y., Swanson, K., Li, Z., and Leskovec, J. Toward universal cell embeddings: Inte- grating single-cell ma-seq datasets across species with saturn. Nature Methods, 21(8):1492-1500 2024. doi: 10.10388441592-0242221999
Zeisel, A., Muñoz-Manchado, A. B., Codeluppi, S., Lönnerberg, P., La Manno, G., Juréus, A., Marques, S., Munguba, H., He, L., Betsholtz, C., al. Cell types in the mouse cortex and hippocampus revealed by single- cell rna-seq. Science, 347(6226): 138-112) 2015. doi: 10. /111///cennce.aaa1999
Segerstolpe, Å., Palasantza, A., Eliasson, P., Andersson, E.-M., Andréasson, A.-C., Sun, X., Picelli, S., Sabirsh, A., Clausen, M., Bjursell, M. K., et al. Single-cell tran- scriptome profiling of human pancreatic islets in health and type 2 diabetes. Cell Metabolism, 24(4):593-60) 2016. doi: 10.1016/j.cmet.20 2016.08.020.

---

Appendix
sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
A. Model Input Processing:
A.1. Cell Encoder Input
As mentioned in Section 3 Methods, the input for cell Encoder is the scRNA-seq data for specific cell population Therefore, Cij denotes the scaled log-normalized expression value of gene j in cell i.
The raw scRNA-seq expression matrix, cmt., is sparse count matrix. For use in sciLaMA, the data after quality control (QC) is processed through library size normalization and feature- wise z-score scaling to achieve zero mean and unit variance. Values beyond +10 are clipped. Specifically, the normalized expression " is calculated as:
log. (1+10' x
borm C
Dk=1 C
Th
Here, cmw represents the raw count value of gene cell i, and Σ
Jaw is the total expression counts number for cell i. C.k:
The multiplication by 10t ensures standardized size factor for normalization. This normalization procedure adjusts for library size differences across cells and prepares the data for following analysis.
A.2. Gene Encoder Input
In this study, sciLaMA integrated static gene embeddings from six external sources across three distinct modalities.
Source
Dimensionality
Modality
ChatGPT
1536
Text
GenePT (NCBI)
1536
Text
ESM
5120
Protein Sequence
ProtTrans
1024
Protein Sequence
scGPT
512
Single Cell
CelIPLM
1024
Single Cell
Table SI. Gene Embedding Sources and Characteristics.
A.2.1 Natural Language Embeddings
We acquired text description-based gene embeddings from two studies: GenePT and scELMo (Chen & Zou, 2024; Liu et al., 2023), utilizing the OpenAl rext-embedding-ee ada-002 model (OpenAI, 2022). These embeddings were generated using two distinct text corpora: GPT-3.5 generated descriptions (referred to as CharGPT) and National Center for Biotechnology Information (NCBI) gene card summaries (referred to as GenePT). We obtained 1,536-dimensional static embeddings for each gene 1,536).
A.2.2 Protein Language Embeddings
We derived protein sequence-based gene embeddings from two protein language models: ESM2 148_15B.URS00 with 5,120- dimensional embeddings per gene (Lin et al., 2023), and ProtXI.Net from ProtTrans with 1,024-dimensionall embeddings (Elnaggar et al., 2022) from the SATURN study (Rosen et al., 2024). These embeddings were generated using the amino acid sequences of each corresponding gene.
A.2.3 Single-Cell Gene Language Embeddings
12

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
For single-cell foundation models, retrieved static gene embeddings from two pretrained models: seGPT- 5512-dimensional embeddings) (Cui et al., 2024) and cellPLM (1.024-dimensional embeddings) (Wen et al., 2023). The scGPT embeddings were obtained using the model's GitHub tutorial, while cellPLM embeddings were extracted from the embedder module's feature encoder parameters, directed by the authors.
B. Model Optimization Illustration:
The sciLaMA model optimization process, comprehensively described in Section Methods, illustrated through stepwise training strategy visualization (Figure S1). The optimal hyperparameter values chosen for experiments scalar 0.05 and latent dimensionality K 40, based on the evaluations presented in Table S8 and Table S9.
(1)
(2)
Cln
Gene (M]
Gane IH
Pretrin oBIMEE
Freeze sellikE, preanih genelAEE
Johrty refine paired WiSs of sol.MMA
Figure $1. Schematic representation of the progressive optimization workflow for the sciLaMA framework. (Box indicates freezing parameters.)
C. Dataset Introduction:
C.1. Experiment Cell Representation Learning Benchmarking
This experiment benchmarks cell representation learning methods using combination of single-cell RNA sequencing datasets derived from five studies focused on the pancreas. The data includes total of 14,767 cells spanning 13,062 genes (after intersection with precomputed static gene embeddings).
Datasets Used: Baron et al.: 8,569 cells (Baron et al., 2016); Segerstolpe et al.: 2,127 cells (Segerstolpe et al., 2016); Muraro et al.: 2,122 cells (Muraro et al., 2016); Xin et al.: 1,492 cells (Xin et al., 2016); Wang et al.: 457 cells (Wang et al., 2016).
The aggregated dataset was gold-standard benchmarking dataset originally analyzed in the context of batch-effect correction, as described Tran al., in 2020 (Tran al., 2020). This benchmarking experiment evaluates the performance of cell representation learning in mitigating batch effects while preserving biological signal.
C.2. Experiment 2: Gene Expression Imputation Benchmarking
This experiment evaluates the accuracy of gene expression imputation approaches by leveraging two complementary datasets (Zeisel et al., 2015; Codeluppi et al., 2018):
Reference se RNA-seq Dataset (Zeisel et al., 2015):
Number of cells: 3,005
Total genes: 19,972, with 3,654 highly variable genes selected for benchmarking.
Validation: A 10% validation split is used for early stopping during model training.
Spatial Transcriptomics Dataset (Codeluppi et al., 2018):
Number of spatial spots: 4,530
13

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
Genes: 30, analyzed using leave-one-gene-our approach simulate imputation scenarios.
This setup allows for assessing the generalizability of gene imputation models.
C.3. Experiment 3: Marker Gene Identification
This experiment focuses on identifying marker genes for distinct cell types using the human Peripheral Blood Mononu- clear Cell (PBMC) 3K dataset from 10x Genomics, a legacy dataset widely utilized in tools like Seurat (Butler et al., 2018) and seanpy (Wolf et al., 2018) tutorials. The ground truth gene markers and cell type annotations were ob- tained from the tutorials httpk////wtpp///..aa.a/aaaaaaaaa // //scappy... htt.ttttwc....eeeeeeaaaaaaaaaaaa
tutorial, and https: html).
Dataset Details:
Initial Size: 2,700 cells x 32,738 genes
Final Size: 2,700 cells 9,540 genes (post-filtering and intersection with static gene embeddings).
Cell Type
Number of cells
CD4 T cells
1158
CD14 Monocytes
487
B cells
357
CD8 T cells
329
FCGR3A Monocytes
160
NK cells
160
Dendritic cells
36
Megakaryocytes
13
Table S2. Cell Type Statistics of human PBMC 3K dataset
C.4. Experiment 4: Trajectory Analysis and Temporal Dynamic Gene Discovery
This experiment investigates gene dynamics along developmental trajectories using the PO mouse cortex dataset from the SNARE-seq study (Chen et al., 2019). The original SNARE-seq dataset includes both transcriptomic and epigenomic information from the same single cells, but we only utilized the transcriptomic data with 1,469 cells and 8,293 genes (after intersection with precomputed static gene embeddings). This experiment focuses on uncovering temporally dynamic genes critical for neurodevelopmental processes. The ground truth gene markers and cell type annotations were obtained from the original study.
Cell Type
Number of cells
IP Im_gnn2
214
UP.Gadd45g
99
IP_Eomes
437
Ex23 Cntn2
177
Ex23_Cux1 542
Table $3. Cell Type Statistics of mouse PO cortex dataset
D. Benchmarking Metrics Introduction:
To comprehensively evaluate the performance of various methods, employ metrics tailored to different aspects of single- cell data analysis, including cluster annotation accuracy, cell type separation, batch mixing quality, and predictive/imputation accuracy (Li et al., 2022; Luecken et al., 2022), briefly summarized below:
D.1. Clustering and Annotation Accuracy
To assess the biological relevance of clustering and annotation based on the learned embeddings, we employ:
14

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
Adjusted Rand Index (ART): Measures the agreement between predicted and ground-truth cluster labels, adjusted for chance. A higher ART indicates better alignment between predicted clusters and original biological annotations, reflecting more accurate and biologically meaningful clustering.
Normalized Mutual Information (NMI): Quantifies the mutual dependence between predicted clusters and ground truth cell type annotation labels, normalized to account for the total number of clusters. A higher NMI indicates better clustering accuracy.
D.2. Cell Type Separation
To evaluate how well methods preserve separation between distinct cell types, employ:
Average Silhouette Width (ASW): Evaluates the cohesion within clusters and the separation between them. Higher ASW scores indicate that cells within the same cluster are more similar to each other than to cells in other clusters, signifying well-defined clusters.
Graph Cell-Type Integration Local Inverse Simpson's 's Index (cLISI)): Measures the local diversity of cell types within neighborhoods an integrated graph representation. High cLISI values suggest better grouping similar cell types the embedding space.
D.3. Batch-Effect Correction Quality
To evaluate batch effect removal while preserving biological variance, we apply:
Batch-Adjjsted Silhouette Width (batchASW): Evaluates the extent of batch mixing while penalizing over-mixing of unrelated cells. Higher batchASW scores indicate better batch integration without compromising biological separation.
Graph Integration Local Inverse Simpson's Index (iLISI): Measures the diversity of batch labels within local neighbor- hoods of an integrated graph. Higher iLISI scores indicate more uniform batch mixing, reflecting better integration while preserving cell type integrity.
D.4. Predictive Accuracy and Divergence Metrics
For imputation and gene expression prediction tasks, we employ:
Pearson Correlation Coefficient (PCC): Assesses linear relationships between predicted and observed gene expression values, with higher values indicating stronger correlations.
Evaluates rank-based relationships, capturing monotonic correlations between predicted and observed values, providing insights into the consistency of expression patterns.
Spearman Correlation Coefficient (SCC):
JensennShannon Divergence (JSD): Measures the similarity between predicted and true gene expression distributions. Lower ISD values indicate better agreement between the two distributions.
Root Mean Square Error (RMSE): Quantifies the average magnitude of errors between predicted and observed values. Lower RMSE scores reflect higher accuracy
D.5, Clustering Quality Metries To evaluate the geometric coherence and separation of clusters in the learned gene embedding space, we include two additional metrics:
Davies-Bouldin Index (DBI): Quantifies the ratio of intra-cluster dispersion to inter-cluster separation. Lower DBI values indicate better-defined clusters with high intra-cluster similarity and distinct separation between clusters.
Salinski-Harabar Score (CHS): Measures the ratio of between-cluster dispersion within-cluster dispersion. Higher CHS values reflect dense, well-separater clusters
E. Supplementary Results:
15

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
Methods
w external knowledge
ARI 1
NMI
ASW 1
batchASW
iLISI f
sciLaMAAGenePPT
0.545
0.767
0.539
0.863
0.240
scLLaMA-CelILMM
0,479
0.723
0.541
0.871
0.257
sciL.MMA-ProoTrrnss
0.547
0.749
0.538
0.864
0.229
sciL.aMA-CharGPT
0.545
0.762
0.534
0.863
0.225
scil.aMA--cGPPT
0.522
0.746
0.526
0.867
0.223
sciLaMA-ESM
0.494
0.722
0.529
0.864
0.253
sciLaMM (s.i.)
0.436
0.698
0,539
0.832
0.210
Table S4. Cell representation learning and integration performance on human pancreatic datasets across variants of sciLaMA models
Methods
cLISI
SciLaMMA-GenePi
0.995
SciLaMA-CelIPLM
0.995
SciLaMA-PPotTrans
0.992
sciL.MMA-ChaGGPT
0.993
sciLaMA- scGPT
0.993
sciL.aMA- ESM
0.993
sciLaMA (s.i.)
0,987
scGPT fine-tuned
0.998
sc scVI-batch
0.982
Ve-raw
0.972
scGPT zero-shot
0.951
CellPLM zero-shot
0.961
GenePT-w
0.838
Table S5. Graph Cell-Type Integration Local Inverse Simpson's 's Index (cLISI) scores across methods (listed supplementary result due to the low variance of 0.001714)
Methods
W| external knowledge
PCC (f)
SCC (f)
JSD (1)
RMSE (L)
SciLaMA-GenePP
0.220 0.029
0.214 0.031
0.280 # 0.009
1.243 0.023
scil MMA
0.222 0.027
0.218 0.028
0.286 # 0.009
1.242 # 0.022
sciLaMA-ProoTrrnss
0.218 # 0.026
0.211 = 0.028
0.283 # 0.009
1.246:0.00
SciLaMA-ChaIGPT
0,219 + 0.027
0,217 = 0.027
0,282 = 0.009
1.244
# 0.022
scil.aMA-sGGPT
0,219 + 0.027
0.217 = 0.027
0.285 # 0.009
# 1244400000
sciLaMA-ESM
0.233 0.026
0.227 = 0.027
0.282 # 0.009
1,233 # 0.022
sciLaMA (s.i.)
X
0.202 0.027
0,212 = 0.025
0,286 = 0.009
1.258 + 0.022
scil.aMA (random)
0.051 0.027
0.049 0.031
0.289 + 0.009
1.374 0.020
sciLaMA (shuffled)
0.056 0.036
0.043 0.037
0.288 = 0.009
1.366 0.027
Table S6. Evaluation of gene expression imputation performance spatial transcriptomies data across variants of sciLaMA models
Methods
w/ external knowledge
Davies-Bouldin Index (4)
Calinski-Harabass Score (1)
SciL.MM-GGenePT
0.852
16.376
SciLaMA-CeIPLLM
0.727
19.610
sciLaMA-PProTTrans
0.802
19.947
sciLaMA-ChaGGPT
0.874
16.522
scilLaMA-sGGPPT
0.780
17.973
sciLaMAEESM
0.780
16.920
sciLaMA (s.i.)
X
0.977
13.087
Table S7. Clustering performance comparison for marker gene identification across variants sciLaMA models
16

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
7
ARI mean
ARI std
NMI mean
NMI std
ASW mean
ASW std
cLISI mean
cLISI std
0.464
0.371
0.513
0.390
0.589
0.087
0.881
0.170
0.01
0,582
0.021
0.734
0.022
0.645
0.010
0.990
0.002
0.05
0.665
0.114
0.763
0.041
0.654
0.005
0.991
0.002
0.1
0.634
0.088
0.751
0.023
0.658
0.012
0.992
0.002
0.25
0.581
0.024
0.743
0.013
0.655
0.006
0.990
0.002
0.5
0,592
0,010
0.747
0,004
0.658
0,012
0,993
0.001
0.75
0,590
0.015
0.748
0,013
0.656
0,012
0,992
0.002
1
0.647
0.107
0.762
0.038
0.651
0.025
0.993
0.003
Table S8. Effect of scalar yy clustering performance across multiple metrics.
K Latent dim
ARI mean
ARI std
NMI mean
NMI std
ASW mean
ASW std
cLISI mean
cLISI std
10
0,651
0.110
0.756
0.042
0.654
0.005
0.991
0,002
20
0.627
0.082
0.761
0.025
0.631
0.015
0.991
0,000
30
0,583
0.017
0.742
0.022
0.633
0,014
0.991
0.001
40
0.680
0.114
0.771
0.039
0.637
0.009
0.991
0.002
50
0.606
0.085
0.743
0.023
0.631
0.010
0.990
0.002
60
0.649
0.100
0.757
0.030
0.631
0.015
0.990
0.002
70
0.590
0.037
0.738
0.017
0.635
0,009
0.991
0.001
80
0.649
0.095
0.753
0.020
0.632
0.009
0.991
0.001
90
0.656
0.102
0.756
0.030
0.637
0.008
0.991
0.001
100
0.641
0.108
0.751
0.027
0.635
0.006
0.991
0.002
Effect of varying latent dimension K on clustering performance across multiple metrics. Table $9,
17

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
Call Erbeeciingg Bandhrrrt
ASV
101
6.8
ARI
Dath 2AURO
ACTUREY
Reseectuoaa MNC
RJSI
WILPNA-CMPPM
CHarOPT
VcRUUMA
Figure $2. Benchmark of cell representation learning, (a) Radar plot showing the performance six established metrics, comparing single cell SOTA methods (scVI and scVI-batch), zero-shot models (GenePT-w, CellPLM, and scGPT), fine-tuned model (scGPT), and comparablc sciLaMA-basee models DAMAA-GeePT/CClIILMMMMaaa various models, with colors indicating cell types (top) and batch origins (bottom). (b) includes foundation models in zero-shot mode, while (c) presents acilaMA-basedd models in additional those from Figure 2c.
(b-e) UMAP visualizations of cell embeddings derived from
Cell embedding (pseudotime)
Temporal gene expression (alongside peeudotime)
soILaMA
Tempora
soLaMM-based peuedolime
Tempora
oor
UMAP_?
psuedotime
Figure Enhanced developmental trajectory analysis with sciLaMA. (a) UMAP visualizations of cell embeddings from sciLaMA (top) and scVI (bottom) colored by inferred pscudotime via Palantir. (b) Heatmaps of dynamic gene expression changes by pseudotime (x-axis) with genes ordered by temporal specificity (y-axis). Top shows sciLaMA-based pseudotime, bottom shows scVI results.
18

---

sciLaMA: A Single-Cell Representation Learning Framework to Leverage Prior Knowledge from Large Language Models
Cell Embedding
Contextual Gene Embedding
Hepatocyte
sciLaMA 2
Kupffer
sciLaMA 2
Mac NK
Mono-NK
cell
cell
sciLaMA_1
sciLaMA
expr.
Figure $4. This visualization presents results from sciLaMA (with latent dimensionality K=2) trained exemplar fetal liver dataset (Choi et al., 2023). The cell embeddings (middle-left; dots represent individual cells) are colored according to annotated cell types, while the contextual gene embeddings (middle-right; dots represent individual genes) show corresponding embedding dimensions. The top and bottom panels illustrate cell embeddings colored by expression levels of genes sampled from distinct regions of the contextual gene embedding space. We quantitatively assessed scil sciLsMA's interpretability by computing feature attribution scores using Integrated Gradients applied to the pretrained cell VAE encoder. This resulted in gene-by-latcnt-non attribution matrix, which we summarized into single gene vector by aggregating absolute attribution embeddings into comparable vector form using vector normalization. The strong correlations (Pearson r = 0.43, Spearman = 0.46) indicate that sciLaMA effectively captures key gene features in alignment with traditional stepwise attribution methods, yet does so within more efficient, unified framework.
To facilitate comparison, projected scil.aMAss contextual gene
19


Reviews:
[{"review_id": "cR2q1uVeXT", "reviewer": "Reviewer_7H8M", "summary": "The paper introduces sciLaMA, a novel framework for single-cell RNA sequencing analysis that integrates gene embeddings from large language models (LLMs) with scRNA-seq data using a Variational Autoencoder based architecture. This approach allows for context-aware representations of both cells and genes, enhancing tasks such as batch effect correction, cell clustering, imputation and gene module identification. The framework demonstrates superior performance in various single-cell analysis tasks, offering a flexible and interpretable solution for different single-cell tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Below, we address the three major points raised:\n\n(1) “How does sciLaMA perform if the gene embeddings are random vectors?” \n\nUsing random gene vectors will prevent the model from generalizing to unseen genes. To further clarify this point, we conducted additional experiments: (i) using random gene embeddings as suggested by the reviewer, and (ii) randomly shuffling individual features within LLM-derived external gene embeddings for each gene. The results (Response Table 1) demonstrate that random external gene embeddings significantly degrade performance, which reflects the utility of LLM-derived external gene embeddings.\n\n(2) “Compare against a simple kNN or linear factorization method like MOFA for the imputation task.”\n\nWe have included the imputation results using MOFA+ in the same table (**Response Table 1**). The results are not competitive with sciLaMA models.\n\n(3) “What is the reason why you used scVI in the pseudotime estimation?”\n\nPseudotime inference (Section 4.4, Figure 5) was performed using the Palantir framework, but requires cell embeddings to be pre-computed as input. The goal of pseudotime inference in our manuscript was not to show that “sciLaMA is the best performing pseudotime inference method”, but to show “incorporating external gene information via LLMs into existing approaches improves their pseudotime inference”. We chose scVI because of its VAE-based inference, which matches sciLaMA’s VAE based inference, with the key distinction being that sciLaMA integrates gene information from LLMs. To further address this concern, we now included additional methods here to learn cell embeddings (**Response Table 2**).\n\nRegarding other figure-related comments:\n\nFigure 3c: We have provided a zoomed-in version at the following anonymous link (**Response Figure**:\nhttps://github.com/anonymous-ICML2025/rebuttal_April1st/tree/main/7H8M) and will include this in the revision.\n\nFigure 5b: No developmental time or cell state information was provided to any model. This ensures that differences in developmental states are captured only from gene expression and gene embeddings, which maintains a fair comparison.\n\nFor the rest of comments on notation and typos:\n\n(a) Yes, each gene’s $h_j$ has a dimensionality of $l$, which corresponds to $W^{cell}$ with dimensions $l \\times M$. We will clarify this in the manuscript.\n\n(b) We have corrected the typos pointed out by the reviewer in the revision.\n\n\n**Response Table 1** to answer `Methods And Evaluation Criteria` Q1 & Q2\n|Methods| PCC (↑) | SCC (↑) | JSD (↓) | RMSE (↓) |\n| ------------- | ------------- | ------------- | ------------- | ------------- | \n| sciLaMA-random (i) | 0.051±0.027 | 0.049±0.031 | 0.289±0.009 | 1.374±0.020 |\n| sciLaMA-shuffled (ii) | 0.056±0.036 | 0.043±0.037 | 0.288±0.009 | 1.366±0.027 |\n|sciLaMA-LLM (avg.)\t|0.222±0.027|\t0.217±0.028\t|0.283±0.008|\t1.242±0.022\n| MOFA+ | 0.173±0.025 | 0.192±0.023 | 0.466±0.036\t| 1.282±0.019 |\n\n**Response Table 2** to answer `Methods And Evaluation Criteria` Q3\n|Methods| ARI (↑) | NMI (↑) | ASW (↑) | cLISI (↑) |\n| ------------- | ------------- | ------------- | ------------- | ------------- | \n|scVI\t| 0.284\t|0.291\t|0.501\t|0.501 |\n|cellPLM |0.216\t|0.281\t|0.513\t|0.684 |\n|GenePT-w\t|0.000\t|0.017\t|0.409\t|0.514 |\n|scGPT\t|0.275\t|0.310\t|0.530\t|0.716 |\n|scGPT-finetuned\t|0.033\t|0.061\t|0.449\t|0.535 |\n|sciLaMA-scGPT\t|0.341\t|0.374\t|0.523\t|0.738 |\n|sciLaMA-ESM\t|0.299\t|0.369\t|0.530\t|0.833 |\n|sciLaMA-ProtTrans\t|0.355\t|0.366\t|0.529\t|0.775|\n|sciLaMA-GenePT\t|0.237\t|0.321\t|0.514\t|0.776 |\n|sciLaMA-ChatGPT\t|0.316\t|0.351\t|0.518\t|0.738 |\n|sciLaMA-CellPLM\t|0.343\t|0.381\t|0.524\t|0.793|", "claims_and_evidence": "The claims the paper is making, which are:\n\n\"(1) We introduce a novel framework that incorporates diverse, external gene knowledge from pretrained LaMs with scRNA-seq data, facilitating context-aware cell and gene representation learning\"\n\"(2) We demonstrate that our approach reduces computational requirements while improving performance compared to existing state-of the-art methods across various single-cell tasks\"\n\nare both underlined by the provided experiments.", "methods_and_evaluation_criteria": "- the proposed objective function seems solid. I believe the inference task was split into three training steps to improve robustness.\nThree further comments regarding the experiments:\n1) Robust cell representation learning and integration: It would be interesting to see how sciLaMA performs if the gene embeddings are random vectors (instead of none or LLM-based).\n2) Imputation task: it would be interesting to see a simple kNN and/or linear approach as a benchmark as well. E.g., a simple linear factor analysis based approach such as MOFA [https://www.embopress.org/doi/full/10.15252/msb.20178124] also in the context of the gene set enrichment analysis, the imputation and the embeddings.\n3) There are much more advanced tools than scVI to estimate pseudotime. Why did you go with scVI here?", "theoretical_claims": "There are no proofs, but also no necessity for any.", "experimental_designs_or_analyses": "- Figure 3c: Could you provide a zoom-in side by side comparison of the imputation task, to present the differences a bit more nuanced and see what/how was imputed?\n- Figure 5b: did you provide the time as an input to any of the models? (IP or EN, i.e. 0 or 1)", "relation_to_broader_scientific_literature": "%", "essential_references_not_discussed": "%", "other_strengths_and_weaknesses": "- I guess `h_j` is also of dimension l? Same for `W_{cell}`, whis is of dimension `l \\times M`? Clarifying this early in the manuscript helps to understand the paper quicker.", "other_comments_or_suggestions": "- \"discvovery\" -> \"discovery\"\n- \"that across\" -> \"across\"", "questions_for_authors": "Please see questions above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z9JeSZOEnR", "reviewer": "Reviewer_zJyw", "summary": "This paper proposes a new method for single cell embedding that leverages external textual information (or internal knowledge representation from large language models). The method works by considering a learnable matrix decomposition of the single cell data, where cell embeddings are produced from the transcriptomics data and the gene embeddings are produced from textual data. The authors then evaluate their methods on a wide range of downstream tasks, such as cell type clustering, gene expression reconstruction, marker gene identification, and trajectory analysis.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Below are our responses to the questions:\n\n(1) “Why do you not use siVAE for comparison?”\n\nWe now clarify this more explicitly in the manuscript. siVAE is functionally equivalent to sciLaMA-self-informed (sciLaMA-s.i.), which instead of using LLM-based external gene embeddings as the input to the gene encoder, sciLaMA-s.i instead uses the transposed gene expression matrix as input to the gene encoder. The two models share similar architectures, with some implementation differences: siVAE is built on TensorFlow, while sciLaMA is implemented in PyTorch. In addition to this, siVAE’s cell decoder produces probabilistic outputs (both mean and variance for reconstructing scaled gene expression), whereas sciLaMA’s cell decoder outputs only the mean with the assumption of unit variance after scaling. We introduced sciLaMA-s.i. as a non-LLM-based version specifically to reproduce siVAE, and we have clarified this more clearly in the revised manuscript.\n\n(2) “Explain sciLaMA-s.i. more clearly.”\n\nAs addressed in response to question (1), sciLaMA-self-informed (sciLaMA-s.i.) uses the transposed gene expression matrix as input to the gene encoder instead of LLM-based external gene embeddings. Based on the reviewer’s suggestion, we have revised the manuscript to provide a clearer explanation of this.\n\n(3) “Try using a learnable gene embedding with the same architecture.”\n\nIf we understand correctly, the reviewer is suggesting replacing the LLM-based external gene embeddings with randomly initialized values and allowing the model to learn contextual gene embeddings from scratch. If so, this question aligns with a similar request from another reviewer (Reviewer 7H8M, Q1), and we have included the corresponding results in our response there (**Response Table**: https://github.com/anonymous-ICML2025/rebuttal_April1st/tree/main/zJyw). If this is not what the reviewer intended, we would appreciate further explanation so that we can properly address the question and plan additional experiments if needed.\n\n**Response Table** to answer `Weaknesses` and `Questions For Authors` Q3\n|Methods| PCC (↑) | SCC (↑) | JSD (↓) | RMSE (↓) |\n| ------------- | ------------- | ------------- | ------------- | ------------- | \n| sciLaMA-random initialization| 0.051±0.027 | 0.049±0.031 | 0.289±0.009 | 1.374±0.020 |\n| sciLaMA (avg.)\t|0.222±0.027|\t0.217±0.028\t|0.283±0.008|\t1.242±0.022|", "claims_and_evidence": "The results shown in this paper suggest sciLaMA outperforms previous state of the art models on a large scope of tasks.", "methods_and_evaluation_criteria": "Evaluation criteria and experimental methods make sense.", "theoretical_claims": "There were no theoretical claims in this paper.", "experimental_designs_or_analyses": "The cell-level analyses are established benchmarks. The gene reconstruction task is also standard in the spatial transcriptomics literature.", "relation_to_broader_scientific_literature": "This paper extends previous works in single cell embedding such as scVI and foundation models for single cell data such as scGPT.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper addresses an interesting challenge of leveraging large language models in single cell representation learning. The authors also produced an in-depth analysis of their method.\n\nWeaknesses:\n\n- Despite noting the inspiration from siVAE, the authors do not compare against that method. Could the authors explain why ? \n- The sciLaMA-si is not clearly explained. What input is used in this case ? In my opinion, the most meaningful baseline would be to use a learnable gene embedding using the same architecture. That is, making $\\mathbf{g}_j$ learnable, everything else being equal. This would represent the most direct assessment of the impact of external knowledge base.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "1. Despite noting the inspiration from siVAE, the authors do not compare against that method. Could the authors explain why ? \n\n2. The sciLaMA-si is not clearly explained. What input is used in this case ?  Could the authors clarify this here and in the text ?\n\n3. In my opinion, the most meaningful baseline would be to use a learnable gene embedding using the same architecture. That is, making $\\mathbf{g}_j$ learnable, everything else being equal. This would represent the most direct assessment of the impact of external knowledge base. Could the authors argue against this, or compare against it if indeed relevant ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "hTnJg5jjqr", "reviewer": "Reviewer_mrfK", "summary": "The authors propose a new method that combines gene embeddings from LLMs with scRNA-seq data to generate context-aware representations for cells and genes. This approach outperforms existing methods in batch effect correction, cell clustering, and gene module discovery, while remaining computationally efficient. By incorporating biological knowledge, sciLaMA enhances single-cell analysis and improves interpretability in gene expression studies.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The reviewer mentioned that we did not provide enough evidence to illustrate how text information contributes to cell representations. In the manuscript we showed text-based gene information helped with removing batch effects (while preserving biologically meaningful information through sciLaMA framework). We are uncertain about the criticism “They only evaluated that text information can enhance the gene expressions in the clustering task”, could the reviewer kindly elaborate further? \n\nFor the criticism “with text information, it is worse than without in ASW metrics”, in the single cell field, it’s widely acknowledged that no single metric holistically captures good performance and multiple metrics are often used for evaluation. As other reviewers have motioned, our work is evaluated on a wide range of downstream tasks and contains experiments on multiple tasks with extensive evaluations. We beat other methods in cell type clustering accuracy tasks (ARI and NMI) and integration tasks (batchASW, and iLISI), only performing worse than fine-tuned scGPT in the ASW metric (cell type separation). The fact that fine-tuned scGPT does a great job on cell type separation alone does not imply good overall performance because a method can high ASW when it separates both cell types and batches, which is undesirable (batches should be well mixed). That is why we measure both batchASW and iLISI to evaluate the integration performance more holistically and keep a good balance between removing batch effects (batches should not be separated) while preserving biologically meaningful information (cell types should be well separated). \n\nTo the comment “the paper does not mention how to embed the molecular functions, but they show it in the framework figure”. We clarified in the manuscript that such information is derived from gene text descriptions, either sourced from ChatGPT-like LLMs responses, or NCBI GeneCard summaries (similar to GenePT and scELMo). If the reviewer is specifically referring to line 350 (right column, page 7), there obtained molecular function data directly from the Gene Ontology (GO) database. \n\nThe review also criticized the experimental design by stating “the efficiency of cell representation should be evaluated by zero-shot setting in the cell classification”. We did not perform cell classification using zero-shot learning because the approach is inherently biased, and instead we followed well-established benchmarking frameworks including scIB (https://doi.org/10.1038/s41592-021-01336-8), as well as what have been mentioned in https://doi.org/10.1186/s13059-019-1850-9 by Tran et al. Both studies indicate strong rationales for using our selected metrics. We are not aware of any publication that explicitly argues for zero-shot cell classification, but would be happy to review any such papers and learn further insights if the reviewer has some in mind. We also looked at the scDiff paper by Tang et al., which the reviewer suggested us to cite. This paper covered “one-shot cell type annotation”, where the prior knowledge of cell type was utilized. This makes us unsure how “zero-shot setting in the cell classification” fits to our study, as our model does not incorporate cell type prior knowledge, and “zero-shot classification” is not one of our goals in this work.\n\nAnother critique was that “baseline models are setting to zero-shot”, which is not true, as we included fine-tuned scGPT in our main figure and table. The reason why we did not fine-tune CellPLM was because running their code yields the error 'Currently CellPLM only supports zero shot embedding instead of fine-tuning’.\n\nFor the spatial experiments, the reviewer stated that “they only include the non-pretrained models”. But every benchmarked model in this part was trained on the same dataset. To address the reviewer’s concern, we now compare against the pretrained foundation model CellPLM spexifically, and have added the results (**Response Table**: https://github.com/anonymous-ICML2025/rebuttal_April1st/tree/main/mrfK). \n\nFor the reason why we compared sciLaMA and scVI in Table 4, please refer to (Reviewer 7H8M, Q3), and we provided more methods there.\n\nThe reviewer stated, “The authors claim that they can use prior knowledge to enhance performance, but the experiments are not strong enough to support their claims.” Could the reviewer kindly clarify what would constitute strong enough evidence?  In one of our head-to-head comparisons, we evaluate models with and without prior knowledge using the same framework. Our results demonstrate that incorporating prior knowledge improves both performance (Tables 1, S6, S4, S7, Figure 2) and computational efficiency (Table 2). \n\nFinally, the reviewer also mentioned that the scDiff paper is one essential reference we did not discuss, we would like to cite this paper in our revision.", "claims_and_evidence": "The authors do not provide enough evidence to illustrate that the text information to the cell representations. They only evaluated that text information can enhance the gene expressions in the clustering task, and with text information, it is worse than without in ASW metrics.", "methods_and_evaluation_criteria": "The paper does not mention how to embed the molecular functions, but they show it in the framework figure. The experiments do not show the efficiency of the molecular functions.", "theoretical_claims": "The paper does not have the theoretical claims.", "experimental_designs_or_analyses": "I’ve checked the experiments. The experiments are not well designed. For example, the efficiency of cell representation should be evaluated by zero-shot setting in the cell classification, as the clustering task is weak in showing the results. Besides, it is wired that other baseline models are setting zero-shot, but the proposed method is trained. The following experiments are still weak. For example, in the spatial transcriptomic data comparison, they only include the non-pretrained models. However, at least a pretrained model, CellPLM, also includes the spatial transcriptomic data in the pertaining data, which needs to be considered. Similar to Table 4, it is too weak to only compare with scVI.", "relation_to_broader_scientific_literature": "The contribution is limited. The reconstruction framework, like scVI and CellPLM, is widely used in the single cell field. The authors claim that they can use the prior knowledge to enhance the performance, but the experiments are not strong enough to support their claims.", "essential_references_not_discussed": "single cell with prior knowledge have been considered in [1]\n\n[1] Tang W, Liu R, Wen H, et al. A general single-cell analysis framework via conditional diffusion generative models[J]. bioRxiv, 2023: 2023.10. 13.562243.", "other_strengths_and_weaknesses": "no", "other_comments_or_suggestions": "no", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VoZbjICP64", "reviewer": "Reviewer_H2Qr", "summary": "The paper introduces a VAE approach to integrate external gene knowledge from pretrained large language models (LLMs) with single‐cell RNA sequencing (scRNA‐seq) data. By adapting static gene embeddings (derived from multiple modalities such as textual descriptions, protein sequences, and single‐cell models) into a context‐aware paired-VAE architecture, sciLaMA generates latent representations for both cells and genes. This combined embedding is then used for single-cell downstream tasks including cell clustering, batch effect correction, gene expression imputation, marker gene identification, and trajectory analysis. Benchmarking on several datasets demonstrates that embeddings from sciLaMA outperforms other approaches in terms of predictive accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We address each of the concerns below with response figures and tables: https://github.com/anonymous-ICML2025/rebuttal_April1st/tree/main/H2Qr: \n\n**Interpretability**:\nThe reviewer is correct that interpretability primarily relies on implicit gene module detection and clustering in the current manuscript. The alignment regularization term in our objective function is designed to align each dimension of cell and gene embeddings to ensure meaningful correspondence between them. In doing so, we are ‘encouraging’ genes with high weight in a particular dimension of the gene embeddings to explain variation in cell embeddings in the corresponding dimension (Response Fig 1).\n\n**Dataset scope**:\nWe included a diverse set of datasets in this study, including human pancreatic, mouse spatial, human PBMC, and mouse neuronal datasets. We understand that expanding the dataset scope could further strengthen our findings, so in the revision we further incorporated additional datasets [See 5].\n\n**Sensitivity to hyperparameters**:\nIn the revision, we tested a range of hyperparameter values [See 6].\n\n(1) “Matrix factorization method might suffice.”\n\nWe did an experiment using a factorization-based approach in response to another reviewer, and showed sciLaMA is better performed (Reviewer 7H8M, Q2). Regarding using siVAE representations with LLM embeddings, this is conceptually similar to what sciLaMA is doing, as we clarified with one of the previous reviewers. sciLaMA can be considered as similar to siVAE but starting with LLM embeddings instead of learning them only from the training scRNAseq data. \n\n(2) “Detailed discussion on the robustness of the embedding alignment process when the external gene embeddings are noisy or incomplete.”\n\nThis is a great point, which is why we included multiple types of external gene embeddings across different modalities. For instance, text-based gene embeddings are derived from ChatGPT responses or NCBI GeneCards summaries, protein sequence embeddings are derived from protein sequences through ESM and ProtTrans, and for single-cell foundation models, we utilize both scGPT and CellPLM to get static gene embeddings. To assess robustness, we evaluated sciLaMA using embeddings from all 3 modalities. The relatively small performance variance across these variants, as seen in our results, suggests sciLaMA is robust to noise in the input LLM embeddings. We also did experiments with additional hyperparameter choices for the alignment process to further validate this robustness [See 6].\n\n(3) “The external gene knowledge may not always align perfectly with the biological nuances present in scRNAseq data.”\n\nWe agree, it is a limitation of external LLMs, but we consider this a strength of sciLaMA. Different biological contexts may require different knowledge sources, and sciLaMA allows users to explore multiple modalities and determine which functional context is most relevant to their dataset. We view our approach as a hypothesis testing strategy for checking which context is most important.\n\n(4) “Quality and bias of these pretrained embeddings can propagate into the sciLaMA model.”\n\nAs mentioned above, sciLaMA provides flexibility in choosing external gene embeddings from different modal-specific LLMs, which enables users to select embeddings most relevant to their dataset, which overcomes potential biases introduced by any single knowledge source.\n\n(5) “Evaluation focus remains on specific tissues and conditions, unclear to show how well it can be generalized to other contexts.”\n\nIn this manuscript, we have included data from diverse sources as described in [Dataset scope], and now further added comprehensive evaluations on two more datasets across diverse methods. We demonstrated sciLaMA robustly performs well in these datasets too (Response Fig 2-3, Repones Tables 1-2)\n\n(6) “Sensitivity of sciLaMA to hyperparameters such as latent space dim, KLD weight, and gamma.”\n\nFor KLD, we follow the standard VAE framework and apply a beta value of 1, with a beta warm-up mechanism during training.\nFor gamma, we set it to a relatively small value to prevent the alignment regularization term from dominating the loss function. Below, we present results using different gamma values (Response Fig 2, Response Table 3).\nFor latent dim, we also evaluate performance across different values of $K$, as shown in the Response Fig 3, Response Table 4.\n\n(7) “Gene identification and trajectory analysis rely primarily on visualizations and cluster-level metrics. Provide a quantitative measure.”\n\nWe have included quantitative validation metrics for gene module identification (Table S7). For gene trajectory analysis, we applied the Curvature-Based Smoothness Score (Response Table 5), showing after alignment, gene trajectories are more consistent than before.\n\nWe also appreciate the reviewer’s careful attention to writing consistency and notation. We will thoroughly revise the manuscript to improve clarity and ensure consistency.", "claims_and_evidence": "Paper provides evidence for improvements in cell-level clustering, gene imputation, and computational efficiency compared to existing approaches. However, the claims regarding enhanced biological interpretability and broad generalizability are less convincingly supported, as they rely mainly on qualitative visualizations and indirect metrics without robust independent validation.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are mostly suitable. However, the evaluation of biological interpretability remains somewhat indirect and might benefit from further targeted validation.", "theoretical_claims": "No theory in the paper.", "experimental_designs_or_analyses": "Yes I reviewed the experimental designs. Overall, these include multiple benchmark datasets and relevant quantitative metrics, but potential issues include the reliance on indirect measures of interpretability, sensitivity to hyperparameter tuning in the multi-stage training process, and a relatively narrow dataset scope that may limit generalizability.", "relation_to_broader_scientific_literature": "sciLaMA builds upon the foundational work of VAEs in single-cell analysis. It addresses a key limitation of these models (the inability to integrate external gene knowledge) by incorporating pretrained gene embeddings from LLMs.", "essential_references_not_discussed": "not the one I am aware of", "other_strengths_and_weaknesses": "### Strengths:\n\nBy incorporating precomputed static gene embeddings from diverse modalities (text, protein sequence, and single-cell foundations), the framework enriches gene representation beyond what traditional VAEs (such as scVI or siVAE) offer. \n \nThe paired encoder-decoder architecture for both cells and genes allows sciLaMA to jointly learn context-aware representations.\n\nExperiments on multiple tasks with extensive evaluations.\n\nPseudotime analysis demonstrates its utility in uncovering biologically relevant gene modules and temporal dynamics.\n\n### Weaknesses\n\nThe motivation to combine features from a VAE with those from large language models (LLMs) seems somewhat arbitrary. One could argue that simpler methods, such as matrix factorization or even using siVAE-derived representations combined with LLM embeddings, might suffice. It is not clear what is an added value of using a VAE based probabilistic latent space.\n\nThe process of adapting static, fixed-length gene embeddings into a context-aware space through a paired-VAE is complex. The paper would benefit from a more detailed discussion on the robustness of the embedding alignment process, especially in scenarios where gene annotations or external descriptions are noisy or incomplete.\n\nThe framework depends heavily on static gene embeddings produced by LLMs. These embeddings capture information from texts, protein sequences, or prior single-cell studies, but their relevance to the intricacies of gene regulation and cellular context may be limited. In other words, the external knowledge may not always align perfectly with the biological nuances present in scRNA-seq data.\n\nFurthermore, the quality and bias of these pretrained embeddings can propagate into the sciLaMA model, potentially skewing downstream interpretations if the external source is not well-calibrated for the target biological context.\n\n\nThe evaluation focus remains on specific tissues and conditions. It is unclear how well sciLaMA generalizes across other biological contexts or more heterogeneous datasets. A broader set of experiments could reinforce claims of universality.\n\nDoes not thoroughly explore the sensitivity of sciLaMA to key hyperparameters (e.g., latent space dimensionality, KL-divergence weights, alignment loss weight γ). \n\n The framework is presented as yielding “biologically interpretable” gene modules and cell representations. However, the review of marker gene identification and trajectory analysis relies primarily on visualizations and cluster-level metrics. Perhaps, a quantitative measure?", "other_comments_or_suggestions": "Typos or other errors:\nIn several places, terms like “large Language Models” are used inconsistently (“large language models” or “LLMs” or “large LaM”, “LaM”); it is preferable to use them consistently.  \nLine 156 page 3 the gene-level quantities are indexed with “j” and sometimes with “i” (e.g. “μ_gene_i”). Likewise in step 2. This appears to be a notational inconsistency that could confuse readers regarding whether the reconstruction error is computed per cell or per gene.\n\nWhile not an error per se, the paper would benefit from a clearer, more unified notation that explicitly distinguishes between cell-level and gene-level variables throughout the loss functions. \n\nFor consistency, “cell by gene ” → “cell‐by‐gene “\n\nLine 99 on the right side, it is unclear what “adapts and contextualizes” precisely means in this setting.\n\nLine 207 page 4, “correspond meaningfully” is not quantified. It is unclear how one would assess whether a specific dimension of the cell latent space aligns with that of the gene latent space, or what “meaningfully” means in a rigorous sense.\n\n“aligning each dimension” is not elaborated upon. Does this imply a one-to-one correspondence between latent dimensions, or does it involve a more complex mapping? More detail would help clarify the author's intention.\n\nLine 259 and 260 make it clear “self-informed” → s.i. \n\nsciLaMA (s.i.) is not immediately obvious how this variant is constructed or what “self-informed” entails compared to the other variants that integrate external knowledge.", "questions_for_authors": "Already covered above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents sciLaMA, a novel framework for single-cell RNA sequencing analysis that integrates pretrained gene embeddings from large language models (LLMs) with a variational autoencoder-based architecture. The core idea, leveraging external textual and biological knowledge to improve context-aware representation of genes and cells, is timely. The method is evaluated across multiple downstream tasks, including cell clustering, gene imputation, and trajectory inference, and demonstrates competitive or superior performance compared to existing approaches.

While some concerns were raised regarding baseline comparisons, evaluation depth for interpretability, and broader generalizability, the authors have proactively addressed these in their rebuttal, providing additional experiments and clarifications. Remaining points for improvement, as emerged during discussion, are:
- a careful evaluation using classification tasks 
- better description of the novelty, as previous work has already demonstrated the utility of injecting text information
- better articulate the rationale behind the chosen approach: why was this particular model adopted

Overall, this is a well-executed and promising contribution that will be of interest to the machine learning and computational biology communities. I recommend acceptance.
Decision: Accept (poster)