Contextualizing biological perturbation experiments through language
Authors: Menghua Wu
Russell Littman
Jacob Levine
Lin Qiu
Tommaso Biancalani
David Richmond
Jan-Christian Huetter
Venue: ICLR 2025 Poster (2025)
Keywords: large language models
Perturb-seq
perturbation experiments
knowledge graphs
retrieval-augmented generation
chain of thought prompting
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose Perturb-seq predictions as a novel set of real-world tasks for large language models and provide a proof-of-concept method with favorable performance.

Abstract: High-content perturbation experiments allow scientists to probe biomolecular systems at unprecedented resolution, but experimental and analysis costs pose significant barriers to widespread adoption. Machine learning has the potential to guide efficient exploration of the perturbation space and extract novel insights from these data. However, current approaches neglect the semantic richness of the relevant biology, and their objectives are misaligned with downstream biological analyses. In this paper, we hypothesize that large language models (LLMs) present a natural medium for representing complex biological relationships and rationalizing experimental outcomes. We propose PerturbQA, a benchmark for structured reasoning over perturbation experiments. Unlike current benchmarks that primarily interrogate existing knowledge, PerturbQA is inspired by open problems in perturbation modeling: prediction of differential expression and change of direction for unseen perturbations, and gene set enrichment. We evaluate state-of-the-art machine learning and statistical approaches for modeling perturbations, as well as standard LLM reasoning strategies, and we find that current methods perform poorly on PerturbQA. As a proof of feasibility, we introduce Summer (SUMMarize, retrievE, and answeR, a simple, domain-informed LLM framework that matches or exceeds the current state-of-the-art. Our code and data are publicly available at https://github.com/genentech/PerturbQA.

Published as a conference paper at ICLR 2025
CONTEXTUALIZING BIOLOGICAL PERTURBATION EXPERIMENTS THROUGH LANGUAGE
Menghua Wu"
Russell Littman, Jacob Levine Biology Research & Al Development, Genentech South San Francisco, CA, USA
Massachusetts Institute of Technology Cambridge, MA, USA
Lin Qiu
David Richmond, Tommaso Biancalani, Jan-Christian Hütter*
Meta AI
Biology Research & Al Development, Genentech South San Francisco, CA, USA
Menlo Park, CA, USA
ABSTRACT
High-content perturbation experiments allow scientists to probe biomolecular sys- tems at unprecedented resolution, but experimental and analysis costs pose signif- icant barriers to widespread adoption. Machine learning has the potential to guide efficient exploration of the perturbation space and extract novel insights from these data. However, current approaches neglect the semantic richness of the relevant biology, and their objectives are misaligned with downstream biological analyses. In this paper, hypothesize that large language models (LLMs) present ural medium for representing complex biological relationships and rationalizing experimental outcomes. We propose PERTURBQA, benchmark for structured reasoning over perturbation experiments. Unlike current benchmarks that primar- ily interrogate existing knowledge, PERTURB QA is inspired by open problems in perturbation modeling: prediction of differential expression and change of di- rection for unseen perturbations, and gene set enrichment. We evaluate state-of- the-art machine learning and statistical approaches for modeling perturbations, well as standard LLM reasoning strategies, and we find that current methods per- form poorly PERTURBQA. As proof of feasibility, introduce SUMMER (sUMMarize, retrievE, and answeR), a simple, comain-informel LLM framework that matches exceeds the current state-of-the-aut
INTRODUCTION
A fundamental paradigm for discovering causal relationships in molecular biology is intervention followed by measurement. Recent experimental methods like Perturb-seg allow biologists to ma- nipulate the RNA and protein expression levels of each gene, and read out the effects on every other gene (Diit et 2016, DDalinger et al., 2017 Replogle et al.] 2022 While these experiments promise large-scale, unbiased insights, the measurement modality (single-cell sequencing) poses significant cost burden and yields datasets of varying statistical power (Nadig et al., 2024). These challenges motivate in-silico approaches for predicting cellular responses to novel perturbations, and for automatically extracting high-level findings from perturbation data.
Current approaches for perturbation response prediction generalize to unseen perturbations by con- necting them to perturbations that have been seen, often via knowledge graphs (Roohani et al.] 2023). However, these approaches reduce textually-rich biological relationships to adjacency matri- ces, leading to loss of information. Furthermore, these methods are trained to regress the change in levels of genes upon perturbation: task that is precursor, but does not directly translate to down- stream analyses like differential gene expression (Love et al. 2014) and gene set enrichment (Subra-) manian et al. 2005). Finally, most existing methods are black-box, revealing little about the learned biology without post-hoc probing.
" "Correspondence rmwu mmit edu and huetter. juatteer.jaacrriitiii---aa ggene com. Work completed while employed at Genentech.
Our code and data publicly available at

---

Published as a conference paper at ICLR 2025
A) PERTUREATION EXPERIMENTS
B) BIOLOGICAL KNOWLEDGI GRAPHS
PERTURB QA
Real-valued gene expression
Genes pathways
Description of gene:
Differential expression
Genes
subfenily
binding cassette
is knockdown of ABCE1 K562 cells Iikey to result differential expression of GABARAR?
Control
Quaternary structure: (Microbial
with proteins and Gag Function: [.]
Direction of change
Differential gene expression analysis
Perturbations
Is knockdown of ABCE1 K582 callss Ikaly result decrease or Increase of GABARAP?
Biological relationships
Relations to other genes pathways:
ABCE enables rrnn a sulur cluster sinding, AlP binding AIP hycretyst sctivtty C1Pase
single-cell transcriptomics datascts
Gene sct enrichment
Down N/A
actirly, Gilese actvty. L-]
evidence experimenta evidence humans may physically intorct
Gene set enrichment
How are the following genes related, and why do they respond similarly to parturbation? CFLAR, VIM, CAPG.
IXX,
EIF3C, mitcchondrial matri,
Gene sets
Interpretation via discrete outcomes
inhibiting
arphabbtta signaling
algha [-]
sets of unnotations
A) Perturb-seq experiments result in a matrix of gene expression levels, which are inter- preted through discrete outcomes. B) Textually-rich, biological know graphs can help explain these outcomes. C) Based on this premise, PERTURBQA introduces three tasks: predicting differen- tial expression and direction of change for unseen perturbations, and summarizing data-driven gene clusters into cohesive sets.
Figure I:
We posit that language is natural medium for traversing the structured, biological knowledge rele- vant to perturbation experiments. Based on this hypothesis, we propose PERTURBQA, a set of bio- logical tasks that query discrete outcomes of perturbation experiments through question-answering For example, rather than predicting the realevalued change in gene, we might ask, "does perturbation p cause differential expression of g?" These tasks inspired by the standard analysis pipeline for interpreting perturbation experiments, and compared to most biological reasoning benchmarks (Rein)
al. 2023). ) they predictive in nature. Ground truth labels are derived from five high quality
single-cell RNA sequencing datasets with CRISPR interference (CRISPRi) perturbations (Replogle)
al. 2022 Nadig et al.) 2024), based strict statistical considerations. Evaluation of state-of-the- art statistical, graph, and language-based methods reveal that these tasks are still far from solved.
To demonstrate that language-based reasoning can be effective these tasks, develop sim- ple LLM-based framework that matches or exceeds the current state-of-the-art on PERTURBQA. SUMMER (SUMMarize, retrievE, and answeR) is an inference-tine strategy that incorporates stan- dard LLM techniques alongside experimental data and biological knowledge graphs. An LLM is first asked to summarize textual descriptions associated with genes as well their impacts on and influences from other biological entities. In addition to "featurizing" genes, this step can be applied iteratively to characterize clusters of genes that exhibit similar responses or effects. Next, inspired by retrieval-augmented generation (Gao et al.) 2023), retrieve perturbation--gnee pairs from existing experimental data based on knowledge graph proximity. While perturbation exper- iments are not textual in nature, their discretized outcomes ground the LLM's reasoning and prevent hallucinations. Finally, inspired by chain-of-thoughh (Wei et af., (002)), the LLM answers the biological question via guided prompting. incorporating the knowledge graph summaries and retrieved experimental outcomes. To summarize, our contributions are three-fold.
1. We propose that biological perturbations should be modeled the level of discrete out- comes that reflect downstream analyses, and that language models are suitable for capturing the relevant biology.
2. PERTURBQA is set of real and currently unsolved tasks that evaluate how models reason over textually-rich, structured knowledge to discover new biology. We find that the cur- rent sfate-of-the-art performs poorly on this benchmark, which we hope will increase the accessibility and interest in machine learning modeling for biological perturbations.
3. We introduce SUMMER, a domain-informen LLM baseline, which matches exceeds the state-of-the-ar without any finetuning. SUMMER is implemented using a lightweight 8B model and operates natively in biologist-imterppetablle language.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Predicting perturbation responses Since experimental costs scale with the number of experi- mental contexts (cell lines) and perturbations, a number of works have been proposed to infer the cost-intervention distribution of cells. Their goal is to generalize to unseen perturbations (Roohani ct al.
2023 Bai et al., [024] Märtens et al., 2024), or unseen contexts (Bunne et al. 2023 Lotfollahi
et al. 2019). This paper focuses on the former setting, as we aim to optimize, not replace, experi- ments. An orthogonal direction is to predict the combined effects of multiple perturbations (Roohani et al. 2023 Gaudelet et al., 2024 Lotfollahi et al. (002)). While these models could be particularly helpful for alleviating experimental burden and rationally designing drug combinations, there are limited evaluation data for combinatorial perturbations (< 150 pairs in Norman et al.) (2019)). Since our goal is to create a trustworthy benchmark for perturbation modeling, we choose to focus on single gene perturbations, and leave this as an opportunity for when better datasets are available.
Language modeling and biology LLMs have been applied to many siology-adjaceed tasks, with several relevant directions included here. Biological question-answering (Hendrycks et al. 2021 Hao et al. 2024a) and scientific coding (Laurent et al.] 2024) Hou & Ji. (023) are common bench- marks assess LLM reasoning, but these works primarily focus on tasks that human specialists are already able to do. Hsu et al. (024) uses LLMs to predict Gene Ontology terms (Ashburner et al.) 2000) associated with known gene sets. In contrast, the focus of our gene set enrichment task is to characterize data-driven gene clusters, which may not be significantly enriched for any known gene set, but are of biological interest to understand. Finally, LLMs have been used towards active experimental design (Roohani et al.) 2024 Qu et al. 2024).
More broadly, there are a number of single-cell (Rosen et al. 2024 Hao et al. 2024b Kalfon al, 2024 Cui et al.] 2024) and biological sequence (Lin et al.) [023] Nguyen et al. 2024 Schiff et al., 2024 foundation models, trained over raw biological data (gene count matrices, protein/DNA se- quences). In this paper, approach biological knowledge through natural language. but multimodal integration of foundation models could be a promising future direction (Wang et al. 2024b).
Language-based reasoning and structured knowledge Significant research efforts have focused on improving LLM reasoning and reliability. Chain of thought (Wei et al. (022) first demonstrated that explicitly instructing L.LMs reason enables them solve much more complex tasks, com- pared to directly asking for answers. Subsequent works have explored how to navigate the space of "thoughts," " or in-progress generations (Yao et 2024 Zhang et al.. 2024). Orthogonally, retrieval augmented generation (RAG) Lewis et al. (2020) was developed to increase LLM reliability. Rele- vant documents are first identilied from corpus, typically based on an embedding index, to act as
source material for reasoning. Instead of querying an index, Graph RAG (Edge et al. 2024) sum-
marizes corpuses into hierarchical graph structures, for richer language- based comparisons. While these methods have seen significant success in natural language applications (Gao et al. [023] [iang jet al., 2023), they are less straightforward to implement in biology, where the vast majority of pa- pers are inaccessible behind paywalls, and only abstracts are available (Wang et al.) (024a). As result, document retrieval- based methods are difficult to evaluate in this paper's s setting. Our method primarily retrieves experimental outcomes (binary labels), rather than relevant literature.
Textually-rich knowledge graphs have been probed through language, both in the classical natural language processing literature (Guu et al., 2015) and in modern LLM literature (Jin et al.] 2024] Besta et al. 2024). Language can provide embeddings to be processed by downstream graph models, or directly serve the medium for prediction (Chen al. 2024 Kau al. 2024). In this work, we take the latter approach, which opens several design choices. Graphs can be embedded alongside text via parameter-efficite
finetuning (He et al. [2024] Perozzi al. 2024). directly serialized into
language (Zhao et al. 2023). inform retrieval (Mavromatis & Karypis 2024), or any combination of these options. To minimize the computational cost of our proof of concept, we serialize graph-based knowledge into text and use graph structure to inform retrieval.
BACKGROUND
Modeling perturbations A perturbation experiment can be represented by matrix X NXD. where N is the number of perturbations, D is the number of measured entities, and entries 2p.g E X represent the change in levels of entity g under perturbation p, relative to control po (Figure A)
3

---

Published as a conference paper at ICLR 2025
For example, in CRISPRi Perturb-seq experiment (Replogle et al. 2022) the level of gene p is decreased, and the resultant change Fp.y in gene g is measured, for all genes g E G.
Roohani et al. (202) (GEARS) first proposed the task of predicting unseen perturbation outcomes in Perturb-seq data. Given Xuain S X, whose rows correspond to perturbations Pirain. their goal was complete the rows Xies, corresponding to unseen perturbations Piest To generalize to Pies. GEARS and subsequent works leverage knowledge graphs that relate the two sets of perturbations B. Specifically, they address a node-level regression task, over the graph G EV, where is a set of biological entities (e.g., genes and pathways) and E is a set of relationships.
(Figure D
There are several aspects of the prevailing formulation that diverge from the findings biologists derive from these experiments. First, when converting knowledge graphs into adjacency matrices, the semantics of each edge are discarded, as they are typically annotated in free text (Ashburner et al., (000). This loss of information may negatively impact model performance, especially in finite data regimes, as biological knowledge graphs often contain hierarchical relationships of conflicting semantics. Second, a common objective (and metric) is the real-valued error between the predicted and true responses, computed over genes that actually do respond (Roohani et al.) 2023 Bai et al.) 2024). However, these genes are not known prior to actual experimentation, and their identity is of high biological interest. Log-fold change is also known to be noisy, and can inconsistent across biological replicates (Nadig et al. 2024). Finally, the goal of perturbation experiments is to understand the underlying biology, but current methods focus solely on recapitulating the data distribution, errors in which may propagate to downstream analyses. These considerations motivate the creation of PERTURBQA, which is centered around higher level outcomes, whose significance can be statistically quantified.
Biologists draw conclusions of the form "p impacts gene or pathway g" through statistical techniques like differential expression (Love al. 2014) and gene set (Subrama- nian et al. 2005) analyses. In differential expression analysis, one assumes that I ~ Pa. where Pa often taken be approximately normal (Cui & Churchill] 2003) or negative binomial (Love al., 2014
Statistical conclusions
Ahlmann-Elizz & Huber 2020). The goal is to test between
Ho Fp.g
and H1 : :p.s +
where rejection of Ho translates to "y is differentially expressed under perturbation compared to the control perturbation po-" Differentially expressed genes may also be assessed by their direction of change, i.c. 2p.9 $ O.
Due to biological and technical noise, the measurement of individual genes may be unreliable, moti- vating statistical analyses at the level of gene sets. A data-driven approach for identifying gene sets to cluster the rows and/or columns of the expression matrix X and test whether more members of well-characterized sets are present in these clusters than expected by chance (Huang et al.) 2008). While these "enriched" gene sets serve as the basis for annotating data-driven clusters, they do not consider the context of each experiment, e.g., the profiled cell line. Furthermore, significance cutoffs are difficult to assess, as the inclusion or exclusion of genes in gene sets was determined manually. As a result, data-driven clusters may exhibit consistent behavior in the experiment but fail to be enriched for known biological phenomena, thus eluding annotation (Replogle et al. 2022).
4 CONTEXTUALIZING BIOLOGICAL PERTURBATIONS
Our hypothesis is that traversing biological knowledge through language not only enables us to pre- dict perturbation effects, but also rationalize perturbation outcomes. We develop PERTURBQA. benchmark to assess structured reasoning over semantically-riee graphs, in the context of molecular biology (Section 4.11. These tasks are non- nriviai both for graph- methods and naive large lan- guage model (LLM) applications (Section 6). To validate our hypothesis, we introduce SUMMER, a simple I.I. approach that matches or exceeds the current state-of-the-art on PERTURBQA. by considering experimental outcomes in the context of domain knowledge (Section
4.1 PERTURBQA
PERTURBQA is composed of three primary tasks evaluated over five real datasets (Figure These tasks reflect the experimental and computational workflow associated with perturbation experiments.
4

---

Published as a conference paper at ICLR 2025
A) Label distribution for DE and Dir
Knowledge graph size
K362
GO
RPEI
CORUM
HepG2
STRING
Jurkat
Reactome
K562-Set
BioPlex
10000
1000 2000 Direction of change
Differential expression
20000
500000
Nodes
Edges
B) Gene set cluster sizes and examples
D) Proportion (p, g) with known interaction
(physical or network effects), stratified by y
m6λ mRNA methylation, nonsense mediated decay, listone acetylation.
Pattrr,
K562
RPE1
HepG2
Jurkat
mitochondrial respiratory chain, targets of nonsonse-mediatted docny, unfolded protein response,
0.094
0.063
0.075
0.106
physical
Gene
0.032
0.025
0.027
0.029
0.214
0.204
0.218
0.253
network 0.222
# Genes in cluster
Human annotations
0.209
0.220
0.208
Figure 2: PERTURBQA dataset statistics. A) Differential expression and direction of change. B)
Distribution of genes per cluster (gene set enrichment), with sample annotations. C) Knowledge graph sizes. D) DE genes are more likely to interact physically, but presence of interaction mini- mally predictive (Table T. There is little difference in network connectivity.
Differential expression: Given a pair of entities g), the goal is to predict wet E (0, 1}. where 0 indicates that perturbing results in no change and 1 indicates that perturbing p results in differential expression of g.
dir E {0, 1}, 2. Direction of change: Given a pair of entities (p.g), g). the goal is to predict Vp.o
where indicates that levels of g decrease under perturbation p, and indicates that levels of increase. This task is only evaluated on pairs for which =
3. Gene set enrichment: Given set of genes P. the goal is to identify description s that characterizes why members of P exert a consistent effect when perturbed ("perturbation cluster"), or respond similarly to perturbation ("gene cluster"). As ground truth, we lever- age expert gene set annotations, published by the authors of Replogle et al. (022).
Perturb-seq datasets We constructed our benchmark based on five Perturb-seq datasets, derived from Replogle al. (2022) and Nadig al. (222)). For each dataset, identified differentially- expressed genes (DEGs) per perturbation using the Wilcoxon signed-rank test (Wilcoxon) 1945). resulting pairs (p: 9) with associated labels vo., and upt. Datasets are 75:25 into train and test along the perturbation axis, with similar distributions of number of DEGs. To ensure label quality, set a rigorous cut-off for DEGs and non-DEGs based on consistency across biological replicates and/or adjusted p-value (details and statistical analyses in Appendix A.2). The label distribution on the test set is depicted in Figure A.
Differential expression and direction of change are assessed at the granularity of single genes (K562, RPE1, HepG2, Jurkat) and gene sets (K562-Set), where the gene set is represented as single entities, with the mean expression of their constituents. Gene set enrichment is evaluated over K562-Sett, where human annotations are taken as the ground truth. Figure 2 illustrates the distribution of the cluster sizes and example annotations.
Domain knowledge PERTURBQA tests whether models can effectively leverage structured domain knowledge and contextual information. Thus, in addition to test examples, we provide:
Harmonized and parsed knowledge graphs, with identifiers aligned to the perturbation data (F. These provide high-quality, biological insights to aid reasoning.
2. Train examples (observation outcomes), to be used retrieval corpus for model train- ing. These may be useful for conditioning the predictions on each dataset, as perturbation responses may differ by cell line (Nadig al. 2024
5

---

Published as a conference paper at ICLR 2025
A) SUMMARIZE
BY RETRIEVE
CJ ANSWER
Ippt (fom
Ferurbed gene ABCE1
LLM
P GREI
Gere offilerest: BABARAR
Enample
Perturbed gene: RUS14
P"
Neighbochood perturbation
Petturbation summary
Gene Infernttt FAME3A
Easmple
Chain-of Chaln-af-thughh (CoT) prompts:
1) Which pentubed genes mhot sami ABCE1 simlar? Which dww.mbbwam 4) Whht ane the upstwar perfattertone that aelect GADARAE?
LLM Limalt mine
GAAAAAA
P'
G¹
kandier BABARAR
queny:
Neighborhood of downstrrem gone
Downstrzamm Gene summary
ABCE1 K562 expression GABARAR?
difrrntiaal
6',
apptasee 62;
Generate summaries every the graph.
Retrieve summaries and experimental outcomes for neighboring nodes in training
Generate prompis containing
L.IM Fimelt pi
Ansrer
summmaries and retrieved outeomme
from D. then query LLM.
Figure 3: Overview of SUMMER. A) Knowledge graph entries are summarized per gene as both a perturbation p and as a downstream gene g. B) Given a new pair (p. g), sample related pairs (p', g') with associated experimental outcomes. C) Concatenate summaries, examples, and guiding questions as prompt for LLM. Depicted prompt edited for concision. Full prompts in Appendix C
In contrast to scientific literature, knowledge graphs densely distill key findings, either through hu- man curation, or from individual (i.e. internally consistent) large-scale experiments. To ensure cov- erage of poorly characterized genes, we took the union of multiple sources, maintaining attribution.
These include UniProt (Consortium) 2022). Ensembl (Martin et al.] 2023), Gene Ontology (Ash-
burner et al. 2000 Aleksander er al.) 2023). CORUM (Tsitsiridis et al. [0022), STRING (Szklarczyk
ct al. 2022). Reactome (Milacic et al. (023), and BioPlex (Huttlin et al. 2021).
A benchmark for discovery PERTURBQA draws upon experimental assays and knowledge graphs to "connect the dots" between known biology and unanswered questions. A key concern is whether these questions indeed unanswered, within the broader literature and by extension, current LLM weights. To quantify this, we compare the connectivity of pairs (p. 9). stratified by the differential expression label (Figure (D). Only ~3% of gene pairs in our test physically interact in any con- text, including other animals, and only ~00% share any annotation, including at the coarsest levels. There is little difference between positive and negative pairs terms of higher-level connectivity. Physically interacting genes are more likely to result in differential expression in our dataset, but presence of a physical interaction minimally predictive of differential expression (Table [T]. Fi- nally. Nadig et al.) (2024) was published strictly after we downloaded the knowledge graphs (June 2024). While the cell lines question have been studied in other contexts, Nadig et al. (2024) re- leased the first large-scale Perturb-seq screens in these two cell lines. Therefore, we conclude that these are indeed predictive tasks, rather than knowledge recall.
4.2 OVERVIEW OF SUMMER
SUMMER is inference-time framework that consists of three steps, which draw inspiration from different aspects of message-passing neural networks and large language model reasoning strategies (Figure 3. First, we "featurize" each gene by summarizing database descriptions of their known function, and their relationships to other entities. To capture directionality, we generate two sum- maries for each gene: as a perturbation, and downstream gene. Next, we use the "training" set of seen perturbations as a retrieval corpus, where relevant perturbation-en -gnn pairs are selected based on knowledge graph proximity. These pairs contribute both gene summaries and discretized experimental outcomes. Finally, we formulate a set of questions to guide the L.M'' reasoning via chain-of-thought style prompting. Full prompts may be found in Appendix C
Neighborhood aggregation as summarization Inspired by message-passing Welling 2017), we characterize genes and their relationships to other biological entities by summa- rizing their graph neighborhoods. Given gene v. we convert to natural language: knowledge graph entries (t.) ). describing node v. and (ty-ser }. describing its relationships with neighbors v'. As il- lustrated in Figure 3A,, we concatenate all entries t generate two versions of each gene summary s,. One asks for the downstream pathways that may be affected by the gene (v as perturbation p), and the other focuses on the upstream pathways that may affect the gene (u downstream gene g)-
graphs (Kipf &

---

Published as a conference paper at ICLR 2025
We can apply this step hierarchically to characterize gene sets. Specifically, to summarize cluster P. we concatenate single gene summaries [So] as input to the LLM, with an emphasis on down- stream/upstreem effects, as appropriate. A variant of this is used to address the gene-set set enrichment task, asking the L.M for both a short description and a name for the gene set (example in D.I.
Graph-based retrieval of relevant training samples Let M(v) represent the top k 10 nodes v/ that share the most neighbors with v in G. For each pair (p, g), we sample up to 15 examples (p', g') from Xiain that match the following criteria 2
1. Up to 5 pairs where both the perturbation and the downstream gene are related to and g respectively, i.e. p' E (p) and g' (g)....
2. Up to 5 pairs with any g' and where the perturbation is related to p. i.e. p' E EV()).
3. Up to 5 pairs with any p' and where the downstream gene is related to g. i... g' E N(g))
These pairs are inserted into the prompt through their respective summaries 8g') and the dis- cretized experimental outcome 3'''s' (Figure
Perturbation outcome prediction as question-answeringg To avoid hallucinations, we found it necessary to dictate that the LLM should consider both the textual summaries and experimental outcomes. Specifically, for both p and g. we ask the LLM identify the most similar p' and g' among those sampled, and to summarize their observed effects. For example, the prompt notes that the LLM should consider compensatory mechanisms, in addition to upstream or downstream pathways. Finally, we ask the LLM to answer the overall question, while citing specific retrieved examples. Figure 3 depicts a sketch of the process (example in D.2). Implementation details We ran all experiments with Llama3 (Dubey et al. 2024h with default parameters of top p 0.9 and temperature 0.6, using the LMDeploy framework (Contributors 2023). Due to computational limitations, the gene summaries were generated by the 70B model, while all other inference utilized the 8B model. To quantify uncertainty, we ran the retrieval step three times and report the average prediction over these three inference runs.
EXPERIMENTAL SETUP
5.1 BASELINES
Differential expression and direction of change We benchmark variety of baselines for unseen perturbation response prediction. All baselines were run with their published code and best reported hyperparameters, where applicable. PHYSICAL is a naive baseline that predicts 1 if (p. q) are known to physically interact in any animal (Figure 2D. STRINGDB (Szklarczyk et al., (022)) and 0 other- wise (DE only). GAT (Veličković et al. 2017) is graph attention network trained with ternary (up, down, no change) classification objective over the same knowledge graphs used to generate the prompts for SUMMER. This setup quantifies the information content contained in graph adjacencies alone. GEARS (Roohani et al. 2023) is a state-of-the-art graph attention network trained over the Gene Ontology (Ashburner et al. 2000) and gene co-expression graphs with a regression objective that focuses on differentially expresses genes. We use absolute predicted log-fold change for differ- ential expression, and signed log-fold change for direction of change. SCGPT (Cui et al. 2024 is a Transformer-basee single-cell foundation model, which is finetuned for perturbation effect predic- tion with the GEARS regression objective.
We also consider ernguage-centric baselines and ablations for SUMMER. GENEPT (Chen & Zou 2024) encodes textual descriptions of genes (-GENE) and their protein (-PROT) products using com- mercial OpenAl embedding models, trained on natural language. These embeddings are input to logistic regression classifier, trained separately for differential expression and direction of change. Recent work has reported that this baseline achieves the ttate-of-the-art lation (Märtens et al. 2024). LLM (No CoT) provides the LLM with two examples (one of cach label) and directly asks for the final answer without explanation. LLM (No retrieval) emulates our chain-of- -fhhuughhh style questioning, but does not retrieve any experimental outcomes. Instead, we
the regression formu-
The number of examples was chosen heuristically, so that all input prompts and potential outputs fit within the llama3 8k token context window.
7

---

Published as a conference paper at ICLR 2025
provide the LLM with hypothesis (each answer option is sampled twice) and ask the LLM to extract supporting and refuting evidence from the gene summaries, before answering the question. Finally, to understand the information content in our retrieved samples, Retrieval (No LLM) takes the mean label over (p',g') without appealing to the LLM for further processing.
Gene set enrichment We compare to gene over-expression analysis (Fang et. 2022 run the gene clusters with a variety of gene set libraries Gene Ontology, Reactome, CORUM as well as their union (Combined). We take the concatenation of the top k gene set names as the predicted summary, where gene sets are ordered by the size of their intersection with each cluster.
5.2 METRICS
Differential expression and direction of change has been reported that gene responses tend to be correlated across perturbations (Kernfeld et al.) (2023), c.g., stress response genes respond promiscuously. In addition, methods like GEARS and SCGPT predict real-valued change in genes, which yield rankings rather than strict probabilities. Thus, we compute binary AUROC over the predictions associated with each downstream gene, and take the average over downstream genes, corresponding to macro AUROC score over downstream genes.
Gene set enrichment We consider both automated and human evaluation. Our ground truth consists of short textual descriptions (under 10 words), while our predictions and baseline outputs are long and vary in style. Standard text generation metrics like BLEU (Papineni et al., 2002) or ROUGE Fl (Lin 2004) do not account for this difference in length as they were designed for machine trans- lation. To assess whether the predictions adequately cover the ground truth annotation without pe- nalizing for longer lengths, we report ROUGE-1 1 recall and BERT Score (Zhang al. 2020), using BioBERT 1.2 (Lee et al. 2020), which was finetuned on 1M biological texts (18B words).
Due to the open-ended nature of the gene set task, automated evaluation methods are limited in their ability to reflect practical utility. Since this paper focuses on providing value to biologists, we recruited a domain specialist (molecular biologist, not the original annotator). We asked them to decide whether the top gene sets or LLM summaries were more informative, and whether the LLM summaries captured the same biology as the manual annotation (Section B.I). For future works, we share all LLM summaries in the data distribution for independent evaluation. If access to human experts is challenging, we also encourage LLM assessment of these questions.
RESULTS
6.1 DIFFERENTIAL EXPRESSION AND DIRECTION OF CHANGE
We evaluated a number of state-of the baselines on the differential expression and direction of change tasks [ and the results indicate that PERTURBQA tasks are largely unsolved. GEARS and SCGPT performances are close to random on differential expression, often exceeded by the naive PHYSICAL baseline. This may be due to the focus on change in differentially-xxpresedd genes in their objective, instead of distinguishing between DEGs and non-diffcrentially expressed genes. On other hand, GEARS decent at direction of change in 3 of cases, reflecting that its directionality loss may be more effective here.
GENEPT is strong baseline, demonstrating the benefits of textual information towards these tasks. In terms of language- baasegee-basdd based reasoning, however, we observe that LLM (No CoT) and LLM (No retrieval) both perform no better than random guessing highlighting that retrieving experimental outcomes and guiding LLM reasoning are both essential to completing this task. This is also reflected in the strong performance of Retrieval (No LLM)). SUMER able to extract more value than "sum of its parts" in 7 of 10 cases, achieving the highest AUC in 8 of 10 cases.
Compared to methods that exclusively model knowledge graph connectivity, LLM outputs are di- rectly interpretable by domain experts (Appendix DD. to understand the model's shortcomings and provide context for the observed experimental outcomes. We studied 300 generations (3 trials of 100 DE examples) to identify primary failure modes (Appendix common error. For example, A is related to C, which is upstream of B, A should not be affected when perturb B. However, the LLM is unaware that C is upstream of B, it predicts that A
Incorrect causal directionality is
8

---

Published as a conference paper at ICLR 2025
Table 1: Results differential expression and direction of change binary prediction. AUROC is computed over the predictions associated with each gene, and averaged over perturbations. Standard deviation is reported over 3 runs (where applicable) rounds of sub-sampling. For more details, see Appendix A.4
Task
Model
K562
RPE1
HepG2
Jurkat
K562-Set
PHYSICAL
0.53
0.52
0.52
0,54
0.55
GAT
0.55+.02
0.57
0.57+.02
0.55+.0z
0.54+.01
GEARS
0.54+.01
0.50=.01
0.48
0.51
0.49:
SCGPT
0.52+.00
0.52
0.48+.00
0.51 00
0.52: 00
Differential
DENEPT-GENC
0.57+t.02
0.54
0.55+.0z
0.55
0.58#
expression
GENEPT-PROD
0.574.01
0.56
0.54+.01
0.55=.01
0.58:.01
LLM (No CoT)
0.52..01
0.51
0.51:.01
0.52
0.50:.00
LLM (No retrieval)
0.511.01
0.48
0.491.01
0.49
0.50..01
Retrieval (No LLM)
0.58+.02
0.58=.o1
0.55+.00
0.55
0.64
SUMMER
0.60
0.58
0.61 4:000
0.58
0.61
GAT
0.584.06
0.60 04
0.64
0.59 04
0.53
GEARS
0.641.01
0.60 01
0.52+.01
0.51
0.59a.02
SCGPT
0.48+.00
0.53=
0.51+.00
0.51
0.54+.00
GENEPT-GENS
0.531.00
0.57 .03
0.58
0,57
0.561.00
Direction of change
GENEPT-
0.57...0
0.57=.00
0.55
0,58
0.574.02
LIM (No CoT)
0.50+.01
0.49=.00
0.49
0.50=.o1
0.50=t001
L.MM (No retrieval)
0.49+.04
0.52=
0.51 #:06
0.53 05
0.45#.18
Retrieval (No LLM)
0.50#t.00
0.50=.00
0.502.00
0.50
0.50=.00
SUMMER
0.62+.01
0.642.01
0.65#.00
0.66
0.69
Table 2: Gene set enrichment on K562 genome-wide clusters. Metrics reported are ROUGE- recall, as well as BERT Score precision, recall, and F1, computed with BioBERT-112. Since the baselines statistical methods, they not subject stochasticity.
Gene clusters
Perturbation elusters
Enrichment
Тop
RROUGEr
PauRrT
RBERIT
FBERI
R
Parkrt
RBERTT
FaURIT 1
Gene Ontology
0.17
0.64
0.66
0.62
0.38
0.66
0.68
Gene Ontology
10
0.32
0.60
0.65
0.60
0.60
0.62
0.71
0.65
Reactome
5
0.18
0.60
0.65
0.60
0.49
0.60
0.68
0.62
Reactome
10
0.27
0.54
0.64
0.56
0.59
0.56
0.67
0.60
CORUM
5
0.07
0.63
0.45
0.42
0.45
0.64
0.63
0.60
CORUM
10
0.07
0.61
0.44
0.41
0.47
0.61
0.62
0.58
Combined
5
0.14
0.62
0.65
0.61
0.41
0.63
0.71
0.66
Combined
10
0.27
0.59
0.65
0.59
0.63
0.57
0.69
0.62
SUMMER (8h)
desc
0.57
0.63
0.76
0.69
0.26
0.63
0.75
0.68
SUMMER (8h)
name
0.20
0.74
0.76
0.75
0.12
0.75
0.76
0.75
SUMMER (70b)
desc
0.45
0.63
0.77
0.69
0.59
0.65
0.80
0.72
SUMMER (70b) name
0.15
0.73
0.76
0.74
0.37
0.77
0.82
0.79
changes when B is changed. This suggests that additional domain knowledge more structured reasoning strategies may be necessary.
6.2 GENE SET ENRICHMENT
SUMMER is able to recapitulate human cluster annotations, exceeding over-representation analysis across automated metrics (Table In particular, SUMMER performs much better than the baselines on gene clusters, which are larger than the perturbation clusters (Figure 2 and more enriched for generic gene sets (Table 3 row 1). Based on independent human evaluation, the LLM --M--eenertedd summary similar more informative than the classical gene set enrichment results in 92% of cases, and agrees with the independent annotator 72% of cases (Appendix [B.I). In the 28% that disagreed with the independent annotator, many clusters contain niche information, not found

---

METTL3, Published as a conference paper at ICLR 2025
Table 3: Gene set enrichment K562, example outputs. Left: Human annotation. Middle: Top over-represented gene sets (combined). Right: Proposed gene set name in bold, followed by de- scription. Row 1 is gene cluster. Rows 2-3 are perturbation clusters.
Genes
Annotation
Gene sets
SUMMER (8b)
(combined)
SLC25A5,
translation cytosol, RNA binding, cytoplasm, membrane, GTP hydrolysis
Ribosomal and Translation Regulation Gene Set The gene set is comprised of genes involved in protein syn- thesis, translation, and ribosomal function, with focus the regulation of protein synthesis and the assembly ribosomal subunits.
BIF4B,
SLC25A3,
RPL3,
RPL41
ZC3H13,
m6A mRNA methylation
nucleus, nucleo plasm, RNA N6- methyladenosine methyltransferase complex, mRNA processing
M6A Methylation Complex-Associater Genes. The gene set is composed of genes involved in the regulation of fe-mmethyladenosse fluencing mRNA splicing and RNA processing. These genes associated with the WMM complex and inter- act with each other to modulate gene expression.
CBLLI,
METTL14,
(m6A) methylation of RNAs, in-
PSMG1,
RBM15
CMTR2,
unknown no significant sets
Regulation Gene Set. The gene set composed genes involved RNA processing and regulation, including mRNA cap modification, alter- native splicing, and RNA-binding activities. These genes converge on pathways related to mRNA stability, trans- lation, and cellular differentiation.
RNA Processing
RBM14-
RBM4,
RBM4,
UNCX.
WDFY3
in typical databases (Table 7 We also observe that in difficult cases, gene set over-representation analysis tends to focus highly specific gene sets, which cover subsets of these clusters. The LLM takes the opposite approach, and its summaries tend to "lift" the description to higher levels of hierarchy (T) While the strategies provide orthogonal information, the LLM's s outputs more coherent. Finally, SUMMER also characterizes clusters for which no gene sets were enriched, and thus could not be annotated manually (Table row 3). These clusters tend to be smaller, or exhibit lower agreement.
7 CONCLUSION
In this work, proposed PERTURBQA, benchmark for language- basee reasoning over structured data that arise from real biological problems. We evaluated a variety of satt--of-the-asd methods and showed that while these problems feasible, they are far from solved. To address these tasks, we also introduced SUMMER, a LLM-based framework that draws upon both biological knowl- edge graphs and existing experimental data. SUMMER outperforms basclines on PERTURBQA. but leaves ample room for future study. We hope that this work will lower the barrier of entry into com- putational modeling of biological perturbation experiments and enable richer, more interpretable methods for these applications.
ETHICS STATEMENT
While this work focuses broadly discovering causal relationships in molecular biology, the methods described do not involve the design of potentially harmful chemical agents or other biomolecules. Our work uses publicly available datasets, which were generated in in-vitro labo- ratory settings.
REPRODUCIBILITY STATEMENT
All datasets and code can be found at repository: PerturbOA] Our data processing pipeline is described detail in Appendix AA22] Templates of all prompts used for LLM experiments be found in the code distribution and in Appendix C
10

---

We would like to thank Sandra Melo-Carlos and Jack Kamm for an introduction to Perturb-seq and guidance on data analysis; Aviv Regev for inspiring the gene set enrichment task; Umesh Padia for evaluating and providing feedback on our model outputs; and Romain Lopez, Alexander Wu, Xiaotian Ma, Taro Makino, Martin Rohbeck for continual Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Heming Yao, Patrick
feedback on this project.
This work was funded by Genentech. M.W., R.L., J.L., L.Q., T.B., and J.-C. H. were employ- ees of Genentech while working on this project. J.L., L.Q., T.B., D.R., and J.-C. H. have equity in Roche.

---

Published as a conference paper at ICLR 2025
REFERENCES
Constantin Ahlmann-Elie and Wolfgang Huber. gimGamPoi: fitting Gamma-Poisson generalized linear models on single cell count data. Bioinformatics, 36024):5001-5702, 12 2020. ISSN 1367- 4803. doi: 10. htl.:///ioinformmttc///taaae
Suzi A Aleksander, James Balhoff, Seth Carbon, J Michael Cherry, Harold J Drabkin, Dustin Ebert, Mare Feuermann, Pascale Gaudet, Nomi L. Harris, et al. The Gene Ontology knowledgebase in 2023. Genetics, 224(() 11:yya11 2023.
M Ashburner, Ball, A Blake, Botstein, Butler, M Cherry, P Davis, K Dolinski, S Dwight, J T Eppig. M A Harris, D P Hill, L. Issel-Tarver, A Kasarskis, S Lewis, J C Matese, J E Richardson, Ringwald, G M Rubin, and Sherlock. Gene Ontology: tool for the unification of biology. Nat Genet, 25(1):25-29 May 2000. doi: 10.1038/75556.
Ding Bai, Caleb N Ellington, Shentong Mo, Le Song, and Eric P Xing. AttentionPert: accurately modeling multiplexed genetic perturbations with multi-scalc effects. Bioinformatics, 40:i453~ i461, 06 2024. ISSN 1367-4811.
doi:
Yoav Benjamini and Yosef Hochberg. On the adaptive control of the false discovery rate in multiple testing with independent statistics. Journal of Educational and Behavioral Statistics, 25(1))60- 83, 2000. ISSN 10769986, 19351054.
Maciej Besta, Nils Blach, Ales Kubicek. Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyezyk, and Torsten Hoefler. Graph of Thoughts: solving elaborate problems with large language models. Proceedings of the AAAI Conference on Artificial Intelligence 38(16) Mar 2024. doi: 10.1609/aaai. v38i16.29720.
Charlotte Bunne, Stefan G Stark, Gabriele Gut, Jacobo Sarabia Del Castillo, Mitch Levesque, Kjong-Van Lehmann, Lucas Pelkmans, Andreas Krause, and Gunnar Rätsch. Learning single- cell perturbation responses using neural optimal transport. Nature Methods, 20(11):1799-1768, 2023.
Yiqun Chen and James Zou. GenePT: A simple but effective foundation model for genes and cells built from ChatGPT. bioRxiv, 2024. doi: 00.11012002310.16..23
Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, and Jiliang Tang. Exploring the potential of large language models (LLMs) in learning on graphs. SIGKDD Explor: Newsl., 25(2):42-6) 61, March 2024. ISSN 1931- 0145. doi: 00.14453655103336555111
The UniProt Consortium. UniProt: the Universal Protein Knowledgebase in 2023. Nucleic Acids Research, 51(D1):D533-D53, 2022. ISSN 0305-1048. doi: 10.1093/nar/gkac1055.
LMDeploy Contributors. LMDeploy: A toolkit for compressing, deploying, and serving LLMss. htttp:://qithin com Coom/Innernnan 1mdeploy 2023.
Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang- ScGPT: toward building a foundation model for single-cell multi-omics using generative Al. Na- ture Methods, pp. 1-11. 2024.
Xiangqin Cui and Gary Churchill. Statistical tests for differential expression in cDNA microarray experiments. Genome Biology, 4:210. 02 2003. doi: 10. 10.11866-0-20034444-10
Paul Datlinger, André Rendeiro, Christian Schmidl, Thomas Krausgruber, Peter Traxler, Johanna Klughammer, Linda C Schuster, Amelie Kuchler, Donat Alpar, and Christoph Bock. Pooled CRISPR screening with single-cell transcriptome readout. Nature Methods, 14(3):27-331, 2017
Atray Dixit, Oren Parnas, Biyu Li, Jenny Chen, Charles P Fulco, Livnat Jerby-Amon, Nemanja D Marjanovic, Danielle Dionne, Tyler Burks, Raktima Raychowdhury, et al. Perturb-seq: dissecting molecular circuits with scalable single-cell RNA profiling of pooled genetic screens. Cell, 167 (7):1853-1866 1866, 2016.
12

---

Published as a conference paper at ICLR 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXhv:240.217783, 2024.
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph RAG approach to query-focused summariza- tion. arXiv preprint (1XXx:2444.161333 2024.
Zhuoqing Fang. Xinyuan Liu, and Gary Peltz. GSEApy: a comprehensive package for performing gene set enrichment analysis in Python. Bioinformatics, 39(1):btac777, 11 2022. ISSN 1367- 4811. doi: 10. 10000biiinooommttss/tass/55.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. Retrieval-augmentee generation for large language models: A survey. arXiv preprint arXiv:2312.1099 2023.
Thomas Gaudelet, Alice Del Vecchio, Eli M Carrami, Juliana Cudini, Chantriolnt-Andeen Kapourani, Caroline Uhler, and Lindsay Edwards. Season combinatorial intervention predictions with Salt & Peper. In ICLR 2024 Workshop Machine Learning for Genomics Explorations, 2024.
K. Guu, J. Miller, and P. Liang. Traversing knowledge graphs in vector space. In Empirical Methods in Natural Language Processing (EMNLP), 2015.
Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo. Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Le Song, and Xuegong Zhang. LAB-Bench measuring capabilities of language models for biology research. Nature Methods, 2024a. doi: 10.
Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Le Song, and Xuegong Zhang. Large-scale foundation model on single-cell tran- scriptomics. Nature Methods, 2024b. doi: 10. 101108/441199222444000377..
Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V Chawla, Thomas Laurent, Yann LeCun, Xavier Bres- son, and Bryan Hooi. G-retriever: Retrieval-augmented generation for textual graph understand- ing and question answering. arXiv preprint arXiv:2402.07633 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
Wenpin Hou and Zhicheng Ji. Gene GeneTuring tests GPT models in genomies. BioRxiv, 2023.
Chi-Yang Hsu, Kyle Cox, Jiawei Xu, Zhen Tan, Tianhua Zhai, Mengzhou Hu, Dexter Pratt, Tianlong Chen, Ziniu Hu, and Ying Ding. Thought graph: Generating thought process for biological reasoning. In Companion Proceedings of the ACM on Web Conference 2024, WWW '24. ACM, May 2024. doi: 10. 10.1145/55893353355155
Da Wei Huang, Brad T. Sherman, and Richard A. Lempicki. Bioinformatics enrichment tools: paths toward the comprehensive functional analysis of large gene lists. Nucleic Acids Research, 37:1- 13, 2008
Edward L.. Huttlin, Raphael J. Bruckner, Jose Navarrete-Peren Joe R. Cannon, Kurt Baltier, Fana Gebreab, Melanie P. Gygi, Alexandra Thornock, Gabriela Zarraga, Stanley Tam, John Szpyt, Brandon M. Gassaway, Alexandra Panov, Hannah Parzen, Sipei Fu, Arvene Golbazi, Eila Maen- paa, Keegan Stricker, Sanjukta Guha Thakurta, Tian Zhang, Ramin Rad, Joshua Pan, David P. Nusinow, Joao A. Paulo. Devin K. Schweppe, Laura Pontano Vaites, J. Wade Harper, and Steven P. Gygi. Dual protcome-scale networks reveal cell-specific remodeling of the human in- teractome. Cell, 184(1 1):3022-300000000 cell.2021.04.011
2021. ISSN 0092-8674. doi: https://doi.org org//10. 1016/..
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7969-7992, 2023.
13

---

Published as a conference paper at ICLR 2025
Bowen Jin, Gang Liu, Chi Han, Meng Jiang, Heng Ji, and Jiawei Han. Large language models on graphs: A comprehensive survey. IEEE Transactions on Knowledge and Data Engineering, 2024.
Jérémie Kalfon, Jules Samaran, Gabriel Peyré, and Laura Cantini, ScPRINT: pre-training on 50 million cells allows robust gene network predictions. bioRxiv, 2024. doi: 10.1101/202..7.299.
605556.
Amanda Kau, Xuzeng He, Aishwarya Nambissan, Aland Astudillo, Hui Yin, and Amir Aryani. Combining knowledge graphs and large language models. arXiv preprint arXiv:2407.66304 065644 2024.
Eric Kernfeld, Yunxiao Yang, Joshua S. Weinstock, Alexis Battle, and Patrick Cahan. A systematic comparison of computational methods for expression forecasting. bioRxiv, 2023. doi: 10.1101/ 2023.07.28.55111
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations, 2017.
Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. LAB- Bench: Measuring capabilities of language models for biology research, 2024.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- woo Kang. BioBERT: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 1234-11234- 2020.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. sertival--uugentee gener- ation for anowledge-intensie NLP tasks. Advances in Neural Information Processing Systems, 33:9459-9474, 2020.
Chin-Yer Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Zarel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Eyolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379666377:1123-1130 2023. doi: 10.1 sci5n4e.ade2574.
Mohammad Lotfollahi, F. Alexander Wolf, and Fabian J Theis. scGen predicts single-cell perturba- tion responses. Nature Methods, 16:715 721, 2019.
Mohammad Lotfollahi, Anna Klimovskaia Susmelj. Carlo De Donno, Leon Hetzel, Yuge Ji, Igna- cio L Ibarra, Sanjay R Srivatsan, Mohsen Naghipourfar, Riza M Daza, Beth Martin, Jay Shendure, Jose L McFaliie-FFiguroo. Pierre Boyeau, F Alexander Wolf, Nafissa Yakubova, Stephan Gün- nemann, Cole Trapnell, David Lopez-Paz and Fabian J Theis. Predicting cellular responses to complex perturbations in tightthroughhut screens. Molecular Systems Biology, 19(6) 2023. doi: https:///oo.org/110/ 5255/mmsb.022211555
Lovev W Huber, and S Anders. Moderated estimation of fold change and dispersion for RNA-seq data with DESeq2. Genome Biol, 15(12):550-555 550, 2014. doi: 10. 18.1186/s1.0 086///13559-000 3059-00140550-88
Kaspar Märtens, Rory Donovan-Maiye, and Jesper Ferkinghoff-Bood turbation models with LLM-informed gene embeddings. In ICLR 2024 Workshop Machine Learning for Genomics Explorations, 2024.
Enhancing generative per-
Fergal J Martin, M Ridwan Amode, Alisha Aneja, Olanrewaju Lustine-Orimoloyy Andrey G Azov, If Barnes, Ame Becker, Ruth Bennett, Andrew Berry, Jyothish Bhai, et al. Ensembl 2023. Nucleic Acids Research, 51(D1):D933-DD44 2023.
Costas Mavromatis and George Karypis. GNN-RAG: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20133 2024.
14

---

Published as a conference paper at ICLR 2025
Marija Milacic, Deidre Beavers, Patrick Conley, Chuqiao Gong, Mare Gillespie, Johannes Griss, Robin Haw, Bijay Jassal, Lisa Matthews, Bruce May. Robert Petryszak, Eliot Ragueneau, Karen Rothfels, Cristoffer Sevilla, Veronica Shamovsky, Ralf Stephan, Krishna Tiwari, Thawfeck Varu- sai, Joel Weiser, Adam Wright, Guanming Wu, Lincoln Stein, Henning Hermjakob, and Peter D'Eustachio. The Reactome Pathway Knowledgebase 2024. Nucleic Acids Research, 52(D1): D672-D678, 11 2023, ISSN 0305-1048. doi: 10. 10.109//aargkaa
Ajay Nadig, Joseph Replogle, Angela Pogson, Steven Mccarroll, Jonathan Weissman, Elise Robin- son, and Luke O' Connor. Transcriptome-wide characterization of genetic perturbations. bioRxiv, 07 2024. doi: 10.1101/2024.07.0 07..0000000.60030...
Eric Nguyen, Michael Poli, Matthew G Durrant, Brian Kang, Dhruva Katrekar, David B Li, Liam J Bartie, Armin Thomas, Samuel H King, Garyk Brixi, et al. Sequence modeling and design from molecular to genome scale with Evo. Science, 386(6723):eado93366 2024.
Thomas M. Norman, Max A. Horlbeck, Joseph M. Replogle, Alex Y. Ge, Albert Xu, Marco Jost, Luke A. Gilbert, and Jonathan S. Weissman. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science, 365(6455)7766-99, 2019. doi: 10. 126/science.
aax4438.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguisties, pp- 311-318, Philadelphia, Pennsylvania. USA, July 2002. Association for Computational Linguis- tics. doi: 10.31 (0.3115/1073083.10000
Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami Al-Rfou, and Jonathan Halcrow. Let your graph do the talking: Encoding structured data for LLMs. arXiv preprint arXhv:2402.05862, 2024.
Yuanhao Qu, Kaixuan Huang, Henry Cousins, William A Johnson, Di Yin, Mihir Shah, Denny Zhou, Russ Altman, Mengdi Wang, and Le Cong. CRISPR-GPT: An LLM agent for automated design of gene-editing experiments. bioRxiv, pp. 2024-04, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. arXiv preprint arXiv:2311.1202. 2023.
Joseph M. Replogle, Reuben A. Saunders, Angela N. Pogson, Jeffrey Hussmann, Alexander Lenail, Alina Guna, Lauren Mascibroda, Eric J. Wagner, Karen Adelman, Gila Lithwick- Yanai, Nika Iremadze, Florian Oberstrass, Doron Lipson, Jessica L. Bonnar, Marco Jost, Thomas M. Norman, and Jonathan S. Weissman. Mapping information-rich genotype-phenotype landscapes with genome- scale Perturb-seq. Cell, (85(14):259--255544 2022. ISSN 0092-8674. doi: https//doi..or/100. http:///bboogg111.00011.....l..d
Yusuf Roohani, Kexin Huang, and Jure Leskovec. Predicting transcriptional outcomes of novel multigene perturbations with GEARS. Nature Biotechnology, 2023.
Yusuf H Roohani, Jian Vora, Qian Huang, Percy Liang, and Jure Leskovec. BioDiscovery Agent: An Al agent for designing genetic perturbation experiments. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations, 2024.
Yanay Rosen, Maria Brbić, Yusuf Roohani, Kyle Swanson, Li Ziang, and Jure Leskovec. Towards universal cell embeddings: Integrating single-cell RNA-seq datasets across species with SAT- URN. Nature Methods, 2024. doi:
Yair Schiff, Chia Hsiang Kao, Aaron Gokaslan, Tri Dao. Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range DNA sequence modeling. In Forty-first Interna- tional Conference on Machine Learning, 2024.
Aravind Subramanian, Pablo Tamayo, Vamsi K. Mootha. Sayan Mukherjee, Benjamin L. Ebert, Michael A. Gillette, Amanda Paulovich, Scott L. Pomeroy, Todd R. Golub, Eric S. Lander, and Jill P. Mesirov. Gene set enrichment analysis: A knowledge-based approach for interpreting
15

---

Published as a conference paper at ICLR 2025
genome-wide expression profiles. Proceedings of the National Academy of Sciences, 102(43): 15545-15550, 2005, doi: 10.073//paa.005058800022
Damian Szklarezyk, Rebecca Kirsch, Mikaela Koutrouli, Katerina Nastou, Farrokh Mehryary, Radja Hachilif, Annika Gable, Tao Fang, Nadezhda T Doncheva, Sampo Pyysalo, Peer Bork, Lars Jensen, and Christian von Mering. The STRING database in 2023: protein-protein association networks and functional enrichment analyses for any sequenced genome of interest. Nucleic Acids Research, 31(D))D633--DO)
11 2022. ISSN 0305-1048. doi: 10. 10.1093/aar/kkac1000
George Tsitsiridis, Raiph Steinkamp, Madalina Giurgiu, Barbara Brauner, Gisela Fobo, Goar Frish- man, Corinna Montrone, and Andreas Ruepp. CORUM: the comprehensive resource mam- malian protein complexes-2022. Nucleic Acids Research, 51(DI):D539-DD 11 2022. ISSN 0305-1048. doi: 10.10933nar//kacc0155
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 10903, 2017.
Marie-Claire Wagle, Daniel Kirouac, Christiaan Klijn, Bonnie Liu, Shilpi Mahajan, Melissa Junttila, John Moffat, Ling Huw, Matthew Wongchenko, Kwame Okrah, Shrividhya Srinivasan, Zineb Mounir, Teiko Sumiyoshi, Peter Haverty, Robert Yauch, Yibing Yan, Omar Kabbarah, Garret Hampton, and Shih-Min Huang. A transcriptional MAPK pathway activity score (MPAS) is clinically relevant biomarker in multiple cancer types. npj Precision Oncology, 2, 12 2018. doi: 10.1038/s41698-00 1900--0001111--01-.
Chengrui Wang, Qingqing Long, Meng Xiao, Xunxin Cai, Chengjun Wu, Zhen Meng, Xuezhi Wang. and Yuanchun Zhou. BioRAG: A RAG-LLM framework for biological question reasoning. arXiv preprint ar arXiv:2488.01100 2024a.
Zifeng Wang, Zichen Wang. Balasubramaniam Srinivasan, Vassilis N Ioannidis, Huzefa Rangwala, and Rishita Anubhai. BioBridge: Bridging biomedical foundation models via knowledge graphs. In International Conference on Learning Representations, 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. sfain-of-thoughh prompting elicits reasoning in large language models. 36th Conference on Neural Information Processing Systems, 2022.
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945. ISSN 00994987.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances Neural Information Processing Systems, 36, 2024.
Di Zhang, Xiaoshui Huang, Dongzhan Zhou, Yuqiang Li, and Wanli Ouyang. Accessing GPT-4 level mathematical olympiad solutions via Monte Carlo tree self-refine with LLaMa-3 8B, 2024.
Tianyi Zhang, Varsha Kishore. Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. BERTScore: Eval- uating text generation with BERT. In International Conference on Learning Representationss 2020.
Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, and Jian Tang. GraphText: Graph reasoning in text space. arXiv preprint arXiv:2310.01085 2023.
16

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL DETAILS
A.1 K562 GENE SETS
We downloaded K562 genome-wide gene sets from the supplementary data associated with Figure 4B of Replogle et al. 2022). These gene sets were obtained by "cluster[ing] genes into expres- sion programs based on their co-regulation." " We evaluated gene set enrichment over clusters that were manually annotated, though we provide predictions over all gene sets (regardless of annotation status).
For differential expression analysis, we took the average of the Log(TP10k+)) values over each gene set, for each cell, similar to a pathway activity score (Wagle et 2018).
A.2 DATA PROCESSING
We normalized all gene counts Log(TPIOk+1) values (log- trggtrranformed UMI count per 10k), where the count Cij of gene j in cell is mapped to
Cj 10, 000 - 1 Σ Cij
log
(2)
To determine differentially expressed genes (DEGs), we the Wilcoxon signed-rank
test [Wilcoxon] 1945) with Henjamini-Hochbere correction (Benjamini & Hochberg 2000 between
non-targeting control (NTC) cells and perturbed cells, for each perturbation.
To focus on perturbations with phenotypic effect, we excluded all perturbations that resulted in < DEGs (p < 0.05), with the exception of 100 negative control perturbations (0 DEGs), sampled uniformly at random per dataset.
We selected thresholds based statistical consistency (Section A.5 For the two K562 datasets, we defined "DE" perturbation gene pairs as those shared between the datasets at < 0.05. Non differentiallyxxpressed pairs were sampled from those that were not differeettalll--xppeesed in ei- ther dataset. Since the remaining datasets lacked biological replicates, we defined "DE" pairs as those with p 0.01, and sampled "non-DE" pairs from those with p > 0.1. We selected the top 20 DEGs per perturbation ranked by adjusted p-value as "positives." We sampled 100 non-DEG; per perturbation as "negatives."
To ensure similar distributions between our training and testing splits, we sorted both selected pertur- bations and DEGs based on prevalence. We split perturbations 75:25 between training and testing. Validation data were sampled at random during training (10% of training). Further details regarding dataset and data split statistics may be found Tables and
Table 4: Data statistics. K562" non-targeting control cells were subsampled due the size of the gedeme-wide dataset. K562-es* (essential) was only used to filter K562 DE and non-DE genes.
Dataset
Cells
Perturbations Train
Control
Perturbed
Total
Test
Features
K562*
5,000
919,124
9851
1564
267
4136
K562-es*
10,691
299,645
2049
RPE1
11,485
236.164
2354
1596
406
4760
HepG2
4.976
140,497
2393
1086
278
7435
Jurkat
12,013
250,943
2392
1227
313
6842
K562-Set
5,000
919,124
9851
1401
357
20
A.3 LLM DETAILS
Due to the stochastic nature LLM generations, we noticed that the LLM would occasionally abstain from selecting one of the intended labels, due to insufficient evidence for either. To account
17

---

Published as a conference paper at ICLR 2025
Table 5: Differential gene expression data split statistics, Number of perturbation gene (set) pairs.
Dataset
Split
Total
non-DE
Differentially expressed
Total
Up
Down
K562
Train
134,467
117,606
16,861
11,041
5,820
Test
23,212
20,093
3,119
2,530
589
RPEI
Train
149,147
127,860
21,287
8,381
12,906
Test
37,942
32,577
5,365
2,121
3,244
HepG2
Train 101,140
86,883
14,257
6,249
8,008
Test
25,749
22,146
3,603
1,599
2,004
Jurkat
Train 113,684
97,747
15,937
5,119
10,818
Test
29,138
25,017
4,121
1,379
2,742
K562-Set
Train 20,606
9,367 2,403
11,239 2,832
3,953
7,286
Test
5,235
995
1,837
Table 6: datasets.
Abstain rate on differential expression (DE) and direction of change (Dir) across all
Model
DE
Dir
LLM (No Retrieval)
0.02
0.36
LLM (No CoT)
3.3 X 10
SUMMER
8.9
10-4
0.03
for this, we intentionally added "insufficient information" as a third answer option. We ran inference on each input sample at least 3 times and took the mean predicted label, after removing all abstaining outputs.
A small fraction of inputs (p. g) resulted in no predictions after this filtering, or were unable to be parsed by our rule-based parsing. The latter due the insufficient capacity of Llama3 8B (rela- tively small LLM) to follow instructions. Since these examples differed by model, we substituted the prediction with an uninformed baseline (the mean label of over the training set) for evaluation. The final abstain rate varied based on L.LM prompting strategy (Table 66. L.M (No Retrieval) abstained nearly a third of the time on direction change. In contrast, LLM (No CoT) only abstained a single time, over all datasets. The improved instruction following may be due to the concise nature of the expected output (only a single answer). Finally, SUMMER nearly always produced prediction over 3 runs (e.g. 0.08% abstain DE).
A.4 BASELINES
For GAT, we grid searched over the number of layers (1, 2, 4, 8) and hidden dimension (64, 128, 256). We used FFN dimension 1024 (memory constraint), GELU activation, dropout of 0.1, weight decay 1e-6, learning rate le-4, and residual connections. We selected the top models based on validation performance (arbitrary 10% of train). In addition to node features, GAT also learned edge attributes, which indicated the source knowledge graph of each edge.
For K562-Set, we pooled the mean embedding each gene set's genes before the prediction head in GAT. On GEARS and SCGPT, used the mean predicted log-fold change over each gene set's genes (mirrors data pre-processing).
small number of genes (97 out I 11,234) did not map to GENEPT embeddings. We set the embeddings for these genes the mean perturbation gene embedding in their respective training sets.
For uncertainty quantification, used the top runs for GAT. For GENEPT, since logistic regres- sion does not inherently introduce randomness (unless it fails to converge; always converges here), subsampled 80% of the training for each of 3 runs. Since GEARS and SCGPT operate
18

---

Published as a conference paper at ICLR 2025
single cells, rather than pseudo-bulk estimates, subsampled 80% of the single cells before taking the average for each evaluations.
A.5 STATISTICAL ANALYSIS
We provide empirical analyses on the quality of our datasets and labels. Figure shows that the Wilcoxon rank sum test is relatively well-calibrated on our data, though the test tends to be conser- vative, erring on the side of identifying fewer DEGs. Thus, we selected relative higher p-value threshold for negative examples. Figures 5 and 6 illustrate that K562 gene clusters and top DEGs are consistent across near-biological replicates (two experiments in the same cell line, by the same lab). This motivates both the K562-set setting, as well as our selection of the top DEGs as positives.
K352
KC562c
RPEI
HepP2
Juskat
0036
0006
0.04
QUA4
P-sllux
P-volocc
P-rodace
P-vollae
P-oulaze
Figure 4: Assessing p-value calibration over single-cell datasets. We split the non-targeting controls (NTCs) randomly half, and the Wilcoxon test to compare the two halves. We would expect to see that the (non-adjusted) p-values are uniformly distributed between and 1. Here, we see that the Wilcoxon test is slightly conservative, i.e. it leans towards reporting "non-differentially expressed."
Overlap lop
Overlap lop
Overlap lop sets
Overlap top 10 sets
800
800
600
600
400
Count
400
200
0
1.00
in gW
gW
66 gw
Figure 5: K562 gene clusters show consistent response between biological replicates. We compute the top & 5, 10 significant gene clusters, sorted by adjusted p-value, for both K562 genome- wide and K562 essential. For each perturbation, we compute the percentage of shared gene clusters (normalizing by genome-wide and essential, respectively). We that the clusters relatively consistent across both datasets, with high fraction of perfect overlaps.
B QUALITATIVE ANALYSIS
B. HUMAN EVALUATION SUMMARIZATION RESULTS
Due to the open-ended nature of the gene set task, automated evaluation methods are limited in their ability to reflect practical utility. Since this paper focuses on providing value to biologists, we recruited domain specialist (molecular biologist, trained in wet lab and computational biology) for this task. We presented them with document formatted as follows and asked two questions.
1) Is A or more informative, or about the same? Options: A, B, same
2) Does B capture the same biology as the bolded annotation? Options: yea, n10 (ground truth label): (list genes)
(top gene
databasee))
(LLM- generated name): (LLM generated descriptions)
Overall, the Lee-grnertted summary is equal or better to the classical gene set enrichment results in 92% of cases, and agrees with the independent annotator in 72% of cases.
19

---

Published as a conference paper at ICLR 2025
% Top es DEGs in gw
% Top gw DEGs in es
Overlap of top 10 sets
Overlap of top 10 sets
600
1300
610
Count
Count
800
Count
Count
1000
400
200
200
200
0.00
0.25
0.50
0.75
1,00
0.00
0.25
0.20
0.75
1.00
3.0
%
in
Top 20 es DEGs in gw
% Top
20 gw DEGs in es
Top 100 es DEGs in gw
% Top 100 gw DEGs es
2000
1:00
400
1500
100
Crant
Count
1000
200
200
300
100
0.0
0.5 1.0
0.0
0.5 1.0 0.0
1.0
0.0
0.5
Figure 6: K562 DEGs are reasonably consistent at the top (ranked by p-value). For each perturba- tion, we plot the percentage of shared top 5, 10, 20, 100 DEGs (normalizing by genome-widk and essential, respectively). In our final dataset, we took the intersection of the top 20 DEGs as "DE" and the intersection of the negatives "non-DE."
1. In 21/25 cases, the biologist reported that the LLM-generated summary was more informa- tive. In 2/25 cases, they contained the same amount of information; and in 2/25 cases, gene set contained more information.
2. In 18/25 cases, the biologist reported that the LLM summary captured the same biology as the original human annotation (our ground truth labels).
We analyze the cases in which the LLM provides less information, or fails to capture the independent human annotation.
1. In the 2 cases where the gene sets contained more information, list of specific protein complexes were discovered, c.g. below.
Eukaryotic Translation Termination, EIF2AK4 (GCN2) dimer autophosphorylate, EIFZAK4 (GCN2) binds ARNA, Aminoacyl-RRNA binds the ribosome st the A-site, A SECISBPPSS BENA(SS)
is hydrolysed to
and EEFSEC: GDP by
EEPSEC, UPP1 binds mRNP with termination codon pracoding Exon Junction Complex, Transloca- tion of riboaome by 3 bases in the 3' direction, Translation of ROBO3.2 MRNA initiates NMD, Translation of ROBO3.2 complex
negatively regulated by NMD,
receptor binds SRPmaacent peptide:ribosome
However, this output is difficult to parse, compared to the LLM-generated output, which faithfully and concisely represents the original annotation of "translation."
Ribosomal Protein Components Involved in Translation: This gene set is comprised of companents of the large amall ribosomai subunita, which are essential protain syntheala and translation. 'Thasa ganes are involved the assembly and function the ribosome, facilitating the tramslation of meesenger RNA into protein.
2. In the 7/25 cases where the LLM summary differed from the human annotation (Table the LLM annotation tended to miss some highly specific terms, e.g. "targets of nonsense- mediated decay" was generalized "stress response," and "dysregulated IncRNA antisense transcripts" was generalized to "nuclear gene regulation." Related terms tend to be sparsely annotated Gene Ontology, so this indicates that it would be useful to tune the granularity of generations in the future, or to generate multiple candidates for specific descriptions.
B.2 CAPTURING THE GREATEST COMMON DENOMINATOR
By construction, clusters of genes may vary in their degree of specificity and agreement. For ex- ample, "translation" contains over 20 million annotations in the Gene Ontology, while variants of
20

---

Published as a conference paper at ICLR 2025
Table 7: Error analysis of gene set summaries that differed from the manual annotation, based on independent human evaluation. These summaries tend to be more generic than the original annotation.
Annotation
SUMMER (8b) gene set name
targets of nonsense-mediated decay dysregulated IncRNA antisense transcripts unfolded protein response mTORCI signaling
RNA Processing and Stress Response Regulators Mitochondrial & Nuclear Gene Regulation and Metabolism
Amino Acid Metabolism and Protein Synthesis Regulators Regulators of Cellular Homeostasis and Signaling Actin Cytoskeleton and Immune Response Regulators Protein Trafficking and Transcriptional Regulation RNA Processing and Translation Regulation
NFKB signaling
myeloid differentiation
growth signaling
growth targets of Myc
Table 8: Unannotated gene clusters are typically less specific and coherent. Compared gene set enrichment, which tends to focus highly-specific subsets of each clusters, SUMMER tends to identify the "greatest common denominator" among the genes.
Genes Gene sets (combined)
SUMMER (8b)
HCCS, 2Fe-2S is inserted in
Ribosomal and Mitochondrial Gene Set. The gene set is com- posed of genes involved in various cellular processes, including protein synthesis, translation, and regulation of gene expression. Many of these genes are involved ribosomes, the translation mitochondrial -NA-endoded genes, and the regulation of protein synthesis in response to cel- lular signals.
AGPS, UQCRFSI; 39S ribosomal
FAM136A,
subunit; 3" incision by
MTREX, ERCC5 (XPG)
assembly and function
PITHD1,
TC-NER; 55S rihosome;
KIF2A,
I-palmitoylglycerroee
REXO2,
phosphate hexadecanol
E1F6
O-hhexadecyglyyceronee phosphate palmitate,
ATP6AP1,
keratan sulfate catabolic process; transcription
Protein Processing and Trafficking Gene Set. The gene set is comprised of genes involved in various cellular processes, in- cluding protoin transport, immune response, and cellular differ- entiation. Common pathways among these genes include protein processing and modification, well cellular trafficking and signaling.
ITM2A,
NPC2,
corepressor activity;
CREGI,
transcription regulator
UQCRB, complex; dermatan sulfate catabolic process,... ...
НEXA
RALA, Expression of Perilipin
Regulatory Genes Cellular Processes and Signaling. The gene set comprised genes involved in various cellular pro- cesses, including protein synthesis and degradation, gene ex- pression, transcription regulation, and cellular signaling. These genes also involved in maintaining proper calcium home- ostasis, regulating mitochondrial function, and facilitating pro- tein trafficking and targeting.
STARD3NI, (PLIN); Dephosphorylation
AKAPSI, of inactive SRC by PTPB1;
ATP2B1, Expression of Phosphoeuolpyruvate YTHDF2 carboxykinase (PEPCK-C)...
"nonsense-mediated decay" vary from 27 to 74,097 annotations (as of November 2024) (Ashburner
al. 2000).
Clusters that are smaller, or exhibit lower agreement, tend to clude manual annotation. In these cases, gene set over-representatioon analysis tends to focus on highly specific gene sets, which cover subsets of these clusters. The LLM takes the opposite approach, and its summaries tend to "lift" the description to higher levels of hierarchy (Table 8).
B.3 ERROR AND INCONSISTENCY ANALYSIS
We analyzed 300 generations (3 trials of 100 DE examples) to understand the failure modes of SUMMER. Errors and inconsistencies primarily resulted from deductions backed by overly-generic information. For example, the LLM may list an excessively broad set of factors that may influence expression.
21

---

Published as a conference paper at ICLR 2025
Perturbations of genes involved in mitochondrial function, protein synthesis, or transcriptional regulation may potantially CC2D11 expression.
In several instances, the LLM may be confused between concepts which are loosely connected, but not in the same context.
The genes of interest in the ezamples are QKI, MT-CYB, ADA, MT-ND1, and RPL12. Among theee, MT-CYB and MT-ND1 similar TXNIP they
involved mitochondrial function and electron transport, related process to TXNIP's role in oxidative stress and mitochonddial fanction.
Here, "oxidative stress" is related to the mitochondria, but TXNIP is upstream of number of stress response genes and unrelated healthy mitochondrial function. Thus, it should not respond similarly to MT-CYB. This aspect might be resolved with a higher-capacity model, but the 8b LLM cannot distinguish.
Finally, we observe that there is fine line between abstaining and predicting "no effect" due to lack of information.
differential expression of genes, including CDK11A. Additionally, perturbing EIP3J resulted in diflerential expression of HSP90AA which ta similar CDK11A. However, relationahlp between ABCE1 and CDK11A not well-represented these examples. Therefore, is unlikely that CRISPRI knockdown ABCE1 in KS62 cells will result in differential expression of CDXI1A.
Based analysis above, can perturbing ABCEL
genes
Final answer: There is insufficient evidence to determine how knockdown of ABCEL affects CDKIIA.
C PROMPTS
We share the prompts used by each stage of SUMMER, as well the LLM baselines. All text surrounded by {...} are replaced with example-specific inputs (Appendix D).
C.I SUMMARIZATION TEMPLATES
We generate gene summaries at two levels: single node and 1-hop knowledge graph neighborhoods. For both levels, we use two prompts per gene (as a perturbation, and as a downstream gene). Wher- ever gene sets are the downstream entities, we replace "gene" with "gene set."
The following an example of a single-node summary of perturbed gene.
You expert
biologiat. who
genes
using Perturb-seq.
Task: You are writing brief overview the human gene (gene), with focus on molecular and cellular functions. You will be provided knowledge.
databaze entries about the gene. Ensure that your overview remaine faithful to this domain
Format:
Write two sentences describing the primary molecular and celluiar function of (gene).
describing the potential downsiream impact perturbing gene (gene) gene knockdown.
Constrainte:
Maintain a professional tone throughout.
Do not comment on own writing.
refarances. Do up additional information. Do not add any importance impact gene. only
not
Domain knowledge: (description)
Brief overview of gene (gene):
To generate downstream gene descriptions, we replace the second formatting instruction with the following.
Write one sentance describing what types of parturbations might impact the expreasion of gane {gana). For example, you might consider pathways that are upstream of the gene or compensatory mechaniems.
Single-node summaries are input alongside additional knowledge graph relationships for 1-hop neighborhood summaries.
You are an expert ioolecular biologiet who studiee how genes are related using Perturb-seq.
Task: You are writing a brief overview of the human gane (gena), with focus on the downstream effacts of parturbing of function).
(gene) gene knockdown
Inputa: You are provided
Description of perturbed gane {geme?
22

---

Published as a conference paper at ICLR 2025
relating (gens) genes or pathways
Write up to live sentences describing the molecular and cellular impact of perturbing gene (gene) via gene knockdown.
Format:
Constrainte:
Remain faithful to all domain knowledge. Do not make up additional information.
Summarize all common aspects succinctly, but point out notable differances within these sats of genes.
Maintain professional tone throughout. Do not comment on your writing. Do not add any notea or references.
Omit importance impact gene. only ita function.
Omit non specific information and
statements, e.g. "this gene involved cellular processes."
Description of gene (gena); (single-node summary)
Relations to other gener: (relationships)
Downstream effecte of perturbing (gene) via knockdown:
To generate downstream gene descriptions, we replace the "Task" instruction with the following.
Task: You are writing brief overview of the human gene (gene), with a focus on molecular and cellular perturbations that may affect the levels gene {gens). For exampla, you might consider pathways that upstream the gane compansatory machanisma.
To generate gene summaries (for differential expression and direction of change), we used the following prompt, where the manual annotation is provided as input.
You are an expert molecular biologist who studies how genes and pathways are related using Perturb
Task: You writing brief overview gene associated with (manual annotation) human cells. You will be providad dascriptions of the constituent genes, Ensure that your overview remains faithful to this domain knowledge.
Format:
Write to sentences summarizing how the genes related. Be characterize how they associated with (manual annotation).
Write one to two sentences describing what types of perturbations might impact the expression of the genes in this gene sat. For example, you might consider pathways are upstream of ganes compansatory mechanisma.
Constraints:
Maintain professional tone throughout.
Do not comment your writing.
Do not any or references.
up additional information.
Do not discuss the importance or impact of the gene set, Focus only on its function. Descriptions of the constituent genee: Cliat of gene summaries)
Brief overview
aet (<manual annotation)):
Finally, for the gene set enrichment task, we use the following prompt for hierarchical summariza- tion of gene sets. Note that the manual annotations are not provided here. We parse the outputs of "Brief overview of gene set" and "Name of gene set" as the description and name in our evaluation.
[Start of Prompt]
You expert molacular biologist who studies how genes and pathways ralated using Perturb-seg
are writing brief overview gene are perturbed.
to similar transcriptional response upstream
Input: You will provided descriptions of the constituent genes. Ensure that your overview remains faithful to this domain knowledge.
Output format: Pill in each of these three sections.
1) Brief overview of gena set: Write
sentences summarizing how given genes related. Focus the
most apecific pathways that are common among these genes.
2) Upstream pathways may affect this gone set: Write one to two sentences describing what types of perturbations might
impact the expression of the genes in this gene ees. For ezample, you might consider pathways that are upstream of these genes compensatory mechanisms.
3) gene Summarize gene Constrainte:
worda.
Maintain profeceional tone throughout.
Do not comment on own writing.
Do not add any notas references. Do not make up additional information.
Do not discuss importance impact of gane sat. Focus only on function.
[Ead of Prompt]
[Staxt of Impus] (list of gene summaries) End of Imput)
C.2 QUESTION-ANSWERING TEMPLATES
For differential expression and direction of change, we used the following template for SUMMER.
23

---

Published as a conference paper at ICLR 2025
[Start of Prompt]
You expexs molecular biologist studiee genee related using Perturb-eeg. goal
Does CRISPRi knockdown of (perturbation) in (cell line) result in differential expression of (gene)?
You input:
Description perturbed geze (perturbation)):: description of gene
perturbed CRISPRi knockdown Description of gene Interest ((gene)): description gene, the Impact on which you to Infer
Context: description of cell line in which the genes are expressed
Examples: set of experimental observations that describe the impact of CRISPRi perturbations related genes
Output: Please extract the relevant parts of the examples that address these fire questions. Be specific.
1) Which
perturbed genes are most similar (parturbation) any, including (perturbation) itself)?
2) When parturbing (perturbation) or similar genes, what downstream pathways or gener are differentially expressed?
Justify your answer the observed outcomes.
3) Which
observed genea interest most similar [gens] any, including (gane) itself)?
4) What perturbations of upstream pathways or genes result in differential expression of (gene) or similar genes any)? Justify your
with the observed outcomes. 5) CRISPRI
(perturbation) {call line] likely example, if 2) and 4) are unrelated or only indirectly related, it is uniikely we will observe differential expression. On
expression {gane}?
other hand, if and 4) significantly overlap in specific genes or pathwaye, we may observe differential expression. Your final
should end with one of these three options and nothing else.
Knockdown (perturbation)
impact (gene).
Yes. Knockdown of (perturbation) results in differential expression of (gene).
There la insufficient evidence to determine how knockdown of {perturbation) affecta (gene).
[Ead Prompp]]
[Start Input]
Description parturbed gene (ppatuurattion)): (summary perturbation)
Description of gane of interest (fgene)): (summary of downstream gana)
Context: (sentence describing cell lina)
Examples: {liat examples)
[Ead Iaput]
For direction of change, we change the answer options to the following.
A) Knockdown of [perturbation) resulta in decrease in expression of (gene).
B) Knockdown (parturbation) remults increase in expression (gene).
The following template was used for the LLM (No CoT) baseline on differential expression.
You are an expert molecular biologiet who etudiee how genes are related using Perturb-seg.
You are given as Iaput:
Perturbed gene: description
that parturbed via CRISPRI knockdown
deacription gene, impact which you wish Context: faentence describing call line)
Clane
Question: If you knockdown the perturbed gene using CRISPRi, how does the gene of interest's expression change?
Anzwer: Your answer must end with one of these two choices and nothing else.
A) Knockdown of the parturbed gane does impact the gone of interest.
perturbed gene Format: Follow the same format Examplee and 2, and complete Example 3. Example 1.
Kuockdown
differential expression the gene
Input:
Parturbed gene: (ummmar parturbation)
(summary downstream gens)
Answer: (either ... or B) ...)
Example 2. (samn format as Example 1, opposite Answer)
Example (same format Example empty Answer)
The following template was used for the LLM (No retrieval) baseline on differential expression. Both answer options are provided twice each hypotheses, regardless of the ground truth answer.
You expert molecular biologiet who studiee how
related using Perturb-seg.
You given as Input:
gene: description gane
parturbed CRISPRI knockdown
Gane of interest: description of gane, the impact an which you wish to Infar
Hypothesls: hypothesta regarding how the specified perturbation affects the gane of interest
Context: (sentence describing cell line)
Question: you knockdown perturbed gene using CRISPRi,
gene expression change?
24

---

Published as a conference paper at ICLR 2025
goal identify
input supports the hypothesis, and explain
hypothesin likely to be true.
Output format: Please in the following four sections. Preserve the formatting and add the corresponding content.
Supporting evidence: Identify all relevant parts imput support the hypothesis.
2) Refuting evidence: Identify all relevant parts the iaput that refute the hypotheeis.
3) Explanation: Eased on the avidence, explain how to answer the queation, step by step. In particular,
there ls causal relationship from the perturbed gane to the gene of interest, explain how biological mechaniama relate the perturbed gene to the gene of interest.
there is no causal relationship from the perturbed gene to the gene of intereet, explain why. For example, the gene interest, there may be relationship between genes. if there insufficiett evidence to answer the question, say so.
perturbed gene may be downstream
4) Answer: Your answer muat end with one of these three choices and nothing else.
A) Knockdown of the perturbed gene does not impact the of interest.
B) Kuockdown
perturbed gene
differential expression the gene of knockdown parturbed gane gene
C) is insufficient evidance
Input:
Perturbed gene: (nummary of perturbation)
Gene of intereet: (summary downetresm gene)
Hypothesia:
(either A) ... ...)
For direction of change, we change the answer options to the following.
A) Kaockdown the perturbed gene results down-regulation of the gene of intereet,
B) Knockdown of the perturbed gene reeults in up-regulation of the gene of interset.
D EXAMPLES
Full LLM outputs are available in our data distribution. We sample real inputs/outputs for illustrative purposes.
D.I SUMMARIZATION EXAMPLES
Here is example of the knowledge-bass description for ABCE1, used as the [description] in the single-node summary.
Description gene: ATP binding cassette subfamily member 1
Gene products: ATP-binding cassette sub family E member 1
Punctions: (Microbial infection) May act chaperone post-tranelatioaa (Microblal infaction) Plays a rola in the down-regulation of the 2-5A/RNASHL pathway during encaphalomyeaardiis (EMCV) and HIV-1 infections., Muclecaide-triphhepphe ribosome disaseebby..
during HIV-1 capsid azsembly,
(NTPase) involved in ribosome recycling mediating
Qusternary structure: (Microbial infection) Interacte with HIV-1 proteins Vif and (Microbial infection) Interacte with HIV-2 protein Gag, Interacts with PINK1.
Interacta with: ABCEI, EIFaJ, GAG__HVIBL MATRX_ CHAV, PHOSP RROSP_RRABVE RABVE
Biological procese: enables endoribonuciesse inhibitor activity (Binds to and stops, prevente reduces the activity of eadoribemucieaze.
Here the unedited output for the single-node summary, which is used as input to the I-hop neigh- borhood summary.
The primary molecular and celiular function of gone ABCE1 facilitate ribonome recycling mediating ribosome disassembly through its mucleaidde-rrphhophaaad (NTPaase) activity, and to act as chaperone for poat-transiational evente during HIV-1 capeid assembly. Additionally, ABCEI playe pathway during viral infections. Perturbing gene ABCEI via gene knockdown capsid assembly, potentially leading to protein synthesis and altered viral replication dynamics.
the down-regulation
2-5A/RNASES disrupt ribosome recycling and viral
Here is an example of the knowledge graph relationships for ABCEI.
Relations to other genea:
ABCE1 enables Iron, sulfur cluater binding, ATP binding, ATP hydrolysis activity, CTPase activity, GTPase activity, eadoribonuclesse inhibitor activity, iron loa binding, ribonucleceide triphosphate phoephatase activity, riboeomal small subunit binding.
ABCE1 is
cytosolic ribosome
ABCE1 is involved in negative regulation endoribonuckesse activity, regulation of trenslation, rescue of stalled ribo- some, ribosomal subunit export from nucleus, ribasome disassebbly, translational initiation, translational terminationn, cytoplaam, cytosol, membrane, mitochondrial matrix, mitochondrion.
25

---

Published as a conference paper at ICLR 2025
Based evidence from experimental evidence humans, database evidence humans, ABCEI may physically interact with RNASEL.
Bazed evidence from experimental evideace in humane, experimental evidence in other animals, ABCEI physi- cally interact with EIFIAX, EIFJA, EIFSB, EIFSC, EIF3D, EIFSE, EIFaF, EIFSG, EIFSH, EIFII, EIFaK, EIFSL, EIF3M, G3BP2, LTO1, MFGES, PSMD14, RACKL, RPL12, RPL23, RPL4, RPL7A, RPL9, RPL9P7, RPL9P8, RPL9P9, RPS10, RPSIO-NUDTO, RPS11, RPS12, RPS15, RPS14, RPS15, RPSISA, RPS16, RPS17, RPS18, RPS19, RPS2, RPS20, RPS21, RPS24, RP825, RPS26, RPS27, RPS27A, RPS28, RPS29, RPS3, RPS3A, RPS4X, RPS5, RPS6, RPS7, RPS8, RPS9, RPSA, YAE1.
Based evidence experimental evidence in humans, experimental evidence other animala, literature evidence in humane, ABCEI may physically interact with EIFS, RPS23.
Besed on evidence from experimental evidence in humans, experimental evidence in other animale, literature evidence in humane, litersture evidence in other animals, ABCEI may paysically interact with EIFI, ETFI
Based on evidence from experimental evidence in humans, experimental evidence in other animals, literature evidence other animals, ABCEI may physically interact with EIFaJ, RPLPO.
Based on evidence from expetimental evidence in humane, literature evidence in humans, ABCEI may physically interact with DCP2, EIP2SS, PDPS, PSMDI, PSMD4.
Based on evidence from experimental evidence humans, literature evidence in humane, literature evidence other animals, ABCE1 may physically interact with EIF2S1, EIF2S2.
Based avidance from experimental evidence other animaln, litarature evidance humans, ABCE1 may physically interact with UBL4A.
Based on evidence from experimental evidence in other animals, literature evidence in other animals, ABCE1 may physically interact with DENR, EIFIE, MRTO4
In 293T calla, ABCEI may form complax with IGF2R
In the cytosol, ABCEI enables OAS antiviral reaponse, RNASEL binds 2'-5' ollgoadenylate, ABCE1 binda RN RNASEL, OAS antiviral response, RNASEL binda 2'-5' oligoadenylate, ABCE1 binds RNASEL
In the mitochondrial matrix, ABCEI enables Interferon alpha/beta signaling, ABCE1 binde RNASEL, inhibiting it, Interferon alpha/beta sigualing, ABCEI binds RNASEL, inhibiting
Here the unedited output for the 1-hop neighborhood summary.
Perturbatione that may impact the levels of ABCEI include allerations in the 2-5A/RNASEL pathway, HIV-1 capeid assembly, ribosome recycling, well changes in the interactions with its binding partners, as HIV-1 proteins Vif and Gag, HIV-2 protain Gag, PINKI, EIF3J, and MATRX_ CHAV. Additionally, parturbations in the regulation of translation, translational initiation, and translational tarmination may affect ABCE1 levels. Furthermore, changes in the activity of ite binding partnere, such as EIFIAX, EIF3A, EIF3E, and RPL12, may also impact ABCE1 expression.
D.2 QUESTION-ANSWERING EXAMPLES
Here is an example of the Clist of examples) used in the QA template the impact of AARS2 knockdown on AAKI.
C.2 retrieved to predict
Example 1: Impact of knocking down MTIP2 on SPHK1
Description of perturbed gens (MTIF2): The primary molecular and cellular function of gene MTIF2 is to facilitate the initiation protein synthesia in mitochondria by protacting formpimathlonyl.eNN from spontaneous hydrolysta and promoting Its binding to the 30S ribosomal subunita, ultimately leading to the formation of the 70S ribosomal complex. This process is crucial for the tranelation of mitochondrial DNA into functional proteinz. Perturbing gene MT1F2 via gene knockdown would likely disrupt the initiation of protein synthesis mitochondria, potentially leading to impaired mitochondrial function and protein syutheeis.
Description of gene of interest (SPHK1): The primary molecular and cellular function of gene SPHK1 is to catalyze the phosphorylation of aphingosine to form sphingosine 1-phosphate, a lipid mediator with both Intra- and extracellular functione, and to interact with varioue proteins to regulate downstream cellular processes Additionally, SPHK1 playe receptor signaling pathway, which reguiates transcription and other callular pro-
role in the pphinggsin----pheee ceesea. Perturbations that might impact expression gene SPHK1 include allerations in the sphingosine-1ppoooe --phoeee--p receptor tignaling pathway, changes in calmodulin binding, or disruptions to its interactions with other proteins such as ACY1, CATB, EPIA1, FHL2, LAMP2, SHLBI, SPHKAP, or TRAF6.
Outcome: A) Knockdown of MTIP does not impact SPHK1.
Example 2: Impact of knocking down CLPX on PTCD1
Description of perturbed gene (CLPX): The primary molecular and cellular function of gene CLPX is to ATP-dependent specificity component of the Clp protease complex, hydrolyzing ATP and forming a homcheramer ring structura that assembles with CLPP rings form the Cip complex. This complex is involved in protein degradation and quality control in the mitochondrial matrix. Parturbing gane CLPX via gene knockdown may disrupt the proper functioning
Clp protease complex, leading to impaired protein degradation and potential accumulation misfolded or damaged proteins in the mitochondrial matrix.
Description of gene of interest (PTCD1): The primary molecular and cellular function of gene PTCDI is negatively regulste leucine URNA levels, mitechondria-enoddel proteins, and COX activity while also affecting the 3'-proceesing of mitochondrial tRNAs, thereby infuencing mitochondrial protein symtheei.. As mitochondrial protein, PTCD1 associates with mitochondrial laucine tRNAs and interacts with various proteins, including ELAC2, to modulate its functions. Parturbationa that might impast expression gana PTCD1 include diaruptions mitochondrial IRNA metabolism, alterations in COX activity, or changes in the levela of interacting proteins, such as ELAC2, which could in turn aflect PTCDI's regulatory roles in mitochondrial protein syntheeis. Outcome: A) Knockdown of CLPX does not impact PTCDI.
(...)
26

---

Published as a conference paper at ICLR 2025
Example Impact of knocking NARS2 on REPS1
Deecription of perturbed gene (NARS2): The primary molecular and cellular function of gene NARS2 is to encode mitochondrial sminoscyl-tRNA synthelaze that catalyzes the attachment of asperagine to cognate IRNA, thereby participating in protein synthesia. This enzyme enables asparagin-tRRA ligase activity, facilitating the formation of Aan- L-separagine
(RNA(As))
IRNA(Aem). Perturbing gone NARS2 gene knockdown may disrupt proper charging of tRNA with asparagine, potentially leading to aberrant protein synthesin and mitochondrial dysfunction.
Description gene interest. (REPS1): The primary molecular and cellulrr function of gene REPS1 is to coordinate the celluaar actions of activated EGP receptors and Ral-GTFanes, potentially linking them to an EGF-responsive tyresine kinase through its interactions with CRK and GRB2. Additionally, REPS1 may be involved in hathrin--ediated endocy- toels through its interactions with AMPH, ITSN1, and SGIPI. Perturbations to the EGF signaling pathway, Pal-GTPaae activity, or clathrin mediate endocytosia may impact exprestion of gene REPSI.
Outcome: A) Knockdown of NARS2 does not impact REPSI.
Here is the unedited output for the above prompt.
(Start of Output]
Bazed the provided examples, will extract the most relevant parts addrees the five questions.
Which observed perturbed genes are moet similar AARS2 (if any, including AARS2 iteelf)?
From the examplas, we can sae that AARS2 is mitochondrial amtnoacyi-tRNA aynthetase, specifically responsible stlaching alanine (RNA(Al)). The most similer genes to AARS2 are likely to be other mitochondrial aminoecyl-LRNA synthatases, such as TARS2 (threonine-tRNA ligase), RARS2 (arginine:tRNA ligase), and NARS2 (Rsparaghe-tRMM similar functions When parturbing AARS2 similar genes, what downstream pathways or genes are differentially expressed? Justify your answer with the observed ouscomes.
ligase). There genes
involved protein ayathesis.
perturbing AARS2 similar genea genee. For instance, knocking down AARS2 does not affect the expresion of SPHK1, PTCDI, SLC35G2, LYRM7, RIN2, AAK1, or NECAP2. Similarly, knocking down TARS2, RARS2, or NARS2 does impact expression SLC35G2, REPS1, or NECAP2, reepectively. This suggests that the perturbation of AARS2 similer genes does not significantly affect downstream pathways or genes.
the exampies,
impact expression of
3. Which of the observad genas of Interest are most similar to AAKI (if any, including AAK1 itsalf)?
From the examples, we can see that AAXI gene involved in regulating clachrin-mediatdd endocytoals and interacts with clathrin and other proteins. The moet similar genes to AAK1 are likely to be other genes involved in clathrin- mediated endocytoeis, such NECAP2, which facilitates endocytosis through its interactions with adapser protein complexee AP-1 and AP-2, Other genes that interact with clathrin, such as RIN2, may also be slmiler to AAKI.
4. What perturbations of upstream pathways or genas result in differential expression of AAK1 or similar genes (if any)? Juatify your answer
the observad outcomes.
From the examples, we can sea that parturbing ganes invoived in clathrin-medlated endocytosia, such as NECAP2, does impact the expression of AAKI. Similarly, perturbing genee involved protein synthesia, affect the expression of AAKI This suggests that the perturbation of upstzeam pathways or genes does not significantly the expression of AAK1 or similar genes.
RPL10,
5. Is a CRISPRi knockdown of AARS2 in K562 colls likely to result in differential expression of AAK1?
Based on the obsarved outcomes, it is unlikell that CRISPRI kmockdown of AARS2 in K662 cells will result in diflerential expresston AAK1. The parturbation AAR similar ganes doea not impact expresston of downstream genas, including AAKI. Therefore, the answer ls: Knockdown AARS2 does not impact AAKI. [Ead Output]
D.3 OTHER EXAMPLES
We use the following for (sentence describing cell line), derived from Wikipedia.
X662 cells are immortalised myclogenous leukomia colls of the erythroleukemia type.
RPEI cells non-cancerous, bTEETTTTT-aat iimEo--immttaliieed near-euploid, adherent, and p53-positive cell line.
Jurkat are an immortalizad of human lymphocyta colls.
HepG2 cells are a human liver cancer cell line, derived from patient with a well differentiated hopatocellular carcinoma.
27


Reviews:
[{"review_id": "jOPBi0DPd1", "reviewer": "Reviewer_JGsH", "summary": "The paper introduces a benchmark for evaluating LLM reasoning on biological perturbation experiments, and an LLM-based framework that uses knowledge graphs and prior data to outperform current methods in interpretability and performance on these tasks.", "questions": "1. How well does SUMMER generalize to new or sparsely annotated biological datasets? Are there performance drops when applied to less-studied cell lines or organisms?\n2. Did the authors perform an error analysis to determine whether specific types of perturbations or gene functions are harder for SUMMER to predict accurately?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method appears to be sound, combining RAG and COT prompting with knowledge graphs for handling complex biological relationships.\n2. The proposed ramework emphasizes interpretable outputs, which is beneficial to biological research.\n3. The evaluation was thorough, with both graph and language-centric baselines across multiple datasets.", "weaknesses": "1. The model focuses on discrete perturbation outcomes and does not address combinatorial perturbations, which are common in biological studies.\n2. The interpretability of the method comes at the cost of additional complexity in prompt engineering, and it is not known how the performance is sensitive to the prompt design.\n3. The paper acknowledges limitations in current evaluation metrics for enrichment tasks. It would be better to see the utilization of new, domain-specific metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fRPFlCk1Gq", "reviewer": "Reviewer_2VEF", "summary": "This paper introduces PERTURBQA, focusing on prompting LLMs for gene perturbation and gene set enrichment. It also presents a reasoning method, SUMMER, designed for this task. Experiments demonstrate the effectiveness of the proposed approach.", "questions": "Retrieved Content for Reasoning: Is the retrieved content primarily focused on the gene's function? If so, could this lead to information leakage in the question? For example, if asked about the influence of gene A on gene C, and the retrieved content on gene A directly states that A turn on C, wouldn’t this turn the task into reading comprehension rather than genuine prediction?\n\nMulti-Hop Reasoning: How does SUMMER handle multi-hop reasoning if it only retrieves information on the one hop neighbors of the perturbation and target gene?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed task of using large language models (LLMs) to address gene-related tasks is innovative and engaging. \n\nThe paper is clearly presented, well-written, and easy to follow.", "weaknesses": "Limited Experimental Insight: The experiments primarily conclude that SUMMER outperforms baselines but provide few additional insights into the new task. Given the novelty of using LLMs for this type of gene analysis, more extensive experiments and in-depth analysis would be valuable.\n\nInsufficient Baselines and Model Comparisons: The baselines selected are not sufficiently comprehensive. Numerous studies have explored LLM reasoning with graph data or text retrieval, which are closely related to this method. Including these in the comparisons could yield deeper insights into the effectiveness of LLMs for gene-related tasks.\n\nMetrics for Gene Set Enrichment: The suitability of ROUGE-1 recall and BERT Score for measuring the accuracy of gene set enrichment results is questionable. Human evaluation or evaluations with LLMs may offer more reliable assessments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iatrDXhwnb", "reviewer": "Reviewer_psYt", "summary": "This paper introduces PERTURBQA, a benchmark suite for evaluating LLMs in reasoning over structured biological data from genetic perturbation experiments. The paper also presents SUMMER, a language-based framework designed to predict differential expression and direction of gene changes as well as to perform gene set enrichment. SUMMER combines knowledge graphs and retrieval-augmented generation to enhance interpretability, matching or surpassing state-of-the-art models on PERTURBQA tasks. The benchmark is designed to help researchers interpret outcomes in high-content genetic perturbation experiments and understand model limitations.", "questions": "NA", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Novel Application of Language-Based Reasoning in Pertburbation Task:** The paper offers a novel application of language-based reasoning to biological data, allowing PERTURBQA tasks to be approached in an interpretable way that benefits domain experts.\n2. **Comprehensive Benchmark Design:** PERTURBQA includes real-world tasks relevant to differential expression, gene direction, and gene set enrichment, providing a holistic assessment of model reasoning on biological data.\n3. **Interpretable Model Outputs:** SUMMER’s use of knowledge graphs and retrieval-augmented generation produces outputs that domain experts can readily interpret, addressing the limitations of black-box models in biological contexts.", "weaknesses": "1.**Insufficient Related Work Discussion:** The paper could better situate its contribution within the existing literature, especially regarding related work on several aspects: 1. graph-to-text works: such as [Zhao et al., 2023](https://arxiv.org/abs/2310.01089), [Chen et al., 2023](https://arxiv.org/abs/2307.03393), check the [survey](https://arxiv.org/pdf/2407.06564) for details. 2. graph RAG works, e.g. [GraphRAG](https://arxiv.org/pdf/2404.16130), [He et al., 2024](https://arxiv.org/abs/2402.07630), [Mavromatis et al., 2024](https://arxiv.org/abs/2405.20139) . Discussing these works would strengthen the contextual grounding of the proposed approach.\n\n2.**Marginal Technical Contribution:** As stated in W1, both ideas of using LLM for graph tasks and graph RAG are widely studied before, which renders the contribution of this paper marginal.\n\n3.**Limited Baselines:** A wider range of recent baselines, especially from the graph domain (e.g. those mentioned above in W1 and W2), should be included to provide a more comprehensive evaluation and enable better comparison of SUMMER's effectiveness.\n\n4.**Potential Data Leakage Concerns:** Given that LLaMA3 might have been exposed to substantial amounts of biological data, including related gene interactions, there’s a risk of data leakage. A clearer evaluation of SUMMER’s performance independent of potential pre-trained biases could help clarify if its good performance stems from model design or from pre-existing knowledge in the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jghMGeOjAl", "reviewer": "Reviewer_ii1Q", "summary": "The paper introduces PERTURBQA, a benchmark for using language models to interpret genetic perturbation experiments. These experiments reveal gene functions but are expensive. The proposed SUMMER framework combines biological text data and experimental results, outperforming knowledge-graph-only models on tasks like predicting gene expression changes and gene set enrichment. SUMMER’s language-based outputs offer clearer insights for biologists, enhancing interpretability and accessibility in modeling biological data.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well-organized and easy to follow.\n\nThe paper proposed an interesting problem of using LLMs to predict gene pertubation.", "weaknesses": "1. The biggest concern of the proposed method is the problem itself. For example, SUMMER relies on the biology knowledge graph. However, the biology knowledge graph is built based on the experiments and analysis. Compared with the gene perturbation prediction from scRNA, the knowledge-building quiring updates the knowledge at once to build the possible links among genes. This indicates that this method has a risk of overfitting the known knowledge and being unable to discover new perturbation genes; it is more like a retrieval system for gene interactions we already know. \n\nTo further explain how the proposed method can generate more situations, my suggestion is to include more datasets to avoid overfitting. For example, in scGPT[1], the evaluation for perturbation is across 3 different perturbation datasets. \n\n2. In the previous study of gene perturbation, one gene input can output several gene expressions, which are judged at one time. However, the LLM framework suggests that the source and target are a pair of data. This means that if we want to see every gene expression after perturbation, we should go through every combination of gene pairs with LLMs. This is really time-consuming in this setting. Besides, the experiments are really limited to a small size of genes.\n\nIn GEARS, scGPT, and CellPLM, they set the perturbation tasks for both 1 gene unseen and 2 genes unseen, and they can claim that the proposed method can handle different perturbation situations. However, in this paper, there are no such illustrations. Besides, in GEARS, the authors predict 102 genes at one time for novel gene perturbation analysis (Fig5a in GEARS). In scGPT, they predict 210 combinations (Fig3 in scGPT). In this paper, there is no evidence that the proposed method can do such things. \n\n3. The authors should include gene pretrained methods such as scGPT[1] and Geneformer[2]. The text-based methods are pretrained on lots of textual information, the comparing method is unfair with only MLP, GAT, and GEARS. \n\nAs I mentioned before, the pretrained methods on the single-cell data are considered one of the most powerful methods in gene perturbation tasks at this moment. In scGPT, they report scGPT can achieve 50% higher than GEARS. Instead of training on lots of textual information, they trained directly from gene expressions. If the authors want to claim the textual-based pretrained method is more powerful than we predict genes from scRNA data, they need to discuss these models.\n\n4. Compared with other work in gene expression from ML method, such as CellPLM[3], LanCell[4], CellSentence[5] this paper lacks biology analysis in bio view. This limited the application to further applications.\n\nIn scGPT and GEARS, they have a figure of predicted gene expression profiles of the perturbation conditions. Other papers are not perturbation-specific methods, but they are ai4biology papers that are published in the top AI conferences. They also show some biological analysis to strengthen their methods for their own contribution, such as marking genes on the cells to show the cell states.\n\n[1] Cui H, Wang C, Maan H, et al. scGPT: toward building a foundation model for single-cell multi-omics using generative AI[J]. Nature Methods, 2024: 1-11.\n\n[2] Theodoris C V, Xiao L, Chopra A, et al. Transfer learning enables predictions in network biology[J]. Nature, 2023, 618(7965): 616-624.\n\n[3] Wen H, Tang W, Dai X, et al. CellPLM: Pre-training of Cell Language Model Beyond Single Cells[C]//The Twelfth International Conference on Learning Representations.\n\n[4] Zhao S, Zhang J, Wu Y, et al. LangCell: Language-Cell Pre-training for Cell Identity Understanding[C]//Forty-first International Conference on Machine Learning.\n\n[5] Levine D, Rizvi S A, Lévy S, et al. Cell2Sentence: Teaching Large Language Models the Language of Biology[C]//Forty-first International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZYnAD1xj8W", "reviewer": "Reviewer_qAqz", "summary": "This paper proposed an in-silico approach to solving the biological perturbation experiment results prediction. The original idea is to enrich the graph-structure-based cellular responses textually. In terms of technique details, the authors proposed PertubQA as a pre-processing step and omitted minor changes in the gene expression level. Then, the authors adopted prompt engineering to retrieve the structured knowledge crossing gene-gene interaction result and gene description. Following a CoT manner, the LLM generates the final answers.", "questions": "Q1. Please describe the distinct contributions of PerturbQA beside a RAG- or a meta-path-walk-like text description generation. How do you ensure the description is correct? For instance, PubmedQA is generated from many articles and certified by human experts. \n\nFor the rest of the question, please refer to weak points.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "S1. Adopting LLM in biological analysis is an interesting topic. This study follows a combination of LLM-related tech fashion. \n\nS2. The authors developed a PertubQA dataset to validate the following research on textual enrichment of perturbation analysis. \n\nS3. Detailed examples and case studies are provided to show the effectiveness of the framework.", "weaknesses": "W1. Part of the technical contribution is related to GenePT, which adopted a textual description and gene expression value to extract the cell representation, and GEARs, which combined prediction on a genetic relation graph (gene-coexpression graph and GO graph).  \n\nW2. The contributions of the proposed PerturbQA dataset are unclear. Why do we need a vague textual description of gene perturbation instead of a formalized graph structure with exactly numeric data? \n\nW3. Missing technical details. I couldn't find the setting (or formal definition) of the study problem. Those make this paper hard to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)