Contextualizing biological perturbation experiments through language

Authors: Menghua Wu, Russell Littman, Jacob Levine, Lin Qiu, Tommaso Biancalani, David Richmond, Jan-Christian Huetter
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: large language models, Perturb-seq, perturbation experiments, knowledge graphs, retrieval-augmented generation, chain of thought prompting

Abstract:
High-content perturbation experiments allow scientists to probe biomolecular systems at unprecedented resolution, but experimental and analysis costs pose significant barriers to widespread adoption. Machine learning has the potential to guide efficient exploration of the perturbation space and extract novel insights from these data. However, current approaches neglect the semantic richness of the relevant biology, and their objectives are misaligned with downstream biological analyses. In this paper, we hypothesize that large language models (LLMs) present a natural medium for representing complex biological relationships and rationalizing experimental outcomes. We propose PerturbQA, a benchmark for structured reasoning over perturbation experiments. Unlike current benchmarks that primarily interrogate existing knowledge, PerturbQA is inspired by open problems in perturbation modeling: prediction of differential expression and change of direction for unseen perturbations, and gene set enrichment. We evaluate state-of-the-art machine learning and statistical approaches for modeling perturbations, as well as standard LLM reasoning strategies, and we find that current methods perform poorly on PerturbQA. As a proof of feasibility, we introduce Summer (SUMMarize, retrievE, and answeR, a simple, domain-informed LLM framework that matches or exceeds the current state-of-the-art. Our code and data are publicly available at https://github.com/genentech/PerturbQA.

Decision: accept (poster)
Reviewer scores: 6, 3, 5, 6, 3

Reviews and discussion:
<thread>
creation date:
1730175478643

thread_type:
official_review

<thread_object>
creator: reviewer_qaqz

<content>
summary:
This paper proposed an in-silico approach to solving the biological perturbation experiment results prediction. The original idea is to enrich the graph-structure-based cellular responses textually. In terms of technique details, the authors proposed PertubQA as a pre-processing step and omitted minor changes in the gene expression level. Then, the authors adopted prompt engineering to retrieve the structured knowledge crossing gene-gene interaction result and gene description. Following a CoT manner, the LLM generates the final answers.

strengths:
S1. Adopting LLM in biological analysis is an interesting topic. This study follows a combination of LLM-related tech fashion. 

S2. The authors developed a PertubQA dataset to validate the following research on textual enrichment of perturbation analysis. 

S3. Detailed examples and case studies are provided to show the effectiveness of the framework.

weaknesses:
W1. Part of the technical contribution is related to GenePT, which adopted a textual description and gene expression value to extract the cell representation, and GEARs, which combined prediction on a genetic relation graph (gene-coexpression graph and GO graph).  

W2. The contributions of the proposed PerturbQA dataset are unclear. Why do we need a vague textual description of gene perturbation instead of a formalized graph structure with exactly numeric data? 

W3. Missing technical details. I couldn't find the setting (or formal definition) of the study problem. Those make this paper hard to follow.

questions:
Q1. Please describe the distinct contributions of PerturbQA beside a RAG- or a meta-path-walk-like text description generation. How do you ensure the description is correct? For instance, PubmedQA is generated from many articles and certified by human experts. 

For the rest of the question, please refer to weak points.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730342098353

thread_type:
official_review

<thread_object>
creator: reviewer_ii1q

<content>
summary:
The paper introduces PERTURBQA, a benchmark for using language models to interpret genetic perturbation experiments. These experiments reveal gene functions but are expensive. The proposed SUMMER framework combines biological text data and experimental results, outperforming knowledge-graph-only models on tasks like predicting gene expression changes and gene set enrichment. SUMMER’s language-based outputs offer clearer insights for biologists, enhancing interpretability and accessibility in modeling biological data.

strengths:
The paper is well-organized and easy to follow.

The paper proposed an interesting problem of using LLMs to predict gene pertubation.

weaknesses:
1. The biggest concern of the proposed method is the problem itself. For example, SUMMER relies on the biology knowledge graph. However, the biology knowledge graph is built based on the experiments and analysis. Compared with the gene perturbation prediction from scRNA, the knowledge-building quiring updates the knowledge at once to build the possible links among genes. This indicates that this method has a risk of overfitting the known knowledge and being unable to discover new perturbation genes; it is more like a retrieval system for gene interactions we already know. 

To further explain how the proposed method can generate more situations, my suggestion is to include more datasets to avoid overfitting. For example, in scGPT[1], the evaluation for perturbation is across 3 different perturbation datasets. 

2. In the previous study of gene perturbation, one gene input can output several gene expressions, which are judged at one time. However, the LLM framework suggests that the source and target are a pair of data. This means that if we want to see every gene expression after perturbation, we should go through every combination of gene pairs with LLMs. This is really time-consuming in this setting. Besides, the experiments are really limited to a small size of genes.

In GEARS, scGPT, and CellPLM, they set the perturbation tasks for both 1 gene unseen and 2 genes unseen, and they can claim that the proposed method can handle different perturbation situations. However, in this paper, there are no such illustrations. Besides, in GEARS, the authors predict 102 genes at one time for novel gene perturbation analysis (Fig5a in GEARS). In scGPT, they predict 210 combinations (Fig3 in scGPT). In this paper, there is no evidence that the proposed method can do such things. 

3. The authors should include gene pretrained methods such as scGPT[1] and Geneformer[2]. The text-based methods are pretrained on lots of textual information, the comparing method is unfair with only MLP, GAT, and GEARS. 

As I mentioned before, the pretrained methods on the single-cell data are considered one of the most powerful methods in gene perturbation tasks at this moment. In scGPT, they report scGPT can achieve 50% higher than GEARS. Instead of training on lots of textual information, they trained directly from gene expressions. If the authors want to claim the textual-based pretrained method is more powerful than we predict genes from scRNA data, they need to discuss these models.

4. Compared with other work in gene expression from ML method, such as CellPLM[3], LanCell[4], CellSentence[5] this paper lacks biology analysis in bio view. This limited the application to further applications.

In scGPT and GEARS, they have a figure of predicted gene expression profiles of the perturbation conditions. Other papers are not perturbation-specific methods, but they are ai4biology papers that are published in the top AI conferences. They also show some biological analysis to strengthen their methods for their own contribution, such as marking genes on the cells to show the cell states.

[1] Cui H, Wang C, Maan H, et al. scGPT: toward building a foundation model for single-cell multi-omics using generative AI[J]. Nature Methods, 2024: 1-11.

[2] Theodoris C V, Xiao L, Chopra A, et al. Transfer learning enables predictions in network biology[J]. Nature, 2023, 618(7965): 616-624.

[3] Wen H, Tang W, Dai X, et al. CellPLM: Pre-training of Cell Language Model Beyond Single Cells[C]//The Twelfth International Conference on Learning Representations.

[4] Zhao S, Zhang J, Wu Y, et al. LangCell: Language-Cell Pre-training for Cell Identity Understanding[C]//Forty-first International Conference on Machine Learning.

[5] Levine D, Rizvi S A, Lévy S, et al. Cell2Sentence: Teaching Large Language Models the Language of Biology[C]//Forty-first International Conference on Machine Learning.

questions:
see weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638971293

thread_type:
official_review

<thread_object>
creator: reviewer_psyt

<content>
summary:
This paper introduces PERTURBQA, a benchmark suite for evaluating LLMs in reasoning over structured biological data from genetic perturbation experiments. The paper also presents SUMMER, a language-based framework designed to predict differential expression and direction of gene changes as well as to perform gene set enrichment. SUMMER combines knowledge graphs and retrieval-augmented generation to enhance interpretability, matching or surpassing state-of-the-art models on PERTURBQA tasks. The benchmark is designed to help researchers interpret outcomes in high-content genetic perturbation experiments and understand model limitations.

strengths:
1. **Novel Application of Language-Based Reasoning in Pertburbation Task:** The paper offers a novel application of language-based reasoning to biological data, allowing PERTURBQA tasks to be approached in an interpretable way that benefits domain experts.
2. **Comprehensive Benchmark Design:** PERTURBQA includes real-world tasks relevant to differential expression, gene direction, and gene set enrichment, providing a holistic assessment of model reasoning on biological data.
3. **Interpretable Model Outputs:** SUMMER’s use of knowledge graphs and retrieval-augmented generation produces outputs that domain experts can readily interpret, addressing the limitations of black-box models in biological contexts.

weaknesses:
1.**Insufficient Related Work Discussion:** The paper could better situate its contribution within the existing literature, especially regarding related work on several aspects: 1. graph-to-text works: such as [Zhao et al., 2023](https://arxiv.org/abs/2310.01089), [Chen et al., 2023](https://arxiv.org/abs/2307.03393), check the [survey](https://arxiv.org/pdf/2407.06564) for details. 2. graph RAG works, e.g. [GraphRAG](https://arxiv.org/pdf/2404.16130), [He et al., 2024](https://arxiv.org/abs/2402.07630), [Mavromatis et al., 2024](https://arxiv.org/abs/2405.20139) . Discussing these works would strengthen the contextual grounding of the proposed approach.

2.**Marginal Technical Contribution:** As stated in W1, both ideas of using LLM for graph tasks and graph RAG are widely studied before, which renders the contribution of this paper marginal.

3.**Limited Baselines:** A wider range of recent baselines, especially from the graph domain (e.g. those mentioned above in W1 and W2), should be included to provide a more comprehensive evaluation and enable better comparison of SUMMER's effectiveness.

4.**Potential Data Leakage Concerns:** Given that LLaMA3 might have been exposed to substantial amounts of biological data, including related gene interactions, there’s a risk of data leakage. A clearer evaluation of SUMMER’s performance independent of potential pre-trained biases could help clarify if its good performance stems from model design or from pre-existing knowledge in the model.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730653580330

thread_type:
official_review

<thread_object>
creator: reviewer_2vef

<content>
summary:
This paper introduces PERTURBQA, focusing on prompting LLMs for gene perturbation and gene set enrichment. It also presents a reasoning method, SUMMER, designed for this task. Experiments demonstrate the effectiveness of the proposed approach.

strengths:
The proposed task of using large language models (LLMs) to address gene-related tasks is innovative and engaging. 

The paper is clearly presented, well-written, and easy to follow.

weaknesses:
Limited Experimental Insight: The experiments primarily conclude that SUMMER outperforms baselines but provide few additional insights into the new task. Given the novelty of using LLMs for this type of gene analysis, more extensive experiments and in-depth analysis would be valuable.

Insufficient Baselines and Model Comparisons: The baselines selected are not sufficiently comprehensive. Numerous studies have explored LLM reasoning with graph data or text retrieval, which are closely related to this method. Including these in the comparisons could yield deeper insights into the effectiveness of LLMs for gene-related tasks.

Metrics for Gene Set Enrichment: The suitability of ROUGE-1 recall and BERT Score for measuring the accuracy of gene set enrichment results is questionable. Human evaluation or evaluations with LLMs may offer more reliable assessments.

questions:
Retrieved Content for Reasoning: Is the retrieved content primarily focused on the gene's function? If so, could this lead to information leakage in the question? For example, if asked about the influence of gene A on gene C, and the retrieved content on gene A directly states that A turn on C, wouldn’t this turn the task into reading comprehension rather than genuine prediction?

Multi-Hop Reasoning: How does SUMMER handle multi-hop reasoning if it only retrieves information on the one hop neighbors of the perturbation and target gene?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665035210

thread_type:
official_review

<thread_object>
creator: reviewer_jgsh

<content>
summary:
The paper introduces a benchmark for evaluating LLM reasoning on biological perturbation experiments, and an LLM-based framework that uses knowledge graphs and prior data to outperform current methods in interpretability and performance on these tasks.

strengths:
1. The method appears to be sound, combining RAG and COT prompting with knowledge graphs for handling complex biological relationships.
2. The proposed ramework emphasizes interpretable outputs, which is beneficial to biological research.
3. The evaluation was thorough, with both graph and language-centric baselines across multiple datasets.

weaknesses:
1. The model focuses on discrete perturbation outcomes and does not address combinatorial perturbations, which are common in biological studies.
2. The interpretability of the method comes at the cost of additional complexity in prompt engineering, and it is not known how the performance is sensitive to the prompt design.
3. The paper acknowledges limitations in current evaluation metrics for enrichment tasks. It would be better to see the utilization of new, domain-specific metrics.

questions:
1. How well does SUMMER generalize to new or sparsely annotated biological datasets? Are there performance drops when applied to less-studied cell lines or organisms?
2. Did the authors perform an error analysis to determine whether specific types of perturbations or gene functions are harder for SUMMER to predict accurately?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732136426042

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
To all reviewers

comment:
Dear reviewers,

Thank you for reading our paper and providing thorough feedback. We have updated our paper with a number of illustrative examples, which we hope will provide further insight into the questions raised here.

We would also like to clarify the scope and contribution of this work, which we believe are central to its interpretation.

1. **The primary contribution of this work is PerturbQA, a carefully curated benchmark for language-based reasoning and scientific discovery, in the context of single-cell perturbations.** Compared to existing benchmarks, which focus on scientific coding tasks [1] or reasoning over known facts [2], PerturbQA is *predictive* in nature, and its tasks are unsolved. PerturbQA draws upon experimental assays and harmonized knowledge graphs to replicate the reasoning required to "connect the dots" between known biology and unanswered questions.

    **Information leakage is the primary argument against this vision. However, we believe that the experimental outcomes from which we derive PerturbQA are minimally represented in existing knowledge bases and LLM pretrained weights.** Biological knowledge graphs tend to report relationships that have been well-validated by targeted studies, rather than the outcomes of large-scale screens. The gene pairs whose relationship we query are very much a *superset* of genes with well-characterized relationships. Specifically, only ~3% of gene pairs in our test sets physically interact (row 2, in *any* context, including other animals), and only ~20% share any pathway/annotation, including at the coarsest levels (row 4).

    | | K562 | RPE1 | HepG2 | Jurkat |
    |-|-|-|-|-|
    | physical, DE | 0.094 | 0.063 | 0.075 | 0.106 |
    | physical, total | 0.032 | 0.025 | 0.027 | 0.029 |
    | network, DE | 0.214 | 0.204 | 0.218 | 0.253 |
    | network, total | 0.222 | 0.209 | 0.220 | 0.208 |

    There is little difference between the positive/negative pairs in terms of higher-level connectivity (row 3 vs. 4). Physically interacting genes are more likely to result in differential expression in our dataset (row 1 vs 2), but **having a physical interaction is minimally predictive of DE,** as AUC hovers around 0.5, while Ours is consistently better than random guessing (from Table 1).

    | | K562 | RPE1 | HepG2 | Jurkat |
    |-|-|-|-|-|
    | physical = 1 | 0.53 | 0.52 | 0.52 | 0.54 |
    | Ours | 0.60 | 0.58 | 0.61 | 0.58 |

    Finally, Nadig et al. 2024 was published strictly after we downloaded the knowledge graphs. While the cell lines in question have been studied in prior work, in other contexts, Nadig et al. 2024 released the *first* large-scale Perturb-seq screens in these two cell lines.

2. **Classic LLM reasoning strategies achieve near-random performance  out-of-the-box on PerturbQA, and it is non-trivial to adapt them in *domain-aware* ways.** Within the past two years, there has been a plethora of brilliant, inference-time LLM strategies, from in-context learning, to CoT, ToT, (Graph) RAG, and more. However, as we demonstrate for ICL and CoT, naively applying existing templates to biological reasoning leads to near-random performance. **This also demonstrates the the *lack* of answers within the pretrained weights themselves.** With regards to retrieval-based baselines, a vast amount of biological literature is unfortunately inaccessible behind paywalls, or otherwise [subject to terms unfavorable for AI development](https://www.genecards.org/). As a result, it is difficult to benchmark standard retrieval-based strategies on equal footing.

3. To demonstrate that language-based reasoning is *feasible* on PerturbQA, **SUMMER integrates standard LLM techniques with domain-specific ways to query structured knowledge.** To the best of our knowledge, **SUMMER is the first fully LLM-based method for unseen perturbation prediction and rationalization**, without relying on any external classifiers or embedding models. Throughout our work, we acknowledge that techniques like CoT and retrieval are common in current LLM systems. The key contribution lies in *what* information is useful to retrieve, and *how* to frame the prompts to encourage reasonable reasoning.

[1] Rein et al. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. 2023.

[2] Laurent et al. LAB-Bench: Measuring Capabilities of Language Models for Biology Research. 2024.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This is a new benchmark contributing to understanding the reasoning capability of large language models in structured biological data.
This task will be of interest to communities interested in the space of AI and Biology.  
However the recommendation at this point is borderline
Since the contribution is not so much methodological, it is unclear to make a strong case for this paper. On the other hand, ICLR has  Datasets as one of the topics in the call for papers and one could fit this paper under that category.