BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments

## Abstract

Abstract Agents based on large language models have shown great potential in accelerating scientific discovery by leveraging their rich background knowledge and reasoning capabilities. In this paper, we introduce BioDiscoveryAgent , an agent that designs new experiments, reasons about their outcomes, and efficiently navigates the hypothesis space to reach desired solutions. We demonstrate our agent on the problem of designing genetic perturbation experiments, where the aim is to find a small subset out of many possible genes that, when perturbed, result in a specific phenotype (e.g., cell growth). Utilizing its biological knowledge, BioDiscoveryAgent can uniquely design new experiments without the need to train a machine learning model or explicitly design an acquisition function as in Bayesian optimization. Moreover, BioDiscoveryAgent using Claude 3.5 Sonnet achieves an average of 21% improvement in predicting relevant genetic perturbations across six datasets, and a 46% improvement in the harder task of non-essential gene perturbation, compared to existing Bayesian optimization baselines specifically trained for this task. Our evaluation includes one dataset that is unpublished, ensuring it is not part of the language model’s training data. Additionally, BioDiscoveryAgent predicts gene combinations to perturb more than twice as accurately as a random baseline, a task so far not explored in the context of closed-loop experiment design. The agent also has access to tools for searching the biomedical literature, executing code to analyze biological datasets, and prompting another agent to critically evaluate its predictions. Overall, BioDiscoveryAgent is interpretable at every stage, representing an accessible new paradigm in the computational design of biological experiments with the potential to augment scientists’ efficacy.

## 1 Introduction

Scientific discovery often resembles a search problem, where multiple hypotheses are tested to find one that leads to informative outcomes (Simon et al., ; Langley et al., ). AI agents, particularly those built using large language models (LLMs), show promising capabilities for this task (Wang et al., ). Previous studies have demonstrated that LLMs can effectively learn from the scientific literature (Wang et al., ; Boiko et al., ; Moor et al., ), compare various experimental plans (Liu et al., ; ; Huang et al., ) and use tools such as executing code (Schick et al., ) or accessing databases (Jin et al., ).

Biomedical research, which heavily relies on iterative experimentation, stands to benefit significantly from such agents. One key challenge in this field is identifying drug targets—typically proteins that, when engaged by a drug, lead to a desired change in disease phenotype. Misidentification of these drug targets is a major cause of failure in clinical trials (Nelson et al., ). CRISPR-based genetic perturbation experiments are instrumental in addressing this challenge. These experiments involve the repression or activation of genes that code for proteins, followed by the measurement of the resulting biological effects, helping to identify drug targets that could reverse disease effects (Przybyla & Gilbert, ). These perturbation screens or forward genetics screens (Schneeberger, ; Moresco et al., ), have been transformative in areas such as drug target discovery (Wang & Doudna, ), elucidating disease mechanisms (Mamedov et al., ), cell engineering (Lim, ), gene therapy (Kalos et al., ), and immunotherapy (Goodman et al., ).

However, experimentally perturbing every single gene is costly. A perturbation screen typically perturbs around 19,000 protein-coding genes, yet anywhere between a handful of genes to a few thousand may exhibit the desired phenotype. When perturbing combinations of genes, this search space is even larger. By strategically designing these experiments in smaller batches that prioritize genes likely to result in meaningful phenotypic effects, it is possible to enhance the efficiency of the search process (King et al., ; Cleary et al., ; Huang et al., ; Roohani et al., ).

Recent work has benchmarked Bayesian optimization algorithms for this task, but this requires training bespoke machine learning models that are often difficult to interpret, on small datasets (Mehrjou et al., ; Lyle et al., ) (Figure 1a). Moreover, these models are unable to leverage the vast biological prior knowledge contained within the scientific literature that is valuable during early experimentation. In contrast, LLMs have been exposed to the scientific literature and can recall biomedical knowledge (Gao et al., ). Such models have shown state of the art performance in information retrieval for clinical question answering (Moor et al., ), patient matching for clinical trials (Wornow et al., ), gene set identification (Hu et al., ), gene function prediction (Chen & Zou, ) and cell type annotation using gene expression data (Hou & Ji, ). However, LLM-based agents have not yet been used for closed-loop biological experiment design.

For effective experiment design, agents require both domain-specific knowledge and the ability to interpret and reason over experimental results. While LLMs demonstrate strong capabilities in these areas, their full applicability across diverse biological contexts requires access to not only the literature but also external sources like tabular datasets. A careful balance must be struck between granting the LLM freedom to explore the action space of genes to perturb and ensuring the selected genes are biologically valid and well-motivated. The LLM must carry information across successive prompts to maintain a consistent experimental strategy. Additionally, the agent’s decision-making should be interpretable, ideally including literature citations and mechanisms for human feedback.

Here, we overcome some of these challenges and introduce *BioDiscoveryAgent*, an agent that designs genetic perturbation experiments using only an LLM paired with a suite of tools (Figure 1b). In each round, the agent constructs a prompt that includes both the task description and experimental results from previous rounds. This prompt is fed into the LLM, and the response identifies genes to perturb for the next round of experiments. BioDiscoveryAgent can also leverage different tools: it can search the scientific literature for relevant articles, execute code to analyze datasets that are inaccessible through textual sources and prompt another agent to critique the predictions of the initial agent.

BioDiscoveryAgent uniquely designs genetic perturbation experiments without relying on a specifically trained machine learning model or an explicitly defined acquisition function. It accomplishes this by leveraging an LLM that can effectively integrate information from both its prior knowledge and experimental results. Moreover, BioDiscoveryAgent using Claude 3.5 Sonnet outperforms baseline methods for experiment design, identifying 21% more experimental hits after five experimental rounds of 128 genes each, which is approximately 17 additional phenotypically relevant genes per dataset. On the harder task of only predicting non-essential genes, it identifies 46% more hits than baselines. BioDiscoveryAgent (Claude 3.5 Sonnet) also shows more than twice as high performance in predicting responses to combinatorial gene perturbations compared to a random baseline, exploring a new setting not previously considered. Unlike conventional approaches that are entirely black-box, the agent’s decision-making is fully transparent at every stage and can be enhanced through using tools such as LLM-based model critique. Overall, BioDiscoveryAgent utilizes its vast biological knowledge along with the ability to reason over insights from previous experimental results to offer an accessible and interpretable method for designing genetic perturbation experiments.

Figure: Figure 1: An AI agent for closed-loop experiment design. (a) Conventional Bayesian optimization approach for experiment design involves training a machine learning model in every experimental round, scoring all perturbations and defining an acquisition function for selecting genes to perturb in the next round. (b) Overview of BioDiscoveryAgent. In every round, the agent is given a prompt that describes the research problem, its role and task as well as experimental results from previous rounds. It generates a set of genes to perturb in the next round and provides reasoning for its prediction. The agent primarily makes use of a large language model for its predictions but also has access to additional tools such as the ability to search for relevant scientific papers on the internet.
Refer to caption: x1.png

## 2 Problem formulation

We study the following closed-loop experimental design task: an agent determines which genes to perturb in each experimental round, using the results to plan subsequent rounds. The objective is to maximize the number of hit genes or gene combinations that are detected over the full course of rounds. In particular, we test our model on two real-world experimental settings:

Single-gene perturbation: Let $\mathcal{G}$ represent the set of all protein-coding genes in humans, where $|\mathcal{G}|\approx$ 19,000. The effect of perturbing a gene $g\in\mathcal{G}$, is denoted by a phenotypic response function $f(g)$, where $f:\mathcal{G}\rightarrow\mathbb{R}$ maps each gene to a real-valued phenotypic response. In this setting, we define an experimental round as the perturbation of a set of genes $\{g_{1},g_{2},\dots,g_{B}\}\subseteq\mathcal{G}$. Each gene in this set is perturbed individually within separate cells. The goal of the experiment is to identify those genes that upon perturbation produce a desired phenotype, $f(g)>\tau$ for some threshold $\tau$. These genes are referred to as hits for that phenotype.

The goal for the agent is to guide the design of these experiments, such that over $t=1,2,\dots,T$ experimental rounds, the total number of hits identified can be maximized (Mehrjou et al., ). The experiment design procedure involves selecting $b$ genes to perturb at each round $t$. Let $B_{t}$ refer to the set of genes selected at round $t$. Following this selection, the phenotypic response of perturbing each of these genes $f(g)$ for each $g\in B_{t}$ is revealed. In the next round $(t+1)$, the agent has access to all phenotypic responses for genes tested in previous rounds: $1,2,\dots,t$.

At the end of $T$ rounds, the total hit ratio is computed as the fraction of true hits that were discovered through the series of experiments (similar to recall). Let $\mathcal{G}_{a}=\cup_{t=1}^{T}B_{t}^{+}$, where $B_{t}^{+}=\{g\in B_{t}:f(g)>\tau\}$ represents the hits identified in round $t$. The hit ratio can then be formally expressed as $\text{hit ratio}=\frac{|\mathcal{G}_{a}|}{|\mathcal{G}_{p}|}$, where $\mathcal{G}_{p}$ is the set of all true hits for the phenotype, $\mathcal{G}_{p}=\{g\in\mathcal{G}:f(g)>\tau\}$.

Two-gene perturbation: We also consider a new problem setting of predicting gene pairs (2-gene combinations) to perturb. Here, each query consists of two genes perturbed in a single cell simultaneously. This formulation is both more computationally challenging given the larger search space as well as biologically impactful. Let $\mathcal{G}^{2}=\mathcal{G}\times\mathcal{G}$ denote the set of all possible gene pairs that can be perturbed. Each element in $\mathcal{G}^{2}$ is a pair of genes $(g_{a},g_{b})$ where $g_{a},g_{b}\in\mathcal{G}$. The perturbation experiment in this context is represented by a function $f_{c}:\mathcal{G}^{2}\rightarrow\mathbb{R}$. This function $f_{c}$ maps each gene pair to a real-valued phenotypic response, which measures the combined effect of perturbing both genes simultaneously. The goal for the agent in this setting is to identify pairs of genes whose combined perturbation results in the desired phenotypic results, i.e. $f_{c}(g_{a},g_{b})>\tau$. Thus, hits also correspond to gene pairs instead of single genes.

## 3 BioDiscoveryAgent

We now present BioDiscoveryAgent which uses an LLM to automate the scientific discovery process for this biological setting. This includes accessing scientific knowledge, generating hypotheses, planning experiments and interpreting results (Algorithm 1). In the single gene setting, at each step $t$, the agent’s objective is to select a batch of $B$ genes for testing in the next step. The agent receives a prompt that describes general information about the experimental setup and the biological hypothesis being tested (Figure 1b, Appendix A, B). The results from each experiment are incorporated into the next prompt, along with the same information about the experimental setup. This prompt creation draws from established methods in developing other LLM-based agents, such as pre-action reasoning (Yao et al., ), reflective thinking (Shinn et al., ), and stepwise planning (AutoGPT, ).

We tested 9 different LLMs across varying levels of complexity for use in BioDiscoveryAgent (Claude v1 (Anthropic, ), Claude 3 Haiku, Claude 3 Sonnet, Claude 3 Opus (Anthropic, ), Claude 3.5 Sonnet (Anthropic, ), GPT-3.5-Turbo (OpenAI, ), GPT-4o (OpenAI, ), o1-mini (OpenAI, ), o1-preview (OpenAI, )). Due to the lack of transparent parameter counts from most organizations, we used the price per token as a proxy for model size. Unless otherwise noted, results are presented for the best performing high and low cost models which are Claude 3.5 Sonnet (Price per 1M output tokens: $15) and Claude 3 Haiku (Price per 1M output tokens: $1.25) respectively.

Processing gene list and experimental observations: When choosing genes for perturbation, it is not always feasible to include all possible genes to choose from. At the same time, we want to avoid arbitrarily narrowing down the gene list based on our existing knowledge. Therefore, we adopt a two-step approach: Initially, we allow BioDiscoveryAgent to suggest genes without restrictions (Appendix Figure 4a). This enables it to draw on its comprehensive understanding of biology freely. If the agent is unable to produce the required gene list after many trials due to invalid or repeated sampling of genes (Appendix Figure 4b,c), BioDiscoveryAgent summarizes the list of all remaining genes and adds it to the prompt to aid in gene selection (Appendix Figure 4d). The summarized gene list is designed to cover a broad range of biological pathways and functions. A similar summarization technique is used for the experimental observations when they exceed the LLM’s context.

Agent Response Format: To ensure interpretability and to guide the agent’s thought process, a consistent response format is defined across all prompts. We direct the LLM to structure its responses into several parts: Reflection, Research Plan, Solution (Appendix A, Figure 1b), similar to (Huang et al., ). Research Plan helps in effective planning and monitoring progress. Through the Reflection and Research Plan entries, the model is able to provide additional reasoning behind a particular prediction. This also helps to rule out predictions that may be hallucinations or not well-motivated. Solution contains a formatted list of genes to perturb next.

### 3.1 Agent tools

The primary mechanism by which BioDiscoveryAgent interacts with the user is through natural language (Appendix Figure 5a). The agent also has access to additional resources to aid in this decision-making beyond what is stored within its weights. These include access to the biomedical literature, a second agent to critique its predictions, and the ability to analyze tabular datasets:

Literature search: As part of the input, the user can choose to have the agent query the literature at each experimental cycle to inform its predictions (Appendix Figure 5b,c). In this case, the agent uses the PubMed API (Wobben, ) to search for papers containing the most pertinent literature for the experiments that the agent was asked to design. The search terms are chosen by the agent. Once the top 5 papers have been identified, the agent summarizes the information within these papers including parsed title, abstract, methods, results, and conclusion sections. It then attaches the summarization to the prompt and use it to identify additional genes to perturb for the given experimental round. The citations to these papers are retained and returned along with the model predictions.

Gene search based on biological databases: LLMs are trained on text-based data and do not have access to many biological databases that are stored in the form of tabular data. We provide the agent with the ability to query databases to search for other genes with similar biological properties as hit genes from previous experimental rounds (Appendix Figure 5d). First, the API is called to perform enrichment analysis for biological processes on the Reactome 2022 database (Gillespie et al., ) to identify the most relevant biological pathways. For each identified pathway, the agent queries other genes that participate in the same pathway. The top K genes that appear most frequently in these pathways are then selected and concatenated to the prompt for the main agent. Gene search based on other criteria (co-essentiality profiles, expression patterns, KEGG pathways) were also tested (Appendix Table 5). Among these, Reactome enrichment analysis gave the best result.

AI critic: An LLM prompted to behave as a research assistant may respond very differently from one that is prompted to mainly critique a prediction. To benefit from these contrastive prompting strategies, we make use of an AI critic similar to the idea by (Weng et al., ). The goal is to identify mistakes and enhance the quality of the final prediction made by the agent (Appendix Figure 5e). At every round, once BioDiscoveryAgent selects a batch of genes to be tested, a critic agent (which is also an LLM) is prompted to critique the choice of the main agent i.e. it can change some or all the genes in the batch and come up with a new set of genes.

## 4 Experiments

We assess model performance using data from past genetic perturbation experiments. We simulate the perturbation of a gene $g$ by retrieving the relevant observation of the perturbation-induced phenotype $f(g)$ from this dataset. In every experimental round we perturb $128$ genes, representing a reasonably sized small-scale biological screen. Since each round of experimentation can incur additional costs and introduce unwanted experimental variation, we focus our evaluations on fewer experimental rounds (5) to more accurately reflect a real biological setting. For each dataset, after each round, we calculate the hit ratio as the proportion of discovered hits out of the total true hits for that dataset.

### 4.1 Datasets and Baselines

For the single-gene perturbation setting, we make use of five different datasets spread across different cell types, publication dates and data generation sites. Each of the datasets contains the phenotypic response of knocking-down over 18,000 individual genes in distinct cells, with the exception of (Scharenberg et al., ) which contains data for 1061 perturbations. All datasets were released after 2021, apart from one dataset (CAR-T (^1^11generated by the authors of this paper)) which is so far unpublished.

Each of the datasets measure a distinct biological process. The (Schmidt et al., ) dataset measures the changes in the production of two key cytokines involved in immune signaling: Interferon-$\gamma$ (IFNG) and Interleukin-2 (IL-2) under different genetic perturbations performed in primary human T-cells. The (Carnevale et al., ) dataset includes perturbation screens for identifying genes that render T cells resistant to inhibitory signals encountered in the tumor microenvironment. Unpublished data (CAR-T dataset) studies the impact of genome-wide perturbations on CAR-T cell proliferation. The (Scharenberg et al., ) dataset measures the effect of perturbation on mediating lysosomal choline recycling in pancreatic cells, and the (Sanchez et al., ) dataset studies the change in expression of endogenous tau protein levels in neurons.

For the two-gene perturbation task, we use a dataset from a screen that knocked down 100,576 gene pairs in K562 cells (Horlbeck et al., ). For each gene pair, we are interested in the synergistic effects on cell fitness upon combinatorial knockdown. Synergy is determined by the deviation between the observed cell fitness and the expected fitness, which is calculated by summing the average impact of knocking down each gene in the combination individually (Horlbeck et al., ).

For baseline models, we use the GeneDisco benchmark (Mehrjou et al., ) and the DiscoBax method (Lyle et al., ). Every baseline includes a multi-layer perceptron $M$ for predicting experimental outcomes using gene features. This is then combined with one of seven different acquisition functions for designing each round of experiments using Bayesian optimization (Appendix E).

## 5 Results

Figure: Figure 2: Per-round performance comparison to machine learning baselines for 1-gene perturbation experiments: Each line averages the hit ratio from 10 runs, with error bars indicating 1 standard deviation. 128 genes were predicted per round over 5 rounds. For (Scharenberg et al., ), a batch size of 32 was used due to its smaller size of 1061 perturbations.
Refer to caption: x2.png

**Table 1: Performance comparison to machine learning baselines for 1-gene perturbation experiments. Hit ratio for experimental round 5 averaged over 10 runs, with 128 genes predicted in each round. ’All’ refers to results across all genes, ’N/E’ only considers non-essential genes. The best performing machine learning model and the best performing LLM for BioDiscoveryAgent across all datasets are both highlighted in gray. ^∗For (Scharenberg et al., ), a batch size of 32 was used due to its smaller size of 1061 perturbations. Schmidt1 refers to the screen measuring Interferon-$\gamma$ (IFNG) and Schmidt2 measures Interleukin-2 (IL-2) following perturbation (Schmidt et al., ). ^†CAR-T refers to an unpublished dataset. LLMs are sorted by price per 1M output tokens (Table 4). See Appendix Table 6 for error intervals.**
| Model | Schmidt1 | Schmidt2 | CAR-T^† | Scharen.^∗ | Carnev. | Sanchez |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | All | N/E | All | N/E | All | N/E | All | N/E | All | N/E | All | N/E |
| Random | 0.037 | 0.035 | 0.031 | 0.031 | 0.033 | 0.020 | 0.160 | 0.136 | 0.036 | 0.039 | 0.034 | 0.034 |
| Baseline Models |  |  |  |  |  |  |  |  |  |  |  |  |
| Soft Uncertain | 0.037 | 0.037 | 0.037 | 0.035 | 0.034 | 0.032 | 0.205 | 0.161 | 0.031 | 0.035 | 0.029 | 0.029 |
| Top Uncertain | 0.057 | 0.042 | 0.072 | 0.050 | 0.058 | 0.044 | 0.294 | 0.236 | 0.037 | 0.033 | 0.039 | 0.028 |
| Margin Sample | 0.054 | 0.040 | 0.061 | 0.047 | 0.054 | 0.045 | 0.285 | 0.227 | 0.036 | 0.032 | 0.033 | 0.028 |
| Coreset | 0.072 | 0.066 | 0.102 | 0.084 | 0.069 | 0.059 | 0.243 | 0.197 | 0.047 | 0.038 | 0.061 | 0.054 |
| Badge | 0.060 | 0.050 | 0.077 | 0.058 | 0.042 | 0.038 | 0.258 | 0.211 | 0.044 | 0.036 | 0.039 | 0.035 |
| K-Means (E) | 0.045 | 0.035 | 0.064 | 0.053 | 0.028 | 0.024 | 0.170 | 0.154 | 0.036 | 0.029 | 0.037 | 0.032 |
| K-Means (D) | 0.048 | 0.035 | 0.074 | 0.060 | 0.025 | 0.021 | 0.281 | 0.240 | 0.039 | 0.030 | 0.043 | 0.037 |
| DiscoBax | 0.088 | 0.069 | 0.074 | 0.057 | 0.047 | 0.021 | 0.201 | 0.200 | 0.045 | 0.038 | 0.058 | 0.049 |
| BioDiscoveryAgent (No-Tools) |  |  |  |  |  |  |  |  |  |  |  |  |
| Claude 3 Haiku | 0.064 | 0.072 | 0.072 | 0.083 | 0.116 | 0.119 | 0.209 | 0.200 | 0.032 | 0.034 | 0.056 | 0.056 |
| GPT-3.5-Turbo | 0.044 | 0.048 | 0.061 | 0.073 | 0.064 | 0.066 | 0.230 | 0.188 | 0.032 | 0.034 | 0.039 | 0.038 |
| Claude v1 | 0.067 | 0.086 | 0.089 | 0.110 | 0.110 | 0.126 | 0.292 | 0.222 | 0.038 | 0.045 | 0.053 | 0.055 |
| o1-mini | 0.081 | 0.092 | 0.096 | 0.120 | 0.127 | 0.139 | 0.279 | 0.286 | 0.040 | 0.041 | 0.074 | 0.073 |
| Claude 3 Sonnet | 0.076 | 0.082 | 0.088 | 0.111 | 0.115 | 0.118 | 0.302 | 0.265 | 0.041 | 0.042 | 0.064 | 0.063 |
| Claude 3.5 Sonnet | 0.095 | 0.107 | 0.104 | 0.122 | 0.130 | 0.133 | 0.326 | 0.292 | 0.042 | 0.044 | 0.066 | 0.063 |
| GPT-4o | 0.059 | 0.068 | 0.074 | 0.090 | 0.116 | 0.120 | 0.311 | 0.286 | 0.037 | 0.040 | 0.030 | 0.043 |
| o1-preview | 0.081 | 0.091 | 0.091 | 0.114 | 0.141 | 0.145 | 0.283 | 0.259 | 0.041 | 0.043 | 0.068 | 0.063 |
| Claude 3 Opus | 0.094 | 0.106 | 0.104 | 0.132 | 0.126 | 0.129 | 0.247 | 0.206 | 0.043 | 0.043 | 0.059 | 0.058 |

BioDiscoveryAgent outperforms baselines based on hit ratio for 1-gene perturbation experiments: We evaluate the performance of BioDiscoveryAgent without any tools (No-Tools) against a range of machine learning benchmarks, including random sampling. When measuring hit ratios across all genes at experimental round 5, BioDiscoveryAgent consistently demonstrates superior performance compared to the best baseline approach, across 7 out of 9 LLMs tested (Table 1, Appendix Table 6). The best performing LLM, Claude 3.5 Sonnet, outperforms the best baseline for each dataset by 21% on average. Performance improvement is observed across 5 of the 6 datasets. This gap is especially large at earlier rounds, where the LLM can leverage its biological knowledge to select genes, in contrast to baseline methods that suffer from the cold start problem (Figure 2).

To verify that the model isn’t relying on uninformative hits, we measured model performance in predicting hits that are non-essential genes. Essential genes are likely to be detected as hits under any perturbation screen given the strong phenotypic effect of perturbing these genes. On the other hand, the response of non-essential genes to perturbation, is harder to predict and often more useful biologically. When filtering for non-essential genes, BioDiscoveryAgent (Claude 3.5 Sonnet) shows an improvement across all 6 datasets with an even higher average performance improvement of 46% over baselines (Table 1, Appendix Table 6).

Additionally, we tested the model’s tendency to predict distinct genes when prompted differently and upon observing different experimental results. We observe a low Jaccard similarity between all predicted genes after five rounds of experiments for any pair of datasets (Appendix Figure 6), suggesting that gene selection is not invariant to the task prompt and experimental observations.

BioDiscoveryAgent can guide 2-gene combinatorial perturbation experiments: In addition to 1-gene perturbation experiments, we also demonstrate that BioDiscoveryAgent can guide 2-gene combinatorial perturbation experiments, which is significantly more difficult due to the much larger combinatorial search space (100,576 gene pairs considered in (Horlbeck et al., )). As shown in Table 2, BioDiscoveryAgent (Claude 3.5 Sonnet) significantly outperforms the random sampling baseline by 170% on average.

**Table 2: Performance evaluation for 2-gene perturbation experiments. 32 gene pairs out of 100,576 pairs predicted per round averaged over 10 runs (Horlbeck et al., ). Evaluation using cumulative number of hits across 5 experimental rounds. Error intervals correspond to 1SD.**
| Model | Round 1 | Round 2 | Round 3 | Round 4 | Round 5 |
| --- | --- | --- | --- | --- | --- |
| Random | 2.6 $\pm$ 1.43 | 5.7 $\pm$ 2.83 | 8.9 $\pm$ 3.67 | 12.8 $\pm$ 3.74 | 16.4 $\pm$ 3.8 |
| BioDiscoveryAgent (No-Tools) |  |  |  |  |  |
| Claude 3 Haiku | 5.1 $\pm$ 2.2 | 14.8 $\pm$ 2.9 | 23.6 $\pm$ 4.1 | 34.1 $\pm$ 5.8 | 45.1 $\pm$ 5.3 |
| Claude 3.5 Sonnet | 8.4 $\pm$ 1.36 | 15.7 $\pm$ 1.1 | 23.1 $\pm$ 2.59 | 31.7 $\pm$ 2.9 | 40.0 $\pm$ 3.77 |

**Table 3: Agent performance improvements when using different tools for 1-gene perturbation experiments. Results show hit ratio for experimental round 5 averaged over 10 runs. See Table 1 caption for notes on specific datasets. See Appendix Table 7 for error intervals.**
| Tools Used | Schmidt1 | Schmidt2 | CAR-T | Scharen. | Carnev. | Sanchez |
| --- | --- | --- | --- | --- | --- | --- |
| Random | 0.037 | 0.031 | 0.033 | 0.160 | 0.036 | 0.034 |
| BioDiscoveryAgent (Claude 3.5 Sonnet) |  |  |  |  |  |  |
| No-Tools | 0.095 | 0.104 | 0.130 | 0.326 | 0.042 | 0.066 |
| Literature | 0.096 (+1%) | 0.098 (-6%) | 0.138 (+6%) | 0.309 (-5%) | 0.042 (+0%) | 0.069 (+5%) |
| AI Critic | 0.088 (-7%) | 0.092 (-12%) | 0.126 (-3%) | 0.309 (-5%) | 0.042 (+0%) | 0.059 (-11%) |
| Gene Search | 0.096 (+1%) | 0.100 (-4%) | 0.123 (-5%) | 0.348 (+7%) | 0.043 (+2%) | 0.062 (-6%) |
| All-Tools | 0.096 (+1%) | 0.090 (-13%) | 0.121 (-6%) | 0.234 (-28%) | 0.043 (+2%) | 0.054 (-18%) |
| BioDiscoveryAgent (Claude 3 Haiku) |  |  |  |  |  |  |
| No-Tools | 0.064 | 0.072 | 0.116 | 0.209 | 0.032 | 0.056 |
| Literature | 0.053 (-17%) | 0.069 (-4%) | 0.091 (-22%) | 0.164 (-22%) | 0.035 (+9%) | 0.057 (+2%) |
| AI Critic | 0.061 (-5%) | 0.070 (-3%) | 0.113 (-3%) | 0.219 (+5%) | 0.043 (+34%) | 0.054 (-4%) |
| Gene Search | 0.080 (+25%) | 0.098 (+36%) | 0.114 (-2%) | 0.249 (+14%) | 0.046 (+44%) | 0.065 (+16%) |
| All-Tools | 0.084 (+31%) | 0.099 (+38%) | 0.128 (+10%) | 0.259 (+24%) | 0.043 (+34%) | 0.058 (+4%) |

**Table 4: Effect of tools on different LLMs. Results show increase in hit ratio when using all-tools as compared to results for the same agent when using no-tools. Models are sorted by increasing price per 1M output tokens. ^∗For o1-mini and o1-preview, output tokens include internal reasoning tokens generated by the models that are not visible in API responses (OpenAI, ). Results are shown for experimental round 5 averaged over 10 runs. See Appendix Table 8 for the actual hit rates.**
| Tools Used | Schmidt1 | Schmidt2 | CAR-T | Scharen. | Carnev. | Sanchez | Price per 1M Token ($) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Claude 3 Haiku | +31% | +38% | +10% | +24% | +34% | +4% | 1.25 |
| GPT-3.5-Turbo | +41% | +64% | -2% | -5% | +16% | +15% | 3.00 |
| Claude v1 | +42% | +37% | +4% | +14% | +42% | +9% | 11.00 |
| o1-mini | +2% | -22% | -10% | -5% | 0% | -3% | 12.00^∗ |
| Claude 3 Sonnet | -3% | +3% | -9% | 0% | +15% | +14% | 15.00 |
| Claude 3.5 Sonnet | +1% | -13% | -6% | -28% | +2% | -18% | 15.00 |
| GPT-4o | -17% | -14% | -11% | -8% | -5% | -13% | 15.00 |
| o1-preview | -36% | -13% | -30% | +2% | -41% | -54% | 60.00^∗ |
| Claude 3 Opus | +2% | -3% | -8% | +18% | -5% | -7% | 75.00 |

The performance of BioDiscoveryAgent can be augmented with tool-use, but the improvement varies by choice of LLM: We evaluated the impact of integrating three different tools—literature search, gene search, and AI critic into BioDiscoveryAgent (Table 3, Appendix Table 7). For the agent using Claude 3.5 Sonnet, tools applied individually or in combination did not lead to significant changes in performance and in some cases hurt performance. In contrast, for Claude 3 Haiku, incorporating all tools consistently enhanced performance.

The effects of different tools varied. In the case of Claude 3 Haiku, using only the literature search tool often resulted in fixation on a few simple keywords and irrelevant papers, leading to less effective searches. However, the literature search still provided verifiable citations for the gene prediction process, which is beneficial for scientists. Performance significantly improved with the use of a gene search based on shared biological pathways. This improvement likely stems from the gene search relying on tabular datasets not fully represented in text-based sources used for language model training, which the No-Tools agent cannot access. Using only the AI critic tool slightly improved performance over the No-Tools model by diversifying predictions or focusing on specific gene sets.

We further evaluated the impact of incorporating tools across a broader range of LLMs (Table 4, Appendix Table 8). We observe that while some models such as Claude v1, Claude 3 Haiku and GPT3.5-Turbo showed a significant improvement in performance with the incorporation of all tools ($24.7\%$, $23.5\%$ and $21.5\%$ respectively), others such as Claude 3.5 Sonnet, Claude 3.5 Opus and GPT4o showed a decrease in performance or no significant change (-$10\%$, -$1\%$ and -$11\%$ respectively). This contrast suggests a potential relationship between model size and the benefit of tool use—smaller models appear to benefit more from tool-assisted retrieval, whereas larger models may derive this information directly from their trained weights.

We further investigated this relationship by measuring the percentage of new genes predicted by Claude 3 Haiku only when using tools, that are also predicted by Claude 3.5 Sonnet with no-tools (Appendix Figure 7). For some datasets, we see a large proportion of such tool-derived genes being predicted by Claude 3.5 Sonnet without any tool use (14-28%). This suggests that larger models are capable of intrinsically retrieving additional biological information that is otherwise provided by the tools. However, this wasn’t the case in all datasets indicating that some orthogonal information from the tools reamins uncaptured by training on scientific text alone. Therefore, a more strategic design and selection of tools could lead to further improvements in agent performance for larger models.

BioDiscoveryAgent accounts for prior knowledge and observations in decision-making : Next, we investigate the use of prior knowledge versus observations from previous experiments in the agent’s decision-making. We examine three scenarios using BioDiscoveryAgent (Claude 3.5 Sonnet): 1) Prompt + Observation, where the agent utilizes both previous experiment results and detailed information about the experiment’s goal; 2) Prompt Only, where the agent ignores all experiment results; 3) Observation Only, where the agent is unaware of the current experiment’s goal and only conditioned on observations. To accurately capture these trends at fine resolution, we use a larger number of experimental rounds with smaller number of perturbations in each round: 30 rounds of experimentation with 32 genes in each round on the IFNG dataset from (Schmidt et al., ).

Results show that Prompt + Observation outperforms the other two scenarios, highlighting the significance of integrating prior knowledge and observations (Figure 3a). Interestingly, Prompt + Observation and Prompt Only benefit from prior knowledge early on, unlike Observation Only, which lacks the experiment’s goal, underscoring the vital role of prior knowledge in the initial experiment phases. However, as experiments progress, Observation Only surpasses Prompt Only, showcasing the agent’s capacity to adapt based on observations. For further validation, we also perform the same experiment using a different LLM and observe a similar trend (Appendix Figure 8, Table 9)

Additionally, we find that access to observations results in more similar gene predictions across different trials compared to experiments without access to observations (Figure 3b). This consistency was quantified using the Jaccard similarity index between all predicted genes after 30 rounds of experimentation. This suggests that observations significantly influence BioDiscoveryAgent’s decision-making, leading to more uniform choices across separate trials.

BioDiscoveryAgent provides interpretable predictions with references to the literature. BioDiscoveryAgent provides interpretable predictions at various stages. In one such example (Appendix F, Figure 5), the agent is tasked with identifying genes regulating the production of Interferon-gamma (IFNG). The agent explicitly reasons that it will focus on genes involved in mitochondrial respiration and the electron transport chain (Appendix Figure 5c). Utilizing the literature search tool, it accesses relevant literature to support its predictions, citing specific papers and line numbers (Appendix Figure 5b). For example, the agent highlights STUB1 as a gene for potential perturbation, supported by references to specific lines in a relevant scientific paper. Alternatively, the agent is also able to indicate when the paper being summarized does not contain actionable information: "In this segment, I cannot find specific gene targets or pathways to focus on for the initial experiments" (Appendix F).

The LLM critic tool provides valuable insights into the predicted set of genes, identifying potential issues such as randomness or too narrow of a focus. In this example, the critic LLM suggests diversifying the selected genes by also considering pathways involved in interferon regulation, such as NF-kB signaling and MAPK signaling (Appendix Figure 5e). Thus, not only does the critic LLM provide clear reasons for selecting specific genes but also further opens avenues for human-in-the-loop feedback by a subject-matter expert. In another example, the critic LLM guides the agent to limit the randomness in gene selection: "The selection of genes seems somewhat random and not focused enough on likely candidates based on known lysosomal and endolysosomal genes…" (Appendix F).

Figure: Figure 3: Agent accounts for both prior knowledge and observations in decision-making Three scenarios are considered: BioDiscoveryAgent (Claude 3.5 Sonnet) has access to task description and experimental observations (Prompt + Observation); only has access to experimental observations (Observation Only); only has access to the task description (Prompt Only). (a) Hit ratio at each experimental round across 30 rounds with 32 genes predicted in each. Each line corresponds to the average over 8 runs with error bars representing 1SD. (b) Jaccard similarity index between all predicted genes at 30 rounds of experimentation. Each cell corresponds to a different model run.
Refer to caption: x3.png

## 6 Related work

Artificial intelligence has demonstrated significant potential across various scientific domains, from realistic simulations of human behavior (Park et al., ) to exploring mathematical function spaces (Romera-Paredes et al., ). Research has shown the utility of these models in mining and querying the scientific literature (Lála et al., ; Schick et al., ), as well as in general research tasks such as analyzing large datasets, reasoning about data, and generating reports (Shakked & Zhang, ; Lehr et al., ). Additionally, closed-loop AI-driven lab experimentation has made notable advances, particularly in chemical synthesis (Boiko et al., ; M. Bran et al., ) and materials discovery (Tom et al., ).

In the biological domain, research has shown that LLMs can capture significant information about biological pathways and processes (Park et al., ), and are valuable in simulating biological processes at various scales (Schaefer et al., ). There have been benchmarking efforts for answering questions in genomics (Hou & Ji, ), performing bioinformatic analyses (Sarwal et al., ) as well as assessing broader biological research capabilities (Laurent et al., ). Despite long-standing interest in developing autonomous AI systems for genetic perturbation experiment design (King et al., ), the use of LLM-based agents has yet to be explored for closed-loop biological experiment design. Some work has focused on designing individual gene editing experiments (Huang et al., ), primarily by optimizing experimental parameters and protocols. In contrast, our work, BioDiscoveryAgent, aims to generate and refine biological hypotheses through a series of perturbation experiments, leveraging the agent’s biological understanding and reasoning capabilities. Although agents have not been used in this setting, past research has investigated machine learning and Bayesian optimization techniques for similar purposes (Mehrjou et al., ; Lyle et al., ).

## 7 Discussion

BioDiscoveryAgent represents a new paradigm in the design of biological experiments, aimed at augmenting scientists’ capabilities. Traditionally, this process employs a multi-stage pipeline using an acquisition function combined with a machine learning model. This model, often challenging to interpret, requires manual design and periodic retraining with handpicked gene features. In contrast, we demonstrate that an agent powered by an LLM can simplify the entire experimental design process into a single prompt from the researcher. Through its enhanced interpretability at every stage, the researcher can meaningfully engage with the model and augment its predictions.

Moreover, the agent is equipped with prior biological knowledge, solving the cold start problem at the beginning of a new round of experiments while efficiently utilizing observational data as the rounds progress. By using various tools, it can explicitly search for and integrate information from the scientific literature and existing tabular datasets. This capability results in a research assistant capable of speeding up biological research through utilizing information from diverse sources.

While BioDiscoveryAgent shows strong capabilities in enhancing experiment design, it still has room for improvement. It tends to perform better with certain cell types over others, likely due to variations in scientific literature coverage. Most performance benefits are observed in the early stages of experimentation rather than in prolonged rounds. Additionally, there is a need to develop better tools or fine-tuning methods to improve information extraction from non-text-based sources.

Overall, BioDiscoveryAgent offers a complementary approach to existing experimental design methods, significantly improving model performance in the low data regime while enhancing overall model interpretability. Amid increasing interest in active experimental design for biological research, BioDiscoveryAgent provides compelling evidence that language models could become essential components of such systems. By facilitating improved reasoning and interpretability and leveraging a broad understanding of the scientific literature, AI tools like BioDiscoveryAgent are poised to become integral to experimental design strategies in the near future.

## Ethics Statement

This study did not involve any animal or human subjects. All experiments were conducted using cell cultures under controlled laboratory conditions.

## Reproducibility Statement

Our results are fully reproducible with the code provided. The required prompts are included in the repository, and all datasets are available in the supplementary materials. The only exception is the CAR-T dataset since it is so far unpublished. Authors are willing to provide an anonymized version of that dataset upon request.

## Acknowledgements

We thank Yanay Rosen, Hamed Nilforoshan, Charilaos Kanatsoulis, Rok Sosic, Hanchen Wang, Ayush Agrawal, Charlotte Bunne, Michael Moor, Michael Bereket, Yash Dalmia, Jens Magnusson, Julia Carnevale, Mineto Ota, Ralf Schmidt, Carl Ward and attendees of the ICLR Workshop
on Machine Learning for Genomics Explorations, 2024 for discussions and for providing feedback on our manuscript. J.L. gratefully acknowledge the support of DARPA under Nos. N660011924033 (MCS); NSF under Nos. OAC-1835598 (CINES), CCF-1918940 (Expeditions), DMS-2327709 (IHBEM); Stanford Data Applications Initiative, Wu Tsai Neurosciences Institute, Stanford Institute for Human-Centered AI, Chan Zuckerberg Initiative, Amazon, Genentech, GSK, Hitachi, SAP, and UCB. The content is solely the responsibility of the authors and does not necessarily represent the official views of the funding entities.

## Conflicts of Interest

A.M. is a cofounder of Site Tx, Arsenal Biosciences, Spotlight Therapeutics and Survey Genomics, serves on the boards of directors at Site Tx, Spotlight Therapeutics and Survey Genomics, is a member of the scientific advisory boards of Site Tx, Arsenal Biosciences, Spotlight Therapeutics, Survey Genomics, NewLimit, Amgen, and Tenaya, owns stock in Arsenal Biosciences, Site Tx, Spotlight Therapeutics, NewLimit, Survey Genomics, Tenaya and Lightcast and has received fees from Site Tx, Arsenal Biosciences, Spotlight Therapeutics, NewLimit, 23andMe, PACT Pharma, Juno Therapeutics, Tenaya, Lightcast, Trizell, Vertex, Merck, Amgen, Genentech, GLG, ClearView Healthcare, AlphaSights, Rupert Case Management, Bernstein and ALDA. A.M. is an investor in and informal advisor to Offline Ventures and a client of EPIQ. The Marson laboratory has received research support from the Parker Institute for Cancer Immunotherapy, the Emerson Collective, Juno Therapeutics, Epinomics, Sanofi, GlaxoSmithKline, Gilead and Anthem and reagents from Genscript and Illumina.

## Appendix A Prompt

The prompt includes the task information and response format as this example shown below.

## Appendix B Dataset Specific Prompts

For each dataset, the research problem and the type of measurement outcome are used to create the prompt.

## Appendix C Algorithm for BioDiscoveryAgent

Figure: Algorithm 1 BioDiscoveryAgent: AI Agent for Biological Experiment Design (using all tools)

Figure: Figure 4: Gene selection strategy: (a) The space of genes that can be tested in a given experiment is constrained by expeirmental limitations. BioDiscoveryAgent can take a few tries to select genes within this limited space. (b) A common error is repeating previously tested genes. (c) Often this will result in the agent getting stalled and unable to make successful selections, especially in the case of large batch sizes. (d) After several failed attempts, we summarize the space of genes that can be tested while prioritizing gene diversity.
Refer to caption: x4.png

Figure: Figure 5: BioDiscoveryAgent workflow with all tools over a single experimental round. Prompts and agent responses have been summarized. See Appendix F for full trace. (a) The input to the agent is the description of the problem. (b) In case of the literature search tool, the LLM first determines appropriate search terms for finding relevant scientific papers. Top ranked retrieved papers are summarized by the LLM (along with line numbers that reference directly to text in the paper) and provided as additional context to the agent when predicting the set of genes to perturb.(c) The agent predicts the set of genes to perturb in the next experiment. Response is structured containing Reflection, Research Plan and Solution. (d) In case of using the gene search tool, the LLM identifies a query gene with which to search for other genes (e) The LLM can also be prompted differently to function as a critic and analyze the predictions so far from a different perspective (f) Final output gene list after using different tools. We used Claude v1 for this analysis since it showed high absolute score as well as high performance gain through tool use for this dataset (Schmidt et al., ).
Refer to caption: x5.png

## Appendix D Tool Descriptions for BioDiscoveryAgent

In this section, we provide some more details about the tools provided to BioDiscoveryAgent to aid it in making its predictions, along with the reasoning for adding those tools.

- 1.
AI critic: LLMs are very sensitive to the prompt they are presented with. Past research has shown varying performance depending on the setting in which the LLM is queried. Thus, an LLM prompted to behave as an agent for a human researcher may behave very differently from one that is prompted to mainly critique the prediction made by another LLM. To benefit from these contrastive prompting strategies, we make use of an AI critic similar to the idea by (Weng et al., ) to identify mistakes and enhance the quality of the final prediction made by the agent.
At every round, once BioDiscoveryAgent comes up with a batch of genes to be tested, a critic agent (which is also an LLM) is prompted to critique the choice of the main agent and it can change some or all the genes in the batch and come up with a new set of genes (Figure 5e). Having such an agent improved performance giving the system additional opportunities to reflect on its reasoning. The system prompt for the critic agent was as follows:
[⬇](data:text/plain;base64,QXMgYW4gYWR2aXNvciwgcGxlYXNlIGNyaXRpcXVlIHRoaXMgcGxhbiBhbmQgc3VnZ2VzdCBzb21lIGNoYW5nZXMgdG8gaXQuIFVzZSB0aGlzIGZvcm1hdDoKMS4gQ3JpdGlxdWU6IGluY2x1ZGUgYWxsIHJlbGV2YW50IGRldGFpbHMgb2YgdGhlIGNyaXRpcXVlLgoyLiBVcGRhdGVkIFNvbHV0aW9uOiBHaXZlIGFuIHVwZGF0ZWQgc2VsZWN0aW9uIG9mIHthcmdzLm51bV9nZW5lc30gZ2VuZXMgYmFzZWQgb24gdGhlIGNyaXRpcXVlIHNlcGFyYXRlZCBieSBjb21tYXMgaW4gdGhpcyBmb3JtYXQ6OiAxLiA8R2VuZSBuYW1lIDE+LCAyLiA8R2VuZSBuYW1lIDI+IC4uLiBcbgoKVHJ5IHRvIGZpcnN0IGZvY3VzIGEgbG90IG9uIHRyeWluZyB2ZXJ5IGRpdmVyc2UgZ2VuZXMgdG8gZ2V0IGEgc2Vuc2Ugb2Ygd2hpY2ggdHlwZXMgb2YgZ2VuZXMgYWZmZWN0IHRoZSByZXNlYXJjaCBwcm9ibGVtIHRoZSBtb3N0LiBGcm9tIHRoZSBvYnNlcnZhdGlvbnMsIHVwZGF0ZSB5b3VyIGJlbGllZnMgcXVpY2tseSBhbmQgc21hcnRseSBhbmQgdGhlbiBkb3VibGUgZG93biBvbiBnZW5lcyB0aGF0IHlvdSB0aGluayBzaGFsbCBiZSBoaXRzIGluIG9yZGVyIHRvIGluY3JlYXNlIHRoZSBjdW11bGF0aXZlIGhpdCByYXRpby4KClBsZWFzZSBkbyBub3QgY3JpdGlxdWUvbWFrZSBhIGxvdCBvZiBjaGFuZ2VzIGlmIHRoZXJlIGlzIG5vIG5lZWQgdG8gbWFrZSBhIGNoYW5nZS4=)
As an advisor, please critique this plan and suggest some changes to it. Use this format:
1. Critique: include all relevant details of the critique.
2. Updated Solution: Give an updated selection of {args.num_genes} genes based on the critique separated by commas in this format:: 1. <Gene name 1>, 2. <Gene name 2> ... \n
Try to first focus a lot on trying very diverse genes to get a sense of which types of genes affect the research problem the most. From the observations, update your beliefs quickly and smartly and then double down on genes that you think shall be hits in order to increase the cumulative hit ratio.
Please do not critique/make a lot of changes if there is no need to make a change.
In addition to the above prompt, the critic agent was also provided with a list of all genes that were tested in the previous rounds along with genes that were identified as hits.
- 2.
Literature Search: Scientific literature captures prior knowledge that can be leveraged to design experiments. A scientist typically reads literature relevant to a problem, builds a hypothesis, and cites relevant past work. We aimed to provide BioDiscoveryAgent with similar capabilities that would allow it to search for relevant papers on the web, learn from them, and incorporate its learnings in designing the experiments.
As part of the input, the user can choose to have the agent query the literature at each experimental cycle to inform its predictions. In this case, the agent uses the PubMed API (Wobben, ) to search for papers containing the most pertinent literature for the experiments that the agent was asked to design. The search terms are chosen by the agent. Once the top 5 papers have been identified, the agent summarizes the information within these papers including parsed title, abstract, methods, results, and conclusion sections. It then attaches the summarization to the prompt and use it to identify additional genes to perturb for the given experimental round (Figure 5b). The citations to these papers are retained and returned along with the model predictions. Over time, the agent accumulated these summaries, granting it access to all literature surveys conducted in previous rounds to propose a set of genes for each specific round.
The inclusion of a literature review tool enhanced interpretability and improved grounding as the agent frequently cited papers it had previously surveyed when predicting genes for the batch. Despite these benefits, the use of a literature review tool had its drawbacks. It tended to make the agent less exploratory, heavily biasing its reasoning towards the papers retrieved. The agent struggled to generate innovative queries for the literature survey API, and the lack of additional re-ranking on API outputs further limited the diversity of papers presented to the agent. This raised some important questions about the most effective use of scientific literature by an AI agent, a topic for future work.
- 3.
Gene Search: LLMs are trained on text-based data and do not have access to many biological databases that are stored in the form of tables. We provide the agent with the ability to search for top $10$ genes based on different criteria. The selected genes are concatenated to the prompt given to the main agent (Figure 5d). The results of augmenting the agent with these different gene search methods are summarized in Table 5.
•
Similar/Dissimilar Genes: The agent selects a gene for which it would like to search for similar or dissimilar genes. This is computed using the cosine similarity between the provided gene features (gene co-essentiality profiles (Tsherniak et al., )). This process first computes the inner product of gene features with the queried gene’s features and sorts the results based on the desired similarity or dissimilarity.
•
Correlated Genes: The agent accesses ARCHS4 (Lachmann et al., ), a public database that provides RNA-seq expression data from human and mouse samples. The API call ranks genes based on the Pearson correlation coefficient, a linear relationship between two genes’ expression levels across multiple samples. Then the call returns top 10 most correlated genes to a gene that the agent selects for query.
•
Genes in Common Tissues: The agent accesses ARCHS4 to retrive a list of tissue types where the gene is expressed. Then the API call looks for top 10 genes that also show strong expression in these tissues, based on the median TPM (transcripts-per-million) value.
•
KEGG Enrichment Analysis: The agent accesses the KEGG (Kanehisa et al., ) enrichment database, which provide curated information on biological pathways and functions associated with genes. For hit genes from previous rounds, the top 10 pathways that are over-represented by statistical significance are returned. For these pathways, the API call will generate the top 10 genes that are associated with them.
•
Reactome Enrichment Analysis: The agent perform similar enrichment analysis to the above, but uses the Reactome database (Gillespie et al., ) instead.

**Table 5: Effect of different gene search tools. Results show hit ratio for experimental round 5 averaged over 10 runs, with error intervals showing 1 standard deviation. ^∗For (Scharenberg et al., ), a batch size of 32 was used due to its smaller pool of 1061 relevant genes. Schmidt1 refers to the screen measuring Interferon-$\gamma$ (IFNG) and Schmidt2 measures Interleukin-2 (IL-2) following perturbation (Schmidt et al., ). ^†CAR-T refers to an unpublished dataset.**
| Tools Used | Schmidt1 | Schmidt2 | CAR-T | Scharen. | Carnev. | Sanchez |
| --- | --- | --- | --- | --- | --- | --- |
| Random | 0.037 | 0.031 | 0.033 | 0.160 | 0.036 | 0.034 |
| Claude 3.5 Sonnet |  |  |  |  |  |  |
| No-Tools | 0.095 | 0.104 | 0.130 | 0.326 | 0.042 | 0.066 |
| Similar | 0.091 (-4%) | 0.098 (-6%) | 0.145 (+12%) | 0.342 (+5%) | 0.043 (+2%) | 0.061 (-8%) |
| Dissimilar | 0.091 (-4%) | 0.096 (-8%) | 0.123 (-5%) | 0.302 (-7%) | 0.042 (+0%) | 0.069 (+5%) |
| Correlated | 0.088 (-7%) | 0.094 (-10%) | 0.138 (+6%) | 0.358 (+10%) | 0.044 (+4%) | 0.058 (-12%) |
| Tissues | 0.091 (-4%) | 0.097 (-7%) | 0.130 (+0%) | 0.310 (-5%) | 0.042 (+0%) | 0.066 (+0%) |
| KEGG | 0.089 (-6%) | 0.091 (-13%) | 0.122 (-6%) | 0.326 (+0%) | 0.045 (+6%) | 0.057 (-14%) |
| Reactome | 0.096 (+1%) | 0.100 (-4%) | 0.123 (-5%) | 0.348 (+7%) | 0.043 (+2%) | 0.062 (-6%) |
| Claude 3 Haiku |  |  |  |  |  |  |
| No-Tools | 0.064 | 0.072 | 0.116 | 0.209 | 0.032 | 0.056 |
| Similar | 0.065 (+2%) | 0.093 (+29%) | 0.095 (-18%) | 0.206 (-1%) | 0.043 (+34%) | 0.054 (-4%) |
| Dissimilar | 0.066 (+3%) | 0.073 (+1%) | 0.081 (-30%) | 0.215 (+3%) | 0.046 (+44%) | 0.054 (-18%) |
| Correlated | 0.068 (+6%) | 0.089 (+24%) | 0.108 (-7%) | 0.259 (+19%) | 0.043 (34%) | 0.047 (-16%) |
| Tissues | 0.063 (-2%) | 0.081 (+13%) | 0.119 (+3%) | 0.226 (+8%) | 0.037 (+16%) | 0.053 (-5%) |
| KEGG | 0.070 (+9%) | 0.074 (+3%) | 0.099 (-15%) | 0.232 (+11%) | 0.036 (+13%) | 0.047 (-16%) |
| Reactome | 0.080 (+25%) | 0.098 (+36%) | 0.114 (-2%) | 0.249 (+14%) | 0.046 (+44%) | 0.065 (+16%) |

## Appendix E Baselines

For baseline models, we use the methods implemented in the GeneDisco benchmark (Mehrjou et al., ) and (Lyle et al., ). Every baseline includes a multi-layer perceptron $M$ for predicting experimental outcomes using gene features. This is then combined with one of seven different acquisition functions for designing each round of experiments:

- •
Soft Uncertain: Prioritizes genes with higher uncertainty under $M$, using a softmax function with temperature.
- •
Top Uncertain: Selects genes with the highest uncertainty under model $M$.
- •
Margin Sample: Selects genes for which the model $M$ has the smallest margins between different classes.
- •
Coreset: Selects genes which are the most distant from previously selected genes based on their embedding representation in $M$.
- •
Badge: Uses a modified k-means algorithm on the gradient embeddings of the data points to select genes. The aim is to diversify the batch based on the model’s gradients.
- •
Kmeans: Selects genes that are closest to the cluster centers determined by K-means. Two baselines apply K-means either to an embedding of the data or the raw data directly.
- •
DiscoBax (Lyle et al., ): Selects genes with high expected change to the phenotype of interest as well as high diversity. Implemented as a set-value maximization problem.

.

Figure: Figure 6: Jaccard similarity index between all predicted hits at 5 rounds of experimentation Each cell corresponds to the union of predicted hits across 10 model runs for each dataset.
Refer to caption: extracted/5896911/figures/heatmap_new.png

**Table 6: Performance comparison to machine learning baselines for 1-gene perturbation experiments. Results show hit ratio for experimental round 5 averaged over 10 runs, with error intervals showing 1 standard deviation. ^∗For Scharenberg22, a batch size of 32 was used due to its smaller pool of 1061 relevant genes. ^†CAR-T refers to an unpublished dataset. BDAgent stands for BioDiscoveryAgent.**
| Model | Schmidt1 | Schmidt2 | CAR-T^† | Scharen.^∗ | Carnev. |
| --- | --- | --- | --- | --- | --- |
| Random | 0.037 $\pm$ 0.013 | 0.031 $\pm$ 0.002 | 0.033 $\pm$ 0.003 | 0.160 $\pm$ 0.028 | 0.036 $\pm$ 0.001 |
| Baseline Models |  |  |  |  |  |
| Soft Uncertain | 0.037 $\pm$ 0.006 | 0.037 $\pm$ 0.006 | 0.034 $\pm$ 0.007 | 0.205 $\pm$ 0.006 | 0.031 $\pm$ 0.006 |
| Top Uncertain | 0.057 $\pm$ 0.007 | 0.072 $\pm$ 0.014 | 0.058 $\pm$ 0.010 | 0.294 $\pm$ 0.030 | 0.037 $\pm$ 0.005 |
| Margin Sample | 0.054 $\pm$ 0.006 | 0.061 $\pm$ 0.009 | 0.054 $\pm$ 0.013 | 0.285 $\pm$ 0.019 | 0.036 $\pm$ 0.003 |
| Coreset | 0.072 $\pm$ 0.007 | 0.102 $\pm$ 0.005 | 0.069 $\pm$ 0.008 | 0.243 $\pm$ 0.031 | 0.047 $\pm$ 0.006 |
| Badge | 0.060 $\pm$ 0.008 | 0.077 $\pm$ 0.008 | 0.042 $\pm$ 0.017 | 0.258 $\pm$ 0.032 | 0.044 $\pm$ 0.006 |
| Kmeans Embed. | 0.045 $\pm$ 0.004 | 0.064 $\pm$ 0.007 | 0.028 $\pm$ 0.011 | 0.170 $\pm$ 0.032 | 0.036 $\pm$ 0.004 |
| Kmeans Data | 0.048 $\pm$ 0.005 | 0.074 $\pm$ 0.009 | 0.025 $\pm$ 0.012 | 0.281 $\pm$ 0.042 | 0.039 $\pm$ 0.004 |
| DiscoBAX | 0.088 $\pm$ 0.000 | 0.074 $\pm$ 0.049 | 0.047 $\pm$ 0.000 | 0.201 $\pm$ 0.018 | 0.045 $\pm$ 0.000 |
| BioDiscoveryAgent (No-Tools) |  |  |  |  |  |
| Claude 3 Haiku | 0.064 $\pm$ 0.005 | 0.071 $\pm$ 0.018 | 0.116 $\pm$ 0.014 | 0.209 $\pm$ 0.030 | 0.032 $\pm$ 0.004 |
| GPT-3.5-turbo | 0.044 $\pm$ 0.007 | 0.061 $\pm$ 0.009 | 0.064 $\pm$ 0.012 | 0.230 $\pm$ 0.032 | 0.032 $\pm$ 0.004 |
| Claude v1 | 0.067 $\pm$ 0.010 | 0.089 $\pm$ 0.014 | 0.110 $\pm$ 0.023 | 0.292 $\pm$ 0.064 | 0.038 $\pm$ 0.005 |
| o1-mini | 0.081 $\pm$ 0.007 | 0.096 $\pm$ 0.014 | 0.127 $\pm$ 0.016 | 0.279 $\pm$ 0.040 | 0.040 $\pm$ 0.004 |
| Claude 3 Sonnet | 0.076 $\pm$ 0.006 | 0.088 $\pm$ 0.007 | 0.115 $\pm$ 0.012 | 0.302 $\pm$ 0.036 | 0.041 $\pm$ 0.003 |
| Claude 3.5 Sonnet | 0.095 $\pm$ 0.004 | 0.104 $\pm$ 0.004 | 0.130 $\pm$ 0.010 | 0.326 $\pm$ 0.034 | 0.042 $\pm$ 0.003 |
| GPT-4o | 0.059 $\pm$ 0.006 | 0.074 $\pm$ 0.006 | 0.116 $\pm$ 0.031 | 0.311 $\pm$ 0.037 | 0.037 $\pm$ 0.004 |
| o1-preview | 0.081 $\pm$ 0.003 | 0.091 $\pm$ 0.004 | 0.141 $\pm$ 0.022 | 0.283 $\pm$ 0.015 | 0.041 $\pm$ 0.001 |
| Claude 3 Opus | 0.094 $\pm$ 0.009 | 0.104 $\pm$ 0.007 | 0.126 $\pm$ 0.011 | 0.247 $\pm$ 0.037 | 0.043 $\pm$ 0.005 |

**Table 7: Agent performance improvements when using different tools for 1-gene perturbation experiments. Results show hit ratio for experimental round 5 averaged over 10 runs, with the error intervals. See Table 1 caption for notes on specific datasets.**
| Tools Used | Schmidt1 | Schmidt2 | CAR-T | Scharen. | Carnev. | Sanchez |
| --- | --- | --- | --- | --- | --- | --- |
| Random | 0.037 $\pm$ 0.013 | 0.031 $\pm$ 0.002 | 0.033 $\pm$ 0.003 | 0.160 $\pm$ 0.028 | 0.036 $\pm$ 0.001 | 0.034 $\pm$ 0.004 |
| BioDiscoveryAgent (Claude 3.5 Sonnet) |  |  |  |  |  |  |
| No-Tools | 0.095 $\pm$ 0.004 | 0.104 $\pm$ 0.004 | 0.130 $\pm$ 0.010 | 0.326 $\pm$ 0.034 | 0.042 $\pm$ 0.003 | 0.066 $\pm$ 0.007 |
| Literature | 0.096 $\pm$ 0.005 | 0.098 $\pm$ 0.010 | 0.138 $\pm$ 0.019 | 0.309 $\pm$ 0.041 | 0.042 $\pm$ 0.002 | 0.069 $\pm$ 0.006 |
| AI Critic | 0.088 $\pm$ 0.004 | 0.092 $\pm$ 0.010 | 0.126 $\pm$ 0.019 | 0.309 $\pm$ 0.037 | 0.042 $\pm$ 0.002 | 0.059 $\pm$ 0.006 |
| Gene Search | 0.096 $\pm$ 0.003 | 0.100 $\pm$ 0.009 | 0.123 $\pm$ 0.009 | 0.348 $\pm$ 0.025 | 0.043 $\pm$ 0.004 | 0.062 $\pm$ 0.007 |
| All-Tools | 0.096 $\pm$ 0.005 | 0.090 $\pm$ 0.003 | 0.121 $\pm$ 0.020 | 0.234 $\pm$ 0.080 | 0.043 $\pm$ 0.001 | 0.054 $\pm$ 0.004 |
| BioDiscoveryAgent (Claude 3 Haiku) |  |  |  |  |  |  |
| No-Tools | 0.064 $\pm$ 0.005 | 0.072 $\pm$ 0.018 | 0.116 $\pm$ 0.014 | 0.209 $\pm$ 0.030 | 0.032 $\pm$ 0.004 | 0.056 $\pm$ 0.005 |
| Literature | 0.053 $\pm$ 0.005 | 0.069 $\pm$ 0.011 | 0.091 $\pm$ 0.031 | 0.164 $\pm$ 0.068 | 0.035 $\pm$ 0.006 | 0.054 $\pm$ 0.009 |
| AI Critic | 0.061 $\pm$ 0.009 | 0.070 $\pm$ 0.009 | 0.113 $\pm$ 0.012 | 0.219 $\pm$ 0.046 | 0.043 $\pm$ 0.006 | 0.054 $\pm$ 0.009 |
| Gene Search | 0.080 $\pm$ 0.013 | 0.098 $\pm$ 0.025 | 0.114 $\pm$ 0.016 | 0.249 $\pm$ 0.067 | 0.046 $\pm$ 0.006 | 0.065 $\pm$ 0.009 |
| All-Tools | 0.084 $\pm$ 0.006 | 0.099 $\pm$<br>0.019 | 0.128 $\pm$ 0.031 | 0.259 $\pm$ 0.039 | 0.043 $\pm$ 0.008 | 0.058 $\pm$ 0.011 |

**Table 8: Effect of tools on different LLMs. Results show hit ratio for experimental round 5 averaged over 10 runs, with error intervals showing 1 standard deviation. ^∗For (Scharenberg et al., ), a batch size of 32 was used due to its smaller pool of 1061 relevant genes. Schmidt1 refers to the screen measuring Interferon-$\gamma$ (IFNG) and Schmidt2 measures Interleukin-2 (IL-2) following perturbation (Schmidt et al., ). ^†CAR-T refers to an unpublished dataset.**
| Tools Used | Schmidt1 | Schmidt2 | CAR-T | Scharen. | Carnev. | Sanchez |
| --- | --- | --- | --- | --- | --- | --- |
| Random | 0.037 | 0.031 | 0.033 | 0.160 | 0.036 | 0.034 |
| Claude 3 Haiku |  |  |  |  |  |  |
| No-Tools | 0.064 | 0.072 | 0.116 | 0.209 | 0.032 | 0.056 |
| All-Tools | 0.084 (+31%) | 0.099 (+38%) | 0.128 (+10%) | 0.259 (+24%) | 0.043 (+34%) | 0.058 (+4%) |
| GPT-3.5-Turbo |  |  |  |  |  |  |
| No-Tools | 0.044 | 0.061 | 0.064 | 0.230 | 0.032 | 0.039 |
| All-Tools | 0.062 (+41%) | 0.100 (+64%) | 0.063 (-2%) | 0.218 (-5%) | 0.037 (+16%) | 0.045 (+15%) |
| Claude v1 |  |  |  |  |  |  |
| No-Tools | 0.067 | 0.089 | 0.110 | 0.292 | 0.038 | 0.053 |
| All-Tools | 0.095 (+42%) | 0.122 (+37%) | 0.114 (+4%) | 0.333 (+14%) | 0.054 (+42%) | 0.058 (+9%) |
| GPT o1-mini |  |  |  |  |  |  |
| No-Tools | 0.081 | 0.096 | 0.127 | 0.279 | 0.041 | 0.074 |
| All-Tools | 0.083 (+2%) | 0.075 (-22%) | 0.114 (-10%) | 0.264 (-5%) | 0.041 (+0%) | 0.072 (-3%) |
| Claude 3 Sonnet |  |  |  |  |  |  |
| No-Tools | 0.076 | 0.088 | 0.115 | 0.302 | 0.041 | 0.064 |
| All-Tools | 0.074 (-3%) | 0.091 (+3%) | 0.105 (-9%) | 0.302 (+0%) | 0.047 (+15%) | 0.073 (+14%) |
| Claude 3.5 Sonnet |  |  |  |  |  |  |
| No-Tools | 0.095 | 0.104 | 0.130 | 0.326 | 0.042 | 0.066 |
| All-Tools | 0.096 (+1%) | 0.090 (-13%) | 0.121 (-6%) | 0.234 (-28%) | 0.043 (+2%) | 0.054 (-18%) |
| GPT-4o |  |  |  |  |  |  |
| No-Tools | 0.059 | 0.074 | 0.116 | 0.311 | 0.037 | 0.030 |
| All-Tools | 0.049 (-17%) | 0.064 (-14%) | 0.103 (-11%) | 0.285 (-8%) | 0.035 (-5%) | 0.026 (-13%) |
| GPT o1-preview |  |  |  |  |  |  |
| No-Tools | 0.081 | 0.091 | 0.141 | 0.283 | 0.041 | 0.068 |
| All-Tools | 0.052 (-36%) | 0.079 (-13%) | 0.098 (-30%) | 0.289 (+2%) | 0.024 (-41%) | 0.031 (-54%) |
| Claude 3 Opus |  |  |  |  |  |  |
| No-Tools | 0.094 | 0.104 | 0.126 | 0.247 | 0.043 | 0.059 |
| All-Tools | 0.096 (+2%) | 0.101 (-3%) | 0.116 (-8%) | 0.291 (+18%) | 0.041 (-5%) | 0.055 (-7%) |

Figure: Figure 7: Percentage of new genes predicted by Claude Haiku only when using tools $H_{a}-H_{n}$ that are also predicted by Claude Sonnet with no-tools ($S_{n}$), where $H_{n}$ is the set of genes predicted by Claude Haiku with no-tools and $H_{a}$ is the set of genes predicted by Claude Haiku with all-tools
Refer to caption: x6.png

Figure: Figure 8: Agent accounts for both prior knowledge and observations in decision-making Three scenarios are considered: BioDiscoveryAgent (Claude v1) has access to task description and experimental observations (Prompt + Observation); the agent only has access to experimental observations (Observation Only); the agent only has access to the task description (Prompt Only). (a) Hit ratio at each experimental round across 30 rounds of experiments. 32 genes predicted per round. Each line corresponds to the average over 10 runs with error bars representing 1 standard deviation. (b) Jaccard similarity index between all predicted genes at 10 rounds of experimentation. Each cell corresponds to a different model run.
Refer to caption: x7.png

**Table 9: Agent uses both prior knowledge and experimental observations to make predictions. Three scenarios: BioDiscoveryAgent (Claude v1) has access to task description and experimental observations (Prompt + Observation); the agent only has access to experimental observations (Observation Only); the agent only has access to the task description (Prompt Only). (a) Hit ratio at each experimental round across 30 rounds of experiments. 32 genes predicted per round across 30 rounds.**
| Dataset | Schmidt22 (IL2) | Schmidt22 (IFNG) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Rounds | 10 | 20 | 30 | 10 | 20 | 30 |  |
| Model | Setting | Avg. Hit Ratio |  |  |  |  |  |
| BioDiscoveryAgent | Observ. Only | 0.0449 | 0.0872 | 0.1037 | 0.0291 | 0.0702 | 0.1061 |
| BioDiscoveryAgent | Prompt Only | 0.0605 | 0.0846 | 0.1071 | 0.0484 | 0.0727 | 0.0872 |
| BioDiscoveryAgent | Prompt + Observ. | 0.0596 | 0.0908 | 0.1143 | 0.0559 | 0.0842 | 0.1077 |

## Appendix F Complete Agent Trace

In this section, we provide a few initial steps of the agent run equipped with all tools, and prompts seen by the agent at each step, as well as the agent’s response. We used BioDiscoveryAgent (Claude v1) for this analysis since it shows high absolute score as well as high performance gain through tool use (Appendix Table 8).

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness.
soundness: 3
strengths: Strengths of this paper includes: 
1. BioDiscoveryAgent introduces a unique approach to closed-loop experimental design, leveraging LLMs to suggest gene perturbations and adapt based on feedback from prior rounds. This paper claims this application of LLMs is novel in experimental biology, particularly for genetic perturbation studies.
2. The agent demonstrates superior performance compared to traditional Bayesian optimization methods. It identify 21% more relevant genes and 46% more non-essential gene hits.
3. BioDiscoveryAgent can draw from diverse information sources with high adaptability and versatility.
summary: This paper introduces BioDiscoveryAgent,an AI-based tool for automating the design of genetic perturbation experiments. Leveraging large language models (LLMs) and biological knowledge, BioDiscoveryAgent identifies genes for perturbation to induce specific phenotypes, such as cell growth, without requiring custom machine learning models or acquisition functions.
Contribution: 1. BioDiscoveryAgent innovates in closed-loop experimental design by using LLMs to propose genetic perturbations in each experimental round, adapting its selections based on prior results and biological insight. 2. The agent outperforms Bayesian optimization baselines by identifying 21% more relevant genes across multiple datasets and achieves a 46% improvement on the more challenging task of identifying non-essential genes. 3. It incorporates external tools like literature search, code execution, and critique from a secondary agent, enhancing interpretability and decision-making.
weaknesses: 1. Although the model demonstrates strong performance on simulated and existing datasets, the paper lacks validation through real-world, wet-lab experiments.
2. BioDiscoveryAgent’s effectiveness is highly dependent on the underlying LLMs’ pre-existing biological knowledge and reasoning abilities. Limitations in the LLM’s training data or biological understanding could restrict the agent's performance.
3. The use of large-scale LLMs, along with additional tools like literature search and gene databases, results in considerable computational overhead.
4. While the agent performs well overall, the paper observes that the greatest performance improvements occur in the early rounds of experimentation, likely due to the LLM’s pre-existing knowledge. However, as rounds progress, these benefits decline, suggesting potential limitations in its ability to learn effectively from experimental feedback over time.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Potential Information Leakage: Could the use of literature and gene search cause information leakage? Specifically, is there a risk that the task query might already be covered in the literature or database, thus affecting the validity of the results?

Linear Performance Curve: Figures 2 and 3(a) show an almost linear improvement in performance across up to 30 turns, without any indication of saturation. This is surprising, as LLM-based multi-turn trials typically reach saturation more quickly. Could you elaborate on why this linear improvement occurs and when, if at all, it might begin to saturate?

Method Validity: The experimental results in Table 1 indicate that different backbones show nearly random performance variations, and the tool usage results in Table 3 also exhibit inconsistent improvements across backbones, which raises concerns about the validity of the proposed method. One possible reason for this could be the low 1-turn accuracy, leading to a low signal-to-noise ratio. Given that the linear performance curve suggests the LLM can effectively address this task with more turns, it would be beneficial to compare the performance of different backbones and tool usage in a multi-turn setting. This could provide more stable results and more insight about backbone’s influence.
soundness: 3
strengths: Backbone Testing: The experiments are conducted across multiple backbone models, enhancing the method's credibility.

Comprehensive Pipeline: The LLM agent is equipped with a variety of tools, resulting in a well-rounded approach.

Clear Writing: The writing is clear, making the method and results easy to understand.
summary: This work proposes BioDiscoveryAgent, a reflection-based LLM agent designed to conduct gene perturbation experiments. It iteratively predicts gene perturbation outcomes, selects genes to perturb, receives experimental results, and continues with subsequent trials based on updated predictions. The study evaluates the effectiveness of BioDiscoveryAgent using experiments on six cell perturbation datasets.
weaknesses: Insufficient Baseline Comparisons: The baseline methods are somewhat distant from the proposed approach. Comparisons with interactive LLM agents, such as ReAct or multi-turn frameworks like reflection-based methods or LLMs as optimizers, would strengthen the evaluation. Additionally, as the method incorporates retrieval augmentation, a comparison with Retrieval-Augmented Generation (RAG) based on the same corpus would be valuable.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: No questions.
soundness: 3
strengths: The authors present an interesting method of using LLMs to help guide perturbation experiments. The authors present extensive experiments with LLMs to pick the LLM best suited to coordinate the tasks to decide which genes will be perturbed. The LLM has access to literature and Reactome to make the decisions of the next gene to perturb. This work follows an increasing movement to design LLM agents to automate parts of the science process and, upon addressing of my points, represents a fruitful direction the community should head towards.
summary: The authors present a LLM agent that can design single-cell experiments. At every step the agent uses literature and Reactome to narrow down a list of genes until it converges, taking the output of the previous step as context.
weaknesses: **Major points:**

1. I am confused what does this paper add when there exists models like GEARS which can predict gene expression that results from perturbations? I realize the authors would argue that this model helps designs experiments as seen by its higher hit ratio but with this logic doesn't GEARS have a very high hit ratio, in one round of in-silico experiments getting many hits? 

2. I am also confused by the metric, what about the genes that the model suggests that are not hits? Do we not care about precision? The genes that the model suggests that are not hits should also be considered. The more of these the worse off. 

3. In line 1855 in the sample agent conversation where does the agent get log-fold change? Does it have a tool that gives it access to this information? From the main text it was just literature and reactome?

4. From Appendix Table 7 it is not clear to me that adding tools helps, a lot of the confidence intervals overlap? Is the difference in the numbers significant here? Also these differences vary by LLM which makes it hard to discern which to do for a particular task.

5. What is the agent adding that cannot be done by simply doing a more traditional pathway enrichment analysis and google? 

**Minor points:**

1. Two-gene perturbation prediction is not a new problem. GEARS introduces this problem already.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. (Same as Weakness 2) Can you provide an intuitive explanation why the effect of using tools differs with LLMs?
2. (Same as Weakness 3) Can you explain a bit the method used by the critique agent, and why you think it may improve performance?
3. You mentioned in the Discussion section that much of the improvements brought by the agent are observed in the early stages, which I think may be related to a better cold start compared to traditional methods. Do you think we can probably use the cold starts from agents to enhance traditional methods?
soundness: 3
strengths: 1. The motivation is clear, and the method design seems intuitive;
2. The experiments are thourough, including different LLMs, both seen and unseen datasets, single and double gene settings, and different tool utilization. The results with the strongest LLM shows a large margin over traditional methods, supporting the effectiveness of their approach.
3. Detailed prompts were provided in the Appendix, which is good for reproducibility.
4. The writing is clear and easy to follow.
summary: This paper presents an LLM-based agent to design gene perturbation experiments for drug targeting, using the LLM's inherent knowledge, previous observations and external tools to improve hit rates over various baselines.
weaknesses: 1. Given the application scenario of the problem, most of the prompting techniques (e.g. to include previous outcomes, to reflect and plan for the next step) are quite intuitive. I do not find anything particularly novel in this part. In particular, you included the literal representation of the experiment outcomes into the prompt for the agent's observation, which is basically an array of numerial data. I wonder if this is the best way to do it.
2. The tool utilization techniques (e.g. searching for genes in the same pathway, searching PubMed articles) are reasonable, but they sometimes does not improve or even hurts the performance. This is concerning when happening with strong LLMs such as Claude 3.5 Sonnet, if we want to explore an upper bound on the abilities of the agent. Moreover, there lacks an explanation why the effects of tools differs with LLMs. From an application perspective, it would be good to predict when using tools helps and when it does not.
3. There lacks an intuitive explanation why using critiques could benefit the agent's predictions, given that the critique is yet another LLM that is not necessary stronger than the actor, and can make mistakes with the same probablity. Unless it is prompted in a way complementary to the actor, or is accessible to some information unavailable to the actor.
4. In general, the paper makes several reasonable attempts to enhance the agent, yet their method does not seem refined enough to contribute substantially to the agent methodology in this application scenario. There also lacks an in-depth discussion on the inconsistency in their empirical findings.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. Why don’t you include all baselines in the 2-gene perturbation experiments (i.e., why do you only compare to random sampling)?
soundness: 4
strengths: 1. **Significant contribution to an important problem.** The problem of efficiently designing genetic perturbation experiments is important, with wide-reaching applications to drug target discovery, gene therapy, immunotherapy, etc. The authors’ experiments show that their proposed method yields a considerable performance improvement over baselines on the task of identifying relevant gene perturbations.
2. **Novel application of LLMs.** While using LLMs for experiment design is not a new idea, to my knowledge, it has not yet been explored for the design of gene perturbation experiments. The author’s approach is considerably different from existing approaches for this task, which primarily focus on Bayesian optimization.
3. **Experiments are thorough and produce interesting insights.** The authors not only show that their method obtains strong performance, they also conduct a series of experiments that shed light on why their method is able to perform well. In particular, the experiments that examine the similarity of the predicted genes across datasets and across experimental rounds are very interesting. The prompt/observation ablations are also very helpful in understanding which parts of the method contribute to the strong performance.
4. **Clear presentation.** The paper is clearly written. The motivation, contribution, and experiment design are all clearly and thoroughly described. It is written in a way such that it is understandable even for someone with a limited background in biology.
summary: The authors propose a new, LLM-based approach for designing gene perturbation experiments, where the goal is to efficiently select a subset of genes that result in a desired phenotype when perturbed. Their approach is based on the creation of BioDiscoveryAgent, an LLM-based agent that can generate, critique, and refine hypotheses about which genes should be selected. They show that BioDiscoveryAgent achieves a large performance improvement over baselines on the task of predicting relevant genetic perturbations across six datasets.
weaknesses: 1. **Lack of technical/methodological novelty**: The two main parts of the method are (1) task-specific prompts and (2)  an algorithm that involves repeatedly calling an LLM with prompts that are adapted based on the outputs at previous steps. This basic process is shared by other work on LLM agents. However, I don’t consider this a major weakness, since the goal of the paper is to design an approach that works well for gene perturbation experiments, and the authors’ approach both works well and is novel within this context.
2. **Claims about interpretability seem overstated**: The authors claim multiple times that their approach offers the benefit of interpretability. For example, they claim that by having the model output a “research plan” and “reflection” in addition to the "solution", this can help to increase interpretability and “rule out predictions that may be hallucinations or not well-motivated.” The problem with this logic is that it assumes that the model’s stated reasoning/reflections are indicative of the true reasoning the model used to arrive at its solutions. However, prior work has shown that an LLM’s stated reasoning/explanations may be misleading/unfaithful to its true decision making process (e.g., [1]). In light of these findings, the author’s claims about interpretability seem overstated to me. I think the paper would be strengthened if the authors discussed the risk that these “reflection” outputs could be misleading. In addition, the authors claim that the LLM’s use of the literature search tool enhances interpretability. However, if the LLM incorrectly summarizes an article or hallucinates information that is not present in the article, this could actually harm interpretability (because it would be misleading). While the user could go to the article and check the correctness of LLM summaries, this is a time-consuming process, so it may not be realistic to expect a user to do this.

[1] Turpin, Miles, et al. "Language models don't always say what they think: unfaithful explanations in chain-of-thought prompting." Advances in Neural Information Processing Systems 36 (2024).

Meta-review: The paper introduces BioDiscoveryAgent, an AI-based tool leveraging large language models (LLMs) to design genetic perturbation experiments, "reason" about their outcomes, and compare and contrast hypotheses explaining them. The agent identifies genes for perturbation to induce specific phenotypes, such as cell growth, without requiring custom machine learning models or acquisition functions. The authors demonstrate BioDiscoveryAgent’s effectiveness across multiple datasets, including a novel one introduced here, showing significant improvements in predicting relevant genetic perturbations compared to existing Bayesian optimization baselines.

Strengths:
1. **Empirical Performance**: The main selling point of the paper is the effectiveness of the method; BioDiscoveryAgent shows substantial improvements over baselines across multiple settings.
2. **Comprehensive Evaluation**: The paper includes extensive experiments, covering multiple datasets (seen and unseen), different LLM backbones, and various tool utilizations.
3. **Presentation**: The writing is clear and easy to follow, the results are well presented.
4. **System Interpretability***: The agent provides detailed reasoning trace for its predictions, enhancing transparency and aiding in the validation of its outputs. However, whether those explanations are truly faithful to the model's internal mechanisms is unclear (and an open problem, as pointed out by reviewer  biwg).


Weaknesses:
1. **Limited Novelty in Methodology**: While the specific application and setting is novel, the underlying methods, such as using LLMs and tools, is certainly not new, with a very rich literature in the past year.
2. **Dependence on underlying LLM**: The agent’s performance heavily relies on the pre-existing knowledge and reasoning capabilities of the LLMs, which may limit its effectiveness in less-studied areas, obfuscates its operation (compared to alternative non-LLM bio methods), and makes it susceptible to known failure modes of LLMs (hallucinations, lack of robustness to prompt variation, etc).
3. **Unclear Interpretation**: Certain aspects of the method, design choices, and trends in results are missing explanation or discussion. 
4. **No Wet-lab Validation**: The paper lacks real-world, wet-lab validation of the agent’s predictions, relying solely on computational benchmarks.

During the rebuttal period, the authors made significant efforts to address the reviewers’ concerns, including running additional experiments, clearing up many misunderstandings, providing thoughtful additional intuition, and improving the clarity of their presentation. Their diligent and meticulous rebuttal prompted two reviewers to increase their scores, and greatly mitigated weakness 3. Although weakness 4 is important, I do not consider it a deal-breaker considering the costs associated with wet-lab validation and the scope of this conference.

Overall, this paper addresses an important problem and provides a promising direction for the use of AI agents in scientific discovery. While some fundamental issues regarding novelty and generalizability remained, the overall positive feedback and the substantial empirical gains justify the decision to accept the paper, with the understanding that the additional results and discussion generated during the rebuttal will be included in the final version of the paper.

Decision: Accept (Poster)