TusoAI: Agentic Optimization for Scientific Methods

## Abstract

Abstract Scientific discovery is often slowed by the manual development of computational tools needed to analyze complex experimental data.
Building such tools is costly and time-consuming because scientists must iteratively review literature, test modeling and scientific assumptions against empirical data, and implement these insights into efficient software.
Large language models (LLMs) have demonstrated strong capabilities in synthesizing literature, reasoning with empirical data, and generating domain-specific code, offering new opportunities to accelerate computational method development.
Existing LLM-based systems either focus on performing scientific analyses using existing computational methods or on developing computational methods or models for general machine learning without effectively integrating the often unstructured knowledge specific to scientific domains.
Here, we introduce TusoAI, an agentic AI system that takes a scientific task description with an evaluation function and autonomously develops and optimizes computational methods for the application.
TusoAI integrates domain knowledge into a knowledge tree representation and performs iterative, domain-specific optimization and model diagnosis, improving performance over a pool of candidate solutions.
We conducted comprehensive benchmark evaluations demonstrating that TusoAI outperforms state-of-the-art expert methods, MLE agents, and scientific AI agents across diverse tasks, such as single-cell RNA-seq data denoising and satellite-based earth monitoring.
Applying TusoAI to two key open problems in genetics improved existing computational methods (40% power improvement to scDRS in associating cells to disease in simulations and 10.5% enrichment improvement to pgBoost for identifying ground-truth variant-gene pairs) and uncovered novel biology, including 9 new associations between autoimmune diseases and T cell subtypes (e.g., primary biliary cirrhosis with central memory T cells) and 7 previously unreported links between disease variants linked to their target genes (e.g., glucose/HbA1c risk variant rs138917529 with GCK ).
Our code is publicly available at https://github.com/Alistair-Turcan/TusoAI .

## 1 Introduction

Scientific discoveries are often bottlenecked by the slow, manual development of computational tools needed to analyze experimental data.
For example, genetics studies have uncovered tens of thousands of disease-associated variants, yet robust computational methods are critically needed to harmonize multi-modal, multi-scale data and uncover the underlying mechanisms (Lappalainen & MacArthur, 2021).
Developing such tools is slow and costly because scientists must iteratively (i) review extensive literature, (ii) test modeling and scientific assumptions against empirical data, and (iii) implement these insights into efficient, scalable code.
For instance, building robust computational methods to link enhancers with target genes from single-cell multiome data has taken multiple expert groups many years (Dorans et al., 2025), hindered by challenges such as *cis*-regulatory modeling, latent confounding, noisy data, and computational scalability.
Large language models (LLMs) have demonstrated strong capabilities in performing human-like analysis (Luo et al., 2025), such as synthesizing relevant literature (Asai et al., 2024), reasoning about biological and modeling assumptions using empirical data (Gao et al., 2024), and generating efficient, domain-specific code (Rasheed et al., 2025).
Integrating LLMs with scientific domain knowledge and iterative data experimentation holds great promise to accelerate computational method development, thereby advancing discoveries in science and medicine.

Existing work has produced general-purpose AI agents across scientific domains, including biomedicine (Huang et al., 2025; Jin et al., 2025) and chemistry (M. Bran et al., 2024).
These systems primarily focus on performing scientific data analyses rather than developing new computational methods; the former involves assembling and executing pipelines of data formatting and existing tools, whereas the latter requires creating new algorithms or models for specific pipeline steps, involving substantial design, optimization, and validation.
In parallel, several studies have developed machine learning engineering (MLE) agents that can design new algorithms for general ML applications (Guo et al., 2024; Trirat et al., 2024; Jiang et al., 2025; Nam et al., 2025), but these approaches do not address domain-specific challenges inherent in scientific research.
Developing *AI agents for scientific method development* that integrate structured domain knowledge and systematically explore data-specific assumptions has considerable potential to accelerate the creation of robust computational methods for science and medicine.

Here, we introduce TusoAI, an agentic AI system that takes a scientific task description with an evaluation function, and autonomously develops and optimizes computational methods for the application (Figure [1](https://arxiv.org/html/2509.23986v1#S1.F1)).
TusoAI integrates structured domain knowledge with iterative, domain-specific optimization and model diagnosis, improving performance over a pool of candidate solutions. We demonstrate that TusoAI achieves superior performance across a range of algorithmic, statistical, machine learning, and deep learning applications in science.
Our key contributions are:

- 1.
We develop TusoAI, an AI agent specifically tailored for scientific method discovery by integrating structured domain knowledge.
- 2.
We propose a novel framework, featuring (i) knowledge tree for structured representation of domain knowledge, (ii) hierarchical planning with Bayesian updates to balance solution quality and diversity, and (iii) fine-grained generation that integrates model optimization with diagnostic feedback.
- 3.
We benchmark TusoAI on 6 single-cell analysis tasks and 5 scientific deep learning tasks, consistently outperforming baseline methods and frequently surpassing existing expert-designed algorithms with on average 16% improvement in single-cell tasks.
- 4.
Applying TusoAI to two key open problems in genetics improved existing computational methods (40% power improvement to scDRS in associating cells to disease in simulations and 10.5% enrichment improvement to pgBoost for identifying ground-truth variant-gene pairs) and uncovered novel biology, including 9 new associations between autoimmune diseases and T cell subtypes (e.g., primary biliary cirrhosis with central memory T cells) and 7 previously unreported links between disease variants linked to their target genes (e.g., glucose/HbA1c risk variant rs138917529 with *GCK*).

Figure: Figure 1: Scientific method development with TusoAI. (A) Method overview. (B) Example domain knowledge tree (categories and instructions per category), feedback, and diagnostics.
Refer to caption: figs/method_overview.png

### 1.1 Related work

LLM-based scientific AI agents.
Several works have developed general-purpose AI agents capable of autonomously executing various scientific research tasks.
Biomni (Huang et al., 2025) provides a unified agentic environment with tools and databases spanning 25 biomedical domains, integrating LLM reasoning with retrieval-augmented planning and code execution to compose complex workflows.
Stella (Jin et al., 2025) employs a multi-agent architecture for autonomous biomedical data analysis, achieving self-evolution by dynamically updating its template library and tool collection.
ChemCrow (M. Bran et al., 2024) is a chemistry-focused agent that integrates 18 expert-designed tools and follows the “Thought, Action, Action Input, Observation” format to iteratively reason toward answers.
These methods emphasize end-to-end data analysis with established tools, whereas our work focuses on developing new computational methods for domain-specific tasks.
Other works have leveraged LLMs to develop application-specific methods, such as single-cell perturbation prediction (Tang et al., 2025), diagnosis prediction (Tan et al., 2025), and mathematical discovery (Romera-Paredes et al., 2024).
In contrast, TusoAI targets computational method development across scientific tasks. InternAgent (Team et al., 2025) and its precursor Dolphin (Yuan et al., 2025) iteratively evolve and implement research ideas through an optimization process augmented with literature review.
As a concurrent effort, Aygün et al. (2025) combine LLMs with tree search and existing model ensembles to improve scientific algorithms, addressing a similar problem but with a different approach from ours, which integrates a domain knowledge tree with fine-grained iterative optimization and Bayesian updates. As their code is not publicly available, direct comparison is not possible.

LLM-based general machine learning agents.
Several recent works have developed AI agents for general machine learning engineering.
AIDE (Jiang et al., 2025) frames ML engineering as a code optimization problem, combining an LLM with tree search to iteratively improve solutions.
R&D Agent (Yang et al., 2025) similarly explores ML architectures in a dynamic feedback loop.
DS-Agent (Guo et al., 2024) combines an LLM with case-based reasoning (CBR), retrieving potentially successful solutions from top-ranked Kaggle solutions, and refining them through iterative optimization.
MLE-STAR (Nam et al., 2025) retrieves candidate models from the web to form an initial solution, then improve it by targeting specific ML components and ensembling.
AutoML-Agent (Trirat et al., 2024) employs retrieval-augmented planning and multi-agent coordination to generate an optimal plan, but executes the plan once without iterative refinement.
These methods are less suited to scientific method development, where domain knowledge is unstructured, existing ML models may be unavailable, and search spaces are continually evolving. We address these challenges through structured domain knowledge representation and hierarchical planning with Bayesian updates during iterative optimization.

Classical automatic machine learning (AutoML) frameworks.
Classical (non-LLM) AutoML frameworks aim to construct high-performing ML models from scratch by searching over key components such as feature preprocessing, model architectures, hyperparameters, and pipeline composition.
Notable examples include auto-sklearn (Feurer et al., 2015), H2O (LeDell et al., 2020), AutoGluon (Erickson et al., 2020), and TPOT (Olson & Moore, 2016).
Within deep learning, neural architecture search (NAS) methods specialize in optimizing neural architectures, with examples such as DARTS (Liu et al., 2018) and AMBER (Zhang et al., 2021).
While effective for standard ML tasks, these approaches are constrained by predefined search spaces and are less suited to scientific domains, where domain knowledge and optimization objectives are unstructured and continually evolving, making LLM-based agents a more natural fit.

## 2 Problem formulation

We consider the problem of automatic scientific algorithm optimization with LLMs.
Given a general solution space $\mathcal{S}^{\text{full}}$ (e.g., all Python scripts) and an evaluator $h(\cdot):\mathcal{S}^{\text{full}}\mapsto\mathbb{R}$, the objective is to find the optimal solution $s^{*}=\arg\max_{s\in\mathcal{S}^{\text{full}}}h(s)$.
$h(\cdot)$ can be any evaluation metric, such as AUC, average of several metrics, or domain-specific measures (e.g., enrichment of inferred disease genes against an expert-curated set).
We assume access to a task description $\mathcal{T}$ (e.g., “single-cell RNA-seq imputation”), a domain-specific knowledge base (e.g., scientific papers), and a general LLM that can be instantiated as agents.
The agent can, for example, summarize domain priors from $\mathcal{T}$, retrieve information from the knowledge base, and refine a candidate solution $s$ based on instructions.
The goal is to iteratively implement and improve solutions to maximize $h(\cdot)$ within a time budget.
We consider two settings: a *cold start*, where optimization begins from scratch, and a *warm start*, where an initial solution $s_{\text{init}}$ (e.g., a state-of-the-art method) is given for further improvement.

## 3 Methods

TusoAI takes as input a task description $\mathcal{T}$, a dataset $\mathcal{D}$, an evaluator $h(\cdot)$, and optionally an initial solution $s_{\text{init}}$.
It outputs an optimized solution $s^{*}$ (Algorithm [1](https://arxiv.org/html/2509.23986v1#alg1)).
Developing computational methods for scientific domains poses several challenges.
First, domain-specific knowledge is often unstructured, which we address using a knowledge tree that organizes information into categories and within-category instructions.
Second, approaches and optimization strategies can vary widely, which we manage through hierarchical planning with Bayesian updates to promote diversity while ensuring solution quality.
Third, understanding complex data patterns is challenging, which we mitigate with fine-grained generation that integrates model optimization with diagnostic feedback.

TusoAI consists of 3 steps.
First, it gathers domain knowledge by summarizing key scientific papers, ensuring that optimization instructions reflect established best practices and recent advances rather than relying solely on LLM priors.
Second, it builds a two-level knowledge tree of structured instructions: (1) categories of optimization strategies and (2) specific instructions within each category, promoting both diversity and relevance. Categories and instructions are first drafted by the LLM and then refined through additional LLM queries in conjunction with paper summaries to ensure diversity and scientific rigor; we also predefine a diagnostic category $\mathcal{I}_{\text{diag}}$ to guide data logging and model diagnosis.
Third, after initializing candidate solutions, it iteratively selects diverse top performers and improves them through either instruction-based or diagnostic-based optimization.
Instruction categories are sampled adaptively via a Bayesian strategy informed by past performance, while feedback comparing new and prior solutions helps discourage repetition.
Examples of instructions generated are provided at Appendix [B](https://arxiv.org/html/2509.23986v1#A2).

Figure: Algorithm 1 TusoAI

Step 1: Gather domain knowledge.
TusoAI first retrieves up to 10 key papers from Semantic Scholar (Allen Institute for AI, 2025) relevant to $\mathcal{T}$, ranked by citation count.
For each paper, an agent $A_{\text{paper}}$ creates a 15-point technical summary from the abstract and iteratively refines it using each paragraph of the paper’s Methods section (up to 1,200 words to focus solely on technical content without relying on costly deep research agents parsing the entire document).
This produces $\mathcal{P}=\{\mathcal{P}_{i}\}$, where each $\mathcal{P}_{i}$ is a refined 15-point summary of paper $i$’s method.

Step 2: Build structured instructions.
TusoAI uses a draft-then-refine strategy to construct optimization categories, where an agent $A_{\text{cate}}$ first drafts candidate categories from the task description $\mathcal{T}$, then refines them by iterating through each paper summary $\mathcal{P}_{i}\in\mathcal{P}$, adjusting existing categories or adding new ones as needed.
Categories are task-specific and can be general (e.g., “regularization”, “model architectures”) or domain-specific (e.g., “single-cell noise modeling”, “genetic feature interactions”).
Each category is assigned a probability $\pi_{c}$ representing its utility in the optimization process; $\pi_{c}$ is initialized by $A_{\text{cate}}$ so that tasks earlier in the pipeline (e.g., “feature preprocessing”) receive higher weight than later ones (e.g., “hyperparameter tuning”).
Similarly, TusoAI uses a draft-then-refine strategy to initialize instructions for each category, where an agent $A_{\text{instr}}$ first drafts 10 candidate instructions $\mathcal{I}_{c}$ from the task description $\mathcal{T}$. These instruction lists are then refined by incorporating 10 additional instructions for each paper summary $\mathcal{P}_{i}\in\mathcal{P}$.
For feedback, TusoAI initializes an empty list $\mathcal{F}_{c}\leftarrow\emptyset$ for each category, which is updated with category-specific feedback during optimization.
A special predefined diagnostic category $\mathcal{I}_{\text{diag}}$ provides instructions for logging diagnostic information useful for model updates, with its own feedback list $\mathcal{F}_{\text{diag}}$.

Step 3.1: Initialize solutions.
The initialization agent $A_{\text{init}}$ drafts 5 candidate solution descriptions from $\mathcal{T}$ and iteratively refines them using each paper summary in $\mathcal{P}$, adding new descriptions or improving existing ones (e.g., “zero-inflated Poisson with kNN smoothing”). These are basic descriptions designed to start from scratch and explore to avoid simply re-implementing existing solutions.
It then attempts to implement and debug each solution; those that successfully compile form the initial solution pool $\mathcal{S}$.
Each implementation attempt is limited to 10 minutes with up to 4 bug-fix attempts.

Step 3.2: Iterative optimization.
Given the current solution set $\mathcal{S}$, TusoAI selects diverse top solutions by clustering them based on code-text similarity and, within each cluster, choosing the shortest solution whose performance is within 0.1% of the cluster’s best model; this helps discourage overfitting and randomness while maintaining diversity and concise code.
For each cluster’s top solution $s$, TusoAI performs either instruction-based optimization (80% probability) or diagnostic-based optimization (20% probability).
The resulting solution $s^{\prime}$ is added to the pool $\mathcal{S}\leftarrow\mathcal{S}\cup{s^{\prime}}$.
Each implementation attempt is limited to 10 minutes with up to 2 bug-fix attempts. This time regularization ensures the optimization period is not wasted on a few inefficient implementations, and encourages the final method to be scalable.

- •
Instruction-based optimization.
The optimization agent $A_{\text{optim}}$ selects an instruction by first sampling an instruction category $c\sim\text{Cat}(\{\pi_{c}\}_{c\in\mathcal{C}})$, then uniformly draw 3 candidate instructions from $\mathcal{I}_{c}$, and finally choosing the most promising among them.
It then optimizes $s$ to produce $s^{\prime}$ using the selected instruction in conjunction with 5 most recent feedback entries from $\mathcal{F}_{c}$.
If $h(s^{\prime})>h(s)$, TusoAI updates the category utility by setting $\pi_{c}\leftarrow 1.1\pi_{c}$ and renormalizing $\{\pi_{c}\}_{c\in\mathcal{C}}$.
Finally, the feedback agent $A_{\text{feedback}}$ summarizes the change from $s$ to $s^{\prime}$ and appends it to $\mathcal{F}_{c}$ (e.g., “this optimization constructed a kNN on the top 50 PC’s rather than on all genes, improving performance by 15%”).
- •
Diagnostic-based optimization.
The optimization agent $A_{\text{optim}}$ selects an instruction by first uniformly draw 3 candidate instructions from $\mathcal{I}_{\text{diag}}$ and then choosing the most promising among them (e.g., “training curves”, “distribution checks”, “validation of assumptions”).
It then diagnoses and improves $s$ to produce $s^{\prime}$ using the selected instruction in conjunction with 5 most recent feedback entries from $\mathcal{F}_{c}$: it runs $s$ to collect diagnostic logs and then uses this information to produce an improved model $s^{\prime}$.
Finally, the feedback agent $A_{\text{feedback}}$ summarizes the change from $s$ to $s^{\prime}$ and appends it to $\mathcal{F}_{\text{diag}}$.

## 4 Experiments

We evaluate TusoAI on 11 scientific applications spanning diverse domains, including 6 single-cell analysis tasks (Luecken et al., 2025) and 5 scientific deep learning tasks (Tu et al., 2022).
The single-cell tasks include denoising (Denoise), cell-type label projection (Label), batch integration (Batch), identification of spatially variable genes (SVG), decomposition of spot-level spatial data into specific cell types (Decomp), and dimensionality reduction for visualization (Visual).
The scientific deep learning tasks include omnidirectional vision (Spherical), prosthetics control (NinaPro), medical diagnostics (ECG), earth monitoring (Satellite), and genetic prediction (DeepSea). In each task, we run TusoAI for 8 hours, optimizing performance on a validation dataset, and evaluating final performance on a separate deployment dataset.
We conduct comprehensive ablation studies to assess the contribution of different components of TusoAI (Subsection [4.2](https://arxiv.org/html/2509.23986v1#S4.SS2)), and two case studies demonstrating how TusoAI can reveal new biological insights in genetics (Section [5](https://arxiv.org/html/2509.23986v1#S5)).
Additional details are provided in Appendix [C](https://arxiv.org/html/2509.23986v1#A3), [D](https://arxiv.org/html/2509.23986v1#A4), [J](https://arxiv.org/html/2509.23986v1#A10), [K](https://arxiv.org/html/2509.23986v1#A11).

Baseline methods.
We compare TusoAI against the state-of-the-art MLE agent AIDE (Jiang et al., 2025), scientific agents Biomni (Huang et al., 2025) and ChatGPT-Agent (OpenAI, 2025), and top-performing application-specific methods.
Biomni (LLM backbone Claude-4-Sonnet) and ChatGPT-agent (LLM backbone GPT-5) are used to iteratively build models on data for single-cell tasks; for deep learning tasks, where Biomni and ChatGPT-agent were unable to operate, we substitute the best of ten models constructed by Claude-4-Sonnet and GPT-5.
For application-specific baselines, we use the “top-performing expert” method for single-cell tasks (Luecken et al., 2025), and all baseline methods, including expert models and NAS methods, for the scientific deep learning tasks (Tu et al., 2022). This set of baselines is consistent with related work in scientific optimization (Aygün et al., 2025) and a recent benchmark that identified AIDE and Biomni as top-performers (Miller et al., 2025).
For details on baseline implementation, see Appendix [E](https://arxiv.org/html/2509.23986v1#A5).

### 4.1 Performance across benchmark experiments

Results for the 6 single-cell tasks and 5 scientific deep learning tasks are reported in Tables [1](https://arxiv.org/html/2509.23986v1#S4.T1) and [2](https://arxiv.org/html/2509.23986v1#S4.T2).
We reached 2 main conclusions.
First, TusoAI consistently outperformed baseline methods across benchmarks when generating code from scratch (average rank of 1.2 for single-cell tasks and 2.8 for scientific deep learning tasks, vs. 3.0 and 4.0 for the second best, resp.).
Second, the methods constructed by TusoAI are novel rather than simple re-implementations of existing approaches or calls to standard packages.
Examples include:
(i) in single-cell denoise, TusoAI designed a non-negative matrix factorization (NMF) approach that models dropout rates, Poisson noise, and performs iterative refinement, distinct from the only other NMF-based approach in the OpenProblems benchmark, ALRA (Linderman et al., 2022);
(ii) in SVG, TusoAI adapted known techniques such as modeling expression as a function of spatial coordinates and neighborhood summaries to create a custom, high-performing method;
(iii) in Satellite, TusoAI combined preprocessing, training procedures, loss functions, and ensembling techniques to build the top-performing model;
and (iv) in Spherical, TusoAI fine-tuned layers of ResNet-50 and augmented the data with random flips and rotations.
Third, all methods constructed by TusoAI are computationally efficient, owing to the runtime constraints imposed during optimization.

**Table 1: Single-cell benchmarks. We report performance across 6 single-cell tasks. “*” denotes agentic methods. Best in bold, second-best underlined. 95% CIs across 3 random seeds all under 0.01 and thus not shown.**
|  | Denoise | Label | Batch | SVG | Decomp | Visual | Avg | Avg rank |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Expert | 0.28 | 0.85 | 0.71 | 0.66 | 0.49 | 0.44 | 0.57 | 3.7 |
| AIDE* | 0.30 | 0.87 | 0.71 | 0.73 | 0.06 | 0.44 | 0.52 | 3.0 |
| Biomni* | 0.16 | 0.89 | 0.82 | 0.16 | 0.53 | 0.35 | 0.49 | 3.7 |
| ChatGPT-Agent* | 0.03 | 0.81 | 0.83 | 0.60 | 0.74 | 0.38 | 0.57 | 3.5 |
| TusoAI* | 0.35 | 0.89 | 0.83 | 0.80 | 0.64 | 0.44 | 0.66 | 1.2 |

**Table 2: Scientific deep learning benchmarks. We report performance across 5 scientific deep learning tasks. “*” denotes agentic methods. Performance of non-agentic methods extracted from NASBENCH-360. Best in bold, second-best underlined. 95% CIs provided across 3 random seeds.**
|  | Spherical | NinaPro | ECG | Satellite | DeepSEA | Avg | Avg<br>rank |
| --- | --- | --- | --- | --- | --- | --- | --- |
| WRN default | 0.14 $\pm$ 0.01 | 0.93 $\pm$ 0.00 | 0.57 $\pm$ 0.00 | 0.85 $\pm$ 0.00 | 0.60 $\pm$ 0.00 | 0.62 | 6.9 |
| DenseNAS random | 0.29 $\pm$ 0.02 | 0.92 $\pm$ 0.01 | 0.58 $\pm$ 0.00 | 0.86 $\pm$ 0.00 | 0.60 $\pm$ 0.00 | 0.65 | 5.4 |
| DenseNAS original | 0.27 $\pm$ 0.01 | 0.90 $\pm$ 0.01 | 0.60 $\pm$ 0.00 | 0.86 $\pm$ 0.01 | 0.60 $\pm$ 0.00 | 0.65 | 5.8 |
| Perceiver IO | 0.17 $\pm$ 0.00 | 0.78 $\pm$ 0.02 | 0.34 $\pm$ 0.00 | 0.84 $\pm$ 0.00 | 0.62 $\pm$ 0.00 | 0.55 | 9.6 |
| XGBoost | 0.03 $\pm$ 0.00 | 0.78 $\pm$ 0.01 | 0.44 $\pm$ 0.00 | 0.64 $\pm$ 0.00 | 0.50 $\pm$ 0.00 | 0.48 | 11.6 |
| WRN ASHA | 0.25 $\pm$ 0.00 | 0.93 $\pm$ 0.01 | 0.57 $\pm$ 0.00 | 0.84 $\pm$ 0.01 | 0.59 $\pm$ 0.00 | 0.63 | 7.1 |
| DARTS | 0.52 $\pm$ 0.03 | 0.82 $\pm$ 0.01 | 0.66 $\pm$ 0.00 | 0.87 $\pm$ 0.00 | 0.68 $\pm$ 0.00 | 0.71 | 4.0 |
| AMBER | N/A | N/A | 0.67 $\pm$ 0.00 | 0.87 $\pm$ 0.00 | 0.68 | N/A | N/A |
| Expert | 0.33 $\pm$ 0.01 | 0.91 $\pm$ 0.01 | 0.72 $\pm$ 0.00 | 0.80 $\pm$ 0.00 | 0.70 $\pm$ 0.00 | 0.69 | 4.6 |
| AIDE* | 0.16 $\pm$ 0.01 | 0.86 $\pm$ 0.00 | 0.52 $\pm$ 0.01 | 0.83 $\pm$ 0.01 | 0.57 $\pm$ 0.00 | 0.59 | 9.8 |
| GPT-5* | 0.36 $\pm$ 0.00 | 0.89 $\pm$ 0.00 | 0.58 $\pm$ 0.03 | 0.86 $\pm$ 0.01 | 0.66 $\pm$ 0.00 | 0.67 | 5.8 |
| Claude-4-Sonnet* | 0.40 $\pm$ 0.00 | 0.90 $\pm$ 0.00 | 0.50 $\pm$ 0.01 | 0.88 $\pm$ 0.00 | 0.73 $\pm$ 0.00 | 0.68 | 4.6 |
| TusoAI* | 0.42 $\pm$ 0.01 | 0.90 $\pm$ 0.00 | 0.61 $\pm$ 0.00 | 0.89 $\pm$ 0.01 | 0.70 $\pm$ 0.00 | 0.70 | 2.8 |

We conducted 2 secondary analyses.
First, we assessed the diversity of code produced by TusoAI and AIDE over 8 hours of optimization, quantifying code diversity using cosine similarity of text embeddings between each candidate and its 10 previous and 10 subsequent iterations (Figure [2](https://arxiv.org/html/2509.23986v1#S4.F2)A). TusoAI achieved substantially higher diversity than AIDE throughout the optimization process.
For example, in the batch integration benchmark, AIDE repeatedly proposed small variations of UMAP-based dimensionality reduction, whereas TusoAI explored a wide variety of dimensionality reduction, transformation, and scaling techniques.
This higher diversity is perhaps due to TusoAI’s instruction sampling, feedback, and diagnosis procedures, which encourage diverse solutions.
In contrast, AIDE promotes incremental changes at each optimization step to facilitate traceability, which may bias the search toward local tuning rather than full exploration.
Details of the diversity procedure is provided in Appendix [G](https://arxiv.org/html/2509.23986v1#A7).
Second, we characterized the optimization trajectory of TusoAI on the single-cell denoising task (Figure [2](https://arxiv.org/html/2509.23986v1#S4.F2)B).
We identified 5 key developments that led to strong performance: (1) introducing NMF, (2) modeling dropout, (3) modeling Poisson noise, (4) adding iterative refinement, and (5) incorporating a sparsity-balancing step.
Notably, during optimization, TusoAI generated many methods that reduced performance before converging on high-performing solutions.
Together with the feedback mechanism, this broad exploration allowed TusoAI to efficiently search the solution space and identify top-performing methods.

Figure: Figure 2: Behavior of code generated by TusoAI. (A) Code diversity of TusoAI and AIDE over optimization time, as measured by $1-$ cosine similarity. Each line corresponds to a dataset. (B) Performance of the current code and the best code over optimization time for a representative task “Denoise”. Key optimization changes with their occurrence times are annotated.
Refer to caption: figs/secondary_analysis_fig.png

### 4.2 Ablation studies

We conducted extensive ablation studies to evaluate the impact of each novel component of TusoAI, including:
(i) removing the categorical structure and placing all instructions and feedback into a single category (No categories);
(ii) disabling the Bayesian sampling strategy across categories (No Bayesian);
(iii) disabling the model diagnosis capability (No diagnosis); and
(iv) discarding domain knowledge altogether, such that each iteration simply applies a generic instruction (e.g., “Optimize this model”; No knowledge).
Removing these components each negatively affected overall performance (Table [3](https://arxiv.org/html/2509.23986v1#S4.T3)).
We attribute this to reduced code diversity (mean diversity 0.48 vs. 0.44/0.39/0.38/0.33 for ablated versions, resp.) and computational efficiency (mean time to optimize 2.3 hours vs. 2.4/3.0/2.6/2.4 for ablated versions, resp.).
Removing domain knowledge had the strongest impact on performance and diversity, while removing Bayesian updates most reduced TusoAI’s computational efficiency. See Appendix [H](https://arxiv.org/html/2509.23986v1#A8) for further ablation details.

We next assessed the impact of LLM backbones used by TusoAI, testing across 5 different LLMs: low-latency models GPT-4o-mini (default) and Claude-3.5-Haiku; state-of-the-art reasoning models GPT-5 and Claude-4-Sonnet; and open-source GPT-oss-120b.
Results are shown in Table [3](https://arxiv.org/html/2509.23986v1#S4.T3).
Apart from GPT-oss-120b, TusoAI achieved relatively consistent performance across all LLMs for most tasks, demonstrating robustness.
Interestingly, LLMs such as GPT-5 and Claude-4-Sonnet did not consistently outperform their lower-latency counterparts, GPT-4o-mini and Claude-3.5-Haiku. This may be because, while reasoning models can construct highly complex code, their tendency to over-build (e.g., each of GPT-5’s methods are 300+ lines of code) makes subsequent iterations difficult to refine; in contrast, low-latency but capable models like GPT-4o-mini and Claude-3.5-Haiku, when paired with an appropriate system design, performed just as well at a fraction of the cost (e.g., optimizing denoising for 8 hours costs 0.24$ with GPT-4o-mini and 22.3$ with GPT-5). See Appendix [I](https://arxiv.org/html/2509.23986v1#A9) for further LLM details.

**Table 3: Ablation studies (top) and varying LLM backbone (bottom). Best in bold, second-best underlined.**
|  | Denoise | SVG | Decomposition | ECG | Satellite | Avg | Avg<br>rank |
| --- | --- | --- | --- | --- | --- | --- | --- |
| TusoAI (default) | 0.35 | 0.80 | 0.64 | 0.61 | 0.89 | 0.66 | 2.0 |
| No categories | 0.09 | 0.72 | 0.56 | 0.63 | 0.86 | 0.57 | 3.2 |
| No Bayesian | 0.36 | 0.77 | 0.22 | 0.57 | 0.84 | 0.55 | 3.4 |
| No diagnosis | 0.26 | 0.77 | 0.68 | 0.63 | 0.86 | 0.64 | 2.0 |
| No knowledge | 0.17 | 0.51 | 0.07 | 0.68 | 0.85 | 0.46 | 3.8 |
| GPT-4o-mini (default) | 0.35 | 0.80 | 0.64 | 0.61 | 0.89 | 0.66 | 2.2 |
| GPT-5 | 0.31 | 0.80 | 0.82 | 0.67 | 0.87 | 0.69 | 2.2 |
| Claude 3.5 Haiku | 0.41 | 0.78 | 0.70 | 0.63 | 0.89 | 0.68 | 1.8 |
| Claude 4 Sonnet | 0.32 | 0.78 | 0.53 | 0.59 | 0.84 | 0.61 | 4.2 |
| GPT-oss-120b | 0.39 | 0.74 | 0.13 | 0.61 | 0.85 | 0.54 | 3.8 |

## 5 Case Studies in Genetics

We applied TusoAI to address 2 key challenges in genetics: detecting disease-critical cell populations and linking genetic variants to their target genes; these are central to understanding disease etiology but limited by current computational models.
We initialized TusoAI with state-of-the-art methods (scDRS (Zhang et al., 2022) and pgBoost (Dorans et al., 2025), resp.) and evaluated its ability to improve these approaches and generate new biological insights.

#### Detecting disease-critical cell populations.

scDRS (Zhang et al., 2022) is a state-of-the-art method that integrates genome-wide association studies (GWAS) with single-cell RNA-seq (scRNA-seq) to identify disease-associated cell populations, but its power is limited by the high noise of scRNA-seq data.
Here, we apply TusoAI in conjunction with scDRS and task it with optimizing scDRS’s association scoring function.
Results are reported in Figure [3](https://arxiv.org/html/2509.23986v1#S5.F3).
We reached 3 main conclusions.
First, the TusoAI-optimized version substantially outperformed the original scDRS in both simulations and real-data benchmarks: it achieved over 40% higher power in causal simulations (Figure [3](https://arxiv.org/html/2509.23986v1#S5.F3)A) while retaining calibration in null settings (Appendix [J](https://arxiv.org/html/2509.23986v1#A10)), and identified 21% more true cell type–disease associations (17 vs. 14) without false associations in a real-data benchmark (Li et al., 2025).
Second, the TusoAI-optimized scoring function is concise and interpretable.
It computes association scores in *log–log* rather than *log* space, likely because this transformation better captures polygenic disease signals across many genes, avoiding domination by a few highly expressed genes.
This improvement reflects TusoAI’s ability to efficiently explore function space: it tested 167 unique versions in 24 hours and at a cost of $0.37, whereas the original authors evaluated fewer than 10 versions over 3 months.
Third, applying the TusoAI-optimized scDRS to a T cell dataset (Cano-Gamez et al., 2020) revealed 26 disease-associated T cell subpopulations (vs. 17 by the original method), including regulatory T cells, central memory T cells, and effector memory T cells associated with primary biliary cirrhosis, consistent with the roles of these T cell populations in autoimmunity (Dominguez-Villar & Hafler, 2018; Seo et al., 2025).

Figure: Figure 3: Optimizing scDRS for detecting cell-disease associations. (A) Assessing power in causal simulations. 95% CI’s are calculated across 30 replicates at each perturbation effect size. (B) Number of discovered ground-truth trait-cell type pairs at FDR 0.05. (C) Number of discovered trait-T cell subtype pairs at FDR 0.05.
Refer to caption: figs/scdrs_fig.png

#### Linking genetic variants to genes using single-cell multiome.

pgBoost (Dorans et al., 2025) is a state-of-the-art method for linking genetic variants to target genes using single-cell multiome data; it integrates variant–gene distance with multiple linking strategies, but the task remains challenging due to the complexity of genetic regulation (Gazal et al., 2022).
Here, we apply TusoAI in conjunction with pgBoost, providing additional positional information for variants and genes, and task it with optimizing distance-based features.
Results are reported in Figure [4](https://arxiv.org/html/2509.23986v1#S5.F4).
We reached 3 main conclusions.
First, the TusoAI-optimized model significantly outperformed the original pgBoost, achieving 13.8% higher enrichment of gold-standard links from fine-mapped eQTLs and 7.2% from activity-by-contact (ABC) links, with particularly large gains across longer variant–gene distances (Figure [4](https://arxiv.org/html/2509.23986v1#S5.F4)A,B).
Second, the distance-based features generated by TusoAI are concise and interpretable: 3 are transformed versions of existing features (inverse, squared, and normalized terms), 2 are interactions of gene annotations with distance terms, and the sixth indicates whether the SNP is $<$50kb from the gene’s transcription start site, consistent with literature suggesting the typical enhancer–promoter range of around 70kb (Bower et al., 2025).
TusoAI discovered these features by testing 511 combinations of 153 novel distance features within 24 hours at a cost of $0.41, whereas the original authors evaluated 5 features over 1.5 months.
Third, applying the TusoAI-optimized pgBoost to fine-mapped SNPs for 94 diseases/traits identified 7 novel variant–gene links missed by previous methods.
For example, a fine-mapped variant rs138917529 for glucose and HbA1c was linked to *GCK*, consistent with the roles of Glucokinase in regulating glucose levels related to both glucose metabolism (Froguel et al., 1993) and HbA1c variation (Chakera et al., 2015).

Figure: Figure 4: Optimizing pgBoost for SNP-gene link discovery. (A) Area under the enrichment-recall curve (AUERC, as defined in pgBoost) across distance thresholds for ground truth eQTL variant-gene links. (B) AUERC across distance thresholds for ground truth ABC variant-gene links. (C) Locus plot of rs138917529 and surrounding genes. Red dashed line indicates cutoff for SNP-gene linking.
Refer to caption: figs/pgboost_fig.png

## References

- Allen Institute for AI (2025)
Allen Institute for AI.
Semantic scholar.
[https://www.semanticscholar.org](https://www.semanticscholar.org), 2025.
Accessed: 2025-09-16.
- Asai et al. (2024)
Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’arcy, et al.
Openscholar: Synthesizing scientific literature with retrieval-augmented lms.
*arXiv preprint arXiv:2411.14199*, 2024.
- Aygün et al. (2025)
Eser Aygün, Anastasiya Belyaeva, Gheorghe Comanici, Marc Coram, Hao Cui, Jake Garrison, Renee Johnston Anton Kast, Cory Y McLean, Peter Norgaard, Zahra Shamsi, et al.
An ai system to help scientists write expert-level empirical software.
*arXiv preprint arXiv:2509.06503*, 2025.
- Bower et al. (2025)
G. Bower, E. W. Hollingsworth, S. H. Jacinto, et al.
Range extender mediates long-distance enhancer activity.
*Nature*, 643:830–838, 2025.
doi: 10.1038/s41586-025-09221-6.
- Cano-Gamez et al. (2020)
Enrique Cano-Gamez, Blagoje Soskic, Theodoros I. Roumeliotis, et al.
Single-cell transcriptomics identifies an effectorness gradient shaping the response of cd4+ t cells to cytokines.
*Nature Communications*, 11(1):1801, 2020.
doi: 10.1038/s41467-020-15543-y.
URL [https://doi.org/10.1038/s41467-020-15543-y](https://doi.org/10.1038/s41467-020-15543-y).
- Chakera et al. (2015)
Ali J. Chakera, Anna M. Steele, Anna L. Gloyn, Maggie H. Shepherd, Beverley Shields, Sian Ellard, and Andrew T. Hattersley.
Recognition and management of individuals with hyperglycemia because of a heterozygous glucokinase mutation.
*Diabetes Care*, 38(7):1383–1392, 06 2015.
ISSN 0149-5992.
doi: 10.2337/dc14-2769.
URL [https://doi.org/10.2337/dc14-2769](https://doi.org/10.2337/dc14-2769).
- Dominguez-Villar & Hafler (2018)
Margarita Dominguez-Villar and David A. Hafler.
Regulatory t cells in autoimmune disease.
*Nature Immunology*, 19:665–673, 2018.
doi: 10.1038/s41590-018-0120-4.
URL [https://doi.org/10.1038/s41590-018-0120-4](https://doi.org/10.1038/s41590-018-0120-4).
- Dorans et al. (2025)
Elizabeth Dorans, Karthik Jagadeesh, Kushal Dey, and Alkes L Price.
Linking regulatory variants to target genes by integrating single-cell multiome methods and genomic distance.
*Nature Genetics*, pp. 1–10, 2025.
- Erickson et al. (2020)
Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola.
Autogluon-tabular: Robust and accurate automl for structured data.
*arXiv preprint arXiv:2003.06505*, 2020.
- Feng et al. (2020)
Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou.
Codebert: A pre-trained model for programming and natural languages.
*CoRR*, abs/2002.08155, 2020.
URL [https://arxiv.org/abs/2002.08155](https://arxiv.org/abs/2002.08155).
- Feurer et al. (2015)
Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter.
Efficient and robust automated machine learning.
*Advances in neural information processing systems*, 28, 2015.
- Froguel et al. (1993)
Philippe Froguel, Habib Zouali, Nathalie Vionnet, Gilberto Velho, Martine Vaxillaire, Fang Sun, Suzanne Lesage, Markus Stoffel, Jun Takeda, Philippe Passa, M. Alan Permutt, Jacques S. Beckmann, Graeme I. Bell, and Daniel Cohen.
Familial hyperglycemia due to mutations in glucokinase – definition of a subtype of diabetes mellitus.
*New England Journal of Medicine*, 328(10):697–702, 1993.
doi: 10.1056/NEJM199303113281005.
URL [https://www.nejm.org/doi/full/10.1056/NEJM199303113281005](https://www.nejm.org/doi/full/10.1056/NEJM199303113281005).
- Gao et al. (2024)
Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaie, Jovana Kondic, and Marinka Zitnik.
Empowering biomedical discovery with ai agents.
*Cell*, 187(22):6125–6151, 2024.
- Gazal et al. (2022)
Steven Gazal, Omer Weissbrod, Farhad Hormozdiari, Kushal K. Dey, Joseph Nasser, Kumar A. Jagadeesh, Daniel J. Weiner, Huwenbo Shi, Charles P. Fulco, Luke J. O’Connor, Bogdan Pasaniuc, Jesse M. Engreitz, and Alkes L. Price.
Combining snp-to-gene linking strategies to identify disease genes and assess disease omnigenicity.
*Nature Genetics*, 54(6):827–836, June 2022.
doi: 10.1038/s41588-022-01087-y.
URL [https://doi.org/10.1038/s41588-022-01087-y](https://doi.org/10.1038/s41588-022-01087-y).
- Guo et al. (2024)
Siyuan Guo, Cheng Deng, Ying Wen, Hechang Chen, Yi Chang, and Jun Wang.
Ds-agent: Automated data science by empowering large language models with case-based reasoning.
*arXiv preprint arXiv:2402.17453*, 2024.
- Huang et al. (2025)
Kexin Huang, Serena Zhang, Hanchen Wang, Yuanhao Qu, Yingzhou Lu, Yusuf Roohani, Ryan Li, Lin Qiu, Gavin Li, Junze Zhang, et al.
Biomni: A general-purpose biomedical ai agent.
*biorxiv*, pp. 2025–05, 2025.
- Jiang et al. (2025)
Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu.
Aide: Ai-driven exploration in the space of code.
*arXiv preprint arXiv:2502.13138*, 2025.
- Jin et al. (2025)
Ruofan Jin, Zaixi Zhang, Mengdi Wang, and Le Cong.
Stella: Self-evolving llm agent for biomedical research.
*arXiv preprint arXiv:2507.02004*, 2025.
- Lappalainen & MacArthur (2021)
Tuuli Lappalainen and Daniel G MacArthur.
From variant to function in human disease genetics.
*Science*, 373(6562):1464–1468, 2021.
- LeDell et al. (2020)
Erin LeDell, Sebastien Poirier, et al.
H2o automl: Scalable automatic machine learning.
In *Proceedings of the AutoML Workshop at ICML*, volume 2020, pp.  24, 2020.
- Li et al. (2025)
Ang Li, Tian Lin, Alicia Walker, Xiao Tan, Ruolan Zhao, Shuyang Yao, Patrick F. Sullivan, Jens Hjerling-Leffler, Naomi R. Wray, and Jian Zeng.
Benchmarking methods integrating gwas and single-cell transcriptomic data for mapping trait-cell type associations.
*medRxiv*, 2025.
doi: 10.1101/2025.05.24.25328275.
URL [https://www.medrxiv.org/content/early/2025/06/05/2025.05.24.25328275](https://www.medrxiv.org/content/early/2025/06/05/2025.05.24.25328275).
- Linderman et al. (2022)
George C. Linderman, Jiajun Zhao, Maria Roulis, et al.
Zero-preserving imputation of single-cell rna-seq data.
*Nature Communications*, 13:192, 2022.
doi: 10.1038/s41467-021-27729-z.
URL [https://doi.org/10.1038/s41467-021-27729-z](https://doi.org/10.1038/s41467-021-27729-z).
- Liu et al. (2018)
Hanxiao Liu, Karen Simonyan, and Yiming Yang.
Darts: Differentiable architecture search.
*arXiv preprint arXiv:1806.09055*, 2018.
- Luecken et al. (2025)
M. D. Luecken, S. Gigante, D. B. Burkhardt, et al.
Defining and benchmarking open problems in single-cell analysis.
*Nature Biotechnology*, 43:1035–1040, 2025.
doi: 10.1038/s41587-025-02694-w.
URL [https://doi.org/10.1038/s41587-025-02694-w](https://doi.org/10.1038/s41587-025-02694-w).
- Luo et al. (2025)
Xiaoliang Luo, Akilles Rechardt, Guangzhi Sun, Kevin K Nejad, Felipe Yáñez, Bati Yilmaz, Kangjoo Lee, Alexandra O Cohen, Valentina Borghesani, Anton Pashkov, et al.
Large language models surpass human experts in predicting neuroscience results.
*Nature human behaviour*, 9(2):305–315, 2025.
- M. Bran et al. (2024)
Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller.
Augmenting large language models with chemistry tools.
*Nature Machine Intelligence*, 6(5):525–535, 2024.
- Miller et al. (2025)
Henry E. Miller, Matthew Greenig, Benjamin Tenmann, and Bo Wang.
Bioml-bench: Evaluation of ai agents for end-to-end biomedical ml.
*bioRxiv*, 2025.
doi: 10.1101/2025.09.01.673319.
URL [https://www.biorxiv.org/content/early/2025/09/07/2025.09.01.673319](https://www.biorxiv.org/content/early/2025/09/07/2025.09.01.673319).
- Mudge et al. (2025)
J. M. Mudge, S. Carbonell-Sala, M. Diekhans, et al.
Gencode 2025: reference gene annotation for human and mouse.
*Nucleic Acids Research*, 53(D1):D966–D975, 2025.
doi: 10.1093/nar/gkae1078.
- Nam et al. (2025)
Jaehyun Nam, Jinsung Yoon, Jiefeng Chen, Jinwoo Shin, Sercan Ö Arık, and Tomas Pfister.
Mle-star: Machine learning engineering agent via search and targeted refinement.
*arXiv preprint arXiv:2506.15692*, 2025.
- Olson & Moore (2016)
Randal S Olson and Jason H Moore.
Tpot: A tree-based pipeline optimization tool for automating machine learning.
In *Workshop on automatic machine learning*, pp. 66–74. PMLR, 2016.
- OpenAI (2025)
OpenAI.
Introducing chatgpt agent: Bridging research and action.
[https://openai.com/index/introducing-chatgpt-agent/](https://openai.com/index/introducing-chatgpt-agent/), July 2025.
- Rasheed et al. (2025)
Zeeshan Rasheed, Muhammad Waseem, Kai Kristian Kemell, Aakash Ahmad, Malik Abdul Sami, Jussi Rasku, Kari Systä, and Pekka Abrahamsson.
Large language models for code generation: The practitioners perspective.
*arXiv preprint arXiv:2501.16998*, 2025.
- Romera-Paredes et al. (2024)
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al.
Mathematical discoveries from program search with large language models.
*Nature*, 625(7995):468–475, 2024.
- Seo et al. (2025)
Eui S. Seo, Sang K. Lee, and Young M. Son.
Multifaceted functions of tissue-resident memory t cells in tumorigenesis and cancer immunotherapy.
*Cancer Immunology, Immunotherapy*, 74(6):184, April 2025.
doi: 10.1007/s00262-025-04035-x.
URL [https://doi.org/10.1007/s00262-025-04035-x](https://doi.org/10.1007/s00262-025-04035-x).
- Tan et al. (2025)
Yanchao Tan, Hang Lv, Yunfei Zhan, Guofang Ma, Bo Xiong, and Carl Yang.
Boxlm: Unifying structures and semantics of medical concepts for diagnosis prediction in healthcare.
In *Forty-second International Conference on Machine Learning*, 2025.
- Tang et al. (2025)
Xiangru Tang, Zhuoyun Yu, Jiapeng Chen, Yan Cui, Daniel Shao, Weixu Wang, Fang Wu, Yuchen Zhuang, Wenqi Shi, Zhi Huang, et al.
Cellforge: Agentic design of virtual cell models.
*arXiv preprint arXiv:2508.02276*, 2025.
- Team et al. (2025)
InternAgent Team, Bo Zhang, Shiyang Feng, Xiangchao Yan, Jiakang Yuan, Runmin Ma, Yusong Hu, Zhiyin Yu, Xiaohan He, Songtao Huang, Shaowei Hou, Zheng Nie, Zhilong Wang, Jinyao Liu, Tianshuo Peng, Peng Ye, Dongzhan Zhou, Shufei Zhang, Xiaosong Wang, Yilan Zhang, Meng Li, Zhongying Tu, Xiangyu Yue, Wangli Ouyang, Bowen Zhou, and Lei Bai.
Internagent: When agent becomes the scientist – building closed-loop system from hypothesis to verification, 2025.
URL [https://arxiv.org/abs/2505.16938](https://arxiv.org/abs/2505.16938).
- Trirat et al. (2024)
Patara Trirat, Wonyong Jeong, and Sung Ju Hwang.
Automl-agent: A multi-agent llm framework for full-pipeline automl.
*arXiv preprint arXiv:2410.02958*, 2024.
- Tu et al. (2022)
Renbo Tu, Nicholas Roberts, Mikhail Khodak, Junhong Shen, Frederic Sala, and Ameet Talwalkar.
NAS-bench-360: Benchmarking neural architecture search on diverse tasks.
In *Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2022.
URL [https://openreview.net/forum?id=xUXTbq6gWsB](https://openreview.net/forum?id=xUXTbq6gWsB).
- Yang et al. (2025)
Xu Yang, Xiao Yang, Shikai Fang, Bowen Xian, Yuante Li, Jian Wang, Minrui Xu, Haoran Pan, Xinpeng Hong, Weiqing Liu, Yelong Shen, Weizhu Chen, and Jiang Bian.
R&d-agent: Automating data-driven ai solution building through llm-powered automated research, development, and evolution, 2025.
URL [https://arxiv.org/abs/2505.14738](https://arxiv.org/abs/2505.14738).
- Yuan et al. (2025)
Jiakang Yuan, Xiangchao Yan, Shiyang Feng, Bo Zhang, Tao Chen, Botian Shi, Wanli Ouyang, Yu Qiao, Lei Bai, and Bowen Zhou.
Dolphin: Moving towards closed-loop auto-research through thinking, practice, and feedback, 2025.
URL [https://arxiv.org/abs/2501.03916](https://arxiv.org/abs/2501.03916).
- Zhang et al. (2022)
Martin Jinye Zhang, Kangcheng Hou, Kushal K Dey, Saori Sakaue, Karthik A Jagadeesh, Kathryn Weinand, Aris Taychameekiatchai, Poorvi Rao, Angela Oliveira Pisco, James Zou, et al.
Polygenic enrichment distinguishes disease associations of individual cells in single-cell rna-seq data.
*Nature genetics*, 54(10):1572–1580, 2022.
- Zhang et al. (2021)
Zijun Zhang, Christopher Y Park, Chandra L Theesfeld, and Olga G Troyanskaya.
An automated framework for efficiently designing deep convolutional neural networks in genomics.
*Nature Machine Intelligence*, 3(5):392–400, 2021.

## Appendix A Example TusoAI code template

The below code template is for single-cell denoising. The only function TusoAI ever sees and optimizes is the tuso_model function.

Figure: Listing 1: Single-cell denoising template file.

## Appendix B Example instructions generated by TusoAI

Figure: Listing 2: Example categories for single-cell denoising.

Figure: Listing 3: Example instructions within a category.

### B.1 Predefined diagnostic instructions

Figure: Listing 4: Example predefined diagnostic instructions.

## Appendix C Single-cell analysis tasks setup

The OpenProblems benchmark (Luecken et al., 2025) contains 12 single-cell analysis tasks with numerous testing datasets and benchmark metrics for each. We select 6 tasks: single-cell denoising, label projection, batch integration, spatially variable gene identification, spatial decomposition of cell types, and visualization. These were selected with the following criteria. First, we required more than one dataset, such that we can optimize on one dataset, and deploy the learned method on the remaining testing datasets, excluding the 2 cell-cell communication tasks and perturbation prediction. Second, a publicly available Github to ensure we are reproducing the testing procedures correctly, excluding multimodal integration and modality prediction. Third, a method for the task should be able to run in a reasonable amount of time on a CPU, excluding the foundation model benchmark.

In each task, we performed optimization on one dataset which could be run in a reasonable amount of time ($<2$ for a simple baseline model). The learned methods of each baseline were then applied to the deployment datasets. In selecting benchmark metrics for each task, we had three criteria. First, the metric should not have unavoidable trivial solutions, excluding the Poisson loss metric from denoising, as this can be easily minimized by simply down-weighting lowly expressed genes, including by just scaling genes by their variance or re-normalizing the data. Second, the metrics should be computationally efficient to run, so optimization speed of each method will not be dominated by running metrics. This excluded several metrics from batch integration and visualization. Third, the metric should line up with the task. In the SVG task, it is initially measured in correlation with spatial variability scores, however, the simulation procedure generates binary 0/1 labels of spatial variability, thus we use accuracy of classifying a gene as SVG instead. We also normalize metrics such that each is between 0 and 1 and a higher score is better. The score in denoising is 1-MSE, normalized so that no denoising is 0, and perfect denoising is 1. The score in spatial decomposition is normalized so that a random cell type assignment is 0, and perfect decomposition is 1. See Table [4](https://arxiv.org/html/2509.23986v1#A3.T4) for a full breakdown of datasets and metrics used in single-cell tasks.

**Table 4: Single-cell benchmark setup. Datasets and metrics refer to the setup on the OpenProblems webpage.**
|  | Optimization dataset | Testing datasets | Benchmark metrics |
| --- | --- | --- | --- |
| Denoise | 1K PBMC | 5K PBMC | MSE |
|  |  | Pancreatic |  |
| Label | 5k cells from Immune Cell Atlas | Diabetic Kidney | Accuracy |
|  |  | GTEX v9 | F1 macro |
|  |  | HypoMap | F1 micro |
|  |  | Mouse Pancreatic Islet Atlas | F1 weighted |
|  |  | Tabula Sapiens |  |
| Batch | 5k cells from Immune Cell Atlas | Diabetic Kidney | Graph connectivity |
|  |  | GTEX v9 | ASW label |
|  |  | HypoMap | ASW batch |
|  |  | Mouse Pancreatic Islet Atlas |  |
|  |  | Tabula Sapiens |  |
| SVG | Drosophila Stereo-seq E5 | Drosophila Stereo-seq E10 | Accuracy |
|  |  | Drosophila Stereo-seq E9 |  |
|  |  | Drosophila Stereo-seq E6 |  |
| Decomp | TMS Lung (alpha=1.0) | TMS Lung (alpha=0.5) | R^2 |
|  |  | TMS Lung (alpha=5.0) |  |
|  |  | Pancreas (alpha=0.5) |  |
|  |  | Pancreas (alpha=1.0) |  |
|  |  | Pancreas (alpha=5.0) |  |
| Visual | Mouse HSPCT | 5K PBMC | Trustworthiness |
|  |  | Mouse Myeloid | Distance correlation |
|  |  | Zebrafish | Density Preservation |

## Appendix D Deep learning tasks setup

The NASBENCH-360 benchmark Tu et al. (2022) contains 10 deep learning tasks across scientific domains with predefined training, validation, and testing splits, as well as evaluation procedures. We select 5 tasks: Spherical, NinaPro, DeepSEA, Satellite, and ECG. These were selected with the following criteria. First, the task should be scientific and somewhat understudied compared to standard ML tasks, excluding the 2 standard image and audio classification tasks. Second, to ensure fair comparison against the precomputed baselines, we removed tasks where we were uncertain about reproducing the evaluation procedure, partly due to recent GitHub or package updates requiring debugging, excluding Cosmic, PSICOV, and DarcyFlow.

In each task, we performed optimization by training a model on the predefined training set and attaining a score on the validation set. The final testing accuracy of optimized models is attained when deploying the model on the predefined test set. We use the same splits and metrics defined in the original paper.

## Appendix E Baseline implementations

AIDE. AIDE takes as input a data folder, task description, and evaluation metric. While originally designed for whole-workflow construction in ML tasks, this can be adapted to general optimization in the following ways. First, AIDE can operate on any data input in the data folder. If specific preprocessing information was needed, we could input this code to the task description. Second, in place of specifying an accuracy metric (e.g., ”F1 score”), we instead simply input the entire evaluation function in Python, and found this worked well. As our goal is optimization and not construction, AIDE’s initial prompt is tuned until code was consistently generated and optimized upon, typically requiring the same formatting information as other methods. AIDE is run for the same length as TusoAI (8 hours) in the same conda environment on the same CPU (Optimization for AIDE and TusoAI is performed on the same Intel(R) Xeon(R) Gold 5416S.) or GPU (Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz), given 4 threads and 50GB of memory. While AIDE does have a default timeout per execution of 1 hour, on attempting to set this to the same time as TusoAI led to consistent crashes on more of half of tasks, thus we left it as is.

Biomni. We access Biomni through its web page. Biomni runs on a CPU and can take input files up to a limit, has a runtime execution of 1 hour. While not specifically designed for optimization, we can upload the same template code and data as TusoAI then ask Biomni to perform an iterative process of updates. In practice, this led to between 2 and 10 iterations per task between 20 minutes and 4 hours.

ChatGPT-Agent. We access ChatGPT-Agent through its web page. ChatGPT-Agent can accept input files up to 25MB and has a runtime execution of 1 hour. We upload the same template code and data as TusoAI and ask ChatGPT-Agent to perform an iterative process of updates. In practice, this led to between 2 and 7 iterations per task between 10 minutes and 5 hours.

Expert. The expert baselines for NASBench-360 are pre-computed from their paper. For single-cell tasks, we selected the expert method with the following criteria. First, it should be within the top 3 methods as defined by the existing OpenProblems benchmarking. Second, the OpenProblems Github should have code for reproducing this method. Third, we selected the method that was particularly efficient compared to others, if applicable, defined by a runtime of less than 10 minutes on OpenProblems, with others having greater than 1 hour. This left us with the following expert methods, whose code we extracted from the OpenProblems Github:

- 1.
Denoise - MAGIC
- 2.
Batch - Combat
- 3.
Label - Logistic Regression
- 4.
Decomposition - NNLS
- 5.
SVG - SPARK-X
- 6.
Visualize - T-SNE (log10CP10K)

Claude-4-Sonnet and GPT-5. In deep learning tasks where Biomni and ChatGPT-Agent cannot apply due to computational limitations (file size, runtime, GPU access), we substitute the best of 10 models generated by Claude-4-Sonnet and GPT-5. 10 models are generated by prompting these LLMs using the same template that would have been used in Biomni and ChatGPT-Agent. The best is decided by the top performing model on the validation dataset which ran in less than one hour.

## Appendix F Stability across replicates

Figure: Figure 5: Validation performance across 3 replicates. Final validation performance after running TusoAI 3 separate times on each single-cell task.
Refer to caption: figs/replicate_fig_points.png

## Appendix G Code diversity

We measure the diversity of generated code, as measured by the cosine similarity of the text embedding of one generated code versus all others. This is performed for TusoAI and AIDE. For each, we first filter out repetitive/uninformative code strings, including comments, imports, evaluation functions and data loading procedures (which will not change over iterations). We then apply sklearn’s TfidfVectorizer function to each cleaned code to obtain a text embedding. We can then compute the cosine similarity between pairs of code. Diversity is measured as 1-cosine similarity. We opt for TF-IDF instead of more sophisticated methods like CodeBERT (Feng et al., 2020) which measure semantic similarity, as we observed this overestimated the similarity between code (all cosine similarity $>0.997$ for all tasks). This is likely due to each iteration always being a slight permutation of the same python method performing the same task. TF-IDF better captures a measure of difference between algorithmic procedures in this case.

## Appendix H Ablation analysis

Figure: Figure 6: Additional ablation information. (A) Box plot across 5 tasks of the mean code diversity. (B) Box plot across 5 tasks of the mean time to optimize.
Refer to caption: figs/secondary_appendix.png

## Appendix I LLM analysis

Figure: Figure 7: Additional LLM information. (A) Average length of generated methods for each task and LLM versus the total count of how many methods were generated. (B) Average length of generated methods for each task and LLM versus the final deployment performance.
Refer to caption: figs/llm_appendix.png

## Appendix J scDRS analysis

Optimization setup. scDRS’ codebase consists of several files. We construct a version of compute_score.py that exposes the compute_raw_score function. This is the only function TusoAI operates upon during optimization. For optimization, we construct causal simulations similar to the scDRS paper, subsampling 10k cells from TMS, perturbing 1000 disease genes in a cluster of cells, setting the geneset overlap to 25%, and varying effect size from 5 to 50%. TusoAI optimizes the compute_raw_score function based on the average (F1 + AUPRC)/2 across 3 replicates at effect size 15%, where scDRS has lower power. We run this experiment for 24 hours using default parameter settings for TusoAI.

Additional simulation results. We apply scDRS and the learned version by TusoAI to all 30 replicates of each effect size in causal simulations. We additionally apply it to 100 replicates of null simulations, identical to scDRS, where 1000 random genes are selected with no perturbation. Additional metrics in these simulations are reported in Figure [8](https://arxiv.org/html/2509.23986v1#A10.F8).

Figure: Figure 8: Additional scDRS metrics. (A) Q-Q plot of -log10 p-values in null simulations. 95% CI’s are calculated at each point across 30 replicates. (B) AUPRC of associating individual cells in causal simulations. 95% CI’s are calculated at each point across 30 replicates. (C) FDR of associating individual cells in causal simulations. 95% CI’s are calculated at each point across 30 replicates.
Refer to caption: figs/scdrs_appendix.png

## Appendix K pgBoost analysis

Optimization setup. pgBoost discovers that distance-based features are critical for modeling SNP-gene distances. It’s samples are SNP-gene pairs, and features include over 30 features derived from single-cell multiome methods and 2 distance features, the SNP distance to the gene’s transcription start site (TSS), and a binary indicator of if this is the closest TSS of any gene to the SNP. We augment pgBoost’s script with gene annotations from GENCODE V48 (Mudge et al., 2025), specifically the SNP’s position, the gene’s TSS, and the gene’s transcription end site (TES). During both the knowledge tree construction and optimization process, TusoAI is encouraged to come up with instructions/optimizations relevant to distance-based modeling of SNP-gene links and avoid other model changes. Optimization is performed by increasing the average enrichment in pgBoost’s primary evaluation of gold-standard links (eQTL and ABC) relative to the original pgBoost’s enrichment. We run TusoAI for 24 hours using default parameter settings.

## Appendix L Prompt templates

### L.1 Instructions for parsing literature

Figure: Listing 5: Initializing paper description with abstract.

Figure: Listing 6: Updating paper description with methods section.

### L.2 Instructions for constructing categories

Figure: Listing 7: Initializing categories with LLM.

Figure: Listing 8: Updating categories with papers.

### L.3 Instructions for constructing within-category instructions

Figure: Listing 9: Initializing within-category instructions with LLM.

Figure: Listing 10: Refining within-category instructions with LLM.

### L.4 Instructions for constructing initial solutions

Figure: Listing 11: Initializing solutions with LLM.

Figure: Listing 12: Refining initial solutions with LLM.

### L.5 Prompt template for Biomni and ChatGPT-Agent

Figure: Listing 13: Single-cell denoising prompt template for scientific agents.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How does TusoAI handle the cold start problem for truly novel scientific domains? The knowledge tree construction relies on retrieving papers from Semantic Scholar based on task descriptions. For emerging research areas with limited literature, how does the system bootstrap domain knowledge? Have you tested TusoAI on tasks where relevant literature is scarce or the task description is deliberately vague?

2. Can you provide more details on the comparison with Aygun et al. (2025) and clarify the key technical differences? Beyond the unavailability of their code, what are the conceptual and methodological differences between your approach and theirs? This seems critical for establishing the novelty of your work.
soundness: 2
strengths: 1. The paper evaluates TusoAI on 11 tasks spanning single-cell analysis and scientific deep learning, demonstrating broad applicability. The inclusion of both cold-start and warm-start settings, along with real-world case studies in genetics, strengthens the practical relevance.

2. The knowledge tree representation provides a structured way to organize domain knowledge, and the Bayesian update mechanism offers a principled approach to balancing exploration and exploitation. The diagnostic-based optimization is a nice addition that grounds the system in empirical data patterns.

3. TusoAI consistently outperforms strong baselines including AIDE and domain-specific expert methods. The ablation studies effectively demonstrate that each component (categories, Bayesian updates, diagnostics, domain knowledge) contributes meaningfully to overall performance.
summary: This paper introduces TusoAI, an LLM-based agentic system for automatically developing and optimizing computational methods for scientific applications. The system structures domain knowledge into a hierarchical knowledge tree, uses Bayesian updates to sample optimization strategies across categories, and incorporates diagnostic feedback during iterative refinement. The authors evaluate TusoAI on 11 scientific benchmarks (6 single-cell analysis tasks and 5 deep learning tasks), showing consistent improvements over baselines including AIDE, Biomni, and expert-designed methods. Two case studies in genetics demonstrate practical utility by improving existing methods (scDRS and pgBoost) and uncovering novel biological insights.
weaknesses: 1. The core algorithmic components are largely existing techniques (LLM agents, tree-based search, iterative optimization), and the main contribution is their combination for scientific method development. The concurrent work by Aygun et al. (2025) appears to address very similar problems, but the paper dismisses direct comparison due to unavailable code without sufficiently differentiating the approaches conceptually.

2. The 8-hour optimization budget is relatively short and may favor methods that converge quickly over those requiring longer refinement. The diversity analysis (Figure 2A) shows TusoAI explores more than AIDE, but it's unclear whether this translates to better generalization. The selection strategy (shortest code within 0.1% of best performance) seems ad-hoc and could inadvertently favor simpler but less robust solutions.

3. The paper doesn't discuss when TusoAI fails or performs poorly, nor does it address practical concerns like how to set appropriate evaluation functions for novel scientific problems, how sensitive the system is to task description quality, or how to validate generated methods when ground truth is unavailable. The computational cost analysis is limited to one brief mention ($0.37-$0.41 for case studies).

4. The case studies claim to discover "9 new associations" and "7 previously unreported links," but these are computational predictions that would require experimental validation to be considered true biological discoveries. The statistical methodology for declaring associations as "novel" versus "missed by previous methods" is not clearly described.

5. While the authors promise to release code upon publication, key details are missing: which specific LLM API versions were used, how were papers selected from Semantic Scholar (just citation count?), what are the exact prompts for different agents, and how stable are results across different random seeds beyond the reported 95% CIs?

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 2
questions: - Can the authors clearly specify what metric is used in each column of Tables 1 and 2, and how the “Avg” and “Avg rank” values are computed across heterogeneous tasks? If normalization was applied (e.g., 1 – MSE, scaled decomposition R²), please make this explicit in the main text or table captions.
- How do the qualitative examples, such as NMF with dropout/Poisson modeling and Satellite ensembling, support the narrative of the methods being "distinct" or "custom"?
- Is using text similarity (Fig. 2A) a sufficient proxy for diversity, or should a more principled novelty metric be employed in the evaluation?
- This work’s broad, application-driven scope and descriptive evaluation would be better suited for an applied-science journal rather than a methodology-focused venue like ICLR.
soundness: 3
strengths: - Clear motivation and problem framing. The paper focuses on creating methods rather than just executing pipelines. It specifies the objective and agent loop, including the task description and the evaluator $h(\cdot)$, with cold/warm start.
- Breadth of benchmarks with application case study. Empirical scope spans 11 tasks across two domains (six single-cell, five scientific deep learning).
- The warm-start improvements on scDRS and pgBoost are interesting. It is interesting to consider concrete empirical deltas and biological findings (e.g., new T-cell disease associations; rs138917529→GCK link).
summary: This paper introduces TusoAI, an agentic AI framework that automatically develops and optimizes computational methods for scientific tasks. Given task descriptions and evaluation functions, it can generate new algorithms rather than just run existing analysis pipelines. The main idea is to structure domain knowledge into a “knowledge tree,” combine it with Bayesian-guided iterative optimization and diagnostic feedback, and iteratively generate, implement, and evaluate candidate solutions. On 11 benchmarks, TusoAI achieved a higher average rank compared to expert-designed methods and existing AI agents.
weaknesses: - Too many symbols in Algorithm 1. If the authors want to introduce these annotations, they should include an appendix table.
- Baseline coverage is insufficient, mainly including general-purpose agents (AIDE, Biomni, ChatGPT-Agent) and simple expert or AutoML models. It lacks domain-specific, publication-level baselines for each scientific task. Including parts of the leaderboard is recommended.
- Ablation and diagnosis analyses lack depth. The ablation study in Table 3 mixes tasks from unrelated domains. It omits per-task variance or statistical testing, making it impossible to isolate the contribution of each component (knowledge tree, Bayesian updates, diagnosis). 
- No detailed analysis of failure cases or performance drops is provided, so readers cannot understand when the system fails or regresses.
- Case studies serve as illustrations but lack control. The genetic applications (scDRS, pgBoost) show improvements, yet they lack independent validation or error analysis.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 2
questions: 1) The key motivation behind TusoAI is to develop new computational solutions to existing problems in science.  Do the results show a new computational method developed beyond your expectations? One that is far beyond the reach of human ML engineers?

2) Is there an underlying optimization problem that the Bayesian framework is solving? 

3) What is the convergence criterion of the system? Figure 2A and 2B suggest that TusoAI never really converges, as code diversity oscillates and performance dips. Is this expected or desired? How did you pick when the algorithm should be terminated? Are there any prompting guardrails that avoid oscillations?

4)  Does increasing the number of papers retrieved from Semantics Scholar (from 10) improve the results?

5) Can you elaborate on how the paper ensures the model selection has been done in an apples-to-apples way across the datasets/baselines? Is model evaluation also handled by TusoAI or externally?
soundness: 3
strengths: 1) Mixing Bayesian optimization in the loop of an agentic AI system is interesting. I wish the paper had substantiated whether this is the first agent of this type or whether the idea is inspired by existing agents. 

2) The results show that some of the methods constructed by TusoAI are novel rather than simple re-implementation of existing approaches or calls to standard packages.

3) The agentic system is tested in many scenarios, from final performance to behavior over time.

4) Single-cell results show promising biological results.
summary: The paper develops TusoAI, an agentic AI system that, given a scientific task description and an evaluation function, autonomously develops and optimizes computational methods for the application. TusoAI is tested across several single-cell tasks and some general deep learning tasks.
weaknesses: 1) The writing can be improved. For example, the Abstract is excessively long. Instead of listing all results, please summarize the key points and condense the Abstract's opening.

2) While the evaluations involve tasks beyond a single cell, it seems like TusoAI's performance is more pronounced in the single cell case, which might suggest the paper could have made that the central focus rather than a general-purpose multi-agent AI.

3) The code is not available for a full review of the paper.

4) Narrow scope to problems that can be solved using off-the-shelf ML models and strategies rather than those that need in-depth model development (e.g., new deep learning models rather than say fine-tuning Restnet). 

5) As discussed in the *LLM-based general learning agents* section, there are already agents developed for ML engineering. This work differentiates itself from those by focusing on scientific domains, but some baselines are still missing (R&D, DS, etc).

6) Limitation in scale: only 10 papers retrieved from Semantic Scholar.

7) I have some concerns about model selection and evaluation, which I have asked in the box below.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 1
questions: - what does your acronym TusoAI stand for?
- Table 1 and 2: How is performance defined in these tasks?
- Is the comparison in the experimental section fair in terms of computational load? How long are baseline methods run compared to the 8h for TusoAI?
- Figure 2, 3 and 4: font size is too small and hard to read.
- line 323: How were the text embeddings computed?
- Figure 2B: Does this "optimization trajectory" look similar for all other tasks? Are there examples where the algorithm converges fast and then does not find a better solution?
- In the methods there is no explanation of how the Bayesian update is performed. Can you explain where and how exactly a Bayesian update is performed and what the prior and posteriors involved are?
- Table 3: Do the ablations only disable one of the parts (i.e. "No Bayesian" only disables the Bayesian part but keeps all other parts)?
- line 380: How do you define the mean time to optimize? What is the criterion that optimization is achieved?
- line 385: "Results are shown in Table ??"
- Figure 3 B/C: What does the circles show? How should we read and interpret those figures?
- Figure 4:
  - what does the distance threshold mean?
  - How should read and interpret figure 4C?
soundness: 2
strengths: - the problem of automated computational model design is important 
- ablation studies are provided to reveal some insight into the importance of the methods components
- strong results are achieved on a set of tasks
summary: The paper presents TusoAI, a method that hand designs a set of steps for automated model discovery using LLMs. The authors compare their method against baseline methods on set of task and find that their approach achieves good performance with high diversity in the suggested solutions.
weaknesses: - the paper lacks any discussion or conclusion
- the Bayesian update step is unclear and not described
- the method seems overall very heuristic. A (theoretic) justification of the algorithm would be of great value.
- additionally the algorithm is rather complex and the explanation are difficult to follow at times
- figure are sometimes unclear and would need re-working

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: ## Questions
- Was an ablation study performed on the number of points saved for each paper's methods in Step 1 "Gather domain knowledge"? How does increasing/ descresing point summaries affect the final model performance?
- Why were other models not tested on the case studies? A lot seems to be riding on TusoAI's new findings, but other models were not given a chance.
- How is the iterative refinement done for "The initialization agent Ainit drafts 5 candidate solution descriptions from T and iteratively refines them using each paper summary in P"? It is not clear and appears to be super important.
- How many hyperparameters are there total for this agent? It appears to have a lot, and I would be curious how those hyperparameters affect the solutions. (ie. run time, integer values used in various places, utility scalars, probabilities, bug-fix attempts, etc..)
- Which hardware is used for all of this? 8 hours means a totally different thing on a laptop vs. a data center. Are all models tested on the same hardware? I see some models are only accessible on the web: how do you account for different compute capabilities?
- How many runs were used for ablation analysis (Table 3)? Was it just run once, or several times with mean score? If it was run once then it needs to be run more times, since LLMs are very stochastic and the results can vary significantly.
- Where is the conclusion/future work section? The paper seems to end abruptly.

## Feedback
- The code is a mess. There are import statements all over the place, comments are unstructured and lacking in many places, and there are no clear instructions on how to reproduce results. Please fix this so I can take a look at the code myself and run things locally.
- I_{diag} is not updated in Algorithm 1, it appears to remain as an empty list. Please consider removing or correct this.
- Line 265: Change "instruction by first uniformly draws" to "instruction by first uniformly drawing".
- The claims (line 297, first paragraph in section 4.1) that TusoAI provides "novel" and "computationally efficient" methods are not substantiated. Provide quantitative results for computational efficiency and a better empirical comparison of the results across all models. Simply stating the method TusoAI came up with does not mean it's novel.
- Please make it clear which model is presented in figure 2B in the caption. This specific sub-figure would benefit from a better description in general.
- Please provide motivation for why we care about generating diverse code. I would argue that if an agent can produce better methods, then it doesn't matter how diverse the methods produced along the way are. This is a central point studied in the paper that needs to be substantiated. In fact, producing lots of diverse models is at odds with the 8 hour time limit, since I can also imagine a model would benefit from trying to iterate on a model that works decently well instead of making smaller changes to a larger amount of models in the same period of time.
- Line 386, please correct: "Table ??".
- Line 430: "This improvement reflects TusoAI’s ability...", is not really a fair comparison. TusoAI started from the work the original authors had already done. Please correct this sentence to show how TusoAI expands upon the original work in a short amount of time, instead of making it seem it tested 167 unique versions _from scratch_.
  - Same comment for line 464 for the second case study.
- The text in the figures are tiny. Please make the text larger so it's easier to read instead of having to zoom in 300% to make out what it says.
- Line 1244: "It's samples" change to "Its samples".
soundness: 2
strengths: ## Strengths
- Model architecture is properly motivated and described in sufficient detail.
- The utility of a model is conveyed properly, I can see the usefulness of this.
- Code is provided.
summary: ## Summary
- The paper introduces TusoAI, an LLM agent for developing computational methods for domain-specific scientific tasks. They apply it to several biological and deep learning problems and demonstrate improvement over other agentic and LLM models. They also demonstrate its performance on iterating on existing state-of-the-art methods and show new scientific insights.

## Recommendation
- Reject. While results comparing with other agentic models are there, the actual performance isn't that much better compared to existing models on the specific tasks. The most interesting result is the case studies, but the other models were not run on the same case studies. There is a lack of case studies in other domains. Code is poorly written and not amenable to open source.
weaknesses: ## Weaknesses
- TusoAI is designed to work with any domain as long as domain knowledge is provided, however, there is a strong focus on biological applications. This paper would benefit from showing TusoAI's performance in other scientific domains similar to the two biological case studies. I see that it worked on deep learning but only a cold start was performed. I am specifically asking for a warm start case study in a non-biology domain.
- Diversity in methods produced is a core architecture consideration, but it is not substantiated in the text. While evidence is provided that TusoAI writes diverse code, there are no references supporting the assumption that generating more diverse code is better.
- Code is poorly written / formatted.


Decision: Accept (Poster)