scPilot: Large Language Model Reasoning Toward Automated Single-Cell Analysis and Discovery
Authors: Yiming Gao
Zhen Wang
Jefferson Chen
Mark Antkowiak
Mengzhou Hu
JungHo Kong
Dexter Pratt
Jieyuan Liu
Enze Ma
Zhiting Hu
Eric P. Xing
Venue: NeurIPS 2025 poster ()
Keywords: Large Language Models
Scientific Reasoning
Automation of Science
Single-cell Transcriptomics
Cell Type Annotation
Trajectory Inference
Gene Regulatory Network Inference
Primary area: machine_learning_for_sciences
TL;DR: We present scPilot, a framework for omics-native reasoning in which a large language model converses, invokes bioinformatics tools, and iteratively explains its decisions to for single-cell analysis.

Abstract: We present scPilot, the first systematic framework to practice \textit{omics-native reasoning}: a large language model (LLM) converses in natural language while directly inspecting single-cell RNA-seq data and on-demand bioinformatics tools. scPilot converts core single-cell analyses, i.e., cell-type annotation, developmental-trajectory reconstruction, and transcription-factor targeting, into step-by-step reasoning problems that the model must solve, justify, and, when needed, revise with new evidence. To measure progress, we release \scbench, a suite of 9 expertly curated datasets and graders that faithfully evaluate the omics-native reasoning capability of scPilot w.r.t various LLMs. Experiments with o1 show that \textit{iterative} omics-native reasoning lifts average accuracy by 11\% for cell-type annotation and Gemini 2.5 Pro cuts trajectory graph-edit distance by 30\% versus one-shot prompting, while generating transparent reasoning traces that explain marker gene ambiguity and regulatory logic. By grounding LLMs in raw omics data, scPilot enables auditable, interpretable, and diagnostically informative single-cell analyses.

SCPILOT: Large Language Model Reasoning Toward Automated Single-Cell Analysis and Discovery
Yiming Gao 14=t, Zhen Wang 12+3 Jefferson Chen', Mark Antkowiak', Mengzhou Hu',
JungHo Kong', Dexter Pratt", Jicyuan Liu', Enze Ma', Zhiting Hu Eric P. Xing
'CC San Diego, MBZUAI, 'CMU, *Texas A&M
yiminggaa661@ttmm. edu, zhw0858ucad.
Abstract
We present SCPILOT, the first systematic framework to practice omics-native rea- soning: a large language model (LLM) converses in natural language while directly inspecting single-cell RNA-seq data and on-demand bioinformaties tools. SCP1- LOT converts core single-cell analyses, i.e., cell-type annotation, developmental- trajectory reconstruction, and transcription-factor targeting, into step-by-step rea- soning problems that the model must solve, justify, and, when needed, revise with evidence. To measure progress, release SCBENCH, suite of expertly curated datasets and graders that faithfully evaluate the omics-native reasoning capability of SCPILOT various LLMs. Experiments with o1 show that iterative omics-native reasoning lifts average accuracy by 11% for cell-type annotation and Serrini-2.5-Prz prompting, while generating transparent reasoning traces explain marker gene ambiguity and regulatory logic. By grounding LLMs in raw omics data, SCP1- LOT enables auditable, interpretable, and diagnostically informative single-cell analyses.
trajectory graph-edit distance by 30% versus one-shot
Introduction
In the of exponential growth in biological data, the quest for artificial intelligence (AI) that function as a true scientific assistant to automate and interpret complex scientific analyses has never been more urgent. Recently, large language models (LLMs) have demonstrated surprising breadth in factual recall and reasoning prowess [73 26 32 22), prompting the question: Can LLMs be harnessed as genuine scientific partners revolutionize traditional biological discovery pipeline?
Yet, translating these general LLM capabilities into the realm single-cell biology remains chal- lenging. The surge of single-cell omics has shifted biology from bulk averages to million-ce matrices 663 [4]. but analysis pipelines still depend on implicit, human-only reasoning [39 [4] [7]
(Figure D. While LLMs excel natural-language explanation and reasoning, most current in computational biology utilize LLMs simply as interfaces that invoke existing bioinformatics
tools [75 [1 31), relying solely these tools' inherent functionalities. Other approaches heav-
ily train foundation models to embed single-cell counts into opaque, high-dimennional vector spaces 77 15 671. resulting less interpretable analyses critical to biological discovery.
propose to bridge this gap with omics-native reasoning (ONR)aa new interactive paradigm in which LLM (i) receives concise textual summary derived from the single-cell expression matrix, (ii) explicitly articulates biological hypotheses in natural language, (iii) invokes targeted
"qqual contribution
i Work done during his time at UCSD
i CCorrsppnndigg author
Cooee data, and package are available at
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

bioinformatics operations directly on the raw data, (iv) evaluates and interprets numerical evidence, and (v) iteratively refines its reasoning until arriving at biologically coherent conclusions. As shown in Figure 1 by closely coupling reasoning to raw omics data, ONR generates transparent and auditable analyses, facilitating interpretability, scientific rigor, and human validation.
This paper operationalizes ONR through SCPILOT, a systematic frame- work that harnesses the reasoning ca- pabilities of an off- fff-the-shelf LLM in- tegrated with problem- moblem-tt-teext con- verter and curated bioinformatics tool library. SCPILOT explicitly for- mulates and iteratively refines hy- potheses, addressing three canonical single-cell challenges: cell-rype an- notation, develappmental-rrrjetoooy re- construction, and transcription-face largeting, producing transparent and biologically insightful reasoning pro- cesses. To systematically quantify progress, we further introduce SCBENCH, the first benchmark for omics-native reasoning that scores numerical accuracy and reveals the biological validity of the model's 's narrative across nine expertly curated single-cell tasks. Our contributions are fourfold:
Single
Traditional manual pipcline
Paageented Tools
Houristic dependence
Limited intergretability
Extating tools
Interpretability
|x scPilot
SCONCC
Natural Langunge
Blological Discovery
Figure 1: Human-lik reasoning + established bioinformatics tools hands-free single cell analysis
LLM-driven single-cell analysis framework. We formulate the first omics-native reasoning, language-centric workflow that automates key analytic stages-cell-ypp annotation, trajectory inference, and gene-regulatory network prediction-whie preserving scientific transparency.
Comprehensive benchmark suit. SCBENCH offers task-specific metrics and expert-verified ground truth, enabling objective comparison LLMs on biologically meaningful problems.
Empirical insights and validation. Comprehensive experiments across nine benchmark datasets demonstrate the effectiveness of SCPILOT: iterative omics-native native reasoning lifts average cell-type annotation accuracy by 11%, reduces trajectory graph-edit distance by 26%, and improves GRN prediction AUROC by 0.03 over direct prompting and conventional baselines.
Biological interpretability and diagnostic reasoning. SCPILOT generates transparent reasoning traces that expose marker ambiguities, lineage inconsistencies. and tissue-specific regulatory logic, enabling biologically interpretable and diagnostically informative single-cell analyses.
2 Related Work
Large Language Models in Single-Cell Analysis. Early biomedical LLMs, c.g., BioGPT [48). BioMedLM (8). and Galactica (66), showed that pre- pre-training on PubMed abstracts or full-text markedly improves factual recall and zero-shot QA, while newer general LLMs (e.g., GPT-4o, Claude-3) now rival or exceed them with broader literature coverage. In parallel, a growing family 41
of single-cell foundation models [77 21 15| 67
6 65| 37). mostly encoder-style LLMs that treat genes as tokens to learn gene- and cell-level embeddings for imputation, perturbation prediction, and cross-dataset transfer. Cell2Sentence and C2S-Scale [41]]]] encode each cell as "sentence," enabling natural-language queries, while other works build LLM interfaces for single-cell
59
25
58
data via fine-tuning [46] 61 42 autonomous tool agents 27
19
57 75 [1]. General-purpor
biomedical agents such as Biomni
31 demonstrate autonomous problem-solving across domains.
Despite their progress, these approaches sidestep the core cognitive load of single-cell analysis: embedding models speak in vectors with no explanations, chat wrappers and tool agents re-package fixed results from traditional tools. Yet we need more language-native reasoning for single-cell analysis, the ability for an LLM to argue, justify, and iteratively refine biological conclusions. Prior work (28) showed GPT-4 can label cell types directly from marker genes. SCPILOT pushes this paradigm beyond a single downstream task to the entire analytic workflow systematically.
Automated Single-Cell Analysis Pipelines. Modern single-cell workflows rely comprehensive toolkits like Seurat and Scanpy 774 62] as the backbone. Specialized modules like CellTypist (cell- type annotation), Monocle (developmental trajectory reconstruction), and SCENIC (gene regulatory

---

Prompt Template Dynamic Input Standard Tool Outputs (ISON)
Standard Pipeline Tools
Problem-to-Text Converter
GRN TF-gene prediction
Cell type annotation
Trajectory Inference
Domain Knowledge
60 scPilot call SCENIC to generate
SCENIC+ trols
SCRNA
(0 scPitot gather annotation preparation
Expert Heuristies
Community Standards
(10 scPilot form question
Oit LLM thaun-of-thought construction
Bio Tool Library
Iii acPilot
(i) schilot form dynamic prompting
Incorporate Monocla preudotime
Incorporate
GENDONIOLOGY
chain-aftthough
function of TF and gene
plan:
gana of interest
LLM Planner
dosplot
evaluation
00 LLM self-reflection annotation and tree
Thaught
LLM gene relationship
finalize trajectory tree
Long context Window
(iv) LLM annotation on clusters
Self-rufacttio Loop
scBench
Pacl
Figure 2: Overview of the SCPILOT framework. The system integrates problem-tee -ot-tttttetttt converter, an LLM planner, and a bio-tool library to perform iterative reasoning and tool calls for three workflows: cell-type annotation, trajectory inference, and gene-regulatory prediction.
networks) 16 68
address specific subtasks but expose many hyperparameters and opaque defaults.
Web-based platforms (e.g., ASAP (20) and one-click frameworks (e-g.. SPEEDI (71) simplify execution yet still embed rigid heuristics that may fail on new tissues or perturbations. Recent LLM
tool agents case this burden by writing code and invoking domain packages on demand [27 19 64
57 CellAgent [75). for example, uses GPT-4 to automatically select tools and hyperparameters. While impressive, these systems mainly wrap default heuristics and offer limited biological insights behind tool calling. SCPILOT advances from scripted automation co-piloting: not only call tools, the model needs to interpret the output of Monocle or SCENIC and perform profound biological reasoning for discovery.
Benchmarks for Biological Reasoning Existing benchmarks for single-cell foundation models or al- gorithms emphasize embedding quality or numeric metrics
44
7
47
60 offering little transparency
biologically meaningful interpretation [35] 70, 7]. LLM benchmarks such as BioASQ [38],
PubMedQA 34 McdQA-USMIL [33), and recent GPQA (55] and LAB-Bench 44 test factual recall but not operation on raw omics data. More recent agent-centric suites (BixBench [51]. Sci- AnceABenten [12], FIRE-Bench (122] test LLMs to orchestrate code execution across natural language bioinformatics problems, but their coverage is shallow, and their evaluation is without domain-deep omics reasoning. Isolated efforts have probed language-naten -ative biology. e.g., GPT-4 for cell-type annotation [28] and LLMs for gene set function discovery [30], yet no public benchmark covers multiple single-cell workflows with ground-truth answers and demands explicit biological justification. To fill this gap, our proposed SCBENCH systematically evaluates language-native reasoning across multiple single-cell workflows with curated datasets, numeric ground truth, and automatic metrics-provideee providdig the first rigorous benchmark for the co-pilot paradigm in SCPIL.T.
3 SCPILOT: Automation of Single-Cell Cell Analysis by LLMs
Let X E RxNN be single-cell expression matrix with G genes and N cells, and let g denote biological query (e.g. "What is the cell type of cluster 5?" or "Does TF Z regulate gene Y?").
Classical bioinformatics methods typically address query q by executing predetermined pipeline:
Swel X: 8),
where fooot is selected from established bioinformatics tools such as Scanpy or Monocle, and is set of hyperparameters manually tuned based on implicit biological assumptions and analyst expertise. Although effective, this traditional practice obscures the underlying biological rationale behind the chosen parameters, limiting reproducibility, transparency, and interpretability.
3

---

Recent LLM-based "tool agents" tackle bioinformatics tasks by writing code on the user's is behalf:: the model produces Python or R snippet, executes prints the numeric output, and (optionally) summarizes the result in text. Formally, given query q and an expression matrix X E &RGNN the agent generates code src,, , srek such that
(2) where each function gares, corresponds to invoked bioinformatics operator (e.g., clustering via scanpy. t1. tlliedd)n) The model's reasoning lives mainly in comments and chat text surrounding the code; intermediate raw data and numerical results the human analyst must read both Python and text to audit a run, and the causal logic link between numeric evidence and biological claims is easily lost.
(arex(9arex...... Jere, ())
hidden unless explicitly printed. Consequently,
Omics-Native Reasoning (ONR). We instead require the LLM to reason directly over the omics space and to record every claim, evidence pair in a transparent trace. Let So X. At step k. the reasoner emits a pair (Ck, Ox),
S = (S
where Cj is natural-language claim, justification, or decision, and og is a primitive omics operator. single, verifiable action applied to the current data state S (e.g. filtering, clustering, scoring, look-ups, etc). The sequence
R= = (1), (ck. ok)]
(4)
constitutes a verbal + computational proof. The final state Sy mapped prediction ÿ = h(5k) that answers The final evaluation is conducted by comparing ÿ with a ground truth answer
3.1 The SCPILOT Framework
To operationalize omics-native reasoning for large-scale single-cell experiments, we introduce SCPILOT, a modular framework that systematically transforms high-dimensional omies data into concise textual summaries, guides reasoning via LLMs, and selectively invokes computational biology tools to iteratively gather and assess evidence. Formally, given a biological query q and an expression matrix X, SCPILOT produces prediction ý alongside transparent reasoning trace R. The architecture comprises three interacting components (Figure 2):
Problem-to- -exxet Converter C. Single-cell datasets routinely contain N ~ 102-6 cells, far beyond any context window of current L.MMs. Thus, for each query q, we implement an algorithmic mapping:
p R GxN Sgg
(5)
which produces a semantic sketch 8, = map is algorithmic. not learned (examples presented in Table Crucially, o reduces volume while preserving biological salience, (c.g., reporting cluster sizes, top-ranked marker genes, developmental trajectory connections, transcription factor-target scores) rather than presenting the full expression matrix, thus significantly reducing data dimensionality while retaining critical biological context.
digestible by the LLM within a single prompt. The
Bio-Tool Library T. This curated library provides a set of primitive omics operators E Ω that encapsulate eellbistallished bioinformatics routines (e.g.. Scanpy, Seurat via Reticulate, Monocle PySCENIC), and lightweight plotting utilities. Each operator returns structured, machine- machine-parsable parsable ISON outputs (e.g., numeric scores, ranked gene lists, graphs) accompanied by succinct natural- language description, enabling the reasoner to seamlessly integrate fresh computational evidence into subsequent reasoning steps in Ck+1-
LLM Reasoner Rs. The reasoning module R. instantiated by powerful LLMs (such as ol), receives the textual summary, user-defined biological query, and a structured reasoning scaffold refined through iterative prompting and domain-expert heuristics. These elements jointly establish closed-loop reasoning workflow:
X S Prompt R. (Thought,, C I, R - ÿ.
(6)
Design Principles. SCPILOT provides a modular, flexible blueprint that enables researchers to
customize reasoning workflows tailored explicitly to their biological queries. The framework adheres to three rigorously validated design principles essential for achieving optimal performance: (a) Bio- logical context first: Prompts consistently incorporate key biological metadata, such species, tissue

---

Table 1: Summary of SCBENCH. Each row defines a computational task, how it's compressed into text, datasets used for evaluation, metrics for assessment, and ground truth sources.
Task
Compression (Problem - Text)
Datasets
Metric
Ground truth
Cype- annota-
Scanpy Leiden clusters + top-k marker genes per cluster (k: 10)
PBMC
Liver
43
Retina
60
Cluster level accuracy (1 /0.5/0
Authoe prodvied labels
tion
Trajectory infer-
PAGA or Manocle-3 lineage Pancreas
Liver
Neocortex
Node-Jaccard f. Graph. Human-curated lin- Edit Spectral Dist.
ence
graph and pscudotime; land-
3
cage tree in original study
mark genes varying monoton- ically
GRN edge predie-
Top-M TF-gene pairs from GRNdb stomach. liver, kidney 18
AUROC
Experiment-validen
tion
PySCENIC with motif support TRRUST validation (M S 150)
24
edges from TRRUST v2 24
type, and experimental protocol. Expert knowledge is required to choose the context, from previous reasoning tool calls. (b) Iterative reasoning: Reasoning and reflection are unfolded iteratively, systematically refining hypotheses based on accumulating computational evidence and even pervious mistakes. (c) Minimal manual heuristics: We seed each task with high-level prompts distilled from do- main best practices, without task-specific fine-tuning of LLM parameters; performance improvements arise exclusively through enhanced prompting strategies and richer evidence.
3.2 SCBENCH: Benchmarking SCPILOT with World Biological Meaningful Tasks
We introduce SCBENCH (summarized in Table 1 a comprehensive benchmark designed explicitly to evaluate the biological reasoning capabilities of SCPILOT across representative single-cell RNA sequencing (scRNA-seq) analysis tasks. These tasks encapsulate the analytical complexity and experimental challenges commonly encountered in practical single-cell studies. Datasets included in SCBENCH are carefully selected from high-quality, publicly available scRNA-seq studies, providing a realistic and diverse platform for assessing SCPILOT 's utility and robustness in bioinformatics discovery. To ensure fairness, reproducibility, and rigorous evaluation, termination conditions for each task within SCBENCH are pre-specified rather than being autonomously determined by the LLM.
Cell Type Annotation. Given a ScRNA expression matrix, the goal of this foundational task is to assign biologically accurate cell-type labels to each cell. Traditionally. this has relied heavily on manual annotation due to limitations in automated tools. We thus curated manually annotated scRNA datasets from published pa- pers: PBMC3K dataset [ from 10x Genomics, Liver 43), and Retina [50].
Table 2: Cell type annotation scores across datasets. Values represent mean where available. Higher values indicate better performance. The top three performances for each col- umn are highlighted with decreasing background intensity.
Method
Liver
PBMC
Retina
Cell'yyist
0.464
0,563
0.388
GPTCelIType
0.404
+ 0.141
0.613 = 0.288
0.300 : 0.297
CellMarker 2.0
0.304
0.250
0.632
Biomni (Gemini -2.5 Pro) 0.464
i: 0007
0.646
0.095
0.570
: 0.135
SCPILOT employs a fixed maximum of three reasoning iterations, provid- ing the LLM sufficient scope to iteratively refine hypotheses and self-correct without ""vvertinkiin"" or excessive computational cost.
Direct (ol)
0.560 1: 0.032
0.667
0.071
0.474 1: 0.045
SCP11OT (ol)
0.518 = 0,022
0.792
0.071
0.728 = 0.084
Trajectory Inference. This task involves reconstructing cellular developmental progression paths, typically structured as lineage trees. Conventionally, trajectory reconstruction relies on statistical tools such as Monocle [10] and manual validation by domain experts. We selected three scRNA-sce datasets adhering stringent criteria: 1) datasets representing clear cellular differentiation processes, 2) original studies that explicitly included trajectory analysis, and 3) availability of expert-curated trajectory lineages as ground truth. The selected datasets are Pancreas (5). Liver 45). and Neocortex [33. This task is implemented as a single-pass reasoning process, incorporating an initial trajectory construction followed by a controlled refinement step guided by Monocle's 's output.
Gene Regulatory Network Prediction. Given a transcription factor (TF) and target gene pair, the task is to predict the existence of a regulatory relationship. Standard approaches typically utilize computational pipelines like SCENIC 4 for candidate predictions, subsequently validated through laboratory experiments and consolidated in reference databases such as TRRUST (24. For
5

---

Table 3: Cell Type Annotation Performance across Different LLMs and Datasets. Values represent performance metries with standard deviations. The top three performances for each dataset and task type are highlighted with decreasing background intensity.
PBMC3K
Liver
Retina
LLM
Direct
SCPILOT
Direct
SCPILOT
Direct
SCPILOT
GPT-4o
0.604 0.005
0.646 + 0017
0.440 + 0,02
0.512 + 0.002
0.439 = 0.002
0.675 0.011
GPT-4o-mini
0.625 0.000
0.604 1 0.001
0.339
0.03
0.387
+: 0.04
0.404
0,000
0.447
0.006
01
0.667 0.005
0.792 + 0.055
0.560
0001
0.518 + 0.011
0.474
0,002
0.728 0.007
Ol-mini
0,646 0.009
0,521 4: 0.001
0.351
0.000
0.435
1: 0.005
0,456
0,001
0.649 0,000
Gemini 2.0 Pro
0.604 + 0.005
0.792 + 0.009
0.494 + 0.001
0.509 + 0,088
0.491
+ 0,000
0.763
Gemini 2.0 Flash
0.604 + 0.001
0,500 + 0.004
0.411 1 + 0.000
0.435 = 0.001
0,500
= 0.001
0.623 = 0.006
Gemini 2.5 Pro
0.583 + 0.001
0.708
+ 0.021
0.494 + 0.007
0.488 + 0.001
0.482
= 0.001
0.675 0.003
Gemma 27B
0.479 0.001
0.500 + 0.016
0.345 0.000
0.393 + 0.004
0.526 1: 0.005
0.579 = 0.002
benchmarking, we compiled GRN data from GRNdb [8], a comprehensive database containing TF-gene predictions derived from omies data via SCENIC. We selected three representative tissues from SoNddb-Stomache Liver, and Kidney-and incorporated experimentally validated TF-gene pairs from TRRUST ground truth. This task is structured single-pass reasoning exercise, with all relevant evidence presented upfront for a thorough, integrated reasoning process.
4 Experiments
Benchmarked Models. We evaluated eight models, including seven proprietary and one prominent open-source model, to represent diverse performance tiers and availability. Proprietary models include GPT-4o, GPT-4o-mini, o1, o1-mini [52], Gemini-2, Oroo, Gemini-2 www.--Fa--Thiimmn and Gemini-2 -2.5Poo [I3]. The open-source one is Gemma-3-
[3 among the best available at the time of experimentation. To facilitate direct comparison, each primary model was evaluated alongside its lightweight variant (c.g., -mini). Further details regarding model versions provided in the Supplementary Material.
Baseline Methods. To rigorously assess performance, we benchmarked SCPILOT against relevant baseline methods across traditional bioinformatic methods and recent LLM- methods.
Tyll-Type Annotation:
Four established baseline approaches were included traditional machine
learning and database-driven methods (Celltypist 1.7.1 (76), CellMarker 2.0 29 ) and LLM-based
methods (GPTCelllype 28). Biomni (31).
Trajectory Inference: Two baseline methods were utilized-py Py-Monocle 100, a conventional trajec- tory inference method, and Biomni, representing an advanced LLM-driven pipeline.
Gene Regulatory Network (GRN) Prediction: We implemented and evaluated three graph neural network architectures Graph Convolutional Networks (GCN) (36), Graph Attention Networks (GAT) [69], and GraphSAGE 23 each trained on the GRNdb dataset. Additionally, we compared
our approach with two contemporary LLMM-based methods: LLM4GRN (2 and BioGPT 449
Direct Prompting. To further contextualize SCPILOT'S performance, we implemented straightforward direct L.M-prompting baselines for each task. All prompts are provided in Appendix.
Cell-type annotation: A single LLM call directly assigns cell-type labels based on differentially expressed genes per cluster, supplemented by high-level dataset descriptions.
Trajectory inference: Consists of three sequential, independent LLM calls: initial cell-type an- notation, subsequent trajectory inference using the annotations, and joint reconsideration step informed by py-Monocle results.
GRN prediction: A single-step LLM prompt directly predicts TF-gene regulatory relationships without iterative refinement.
4.1 Main Result Analysis
Cell-type Annotation. Table demoostrattsss that both direct prompting and the SCPILOT framework outperform traditional annotation tools. Table summarizes the mean accuracy + variance across

---

Table 4: Trajectory reconstruction performance across different LLMs. Values represent mean standnr devintion. Jaceard, higher values (f) are better; for GED-nx (10s) and Spectral Distance, lower values (L) are better. The top three performances for each column are highlighted with decreasing background intensity.
Pancreas
Liver
Neocortex
LLM
Direct
SCPILOT
Direct
SCPILOT
Direct
SCPILOT
Jaccard (t)
GPT-4o
0.923
0.872
0,118
0.956 + 0.978
0.792
0.917 0.01
GPT-4o-mini
0.923 + 0.134
0.718
+ 0.045
0.889
+ 0.138
0.778
+ 0.100
0.792
0.130
0.875 0.110
01
1,000 + 0.000
1,000
= 0.000
1.000
+ 0.000
1.000
+ 0.000
1.000
0,000
0.833 + 0.071
01-mini
0.846 + 0.5
0.744
= 0.221
0.956
+ 0.077
0.911
+ 0.077
0.708
0.032
0.813
0.063
Gemini 2.0 pro
0.949 0.089
1,000
0.978 +: 0.033
1,000 0.000
0.958
1,000 0.000
Gemini 2.0 Flash
0.974 i: 0.045
0,923
=: 0.134
0.933 +: 0.063
1,000
1: 0.000
0.854
0.145
0.958 0.071
Gemini 2.5 Pro
0.949 + 0.089
1,000 = 0.000
1.000 + 0.000
1.000 + 0.000
0.949
0.089
1.000 + 0.000
GED-nx (10s) (L)
GPT-40
13.0 1: 4,00
10.67 1.53
10.0 :: 2,00
8.67 4.04
14.0 5.29
16.33 1 4.01
GPT-4o-mini
19.33 +: 8.08
20.67
= 5.77
12.67 1.15
16.00 1.6
17.33 = 5.03
15.33 5.13
01
6.67 + 2:31
5.33 = 1.15
10.67 5.77
8.00 + 3.46
14.00 6.93
13.33 + 1.15
O1-mini
22.67 2.31
12.00
10.67
+ 3,05
13.00
18.00
2.65
18.00 0.00
Gemini 2.0 pro
8.33 2.08
7.00
10.00
6.67
14.00
12.67
Gemini 2.0 Flash
16.33 + 8.02
13.33
1.53
11.00 1.00
11.33 6.43
17.00 6.08
14.00 4 2.68
Gemini 2.5 Pro
8.33 + 2:08
5.00
+ 1.73
8.00
+ 4.00
3,33 4 2.311
13.33
1.15
9.50 #:2.2
Spectral Distance (4)
GPT-4o
0.640 + 0.295
0.772 # 0,447
0.469
+ 0.105
0.383 + 0.286
1.313
= 0.346
0.946
0.622
GPT-4o-mini
1.704 + 0.640
1.482
+ 0.462
0.745
+ 0.447
1.072 + 0.362
1.428
0.363
1.055 + 0.110
01
0.271 +: 0.077
0.271
= 0,077
0.634
+: 0.158
0,567 1: 0.451
1,005
0,032
1.261 0:26
O1-mini
1.993
= 0.431
1.219
0.670
0.155
0.636
1.624
1,576 0.071
Gemini 2.0 pro
0.453 1: 0.192
:: 0.190
0.362
1: 0.349
0.192 1: 0.000
0.842
0.454
1.064 1: 0.071
Gemini 2.0 Flash
1.026 + 1.123
0.901
= 0.148
0.501
+ 0.197
0.289 + 0.138
1.137
0.497
0.561 + 0.479
Gemini 2.5 Pro
0.453 + 0.192
0.310 0.029
0.388
+ 0.205
0.199 + 0.032
0.977
0.134
1.052 0.055
Table 5: AUROC Scores for Gene Regulatory Network Inference.
Stomach
Liver
Kidney
LLM
Direct
SCPILOT
Direct
SCPILOT
Direct
SCPILOT
GPT-40
0.623 0.001
0.800 1 0.001
0.577 :: 0.001
0.743 :: 0.000
0.570 = 0.001
0.707 0.002
GPT-4o-mini
0.583 0.000
0.697 + 0.01
0,600 = 0.004
0.683
0001
0.567
0.000
0.733 0.011
0.827 + 0.002
0.873 + 0.004
0.753 + 0.001
0.760
+: 0.000
0.777
= 0.000
0.797 +: 0.001
O1-mini
0.690
0.783 1: 0.660
0.700
0.000
0.640
0.000
0.727 0.001
Gemini 2.0 Pro
0.949 = 0.008
1.000 +: 0.000
0.600 = 0.000
0.737 +: 0.000
0.600
= 0.000
0.743 1: 0.001
Gemini 2.0 Flash
0.690
0.697
0.690
0.753
0.683
0.730
Gemini 2.5 Pro
0.610 0.006
0.820 +: 0,000
0.637 #: 0.000
0.753 4: 0.001
0.623 0.001
0.727 à: 0.002
three benchmark datasets. Among direct approaches, o1 achieved the highest overall accuracy (0.667 on PBMC3K, 0.560 on Liver, 0.474 on Retina), underscoring the importance of model capacity.
Implementing the SCPILOT's pipeline further improved accuracy for 19 out of 24 model-dataset combina- tions. The Retina dataset showed the most significant median accuracy gain (+0.180), followed by PBMC3k (+0.042) and Liver (+0.024). The sub- stantial improvement Retina is largely attributed to SCPILOT'S iterative reasoning process, which effectively differentiated major cell populations such rod photoreceptors, Müller glia, and bipolar cells by accessing the data and evaluating based on dotplot expression. Conversely, the one-step direct approach, limited to top marker genes, struggled with such detailed distinctions. Overall, SCPILOT
Table 6: Trajectory metrics across methods
Metric
ScPiloT Gemini-2,5 Pro
Biomni Gemini-2,5
py Momooel
Pro
Taccard (1)
GFD-nx (1)
3.33 22311
8.33 23.21
20
Spectral
(1)
2000
20.379
0.469
7

---

implementations using the o1 and Gemini-2. O-Pro models ranked highest (0.792 on PBMC3k, 0.728/0.763 on Retina, 0.518/0.509 on Liver).
Trajectory Inference. Table demonstrates SCPILOTT's superior performance compared to baseline methods Biomni and Monocle. Comprehensive evaluation metriis--including node overlap (Jaccard), graph-edit distance (GEDD-structure-awarr scores, and spectral distance-are summarized in Table 4 For the direct approach, the o1 model and Gemini-2 P-rro achieved the best performance, with o1 obtaining perfect Jaccard scores (1.000) and superior structural accuracy on Pancreas and Neocortex.
When adopting the SCPILOT pipeline, structural errors were further reduced in 10 of 21 model- metric pairs (median improvements: GED -2.0, spectral distance -0.14). Gemini-2.. 5-roo consistently delivered optimal results, closely followed by Gemini-2.0 Orro.
GRN TF-Gene Prediction. Average AUROC results for
GRN prediction across three tissues are summarized in Ta-
5 prroides baseline comparisons against two types of methods: graph neural network (GNN) models trained on GRN data and LLM-based tools (LLM4GRN, BioGPT). The SCPILOT pipeline consistently outperformed these baseline methods, except when utilizing smaller models (GPT-40-mini Gemini-2.0 0.. FFaash). Compared to direct prompting, scP1- LOT demonstrated an average AUROC improvement of +0.098.
Table 7: GRN prediction perfor- mance on Stomach dataset across methods. Note that BioGPT* refers to LoGGP--aarg-PubMeeeee
Method
AUROC (Stomach)
GCN
0.723 10071
Again, the o1 model under the SCPILOT pipeline achieved the highest overall accuracy (AUROC: 0.873 stomach, 0.760
GraphSAGE
0.713 +0003
GAT
0.683 10071
liver, 0.797 kidney), with Gemini-2 P-ro ranking second. GPT-4o exhibited the greatest relative improvement (+0.162 average AUROC), underscoring the effectiveness of iterative reasoning in harnessing latent regulatory insights.
LLM4GRN
0.727 +0025
BioGPT*
0.660
Direct (ol)
0.827 10002
SCPILOT (ol)
0.873 4:0004
Cross-Task Trends. Three consistent patterns emerged across tasks: (1) Superior results arise from combining large-scale
models (e.g.. o1, Gemini 2. 2/2.2. 5 Pro) with structured, iterative reasoming in SCPILOT. (2) Mini or latency-optimized variants frequently produced unreliable outputs, including over over-generation and hallucination, during extended reasoning chains. Overall, the results clearly indicate that iterative, reflective prompting significantly elevates state-of-the-art LLMS from competitive to decisively outperforming traditional bioinformatics methods in annotation, trajectory inference, and GRN prediction. (3) In rare cases, simpler direct prompting surpassed SCPILOT; these instances, though uncommon, offer valuable insights, discussed systematically in Appendix C.T
Challenges with Local Open-Source LLMs. While open-soures LLMs offer model transparency and data control advantages, our assessment of Gemma-3 3 for automated annotation tasks highlighted critical limitations. Performance evaluations revealed consistent inferiority to proprietary models such as GPT-4o and Gemini (Table 3. suggesting that significant domain-speciff specific fine-tuning is essential for accurate biological reasoming. Computational efficiency posed additional challenges: inference on the PBMC3k dataset required 135.7 seconds per evaluation using four NVIDIA A A100 (80 GB) GPUs, compared to only 8.8 seconds for GPT-4o-a more than 15-fold difference. The combination of high hardware demand, prolonged runtime, and limited predictive accuracy renders fully on-premise deployments financially and operationally impractical for most laboratories. Thus, SCPILOT employs API-based models as its backbone, while local open-source LLMs were not pursued further.
Efficiency and Cost Analysis. To evaluate the practical accessibility of SCPILOT, we conducted a detailed cost and efficiency analysis using Gemini-2 2- P-rro. As summarized in Table [3] executing the most complex tasks requires minimal financial outlay-mere cents per task high-iinngg the affordability and scalability of our framework. Token counts were approximated using tiktoken, with cost rates of $1.25 per million input tokens and $10 per million output tokens, without caching. Furthermore, compared to the general-purpose agent Biomni, SCPILOT achieves up to 30x lower cost and substantially faster performance due to its targeted reasoning and optimized toolchain (Table Importantly, SCPILOT consistently succeeds in complex tasks like GRN prediction, where Biomni often fails, underscoring SCPILOT's advantage in specialized biological reasoning. Overall, our analyses demonstrate that SCPILOT offers accessible and economically viable platform, significantly outperforming general-purpose methods in efficiency, cost, and reliability.
8

---

4.2 Ablation Studies
We performed three ablation experiments (Figure 3 Figure rigorously assess the contributions of contextual metadata, domain context-Gene Ontology (GO) knowledge, and trajectory priors to the overall accuracy and robustness of SCPILOT 's reasoning.
Contextual Metadata Ablation. We first investi- gated the impact of dataset-level contextual meta- data (such dataset size, tissue origin, and exper- imental conditions) on cell-type annotation accu- racy using the PBMC3K dataset. The full SCPILOT pipeline achieved strong baseline accuracy, with scores of 0.792 (o1), 0.646 (GPT-40), and 0.604 (4o-mini). Removing contextual metadata led to noticeable declines accuracy: 0.104 points (o1), 0.063 points (GPT-40), and 0.188 points Figure 3: Ablation on metadata and GO context.
(4o-mini). Despite these performance drops, all models continued to outperform traditional single- cell baselines, indicating intrinsic robustness of LLM reasoning even with incomplete metadata. These findings underscore two critical observations: (i) high-capacity models such as o1 significantly depend on contextual information for precise biological interpretation, and (ii) smaller models, no- tably 4o-mini, exhibit heightened sensitivity to absent metadata. Thus, comprehensive contextual metadata integration is crucial, particularly for smaller or mid- sized LLMs, to facilitate biologically coherent reasoning.
Gene Ontology Perturbation. Next, we assessed the significance of accurate Gene Ontology (GO) information by perturbing the GO database utilized in the GRN prediction module. Specifically, annotations were randomized to simulate erroncous overlaps. This perturbation resulted in substantial reductions in GRN prediction accuracy: in the Stomach dataset, AUROC decreased from 0.873 to 0.813 (o1), from 0.800 to 0.710 (GPT-4o), and from 0.697 to 0.617 HUPT-4o-minl) Although accuracy declined, all models maintained higher performance compared to
genuine peaucriptionffcctrr-eeeee
baselines, suggesting even perturbed GO data provides partial structural guidance. These results highlight that precise GO annotations between TFs and target genes are essential for robust GRN inference, with smaller models disproportionately affected by annotation inaccuracies.
Trajectory Input Perturbation. Finally, evaluated the role of trajectory priors by cor- rupting the py-Monocle input used in the liver dataset's 's trajectory inference task. Py-Monocle typically provides statistical insights into inter- cluster relationships and pseudotime hierarchy, crucial for informing accurate trajectory rea- soning. The corrupted Monocle inputs led to reduced reasoning accuracy for both o1 and GPT-4o-mini based SCPIL.OT models relative to unperturbed conditions. These findings con- firm the importance of accurate trajectory cues cues-specff cluster connectivity and pseudotime structure- in enhancing LLM-based biological reasoning. Consequently, robust integration with established computational tools like Monocle is essential for high interpretability and analytical performance.
Figure 4: Ablation on trajectory inference input.
4.3 Biological Interpretability and Insights
We further investigated how SCPIL.OT transforms benchmark accuracy into biologically interpretable insights across representative single-cell analysis tasks using the SCBENCH evaluation suite.
Multi-gene logic resolves marker ambiguity. In cell-type annotation tasks, SCPILOT leverages an iterative propose - filter- + solve reasoning loop, enabling systematic construction and validation of combinatorial marker hypotheses (Figure 5). Specifically, on the PBMC3k dataset, the 01 based SCPiLOT model initially proposed candidate marker sets (e.g., NK cells: NKG7, GNLY, GZMB; CD8 T cells: CD3D, CD8A), filtered out absent markers (e.g.. excluding plasma cells due to missing SDCI), and resolved marker expression ambiguity through dotplot reasoning. This meticulous process resulted in correct annotation for 7 out of 8 clusters. The model notably identified that NKG7 alone is insufficiently specific; however, combining it with CD3D and GNLY reliably distinguishes NK cells from cytotoxic T cells-at an important distinction often overlooked by single- marker methodologies,

---

particularly when CD8A exhibits weak expression. Further qualitative analysis and case studies are detailed in Appendix
Self-auditing via monocle diagnostics sharpens tra- jectory accuracy. For trajectory inference tasks, Gemini-2 Serinni-..5-Proe P-roo-aadd SCPILOT first constructed an initial lineage tree and subsequently self-audited the in- ferred structure utilizing py-Monocle diagnostic outputs, Through targeted refinements including correction of the tree root, restoration of canonical hepatic lineage se- quences, and hierarchical adjustments- SSPPLOT substan- tially improved trajectory accuracy, reducing the GED-nx nx metric by six edits and decreasing the Spectral Distance by 0.32. These improvements resulted in a trajectory struc- ture closely aligned with the developmental topology es- tablished in original biological studies. Comprehensive edge-level analyses are provided in Appendix D.2
8
Annotate this PBMC dataset
x
Top of clusters
LLM propose call types of Interess, with their gones:
Octx) 0319 RE431 CEM be
clanted (eeenitte catta BMc: ON 18A-ER 208 18921
(tryshrocstes 113 1:: 190: A.A52
Filtering Plasma Cell genes removed: they are not in data
Tissue-specific TF reasoning in GRN prediction. In GRN prediction, SCPILOT effectively employed tissue- specific retrieval module that filtered out spurious Gene Ontology (GO) overlaps, seamlessly integrating expres- sion context with known regulatory pathway informa- tion. This approach demonstrated biologically informed, context-sensitive reasoning. For instance, in predicting Statl's regulation Irf7, SCPILOT accurately captured the GO functional overlap. The prediction of Kif4 regulating MucSac illustrated the model's nuanced tissue-specific un- derstanding. More examples of successful and unsuccess- ful predictions are discussed thoroughly in Appendix D.3
analyze gene expression data
LLM finalize annotation:
In summay. SCPILOT not only achieved competitive ac- curacy (e.g., a score of 0.789 on the retina atlas) but also elucidated the mechanistic bases underlying its predictions. By clearly identifying ontology gaps, marker ambiguities, and rare-cell misclassifications, SCPILOT delivers diagnos- tic transparency, significantly advancing biological inter- pretability. This transparency not only enhances biological discovery but also informs the iterative development of future omics-native reasoning models.
Eval: correct,
1 Incorrect
Figure 5: Example of SCPILOT multi- gene reasoning PBMC3K annotation.
5 Conclusions
We introduced Omics-Natee Native Reasoning (ONR), novel LLM scientific reasoning paradigm wherein LLMs directly inspect raw single-cell data, invoke specialized analytic tools. and clearly articulate every biological inference in natural language. Operationalized through SCPILOT, ONR redefines critical analytical processes- cell-type annotation, trajectory inference, and gene regulatory network prediction- by transitioning them from opaque, black box methods to transparent and interpretable conversational workflows. Our newly established benchmark, SCBENCH, quantitatively demonstrates significant improvements: iterative reasoning via SCPILOT employing the o1 model boosts annotation accuracy by 11%, while using Gemini-2.. -2. 5-rro reduces trajectory graph-edit distance by 30% compared to traditional one-shot prompting approaches. By shifting analytical processes from implicit heuristics to explicit, data-informed reasoning, SCPILOT offers a robust, transparent, and continually improving foundation for single-cell biological discovery, paving the way for Al systems that actively reason alongside scientists as genuine scientific partners.
Limitations and Future Work. Despite progress, significant challenges persist. Current data compression may miss subtle signals from rare cell populations, requiring enhanced representation methods. Scaling ONR to billion-token contexts and integrating retrieval-augman -ugmented reasoning chains pose major scalability issues. Furthermore, ensuring trustworthiness requires robust methods to mitigate LLM hallucinations and incorrect claims. Finally, key future direction is integrating exper- imental wet-lab feedback to validate computational predictions in vitro, confirming the biological validity of ONR frameworks.
10

---

References
[1] 10x Genomics. 10x genomics 3k pbmes dataset. https://support. 10xgenomics com/
2017. Accessed: 2025-05-
13.
[2] Tejumade Afonja, Ivaxi Sheth, Ruta Binkyte, Waqar Hanif, Thomas Ulas, Matthias Becker, and Mario Fritz. Llm4grm: Discovering causal gene regulatory networks with llms evaluation through synthetic data generation, 2024.
[3] Google AI. Gemma 3.
https::/blog.gont http://www....oooooooooooo/////a
3/
2025.
[4] Sara Aibar, Carmen Bravo González-Blas, Thomas Moerman, Vân Anh Huynh-Thu, Hana Imrichova, Gert Hulselmans, Florian Rambow, Jean-Christophe Marine, Pierre Geurts, Jan Aerts, Joost van den Oord, Zeynep Kalender Atak, Jasper Wouters, and Stein Aerts. Scenic: single-cell regulatory network inference and clustering. Nature Methods, 14(11):1033-1066 2017.
[5] Aimée Bastidas-Ponce Sophie Tritschler, Leander Dony, Katharina Scheibner, Marta Tarquis- Medina, Ciro Salinno, Silvia Schirge, Ingo Burtscher, Anika Böttcher, Fabian J. Theis, Heiko Lickert, Mostafa Bakhti, Allon Klein, and Barbara Treutlein. Comprehensive single cell mrna profiling reveals a detailed roadmap for pancreatic endocrinogenesis. Development, 146(12)xdev 17849,, 06 2019.
[6] Haiyang Bian, Yixin Chen, Xiaomin Dong, Chen Li, Minsheng Hao, Sijie Chen, Jinyi Hu, Maosong Sun, Lei Wei, and Xuegong Zhang. scmulan: multitask generative pre-trained lan- guage model for single-cell analysis. International Conference on Research Computational Molecular Biology, pages 479-482. Springer, 2024.
[7] Rebecca Boiarsky, Nalini M Singh, Alejandro Buendia, Ava P Amini, Gad Getz, and David Sontag. Deeper evaluation of a single-cell foundation model. Nature Machine Intelligence, 6(12):1443-1446, 2024.
[8] Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, et al. BiomedIm: A 2.7 parameter language model trained biomedical text. arXiv preprint 2ii::2403.184211 2024.
[9] Junyue Cao, Malte Spielmann, Xiaojie Qiu, Xingfan Huang, Daniel M Ibrahim, Andrew J Hill, Fan Zhang, Stefan Mundlos, Lena Christiansen, Frank Steemers, et al. The single-cell transcriptional landscape of mammalian organogenesis. Nature, 566(7745):46-500,,, 2019..
[10] Junyue Cao, Malte Spielmann, Xiaojie Qiu, Xingfan Huang, David M Ibrahim, Aaron Hill, Fan Zhang, Stefan Mundlos, Lars Christiansen, Frank J Steemers, et al. The single-cell transcriptional landscape of mammalian organogenesis. Nature, 566(7755))49665502 2019.
[11] Yiqun Chen and James Zou. Genept: simple but effective foundation model for genes and cells built from chatgpt. bioRxiv, pages 2023-10, 2024.
[12] Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, al. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. arXiv preprint axXiv224100055080,2222 2024.
[13] Google
Cloud. Gemini Models.
http::///lloo.googgll..co//eeere ai/
wwtecess/ooelss/eeeo httddd/s///odellggmmiiiiiia gemini/ 2025.
[14] The Tabula Sapiens Consortium*, Robert Jones, Jim Karkanias, Mark A Krasnow, An- gela Oliveira Pisco, Stephen R Quake, Julia Salzman, Nir Yosef, Bryan Bulthaup, Phillip Brown, al. The tabula sapiens: A multiple-organ, single-cell transcriptomic atlas of humans. Science,
37666594)cabl4896, 2022.
[15] Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang. segpt: toward building foundation model for single-cell multi-omies using generative ai. Nature Methods, 2178):1470-1488 2024.

---

[16] C *Dominguez Conde. Chao Xu, Louie B Jarvis, Daniel B Rainbow, Sara B Wells, Tamir Gomes, SK Howlett, O Suchanek, K Polanski, HW King, et al. Cross-tissue immune cell analysis reveals sissue- features in humans. Science, 376(65944 2022.
[17] Xiaoru Dong, Jack Leary, Chuanhao Yang, Maigan A Brusko, Todd M Brusko, and Rhonda Bacher. Data-driver selection of analysis decisions in single-cell ma-seq trajectory inference. Briefings in Bioinformatics, 25(3), 2024.
[18] Li Fang, Yunjin Li, Lu Ma, Qiyue Xu, Fei Tan, and Geng Chen. Grndb: decoding the gene regulatory networks in diverse human and mouse conditions. Nucleic Acids Research, 49(D1):D9--D103, 11 2020.
[19] Shanghua Gao, Ada Fang, Yepeng Huang, Valentina Giunchiglia, Ayush Noori, Jonathan Richard Schwarz, Yasha Ektefaic, Jovana Kondic, and Marinka Zitnik. Empow- ering biomedical discovery with ai agents. Cell, 125--111,1 2024.
[20] Vincent Gardeux, Fabrice PA David, Adrian Shajkofei, Petra C Schwalie, and Bart Deplancke. Asap: web-based platform for the analysis and interactive visualization of single-cell ma-seg data. Bioinformatics, 33(19):3123-3125 2017.
[21] Jing Gong, Minsheng Hao, Xingyi Cheng, Xin Zeng, Chiming Liu, Jianzhu Ma, Xuegong Zhang, Taifeng Wang, and Le Song. xtrimogene: an efficient and scalable representation learner for single-cell ra-seq data. Advances in Neural Information Processing Systems, 36:69391-69903, 2023.
[22] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arX1v:2501.12948, 2025.
[23] William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, volume 30, pages 1024-1034, 2017.
[24] Heonjong Han, Jae- Won Cho, Sangyoung Lee, Ayoung Yun, Hyojin Kim, Dasom Bae, Sunmo Yang, Chan Yeong Kim, Muyoung Lee, Eunbeen Kim, Sungho Lee, Byunghee Kang, Dabin Jeong, Yaeji Kim, Hyeon-Nae Jeon, Haein Jung, Sunhwee Nam, Michael Chung, Jong-Hoon Kim, and Insuk Lee. Trrust v2: an expanded reference database of human and mouse transcrip- tional regulatory interactions. Nucleic Acids Research, A6(DI):D300-D33) 10 2017.
[25] Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo, Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Xuegong Zhang, and Le Song. Large-scale foundation model on single-cell transcriptomics. Nature methods, 31(8):1481-18
2024.
[26] Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhit- ing Hu. Reasoning with language model is planning with world model. arXiv preprint 149. 2023.
[27] Shibo Hao, Tianyang Liu, Zhen Wang. and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Advances in neural information processing systems, 36:45870-45894 2023.
[28] Wenpin Hou and Zhicheng Ji. Assessing gpt-4 for cell type annotation in single-cell rna-seq analysis. Nature methods, 21 21(8):1462-1455 2024.
[29] Congxue Hu, Tengyue Li, Yingqi Xu, Xinxin Zhang, Feng Li, Jing Bai, Jing Chen, Wenqi Jiang, Kaiyue Yang, Qi Ou, Xia Li, Peng Wang, and Yunpeng Zhang. Cellmarker 2.0: an updated database of manually curated cell markers in human/mouse and web tools based on scrna-seq data. Nucleic Acids Research, D1700) January 2023.
[30] Mengzhou Hu, Sahar Alkhairy, Ingoo Lee, Rudolf T Pillich, Dylan Fong, Kevin Smith, Robin Bachelder, Trey Ideker, and Dexter Pratt. Evaluation of large language models for discovery of gene set function. Nature methods, 22(1):22-11, 2025.
12

---

[31] Kexin Huang, Serena Zhang. Hanchen Wang, Yuanhao Qu, Yingzhou Lu, Yusuf Roohani, Ryan Li, Lin Qiu, Gavin Li, Junze Zhang, Di Yin, Shruti Marwaha, Jennefer N. Carter, Xin Zhou. Matthew Wheeler, Jonathan A. Bernstein, Mengdi Wang, Peng He, Jingtian Zhou, Michael Snyder, Le Cong. Aviv Regev, and Jure Leskovec. Biomni: A general-purposs biomedical ai agent. bioRxiv, 2025.
[32] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Bl-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.1672)) 2024.
[33] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences. 11(14):6421, 2021.
[34] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint
2019.
[35] Kasia Z Kedzierska, Lorin Crawford, Ava Amini, and Alex X Lu. Zero-shot evaluation reveals limitations of single-cell foundation models. Genome Biology, 26(1): 2025.
[36] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (JCLR), 2017.
[37] JungHo Kong, Ingoo Lee, Dean Boecher, Akshat Singhal, Marcus Kelly, Jimin Moon, Chang Ho Ahn, Chan- Young Ock, Tannavee Kumar, Timothy John Sears, al. Translating clinical gene sequencing into a foundational representation of tumor subtype. bioRxiv, pages 2025-09, 2025.
[38] Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. Bio A manually curated corpus for biomedical question answering. Scientific Data, 10(1):170, 2023.
[39] David Lähnemann, Johannes Köster, Ewa Szezurek, Davis J McCarthy, Stephanie C Hicks, Mark D Robinson, Catalina A Vallejos, Kieran R Camphell, Niko Beerenwinkel, Ahmed Mahfouz, et al. Eleven grand challenges in single-cell data science. Genome biology, 21(1):31, 2020.
[40] Jon M Laurent, Joseph D Janizek, Michael Ruzo, Michaela M Hinks, Michael J Hammer- ling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D White, and Samuel G Rodriques. Lab-bench: Measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10366 2024.
[41] Daniel Levine, Syed Asad Rizvi, Sacha Lévy, Nazreen Pallikkavaliyaveetii, David Zhang, Xingyu Chen, Sina Ghadermarzi. Ruiming Wu, Zihe Zheng, Ivan Vrkic, al. Cell2sentence: teaching large language models the language of biology. BioRxiv, pages 2023-09, 2024.
[42] Cong Li, Qingqing Long, Yuanchun Zhou, and Meng Xiao. screader: Prompting large language models to interpret serna-seq data. arXiv preprint arXiv:2412.18155 2024.
[43] Yan Liang, Kota Kaneko, Bing Xin, Jin Lee, Xin Sun, Kun Zhang, and Gen-Sheng Feng. Temporal analyses of postnatal liver development and maturation by single-cell transcriptomics. Developmental Cell, 57(3):398 414.e5. 2026/01/29 2022.
[44] Tianyu Liu, Kexing Li, Yuge Wang, Hongyu Li, and Hongyu Zhao. Evaluating the utilities of large language models in single-cell data analysis. biorxiv. preprint, 8:2023, 2023.
[45] Jeremy Lotto, Sibyl Drissler, Rebecca Cullum, Wei Wei, Manu Setty, Erin M. Bell, Stéphane C. Boutet, Sonja Nowotschin, Ying- Yi Kuo, Vidur Garg. Dana Pe'er, er, Deanna M. Church, Anna- Katerina Hadjantonakis, and Pamela A. Hoodless. Single-cell transcriptomics reveals early emergence of liver parenchymal and non-parenchymal cell lineages. Cell, (830))702-716.14 2026/01/29 2020.
[46] Yen-Chun Lu, Ashley Varghese, Rahul Nahar, Hao Chen, Kunming Shao, Xiaoping Bao, and Can Li. sechat: A large language model-powered co-pilot for contextualized single-cell ma sequencing analysis. bioRxiv, pages 2024-10, 2024.
13

---

[47] Malte D Luecken, Maren Büttner, Kridsadakor Chaichoompu, Anna Danese, Marta Interlandi, Michaela F Müller, Daniel C Strobl. Luke Zappia. Martin Dugas, Maria Colomé-Tatché, et al. Benchmarking atlas-level data integration in single-cell genomies. Nature methods, 19(1):41-50, 2022.
[48] Renqian Luo, Liai Sun, Yingee Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie- Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings bioinformatics, 23(6):bbac409 2022.
[49] Renqian Luo, Liai Sun, Yingee Xia, Tao Qin. Sheng Zhang, Hoifung Poon, and Tie- Ran Liu. BioGPT: generative pre-trained transformer for biomedical text generation and mining, Briefings in Bioinformaties, 23(6), 09 2022. bbac409.
[50] Manikandan Menon, Saad Mohammadi, Jose Davila-Velderrail Velderrain, Brian A Goods, Timothy D Cadwell, Yujia Xing, Anat Stemmer-Rachaminn Alex K Shalek, J Christopher Love, Manolis Kellis, and David A Hafler. Single-cell transcriptomic atlas of the human retina identifies cell types associated with age-related macular degeneration. Nature Communications, 10( 1091), 2019.
[51] Ludovico Mitchener, Jon M Laurent, Benjamin Tenmann, Siddharth Narayanan, Geemi Wellawatte, Andrew White, Lorenzo Sani, and Samuel G Rodriques. Bixbench: a comprehensive benchmark for llm-based agents in computational biology. arXiv preprint arXiv:2503.00096 2025.
[52] OpenAl. OpenAl Models.
appenai.com/ddocs//mddeeee 2024.
[53] Dimitrios Polioudakis, Luis de la Torre-Ubietaa, Jonah Langerman, Aaron Elkins, Xiaoyu Shi, Jason L Stein, Casey K Vuong, Sehastian Nichterwitz, Margarita Gevorgian, Christopher K Opland, Derek Lu, William Connell, Elizabeth K Ruzzo, Jennifer K Lowe, Tanja Hadzic, Fiona I Hinz, Sahar Sabri, William Lowry, Mark Gerstein, Kathrin Plath, and Daniel H Geschwind. A single-cell transcriptomic atlas of human neocortical development during mid-gestation. Neuron, 103(5):785-001 2019.
[54] Fei Quan, Xin Liang, Mingjiang Cheng, Huan Yang, Kun Liu, Shengyuan He, Shangqin Sun, Menglan Deng, Yanzhen He, Wei Liu, et al. Annotation of cell types (act): convenient web server for cell type annotation. Genome Medicine, 15(1):91, 2023.
[55] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
[56] Syed Asad Rizvi, Daniel Levine, Aakash Patel, Shiyang Zhang, Eric Wang. Sizhuang He, David Zhang, Cerise Tang, Zhuoyang Lyu, Rayyan Darji, et al. Scaling large language models for next-generation single-cell analysis. bioRxiv, pages 2025 04, 2025.
[57] Yusuf H Roohani. Andrew H. Lee, Qian Huang, Jian Vora. Zachary Steinhart, Kexin Huang, Alexander Marson, Percy Liang, and Jure Leskovec. Biodiscoveryagent An Al agent for designing genetic perturbation experiments. In The Thirteenth International Conference Learning Representations, 2025.
[58] Yanay Rosen, Maria Brbić, Yusuf Roohani, Kyle Swanson, Ziang Li, and Jure Leskovee. Toward universal cell embeddings: integrating single-cell rna-seq datasets across species with saturn. Nature Methods, 21 2188) 2024.
[59] Yanay Rosen, Yusuf Roohani, Ayush Agarwal, Leon Samotorčan, Tabula Sapiens Consortium, Stephen R Quake, and Jure Leskovec. Universal cell embeddings: A foundation model for cell biology. bioRxiv, pages 2023-11, 2023.
[60] Wouter Saelens, Robrecht Cannoodt, Helena Todorov, and Yvan Saeys. A comparison of single-cell trajectory inference methods. Nature biotechnology, 37(5):547-55,, 2019.
14

---

[61] Moritz Schaefer, Peter Peneder, Daniel Malzl, Mihacla Peycheva, Jake Burton, Anna Hakobyan, Varun Sharma, Thomas Krausgruber, Joerg Menche, Eleni M Tomazou, al. Multimodal learning of transcriptomes and text enables interactive single-cell rna-seq data exploration with natural-language chats. bioRxiv, pages 2024-10, 2024.
[62] Tim Stuart, Andrew Butler, Paul Hoffman, Christoph Hafemeister, Efthymia Papalexi, William M Mauck, Yuhan Hao, Marlon Stoeckius, Peter Smibert, and Rahul Satjja. Com- prehensive integration of single-cell data. cell, (117)))18888199222 2019.
[63] Valentine Svensson, Roser Vento- and Sarah A Teichmann. Exponential scaling of single-cell rna-seq in the past decade. Nature protocols, 13(():599-60)) 2018.
[64] Kyle Swanson, Wesley Wu, Nash L Bulaong, John E Pak, and James Zou. The virtual lab: Ai agents design new sars-cov-2 nanobodies with experimental validation. bioRxiv, pages 2024-11, 2024.
[65] Artur Szałata, Karin Hrovatin, Sören Becker, Alejandro Tejada-Lapuerta, Haotian Cui, Bo Wang. and Fabian J Theis. Transformers in single-cell omics: review and new perspectives. Nature methods, 21 (1) 1443, 2024.
[66] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:221 1.0900,, 2022.
[67] Christina V Theodoris, Ling Xiao, Anant Chopra, Mark D Chaffin, Zeina R Slyed, Matthew Hill, Helene Mantineo. Elizabeth M Brydon, Zexian Zeng, X Shirley Liu, et al. Transfer learning enables predictions in network biology. Nature, 618(7965):616-624 2023.
[68] Koen Van den Berge, Hector Roux de Bézieux, Kelly Street, Wouter Saelens, Robrecht Can- noodt, Yvan Saeys, Sandrine Dudoit, and Lieven Clement. Trajectory-basse differential expression analysis for single-cell sequencing data. Nature communications, 11(1):1201, 2020.
[69] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR), 2018
[70] Hanchen Wang, Jure Leskovec, and Aviv Regev. Metric mirages in cell embeddings. BioRxiv, pages 2024-04, 2024.
[71] Yuan Wang, William Thistlethwaite, Alicja Tadych, Frederique Ruf-Zamojski, Daniel Bernard, Antonio Cappuccio. Elena Zaslavsky, Xi Chen, Stuart C Scalfon, and Olga G Troyanskaya. Automated single-cell omics end-to-end framework with data-driven batch inference. Cell Systems, 15(10)9822999))) 2024.
[72] Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Weiqi Liu, Albert Chen, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Eric P. Xing, and Zhiting Hu. FIRE-bench: Evaluating research agents on the rediscovery of scientific insights. 2025.
[73] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. of prompting elicits reasoning in large language models. Advances neural information processing systems, 35:24824 -24377, 2022.
[74] F Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology, 19:1-5, 2018
[75] Yihang Xiao, Jinyi Liu, Yan Zheng, Xiaohan Xie, Jianye Hao, Mingzhi Li, Ruitao Wang, Fei Ni, Yuxiao Li, Jintian Luo, et al. Cellagent: An Ilm-driven multi-agent framework for automated single-cell data analysis. BioRxiv, pages 2024-00 05, 2024.
[76] Chuyao Xu, Marco Prete, Simon Webb, Lauren Jardine, Benjamin J Stewart, Renjie Hoo, Peng He, Kathryn B Meyer, and Sarah A Teichmann. Automatic cell-type harmonization and integration across human cell atlas datasets. Cell, 186(6))55876-5881 .e20, December 2023.
[77] Fan Yang, Wenchuan Wang, Fang Wang, Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scbert as large-scale pretrained deep language model for cell type annotation of single-cell ma-seq data, Nature Machine Intelligence, 4(10):852-80 866, 2022.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the abstract and introduction of this paper faithfully and accurately reflect the paper's ' contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, we thoroughly discuss the limitations and potential future improvements in the supplementary.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Yes, disclose all details to reproduce the main experiments across both the main text and the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Yes, we open-source and upload our code and data in the supplementary.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wwtpcccpppppppppeeeeeeeeeeeeeeee httae//ggiieee/eeeeebbsssssssss /Cc////oddiiisssseeeeeee ssiiiisiiiyiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Yes, we specify all experiment details across the main text and the supplemen- tary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, we report the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we report the running cost in our supplementary to reproduce our experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, strictly follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we assign proper credits to all assets used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification: Yes, we thoroughly document our data and code the supplementary. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: Yes, our framework is built on top of the reasoning capability of LLMs as reasoning engine for the framework.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standart components.
Please refer to our LLM policy (https: //euriip. wwwpuraippp..oooeeeeeeeeeeee cc/Confeerences/20000 for what should or should be described.
22

---

A Data and Code Availability
All of the raw and processed single-cell RNA-seq datasets used SCPILOT are publicly sourced collections. The Cell type annotation datasets are detailed in Table 9 the Trajectory inference collections Table 10 and the GRN TF-gene prediction cohorts in Table For each dataset, we release the processed objects (HSAD format for scRNA datasets, and csv format for GRN data), along with cell-type labels, trajectory gold-standards, and GRN reference networks.
The complete source code for dataset preprocessing, automatic graders. evaluation metrics, and benchmark drivers is released under the MIT license and available our SCPILOT github https //github. www.mmoiitiii ottpiiiooooo//oooooooooooo
B
Additional Details of SCPILOT and SCBENCH
B.1
Model zoo
We specify the exact versions of the 7 proprietary models and open-source model in the scPiL.T. This model zoo spans both large-scale multimodal systems and lightweight, inference-optimized variants, ensuring comprehensive evaluation of performance trade-off's single-cell analysis tasks.
Table 8: Large Language Model Zoo: Current State-of-the-Arr Models
Model Name
Version/ID
Description
OpenAl Models
GPT-4o
go 202-400-220------
Omni model with multimodal capabilities Lightweight variant of GPT-4o Advanced reasoning model Efficient version of
GPT-4o-mini
1g0-44-minn--222--77-1
01
01-2024-12-17
O1-mini
o1-mini-2024-09-1
Google DeepMind Models
Gemini 2.0 Pro
gemini-2 1atpiii-220----eeepp- gemini-2 2emini-2.0-flashh
Experimental professional model Optimized for fast inference Latest pro model iteration
Gemini 2.0 Flash
Gemini 2.5 Pro
-et.---.--p---p------- 03-25
Google Open Models
Gemma 27B
parameter instruction- tuned model
B.2 Dataset description
We elaborate the details of the dataset in SCBENCH.
Cell Type Annotation. We selected datasets, and their details are in in The "# Cell types" column refers to the number of ground truth cell types. The "Celltypist Model" column refers to the Celltypist model we used to evaluate Celltypist baseline performance on this dataset.
Table Summary of Cell Type Annotation Datasets
Dataset
Tissue
Size
Cell Types
Cell'yyiist Model
Source
Liver
Mouse liver
41,000 cells x 2,000 genes (HVGs)
31
Healthy_Mouse, Mouueouuu_uaae Liver
43
PBMC3K
Human PBMC
2,638 cells x 13.714 genes 20,091 cells x 19,719 genes
8
Inmune _emune_All_Lov
Retina
Human retina
9
Retina
[S0]
Trajectory Inference. We selected 3 datasets, and their details are in 10 The "# Timepoints" column represents the innate developmental stage design in the single-cell sequencing experiment. For example, in the Pancreas dataset, there are 4 timepoints, from E12.5 to E15.5, corresponding to Day 12.5 to Day 15.5 in the embryo. The Trajectory Nodes" represents the number of cell type, and thus the number of nodes the trajectory tree.
GRN Te-gene Prediction. From the GRNdb database, we selected tissues, and their details are in For each tissue, compared its SCENIC-generatee TF-gene pairs with the CHiP-Seg verified
11
23

---

Table 10: Summary of Trajectory Datasets
Dataset
Tissue
Size
Timepoints
Trajectory Nodes
Source
Pancreas
Mouse pancreas
36,351 cells x 17.327 genes 44,010 cells x 2,000 genes (HVGs)
4
14
5
Liver
Mouse liver
15
45
Neocortex
Human neocortex
33,976 cells x 35,543 genes
2
16
S3
database TRRUST. If this pair exists in TRRUST, we recognize it as positive edge. Then, randomly sample amother gene that does not result verified or SCENIC-generated TF-gene edge. So have half questions positive (answer is yes, TF-gene relationship exists) and half as negative (answer is no, there no regulation).
B.3
Evaluation Metrics
B.3.1 Cell Type Annotation
The key to calculating the Cluster-level accuracy is similarity based on the GO database.
Cleaning and Standardizing Cell
Table 11: GRN task summary by held-out context
Type Names. Raw cell type names often exhibit inconsistencies such as mixed casing, redundant suffixes, ambiguous abbreviations. To address
Dataset name
Verified TF gene Edge
Total Questions
Unique TFs
Stomach
23
46
10
Liver
142
Kidney
49
98
21
this, cleaning and standardization process was applied:
Plural forms (e.g., "cells") were converted to singular (e.g.. "cell"), redundant whitespace and punctuation were removed, and biologically meaningful symbols (e.g., slashes "/")
Automatic Cleaning:
were retained.
Standardized Mapping: Cleaned names were first matched against predefined dictionary of known cell type nomenclature. For names not found in the dictionary, a language model was used to generate standardized mappings dynamically, ensuring coverage of uncommon or ambiguous terms.
This process harmonized all cell type names, enabling consistent downstream analysis.
Mapping to Cell Ontology Identifiers. Standardized cell type names were mapped to terms the Cell Ontology (CL) to integrate annotations with structured biological knowledge:
Ontology Querying: Names queried against the Cell Ontology using automated search tool, retrieving corresponding identifiers (CLIDs) and high-level categories.
Handling Unmapped Names: Names that could not be mapped directly to the ontology were retained for further review or additional processing.
This mapping ensured systematic alignment between predicted and reference annotations.
Construction of the Ontology Tree. The hierarchical relationships within the Cell Ontology were leveraged to evaluate lineage-based relationships between predicted and reference annotations:
Ontology Hierarchy Parsing: The Cell Ontology downloaded in OWL format, and parent- child relationships between ontology terms were extracted.
Graph Construction: A directed acyclic graph (DAG) was built, where nodes represented CLIDs, and edges denoted parent-child relationships.
This hierarchical structure enabled evaluation beyond direct matches, incorporating extended lineage- based relationships.
Ontology-Basee Scoring Framework. Inspired by the ontology- scoring methodology in GPTcelltype, a scoring framework was developed incorporate both exact and hierarchical matches:
Exact Match: score of 1.0 was assigned if the predicted CLID(s) exactly matched the reference CLID(s).

---

Partial Match: A score of 0.5 was assigned if the predicted CLID(s) overlapped with the reference Cs)D(s) or their relatives (parent or child terms) the ontology hierarchy.
No Match: A score of 0.0 was assigned no overlap was observed.
This scoring framework ensured biologically meaningful comparisons while accounting for the hierarchical structure of the Cell Ontology.
Biological Context Validation. To ensure the biological relevance of predictions, an additional layer of validation was performed:
Using the ontology graph, predictions were cross-checked against all known relatives of the reference CLIDs, including parents and children.
Relative Identification:
Broad Type Consistency: Predictions were compared at higher categorization levels (e.g., "im- mune cells," "stromal cells") to ensure consistency with the broader biological context.
Summary. The evaluation methodology integrates systematic name cleaning, dynamic mapping, and ontology-aware scoring to ensure biologically accurate assessments. By leveraging both direct matches and hierarchical relationships within the Cell Ontology, the framework provides a robust and biologically meaningful evaluation of cell type annotations.
B.3.2 Trajectory Inference
To evaluate the predicted trajectory tree against the ground truth, we employ three distinct metrics capturing structural and spectral similarities:
Jaccard Similarity (Nodes). We quantify the structural similarity between predicted and ground truth trees at the node level using the Jaccard similarity coefficient. For two trajectory graphs with node sets Freed and Vit. representing the predicted and ground truth trees, respectively, the Jaccard similarity is defined as:
Vpreed O Vigl
J(preed pred, Vgl)
preed
Vgdl
where | denotes set cardinality. This metric ranges from 0 with higher values indicating greater overlap between the node sets of the two trees.
Graph Edit Distance (GED). Graph Edit Distance (GED) quantifies the structural dissimilarity between two graphs by measuring the minimum number of edit operations required to transform one graph into another. For predicted and ground truth graphs Gpred pred: Eprea) and G = (Vgs Eg). the GED is formally defined as:
GED(Gpred Ggt)= =
mii Σ c(e) GGed, CET
(8)
where T(Gpood Gg) denotes the set of all possible edit paths transforming Gpeed into Gg. and c(e) represents the cost of edit operation e (node/edge insertion, deletion, or substitution). For computational tractability, we impose a timeout constraint of 10 seconds. Lower GED values indicate greater structural similarity between the graphs
Spectral Distance (Euclidean). Spectral distance captures global structural properties of graphs by comparing the eigenvalue distributions of their normalized Laplacian matrices. For graphs Greed and Gg. let Epeed and Eu denote their respective normalized Laplacian matrices. The spectral distance is defined as the Euclidean distance between their ordered eigenvalue spectra:
dspocerall (Gprod, Gg) = |AApred Agella
(9)
where Apred pred ..... Xpred and Ag. = (A ..... ) are the eigenvalues of Eped and Leg arranged ascending order. Lower spectral distances indicate greater similarity the global topological structure of the graphs.
25

---

B.3.3 GRN -e-eene Prediction
We evaluate the prediction performance of Gene Regulatory Network (GRN) transcription factor (TF)-gene interactions using standard binary classification metries:
Area Under the ROC Curve (AUROC). The AUROC quantifies the classifier's discriminative ability across all possible decision thresholds. For binary classifier with varying threshold T, the AUROC is computed as:
AUROC L. (P)(() dFPR(()
(10)
where the True Positive Rate (sensitivity) and False Positive Rate (1 (I-specificity) are defined as:
TP(T) TPR(T) TP(T) FN(T)' FPR(T) FP(T) TN(T)
FP(T)
(11)
Confusion Matrix. The confusion matrix provides comprehensive breakdown of prediction outcomes for given threshold, capturing the counts of true negatives (TN), false positives (FP), false negatives (FN), and true positives (TP):
TN FP] - 0 FN
| 1= 0 = 10-1,y-1
C =
(12)
TP
where and i denote the ground truth and predicted labels, respectively.
C Additional Results
C.1 Occasional suboptimal performance
Occasionally, simpler 'Direct' prompt can outperform SCPILOT. These cases are rare, systematic, and informative. SCPILOT overwhelmingly outperforms the baseline (wins in 87 of 108 total comparisons). The rare losses are systematic: 13 of 21 (62%) are from less-capable "mini" models. Their limited capacity for sustained logic leads them "over-explore" and make mistakes with SCPiLOT's extended reasoning.
For the few remaining cases with powerful models, the cause is high dataset nuances, inducing "overthinking." The only powerful-mone losses in cell-type annota- tion occurred on our most complex dataset, Liver. The characteristics of datasets explain this pattern:
Table 12: Runtime of various LLMs.
LIM
Runtime
Std. Dev.
GPT-4o
3.9701
0.1045
GPT-4o-mini
3.7513
0.1673
o1
11,4210
0,4140
PBMC3k contains clusters and 8 cell types. ple dataset characterized by clear 1:1 mapping between clusters and cell types, with distinct marker genes. scP1- LOT performs well on this dataset, although mini models may already saturate its performance ceiling.
sim-
o1-mini
4.0072
0.1827
Gemini-2.0-pro
30.6115
0,4351
Gemini- 1etini-2.0-fit 2.0-flash
11.0586
0,1445
Gemini-2.5-p 2.5-pr
29.7689
0,2197
Retina consists of 18 clusters and 9 cell types. It has moderately clear cell types with slight ambiguity. On this dataset, SCPILOT consistently improves accuracy over simpler baselines.
Liver includes 28 clusters and 31 cell types. This dataset is highly complex, with overlapping developmental lineages and noisy expression patterns. On such data, SCP11.TT can sometimes "overthink" and merge distinct subtypes, reducing performance.
In the complex Liver dataset, SCPILOT using the powerful o1 model achieved a score of 0.518, while the simpler Direct method scored 0.560. A key error involved confusing developmentally related hepatocytes and hepatoblasts. Because they share many markers, SCPILOT's deep reasoning amplified this ambiguity and wrongly merged them, whereas the Direct method incidentally correct by ignoring this nuance.
This analysis delineates the boundaries for applying LLMs complex biological data and will be incorporated into the manuscript. Future improvements include adaptive reasoning depth and marker clarity assessments.
26

---

C.2
Time Cost Analysis
In Table 12, we recorded average time cost of different LLM API calls, during the TF-gene prediction task. The OpenAl family consistently returns results in 3.8 4.0s except o1, whereas the full-scale Gemini-2. O-pro and Gemini- 2-pprr take about 30s per run-way slower than OpenAl models. Even Gemini "flash-thnnking" (=11s) still nearly three times slower than OpenAl's baseline models. o1 reasoning took 3x extra time compared with its mini variant.
This significant runtime gap indicated clear trade-offs among reasoning depth, inference la- tency, and model scale. Gemini's 's substantial de- lay likely arose from Google Al's more complex internal processing
Annotate this PBMC dataset
|x|
Top gane of clusters using scanpy
longer input handling, as even its optimized "flahh-tinkknng variant failed to match OpenAl's baseline speed. Mean- while, i'ss roughly threefold increase in latency compared its mini variant suggested that de- tailed step-by-step reasoning incurs notable over- head, potentially due to longer context handling and more elaborate token-by-toker generation. Thus, selecting appropriate LLMs for biological tasks involved balancing performance demands: faster models like GPT-4o or mini variants for efficiency, versus slower, larger models when sophisticated reasoning outweighs runtime con- siderations. While Gemini was substantially slower in its Al infrastructure.
LLM propose cell types of interess, with marker gernes:
ragie p-----
Problem reasoning: too many cell types
proposed! but missing Monocyte subtypes...
Filtering with data
analyze gene expression data
100% 1O07T, YIMT).
C.3 Accessibility
and Financial Cost Analysis
annotation:
Our novel omics-native reasoning (ONR) paradigm is inherently more demanding than standard text analysis and requires powerful LLMs, point confirmed by our experiments on Gemma 27B (the best open-source model testable on 8xH100s at the time). Our experi- ments reveal two reasons why ONR is challeng- ing for current open-source models:
Eval: correct, partial, 2 Incommect
Confusion NK/ CDB Overwhelmeed Cluster 5
Figure 6: SCPILOT GPT-4o-mini reasoning in PBMC3K annotation, with problem noted
Insufficient Domain Knowledge: Weaker models lack the nuanced, pre-trained understanding of biology, such as raw markers and pathways, to interpret omics data correctly.
Poor Instruction Following: Weaker models often fail to follow complex instructions and adhere biologist-defined formats (e.g., breaking the reasoning chain.
These insights are a key contribution, offering guidance for future model development. How- ever, the cost of using powerful models via SCPILOT is negligible. Our detailed cost analysis for Gemini-2.5 2.rrr shows that a complete run for our most complex tasks costs only few cents, making the framework accessible 13] Token counts were approximated with riktoken. Cost for Gemini-2.5-PP is 1.25 / 1M input, 10 / 1M output; caching was used in the cost analysis. The negligible cost allows any lab to leverage state-of-the-an art Al without expensive local GPUs. Developing cheaper, specialized open-source models for ONR is promising future direction; in parallel, our framework provides an immediate, accessible tool for the community.
C.4 Additional ablation studies
C.4.1 Sensitivity analysis Cell type annotation marker gene selection
For cell-type annotation, key hyperparameter K, the number of top marker genes. We tested performance sensitivity on the PBMC3k dataset for K 5, 10 (our default), and 20 in table [4
27

---

Task (single run)
In token
Out token
In $
Out
Total
Cell-type annotation (retina)
6,155
2,197
0.008
0.022
0.03
Trajectory inference (neocortex)
8,221 17,877
2,702
0.010
0,027
0.04
GRN Te-ge (stomach, 46 pairs)
9,276
0.022
0.093
0.12
Table 13: Cost analysis of SCPILOT on all three tasks
Performance peaks at our chosen K = 10. More importantly, the strong results across different values demonstrate that approach is robust and not highly sensitive this hyperparameter.
Table 14: Annotation Accuracy (PBMC3K) vs. Number of Top Genes (K)
K
Model
Avg Accuracy
Variance
5
o1 GPT-4o-mini
0.771
0,001
5
0.500
0.016
10 (Default)
o1
0.792
0.005
10 (Default)
GPT-4o-mini
0.604
0.001
20
o1 0.750
0.016
20
GPT-4o-mini
0.542
0.009
C.4.2 Validation of Intermediate Tool Calls
To validate that SCPILOT's reasoning is grounded in its tool outputs, we performed perturbation studies on two critical components. We will add these results to the manuscript.
1. Perturbation of Gene Ontology (GO) Database in GRN Prediction the GO database for Gene Regulatory Network (GRN) prediction, we shuffled its term associa- tions with random noise Corrupting GO data significantly degrades AUROC (e.g., p 0.044 for GPT-4o-mini), confirming that SCPILOT's reasoning relies on accurate information from this intermediate step.
To test dependency
Table 15: Impact of GO Perturbation GRN Prediction (AUROC)
Model
AUROC (Original)
AUROC (GO Shuffled)
A AUROC
o1 gpt-4o
0.873
0.813
-0.060
0.800
0.710
-0.090
GPT-4o-min
0.697
0.617
-0.080
Perturbation My-Moneel Output in Trajectory Inference We tested dependency on py py-Monocle by providing SCPILOT with corrupted report (randomized cluster relationships and pseudotime order) for the liver dataset The performance drop across metrics demonstrates that accurate outputs from tools like Monocle are critical for SCPILOT.
Summary: These experiments prove that SCPILOT's success is fundamentally dependent on the integrity of the data provided by the bioinformatics tools it integrates.
C.4.3 Additional Experiments in Error Propagation and Uncertainty Assessment
Error Propagation via Input Perturbation We highlight the input perturbation experiments from main text (Figure 4a), where we removed all contextual metadata (e.g.. tissue type) from the input prompt for the PBMC3k annotation task. In the Supplementary experiment, we removed the key input context, witnessing significant performance drops, and confirming that rich metadata is critical for reliability [7
2. Uncertainty and Reliability Assessment We performed a new reliability analysis on the GRN prediction task (Stomach dataset). We calculated 95% confidence intervals (CIs) from 10 trials, performed 1000-bootstrap resampling, and assessed calibration using Expected Calibration Error
28

---

Table 16: Impact of Monocle Perturbation on Trajectory Inference (o1 model)
Metric
Original Avg
Perturbed Avg
Performance Change
Jaccard
1.000
0,933
Worse
GED-nx (10s)
8.00
11.33
Worse
Spectral Distance
0.567
0.593
Worse
Table 17: Results of Context Removal on PBMC Dataset Annotation Accuracy
Model
Accuracy (Original)
Accuracy (No Context)
Performance Drop
o1 GPT-4o
0.792
0,688
0.104
0.646
0.583
0.063
GPT-4o-mini
0.604
0,416
0.188
(ECE) and Brier scores. The o1 model is more accurate, stable (tighter CI), and reliable (lower ECE
and Brier scores) 18
Table 18: Uncertainty & Calibration Metrics for GRN Prediction
Model
95% CI (10 runs)
Bootstrap (+)
ECE
Brier Score
01 GPT-4o-mini
0.84-0.90 0.65-0.75
+0.05
0.05
0.14
+0.10
0.12
0.20
C.5
SCPILOT vs. A General-purpose Biomedical Agent Biomni
We conducted detailed qualitative and quantitative comparison against Biomni, powerful state-of- the-art general-purpose biomedical LLM agent. This comparison highlights the value of SCPILOT's specialized, reasoning-fire first approach.
Contrasting Design: Reasoning-First vs. Tool-First The core difference is our design philosophy. SCPILOT is specialized, reasoning-firs/ agent, whereas Biomni is purpos,,, tool-first agent. We described in detail in table [9]
Impact Scientific Accuracy This design difference leads to significant accuracy gaps. In all 3 tasks, SCP1LOT significantly outperformed Biomni. The results have been included the main text tables
6 andd
For two specific reasoning differences: In Retina anno- tation: SCPILOT correctly distinguishes fine- ann-ngri sub- types (e.g., ON- vs. OFF-bipolar cells), whereas Biomni makes clear errors, mislabeling Müller glia as amacrine cells. In Liver trajectory inference: SCPILOT correctly identifies the Epiblast root and reconstructs faithful lin- eages, while Biomni misplaces cell types (e.g.. "Cardiac muscle") in the lineage, ignoring gene evidence.
Ok
Value in Efficiency and Cost SCPILOT's specialized ap- proach is also vastly more efficient and cost-effective. As shown in table 20 SCPILOT is up to 30x cheaper and significantly faster because curated toolchain and reasoning-first method avoid costly, broad tool searches. It also succeeds on the complex Gene Regulatory Network (GRN) prediction task, where the pprrpss agent fails. This comparison shows that while ceneral-purposs agents are flexible, SCPILOT's specialized reasoning delivers higher scientific fidelity at fraction of the cost.
Figure 7: SCPILOT o1 Liver Trajectory
29

---

Table 19: Design comparison of SCPILOT (reasoning-fing first) and Biomni (tool-first).
Dimension
SCPILOT (Reasoning-First)
Biomni (Tool-First)
Workflow
Hypothesis formulation Omics- native reasoning + Iterative refine-
Run tool Keyword lookup Hard-coded template
ment
Error Handling
Self-diagnoses biologically implau- sible steps and revises
Falls back heuristics Python errors; no biological validation
Output
Transparent Chain-of-Thoughh
Data Dump: Dictionaries, tables, templated outputs
Explanations, confidence scores
Table 20: Efficiency and cost comparison between SCPILOT and Biomni.
Task (vs. Biomni Gemini- 2.5 Pro)
SCPILOT
SCPILOT Cost
Biomni Cost
Cost Ratio
Time
Cell-type annotation
1-3 min
$0.03
$0.80-$1.00 $1.00
~~2-333
Trajectory inference
1-3 min
$0.04
$0.80-$1.0
~20-25
GRN prediction
3-5 min
$0.12
Unsuccessful
N/A
D Biological Insights
D.1 Annotation
Figure 8 analyzes the component impact on retina annotation. SCPILOT GPT-4o scored 0.789 on retina, correctly annotating 14 clus- ters (e.g., Rod Photoreceptors), with 2 partially matched and 3 incorrect (e.g., Cluster 19). In cluster 0,1,7,8, the model correctly picked up rod-specific phototransduction cascade genes. Then, in cluster 14, the model demonstrated its ability to resolve low-abundance cones, despite much larger rod pool. In cluster 17, SCPILOT successfully found horizontal Cells with expres- sions of genes like PROXI and GADI, meaning it was capable of correctly identifying rare cell types (+s less than 1% of retina cells).
GPT4 attencac
ToM0l Sas
Cals
Fubalyy
Figure 8: SCPILOT GPT-4o Retina Annotation
In the main text, we displayed the superior annotation reasoning of SCPILOT in SCPILOT using the o1 model. Here we present the inferior GPT-40-min reasoning proposed too many cell subtypes and confused by multiple gene expressions 6 labeling MKG7/G/MB/FCGRRM + with CD8 T (Cytotoxic T Cells), and NKG7, CXCR4, CD3E with broad cell. In cluster 5, GPT-4o-minn collected distinct highly expressed genes, and failed to find their relationship, labeling it with T cell again incorrectly.
D.2 Trajectory Inference
Figure 7 shows sample trajectory tree analysis in liver dataset with SCPILOT utilizing the o1 model. compares the prediction to the ground truth, with shared edges annotated solid black. The sharing 10 edges, with 3 missing (false negatives, blue) and 5 extra (false positives, red), showing SCPiLOT's role in structural accuracy.
Table 21 these ten faithful edges preserve the global skeleton of liver organogenesis: extra-embryomic tissues split early, mesoderm feeds into hepatic and vasculogenic branches, and the hepatoblast lineage matures correctly down to terminal hepatoblasts.
In Table 22 collectively, three missed and five extra edges all cluster around the mid-level mesodermal hub (nodes 7, 12, O, 4). The predictor correctly recognized the existence of these cell types (so Jaccard 1) but mis-ordered their emergence, compressing swapping developmental stages.
30

---

Analysis
Jaccard simaniit
Simliarty inocesi GED-ra (
GEU 100 Souctraa Distance
Key Changes
Correcting developmental
Recanstructing canonical hepatic lineage
legael thet www.ihndddaaariiii were
100 laid connect crep-ttt -it ifglbtur Endodern- Faragu tiybprria int Hap aoc Hoog 11/00 Devdloge Hesucitkatt Stluuing
colloaree ve true somsitstte orglen
Roor row consains two Epioben Plur ponart branches jutt)s POcU have supo ad antinely from the mann
repatoogte marors the teoboot chro logy marker progressen.
biologically impossible jumps
Refining terminal-state definition
/PAC3 neted earty propentoon hors were clase
hapslcbiad distidddd ai tnmerrvecta
DSt now genkaine collett hepatayyes, rodee.
Those crose pyye adger are gorvt cendneyyy po poe mexuder deried smanhh Printtie and anangictters
tortyicyytt, wehiegg tors /hnmnngno ioaeia
heparchiace
Figure 9: Gemini-2.5-PP omics-driven reasoning in Trajectory Inference, with/without py- Monocle
Table 21: Strengths of the prediction
Developmental branch
Correct edge(s)
Biological interpretation
Early embryonic lineage
extra-
root-+13 (visceral en- doderm)
Visceral-endoderm branch is captured exactly.
Primitive streak II cascade
root root->10-11> 10 (primitive streak II + yolk sac/mesoderm)
Temporal progression from primitive streak into yolk-sac lineage and generic mesoderm re- produced.
Hepatic trajectory
+6, 12-11
splanchnic mesoderm-> gut-tube endoderm migrating definitive hepatoblasts; side branch to septum transversum mesenchyme Full liver lineage-including side mesenchymal without error.
Hemangioblast bi- 4-+2, 4-55 (endothe- furcation
Correct divergence of endothelial vs. hematopoi- etic fates from hemangioblast.
lium / hematopoietic)
Concluding the biological picture in this predicted tree: Early specification (root - primitive streak II / visceral endoderm) and late hepatic maturation (hepatoblast lineage) were handled very well. Mesodermal diversification, the branching that creates cardiac, vasculogenic, and carly streak I derivatives, was the main failure point, leading to an over-connected mesh among mesodermal descendants. Outside this mid-zone, errors occurred. Thus, SCPILOT's largest contribution to the errors (3 FN + FP) was a localized mis-routing rather than global disorganization.
In short, the predictor gave accurate outline of liver development but compresses the timeline of mesodermal commitment, underscoring the need for finer temporal cues distinguish closely related mesoderm-derived cell types. SCPILOT successfully integrated functional relevance, expression context, and known regulatory pathways to make accurate predictions, demonstrating their potential for biologically informed inference when provided with sufficient cues.
D.3 GRN Prediction
The correct predictions in Table 23 were explained in the main text.
31

---

Table 22: Weak spots in the prediction
Issue
Edge(s) concerned
Error type
Consequence
Primitive streak or- phaned
root-79 (primitive streak I)
Missing (FN)
Early streak I appears later than it should, breaking its parallel origin with streak II.
Mesodermal diversification under-represented Spurious fan from mesoderm
7-+0, 7-+4 (cardiac mesoderm, heman- gioblast) 7-+12, 7-9
Missing (FN)
Fails to branch cardiac and vasculo- genic fates directly from mesoderm.
Extra (FP)
Mesoderm now feeds splanchnic meso- derm and streak conflating separate temporal stages.
Mis-routing via splanchnic meso- derm
12-+0, 12->9
Extra (FP)
Pulls cardiac mesoderm and streak into downstream compartment.
Looping of vasculo- genic branch
9-+4 gioblast)
(heman-
Extra (FP)
Places hemangioblast downstream of streak instead mesoderm, forming artificial detour.
The incorrect predictions 24 frequently arose from insufficient search for domain knowledge or misleading biological signals. For instance, the prediction of Usf2 regulating Pigr occurred when models lacked detailed searching for their relationship, limiting their ability justify reject link. In predicting Fos regulating Hmox/, shared GO term annotations between a TF and its candidate target gene falsely suggested functional association, leading incorrect reasoning.
D.4 Conclusion on SCP1LOT accelerating discovery.
1. Transparent agent loops. All intermediate outputs and reasoning steps logged, letting biologists audit each decision.
Cross-task generality. The same kernel framework solved annotation, trajectory inference, and GRN prediction tasks, hinting at broad utility.
Plug-and aaee-and-play tool integration. The agent can ingest outputs from Monocle, SCENIC, and more traditional bioinformatics analysis tools without code changes, making it easy to layer domain heuristics on top of LLM reasoning.
E Prompt Templates
E.1
Cell Type Annotation Direct Prompting
Below the one-step prompting for cell type annotation, purely based on the top marker genes for each cluster, and the context.
Direct Prompt for Cell type Annotation
system role "You are expert in scRNA sequencing cell type annotation."
content = this is background information:
look this dict: differential genes of each cluster. Please provide cell type annotation for each cluster. Output text dict format just like the input dict. Keys are number of cluster, and Values are strings of cell type names. Output should be text diet, no other word should exist.
ciatter_geee_diie dict. This is cluster number and the corresponding top
F.2
Cell Type Annotation SCP1L.OT
Below are the prompts in the multi-agent framework for SCPILOT cell type annotation. In the hypothesis generation, SCPиLOT integrates the top marker genes per cluster, dataset context and potentially, information from previous iterations create hypothesis about the dataset.
32

---

Table 23: SCPILOT o1 successful reasoning in TF-Gene Prediction
TF + Gene
Reasoning
Why the reasoning succeeded
(Tissue)
Statl + Irf7
Statl a key regulator of interferon signaling, and Irf7 is essential for type interferon induction. Both genes share the GO term defense response to virus, suggesting func- tional overlap. Although direct evi- dence in stomach tissue limited, shared antiviral pathways make di- rect regulation plausible.
The answer recognizes that STATI sits at the hub of type-I interferon signaling and that IRF7 is the master amplifier of the same pathway. It explicitly points their shared GO term "defense response to virus", connecting transcription factor activation, downstream gene induction, and antiviral state. That functional con- vergence is exactly what experimental immunologists rely on when mapping STATI-IRF7 regulatory cascades in mu- cosal tissues.
(Stomach)
Kif4
Kif4 known to maintain epithe- lial homeostasis and can regulate mucin genes in the gastrointesti- nal tract. Muc5ac is the principal mucin in the stomach and is crucial for gastric barrier function. Given Klf4's role in controlling mucin ex- pression (e.g., Muc2) and epithelial differentiation, it is plausible that it may also directly regulate MueSac. However, definitive experimental evidence for direct regulation in the stomach remains limited.
Instead of merely noting that both are "ep- ithelial," the model explains that KLF4 maintains gastric epithelial homeostasis and has precedent for activating mucin genes (c.g., Muc2 in the intestine). then argues that MUC5AC is the prin- cipal gastric gel-forming mucin, mak- ing KLF4-driven regulation biologically plausible. By marrying KLF4's differ- entiation role with the barrier function MUC5AC, the reasoning shows a mu- anced grasp of gastrie physiology.
MucSac (Stom- ach)
SCPILOT Prompt for Cell type annotation hypothesis
content Too (len(self.top_gene) _egess) differentially expressed genes: [self.top_genes] genes)) self.referencce_dii
content += You refer to the possible cell types of these top genes in this dictionary- self.referencee _iiteetcceeeee""""
content +ontent Currrennt HypothesisseeE.yyohhess
annotation_diie
content += Th cell type annotation from previous iterations annotation_des _dict"
ho__gene_clustee content += f'Clusters without need to focused on: fro__gene_clustriei if surraiio_ssummay:
content += f"This is summary of previous iteration annotation, with information of next steps take. iteration_satt iitration_summar"
system_role "You are research assistant specializing in cell biology. Based on top differentially expressed genes, previous cell type annotation (if provided), Clusters without need to be focused on (if provided), summary of previous iteration annotation (if provided), and failed genes (if provided), refine the given hypothesis be more accurate and specific."
In the marker gene proposal step, SCPILOT specifically proposes a marker gene list for the cell types of interest.
SCPILOT Prompt for Cell type annotation marker gene proposal
prompt f You are bioinformatics expert specializing in liver cell annotation. Your task to propose an experiment for cell type annotation based on the following information: Refined hypothesis: self.hypothesis
33

---

Table 24: SCPILOT o1 failed reasoning in TF-Gene Prediction
TF + Gene
Reasoning
Why the reasoning failed
(Tissue)
Usf2 + Pigr
Usf2 is a bHLH-LZ tran- scription factor best character- ized for regulating metabolic and some housekeeping genes. Pigr encodes an immunoglob- ulin receptor primarily active in mucosal immunity.
The model claimed 'no known functional over- lap' because PIGR is mucosal-immunt gene and USF2 'mostly regulates metabolic genes'. In reality, USF family bHLH factors bind canonical E-box motifs present in the Pigr promoter and are documented activators of several immune-related transporters. By re- stricting USF2 'metabolic housekeeping', the explanation ignored promoter architecture studies that tie USF2 to Ig-transport regulation in gut epithelium-henee the false negative. Actually, one published journal [? has ex- plicitly mentioned this TF-gene relationship, indicating that the current SCPILOT is not an expert in searching literature.
(Stomach)
There is no known fune- tional overlap or evidence co-regulation between Usf2 and Pigr in stomach tissues. Hence, direct Usf2 Pigr regulation in the stomach is unlikely.
Fos Hmoxl (Stomach)
Fos is key component of the AP-I complex, known to reg- ulate stress-responsive genes, including those involved cellular response cadmium ion. Hmox1 is upregulated by oxidative and heavy metal stress and shares this func- tional pathway with Fos tar- gets, This overlap suggests moderate potential for direct regulation of Hmox by Fos in the stomach.
The model argued that both genes "stress-responsvve" and share the GO term cellular response to cadmium ion, cluding a plausible AP-1-driven induction. HMox1 transcription in fact, dominated by NRF2/KEAPI and small-Maf binding at antioxidant-responss elements, not by AP-1 sites. While AP-1 motifs exist in the far up- stream enhancer, they tric tissue for HMox1. The explanation, there- fore, over-generalized "stress overlap" without checking the principal regulatory axis, yield- ing false positive.
functional in gas-
Instructions:
0. We have already labeled some of clusters, the information is annotation dict
1. a) The most important unsolved clusters are gengeee cluster.
b) Other cell types in annotation, dict is cell types we labeled with high or low confidence.
We first need to decide whether should annotate them again.
2. For the cell types we have already successfully used, the gene marker list is success- ful _el_genes.
For large cell types like hepatocyte or cell, the remaining clusters might still contain them, but for smaller cell types you don't need to think about them again. For the marker genes already used but did not detect any expression, the list is miile_geness, So you don't need to try these gene and related cell type again.
3. Do not specify the cluster with cell type here. You can just output cell type and related marker genes.
4. Only provide proposal of unlabeled clusters, consider potential overlaps in marker gene expression between cell types:
a) Name of the cell type b) 3-5 marker genes
Output format: List of cell types with their markers: [Cell Type 1]: Gene A; Gene B; Gene
[Cell Type 2]
Python list of marker genes: MARKER BARKER_GENESS ['GeneA', 'GeneB', 'GeneC,, ...] Remember: Be specific and concise in your descriptions. Ensure all cell types have at least marker genes. Include the Python list of all marker genes at the end of your response, don't 't use any backtick.
34

---

system role "You are Al trained to design scientific experiments based hypotheses and background information."
In the evaluation step, SCPILOT analyze the dotplot expression data, and make comprehensive evaluation about the expression and thus perform prediction.
SCPILOT Prompt for Cell type annotation evaaluation
content f Context: You are working single-cell RNA sequencing cell type annotation task. The goal is to identify distinct cell types based on gene expression patterns. You have created dotplot using a of marker genes visualize gene expression across different clusters.
MUST remember: you should list the cell types here. All cell types you refer to, should be in here. possible_ cctp__tllelleleeeeee__eeeeeeeeee
Data: For each cluster, the top genes are in this dictionary: marker gennss The clusters cannot find top genes are: empty_ keys These are the genes that are successfully (highly) expressed in some clusters: success list These are the genes that are failed to express in any clusters: fail list There are some clusters that have similar expression, we did differential expression analysis for these cluster pairs. The pairs and top differential genes are in: similar_clusters__ wwwssterussseeeeeeee __dicttcttttc
Add-on: Duplet and Contamination in dotplot: duplet_rule rule Any other important instructions: contamination rule
Please give greatest attention to the Add-on part, if they are provided. Make sure to use them in your analysis.
Instructions: Please analyze the provided data and answer the following questions:
1. Gene-level analysis: a) Which genes are highly expressed in specific clusters? Provide detailed description. b) Are there any genes that show differential expression across clusters (high in some, low in others)? c) Are there any genes that not informative for cell type annotation (low expression across all clusters)? You should answer this question based on negation of lb.
Cluster-level analysis: a) Are there any clusters that lack high expression of any marker gene? If so, list the cluster numbers. There in total cluster_size clusters, index from 3. Overall assessment: a) Based on the gene expression patterns, are there distinct clusters that potentially represent different cell types? b) Can you assign specific cell type identities to any of the clusters based on the marker gene expression? If so, provide your cell type annotations. You should only assign one cell type to each cluster. No doublet or contamination here. e) To refine the cell type annotation, recommend possible additional cell types. d) To refine the cell type annotation, recommend any particular cluster to perform subgrouping.
4. Confidence assessment: a) What are confidence levels of your annotation? Please also assign confidence score to the process name you selected. This score should follow the name in parentheses and range from 0.00 to 1.00. A score of 0.00 indicates the lowest confidence, while 1.00 reflects the highest confidence. This score helps gauge how accurately the annotation is. Your choices of confidence score should be normal distribution (average 0.5) You should consider if the annotation widely seen, using deterministic wording and following background of dataset. For instance, if you label doublet, or using word "probable", the score should be lower.
b) Based confidence levels, what the annotation results that you want to "stabilize". that is not change in next steps? if iteration is 1, you should choose top 1/3 confident clusters. if iteration is and beyond, you should choose top 2/3 or more clusters. You can choose this threshold.
Please provide your answers in structured ISON format, addressing each question separately using "la" "2a" etc.
Remember, this one iteration cell type annotation for a liver scRNA-seq dataset. Your insights will guide further refinement of the analysis. "" ssysemm_rol soaeerroleee--ee = "You are an expert bioinformatician specializing in single-cell RNA sequencing data analysis and cell type annotation."
35

---

E.3
Trajectory Inference
Both Direct prompting and SCPILOT use the same simple annotation prompt to generate annotated clusters for trajectory analysis.
Direct and SCPILOT Prompt for Trajectory annotate
query = p* You are analyzing single-cell RNA-seq data. The dataset contains clusters of cells identified by Leiden clustering. Given these clusters and reference cell types, predict the most likely annotation for each cluster. First, read the context. Then, use the top genes annotate each cluster. Finally, refine the annotation with the percentage of timepoint in each cluster, so you will know the cell type in the cluster is more likely in the proliferating or mature stage.
Please refer to the context information of the dataset: context Please base your annotation on the top 5 genes of each cluster: top5_dict Here is the percentage timepoint in each cluster: day_percentage
Remember, every cluster a distinct cell type.
First output your chain of thought, then provide output dictionary mapping cluster IDs to cell type annotations. For the dictionary, ONLY output a python code dictionary, do not include pytthon
The Direct Prompting use a one-step tree construction prompt, trying to connect all annotated clusters in one tree.
Direct Prompt for Trajectory build tree
query = f"" Construct a developmental trajectory tree for the clusters in the single-cell dataset. Here is the context: context Ensure: 1. The tree starts from the root (youngest stage). 2. Progression follows biologically meaningful paths. 3. Misplaced branches are flagged. Data: Cell Type annotation for each cluster: annotated ammoaated_clustrrss clusters Developmental stages: day_percentage
In your trajectory tree, do not include any time stage, only use the Cell clusters. We need to set dummy node root (use the name "root"), and then add all cell types iteratively as leaves and leaves of leaves. DO NOT include any other node except root and cell types. First output your chain of thought, then provide output trajectory tree nested dictionary at END. For the nested dictionary, ONLY output python code dictionary, no backslash to make new line. do not include "'python" "". Do not put anything else after the nested dict.
SCPILOT adopts a multi-step tree construction prompting. First, find the root node (initial cell type). Then iteratively add the leaves to the tree and then finalize the trajectory.
SCPILOT Prompt for Trajectory multi-step tree reconstruction
root_query = p""" You are analyzing single-cell RNA-seq dataset with developmental progression. Context: context
Given the following: Cell Type annotation for each cluster: annotated_cluste Develop-
mental stages as percentage across timepoints: day jey_eeeenttgee
Task: Identify the single most appropriate root cluster for a developmental trajectory tree.
This should represent the developmentally earliest or least differentiated cell population. Only return the name of the root cluster as a string. Do not include any additional explanation.
tree_query " Construct developmental trajectory tree starting from the root cluster: "root_ "root_cluster""
Context: context
Given: Cell Type annotation for each cluster: aanotated_clustery Developmental stages:
day _ey_peecnttgee
Task: Iteratively construct a trajectory tree: 1. The tree should begin at the root cluster: Troot_clusted 2. Add directional edges from the root to other clusters based on develop-
36

---

mental progression. 3. Each edge should represent a biologically meaningful transition. Continue expanding the tree until all clusters are connected. Flag any biologically implausible transitions (e.g., backward differentiation).
Output Format: Return a Python dictionary where: Keys are parent nodes (clusters). Values lists of child clusters. Ensure every cluster appears least once in the tree. In your trajectory tree, do not include any time stage, only use the Cell clusters. We need a dummy node root (use the name "root"), and then add all cell types iteratively as leaves and leaves of leaves. DO NOT include any other node except root and cell types. .." finalize_gn _errrre__rrrr__e f Please extract the trajectory tree inside the input. The input containing the tree is whole tree IN YOUR REPLY, ONLY output python code tree, DO NOT include "'python You can represent tree using nested dict. DO NOT USE any square brackets. ALWAYS use nested curly brackets.
After receiving the report from py-Monocle, SCPILOT will generate analysis summarizing how to improve based on Monocle suggestions.
SCPILOT Prompt for Trajectory analysis monocle
prompt = f'" Analyze these components for trajectory improvement: Biolog- ical context: context 2. Current trajectory: trajectory _tre 3. Cluster annotations: anno-
analysis-prompt
gated_cluster 4. Analytical report: trajectory_repot Identify: Missing progenitor rela- tionships Cluster-cell type mismatches Marker gene contradictions Structural hierarchy errors Root node completeness u..
Then, SCPILOT will reconsider the cell type annotation and trajectory with the Monocle analysis.
SCPILOT Prompt for Trajectory reconsider with monocle
query = p""" Perform comprehensive validation of cell trajectory and annotations using biological context and analytical report insights.
Requirements: 1. Analyze context to identify key gene markers for cell fate determination. Validate hierarchical structure in trajectory_ tree matches differentiation pathways. 3. Resolve any inconsistencies in annotated clusters using markers from trajectory report. 4. Ensure root node contains all initial progenitor states. 5. Remove temporal references, focus lineage relationships.
Current trajectory structure: trajectory_tree Existing cluster annotations: annotated _eoouttee_dlssters This is the analysis about how improve: analysis
No matter how you modify, you should make sure all the cell types in the final cell type annotation dict
the final trajectory structure, and vice versa.
Output Format requirement: DO NOT use ""python" to wrap your python code. - Return as string of two dictionaries: trajectory_die dict and annotation dict. Trajectory must be hierarchical dict. set a dummy node as root (use the name "root"), and then add all cell types as leaves. In your trajectory, do include any square brackets. use 'str' to contain nodes, and curly bracket to show tree trajectories. In the trajectory, Leaf node values should always be empty dictionaries (empty curly brackets). Annotation diet must map ALI. cluster IDs to terminal cell types.
ONLY output the string of two dictionarics, no additional text/formatting. Be extra careful with the format of curly brackets in the nested dict of trajectory dict.
scPilOT will have final synthesis step that keep all the reconsidered results consistent.
SCPiLOT Prompt for Trajectory final synthesis
synthesis_ _rriiissppppteeee = r""" Current adjusted trajectory and refined annotations: response
You may see is a string containing one nested dictionary for trajectory, and one dictionary for annotation. We want to extract these dictionary from the string.
37

---

Make sure to output the current input materials in correct format: Make sure all the cell types in the final cell type annotation dict are in the final trajectory structure, and vice versa. Output Format requirement: DO NOT use "'python" to wrap your python code. Return as a single Python tuple with (trajectory_dee _dct, annotation_ sil_dict)-- Trajectory must be hierarchical dict. set dummy node root (use the name "root"), and then add all cell types leaves. In your trajectory, do not include any square brackets. use 'str' to contain nodes, and curly bracket show trajectories. In the trajectory. Leaf node values should always be empty dictionaries (empty curly brackets) Annotation dict must map ALL cluster IDs to terminal cell types
ONLY output the Python tuple (trajectory_duee _dcc, annotation -itiattiooddii)),..oo no additional text/formatting.
For GRN TF-gene prediction, the Direct Prompting only asks a simple question regarding this TF-gene relationship.
Direct Prompt for GRN prediction
prompt f""" Decide how much possible directly regulates gene in (ctxB):
The possibilityy is number from 0 to 1.
Return exactly: Reasoning: <your reasoning> Possibility is: <your possibility> """ strrpptp))
For SCPILOT, the GRN TF-gene prediction incorporates more inputs. There is more context about the tissue of TF and gene, and the GO database functional overlap for TF and gene.
SCPILOT Prompt for GRN prediction
prompt sww__shot_blockee + f""" *Task*: TF: tf and Context tissues (ctxA) Functional
overlap (shared GO BP terms): overlap
Decide how much possible tf directly regulates gene in (ctxB):
The possibility a number from 0 to
Think step by step: 1. Recall TF tf's biological role. 2. Compare gene with known tf targets.
Conclude which statement fits better (<= 4 sentences).
Return exactly: Reasoning: <your reasoning> Possibility is: <your possibility> """ """,strip()
38


Reviews:
[{"review_id": "CR6wQEUzdU", "reviewer": "Reviewer_RBqf", "summary": "This paper introduces SCPILOT, a novel framework that utilizes Large Language Models (LLMs) to automate and enhance the analysis of single-cell RNA-sequencing (scRNA-seq) data. The core concept is \"omics-native reasoning\" (ONR), where an LLM not only processes natural language inputs but also directly interacts with omics data and bioinformatics tools to perform complex analyses. The framework addresses three key challenges in single-cell analysis: cell-type annotation, developmental-trajectory reconstruction, and transcription-factor targeting.", "questions": "1. Does the LLM in the SCPILOT framework autonomously decide which bioinformatics tools to call upon, or does it follow a pre-determined workflow where the sequence of tool usage is set by human engineers?\n2. What is the paper's assessment of the depth of biological knowledge in state-of-the-art LLMs, and how correct is the \"chain-of-thought\" reasoning they produce in this specialized domain?\n3. How does SCPILOT determine the termination condition for its iterative loop? Is the decision made autonomously by the model itself or by an external program?\n4. According to the results, even with a powerful model like GPT-4o, the 'Direct' method occasionally yields better results than the SCPILOT pipeline. What are the potential reasons for this?", "limitations": "The framework relies on a \"Problem-to-Text Converter\" to create a \"semantic sketch\" of the massive single-cell data matrix to fit within the LLM's context window. A potential issue, acknowledged by the authors, is that this compression step may discard subtle but important biological signals, particularly from rare cell types.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Instead of simply using LLMs as wrappers for existing tools, SCPILOT enabling the LLM to directly interact with data and reason through biological problems. \n\n2. The development of SCBENCH provides a standardized and rigorous platform for evaluating LLMs on complex, real-world biological tasks. This is a valuable contribution to the field, allowing for objective comparisons of different models and methods. \n\n3. A key advantage of SCPILOT is its ability to produce a transparent \"chain of thought,\" which makes the analysis process auditable and easier for researchers to understand and verify. This addresses the \"black-box\" nature of many computational biology tools. \n\nWeaknesses:\n1. The best results were achieved using powerful, proprietary LLMs like Google's Gemini 2.5 Pro and OpenAI's models. The paper itself notes that the open-source model tested (Gemma 3 27B) underperformed significantly and had issues with latency and cost, limiting the accessibility of the framework for labs without access to these expensive APIs. \n\n2. The paper notes that smaller or latency-optimized \"mini\" models occasionally performed worse with the SCPILOT pipeline, suggesting that the extended reasoning process can lead them to \"over-explore\" and make mistakes.", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful to Reviewer RBqf for highlighting three pillars of our contribution.\n1. Novelty of approach – \"enabling the LLM to directly interact with data and reason through biological problems.\"\n2. Community value of scBench – a \"standardized and rigorous platform for evaluating LLMs on complex, real-world biological tasks.\"\n3. Transparency & auditability – our fully exposed “chain-of-thought,” which explicitly tackles the usual “black-box” critique by making every step \"auditable and easier for researchers to understand and verify.\"\n\n### **(a) Accessibility of scPilot and Additional Cost Analysis**\n\nWe acknowledge the reviewer's valid concern about relying on powerful proprietary LLMs. Our novel omics-native reasoning (ONR) paradigm is inherently more demanding than standard text analysis and requires powerful LLMs, a point confirmed by our experiments on **Gemma 3 27B** (the best open-source model testable on 8xH100s at the time). Our experiments reveal two reasons why ONR is challenging for current open-source models:\n\n* **Insufficient Domain Knowledge**: Weaker models lack the nuanced, pre-trained understanding of biology, such as raw markers and pathways, to interpret omics data correctly.\n* **Poor Instruction Following**: Weaker models often fail to follow complex instructions and adhere to biologist-defined formats (e.g., JSON), breaking the reasoning chain.\n\nThese insights are a key contribution, offering guidance for future model development. However, the cost of using powerful models via scPilot is negligible. Our detailed cost analysis for Gemini 2.5 Pro shows that a complete run for our most complex tasks costs only a few cents, making the framework accessible.\n\n**Table 1. Cost analysis of scPilot on all three tasks (Gemini-2.5 Pro)**\n| Task (single run) | Input tokens | Output tokens | Input cost | Output cost | **Total** |\n| :--- | :--- | :--- | ---: | ---: | ---: |\n| **Task 1 – Cell‑type annotation (retina)** | 6,155 | 2,197 | \\$0.008 | \\$0.022 | **\\$0.03** |\n| **Task 2 – Trajectory inference (neocortex)** | 8,221 | 2,702 | \\$0.010 | \\$0.027 | **\\$0.04** |\n| **Task 3 – GRN TF→gene prediction (stomach, 46 pairs)** | 17,877 | 9,276 | \\$0.022 | \\$0.093 | **\\$0.12** |\n*Note: Token counts approximated with `tiktoken`. Cost = tokens / 1,000,000 * unit price (\\$1.25 / 1M input, \\$10.00 / 1M output; no caching).*\n\nThe negligible cost allows any lab to leverage state-of-the-art AI without expensive local GPUs. Developing cheaper, specialized open-source models for ONR is a promising future direction; in parallel, our framework provides an immediate, accessible tool for the community.\n\n### **(b) Additional analysis on occasional suboptimal performance**\n\nWe thank the reviewer for this sharp question. The observation that a simpler ‘Direct’ prompt can occasionally outperform scPilot is a key finding, not a flaw. These cases are rare, systematic, and informative.\n\nscPilot overwhelmingly outperforms the baseline (wins in 87 of 108 total comparisons). The rare losses are systematic: **13 of 21 (62%)** are from less-capable \"mini\" models. Their limited capacity for sustained logic leads them to \"over-explore\" and make mistakes with scPilot's extended reasoning. \n\nFor the few remaining cases with powerful models, the cause is high dataset nuances, inducing \"overthinking.\" The only two powerful-model losses in cell-type annotation occurred on our most complex dataset, Liver. The characteristics of our datasets explain this pattern:\n\n**Table 2: Characteristics of three datasets in Cell Type Annotation**\n| **Dataset** | **# Clusters** | **# Cell Types** | **Description** | **scPilot's Behavior** |\n| :--- | :--- | :--- | :--- | :--- |\n| **PBMC3k** | 8 | 8 | Simple: 1:1 mapping, distinct markers | Good, though mini models may saturate performance |\n| **Retina** | 18 | 9 | Moderate: Clear types, slight ambiguity | **Consistently improves accuracy.** |\n| **Liver** | 28 | 31 | Complex: Overlapping lineages, noisy expression | **Can \"overthink\" and merge distinct subtypes.** |\n\n**A Concrete Example of \"Overthinking\"**: In the complex Liver dataset, scPilot with the powerful o1 model scored 0.518, while the Direct method scored 0.560. A key error was confusing developmentally related `hepatocytes` and `hepatoblasts`. Because they share many markers, scPilot's deep reasoning amplified this ambiguity and wrongly merged them, whereas the simpler Direct method was incidentally correct by ignoring this nuance.\n\nThis analysis defines the boundaries for applying LLMs to complex biological data and will be added to the manuscript. Future improvements include adaptive reasoning depth and marker clarity assessments.\n\nWe provide a more detailed analysis in Reviewer mMx1's rebuttal (a) about why o1's overthinking behaviors in Liver dataset.\n\n### **(c\\) Clarification on LLM Tool Selection**\n\nThis is related to the design philosophy of scPilot. scPilot uses a pre-determined tool workflow as a deliberate design choice to rigorously evaluate the LLM's core **omics-native reasoning**, not its ability to use tools.\n1. Our primary goal is to test the model's ability to interpret complex, numerical outputs from established bioinformatics tools and formulate a coherent scientific argument.\n2. By standardizing the pipeline with gold-standard tools (e.g., Monocle, SCENIC), we isolate this core reasoning capability. This prevents our evaluation from being confounded by trivial tool-use errors, like incorrect API calls, which is a separate research problem.\n3.  Our comparison against a tool-calling agent, Biomni, directly demonstrates the negative impact of such tool-use errors on performance; response (c) to Reviewer \n\n### **(d) Additional analysis on depth of biological knowledge and reasoning in LLMs**\n\nThis is an excellent question that gets to the heart of our contribution. First, we should note that the internal \"chain-of-thought\" of proprietary models (o1) is not accessible.\n\nSecond, our assessment is that STOA LLMs possess a surprising depth of latent biological knowledge, and the scPilot framework is crucial for structuring this knowledge into a correct, verifiable, and scientifically rigorous reasoning process. We demonstrate this across the following qualitative analysis\n\n1.  **Sophisticated Multi-Gene Logic**: In cell-type annotation, scPilot moves beyond simple marker matching to distinguish **NK cells from CD8 T-cells** using a combination of `NKG7`, `GNLY`, and `CD3D`—a task where single-marker tools often fail. It can also identify extremely rare cell types (~1% abundance).\n\n2.  **Deep System-Level Understanding & Self-Correction**: In trajectory inference, scPilot reconstructed the liver developmental tree and, more importantly, used diagnostic outputs from Monocle to **self-audit and correct its own reasoning**. This improved trajectory accuracy significantly (e.g., reducing graph-edit distance by 6 and spectral distance by 0.32).\n\n3.  **Contextual, Tissue-Specific Reasoning**: For GRN prediction, scPilot applies **tissue-specific reasoning**, such as understanding `Klf4`'s role specifically in the stomach, to make more accurate predictions.\n\nIn conclusion, our framework produces a transparent reasoning trace—a step-by-step record of claims, tool outputs, and decisions—that serves as a fully auditable line of reasoning.\n\n### **(e) Clarification on Termination Condition**\n\nThis is an excellent question regarding the mechanics of our framework.\n\nFor our benchmark, `scBench`, the termination conditions are pre-defined for each task to ensure fair, reproducible, and rigorous evaluation, rather than being determined autonomously by the LLM.\n\nThe specific conditions are tailored to the nature of each task:\n- Cell-Type Annotation: The process is set to a maximum of 3 reasoning iterations. This fixed number provides a sufficient window for the LLM to perform self-correction and refine its initial hypotheses without leading to \"overthinking\" or excessive computational cost.\n\n- Trajectory Analysis & GRN Prediction: These are structured as single-pass reasoning tasks. For trajectory analysis, this involves an initial tree construction followed by a controlled self-reconsideration step prompted by Monocle's output. For GRN, all evidence is provided upfront, making a single, comprehensive reasoning pass the most effective approach.\n\nThis deterministic design is crucial for a benchmark. It ensures models are evaluated under identical conditions, preventing variability that confounds results, an issue seen with the autonomous agent, Biomni.\n\n### **(f) Potential Information Loss from Data Compression**\n\nWe agree with the reviewer on this important point. The \"Problem-to-Text Converter\" is a necessary trade-off to fit large omics datasets into current LLM context windows. While this compression is inherent, our framework is both highly effective today and designed for future extension.\n\nFirst, our current \"semantic sketch\" approach is remarkably effective at preserving critical biological signals. For instance, in our Retina analysis, scPilot successfully identified rare Horizontal Cells that constitute less than 1% of the total cell population. This demonstrates that our converter retains the necessary resolution for nuanced discovery, even for subtle signals.\n\nThis trade-off represents an exciting research frontier. We envision several strategies to enhance this component:\n* **Dynamic, RAG-like Retrieval**: Allow the LLM to query the full data matrix for fine-grained details on demand.\n* **Hybrid Foundation Models**: Integrate scPilot with specialized single-cell models (e.g., scGPT) that can process raw data, with our LLM acting as the high-level reasoning engine.\n\nOur `scBench` provides the ideal platform to develop and test these advanced strategies. Thus, while data compression is a challenge, our work offers an effective current solution and a foundational testbed for future innovation."}, {"review_id": "J03526hcjs", "reviewer": "Reviewer_4XCY", "summary": "The authors proposed an agent based cell annotation framework to leverage LLMs to use tools and do inference over raw input data. The experiments are conducted across three applications: cell type annotation, trajectory inference, and GRN prediction. The performances are better than previous LLMs based models.", "questions": "1. See weeknesses.\n2. It is unclear about how the Problem-to-Text Converter work. The mapping process is unclear and I have tried to find unswer in the following content but it seems missing.", "limitations": "See weeknesses", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. It is important to incorporate the raw gene data into the analysis which is important and essential in the bioinformatics pipeline. And this point is ignored by most previous LLMs based single-cell analysis framework.\n2. The experiments are a lot including three different tasks with detailed analysis.\n\nWeaknesses:\n1. This work is more like an application of agent on LLMs, which doesn't propose further machine learning algorithm improvement.\n2. The experiments are conducted solely on LLM based method. It is important to verify that whether the LLMs driven methods can be better than traditionally non-LLMs based methods.", "was_revised": true, "final_justification": "I appreciate authors' effort on providing detailed response. Regarding the Reasoning Paradigm in the innovation, which is claimed as the core contribution in this paper, I don't think the current manuscript successfully verifies whether the reasoning path produced by LLMs is really biologically reasonable. It would be better to provide detailed reasoning paths and examing them by biology experts to see whether it provides biologically plausible reasoning results. Probably, these can be checked by reviewers from computational biology journals. Therefore, I decided to keep my original score.", "rebuttal": "We sincerely thank Reviewer 4XCY for highlighting our significant contributions:\n1.  **Methodological Significance**: Recognizing our novel approach of \"incorporating raw gene data\" as \"important and essential\"—a critical step that has been \"ignored by most previous LLM-based frameworks.\n2.  **Experimental Rigor**: Acknowledging that our \"experiments are a lot,\" spanning three different tasks with \"detailed analysis.\"\n3.  **Demonstrated Improvement**: Confirming that scPilot's performance is \"better than previous LLMs based models.\"\n\n\nWe appreciate the reviewer's questions, which touch upon the nature of our contribution, the breadth of our experimental baselines, and the mechanics of our data-to-text conversion. We are confident that the detailed clarifications below will fully resolve these concerns and reinforce the significance and novelty of our work.\n\n*** \n\n**(a) Clarification on Machine Learning Contribution and Originality**\n\nWe thank the reviewer for this comment, which allows us to clarify our core contribution. While we do not propose a new, standalone ML algorithm in the traditional sense, such as a novel neural network architecture or loss function, our work introduces a **novel reasoning paradigm** and a **foundational benchmark, `scBench`**, which are crucial and non-trivial advancements for applying AI to science.\n\n**1. First, A Novel Reasoning Paradigm (Not a Pure Application)**\n\nOur primary contribution is **Omics-Native Reasoning (ONR)**, the first systematic framework for an LLM to directly interact with raw omics data in an interpretable reasoning loop. \n\n\nThis is fundamentally different from prior work and is a non-trivial advancement:\n* Unlike simple **tool-calling agents** that merely wrap existing bioinformatics tools, scPilot deeply *interprets the numerical output* of those tools to form and revise hypotheses.\n* Unlike **embedding-based foundation models** that produce opaque \"black-box\" vectors, scPilot generates a fully transparent and auditable reasoning trace, addressing a key limitation in computational biology.\n\nAs the reviewer noted, creating a framework where an LLM can directly reason with raw gene data is an \"important and essential\" challenge, ignored by most previous work. Solving this is our core methodological contribution.\n\n\n**2. Second, Tangible Performance Gains from a New Paradigm**\n\nThis new paradigm is not merely conceptual; it delivers substantial, measurable improvements in analytical accuracy. Our experiments show that the iterative reasoning enabled by scPilot **lifts average cell-type annotation accuracy by 11%** and **cuts trajectory graph-edit distance by 30%** compared to simpler, one-shot prompting methods. This demonstrates that changing the *reasoning process* is a powerful vector for performance improvement, distinct from changing the model architecture.\n\n**3. Last but not least, A Foundational Benchmark for Future ML Improvements**\n\nCrucially, our work is **foundational for future machine learning algorithm improvements**. Before algorithms can be improved for a task, that task must be rigorously defined and benchmarked.\n1.  **A New Task & Benchmark**: Our ONR paradigm defines the task, and `scBench` provides the first-ever leaderboard-style benchmark to measure performance on this complex scientific reasoning.\n2.  **Enabling Future Models**: The transparent reasoning traces generated by scPilot can serve as high-quality, structured data for fine-tuning the next generation of smaller, more efficient, open-source scientific AI models to become expert \"omics-native reasoners.\"\n\nAs a result, our framework further provides the essential blueprint and testbed for these future ML advancements.\n\n***\n\n**(b) Additional Experiments with Non-LLM Baselines**\n\n\nWe thank the reviewer for emphasizing the importance of comparing against traditional non-LLM methods. We would like to clarify that our paper already includes these crucial baselines across all three tasks, and our results consistently show that scPilot outperforms them.\n\nIn our original submission, we benchmarked scPilot against several established non-LLM methods:\n* **Cell-Type Annotation:** `CellTypist` and `CellMarker 2.0` **(Table 2, Line 170)**\n* **Trajectory Inference:** `py-Monocle` (Monocle3) **(Table 6, Line 231)**\n* **GRN Prediction:** Graph Neural Networks including `GCN`, `GraphSage`, and `GAT` **(Table 7, Line 260)**\n\nTo further strengthen this analysis, we have now added the latest version of **Celltypist (1.7.1)** as an additional baseline. The results continue to demonstrate scPilot's superior performance.\n\n**Cell Type Annotation Accuracy Comparison**\n| Dataset | scPilot o1 | scPilot Gemini-2.5 Pro | Celltypist 1.6.3 (original) | Celltypist 1.7.1 (new) |\n| :--- | :--- | :--- | :--- | :--- |\n| **Liver** | **0.518** | 0.488 | 0.429 | 0.464 |\n| **PBMC** | **0.792** | 0.708 | 0.500 | 0.563 |\n| **Retina** | **0.728** | 0.675 | 0.263 | 0.368 |\n\nAs this table and the results in the main paper show (e.g., Table 6 for trajectory, Table 7 for GRN), our LLM-driven scPilot framework consistently and significantly outperforms these state-of-the-art traditional methods.\n\n\n***\n\n**(c\\) Clarification on Problem-to-Text Converter**\n\n\nWe thank the reviewer for this question and are happy to clarify the mechanics of our Problem-to-Text Converter. Its purpose is to create a compact, information-rich \"semantic sketch\" of the massive raw data matrix, which can contain over 100,000 cells, so that an LLM can process it within its context window.\n\n**Key Steps:**\n\n* **Clustering:** The converter first groups single cells into clusters (e.g., Leiden algorithm via Scanpy). This reduces the dataset from 10⁵ cells to a manageable number of clusters (typically 10–30).\n* **Feature Selection:** For each cluster, the converter identifies the top marker genes (e.g., top 10 genes by differential expression), which define the cluster’s identity.\n* **Biological Metadata:** The converter includes essential metadata such as species, tissue, protocol, and timepoints, providing biological context.\n* **Structured Prompt Formation:** The output is formatted as a structured prompt summarizing for each cluster:\n\n  * Cluster size (number of cells)\n  * Top marker genes (ranked)\n  * Other relevant features (e.g., mean expression of key genes, pseudotime, etc.)\n  * Any available prior annotations\n\n**Example:**\n\n> “Cluster 0: 1,050 cells, Top markers: \\[Cd14, Lyz, Fcer1g, ...];\n> \n> Cluster 1: 900 cells, Top markers: \\[Cd3d, Cd3e, Trac, ...];\n> ...\n> \n> Species: Mouse; Tissue: Liver; Timepoints: E9.5, E10.5 ...”\n\n**Generalization to All Tasks:**\n\n* For cell type annotation, clusters and their marker genes are provided.\n* For trajectory inference, the cluster connectivity (e.g., from py-Monocle) and key developmental genes are included.\n* For GRN prediction, the converter extracts top TF–target gene pairs (from tools like SCENIC) with supporting NES scores/motifs.\n\nThe resulting compressed prompt is small enough to fit in the LLM’s context window, yet detailed enough for meaningful biological reasoning. The LLM then reasons over this digest, proposes hypotheses, and iteratively calls tools for further evidence if needed."}, {"review_id": "3N5JNkuUk2", "reviewer": "Reviewer_82g2", "summary": "The core idea of ​​this paper is to use the big language model as a \"bioinformatics assistant\" so that it can understand the single-cell sequencing data by itself, and then output reliable biological analysis for you step by step. It did three things:\n\n1. **Data condensed into a \"easy-to-read version\"**\nThe gene expression matrix of millions of cells is first squeezed into JSON with key information such as cluster size, marker genes, and differentiation pathways by algorithms, so that the model can grasp the key points at once and not be overwhelmed by massive numbers.\n\n2. **\"Dialogue + tool\" closed loop**\nEvery step the model takes, it will first mutter in its mind \"This cluster of cells looks like XXX\", and then automatically call the bioinformatics tools (such as Scanpy, Monocle, SCENIC, etc.) to run it, and then correct its own conjecture after getting the results back - just like discussing with colleagues, proposing hypotheses while testing hypotheses.\n\n3. **Realistic evaluation routine - SCBENCH**\nComparison was made for three classic tasks:\n\n* Labeling cell clusters (cell type annotation)\n* Reconstructing cell development routes (developmental trajectory)\n* Predicting gene regulatory networks (TF-target gene relationships)\nThe results show that after applying this \"dialogue + tool\" process, the accuracy of various tasks can be increased, some graph structure errors can be greatly reduced, and the common pitfalls of the model can be seen.\n\nIn short, SCPILOT turns the \"black box script running\" bioinformatics analysis into a traceable, iterative, chat-like scientific workflow, allowing us to truly treat large language models as \"partners\" in bioinformatics, rather than robots that can only move literature and type code.", "questions": "As listed in weakness", "limitations": "As listed in weakness", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n* Quality\n\n  * Fairly thorough experimental comparison\n\n    * They compared SCPILOT against several common baselines on three classic tasks—cell type annotation, trajectory reconstruction, and gene regulatory network prediction—and used metrics like accuracy and graph error to show improvements. The empirical work looks pretty solid.\n    * The experiments cover a handful of public datasets, and they also released their code and data-processing scripts, which should help others reproduce the results.\n  * Modular design\n\n    * Breaking the system into “data summarization,” “model reasoning,” “tool calls,” and “result verification” keeps each part’s responsibility clear. That should make it easier to swap out or upgrade individual pieces down the line.\n\n* Clarity\n\n  * Intuitive flowcharts and example dialogues\n\n    * Using a flowchart (Figure 2) and multi-turn example conversations (Figure 3) to show the loop makes it easy to follow the overall approach.\n    * They even list API snippets and input/output JSON for each tool call, so you can pretty quickly see how to implement it.\n  * Logical structure\n\n    * The paper walks you through data preprocessing and summarization first, then the dialogue strategy and toolchain, and finally the evaluation results. The roadmap is easy to follow.\n\n* Significance\n\n  * Filling the “LLM + bioinformatics” gap\n\n    * Most work so far has focused either on pure dialogue or on bioinformatics pipelines; SCPILOT’s idea of combining both in single-cell analysis is quite fresh.\n    * Merging auditable tool calls with model reasoning could really help push explainable AI in the single-cell field.\n  * Potential community value\n\n    * They open-sourced the SCBENCH benchmark suite, giving others a starting point for comparison and hopefully spurring more research.\n\n* Originality\n\n  * Introducing a “dialogue + tools” paradigm\n\n    * While there are earlier examples of LLMs calling external APIs, this is one of the most complete end-to-end demos in single-cell sequencing, tying together community-favorite bioinformatics tools in a loop.\n  * Structured data summarization\n\n    * Compressing high-dimensional expression matrices into structured JSON summaries is a neat idea that could guide future LLM applications on large-scale biological data.\n\n## Weakness\n* * * Insufficient transparency of method details\n\n    * The empirical thresholds and hyperparameters used in the \"marker gene selection\" and \"branch point extraction\" phase of summary generation do not provide mathematical formulas or selection criteria, and no sensitivity analysis is performed on different settings;\n    \n    * The tool call link lacks intermediate verification (such as expression distribution diagrams, statistical tests or p-values), and it is impossible to judge whether the hypothesis verification is sufficient.\n    \n    2. Limitations of experimental design and evaluation\n    \n    * The baselines used for comparison are mostly older versions (such as Monocle2 and classic SCENIC), and there is no direct comparison with the latest LLM+bioinformatics work of the same period, which may overestimate the performance improvement;\n    \n    * The lack of error propagation experiments (such as artificially introducing summary or dialogue errors to observe the fluctuation of the final results) and uncertainty assessment (confidence interval or probability calibration) makes it difficult to measure the reliability of the overall process.\n    \n    3. Ambiguity in dialogue strategy and automation process\n    \n    * It is not clear when the dialogue stops (fixed rounds or dynamic decision) and how to balance \"exploration\" and \"verification\";\n    \n    * The method claims to be \"fully automatic and reproducible\", but it also mentions that \"manual review of dialogue output is required\" to prevent miscalling, which is inconsistent with the previous and subsequent statements.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 82g2 for their detailed and insightful summary. We are particularly grateful that the reviewer recognized the core contributions of our work on multiple levels:\n\n1.  **Paradigm Shift**: For highlighting scPilot's success in transforming \"black box\" analysis into a \"traceable, iterative, chat-like workflow,\" creating a \"bioinformatics assistant.\"\n2.  **Methodological Innovation**: For appreciating the originality of our \"complete end-to-end demo\" that unites community tools, the \"neat idea\" of structured data summarization, and the clarity of our \"modular design\" and \"intuitive flowcharts.\"\n3.  **Solid Evaluation and Community Value**: For acknowledging our \"fairly thorough experimental comparison\" as \"pretty solid\" and reproducible, and for noting the \"potential community value\" of the open-sourced `scBench` benchmark in spurring future research.\n\n\n***\n\n**(a) Clarification on empirical thresholds and hyperparameters with sensitivity analysis**\n\nWe thank the reviewer for this excellent suggestion. In response, we have conducted a new sensitivity analysis for key hyperparameter requested (**marker gene selection**) and are happy to clarify the criteria used for our experimental thresholds. We will add this analysis to the final manuscript to improve transparency.\n\n1. **Sensitivity Analysis on Marker Gene Selection ($K$)**\n\nFor cell-type annotation, a key hyperparameter is $K$, the number of top marker genes. We tested performance sensitivity on the PBMC3k dataset for $K=5$, 10 (our default), and 20.\n\n**Table 4. Annotation Accuracy (PBMC3k) vs. Number of Top Genes ($K$)**\n\n| **$K$** | **Model** | **Avg Accuracy** | **Variance** |\n| :--- | :--- | :--- | :--- |\n| 5 | o1 | 0.771 | 0.001 |\n| 5 | gpt-4o-mini | 0.500 | 0.016 |\n| **10 (Default)** | **o1** | **0.792** | **0.005** |\n| **10 (Default)** | **gpt-4o-mini** | **0.604** | **0.001** |\n| 20 | o1 | 0.750 | 0.016 |\n| 20 | gpt-4o-mini | 0.542 | 0.009 |\n\nPerformance peaks at our chosen $K=10$. More importantly, the strong results across different values demonstrate our approach is robust and not highly sensitive to this hyperparameter.\n\n2.  **Clarification on Trajectory Branch Point Extraction**\n\nFor trajectory branching, our method uses **no** tunable threshold for branch point extraction. Instead, we directly compare scPilot's predicted graph against the ground-truth lineage curated in the original publications (Pancreas, Liver, Neocortex). The evaluation is a direct comparison to an established biological standard, not a process dependent on an internal parameter.\n\n***\n\n**(b) Validation of intermediate tool calls**\n\nWe thank the reviewer for this insightful comment. To validate that scPilot's reasoning is grounded in its tool outputs, we performed perturbation studies on two critical components. We will add these results to the manuscript.\n\n1.  **Perturbation of Gene Ontology (GO) Database in GRN Prediction**\n\nTo test dependency on the GO database for Gene Regulatory Network (GRN) prediction, we shuffled its term associations with random noise.\n\n**Table 5. Impact of GO Perturbation on GRN Prediction (AUROC)**\n\n| Model | AUROC (Original) | AUROC (GO Shuffled) | $Δ$ AUROC |\n| :--- | :--- | :--- | :--- |\n| o1 | **0.873** | 0.813 | -0.060 |\n| GPT-4o | **0.800** | 0.710 | -0.090 |\n| gpt-4o-mini | **0.697** | 0.617 | -0.080 |\n\nCorrupting GO data significantly degrades AUROC (e.g., $p=0.044$ for gpt-4o-mini), confirming that scPilot's reasoning relies on accurate information from this intermediate step.\n\n2.  **Perturbation of py-Monocle Output in Trajectory Inference**\n\nWe tested dependency on `py-Monocle` by providing scPilot with a corrupted report (randomized cluster relationships and pseudotime order) for the liver dataset.\n\n**Table 6. Impact of Monocle Perturbation on Trajectory Inference (o1 model)**\n\n| Metric | Original Avg | Perturbed Avg | Performance Change |\n| :--- | :--- | :--- | :--- |\n| Jaccard | **1.000** | 0.933 | Worse |\n| GED-nx (10s) | **8.00** | 11.33 | Worse |\n| Spectral Distance | **0.567** | 0.593 | Worse |\n\nThe performance drop across metrics demonstrates that accurate outputs from tools like Monocle are critical for scPilot.\n\n**Summary**: These experiments prove scPilot's success is fundamentally dependent on the integrity of the data provided by the bioinformatics tools it integrates.\n\n***\n\n**(c) Additional experiments with new baselines (traditional bio tools and latest LLM+bioinformatics works)**\n\nWe thank the reviewer for this crucial suggestion. In response, we conducted extensive new experiments against state-of-the-art (SOTA) baselines.\n\n**Expanded Benchmarking with SOTA Baselines**\n\nWe added several new baselines:\n* **Updated Traditional Tool**: `Celltypist (1.7.1)`.\n* **SOTA LLM Methods**: `Biomni` (biomedical agent), `LLM4GRN` (GRN pipeline), and `BioGPT-Large` (foundation model).\n\nThe results confirm scPilot's superior performance holds against these newer, challenging baselines.\n\n**Table 7. New Baselines for Cell-Type Annotation**\n\n| Dataset | scPilot (o1) | Celltypist 1.7.1 (new) | Biomni (new) |\n| :--- | :--- | :--- | :--- |\n| **Liver** | **0.518** | 0.464 | 0.464 |\n| **PBMC** | **0.792** | 0.563 | 0.646 |\n| **Retina** | **0.728** | 0.368 | 0.570 |\n\n**Table 8. New Baseline for Trajectory Inference**\n\n| Model (on Liver Dataset) | Jaccard (Avg) | GED-nx (Avg) | Spectral Dist (Avg) |\n| :--- | :--- | :--- | :--- |\n| **scPilot (Gemini-2.5 Pro)** | 1.00 | **3.33** | **0.199** |\n| **Biomni (Gemini-2.5 Pro)** | 1.00 | 8.33 | 0.482 |\n\n**Table 9. New Baseline for GRN TF-Gene Prediction**\n\n| Model (on Stomach Dataset) | AUROC (Avg) |\n| :--- | :--- |\n| **scPilot (o1)** | **0.873** |\n| **LLM4GRN** | 0.727 |\n| **BioGPT-Large** | 0.66 |\n\nThis comprehensive comparison confirms scPilot's superior performance and the advantage of our omics-native reasoning approach. We will include all new results in the new version of the manuscript.\n\n***\n\n**(d) Additional Experiments in Error Propagation and Uncertainty Assessment**\n\nWe thank the reviewer for these suggestions. In response, we highlight existing work and provide new analyses on error propagation and uncertainty.\n\n**1. Error Propagation via Input Perturbation**\n\nWe highlight the **input perturbation experiments** from our main submission (Figure 4a), where we removed all contextual metadata (e.g., tissue type) from the input prompt for the PBMC3k annotation task.\n\n**Table 10. Results of Context Removal on Annotation Accuracy**\n\n| Model | Accuracy (Original) | Accuracy (No Context) | Performance Drop |\n| :--- | :--- | :--- | :--- |\n| o1 | **0.792** | 0.688 | **↓ 0.104** |\n| GPT-4o | **0.646** | 0.583 | **↓ 0.063** |\n| gpt-4o-mini | **0.604** | 0.416 | **↓ 0.188** |\n\nThis directly shows that removing key input context significantly drops performance, confirming rich metadata is critical for reliability.\n\n**2. Uncertainty and Reliability Assessment**\n\nWe performed a new reliability analysis on the GRN prediction task (Stomach dataset). We calculated 95% confidence intervals (CIs) from 10 trials, performed 1000-bootstrap resampling, and assessed calibration using Expected Calibration Error (ECE) and Brier scores.\n\n**Table 11. Uncertainty & Calibration Metrics for GRN Prediction**\n\n| Model | 95% CI (10 runs) | Bootstrap (±) | ECE | Brier Score |\n| :--- | :--- | :--- | :--- | :--- |\n| **o1** | **0.84–0.90** | **±0.05** | **0.05** | **0.14** |\n| **gpt-4o-mini** | 0.65–0.75 | ±0.10 | 0.12 | 0.20 |\n\nThe powerful `o1` model is not only more accurate but also more stable (tighter CI) and reliable (lower ECE and Brier scores). These analyses provide the requested quantitative assessment of scPilot's reliability, which we will add to the manuscript.\n\n***\n\n**(e) Stopping Criterion**\n\nWe thank the reviewer for this question. The dialogue stopping condition is pre-defined for each task to ensure reproducibility for our `scBench` benchmark. Please see (e) in our rebuttal to reviewer **RBqf**.\n\nThe balance between **\"exploration\" and \"verification\"** is managed by our structured reasoning loop:\n\n* **Exploration**: The LLM first generates broad hypotheses from summarized data (e.g., proposing potential cell types).\n* **Verification**: Subsequent tool calls gather evidence to verify these hypotheses (e.g., inspecting data to confirm cell types, as in Figure 3).\n\nThis `propose -> call tool -> analyze` cycle channels the LLM's ideas into data-grounded inquiries, preventing ungrounded tangents and ensuring a scientifically sound workflow.\n\n***\n\n**(f) Consistency and Clarity in Manual Intervention**\n\nWe thank the reviewer for this opportunity to clarify. We believe there may be a misunderstanding, as our paper does **not** use the term \"fully automatic\" nor state that \"manual review... is required.\" We clarify \"automatic and reproducible\" and the role of human oversight below.\n\n1.  **How scPilot is Automatic and Reproducible**\n    * **Automatic Execution**: \"Automatic\" means scPilot runs an entire workflow end-to-end **without human intervention during execution**.\n    * **Reproducible by Design**: The process is reproducible because prompts, tools, and stopping conditions are fixed. This ensures the same inputs produce the same outputs, a cornerstone of `scBench`.\n\n2.  **The Role of Human Auditing vs. Required Intervention**\n    The confusion may stem from our emphasis on **auditability**. A key strength is producing a transparent reasoning trace that a scientist *can* review for validation, much like reviewing a colleague's work. This is fundamentally different from requiring intervention *during* the process. The system runs automatically; transparency is a feature for trust and insight, not a bug requiring manual patches.\n\nIn summary: scPilot is **automatic in operation**, with outputs that are **auditable by design**, not dependent on intervention."}, {"review_id": "ByvI3kmeir", "reviewer": "Reviewer_mMx1", "summary": "The scPilot system provides human-like reasoning to orchestrate established single cell transcriptomics analysis methods to complete specific analysis tasks. The system is implemented using a problem-to-text converter to support natural language reasoning, use of Bio-Tool library of available functions, and an LLM planner.", "questions": "Can the authors better show the value of the reasoning system compared to similar recently published methods (e.g. C2S-Scale or CellAgent).\n\nIt would be nice to show an example of how the reasoning works, even in supplementary material. Currently, the main basis of the paper is the addition of reasoning to the model, but other than performance assessments, there is not exposition of the value of reasoning, for example in terms of interpretation, definition of workflows, manual checking of model decisions.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength (major): development of the scBench omics reasoning benchmark targeting 3 biologically relevant tasks.\n\nStrength: extends past established “chat with your data” systems for single cell genomics to cover multi-step reasoning directly on the omics data.\n\nStrength: Paper analyzes error modes and proposes augmentation strategies to address performance gaps.\n\nStrength: performance on diverse tasks is high\n\nWeakness (major): Value of reasoning is unclear (could be better explained). For example, direct use of O1 beats scPilot in cell type annotation.\n\nWeakness (major): based on the arguments in the related work section, more appropriate baselines should be selected from state-of-the-art tools that are cited (e.g. chat front ends or C2S-Scale).", "was_revised": true, "final_justification": "Increase based on authors emphasis in the rebuttal on the value of the reasoning trace and the author's intent to emphasize this more in the paper.", "rebuttal": "We sincerely thank Reviewer mMx1 for recognizing several core contributions of our work. We are particularly grateful for the acknowledgment of:\n\n1.  **A Major Foundational Contribution**: Highlighting the \"development of the scBench omics reasoning benchmark\" as a **major strength**.\n2.  **A Methodological Leap Forward**: Recognizing that our framework \"extends past established ‘chat with your data’ systems\" to enable **\"multi-step reasoning directly on the omics data.\"**\n3.  **Thorough Validation and High Performance**: Appreciating our comprehensive \"analysis of error modes\" and, importantly, that scPilot's \"performance on diverse tasks is **high**.\"\n\n---\n\n**(a) Value of Reasoning: Why scPilot (o1) performs worse sometimes**\n\nWe thank the reviewer for this critical question. It highlights the value of scPilot's reasoning beyond raw accuracy.\n\n1.  **Why 'Direct' Prompting Occasionally Outperforms scPilot**: As detailed in our **response (b) to Reviewer RBqf**, the rare cases where the 'Direct' method outperforms scPilot are on datasets with ambiguous markers (e.g., Liver). Here, scPilot's deep reasoning can cause it to \"overthink\" complex signals—a trait mirroring expert caution that can lower a strict accuracy score. The 'Direct' method, by forcing a single \"best guess,\" can be incidentally correct in such noisy conditions.\n\n2.  **The Scientific Value of \"Overthinking\"**: The value of scPilot's reasoning lies in its transparent, scientific process, not just the final label. Its \"overthinking\" provides crucial insights where direct methods fail. For example, on the Liver dataset:\n    * **Cluster 16 (Ambiguous Identity):** Faced with markers for multiple cell types (`hepatoblasts`, `hepatocytes`), scPilot's log noted the cluster \"may need finer analysis,\" correctly identifying the biological ambiguity instead of forcing a likely incorrect label.\n    * **Cluster 19 (Insufficient Evidence):** Lacking strong markers, scPilot flagged this cluster as un-annotatable and a \"prime candidate for subgrouping,\" correctly identifying the limits of the data and **preventing a confident error** where a direct method would guess.\n\nIn conclusion, scPilot's value is its transparent, cautious, and scientifically rigorous analysis. The reasoning trace reveals *why* a decision is made—or not made—which is invaluable for human trust and guiding future experiments, a key advantage over opaque methods.\n\n---\n\n**(b) Additional experiments to compare with appropriate STOA tools - check response (c) to the reviewer 82g2**\n\nWe thank the reviewer for this crucial suggestion. As detailed in our **response (c) to Reviewer 82g2**, we have conducted new experiments comparing scPilot against the latest generation of LLM-based bioinformatics tools.\n\n#### Expanded Comparison with State-of-the-Art (SOTA) Baselines\n\nWe expanded our evaluation to include new SOTA baselines: **Biomni** (a powerful LLM agent), **LLM4GRN** (a specialized pipeline), and **BioGPT** (a foundation model). Our results, detailed in the response to Reviewer 82g2, show that scPilot consistently outperforms these recent methods.\n\n#### Regarding C2S-Scale and CellAgent\n\nWe made significant efforts to include these specific tools but could not perform a fair benchmark due to technical limitations:\n\n1.  **C2S-Scale:** This tool is not fully open-source. The only public version is a 1B model, a class which our experiments show lacks the reasoning capability for `scBench`, making a comparison uninformative.\n2.  **CellAgent:** Evaluation was impossible as its web demo's data upload is non-functional and its source code is unavailable, preventing reproducible analysis.\n\nIn summary, we have broadened our comparisons to include relevant and accessible SOTA frameworks. We are committed to benchmarking against tools like C2S-Scale and CellAgent in `scBench` as they become openly available.\n\n---\n\n**(c) Value of reasoning: qualitative comparison with recent methods (Biomni)**\n\nWe thank the reviewer for this excellent question. While a direct benchmark against `C2S-Scale` and `CellAgent` was not feasible due to their limited accessibility (as explained above), we have conducted a detailed qualitative and quantitative comparison against **Biomni**, a powerful, state-of-the-art general-purpose biomedical LLM agent (technically, Biomni comes after our submission). This comparison clearly demonstrates the value of scPilot's specialized, reasoning-first approach. (Biomni has a very similar design philosophy to CellAgent, but much powerful).\n\n#### Contrasting Design: Reasoning-First vs. Tool-First\n\nThe core difference is their design philosophy. scPilot is a **specialized, reasoning-first agent**; Biomni is a **general-purpose, tool-first agent**.\n\n| Dimension | **scPilot (Reasoning-First)** | **Biomni (Tool-First)** |\n| :--- | :--- | :--- |\n| **Workflow** | Hypothesis formulation → Omics-native reasoning → Iterative refinement | Run tool → Keyword lookup → Hard-coded template |\n| **Error Handling** | Self-diagnoses biologically implausible steps and revises | Falls back to heuristics on Python errors; no biological validation |\n| **Output** | **Transparent Chain-of-Thought**: Explanations, confidence scores. | **Data Dump**: Dictionaries, tables, templated outputs. |\n\n#### The Impact on Scientific Accuracy\n\nThis design difference leads to significant accuracy gaps.\n* **Retina Annotation**: scPilot correctly distinguishes fine-grained subtypes (e.g., ON- vs. OFF-bipolar cells). Biomni makes clear errors, mislabeling Müller glia as amacrine cells.\n* **Liver Trajectory**: scPilot correctly identifies the Epiblast root and reconstructs faithful lineages. Biomni misplaces cell types (e.g., \"Cardiac muscle\") in the lineage, ignoring gene evidence.\n\n#### The Value in Efficiency and Cost\n\nscPilot's specialized approach is also vastly more efficient and cost-effective.\n\n| Task (vs. Biomni on Gemini-2.5 Pro) | scPilot Time | scPilot Cost | Biomni Cost | **Cost Ratio** |\n| :--- | :--- | :--- | :--- | :--- |\n| **Cell-type annotation** | 1–3 min | **$0.03** | $0.80–$1.00 | **~27–33x** |\n| **Trajectory inference** | 1–3 min | **$0.04** | $0.80–$1.00 | **~20–25x** |\n| **GRN prediction** | 3-5 min | **$0.12** | Unsuccessful | **N/A** |\n\nscPilot is up to **30x cheaper** and significantly faster because its curated toolchain and reasoning-first method avoid costly, broad tool searches. It also succeeds on the complex Gene Regulatory Network (GRN) prediction task where the general-purpose agent fails. This comparison proves that while general-purpose agents are flexible, scPilot's specialized reasoning delivers higher scientific fidelity at a fraction of the cost.\n\n---\n\n**(d) Additional Examples of Reasoning in Action**\n\nWe are happy to provide concrete examples of scPilot's reasoning, demonstrating its value in creating interpretable and verifiable workflows. The value lies not just in quantitative performance, but in its transparent, multi-step biological analysis.\n\n#### Multi-Gene Logic for Accurate and Interpretable Annotation\n\nInstead of simple marker matching, scPilot uses a `propose -> filter -> analyze` loop for nuanced interpretation, which we detail with examples in **Figure 3 and Supp. Figures 1-2**.\n\n* **How it Works:** scPilot proposes cell types from top markers, filters hypotheses against data (e.g., requires key marker presence), and then analyzes multi-gene combinations for a final call.\n* **Value:** This process correctly distinguishes NK cells from CD8 T-cells in the PBMC3k dataset by analyzing the *conjunction* of `NKG7`, `GNLY`, and `CD3D` expression. This explicit logic, recorded in the trace, allows scientists to verify the decision and is powerful enough to find rare cell types (<1% abundance).\n\n#### Self-Auditing for Deeper System-Level Understanding\n\nIn trajectory inference, scPilot uses external tool outputs to self-correct its own predictions.\n\n* **How it Works:** scPilot first constructs a lineage tree, then uses a diagnostic report from a tool like `py-Monocle` to perform a \"reconsideration\" step, auditing and repairing its initial tree.\n* **Value:** This self-correction significantly improves biological accuracy (graph-edit distance reduced by 6, spectral distance by 0.32), correcting the tree's root and repairing the canonical hepatic developmental ladder. The reasoning trace (detailed in **Figure 7** and **Supp. Table 6**) allows direct comparison of predicted branches against known biology.\n\n#### Contextual Reasoning for Tissue-Specific Predictions\n\nFor Gene Regulatory Network (GRN) prediction, scPilot integrates multiple data types for context-aware analysis.\n\n* **How it Works:** The framework assesses Transcription Factor (TF)-gene relationships by combining Gene Ontology, expression context, and known regulatory pathways.\n* **Value:** This enables true **tissue-specific reasoning**. For example, when predicting the `Klf4` → `Muc5ac` regulation, scPilot leverages knowledge that this interaction is key in stomach tissue, yielding a more accurate prediction than a generic model would make (see **Supp. Table 7**).\n\nThese examples show scPilot’s value is in its interpretable workflows, evidence-based self-correction, and nuanced biological interpretation, all exposed for verification. We will highlight these in the supplement, scuh as figure 3, Supp. Fig1, 2, Supp. Table 6, 7."}, {"review_id": "wBOvGuEz6o", "reviewer": "Reviewer_VK46", "summary": "The work converts omics data into step-by-step reasoning problems for LLMs to solve. The addition of the raw omics data creates interpretable analyses and yields better results than direct LLM use. The work also introduces scBench, which is a benchmark that evaluates biological reasoning.", "questions": "1. Are the tools within the bio-tool library crafted beforehand, or is the tool agent able to dynamically add new tools into the library? In other words, is scPilot able to be used for other bioinformatics related tasks, or would that require manual modification to the tool library?\n\n2. What are the data statistics for each of the datasets in the benchmark? I.e. number of data points, number of label types, and input feature distribution like lineage graph stats for the trajectory inference.\n\n3. Was there any additional filtering performed on the benchmark datasets? For the collected annotations such as in the cell type annotation task, was the only qualification that the label be manually annotated?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The problem domain is one not frequently investigated. The benchmark on its own is a noteworthy contribution. The use of this benchmark to evaluate the reasoning capability of LLMs on biological tasks is important in the effort to create scientific reasoning models.\n\nThe scPilot framework, which consists of a problem-to-text converter, tool library, and tool planner, is fairly standard in related domains. The novelty comes from the explicit requirement to force the LLM to record its claims in a transparent trace and in its application to this domain.\n\nThere is a lack of examples for the reasoning trace, which is one of the design features of scPilot. Figure 3 provides the reasoning for a PBMC3k annotation task, but it is unclear if this is the full reasoning trace, or if there are additional natural language claims/justifications.", "was_revised": true, "final_justification": "The authors have provided a detailed response and an example of the reasoning trace. Most of the concerns have been addressed, and I would like to keep my score as 4.", "rebuttal": "We sincerely thank Reviewer VK46 for their positive evaluation and for recognizing the key contributions of our work. We are especially grateful for the acknowledgment of:\n\n1.  **Our Novel Domain and Foundational Benchmark:** For highlighting that we are addressing a \"problem domain... not frequently investigated\" and that our `scBench` benchmark is a \"noteworthy contribution\" in the \"important... effort to create scientific reasoning models\".\n2.  **Our Core Methodological Innovation:** For identifying the specific novelty of our framework in its \"explicit requirement to force the LLM to record its claims in a transparent trace\" and its novel application to this domain.\n\n\n\n**(a) Novelty and Clarification on Reasoning Trace**\n\n\nWe thank you for pointing out the need for more clarity on the reasoning trace. Figure 3 is a high-level schematic, and you're correct that the full trace contains additional, detailed natural language claims and justifications for each step. The value of this transparent trace is best shown through concrete examples of the reasoning process in action.\n\nWe kindly refer the reviewer to previous response (d) to Reviewer RBqf, response (c) (d) to Reviewer mMx1.\n\n#### Cell-Type Annotation: Multi-Gene Logic**\n\nscPilot's reasoning trace for this task records a `propose -> filter -> analyze` workflow that allows for sophisticated biological interpretation.\n* **How it Works**: The trace first records the LLM's initial **claim** (proposing potential cell types). It then logs the **tool call** and its output (e.g., noting that the key marker for Plasma cells, `SDC1`, is absent in the data). Finally, the trace includes the LLM's explicit **justification** for the final annotation, such as using the *combination* of `NKG7`, `GNLY`, and `CD3D` expression to distinguish NK cells from CD8 T-cells—a nuance often missed by simpler methods.\n* **Value**: This trace provides a fully auditable record of the claims, evidence, and justifications, allowing a scientist to manually check and understand each decision.\n\n#### Trajectory Inference: Self-Auditing with Tool Diagnostics**\n\nThe reasoning trace for trajectory inference showcases an even more complex workflow involving self-correction.\n* **How it Works**: The trace logs the initial tree construction, the call to `py-Monocle` for a diagnostic report, and then the LLM's detailed justification for why it is making specific repairs to the tree (e.g., correcting the root, fixing the hepatic developmental ladder).\n* **Value**: This provides an interpretable record of how scPilot uses evidence from traditional tools to improve its own results, leading to a more accurate and biologically faithful output (e.g., reducing graph-edit distance by 6).\n\n#### GRN Prediction: Contextual Reasoning**\n\nFor GRN prediction, the reasoning trace captures how scPilot integrates multiple data types to make a context-aware decision.\n* **How it Works**: The trace shows the framework assessing a potential TF-gene relationship by combining functional data from the Gene Ontology, gene expression context from the input data, and information from known regulatory pathway databases.\n* **Value**: This allows for true **tissue-specific reasoning**, and the trace makes it clear how the final prediction was weighted by this contextual evidence, moving beyond generic associations.\n\nWe will add these detailed examples of the complete reasoning traces to the supplementary material to fully illustrate this key feature of our work.\n\n\n\n**(b) Flexibility and Extendibility of Bio-tool Library**\n\nThis is an excellent question about the framework's flexibility. To clarify, the tools within the **bio-tool library are curated beforehand**, and the agent does not dynamically add new tools during a run. As we explained in our response to Reviewer RBqf (c), this was a deliberate design choice to ensure our `scBench` benchmark is rigorous and reproducible, focusing on the LLM's reasoning capability rather than on tool discovery.\n\nRegarding its use for **other bioinformatics tasks**, scPilot is designed to be **modular and easily extendable**, though this does require manual modification by a developer. As noted by Reviewer 82g2, the modular design of the \"Problem-to-Text Converter,\" \"Bio-Tool Library,\" and \"LLM Planner\" keeps each part’s responsibility clear and facilitates extension.\n\nTo adapt scPilot for a new task, a developer would simply need to:\n1.  **Wrap the new bio-tool** in a simple Python function that formats its output into the standardized JSON that scPilot expects.\n2.  **Register the tool** by adding a description (e.g., a docstring) to the library so the LLM planner understands its purpose and how to use it.\n\nIn summary, while our `scBench` implementation uses a fixed toolset for evaluation integrity, the underlying scPilot framework is intentionally designed to be an extensible platform, allowing developers to apply omics-native reasoning to a wide range of future bioinformatics challenges.\n\n**(c\\) Detailed Benchmark Dataset Statistics - already in supplementary**\n\nThank you for this question. Detailed statistics for all datasets used in `scBench`—including the number of cells, number of cell types, and graph statistics for the trajectory inference tasks—are provided in Tables 2, 3, and 4 of our supplementary material.\n\n\n**(d) Data Filtering and Annotation Criteria**\n\nOur benchmark datasets were curated from high-quality, peer-reviewed studies without additional computational filtering to ensure they reflect established community standards.\n\n#### **Data Filtering**\n\nWe did **not** perform additional computational filtering on the datasets. Our selection process focused on using data directly from **high-quality, peer-reviewed publications** where the data were already processed and filtered according to the standards of the original study. Instead, all datasets for scBench were curated and processed through a uniform, four-step pipeline to ensure quality and consistency.\n\n1.  **Data Standardization and Compression**: Raw data from high-quality, published studies is first standardized and stored. It is then compressed into a uniform text format that the LLM can process.\n\n2.  **Ground-Truth Harmonization**: The ground-truth criteria are stricter than simply \"manually annotated.\" Annotations are mapped to standard databases like the Cell Ontology, and published lineage graphs are standardized to ensure a consistent, high-quality ground truth.\n\n3.  **Prompt Sketch Creation**: Task-specific converters generate compact JSON \"sketches\" that summarize key data (e.g., cluster statistics, marker genes) to fit within a single LLM prompt.\n\n4.  **Packaging**: Each task is bundled with its prompt sketch, data configuration, a programmatic grader, and instructions to allow any method to be run reproducibly.\n\nThis systematic pipeline ensures that `scBench` is built upon well-vetted data and expert-validated, harmonized ground truths.\n\n#### **Ground-Truth Annotation Criteria**\n\nOur criteria for ground-truth labels were stricter than simply being \"manually annotated\" and were tailored to each task:\n\n* **Cell-Type Annotation**: The labels were required to be **expert-curated and published** in a peer-reviewed study, ensuring they have been vetted by the scientific community.\n* **Trajectory Inference**: Datasets were required to include a **human-curated lineage tree** as ground truth within the original publication.\n* **GRN Prediction**: The ground truth is based on **experimentally-validated** TF-gene interactions from the trusted TRRUST v2 database.\n\nThis curation strategy ensures that `scBench` is a reliable and realistic testbed for evaluating scientific reasoning."}]

Author rebuttal:

Meta-review: The authors propose an LLM-driven set of workflows and tool use that automates analysis of single cell transcriptomic data. The method converts core problems such as cell type annotation into a natural language reasonign workflow that the LLM can manipulate. The paper also introduces a novel benchmark (scbench), that evaluate these capabilities across a suite of different tasks and using various different base LLMs. Discussion between authors and reviewers clarified the accessibility of the technique, the role and nature of the reasoning traces, and yielded important additional baseline comparisons to state of the art bioinformatics methods.
Decision: Accept (poster)