SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation
Authors: Jiahao Zhao
Feng Jiang
Shaowei Qin
Zhonghui Zhang
Junhao Liu
Guibing Guo
Hamid Alinejad-Rokny
Min Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Single-cell Foundation Models
Scientific AI Benchmark
Knowledge-augmented Evaluation
Primary area: datasets and benchmarks

Abstract: Large language models (LLMs) are increasingly applied in scientific research, offering new capabilities for knowledge discovery and reasoning. In single-cell biology, however, evaluation practices for both general and specialized LLMs remain inadequate: existing benchmarks are fragmented across tasks, adopt formats such as multiple-choice classification that diverge from real-world usage, and rely on metrics lacking interpretability and biological grounding.  We present **SC-ARENA**, a natural language evaluation framework tailored to single-cell foundation models. SC-ARENA formalizes a *virtual cell* abstraction that unifies evaluation targets by representing both intrinsic attributes and gene-level interactions. Within this paradigm, we define five natural language tasks — cell type annotation, captioning, generation, perturbation prediction, and scientific QA — that probe core reasoning capabilities in cellular biology.  To overcome the limitations of brittle string-matching metrics, we introduce **knowledge-augmented evaluation**, which incorporates external ontologies, marker databases, and scientific literature to support biologically faithful and interpretable judgments.  Experiments and analysis across both general-purpose and domain-specialized LLMs demonstrate that: (i) under the *Virtual Cell* unified evaluation paradigm, current models achieve uneven performance on biologically complex tasks, particularly those demanding mechanistic or causal understanding; and (ii) our knowledge-augmented evaluation framework ensures biological correctness, provides interpretable, evidence-grounded rationales, and achieves high discriminative capacity, overcoming the brittleness and opacity of conventional metrics. **SC-ARENA** thus provides a unified and interpretable framework for assessing LLMs in single-cell biology, pointing toward the development of biology-aligned, generalizable foundation models.

Published as a conference paper at ICLR 2026
SC-ARENA: A NATURAL LANGUAGE BENCHMARK FOR SINGLE-CELL REASONING WITH KNOWLEDGE- AUGMENTED EVALUATION
Jiahao 1.2.3 Feng Jiang". Shaowei Qin", Zhonghui Zhang", Junhao Liu",
Guibing Guol-t, Hamid Mlinejad-Rokns) Min Yang?.3f
Software College, Northeastern University, China
2 AAttificial Intelligence Research Institute, Shenzhen University of Advanced Technology, Shenzhen, China "Shenzhen Key Laboratory for High Performance Data Mining,
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China University California, Irvine, CA, USA "School of Biomedical Engineering, UNSW Sydney, Sydney, Australia
zhaoji ahao@ma neu edu. cn, jiangfeng@ suat-sz www.ssz..udccoo en, guogbl neu. edu. on, min. yange cn. cn
neu.
ABSTRACT
Large language models (LLMs) are increasingly applied in scientific research, offering new capabilities for knowledge discovery and reasoning. In single-cell biology, however, evaluation practices for both general and specialized LLMs re- main inadequate: existing benchmarks are fragmented across tasks, adopt formats such as multiple-choice classification that diverge from real-world usage, and rely on metrics lacking interpretability and biological grounding. We present SC- ARENA, a natural language evaluation framework tailored to single-cell foun- dation models. SC-ARENA formalizes a virtual cell abstraction that unifies eval- uation targets by representing both intrinsic attributes and gene-level interactions. Within this paradigm, we define five natural language tasks (cell type annotation, captioning, generation, perturbation prediction, and scientific QA) that probe core reasoning capabilities in cellular biology. To overcome the limitations of brittle string- matching metrics, we introduce nowwldge-auugeent incorporates external ontologies, marker databases, and scientific literature sup- port biologically faithful and interpretable judgments. Experiments and analysis across both general-pen perepose and domain-speccille under the Virtual Cell unified evaluation paradigm, current models achieve un- even performance on biologically complex tasks, particularly those demanding mechanistic or causal understanding; and (ii) our knowledge-augmentee evalua- tion framework ensures biological correctness, provides interpretable, evidence- grounded rationales, and achieves high discriminative capacity, overcoming the brittleness and opacity of conventional metrics. SC-Arena thus provides a unified and interpretable framework for assessing L.LMs in single-cell biology, pointing toward the development of biology-aligned, generalizable foundation models. Our code is available at: https//ggiibbboommSSUUTTAAA////C
evaluation, which
L.MSs demonstrate that (i)
INTRODUCTION
Large language models (LLMs)) are increasingly being applied biological research, enabling knowledge extraction (Garcia et al. 2024), reasoning (Gong et al.) 2023), and hypothesis generation (Abdel-Rehimg et al. 2025 in diverse modalities. In cellular biology, researchers are actively inves-
'Coresponding authors.

---

Published as a conference paper at ICLR 2026
tigating how to leverage LLMs to integrate high-dimensional molecular data with mechanistic un- derstanding. which is crucial for tasks such as cell type annotation (Wu & Tang 2025), y perturbation analysis (Istrate et al. 2024). and mechanistic question-answer answering (Wang et al.] 2024p. Collectively, these efforts reflect the aspiration to construct a virtual cell (Roohani et al. 2025), a computational model that enables in silico analyses through simulations, thereby accelerating the scientific dis- covery process. However, realizing this vision requires the development of fair and comprehensive benchmarks that provide interpretable, task- gosk-grounded, and biologically faithful evaluations, capable of accurately assessing LLMs' ability to interpret biological signals and mimic single-cell behaviors beyond generic NLP metrics.
Existing benchmarks for comprchensively evaluating LLMs' ability to process heterogeneous single-cell biological data remain limited. Most focus on narrow tasks (e.g., cell type annotation (Yuan et al.) 2024)) without assessing whether models acquire holistic understanding of cellular identity and dynamics. Broader scientific QA benchmarks, such as SciBench (Wang et al., 2023) and PubMedQA (Jin et al. 2019), probe reasoning but remain domain- and fail to capture the demands of single-cell analysis. More recent efforts, such as CELLVERSE (Zhang et al. 2025) and SOAR (Liu al. 2024), extend evaluation to multi-omics tasks, yet still lack systematic coverage of reasoning and generative capabilities. Consequently, a principled framework still absent for de- termining whether LLMS can operate reliably across heterogeneous biological tasks while faithfully capturing biological attributes, dynamics, and reasoning.
Inspired by recent progress toward constructing virtual cells, which require models account for cellular states (i.e., attributes) and generate corresponding responses (i.e., actions) to environmental cues, we introduce SC-ARENA, a benchmark that evaluates LLMs through the abstraction of Virtual Cell within an arena-style evaluation setting. This paradigm reconceptualizes evaluation as a selection process: Can an LLM serve as a virtual cell by faithfully capturing biological attributes, dynamics, and reasoning? Concretely, we define minimal requirements for a virtual cell and design five representative natural language tasks: captioning. cell type annotation, cell generation, scientific QA, and perturbation prediction that jointly probe static properties and dynamic behaviors. In these tasks, unlike prior benchmarks that rely on constrained multiple-choice formats, we adopt open- ended QA to better reflect practical use cases and capture reasoning depth. Regarding evaluation, standard metrics such as accuracy or BLEU, widely used in previous works (Liu et al. 2024), cannot capture these aspects. We therefore adopt LLM-as-a-judge (Gu et al. 2022) while mitigating bias through a knowledge-augmented framework inspired by Eval-RAG et al. (2023) grounded in external databases and ontologies, resulting in evaluations that are interpretable, reproducible, and biologically faithful.
Experiments across multiple state-of-the-arr LLMs demonstrate the utility of SC-ARENA. We find that (i) models perform well on text-aligned tasks such as captioning but struggle on perturbation prediction and mechanistic QA, revealing gaps in causal reasoning; (ii) knowledge-augmentee eval- uation correlates more strongly with expert judgments than string- based metrics; and (iii) even the strongest general-purpose purpose LLMs fail to consistently simulate the attributes and methods of the Vir- tual Cell, underscoring the need for domain-speciaaized approaches. Our contributions are threefold:
Virtual Cell abstraction. We introduce the Virtual Cell as a unified evaluation object for single-cell reasoning, inspired by object-oriented modeling. This abstraction jointly encodes cellular attributes (identity, state) and actions (responses, interactions), enabling principled and extensible framework for evaluation. Starting from five representative tasks, SC-ARENA systematically probes both static and dynamic aspects of cellular biology.
Natural language and mmowledge-augmented evaluation. We reformulate single-cell bench- marks into natural language QA tasks, moving beyond rigid classification or multiple-choice for- mats to better reflect real-world usage. To ensure biological fidelity and interpretability, we design evaluation scheme that integrates ontologies, marker databases, and literature evidence, providing Nomain-groundee scoring and explanatory rationales.
Comprehensive empirical study. We compare several popular general-purposs purpose and single-cell specialized LLMs under SC-ARENA. The results show a strong alignment between our knowledge- augmented evaluator and expert judgments, while revealing systematic gaps, particularly in mecha- nistic reasoning, that highlight directions for future biology-aligned foundation models.
2

---

Published as a conference paper at ICLR 2026
Table Comparison of existing single-cell LLM evaluation frameworks.
Framework
Evaluation Paradigm
Evaluation Format
Evaluation Metrics
C2S-Scale
Coll Sentenee
Prompted Completion
Lexical & Statistical Matching
Cell-ol
Reasoning Agent
Constrained Selection
Classification Matching
SOAR
Single-Task Agent
QA-hased
Surface String Matching
Classification
CELIVERSE
Multi-omies Cell Sentence
Multiple Choice
Classification Matching
Questions
SC-ARENA (Ours)
Virtual Cell
Open-Ended Natural Language QA
Grooledge-Grounded
Matching
RELATED WORK
2.1
SINGLE-CELL MODELING APPROACHES
The evolution of single-cell modeling has progressed from embedding- based architectures to natu- ral language-driven reasoning. Early efforts applied Transformer architectures to large-scale single- cell RNA-seq corpora, encoding expression profiles into latent embeddings for downstream tasks. Geneformer (Theodoris et al. 2023) and scFoundation (Hao et al. 2024) trained transformer mod- els from scratch on tens of millions of cells, whereas scBERT (Yang et al. 2022) and scGPT (Cuil jet al. 2024) adapted existing NLP architectures- BERT and GPT-2, respectively- robust cell type annotation and perturbation prediction. CellFM (Zeng et al. 2025) further expanded model ca- pacity, training 800M parameters on 100M cells to improve robustness and generalization. Beyond purely embedding-haie -hasee paradigms, hybrid approaches such as scGenePT (Istrate al. 2024) and Cellliama (Choi et al. 2024) integrated textual biological knowledge, marking an early transition toward language-driven modeling in single-cell research.
Building on the demonstrated effectiveness of general LLMs in cell annotation (e.g., GPTCell-type (Hou & Ji 2024)), recent studies have reformulated single-cell modeling directly in natural lan- guage to enhance interpretability and reasoning. Cell2Sentence (Levine et al. 2024) and its ex- tension Cell2Sentence-scll (Rizvi al.) 2025) introduced the concept of "cell sentences," con- verting gene expression profiles into textual representations and thereby laying the foundation for language-basee modeling. More recent efforts, such as CellReasoner (Cao al.] 2025) and Cell-ol (Fang et al. 2025), further emphasize reasoning, incorporating mechanisms for structured inference in single-cell tasks. However, as summarized in Table [ these approaches operate under distinct paradigms that limit holistic assessment: C2S-Scale focuses static encoding and scaling laws, evaluating prompted completion via lexical and statistical metrics (e.g.. BERTSScore, Gene Over- lap); while Cell-ol functions reasoning agent focusing batch-level logical consistency, yet is constrained to selection-based formats (requiring candidate lists) rather than open-ended genera- tion. Consequently, they lack unified evaluation framework that systematically assesses biological reasoning capabilities through the lens of a dynamic Virtual Cell.
2.2 SINGLE-CELL BENCHMARKS
Several scientific QA benchmarks have been developed to evaluate LLMs in biomedical and STEM domains. General-purposs resources such as SciBench (Wang et al.) 2023) and PubMedQA (Jin af. 2019) adopt natural language QA formats and provide broad coverage of biomedical reason- ing. However, they remain largely agnostic single-cell contexts and lack the mechanistic depth required for cellular-level evaluation.
To enable more precise assessment in single-cell biology, specialized benchmarks have recently emerged. CELLVERSE (Zhang et al., 2025) focuses on unifying cross-modaal modslitt data via multi- omics cell sentences. but relies multiple-chiice questions (MCQ), often converted from open queries to maintain stability, which constrains reasoning and does not reflect real-world usage. Sim- ilarly, SOAR (Liu al. 2024) evaluates models single-task agents focusing only on cell type
3

---

Published as a conference paper at ICLR 2026
(o) Participate
(b) Formal Examination
(c) Knowledge Augnented Evolustion
Virial Gol
Afriluus
Verlod
Anadi
Svalutiin
Nome
O plns
Truth Colterigg
1 on f
Talefe belicto Prolcter
Noci Aboue
②
Exteno Gurbay
Quar
of
Information
SC- Data
SC- Knowledge
Leader bacr
del Servece
Y Everdd
Pupited Articies
OhtolegydCel Marker
GELLYGENE
NCBEBunsFrardG
Pubaled Articles
Soura
Tulayy
Seyle coresiin
Figure 1: Overview of the SC-ARENA framework.
notation (targeting specific labels) and relies on string matching metrics (e.g.. BLEU, exact match), reducing complex reasoning to surface-level lexical overlap and offering little interpretability. Cru- cially, as detailed in Table [ these benchmarks utilize standard classification string-matching metrics, lacking biological grounding in their evaluation methodology. and failing to assess whether models truly understand cellular mechanisms or merely memorize superficial patterns.
These limitations highlight the need for an evaluation framework that unifies assessment targets via Virtual Cell (simulating attributes and methods), employs open-ended natural language reasoning without candidate lists, and incorporates domain-specific knowledge to ensure biological fidelity-a gap that our SC-ARENA framework addresses.
3 THE SC-ARENA EVALUATION FRAMEWORK
These limitations motivate the design of SC-ARENA, which builds on the virtual cell abstraction to unify evaluation targets and employs knowledge- knowledge-grounded grounded QA for interpretable and biologically faithful assessment, as shown in Figure The benchmark consists of three components: (i) fram- ing the participant model as a virtual cell, (ii) constructing formal examination comprising five representative tasks, and (iii) applying our mooleddg-aaugmente evaluation framework.
3.1 KNOWLEDGE CELL CLASS: DEFINING THE PARTICIPANT AS A VIRTUAL CELL
To evaluate whether LLMs acquire biologically grounded knowledge rather than memorizing perficial patterns, we introduce the notion of a Virtual Cell. A virtual cell serves as an abstraction of biological entities, defined an instance of a Knowledge Cell class that encapsulates both static attributes and dynamic methods, which are defined as below:
Attributes Attributes represent the intrinsic identity and state of a cell, capturing multimodal bi- ological information: (i) Exppression-beeee structured "cell sentences" (Rizvi et al. 2025). (ii) Text-hased descriptions of morphology, function, localization, and role, curated from literature and databases, and (iii) Ontology-basee hierarchical annotations from resources such as the Cell Ontology (CL).
features derived from scRNA-seg profiles, encoded as
Methods Methods represent the extrinsic dynamics of a cell, modeling its interactions with the environment: (i) Cell -> Environment processes, including cytokine secretion, signaling, antigen presentation, and immune activation, and (ii) Environment > Cell responses, such as transcrip- tional changes under perturbations (e.g., drug treatment or gene knockout).
Leveraging this abstraction, a model that coherently represents both attributes and methods qualifies candidate Virtual Cell LLM, it demonstrates the capacity simulate both static identity
4

---

Published as a conference paper at ICLR 2026
and dynamic behavior within unified Knowledge Cell class. This design establishes principled evaluation unit that integrates heterogeneous tasks into a single framework.
3.2 MULTI-TASK BENCIMARK WITH FORMAL EXAMINATION
Building on this definition, SC-ARENA is designed as multi-task benchmark that operationalizes the evaluation of Virtual Cell LLMs across complementary perspectives. Each task probes a different mapping between modalities or reasoning direction within the Knowledge Cell class, and the tasks are defined as follows:
Cell Type Annotation (Expression Ontology): Assign ontology-grounden labels expression profiles. Given a cell sentence, the LLM predicts the corresponding ontology-basee cell type label.
Cell Captioning (Expression - Language): Generate natural language descriptions from cell sentences. Given a cell sentence, the LLM produces a natural language description of the biological state, testing interpretability and the verbalization of transcriptomic patterns.
Cell Generation (Ontology/I Language -> Expression): Synthesize plausible expression profiles from cell type descriptions ontology terms. Given cell type name, the LLM generates plausible cell sentence, assessing its ability to produce molecular profiles consistent with semantic labels.
Perturbation Prediction (Environment Cell):: Predict expression changes induced by perturba- tions given baseline profiles and perturbation signals. The evaluation requires the LLM to (i) predict up- and nown-regulated genes and (ii) generate the post-perturbation cell sentence.
Scientific QA (Cell + Environment): Answer mechanistic questions regarding cellular functions and intercellular interactions. Questions are derived from scientific literature, requiring the LLM to extract relevant knowledge from prior studies and provide evidence-based explanations.
Together, these tasks assess (i) bidirectional translation between molecular data and semantic de- scriptions, and (ii) reasoning over causal interactions between cells and their environments. By jointly covering static identity, dynamic behavior, and cross-modal reasoning, SC-ARENA provides holistic testbed for measuring whether LLMs achieve biologically meaningful understanding of cellular systems.
3.3 KNOWLEDGE-AUCMENTE EVALUATION
We found conventional NLP metrics (e.g., BLEU, ROUGE, BERTScore) fail to capture biological fidelity in our preliminary experiments (see Appendix A.4 for details). To address this, we intro- duce knowledge-augmennee L.L---s-a-judge framework, inspired by Eval-RAG (Ryu et al.) 2023). which improves judging reliability by conditioning evaluation on retrieved context. Unlike conventional LLL--asjuude approaches (Gu al. (2024) that rely only the prompt and model out- put, our evaluator explicitly integrates curated external resources, including Cell Ontology, UniProt, Gene Ontology, CellMarker, and peer-reviewed literature. Grounding these verifiable references enables the evaluation to capture semantic coherence, penalize biologically implausible outputs, and provide interpretable feedback. Formally, each evaluation instance is represented as
I (g, r, K,9),
where q denotes the task prompt, r the model response, K the retrieved external knowledge, and the ground-truth answer. An evaluator LLM E maps this tuple to
E(Z) [0, 100].
implemented discrete rating in [0.5] linearly rescaled 100). Conditioning on both K and
g allows the evaluator to accommodate linguistic variability, penalize factual errors against trusted references, and assign partial credit to semantically related predictions, yielding more faithful and interpretable evaluation than either string mathhing metrics ungrounded LLM-as-judge baselines.
Our framework prioritizes biological fidelity over speculative novelty: high scores are awarded only to predictions consistent with experimentally validated facts. This design ensures evaluation mea- sures models' alignment with biological reality rather than their capacity for fluent but unsupported claims. The framework remains robust across different knowledge sources by anchoring judgments in curated, consensus- evidence rather than dynamic retrieval, maintaining stability even when underlying databases (e.g., UniProt, NCBI, CellMarker) substituted.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 BENCIIMARK DATASET CONSTRUCTION
All benchmark data sets derived from publicly available high-quality single-cell resources. A detailed summary of these datasets is shown in Table 2
Table 2: Summary of benchmark datasets in SC-ARENA.
Task
Data Source
#Samples
Format (Input + Output)
CTA
CELLxGENE
608
cell sentence (expression) - ontology label (CL)
CC
CELLXGENE
608
cell sentence (expression) - natural- language caption
CG
CELLxGENE
608
Ontology/cell-tyr name + cell sentence (expression)
PP
Norman; Adamson
138
Control & perturbed cell sentences + perturbation spec (i) up/down DEGs; (ii) pesturettubbaiion cell sentence
SQA
PubMed
254
Question -> natural-language answer + evidence rationale
The first three tasks-Cell Type Annotation (CTA), Cell Captioning (CC), and Cell Generation (CG) are constructed from a shared subset of 608 representative profile sampled from the CZ CELLAGENE Discover portal (Program et al.) 2025). Each cell's gene expression profile is con- verted into a natural language "cell sentence." " providing a unified representation across tasks. This design ensures consistency and comparability while establishing a closed-loop interplay among ex- pression profiles, ontological labels, and natural language: models must identify cell identity, ver- balize biological states, and generate plausible single-cell expression profiles.
Perturbation Prediction (PP) is compiled from two large-scale perturbation studies (Norman (Nor- man et al. 2019), Adamson (Adamson et al.) 2016)), covering 138 genetic interventions. For each perturbation, we compute the mean expression profile of control (pre-perturbation) and perturbed (post-perturbation) cells, convert both into "cell sentences," and extract differentially expressed genes (DEGs) as ground-truth up/down-regulated gene sets.
Finally, Scientific QA (SQA) is curated from 100 PubMed articles focused on human genes and cellular biology, yielding 254 questions paired with reference answers and supporting evidence. Each question targets mechanistic reasoning, cell-environment interactions, and single-cell biol- ogy concepts, requiring application of the paper's 's findings rather than simple recall. Following semi-automatic pipelines such as EasyDataset (Miao al., (025). we streamline retrieval, question generation, and evidence linking, enabling interpretable evaluation of functional and mechanistic knowledge.
To evaluate the potential data leakage risk in our dataset, we conducted verification using the C2S- scale series models in CTA, which were trained on source-related but differently formatted datasets. Despite this theoretical overlap, these models show significantly lower character-level similarity to our samples than general-purgy purposs models, while achieving higher task accuracy. It indicates that the models did not memorize our data but instead learned tesk-relevant knowledge, suggesting that the leakage risk in our dataset construction is minimal. A detailed analysis is provided Appendix A.6
4.2 EXTERNAL KNOWLEDGE FOR EVALUATION
To ensure that evaluation reflects biological faithfulness rather than superficial lexical overlap, we ground each task in curated external resources:
Cell Type Annotation: We use hierarchical paths from the Cell Ontology (CL) (Dichl et al. 2016), which define standardized taxonomies of cell identity. By situating predicted types within this ontol- ogy, we can measure semantic similarity between model outputs and gold- goan-st labels, rewarding predictions that are close in the hierarchy even if not exact matches.
Cell Captioning: For evaluating natural language descriptions, we incorporate official definitions of target and ancestral cell types from CL. These definitions provide reference descriptions of mor-

---

Published as a conference paper at ICLR 2026
phology, function, and localization, enabling assessment of whether generated captions capture the essential biological attributes and avoid omissions.
Cell Generation: We validate generated cell sentences against cell-type-specific marker genes rated in the CellMarker database (Zhang et al., 2019). Marker genes act as widely accepted gold standards for distinguishing cell identities, making them ideal reference for judging whether syn- thetic profiles preserve biological distinctiveness.
Perturbation Prediction: To assess the plausibility of predicted differentially expressed genes (DEGs), we integrate functional annotations from NCBI (O'Leary et al. 2016).
UniProt (UniProt Consortium, 2018), and the Gene Ontology (GO) (Carbon al.) [022T These resources capture gene-level functions, pathways, and interactions. allowing us to verify whether predicted perturbation responses align with known biological mechanisms.
Scientific QA: For factual verification, we extract supporting abstracts and key excerpts from the original PubMed articles used construct the questions. This provides ground-trun context for checking whether model answers are both scientifically accurate and widdecc-suppoorte
4.3 EXPERIMENT SETUP
We evaluate both general-purpose and Nomain-speccalliee LLMs on SC-ARENA.
General-
purpose models include the Qwen2.5 (Yang al.) (024) and Qwen3 Yang al. 2025 families
across multiple scales, as well as GPT-4o, DeepSeek-Ri (Guo et al., 2025). ). and Kimi-K2 TTam
al.
2025). Domain-specializze models include C2S-Scale (Rizvi et al. 2025), seGenePT (strate
et al. 2024), the reproduced scGPT (Cui et al. [024). and the reasoning-orient
Cell-O1 (Fng
al. objective.
2025). Each domain-specific checkpoint is evaluated only on tasks aligned with its fine-tuning
Each instance standardized into a cell sentence representation. General-purpose LLMs receive task-specific natural language query, while domainnspecializedd models follow their original in- ference protocols. All outputs are converted into a unified response schema before evaluation to ensure consistency. Additional details the unified formatting, fairness controls, and cross-moded standardization are provided in Appendix A.5
Table 3: Performance of different models across five tasks (with Total Score). Bold 1st place in column, Underline 2nd place in column. Non-numeric entries (-) are excluded from ranking.
Model
CTA
CG
CC
PP
SQA
Total
General-purpose Models
Qwen2.5-7B
12.61
45.98
51.05
28.84
64.09
202.57
Qwen2.5-14B
25.89
51.74
56.05
34.78
66.37
236.06
Qwen2.5-32B
26.78
50.95
55.46
36.23
66.77
238.06
Qwen3-88 BBB
21.45
50.53
57.20
37.39
72.83
254.45
Qwen3-144 14B
29.17
15.50
60.88
32.89
72.20
210.64
Qwen3-32B
31.35
55.39
62.69
37.54
65.03
252.00
Qwen3-235B
37.47
52.76
62.03
35.94
74.48
262.68
GPT-4o
36.29
59.70
63.02
37.24
67.56
263.81
DeepSeek-RI
40.81
62.24
66.51
36.23
70.87
276.66
Kimi-K2
40.00
63.04
67.89
37.10
69.13
277.16
Domain-speccalized Models
C2S- ASS-yyha-4410m((ellyyy pyee--ecicp-eriiioo
47.34
C2S 12S-Pythia-410ml seell-eeneattin)
20.30
41.68
18.55
Cell-ol
34.11
43.91
67.89
24.20
64.09
234.20
ScGPT
21.55
SCGemePTINCCBH+mmPPo))) ScCienPPT(GO-all)
24.13
26.03

---

Published as a conference paper at ICLR 2026
4.4 BENCHMARKING RESULTS
Overall Performance. As shown in T no system reaches the level of a reliable "virtual cell." Even the best-perforning general models, Kimi-K2 (277.2) and DeepSeek-RI (276.7), fall short of a normalized passing threshold (5*60). This highlights both the inherent difficulty of single-cell reasoning and the considcrable headroom for improvement.
Effect of Model Scale and Iteration. Scaling and iteration bring consistent gains. Within the Qwen family, performance rises from 202.6 (Qwen2.5-7B) to 262.7 (Qwen3-235B), a nearly 60-point improvement. Iterative upgrades also matter: Qwen3 models outperform Qwen2.5 at comparable scales, and frontier systems like GPT-4o (263.8) and Kimi-K2 (277.2) surpass earlier baselines. However, no model is uniformly strong, Kimi-K2 excels in generation (63.0) and captioning (67.9), while Qwen3-32B narrowly leads in perturbation prediction (37.5). Scaling enhances fluency and coverage but does not resolve mechanistic reasoning. A task-level visualization using radar plots, along with further analysis, is provided in Appendix A.8.I
Task-wise Differences. Performance varies sharply across tasks. Captioning (up to 67.9 with Kimi- K2) and scientific QA (74.5 with Qwen3-235B) reach the 60-70 range, while cell type annotation lags around 40 (best: DeepSeek-R1 at 40.8) and perturbation prediction remains below 38 (best: Qwen3-32B at 37.5). This asymmetry reflects the "fluent but not faithful" gap: models generate coherent text yet struggle with ontological precision and causal inference. A radar plot providing task-level visualization of these disparities is shown in Appendix A.5
A.5
Domain-specialized systems show complementary strengths. C2S-Pythia (410M, cell-type prediction) reaches 47.3 on annotation, outperforming even GPT-4o (36.3) and Qwen3-235B (37.5), despite being orders of magnitude smaller. By contrast, GenePT variants achieve only 21-26 on perturbation prediction, far below ceneral-purposs leaders, illustrating that specialization is highly task desendennd and not uniformly beneficial.
Domain-Specific
General Models.
Case:
"al_ape""
Rational:
Oveaal
Step
thigher cloer pretitioo)
The prodicions sutllly mate prrprta of wuth rnclute anventied ond doge repultted gones (ngg
Supl:
Belogial plcatsiity #t repiited
ogicnl
contect crione mutton und the dirett target beeucto corvecly prodictd dover- regulaed.
Mept::
of
coprates the care sn regulane sandd grnes bux tidk preditin assorsring persoslert sigethrrntly
Supp::
-24
Supportiion
Condatie::
pnssedjemeen brondaa inclivathy coring the sedeceed exprr onlan clanges. predicing cow reg and peoto-rin fing moderns eakkttier coore of').
Coeduaion
Prediction Score
(a) The data points aggregated across all models in the CTA task.
(b) Example of scoring responses, produced by the evaluator ing external knowledge in the PP task.
Figure 2: (a) Relationship between prediction score and ontology distance (Spearman p 0.6212, p <0.001); (b) Example scoring responses using external knowledge.
5 ANALYSIS
5.1 BIOLUGICAL CORRECTNESS: ONTOLOGY- GROUNDED VALIDATION VIA CELL TYPE
ANNOTATION
To assess whether the nhowledge-augmente evaluator assigns biologically coherent scores, alyze the task of cell type annotation, which naturally leverages the hicrarchical structure of the Cell Ontology (CL). For each prediction, both the predicted and reference cell types are mapped to CL identifiers via the Ontology Lookup Service (OLS). We then compute their shortest-path distance d; within the CL hierarchy, using it proxy for biological relatedness.
We quantify alignment between evaluator scores and ontology distance by computing the Spearman rank correlation (Spearman 1904 Kendall 1970) between the evaluator's score 8; and the negative
8

---

Published as a conference paper at ICLR 2026
ontological distance -d,, shown in Figure 2a The correlation is strongly positive (p 0.6212, p < 0.001), indicating that predictions closer to the ground-truth type in the ontology consistently receive higher scores. This demonstrates that SC-ARENA'S 's scoring scheme faithfully aligns with biological hierarchy, successfully operationalizing and validating the knowledge-augmentt ation paradigm proposed in this work. A concrete example illustrating this scoring mechanism is provided in Appendix A.9) Beyond this CTA-focused analysis, we comprehensively validate the cross-task reliability of our soooledge-augmente evaluator across all five SC-ARENA tasks in Ap- pendix A.7 demonstrating its consistent alignment with biological ground truth and expert judgment despite handling divergent output formats and reasoning demands.
evalu-
5.2 INTERPRETABILITY: STRUCTURED, ANOWLEDGE-GROUTT
RATIONALES
A central advantage of SC-ARENA's natural language- based design is its transparent and inter- pretable evaluation: each score is accompanied by structured rationales grounded in biological knowledge, rather than presented as an opaque number.
As shown in Figure 2h in the perturbation prediction task the evaluator LLM generates biologically informed explanations that explicitly connect its scoring decisions domain wowwledge-ooerriee gene function (e.g., VIM in stress response) and perturbation mechanism ARIDIA in chromatin remodeling).
This design transforms evaluation from a black-box judgment into an auditable and instructive pro- cess: it reveals why predictions succeed or fail, and turns evaluation into teaching signal for it- erative model refinement. It highlights that SC-ARENA not only measures performance but also explains it, enabling systematic error analysis and distinctive interpretability.
5.3 DISCRIMINATIVE CAPACITY: DISTINGUISHING BIOLOGICALLY MEANINGFUL PREDICTIONS
Beyond correctness and interpretabilityy an effective evaluation framework must also demonstrate discriminative capacity: the ability to distinguish models and outputs according their biological plausibility. Traditional NLP metrics fall short in this regard.
For each task, we computed the similarity between model outputs and ground-truth references using several widely adopted NLP metrics, including BERTScore (Zhang et al. 2020), BLEU (Papineni al. 2002). ROUGE (Lin 2004), and METÉOR (Banerjee Lavie 2005). with detailed results provided in Appendix A.4 However, the results reveal clear limitations:: the scores are either uni-
formly close across models, offering little discrininative power, or near zero, failing to capture meaningful differences in biological reasoning quality.
In contrast, SC-ARENA achieves fine-grained discrimination by integrating structured rationales with domain knowledge to evaluate prediction plausibility and relative model strength. For example, in the cell type annotation task, SC-ARENA leverages cell ontology as external knowledge to capture differences in prediction depth. As shown in Appendix A.8.2] larger models tend to generate deeper, more specific cell type predictions, which align with their overall benchmark performance, enhancing the framework's discriminative power.
DISCUSSION
6.1 BRIDGING THE GAP: FROM FLUENT FAITHFUL BIOLOGICAL LANGUAGE MODEL.S
Our results in SC-ARENA reveal a clear dissociation between linguistic fluency and biological faith- fulness in current LLMss. As shown in Table 3. general-purpos models consistently outperform domain-specialized ones on open-ended generation tasks such as Cell Captioning, demonstrating strong surface-level fluency. However, this advantage vanishes on tasks requiring ontological pre- cision causal accuracy: in Cell Type Annotation, most general models are outperformed by spe- cialized counterparts, and performance on Perturbation Prediction remains universally poor across all models. Together, these findings expose systemic "fluent but not faithful" gap: models may speak biology convincingly, yet fail to reason with the precision, hierarchy, and causality that define biological understanding.

---

Published as a conference paper at ICLR 2026
To bridge this gap, our findings indicate that modeling must prioritize structured resources (e.g., tologies, pathways) to encode biological logic. Regarding evaluation, SC-ARENA establishes that benchmarks must (1) assess granularity over exact matching, and (2) require auditable, knowledge- grounded ationales-pprncipllss operationalized in this work. Future integration with dynamic knowledge graphs will further shift models from merely speaking biology truly reasoning.
6.2 SCORING RELIABILITY: ON THE CORRECTNESS OF LLM-AS-A-JUDGGE
The second dimension concerns the reliability of scoring. Our knowledgg-augmennee LLM judge demonstrates measurable alignment with biological hierarchy: in the cell type annotation task, eval- uator scores show a strong positive correlation with ontology distance, and the generated rationales explicitly reference domain knowledge such as gene functions and perturbation mechanisms. This demonstrates that the judge not only distinguishes biologically closer from more distant predictions but also grounds its decisions in interpretable reasoning, moving evaluation from opaque numbers to auditable explanations. Such capacity is critical for systematic error analysis, allowing evaluation reveal not just what model gets wrong, but also why.
Despite these strengths, the judge inherits the probabilistic nature of LLMs and thus exhibits limita- tions. Future improvements could mitigate these weaknesses in several ways: ensembling multiple judges to reduce variance across single models; calibrating against expert-annotated rationale sets to ensure that rationales reflect causal biological truth rather than spurious correlations; and integrat- ing live biological knowledge bases such as GO, CI., and CellMarker so that scoring criteria evolve alongside scientific progress. Taken together, these advances could transform LLLM-aa---judge from promising scaffold into a verifiable instrument for scientific evaluation. Further analyses in Ap- pendix A.8.3 confirm that our evaluator is robust to the choice of judge model and invariant to response length, reinforcing its suitability as a reliable assessment tool.
6.3 FUTURE WORK: EXTENDING SC-ARENA TO EMERGING MODALITIES AND TEMPORAL REASONING
SC-ARENA is designed as a modular framework adaptable to evolving single-cell biology. Fu- ture extensions will incorporate: (1) Spatial transcriptomics for predicting cell-cell interactions within tissue context, extending the Virtual Cell's environmental dynamics; (2) Developmental tra- jectories to evaluate temporal reasoning through tasks like predicting progression in hematopoietic lineages; and (3) Multi-omics integration combining ATAC-seq, proteomics and other modalities to assess cross-modal biological consistency. These expansions will transform SC-ARENA into a living benchmark continuously aligned with emerging technologies and mechanistic questions in cell biology.
7 CONCLUSION
In this work, we present SC-ARENA, a natural language evaluation framework designed assess the capabilities of foundation models on key single-cell biology tasks. By constructing a virtual cell abstraction and designing five representative tasks, including cell-type annotation, gene perturbation reasoning, and biological QA, we enable interpretable and task -roondr evaluation of LLMs in the biological domain. To enhance both precision and insight, we introduce knowledge-augmented metrics that leverage external databases to evaluate model outputs beyond surface correctness. Our experimental results reveal that current LLMs show promising yet uneven performance across tasks, and that knowledge grounding significantly improves evaluation reliability and interoperability. SC- ARENA provides not only diagnostic tool for biological LLMs but also a new perspective on how natural language evaluation be aligned with domain-specifie reasoning. We hope this framework lays the groundwork for future efforts in building and benchmarking trustworthy, hology-alignne large language models.
ACKNOWLEDGMENTS
This work was supported by the project of Shenzhen Application Research and Development Special Fund Support (Grant No. XI.0S00022002270925550008 the National Natural Science Foundation
10

---

Published as a conference paper at ICLR 2026
of China (62376262, 62576083), the Natural Science Foundation of Guangdong Province of China (2024A151000016, 2025B1515020032), Shenzhen Science and Technology Innovation Program (K0TI2001999291733366).
REFERENCES
Abbi Abdel-Rehim, Hector Zenil, Oghenejokpeme Orhobor, Marie Fisher, Ross J Collins, Elizabeth Bourne, Gareth W Fearnley, Emma Tate, Holly Smith, Larisa N Soldatova, et al. Scientific hypothesis generation by large language models: laboratory validation in breast cancer treatment. 2025.
Journal of the Royal Society Interface,
Britt Adamson, Thomas M Norman, Marco Jost, Min Y Cho, James K Nuñez, Yuwen Chen, Jacque- line Villalta, Luke A Gilbert, Max A Horlbeck, Marco Hein, et al. A multiplexed single-cell crispr screening platform enables systematic dissection of the unfolded protein response. Cell, 167(7):1867-1882 2016.
Satanjeev Banerjee and Alon Lavie. Meteor: automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the ACI. Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005. also see LDC TIDES 2003 datasets.
Guangshuo Cao, Yi Shen, Jianghong Wu, Haoyu Chao, Ming Chen, and Dijun Chen. Cellreasoner: A reasoning-enhamne enhanced large language model for cell type annotation. bioRxiv, pp. 2025 05.5-05,77 2025.
S Carbon, E Douglass, Bm Good, Dr Unni, NI Harris, Cj Mungall, S Basu, RI Chisholm, Rj Dod- son, E Hartline, et al. The gene ontology resource: enriching a gold mine. NUCLEIC ACIDS RESEARCH, 49(D1):325-33)) 333, 2021.
Hongyoon Choi, Jeongbin Park, Sumin Kim, Jiwon Kim, Dongjoo Lee, Sungwoo Bae, Haenara Shin, and Daeseung Lee. Cellama: foundation model for single cell and spatial transcriptomics by cell embedding leveraging language model abilities. bioRviv, pp. 0024-05.20 2024.
Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang- segpt: toward building a foundation model for single-cell multi-omics using generative ai. Nature methods, 11700:1400 2024.
Alexander D Dichl, Terrence Meehan, Yvonne M Bradford, Matthew H Brush, Wasila M Dahdul, David S Dougall, Yongqun He, David Osumi-Sutherland, Alan Ruttenberg, Sirarat Sarntivijai, et al. The cell ontology 2016: enhanced content, modularization, and ontology interoperability. Journal of biomedical semantics, 7(1):44, 2016.
Yin Fang, Qiao Jin, Guangzhi Xiong, Bowen Jin, Xianrui Zhong, Siru Ouyang, Aidong Zhang, Jiawei Han, and Zhiyong Lu. Cell-ol: Training lims solve single-cell reasoning puzzles with reinforcement learning. arXiv preprint arXiv:2506.02911. 2025.
Gabriel Lino Garcia, Joao Renato Ribeiro Manesco, Pedro Henrique Paiola, Lucas Miranda, Maria Paola de Salvo, and Joao Paulo Papa. A review scientific knowledge extraction ing large language models in biomedical sciences. arXiv preprint arXiv:2412.03353 2024.
Xinyu Gong, Jason Holmes, Yiwei Li, Zhengliang Liu, Qi Gan, Zihao Wu, Jianli Zhang, Yusong Zou, Yuxi Teng, Tian Jiang, Hongtu Zhu, Wei Liu, Tianming Liu, and Yajun Yan. Evaluating the potential of leading large language models reasoning biology questions, 2023.
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Ying- han Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 2501.129499 12948, 2025.

---

Published as a conference paper at ICLR 2026
Minsheng Hao, Jing Gong, Xin Zeng, Chiming Liu, Yucheng Guo. Xingyi Cheng, Taifeng Wang, Jianzhu Ma, Xuegong Zhang, and Le Song. Large-seale foundation model on single-cell tran- scriptomies. Nature methods, 21(8):1481-1491, 2024.
Wenpin Hou and Zhicheng Ji. Assessing gpt-4 for cell type annotation in single-cell ma-seq analysis. Nature methods, 21(8): 1462-146,,, 2024.
Ana-Maria Istrate, Donghui Li, and Theofanis Karaletsos. segenept: Is language all you need for modeling single-cell perturbations? bioRxiv, pp. 2024-10, 2024.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv: 2019.
Maurice George Kendall. Rank Correlation Methods. Griffin, London, 4 edition, 1970.
Daniel Levine, Syed A Rizvi, Sacha Lévy, Nazreen Pallikkavallyaveeti,, David Zhang, Xingyu Chen, Sina Ghadermarzi, Ruiming Wu, Zihe Zheng. Ivan Vrkic, al. Cell2sentence: Teach- ing large language models the language of biology. In International Conference Machine Learning, pp. 27299-27325. PMLR, 2024.
Chin-Yews Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, 2004. Association for Computational Linguistics.
Junhao Liu, Siwei Xu, Lei Zhang, and Jing Zhang. Single-cell omics arena: A benchmark study for large language models on cell type annotation using single-cell data. arXiv preprint arXiv:2412.029155 2024.
Ziyang Miao, Qiyu Sun, Jingyuan Wang, Yuchen Gong, Yaowei Zheng, Shiqi Li, and Richong Zhang. Easy dataset: A unified and extensible framework for synthesizing 1lm fine-tuning data from unstructured documents. arXiv preprint arXiv:2507.04009 2025.
Thomas M Norman, Max A Horlbeck, Joseph M Replogle, Alex Ge, Albert Xu, Marco Jost, Luke A Gilbert, and Jonathan S Weissman. Exploring genetic interaction manifolds constructed from rich single-cell phenotypes. Science. 365(6455):786-799) 2019.
Nuala A O'Leary, Mathew W Wright, Rodney Brister, Stacy Ciufo, Diana Haddad, Rich Veigh,, Bhanu Rajput, Barbara Robbertse, Brian Smith-White, Danso Ako-Adjei, et al. Reference quence (refseq) database at ncbi: current status, taxonomic expansion, and functional annotation. Nucleic acids research, 44(D1).D733-D775, 2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, 2002. Association for Computational Linguistics.
CZI Cell Science Program, Shibla Abdulla, Brian Aevermann, Pedro Assis, Seve Badajoz, Sidney M Bell, Emanuele Bezzi, Batuhan Cakir, Jim Chaffer, Signe Chambers, al. Cz cellxgene discover: a single-cell data platform for scalable exploration, analysis and modeling of aggregated data. Nucleic acids research, 53(D)):D886 DD006-900,, 2025.
Syed Asad Rizvi, Daniel Levine, Aakash Patel, Shiyang Zhang, Eric Wang, Sizhuang He, David Zhang, Cerise Tang, Zhuoyang Lyu, Rayyan Darji, et al. Scaling large language models for next- generation single-cell analysis. bioRxiv, pp. 2025 04, 2025.
Yusuf H Roohani, Tony Hua, Po- Yuan Tung, Lexi Bounds, Feiqiao Yu, Alexander Dobin, Noam Teyssier, Abhinav Adduri, Alden Woodrow, Brian S Plosky, et al. Virtual cell challenge: Toward a turing test for the virtual cell. Cell, 188(13):330-3374 2025.
Cheol Ryu, Seolhwa Lec, Subeen Pang. Chanycol Choi, Hojun Choi, Myconggee Min, and Jy- Yong Sohn. Retrieval-based evaluation for 1lss: a case study in korean legal qa. In Proceedings the Natural Legal Language Processing Workshop 2023, pp. 132-137, 2023.
Charles Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):22-101, 1904.
12

---

Published as a conference paper at ICLR 2026
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025.
Christina V Theodoris, Ling Xiao, Anant Chopra, Mark D Chaffin, Zeina R Al Sayed, Matthew C Hill, Helene Mantineo, Elizabeth M Brydon, Zexian Zeng, X Shirley Liu, et al. Transfer learning enables predictions in network biology. Nature, 618(7965):16-6224 2023.
The UniProt Consortium. Uniprot: the universal protein knowledgebase, Nucleic acids research, 46 (5):2699-2699, 2018.
Chengrui Wang. Qingqing Long, Meng Xiao, Xunxin Cai, Chengjun Wu, Zhen Meng, Xuezhi Wang, and Yuanchun Zhou. Biorag: A rag-llm framework for biological question reasoning. arXiv preprint arXiv:2408.01 107, 2024.
Xiaoxuan Wang. Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-leve scien- tific problem-solving abilities of large language models. arXiv preprint TrXiv:2307.10633 2023.
Yuxuan Wu and Fuchou Tang. scextract: leveraging large language models for fully automated single-cell ma-seq data annotation and prior-informed multi-dataset integration. Genome Biology, 26( 177,, 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, al. Qwen2. technical report. arXiv e-prints, pp. arXiv-2412. 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388. 2025.
Fan Yang. Wenchuan Wang. Fang Wang. Yuan Fang, Duyu Tang, Junzhou Huang, Hui Lu, and Jianhua Yao. scbert as a large-scale pretrained deep language model for cell type annotation of single-cell ma-seq data. Nature Machine Intelligence, 4(10):852-88) 866, 2022.
Xinyu Yuan, Zhihao Zhan, Zuobai Zhang, Manqi Zhou, Jianan Zhao Boyu Han, Yue Li, and Jian Tang. Cell ontology guided transcriptome foundation model. Advances in Neural Information Processing Systems, 37:6323-6366, 2024.
Yuansong Zeng, Jiancong Xie, Ningyuan Shangguan, Zhuoyi Wei, Wenbing Li, Yun Su, Shuangyu Yang, Chengyang Zhang, Jinbo Zhang, Nan Fang, et al. Cellfm: a large-scale foundation model pre-trained on transcriptomies of 100 million human cells. Nature Communications, 16(1):4679, 2025.
Fan Zhang, Tianyu Liu, Zhihong Zhu, Hao Wu, Haixin Wang, Donghao Zhou, Yefeng Zheng, Kun Wang, Xian Wu, and Pheng-Ann Heng. Cellverse: Do large language models really understand cell biology? arXiv preprint arXiv:2505.0788 2025.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evalu- ating text generation with bert. In International Conference Learning Representations (ICLR), 2020. arXiv:1904.0966:
Xinxin Zhang, Yujia Lan, Jinyuan Xu, Fei Quan, Erjie Zhao, Chunyu Deng, Tao Luo, Liwen Xu, Gaoming Liao, Min Yan, et al. Cellmarker: manually curated resource of cell markers in human and mouse. Nucleic acids research, 47(DI):D71--D728 2019.
A APPENDIX
A.1 ETHICS STATEMENT
All data used in this work are derived from publicly available, open-source datasets, and thus raise no concerns regarding biomedical ethics or data licensing, as mentioned in the Section 4 We ac- knowledge that I.L LLM--as-jude evaluation may still carry inherent biases in the discussion section.
13

---

Published as a conference paper at ICLR 2026
Our contribution aims to mitigate such issues by explicitly grounding evaluation in external biolog- ical knowledge, providing a more objective framework and laying the foundation for more reliable and ethically sound evaluation practices in the future.
A.2 REPRODUCIBILITY STATEMENT
Our experimental setup, including model selections, data preprocessing, prompt templates, inference
protocols, and evaluation procedures, fully described in Section of the main text, Additional
implementation details and evaluation protocols are provided in Appendix A.5 All benchmark
datasets are constructed from publicly available resources as outlined in Section ensuring full reproducibility of our results.
4 and Table
2
A.3
THE USAGE OF THE LLM
In this paper, we only use LLM to polish the content to improve grammar and expression.
A.4 THE DETAILS OF OTHER EVALUATION METRICS
Traditional NLP evaluation metrics, such as BERTScore, BLEU, ROUGE, and METEOR, are widely used for assessing model outputs. However, when applied to biological and domain-specific tasks, these metries exhibit sigmificant limitations. For instance, BERTScore often assigns nearly identical scores to outputs from different models, effectively collapsing biologically distinct pre- dictions into similar numerical values. This evident in Table 4 where models like Qwen3-8B, Qwen3-32B, and GPT-4o achieve comparable BERTScore values despite notable differences in the biological accuracy of their predictions.
Lexical overlap overlap-based based metrics, including BLEU and ROUGE, are equally problematic in this text. A model output such as "CD8 T cell, NK cell, B cell" can receive a high BLEU or ROUGE score against a gold label like "T cell" simply due to shared vocabulary, cally incorrect. This can be observed in the perturbation task (Table 7 where BLEU-1 values are relatively high for several models, yet more detailed n-gram metries (BLEU-2, ROUGE-2) remain low, reflecting partial but misleading lexical overlap rather than true biological fidelity.
though it is biologi-
METEOR, which accounts for synonymy and paraphrasing, provides slightly better differentiation, but it still lacks grounding in domain-specing knowledge and fails to penalize mechanistically im- plausible predictions. Across tasks such cell type prediction, captioning, generation, perturbation, and ScienceQA (Tables (8). we consistently observe that high scores on these metrics do not nec- essarily correspond to biologically accurate or meaningful outputs. For example, DeepSeek-RI often achieves the highest BLEU or METEOR scores in science QA and perturbation tasks, yet other models with slightly lower scores may produce more precise or mechanistically consistent predictions.
In summary, while these metrics provide a rough estimate of linguistic similarity, they are insuffi- cient for evaluating the biological faithfulness of model outputs. Our observations underscore the need for specialized evaluation approaches that integrate domain knowledge and mechanistic con- straints, rather than relying solely on traditional NL.P metrics.
14

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparison of different models across traditional metrics in cell type annota-
tion.
Model
BERTScore
BLEU-1
BLEU-2
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Owen2.5-7B
80.20
0.24
0.02
0.12
0.02
0.12
0.07
Qwen2.5-14
83.45
0.68
0.22
0.05
0.13
Qwen2.5-32B
82.93
0.94
0.23
0.04
0.07
Qwen3-8B
80.98
0.07
0.01
0.17
0.04
0.17
0.09
Qwen3- 14B
80.86
18.38
4.54
0.21
0.06
0.21
0.12
Qwen3-32B
84.18
22.22
7.09
0.23
0.07
0.23
0.16
Qwen3-235H
83.47
11.52
3.04
0.29
0.10
0.29
0.18
Kimi-K2
84.91
31.60
10.91
0.30
0.10
0.29
0.16
GPT-4o
84.11
11.49
2.15
0.28
0.09
0.28
0.12
DeepSeck-RL RI
85.08
41.55
19.55
0.30
0.10
0.30
0.19
Table Performance comparison of different models across traditional metrics cell captioning.
Model
BERTScore
BLEU-1
BLEU-2
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Qwen2.5-7BB
81.72
7.25
0.74
0.12
0.01
0.16
Qwen2.5- 1-e22.5-14B
82.29
8.44
0.89
0.13
0.01
0.09
0.17
Qwen2.5-322
82.22
7.59
0.88
0.12
0.01
0.09
0.17
Qwen3-88
82.16
3.49
0.32
0.12
0.01
0.09
0.15
Qwen3- 14B
81.91
8.73
0.82
0.13
0.01
0.09
0.16
Qwen3-322 3-BBB
81.87
8.07
0.73
0.13
0.01
0.09
0.16
Qwen3-235B
82.10
8.44
0,90
0.13
0.01
0.09
0.17
Kimi-KS K2
81.47
8.50
0,93
0.13
0.02
0.09
0.16
GPT-4o
82.72
8.39
0.88
0.12
0.01
0.09
0.18
DeepSeek-R1
82.18
8.77
1.04
0.13
0.02
0.10
0.18
Table 6: Performance comparison of different models across traditional metrics cell generation.
Model
BERTScore
BLEU-1
BLEU-2
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Qwen2.5-7B
75.17
0.12
0.00
0.00
0.00
0.00
0.00
Qwen2.5-14Bp
77.01
0.36
0.00
0.01
0.00
0.01
0..00
Qwen2.5-322
77.28
0.72
0,00
0,02
0.00
0.01
0.01
Qwen3-8B
75.17
0.08
0,00
0.00
0.00
0.00
0.00
Qwen3-144 14B
71.69
0.30
0.00
0.00
0.00
0.00
0.00
Qwen3-328 32B
77.27
0.33
0.00
0.01
0.00
0.01
0.01
Qwen3-235B
77.65
1.18
0.00
0.03
0.00
0.01
0.02
Kimi-K2
79.24
1.91
0.00
0.04
0.00
0.02
0..02
GPT-4o
77.98
0.77
0.02
0.00
0.01
0.01
DeepSeek-R1
79.60
2.67
0.06
0.00
0.03
Table 7: Performance comparison of different models across traditional metrics in perturbation pre-
diction.
Model
BERTScore
BLEU-1
BLEU-2
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Qwen2.5-78
87.40
88.39
29.92
0..6
0.08
0.21
0..4
Qwen2.5-14P
90.39
90.95
23.45
0.77
0.11
0.28
0.45
Qwen2.5-32B 5-3BBB
90.92
91.99
22.01
0.83
0.12
0.30
0.48
Qwen3-8B
85.71
88.84
27.27
0.51
0.09
0.21
0.30
Qwen3- 14B
88.19
86.11
24,98
0.62
0.10
0.24
0.37
Qwen3-322 32B
89.06
91.19
27.01
0.65
0.11
0.26
0.37
Qwen3-235B
89.20
90.19
25.55
0.70
0.10
0.26
0,42
Kimi-K2
90.74
91.40
22.56
0.81
0.11
0.29
01.49
GPT-4o
90.50
91.53
18.26
0.84
0.12
0.30
0.48
DeepSeek-R1
90.94
91.36
18.01
0.89
0.12
0.30
0.51
15

---

Published as a conference paper at ICLR 2026
Table Performance comparison of different models across traditional metries in ScienceQA.
Model
BERTScore
BLEU-1
BLEU-2
ROUGE-1
ROUGE-2
ROUGE-L
METEOR
Qwen2.5-7B
88.51
43.22
14.10
0.38
0.13
0.23
0.25
Qwen2.5-14B
88.91
48.02
16.20
0.40
0.15
0.25
0.27
Qwen2.5-32B
89.04
52.43
17.42
0..40
0.1.
0.25
0..6
Qwen3-8B
87.81
29.20
9.08
0.39
0.13
0.22
0.31
Qwen3-14B
85.46
32.65
9.73
0.39
0.12
0.22
0.30
Qwen3-32B
74.81
30.46
8.70
0.33
0.10
0.18
0.26
Qwen3-288 235BB
87.55
29.86
8.66
0.39
0.12
0.22
0.30
Kimi-K2 K2
86.83
38.53
8.33
0,31
0,08
0.19
0.19
GPT-4o
89.24
53.73
18.48
0).40
0.15
0.25
0..66
DeepSeek-RI
87.49
34.05
8.85
0.36
0.10
0.21
0.26
A.5 EXPERIMENTAL DETAILS
Before detailing the models and our evaluation protocol, provide comprehensive overview of the data foundational to our benchmark. Table 2 summarizes the data sources, sample sizes, and specific input-output formats for all five tasks within SC-ARENA.
We evaluate a diverse set of both general-purpose and domain-specialized large language models (LLMSs) assess their performance covers range of publicly available foundation models, including DeepSeek-RA (Guo et al. 2025), GPT-4o. Kimi-K2 (Team al., (2025), and the Qwen series models, particularly Qwen2.5 (Yang et al. 2024) and Qwen3 (Yang et al. 2025). This selection spans a spectrum of model scales and architectures, allowing
single-cell biology benchmark suite. Our evaluation
examine performance differences attributable model capacity and
pretraining strategies.
In addition, we include four L.LMs that have been fine-tuned specifically for single-cell genomies: scGPT (Cui et al., 2024), seGenePT (Istrate et al. 2024), C2S-Scale (Rizvi et al., 2025). and Cell- O1 (Fang et al. 2025). The scGPT model used our evaluation was obtained from the version publicly released by the seGenePT authors; this model was fine-tuned on perturbation dataset by the scGenePT team to produce the version used in our experiments.
Our evaluation proceeds in two stages: answer generation and automated scoring. Cell genes in each instance are first standardized into a unified cell sentence format. For general- purpose LLMs, natural language question using task task-specifie prompt templates (see Appendix for full prompt designs). The model receives this formatted input and generates free-text response. For domain-specialized models, inputs are preprocessed according to each model's published inference protocol (e.g., tokenization and input schema), and outputs are post-processed into a unified response format compatible with our evaluation framework.
each benchmark instance is reformulated
To eliminate evaluation artifacts arising from inconsistencies in model output conventions, all model responses, regardless of architecture pretraining background, are converted into standard- ized structured schema before being passed to the evaluator (e.g.. ..... 1. (Up: geneA, geneB, ... This ensures that all models are judged under iden- tical formatting conditions, preventing discrepancies due to prompt conventions, tokenization, or decoding style.
At the input level, general-purpose LLMs uniformly adopt task- natural language prompts, whereas Nomaan-specializen LLMs strictly follow their original, published inference protocols with- out any forced adaptation. This design prevents compatibility-indeee biases and isolates the intrin- sic reasoning or domain-specifit competence each model.
For transparency and interpretability. explicitly categorize the evaluated systems into general- purpose versus domain- -emain--pecialized models the main paper. This separation reflects two meaning- ful axes of capability, general reasoning and biology-specifie expertise, and avoids conflating these factors when interpreting performance differences.
We employ GPT-4o-mini the automated evaluator for all tasks. For each task, a task-specific eval- uation prompt is carefully designed. The evaluator is provided with the input question, the model- generated response, the ground-truth answer, relevant external knowledge, and the task-specific scor-
16

---

Published as a conference paper at ICLR 2026
ing rubric (see Appendix for details). Based this information, the evaluator assigns [0, 5] scale. To facilitate cross- task comparison, these raw scores are normalized by dividing by 5, yielding percentage score. The final task-level performance is obtained by averaging the normal- ized scores across all instances within the task.
To ensure the robustness of our results, each model evaluated twice independently every task. We confirmed that the discrepancy between repeated runs did not exceed 2 percentage points in accuracy, thereby validating the stability of our evaluation. These combined controls ensure that observed performance differences reflect genuine model capability rather than artifacts of training data similarity, imput style alignment, or heterogeneous inference procedures.
The total score reported in Table is the unweighted sum of the five task scores, reflecting the holis- tic nature the Virtual Cell framework: each task annooation, captioning, generation, perturbation prediction, and scientific QA-is considered an equally essential and complementary capability for modeling complete virtual cell. We intentionally avoid arbitrary weighting to prevent introduc- ing subjective biases, and instead present raw task-level performance transparently to allow users to interpret relative strengths according to their specific application needs.
A.6 MITIGATING DATASET LEAKAGE IN SC-ARENA
Our evaluation paradigm is fundamentally different from prior works and is explicitly designed to minimize benchmark dataset leakage risks eesecialpe pertinent given that models like C2S were trained on CELLxGENE thro-hh the following four strategies:
1. New task formulations and prompts eliminate onput- overlap. SC-ARENA in- troduces entirely new task definitions and natural- Lanuual-anguage prompts under the Virtual Cell abstraction. None of the lated training corpora. As a result, even a model has seen the underlying raw scRNA-seq profiles, it has never encountered the linguistic formulation used in SC-ARENA.
pairs, templates, phrasings overlap with C2S re-
2. Evaluation is kmowledge-grounder not string-based. Our LLL--as-a-judge assesses bi- ological correctness using external resources (Cell Ontology, CellMarker, Gene Ontology, UniProt, and peer-reviewed literature), preventing models from gaining high scores through memorized patterns or lexical similarity. To further substantiate this, we computed BLEU scores for the C2S models the Cell Type Annotation (CTA) task (see Table 9 Both models achieve high biological accuracy despite remarkably low BLEU scores scores-lower than those of DeepSeek-RI and Kimi-K2-strongg indicating that their performance reflects acquired biological knowledge rather than memorization of test samples.
3. Randomized sampling reduces systematic overlap. For datasets without predefined splits (e.g., CELLXGENE), we perform uniform random sampling across tissues, cell types, and expression distributions. This prevents overrepresentation of frequent or previously dominant classes and reduces the chance that performance arises from memorizing com- mon patterns.
4. SC-ARENA is a continuously evolving benchmark. We acknowledge that any static benchmark may eventually become overfitted. SC-ARENA is therefore designed as living benchmark: new perturbation datasets, multimodal atlases, and emerging biological tasks (e.g., Perturb-seq challenges) will be progressively incorporated under Virtual Cell abstraction to further mitigate leakage concerns and enhance long-term robustness.
Table 9: BLEU scores for C2S models on the Cell Type Annotation (CTA) task. Low BLEU scores scores-despite strong biological performance support that results are not due data leakage or memorization.
Model
BLEU-1
BLEU-2
CTA-Score
C2S- C2S-Pythia-410m
23.73
17.14
47.34
C2S-Scale-Pythia-ll
13.66
8.29
41.68
DeepSeek-R1
41.55
19.55
40.81
Kimi-K2
31.60
10.91
40.00
17

---

Published as a conference paper at ICLR 2026
A.7 CROSS-TASK RELIABILITY OF THE
EVALUATOR
To address concerns about the generalizability of our knowledge- -ugggggedd evaluator beyond Cell Type Annotation (CTA), we conducted reliability analyses across all five SC-ARENA tasks, cach involving distinct output formats and reasoning demands. As summarized in Table [0] we employ task-specific external validity metrics and expert judgments to validate the evaluator's 's alignment with biological truth and human assessment.
For Cell Generation (CG), we measure the overlap between generated "cell sentences" and known sell-type-specific marker genes from CellMarker. The evaluator's scores show strong positive cor- relation with the percentage of correctly included markers (Spearman 0.6669, p < 10-4).
For Perturbation Prediction (PP), we compute cosine similarity between predicted and ground- truth sets of differentially expressed genes (DEGs). Evaluator scores correlate significantly with this similarity (p- 0.6057, p < 10-1388 138), confirming sensitivity to mechanistic fidelity.
For Cell Captioning (CC) and Scientific QA (SQA), we conduct pairwise human preference stud- ies: we randomly sampled 30 samples containing two responses from different models per task and ask two experts to independently judge which response is better. Crucially, the evaluator' 's rankings align well with aggregated expert preferences: p = 0.8394 (p = 3.6 10-9) for CC and p = 0.8199 (p 3.0 10-$) for SQA.
These results collectively verify that our auggeeeeee evaluator maintains high reliability across diverse reasoning modalities, structured prediction, molecular generation, and open-ended explanation, by grounding judgments domain-specifie knowledge and human expert alignment.
Table 10: Reliability validation of the showledge-augmenter evaluator across SC-ARENA tasks. Metrics reflect correlation between evaluator scores and external validity measures (Spearman and p-value). For open-ended tasks, we report expert agreement rate and correlation with evaluator rankings.
Task
Validation Metric
Spearman
p-value
CTA
Ontology path distance
0.6212
10-3
CG
% correct CellMarker genes
0.6669
< 10-3
PP
DEG cosine similarity
0.6057
< 10 10
CC
Expert preference alignment
0.8394
10-3
SQA
Expert preference alignment
0.8199
< 10-3
A.8 ADDITIONAL. RESLUT ANALYSIS
A.8.1 ADDITIONAL RESULT ANALYSIS WITH RADAR PLOTS
Task-level Visualization Scaling Effects. While the main text highlights numerical improve- ments from scaling and iteration, Radar plots provide complementary, task-level perspective. Fig- ure 3r compares representative general-purper purpose models across the five SC-ARENA tasks. The visu- alization confirms the aggregate trend reported in Table larger and newer models consistently expand the coverage of capabilities - but also exposes uneven gains across tasks. For instance, Kimi-K2 achieves pronounced lead in captioning and generation, whereas Qwen3-32B performs comparatively better in perturbation prediction. These contrasts underscore that model scaling im- proves overall fluency and reasoning breadth, yet does not fully overcome the challenge of mecha- nistic prediction.
Heterogeneity Across Tasks. The radar plots further reveal that gains from scaling are not uni- formly distributed. Open-ended tasks (captioning, scientific QA) show the steepest improvements, whereas deterministic tasks (cell type annotation, perturbation prediction) remain relatively con- strained. This echoes the "fluent but not faithful" gap emphasized in the main discussion, illustrat- ing how visualization helps to highlight task-specific limitations that may be obscured in aggregate scores.
18

---

Published as a conference paper at ICLR 2026
Implications for Model Development. By making task asymmetries visible, radar plots empha- size the importance of fine one-grained evaluation beyond single total scores. They suggest that future progress may require not only scaling and iteration, but also targeted approaches that explicitly address mechanistic reasoning in biology.
despseokr)
gaen?_ _5_78 awer2__5_44
qpwen LA8R
q 328
gpt-4o
quvena2444
qwen3_235B
kimik2
qwen2_5_ 5_828
CG
SQA
Figure Radar-plor comparison of representative general-purpose models across the five SC- ARENA tasks: cell type annotation, perturbation prediction, cell generation, cell captioning, and scientific QA. The visualization highlights the uneven distribution of gains: while models such as Kimi-K2 and DeepSeek-RI excel in captioning and generation, Qwen3-32B performs comparatively better in perturbation prediction. The radar plot provides a task-level perspective that complements aggregate scores and illustrates persistent challenges in mechanistic reasoning.
A.8.2 DISCRIMINATIVE CAPACITY VIA ONTOLOGY PATH LENGTH TO ROOT.
To further validate the discriminative capacity of SC-ARENA, we examined the distribution of ontol- ogy path length to root for predicted cell types across models (Figure Here, the x-axis represents binned intervals of the average path length the ontology root, with shorter values corresponding to more specific and biologically precise annotations, while the y-axis reports the count of predictions falling into each interval.
The results reveal clear scaling trend: Qwen3-32B produces deeper predictions average, fol- lowed by Qwen3-14B and then Qwen3-8B. This hierarchy prediction depth aligns closely with their overall benchmark performance, where larger models consistently outperform their smaller counterparts. Such consistency indicates that as models scale, they not only achieve higher aggre- gate scores but also tend to generate provides additional evidence that SC-ARENA can capture fine- ine-grained distinctions in model behav- ior, delivering discriminative capacity beyond what traditional NL.P metrics offer.
specific and biologically meaningful predictions. This
19

---

Published as a conference paper at ICLR 2026
qpann]
175
Couit
Figure 4: Distribution of ontology path length to root for predicted cell types across models. The x-axis shows binned intervals of the average path length to the ontology root, using left-closed, right-open notation [a,b), and the y-axis indicates the number of predicted cell types falling into each interval. Shorter path lengths indicate closer alignment with the ontology hierarchy and thus more specific predictions.
A.8.3 ROBUSTNESS OF THE KNOWLEDGE BROWLEEDEEAAGMENTEED
AUGMENTED EVALUATOR
To further validate the reliability of our knowledge-augmenter evaluator, we conduct three addi- tional robustness checks.
(1) Cross-model evaluator consistency.
We re-ran the full SC-ARENA benchmark using second, independently trained judge model (DeepSeek-V3. .t..V3...EEppP 2-xxp) the evaluator, replacing the default GPT-4o-mini. The two evalu- ators demonstrate strong agreement across all five tasks and all tested models (metrics in Table [T]. These results indicate that our evaluation outcomes are not artifacts of a specific L.M judge, but reflect stable, task-grounded judgments anchored in external biological knowledge.
(2) Insensitivity to response length.
We explicitly tested whether scoring is biased by answer verbosity. Across all model responses in SC-ARENA, we computed the correlation between token count and evaluation scores. The corre- lation is negligible (Table ITp. confirming that evaluator assesses content based on biological fidelity and reasoning quality, not surface form such as answer length.
(3) Robustness to evaluation stochasticity.
To assess stability against inherent randomness in the evaluation process, we re-ran the evaluation the Cell Type Annotation (CTA) task with different random seed (changing from 2025078 701 to 42). The resulting scores remained highly consistent (Table II), confirming that our evaluator's judgments stable under typical stochastic variations encountered during LLM-bbsed assessment.
Table 11: Quantitative metrics for robustness validation of the knowledgg-augmeetted evaluator.
Robustness Test
Spearman
Kendall's 7
Cosine Similarity
Cross-moo model evaluator consistency
0.6701
0,5719
0.8873
Insensitivity to response length
0.0730
0.0578
Robustness to evaluation stochasticity (CTA task)
0.9070
0.8354
0.9468
20

---

Published as a conference paper at ICLR 2026
A.9 STEP-BY-STER EXAMPLE OF KNOWLEDGE-AUGMENTI BIOLOGICAL PLAUSIBILITY SCORING
To enhance interpretability and clarify how biological plausibility is quantified in SC-ARENA, we present concrete, end-to-end example from the Cell Type Annotation (CTA) task, corresponding to the evaluation logic illustrated in Figure 2b.
Setup. Consider the following instance:
Ground-truth cell type: CDI6-positive, CD56-dim natural killer cell, human
Model prediction: Natural Killer (NK) Cell
Step 1:
Ontology Matching via OLS. We standardize both labels using the Ontology Lookup
Service (OLS) to map them to canonical identifiers in the Cell Ontology (CL) (Dihhl et al. 2016);
Ground-truth + CL 2000001 (a terminal, highly specific NK cell subtype)
Prediction - CL 0000623 (the generic "natural killer cell" parent node)
Step 2: Hierarchical Distance Computation. We compute the shortest path length between the two CL nodes the ontology graph. In this case:
Distance = 2
indicating that the predicted term is two levels above the ground-truth in the CL. coararchy-correctlyy capturing the NK lineage but lacking terminal granularity.
Step 3: Mapping Distance to Plausibility Score. While the raw distance itself is not the final score, it serves biologically grounded proxy for correctness. Our knowledge-augmentt tor (an Eval-RAG-style LLM judge) uses this ontology path-along with functional definitions and marker gene context- assign numan-interpretable rating a 0-5 scale. In this example, the evaluator would typically assign score of 3 (see Figure 10 for the full rubric), reflecting:
evalua-
Correct lineage identification (NK cell),
Lack of terminal specificity (missing CD16+/CD56-dim distinction),
Biological plausibility despite incompleteness.
Step 4: Validation via Correlation Analysis. Across CTA predictions, we validate that such ontology-informee judgments align with human intuition. As shown in Figure 2a, evaluator scores exhibit a strong positive Spearman correlation with negative ontology distance:
p == 0.6212,
< 0.001
This confirms that SC-ARENA'S 's scoring is not arbitrary but systematically reflects biological re- latedness encoded in curated ontologies.
This example illustrates how SC-ARENA transcends brittle string matching by integrating struc- tured domain knowledge into every evaluation seep-yielding scores that are both interpretable and biologically faithful.
A.10 DETAILED PROMPT FOR EACH TASK
To ensure reproducibility and fairness, we provide here the full set of task-specific prompt templates used in SC-ARENA. For each benchmark task, design two categories of prompts: (i) answer generation prompts, which are provided to the tested models to elicit predictions in a standardized format, and (ii) score generation prompts, which mini) to assign task task-specific scores following the rubric in Appendix AA.5] Together, these templates operationalize the Virtual Cell abstraction by unifying inputs, outputs, and evaluation across tasks.
presented to the evaluator model (GPT-40-
21

---

Published as a conference paper at ICLR 2026
Cell Type Annotation (CTA). The answer generation prompt instructs the model to infer the most likely goooulddg cell type from a ranked gene expression list (cell sentence). The evaluation prompt guides the judge compare the predicted type with the gold label, rewarding exact matches or semantically close ontology categories.
Cell Captioning (CC). The answer generation prompt asks the model to produce a concise natural language description of the cell, highlighting marker genes and lineage context. The evaluation prompt checks whether the caption aligns with the ontology definition and expression evidence, penalizing vague or generic responses.
Cell Generation (CG). The answer generation prompt requires the model to synthesize plausible ranked gene list given cell type name and description. The evaluation prompt instructs the judge to verify consistency with marker gene databases and ontology knowledge, assigning partial credit when the generated profile is approximately correct.
Perturbation Prediction (PP). The answer generation prompt asks the model to predict both the perturbed cell sentence and sets of up- and down-regulated genes given a baseline cell sentence and perturbation condition. The evaluation prompt guides the judge to assess predictions against experimental ground truth and external references (e.g., NCBI, UniProt, GO), with scores reflecting both plausibility and mechanistic validity.
Scientific QA (SQA). The answer generation prompt presents the model with a domain-speciinn research question and asks for step-by-step reasoning process leading concise final answer. The evaluation prompt provides the judge with the model's answer, the gold reference, and support- ing PubMed context, instructing it to score factual accuracy, reasoning quality, and alignment with evidence.
Figures 5 illustrate the answer generation prompts for the five tasks, while Figures I0 present the corresponding evaluation prompts.
Cell Type Annotation Answer Generation Prompt
You are given mnu__enne] genes ranked by expression level from (organism] cell.
Cell sentence: (cell_ sentence)Please reason step-by-step to determine the most probable cell type.
Consider known marker genes, expression patterns, and biological context.
After your reasoning, conclude with your prediction the exact format: [Predicted, cece___ell___e__le ...]
Figure 5: Cell Type Annotation Answer Generation Prompt.
22

---

Published as a conference paper at ICLR 2026
Cell Captioning Answer Generation Prompt
The following sentence represents [num_genes] genes from (organism) cell, ranked by expression level.
Cell sentence: (cell_seele sentence)
Generate concise, natural- laanrae- description of this cell that reflects the most specific cell type supported by the expression profile, while remaining biologically accurate and consistent with the Cell Ontology.
Guidelines:
Prioritize lineage-defining markers, key functional modules, and unique biological roles directly
indicated by the gene list.
If gene evidence strongly supports unique terminal cell type, describe it clearly and specifically. If evidence insufficient or ambiguous for the exact terminal type, describe most specific broader parent type supported by data, note uncertainty or possible alternatives. "highly active"
Avoid generic phrases
metabolism" unless specific
entire response be wrapped in the format: [Captioning:...
Figure 6: Cell Captioning Answer Generation Prompt.
Cell Generation Answer Generation Prompt
You are given the name and description of a cell type. Cell Type: [cell_type) Description: (cell description)
Generate sentence that reflects expected gene expression profile this The cell sentence comma-separated
genes, ordered highest expression.
Ensure the gene ordering biologically plausible and consistent with described cell type's known functions and markers.
to state answer using the exact format: [Cell Selell__etteee... ...]
Figure 7: Cell Generation Answer Generation Prompt.
23

---

Published as a conference paper at ICLR 2026
Perturbation Prediction Answer Generation Prompt
You are single-cell transcriptomics expert.
Background
Original expression profile (genes ranked by descending expression): (cell__enneee _entencee
Perturbation applied: jiorturbationndeesciiiinnn// petpcrrptioonnaaniiieee differentially expressed genes (DEGs): (candidate__des list)
Task
1. From the provided candidate DEGs, identify which genes significantly Up-regulated (Up) and which are Down-regulated (Down), based on the perturbation context and prior gene knowledge.
2. Based on these changes, generate the updated cell sentence that reflects the perturbed expression profile.
Output
You may briefly explain your reasoning (s 5 bullets or s 120 words). Conclude with the final answer exactly this format, on a single line: [Up: geneA, geneB, ..own: Sentence: genel gene2 gene3 ...]
geneX, geneY, S...tecel_eennteene:
Figure 8: Perturbation Prediction Answer Generation Prompt.
Scientific QA Answer Generation Prompt
You are domain expert in single-cell blology. You will be given specific type question and the question itself. Please think step by step using relevant biological knowledge before answering. Your answer enclosed the following [Answer: ...] clear, concise, scientifically accurate language.
Input: Question Type: (question_tyee
Question: (question)
Your Response:
Figure 9: Scientific QA Answer Generation Prompt.
24

---

Published as a conference paper at ICLR 2026
Annotation Generation
Input Providd::
"Exect Cell Typa**; The manually annotated ground truth cell type
**Cell Ontology Chain]s] "*; Herarchical paths from the root the exact cell
**9yedicted Type machine-gererated type
Tack:
predicted cell type "*g-5",,
biological and functional prosimity the exect cell type
poseion the provided ontology chain(s), appolcable
knewledge biology immunology ontology chains a suppppring reference** * not riiid constraint.
judement prionitees "*somantic lineage cleseness" using
net Score:
The predkted type "eexactly matches the provided ground truth call type.
ast Score:
The predicted type "*recognized synonym * "*functionally equivalent** **drrct parent/ancestor" ontology subtype**
-lternatively
(eg.
aee Score:
The predicted type is "rrassonable vibling* ** or **ubbtpp within the same Insage"* but is *less specific** or **1-2 levels away"" It appear in the antalogy patts, biologically close based known immurclagy developmental origin.
mee Score:
A6a type belongs
functional category ineage" "ggeneaal
meaning.
The preditted typo is ""vaguely related"", such as belonging to the **aame germ laya or "*broader inmune context" but lacks clear
Ineage relevence.
nett
type *ddffierent Ineage"* *functionally unrelsted"
biologically reaationhhpp"
type.
Also assign for nonsensical, ambiguous, non cel type predictions.
Additional Principles Scoring:
*Onttology
Reference**
Use the antology pathis) to help identily possible matches and hierarchy positions, However do "*"oot rely soldly on ontology incluiin"* prioritie functional and lineage reasening.
"Soologiaal Reasoning Encouraged"":
missing
ontology chain,
type biologically and reasonably
**Uneege Fins, Specifictt Second**,
Priontov whether the prediction belongs the conrect cell Imwage before judging how specific distint
Matching Chain**;
multiple ontology chans are procided,
that to possibtie scone the predicted type.
Provide only final scone in following format:
[Store:X] XI
Where Integer from
Input:
Exact Call Type: Mi
Coll Ontalogy ChainjstSSs
Predicted Call Typertis
Figure 10: Cell Type Annotation Score Generation Prompt.
25

---

Published as a conference paper at ICLR 2026
Captioning Generation
You are biomedical expert in trnserittnns and call type dissiliiatioo, whth deep expartise in the Cell Oktology and hierarchcal structure.
"*ovaluate model description single
*MGene expression profile** ranked from
[cell_sentrn seeleenne)
tnt 2. **Cell Ontology definition path
Nerarchical linsage from broad parent concept down to specifis fine-grained cell type. Each level contains a name and definition.
vel__ooth_chate
s. *Cell description (to evalustedj*
"IMPORTANTT =e This only
you should
emoty contains only whitessacc,, describe type, you must assign "*[Score: O1** without
analy analyss.
(predicted_ captionn)
tn "Lvoluation Objective**
Assess whether the deveription "Section 3** socurately and specifically reflects the **target cell type** stusted the ontelogy considering "one expression
from Section
You must:
CChrk ""ontolagy match" (terminal node or appropr ate ancestor).
Check "gene expression support" for daimed daimedspeciiitty
Bn "Keyy *niciples***
poinzs**
Description esactly matches the "reeminal node" supported by clear marler
polnts"*
Matches terminal
evidence mostly supsortive. siiing type, supported gene data; Overly or vague description limited
ppintss**
"aines**
point"*
Barely relevant or generic tissun/yyttmm reference.
**0 poincs**
Enipyy unrelated, incoherent, wrong cell
""Special rule:*
emoty generic (e.g. cel"" cell') assign immedistely.
**Your should incude the following format:"*
brief justification (2-4 sentencess explaning the reasoning behind your score.
Figure 11: Cell Captioning Score Generation Prompt.
26

---

Published as a conference paper at ICLR 2026
Annotation Generation
Input Providd::
"Exect Cell Typa**; The manually annotated ground truth cell type
**Cell Ontology Chain]s] "*; Herarchical paths from the root the exact cell
**9yedicted Type machine-gererated type
Tack:
predicted cell type "*g-5",,
biological and functional prosimity the exect cell type
poseion the provided ontology chain(s), appolcable
knewledge biology immunology ontology chains a suppppring reference** * not ngiid constraint.
judement prionitees "*somantic lineage cleseness" using
net Score:
The predkted type "eexactly matches the provided ground truth call type.
ast Score:
The predicted type "*recognized synonym * "*functionally equivalent** **drrct parent/ancestor" ontology subtype**
-lternatively
(eg.
aee Score:
The predicted type is "rrassonable vibling* ** or **ubbtpp within the same Insage"* but is *less specific** or **1-2 levels away"" It appear in the antalogy patts, biologically close based known immurclagy developmental origin.
mee Score:
A6a type belongs
functional category ineage" "ggeneaal
meaning.
The preditted typo is ""vaguely related"", such as belonging to the **aame germ laya or "*broader inmune context" but lacks clear
Ineage relevence.
nett
type *ddffierent Ineage"* *funccionally unrelsted"
biologically reaationhhpp"
type.
Also assign for nonsensical, ambiguous, non cel type predictions.
Additional Principles Scoring:
*Onttology
Reference**
Use the antology pathis) to help identily possible matches and hierarchy positions, However do "*"oot rely soldly on ontology incluion"* prioritie functional and lineage reasening.
"Soologiaal Reasoning Encouraged"":
missing
ontology chain,
type biologically and reasonably
**Uneege Fins, Specifictt Second**,
Priontov whether the prediction belongs the conrect cell Imwage before judging how specific distint
Matching Chain**;
multiple ontology chans are procided,
that to possibtie scone the predicted type.
Provide only final scone in following format:
[Store:X] XI
Where Integer from
Input:
Exact Call Type: Mi
Coll Ontalogy ChainjstSSs
Predicted Call Typertis
Figure 12: Cell Generation Score Generation Prompt.
27

---

Published as a conference paper at ICLR 2026
Perturbation Pred ctediction Generation
expert single tronscriptomics gene reguiation, predicted gene expression perturbation in response to speciicc condition. You gieen the following information:
"*biological pousibiitt accuracy"*
** "Ungrrtureed Cell Expression (cell sentenca)** description of the expression profile before perturbation. call.
Condition" experimental factor
Expression (cell sentenos)
1st of signifiaatlly "Tup-reglaate" genes"*
Ust of significntly" ** d'dwnn regulated genes**
"Predicted Result* by
model:
perturbed Expression sentence)
Predicted signficantly uup-reuulted genes**
Prodicted sigrifccnnll Noww-eeullbbe genes"*
"Reforance knowledge**
Brief summaries from NCBI, GeneCards, and UniProt for the invoked
Ontology
Comporent C desuriptio))
Biological Process (60, descciption)
Motecular Function [GO descristion)
response
considering following
Are the ""predicted expression charges** consistent with the true perturbation pattem??
Are the ""up/down regulated penes** biologically plousible given the perturtation and consistent with known gene functions? shifts?
Does prredccted sentence"* paalestvely resemble "supported contradicted"*
terms of
knowledge?
Give a score from *10 5*a based evenal plausbility and alignment ground truth and known biology, where:
Completely ncoerect and boologizllly implassible
Poor prediction and uncorvincing
Somewhat
Ressonable prediction
soundness
4 Mosbly correct with minor inconsistencies
5 Highly accurate and bio logicillly consistent
**oou should indlde
following format"":
[Seore: xI
#dd Input:
"Unserturbed Expression
*PPrttrbation Condition**:
jparturbation description)
*GGround Truth**.
"Pertturoed Cell Expression (cell sentence)**:
jpert_eenterce)
"Ugg-ngulated Genes**:
(up, wot_genee_str
"Down-regubbted Genes**
(down str)
""Preditedd
language
Nerturbed Expresiion (cell sentenca)"* iceil sentence)
**Predicted Up sp-regulated Gones**:
[up ganes]
**Predicted Down-regulated Genes**
genes)
**efferance *nowledge'*:
**eene Summanies (NCII/ GeneCards UniPrat)**,
(NCis card, UniProt summariess
Description.*.
deseriittion
"Bological Process** (30, desccription)
"Moleaular function"*, [GO] deseriptio))
Figure 13: Perturbation Prediction Score Generation Prompt.
28

---

Published as a conference paper at ICLR 2026
Scietiff QA Score Goneration Prompt
single-call biology scientffc reasaning.
the anower provided language model ('Evaluated Model") question accurate, wel- reessredddd and blolog cally sound.
task eveluate
You will be given:
**I Type]"". A
type
reasoning required (e.g.. Marier Reasoning, Logk,
Experieental Design, etc..
**JOriginal Quection)"": "; The actual question that was pased to the model.
(Ground truth Answer]"" A relable, expert-verfee reference answer.
* |Model Answer ** The output from the Evaluated Middll.
RRfereecce Paper The thee the scientfic
from
derived.
RReference Paper Abstract]
abstract
paper, provided as knowledge you assess correctnees.
"Reeeevant Passage]** spectfic section of the paper most doosely related this question [may include results, figures, methods) Use this pacsage as the primary reference for cornectness.
Instractions
Carefully anahyas
Sciietifiially
(check facts, terminology brological mechanisms].
Laogically consistens" with the Original Question.
*ell aligned* with the Ground Truth Answer.
""Appropriote" the
showing the right reasoning depth domain relevance.
**Condistent
supported
Paper Passage copy verbatim,
check cornectness).
ne important Evelustion Rules
Any "acttual or scientfic erro (e.g, meclassffing incorrect pathway direction, biological effect) must lower
5** *CCncoppual or mechanisii enoos" that undermine reasoning le miking up immune stimulatory vs suppr essive rolest should be
ecits, score
condidered mocor flaw, scored "*g3**
If bagelyy correct contradiction),
**minor imprectton" ** wording, lock detail without scientifi
**g**,
**ully correct, scientflic errors strong aliggmmntt* may receive ""g"".
ne Your Evalustion should Include:
did "Weaknesses Erross" explicit
*Strrnnttss":
wrong miskeading.
Impact Errons** How they affect correctness and scoring.
Scoring Rubric
Description
[
**5**
Fully correct, scientffically accurate, no errons, insightful, and well aligned with the ground truth.
**g*
Mostly cornect, but with minor flave imprecsions; no major scentific errons
****
Partially correct, contains at least one clear scietific error noticeable though some cornect reasoning present.) misunderstandnng
Langely
multiple scientific mostly wrong.
Minimally
Completely incornet, imelesant, nonsensccal.
the end of your resparse, you must include the final score in this esact format:
[Score: XI'
Input
*Quusttoo Ty
"question["typ"" [type"]]"
*Origia Ouestion**:
"question] questioonn
Answerr**
[question]" answer 'I'
MMddi Answer".
|model_answer]
*Beference Paper Tole**
"ppaper [itie]"
"Baference Abstracc""
"poper abstract]'
7. "eleevant Fassage**:
'quuestion" rrelevan,, Telesont_passsgee]
Figure 14: Scientific QA Score Generation Prompt.
29


Reviews:
[{"review_id": "7V3FDdYkOH", "reviewer": "Reviewer_k5M8", "summary": "This work proposes an evaluation framework called Virtual Cell which seeks to unify the assessment of LLMs performance on various sub tasks important for single cell analysis like cell type annotation, captioning etc. The authors then evaluated various LLMs on a dataset derived from combining publicly available single-cell databases.", "questions": "- This research area is a useful application of LLMs for biological science but while the authors mention the previous works, proper comparisons to these methods is lacking. Authors should consider expanded the related work section needs to clarify the contributions of this paper in comparing to some of the works mentioned here. \n- Authors should consider incorporating some of the ideas suggested in the discussion on modeling, evaluating and scoring into this work to improve the contribution and novelty of this work.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Combining vast amounts of single-cell data with natural language based knowledge available to gain insights into cellular function is beneficial to the biology community so this is a timely topic. \n2. Authors have benchmarked their proposed framework for evaluating LLM performance across many different LLM models or other domain specific models. \n3. Definition of the knowledge cell class is well thought out in considering multiple sources of information available for analyzing cellular dynamics.", "weaknesses": "- The novelty/value of this framework for evaluation is unclear, many current models like Cell2Sentence already combine single cell rna data with text based information and have shown use cases for downstream tasks like cell type prediction, perturbation response prediction etc. \n- Considering existing methods that can perform some of the tasks mentioned in the multi-task benchmark like Cell2Sentence, CellReasoning etc, to fully evaluate this work, performance of existing methods on individual tasks should be included and discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h7EH3W9Qtd", "reviewer": "Reviewer_6ZRo", "summary": "This paper introduces SC-Arena, a benchmark designed to evaluate large language models (LLMs) as “virtual cells” capable of reasoning over biological knowledge and single-cell data. The benchmark includes multiple question types testing biological plausibility, reasoning consistency, and interpretability. It also introduces a knowledge-augmented evaluation strategy (Eval-RAG) that uses retrieval-augmented generation to penalize biologically implausible answers and reward semantically coherent responses beyond exact-match metrics.", "questions": "How sensitive is Eval-RAG to the retrieval source—does the choice of biological database or text corpus significantly change the evaluation outcome?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "•\tVery creative and well-motivated benchmark that treats LLMs as reasoning agents over biological cell states.\n•\tThe Eval-RAG strategy is an elegant idea that improves evaluation by incorporating biological context and semantic plausibility, moving beyond token-level correctness.\n•\tThe paper provides a valuable framework for comparing different LLMs under biologically grounded tasks.\n•\tThe paper does extensive evaluation of general and domain-specific models.\n•\tDiscussion includes relevant next steps for producing a more biologically robust evaluation benchmark.", "weaknesses": "The paper would benefit from more detailed examples—for instance, elaborating on the process shown in Figure 2, panel B, to clearly explain how the biological plausibility scoring is computed step by step.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TrnAsyC6co", "reviewer": "Reviewer_2kiV", "summary": "This paper introduces SC-Arena, a natural language benchmark for evaluating large language models (LLMs) in single-cell biology. The authors propose a unified “Virtual Cell” abstraction and design five biologically grounded tasks: cell type annotation, captioning, generation, perturbation prediction, and scientific QA. The evaluation leverages a knowledge-augmented LLM-as-a-judge framework, integrating external ontologies and databases to ensure interpretability and biological fidelity. Experiments compare both general-purpose and domain-specialized LLMs, revealing strengths and limitations in biological reasoning.", "questions": "- While the use of open-source datasets is commendable, how do the authors plan to address the risk of benchmark dataset leakage? For example, CS2 used the CELLXGENE dataset for training. Could this explain their performance on the CTA task?  \n\n- Are there plans to expand the benchmark to include additional modalities or more challenging reasoning tasks? \n\n- It’s unclear how the architecture of the models will influence performance in these tasks. My main concern is that performance differences may reflect experimental artifacts rather than true differences in model capability. For example, a domain-specific model might outperform others simply because it was fine-tuned on data similar to the benchmark, or because its input format more closely matches the evaluation protocol. How do the authors plan to address this potential bias? Will this influence which models can or cannot be used for the benchmarking?\n \n- The assessment of the knowledge augmented evaluator is shown only for the CTA task. How do the authors plan to validate the evaluator for the other tasks? How can we be confident that the evaluator is reliable and interpretable for these other tasks, which may involve different types of reasoning, output formats, and biological knowledge?\n\n- As the evaluator is also a model, a proper evaluation of the model should be performed. For example, are there cases where the evaluator produces scores that do not align with expert human judgment, or where it fails to recognize biologically implausible or trivial answers? Does the evaluator systematically favor certain model architectures, output styles, or biological domains? Does it reward verbosity, penalize concise but correct answers, or show preference for models trained on similar data as the evaluator itself?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "-  The Virtual Cell abstraction and multi-task natural language evaluation is an interesting idea to more objectively test different models’ capacity to understand cellular processes. \n\n- Integrating external biological knowledge (Cell Ontology, UniProt, GO, CellMarker, PubMed) into the evaluation pipeline is a major strength and a clever way to address the limitations of string-matching metrics.\n\n- The paper benchmarks a wide range of models (Qwen, GPT-4o, DeepSeek-R1, Kimi-K2, C2S-Scale, scGenePT, scGPT, Cell-O1) and analyzes performance across tasks, model scales, and domains, using only open-source datasets.", "weaknesses": "- While the benchmark covers different tasks and the datasets are open-source, the paper does not address the risk of benchmark dataset leakage; such as, whether the datasets used to construct the SC-Arena benchmark were present in the pretraining or fine-tuning data of the evaluated models. \n\n- The rationale for model selection and the fairness of comparisons (e.g., fine-tuning protocols, input formats) could be better discussed. \n\n- The knowledge-augmented LLM-as-a-judge is promising, but its reliability and potential biases should be discussed further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4yPs3eN2H9", "reviewer": "Reviewer_6o9F", "summary": "This paper proposes SC-ARENA, a new benchmark to evaluate large language models (LLMs) in single-cell biology. The core ideas are:\n1. Virtual Cell abstraction\nTreats a model as if it were a “virtual cell.”\nThe “cell” has attributes (identity/state) and methods (how it responds to the environment).\nThis unifies evaluation across different biological tasks instead of testing each task in isolation. \n2. Five natural language tasks\nEach one probes something biologically meaningful:\nCell Type Annotation (CTA): Given expression → predict ontology cell type.\nCell Captioning (CC): Given expression → describe the cell in natural language.\nCell Generation (CG): Given a cell type description → generate a plausible “cell sentence” (i.e. pseudo-expression profile).\nPerturbation Prediction (PP): Given baseline + perturbation, predict changes and post-perturbation state.\nScientific QA (SQA): Answer mechanistic, literature-grounded biological questions. \nThese map both static identity and dynamic behavior (how a cell changes under interventions), which is exactly what one would expect from a “virtual cell.” \n3. Knowledge-augmented evaluation\nInstead of BLEU / ROUGE / exact match, which fail badly in biology, SC-ARENA uses an LLM-as-a-judge but grounds the judge in external biological knowledge (Cell Ontology, Gene Ontology, UniProt, CellMarker, PubMed evidence, etc.).\nThe evaluator scores answers and produces an interpretable rationale with references, not just a number. \nThey show this judgment correlates with real biological hierarchy (e.g. closer ontology terms → higher score; Spearman ρ≈0.62, p<0.001). \n4. Empirical study\nBenchmarks general LLMs (Qwen2.5/3, GPT-4o, DeepSeek-R1, Kimi-K2) and domain-specific single-cell models (C2S, scGPT, scGenePT, Cell-O1).\nFinds: no model is good at everything; general models are fluent but biologically shaky, and domain models are precise in narrow skills but weak elsewhere. \nFor example, captioning and science QA get the best scores (~60–74/100 for top models), but perturbation prediction and mechanistic reasoning remain very weak (<38/100). \nSmall domain models like C2S actually beat giant general LLMs on cell type annotation, which shows specialization can outperform sheer parameter count for grounded biology.", "questions": "1. Knowledge-Augmented Judging\nWhat prevents the LLM-as-judge from being biased by language fluency rather than biological accuracy?\nHow reproducible are scores across judges (e.g., GPT-4o vs DeepSeek-R1) or seeds?\nHow do you ensure the judge doesn’t “reward” models that use familiar ontology terms but misstate mechanisms?\n\n2. Dataset and Coverage\nAre the 600-sample CellxGene subset and 138 perturbations enough to represent cellular diversity?\nCould SC-ARENA generalize to datasets from other species, tissues, or modalities?\nAre there biases toward well-studied cell types (e.g., immune, epithelial) due to ontology density?\n\n3. Benchmark Design\nWhy were all five tasks weighted equally in the “Total Score”?\n→ Would weighting causal tasks (e.g., perturbation prediction) more heavily yield a different ranking?\nDid the authors test whether models overfit to the language style of prompts rather than underlying biology?\nHow much variance exists across repeated evaluations (inter-run consistency)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Unified evaluation via the Virtual Cell abstraction\nInstead of 5 unrelated leaderboards, SC-ARENA frames everything around one object: the Virtual Cell. The “attributes” vs “methods” split is elegant, and it matches how experimentalists think (cell identity vs response to perturbation). This is novel and, importantly, extensible. \nThis framing makes it possible to talk about “does an LLM qualify as a virtual cell model?” instead of “did it get 2% better BLEU.” That’s a conceptual contribution, not just engineering.\n2. Knowledge-augmented evaluation is thoughtful and genuinely useful\nThe work directly addresses a known failure mode of LLM evals in science: surface-similarity metrics like BLEU and ROUGE often (a) reward buzzwords, (b) fail to punish mechanistic nonsense.\nHere, the evaluator:\nretrieves ontology / marker / pathway / literature evidence, explains why it scored something, and aligns with biological hierarchy (ρ=0.6212 between ontology distance and evaluator score).", "weaknesses": "1. The benchmark leans heavily on LLM-as-a-judge, which is itself another model\nYes, they mitigate this with knowledge retrieval and ontologies. Yes, they validate correlation with ontology distance. But the scoring model is still an LLM (GPT-4o-mini), which raises standard questions:\nHow stable is the score across judge variants / seeds?\nCould a model “game” the judge by mimicking citation-sounding language and pathway buzzwords?\nIs there any adversarial testing (e.g., hallucinated but authoritative-sounding nonsense)?\nThey partially address alignment with expert judgments and ontology distance, but a deeper robustness audit (or inter-judge agreement across two different judges) would make the claim stronger. \n\n2.Ground truth for some tasks is underspecified\nFor perturbation prediction (PP), the model is supposed to say:\nwhich genes go up/down, and\nproduce a plausible “post-perturbation cell sentence.”\nBut evaluating free-form gene-level differential expression is biologically thorny:\nThey say they use DEGs from Norman/Adamson and external knowledge bases (GO, UniProt, NCBI) to judge plausibility. \n\nWhat isn’t fully clear is: how do you distinguish a true novel hypothesis from a hallucination? If a model proposes a plausible but previously unreported compensatory pathway, does it get penalized or rewarded?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)