AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite
Authors: Jonathan Bragg
Mike D'Arcy
Nishant Balepur
Dan Bareket
Bhavana Dalvi Mishra
Sergey Feldman
Dany Haddad
Jena D. Hwang
Peter Jansen
Varsha Kishore
Bodhisattwa Prasad Majumder
Aakanksha Naik
Sigal Rahamimov
Kyle Richardson
Amanpreet Singh
Harshit Surana
Aryeh Tiktinsky
Rosni Vasu
Guy Wiener
Chloe Anastasiades
Stefanus Candra
Jason Dunkelberger
Daniel Emery
Rob Evans
Malachi Hamada
Regan Huff
Rodney Kinney
Matt Latzke
Jaron Lochner
Ruben Lozano-Aguilera
Ngoc-Uyen Nguyen
Smita Rao
Amber Tanaka
Brooke Vlahos
Peter Clark
Doug Downey
Yoav Goldberg
Ashish Sabharwal
Daniel S Weld
Venue: ICLR 2026 Oral (2026)
Keywords: Agents
evaluation
benchmarks
scientific research
Primary area: datasets and benchmarks
TL;DR: We present principles and tooling for rigorous AI agent benchmarking, instantiated in AstaBench—the first holistic measure of agentic ability for scientific research—plus experiments showing AI remains far from solving research assistance.

Abstract: AI agents hold the potential to revolutionize scientific productivity by automating literature reviews, replicating experiments, analyzing data, and even proposing new directions of inquiry; indeed, there are now many such agents, ranging from general-purpose "deep research" systems to specialized science-specific agents, such as AI Scientist and AIGS. Rigorous evaluation of these agents is critical for progress. Yet existing benchmarks fall short on several fronts: they often (1) lack reproducible agent tools necessary for a controlled comparison of core agentic capabilities; (2) do not account for confounding variables such as model cost and tool access; (3) do not provide standardized interfaces for quick agent prototyping and evaluation; (4) fail to provide holistic, product-informed measures of real-world use cases such as science research; and (5) lack comprehensive baseline agents necessary to identify true advances. In response, we define principles and tooling for more rigorously benchmarking agents. Using these, we present AstaBench, a suite that provides a holistic measure of agentic ability to perform scientific research, comprising 2400+ problems spanning the entire scientific discovery process and multiple scientific domains, and including many problems inspired by actual user requests to deployed Asta agents. Our suite comes with the first scientific research environment with production-grade search tools that enable controlled, reproducible evaluation, better accounting for confounders. Alongside, we provide a comprehensive suite of nine science-optimized classes of Asta agents and numerous baselines. Our extensive evaluation of 57 agents across 22 agent classes reveals several interesting findings, most importantly that despite meaningful progress on certain individual aspects, AI remains far from solving the challenge of science research assistance.

Published as a conference paper at ICLR 2026
ASTABENCH: RIGOROUS BENCHMARKING OF AI AGENTS WITH A SCIENTIFIC RESEARCH SUITE
Jonathan Bragg'
Mike D' D'Arey'
Nishant Balepur
Dan Bareket
Bhavana Dalvi!
Sergey Feldman'
Dany Haddad
Jena D. Hwang'
Peter Jansen 1.3
Varsha Kishore 166
Bodhisattwa Prasad Majumder'
Aakanksha Naik²
Sigal Rahamimov!
Kyle Richardson'
Amanpreet Singh* Guy Wiener
Harshit Surana
Arych Tiktinsky'
Rosni Vasu
Chloe Anastasiades
Stefan Candra'
Jason Dunkelberger' Rodney Kinney?
Dan Emery!
Rob Evans*
Malachi Hamada*
Regan Huff?
Matt Latzke"
Jaron Lochner- Ruben Lozano-Aguilera'
Cecile Nguyen* Smita Rao
Amber Tanaka'
Brooke Vlahos'
Peter Clark Doug Downey Yoav Goldberg!.S
Ashish Sabharwal' Daniel S. Weld
1 Asta Team, Allen Institute for AI, University of Maryland, 3 University of Arizona, A University of Zurich, BBar-Ilan University, University of Washington, "Wokk performed while at Ai2 Asta Team.
ABSTRACT
Al agents hold the potential revolutionize scientific productivity by automating literature reviews, replicating experiments, analyzing data, and even proposing new directions inquiry; indeed, there are now many such agents, ranging from general- purpose "deep research" systems to specialized science-specific agents, such as Al Scientist and AIGS. Rigorous evaluation of these agents critical for progress. Yet existing benchmarks fall short on several fronts: they often (1) lack reproducible agent tools necessary for controlled comparison of core agentic capabilities; (2)
do not account for confounding variables such as model cost and tool access; (3) do not provide standardized interfaces for quick agent prototyping and evaluation; (4)
fail provide holistic, producctinformed measures of real-worl world use cases such science research; and (5) lack comprehensive baseline agents necessary to identify define principles and tooling for
advances. In response, benchmarking agents. Using these, we present AstaßBech, suite that provides a holistic measure of agentic ability to perform scientific research, comprising 2400+ problems spanning the entire scientific discovery process and multiple scientific domains, and including many problems inspired by actual user requests deployed Asta agents. Our suite comes with the first scientific research environment with production-gree gradde search tools that enable controlled, reproducible evaluation, better accounting for confounders. Alongside, we provide comprehensive suite of nine science-optimized classes of Asta agents and numerous baselines. Our extensive evaluation of 57 agents across 22 agent classes reveals several interesting findings, most importantly that despite meaningful progress on certain individual aspects, AI remains far from solving the challenge of science research assistance.
rigorously
INTRODUCTION
AI agents increasingly being applied complex real-world cases. In particular, they hold the promise to revolutionize scientific productivity by automating reviews of the literature, replicating complex experiments, analyzing high volumes of data, and proposing avenues explore.

---

which enables defining a benchmark suite and leaderboard with time-invariant cost accounting using model usages logged by Inspect (UK AI Security Institute 2024). a standard agent evaluation framework that provides broad model and evaluation compatibility. Published as a conference paper at ICLR 2026
Large organizations such OpenAl and Google are investing in ceneral-purposs "deep research" systems to help everyone, including scientists, comb through literature much more effectively. We even have specialized science-specific agents, such AI Scientist (Lu et al. 2024 Yamada et al.) 2025) and AIGS (Liu et al., 2024), targeting scientific research. With so many different agents-many behind paywalls and all evaluated in bespoke ways-how how are end users and Al developers know which perform best?
Unfortunately, existing agent benchmark suites have several deficiencies, when considered as general measure of Al skill, including for their ability to do scientific research (Table J. First, suites often lack the standard task environments and tools necessary for realistic, controlled comparison of agents on a level playing field; for example, no large-scale, controlled document retrieval tools exist, making unclear whether a 'winning' agent has superior Al capabilities or merely access to a more relevant information source. Second, they fail to properly account for confounding variables; we are unaware of agent benchmarks that account for variations in tool usage, and only a few like HAL (Kapoor al. 2025) measure cost, which is critical since even simplistic strategies (e.g.. taking majority vote over repeated invocation) can boost accuracy by spending more (Kapoor rarely standardized for use by general agents, since suite developers typically assume either that users will evaluate only agents that come with the suite (and so it is fine for evals to be coupled to agents, as in the case of OpenHands (Wang) iet al. 2025) or AutoGen (Fourney et al. 2024)) or that users will build only specialized agents for specific benchmarks (as is the case with general suites like HAL (Kapoor al. (2255). Mcasuring new agents on a full suite typically requires time-consuming interventions ranging from extensive decoupling manually clarifying task instructions that were not written with general agents in mind; this harms reproducibility and controlled comparison. Fourth, they often lack tasks that are informed by authentic product usage data (typically guarded by technology companies), raising concerns that higher scores may not lead to meaningful real-world benefit. Finally, benchmark suites lack comprehensive agent baselines for proper comparison. As a result, most published evaluations only compare to small number of other agents or ablations, making it difficult to assess whether claimed improvements represent genuine advances.
et al. 2024). Third, benchmark suite interfaces
In response, we present a set of benchmarking principles and benchmark suite, built upon these principles, that overcomes the aforementioned limitations, along with open-source resources that enable more rigorous, comprehensive measurement, Specifically:
We formalize principles for rigorously benchmarking agents (Appendix A which address key limitations of current agent benchmark suites.
Guided by our principles, we present Cecaaeeeeet((cttiiooon 3 more rigorous agent benchmark suite that is holistic measure of scientific research, which exercises a broad spectrum of skills- including literature understanding, data understanding, planning, tool use, coding, and search- and comprises over 2400 problems spanning the full scientific discovery process and multiple scientific domains, including many problems based on real user requests from Asta where we have deployed several of our agents for public use. It is easy to integrate new general agents with AstaBench, which provides a standardized task interface.
AstaBench includes the powerful Asta Environment (Section D. the first agent environment that enables controlled, reproducible evaluation with production-graan search tools for retrieving information from a large corpus of scientific literature.
We also introduce the aeeat-eval Agents Evaluation Toolki (Section 4.2).
We introduce AstaBench Leaderboard built using this Toolkit. It's the first agent leaderboard to properly account for confounding variables such as the tools used by the agent and inference cost.
https://asta.aleen..
https://qithub.com enai http:///tttabbboooo/111eeeeeeeee
2

---

Published as a conference paper at ICLR 2026
Table AstaBench improves over existing agent benchmark suites in several ways. tests holistic scientific reasoning (i.e., a broad spectrum of task types and across more than one scientific domain). Many of its problems are inspired by actual user requests our deployed Asta agents. Its standard tool environment isolates core agentic abilities (e.g., planning, tool-calling, etc.) from information access. AstaBench's 's scoring controls for confounders, such as computational cost, and its tasks are defined using uniform format that supports general-purpose agents. The table's s final column, titled 'Cls.', indicates the number of agent classes (c.g., ReAct) that are used to instantiate (c.g. with specific LLMs) the total number of agents listed in preceding column; AstaBench includes more classes agents than prior benchmarking efforts.
Relevant for all agent benchmarks
Holistic sci. reasoning
Product usage-based
Controlled,
Scoring accounts for confounders
Tasks ready for general agents
Agents Total Cls.
realistic tools
AstaBench
Broad (weighted
Lit. tasks
Prod, -rrdd lit. corpus
Costs, tools, openness
Decoupled, with standard
57
22
towards CS)
formats
AutoGen-
X No science
Coupled to agent framework
11
Bench
BixBench
Bio data
X
X
Non-standard
2
science
notebook tools
BrowserGym
x No science
X
v Ready for
10
2
weh agents
HAL
Coding
X
Costs
Non-standard
113
10
formats
Inspect
Coding, knowledge
X
X
Non-standard
18
Evals
formats
LAB-Bench
Bio
Non-standard
12
formats
OpenHands
Coding, data analysis
X
Costs
Coupled to agent framework Coupled to agents
53
6
Evals
ScienceAgent ~ Data
Costs
17
3
Bench
analysis
Terminal-
Coding
X
X
X
v Ready for
33
12
Bench
terminal agents
Vector Inst.
x No science
X
Non-standard
Leaderboard
formats
AstaBench
Agent
LLM
Pubric LLM Judge
Standard Tools
Compute Cost
Figure 1: Using AstaBench we evaluated 22 agent classes on a diverse set of science tasks while controlling the set of available tools, e.g., to ensure each agent has access to the same set of scientific papers. AstaBench leaderboards record not just agents accuracy but also how much computation required to achieve that performance.
Finally, we present the agent baaett-baeliii bhes Agents Suite (Section 4.3) the most comprehensive standardized agents suite, comprised of nine Asta agent classes that have been optimized for scientific research tasks, well numerous baselines.
Together, the AstaBench benchmark suite, agent environment, agents suite, and leaderboard enable a holistic measurement the current state of LLM agents for scientific research assistance, well as a path for continuous improvement (Fig. I We report on an extensive set of experiments on AstaBench using our agents suite with 57 agents spanning 22 classes of agent architectures. ranging from sasc-spec agents such as Asta Scholar QA and Asta CodeScientist to generic, ReAct-style architectures applicable to the broad range of benchmarks within AstaBench. We find that while
https://githubb.oo
/agent -batt
3

---

Published as a conference paper at ICLR 2026
meaningful progress has been made on many fronts, science research assistance remains far from solved. Section summarizes our findings, with more details in the appendices.
These findings provide a current snapshot of the state of scientific research assistance agents. But this is only a starting point. AstaBench offers the ability to help the community continually and systematically assess progress (or lack thereof) as new agents designed, something that has been difficult to do holistically. We hope AstaBench will continue to serve as a valuable guide for the development of future agents through its clear targets, cost-aware performance reporting, and transparent evaluation regimen.
2 RELATED WORK
Our efforts relate to two recent threads of research: the development of holistic agent evaluations that test wide range of LLM-driven automation (for general review, see Yehudai et al. (2025) and the development of new benchmarks for measuring the scientific reasoning of LLMs and their use as scientific assistants and agents (Wang et al. 2023). We consider each in turn.
Holistic Agent Evaluations The last few years have seen a surge in benchmarks and evaluation frameworks that attempt to holistically measure the reasoning abilities of LLMs (e.g.. Gu al. 2025; Gao et al.] 2024; Habib et al., 2023] Guha et al., 2024). Given the rise of LLM-driven automation, recent efforts have centered around new benchmarks and frameworks for evaluating LLM agents. Ta highhlihts recent efforts that are most closely related to AstaBench in terms of their scope holistic science agent benchmarks: AutoGenBench (Fourney al.) 2024), BixBench (Mitchener et al. 2025), BrowserGym (Le Sellier De Chezelles et 2025), the Holistic
Agent Leadcrboard (HAL) (Kapoor et al. 2025), Inspect Evals (UK AI Safety Institute and Arcadia Impact and Vector Institute (022)) Lab-Bench (Laurent et al. (024)) OpenHands Evals (Wang ctal. ScienceAgentBench (Chen et al., (0255), Terminal CerrminalBBenchh (The Terminal- Team
2025a), and the Vector Institute Leaderboard (Vector Institute 2025 We compare these efforts to AstaBench across the following dimensions: holistic scientific reasoning (i.e., focuses on broad spectrum of task types and across more than one scientific domain), product usage-based (i.e., involves tasks based on product use cases), controlled, realistic tools (i.e., distributes standard, realistic tools that allow for controlled comparison of agents), scoring accounts for confounders (i.e., scores systematically account for cost, controlled tool use, and other confounders). general agents (i.e., tasks have uniform formats that support general- -urppos agents), and number of agents (i.e., total number and number of different classes of agent).
AstaBench stands out these dimensions, which key to advancing scientific Al and increasing benchmarking rigor generally (Appendix(A). In terms science, the other agent benchmark suites all less holistic, either more limited in terms of task category (e.g.. HAL'S only science tasks are coding tasks) or the domain (e.g.. LAB B- B-Bench is limited to biology); AstaBench is also the only benchmark to leverage data from companion product (Asta) in its tasks. Despite its importance, few suites have seriously focused on cost (HAL is an exception), and none have distributed standard tools that are decoupled from agents or agent frameworks. While some leaderboards are scaling up the number of agents they test (again, notably HAL), all test far fewer agent classes (architectures) compared to AstaBench, which also distributes open-source code for these agent classes through agent-basell patasseinne lines Agents Suite.
Science Benchmarks and Agents for Science Naturally, the rise of powerful large language models (LLMs) has led to much recent interest in LLM-driven approaches to scientific rescarch-relater tasks. Many new benchmarks have been developed, often focusing on particular sub-problems in the full
research pipeline, including scientific coding and execution (Tian et al. 2024 Lai et al. [023] Chen
et al.
2025a Chan et al. 2025 Huang et al. 2024), data analysis (Majumder et al. 2025
Xu et al.
2025
research reproduction (Bogin al. 2024| Siegel et al. [025] || Tang et al. 2025 Kon et al.
2025
Xiang et al. 2025; Sttrace et al. 2025 Zhao et al.] 2025 Yan et al.] 2025). ideation and
hypothesis generation (Ruan et al. 2024 Si et al. 2024 Vasu al. 2025), and literature retrieval
and understanding (Shi et al. 2025 He et al. 2025). among others (Zhu et al. 2025). AstaBench
Aggett counts for Table 1 were derived from live leaderboards and repositories accessed August 2025, in addition to the canonical benchmark references (Microsoft) 2024 [ServiceNow] 2025 SAgE Team, Princeton
University 2025| Arcadialmpact / UK Government BEIS Team 2025 All-Hands-Al
2025a The Terminal
Bench Team 2025b).
4

---

Published as a conference paper at ICLR 2026
spans many of these task categories, and provides the most comprehensive evaluation of scientific agent performance to date (Table
Increased LLM capabilities have led emergence of a host of agents for end-to-end, open-ended sci- entific discovery, including Al Scientist (Lu et al. 2024 Yamada et al. 2025), Agent Lab (Schmidgall et al. 2025), AIGS (Liu et al.) 2024), and CodeScientist Jansen et al. 2025 among others (Cheng et al. 2025p. To bring clarity to this area (and accelerate its progress), AstaBench introduces a new end-to-end task that evaluates an agent's 's ability to complete research project, starting from an idea and ending with a written report and code. We believe this task is a useful complement to the many existing benchmarks that focus on more narrow problems in the research pipeline.
ASTABENCH: A HOLISTIC SCIENTIFIC RESEARCH BENCHMARK SUITE
We present AstaBench, the first benchmark suite for holistic evaluation of agents' ability to perform scientific research. Crucially, our suite is reproducible even as science progresses, since comes with the first realistic, reproducible search tools (Section[4). Our suite implements new standard interface for agent benchmark suites and provides time-invariant cost reporting through the agent-eval Agents Evaluation Toolkit (Section 4.2) As such, AstaBench is ready for use by new general agents such as those in our agent baselines suite (Section
AstaBench comprises the following 11 benchmarks (summarized in Table 2 with full details in Appendix E and example inputs in Appendix H note that AstaBench uses slightly modified tests an agent's ability to handle challenging scientific search queries. Li Lit http:///teee//e/ttoollll/a wtwps//----otite------errreeeee- (Skarlinski ct al. 2024) measure an agent's 's ability to answer questions and retrieve papers within the biomed- ical domain. ScholargA-CSS tests agent's ability to answer long-form scientific questions. ArxivDIGESTables -Cean (Newman et al. 2024) tests an agent's ability to create a literature review table.
versions of some of the cited datasets);
(Bogin al., 2024) tests the ability of code agents to up and execute Python machine learning experiments reported in ML and NLP papers. CORE-Bench- Hard Siegel et al. 2025) tests agent's ability reproduce experiments and analyses from papers] DS-1000 (Lai et al. 2023) tests the ability of agents on data science tasks encountered in research. DiscoveryBench MMjumdder al. 2025) tests whether the agent can automatically find and verify hypotheses from given dstaset(s). R2Z-Bench/E2E Bencc-enhh---- Bench-Hard test whether agents can perform the full research pipeline of ideation, planning, (software) experiment design, implementation, execution, analysis, and producing final report.
Full details of how these tasks are scored can found in Appendix Some use LLMs as judges to evaluate outputs against rubrics iepperFiddingBencee ScholarCA- clean, Di piscoveryBench, Ben- 12E-Bench-Harde while others use programmatic eval- uation MedittgA-Fulleeee DS-1000).
Fextt Search, SUPER-Expert, CORE -ORE-
ASTA ENVIRONMENT
Asta Environment is, our knowledge, the first realistic, reproducible scientific research environment for agents. It provides standardized tools, an evaluation toolkit, a leaderboard, and numerous agents.
4.1 STANDARD TOOLS FOR AGENTS
Asta Environment provides a comprehensive set of standard tools for science research assistance, from which each AstaBench task includes a specific subset based on its requirements (Table 2)
Asta Scientific Corpus: A toolset for accessing the scientific literature, which represents the first production grade, reproducible search tools for agents. These tools can restrict outputs to papers preceeding a date; AstaBench uses this feature to limit results the date of benchmark creation so that new papers do not contaminate results (see cutoffs for specific tasks in Table 2 The snippet_
search tool can further restricted to papers with specific IDs so that it can be used as a text retrieval
"ORRK- "CORk-BBenh -ench-Hardd"" omits GPU-requiring tasks from the original CORF-Bennc--e Bench-Hard; Appendix P
5

---

Published as a conference paper at ICLR 2026
Table 2: AstaBench benchmarks, spanning four task categories: Literature Understanding, Code & Execution, Data Analysis, and End-to-End Discovery. Benchmarks are fully reproducible when paired with the Asta Environment tools listed in the "Tools' column, which come standard with each benchmark: Computationa
Notebook (Code) or Asta Scientific Corpus (Corpus) tools that restrict to papers before the specified 'Date Cutoff' (exclusive). (Original datasets were filtered to ensure questions are answerable with the environment.) Forr GESTT Sereesaaall--eeeeeeee es-Clean, corpus tools are restricted to snippet search with specific paper IDs for each problem. indicates created by us, and i indicates previously unreleased.
Name
Task category
Domains
Test
Val
Tools
Date Cutoff
PaperFindinggench "t LitgA2-PullText-Seeae Search
Lit. Und. (search)
CS
267
66
Corpus
2025-06-01
Lit. Und. (scarch)
Biology
75
10
Corpus
2024-10-17
ScholarOA-CS2 "f LitgA2-FullText
Lit. Und. (report)
CS
100
100
Corpus
2025-05-01
Lit. Und. (MC)
Biology
75
Corpus
2024-10-17
ArxivODIGESTablls-CCeea
Lit. Und. (table)
Mixed
100
70
Snippet?
Paper IDs
SUFER-Expert
Code & Exec.
CS
45
50
Code
CoRE-Bench-Hard"
Code & Exec.
Mixed
37
35
Code
DS-1000
Code Exec.
CS
900
100
Code
Discoveryhench
Data Analysis
Mixed
239
25
Code
E2E "2E-Bench "t
End-to-End Disc.
CS
40
10
Code
B2E-Bench Bench-Hardd"" "t
End-to-End Disc.
CS
40
10
Code
mechanism over those papers (useful for detailed literature analysis, e.g., in ArxivDIGESTables= Clean). It provides the following specific tools via the MCP (Model Context Protocol) standard: sni anippet_search search _earch_papers____________ sepperr__y__r_eeeee her_ebyyrreeeeeeeeeeeeeeeeeeeeee get_paper, get__aper__bate get_"
citations, search_ bearch_authors_by.. sartooss_by__mmm_e get_ vet__uuhoo_pppperee search_paper__y__tin
Computational Notebook: A stateful computational (Jupyter) notebook. The tool can execute Python code as well as standard IPython magic commands like $twritefile, $matplot lllpooiibb Python variables and environment are maintained between calls so that the tool can used to solve problems incrementally. By default, the tool returns a timeout message to the agent if single cell takes more than 5 minutes to execute. Since the tool needs to execute code, it lives in a new sandbox image that's 's created by the framework.
inline, and schell_commaal
Our tools feature improved agent compatibility compared other suites. They cleanly decoupled from agents and provide easy integration via MCP. Code executed in our sandbox can call tools provided by the main (host) execution environment (e.g., Asta Scientifis Corpus), enabling
testing of code execution agents, e.g., agents that implement the CodeAct (Wang et al. 2024 pattern.
4.2 AGENT-RVAL EVALUATION TOOLKIT & ASTABENCH LEADERBOARD
We use Inspect (UK AI Security Institute 2024 as the framework for implementing our individual agentic benchmarks, as it provides broad model provider and tool compatibility, useful logging and debugging affordances, and a growing set of compatible evals (UK Al Safety Institute and Arcadia Impact and Vector Institute| 2025). However, Inspect logs only model usages (not normalized dollar amounts) and lacks tooling for defining benchmark suites with unified scoring or leaderboards. To fill this gap, we present the agent-eval ageett leaderboard toolkit, which provides benchmark suite, reporting, and leaderboard layer on top of a suite of Inspect-forme formatted benchmarks; it features:
The agent-eval toolkit computes normalized dollar costs based on model usages logged through Inspect. For mapping model usages to prices, we use a frozen snapshot of the litellll 1m cost map, which is community-sourcel for broad model coverage It factors in cache discounts for agents that take advantage caching, as this an increasingly adopted optimization technique (and providers like OpenAl provide these discounts automatically), however,
Time-invariant cost calculation:
https: / thub. com Tnna /aggenteeeer We supplement the cost map with prices for custom models based on Together Al (ht http://wwww/tt/ toge ther.ar ai/) generie model size-hased pricing.

---

Published as a conference paper at ICLR 2026
it does not factor in any latency-relan related discounts (e.g., service tier batching). Using frozen snapshot allows a fair comparison of evaluation costs even if API prices change between evaluations To
Reporting that accounts for confounders: In addition to cost, the agent - toolkit and leader- boards categorize agent evaluation submissions according to their reproducibility and degree of control based on the following dimensions (full definitions in Appendix (B)
Agent openness (is the agent implementation open?): Open-source, open- weight (V), Open-
source, closed-weight (~)) Closed source & API available (A), or Closed & UI only (x)
Agent tooling (does the agent use the provided standard tools for the tasks?): Standard (V), Custom interface (~~), or Fully custom (x)
Leaderboard web interface: In addition to the agent-eval CLI-based leaderboard interface (which requires authentication currently unavailable to the public for AstaBench), we also include a web application interface for the AstaBench Leaderboard which supports external submissions (with Hugging Face user-based authentication) and provides interactive plots and tables.
4.3 AGENT BASELINES AGENTS SUITE
To enable comprehensive measurement on AstaBench and other benchmarks and advance the state of the art-we provide the agent-based ines Agents Suite["] which consists of a large set of agents from 16 agent classe with standard Inspect-compatible interface. Table lists these agents, grouped into (1) the Asta agents that optimized for scientific research tasks and (2) numerous baseline agents that we evaluate. Detailed descriptions are deferred to Appendix
EXPERIMENTS
We now present experimental results, which we have also used to seed the interactive AstaBench leaderboard 14 Our experiments were conducted over period of several months. Since one may boost scores by using more compute (eg using repetition and majority vote) (Dodge et al.] 2019), report cost as well as accuracy. We also report the standard deviation of our measurements. For brevity, when an agent was tested with multiple different models, we report the top result(s) plus any other significant data points. The entire set of results, plus plots of scores vs. costs including the Pareto frontier (showing the best agent for a given cost), are Appendix D
Some agents (e.g., ReAct) can attempt all 11 benchmarks; others are category-specific benchmark-speciffc. Table shows the overall results for those agents attempting all benchmarks, as well as agents that can solve all the benchmarks in least one category. Category- and benchmark- specific results are presented in Appendix C for space reasons.
As noted above, agents powered by closed weight LLMs currently far exceed the reach of those powered by open weight LLMs. On the other hand, simply switching the underlying LLM with the latest and greatest one isn't 't necessarily a reliable recipe for success on AstaBench. one of the newest LLMss gpt-5, provides only a modest boost over an earlier "reasoning LLM",, c3, except on three benchmarks. In fact, gpt-5 hurts the performance of several specialized agents.
case point,
Tools designed specifically for science research assistance can significantly help Al agents. This is most noticable with Asta vo, which scores ~9% higher than the next best agent, ReAct with 9pt-5 (53.0% vs. 44.0%). However, this comes with the trade-off of significantly higher development (enginecring) cost, and (for some tasks, specifically in to-endddiscooee
higher inference cost.
None of the commercial scientific research agents were able to perform the full range of research tasks in AstaBench. The best such API-based agent (Fut uroHouu Falcon) and the best closed
The cost map snapshot used for the leaderboard may he periodically updated, but we will always re-calculate all costs based on the current snapshot to ensure fair comparison. hhttps://aa lenai wttpeaa....a.aaaaaaaeeeeeeeeeeee
htttps:////t thub. com/allae ena /at///ggetttbtt basel nes
"Slightly less than the 22 we evaluate hecause some are closed source and thus not usable on new inputs; however, we provide ways to reproducing those results hased on cached answers obtained for our experiments. "https://allen lenai www.aaaaa...gg/aaaaaaaaaaoooo Teaderbaa
7

---

Published as a conference paper at ICLR 2026
Table 3: Agent classes in the agent-basel Agents Suite, with Asta agents in the top section and baseline agents the bottom section. "Standard" tooling means that the only tools used are the ones distributed with the AstaBench tasks; "Custom interface" means that standard date-restricted search is used but additional custom tooling may be used; "Fully custom" means that tooling is custom and standard search tools not used.
Name
Task optimization
Open- source
Tooling
Asta Paper Finder
Lit. Und. (scarch)
V Yes
Custom interface
Asta Schclar QA
Lit. Und. (report)
Yes
Custom interface
Asta Scholar QA (w/ Tables)
Lit. Und. (report)
V Yes
Custom interface
Asta Table Synthesis
Lit. Und. (table) V Yes
Custom interface
Asta Code
Code & Execution
V Yes
Custom interface
Asta DataVoyager
Data Analysis
V Yes
Custom interface
Asta Panda
End-to-End Disc.
Yes
x Fully custom
Asta CodeScientist
End-to-End Disc.
Yes
x Fully custom
Asta v0
Multi
V Yes
X Fully custom
ReAct
None (general)
Yes
Standard
Smol moolagents Coder
None (general)
V Yes
Custom interface
You. Search API
Lit. Und. (search)
X
X Fully custom
Elicit
Und. (report)
X
X Fully custom
FutureHcuse Crow
Lit. Und. (report)
X
X Fully custom
FutureHcuse Falcon
Lit. Und. (report)
X
X Fully custom
OpenAI Deep Research
Lit. Und. (report)
X
X Fully custom
OpenSeiLM
Und. (report)
Yes
Custom interface
Perplexity Sonar Deep Research
Und. (report)
X
x Fully custom
SciSpace Deep Review
Lit. Und. (report)
X
x Fully custom
STORM
Lit. Und. (report)
V Yes
X Fully custom
You. Research API
Lit. Und. (report)
X Fully custom
Faker
End-to-Ena Disc.
Yes
Standard
Overall
Literature Understanding
Efficiency Frontier
0.6
Asta Panda (claude sonnet-4)
(mixture)
Score
0.4
Asta, CodeScientist (claude-1--ssenme))
(spt-4.11)
0.2
React (claude- 3-5-haikal
React (gpt-4.1)
(ppt-5)
10-?
10-1
10"
10-7
10-1
React (gpt-5-mini)
Code & Execution
Data Analysis
Smolagents (apt 4.11
0.6
0.4
Smolagents (gpt-5)
Smolagents (apt 5-min))
Score
Smolagents (llama 4-soutt
Asta DataVoyager sonneeesoonet gpt-4ot)
Asta DataVoyager (gpt-4.1t gpt-4ot!
Datavoyager (apt 5t. opt-4ot;
10
10-1
10"
0.0
10-1
10-2
10
Asta DataVeyeger tppt-mreffort:mmnimme gpt-4ot!
End-to-End Discovery
Cost (USD)
Asta Panda (gpt-4.11)
DataVoyager gpt-4ot)
React (cluude-soeeet-4)
0.6
(gpt-4o)
Score
React (gemini-2 5-flash)
(lamm-4-scout)
Smolagents (ccuude3-SSiaae haSih
Smolagents (ellude-sonet--d
Smelagents (gpt-4o)
Smolagents (gemini-2.5-flssh))
10
Cost (USD)
Figure 2: Score vs. cost analysis for overall and category results (from Tables Points indicate means. Points on the Pareto frontier connected with dotted lines, representing optimal quality-cost trade-off's for each category (Literature Understanding, Code & Execution, Data Analysis, End-to-Ent Discovery). i denotes models not pinned stampst version. Note: the x-axis (cost per answer in dollars) uses log scale. For more detailed plots for individual categories and benchmarks, see Appendix D
16 and 17).
8

---

Published as a conference paper at ICLR 2026
one (OpenAI Deep Research) score well on literature understanding, but are unable to perform the full spectrum of science research assistance.
Science research assistance is still far from solved, evidenced by the generally low overall scores for the full gamut of agents, from fully open to fully closed. For example: The best open source agent with open weights LLMs scores terrible 11 11.1% % Emmolagents Coder with 162-Instruct) (Table (4). The best open source agent with closed LLM(s) is much better: 53.0% (Asta vO) (Table 4). While the best API-based agent (FutureHouse Falcon) and closed agent (OpenAI Deep Research) score well on a single benchmark (Table 6). they are stymied by the full range of tasks.
The post-performance tradeoff across agents. highlighted by the Asta leaderboard's Pareto curve provides several interesting insights. The best economical model ReAct with gpt-5-mini. scoring 11%6-wtthi 21 % (absolute) of the best performing models costing over an order of magnitude less at $0.04 per problem.
Powering a general agent with an expensive model can lower the overall cost. Though the per-token cost is to 25 times lower for gemini-flash and llama-scout compared to o3 or sonnet, the weaker models often take more steps get stuck in loops, causing ReAct agent to end up being twice as expensive in addition to lower-performing.
Surprisingly, most of our specialized agents (Asta Scholar QA (Table 6 Asta DataVeyager (Table J. Asta Code (Table 8)) perform worse with got-5 than with previous models, while ReAct performs much better. One possible explanation for this is that gpt-5 has been tuned to do well with now now-common ReAct-style workflows, and conversely may be relatively less adaptive to alternate workflows. If this is indeed true, and trends continue, there may be diminishing value in application-specific workflows.
As the LLM underlying ReAct, gpt-5's boost over o.3 is generally light, with only gain of 0%- 5% across most benchmarks. However, gpt-5 provides a huge boost in 4 benchmarks: +13.4% absolute on Scholarga-CS2 (Table + 24.8% on SUFER-Expert (Table 8) +25.3% on LitQA2- FullText-Search (Table and +21.1 % on E2E G2E-Bench-Hares (Table[0]
In general, today's agents are reasonably good at literature understanding. However, despite some recent progress, coding, experiment execution, data analysis, and data-driven discovery still remain major, unsolved problems for science assistance agents.
Literature Understanding: For literature search agents, Asta Paper Finder stands out as an impressive system, scoring much higher than its closest rival (ReAct) on aaperFLndingBenee and AatQA2-FullTees -Sarchh (Table Despite this, it is clear that the paper-finding task is far from 'solved,' requiring further work achieve truly comprehensive results.
For literature question annheirng agents, our results (Table 6 suggest that (among other things): The best models have relatively good performance this category, scoring around 80%. This likely because literature understanding has been a strong focus of many task-optimized agents in the community (or conversely, the community has targeted literature understanding because this category is particularly well suited for language models), Asta Scholar QA, Elicit, and SeiSpace Deep Review ew are the best tools on these tests (all score about 85% or higher on ScholargA- -C22 Table For all three tools, the higher performance is driven by the citation subscores of the evaluation. The other external/commerrial agents are not far behind, but also do not do significantly better than the best ReAct buseline. This is indeed surprising given ReAct'S 's simplicity. but also an indicator of the challenging nature of the task that requires system responses to be precise and cover the relevant points as well as cite the correct supporting sources for claims as necessary.
For literature review table generation agents, our results (Table 7 suggest that: even the best models do not yet achieve strong performance in this category, with recall scores around 43%, likely due to limited efforts to build task- latt-oot agents in this space. Asta Tabie Synthesis, backed by gpt-5, wins on this task, beating the best general agents. However, Asta Table Synthesis backed by gpt-5-mini also shows competitive performance, at just 13% of the cost.
Code and Execution: Coding and execution far from solved all agents score low on these tasks, e.g., all but two scored below 25% on SUPER-Expee (ReAct with gpt 5 scored 41% and

---

Published as a conference paper at ICLR 2026
gpt-5-mini scored 37%; Tables 8 and 15). Coding and execution thus remain major bottlenecks
for assisting with and automating science.
The impact of using gpt-5 5 is highly unpredictable. Surprisingly, running the general ReAct agent with gpt-5 significantly improves its performance (compared to running with other LLMs), while running the more custom-buil Smo Smolagents Coder with gpt-5 notably decreases performance. One possible pospossi explanation is that gpt-5 has been tuned for the common ReAct-style workflow, making gpt-5 less adaptive alternate workflows.
Similarly, aulomated data analysis and data-driven discovery is a major, unsolved challenge for science assistance agents. We agents struggle with this benchmark, with the maximum score being only 34% (Table despite increased attention in the community.
Data Analysis:
End-to-End Discovery: End-to-end discovery remains far from being meaningfully solved. Although the average research step completion scores appear reasonable (scores to ~70% Table TO). the likelihood of completing all experiment steps remains near zero. For example, given 10 steps per experiment, and a success rate of 70% per step. the success rate to complete all steps in the experiment will Rs 0.710 3% (see Table for actual numbers, reaching maximum of 5%). A lot more work is needed, and we hope these benchmarks will help push research forward in this direction.
CONCLUSION AND FUTURE WORK
In summary, we identify limitations of current approaches to benchmarking agents, and present methodology and tooling for doing so more rigorously. Using this methodology and tooling, we introduce AstaBench, holistic benchmark suite for scientific research that addresses key limitations. AstaBench is the first major agent benchmark suite to come with standard environment and tools that enable controlled comparison of agents: the Asta Environment, the first scientific research environment for agents with realistic, controlled search tools. Alongside, we present the agent- baselines Agents Suite, a large suite of standardized agents, which we used to conduct experiments on AstaBench with 57 agents across 22 architectural classes. This revealed several interesting findings, most importantly that despite meaningful progress certain individual aspects, agentic Al remains far from solving the challenge of scientific research assistance. We invite the community to make submissions the AstaBench Leaderboard, which is powered by our agent-eval Agents Evaluation Toolkit.
This work opens up many exciting possibilities for the agentic AI, scientific research assistance, and automated scientific discovery communities. We are actively pushing the performance-cost frontiers in AstaBench and closing the gap for truly open agents by developing new agent techniques, tools, and open models specialized for scientific research. We are also enhancing agent abilities to manage complex context, from improving on Asta vO simple orchestration techniques to handling long-duration tasks in complex research projects. We are continuing to research how to refine our aa-dde- grading procedures, especially for challenging scientific discovery tasks. We plan to develop fresh benchmark problems that use the latest scientific knowledge, which is contamination- resistant and past the training cut-off date of models. We also plan to build benchmarks that test more aspects of collaboration with humans, and deepen coverage of problems in impactful fields such as biomedicine. Finally, we are committed to continuing to measure the latest advances both by testing the latest LLMs and by adding more agent architectures to agent base lines.
ETHICS STATEMENT
We took care to adhere to a high ethics bar. We obtained legal review for all material presented in this work. The new real-world user queries used in the Literature Understanding tasks were collected with user consent. We also credit any benchmarks that we adapted for use in our suite, as well as agents that we leverage, citing those works. When measuring existing agents, worked with the agent creators where possible to ensure they are measured fairly, including Elicit, Future House, and SciSpace.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We took special care to make this work reproducible; indeed, reproducibility is value proposi- tion of our benchmark suite. AstaBench comes with open source code for all included benchmarks, agents, and core infrastructure-as well logs of all reported experiment. The framework logs and reports specific repository commits, including for data. The agent tools in AstaBench improve reproducibility by providing date-restricted access to the supporting document corpus.
AUTHOR CONTRIBUTIONS
Authors listed in alphabetical order within each section:
Project leadership, framework, and general agent development: Jonathan Bragg, Mike D' Arcy
Research by task category (benchmarks and agents):
-Literature Understanding (paper finding): Dan Bareket, Yoav Goldberg, Sigal
Rahamimov, Aryeh Tiktinsky, Guy Wiener
-Literatrre Understanding (summarization and QA): Nishant Balepur, Doug
Downey, Sergey Feldman, Dany Haddad, Jena D. Hwang, Varsha Kishore, Aakanksha Naik, Amanpreet Singh, Daniel S. Weld
Literature Understanding (table generation):
* Benchmark: Aakanksha Naik
* Agent: Mike D' Arcy, Dany Haddad, Aakanksha Naik
-Code & Execution: Mike D Arcy, Kyle Richardson
-Dat Analysis: Bodhisattwa Prasad Majumder, Harshit Surana
End-to-End Discovery: Peter Clark, Bhavana Dalvi, Peter Jansen, Rosni Vasu
Engineering:
Frameworks and leaderboard data: Chloe Anastasiades, Stefan Candra, Regan Huff, Rodney Kinney
Leaderboard web application:
Jason Dunkelberger, Dan Emery, Cecile Nguyen,
Smita Rao, Amber Tanaka, Brooke Vlahos
Management: Jaron Lochner, Smita Rao, Rob Evans
Design: Matt Latzke
Support and data annotation: Malachi Hamada
Product management: Ruben Lozano-Aguilera
Management, mentorship, and advice: Peter Clark, Doug Downey, Yoav Goldberg. Ashish Sabharwal, Daniel S. Weld
The Use Large Language Models (LLMs)) We used Al-based tools (Claude Code, Github Copilot, ChatGPT) for analyzing results data, generating code to populate plots and tables, identifying emrors and missing references, and (minor) writing assistance.
ACKNOWLEDGMENTS
This work would not have been possible without a broad and supportive community, In particular, we thank: David Albright and Kyle Wiggers for communications support and useful feedback; Crystal Nam for legal support; Ali Farhadi and Sophie Lebrecht for insightful feedback and encouragement; Stephen Kelman for design support; the creators and maintainers of the Inspect evaluation framework: the creators the external datasets that we have integrated; and the data workers who contributed to the creation of those datasets and the datasets that created.
11

---

Published as a conference paper at ICLR 2026
REFERENCES
Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao.
LitSearch: A retrieval benchmark for scientific literature search. In EMNLP, 2024. URL https?
//aclaanthology..rrr/222222 www.::244.mmmmmmammammmmm.mm 840/
All-Hands-Al. OpenHands agent hub, 2025a. http:p//tttttt /ttp:///w/..a/////aaaa.oaaaaaaaa
S/ //genthub Accessed: 2025-08-25.
All-Hands-AL OpenHands evaluation Icaderboard, 2025b. URL utLpttpp:///doc.coooo.e.
m/
6Nt QF 1t:gid=0#gid==0 Accessed: 2025-08-25.
Arcadialmpact UK Government BEIS Team. Inspect Evals Dashboard, 2025. URL https Lanpppeeeeal...abbbbrdd----eeeee 08; site down on 2025-08-25.
wtw.:/.....t..............m. app/ Accessed: 2025-07-
Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D' Arey, David Wadden, Matt Latzke, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke S. Zettlemoee, Graham Neubig, Daniel S. Weld, Doug Downey, Wen tau Yih, Pang Wei Koh, and Hanna Hajishirzi. OpenScholar: Synthesizing scientific literature with retrieval-augmmnned LMs. ArXiv, abs/2411.14199, 2024. URL puttps://app semanticscholar. owt. Corpus 27416666119
Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, and Tushar Khot. SUPER: Evaluating agents on setting up and executing tasks from research repositories. In EMNLP, 2024. URL https: www.llannthlloy.ooo org.y//2rg222000 emn1
p-ma n. 702]
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mádry. MLEE bench: Evaluating machine learning agents on machine learning engineering. ICLR, 2025. URL
https://cpenrevii net 6ss
Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hooi. MLR-Bench: Evaluating AI agents on open-ended machine learning research. 19955 2025a. URL http::s://axxiv..rg//bbbbbb.
9955
Ziru Chen, Shijie Chen, Yuting Ning. Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. SeienceAgemlBench: Toward rigorous assessment language agents for data-driven scientific discovery. ICLR, 2025b. URL Ut..nttpp::/eeeneevvee... wwwpeewwwwwwee.eee..rrr.re 2
Junyan Cheng, Peter Clark, and Kyle Richardson. Language modeling by language models. arXiv:2506.20249, 2025.
wttp:/rr.gaab//......22.. 20249
Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Fernando Campos, and Voorhees Overview of the TREC 2020 deep learning track. ArXiy, ab2/2100.000066 2021. URL
manticschon htt.://aa...arrooopppss...222212
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In EMNLP, 2019.
Adam Fourney, Gagan Bansal, Hussein Mozannar, Cheng Tan, Eduardo Salinas, Erkang Zhu, Friederike Niedtner, Grace Proebsting, Griffin Bassman, Jack Gerrits, Jacob Alber, Peter Chang, Ricky Loynd, Robert West, Victor Dibia, Ahmed Awadallah, Ece Kamar, Rafah Hosn, and Saleema Amershi. Magentic- Onge:ttic-One: generalist multi agett system for solving complex tasks. arXiv, abs/2411 1.411.00 1,04468, 2024. https:s::
abs /411 04468
12

---

Published as a conference paper at ICLR 2026
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac' h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. http:::::: http:///enndo.orrreeeeerrrrree//
Yuling Gu, Oyvind Tafjord, Bailey Kuehl, Dany Haddad, Jesse Dodge, and Hannaneh Hajishirzi. OLMES: A standard for language model evaluations. Findings of NAACL, 2025, URL
htttps:/aaaat org/ abs 2406. 08446
Etash Guha, Negin Raoff, Jean Mercat, Ryan Marten, Eric Frankel, Sedrick Keh, Sachin Grover, George Smyrnis, Trung Vu, Jon Saad Falcon, Caroline Choi, Kushal Arora, Mike Merrill, Yichuan Deng, Ashima Suvarna, Hritik Bansal, Marianna Nezhurina, Reinhard Heckel, Seewong Oh, Tat- sunori Hashimoto, Jenia Jitsev, Yejin Choi, Vaishaal Shankar, Alex Dimakis, Mahesh Sathiamoor- thy, and Ludwig Schmidt. Evalchemy: A post-trained model evaluation framework, November 2024. URL E9f0f61 09900111862224440011111111111111
com http:///ww.a.o.ooooooooo//ooo///
Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. Lighteval:
lightweight framework for LLM evaluation, 2023. URL httpp::///itthht/ htw.:/tuuu.oooomgigiigg..o....
face/1ighteval//e http:///eeeeeeee2//2ee22 11w.1122ff002232 6d36f 2at.aa/2226225228080066666666222
Yichen He, Guanhua Huang. Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, and Weinan E.
PaSa: An LLM agent for comprehensive academic paper search. ACL, 2025.
hww.aa/nhhoooy..or//222..... -ong..nl--11111..0 2ttpg2222...11111..775757771 572/|
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovee. MLAgentBench: Evaluating language agents machine learning experimentation. In ICML., 2024.
Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi, Bodhisattwa Prasad Majumder, Daniel S. Weld, and Peter Clark. CodeScientist: End-to-end semi-automated scientific discovery with code-based experimentation. In ACL Findings, 2025.
Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. Al agents that matter. 40775:240.001502, 2024. URL RtLLattps://azxiv.ooo wttp://rrrraabb//....oo 01502
Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Franck Stéphane Ndzomga, Kangheng Liu, and Arvind Narayanan. HAL: A holistic agent leaderboard for centralized and reproducible agent evaluation. https::////t / thub. com pr ncceoo- https///gg..aooaacccoocooooonaoa
harss 2025.
Patrick Tser Jern Kon, Jiachen Liu, Xinyi Zhu, Qiuyi Ding, Jingjia Peng, Jiarong Xing, Yibo Huang, Yiming Qiu, Jayanth Srinivasa, Myungjin Lee, Mosharaf Chowdhury, Matei Zaharia, and Ang Chen. EXP-Bench: Can AI conduct AI research experiments? wrXiv:2505.24783 2025. URL
htttps://axxis org/ /wtp/bbs////oooooo/oo////oea http//bb/25555...
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. ICML, 2023.
Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati. Andrew D. White, and Samuel G. Rodriques. LAB- Bench: Measuring capabilities of language models for biology research. arX.v.240..10362. 2024.
Thibault Le Sellier De Chezelles, Maxime Gasse, Alexandre Drouin, Massimo Caccia, Léo Boisvert, Megh Thakkar, Tom Marty, Rim Assouel, Sahar Omidi Shayegan, Lawrence Keunho Jang, Xing Han Lù, Ori Yoran, Dehan Kong, Frank F. Xu, Siva Reddy, Quentin Cappart, Graham Neubig, Ruslan Salakhutdinov, Nicolas Chapados, and Alexandre Lacoste. The BrowserGym ecosystem for web agent research. TMLR, 2025. Pid-5295 3
Zijun Liu, Kai Liu, Yiqi Zhu, Xuanyu Lei, Zonghan Yang. Zhenhe Zhang, Peng Li, and Yang Liu. AIGS: Generating science from powered automated falsification. ArXiv, 1.s.2411..199/11...1 2024. htt.hhttp://aae semanti ar http:/cc........ooo.oooppp//aa/a
13

---

Published as a conference paper at ICLR 2026
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. The Al scientist: Towards fully automated open-ended scientific discovery. ArXiv, abs/2408 06292, 2024.
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Sanchaita Hazra, Ashish Sabharwal, and Peter Clark. Data-driven discovery with large generative models. ICML, 2024.
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhi- jeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. DiscoveryBenchh: Towards data-driven discovery with large language models. ICLR, 2025. URL httpss:/cpeereeveew..eee
Microsoft. AutoGen agent igmmmmmmmmmmnnnninnnnnnn 2024 URL http:/:s//tttttt..////i/ioe/eeee /Att:eaaaaeeee/eeaaaaaaaaaaaeeee http://gggeeeetttteeeeeeeeeeceee Bttpg//ss////oooooor/r/////////r
Accessed: 2025-08-25.
Jordan Mitchener, Francisco Pineda, Yuxin Ye, Spyros Maniatis, Kenneth Holstein, Kam Dahlquist, James D. Braza, Andrew D. White, and Samuel G. Rodriques. BixBench: comprehensive benchmark for Ilm-based agents in computational biology. arXiv:2503.00090 2025. URL
/ttpos///axxv.oorr//////33. 00096
Benjamin Newman, Yoonjoo Lee, Aakanksha Naik, Pao Siangliulue, Raymond Fok, Juho Kim, Daniel S Weld, Joseph Chee Chang. and Kyle Lo. ArxivDIGESTables: Synthesizing scientific literature into tables using language models. In EMNLP, 2024. URL https::/acclatthon
gy. 0yg2 emn maannp-main 538/
Vishakh Padmakumar, Joseph Chee Chang, Kyle Lo, Doug Downey, and Aakanksha Naik. Setting the table with intent: Intent-aware schema generation and editing for literature review tables. arXiv:2507.1952) 2025.
Pritika Ramu, Aparna Garimella, and Sambaran Bandyopadhyay. Is this bad table? a closer look at the evaluation of table generation from text. In EMNLP, 2024. URL httpt:tttccsaaahhhhtt
gy. org. 2024
Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunis- mäki. 'smolagents': smol library to build great agentic systems. https::/gitta: com/ 2025.
Soggingface/smollaae
Kai Ruan, Xuan Wang, Jixiang Hong, Peng Wang, Yang Liu, and Hao Sun. LiveldeaBench: Evaluating LLMs' divergent thinking for scientific idea generation with minimal context. arXiv:2412.17596 2024. /tl.lttpss//axx..oorr/a/////////
SAgE Team, Princeton University. HAL: Holistic agent leaderboard, 2025. hRtps:/:://al Accessed: 2025-08-25.
www.ttteederbbaa... wpr.ioetoo..duu/eeeeeeee....eeee cs. .s.princeton.
Samuel Schmidgall, Yusheng Su, 7e Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent Laboratory: Using LLM agents as research assistants. In arXiv, volume 0bss22001.04277 2025.
ServiceNow. BrowserGym leaderboard, 2025. / Sarviiceeor/brooserrmyneeeeeeee
Accessed: 2025-08-25.
Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, and Monica S. Lam. Assisting in writing Wikipedia-like articles from scratch with large language models, 2024, URL
https://arxiv..rgge wttp:/iwwworr/ab//////o/// 14207
14207
Xiaofeng Shi, Yuduo Li, Qian Kou, Longbin Yu, Jinxin Xie, and Hua Zhou. SPAR: Scholar paper retrieval with llmebased agents for enhanced academic search. arXiv:2507.15245. 2025. URL
https: // aaxii. /ttaa//v.orr//b//55///0
15245
Chenglei Si, Diyi Yang. and Tatsunori Hashimoto. Can llms generate novel research ideas? large-scale human study with 100+ nlp researchers. arXiv preprint 2.X.v2240..0411 0490 2024.
14

---

Published as a conference paper at ICLR 2026
Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl, and Arvind Narayanan. CORE- Bench: Fostering the credibility of published research through a computatioonl reproducibiliy agent benchmark. TMLR, 2025- 2025-Jamary:1131, 2025. URL httpps:/ttmt...emme/ wttp://rr...pppppps//s/s/ss
025/01-2025pap2 pdf
Amanpreet Singh, Joseph Chee Chang, Chloe Anastasiades, Dany Haddad, Aakanksha Naik, Amber Tanaka, Angele Zamarron, Cecile Nguyen, Jena D. Hwang, Jason Dunkleberger, Matt Latzke, Smita R Rao, Jaron Lochner, Rob Evans, Rodney Kinney, Daniel S. Weld, Doug Downey, and Sergey Feldman. Ai2 Scholar QA: Organized literature synthesis with attribution. ArXiv, abs/2504. 10861, 2025. URL https://api: be..pss:/ap..ssmantt. semant icsho ar. wrg/CorpossDr 277786810
Michael D. Skarlinski, Sam Cox, Jon M. Laurent, James D. Braza, Michaela Hinks, Michael J. Hammerling, Manvitha Ponnapati, Samuel G. Rodriques, and Andrew D. White. Language agents achieve superhuman synthesis of scientific knowledge, arXiv:2409. 13740, 2024. URL https: org/ abs//bb//// 2409 13740 Introduces the LitQA2 benchmark for evaluating language models on scientific literature research tasks.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, et al. Paperbench: Evaluating ai's ability to replicate ai research. arXiv preprint arXiv:2504.01845 2025.
Jiabin Tang, Lianghao Xia, Zhonghang Li, and Chao Huang. Al Al-Researcher: Autonomous scientific innovation. 0rxiv:2505.187000 2025. URL
The Terminal-Bench Team. Terminal-benche A benchmark for ai agents in terminal environments, com laude- mastitute/termiee bench
Apr 2025a. URL
The ferminal-Bendh Team. Verminal-Bennhh leaderboard, 2025b. URL httph::/tttteeeeeee
leaderboard Accessed: 2025-08-25.
Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong. Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huerta, and Hao Peng. SciCode: A research coding benchmark curated by scientists. arXiv:2407. 222447.1166.. 2024. URL /tRLhht:s://arxttt
org/ abs/ /207 13168
UK AI Safety Institute and Arcadia Impact and Vector Institute. Inspect Evals: Community- contributed evaluations for inspect ai. htttp:: /ttp:////gthbb./// com//UKGoverrreeettBBB.
inspect_evals 2025. Accessed: 2025-08-24.
UK Al Security Institute. Inspect Al: Framework for Large Language Model Evaluations, May 2024. URL/httss:// Btt.a//.....cccooooooooooeenanaa
ai
Rosni Vasu, Chandrayee Basu, Bhavana Dalvi Mishra, Cristina Sarasua, Peter Clark, and Abraham Bernstein. HypER: Literature--rounddd hypothesis generation and distillation with provenance, 2025.
12937
Vector Institute. Vector evaluation leaderboard,
lea 2025. URL httthttss///hgigggggettt co/ spa Accessed: 2025-08-25.
Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-66) 2023.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li. Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. In ICML, 2024.
Xingyao Wang, Boxuan Li, Yufan Song, Frank F, Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma. Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In /CLR, 2025. URL https: http:://oppeeeeeeeeee net wtt.:oreee/frureeee// ?id=0dddeyyDooF
UR//tttp:://ppeeeeiiee
15

---

Published as a conference paper at ICLR 2026
Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, and Yulan He. Scireplicate-bence Bench- marking Ilms in agent-driven algorithmic reproduction from research papers. Proceedings of COLM, 2025.
Tianze Xu, Pengrui Lu, Lyumanshan Ye, Xiangkun Hu, and Pengfei Liu. ResearcherBench: Evaluat- ing deep Al research systems the frontiers of scientific inquiry. (XXi::507.166880 2025. URL atsgg//bb/25000
ht.tps:// http://axxvv/
6280
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Nicolaus Foerster, Jeff Clune, and David Ha. The Al Scientist-v2: Workshop-level automated scientific discovery via agentic tree search. ArXiv, abs/2504.08066 2025. URL http:: //ppi.appp.. semant
org/Cozpus :D:
Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, George Michalopoulos, Yue Zhang, et al. Lmr-bench Evaluating Ilm agent's 's ability on reproducing language modeling research. arXiv preprint arXiv:2506. 17335, 2025.
Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shhueli-Scheuer. Survey on evaluation of LLM- LL-baaed agents, arXiv:2503. 16416, 2025.
URL
abs /20/2/553.. 6416
Xuanle Zhao, Zilin Sang, Yuxuan Li, Qi Shi, Weilun Zhao, Shuo Wang, Duzhen Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. Autoreproduce: Automatic ai experiment reproduction with paper lineage. arXiv preprint arXiv:2505. 20662, 2025.
Kunlun Zhu, Jiaxun Zhang, Ziheng Qi, Nuoxing Shang. Zijia Liu, Peixuan Han, Yue Su, Haofei Yu, and Jiaxuan You. SafeScientist: Toward risk-aware scientific discoveries by LLM agents. arXiv:2505.23552 2025. URL httphhttpt////attte http://aawxivvoooraa//////// wtwp/gg/aab////aooar .2559
A PRINCIPLES FOR BENCHMARKING AGENTS
We propose the following principles for more rigorously benchmarking agents:
1. The task suite must represent the complexity of real-world usage. In order determine whether agents can serve as effective assistants for use case, it is necessary to test broad range of relevant tasks. Real-world product usage provides an informative basis for determining appropriate tasks. but unfortunately such data is typically guarded by product companies (who use to create private evaluations) and unavailable to academic benchmark creators. Moreover, in order to measure progress towards broadly capable agents, the task suite should require exercising a range of advanced, general skills such as reasoning, planning, tool use, search, coding, and data analysis.
2. A standard, realistic, and reproducible environment and tools must accompany the suite for controlled comparison of AI capabilities. The environment should be realistic measure agents' ability to act in the real world. At the same time, the environment and tools must be standard and reproducible to facilitate controlled comparison across different agents. Most existing benchmark suites lack standard tools, leading agent developers to use disparate environments and tools that obscure whether performance differences are due to superior AI capabilities or other enhancements. It particularly important that benchmark suites provide standard search tools with reproducible test-time access to the same document corpus, yet large-scale, scale, optimized search indexes are costly to create and public search tools are not reproducible; we are unaware of any such public, reproducible, large-scale search tools.
3. Reporting must account for confounding and tool usage. It's essential to account for cost, since even simplistic strategies, such as repeating a task many times and taking majority votes, can boost accuracy by burning cash Kaapor et al. (2024 2025). Controlling for tool usage is also essential to separate gains due to model or agent architecture advancements from benefits due to privileged access to specialized information sources.
eepabiial--secialllly computational cost
16

---

Published as a conference paper at ICLR 2026
4. Task interfaces must be standardized to facilitate integration of general agents. General agents that can perform many different tasks are likely to better meet diverse real-world needs. Unfortunately, most previous benchmark suites require general agent developers to adapt agents for individual tasks, introducing developer bias and hindering development. To support the development of general agents, task interfaces should provide "reasonable" accommodation for an intelligent agent that has not been developed specifically for the test tasks: complete task instructions, task-required tools, and submission affordances-all in standard format.
5. Comprehensive agent baselines with standard interfaces are needed to measure state- of-the-art. large integrated suite of agent baselines must be available identify which agents are truly state-of-the-art agents and to provide high-quality starting points for future development, yet is lacking from current agent suites resulting most evaluations comparing only to small number of other agents or ablations on the evaluator's ' own agent.
B EVALUATION TOOLKIT: OPENNESS AND TOOLING
Definitions for the Agent openness and Agent tooling classifications for baseline:
Agent openness describes the transparency and reproducibility of an agent's implementation:
Open-source, open-weight (v): Both agent code and ML model weights are publicly available, enabling full end-to-end reproducibility.
Open-source, closed-weight (~-): Agent code is available but relies on proprietary ML models, allowing partial reproducibility of the approach.
Closed source & API available (AA): Implementation details are proprietary, but the system is accessible via API, enabling result verification but not method reproduction.
Closed & UI only (x): Neither code nor programmatic API access is available.
Agent tooling describes the tool usage and execution environment of an agent during evaluation:
Standard (v): Uses only predefined tools from the evaluation environment (as defined in Inspect's state.tools).
Custom interface (~)): Uses custom tools for accessing equivalent underlying environment, which for AstaBench we define as task-relevant portions of the Asta Environment:
* Literature tasks: Information access limited to date-restricted usage of the Asta Scientific Corpus.
*
Code tasks: Code execution is limited to an IPython shell in a machine environment initialized with the standard Asta Environment sandbox Dockerfile (or equivalent).
Fully custom (x): Uses tools beyond constraints of Standard or Custom interface.
17

---

Published as a conference paper at ICLR 2026
Table 4: Overall results for agents that can solve all the tasks (additional results in Table Reported values are macro averages over benchmark statistics; confidence intervals are omitted. f denotes models not pinned date-stamped version. Bold denotes the agent is on Pareto-optimal frontier for that column pair.
Literature Understanding & Exccution
Code
T Agent
Model
Overall
Analysis
Discovery
Score
Cost
Score
Cost
Score
Cost
Score
Cost
Sonte
Cost
Assá vO
mixture
3.40
0.58
47.6
0.19
33.2
68.8
12.57
ReAct
opt-5
44.0
0,31
54,6
0.30
55.0
0.35
30.5
0.09
36.1
0.49
RoAct
39.4
0.16
46.8
0.35
49.3
0.19
33.7
0.04
28.0
0.07
Smol aganta
claude-sonntt-4
38.1
1.02
42.7
0.71
39.6
1.96
28.8
0.24
41.5
1.19
Coder
Smo lagenta
gpt-5
26.7
Coder
ReAct
spt mini
31.6
0.04
0.05
50.5
26.9
12.6
ReAct
claude-3-S-haikt
21.9
0.04
36.2
0.03
22.4
0.05
24.3
0.01
0.04
mallagant
Llama-4-scout
11.1
0.11
20.0
0.03
0.12
20.2
0.01
0.5
0.27
Coder
Table 5: Literature Understanding search benchmarks results (additional results in Table 12 denotes models not pinned a date-stamped version. Bold denotes the agent is on Pareto-optimal frontier for that column pair.
T
Agent
Model
Papezf Snpe:Fiininngenccl
Litg82 FullText Search
Score
Cost
Score
Cost
Asta Paper
gemini 2 flash,
39.7 3.1
0,063 1003-000 0.005
90.7
6.6
0.112:00007
Pinder
gpt-4o
Asta
37.6 1
3.1
0.033 0.005
90.7 1. 6.6
0,112 1. 0.007
ReAct
gpt-5
264
3.9
0.428 1: 0.048
82.7 8.6
0.389 + 0.055
19.3
0.518 0.067
57.3
0.127
Smo agerts
gpt-4.1
16.5 3.5
0.080 + 0.007
50.7 :2 11.4
0.095 1: 0.037
Coder
Smelagents
claude sonnett-
22.1 3.5
0.975 0.139
52.0 11.4
1.100
Coder
You.
7.2 2.0
36.0 10.9
Search APT
C SUPPORTING EXPERIMENTAL RESULTS
This section contains supplemental tables and figures for the narrative in Section 5 Table shows the overall results for those agents attempting all benchmarks, well as agents that can solve all the benchmarks in at least one category. We then show category-specific results, for Literature Understanding (Tables to 7, Code and Execution (Table 8). Data Analysis (Table 9), and End-to-
End Discovery (Table 10 For details about referenced agents and models, refer to Tables and respectively.
In the Tables, "O" denotes Openness, with values (Open-source, open-weight), ~ (Open-source, closed-weighte AA (Closed source & API available), and x (Closed & UI only). "T" denotes Tooling, with values / (Standard), ~ (Custom interface), and x (Fully custom). The openness values apply to the agent (including the model used). "4" denote 95% confidence intervals. Bold denotes the agent is on Pareto-optimal frontier for that column pair. Our results reveal several noteworthy insights.
18

---

Published as a conference paper at ICLR 2026
Table 6: Literature Understanding QA benchmarks results (additional results in Table 13) Agents without an API could not be evaluated on LitQA2-FT. denotes models not pinned a stae-sd version. Bold denotes the agent is on Pareto-optimal frontier for that column pair.
Ageni
Model
ScholargA-CS2
Lit
Score
Cost
Score
Cost
ReAct
gpt-5
79.8
3.5
0.373 1: 0.034
82.7 1. 8.6
0.276 1: 0.114
0.299
Putureflouae Crow
1pi-4.m 4411mmini, o3 33-mini,
81.1 1.7
0.107 = 0.004
72.0 10.2
0,065 + 0.003
gemini-2.5-rias
X
Futurellouse Falcon
gpt 4.1 mini.
71.6 1.3
0.403 = 0.051
74.7 9.9
0.220 0.011
gemini-2.5
03-mini
ReAct
0.275
80.0
+:
Smolageats Codes
gpt gemini-2.3 2.ssshshh,
68.4 4.4
0,154 0.014
73.3 10.1
0,101 + 0.026
Perplexity Sonar
67.3 1.2
0.416 0.019
73.3 10.1
0.219 0.016
Deep Research
sonar deep research
Smolageats
gpt-4.1
0.016
0.005
Yom..com Research
55.0 2.2
8.0 6.2
API
Asta Scholar QA (w/
claude sonnet 4
87.9 1.2
1.314
QA
gemini flash!
0.126
Asta Scholar QA Claude-sonas sonnet-4
86.2 1.4
0.393 0.030
Asta Scholar QA
spt-51
85,9 =
1.099 + 0.074
Elinit
85.5 1.6
SciSpace Leep
claude-sonnet-4
84.6 = 1.3
Revicw
STORM
9pt-3.
78.3 2.4
0.094 0.002
gpi 40
A OpenAl Deep
o3-/od-mini-
79.4 1.4
1.803 + 0.039
Research
deep-rasearch,
gemin1-2,5-pro
Openšei1x
11ama-3.1-
58.0
2.6
0.004 1: 0.000
openscholar-9h
Table 7: Literature Understanding
task benchmark results (additional results in Table 14p. denotes models not pinned to date-stamped version. Bold denotes the agent is on Pareto-optimal frontier for that column pair.
O
T
Agent
Model
Arxiv001GRSTTablee-CCle
Score
Cost
Asta v0
mixture
42.9
0.517 0.056
Asta Table Synthesis
gpt-5*
42.6 + 3.5
1.281
0.140
Asta Table Synthesis
gpt-5-minil
41.7
0.172 0.019
ReAct
o3
32.9 + 3.3
0.050 0.004
Smolagenta Coder
gpt-5
31.5 + 3.2
0.060 0.004
19

---

Published as a conference paper at ICLR 2026
Table 8: Code & Execution category results (additional results in Table 15
f denotes models
not pinned to a date-stamped version. Bold denotes the agent is on Pareto-optimal frontier for that column pair.
O T Agent
Model
SUPER-EXporH
CCRB-Bonch-Hard
DS-1000
Score
Cost
Score
Cost
Score
Cost
gpt
+ 12.9 0.589 0.140
45.9
16.3 0:43=000
78.0
2.7
0.021 0.0009
ReAct
gpt-S-mini
34,6 13.2
0.105 + 0.06
45,9
16.3
0.047 2 0.014
71.0 = 3.0
0.003 :: 0.0001
ReAct
Asta
mixture
19.4 10.4
0,332
0.057
48.6
16.3
0.226
0.093
74.8 = 2.8
0.011 0.0007
Smolagents
claude-
11.7 8.0 3,559
4: 1.766
32.4 1:
15.3
2.199
0.780
74.7 1:
2.8
0.114 0.0079
Coder
sonnet-4
Bmolagents
gpt-5
3.6 4.8
0.003
13.3
11.2
0.190
0.106
75.7 = 2.8
0.019 = 0. 0.0007
Smolagencs
claude-3-S-
16.8 +: 9.6 0 0.581
0,0000
0.332
0.210
9.9 : 2.0
0.024 + 0.0103
Coder
A Code
gpt-4.1
16.3 9.4 0.285 0.099
Code
gpt
13.5
0.072
Table 9: Data Analysis DiscoveryBench results (additional results in Table 16 t denotes models not pinned to a stae-et version. Bold denotes the agent is on Parcto-optimal frontier for that column pair.
O
T
Agent
Model
DiscoveryBench
Score
Cost
ReAct
o3 mixture
33.7 +
5.1
0.039 + 0.004
X Asta v0
33.2
5.1
0.246 0.071
Asta DataVoyager
o31, got
31.1
0.234 0.061
V ReAct
gpt-5 claude-sonnet-4
30.5
4.8
0.092 1 0.009
Smolagents Coder
28.8 1
4.8
0.237 1 0.019
Table 10: End-to-End Discovery category results (additional results in Table 17 t denotes models
not pinned to a date-stamped version. Bold denotes the agent is on Pareto-optimal frontier for that column pair.
O T Agent
Model
E2E-Bonch
E2E BeB-Bench-Hard
Score
Cost
Score
Cost
X Asta Panda
claude-
70.5 6.2
10.14
68.2 4.4
14.487 + 1.050
sonnet-4
X Asta v0
mixture
70.4 6.3
10.643
67.3 5.3
14.487 1: 1,500
claude-3-7-
65.3
2.760
64.5 + 5.5
3.549
CodeSciention
sonnet
Smolagents
opt-5
62.8 9.8
0.205 0.025
30.3 + 10.5
0,232 : 0.043
Coder
V ReAct
claude-
$2.5 +
0.749 0.072
38.9 + 6.9
0.836 - 0.057
sonnet-4
Smolagents
claude-
47.2 6.1
00.131
35.8 + 7.8
1,512 + 0.307
Coder
sonnet-4
Faker
gpt-4.1
39.2
0.026
25.4+ 4.5
0.029 0.001
ReAct
o3
34.9 10.1
0.065 0.010
21.0+ 7.6
0.075 + 0.019
ReAct
gpt-5
30.0 11.9
0.403+0.055
42.1 11.4
0.584 + 0.072
20

---

Published as a conference paper at ICLR 2026
Table 11: Overall results for agents that can solve all the tasks. Reported values are macro averages over benchmark statistics; cost confidence intervals are omitted for space. f denotes models not pinned a date-stamped version.
Agent
Model
Overall
Licerature Understanding
Code & Exceution
Data Analysis
End-so-End Discovery
Score
Cost
Score
Cost
Score
Cost
Score
Cost
Scure
Cosl
ReAct
clande-3-3-
21.9
1.6
0JM
36.2 =
2.3
0.00
22.4
3.0
0.05
24.3
4,7
0.01
4.6
2.2
0.04
claude-
23.2
sornet-1
RoAct
apt-t.2
31.6
23
0.20
46.4
23
0.34
32.4 +
5.1
0.09
30,5
5.1
0.02
13.1 +
5.0
0.14
RoAct
16.24
14
0.12
318
22
0.15
18.3 1
3.5
0.15
13.2 1
3.7
0.01
1.5
1.3
0.15
apt-a-mins
RGAct
apt-5
44.0
3.0
0.31
$4.6
22
0.30
53.0
7.0
0.35
30.5
0.09
36.1
8.3
0.49
ReAct
gemini-2.5- 3-
15.3
144
0.71
32.8
3.0
0.46
26.0
1
4.1
0.45
1.9 1
1.7
0.10
0.3 1:
1.1
1.83
11ama-(-
Scout
BeAct
39.4
26
0.16
46.8
23
0.35
49.3
+
6.3
0.19
33,7 +
5.1
0.04
25.0
6,3
0.07
Smolacints
claud:-3-3-
12.7
15
0.30
20.9
0.05
8.9
1:
5.3
4.5 1:
2.0
0.73
Smolacants
claude-
38.1
23 1.02 42.7
24
0.71 39.6
5.8
1.96
28.8
0.24
41.5 4:
4.9
1.19
Codor
sonnct-1
328
43.9
23 0.07
28.4
33.3
6.0
Smolagrats apt-to
13.6:1 1.6 0.36 22.7
2.1 0.08
3.1
17.8 1:
0.05
5.3
2.6
0.67
Codaz
Smolacaats apt-3-min1
23 0.06 38.5
27 0.02 28.3 4:
27.7
22.0 1:
6.1
0.08
Codar
Smolaqenta apt-5
37.5 +: 25 0.13 46.0 + 25 0.12 30.9 +: 4.2 0.10 26.7 +: 4.7 0.08
46.5
7.2
0.21
Codar
Snulagents
26,4
25
16.6
24.7
0.02
28.6
6,4
2.21
Smolaccats
11ama-4-
11.1 1.4 0.11 20.0 22 0.00
24 0.12 20.2
0.01
0.3
0.4
0.27
Goder
$3.0 3,40 62.2 20 0.58
68.8
D FULL EXPERIMENTAL RESULTS
Section 5 presented results for the best agents (i.e., agents running with the best underlying model), plus few additional important data points. Here show the full set of results for all configurations of agents that were tested (a superset of the results in Section 5). We also show plots of scores vs. costs, including the Pareto frontier (showing the best agent for given cost). In the Tables, "O" denotes Openness, with values / (Open-source, open- weinh,,ight), (Open-source, flosedwwiigt) and (Closed & UI only). "T"" denotes Tooling, with values (Standard), ~ (Custom interface), and x (Fully custom). "+" denote 95% confidence intervals.
Statistical Methodology All confidence intervals shown are 95% CIs computed as 11.6 x SE, where SE is the standard error. For individual benchmarks, standard emrors calculated from the variance across evaluation samples within each task. For category-leve aggregations, standard errors
are propagated analytically using weighted averaging: SE SEaategrr
Σ wF SE / W,, where the task weights (uniform at 1.0 except for the two LitQA tasks which each have weight 0.5). This propagation assumes independence between tasks, which could slightly underestimate uncertainty.
21

---

Published as a conference paper at ICLR 2026
Benerfindinggench
Efficiency Fronlier
Asta Paper finder (gemini-2-flash, gpt- 4o)
0.40
Smolagents (gpt-S-minii
Smolagents Illama---scout
0.35
Asta (mixture)
0.30
React (cleude-3-S-haikk))
React (claude-sonnet-4)
Score
0.25
React (gpt-4.1) 1]
React (gpt-to)
React (gpt-S)
React (spt-S-mini)
React (gemini-22 5-flash)
React (llama-4-scout)
React (o3)
Smolagents (claude- 3-5-hel--
Smolagents fcloude-sonnet-()
Smolagents (gpt-4.1)
Smolagents (gpt-4a)
10
10-4
10a
Smolagents (gpt-5)
Smolagents Igemini-2. 5-ftssh)
LitQa2-Fulleet-Search
cou.com Search API (no cost)
1.0
0.8
0.4
0.2
0.0
10-3
10-2
10-1
Cost (USD)
Figure 3: Score vs. cost analysis for Literature Understanding search benchmarks (Table 12 Points indicate means; error bars denote 95% confidence intervals. Points the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-offs for each eval (Paperii ndi nggtench I. AittA22Ful Text Search). Note: the x-axis (cost) uses log scale.
22

---

Published as a conference paper at ICLR 2026
ScholarQA-CS2
Effidency Frantier
QA Tables) (gemini flash!
Asta Scholar QA (No Tables) (spt-4o-miri)
0.8
Asta Scholar QA (o.)
OpenScilM
-1-spanschola-8))
React (gpt-5)
0.6
React (gpt
Poolagents (gp8-4.3)
Score
Smo lagents (gpt-5-mirs)
Smo lsgeltenss illmm 4-scout)
0.4
QA (claude--onnee-4))
Asta Scholar QA (No tables) Icla (cluuse-somett)
Scholar (No (mixture)
0.2
futureficuse CROW (gpt-4,1-minn 03-mini, 3ommi-2.5-ffashl
FutureHouse FALCON (gpt 4pp 4.1-mini, gemini flssh 43-min))
10-3
10
10
100
1ommr-2.5-ppm)
Deep Research (o3 (o3-deep seeepreseearch-22 2025 06-26,
LitQA2-FullText
Decp Research desp research!
2.5-flash,
React (caclddcddo-33S-haale))
React (cluude-sonee--)))
React(gpt-4.3)
React (gpt-4a)
React -igemmin-2.--flacae
React React(lama-<< s-soutt
React(o3)
(get-3. S-turbo, get-4al
Score
Smo lagents Ielaude 3-shllluul scnnete-
0.4
lagents (gp5-40)
Smo lagents (gpt-5)
Igemini 2.5 flashi
cost)
0.2
Scippcc tno cost)
You com Research API (no cost)
0.0
Cost (USD)
Figure 4: Score vs. cost analysis for Literature Understanding QA benchmarks (Table 13 Points
indicate means; error bars denote 95% confidence intervals. Points the Pareto frontier connected with dotted lines, representing optimal quality-cost trade-off's for each eval (ScholarOA- cs2, LitgA2-PullTexx Note: the x-axis (cost) log scale. denotes models pinned a date-stamped version.
23

---

Published as a conference paper at ICLR 2026
Table 12: Literature Understanding search benchmarks results. f denotes models not pinned to date-st version.
Agent
Model
Papez Findingeeenh
SiLQAZ PullText Search
Score
Cost
Score
Cost
ReAct
claude
haiku
10.7
2.6
0.061 + 0.005
60.0 11.2
0.069 0.007
React
=laude-soanet-4
20.3 = 3.2
0.541 + 0.025
46.7 11.4
0.606 0.031
ReAct
gpt-4.1
16.5 1. 3.2
0.867 1 0.183
65.3 1 10.8
0.819 1.19:100 0.258
ReAnt
gpt-44
12.9 = 3.
0.267 +: 0.032
66.7 10.7
0.128 1: 0.081
spt-5-mini
22.0 3.7
0.060 0.009
56.0 11.3
0.118
ReAct
get-5
26.4 3.9
0.428 1: 0.008
82.7 8.6
0.389 1: 0.055
ReAct
gemini-2.5-flash
65 2.3
0.214
$7.3
0.650 0,400
ReAct
1lama 4 scout
5.4 2.2
2.816 0.319
37.3 11.0
4.326
ReAct
19.3 3.7
0.518 + 0.067
57.3 11.3
0.790 0.127
Smolagents
claude-3-S-heiku
4.6 2.1
0.070 1 0.011
2.7.1 3.7
0.096 0.060
Code:
Smolagenis
22.1
0.975 0.139
52.0 11.4
0.009
Code
9molagents
9pt 4.1
16.5 3.5
0
+
0.095
Codes
Smolagents
gpt-40
12.5 d: 3.5
0.088 1: 0.010
20.0 9.1
0.105 0.023
Gode
3molagenis
9pt-5-mini
17.2 3.2
0.034 0.012
48.0 11.4
0.036:000000
3molagents
gpt-5
20.0 3.9
0.121 0.012
54.7 = 11.3
0.152
Code:
Smolagents
gemini-2.5-Elamt
14.7 3.4
0.044 + 0.006
36.0 10.9
0,089 0.10
Coder
Smolagents
Lama-4-acout
7.0 2.9
0.003:0.000
6.7 5.7
0.00
Codez
0.063
+
Asta Papes
gemini-2 2-Eashh gpt-4o
39.7 = 3.1
0,063 0.005
90.7 + 6.6
0.112
Finder
A
You.com
7.2 2.0
36.0
Search A21
AriiDDGGESTables-Clenn
Efficiency Frentier
Smolagerte (gpt:5-mini)
Roart (gat-S-mini)
0.4
Synthesis (gpt-S-minit)
Asta vo (mixture)
Asta Ta Synthesis [claude-3 halku)
0.3
Asta Table Synthesis (claude-sonnet-4)
Table Synthesis (gamini flash)
Score
Asta Table Synthesis (gamini-2.5-ppm
Synthesis (gpt-4.1)
0.2
Asta Ta Synthesis (apt51)
Synthesis
scoutt)
Asta Table Synthesis (63)
React (claude:3-S-hakk)
React (claudessonnet-4)
0.1
Reart (gpt-4.1)
(gat-4n)
React (gpt-5)
(gemini-2.5-ffachh
0.0
10-1
10-2
10a
React Illama-4-scoutt
React (o3)
Cost (USD)
Smolagents (claude- h-S-haluu))
Smolagents (claude sonnet
Smolagents (gpt-4.1)
Smolagents (gpt-4o)
Smolagents (gpt-5)
Smolagents (gemini- 2.5-fiahh)))
Smolagents (llama-4-scout)
Figure 5: Score vs. cost analysis for the Literature Understanding benchmark (Table Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-cost trade-off' for each eval. Note: the x-axis (cost) uses log scale. i denotes models not pinned date-stamped version.
24

---

Published as a conference paper at ICLR 2026
Table 13: Literature Understanding QA benchmarks results. Agents without an API could not be evaluated on LitQA2-FT. Models parentheses indicate self-reported models. denotes models not pinned to a date-stamped version.
Agent
Model
SchelazCA
C52
LiLQA2 FullText
Score
Cost
Score
Cost
ReAct
clavde-3-S-hoikt
66.3 1.
2.8
0.019
0.001
32.0 1: 10.6
0,022 1. 0.004
~
ReAct
claude-sonaet-
78.3
2.2
0.390
0.019
68.0 10.6
0.238
0.026
RoAst
gpt-4.1
70.1
3.2
0.733 0.243
77.3 1 9.5
0.222
0.097
ReAct
gpt-44
51.3 =
3.4
0.101
0.012
22.7 = 9.5
0.046 0.015
gpt 5-mini
26.7 7.4
0.022 0.004
74.7 9.9
0.075 0.029
ReAst
gpt-5
79.8 = 3.5
0.373 = 0.034
82.7 = 8.6
0.276 1: 0.114
~
gemini-2.S-fleah
V
ReAct
1ama-4-5cout
24.8 5.1
0.588 0.144
41.3 11.2
0,170 0.104
~
ReAzt
66.4 3.0
0.275 0.039
80.0 = 9.1
0.347 0.083
Smol $molagenta Coder
clatde-3-SShaiike
49.9 4.4
0.042 0.004
25.3 1: 9.9
0.056 0.010
Emolagenta Coder
clande-aonnet-4
7244 2.1
0.794 0.052
50.7 11.4
0.627 1 0.066
Smolagents
gpt
73.7= 2.1
65.3
0.005
Emolagents Coder
gpt-4c
46.3 4.0
0.078 = 0.008
14.7 8.1
0.050 1: 0.010
~
Smolagenta
gpt-S-miri
50.7
Smo agenta Coder
gpt-5
684 4.4
0.154
0.014
73.3 10.1
0, + 0.026
Smolagents Coder
gemini-2.5-flash
63.7 4.6
0.080 0,044
41.3 11.2
0.034 1 0.006
agonat Coder
1lam-4-scoout
39.6 4.8
0.008
42.7 - 11.3
0.013 0.002
Asta v0
mixture
87.7 1.4
1.529 0.291
70.7 = 10.4
0,306 0.093
~ Asta Scholar 0A Iw/ Tables)
88.7 =
Asta Scholar QA tw/ Tables)
claude-sonaet-4
87.9 1.2
1,314 0.281
Asta Scholar la 0A
clande-somnet-4
86.2 = 1.4
0.393 = 0.030
Asta
genini flash"
87.7
0.126
~ Asta Scholar QA
gst mini gpt 5*
78.5
0.012
Asta Scholar
83,9
X X alicit
85.5 = 1.6
A Perplexity Sonaz Deep
gemini 2.5 flash, sonar-deep- research
67.3 1.2
0.416 0.019
73.3 10.1
0.219
Research
A You.com Research
58.0 2.2
8.0 6.2
x
X SciSpace Deep Review OpenScilM
claude sonnet 4
84.6
1.3
11ama-3.1- operscholar-8b
580
A OpenAI Deep Research
o3-/o1 /oo-mini--
79.4 1.4
1,803 0.039
deep-research,
gemin1-2.5-pro
A FutureHouse
gpt-4.1-mini, o3 mini,
81.1
0.107 0.004
72.0 10.2
0.065 0.0003
gemini-2.5-flest
A
X FutureHouse Falcon
gpt-4,1-mint
77.6 1
1.3
0.403 0.051
74.7 1
9.9
0.220 1 0.011
gemint-2.5-tlashh
63-mini
X STORM
gpt-3.5-turbo,
78.3
2.4
0.094
0.002
gpt
25

---

Published as a conference paper at ICLR 2026
Table 14: Literature Understanding wrw. task benchmark results.
T
Agent
Model
Arx.ivDiGRSSTabees-Clleee
Score
Cost
ReAct
1laude-3-5-has
21.7 +
2.6
0.013 + 0.001
ReAct
claude-sonnet-4
25.5 + 3.1
0.069 + 0.005
ReAct
gpt-4.1
27.5
3.2
0.038 0.004
ReAct
gpt-4o
16.3
2.4
0.005
ReAct
gpt-5-mini
32.1 3.3
0.013
ReAct
gpt-5
29.4 3.7
0.064
ReAct
Demini-2.S-flash
25.2 3.1
0.022
ReAct
1lama-4-scout
9.5 2.3
0.100
ReAct
03
32,9 3.3
0.004
Smolagents Coder
Llade-3-5-haakk
15.0 2.8
0.017 0.000
Smolagents Coder
claude-sonnet-4
24.8 2.9
0.204 1.0018
Smolagents Coder
gpt-4.1
27.2 3.1
0.044
Smolagents Coder
gpt-40
14.6 2.5
0.051 0.007
Smolagents Coder
9pt-5-mini
30.0 3.2
0.009
Smolagents Coder
gpt-5
31.5 3.2
0.000
Smolagents Coder
gemini-2.55 5-
25.2 2.8
0.021 +00002
Smolagents Coder
11ama-4 scout
8.7 2.2
0.099
Asta v0
mixture
42,9 3.7
0
Asta Table Synthesis
gpt-4.1
38.8 3.5
0.347 0.033
Asta Table Synthesis
claude-3-S-haiku
31.1 3.6
0.165 0.018
Asta Table Synthesis
claude-sonnet-4
37.2 3.3
0.076
Asta Table Synthesis
gemini-2.5-flas
34,4 3.3
0.133
Asta Table Syrthesis
o3 Table Synthesis gemini-2.5-pro
41.6 3.5
0.517 0.056
35.4 3.5
0.155
Asta Table Synthesis
11ama-4-scout
26.4
3.3
0.025 0025+0.000 0.003
Asta Table Syrthesis
gpt-5t
42.6
3.5
1.281 0.144
Asta Table Synthesis
gpt-5-mini'
41.7 3.7
0.012
26

---

Published as a conference paper at ICLR 2026
Table 15: Code & Execution category results.
T Agent
Model
SUPER -xpexrt
CORE Bench Eard
DS-1000
Score
Cast
Score
Cost
Score
Cost
ReAct
claude-3-5-
13.1 8.3 0.077
0.017
0,0000
0.077 0,021
54.1 ::
3.3
0.006 1: 0,0002
haiku
ReAct
claude-
22.6 11.1
0.448 0.087
40.5 =
0.499 0.08
75.6 2.8
0.044 0.0020
sonnet-4
ReAct
qpc-4.1
11.2 7.5
0.156
189
0.005
67.0
ReAct
4o
5.9 6.7
0.319
5.4 7.4
0.124
43.7 + 3.2
0.010 = 0.0006
ReAct
spt-S-mini
34.6 13,2
0.105
0.046
45.9
16.3
0.077
0.014
71.0:
3.0
0.003 0.0011
ReAct
gpt-5
41.1 12.9 0,589
0.140
45.9
16.3
0.43
0,139
78.0
2.7
0,021 0.0099
ReAct
gemini-2.5-
20.0
10.7
0.875 - 0.295
27
5.3
0.470
55.4
3.2
0.019 1 0.0032
flash
V ReAct
a1mm-44-
4.7. 5.2. 0.175 0.066
0.0000
0.027 0.0018
9.7
1.9
0.110 1 0.0077
scout
ReAct
e3
16.3 9.6 0.369 0.097
56.8 = 16.2
0.196 0.196= 0,076
74.9
2.8
0,010 = 0.0077
~ Smolagents
claude-3-5-
16.8 9.6 0812
0.581
0.0000
0.332
0.210
9.9
2.0
0.024
0.0103
Coder
hoiku
8mo Iggnts
claude-
11.7 8.0 3.559
1.766
324 15.3
2.199
0.780
74.7 +
2.8
0.114 + 0.0079
Coder
sonnet
8mo I agents
gpc-4.1
700 6.9 0.149 0. 166
21.6 13.4 000031 48.0 3.3
0.073 + 0.0230
Codez
Smolagents
gpt-4o
399 4.9 1.351 = 0.715
5.4 = 7.4 0.19 16.8 2.4
0.137 + 0.0422
Coder
agents
9pc-5-m 55min
14.2 8.9 0,240 0.207
5.4 7.4 0.014
65.2 3.1
0.016 0.0066
Codez
Emolagents
9p
0.079
0.110
+ 2.8
0.0007
Coder
Smolagents
gemini 2.5 flash
7.5 6.0 0.796 0.945
13.5
11.2
00.700
28.9
3.0
0.044 0.0127
Coder
~ Smol agents
11amo-4-
8.1.1: 7.0
0.323 0.377
0.0000
0,046 0.034
2.7 1
1.1 0.004 1: 0.0020
Asta vD
mixture
19.4 10.4 0.332 0.057
48.6 16.3 0.093
74.8
+
2.8
0.011 0.000
~ ~ Asta Code
qpc 4.1
16.3 9.4 0.285 0.059
Asta Code
go=-4o
3.6 6.4
0.464 0.113
~ ~ Asta Code
gpt-5
13.5
9.4
0.372 0.072
Asta Code
gpt-5-min'
12.8 9.1 0.067 : 0014
27

---

Published as a conference paper at ICLR 2026
Exppett
Efficiency
Asta Code (gpt-5-mini)
React (gpt-5)
React (gpt-S-mini)
0.4
Smolagents (gpt-5-mll)
Score
0.3
Asta Code (gpt-4.1)
Asta Code. (gpt-4o)
Asta Code (gpt-S)
Asta vθ (mixture)
(claude-3-5-
0.1
React (claude-sonnet-4)
React (gpt-4.1)
0.0
React (gpt-4o)
10
10
10a
(gemmin--2.5-fass)
React (llama-t-scout)
CORE-Bench-Hard
Smolagents (claude-3-5-haiku)
Smolagents (claude-sonnet-)
Smolagents (gpt-4.1)
0.7
Smolagents (gpt-4o)
Smolagents (gpt-5)
Smalagents (gemini-2.5-ftach)
Smolagents (llama--sccutt
Score
0.3
0.1
0.0
10-2
10-1
10a
DS-1000
Score
10-3
10-2
10-1
Cost (USD)
Figure 6: Score vs. cost analysis for Code & Execution benchmarks (Table 15p Points indicate means; error bars denote 95% confidence intervals. Points the Pareto frontier connected with dotted lines, representing optimal quality-cost trade-offs for each eval (coRe-Bench-Hard) SUFER-Expert, DS-1000). Note: the x-axis (cost) uses log scale. f denotes models not pinned to a date-stamped version.
28

---

Published as a conference paper at ICLR 2026
Table 16: Data Analysis DescovezyBecch results.
Agent
Model
DiscoveryBench
Score
Cost
ReAct
Claude-3-5-haije
24.3 +
4.7
0.012 0.001
ReAct
claude-sonnet-4
23.2 4.1
0.132 0.009
ReAct
spt-4.1
30.5
0
ReAct
ept-4o
13.2
0.040 0.010
ReAct
gpt-5-mini
26.9 4.8
ReAct
gpt-5
30.5 4.8
ReAct
gemini-2.5-$laat
1,9 1.7
0.101 = 0.007
ReAct
11ama-4-scout
5.9
0.192 0.021
ReAct
o3
33.7 5.1
0.039 = 0.004
Smolagentss
Coder claude-3-S-haike
16.5 4.1
0.024 0.007
Smolagents Coder claudd-ssonett-
28.8 4.8
0.237 0.019
Smolagents Coder
gpt-4.1
28.4 4.9
0.045 + 0.018
Smolagents Coder
gpt-4c
17.8
0.054 1 0.004
Smolagents Coder
gpt-5-mini
27.7 4.9
0.071 1 0.041
Smolagents Coder
gpt-5
26.7 4.7
0.077 1 0.006
Smolagents Coder
gemini-2.5-flas
24.7 4.7
0.017
Smolagents Coder 11ama-4-scout
20.2 4.5
0
Asta v0
mixture
33.2 5.1
0.246
Asta DataVoyagez
gpt-4. 1pt=4..'.gpt-44" gpt-4o
29.9 5.0
0.147 = 0.020 0.523 + 0.050
Asta DataVoyager
Blaude-sonnes
25.7 + 4.6
gpt-4o!
Asta DataVoyager
o3'. gpt
31.1 5.0
0.234 + 0.061
Asta DataVoyager
pog=s":cffortmminina
27.0 4.7
0.215 = 0.029
spt-4o gpt-5', 4pt-4o
Asta DataVoyager
29.6 4.9
0.057
DiscoveryBench
0.40
Efficiency
Smolagents (lama-4-scout)
0.35
React (gpt-S-mini)
React (gpt-4.1)
React (a3)
0.30
Voyaerr (claude-sormet-4, gpt-4o]]
Asta DataVeyager (gpt-4:11, gpt-4or)
DataVeyager (gpt-5t, gpt-4o))
Score
Asts DataVeyager (gpt-5tteffortmminn gpt-4at)
0.20
Acta DataVeyager (o3t, opt-4ott
(mixture)
0.15
React (cleudde--S-haileu
React (cludde-connet44)
(gpt-4o)
0.10
React (gpt-5)
React (gemini-2.5-fushh
0.05
React
Smolagents (clbude-3--hhileu)
Smolagents (claude-sonnet-4)
Smolagents (gpt-4.1)
10-3
10-2
10-1
Smolegents (gpt-4o)
Cost (USD)
Smolagents
Smolagents (gpt-S-mini)
Smolagents (gemini-2.5-tie S-flash)
Figure 7: Score vs. cost analysis for Data Analysis sub- bobb-bencmmrkks.. Points indicate means; error bars denote 95% confidence intervals. Points on the Pareto frontier are denoted with red triangle markers, representing optimal quality-cost trade-offs for each eval (D scoveryBench). denotes models not pinned stae-st version.
29

---

Published as a conference paper at ICLR 2026
Table 17: End-to-End Discovery category results.
Agent
Model
32E-Benchh
E2E Bench-Bard
Score
Cost
Score
Cost
V ReAct
claude-3-5-
4.5 2.8
0.012
4.8 + 3,4
0.018
haiku
v ReAct
claude-
52.5 = 6.8
0.749 0.072
38.9 + 6.9
0.836
sonnet-4
ReAct
gpt-4.1
19.3 7.3
+ +
14.8 + 6.8
0.139 + 0.034
ReAct
gpt-44
1.6 1.7
0.157
1.4 1.9
0.135 0.028
ReAct
gpt-5-mini
9.5 7.6
0.006
15.7
8.3
0.040 0.008
ReAct
gpt-5
30.0 = 11.9
0,403
42.1 + 11.4
0.584 0.072
V ReAct
gemini-2.5
0.0000
2.401 + 1.149
1.1 +
2.1
1.263 0.672
flash
v ReAct
1lama-4-
1,9 2.1
0.135
0.9 1.1
0.813 -.144
scout
V
ReAct
o3
34.9 1 10.1
0.065 0.010
21.0 + 7.6
0.075 1 0.019
Smolagents
claude-3-5-
5.3 3.1
0.946
3.7 +
2.4
0.505
0.538
Coder
haiku
Smolagents
claude-
47.2 6.1
0.873 0.110
35.8
7.8
1
Coder
sonnet-4
Smolagents
gpt-4.1
36.6 9.3
0.178 - 0.146
30.0 + 7.7
1.955 + 1.773
Coder
Smolagents
gpt-4o
5.4 +
3.9
0.473 0.347
5.1 +
3.3
0.866 + 0.757
Coder
Smolagents
gpt-5-mini
22.3 + 9.6
0.076 0.114
21.6 1 7.5
0.076.1..10
Coder
Smolagents
gpt-5
62.8 + 9.8
0.205 0.025
30.3 1 10.5
0.232 T 0.043
Coder
Smolagents
gemini-2.5-
34.0 = 10.2
1.877 = 0.830
23.2 + 7.8
2.541 + 1.203
Coder
flash
Smolagents
1lama-4-
0.2 + 0.3
0.283 + 0.152
0.7 + 0.7
0.251 + 0.181
Coder
scout
X Asta v0
mixture
70.4
6.3
10.643
0.717
67.3 + 5.3
14.487
1.050
V Faker
gpt-4.1
39.2 + 6.9
0.026 + 0.001
25.4 + 4.5
0.029
0.001
X Asta Panda
gpt-4.1
36.6 + 7.7
7.610 + 1.650
39.3 + 7.0
9.319 + 1.243
Asta Panda
claude-
70.5 - 6.2
10.643 + 0.717
68.2 + 4.4
14.487
1.050
sonnet-4
X Asta CodeScientist
claude-3-7-
65.3
7.1
0.510
64.5 1 5.5
1.6922
sonnet
30

---

Published as a conference paper at ICLR 2026
E2E-Bence
0.8
Efficiency Frontier
slaute-sornet-4)
0.7
Asta (mixture)
Asto, CodeScientist (cloude 3-7-somect
(got-4.11)
0.6
React (gpt-S)
Smolagents (gpt:5)
0.5
Score
Asta Panda (gpt-4.11)
React (claude 3-----35 halku))
React (claude (claude-sonnet-ee son-soo-ntt- 4]
0.3
React (gpt-4o)
(apt-5mmnn))
0.2
React (gemini flash)
(llama-44 4-scout)
0.1
React (o3)
Smolagents (claude 3-5-haiku)
0.0
emalagents sones--)
10
10-1
10
101
Smolagents (gpt-4.1]
Smolagents (gpt.
Har-Bench-Hard
Smolagents (gat-S-mini) 5-min))
Smolagents (gemini-22 )
0.7
Smolagents (llama-4-scout)
0.6
0.5
Score
0.4
0.3
0.1
0.0
10 2
10
10"
10a
Cost (USD)
Figure 8: Score vs. analysis for End-to-End Discovery benchmarks (Table 17
Points indicate
means; error bars denote 95% confidence intervals. Points on the Pareto frontier are connected with dotted lines, representing optimal quality-coss trade-offs for each eval Be2e-Be E2E-Bench- Hard). Note: the x-axis (cost) uses a log scale. i denotes models not pinned to date-stamped version.
31

---

Published as a conference paper at ICLR 2026
E EVALUATIONS
E. SHORT DESCRIPTIONS
PapezFindingBench
PaperFindiaggence tests an agent's ability to handle challenging scien- tific search queries. Given a textual query string, the task is to return a ranked list of papers that satisfy the query. This new benchmark is subset of our own internal evaluation for our literature-searhh agent (Asta Paper Finder). Unlike existing paper- finding benchmarks, which are restricted to semantic search queries, our dataset includes metadata and navigational queries along with a diverse mix of semantic queries. The queries are sourced from PaperFinde and OpenSeiL.M user logs and the LitSearch (Ajith et al. (024) and PaSa (He et al. 2025) datasets. Evaluating retrieval tasks is challenging, and our chosen evaluation metries along with other benchmark details are discussed in appendix [E.2] Briefly, navigational and metadata queries are evaluated in terms of FI over the result set, and semantic queries use the harmonic mean of estimated recall and nDCG. The final evaluation metric is an average of per-query scores.
These benchmarks measure an agent's abil- ity to answer questions and retrieve papers within the biomedical domain. They are based on the LitQ22 dataset (Skarlinski et al., 2024), which contains 199 multiple-choiee questions, each as- sociated with a target paper whose full-text can potentially answer the question. To enable fair comparison for agents using our standard retrieval tools, we filter the original dataset a subset of 85 questions where the associated relevant paper is available in our Asta Scientifis iffc Corpus snippet search index within the specified cutoff date (see Table 2) Following (Skallinkki et al. (2024), LitQA2-Ful Fextt evaluates in terms of accuracy, the fraction of questions with a correct answer. LitQA2-FullTes - -earch isolates the retrieval task aimed at finding K papers such that one of them is the target paper for the question, and evaluates recall @ 30 (as used in (Skarlinski et al. (2024)) To avoid double-counting this benchmark when computing aggregate macro-averaged Liter- ature Understanding scores (compared to other benchmarks in that category), we weight each of these two evals by 0.5 in the macro-average. For additional details and comparisons, see appendix (G.2
ScholargA-CS2 The ScholarQA-Csa benchmark tests agent's ability answer long-form scientific questions. Given complex scientific question like "How is diversity typically evaluated recommendation systems?" the task is to identify relevant prior work and compose long-form answer report that appropriately cites sources. ScholargA- CS2 is a new benchmark that builds upon the recent ScholarQA-CS (Asai et al., (202)) by incorporating real scientific queries and introducing four facets for coverage and precision evaluation of both answers and their attributions, using LLM- as-judge. The average these four facet scores is the final evaluation metric. For more detail, see appendix
benchmark tests an agent's 's ability to create literature review table whose rows are publications and whose columns consist of aspects used to compare and contrast a set of papers. Given a set of related papers and a caption describing the table's intent (e.g.. "Overview of LLM pretraining benchmarks"), the task to automatically output a complete literature review table. We release a new benchmark that builds on ArxivDIGESTable es, the first high-quality dataset for literature review table generation created by Newman et al. (2024) by extracting review tables from ArXiv papers. Our evaluation includes two key improvements: (i) we curate a small clean subset of instances from the original test set, and (ii) we introduce an end-to-end evaluation methodology for the task. Tables are scored by prompting an L.LM "unroll" them into statements. The evaluation metric is the proportion of ground truth statements from the reference table that are entailed (according to an LLM judge) by the unrolled generated table. For more detail, see appendix E.4
ArxiivDIGESTalee--Clee
The
SUPER-Expert The SUPER-Expert benehmark (Bogin al. 2024 (Setting UP and Executing tasks from Research repositories) tests the ability of code agents to set up and execute Python machine learning experiments reported in MI and NLP papers. It targets the common yet often non-trivial and time-consuming task of setting up and running code from sparsely documented repositories
ihttos::/papprrindder www.://arr.rll.aaiaaaaaaa.a /chat
hhtt:s::/ppenscimm.allee
32

---

Published as a conference paper at ICLR 2026
accompanying published papers. Given natural language instruction along with a GitHub repository pointer (e.g., asking to train a model following a paper's 's code at a given URL), the task is to clone the repository, install any needed dependencies, configure, run the requested training/ovaluation, and report the outcome (e.g., model accuracy). In contrast to other repository-centent tasks, the particular focus here is on low-resource research repositories GitHub- like those researchers often encounter when validating and expanding upon prior published work. For more detail, see appendix E.5
code execution
-oRe-Bench-Hard The BeRR--BBcch-ardd benchmark (Siegel et al. 2025) tests an agent's ability to reproduce experiments and analyses from papers. The input is "capsule" from CodeO- cean.com containing code and data released alongside a published paper, as well as a set of instructions indicating specific analyses to perform with the capsule (full example in appendix [...I]. The task is to perform these analyses and write answers in a report. jaon file. The capsules in CORE-Bench- Hard chosen to be highly reproducible and span a variety of domains, including computer science, social science, and medicine, and use Python and R programming languages. For more detail, see appendix E.6
DS-1000 The DS- 000 benchmark (Lai et al. 2023) tests the ability of code models on routine data science tasks encountered in everyday research. The input is a coding question and an incomplete code snippet that the agent must fill in to answer the question (see example in appendix H.S.I). The output code snippet is graded by rumning it against (problem-specific) test case. This benchmark contains 1000 problems involving 7 Python libraries that were originally collected from StackOverflow and perturbed to avoid training leakage. We use the task implementation provided in Inspect evals (UK) Al Safety Institute and Arcadia Impact and Vector Institute| (2025) and report the accuracy of the proposed code passing the target test cases. For more detail, see appendix E.7
The Di DiscoveryBerch (Majunder al. 2025) benchmark aims to test whether the agent can automatically find and verify hypotheses from given dataset(s), performing data-driven analysis. The input to the task is a discovery goal and a collection of datasets and their respective metadata, and the output is hypothesis addressing the goal with the highest specificity for the context, variables, and relationship supported by the dataset(s). Optionally, a workflow for deriving hypothesis can be output to augment information already present in the hypothesis. This is the first comprehensive benchmark to test agents' or language models' ability to perform data analysis- including data preparation, basic statistical analysis, complex data transformation, and modeling on datasets from diverse domains, such as sociology and engineering. We collect task datasets from open public repositories made available by already published works from the 6 domains. The discovery goals are extracted from the associated papers to the datasets, or human-annotated, where each gold output (i.e.. the hypothesis) rigorously verified by data analysis experts. The performance on the benchmark is measured as the alignment of the predicted and gold hypotheses. The final metric, Hypothesis Matching Score, a product of three L----as-uudge scores that measure the alignment of the predicted and the gold hypotheses in the dimensions of their context, associated variables, and the relationship among them. For more detail, see appendix E.8
DiscoveryBench
E2E-Bench The -eB-Bench task aims to test whether agents can perform the full research pipeline of ideation, planning, (software) experiment design, implementation, execution, analysis, and produc- ing a final report, i.e., a complete research cycle. The input to the task is a research question in the domain of AI/NLP and a detailed description of the steps to investigate it, and the output is technical report, a trace of the agent's 's reasoning, and any code or artifacts (e.g., datasets) generated. This is new release and forms the first agent-neutral benchmark (i.c., a benchmark that isn't 't designed to highlight the strengths and scope of a particular agent) designed to compare automatic scientific discovery (ASD) agents. fills a gap in the current research landscape where there are many such agents, e.g., AI Scientist (Lu et al., 2024), AgentLab (Schmidgall et al., 2025). and CodeScientist (Jansen et al. 2025), but no systematic way to compare them. In practice, to allow more controlled system-to-system comparisons, the problems are specified in considerable detail and hence only weakly test the ideation and planning steps. At the same time, these problems are not as prescriptive as typical ML coding problems, e.g., in MLAgentBench (Huang et al. 2024). The problems are created via a mixture of machine generation and human review, and include a detailed task description and a problem-specifie evaluation rubric. The final score is overall M.M--ss-uudee assessment
33

---

Published as a conference paper at ICLR 2026
based on three LLM-as-judge scores obtained by evaluating each relevant agent output (report, code, and artifacts) against the rubric. For more detail, see appendix E.9
E2E-Bench Has-Bench--ard This task is similar to E2E-Bench, except the problems are generally harder. It follows the same task definition, evaluation, baselines, and environment as EZE-Bench, however the data collection method different. For more detail, see appendix[E.1
E.2 PAPERFIND INGBENCH
In the rise of LLM-based agentic workflows, the ability answer challenging scientific search queries, across a wide range of searching criteria, have become possible. However, current paper finding benchmarks largely confine themselves small subset of search query kinds (e.g. LitSearch (Ajith et al. 2024), PaSa (He et al. 2025) and LitQA2 dataset (Skarlinski et al., 2024)). They focus on purely semantic criteria, covering metadata or navigational queries, and they are missing methodological process to cover the different within-semantic challenging types.
FaperFincing is subset of our own internal evaluation for our literature-search agent (Asta Papor Finder), which focuses on challenging queries (the internal evaluation also mixes in a bunch of easier queries. to ensure stability as a product and avoid regressions). PaperF inding is designed to be challenging (including things that our system currently does not perform well on) and realistic (based to the extent possible on real-world queries and information needs). It also aims to be broad and diverse in two axes: first, it covers a broader set of information needs. Unlike existing datasets that focus on semantic queries that search for a set of unknown-to-the--e description of their content, our benchmark includes also "navigational" queries that seek a single known-to to- so--the-ssee user paper based on a short reference ("the alpha-geometry paper"), and queries that define paper sets based on wide set of metadata criteria ("acl 2024 papers that cite the transformers paper"). The second axis of diversity is within the remantic-searce category, in which we seek to include different types of query challenges. The dataset mixes the different categories, and doesn't 't clearly indicate which query belongs to which category (even though a human will very easily tell). This is following our belief that a literature- seauch agent should be able handle all these query types, even if by merely routing them to different sub-agents.
papers based on
PaperFindingBench includes 48 navigational queries, 43 metadata queries, and 242 semantic queries. Some of the metadata queries contain (easy) navigational queries as part of their criteria, but there is currently strict separation between metadata and semantic queries (metadata queries do not involve a semantic component and vice-versa), which may change in future versions.
Dataset Creation The Navigational queries are based on PaperFinder usage logs, to include queries that, at least at some point in time, paper-finder failed on.
The semantic queries are curated from a mix of sources: PaperFinder usage logs, OpenSciLM usage logs, and existing literature-seant search datasets: LitSearch (Ajith et al. (024) and PaSa (He et al. 2025). We first identified a subset of queries that were challenging for the PaperFinder system, by looking for queries that returned few or no results identified by the system as "perfectly relevant", and for which we assessed (for query logs) or know (for the annotated dataset) that relevant papers exist, We then manually inspected a collection of such queries to identify challenge types 19 Finally, we created a set in which all challenge types are represented, while prioritizing queries for which running PaperFinder in an ablation mode with any of its components resulted fewer perfectly- papers for the ones that we do find. The set contains a mix of queries for which we assume there are many relevant results, and queries for which we assume only a handful of results exist. For numerous queries, assessing the relevance of the paper cannot be done solely based on title and abstract, but requires evidence from the paper's 's full text.
Metadata queries These were hand-crafted to achieve broad coverage of semantic-scholar API usage, as well as interaction between APIs, as well as challenges that are solvable but not directly supported
/tttps://paperrfnner..rc httpss hhttps::/opensciin..
/chat
These include, for example, multiple criteria, complex relations between criteria, use of uncommon terms, of incorrect jargon, seeking details that are not part of the main claim of the paper, query providing unnecessary even distracting hackground information.
34

---

Published as a conference paper at ICLR 2026
by the APIs, such as negation ("not citing the transformers paper"). The queries include nesting and recursion of properties, and are inspired by the most complex queries we saw in the dataset, and taken up notch or two. We emphasized queries that require combining multiple APIs.
Evaluation Evaluating retrieval is challenging, as it ideally requires a gold-set of all relevant documents the corpus, which is often not known. Such gold-set is available for the navigational and the metadata queries (each metadata query is internally associated with python code that uses the APIs to solve it completely, and whose results we use as the gold set). For the semantic queries, the full-coverage gold-set does not exist, and we resort to combination of partial annotation and LLM- based judgement. Each query is associated with a (potentially empty) small-set of Enown-to-be-gome matches, well as with weighted set of relevance criteria that should be individually verified by the LLM against evidence from the paper for the paper to be considered a good match. The individual relevance criteria were automatically generated by LLM based on (potentially expanded version of) the original query. For fifth of the queries, the relevance criteria were manually verified and corrected or tweaked. As the tweaks and corrections turned out be mostly minimal, and as the LLM-based relevance criteria were proved to be highly effective for the queries for which manual annotation for some papers is available, we consider all the relevance criteria as reliable, though they may be further improved in future versions. As we aim to assess retrieval and not the judging-LILM's ability to handle long-contexts, we don't t provide the paper's full-text for relevance judgement but rather require each result item to be associated with extracted evidence text (either from the paper itself or from papers citing it), which is then fed to the LLM for relevance judgement.
Scoring Metrics
We use two different scoring metries.
For the navigational and metadata queries, for which the gold-set is known, we use F1 over the result-set to score individual queries.
For the semantic queries, which are based on LLM judgement, we can compute precision, but not recall. One potential metric would be simply the number of returned documents that are L.MM-juged to be relevant, however, this number is unbounded and harder to integrate with other scores in AstaBench. We thus opted to compute recall over an estimated set size for each query (that is, we divide by estimated set size and not definitive onc), to bound the numbers between 0 and 1. The estimated set size is determined by running multiple variations of PaperFinder with very lenient threshold, taking the union of the resulting set, and then multiplying it by factor that ranges from 2 to 10 to estimate an upper bound and allow room for additional papers (smaller initial sets are less reliable and are multiplied by larger number). Note that in extreme cases, this may result in a recall number larger than 1. We bound this by considering the retrieval-adjusted metric of recall@k where we set be the estimated set size (this corresponds to the established recall@ R metric, but we compute estimated - recall@estiman the score in 1, and also discourages submission of "junk" results.
Computing recall @k fulfills two purposes: it bounds
We balance recall @k k not by precision, but by nDCG, as it provides a more relevant signal (favoring ranking relevant documents over irrelevant ones). The combination of nDCG and recall@estimanl makes precision mostly redundant. To provide single score for each individual query, we combine the recall and nDCG numbers using an harmonic mean (FI over estimated-recall and nDCG).
To provide single unified score for the entire dataset, we average the individual query scores, overall queries regardless of their type.
Tools Cutoff Date We encourage participants to use the keyword and snippet search functionalities provided in Asta Scientific Corpus. In any case we expect submissions to follow the same cutoff date as the corpus cutoff date for both these tools which set to June 1st 2025.
Example Input An example input can be found in appendix H.I.I
E.3 SCHOLAROA-CS2
Scientific literature reviews longstanding component of scientific workflows, and today increasingly automated by commercial and open long-form QA services, such as OpenAl Deep Research, ScholarQA (Singh et al. 2025), Elicit, Perplexity, Paper QA (Skarlinski et al. 2024).
35

---

Published as a conference paper at ICLR 2026
and many others. Evaluating long-form answers to literature review questions is challenging problem in natural language processing. Many acceptable long-form answers exist for any given question, and even with dataset of "gold" answers, it is difficult to define how to score given answer across the relevant dimensions of quality (coverage, correctness, attribution, etc.). The task is especially challenging in the scientific domain, where assessing an answer requires deep subject- matter expertise and can change over time. Asai et al. (2024) introduced ScholarQABench, which consists of multiple datasets evaluate scientific QA systems over several dimensions. Only one of its datasets- ScholarQA-CS, which we build on in our work evaluates answer coverage based on a set of target key ingredients (necessary points to cover in comprehensive answer, manually annotated in that work) for each question. The authors of ScholarQA-CS identify several limitations of their dataset, including that the annotated key ingredients could be subject to "gaming" because they reflect specific preferences of the two annotators, and that the full evaluation relies on heuristically set weight terms. In our new dataset, we instead collect a diverse set of key ingredients from a variety of candidate system responses, and also develop new LLM-as-judge approaches for answer relevance and improved citation evaluation.
Evaluation Our ScholarOA-C3S evaluation takes in an answer to question and outputs a score which is an average of four constituent measures of answer quality: citation recall (whether each claim in the answer is fully supported by its citations), citation precision (whether each citation in the answer supports its associated claim, at least partially), answer relevance (whether each paragraph of the answer addresses the question) and answer coverage (the fraction of necessary points covered in the answer).
All four evaluations rely on an LLM as judge, and the prompts are given in appendix H...33 To enable accurate assessment of citation recall and citation precision, we leverage a feature of many evaluated systems: they provide quotes from each cited article intended to support the associated claim. For each claim, if the L.LM judge assesses that the claim is fully supported by any combination of its citations and they include at least one supporting quote, that claim receives a citation recall score of 1.0. If the LLM judge assesses support based on the cited paper's title but there are no supporting quotes (this can happen because the system lacks the quote feature or because the particular sources' texts unavailable to the system e.g. for copyright reasons), the claim receives score of 0.5. Otherwise, the claim receives a score of O. Our final citation recall measure is an average over claims. To compute citation precision, we use the LLM judge assessments of whether citation provides at least partial support for its associated claim. If yes, the citation receives a score of 1 (or 0.5 if it lacks quote), otherwise it gets score of O. Our final citation precision is the average of these scores macro-averaged by claim. Note that these citation metrics do not verify that citations refer to real papers or that quoted snippets actually appear in the cited sources. Accordingly, in our evaluation we discarded snippets for systems that we evaluated which do not have access to real literature and therefore would be likely to hallucinate and receive inflated scores. For answer relevance, we instruct the LLM judge to evaluate the answer, one paragraph at a time, and instruct it to return a list of paragraphs that are not directly relevant for answering the query. Our final answer relevance score is the proportion of relevant paragraphs.
The fourth measure, answer coverage, is more challenging to assess because it requires not only evaluating the answer itself, but also identifying the key elements that correct answer to the question must include. Inspired by the approach taken in TREC information retrieval competitions (Craswell et al. 202T), for each question we gather pool of candidate ingredients from the systems are evaluating question, we ask the LLM judge extract key ingredients from each system's 's answer, identify specific details associated with each ingredient, and classify each ingredient's 's importance as "answer critical" (must-haves for answering the question) or "valuable" (nice to have, but not critical). We then cluster the extracted ingredients by instructing the LLM judge to group semantically similar ingredients together while retaining the importance label. This process results in question-speciff rubrics of ingredient clusters. The ingredient extraction prompts are given in appendix H.3.5
20 and assess the ingredients using an LLM judge. Specifically, for each evaluation
The rubric ingredients are used answer evaluation-tin -iime to measure coverage. For each ingredient cluster, the LLM judge gives a score of O (does not meet the criterion described in the rubric
Speciifically we source from the eight "QA- long" systems listed in Table plus two baseline LLMs without retrieval- Claude Sonnet 4.0 without thinking and Google's Gemini 2.5 Pro. All reports sourced were obtained hefore the cutoff date of June 24, 2025.
36

---

Published as a conference paper at ICLR 2026
ingredient), 1 (somewhat meets the criterion) 2 (perfectly meets the criterion). The final answer coverage score is a weighted average of the individual ingredient scores, with ingredient importance determining the weight (with "answer critical" ingredients counting twice as much the "valuable" ingredients). The answer coverage prompt is shown in appendix H.3.3 with a sample rubric in appendix HH..2
As our test set, we gather 100 user questions issued to OpenšciIM (Asai et al. 2024), filtered for language, quality and topic (we select questions from the computer science domain). The details of the selection process are given in appendix [E.31] As development set, we retain the previously published ScholarQA-CS dataset (Asai et 2022) of 100 questions and update its ingredient lists using the same methodology described above.
Data Collection
Choice of LLM Judge Since our evaluation is based upon LLM as a judge, we selected an LLM that handle long input contexts for processing long-form answers and also follow the various constraints described in our prompts. We choose to use gemini-2.5 models. We correlated the performance of gemini- -2. 5-.55-flat flaassh and gemini-2. meminii-2.. 5-pro the judge on the task optimized systems Section 4.3 evaluated on Schol arQA-CS2, and found that the Pearson correlation was 0.995. We therefore use gemi 12.2.5----
5- f ash as the official evaluator given its lower usage cost.
Validation of ScholarQA-CS2 We empirically validate our evaluation by measuring how well its ranked scores correlate with expert annotator judgments. Specifically, the annotators are presented with a query and answers from three models and are asked to rank them based on answer quality, taking into account the quality of citations, the relevance of the text, as well as, other more subjective preferences like the flow, organization, and structure. We conduct this three-way comparison over all eval test questions, selecting at random answers from pool of six agents-Ase -Asta Scholar QA (w/ Tables). OpenAI Deep Research, Elicit, Perplexity Sonar Deep Research, STORM, and a Qwen3-8B model finetuned on QA pairs collected from production Asta Scholar QA, and calculate win rates. At the system level, we find moderate human numan-model agreement (Kendall r = 0.467), which rises substantially to 0.800 when excluding Elicit outputs for which experts show systematic dispreference. At instance-level,
observe overall agreement of 68. 68.1%. of 0.369) for instances with a clear winner (i.e., human agreement). This agreement is higher than the agreement with individual metrics (38.7%-63.5%%) which suggests that the metries may be working in concert to more accurately capture human counterbalance their individual weaknesses and collectively achieving more than any single metric can on its own.
one another in ways that
For rubric validation, we additionally investigate the concern of bias arising out of sourcing our candidate ingredients from the systems we evaluate. In particular, we examine how the answer coverage scores change for systems when they are held out of the ingredient extraction stage. Specif- ically, we select systems with competitive answer coverage scores (Asta Scholar QA (w/ Ta- bles), OpenAI Deep Research, El Elicit, Perplexity Sonar Deep Research, SciSpace Deep Review) and create five different sets of rubrics for our test questions, where each set holds out one of the five systems and sources ingredients from the remaining nine systems. We recalculate the answer coverage scores using the held-out rubrics and compare them against the answer coverage scores from our full (10-system) reported results. The results show that the effect of being held varies across systems, with three (Asta Scholar QA (w/ Tables), 81 ilicit, t, and sci Space Deep Review) experiencing significant drops in performance in the held-out condition (average 2.5 point drop: p <= 0.01) and two (OpenAI Deep Research and Perplexity Sonar Deep Research) with insignificant drops (< I point drop; p 0.16). The degree of held-in bias decreases as we add more systems: separate hold-out experiments with a 4-system rubric shows a 5 point average drop across all evaluated systems. This suggests that including more systems for rubric creation is helpful for mitigating bias. However, more investigation is necessary to determine the reasons for bias and how to most fairly evaluate systems that we not used for rubric creation.
Tools Cutoff Date Our lorg-form QA task relies on access to the keyword and snippet search functionalities provided in Asta Scientific Corpus. The corpus cutoff date for both these tools set to May 1t 2025 for this task.
Example Input An example input can be found in appendix H.3.1
37

---

Published as a conference paper at ICLR 2026
Field of Study
# Papers
Computer Science
94.3%
Mathematics
21.3%
Engineering
9.1%
Medicine
5.8%
Physics
4.2%
Biology
1.3%
Other
0.8%
Table 18: Distribution of fields of study (FoS) among papers in the Arx.LvDrGeSTTblle--Cle validation and test sets. Note that paper can have multiple FoS tags. Tags with fewer than five papers are grouped into the "Other" category, which includes Geology, Sociology, Materials Science, History, Political Science, Environmental Science and Chemistry.
E.3.1 QUERY SELECTION
Here we outline the procedure for collecting 100 test set queries. We obtained from OpenScholar on Feb 21, 2025 8K random input queries with three words more, and used LLM (Claude Sonnet 3.5) to annotate them over five dimensions: language, field of study, clarity, completeness, and query type Based the generated annotations, we down select to English, Computer Science queries that express clear research request, for total of 3.5K queries. We then random sample 200 instances, which then manually examined by four of our authors for question clarity, quality, and answerability to obtain our final 100 test queries. For detailed prompts, see appendix H.3.4
E.4 ARX.VDIGESTABLES-CCOM
Data Collection Padmakumar et al. (2025) identify that instances in ArxivDIGESTab sometimes contain one of the following issues:
Generic columns (e.g., year of publication, research focus etc.)
Unrecoverable columns containing information that cannot obtained from full-texts of papers in the table (e.g., dataset instances)
Generic columns are trivially easy to generate (over-optimistic performance estimates), while unre- coverable columns are impossible to generate (under-optimistic estimates). Therefore, evaluating on subset free from these issues ensures that obtain a realistic estimate of model performance. Since filtering such instances automatically non-trivial, Padmakumar et al. (2025) manually curate ArxiviIGESTables-CC
subset of 170 instances free of these issues. We use this subset, randomly sampling 100 instances to create the test set and using the remaining as a validation set.
Table 18 presents the distribution of fields of study in ArxivD IGESTabl ee-oees-11 Mas-Ceea.
Evaluation Newman et al. 2024) originally proposed reference-baatte automated evaluation pro- cedure for the task of literature review table generation. Their procedure consists of two components: evaluating the schema (columns) and values (cells) for generated table. However, this decomposed evaluation has two disadvantages.. First, requires agents evaluated on this task to expose the same of components (column generation and cell value generation), instead of allowing flexibility in agent design. Second, cell value evaluation is conducted by providing agents with the set of "gold" columns from the reference table and assessing how well generated cell values match the cell values in the reference table. Therefore, this evaluation component effectively just measures the ability of agents to perform question answering over a single paper. address these disadvantages, we develop an end-to-end evaluation methodology inspired by TABEVAL (Ramu et al.) (024). The TABEVAL protocol first represents a generated table's 's semantics by breaking it down into a list of natural language atomic statements, process referred to as table unrolling. Then, it compares these
21 For query type, we instruct the model to distinguish between queries that contain an identifiable request, queries that resemble search terms, and queries that seek to test the capability of the agent (e.g., "can write ?" "can i speak chinese?"(sic)).
38

---

Published as a conference paper at ICLR 2026
statements against ground truth statements produced from reference table using maaallment-baseee measures. We adopt the same approach, prompting GPT-4o to perform unrolling on generated tables, and then reporting the proportion of ground truth statements from the reference table that entailed by the unrolled generated table (judged by GPT-4o) as recall. The prompts for table unrolling and assessing entailment are provided in appendix H.5.2 and appendix H.5.3
Example Input
An example input can be found in appendix H.5.1
E.5 SUPER-EXPERT
Task Each input in SUPER-Expert consists of (a) question specifying a particular research task to execute within a code repository (see example in appendix [H.6.]], (b) a specification of particular output result to produce, and (c) and details of the corresponding GitHub repository. The goal then is for the agent download the target repository, and perform all of the necessary setup and configuration needed for running the repository code, modify specific details in the code as needed for the task (e.g., dataset name or location), execute the target task, and finally report the result the desired format.
Annotation What makes SUPER-Expert challenging is that such repositories are not well- documented, each repository has its own set of issues, and while it's sometimes possible to make a high-level solution plan, it is very difficult predict what specific error will one encounter during the setup and execution process. Gold solution annotations for these tasks were therefore obtained using high skilled annotators familiar with running ML and NLP experiments, hired through Upwork They produced solutions in the form of Jupyter notebooks which are also available as part of the benchmark.
Evaluation AstaBench includes two of the original splits from Bogin et al. (2024); the Expert split test and the Auto split containing 50 auto- aatto-ennrated
containing 45 end-to-end problems problems (generated based on the README file respositories that pass a certain filter) as our development set. Scoring for the Expert split is done by computing the exact match metric between the produced solution and the annotated gold solution (often a JSON dictionary containing output experiment metrics such as loss values).
Example Input example input can be found in appendix H.6.1
F.6
CORE-BRNCH-HARD
The version of CORE-Bench-Eaze that we include in AstaBench is adapted in a few ways:
The original task comes with three difficulty levels (Easy, Medium, and Hard). We use the Hard version, which makes the task more challenging by removing several files from the capsule (such as the run script and the pre-computed result files), so the agent has to figure out how install and run the code before can do its analyses.
We remove instances that would require a GPU to run, to keep the resource requirements in line with the rest of the tasks. This reduces the dataset to 37 samples instead of the original 45.
Though not mentioned in the paper, the original benchmark code includes standard promp 24 that describes the general task requirements and expected format of the output report. We always include these instructions in the task input to ensure that the task is self-contained.
We use the train split of the original dataset as the validation split in AstaBench.
2 https::/ww..pwwnt com
httts::/fupyter.org
03b hte.aa/aaaat/eccccccc..oo.p.....
39

---

Published as a conference paper at ICLR 2026
The field of study distribution in the test set is 14 Social Sciences problems (37.8%), 12 Medical Sciences (32.4%), and 11 CS (29.7%).
Example Input
An example input can be found in appendix H.7.1
E.7 DS-1000
We use the original version of DS-1000 from Lai et al. (2023) and the task implementation from Inspect evals (UK AI Safety Institute and Arcadia Impact and Vector Institute (025). In contrast
to the original test set, we reserve 100 examples from the original set for validation and system development.
Example Input
example input can be found in appendix H.8.1
E.8 DISCOVERYBENCE
(Mjuumder et al. 2024) provide initial evidence for the automated scientific discovery paradigm within the setting of data-driven discovery, where both search and verification of hypotheses may be carried out using a dataset alone (i.e., after physical experiments and data collection, but the extent of this ability remains unclear. We, therefore, aim to systematically evaluate the following question: How capable are current state-of-the-art LLMs at automated data-driven discovery?.
Answering this question is hard, data-driven discovery in the wild (real-world) is diverse across domains and subject areas, which in turn makes it difficult to build a robust evaluation framework to measure progress. We address this using a pragmatic formalization of data-driven discovery. namely the search for relationship that may hold between variables in a context, where (importantly) the description of those facets may not in the language of the dataset. A data-driven discovery task then has one of these components missing, e.g., "How did urban land use affect the invasion of introduced plants in Catalonia?", Importantly, this formalization allows for systematic, reproducible evaluation over wide variety of real-world problems, by leveraging these facets.
Task DiscoveryBench (Majumder al. 2025) is novel benchmark for discovering data-driven hypotheses. In this benchmark, a data-driver discovery task is defined as follows: Given one or more task dataset(s) and discovery goal, derive hypothesis addressing the goal with the highest specificity for the context, variables, and relationship supported by the dataset(s). Optionally, a workflow for deriving hypothesis be output augment information already present in the hypothesis. Each hypotheses have to be verified programmatically (e.g., using Python) through data analysis workflow.
Data Collection Our goal is replicate the scientific process undertaken by researchers search for and validate a hypothesis from one or more datasets. We focus on six scientific domains where data-driven research is the cornerstone of scientific progress: sociology, biology, humanities. economics, engineering, and meta-science. Our gold trajectories to solve a discovery task carefully follow the published papers' workflows in respective domains. As most of the papers are highly cited, peer-reviewed, and from top venues in the domains, it is reasonable to assume the published workflows are scientifically valid.
The domain distribution in Discoveryßench is shown in Table 19
Evaluation We evaluate task performance by measuring the alignment of the predicted and gold hypotheses natural language. We designed a model-based evaluation strategy using gpt-4- preview-012 0125 as the evaluator, conditioned on our structured formalism of data-driven hypothe- ses, i.c., a hypothesis variables. Critically, the evaluator assesses entailments/cquivalleeee between linguistic elements of predicted and gold hypothesis pair, following several LM-based language entailment as automatic tools for scientific claim verification.
composed of a context, variables, and a relationship between interacting
Example Input
An example input be found in appendix H.9.1
40

---

Published as a conference paper at ICLR 2026
Domain
Percentage
Meta-science
41.8%
Sociology
24.3%
Humanities
15.9%
Biology
6.7%
Engineering 6.3%
Economics 5.0%
Table 19: Distribution of domains in DiscoveryBench.
Agent
Model
E2E-Bensh
B2E-Bench-Hard
Faker
gpt-4.1
0.00
0,00
Asta CodeScientist
claude-3-7-sone
0.05
0.03
Asta Panda
claude-sonnet-4
0.00
0.03
Table 20: Overall end-to-end task completion rates (all required steps completed successfully). While individual step completion accuracy is reasonable (up to ~70%, Table the likelihood of completing all (typically 10-15) steps remains zero due compounding.
E.9
E2E-BANCH
Data and Data Collection Each example is research task the domain of AI/NL,, for example:
"Test whether effective prompts discovered for large language models can directly improve smaller models' performance on classification tasks."
followed by detailed description of the steps to perform this test. Tasks were created using mixture of machine generation (using Asta CodeScientist ideator tool) and human review and editing follows: First, collected all * *ACL conference papers from 2021 later with at least 100 citations and available on arXiv (288 papers). The ideator tool then picks two at random and uses these to LLM-generate up five research ideas from the combination, repeated until we have 4400 ideas, which are then automatically simplified, filtered, and ranked. Finally human expert raters reviewed the top ideas, discarding infeasibleiippossbble ideas or making small edits to repair them (if possible). The top 50 were used for the final dataset.
Evaluation During idea generation, an ceample-specifee scoring rubric is also auto-gettt autto-genrratdd, asking whether all the necessary stages of research were conducted. Each rubric item is scored using LLM-as-judge against three facets the ASD outputs separately (report, code, artifacts), to provide overall score. More details given in appendix F.9
While we primarily report the average research-step completion rate (Table IOP. Table 200sshw the overall task completion scores (when all required rubric items met). These overall scores zero, due to compounding, reflecting the continuing challenge full end-to-end research.
Environment Given the complexity and time/dollar cost of ASD agents, ASTABench supports cache-based agents where (a) answers to all examples are precomputed offline, then (b) run- time cache-based agent simply retrieves cached answers each question, allowing scoring in the ASTABench environment.
Example Input example input can be found in appendix H.10.1
E.10 32E-BINcH-HazR
Data Collection Rather than using Asta CodoScientiste ideator, we instead use the HypER hypothesis generation system (aasu et al. 2025). HypER first identifies a research trend starting from each of the highly cited ACL papers from the above collection. For each research trend it then generates an initial idea, which is then refined further based on relevant paper excerpts to propose novel, underexplored tasks. Unlike H2n-hench, do not apply task simplification step, but keep
41

---

Published as a conference paper at ICLR 2026
the initial proposals unchanged. Next, the proposed tasks are automatically ranked and manually reviewed by human expert raters, who discard or fix infeasible tasks. Finally the top 50 tasks were used for the final dataset.
Example Input An example input can be found in appendix H.11.1
F
AGENTS
We describe the evaluated agents in two parts: (1) the Asta agents that we optimized for scientific research tasks, and (2) numerous baseline agents- agents-both general and science-specific -tht provide access to through the suite.
F.1 ASTA AGENTS
We release nine scientific research-optimiine agent classes, including Asta v0, orchestrator agent that automatically detects the type task and dispatches to an appropriate task-specific sub-agent:
Asta Paper Finder is our paper-secking agent, which is intended assist in locating sets of papers according to content-based and metadata criteria. It is implemented as a pipeline of manual-coded components which involve LLM decisions in several key-points, as well as LLM--ased relevance judgments of retrieved abstracts and snippets. At high-level, a query is analyzed and transformed into a structured object which then fed to an execution planner that routes the analyzed query to one of several workflows, each covering particular paper-seeking intent. Each workflow may involve multiple steps, and returns while weighting content relevance together with other criteria which may appear in the query (e.g.. "early works on", , "influential" etc). This agent is a frozen-in-time and simplified version our live saper-finding agent available to in Asta, which is restricted to single-turn interactions, does not ask for clarifications nor refuses queries, and which is using only the tools exposed in the AstaBench public APIs. It described in more details in appendix F3
dlevance-judger set of papers, which then ranked
Asta Scholar QA is previously published scientific long-form question answering system. composed of three components: retrieval to identify relevant passages from two Semantic Scholar corpora; re-ranker select the most relevant of the retrieved passages; and multi-step LLM pipeline to create the final comprehensive report, including in-line citations. We experiment with several LLMs (including gpt-s!) part of the pipeline and report the best results with claude-scnnet- -4 -4222505144 We further report results with gpt 40-m-4- -mini, and gemini-2... compare the performance and cost against smaller LLM. See Singh et al. 2025) for complete details on the system.
plash-previeew0S--e-
Asta Scholar QA (w/ Tables) a variant of Asta Scholar QA that includes literature review tables. The Scholar QA system generates answers with sections each of which is either long form paragraph or list of items and their descriptions. In the latter case, the corresponding section also includes literature review table comparing the cited papers across multiple dimensions relevant to the query. The creation of tables leads to more LLM calls resulting in higher costs as well. We report our best results with this variant with Sanude-scnnet--. 44 20-2025051- as the backbone LLM.
Asta Table Synthesis is a previously published literature review table generation system. It follows two-step prompting workflow. Step retrieves titles and abstracts all input papers from the Semantic Scholar database and provides this information alongside the table's caption to an LLM to generate suggestions for columns/aspects along which papers can be compared. Step rephrases each column natural language query and prompts an LLM to generate cell values per paper conditioned on snippets relevant to the column retrieved from the paper full-text. We report results with the following backbone LLMs in this two-step workflow: gpt-4. 1, o3, gpt-S-minif spt-st, gemini-2.5
claude-22 23-5-haiee claude- sonnet-4 - 4, gemini-2 2-....55.... -fa-flash-ppp w-w.-aaa--eeeeee----------- pro, and 11 amama- 4 scout. See Singh et al. 2025) for complete details.
42

---

Published as a conference paper at ICLR 2026
Asta Code is an implementation of the React-style code agent in Bogin al. (2024) that was originally designed for the SUPER-Expers evaluation. In addition to implementing a standard ReACT
loop, it also has a built-in tool for file editing and a custom trajectory representation that facilitates fine grained trajectory evaluation. This includes evaluating whether certain landmarks (i.e., expected points in the trajectory trace) have been reached by the agent to measure partial success, as well as the ability to run code agents with partially filled-in gold trajectories. While these evaluation features are currently limited to SUPER-Expez this solver allows for other code tasks to be extended to facilitate this kind of intermediate evaluation, and has an abstract structure that allows for the implementation of other agent workflows beyond ReACT.
Asta DataVoyager a role-based multi-agent system powered by large generative model from (Majumder et al. 2024). Asta DataVoyager can semantically understand a dataset, programmati- cally explore veritiable hypotheses using the available data, run basic statistical tests (e.g., correlation and regression analyses) by invoking pre-defined functions or generating code snippets, and finally analyze the output with detailed analyses. The core components of the system consist of specialized agents that are designed to manage different aspects of the data-driven discovery process pleensnn..nnnn.... programming and code execution, and data analysis. Additionally, to interpret plots generated during analyses, upon generation, we run multi-modal generative model (here, gpt-4o) to produce a natural language summary of such figures so that other subagents can access that information as additional context. We employ the AutoGen framework 25 that allows agents to communicate in arbitrary order, dependent on the context, which is maintained by an Orchestrator agent. See Majumder et al. (2024) for complete details.
Asta Panda performs research via LLM-based plan-and-act (hence "Panda") cycle. Given research task, it first generates a natural language plan, then systematically performs each plan step in turn, then writes a report the outcome. Each plan step is performed using SeAct/CCoeeAct-tyyy loop of (a) write Python code (b) execute it (c) reflect, and either recode (if step failed/incomplete) or move the next plan step depending on the outcome. If there are too many failures the system replans from the failed step. Since the Asta P Fanda source code 26 has not yet been integrated, we grade the cached results.
Asta CodeScientist is an autonomous scientific discovery system for domains comprising computational experiments (e.g., machine learning or NLP) (Jansen et al. 2025). Asta Codešci- entist implements idea creation and experiment construction through a joint genetic search over combinations of research articles and pre-specified codeblocks, which define common actions in the investigative domain (e.g., prompting a language model). Since the Asta CodeScientis source code has not yet been integrated, we grade the cached results.
Asta v0 is an orchestrator agent that automatically detects the type of task and dispatches to an appropriate task-specific sub-agent. It uses simple but effective text similarity approach, that achieves 100% routing accuracy on the validation set. Once the task type is identified, Asta vi hands off control to a specialized solver for that task category, chosen for best expected performance based on our preliminary experiments. The full routing table can be found in appendix F.7
F.2 BASELINE AGENTS
For the set of baseline agents, we provide two general agent classes and 11 scientific research- optimized agent classes:
ReAct a minimum-viable baseline solver that serves to measure the capabilities of LLMs without adding a sophsticated agentic architecture or task-optimized prompt. It is simple ReAct loop: chat-LLM is given a message history (initially just containing its system prompt (see appendix (155) and the task instance input) and provided tools, it generates an output message with some reasoning and attached tool calls, then the results of the tool calls are appended to the message history and the
http:://githhbb.cmmaalleee
43

---

Published as a conference paper at ICLR 2026
LLM is called again. This continues until the submit (answer) tool is called, which breaks the loop and returns the final answer.
The tool calls and responses are written with the native tool-calling format of the LLM (i.e., tool-call JSON objects attached to LLM output messages and special tool message types for responses) The agent truncates tool call outputs to most 16,384 bytes prevent long outputs from causing emrors in the LLM.
Smolagents Coder is the reference CodeAgent from the smo smolagents library (Roucher et al. 2025). It a ReAct agent, and as with the ReAct agent, the input at each step is message history; code rather than via the native tool-calling format of the LLM. Previous work has found that code- based tool calling can outperform other formats in practice (Wang et al., 2024), and it has the theoretical advantages of being able to manipulate values by reference and represent logic structures such as loops in a single step, as opposed to the LLM having to simulate these structures
however, the actions for Smolas lagents Coder represented
long sequence of calls. Smol agents Coder is instructed to produce Python code block to take actions (see appendix F.6 for prompt details); the code block is executed in the stateful Python environment (Section 4.1). and all of the agent's s tools are made available as callable Python functions. In addition, the agent can call final answer function to submit its final answer. The agent's next input includes both the return value of the final statement in the code block as well as any printed output, up to maximum of 20,000 characters.
You. com Search API is a commercial Web and News Search API, which we accessed to obtain their responses.
Elicit is a commercial AI research platform for finding, summarizing, and extracting insights from scientific papers, such as in systematic reviews. Elicit searches the Semantic Scholar database and draws on all major large language model providers to provide Al screening, extraction, and deep research reports with in-line citations. Elicit elected make submission Schola albolario a 04-03-2022 which we processed using an offline cached solver.
FutureHouse Crow general-purpose agent built PaperQA2 that can search the literature and provide concise answers to questions (Skarlinski et al. 2024). It uses a combination of OpenAl's spt-4. 1-mini and c3-mini as the backbone LLMs. Although PaperQA2 is open source, it does not include retrieval. As such, we accessed FutureHouse's API to obtain Crow responses.
FutureHouse Falcon is a closed-source agent for deep literature reviews and hypothesis evalua- tion, designed for long-form question answering 29) Falcon also uses OpenAl's gpt-4. 4 1-mini and o3 03-mini as the backbone LLM. We accessed FutureHouse's API to obtain Falcon responses.
OpenAI Deep Research a commercial deep research system that uses Web search and Ope- nAl's language models to answer scientific questions. We obtained their reports by querying the o3 03-deep-researee model via the OpenAl API for each question.
OpenSciLM is a previously published question answering system based on fine-tuned open models (Asai al. 2024). It uses custom wrapper the snippet and keywords search functionalities of Asta Scientific Corpus for retrieval and a custom reranker. The OpenSei LM paper evaluated multiple variants of its RAG pipeline, here we evaluate the publicly available demo system which uses an open 8B-parameter Llama-3. backbone fine-tuned on synthetic data.
Perplexity Sonar Deep Research is commercial deep research system that on Per- plexity's 's proprietary search and closed LLM (Sonar). We accessed sonar- Perplexity's API to obtain their responses.
via
Fgg. for OpenAl models: http:///s//aaao.o....aaaaaoo///a
1ing
44

---

Published as a conference paper at ICLR 2026
SeiSpace Deep Review is commercial system that searches Semantic Scholar, AMiner and OpenAlex, using multiple models across subtasks. Some models are fine-tuned for task-specific needs (e.g., reranking for relevance). SciSpace elected to make submission to ScholarQA-ae on 06-13-2024 which we processed using a cached solver. In their submission, the LLM was which we report in Table 6
identified as
STORM is an open-source system from Stanford that uses You.com search and synthesizes com- prehensive, Wikipedia-like articles given topics or questions (Shao et al. 2024). STORM uses OpenAl's GPT-4o and GPT-3. 5 as LLM backbones in various parts of its pipeline.
You. com Research API is commercial deep research system that runs You.com's search and unknown LLM. We accessed You.com's 's API to obtain their responses.
Faker is a baseline agent used validate the scoring metries for the End-to-End Discovery tasks. Faker simply prompts a LM to make up the report, code, and artifacts as best it can, to simulate a successful piece of research, without actually doing the work.
F.3 ASTA PAPER FINDER
The Asta Paper Finder agent (PaperFinder) is a frozen-in-time subset of the PaperFinder sub- component of the Asta project ("the PaperFinder Product"). AstaBench PaperFinder follows the overall paper-finding use, and the set of papers available it. also differs in some configuration options, and does not improve over time. Finally, unlike the product, it does not support multi-turn continuations, and is restricted single-turn scenario where the input is a complete query and the response a ranked set of matching documents, and the evidence for each one.
procedure of the product, but differs from it in the indices and APIs it can
PaperFinder is a system designed to locate scientific papers in a large corpus of scientific literature, while integrating several indices, APIs, search strategies and LLM-based judgments in an intelligent and effective manner. It handles three kinds of queries: navigational queries, that aim to find a specific paper known to the user, semantic queries that locates of their content, and metadata queries, that aim to find papers based on metadata criteria. The types not fully isolated, and metadata criteria may intersect with navigational semantic criteria. It also supports modifiers like "central", "recent" or "early", which influence the ranking of the results based on metadata information.
of papers based semantic description
The PaperFinder agent works as a pipeline of manual coded steps which involve LLM decisions At high level, query enters the query analyzer which transforms the query into a structured object reflecting the structure and semantics of the query. The analyzed query (which includes a semantic relevance criteria) is then sent to an execution planner which looks the analyzer output and routes it to one of several sub-work workflors,,,,, each of them dedicated to particular kind of search (navigational, looking for a set of papers based on semantic criteria and potential additional metadata, queries that involve complex metadata criteria, and author-based queries). The result of each of these workflows a set of papers and relevance judgments about each of them. These are then moved to ranker component that orders the papers in order which is consistent with the user's 's request, weighing the relevance scores together with other criteria such as publication time and number of citations for each work, in particular if this is supported by the query (i.c., explicit requests for "recent", "early", "classic", "central", "well known", "little known" etc). The ranked results then returned.
in several key-points 30
The PaperFinder agent uses the search APIs available in AstaBench.
F.3.1 QUERY ANALYSIS
The query analyzer is LLM based and extracts a set of predefined properties of the query. The set of extracted properties is based on manual analysis of user-issued queries, and evolves over time. It
3e found the manual -odiigg approach to be more efficient (in terms of number of LLM calls, number of tokens, and in terms of the ability to parallelize) and more reliable than more dynamic process that grants more autonomy to the LLM,, allowing it to write code and significantly influence the computation flow and search process. We do plan switch at least some component more dynamic workflows in later versions.
45

---

Published as a conference paper at ICLR 2026
covers primarily properties that are of use the downstream components (search sub-flows and final ranker), but also includes some information that is not currently handled but that we would like to be aware of, for allowing to inform the user that given query criteria is not supported (for example, author affiliations).
The query analyzer is implemented several short prompts running in parallel, each targeting different small subset of properties (ranging from 1 to 3). We do not claim this the optimal way of structuring such component, but we found it to be effective and have lower latency compared to longer prompt that extracts all of the information pieces.
The query analyzer extracts the following properties:
Broad vs Navigational Does the query target a specific paper (e.g., a paper's title, "the olmo paper", "the vaswani 2017 paper") or set of papers that matches some criteria? This is similar the distinction in traditional search queries.
Semantic Criteria Semantic criteria is a constraint or a request about the content or title of the paper (papers about X. papers that do Y). Papers in academic scientifie-literauure retrieval benchmarks focus almost exclusively on this criteria. However, real-world queries may include additional details such as metadata constraints or other properties, analyzer is to separate the semantic criteria from the other properties, and populate it in its own dedicated string. Note that the semantic criteria may be complex and include many sub-criteria ("papers about X, Y and Z that do not do W")) The query analyzer treats these single criteria and extracts them as single field. The analysis sub-criteria happens down the line.
discussed below. A major role of the query
Relevance Criteria A main component of the paper-finder is judging the relevance of each individ- ual candidate result. The query analyzer also breaks the semantic query into multiple sub-criteria (based on an LLM call), coupled with importance score and a short description of each one. These criteria will be used for assessing the relevance of the individual results.
Metadata Constraints Simple metadata fields (year, year-range, authors, venues, citation counts) are extracted at fields. For complex metadata constraints (nested, negated, refer to other papers, etc), they exist, are translated into a complex data-structure which beyond the scope of this paper.
Explicitly non-supported metadata constraints These based metadata requests that appear frequently enough and indices. Currently these includes author affiliation information (" ("papers from A12 about language modeling").
in our logs, but for which we do not currently have metadata support in the APIs
Recency and centrality modifiers Common requests that correlate with metadata information, e.g. "cntrall paper", "classic paper", "highly cited", "recent paper", "carly works" etc 3T
F.3.2 NAVIGATIONAL QUERIES
Specific paper requests are handled using a combination of three strategies that run in parallel:
1. The semantic-scholar title API.
2. Asking an LLM and then using the semantic-schllar title API to ground the answers to specific corpus-ids.
3. Extracting key terms from the query, searching for sentences containing these terms, looking for citations within these sentences, and returning the top-cited items as candidates.
Each of these strategies return more results, which then merged and returned.
46

---

Published as a conference paper at ICLR 2026
BrowdSSaarc
Iteette Short Cireut
Dense
Snowhall
Fetck Caooldates
Specific Paper Citations
Alagtive Load (Hr(B)
Keyword (52 Relevwnce) Soarch
Suggest
Relevance Judgement
Post Ritee Lyear/veree)
Score & Sort
Figure 9: PaperFinder semantic query workflow
F.3.3 SEMANTIC QUERIES
On a high-level, the process works by performing series of retrieval steps, where each of them is followed by an LLM-based relevance filtering step. Each retrieval step broadens the scope of the previous ones, and is informed based the relevant documents identified in the preceding steps.
The input to the first retrieval step is the semantic criteria from the user-query, as extracted by the query analyzer. Based on this criteria, an LLM generates k rephrasing it, and the k + 1 queries (rephrasing and initial query) are sent to the semantic search API.
Initial-search.
We now move from snippet-level to paper-level by aggregating the returned snippets according to the papers they come from. All snippets from the representing the paper, in which the snippets are ordered by their order of appearance in the paper's s text. This aggregation is performed across queries: all the snippets in all the k + 1 result sets participate in the aggregation, so that each paper item potentially contains matches from multiple sources.
paper consolidated into single item
Cited papers. For some queries, a non-negligible number of matching snippets refer to other papers ("Doe et al 2023 show that..."). We extract the set of papers mentioned in each snippet, and associate the snippet also to papers from this set. Thus, each snippet may participate in several paper items: both the paper it came from, and the papers it cites. Some paper items contain only evidence mentioned within them, other paper items contain only evidence from citing papers, and some contain a mix.
We now have set of potential papers matching the query, each containing evidence snippets from multiple sources. To each of these we add also the title and abstract of the paper.
The following step relevance judgment, in which we filter the candidate paper set using LLM judgment (see below), resulting in subset containing relevant papers with their relevance judgments. We keep the m most promising papers for the query. The order in which we go over the results matters for efficiency. We model this as multi-armed bandits problem over the different sources (each query is a source).
31 Adjectives that do not correlate with metadata information, e.g., "good paper". "high quality paper". "interesting paper", "a good summary of" are currently ignored, though some of them ("a good summary of") may make their way into the semantic criteria in some cases.
47

---

Published as a conference paper at ICLR 2026
Citation Tracking. The sedevance-judgmet groups the papers to categorical tiers, with highly- relevant being the perfect matches.
This stage takes the top categories relevat and somewhat-relevant and performs forward and backward citation searches (a procedure known in the literature as snowballing). In forward snowballing look for papers that cite the papers in the set, while in backward snowballing we look for papers cited by the papers in the set. These will then also go through relevance judgment.
Followup queries We now formulate new queries based on the returned results. This is done by considering a subset papers that were judged as relevant to the query, whose distance from the query in the embedding space the largest. Intuitively, these are relevant results which boundaries of the current search queries. An LLM reformulates query based on the papers' titles, abstracts and returned snippets, as well the original query. These are then handled like in the initial search step: issuing queries to the vector-based API, adding cited papers, aggregating the results per paper, filtering papers that are already known from previous steps, sending to relevance judgment, and returning result set, which is then combined with the existing result set.
the
Short-cireuiting This process proceeds with iterations of citation tracking and followup queries for up to predetermined number of rounds. During the process we keep track of the number of papers that were sent to relevance judgment, and the number of papers that passed The process stops if the number of found highly-relevant papers is sufficiently high, or if the number of relevance-judgment grows over a predetermined limit.
Relevance Judgment The relevance judgment component is applied separately to each of the found papers, and judges its relevance based its information (title, abstract, extracted snippets, and referring snippets from other papers). The relevance judgment prompt considers each of the sub-criteria identified in query analysis, as well as the original query. Each sub-criteria is ranked as perfectly-relevant, somnewhat-relevat or not-relevant. These are then combined to return a categorical relevance judgment (perfectly relevant, highly relevant, somewhat relevant, not-relevant).
F.3.4 METADATA QUERIES
Simple metadata filters (venue, year) top of semantic queries handled as post-filters the result set, or as ranking criteria (recent, highly cited). Queries that involve only metadata, or queries that involve a semantic criteria and complex metadata criteria, are first sent dedicated metadata retrieval component, and then filtered for semantic match using the relevance judgment component. The metadata component uses LLM calls to analyze the metadata into a structured work-plan, which is then passed to manually-coded executor which translates it to series of API calls.
F.3.5 FINAL RANKING
Finally, we combine the relevance judgements with other criteria, based on the query analysis, using a heuristic that takes into account number of citations, publication date, and the preferences expressed in the query if they exist.
F.4 AGENT SOURCE CODE REFERENCES
Asta Paper Finder
Asta Table Synthesis 33
Asta Scholar QA
61 [T12@aaaaheeettaeeeeeeeaeaaggaaa
48

---

iibllsiie as a conference
conference paper at ICLR 2026
Asta
Code
Code
(atavoy
a
30
panda (cached)
37
Aata
Asta CodeScientist CodeScientist (cached) 58
CodeSciannn
Asta v
39
40
ReAct
Smolag Smolagents Coder
Elicit
Elicit (cached)
Perplexits Sonar Deep Rosearch
SciSpace Deep Review (cached)i 44
OpenSciLM (cached
45
Openkt Deep  (cached
Seteuzeetooooeo Falco:
Crow
61 http::/ecehuaaaaaaooaaaeeeeeeeea
de
t
py
py
py
ed
pyp
py
futurehcuse_solves
49

---

Published as a conference paper at ICLR 2026
STOR
49
You. com Research AP
50
com Search AP
Faker
F.5
REACT PROMPT
The ReAct agent the system prompt from the InspectAI library's 's basic agent, constructed without knowledge of AstaBench.
You are a helpful assistant attempting submit the correct answer. You have answer.
several functions evailable help with finding may perform one Function call, You will see the result of the Tunct right after sending
measage may
message. you need perforn multiple actions, you can always send more messages with aubsequent Tunction calls, Do some reasoning before your actions, describing how they fit into your plan.
function cails you are geing to
When you have completed the task and have answer, call the submit function to report it.
F.6 SMOLAGENTS CODER PROMPT
We use the default smolagents v1.17.0 system prompt, and additionally add tool definitions in the input user message when describing the task (note placeholders for tool_descria pt. ons and task_prompt);
You have access to astabench toels in a sandbox ronmentttt.. You can use these tools in your Python code: (tool_éeccriptions]
Remember that you have 'final_answer (answer: str) function that you your final answer and mark the task completed. The answer passed to the "final_answer function should be a string Formatted according to the task instructions; depending on the task, the string might need to contain structured outputs like JSON or code, and there may be other steps (such as writing files) that you need to perform in addition to calling "final_answer'.
must
[task_prompt
The task task_prompt is simply the input from the task itself. Each available tool is represented in tool cooo__deccrppton as a function signature with the tool description and parameters, For example, for get_paper from Asta scientific Corpus, we have:
get_paper (paper_ids str, fields: str 'title, abstract, corpus authhors, year, venue, citation- Count, referenccttt,,,,,,t,,,,oo wtw.eerrreeotttitttitaaraaaaaaaa
Get details about paper by its id.
Pytooomm solver
py
py
@y@youcom_sclver
6961
y
@afaker_sclve
50

---

Published as a conference paper at ICLR 2026
Args:
paper_id: The the paper to get. The following types of IDs are supported:
<sha> Semantic Schclar ID, e.g.
aawp://eeeeeeeeeeeaaaaaaaaaaeeee CorpusId:215:16145
CorpusIa:<id> Semantic Scholar numerical ID, e.g.
: a Object Identifier, e.g.
+ 101:10018653//1/111--33
ARXIV:<id> arxiv.rg, e.g. ARXIV:2106.1590
MAG:<id> - Microsoft Academic Graph, e.g. MAG:112218234
ACL:<id> Association for Computational Linguistics, e.g.
ACI:M12-3903
EMID:<id> NubMed/Meddlin ine, e.g. PMID:19872477
<MdID:<id> PuaMed Central, e.g. PNCID:2323736
<ul1: URL, from one of the sites 1isted below, e.g. 159288v1
fields: String of sepma-sep fields include in the response. s E.g year, authors".
auar,
Default is "title", Available fields are: abstract, authors, citations, fieldsofstudy, isopenAccese, journal, publicationDate, references, tldr, url, venue, year.
Returns:
The paper object.
F.7 ASTA VO ROUTING TABLE
Asta vO's routing approach starts by predicting task type based on the (character-level) lexical overlap of the input against set of examples from the validation set. This approach sometimes confuses highly similar tasks that have the same answer format (e.g. PaperF indingEench and LitQA2-FuliTex: -eet-aeeee))),, but achieves 100% routing accuracy on the validation set.
want to route such tasks the same sub-agent anyway, it
Once the task type is identified, Asta v0 hands off control to specialized solver for that task category, chosen for best expected performance based on our preliminary experiments 53
Paper search tasks (PaperFindingBesnl per
ALtoA2-Fulll Fel:FFFuuTTe::eeeeeeaa - Asta Pa-
Finder
Long-form QA (scholargA-cs2) - Asta Scholar QA (w/ Tables) with claude- sonnet-4
Table generation (Arxtt001casTT.teee---eaeae + Asta Table Synthesis with c3
Data analysis (DiscoveryBenchh) Asta DataVoyager with configuration
Code repository replication (SUPER-Experty + Asta Code with gpt-4.1
End-to-end discovery (E28- E2E- 12E-Bench-Hard, - Asta Panda with claude- sonnet-4
Other tasks (DS- 1009, CORE- Lit.AA-
ReAct with o3
The orchestrator implements a fallback mechanism enable sub-agents to opt out: if the predicted task-type's sub-agent doesn't 't produce an output, Asta v0 retries with the next most similar task type (up to 3 attempts).
Our Asta vo experiments were started prior to the release of gpt-5, and due to time and the relatively poor performance of GPT-5 on many specialized solvers, we did not evaluate gpt-5 version for this work. We also note that Asta Code chosen hased on very early experiments with relatively old models, despite the final results showing better SUPKR- Expert performance from ReAct with o3.
51

---

Published as a conference paper at ICLR 2026
F.8 VALIDATION OF LITERATURE UNDERSTANDING AGENTS
Some scientific QA agents are not capable of outputting structured data that conforms to given schema. Accordingly, take the plain text output of these QA agents and pass them through "formatting" step. This formatting step uses an LLM (gemin1-2.0 5- 5-slash) to split the plain text report into sections, identifying the inline citations and returns structured output that conforms to SQAResponse schema. There are also some agents that proport to have structured output capabilities but whose output quality drops dramatically when it is enabled. We also use the formatting step for these agents. The list of agents for which we use a formatting step are: You.com, Perplexity DR, OpenAl DR, and FuturHouse Crow and Falcon.
For Asta Paper Finder, an expanded and continuously developed version of the agent- a user interface and additional infrastructure-iis actively used by a growing number of users. Through- out the extended period of development and real-world usage, we have validated the agent repeatedly using an internal eval set (which a superset of the benchmark we now release including some addi- tional simpler regression-testie testing queries). Although this internal set is not an established benchmark has been proven useful to monitor retrieval quality and detect any regressions in recall or ranking performance. The increasing adoption among users serves as additional corroboration of both the effectiveness of the agent and the correctness of our internal evaluation methodology.
For AittAA-FFulieen specifically, since it's multiple-choice QA task, we evaluate the Future- House (creators of the original LitQA dataset) agents, and You.com and Perplexity DR because of api availability and their suitability the task of short-form QA. The system can respond with only the correct choice or a short description with the correct choice as a json to be considered valid. ensure the baseline systems respond in the required format by
For handful of samples, issuing the same input prompt to their UI chat interfaces. Since L 11100A2PPU 1 Text is a subset of the original, direct comparison with results in (Skarlinski et al. 2024) is difficult. Further, at the time, PaperQA2 used gpt -tturoo as the backbone LLM, while Fut Puefuooee Crow, which is based on PaperQA2 uses gpt-4. 1.-...11mm 1-mini. For sanity, we look at the difference between the average accuracy result reported for PaperQA2 (66.0) and FutureHouse Crow (72.0) and conclude that evaluating on fewer questions and with better SOTA models explains it.
For Asta Table Synthesis, we expect scores on our new end-to-ent evaluation metric to gener- ally be in the same range as the results reported by Newman et al. 2024).
For Perplexity Sonar Deep Research, we set "reasoning "reasoning_effforthigghh and "search_context, context_ size=high", maximizing the model's compute and offering the best possible performance on our datasets. The Perplexity API also provides a "search_ search_mmde" parameter which can be set to "academic" to only retrieve academic sources. However, at the time of running the system (August 3rd-7th, 2025), this disabled web search entirely, so we did not set this parameter. Finally, while we found it may be possible to prompt Perplexity Sonar Deep Research to extract quotes in each of its cited sources, the API does not explicitly return these snippets; thus, we evaluate the model as if it only cites the title and URL. of each page.
F.9 VALIDATION OF END-TO-END DISCOVERY AGENTS
To score and validate agents end-to-end tasks, the E2E scorer uses task-specific scoring rubric for each task, listing the key required facets of a valid result (e.g., downloads the right dataset, selects the right baseline, etc.). The rubrics were checked manually (and updated where needed) by human annotators. To apply these, the scorer uses LLM-as-judae -as-dee to score each rubric item on each of three classes of artifact generated by the agent, namely: the generated report, the generated code, and the produced artifacts (c.g., datasets). Scores are easily viewed in a generated HTML page Fig. [T). Each facet is scored for "meets criterion" (green). "fails criterion" (red), "no evidence either way" (yellow). Only if all three facets are consistent and include "met" is the overall criterion considered "met". This three-facet approach adds substantial robustness to scoring, in particular helping avoid false positives (FP). e.g., the report states experiment was run, but the code shows otherwise, and recover from false negatives (FN), e.g., paper doesn't 't mention a criterion, but code shows was indeed implemented, see Table 21 The rubric scores were validated using spot-check sampling and verification by a human (judged 92% correct on a dev set sample of 50 rubric items). Failures include occasional over-optimistic scoring (e.g., the paper only vaguely mentions rubric item, but is still
52

---

Published as a conference paper at ICLR 2026
Figure 10: Graphical presentation of scoring single end-to-end answer. Each row is different rubric item, columns 3, 4, and 5 show whether that rubric item was met (green), not met (red), or unknown (yellow) by the generated paper, code, and artifacts respectively. Column 6 indicates whether 3-5 are consistent (green) or not (red), with the overall verdict in the last column 7. The overall score is the average of the final column cells (green = I, red = 0).
Facets Code
Paper
Artifacts
TP: Meets criterion, and met elsewhere (overall score: 1)
0.44
0.32
0.48
FPs; Meets criterion, but failed elsewhere (overall score: 0)
0.16
0.03
0.03
FN No evidence either way, but met elsewhere (overall score: 1)
0.03
0.16
0.00
TN: No evidence either way, and failed elsewhere (overall score: 0)
0.02
0.24
0.01
TN: Fails criterion (overall score 0)
0.35
0.26
0.48
Table 21: Different ways that the three facets combine (fractions) for scoring an end-to-end rubric
criterion, in particular how items that would have been false positives (FP*) false negatives (FN*) based on a single facet are corrected. For example, for 16% of the answers, the produced paper suggested rubric criterion was met, but the code and/or artifacts showed it was actually not, (desirably) resulting in an overall score of 0 for that criterion, correcting what would have otherwise been a false positive based on the paper alone.
scored 1), or failures in the details (e.g., the required code has been implemented, scoring 1, but the implementation misses an important conceptual nuance of the experiment).
G ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS
G.1
EXPERIMENTAL DESIGN
Table 22 provides a list of models run in our experiments.
G.2 EVALUATION ON FULL SET OF LITQA2 DATASET
This section presents additional details on evaluating on the LitQA2 dataset. When evaluating on literature search agent (PaperFinder), provide it with the question text as is, without including the multiple choices and without attempting to translate the question into a paper- paper-finding qurry-form. We did not do any task-specific modifications tuning of PaperFinder for this task.
53

---

Published as a conference paper at ICLR 2026
Table 22: Models run in our study. Model names are mapped to the model identifiers used during API calls, with $ used to disambiguate models that were called without their date identifiers for full transparency.
Name
Model ID
Organization Been--eiight Inference Provider
gpt opt-3.55 3ot-3.5-turbo
gpt-3.5-turae 0125
OpenAl
x
OpenAI
opt-4o-mini
opt-4o-mini
OpenAl
OpenAl
gpt-4o
gpt-4o-2024-08-0
OpenAl
OpenAI
gpt-4c1
gpt-4o
OpenAl
OpenAl
gpt-4.1
gpt-4.1-2025-04
OpenAl
OpenAl
14
gpt-4.11
gpt-4.1
OpenAl
OpenAl
gpt-4. 1-miini
gpt-4.1-mini
OpenAl
OpenAl
gpt-5-ming
gpt-5-mini-2025-
OpenAl
OpenAl
08-07
gpt-5-mini!
9pt-5-mini
OpenAl
OpenAl
gpt-5
gpt-5-2025-08-07
OpenAl
OpenAl
gpt-5t
gpt-5
OpenAl
X
OpenAl
03-mini
03-mini
OpenAl
OpenAl
03-2025-04-16
OpenAl
OpenAl
o3f
03
OpenAl
X
OpenAl
claude-3-S-haike
Blaude-3-5-haike
Anthropic
Anthropic
20241022
-laudee337-ssonet
claude-3-7-somne
Anthropic
Anthropic
20250219
claude-sonnet-4
claude-sonnet-4
Anthropic
Anthropic
20250514
geminl-2-flash
gemini-2.0-flie
Google
Google Vertex Google Vertex Al
gemin1-2.5-flesh
1omini-2.5-flasi
Google
X
previex-05-20
gcmini-2.5-flash
semini-2.5-llash
Google
Google Vertex Google Vertex Perplexity
gemini-2.5-pro
gemini-2.5-pro
Google
scnar-deep-
sonar-deeg-
Perplexity
X
research
research
1lama-4-scout
Llama-4-Scout-12
Meta
Together AI
16E-Instruct
1lama-3.1-
1lama-3.1-
Meta/
Self-hosted
openscholar-8b
openscholar-9b
Allen AI
54

---

Published as a conference paper at ICLR 2026
As LitQA2 was designed full-text search benchmark, main results the LitoA2- FullText -Saarch subset, for which our corpus contains full-text to all papers. Here we report results also the original LitQA2 dataset of Skarlinski et al. (2024). in which 114 out of the 199 queries have only their abstracts, and not full text, represented our search index. The results in Table show that PaperFinder agent obtains very similar results the agent (f al.) (2024) despite having access to only abstracts for over half the papers, and scores significantly higher on the subsets where full text is available.
Table 23: Retrieval scores on full set of LitQA2 dataset
Name
original-set portion
full-text percentage
recall
recall
30
PaperQA2 (Skarlinski et al. (0244)
full (199)
100%
69.9
62.8
PaperFinder (ours)
full (199)
<50%
70.3
64.3
PaperFinder (ours)
100%
93.3
90.7
Search Test (75)
PaperFinder (ours)
LitQA2-Ful1 -Fl22-uullText
100%
Search Val (10)
H EVALUATION TASK SAMPLES AND PROMPTS
This section provides higher level of detail for evaluation tasks through example problems and rubrics, plus detailed prompts.
H.1 PAPERFIND INGBNENCCH
H.1.1 EXAMPLE PROBLEM
Find papers relevant to the follewing query: Could you suggest research that investigates blustering-based efficient attention mechanism within Transforner models?
Try to be comprehensive in your search yet efficient and accurate, i.e.
find as many highly relevant papers possible, but try to keep efficiency in mind. You may submit up to 250 papers.
If the query asks for specific paper known to the user, i.e. "the Transformer paper", "the BERT paper", "the GPT-3 paper" etc, try to This does not
find that specific paper and only return that apply any query phrased in singular "paper" or "article" those can be general queries and should return multiple relevant papers, e.g. "which paper introduced transformer generative mode! for text generation".
Return your answer as JSON with the following structure, results should be ordered by most relevant first:
json
"output":
"results";
"paper_ict id"; "string; the semantic scholar corpus_id of 4 the paper",
"markdown evidence"; "markdoow"eevdeene""" "string; a markdown snippet with verbatim text from the paper that supports the relevance of the paper the query; the evidence should be concise and limited to the minimum needed to support the paper's relevance"
b
55

---

Published as a conference paper at ICLR 2026
H.2 LITQA2-FULLTER SEARCH
H.2.1 EXAMPLE PROBLEM
Find papers relevant to the follewing query: Active olfactory receptor genes increase their contacts with greek island regions by what factor in mouse olfactory neurons?
Try to be comprehensive in your search yet efficient and accurate, 1.e.
find as many highly relevant papers as possible, but try to keep efficiency in mind. You may submit up to 250 papers.
If the query asks for specific paper known to the user, i.e. "the Transformer paper", "the BERT paper", "the GPT-3 paper" etc, try to This does not apply to any query phrased in singular "paper" or "article" those can be general queries and shculd return multiple relevant papers, e.g. "which paper introduced transformer-basee generative mode! for text generation".
find that specific paper and enly return that
Return your answer as JSON with following structure, results should be ordered by most relevant first:
json
"output";
"results":
"paper_id": "string; the semantic scholar corpus_id + the paper",
"markdown evidence"; "narkkoownevidence"" "string; a
snippet with verbatim text from the paper that supports the relevance of the paper the query; the evidence should be concise and limited the minimum needed to support the paper's relevance"
b
H.3 SCHOLARQA-CS2
H.3.1 EXAMPLE PROBLEM
Generate a report answering the following research question. Be sure to include inline citations for each claim. Return your result valid JSON with single key "sections" which is list sections, each having keys "title' "text", and "citations". Each entry in "citations" should have JSON list of "snippets" extracted from the reference document and an 'id', each of which appears exactly in the text. Each "id" should be inline citation as it appears in the text (with wrapping parentheses or square brackets if appropriate).
Each citation should have a "title" if one is available. Any additicnal information about the citation should go under 'metadata".
Do not create References section.
Here is example "section" to help you with formatting:
"title"; "Background",
"text": "Convolutional neural networks (CNNs) have achieved state-of-the -he---rtt results in image classification [1] [2].", "citations";
56

---

Published as a conference paper at ICLR 2026
"id": "[1]",
"snippets":
"CNNs have become the standard for many visuel tasks."
1,
"Eitle": "ImageNet classification with Deep Convolutional + Neural Networks",
"metadata":
"authors"; "Krizhevsky, et al.", "year": 2012, "arxiv": "1207.0580"
"id": "[2]",
"snippets": "significant improvements image recognition have been - observed with CNNs.
"title": "Very Deep Cenvolutional Networks for Large-Scale Image Recogniticn",
"metadata";
"authors": "Simonyan, K. Risserman, A.", "year": 2014,
"arxiv": "1409. 1556"
Question: Apart from preventing overfitting, are there any side effects (desirable or otherwise) of applying dropout in deep neural networks?
H.3.2 EXAMPLE RUBRIC
"qusstion; "how the hallucination is linked the bias", Mingredients";
"name": "answer_oritical_0
"criterion": "Define AI hallucination and AI bias", "weight"; 0.1428774485714285, "examples";
"factually incorrect, nensensical, or misleading outputs 4 despite appearing confident in their responses", "when an I.M generates content that does not correspond to reality, producing cutputs that are coherent and grammatically correct but factually incorrect nonsensical",
"Ai systems generate outputs that are misleading, biased, or entirely fabricated, despite appearing convincingly real", "systematic errors skewed outputs stemming from inbalances in training data, model architecture, 4 context",
deployment
"an inclination or prejudice for or against a person or group, + especially in a way considered unfair",
"prejudiced unfair outcones due to skewed training data or flawed algorithmic design"
"name"; answor_critical_11
"criterion": "Explain shared root causes linking hallucination and bias, particularly training data issues", "weight": 0.14285/14285/1425,
Texamples":
57

---

Published as a conference paper at ICLR 2026
"biased training data",
"Both originate from the inherent reliance on statistical
+ pattern matching over true semantic understanding",
"Incomplete or biased data can lead to AI models learning 4 incorrect patterns, resulting in hallucinations", byproduct of
"Data-related hallucinatiens generally emerge biases, misinformation, and knowledge gaps, which are 4 fundamentally rooted in the training data",
"If the training data is biased, incomplete, or flawed, the Al model may learn incerrect patterns, leading to inaccurate predictions and hallucinations",
"Both phenomena emerge frem datasets that either incomplete, noisy, or imbalanced"
"name": Sanswer_oritccal_2
"criterion"; "Explain how bias directly contributes hallucination",
"weight"; 0. 1,14887712855112255,
eexamples;
"biases manifest themselves as hallucinations in summarization 4 tasks, leading to factually incorrect summaries", "correlation coefficients reaching 0.81-0.83 between intrinsic bias and extrinsic hallucination rates",
"Language models may generate stereotypical or harmful content + about marginalized groups when trained on internet text a containing systemic biases",
"bias in medical training data leads to models generating plausible out incerrect medical information", model trained data that underrepresents certain + groups or overrepresents particular viewpoints, it may generate hallucinatory content that reflects these imbalances",
language model might assume nurse is female without any
gender cue, hallucinating that detail based on gender-role stereotype"
"name"; "answer_criticaa__",
"criterion": "Explain how hallucination propagates and amplifies bias",
"weight"; 0.1422::. 0,11285714285714255, "examples";
"When an AI model hallucinates, the nonsensical or incorrect information it generates may inadvertently reveal the has learned from biased data", 4 prejudiced assumpticns
"The very act of hallucination, being a deviatien from factual grounding, can scmetimes be fanitttttion of the system's
internal biases, where the 'made-up' information aligns with these learned prejudices",
"Confidence Flawed Outputs: Hallucinations presented 4 confidently by AI can reinforce existing biases", "Data Pollution; Biased or hallucinated outputs fed back into training data create self-reinforcing cycles of inacouracy and prejudice",
"When AI systems hallucinate, they often draw upon learned patterns and associations from their training data that
4 include societal biases",
"AI hallucinations can amplify existing biases in the data, leading to discriminatory outcomes*
58

---

Published as a conference paper at ICLR 2026
"name": "answer critical_4", "criterion"; "Describe the interconnected and bidirectional
nature of the relaticnship",
"weight"; 0.14285742857142855 "examples":
"they represent different manifestations of fundamental
4 limitations in current systems", "addressing one without the other provides incorplete t solutions",
"both stem from systemic issues in data quality, model architecture, and training processes",
"AI bias manifests hallucinations when models are trained unrepresentative or imbalanced data and combined with specific architectural designs"
"name": "valuable _",
"criterion"; "Provide real-world examples demonstrating the link",
"weight": 0.07142857142837142,
"xxmmles";
"Healthcare Diagnostics: AI systems hallacinated symptoms for Black patients 348 more often than for white patients, correlating with underrepresentation in training data", "Recruitment Tools: Amazen's scrapped hiring algorithm +
downgraded resumes containing the word 'women's' while inventing irrelevant skill candidates",
for male
"Mata Avianca legal case where ChatGPT produced nonexistent
4 legal opinions",
"ChatGPT's 'Inner Racist' Incident where the model hallucinated a a hateful rant laced with stereotypes",
"In healthcare; factual hallucinations leading to logical hallucinations and diagnostic errors that can jeopardize patient safety"
"name": "valuable_i",
"criterion": "Discuss mitigation strategies that address both issues", "weight"; 0.07142857142857142,
Texamples":
"data preprocessing, algerithm selection, and model
+ evaluation",
"Training AI models large, diverse, and high-quality + datasets",
"The research community increasingly advecating for + integrated evaluation frameworks that simultaneously assess
+ factual accuracy and fairness",
"Data deduplication, impreved data curation, and augmentation reduce memorization artifacts and balance 4 representation",
"External fact-checking layers and retrieval-augment
4 generation (RAG) frameworks"
"name": "valuable 2",
"criterion"; "Explain specific mechanisms connecting bias and + hallucination", "weight": 0. 0.07142857142855142,,
"examples";
59

---

Published as a conference paper at ICLR 2026
"LVIMs struggle with object hallucinations due their reliance on text cues and learned object co-occurrence + biases",
"RLae is vulnerable te the biases inherent in the human annotators' judgments",
"ebject hallucinations in vision-language medels stem from overconfidence problems closely related to statistical
bias",
"Models rely on token probabilities and Learned correlations
rather than true understanding of underlying knowledge",
"When learned probability distributions are biased, incomplete, overly general, medels produce outputs that statistically probable but factually incorrect or biased", "Modern generative models operate like advanced autocompletion, focusing on producing likely-sounding continuations"
"name": "valuable_3",
"criterion": "Discuss implications for high-stakes domains", "weight"; 0.0714285142287112, "examples":
"can lead to misinformed decisions in critical areas such as 4 healthcare, finance, and security", "Healthcare: Medical AI might. hallucinate treatment.
+ recommendation while reflecting biases against demographic + groups",
"Law: Legal A1 systems might fabricate case precedents while perpetuating systemic biases", "healthcare applications where both phenomena lead to + misdiagnosis and inappropriate treatment zecommendations",
"Legal and judicial contexts where fabricated case citations can mislead practiticners"
1,
H.3.3 EVALUATION PROMPTS
Citation Precision and Recall
You are a claim validator. For each claim made in the following text you wi11 determine if it is supported by the quote from it's
corresponding inline citations. As is typically done in academic share citations. Make sure to alse include claims presented in table format. For references with only the title available (ie no quotes from the reference are included), judge them as "supperting" if the title indicates that the paper is likely relevant to the claim being considered. Return JSON object with single key "claims" which is list of 'claim" objects, one for each sentence the text. Each "claim" object contains the claim itself ("text"), a list of "supporting" inline citations and "non_ chon_supporting" inline citations and finally boolean
writing, assume that consecutive sentences
which indicates if the claim is entirely
supported by the quotations in the associated citations. Each inline citation corresponding to that claim should appear in either "supporting" or "nen_supporting") but net both. Rach claim made in the text should appear in your cutput, but you should skip sentences covering high level introductory information.
Answer Relevance
You are given query and correspending long answer.
60

---

Published as a conference paper at ICLR 2026
Goal: find irrelevant paragraphs in the don't directly answer the query and shouldn't be the answer.
These are paragraphs that
For instance, if the query is about datasets for scientific question answering, paragraph about multilingual question answering datasets that don't contain scientific text would be considered irrelevant.
Explicitly consider whether something may be indirectly relevant. For
example, if the question is about the conditions of horses in South
Africa, a paragraph about general animal welfare in South Africa is potentially relevant while not being precisely about horses. On the other hand, paragraph about pig welfare in South Africa is irrelevant.
Note that subtle differences can make the text irrelevant to the query. For instance, text about scientific survey paper generation is not relevant to query about autcmatic paper review generation, Even though they seem related, they are about very different tasks.
Also, useful background in general is relevant. If the question is about an approach to creating liver-related proteins, some information about liver-related proteins cculd contextualize other parts of the answer. If paragraph contextualizes another part of the answer, then it is relevant.
Go through the answer and output list of irrelevant paragraphs. Every single paragraph needs to be censidered, one by one. Our goal is to catch all the irrelevant paragraphs, Return your result as JSON object with single key paazelevant_paragraaee
please be thorough.
whose value is a 1ist of objects, each having keys 'reascn", and "answer_text' as follews: [["irrelevant "iirelerant_ppraggaphh:'" paragraphs":
[I
"reason": "discuss why something irrelevant (not indirectly relevant)", ",
"answor_text"" "exact ENTIRE paragraph (not just part of it) from the answer that is irrelevant"
11,
}}
Make sure all the irrelevant paragraphs are included.
Answer Coverage
You will be given question somecne asked (in <question></quesiionn
tags) and the corresponding response (in <response></responee tags)
given them by assistant
You will then be given an enumerated list of criteria by which
evaluate the response. Each criterion specifies zequirements that the
answer must satisfy. You will assign score accordingly (see below). tags, below each criterion) that illustrate the type of details that would satisfy the criterion. We de NOT expect any the specified details to necessarily appear in the answer. These are strictly to be used as guidance for locating the answers that satisfy the set requirement.
You will alse be given list of examples (in <examples><<exampless
For each criterion, return score of 0,
2 indicating how appropriate the response based on the given criterion. means the response does not meet the criterion, means the response somewhat meets the criterion, 2 means the response perfectly meets the criterion. Judge only the specified aspect (s) delimited by the critericn, not any other qualities of the
61

---

Published as a conference paper at ICLR 2026
Scoring Example 1:
<question>Common medical NLP papers on clinical text Sanchmarks</questione
<response> application of natural language processing (NIP) and machine learning to medical text presents tremendous apportunities for healtheare tasks such prediction ... [ERNCATEDD<IIeSSee Criteria:
<criterion>
1. Detail the well-known medical NLP datasets
<examples>
i2b2 includes datasets focused on temporal relations in clinical
narratives, CRAFT Corpus is a collection of 97 full-length,
open-access biomedical journal articles with semantic and syntactic annotations.]
</examples>
</critericn>
<griterion>
2. [TRUNCATED]
<examples>
... [TRUNCATED]
</examples>
</criterien>
A 2 point answer would fully satisfy the criterion #1. For example, it would include speci fic names with sone detaila of well-known medical datasets for like those mentioned
the examples.
point answer would only partially satisfy the criterien #1. For example, a dataset (like those in examples) may be mentioned, but no detail would be provided. or datasets may be simply listed without further discussion. would not menticn datasets all.
point
Scoring Example 2:
<question>Nhat are some of the documentation methods used in Linguistics <teel/ww..///usttiiiee. <response>Language documentation, also called documentary Linguistics, is specialized subfield of linguistics (IRONCATED] Criteria:
<criterion>
1.... (TRUNCATED]
<examples>
...RUUCCATED)
</examples>
</eriteric>>
<criterion>
2. Cover elicitation techniques for capturing specific linguistic data.
<examples>
struttured interviews, elicitations based on standard word lists, prompted speech tasks
</ </examles>> </rriterio>>
point answer to criterion #2 would contain common elicitation techniques like (but not limited to) those mentioned in the examples.
The answer specifics don't have to match exactly with the examples, but examples show the types of instances that would count towards satisfying the criterion.
point answer to criterion #2 be incomplete in some way. For exanple, the answer might mention \"elicitation sessions\" during a discussion on audio recording, but it fails to specifically address the requirement. or the answer gives list of standard word lists in the answer as resources, out fails to tie this information to elicitation.
62

---

Published as a conference paper at ICLR 2026
point to criterion #2 would simply not include the discussion in any way, For example, if an answer focuses only on data handling (post clicitation) techniques, it would miss out on techniques for documentation intervicw itself.
Scoring Example 3:
<question>How transformer medels differ from recurrent neural networks (<Rquettiin
<tespouse>rransformee models use self-sttention mechanisms to process input, while RNNs process input sequentially. Transformers are better at handling range dependencies in data because they don't rely on previous time steps to pass information. RNNs may suffer from vanishing gradients and have trouble with long-term dependencies, < responsc>>
Criteria:
<criterion>
1. Must cempare how the architecture and data processing flow ditter between transformers and RNNs. <examples>
Trans use parallel processing and self-attention; RNNS process input tckens one at a time in sequence. Transformers can look at the entire input sequence at once, while RNNs have to pass information step by step.
</eamples>> </ </criterien>
A 2 point answer would accurately and distinctly contrast both architecture and sequence-prooessing style of both model families (e.g., parallelism sequential processing, of self-attention vs. recurrence).
A 1 point answer would provide a partial or imprecise comparison, perhaps only menticning one difference, or being vague (e.g., "Transforners work differently from RNNs in how they process text" without further claboration).
A 0 point answer would explain only one architecture (e.g., only transformers), or describe both but fail to contrast them on the asked criteria.
Return your result as JSON object with single key "scores" whose value is a list of objects, each having keys "criteria_idk' "reasoning', "score" and "evidence" from the text supporting the claim.
H.3.4 QUERY SELECTION
Query Annotation Prompt
query in English? Choices: true false>, "Query Type"; <Choose from query types below or suggest your own>,
"English": <Is this
"Computer Science": <Is the query generally fall under the computer science or closely related field? Choices: true false>
"Field of Study": <Choose from the Field of Study belew>,
"Subfield of Study": kIf you chose Computer Science, Bicredicine, and esychology as the Field of Study, speci fy the subfield of study that this query most related to (examples are below).
more than one subfield, slash delimit and order from highest to lowest importance.> "Fragment"; <Do you think this is full query, or is a part obviously missing in the query? Choices: complete I missing>,
"Clarity": <Is the request clear? Choices: clearly understandable but understandable need clarifications,
vague
63

---

Published as a conference paper at ICLR 2026
"Research Stage:" <Ideation, Topic Understanding, Literature Search and synthesis, Research Design, Data Analysis, Project write up, Can't tell>
...
Query Types:
"request": This is asking the system for information particular topic or supject.
"search terms": This user giving a sequence terms, likely for search. "testing"; This user is asking the system to say something about its abilities or capabilities.
Field of Study:
"Computer Science":
Computer Science the study of computers and computational systems, including theory, design, development, and
application.
"Bi "kiomedicine"" Hiomedigine studies the application of the principles of the natural sciences and especially biology, physiology, and biochemistry to clinical practice. "Psychology"; Psychology
the study of the mind and behavior. study of the mind, how it works, and how it affects behavior. "None of the above": This query belongs to a different field of study.
the
EXAMPLES of Subfield of Study:
Computer Science: artificial intelligence, computer systems and networks, security, database systems, human computer interaction, vision and graphics, numerical analysis, progranming languages, software negineerig, and theory of computing.
Bio Biomecicine: medical microbiology, virology, clinical chemistry,
hematology, immunology, genetics, molecular pathology, microbiology, bioinfomatics, and biomechanics.
Psychology: behavioral psychology, clinical psychology, cognitive
psychology, comparative psychelogy, cultural psychology, developmental psychology, and educational psychology.
H.3.5 KEY INGREDIENT EXTRACTION AND CLUSTERING PROMPTS
Ingredient Extraction
will provide you query that tests literature knowledge and report from a system. You will use the system report to identify key requirements or "ingredients" that the report sees as necessary for answering the question. Each ingredient should include a high level descripter of what is expected in an answer, and 1ist of examp's es or details (if relevant).
How to write a good ingredient:
Each ingredient should include one requirement at a tine. For example, instcad of "The answer should mention the challenges of manual construction of ontology and discuss the use of automated methods for aiding the process. have two ingredients: "The answer should mention the challenges of manual construction of an entology" and "The answer should discuss the use of automated methods for aiding the entclogy construction."
Each ingredient should address different component of the query. If the query requests "gffect of phonemic perceptions is evident in language acquisition, speech comprehension, and second language learning", single ingredient shouldn't try to address all three "language acquisition", "speech comprehension", and "second language learning". Ideally these should be separated out into multiple requirements.
64

---

Published as a conference paper at ICLR 2026
Identify which eritically important ingredients. Critical ingredients are those, if not satisfied, would render the response uscless. This is a judgement call you must make by closely considering what the QUESTION IS REQUESTING. For example, if questicn asks for "coding datasets for assessing LIM capabilities", then identifying the most commen accepted coding evaluation dataset & benchmarks, and possibly also their details (e.g., notable methods used) would be critically important. However, ingredients that, for example, deive into the theoretical background of a particular evaluation discuss future researcn directions would NOT be critically important. For critically important information use SHOULD (e.g., "The answer should cover ..."), otherwise use MIGH. (e.g., "The answer might cover ...").
Use the main verb judiciously accerding to what you observe in the report: if the information shculd be mentioned in passing, you might use language
"The answer should MENTION/TOUCH ON ....,
should be covered in some detail language like "The answer should ..." would be appropriate. If the answer should list items then it would be fitting to write "The answer should LIST/ENUMERATE
Unless specifically required by the question, the ingredient should avoid using specific numbers or qualifiers in the ingredient
description: e.g., "The answer should 1ist the three main challenges that.... "The answer should list the main challenges that OR "The answer should list main challenges such hallucination grounding problems that ...""
An ingredient MUST:
Be agnostic as to where in the report it appears (c.g., "should begin by explaining" --> "should explain"; "might conclude by noting" --> "might nete")
Be self-centained and understandable without needing to know about other ingredients (e.g. In "The answer should also mention other commen approaches" language like "also" and "other" rely on other ingredients for disambiguation))
Not make reference to other ingredients (e.g. proneuns like "these" in "should further describe these approaches" that refer to the previous ingredient should be avoided and be replaced with mentions) Not contain (ultra) specific information, unless the question
specifically calls for it. List them "examples" instead. If ingredient mentioned the need for datasets, the examples would the specific datasets that the report mentions
Refrain from including specific mentions of variants with limited shelf life. For example, put "Honey Smacks" or "Special k" in the examples es under more generic "Kellogg's careals". Try "Apple os" in the ingredients instead of "Big Sur" "Mojave".
Purther Rules and Guidelines:
Step through the report sequentially
In writing your ingredients and examples, only use information
contained in the report.
Cover as much of the relevant pertions of the report possible. the ingredient examples scurce from report (not elsewhere)
Content you include
No references should be made to the reference report itself: e.g., don't t write "The answer should briefly define each of the key concepts introduced in the report" instead write "The answer should briefly define each of the key concepts such as..."
Noto that ingredients are requirements. Phrase them as requirements an
answer should fulfill: start with "The answer should (for answer critical ingredients) or "The answer might (for non answer critical ingredients).
Return jsen as an answer:
65

---

Published as a conference paper at ICLR 2026
"id": sequential numerical ingredient id, "ingredient": description the rtteiiieen/reqqiieeetttttt "examples"; [[ "detail"; examplcs/detale if relevant, "citation"; citaticn if available; null if not available /....
Acceptable forms of citations:
If corpusid specified in the report, cite the number, e.g., "citation"; "13756489"
If the URL (e.g. to arxiv) is specified, cite the URL, e.g., "citation": wtt..org/aab//00..77755555
If Auther and Year as specified: "citation", e.g., " "(Vaswani et al, 2017)"
no citations are available, e.g., "citation": null
Ingredient Clustering
will give you a user query and list ingredients. The ingredients are written requirements for writing good answer. Note that ingredients the writer thought are more critical to answering the query are prefixed with "The answer SHOULD". Useful but not eritical information is marked as "The answer MIGHT".
Do the following:
1. Ident the key concepts, ideas, and named entities that should be covered for this question
2. Carefully consider the query and the ingredients given to you. At this
stage, ONLY look at the ingredient description (do not consider the examples) identify minimal set of non-overlapping key remuirements that either are high-quality ingredients CR are consistently being covered in the ingredient list. Take into consideration concepts identified in 1, especially when deciding if the key requirement should be "SHOULD" "MIGHT" requirement.
3. Next, step through each of the given ingredients, and decide which set requitements it should be associated with, and distribute the examples (see Notes 1 and 2).
4. Frune the exanples: Remove exact near duplicates. Renove examples that you judge are not directly relevant the key requirement.
5. Finally, list ingredients that were left out and why.
Notel: You are allowed and encouraged to place multiple ingredients into
a single key requit This would be fitting in the case of duplicate near duplicate ingredients like "discuss physical commonsense datasets like PIQA" vs. "include discussion of PIQA other physical commonsense datasets", This type grouping can also happen if you have a more general key requirement that can handle multiple ingredients, for example, for a key requirement "discuss success of Al in disease detection" might encompass ingredients like "mention success in diabetic retinopathy prediction" and "point out that machine learning methods have been successfully used on BCG data to identify early signs of strial fibrillation".
Note2: You are allowed to split ingredients into multiple key
requirements, For example, if an ingredient reads "The answer might explain why the engagement dropped, focusing on common mistakes in interface design." ", you may end up placing it under beth the requirement. "The answer might explain the drop in engagement" and the requitement. "The answer might discuss common mistakes in interface design", distributing its examples to the appropriate requirement.
Rules:
Always keep your focus on the query. All key requitements must be relevant for the query.
NEVER include an ingredient in requirement on the basis of the examples alone. ALMAYS make that the ingredient description is prioritized.
66

---

Published as a conference paper at ICLR 2026
Use your best judgement for deciding whether key requirement should be "SHOULD" or "MIGHT" requirement ALMAYS based on the question and the key concepts and ideas you identified early on.
Each requirement should ideally address a different component of the query. If the query requests "Effect of phonemic perceptions is evicent in language acquisiticn, speech comprehension, and second language Learning", a single requirement shouldn't try to address all three "Language acquisition", "speech comprehension", and "second language learning". Ideally these should be separated out. into
multiple requirements.
Remember, the key requirements should not overlapping. For example:
Note that ingredient R1-"The answer should introduce transformer architecture componente, including attention mechanisms and their role in sequence modeling" partially overlaps with R2-"The answer should discuss the role of attention mechanisms in sequence modeling". This should be avoided, when possible: RI could instead be "The answer should introduce transformer architecture compenents" since the rest is covered by R2.
Each key requirement should be self-contained and understandable without needing to know about other requi (e.g. pronouns like "these" in "should further describe these approaches" that refer to the previcus rements should be avoided and be replaced with menticns).
Although "should" ingredients are more important, the "might" ingredients are also valuable to Include those that you think they would (best) help answering the user's query. There should never be a key requirement that has no ingredient associated.
It's okay to have leftover ingredients. Ingredients that you think are not very relevant, too vague, peripherally relevant be left out even if they carry the "shculd" phrasing.
Background or causally related information uniess the query asks expl icitly for them, should be considered "MIGHT" requirements. DO NOT include key requirements that are centrally about paper citations. For example, do not include requirements like "List recent papers... " "Cite the most impactful papers... "Identify and discuss important papers...".
Repeat (THINK) after me!
I will be choosy about "SHOULD" requi rements. "MIGHT" requirements, I
can use liberally.
will base "SHOULD" and "MIGH"" based key concepts judge being central to answering the query.
will always write requirements that are relevant to the query.
Return jsen:
"key_requirements""
"key_requi rement": description designed after the ingredients you group together,
"ingrecients": |the ingredient id list of those ingredients you grouped.
"examples": [concatenated relevant examples from ingredients in this requirement "detail": examples/details if relevant, "citation": citation if available; null if not available ) ...]
feft__outnnngredenite ""ingrecient": id the ingredient that got left out, "reason": brief reason why was left out.l,
67

---

Published as a conference paper at ICLR 2026
H.4 LiTOA2-FULITEXT
H.4.1 EXAMPLE PROBLEM
Active olfactors receptor genes increase their contacts with greek island regions by what factor in mouse olfactory neurons?
A. 2.0 fold
B. fold
C. 1.7 fold
D. 2.7 fold
E. Insufficient. information to answer the question
F. 3.0 fold
Answer with the letter of the chosen answer in JSON: ("answer"; "<letter>").
H.5 ARXIVDIGRSTABIBS-CHI
H.5.1 EXAMPLE PROBLEM
We would like you build table that each paper as a row and, as each column, a dimension that compares between the papers.
You will be given multiple papers labeled Paper 1, 2, and so on. You will be provided with the title and content of each paper. Please create table that compares and contrasts the given papers, that would satisfy the follewing caption: Comparison of Receiver Operation Policies for RFEHNS..
Return the table in the specificd JSON format only.
Make sure that the table has dimensions which are phrases that can compare multiple papers, and papers as rows.
Paper 3343717 title: Wireless Information and Energy Transfer in Multi-Antenna Interference Channel
Paper 3343717 abstract: This paper considers the transmitter design for wircless information and energy transfer (WIET) in a miltiple-input singlc-output (MISO) interference channel (IFC). The design problen is maximize the system thrcughput subject to Individuel energy harvesting constraints and power constraints. It is observed that the ideal scheme, where the receivers simultaneously perform information detection (ID) and energy harvesting (EH) from the received signal, may not always achieve the best tradeoff between information transfer and energy harvesting, but simple practical scnemes based time splitting may perform petter. We therefore propose two practical time splitting schemes, namely the time-division mode switching (TDMS) and time-division multiple access (TDMA), in addition to the existing power splitting (PS) scheme. In the two-user scenario, beamforming is optimal to all the schemes. Moreover, the design problems associated with the TDMS and TDM schemes admit semi semi-amalytical solutions. In the general K-user scenario,
show that
successive convex approximation method is proposed to handle the WIET problems associated with the ideal scheme, the 25 schere and the TDMA scheme, which are known NP-hard in general. simulation results show that none of the schemes under consideration can always dominate another in terms of the sum rate performance. Specifically, it is observed that stronger cross-link channel power improves the achievable sum rate of time splitting schemes but degrades the rate performance the ideal scheme and PS scneme. As a result, time splitting schemes can outperform the ideal scheme and the PS scheme in interference dominated scenarios.
Paper 8313045 title: Wireless Information and Power Transfer in Multiuser OFDM Systems
68

---

Published as a conference paper at ICLR 2026
Paper 8313045 abstract: In this paper, we study the optimal design for simultanecus wireless information and power transfer (SWIPT) in
downlink multiuser orthogonal frequency division multiplexing (OFDM) systems, where the users harvest energy and decode information using the same signals received from a fixed access point (AP). For information transmission, schemes, namely, time divisicn multiple access (TDMA) and orthogonal frequency division multiple access (CFDMA) At the receiver side, due to the practical limitation that circuits for narvesting energy from racio signals
consider two types of multiple access
not yet able to decode the carried information directly, each user appl ipslies either time switching (18) power
splitting (PS) to coordinate the energy harvesting (EH) and
information decoding (ID) processes. For the TDMA-based information transmission, employ TS at the receivers; for the CFDMa-based information transmission, above two scenarios, we address the problem maximizing the weighted sum-rate over all users by varying the time/froquency power allocatien and either TS or PS ratio, subject to minimum harvested energy constraint each user as well as peak and/or total transmission power constraint. For the TS scheme, by appropriate variable transformation the problem is reformulated as a convex problem, for whigh the optima power allocation and TS ratio are obtained by the Lagrange duality method. For the 2s scheme, we propose iterative aigorithm to optimize the power allocation, subcarrier (SC) allocation and the PS ratio for each user. The performances of the two schemes are compared numerically as well as analytically for the special case of single-user setup. It is revealed that the peak power constraint imposed each CFDM sc well the number of users in the system play key roles in the rate-energy performance comparison by the two proposed schemes.
employ PS at the receivers. Under the
Paper 902546 title: Wireless Information and Power Transfer: Energy Efficiency Optimization in OFDMA Systems
Paper 902546 abstract: This paper censiders orthogonal frequency division multiple access (OFDMA) systems with simultaneous wireless information and power transfer. we study the rescurce allocation algorithm design for maximization of the energy efficiency of data transmission (bits/Joule delivered to the receivers). In particular, able to spli:
we focus on power splitting hybrid receivers which the received signals into two power streams for concurrent
information decoding and energy harvesting. Two scenarios are investigated considering different power splitting abilities of the receivers. In the first scenaric, the received power into a continuous arbitrary power splitting ratics. In the second scenario, we examine receivers which can split the received power only into a discrete set of power streams with fixed power splitting ratios. For both scenarics, formulate the corresponding algoritam design a non-convex optimization problem which takes into account the circuit power consumption, the minimum data rate requirements of delay constrained services, the minimum required system data rate, and the minimum amount of power that has be delivered to the receivers. By exploiting fractional programming and dual decomposition, suboptinal iterative resource allocation algorithms are developed to solve the non-convex problems. Simulaticn results illustrate that the proposed iterative resource allocation algorithms approach the optimal solution within a small number of iterations and unveil the trade-off between energy efficiency, system capacity, and wireless power transfer: (1) wireless power transfer enhances the system energy efficiency by harvesting energy in the radio frequency, especially in the interference limited regime; (2) the presence of rultiple
assume receivers which power streams with
split
receivers beneficial for the system capacity, but not necessarily for the system energy efficiency.
69

---

Published as a conference paper at ICLR 2026
Paper 1767525 title: Joint Transmit Beamforming and Receive Power Splitting for MISO SWIPT systems
Paper 1767525 abstract: This paper studies a multi-user multiple-input single-output (MIS0) downlink system for simultaneous wireless information and power transfer (SWIPT), in which set of single-antenna mobile statiens (MSs) receive information and energy simultaneously via power splitting (PS) from the signal sent by multi-antenna base station (BS) We aim to minimize the total transmissie ssion power at 88 by jeintly designing transmit beamforming vectors and receive PS ratios for all MSs under their given ratio (SINR) constraints for information decoding and harvested power constraints for energy harvesting. First, we derive the sufficient and necessary condition for the feasibility of our formulated problem. Next, solve this non-convex problem by applying the technique of semidefinite relaxation (SDR). We prove that
is indeed tight for our problem and thus achieves its global optimum, Finally, we propose two suboptimal solutions of lower complexity than the optimal solution based the principle of separating the optimization of transmit beamforming and receive PS, where the zero-forcing (2F) and the SINR-optima based transmit beamforming schemes are applied, respectively.
Paper 11665681 title: Power efficient and secure maltiuser communication systems with wireless informaticn and power transfer
Paper 11665681 abstract: In this paper, we study resource allocation algorithm design for power efficient secure communication with simultanecus wireless information and power transfer (WIPT) in focus power multiuser commuiccation systems. In particular, splitting receivers which are able to harvest energy and decode information from the received signals. The considered problem is modeled as an optimization problem which takes into account a minimum required
poiss ratio (STNR) at multiple desired receivers, maximum tclerable data rate rultiple multi-antenna potential eavesdroppers, and a minimum required power delivered the receivers. The proposed problem formulation facilitates the dual use of artificial noise in providing efficient energy transfer and quaranteeing secure communication. We aim at minimizing the total transmit power by jointly optimizing transnit beamforming vectors, power splitting ratios the desired receivers, and the covariance of the artificial noise. The resulting non-convex optimization problem is transformed into semidefinite programning (SDP) and solved by SDP relaxation. We show that the adopted SDP relaxation is tight and achieves the global optimum the original problem. Simulation results illustrate the significant power saving obtained by the proposed optimal algorithm compared to suboptima.. baseline schemes.
Paper 125571 title: Wireless Information and Power Transfer: Architecture
Design and Rate-Energy Tradecff
70

---

Published as a conference paper at ICLR 2026
Paper 125571 abstract: Simultanecus information and power transfer
the wireless channels potentially offers great convenience to mobile users. Yet practical receiver designs impose technical constraints on its hardware roalization, as practical circuits for harvesting energy from radio signals are not yet able to decode the carried information directly. make theoretical progress, operation, namely, dynamic power splitting (DPS), which splits the received signal with adjustable power ratio for energy harvesting and information decoding, separately. Three special cases of DPS, namely, time switching (IS), static power splitting (S2S) and on-off power splitting (OPS) are investigated. The and SPS scheres can be treated as special cases of OPS. Moreover, we propose two types of practical receiver architectures, namely, separated versus integrated information and energy receivers. The integrated receiver integrates the frent-end components smaller form factor. The rate-energy tradeoff for the two architectures are characterized by à so-called rate-energy (R-E) region. The optimal transmissien strategy is derived to achieve different rate-energy tradeoffs. With receiver circuit power consumption taken into account, it is shown that the CPS scheme is optimal for both receivers. For the ideal case when the receiver circtit does not consume power, the SPS scheme is optimal for both receivers. In addition, we study the performance for the two types of receivers under realistic system setup that employs practical moculation. Our results provide useful insights the optimal practical receiver design for simultaneous wireless information and power transfer (SWIPT)
propose general receiver
the separated receiver, thus achieving
Paper 3148780 title:
Training-Based SWIPT: Optimal Power Splitting the
Receiver Paper 3148780 abstract: We consider a point-to-point system with simultaneous wireless informaticn and power transfer (SWIPT) over a block-fading channel. Each transmission block consists of & training phase and data transmissien phase. Pilot symbols the receiver. To enable SWIPT, the receiver adopts a power-splitting design, such that portion of the received signal is used for channel estimation or data detection, while the rest is used for energy harvesting, We optimally design the power-splitting ratios for both training and data phases to achieve the best ergodic capacity performance while maintaining a required energy harvesting rate. Our result shows how poscr-splitting receiver can make the best use of the received pilot and data signals to obtain optimal SWIFT performance.
transmitted
during the training phase for channel estimation
Paper 7151441 title: Wireless Information and Power Transfer: A Dynamic
Power Splitting Approach
71

---

Published as a conference paper at ICLR 2026
Paper 7151441 abstract: Energy harvesting is promising solution to prolong the operation time of energy-constraine wireless networks. In particular, scavenging energy from ambient radio signals, namely wircless energy harvesting (WEE), has recently drawn significant consider point-to-peint wireless link
attention. In this paper, over the flat-fading channel, where the receiver has fixed power supplies and thus needs to replenish energy via WEH from the signals sent by the transmitter. We first consider a SISO (single-input single-cutput) system where the single-antenna zeceiver cannot decode information and harvest energy independently from the received. Under this practical constraint, we propose a dynamic power splitting (DPS) scheme, where the received signal is split into two staeams with adjustable power levels for information decoding and energy harvesting separately based on the instantaneous channel condition that is assumed optimal power splitting rule at the receiver to achieve various tradc-offs between the maximum ergodic capacity for information transfer and the maximum average harvested energy for power transfer, which are characterized by the boundary of a so-called "rate-cnergy (R-E) region. Moreover, for the case when the channel state information is also known at the transmitter, we investigate the joint optimization of transmitter power control and receiver power splitting. The achievable R-E region by the preposed DPS scheme is also compared against that by the existing time switching scheme well as a perfornance upper bound by ignoring the practical receiver constraint. Finally, we extend the result for optimal DPS to the SIMO (single-input multiple-output) system where the receiver is equipped with multiple antennas. In particular, we investigate a cowpcoxpyy power splitting scheme, namely antenna switching, which achieves the near-optimal rate-energy trade-offs as compared to the optimal DPS.
signa-
known at the receiver. We derive the
Paper 16191957 title: Wireless Information Transfer with Opportunistic
Energy Harvesting
Paper 16191957 abstract: knergy harvesting is promising solution prol ong the operation of energy-constrained wireless networks. In particular, scavenging energy from ambient radio signals, namely wireless energy harvesting (WEH), has recently drawn significant attention. In this paper, over the narrowband flat-fading channel subject to time-varying co-channel interference. It is assumed that the receiver has no fixed power supplies and thus needs to replenish energy ogportunistically via WEH from the unintended interference and/or the intended signal sent by the transmitter. that can only decode information or harvest energy at any time due to the practical circuit limitation. Therefore, it is important to investigate when the receiver should switch between the two modes of information ion decoding (ID) and energy harvesting (EH), based the instantaneous channel and interference condition. In this paper, we derive the optimal mode switching rule at the receiver to achieve various trade-offs between wireless information transfer and energy harvesting. Specifically, outage probability for delay-limited information transfer and the maximum ergodic capacity for no-delay-limited information transfer versus the maxinum average energy harvested at the receiver, which are characterized by the boundary of so-called "cutage-enerqy" region and "ratc-energy" region, respectively. Morcover, for the case when the channel state informaticn (CSI) is known at the transmitter, we investigate the joint optimization of transmit power control, information and energy transfer schedul ing, and the receiver's mode
consider point-to-point wireless link
further assume single-antenna receiver
determine the minimum transmission
switching: ng. The effects of circuit energy consumption at the receivee on the achievable rate-energy trade-offs are also characterized. Our
results provide useful guidelines for the efficient design of emerging wireless communication systems powered by opportunistic WEH.
72

---

Published as a conference paper at ICLR 2026
Respond with the Following json schema:
"$defs":
"Cell":
"cescription" "A Cell Object consists of paper ID, column name
and\nthe corresponding cell value that column in the - table.",
"properties";
"paper_id"" id":
"title": "Paper Id",
"type": "string"
@column_name""
"title": "Column Name", "type": "string"
b
"cell_value";
"title": "Cell Value", "type": "string"
requurre"; "paper_iae id",
"cell_value"
"title": "Cell", "type": "object"
), "description": "A Table Object is a List of Cell Objects.", ",
"properties": {
"cell_values"
"items":
"$ref": "#/Sdefs/Cel1"
"title": "Cell Values", "type": "array"
"required"; [ Scell_values
"title": "Table",
"type": "object"
H.5.2 TABLE UNROLLING PROMPT
You are helpful assistant that help infer useful information from tables comparing sets of scientific papers. You are given comparison table markdown format. Every row in the table contains information about scientific paper. Your goal is to rewrite the information conveyod by each cell in the table in the form of natural language statements. Each statement is atomic unit of information from table.
Follow the instructions given below to do so:
1. identify the column headers in the table.
2. Identify the various rows in the table.
3. For each row, go through every cell in that row (excluding the first
one that refers to paper ID) and write one atomic statement per cell.
4. Use the paper ID and information from column headers when writing these statements.
73

---

Published as a conference paper at ICLR 2026
5. Write all such statements in natural language (excluding icons/emojis) and output as numbered list.
6. Do not exclude any detail that is present the given table, or add extra details that are not present in the table.
7. Do not include any citation information in the statements.
Table:
[TABLE]
Statements:
H.5.3 EVALUATION PROMPT
Following is series of informative statements about set of ecientific research papers:
[UNROLLED_TARLL,
Given these statements, only state if the following ng statement is true, false or unknown.
Statement: [STATEMENT]
Answer:
H.6 SUPER-EXPERT
H.6.1 EXAMPLE PROBLEM
Run probability-basee prompt selection on the SST-2 dataset using opt-125m as the base model with the script 'run promppr www_prompt_seeection..p selection..y'. Report metrics.
Additional instructions:
If any details are missing from the task description, you are allowed to make arbitrary assumptions and decisions ful fill the task.
To requce execution time:
1. load/preprocess only the first 10 rows of each set in the dataset. 2. Only run a single epoch (when training). 3. Make sure you only run a single experiment, disabling any grid searchor ayperparameter tuning.
Git repository: http:://thhhh.c.c//////////////a selection
H.7 CORE-BENCH-HARE
H.7.1 EXAMPLE PROBLEM
The task input for the agent:
Task: codeccean_har hard
74

---

Published as a conference paper at ICLR 2026
Your goal is test the computaticnal reproducibility of the repository cloned to your current directory, which code from scientific paper. Specifically, you need to Run the manuscript. Rnd file using Rscript and render it as html. Put the results in the "../results" folder. Save your report tc file named report.json in the envirenment directory you started in that contains the capsule itself, where you fill in all of the following fields: dict_key (['Report the final outcones of reproducibility checks at the article level after original authors were contacted (Table 1 of the manuscript). You should report for row 1 in the table (ignore the confidence reproducibilice checks
interval) 'Report the final cutcomes
the article level after original authors were contacted (Table of the manuscript). You should repert n for row 2 in the table (ignore the confidence interval).', 'Report the final outcores of reproducibility checks at the article level after originsl authors were contacted (Table row 3 in the table (ignorc the confidence interval).', 'fig From Figure 1, report the proporticn of articles with fully reproducible target values from the random effects model after author contact. Ignore the confidence intervals']] You should install all of the requirements found in the Readme file and then run the commands necessary to answer the questicns.
the manuscript), You should report n for
The top-level contents of the corresponding capsule (with red items being omitted in the Hard version we use):
REPRODUCING.md
code/
LICENSE
README
config.json
lib.py
1ib2.py
lib2noDTW.py
librun.py
preprocess. py
run
run.ipynb
data/
LICENSE
ces:Preprocessed.pickkk
test:Removeeegiiiiss:
test@emooveeegiiiiiss__11111
testRemovveeeiinlass_555 20
Lestfemooveeegiinlass_220333
eesieemoveeeeiiinnsss__2....3
testRemmoveBBaginnlss__5
testRRemoveBeeginasss_
test_quicktest.
train
jraintrrajMMoeel
srain_quicktest
environment/
L Dockerfile
metadata/
metadata.yml
results/
expResult.pichle
75

---

Published as a conference paper at ICLR 2026
expResslt_noDDW.ppettl
output
out
output_noDIW. txt
run run.html
And the (abridged) content of the README. md file:
HурerÊтA
These are the program of the paper **януperčтй: Non-Deep-Learning Method for Estimated Time of Arrivals**.
Data
## train
Raw trajectories for train.
I triikeeq
## jModel jeedee.el.lccleeeee.....
The trajectories model, includes 3 tables
Hypercube series table Preprocessed trajectories.
Original trajectories table: Original GPS data.
Mapping table It map hypercubes to original trajectories.
H.8 DS-1000
H.8.1 EXAMPLE PROBLEM
Problem:
Given a 3d tenzer, say: batch sentence length x embedding dim
= torch. rand( (10, 1000, 96))
and an array (or tensor) of actual lengths for each sentence
Lengths torch randint (1000, (10,))
outputs tensor 370., 502., 652., 859., 545., 964., 566., 576., 1000., 803.1)
How to fill tensor 'a' with 2333 after certain index along dimension 1 (sentence length) according to tenser 'lengths'
want smth like that
al lengths I 2333
A:
<code>
import numpy as np
import pandas as pd
import torch
tandh (110, 1000, 96))
lengths = Barchhraadime (1000, (10,))
</code>
put solution in this variable BEGIN SOLUTION
76

---

Published as a conference paper at ICLR 2026
<oode>
Write the remaining python code to append to the progran above (but do
not repeat the part of the ccde that is already given in
"coode>... "<oode>...</onne <//</ode>"; just write the new code). Put your answer inside <code> and </code> tags.
H.9 DISCOVERYBENCH
H.9.1 EXAMPLE PROBLEM
Dataset path: nls als_bmi_
Dataset description: The dataset contains information from National Longitudinal Survey of Youth (NLSY79). It includes information about the Demographics, Family Background, Education, Health, Residential al, Financial Criminal Records of the participants.
Brief description of columns:
IDF (range 1-12686) 1979: Unique Identifier of the respondent, Sample ID, 1979 (interviex): Sample Identification Code, Age of respendent, 1979: Age of respondent in 1979,
Age of respondent at interview date, 1981: Age of respendent in 198:, Age of respendent Occupation of adult male in househeld at age 14, 1979: Occupation of the adult male present in the household of the respondent. at age 14 1979. Variable records the occupation of the father figure of the repondent, values include FARMER AND FARM MANAGERS, PROFESSIONAL, TECHNICAL AND KINDRED etc, Highest grade completed by respondent 's mother, 1979: Highest grade or year of regular school that respondent 's mother ever completed till 1979,
interview date, 1989: Age of respondent in 1989,
Highest grade completed by respendent's 's father, 1979: year of regular school that respondent father ever completed till 1979,
Highest grade or
Highest grade completed, 1979: Highest grade or year of regular school that respondent have completed and got credit for till 1979, Racial/ethnic cohort, 1979: Respondent racial/ethnic cohort, contains one of three values 1:BLACK, 2:HISPANIC, 3:NON-BLACK NON-HISPANIC, Sex of respendent, 1979: Sex of the respondent, 1:MALE 2:FEMALE, Family size, 1979: Family size of the respondent Ever convicted of an illegal act variable that indicates if the respondent was convicted of an illegal act in adult court other than minor traffic violations before 1980, Ever been sentenced in any correcticnal institution before. 1980: Boolean
1979,
adult court before 1980: Boolean
variable that cated if the respondent was sentenced spend time in corrections institute, like jail, prison, or youth institution like a training school or reform school or not before 1980,
Height of respondent, 1981: Heiggh of the respondent in inches in 1981,
Reight of respondent, 1985: Height of the respondent in inches in 1985,
Weight of respondent, 1981: Weight of the respondent kilograms in
1981,
Weight of respondent, 1989: Weight of the respondent in kilograms in
1989,
Weight of respondent, 1992: Weight of the respondent in ki kolorrmss in 1992,
Rank in class last year attended at this school, 1981: Respondent rank
in the class that he attended in school last year (in 1980) (variable
recorded in 1981),
Number of students in class last year attended at this school, 1981: Number of students in the respendent 's class for the last year attended this school,
77

---

Published as a conference paper at ICLR 2026
ASVAB Arithmetic Reasoning Score (rounded), 1981: This variable
represents the standardized sccres of respondents on the Arithmetic Reasoning section of the ASVAB test. It provides way to compare individuals' performance on this specific aspect of the test within standardized Framework.,
ASVAB Word Knowledge Score (rcunded), 1981: This variable represents the standardized scores of respendents on the Word Knowledge section of the ASVAB test, allowing for comparison of individuals' performance on this speci fic aspect of the test within a standard: zed framework.,
ASVAB Paragraph Comprehensien z Score (rounded), , 1981: This variable represents the standardized sccres of respondents on the Paragraph Comprehension section of the ASVAB test, allowing for comparison of individuals' performance on this specific aspect of the test within standardized framework.,
ASVAB Mathematics Knowledge Score (rounded), 1981: This variable represents the standardized sccres of respondents on the Mathematics Knowledge section of the ASVAB test, facilitating comparison of individuals' performance on this specific aspect of the test within standardized framework.,
Type of residence respondent is living in, 1981: Type of residence respondent is living in the 1981, contains one of these values 2:BACHELOR, OFFICER QUARTERS, 5:JAIL, 6:0THER TEMPORARY
1:ABCARD SEIP, BARRACKS, 4:HOSPITAL, 12:ON-BASE MIL FAM HOUSING, FRATERNITY, SORORITY,
3:DORM,
QUARTERS, 11:0WN DWELLING UNIT,
13:CFF-BASE MIL FAM HOUSING, 14:ORPHANAGE,
15:RELIGIOUS 17:PARENTAL,
INSTITUTION,
16:0THER INDIVIDUAL QUARTERS, 18:HAI CONDUCTED WITH PARENT,
19:R IN PARENTAL HOUSEHOLD, Type of residence respondent is living in, 1982: Type of residence respendent is living in the 1982, contains 2:BACHELOR, OFFICER QUARTERS, 5:JAIT, 6:0THER TEMPORARY 4:HOSPITAL, 12:ON-BASR MTL FAM HOUSTNN ING, 15:RELIGIOUS 17:PARENTAL,
of these values
1:ABOARD SHIP, BARRACKS,
3:DORM,
FRATERNITY, SORORITY,
QUARTERS, 11:0WN DWELLING UNIT,
13:CFF-BASE MIL FAM HOUSING,
14:0RPHANAGE,
INSTITUTION,
16:0THER INDIVIDUAL QUARTERS, 18:HHI CONDUCTED WITH PARENT,
19:R IN PARENTAL HOUSEHOLD, Type of residence respondent is living in, 1983: Type of residence respondent is living in the 1983, contains one of these values 2:BACHELOR, OFFICER QUARTERS, 5:JAIL,
1:ABCARD SHIP, BARRACKS, FRATERNITY, SORORITY,
3:DORM,
4:HOSPITAL, 12:ON-BASZ MIL FAM HOUSING,
6:CTHER TEMPORARY
QUARTERS, 11:0WN DWELLING UNIT, 13:CEF-BASE MIL FAM HOUSING, INSTITUTION,
HANNAG,,
15:RELIGIOUS 17:PARENTAL,
16:0THER INDIVIDUAL QUARTERS, 18:HHT CONDUCTED WITH PARENT,
19:R IN PARENTAL HOUSEHOLD, Type of residence respondent is living in, 1984: Type of residence respondent is living in the 1984, contains one of these values 2:BACHELOR, OFFICER QUARTERS, 5:JAIL, 6:OTHER TEMPORARY
1:ABCARD SEIP, BARRACKS,
3:DORM,
PRATERNITY, SORORITY,
4:HOSPITAL, 12:ON-BASE MIL FAM HOUSING,
QUARTERS, 11:0WN DWELLING UNIT, 14:ORPHANAGE,
13:CFF-BASE FAM HOUSING,
15:RELIGIOUS 17:PARENTAL,
INSTITUTION, 16:0THER INDIVIDUAL QUARTERS,
19:R IN PARENTAL HOUSEHOLD, Type of residence respondent is living in, 1985: Typc of residence respendent is living in the 1985, contains 2:BACHELOR, OFFICER QUARTERS, 5:JAIL, 6:OTHER TEMPORARY QUARTERS, 11:0WN DWELLING UNIT, 12:ON-BA8R. MIL FAM HOUSTNG 13:CFF-BASE MIL FAM HOUSING, 14:0RPHANAGE, 15:RELIGIOUS INSTITUTION, 16:0THER INDIVIDUAL QUARTERS, 17:PARENTAL, 18:HHI CONDUCTED WITH PARENT,
18:HHI CONDUCTED WITH PARENT,
of these values
L:ABCARD SHIP, BARRACKS,
3:DORM,
FRATERNITY, SORORITY,
4:HOSPITAL,
19:R IN PARENTAL HOUSEHOLD,
78

---

Published as a conference paper at ICLR 2026
Type of residence respondent is living in, 1986: Type of residence respondent is living in the 1986, contains one of these values
2:BACHELOR, OFFICER QUARTERS, 5:JAIL, 6:CTHER TEMPORARY 1:ABCARD SHIP, BARRACKS, A:HOSPITAL, 12:ON-BASE MIL FAM HOUSING, FRATERNITY, SORORITY, QUARTERS, 11:0WN DWELLING UNIT, 13:OFF-BASE MIL FAM HOUSING, 14:ORPHANAGE,
3:DORM,
15:RELIGIOUS 17:PARENTAL,
16:0THER INDIVIDUAL QUARTERS, 18:HHI CONDUCTED WITH PARENT,
INSTITUTION,
19:R IN PARENTAL HOUSEHOLD, Type of resi residence dence respondent is living in, 1987: Type of residence respondent is living in the 1987, contains one of these va values
1:ABOARD SHIP, BARRACKS,
2:HACHBLOR, OFFICER QUARTERS, S:JAIL,
3:DORM,
FRATERNITY, SORORITY,
4:HOSPITAL, QUARTERS, 11:0WN DWELLING UNIT, 12:0N-BASE MIL FAM HOUSING, 14:0RPHANAGE,
6:CTHER TEMPORARY
16:0THER INDIVIDUAL QUARTERS, 15:RELIGIOUS 18:HHI CONDUCTED WITH PARENT,
13:CFF-BASE MIL FAM HOUSING,
INSTITUTION,
17:PARENTAL,
19:8 IN PARENTAL HOUSEHOLD, Type of residence respondent is living in, 1988: Type of residence respondent is living in the 1988, contains one of these va values 2:BACHELOR, OFFICER QUARTERS,
1:ABCARD SHIP, BARRACKS,
3:DORM,
FRATERNITY, SORORITY,
4:HOSPITAL, QUARTERS, 11:0WN DWELLING UNIT, 12:0N-BASE MIL FAM HOUSING, 14:ORPANNAGK,
5:JAIL,
6:CTHER TEMPORARY
13:OFF-RASH MIL. RAM HOUSING,
15: OUS
16:0THER INDIVIDUAL QUARTERS, 18:HHI CONDUCTED WITH PARENT,
INSTITUTION,
17:PARENTAL,
19:8 IN PARENTAL HOUSKHOLD, Type of residence respondent is living in, 1989: Type of residence respondent is living in the 1989, contains one of these values 2:BACHELOR, OFFICER QUARTERS, 5:JAIL, 6:CTHER TEMPORARY 4:HOSPITAL, 12:0N-BASE MIL FAM HOUSING, QUARTERS, 11:0WN DWELLING UNIT, 13:OFF-BASE MIL FAM HOUSING, 14:ORPHANAGE, 15:RELIGIOUS INSTITUTION,
1:ABCARD SHIP, BARRACKS,
3:DORM,
FRATERNITY, SORORITY,
16:0THER INDIVIDUAL QUARTERS, 18:HHI CONDUCTED WITH PARENT,
17:PARENTAL,
19:R IN PARENTAL HOUSEHOLD, Type of resi dence respondent is living in, 1990: Type of residence respondent is living in the 1990, contains one of these values
1:AHOARD SHIP, BARRACKS, FRATERNITY, SORORITY,
2:BACHNIOOR, OFFICER QUARTERS, 5:JAIL,
3:DORM,
4:HOSPITAL, QUARTERS, 11:0WN DWELLING UNIT, 12:ON-BASE MIL FAM HOUSING, 15:RELIGIOUS 17:PARENTAL,
6:CTHER TEMPORARY
13:CFF-BASE MIL FAM HOUSING, 14:0RPHANAGE, INSTITUTION,
16:0THER INDIVIDUAL QUARTERS, 18:HHI CONDUCTED WITH PARENT,
19:8 IN PARENTAL HOUSEHOLD, Type of residence respondent living in, 1991: Type of residence respondent is living in the 1991, contains one of these values 2:BACHELOR, OFFICER QUARTERS, S:JAIL,
1:ABCARD SHIP, BARRACKS,
3:DORM,
FRATERNITY, SORORITY,
4:HOSPITAL, 12:0N-BASZ MIL FAM HOUSING,
6:CTHER TEMPORARY
QUARTERS, 11:0WN DWELLING UNIT, 13:OFF-BASH MIL FAM HOUSING,
15: OUS 17:PARENTAL, 19:R IN PARENTAL HOUSKHOLD, Type of residence respondent is living in, 1992: Type of residence respondent is living in the 1992, contains one of these values 2:BACHELOR, OFFICER QUARTERS, 5:JAIL, 6:CTHER TEMPORARY 4:HOSPITAL, 12:ON-BASE MIL FAM HOUSING, QUARTERS, 11:0WN DWELLING UNIT,
14:ORPHANAGE,
INSTITUTION,
16:0THER INDIVIDUAL QUARTERS,
18:HHI CONDUCTED WITH PARENT,
4 1:ABCARD SHIP, BARRACKS, FRATERNITY, SORORITY,
DOR,
15:RELIGIOUS 17:PARENTAL, 19:R IN PARENTAL HOUSEHOLD, Type of desidence respondent is living in, 1993: Type of residence respondent is living in the 1993, contains one of these va values
13:OFF-BASE MIL FAM HOUSING, 14:ORPAANAGE, INSTITUTION,
16:0THER INDIVIDUAL QUARTERS, 18:HHI CONDUCTED WITH PARENT,
1:ABOARD SHIP, BARRACKS, FRATERNITY, SORORITY,
2:HACHNLORR, OFFICER QUARTERS, SIJAIL,
3:DORM,
6:CTHER TEMPORARY 4:HOSPITAL, 12:0N-BASE MIL FAM HOUSING, 15:RELIGIOUS
QUARTERS, 11:0WN OWELLING UNIT, 13:CFF-BASE MIL FAM HOUSING, 14:0RPHANAGE, INSTITUTION, 16:0THER INDIVIDUAL QUARTERS, 17:PARENTAL, 18:HHI CONDUCTED WITH PARENT, 19:R IN PARENTAL HOUSEHOLD,
79

---

Published as a conference paper at ICLR 2026
Type of residence respondent is living in, 1994: Type of residence respondent is living in the 1994, contains one of these values
2:BACHELOR, OFFICER QUARTERS, 5:JAIL, 6:CTHER TEMPORARY 1:ABCARD SHIP, BARRACKS, A:HOSPITAL, 12:ON-BASE MIL FAM HOUSING, FRATERNITY, SORORITY, QUARTERS, 11:0WN DWELLING UNIT, 13:OFF-BASE MIL FAM HOUSING, 14:ORPHANAGE, INSTITUTION,
3:DORM,
15:RELIGIOUS 17:PARENTAL,
16:0THER INDIVIDUAL QUARTERS, 18:HiI CONDUCTED WITH PARENT,
19:R IN PARENTAL HOUSEHOLD, Type of resi dence respondent is living in, 1996: Type of residence respondent is living in the 1996, contains one of these va values
1:ABOARD SHIP, BARRACKS,
2:HACHBLOR, OFFICER QUARTERS, SIJAIL,
3:DORM,
FRATERNITY, SORORITY,
6:CTHER TEMPORARY QUARTERS, 11:0WN DWELLING UNIT, 12:ON-BASE MIL FAM HOUSING, 13:CFF-BASE MIL FAM HOUSING, 14: INSTITUTION,
4:HOSPITAL,
16:0THER INDIVIDUAL QUARTERS, 15:RELIGIOUS 18:HHI CONDUCTED WITH PARENT,
17:PARENTAL, 19:8 IN PARENTAL HOUSEHOLD, Family net wealth, 1985: Total Net Wealth for Family. Created by sunming 4 all asset values and subtracting all debts for the year 1985, Family net wealth, 1990: Total Net Wealth for Family. Created by surming
all asset values and subtracting all debts for the year 1990, Family net wealth, 1996 (key data peint): Total Net Wealth for Family.
- Created by summing all asset values and subtracting all debts for the year 1996,
Market value of residential property respondent/spouse own, 1985: Market value of residential property that respondent/spouse owned in 1985, Market value of residential property respondent/spoosee own, 1990: Market value residential property that respondent/apouse owned in 1990, Market value of residential property respondent/spouse own, 1996: Market value of residential property that respondent/spouse owned in 1996, Total market value of farm, business, and other preperty, 1985: Total market value of all of the real estate, assets in the business (es), 1985,
farm operation (8)
Total market value of farm, business, and other property, 1990: Total market value of all of the real estate, assets in the business (es),
farm operation (s) in 1990,
Total market value of farm, business, and other property, 1996: Total market value of all of the real estate, assets in the business (es),
farm operation (s) 1996,
Market Value of vehicles respondent/apouse own, 1985: Iotal market value of all vehicles including autemobiles that respondent/spoust owned in 1985,
Market Value of vehicles respondent/spouse own, 1990: Total market value of all vehicles including autembblees that respondent/spouse owned in 1990,
Market Value of vehicles respondent/spout own, 96: Total narket value of
all vehicles including automebiles that respondent/spouse owned in 1996,
Total market value of items over $500, 1985: Total market value all the other assets of the respendent that were worth more than $500 in 1985,
Total market value of items over $500, 1990:
Total market value of all the other assets of the respendent that were worth 1990,
than $500 in
Total market value of items over $500, 1996: Total market value of all the other assets of the respondent that were worth more than $500 in 1996,
Total net family income, previous calendar year, 1979: Total net family income for the previous calendar year (1978) (recorded in 1979),
Total net family income, previous calendar year, 1985: Total net family income for the provious calendar year (1984) (recorded in 1985), Total net family income, previcus calendar year, 1989: Total net family income for the previous calendar year (1989) (recorded in 1989),
80

---

Published as a conference paper at ICLR 2026
Was more money put into or taken R/spouse savings since last interview, 1989: Categorical variable indicating was more money was put into or taken out of seapodenn//ppous savings since last intervicw in 1989.
It contains four values 1:PUT MORE MONEY IN, 2:T00X MORE MONEY OUT, 3:NO CHANGE, 4:NO SAVINGS,
Net amount respondent/ spouse put into ags si since 1ast interview, 1989: put into their savings
Net amount of money that respondent/spone since 1ast interview in 1989,
Net amount respondent/ppouse tock cut of savings since last interview,
1989; Net amount of money that respondent/apouse took out of savings since last interview 1989,
Query: Docs increased time preference leads to higher BKI?
In the final answer, please cutput a ison containing two keys:
'hyptthesss'; SCIENTIFIC HYPOTHESIS,
'workflow'; WORKFLOW SUMMARY
where
the SCIENTIFIC HYPOTHESIS is natural language hypothesis, derived from the provided dataset, clearly stating the context of hypethesis (if any), , variables chosen (if any) and relationship between those variables (if any) including any statistical
significance, Please include all numeric information as necessary to support the hypothesis.
and
the WORKFLOW SUMMARY a summary of the full workflow starting from data loading that led to the final hypothesis.
Make you Load the dataset to analyze it deter to agent that can)
H. E2E-BENCE
1.10.1 EXAMPLE PROBLEM
You are an autonomous agent, tasked to perform the follewing research task:
**TASK :
Name** simple-daag-mmlammment
Short Des:ription*:: Enhancing the static DAG-ERC model with simple content-based edge selection fer improved emotion recognition in conversations.
++ong Descriptione+: This research explores simplified enhancement to the static DAG construction in the DAG-ERC model by implementing basic content-aware edge selection mechanism. Rather than developing ng fully dynamic DAG construction approach, we focus augmenting the existing static DAG with a small number of additional edges based on simple content similarity metrics between utterances. This approach maintains the core structure of the original DAG-ERC model while potentially capturing additicnal relevant connections that may improve cmotion recognition performance.
81

---

Published as a conference paper at ICLR 2026
Hypothesis explore**: Augmenting the static DAG structure with small number of additional edges based on content similarity between utterances will improve emoticn recognition performance compared to the original static DAG-ERC medel, particularly for conversations where important contextual relationships apan beyond the immediate dialogue history.
Metric to use; The primary metrics will be we weighted-averane P1 score and micro-averaged #1 score (excluding the majority class) for emoti on recognition, consistent with the original DAG-ERC paper. We will also analyze the number and distribution of additional edges understand the impact of our enhancement.
B=Bseeines:*: We will compare our enhanced DAG-EN ERC against: (1) the original DAG-ERC with static rules, and (2) fully-cennected graph baseline where all utterances are connected to all previous utterances (up to fixed windew size).
Research Idea Variables=s: Independent. variables include the DAG construction method (original static DAG, our ennanced DAG with content-based edges), the similarity threshold for adding edges, and the maximum number of additicnal edges per utterance. Control variables include the feature extraction method, the emotion recognition model architecture, and the evaluation metrics. The dependent variable is the emetien recognition performance.
Research Idea Design**: Implement simple enhancement to the static DAG construction in the DAC-ERC model by adding content-based edges between utterances. The goal to capture additicnal relevant connections that may improve emction recognition performance while maintaining the simplicity and efficiency of the original model.
*+1. Data Preparationx::
Use the IEMOCAP dataset, follewing the preprocessing steps in the original DAG-ERC paper.
Extract a small subset (e.g., 20 conversations) for the pilot study. +*2. Enhanced DAG Constructiones.
Start with the static DAG constructed using the original rules from the DAG-ERC paper (based speaker identity and positional relations). content similarity with all previous utterances (within a reasonable window, e.g., 10 utterances) using simple metric such as cosine similarity between Roberta embeddings.
For each utterance, compute
Add accitienal edges from previcus utterances to the current utterance
if their similarity exceeds a threshold (e.g., 0.8) and they are not static DAG.
already connected
Limit the number of additional edges per utterance (e.g., maximum 3) maintain sparsity.
++3, Implementation Details**:
Use RoBERTa-Base the feature extractor for both the emotion recognition ii model and the similarity computation.
Implement the enhanced DAG construction as preprocessing step before training the emotion recogniticn model.
Experiment with different similarity thresholds (e.g., 0.7, 0.8, 0.9) and maximum number of additional edges (e.g., 1, 3, 5)
Use the original DAG-ERC model architecture without modifications for the emotion recognition task.
*+4. Training and Evaluation
Train the model on the TEMOCAP dataset. using the enhanced DAG ructure.
Compare the performance with the criginal DAG-ERC model and the fully-cennected baseline.
Analyze the number and distribution of additional edges added by the enhancement.
82

---

Published as a conference paper at ICLR 2026
Identify specific examples where the enhanced DAG leads to correct predictions that were incorrect with the original DAG. **5. Output and Analysis**:
Save the trained models and their performance metrica.
Generate visualizations of the original and enhanced DAG structures for few example conversations.
Analyze relationship between the number of additional edges and the emot inn recognition performance.
investigate which types of conversations benefit most from the enhanced DAG structure.
For the pilet experiment, implement the enhanced DAG construction approach on 20 conversations from the IEMOCAP dataset to validate the approach before scaling to the full experiment. Focus on a single similarity threshold (e.g., 0.8) and single maximur number of additicnal edges (e.g., 3) for simplicity. end of task definition
Please perform this task and produce four results:
report, describing the results your research. The report should + include, among other things, the following parts: Title, Abstract, + Inductcii Approach, Experiments, Results, Conclusion, References.
2. The code you wrote perform the research.
trace/ log of your research, The trace should give step-by-step
4 description of the actions the agent (you) took, e.g., searching the
literature, writing and executing code, analyzing results. The trace should alse include the results of those actions, e.g., the papers found, the experimental results from code execution, etc.
Any other research artifacts (datasets, analyses, results, etc.) that + you generated, to substantiate your report. If these artifacts (e.g.r a dataset) are large, only shew part of them but enough to convey
their contents.
These results will be used to assess how well you performed the task.
Return your ansver the follewing JSCN structure (a dictionary containing a single top-level key, "results", which is dictionary containing the keys 'report', "code", "trace", and "artifacts", in exactly the format described below):
"results";
"report" (str): <report>, "code" " (list):
["filename" (str): <fi <filenamel>, "code" (str): <codel>),
['filename" (str): <filename2>, "code" (str) <code2>),
"trace" (str): <trace>, arttfacts" (list):
("filename" (str): <filenamel>, "artifact" <artifact>>/ ["filename" (str): <filename2>, "artifact" (str): <artifact2>),
where <report> is multiline string that contains the report, <trace> is multiline string that contains trace (or summary of the trace) of the agent's behavior while sclving the task, and the artifacts products the research (created datasets, etc.)
83

---

Published as a conference paper at ICLR 2026
H.11 E2E-BExCE-HARD
H.1111 EXAMPLE PROBLEM
You are autonemous agent, tasked perform the following research
task:
TASK DEFINITION:
Name:
Adaptive Reasoning Enhancement
Short Description: Combining Complexity-Based Prompting and Imitation Demonstration Learning improve language models' generalization unseen tasks.
Hypothesis to explore: Integrating Complexity-Based Prompting with Imitation Demonstration Learning will enhance the generalization capabilities of language models, resulting in improved performance unseen tasks by dynamically adapting reasoning complexity and remonstration selection.
Key Variables:
Indepencent variable: Integration of Complexity-Based Prompting with Imitation Demonstration Learning
Dependent variable: Generalization capabilities language models on unseen tasks
Comparisen groups: Four conditiens: Baseline (standard prompting), CBP-only, IDL-only, and Integrated (CBP+IDL)
Baseline/centrol: Standard prompting without CBP or IDL
Context/setting: Complex multi-stan reasoning problems
Assumptions: Complexity- Prempting enhances reasoning by focusing hign-complexity rationales, while Imitation Demonstration Learning reinforces Learning through imitation
Relationship type: Causal (integration 'will enhance' capabilities)
Population: Language models
Timeframe: Net specified
Measurement method: Primary metric: Accuracy on unseen tasks; Secondary metrics: Reasoning complexity, demonstration effectivenese and response quality
84

---

Published as a conference paper at ICLR 2026
Long Description: Description: The research explores the integration of Complexity-Based Prompting and Imitation Demonstration Learning enhance the generalization capabilities of language models on unseen tasks. Comploxity-Based Prompting involves selecting prompts based on reasoning complexity, guiding the model through intricate reasoning chains. Imitation Demonstration Learning strengthens the learning process by mimicking human review strategies, selecting similar examples for new questions and re-answering based on retrieved examples. The hypothesis posits that combining these nethods will allow the model to dynamically adapt its reasoning complexity and demonstration selection, leading improved performance on unseen tasks. This approach addresses the gap in existing research by offering a novel combination of methods to enhance model adaptability and reasoning capabilities. The expected outcome is that the model will perform better unseen tasks by leveraging complex reasoning chains and effective demonstration selection. This research significant as it provides a new perspective on cnhancing language moccls' reasoning abilitics, petentially leading to nore robust and adaptable AI systems.
Key Variables: [Complexity- Prompt-
ingl (https://www www.stiisccntnttooee Complexity-Based Prompting invelves selecting prompts based on the complexity of reasoning steps. This method enhances model performance on tasks requiring deep reascning by focusing on high-complexity rationales. It involves conducting a voting process among different reasoning paths to determine the most complex and informative one. The prompts guide the model through these complex reasoning chains, ensuring effective handling of intricate tasks. This variable is critical as it directly influences the model ability to process complex reasoning tasks, improving its generalization capabilities.
Imitation Demonstration Learn-
ingl Imitation Demonstration Learning strengthens the learning process by mimicking human review strategies. It involves selecting the most similar example to new questien and re-answering according to the answering steps of the retrieved example. This approach emphasizes interactions between prompts and demonstrations, reinforcing learning through explicit imitation. It requires a mechanism to select similar examples and re-answer questicns, improving the model's ability to
learn from demonstrations. This variable is essential as it enhances the model's ability to generalize from demonstrations by consolidating known knowledge through imitation.
85

---

Published as a conference paper at ICLR 2026
Research Idea Design: The hypothesis will be implemented using the ASD Agent's capabilities by integrating Complexity-Based Prompting Imitation Demonstration Learning. The process begins with defining a set of tasks that require complex reasoning. Comploxity-Based Prompting will be applied by designing prompts that include high-complexity reasoning chains. These prompts will guide the nodel through intricate reasoning steps, ensuring effective handling complex tasks. Imitation Demonstration Learning will be implemented by developing mechanism to select similar examples for new questicns. This involves creating system that identifies similar examples based on semantic similarity and uses them re-answer questicns, reinforcing the learning process. The integration of these methods will occur at the prempt level, where the complexity-based prompts will be combined with imitation demonstration strategies to enhance the model's reasoning capabilities. The data flow will involve feeding the model with complexity-based prompts and using the imitation demonstration mcchanism to select and re-answer questions. The expected outcome is that the model will perform better on unseon tasks by leveraging complex reasoning chains and effective demonstration selection. This approach is novel it combines distinct methods to enhance language models' reascning abilities, providing new perspect ive en improving AI systems' adaptability and performance.
Evaluation Procedure: Please implement an experiment test the hypothesis that integrating Complexity-Based Prompting (CBP) with Imitaticn Demonstration Learning (IDL) will enhance language models' generalization capabilities en unseen reasoning tasks. The experiment should compare four conditicns:
1. Raseline: Standard prompting without CBP or IDL
2. CBP-only: Using only Complexity-Pasee Prompting
3. IDI-only: Using only Imitation Demonstration Learning
4. Integrated (CEP+IDL): The experimental condition combining both approaches
The experiment should include the follewing components:
## Dataset
Use reascning task dataset such as 2WikiMultiHopon that includes complex multi-step reasoning problems. The dataset should be split into training (60%), validation (20%), , and test (208) The test set will represent 'unseen tasks' for final evaluation.
" Pilot Mode Implementation
Implement global variable PILOT_MODE MODE with three possible settings: 'MINI_PILOT" 'PILOT', or 'FULL EXPERIMENT',
MINI_PILOT: Use questions from the training questions from the validation set for evaluation.
for development and
PILOT: Use 100 questions from the training set for development and 50 questions ins from the validation set for evaluate latioon.
FULL EXPERIMENT Use the entire training set for development and the entire test set for final evaluation.
Start with MINI PILOT, then proceed to PILOT successful, not run EXPERIMEN without human verification of the PILCT results.
## Complexity-Basee Prompting Module
Imp lement module that:
Generates multiple reasoning paths for each question the training set
2. implements voting mechanism to determine the most complex and informative reasoning path
86

---

Published as a conference paper at ICLR 2026
3. Creates prompts that guide the medel through these complex reasoning chains
4.
Stores these complexity-based prempts for later use
#* Imitaticn Demonstration Learning System
Implement systen that:
1. Creates database of question-ansver pairs with detailed reasoning
steps from the training set
2. For new questions, calculates semantic similarity to find the most
4 similar examples in the database
3. Retrieves the most similar examples and their reasoning steps
4. Constructs prompts that include these examples to guide the model in answering
questions
## Integrated Approach (CBF+IDL)
Impl ement the integration of CBP and IDI. by:
1. Using CEP to generate complex reasoning chains for the questions
2. Using IDL to select similar examples with their reasoning steps
3. Combining both in unified prompt that includes both the complex reascning guidance and the similar examples
4. Implementing an adaptive mechanism that adjusts the weight given CBP vs. IDL based on questien characteristics
# Rvaluation
Evaluate all four conditions using:
1. Primary metric: Accuracy unseen tasks (percentage of correctlyy answered questions)
2. Secondary metrics:
Reasoning complexity (average number of reasoning steps in responses) Demonstration effectiveness (semantic similarity between selected examples and target questiens)
Response quality (coherence, relevance, and logicality of reasoning), use ROSCOE only if applicable
# Statistical Analysis
Perform statistical analysis to determine if differences between conditions are significant:
1. Conduct paired t-tests between conditions
2. Calculate effect sizes (Cohen's for each comparison
3. Perform bootstrap resampling to establish confidence intervals
## Logging and Reporting
Impl Implement comprehensive logging that captures:
1. All prompts generated for each condition
2. Model responses for each questien
3. Evaluation metrics for each condition
4. statistical analysis results
5. Examples of successful and unsuccessful cases
The final report should include:
1. Summary of results for each condition
2. Statistical significance ficance of differences between conditions
3. Analysis of when and why the integrated approach performs better worse
4. Recommendations for further improvements
## Implementation Details
Use NLTK for text processing and tokenization
Use scikit-learn for semantic similarity cal calculations and statistical
analysis
Use language model (e.g., GPT-4) for generating responses
Implement proper error handling and logging throughout
Please run the experiment in MINI_PILOT mode first, then PILCT mode if successful. Do not proceed te EXPERIMET without human verification.
87

---

Published as a conference paper at ICLR 2026
end of task definition
Now: Please perform this task and produce four results:
1. A report, describing the results of your research. The report should include, among other things, the following parts: Title, Abstract, introduction, Approach, experiments, Results, Conclusion, 4 References.
2. The code you wrote to perform the research.
3. A trace/log of your research. The trace should give a step
description of the actions the agent (you) took, e.g., searching the
literature, writing and executing code, analyzing results. The trace results those actions, c.g., the papers found, the experimental results from code execution, etc.
should alse include
4. Any other research artifacts (datasets, analyses, results, etc.) that you generated, to substantiate your report. If these arti facts (e.g.r large, only shew part of them but enough to convey 4 their contents.
cataset)
These results will be used to assess how well you performed the task.
Return your answer in the following JSON structure (a dictionary containing single top-level key, 'results", which is dictionary containing the keys "report', "code", 'trace', and "artifacts', in exactly the format described below):
"results":
"repert (str): <repert>, "code" " (list):
["filename" (str): <filenamel>, "code" (str): <codel>), ("filename" (str): <fileneme2>, "code" (str): <code2>),
l, "trace" (str): <trace>, "artifact" (str):
("fi FiFamee (str): <filenamel>, Martifacts (str): <arti
['filename" (str): <filename2>, "artifact" (str): <artifact>>),
where <report> is multiline string that contains the report, <trace> is multiline string that contains trace summary of the trace) the agent's behavior while solving the task, and the artifacts are proaucts of the research (created datasets, etc.)
88


Reviews:
[{"review_id": "WSYoOI2jLs", "reviewer": "Reviewer_zxpV", "summary": "This paper puts forward outstanding issues in the evaluation practices for AI science-specific agents. They introduce AstaBench, an evaluation suite that provides a holistic measure of agentic ability to perform scientific research.", "questions": "Why are non-standard tools a confounding variable if we don't focus on comparing models but agent performance?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper does identify open issues in agent evaluations and provides a thorough analysis of the field. The paper provides a holistic evaluation of various models and scaffolds. \n- The set of benchmarks included in the analysis is broad and offers good coverage of the AI science benchmark domain\n- The writing is clear and overall easy to parse.", "weaknesses": "- Treating tools as confounding variables only makes sense when we want to collapse agent evals to model evals. For measuring how well agents with the latest model-backbones can do on a task, we should compile each model with the best-possible scaffold around it. This often involves using the tools the agentic models was e.g. trained with natively (web search, code interpreter, etc.)\n- Other confounders aside from standard tools have been motivated and described in previous work [1]\n- The Pareto efficiency frontier should be convex because every point on the convex envelope can be achieved by mixing between two agents. \n\n[1] https://arxiv.org/abs/2407.01502", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s18iLG1qao", "reviewer": "Reviewer_zo8M", "summary": "Paper features an AstaBench - benchmark for evaluating scientific LLM agents on the variety of tasks. It integrates multiple benchmark for each task and consists not only with raw data, but with tools essential for testing. Benchmark can be used for testing agents on individual tasks (e.g. literature review or data analysis) as well as on end-to-end research.", "questions": "- Could you please provide at least brief description of scoring metrics in the main part of the paper? It's an important part of the benchmark, yet it's currently hidden in the appendix.\n- It probably would be better to replace \"AI\" with \"LLM\" in title and abstract as benchmark made mainly for LLM agents, not for any AI agents (which also may refer to RL and other areas).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- AstaBench provides a set of tools for testing agents, which makes it easier to setup experiments.\n- Benchmark can be used for testing agents for specific tasks as well as for end-to-end research tests.\n- AstaBench have a potential to become strong and stable benchmark for testing scientific LLM agents.", "weaknesses": "- Currently benchmark is heavily weighted for computer science and machine learning domains", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sF5T5I5qXd", "reviewer": "Reviewer_LRec", "summary": "The paper introduces AstaBench, a suite of benchmarks for evaluation AI agents' potential for conducting scientific tasks. The suite aggregates scientific tasks and the authors run a comprehensive evaluation of different models on these tasks with different agent scaffolds. The suite controls for several factors, such as which tools are used, and provides a set of agent baselines that the authors run these evaluations on.\n\nThe paper's contributions to AI for science evaluations are impressive. Evaluating agents is a challenging problem, and the authors make a serious and compelling effort in that direction. I recommend acceptance, and offer suggestions for further improving the paper below.", "questions": "Overall, I think the paper is a great contribution and would be a strong fit for ICLR. I am happy to recommend acceptance. Moreover, I would raise my score further and recommend a spotlight or oral ***if the following points are addressed***:\n- Please make sure the text of the paper matches the contributions of the paper. The results in the paper speak for themselves; there is no need to claim this is the \"first\" suite to do many of these things (in particular with the comparisons against past agent frameworks)\n- Please clarify how you selected the agents you picked for the analysis. What other agents did you consider? Why did you select this subset?\n- Please update the text to clarify the contributions based on my concerns in the weaknesses section (differences in the number of tasks for benchmarks, aggregation of results across benchmarks, and more generally updating the clarity of the writing and presentation).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The authors calculate the cost for all evaluations, which is key for reliable benchmarking. Most existing frameworks don't do this (as the authors correctly identify).\n- The scope of benchmarks is broad, including agents from many stages of the scientific research pipeline\n- The number of models and agent scaffolds tested is impressive\n- The insights from the analysis are solid; I appreciate the focus on in-depth analysis and insights as well as the focus on openly releasing all of the materials from the analysis", "weaknesses": "- Some parts in comparisons with other frameworks are overclaimed. For example, as far as I know, frameworks like HAL and Inspect do have open agents and account for variation in tool use/agent scaffolds. \n- Similarly, some of these frameworks have support for general agents too, for example the Inspect ReAct agent or the HAL generalist agent that are available across benchmarks \n- Claims like benchmarks in other suites not being \"product informed\" also seem like a stretch, especially since the other frameworks also incorporate many of the same benchmarks\n- CORE-Bench has 45 tasks in train and 45 in validation; why does the table say 35 and 37?\n- Could you clarify how the budget or number of steps for agents is set, especially over different models? How do you avoid infinite loops?\n- In figure 2, how are the 11 benchmarks collapsed into 5 figures? What is the aggregation function?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8m73udWja2", "reviewer": "Reviewer_JXif", "summary": "The paper introduces AstaBench, a benchmark suite and surrounding tooling intended to holistically evaluate agents for scientific research assistance. Contributions include: (i) a 2400+ problem task suite spanning literature understanding, coding/execution, data analysis, and end-to-end discovery; (ii) a standardized agent environment with date-restricted, production-grade literature search and a computational notebook to foster controlled comparison; (iii) a leaderboard and cost-accounting toolkit that normalizes API prices and reports efficiency; and (iv) a set of baseline agent classes (including several “Asta” agents) used to evaluate 57 agents across 22 classes. Key empirical finding: despite progress on some sub-skills, science assistance remains far from solved; even competitive agents struggle outside literature tasks, and end-to-end success rates remain low.", "questions": "My questions are reflected in the weaknesses I mentioned above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Ambitious scope & breadth. The suite spans four categories with explicit date cutoffs and tool restrictions, a step toward controlled, reproducible comparison in agent settings where tool access often dominates outcomes. \n\n2. Standardized environment. The Asta Scientific Corpus (date-restricted search/snippet tools) plus the Computational Notebook provides an agent-friendly interface decoupled from bespoke agent code—useful for generalization across architectures. \n\n3. Cost-aware leaderboard. Normalized, time-invariant cost accounting and reporting of confounders (openness, tooling) encourage more honest comparisons across agents/providers. \n\n4. Comprehensive baselines. Evaluation across 57 agents / 22 classes (including ReAct, Smolagents, and specialized Asta agents) provides a broad empirical snapshot rather than narrow, self-selected comparisons. \n\n5. Clear high-level findings. The paper documents that literature tasks are comparatively mature, while coding/execution, analysis, and end-to-end discovery remain weak—useful signal for the community.", "weaknesses": "1. Potential product-driven bias and external validity. Several tasks are “inspired by actual user requests to deployed Asta agents,” which risks distributional alignment with the authors’ own agent design and data pipelines. The paper could perhaps quantify how many tasks are product-derived, how they were sampled, and present held-out non-Asta sources to reduce bias?\n\n2. Judging protocols and reliability. The work leans on a “Rubric & LLM Judge” but provides limited detail on prompting, calibration, adjudication, and human validation (e.g., double-blind spot-checks, disagreement resolution, inter-rater reliability). Perhaps more rigorous reporting (e.g., bootstrap CIs, per-task variance, adjudication rates) could be done to support “rigorous” claims?\n\n3. Fairness of comparisons across tooling. Many agents are evaluated with custom or fully-custom tools, while others are restricted to the standard toolset. Even with “tooling” labels on the leaderboard, this remains an apples-to-oranges comparison. Stronger tool access ablations (standard-tools-only vs customized) could accompany headline numbers?\n\n4. Domain skew & representativeness. The benchmark is weighted toward CS with relatively shallow non-CS coverage (e.g., limited biomed beyond LitQA2). For a “scientific research” suite, the domain diversity is modest. Perhaps the authors could provide a breakdown by field, difficulty, and reasoning type to justify claims of “holistic” science?\n\n5. End-to-end metrics under-specified. The paper argues end-to-end discovery is hard (e.g., step-wise success compounding) but lacks task-level pass criteria, failure taxonomies, and human-rated scientific quality of the final reports/code for E2E-Bench(-Hard). Perhaps clarify grading and include expert review samples?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)