SciNav: A General Agent Framework for Scientific Coding Tasks
Authors: TIANSHU ZHANG
Huan Sun
Venue: ICLR 2026 Poster (2026)
Keywords: science agents; test-time scaling
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Autonomous science agents, built on large language models (LLMs), are increasingly being investigated to generate hypotheses, design experiments, and produce reports. Prior science agents primarily focus on open-ended scientific problems, where such outputs—hypotheses, experiments, or analyses are inherently subjective and thus difficult to evaluate rigorously. In contrast, existing scientific coding benchmarks provide tasks with clearly defined, executable outputs that enable objective assessment. However, current agent-based approaches to these benchmarks remain engineering-driven pipelines, lacking principled framework design. This mismatch exposes a gap: the absence of end-to-end, principled science agent frameworks for scientific coding tasks. We address this gap by focusing on scientific coding tasks, where evaluation can be made rigorously, and introducing an agent framework SciNav (Scientific Navigator) that enables more effective solution exploration. Our framework is designed to operate efficiently under constrained search budgets, moving beyond reliance on pre-defined success metrics and prolonged search cycles. Inspired by findings that comparative judgments often reveal finer-grained quality differences and therefore provide greater discriminative power than absolute scoring, our framework leverages pairwise relative judgments within a tree search process to select top-K promising solution branches, prune low-potential ones, and progressively narrow down the solution candidates on the selected branches guided by relative comparisons. We demonstrate our agent's effectiveness across different types of tasks on two benchmarks. Experiments show that SciNav significantly outperforms direct prompting and prior agents like OpenHands and Self-Debug across different base models, task types, and difficulty levels, and exceeds different frontier comparators such as random selection and LLM absolute scoring. These results confirm the strength of our agent design and highlight the effectiveness of relative judgment–guided top-K search for high-quality scientific coding, marking a step toward more practical science agents.

Published as a conference paper at ICLR 2026
SciNav: A GENERAL AGENT FRAMEWORK FOR SCIEN- TIFIC CODING TASKS
Tianshu Zhang The Ohio State University zhang. 11 115358osu.co .dd
Huan Sun
The Ohio State University sun. 397 @ osuuedu
ABSTRACT
Autonomous science agents, built on large language models (LLMs), increas- ingly being investigated to generate hypotheses, design experiments, and produce reports. Prior science agents primarily focus on open-ended scientific problems, where such outputs- hypohhsss,, experiments, or analyses are inherently subjec- tive and thus difficult to evaluate rigorously. In contrast, existing scientific coding benchmarks provide tasks with clearly defined, executable outputs that enable objective assessment. However, current agent- beseed approaches these bench- marks remain engineering-driven pipelines, lacking structured framework design. This mismatch exposes a gap: the absence end-to-end, structured science agent frameworks for scientific coding tasks. We address this gap by focusing scien- tific coding tasks, where evaluation can be made rigorously, and introducing an agent framework SciNav (Scientific Navigator) that enables more effective solution exploration. Our framework designed to operate under constrained search bud- gets, moving beyond reliance pre-defined success metrics and prolonged search cycles. Inspired by findings that comparative judgments often reveal Liner-graine quality differences and therefore provide greater discriminative power than absolute scoring, our framework leverages pairwise relative judgments within tree search process select top-K promising solution branches, prune low www-potential ones, and progressively narrow down the solution candidates on the selected branches guided by relative comparisons. We demonstrate our agent's effectiveness across different types of tasks on two benchmarks. Experiments show that SciNav significantly outperforms direct prompting and prior agents like OpenHands and Self- Self-Debug across different base models, task types, and difficulty levels, and exceeds different frontier comparators such as random selection and LLM absolute scoring. These results confirm the strength of our agent design and highlight the effectiveness of relative judgment-guided top-K search for high-quality scientific coding, marking step toward more practical science agents
INTRODUCTION
Large language models (LLMs) have recently shown strong potential to advance scientific discovery, giving rise science agents that aim to automate the research process end-to-end. Science agents such as Agent Laboratory (Schmidgall et al., 22025), ResearchAgent (Back et al., 2024), and AlphaEvolve (Cui al., 202I), aspire generate research ideas, design experiments, and draft reports. While this vision of broad-scope automation is compelling, it raises a central challenge: how to evaluate the outputs of such agents. Unlike standard benchmarks with unambiguous correctness criteria, the artifacts produced, including novel hypotheses, experimental protocols, and written analyses, are inherently open-ended and subjective, often demanding expert review or costly human studies to judge their scientific validity.
At the same time, existing scientific coding benchmarks such as DSBench (Ting et al. 2024), DA- Code (Huang et al., 2024), BixBench (Mitchener et al.] 2025). DiscoveryBench (Majumder et al.), Bere-Bench (Siegel et al. 2024) and SciCode (Tian et al. 2024) define tasks with executable outputs that allow objective assessment. However, current agent-based approaches to these benchmarks
Our code is publicly available at http:://gthaaccomoooSS/////////

---

Published as a conference paper at ICLR 2026
Scientific Coding Task
Absolute Judgment
Solution A
Solution B
"Code Annlysie " code
*Cade Analyeis "; code onticcl errorin how hondles North Pactfic selection,
good underssaatnn the sclentific but the tochnicol implomentation entation unaveilable paokoge without fallbuck
Assess burn scars using satellite imagery and perform spatial analysis to understand the impact wilcftres.
the
propptp handle the 0-380
LLLLM Scora""::5
L"LLM 400
s3
Relative Judgment
Sn Solution
36
S8
Explanation: evaluate both socutions no for computing and platting the leading Empirical Orthogonal Fanction (EOF) of sea surface temperature dats. "*Solution Andysight....
Each
Initial selection: geet top-k branches
a. get acluttinn
b. derive the child solution from the best
37
$10
"*Kay Extier wnos
"*Solution 8 Amaiysss
Salanoo i mace poncka and inllutee the axtiaal area son woslatiing for FGG ansusak. which Se accentiicalll moce ancunnts for intuudeehhppenddet date, Siatation & mont thorough in citts meeponnnninng...omm but musses this important oen seighting factor.
solution top-k
Expanded Set
s9
Comparison Set
"*Bettor*::
Figure I: Illustration of Top-K Comparative Tree Search (TKCTS). Left: the search tree expands candidate solutions from an initial set, with relative comparisons (grey dashed arrows) guiding which branches and solutions retain and explore. Right: comparison between absolute scoring, which LLM assigns pointwise scores to individual solutions, and relative judgment, which L.M evaluates solution pairs and provides sharper, more reliable distinctions. Relative judgments guide the search toward higher-quality solutions under constrained budgets.
largely rely on general-purpos agents such as OpenHands (Wang et al. 2024) and Auto-GPT Yang et al. 2023), primarily target ingineringg-rrintee workflows like designing tools for Bash- based environment management and reading or writing files (Huang et al. 2024, Mitchener et al. 2025 Jing et al., 2024). Though these workflows be adapted to the scientific coding benchmarks, they mainly focus on the engineering pipeline, which lack structured agent framework design that enables more effective solution exploration.
exploran
As a result, a gap remains: existing science agents are primarily designed to operate end-to-end on open-ended scientific problems, focusing on generating hypotheses, experimental designs, and reports, while scientific coding benchmarks- direct evaluation, are typically approached using general-purpose agents or engineering-centrre pipelines that lack structured framework design. Our work addresses this gap by focusing on scientific coding tasks, where evaluation can be made rigorous, and by introducing framework that enables more effective solution exploration. Recent efforts such as AIDE (Jiang et al.) (2025) have taken steps this direction by proposing specialized frameworks for machine learning coding tasks. However, these systems often assume (i) well-defined evaluation metries (e.g.. leaderboard accuracy) that agents can directly optimize, and (ii) large exploration budgets (c.g.. 24-hour attempts) that allow exhaustive solution generation and testing. Such assumptions are rarely practical, especially when task types are diverse: different scientific problems demand distinct evaluation criteria, many of which are not known in advance, and prolonged search cycles are prohibitively costly. In contrast, our framework produces high-quality solutions under constrained search budgets without relying on pre-specified metrics provide feedback during the agent exploration.
clearly defined, executable outputs that enable
To address the above limitations, We introduce SciNav (Scientific Navigator), an autonomous agent for scientific coding tasks that leverages Top-K Comparative Tree Search (TKCTS) effectively navigate solution spaces under constrained budgets. Many studies have shown that relative judgments have higher reliability and discriminative power than absolute scoring Yan 2024 Liu et al. 2024 Peyrard et al., (202T). As Figure [ shows, it's easier to compare two outputs and decide which is better than to assign an absolute score, as comparisons provide clearer signals of where one solution succeeds or fails relative to another. By treating one solution as an anchor against which another is evaluated, comparative assessments sharpen distinctions between candidates, leading to more reliable outcomes and closer alignment with task instruction than pointwise scoring. Building on these findings, TKCTS integrates comparative judgments into a structured tree search. In the early stages, the search explores broadly but retains only the Top-K most promising branches by comparative
2

---

Published as a conference paper at ICLR 2026
Agents
Seage
Molti-Plan?
Self-Impoiw Selection
Artifact
Evaluation
Research ol
2066
Full-eycle, spmn-edddd
NourlPS miiew
Exp
Agait (Suhlttytrers
Full-sycle
dhsolate
Fap Paper quality
CadeScientar Dasse 6c
cycle, spen ended
Paper
Accepe/Reject Tly..
eycle open ended
Paper
ressew
AlghuEvvlve (Nvikoy
Full-cycle, apes-ended
Task-dellied mesic
Code
Task-spocifle
SoMestr Chi
with use
answer
Accuraty
AIDE [hang at. 3725
Mashine leamiin
Taek-defined metric
Cade
Accuracyy atc
(005)
Scientific coding tasks
LLM relative judgments
Code Exec..
Task specific
Table 1: A comparison of existing science agents with SciNav, in terms of their focus, research artifacts being evaluated, and automatic evaluations. 'Full-cycle' refers to the stages from literature review, ideation, experimentation to report writing. "Self-Improve Selection' refers the evaluation strategy used to choose solutions from the candidate pool for further refinement. Unlike prior agents that rely task-defined success metrics, our agent is designed for practical scenarios where such criteria are not available at run time. 'Y' indicates yes; 'N' indicates no; indicates not applicable.
judgments, pruning low-potential ones to control cost. As the search progresses, child nodes are generated along the retained branches, and comparative judgments rank frontier nodes that the Top-K solutions are promoted for further expansion while the rest are discarded, until a final solution reached. This design balances systematic exploration with adaptive prioritization, effectively directing computation toward high-quality solutions.
In summary, contributions are three-fold:
We introduce SciNav, an autonomous science agent for scientific coding tasks that leverages relative judgments within a Top-K tree search to efficiently explore solution spaces under constrained budgets, enabling systematic reasoning and high-quality code generation.
We conduct a detailed analysis of each component to better understand how it affects the performance our agent framework.
Our results demonstrate that SciNav achieves substantial improvements over baselines ScienceAgentBenn across different base models. Compared to the strongest baseline agent, Self-Debug SciNav delivers up to a 24% relative gain in success rate (SR) and 7.8 absolute points improvement in VER, and it surpasses OpenHands by 22.9% relative gain in SR. Besides, on DA-Code, SciNav achieves 29 absolute points gain on data manipulation and statistical analysis tasks, 13 absolute points improvement on average and 10%-23% absolute gain across different task difficulties over baselines.
2 RELATED WORK
Science Agents. To situate our work, we compare existing science agents along their scope, planning mechanisms, retrieval usage, evaluation strategies, and target artifacts. As shown Table [ prior systems largely operate in full-cycle, open-ended setting, aiming to cover the entire research pipeline that produces outputs such as hypotheses, experiments, or reports, which resist systematic automatic evaluation (Baek et al. 2024 SShhiddaall et al. 2025 Jansen al., 2025 Yamada et al.. 2025; Cui et al., 2021). Some efforts focus on narrower domains (e.g., machine learning coding tasks) and primarily rely on commonly used task-defined metrics (e.g., accuracy and FI) that assume well-specified success criteria (Jiang et al. (2025), or focus on simple question answering tasks and integrate tool use and retrieval produce final answers (Chai et al. (025). In contrast, our work aims to have principled agent framework to help solving scientific coding tasks, where solutions take the form of executable programs that can be rigorously assessed against ground truth outputs. We present SciNav, an agent that integrates relative judgments guided top-K tree search to navigate solution spaces under constrained computational budgets effectively.
Test-time Scaling in LLMs. Prior work such as PlanSearch (Wang et al., 2025) and CodeMonkeys (Ehrlich et al., 2025) demonstrates that increasing the number of generated candidate solutions leads to approximately log-linear improvement in the proportion of problems successfully solved by at least one candidate. This test-time compute scaling phenomenon significantly enhances overall solution coverage, that can be measured by metrics such as Pass @ K in code generation tasks. SFS (Light et al.) 2025) reveals performance gains on programming tasks by enhancing the solution diversity and leveraging prior search experiences. [nell et al. 2025 shows that scaling L.M test-time
3

---

Published as a conference paper at ICLR 2026
compute optimally be more effective than scaling model parameters. Inspired by these work, we leverage the test-time scaling across multiple components of our agent. addition, we uniquely leverage relative judgments-bass bassed frontier comparator to select and refine candidate solutions during tree search. This enables effective exploration and improvement without requiring explicit success criteria, making our approach more generalizable to real-world scientific tasks where such gold signals are often unavailable.
3 AGENT FRAMEWORK
Algorithm 1 Top-K Comparative Tree Search (TKCTS)
Input: Task T' initial candidates Sg; comparison budget B; beam size
Output: Final solution s*
Initialize priority queue Q <- -So:
// initialization
while Q * 0 and B > 0 do
P + SELECTPAIRS(Q))
// choose candidate pairs to compare
foreach (3) E P do
w + CCMMPARE 8;, 8;) )
// LLM returns which is better and why // rank candidates
UPDATERANKING(Q.S.. S4, 8y, w)
B-B-1
Skeep < TOPK K); Strop + Q Skeepi (RUNEE(Surpp))
// discard low-potential // generate children frontier nodes // return the best among remaining
candidates from priority queue
E + EXPAND(S(ep)))
INSERT(Q, E)
SELECTFINAL(Q)
return S*
3.1 DESIGN MOTIVATION
Scientific discovery is rarely linear: it depends exploration, revision, and reflection. Yet, most current reasoning agents make shallow, one-shot decisions, lacking the capacity to backtrack, refine, evaluate solutions strategically. SciNav is designed to overcome these limitations by treating scientific reasoning a trajectory-driven process, where multiple candidate paths are explored, compared, and refined. Not all paths are equal: some lead to breakthroughs, others dead ends. SciNav introduces relative judgments-guidee top-K tree search that compares, prunes and expands solution trajectories.
SciNav is inspired by human-like reasoning processes, where initial rough ideas or high-level plans are generated first, followed by iterative debugging and refinement into more rigorous solutions. This approach allows the agent to progressively improve its solutions through self-correction. The agent explores candidate solutions via tree search, with a top-K selection strategy that both enables controlled backtracking and reduces the cost of subsequent comparative evaluations. Backtracking allows the agent to revisit earlier solutions and reconsider alternative paths when necessary, while relative judgments provide more reliable guidance which branches to pursue, how prioritize exploration, and which solutions merit further refinement. Together, these mechanisms enable more deliberate and adaptive problem-solving than prior one-shot purely engineering pipeline-based agents.
3.2 COMPONENTS
As Algorithm D shows: TKCTS emphasizes relative judgments as the primary evaluation signal. In each iteration, candidate pairs are compared, rankings are updated, the Top-K branches or candidates are retained, low-potential branches and candidates are pruned, and children of retained branches are expanded. The loop continues under comparison budget, yielding final solution selected by pairwise preference. While Algorithm outlines the overall loop of TKCTS, its effectiveness relies several interacting components. In what follows, describe four major components- iintonant-iiiiiall planning and code generation, self-debug, coff-impovee and the frontier comparator that together implement the framework.
4

---

Published as a conference paper at ICLR 2026
Initial Planning and Solution Generation. Existing work such as PlanSearch WWagg et al. 2025 and CodeMonkeys (Ehrlich et al.] 2025) show that as the number of generated solutions increases, the fraction of problems in a dataset that are successfully solved by at least one candidate often grows approximately log-linearly. This test-time compute scaling effect of generating more candidate solutions can significantly improve the overall success solution coverage such Pass K in coding tasks. Hence, we follow (Jiang et al.) 2025) to generate a rich pool of diverse solution candidates to give more starting points to increase the agents' exploration success. We first prompt LLM to generate multiple high-level plans, and ensure that previously generated plans are visible to the model avoid repeated plans. Then for each plan, the LLM generates corresponding program as a candidate solution. This approach unlocks hidden insights that would be missed by deterministic one-pass inference alone.
Self-Debug for On-the-Fly Error Correction. Scientific coding tasks require the agent to produce executable code. SciNav incorporates self-dehugging mechanism leverage code interpreter to detect and repair bugs during tree search. This reflective capability enables the agent to revise faulty steps without discarding entire trajectories.
Iterative Self-Improve through Reflective Reasoning. Reasoning is not just about fixing mistakes, but about getting better with each step. SciNav employs iterative self-refinement by prompting the model to identify specific refinement point within selected frontier solution, based on the task description. This process mirrors how humans iteratively refine solutions, progressively improving them toward correctness and completeness.
Frontier Comparator. Effective trajectory selection is critical to the success of SciNav. At each step of the search, the agent must decide which frontier solutions expand, refine, or prune. We design the Frontier Comparator around comparative judgments, where candidate solutions are directly contrasted against each other rather than scored in isolation. This relative evaluation provides sharper distinctions, greater stability, and closer alignment with human preferences than noisy absolute scores. Concretely, given a pool of candidate solutions, the comparator selects promising branches through iterative pairwise comparisons. The top-K branches are retained for further exploration, while low-potential ones are pruned. As the search deepens, new child solutions are introduced into the pool and evaluated in the same pairwise manner. At each step, we prioritize only the top-K candidates for relative judgments. This strategy enables controllable backtracking by allowing the search to revisit up to K carlier solutions, while also reducing the cost of pairwise relative judgments every time. Crucially, backtracking ensures that if the currently preferred path stagnates or fails, the agent can return previously lower-ranked candidates and resume exploration from there. This prevents premature commitment to suboptimal solutions and supports adaptive recovery, making the search more resilient. The mechanism also allows the agent dynamically allocate its limited budget toward the most promising directions. In summary, the Frontier Comparator transforms trajectory selection from one-shot scoring into comparative, iterative process that systematically navigates solution spaces under constrained resources. After each round of pairwise judgments, ranking algorithm is applied to the candidates in the priority queue update their quality ordering (see Appendix (F).
EXPERIMENTAL SETUP
Agent-level Baselines. To evaluate the effectiveness of SciNav, we compare it against the following three baselines, each representing a different strategy for program generation and reasoning: 1) Direct Prompting: This baseline uses an LLM to generate a solution in a single pass using task-specific prompt. It reflects the standard zero-shot setting commonly used prior work. This method serves as a simple and widely adopted baseline for evaluating initial model generation quality. 2) Deff-Debug (Chen et al. 2024): In this baseline, the model generates an initial solution and then attempts to improve it via self-debugging based on the Python interpreter execution feedback. While this method allows for limited reflection, it does not incorporate trajectory search, external evaluation, or ranking. It evaluates the isolated effect of a self-correction loop without broader solution exploration. 3) OpenHands (Wang et al. 2024] OpenHands is a general agent that is designed for multiple domains including Web and software engineering tasks. It builds on the ReAct framework (Yao et al.) (023) to generate the next action based on the previous observation. Instead of directly generating the entire program solution once, OpenHands gradually finishes the solution step by step.
5

---

Published as a conference paper at ICLR 2026
Frontier Comparator Baselines. We compare our Frontier Comparator against several alternative selection strategies: (1) Random Selection, which randomly picks a candidate solution from the pool; 2) LLM-Absolute: The LLM assigns numerical score to each candidate solution, and the highest-scoring solution is selected for further refinement. 3) Rubric-Abxolute (Chen et al. (025): The LLM scores each candidate according to structured rubric derived from the ground truth program. These serve as baselines to evaluate the advantage of pairwise comparative judgments.
Datasets. To evaluate SciNav in realistic scientific coding scenarios, we use two benchmarks: SeienceAgemtBench (Chen et al. 2025) and DA-Code (Huang et al. 2024). Science a curated benchmark designed to assess agents' capabilities in scientific discovery. This dataset includes a diverse set of tasks that cover the entire workflow such as model development, data analysis, and visualization, spanning from four scientific disciplines: Bioinformaties, Computational Chemistry, Geographical Information Science, and Psychology & Cognitive Neuroscience. All of our experiments are done on their "without expert-provided knowledge" setting. DA-Code encompasses a diverse set of challenging data wrangling and analytics tasks that require advanced data science programming techniques for intricate data processing and answer derivation. To evaluate the effectiveness of Nci, randomly sample 100 tasks across different categories such as data insight, data manipulation, data wrangling, and statistical analysis from DA-Code, while also covering different levels of difficulty.
Experiment Details. We experiment with GPT-4o (both the 0513 version and the 1 1120 version) OpenAA| 2024), Claude-3.7 2ommett20225011-xv11d version) (Anthropic) 2025) and DeepSeek-RI (DeepSeck-Al) 2025). Since GPT-4o (2024-11-20) version is much cheaper than GPT-4o (2024-05- 13) version, we try both versions for the main experiments, but use GPT-4o 1002--11220) version for the ablation study and frontier comparator part. For all experiments, we use the same hyperparameters. For temperature, we use 0.5 for code generation and 0.5 for debug, analysis and summary, and when leveraging LLMs to compare or judge the solutions. For top-p, we use top 0.95, and perform O-shot prompting via the APIs. For each baseline, we run three times to get the mean performance. For each frontier comparator, we run the agent twice to get the mean performance. To constrain the budget, we set the initial solution number to 5, maximum debug step to and total exploration step to 10, which includes the self- saff-improvement step if the budget has not been exhausted by self-debug.
Evaluation Metrics. We follow previous work (Chen et al. 2025 Huang et al. 2024) to comprehen- sively evaluate each generated program. For Beich Execution Rate (VER) measures whether program can execute without errors. (2) Success Rate (SR) assesses whether the output satisfies the specific task goal, such as passing predefined task success criteria, matching expected predictions, or producing a high-quality visualization. These criteria are implemented as task-specific evaluation programs during the benchmark annotation process. Among the reported metrics, SR (Success Rate) is the most important as it directly reflects task success. For DA-Code, we leverage their evaluation suite, which supports multiple tasks through configurable setups, where each task is uniquely identified and defined with its required outputs, metrics, and options. Their tailored scoring methodology assesses agent performance across diverse outputs such as tables, charts, and machine learning predictions, with metrics customized for each output type. (3) API Cost (Cost) reports the average dollar cost required to complete a single task using the agent. This metric accounts for API usage and serves to highlight the importance of designing cost-efficient agents, as emphasized by Kapoor et al. 2024
use two key metrics. (1) Valid
RESULT ANALYSIS
5.1
MAIN RESULTS
Table 2 compares the performance of different agent strategies using two versions of GPT-4o (2024- 13 and 2024-11-20) across two metries: Success Rate (SR) and Valid Execution Rate (VER). The experiment results show that our proposed agent SciNav consistently outperforms baseline approaches across both model versions. Under the 2024-05-13 model, SciNav achieves an SR of 16.1%, %. surpassing Self-Debug (14.7%) and OpenHands (13.1%), while maintaining strong VER of 66.0%. Although the cost (0.512) is higher than Self-Debug (0.057) and Direct Prompting (0.011), SciNav offers better balance between performance and cost-effectiveness compared to OpenHands (1.093), which is substantially more expensive despite lower SR and VER. With the updated GPT- 4o (2024-1 1-20), due to the constrained budget, we only choose the best baseline Self- Self-Debug for

---

Published as a conference paper at ICLR 2026
Base Model
Agents
SR
VER
Cost 4
Direct Prompting
7.50
42.2
OpenHands
13.1
62.8
1.093
GPT-4o (2024-05-13)
Self-Debug
14.7
71.2
0.057
SciNav (Ours)
16.1
66.0
0.512
Self-Debug
15.0
67.0
0.030
GPT-4o (2024-11-20)
SciNav (Ours)
18.6
69.9
0.342
Self-Debug SciNav (Ours)
22.5
84.3
0.066
Claude-3.7
25.5
72.5
0.893
Self-Debug
18.6
59.8
0.023
DeepSeck-RI
SciNav (Ours)
19.6
67.6
0.298
Table 2: Mean performances of each agent ScienceAgentBened (Chen et al. 2025 Among the reported metrics, SR (Success Rate) the most important as directly reflects task success. VER (Valid Execution Rate) indicates whether program can be executed without errors and is closely related to the number of debugging steps. Note that we run SciNav with debug steps, while Self-Debug is run with 10 debug steps, which may occasionally allow Self-Debug to achieve higher VER than SciNav.
Performance on by Task Type and Average
Performance by Difficulity
Debug
Dobus
0.31
0.10
D11
0,23
0.14
Data Insight
Data Manipulation
Data Wrengling
Statistical Analysia
Average
Fany
Medkum
Hard
(a) Performance by Task Type
Performance by Task Difficulty
Figure 2: Performance on DA-Code (Huang et al. 2024): (a) by task type, (b) by task difficulty. Model: GPT-4o (2024-11-20).
comparison. SciNav achieves large performance gain compared to Self-Debug, with 24% relative improvement for SR and a 2.9-point absolute gain for VER.
Generalization to Other Base Models. We further evaluate SciNav Claude-3.7 and DeepSeek-RI to assess its generalization beyond GPT-4o. Claude-3.7 is recognized for strong code generation, while DeepSeek-R11 recognized for strong reasoning. As shown in Table / SciNav consistently outperforms Self-Debug on both models, with gains of up to 3 points in success rate (SR) and points in VER. These results indicate that SciNav generalizes effectively across different base models, despite their distinct strengths.
Generalization to Other Datasets. Figure 2 presents the performance comparison between SciNav and Self-Debug on DA DA-Code across different task dimensions. Figure shows that SciNav con- sistently outperforms Self-Debug most task categories and on average, achieving notable gains in data manipulation (29% absolute improvement). statistical analysis (29% absolute improvement) and on average (13% absolute improvement), while maintaining comparable performance in data wrangling. Figure 2b further demonstrates that SciNav adapts well to varying task difficulty, yielding strong improvements across easy (12% absolute improvement). medium (10% absolute improvement), and especially hard tasks (23% absolute improvement). These results confirm that SciNav delivers superior performance across diverse task types and task difficulty, underscoring its effectiveness as a scientific coding agent for DA-Code.
7

---

Published as a conference paper at ICLR 2026
Models
Frontier Comparator
SR
VER
Random Selection
15.2
64.7
GPT-4o (2024-11-20)
LLM-Absolute
16.2
69.1
Relative Judgments
18.6
69.9
GPT-4o (2024-1
Rubric-Absolute (w/ GT) 21.1
74.5
Table 3: The effect different frontier comparators on our agent. "w/ GT means using ground truth program information in the frontier comparator: We include Rubric-Absolute only reference, since it impractical in real-world settings.
5.2 FRONTIER COMPARATOR ANALYSIS
Table 3 evaluates the impact different frontier comparators on agent performance using GPT-4o 22024-11-20) Among the Random Selection, LLM--bsolute and Relative Judgments, Relative Judgments achieves the highest SR (18.6%) and VER (69.9%). outperforming both other two baselines, which attain lower SR of 15.2% and 16.2%, respectively and lower VER of 64.7% and 69.1% respectively.
The Rubric-Absolute achieves the highest SR and VER among all frontier comparators, benefiting from access to ground truth rubric descriptions that detail the correct solution steps. While this method partially leverages ground truth signals, highlights the potential performance gains from incorporating reliable supervision when available. However, such rubric-based evaluations are often impractical real-worke scientific tasks, where detailed grading criteria are rarely accessible. In contrast, our relative judgments- -bsed frontier comparator offers more generalizable solution, as operates independently of ground truth labels while still delivering strong performance. Due to its effectiveness and applicability in reality, we adopt relative judgments the default frontier comparator in our main agent framework.
5.3 COMPONENT ABLATION STUDY
of Init. Solutions
Avg Successful Init. Solutions
Avg # Successful Nodes
Use Self-Improve?
Success Rate
0.24
0.40
0.40
0.57
40.5
No
0.98
1.14
45.2
Yes
1.17
2.69
57.1
Table 4: Component ablation study on SciNav. Model: GPT-4o (2024-11-20), The experiment is conducted on 40 tasks ScienceAgentBench, each of which has been successfully solved least once by either baseline agent SciNav.
Table presents component-wise ablation study evaluating the impact of the number of initial solutions and the use of the self-improvement mechanism in our agent framework. We observe that:
(1) Number of successful initial solutions directly contributes to end-to-end success rate. The table shows strong correlation between the average number of successful initial solutions and the overall success rate. Without self-improvement,, increasing the number of initial solutions from to 5 yields steady improvement in the average number of successful initial solutions (from 0.24 to 0.98), the average number of successful nodes (from 0.40 to 1.14) and the overall success rate (from 40.5% to 45.2%) This indicates that generating multiple initial solutions increases the chance that least one initial solution is close to correct, thus improving the agent's 'tss final performance. The more successful starting points the agent has, the more likely it is to select or build upon a valid reasoning path.
(2) Self-improvement enables the agent to generate more correct programs and achieve the highest success rate. The final row of Table iisolates the effect of enabling self infroreprovetttt for the same initial solution size 5, when self-immeeeeeee ieff-mmproveeent is disabled (row 3), the agent achieves average of 1.14 successful nodes and 45.2% success rate, while when self eef--improvemntt is enabled (row 4), the number of successful nodes jumps to 2.69, and the success rate increases significantly to 57.1 17.. This
8

---

Published as a conference paper at ICLR 2026
demonstrates that self- ieelovmmpoveeentt nearly doubles the number of correct programs, allowing the agent to refine and expand upon flawed or incomplete initial solutions. The improvement in both node-level correctness and overall task success confirms that self-improvement is key driver of end-to-end performance.
Overall, the findings highlight the complementary roles of initial solutions and iterative self-refinement in enhancing agent performance on the tasks.
5.4 ERROR ANALYSIS
To evaluate the effectiveness of our different frontier comparators, we conducted an error analysis on 20 randomly selected unsuccessful tasks from AcinneeAgent using SciNav. We categorize errors into two main types: exploration errors and verification enrors. An exploration error occurs when the agent's s entire trajectory fails to produce any solution that meets success criteria. In contrast, verification error arises when least successful solution exists in the trajectory, but the agent fails to identify or select it as the final output.
Analysis on 20 Unsuccessful Tasks
Verification Error 15.8%
Exploration Not Executable
15.8%
Exploration Error: Executable but Unsuccessful 68.4%
Figure Error analysis on 20 randomly selected unsuccessful tasks of ScienceAgentBench.
We further subdivide exploration errors into categories: executable, where all generated solutions are buggy, and executable but unsuccessful, where some solutions are runnable but do not satisfy the task requirements. As shown in Figure only 15.8% of the failures were due verification errors, suggesting that our relative judgments based frontier comparator is generally effective at recognizing correct solutions. The remaining 84.2% of failures were attributed to exploration errors, with 15.8% resulting from non-executable programs and 68.4% from executable but incorrect outputs. This analysis highlights that the dominant source of failure lies in the exploration stage. suggests that more test-time compute is needed in order to cover successful solution in the trajectory, such as increasing initial solution size or self-improve more steps.
INFLUENCE OF K ON SciNav
Figure 4 examines how the top-K K solution comparison factor influences SciNav's performance and computational cost. As shown in Figure increasing K initially improves average task accuracy, with K = 2 yielding the highest performance across our preliminary study 30 randomly sampled DA-Code tasks. Beyond K 2. performance gains do not continuellarger K values either plateau or slightly degrade accuracy. Figure "bgure/biillutraee illustrates the corresponding cost trend: the number of required LLM calls grows along with K. meaning larger K values incur substantially higher computational overhead. Taken together, these results show that K = offers the best trade-off between performance and cost, providing the most effective search behavior under constrained budgets. We therefore adopt K = as the default setting for our experiments.
7 CONCLUSION
In summary, we proposed SciNav, a principled autonomous science agent framework for scientific coding tasks. Unlike prior science agents that mainly operate open-ended scientific problems

---

Published as a conference paper at ICLR 2026
Influence Agent Performance
Agont
0.25
24
0.30
60.00
0.15
0.05
03
3.0
3.5
2.0
2.5
3.0 3.5
6.0
4.5
5.0
1.5
3.0
3.0
35
(a) Average score across different values K E 3,, 4,5) Agent cost across different values of K a preliminary study on randomly sampled DA-Code tasks.
31,25
K = 2 achioves the highest average performance, and is there- fore used for all subsequent experiments.
Figure 4: Agent performance and cost across different values K E (1,2,3,4,5)
with subjective evaluation criteria, scientific coding tasks enable rigorous assessment through code execution against ground truth. Yet, existing agents for these tasks remain largely general-purpose or depend on purely engineering-die that integrates relative judgments into Top-K tree search. Our proposed agent SciNav systematically explores solution spaces under constrained budgets, enabling both efficient search and high- high-quality code generation. We provide detailed component-leved analysis which shows how each design choice contributes to the overall effectiveness of the framework, offering insights into the mechanisms behind improved agent performance. Through extensive experiments DA-Code, we show that SciNav consistently outperforms direct prompting and prior agent baselines such as OpenHands and Self-Debug across diverse base models, task types, and difficulty levels. while also surpassing alternative frontier comparators. These results confirm the effectiveness of relative gudgment-guide top- K search and represent step toward science agents that are more reliable, principled, and practical.
pipelines. We bridge this gap with a principled agent framework
and
ACKNOWLEDGMENTS
The authors would thank colleagues from the OSU NLP group and the Amazon AGI team for constructive feedhack. This research was sponsored in part by NSF OAC 21 12606, Amazon, Cisco, and Ohio Supercomputer Center (Center 1987). The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.
REPRODUCIBILITY STATEMENT
We have made efforts ensure the reproducibility of our work. The paper provides detailed descriptions the proposed framework (Section [ experimental setup, evaluation protocols and implementation details (Section 4 in the main text and Appendix G All datasets used in experiments are publicly available, and we describe all the prompts we used for each component in Appendix G To further facilitate reproducibility, will release the full source code upon acceptance of the paper.
ETHICS STATEMENT
This work introduces SciNav, framework for autonomous science agents tailored to scientific coding tasks. Our study is conducted entirely on publicly available scientific coding benchmarks (SeienceAgentBencc and DA-Code) that involve well-defined programming tasks with objectively
10

---

Published as a conference paper at ICLR 2026
verifiable outputs. No human subjects, personal, sensitive data were used in any part of this research.
We acknowledge the broader ethical considerations of autonomous science agents. While framework designed for controlled scientific coding benchmarks, we recognize that autonomous code generation systems may be misused to produce insecure, biased, or harmful code if applied irresponsibly. To mitigate such risks, we restrict our experiments to open benchmarks with safe and bounded tasks, and we clearly document the evaluation setup and intended scope of use.
We affirm that this research complies with the ICLR Code of Ethics. All results and analyses were produced by the authors without external conflicts of interest undisclosed sponsorship. Our goal is to advance principled, transparent, and rigorously evaluated science agents, while encouraging the community to continue addressing fairness, safety, and responsible deployment in broader applications.
REFERENCES
Anthropic. Claude 3.7 sonnet. 2025. URL /tt//ttss//wwwaaarrrrrrr...ooCCt
Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint
wwX7v:24004007338, 2024.
Ohio Supercomputer Center. Ohio supercomputer center, 1987. URL http://osc.eeu//rr edu/akk /19495/F5s1ph73
Jingyi Chai, Shuo Tang, Rui Ye, Yuwen Du, Xinyu Zhu, Mengcheng Zhou, Yanfeng Wang, Yuzhi Zhang, Linfeng Zhang, Siheng Chen, et al. Scimaster: Towards general-purpost scientific ai agents, part i. x-master as foundation: Can we lead on humanity's last exam? arXiv preprint arXiv:2507.05241, 2025.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representutions, 2024. URL
https::/opperewwe wwwpinreview.nneeeooreeeeeeeeeee
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. Sci- enceagentbench: Toward rigorous assessment of language agents for data-driven scientific dis- covery. In The Thirleenth International Conference Learning Representations, 2025. URL
https://openrevia
Can Cui, Wei Wang, Meihui Zhang, Gang Chen, Zhaojing Luo, and Beng Chin Ooi. Alphaevolve: A learning framework to discover novel alphas in quantitative investment. In Proceedings of the
2021 International conference on management of data, pp. 2208 2216, 2021.
DeepSeek-AL. Deepseek-rl. 2025. URL httphhttts//hhugiiggggaeee http://c.....eeeeeeeeeaaeeeeeeee http://tts////ggiinoaa../ce////e
Ryan Ehrlich, Bradley Brown, Jordan Juravsky, Ronald Clark, Christopher Ré, and Azalia Mirhoseini. Codemonkeys: Scaling test-time compute for software engineering, 2025. URL /ttpss//arxivv
htggabb//501.144 14723
Arpad E. Elo. The Rating of Chessplayers, Past and Present. Arco Pub., New York, 1978. ISBN 0668047216 9780668047210.
URL http://www. amazon. com/
Rating- Chess Chweshhhlaeeeeeeeeeeeeeeeeeeeeee Playelll PaspeerssaatttPeeeeee/////66161
Yiming Huang, Jianwen Luo, Yan Yu, Yitong Zhang, Fangyu Lei, Yifan Wei, Shizhu He, Lifu Huang, Xiao Liu, Jun Zhao, et al. Da-code: Agent data science code generation benchmark for large language models. arXiv preprint arXiv:2410.07333 2024.
Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S Weld, and Peter Clark. Codescientist: End-to-end semi- automated scientific discovery with code- based experimentation. arXiv preprint arXiv:2503.22708 2025.
11

---

Published as a conference paper at ICLR 2026
Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code, 2025. URL /RRLhhtps///rrxvvorrr
abs/2502. 13138
Liqiang Jing, Zhehui Huang, Xiaoyang Wang. Wenlin Yao, Wenhao Yu, Kaixin Ma. Hongming Zhang, Xinya Du, and Dong Yu. Dsbench: How far are data science agents from becoming data science experts? arXiv preprint arXiv:2409.00
2024.
Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter. arXiv preprint arXiv:2407.01502, 2024.
Jonathan Light, Yue Wu, Yiyou Sun, Wenchao Yu, Yanchi Liu, Xujiang Zhao, Ziniu Hu, Haifeng Chen, and Wei Cheng. SFS: Smarter code space search improves LLM inference scaling. In The Thirteenth International Conference on Learning Representations, 2025. URL https///
Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulić, Anna Korhonen, and Nigel Collier. Aligning with human judgement: The role of pairwise preference in large language model evaluators, arXiv preprint ar arXiv:2403.16950 2024.
Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhi- jeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discoverybench: Towards data-driven discovery with large language models. In The Thirleenth International Conference on Learning Representations.
Ludovico Mitchener, Jon M Laurent, Benjamin Tenmann. Siddharth Narayanan, Geemi Wellawatte, Andrew White, Lorenzo Sani, and Samuel G Rodriques. Bixbench: comprehensive benchmark for llm-based agents in computational biology. arXiv preprint arXiv:2503.00090 000966 2025.
Alexander Novikov, Ngån Vū, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang. Adam Zsolt Wag- ner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al. Alphae- volve: A coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506. 13131, 2025.
OpenAl. Gpt-4o. 2024. URL httt:s:://oenaaa.cmmmm//e///e///
Maxime Peyrard, Wei Zhao, Steffen Eger, and Robert West. Better than average: Paired evaluation of nlp systems. arXiv preprint 10.11:2110.10746, 2021.
Samuel Schmidgall, Yusheng Su, 7e Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Michael Moor, Zicheng Liu, and Emad Barsoum. Agent laboratory: Using Ilm agents as research assistants, arXiv preprint XrXiv:2501 10000.0422...20 2025.
Zachary S Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, and Arvind Narayanan. Core- bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. arXiv preprint arXiv:2409.1 11363, 2024.
Charlie Victor Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. The Thirteenth Interna- tional Conference on Learning Representations, 2025. URL https::/oppnnreiieeeee www.:/.w..eef.omnmt/ id=4FWWwZtd22n
Minyang Tian, Luyu Gao, Shizhuo Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. Scicode: A research coding benchmark curated by scientists. Advances in Neural Information Processing Systems, 37:30624-333 -30650, 2024.
Evan Z Wang, Federico Cassano, Catherine Wu, Yunfeng Bai, William Song, Vaskar Nath, Ziwen Han, Sean M. Hendryx, Summer Yue, and Hugh Zhang. Planning in natural language improves LLM search for code generation. The Thirteenth International Conference on Learning Repre-
sentations, 2025. URL httpp:/:/ppeerrreeeee/e//rrrrrra
12

---

Published as a conference paper at ICLR 2026
Xingyao Wang. Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingehen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for software develop- generalist agents. In The Thirteenth International Conference Learning Representations, 2024.
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search. arXiv preprint arXiv:2504.0806, 2025.
Ziyou Yan. Evaluating the effectiveness of llm-evaluuttors (aka llm-as-judge). eugeneyan.com, Aug 2024. httt:s/swaaggagggooooooo/oo//m/e
Hui Yang. Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions, 2023. URL httpp:://aaxxw.rrrrrbb////////// /ttbbb//233.0000 02224
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL httpttts////oeeeeeeeee net/forum?id-W
vluYUL-X
13

---

Published as a conference paper at ICLR 2026
A LLM USAGE STATEMENT
In preparing this manuscript, we used LLM as an assistive tool for polishing the writing. Specifically, the LLM was employed improve clarity, grammar, and flow of text that had been originally drafted by the authors. The LLM did not contribute to research ideation, experimental design, data analysis, or the development of algorithms. All technical content, results, and conclusions were produced solely by the authors.
B COST ANALYSIS
Base Model
Agents
SR
VER
Cost
Direct Prompting
7.50 (0.5)
42.2 (1.6)
0.011 (0.000)
OpenI
13.1 (2.6)
62.8 (2.9)
1.093 (0.071)
GPT-4o (2024-05-13)
Self-Debug
14.7 (3.2)
71.2)
0.057 (0.001)
SciNav (Ours)
16.1 (1.2)
66.0 (3.5)
0.512 (0.009)
Self-Debug
15.0 (4.8)
67.0 (7.4)
0.030 (0.010)
GPT-4o 12024-11-200
SciNav (Ours)
18.6 (3.3)
69.9 (0.6)
0.342 (0.008)
Table Mean performances each agent and standard deviations on ScienceAgentBench (Chen et al. 2025). Among the reported metrics, SR (Success Rate) is the most important it directly reflects task success. VER (Valid Execution Rate) indicates whether a program can be executed without errors and is closely related to the number of debugging steps. Note that SciNav with 3 debug steps, while Self-Debug run with 10 debug steps, which may occasionally allow Self-Debug achieve higher VER than SciNav.
C STATISTICAL SIGNIFICANCE TEST
We conducted statistical significance test quantify the reliability of the observed performance differences. Specifically, we compared our agent SciNav with the strongest baseline, Self-Debug, on the DA-Code benchmark using the Mann-Whitney U test. The resulting p-value 0.0177, indicating that the improvement achieved by our agent is statistically significant (p < 0.05).
D CROSS-MODEL EVALUATION
Models
SR
VER
Solution GenerationnRefinement
Judge/Verify
GPT-4o
GPT-4o
18.6
69.9
mixture of GPT-4o and Claude-3.7
GPT-4o
19.6
63.7
Claude-3.7
GPT-4o
18.6
72.5
Table 6: Model selection of solution generatiin/rrefiemment and judgment/vvrrffiatton effect agent on Acenticege CCen et al. 2025).
14

---

Published as a conference paper at ICLR 2026
E DA-CODE SAMPLED CASES
We release id of our sampled 100 DA-Code tasks as following: di-csv-005, di-csv-007, di-csv- 008, di-csv-009, di-csv-010, di-csv-011, di-csv-012, di-csv-013, di-csv-0l6, di-csv-018, di-csv-019, di-csv-022, di-csv-023, di-csv-027, di-csv-028, di-csv-029, di-csv-032, di-csv-033, di-csv-034, di- csv-035, dm-csv-002, dm-cssv005, dm-csv-007, dm-csv-014, dm-csv-017, dm-csv-018, dm-csv-019, dm-csv-020, dm-csv-026, dm-csv-029, dm-csv-030, dm-csv-031, dm-csv-033, dm-csv-036, dm-csv- 037, dm-csv-039, dm-csv-040, dm-csv-046, dm-csv-054, dm-csv-056, dm-csv-061, dm-csv-062, dm-csv-066, dm-csv-068, dm-csv-069, dm-csv-072, tata-wranglingO00 wran-wwang -00,, data- wrangling- 0100, data- data-wrangen wrangling- 012, cata-wrangling-O11
waaaalawnnnnnnnn-nn 020, data taattawwanllin--00 lata-wrangling-O13 data-wrah www.-wlinn-------e -035 data- wrangling-OO3e data- data-wrang wrangling-O41 data wrangling-
042, Data-wrangling-Oke data- siaaa-waagllin--44
Muttawrangling-O44 Gata-wrangling-D0 data-
wrangling-0555 data wrangling-Osee data- www.wwiingggnnnnn data- wrangling- 0059, data- lata-wrangling--e
data-wrangling-06:
waaa--rragling-067. data-wrangling- wareellwaalggnn---e cata-wrangling-053
068, data- gatta-wrangling-07.
Sata-wrangling-076 data- weaaeawwanllnng-777 data-wrangling-OS5 data-
wrangling- 085, data- data-wrand wrangling-O88 data- waaaaaawwanggggnnnn 095, waata-wrngging---e data sitiaa-waagiin---0
data- wrangling- 100, data-wrangling- 101, data-sa-001, data-sa-003, Kata-sa-004 data-sa-009 data-
sa-012, data-sa-014 4, data-sa-016, data-sa-01 8, data-sa-020, data-sa-022, data-sa-023, data-sa-024, data-sa-030, data-sa-031, data-sa-033, data-sa-039, data-sa-040.
F ELO ALGORITHM
For completeness, we provide the details of the Elo Rating algorithm (Elo 1978) used to maintain a ranking list after relative judgments in SciNav. Firstly, we initialize the Hlo score F, 1500 for each code solution P.. Then, we iteratively update the Elo scores by using the relative scores between any two solutions. Taking the solution pair P. and P3 as an example, we obtain their comparative judgment results S; and Sy:
Expected scores: When solution P3 faces solution Pj. the expected score for P. (denoted Eş) is:
E, 1 10 R2)/400 Similarly, the expected score for P, is E, =1 E,.
(1)
After each comparative judgment:
If P; is better: 5; S, =0
If Py is better: S, 0, 0, =1
If P: and Pj are equally better: S; = 0.5, S, 0.5
Rating update rule: Use the standard Elo update formula. For solution P::
R R4 + (S. 1.))
(2)
Where:
R2 is the old rating, R2 is the new rating.
S: is the actual score.
E, is the expected score.
K is constant. We set it to 32 to determine how fast ratings change.
For each pair of solutions, we update both of their Elo scores once. After all pairwise comparisons, we obtain the final Elo scores for all solutions, which can be used to derive ranking.
G PROMPTS
15

---

Published as a conference paper at ICLR 2026
Initial Planning and Code Generation
You are scientific coding expert to write code based on the task description. You need to come up with an excellent, reasonable and creative plan for a solution and then implement this solution in Python. We will now provide description of the lask.
Task description:
Task goal: You are an expert Python programming assistant that helps scientist users to write high- quality code to solve their tasks. Given request, you expected to write complete program that accomplishes the requested task and any outputs in the correct format. Please wrap your program in code block that specifies the script type: python. For example:
python World!")
Please keep your response concise and do not use a code block if it's not intended to be executed. Please do not suggest few line changes, incomplete program outline, or partial code that requires the user to modify.
Please do use any interactive Python commands your program, such as "lpip install numpy', which will cause execution errors.
Memory: (previous generated plans, execution results and summaries]
Response format:
Your response should be a brief outlinessketch your proposed solution natural language (3-5 sentences), followed by a single markdown code block (wrapped in -') which implements this solution and if it's machine learning task, then prints out the evaluation metrie. There should be no additional headings text in your response. Just natural language followed newline and then the markdown code black.
Solution sketch guideline:
Take the Memory section into consideration when proposing the design, don't propose the solution.
it's a machine learning task, then keep the evaluation the same. The solution sketch should be 3-5 sentences.
If the task is machine learning task, propose evaluation metric that is reasonable for this task. The data is already prepared and available the 'input' directory. There need unzip any files. For any provided file path, you should replace with '//nnpt'' plus the file name and ignore other dictionary or subdirectionary name in the path.
Implementation guideline:
The code should # *mmplement the proposed solution** If the task is machine learning task, you should **print the value the evaluation metric computed on a hold-out validation set**, The code should be single-file python program that is self-contained and can be exccuted as-is. No parts of the code should be skipped, don't I terminate before finishing the script. Your response should only contain single code block.
Be aware the running time the code, it should complete within 15 minutes.
All the provided input data stored in '/input' directory. * "IIf there is test data provided for this task, please save the test predictions in submission.csv file in the '/working' directory as described in the task description ** This is extremely important since this file is used for grading/cvaluation. DO NOT FORGET THE submission.cs. file!
can also use the /working" directory to store any temporary files that your code needs to create. The evaluation should he hased on 5-fold cross-validation but only if that's an appropriate evaluation for the task at hand.
Installed Packages:
Your solution can use any relevant machine learning packages such as: torchvision, scikit-lcarn, torch, torch- mrcch-geometric, lightGBM, timm, xgboost, pandas, numpy, eeyesian-optimizatinn statsmodels. Feel free any other packages too (all packages are already installed!). For neural networks suggest using PyTorch rather than TensorFlow.
Data Overview: [)
16

---

Published as a conference paper at ICLR 2026
Self-Debug Prompt
You are a scientific coding expert to write code based on the task description. Your previous solution had bug, so based on the information below, you should revise it in order to fix this bug. Your response should be an implementation outline in natural language, followed by single markdown code block which implements the bugfix/solution.
Task goal: You are an expert Python programming assistant that helps scientist users to write high- quality code to solve their tasks. Given user request, you are expected to write complete program that accomplishes the requested task and save any outputs in the correct format. Please wrap your program in code block that specifies the script type: python. For example:
python print ("Hello World!")
Please keep your response concise and use code block it's not intended to be executed. Please do not suggest few line changes, incomplete program outline, or partial code that requires the user to modify. Please do not any interactive Python commands in your program, such as "lpip install numpy', which will
execution errors.
Here's the user request you need to work on: [task description)
Previous (buggy) implementation:
Execution output:
Traceback (most recent call last):
File "runfile. py", 1ine 179, in <module>
pair_plot_img np. frombuffer (pair_plot. canvas. tostriigg.gbb)) dtype=np. uint8 uints8
AttributeError:
cnnteerroor:Figuuree 'FiuureCavvaMac' object has no attribute 'tostring_rgb'. Did you 'tostringaarg"
Execution time: 13 seconds seconds (time limit is minutes).
Response format:
Your response should be a brief outline/sketch your proposed solution in natural language (3-5 sentences), followed by a single markdown code block (wrapped in "** which implements this solution and if it's machine learning task, then prints out the evaluation metrie. There should be no additional headings or text in your response, Just natural language followed newline and then markdown code black.
Bugfix improvement sketch guideline: You should write brief natural language description (3-5 sentences) of how the issue in the previous implementation can fixed.
The data is already prepared and available in the '/input' directory. There is no need to unzip any files. For any provided file path, you should replace it with '/input' plus the file name and ignore other dictionary or subdirectionary name in the path.
Implementation guideline:
The code should immplement the proposed *o If the task is a machine learning task, you should #*print the value of the evaluation metric computed on hold-out validation set The code should be single-file python program that is self-contained and can be executed as-is.
No parts the code should skipped, don't terminate before finishing the script. Your response should only contain a single code block.
Be aware of the running time of the code, it should complete within 15 minutes. All the provided input data is stored in '//nput' directory.
*fIf there is test data provided for this task, please the predictions in 'submission.csv" file in "/working" directory as described in task d This extremely important since this file is used for grading/evaluatte You can also use the '/working' directory to store any temporary files that your code needs to create. The evaluation should be based 5-fold mooss-vllidttion but only if that's appropriate evaluation for the task hand.
DO NOT FORGET THE submission.esv file!
Data Overview: (
17

---

Published as a conference paper at ICLR 2026
Self-Improvement Prompt
You scientific coding expert write code based the task description. You provided with previously developed solution below and should improve it order to further increase the performance. For this you should first outline brief plan in natural language for how the solution can be improved and then implement this improvement in Python based on the provided previous solution.
Task goal: You are an expert Python programming assistant that helps scientist users write high- quality code solve their tasks.
Given a user request, you are expected to write a complete program that accomplishes the requested task and save any outputs in the correct format. Please wrap your program in code block that specifies the script type: python, For example:
"ppyhhnn print("Hello World!")
Please keep your response concise and do code block if it's intended be exccuted. Please do not suggest few line changes, incomplete program outline, or partial code that requires the user to modify. Please do not use any interactive Python commands in your program, such as "lpip install numpy', which will cause execution emror..
Here is the user request you need to work on: (task description)
Memory: (previous <plan, code, results>)
Response format:
Your response should be hrief outline/sketch your proposed solution natural language (3-5 sentences), followed by single markdown code block (wrapped in >*)) which implements this solution and if it's machine learning task. then prints out the evaluation metric. There should be additional headings or text in your response. natural language text followed by newline then the markdown code block.
Solution improvement sketch guideline:
The solution sketch should brief natural language description how the previous solution can be improved.
You should be very specific and should only propose a single actionable improvement.
This improvement should be atomic that we can experimentally evaluate the effect of the proposed change.
Take Memory section into consideration when proposing the improvement.
The solution sketch should be 3-5 sentences. The data is already prepared and available in the '/input' directory. There is no need to unzip any files. For any provided file path, you should replace it with "/input'' plus the file name and ignore other dictionary or subdirectionary name in the path.
Implementation guideline:
The code should simplement the proposed solution If the task is machine learning task, you should ** **print the value of the evaluation metric computed on hold-out validation set**, single-file python program that self-contained and can be exccuted as-is.
The code should No parts of the code should skipped, don't terminate before finishing the script. Your response should only contain a single code black.
Be aware of the running time of the code, it should complete within 15 minutes. All the provided input data is stored in '/input' directory. **If there is test data provided this task, please save the test predictions 'submission.csv' the "/working' directory as described in the task description*** This extremely important since this file is used for grading/evaluation You can also use the '/working' directory to store any temporary files that your code needs to create. The evaluation should be hased 5-fold cross-validation but only if that's an appropriate evaluation for the task at hand.
DO NOT FORGET THE submission.csv file!
Previous solution: (program)
18

---

Published as a conference paper at ICLR 2026
Feedback Prompt
are scientific coding expert write based on description. You written code to solve this task and now need to evaluate the output of the code exccution. You should determine if there were any bugs as well as report the empirical findings.
Task goal: You
write high- quality code solve their tasks. Given user request, you are expected to write complete program that accomplishes the requested task and save any outputs in the correct format. Please wrap your program in code block that specifies the script type: python. For example:
expert Python programming assistant that helps scientist
python
print("Hello ("Hello World!")
Please keep your response concise and do code block if is not intended to be executed. Please do not suggest few line changes, incomplete program outline, partial code that requires the user to modify. Please do not use any interactive Python commands in your program, such as "lpip install numpy', which will cause execution errors.
Here user request you need to work on: [lask description)
Implementation: [program]
Execution output:
Analysis complete. Figure saved to predd pallecular_analsiiie Execution time: 13 seconds seconds (time limit is 15 minutes).
Relative Judgments Prompt
Please act impartial judge and evaluate the quality of the code responses provided by two Al assistants the programming question displayed below.
You will be given assistant A's answer, and assistant B's answer. Your job is to evaluate which assistant's answer is better. Avoid any position biases and ensure that the order in which the responses presented does not influence your decision. Do allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be objective possible. After providing a fine-grained analysis
differences between two code responses, output your final verdict and score by strictly following this format: Rating A: [[1-10]]
Rating B: [[1-10]]
Better: [[A or B]]
[Question] ('Tas description)
[The start of Assistant A's RESPONSE]|responss of assistant [] end of Assistant A's RESPONSE]
[The start of Assistant B's RESPONSE] of assistant B) [Tee end of Assistant B's RESPONSE]
You need to the following output format: ****
Explanation: Here an explanation.
Rating A: [[1-10]]
Rating B: [[1-10]]
Better: [[A B]] #:t:
19

---

Published as a conference paper at ICLR 2026
LLM-Absolute Prompt
Task description: ()
Response summary: {}
The generated code program is as following: {program}
Please generate between to 100 indicate how good and suitable the generated code matches request.
You can have the explanation, reasoning or analysis, but please explicitly generate the score using the format S*ore:: ***, e.g.. *"Coore 37** in your response.
Rubric-Absolute Prompt
Task description: ()
Response summary: {}
The generated code program as following: [program]
The grading rubric is provided as JSON: (rubric description)
Please generate a score between 0 to 100 based on the given grading rubric (criteria) indicate how good and suitable the code solution matches the request.
You can have the explanation, reasoning or analysis, but please explicitly generate the score using the format Score: *t,, e.g., Score: 37 in your response.
20


Reviews:
[{"review_id": "MAULRrJWPJ", "reviewer": "Reviewer_2cow", "summary": "This paper introduces SciNav (Scientific Navigator), a framework for autonomous science agents designed to tackle scientific coding tasks. The core contribution is the Top-K Comparative Tree Search (TKCTS) algorithm, which replaces absolute scoring with pairwise relative judgments during solution exploration. SciNav integrates several components: initial multi-plan generation, self-debug, iterative self-improvement, and a frontier comparator based on relative LLM judgments, to progressively refine code solutions under constrained computational budgets.\n\nExperiments on ScienceAgentBench and DA-Code show that SciNav performs best compared to baselines such as OpenHands and Self-Debug. Ablations also show that relative comparison helps compared to random selection and absolute scoring.", "questions": "What is the K in top-K and the comparison budget for the experiments?\nHow does changing this impact the performance? Given the main contribution is the comparator it would be helpful to see how much performance is impacted by these hyperparamter choices.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "S1. The relative judgment–guided Top-K search is a well-motivated methodological idea that builds on prior insights about the reliability of pairwise evaluation and applied in an agentic setting.\n\nS2. The experiments are reasonable, covering two benchmarks, several LLM backbones, and detailed component ablations. The experiments for the contributions of each component, including initial plan diversity, self-improvement, and the comparator strategy are appreciated.", "weaknesses": "While the results and experiments are good, my main concerns center around how much we can interpret from them which I'm happy to change with some clarification.\n\nFirst, the paper does not report error bars or statistical significance. This makes it hard to assess whether observed performance differences are meaningful or consistent across runs.\n\nSecond, it is important, especially when we consider deployment to also compare the cost of each agent/ablation involved. How many extra LLM calls/tokens are used for SciNav vs. Self-debug to obtain the performance increases? What is the cost of inference time or $ cost to have extra LLM calls? \n\nWithout this information, it is hard to assess the tradeoff/value of SciNav. For instance, how much of this performance gain is just a result of scaling test-time compute.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J7Vh77J3xh", "reviewer": "Reviewer_2AeN", "summary": "In this paper, the authors focus on improving scientific coding agents by proposing SciNav, a framework that treats problem-solving as a structured search guided by relative evaluation. They introduce Top-K Comparative Tree Search (TKCTS), which allows the agent to explore, compare, and refine code solutions through pairwise relative judgments rather than absolute scoring. SciNav integrates components for planning, self-debugging, self-improvement, and frontier selection using an Elo-based ranking mechanism. The authors evaluate SciNav on scientific coding benchmarks to demonstrate that this principled, comparison-driven approach leads to more effective and reliable solution generation than the baseline agents.", "questions": "* Was a cost or runtime comparison performed to quantify the additional computation introduced by pairwise judgments and iterative search relative to baselines?\n* The TKCTS relies on relative judgments by an LLM-as-judge. How consistent are these judgments across multiple runs or judging models? Would cross-model evaluation (e.g., using a different LLM as the judge) yield stable rankings?\n* The framework uses a fixed budget (five initial solutions, three debug steps, ten total exploration steps). Why were these values chosen, and have the authors tested sensitivity to these parameters?\n* Given that relative judgments guide the search process, was any validation (e.g., human evaluation or ground-truth correctness checks) performed to verify that the LLM-judge’s preferences align with actual code quality?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper is well-motivated and clearly defines the need for principled frameworks for scientific coding tasks with verifiable outputs.\n\n(2) It presents a structured search method combining relative judgments and iterative refinement, supported by consistent quantitative improvements over existing agent baselines.", "weaknesses": "* Evaluation is limited to two controlled benchmarks, leaving uncertainty about generalization to real-world or open-ended scientific tasks.\n* Reliance on LLM-as-judge comparisons may introduce bias, as the same models both generate and evaluate solutions.\n* The fixed and narrow search budget restricts exploration, and scalability to more complex tasks remains unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z94W05wcpO", "reviewer": "Reviewer_Eeg5", "summary": "The authors propose and evaluate SciNav, a \"framework\" (agent) that performs scientific coding. The agent performs search taking into account a constrained budget, and uses pairwise (comparative) jugements rather than absolute judgements to guide the search. They show SciNav outperforms prior agents like OpenHands.", "questions": "- A common approach in other agent-based coding tasks is to use genetic algorithms to merge different coding ideas (e.g., AIScientist), rather than expanding a single parent. It'd be nice to know how your \"single parent\" approach would compare. Do you have any intuitions about this?\n - It seems that your contribution is more proposing and evaluating TKCTS as a great framework for agent coding, rather than the (narrower) use of comparative judgements. Is that a reasonable reframing of the contribution?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The empirical result of outperforming OpenHands and Self-Debug is quite compelling\n - nice to see search budgets taken into account in the framework\n - nice to see you leveraging existing benchmarks rather than creating a new one\n - ablation that suggests relative judgements are helping (a little bit)", "weaknesses": "- Gains are somewhat modest (~2%-3%), so the impact of the work seems a little limited\n - Comparison with genetic algorithm approaches to coding (e.g., in AI Scientist) would be useful \n\nMinor:\n - Abstract takes way to long too get to the goal and contribution - should be stated in first or second sentence. (The abstract gives the impression at first you're going to propose a benchmark)\n - Would be worth expanding on use of relative judgements in AI, e.g., it's the basis of A/B testing, preference optimization (e.g., DPO), and other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hVZA51yqV8", "reviewer": "Reviewer_9HVs", "summary": "The paper proposes SciNav, an agent for scientific coding problems that frames solution search as a Top-K Comparative Tree Search (TKCTS). Instead of relying on absolute LLM scores or task-specific metrics during exploration, SciNav repeatedly performs pairwise (relative) judgments among candidate programs, prunes low-potential branches, and refines the top-K trajectories via self-debug and self-improvement loops. The system is evaluated on ScienceAgentBench and DA-Code, claiming consistent gains over Direct Prompting, Self-Debug, and OpenHands, with ablations suggesting relative judgments beat random or absolute scoring for frontier selection.", "questions": "1. Compute parity:\nCan you report tokens, wall-clock time, and $ cost per task for each method? Do results hold under strict compute-budget matching?\n\n2. Judge/model decoupling:\nDid you evaluate cross-model judging (e.g., Claude evaluates GPT solutions and vice versa)? If not, please include — this is crucial to rule out style bias.\n\n3. Pair selection policy:\nHow exactly are candidate pairs chosen for comparison? Uniform random? Score-based? Uncertainty-based? Please add an ablation isolating this choice.\n\n4. Comparator stability:\nHow sensitive performance is to the Elo update parameters and number of comparison calls? A plot of success-rate vs. comparison budget would help.\n\n5. DA-Code sampling clarity:\nHow were the 100 DA-Code tasks selected and stratified? Please release task IDs and sampling seeds to enable exact replication.\n\n6. Hybrid signal experiment:\nHave you tested combining pairwise judgments with cheap verification signals (static checks, partial tests)? This seems easy to add and directly addresses observed failure modes.\n\n7. Baseline strengthening:\nCan you add tournament-selection, majority-vote ranking over best-of-N, or MCTS-style search? If not, please justify why these are not relevant or already covered by TKCTS.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Clear problem focus: Targets scientific coding where outputs are executable and evaluable, avoiding the fuzziness of “end-to-end science agents.” \n\n2. Methodical framing: TKCTS with self-debug/self-improve is a clean, modular agent design; Algorithm 1 is easy to follow and implement. \n\n3. Relative judgments: Sensible use of pairwise comparisons; the frontier-comparator ablation shows consistent benefits over random and absolute scoring. \n\n4. Cross-dataset evaluation: Evidence on ScienceAgentBench and DA-Code, with task-type and difficulty breakdowns and an informative error analysis.", "weaknesses": "1. Compute fairness & unclear budgets.\nThe paper fixes step counts but does not report token, runtime, or dollar budgets across methods. Without normalized compute, it’s unclear if performance gains stem from the algorithm or simply more compute. Reporting tokens / wall-clock / $ per task and re-running under a fixed compute budget is necessary.\n\n2. Small absolute gains and low success-rate regime.\nImprovements are modest (e.g., \\~2–4% absolute SR gains) and overall SR remains low (\\~15–19%), as seen in Table 2. The practical significance is unclear without confidence intervals, per-task breakdowns, or hypothesis testing.\n\n3. LLM-as-judge bias / circularity risk.\nIf the same model family both generates and judges, relative scoring may reflect stylistic familiarity rather than correctness signal. This risks overestimating benefit of pairwise comparison. Cross-model judging or position-randomization baselines are missing.\n\n4. Baselines not fully representative.\nThe paper omits competitive selection/reranking baselines such as tournament selection over best-of-N, MCTS with learned value, or verification-guided heuristics. These are relevant comparisons that could challenge the novelty claim.\n\n5. Sparse statistical reporting.\nOnly 2–3 runs per setting, no reported CIs, no significance tests, and unclear sampling protocol for DA-Code. More rigorous variance reporting is needed, especially given the stochastic nature of LLM benchmarking.\n\n6. Under-leveraged partial execution signals.\nThe paper emphasizes “no task-specific metric at run time,” but many tasks allow cheap checks (import/compile, partial test subsets, lints). A hybrid relative-judgment + lightweight execution signal could materially improve the agent — and the omission feels like an avoidable limitation rather than a principled choice.\n\n7. Reproducibility gap.\nPrompts are included, but code is “will be released upon acceptance.” Given the importance of queueing, frontier selection, and Elo parameters, anonymized code or pseudocode for comparator internals would improve credibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DLgVpVsIsB", "reviewer": "Reviewer_2Rev", "summary": "This study introduces SciNav, an LLM-based agent framework designed for scientific coding tasks. SciNav employs Top-K Comparative Tree Search to address these tasks under constrained computational budgets. Experimental results demonstrate that SciNav surpasses baseline methods, achieving up to a 24% increase in success rate and a 7.8-point absolute improvement in valid execution rate.", "questions": "Please see comments above. One additional question: can scientific coding serve as a valid representation of scientific reasoning tasks? How does the proposed system enhance scientific reasoning in open-ended environments?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-\tThis paper addresses a clearly defined and underexplored problem: scientific coding tasks, which connect scientific reasoning with executable code generation and allow for objective evaluation.\n-\tThe proposed Top-K Comparative Tree Search introduces a novel use of relative LLM judgments for iterative code refinement. It shows consistent empirical improvements over strong baselines.\n-\tThe paper is well-organized with detailed component analysis, ablation studies, and transparent experimental setup across multiple benchmarks and LLMs.", "weaknesses": "-\tThe primary concern with this study is the gap between the authors’ claims and their actual contributions: the paper frames SciNav as a general “principled agent framework,” but the work focuses narrowly on code-generation heuristics without formal theoretical grounding or demonstration of broader scientific reasoning.\n-\tThe use of the term “principled” in the title and narrative is inappropriate, as the method is not derived from explicit first principles or formal justification. Currently it is a structured heuristic rather than a principled framework in the scientific sense. I highly recommend the authors to use more appropriate terms.\n-\tThe paper lacks quantitative analysis of computational cost and statistical significance of performance gains brought by the proposed method, making claims of efficiency and reliability only partially supported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)