NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents
Authors: Tianshi Zheng
Kelvin Kiu Wai Tam
Newt Nguyen Kim Hue Nam
Baixuan Xu
Zhaowei Wang
Cheng Jiayang
Hong Ting Tsang
Weiqi Wang
Jiaxin Bai
Tianqing Fang
Yangqiu Song
Ginny Wong
Simon See
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
benchmark
virtual environment
generalization
agent
scientific law discovery
Primary area: datasets and benchmarks
TL;DR: We introduce NewtonBench, a benchmark for evaluating LLMs’ scientific law discovery via interactive, generalizable experiments.

Abstract: Large language models (LLMs) are emerging as powerful tools for scientific law discovery, a foundational challenge in AI-driven science.
However, existing benchmarks for this task suffer from a fundamental methodological trilemma, forcing a trade-off between scientific relevance, scalability, and resistance to memorization. Furthermore, they oversimplify discovery as static function fitting, failing to capture the authentic scientific process of uncovering embedded laws through the interactive exploration of complex model systems. To address these critical gaps, we introduce **NewtonBench**, a benchmark comprising 324 scientific law discovery tasks across 12 physics domains. Our design mitigates the evaluation trilemma by using counterfactual law shifts - systematic alterations of canonical laws - to generate a vast suite of problems that are scalable, scientifically relevant, and memorization-resistant.
Moreover, we elevate the evaluation from static function fitting to interactive model discovery, requiring agents to experimentally probe simulated complex systems to uncover hidden principles. Our extensive evaluation of 11 state-of-the-art LLMs reveals a clear but fragile capability for discovery in frontier models: this ability degrades precipitously with increasing system complexity and exhibits extreme sensitivity to observational noise. Notably, we uncover a paradoxical effect of tool assistance: providing a code interpreter can hinder more capable models by inducing a premature shift from exploration to exploitation, causing them to satisfice on suboptimal solutions. These results demonstrate that robust, generalizable discovery in complex, interactive environments remains the core challenge for the future of automated science. By providing a scalable, robust, and scientifically authentic testbed, NewtonBench offers a crucial tool for measuring true progress and guiding the development of next-generation AI agents capable of genuine scientific discovery.

Published as a conference paper at ICLR 2026
NEWTONBENCH: BENCHMARKING GENERALIZABLE SCIENTIFIC LAW DISCOVERY IN LLM AGENTS
Tianshi Zheng"), Kelvin Kiu-Wai Tam"l Newt Hue-Nam K. Nguyen" Baixuan Xu', Zhaowei Wang', Jiayang Cheng', Hong Ting Tsang', Weiqi Wang' Jiaxin Bai', Tianqing Fang, Yangqiu Song"', Ginny Y. Wong", Simon See"
'The Hong Kong University of Science and Technology, ?NVIDIA (tzhengad, kwtamai, <hanguyyeJ/cconnect. ust.eecctt.............. yqsongåcse .uteesseeess............. .h {gwong, ssee ssee @avidia com
Code and Data:
ABSTRACT
Large language models (LLMs) are emerging powerful tools for scientific law discovery, a foundational challenge in Al-driven science. However, exist- ing benchmarks for this task suffer from fundamental methodological trilemma, foreing trade-off between scientific relevance, scalability, and resistance mem- orization. Furthermore, they oversimplify discovery static function fitting, failing to capture the authentic scientific process of uncovering embedded laws through the interactive exploration of complex model systems. To address these critical gaps, we introduce NEWTONBENCH, a benchmark comprising 324 sci- entific law discovery tasks across 12 physics domains. Our design mitigates the evaluation trilemma by using counterfactual law shifts-systematie alterations of canonical laws-to generate vast suite of problems that scalable, scientifi- cally relevant, and memorization-reeiitane from static function fitting to interactive model discovery, requiring agents ex- perimentally probe simulated complex systems to uncover hidden principles. Our extensive evaluation of state-of-the-arr L.LMs reveals clear but fragile capa- bility for discovery in frontier models: this ability degrades precipitously with increasing system complexity and exhibits extreme sensitivity to observational noise. Notably, uncover paradoxical effect of tool assistance: providing a code interpreter can hinder more capable models by inducing a premature shift from exploration exploitation, causing them satisfice suboptimal solu- tions. These results demonstrate that robust, generalizable discovery in complex, interactive environments remains the core challenge for the future of automated science. By providing a scalable, robust, and scientifically authentic testbed, NEWTONBENCH offers a crucial tool for measuring true progress and guiding the development of next sekeraeneration Al agents capable of genuine scientific discovery.
Moreover, we elevate the evaluation
INTRODUCTION
Scientific law discovery, the process of distilling complex natural phenomena into concise, pre- dictive mathematical principles, represents a cornerstone of human intellectual achievement, having consistently catalyzed paradigm shifts across the sciences (Langley et al. 1987 Popper 2002). This endeavor demands sophisticated interplay of abilities central to the scientific method: formulat- ing hypotheses, methodical experimentation, and synthesizing empirical findings into universal laws
(Peirce 1958 Schmidt & Lipson 2009).
The rapid advancement of Large Language Models (LLMs)) has brought this grand challenge into sharp focus (OpenAI 20244 Google 2025). Exhibiting remarkable emergent capabiliiiess in math- ematical reasoning (Ahn et al.] 2024). logical reasoning (Yu et al.) 2024). and agentic planning (Huang et al. 2024b), these models appear to possess foundational skills for scientific inquiry. This
"Equal Contribution.
1 "ooresponding Author.

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of benchmarks for scientific law discovery. The "Sci. Rel." column (scientific relevance) indicates if problems are derived from real-world scientific laws (versus being synthetic). The "Mem--Free column indicates whether the benchmark fully prevents memorization.
Benchmark
# Problems Diseovery Scope Sei. Rel.
Mem..Free Sealable
Diffieuity
Data Acquisitinn
Traditional World Laws
Foynman (Udrosa & Teunark
2020
120 y
Function Hunction
X x x
Partially Tumable Fixed
Passive Otservation
Empiricallennan Cramin
21125
X
Passive Observation
Tnd-of-Durribuii Laws
LLM-SR (Shuface et
4
Function Function
X
Fixed Fixed
Passive Observation
EvoSLD Lan at ml.
S
X
Passive Otservation
Transformmed Lovs
LSR Transform (Shojare al. 2025b
111
Function
x
V
Fixed
Passive Observation
Synthetic
LSR-Synhh (Shojcee et al. 2025h
128
Function
x
5
3
Fixed
Passive Observation
PhysSymbol [au Sian
5,000
Function
Tunable
Passive Observation
NEWTONBENCH (Ours)
324
Model System
/
/
/
Multi-dimensiomal Active Exploration
has sparked widespread curiosity (Armitage) 2025 Fang et al., 2025), crystallized by provocative questions such as whether an LLM could "rediscover Newton's 's laws from ils own intelligence" (He) 2023). However, answering such questions requires moving beyond anecdotal observations. To foster and guide progress toward automated science, it is critical to develop a robust testbed to rigorously benchmark the true abilities and inherent limitations of current LLMs in this domain.
Nevertheless, our closer inspection reveals that existing benchmarks for LLM-driven scientific law discovery exhibit substantive shortcomings that prevent a robust evaluation (as illustrated in Ta- ble D. Current approaches fall into three paradigms, each with critical limitation. First, the Dfitt--Distiiuttin (OOD) Laws paradigm uses complex, lesser-known principles to ensure rel- evance and memorization resistance, but their scarcity makes it impractical to scale. Second, the Transformed Laws paradigm provides scalability by rewriting known principles, yet remains vul- nerable to memorization, as models may simply recall the original law rather than reason from the provided data (as evidenced in Appendix E.II. Finally, the Synthetic Laws paradigm achieves maximum scalability and inherently memorization- free, but the cost of scientific relevance, as relies on algorithmically generated equations that lack correspondence to real-world phenomena. Collectively, these paradigms expose fundamental methodological trilemma: a forced trade-off between scientific relevance, scalability, and resistance to memorization.
While the trilemma highlights the trade-offs between existing paradigms, a more fundamental limi- tation underlies them all: they frame scientific discovery as a problem of static function discovery, where the goal is simply to fit a mathematical formula to variables in a tabular dataset. This static framing is stark departure from the authentic scientific process, which relies on the interactive exploration of complex model systems (Cartwright 1983 de Silva et al. 2020) for instance, J.J. Thomson's 's inference of the e/m ratio from cathode ray manipulation. Therefore, to meaning- fully benchmark L. LLMs, evaluations must move beyond static function fitting and embrace the more scientifically authentic challenge of model discovery in interactive environments (Fang al. 2025).
To address these limitations and facilitate a more rigorous evaluation, we introduce NEWTON- BENCH. Our design built on two core principles, each targeting a fundamental shortcoming in prior work. First, to resolve the methodological trilemma, we introduce the counterfactual law shift-a technique that systematically alters the mathematical structure of canonical physical laws, for instance, by modifying operators or exponents (see examples in Figure D). This generates a vast suite of laws that are conceptually grounded but physically novel. This approach simultaneously achieves the scalability of synthetic laws, maintains the scientific relevance of real-world princi- ples, and ensures resistance to memorization by creating problems that cannot be solved by recall, thus forcing models to reason from first principles.
Complementing this approach, our second principle tackles the limitation of static function discov- ery. To this end, we designed NEWTONB ENCH as an interactive, system-oriented environment. Rather than fitting a formula to a pre-existing table, an agent must actively design experiments by specifying imput parameters (e.g., setting the mass of an object the initial velocity in simulation) and interpreting the resulting feedback from the virtual environment. Critically, the target law is embedded within complex model with confounding variables. This design compels the agent to
2

---

Published as a conference paper at ICLR 2026
engage in the interactive exploration of complex model system, elevating the core challenge from simple function fitting to the more scientifically representative task of model discovery.
Furthermore, NEWTONBENCH is designed to precisely probe the limits of an LLM's scientific capa- bilities. It features two independent dimensions of difficulty control: the intrinsic complexity of the target law (tuned via the counterfactual law shift) and the extrinsic complexity of the surrounding experimental system. This allows for a fine-graine mne-grained analysis of a model's 's breaking points. To isolate the challenge of discovery itself, we introduce an advanced setting where the LLM is provided with a code execution interface for tasks like numerical regression or hypothesis testing. By removing computational limitations primary bottleneck, this setup is designed to push models from being merely compute-bon buuud to being truly discovery-bound. revealing the genuine frontiers of their scientific reasoning abilities.
Experimental results from 11 state-of-the-art LLMs on NEWTONBENCH reveal clear but frag- ile capability for scientific law discovery. While frontier models like GPT-5 and Gemini-2.5-pr demonstrate proficiency in simple, isolated systems, their performance degrades precipitously as the intrinsic complexity of the law or the extrinsic complexity of the experimental system increases. This fragility is further underscored by extreme sensitivity to observational noise, where even minimal perturbations cause a sharp decline symbolic accuracy. Furthermore, our analysis un- covers paradoxical effect of code assistance: providing code interpreter boosts weaker models by offloading computation, but paradoxically hinders stronger models. We trace this performance degradation premature shift from exploration exploitation, where capable agents over-rely on the tool for local optimization at the expense of discovering the globally correct law. Collectively, these results demonstrate that while LLMs are beginning to develop foundational scientific reason- ing skills, achieving robust, generalizable discovery in complex, interactive environments remains the core challenge for the future of automated science.
TASK FORMALIZATION
We formalize the discovery task in NEWTONBENCH around two core concepts: Equation and Model. An agent is required to discover a target equation by experimenting within a given model. Definition 1 (Equation). An equation is treated as a mathematical expression that acts as function. mapping set of input variables to single numerical output. It is structurally represented by an Expression Tree T, a form of Abstract Syntax Tree (AST).
The nodes of T are categorized as follows: 1) Leaf Nodes Ny. These represent the operands of the equation. They can be either input variables v E V (e.g., mass, distance) or numeric literals E C (e.g.. coefficients, known constants). 2) Internal Nodes Ny: These represent mathematical operators o E O that are applicd to their child nodes. The operators can be unary (e.g., sin(z). vx) or binary (e.g., + y. I X y). The complete set of operators O used in NEWTONBENCH is detailed in Appendix A.322 The evaluation of an equation f for a given assignment of values to its input variables V is performed via a post-order traversal of its expression tree T. This recursive evaluation process is formally described in Appendix A.3.
Definition (Model). A model M represents an experimental system and is defined as an ordered sequence of k equations, M = ff1..2......e)) fk). The model takes a set of lesell input vari- ables VM and produces a set of final outputs Y.M-
The computation within M proceeds sequentially through its ordered equations. The inputs for any equation fi can be drawn from the model's primary inputs VMM or the outputs of any preceding equations {v; j < i). The model's final output, V/M is designated subset of the collected outputs from all equations, i.e., VM C (y1....,y1.
Based these definitions, in our benchmark, one equation furgat is designated as the hidden physical law for the agent to discover. All other equations the given model, forming the set of assisting {furge)). The physical laws in assisting equations are provided
equations Fasist = {fi,.... ... fie) as known information via prompt to foster discovery.
The task's 's complexity is determined by the structure of M (illustrated in Figure Ip. which we define across three parallel settings. The simplest setting is Vanilla Equation, where the model contains only the target law (M- (Junge))) The challenge escalates in the Simple System and Complex
3

---

Published as a conference paper at ICLR 2026
Experimentation Complexity
Physical Law Curation
Vanilla Equation
Expression
Counterfactual
difficulty
Orignal
Big Wiggg
I3ON
Bay
I G
My 1il
M
F (Wty, Way)
Simple System
NewtonBench
Agentic Exploration
Physics Domains
N J0ly
Domney
VENNCUAEE Tochare
Complex System
Cam
Foree
Cer
M
Soaft
Redowatte Deney
Horreek Metee
Make Lom
Pea'L Suberationn
Law
Dianhvtton
Suesder
ENoY
Figure 1: An illustration of core designs in NEWTONBENCI: experimentation with model system, counterfactual shifts in physical laws from various domains, and agentic exploration settings.
System settings, where furget is embedded within a larger model (I: 1). In these more advanced model discovery tasks, the agent must leverage assisting equations to disentangle confounding vari- ables and uncover the target law. Details on system configurations are provided in Appendix A.2 The solvability proof of our task settings is provided in Appendix E.2
3 NEWTONBENCH
In this section, we introduce the construction of NEWTONBENCH, detailing its three core compo- nents: physical law curation (33.T). the virtual environment ($3.2). and the evaluation metrics ($3.3).
with full implementation details provided in Appendix
3.1 PHYSICAL LAW CURATION
Our benchmark's 's physical law curation process involves two primary stages: first, the collection of canonical seed laws, and second, the application of Counterfactual Law Shifts generate novel variations. For the first stage, the selection of our seed laws is guided by three criteria: they must be foundational to their respective fields, span diverse range of physics domains, and, critically, each must allow for dimensionally consistent mutations (often via adjusting physical constants). The second stage employs Counterfactual Law chifts-anallgouus to counterfactual reasoning in physics (Elwenspock] 2012) to systematically alter these canonical laws. This approach prevents agents from solving tasks by memorizing known laws, effectively creating a novel hypothetical universe for the agent to discover in each task.
Operationally, these shifts are implemented as mutation operations (ooza 1994) on the equation's expression tree. These mutations can alter either the mathematical operators (e.g., changing an addition to a multiplication) or the values of numeric constants (e.g.. modifying an exponent to change quadratic relationship to cubic one). A critical consequence of such alterations is the potential disruption of dimensional consistency. To preserve the physical coherence of the modified equation, this disruption is offset by systematically adjusting the dimensional units of the embedded

---

Published as a conference paper at ICLR 2026
physical constant. This necessity dictates that every target law in our benchmark includes least one such constant to serve this compensatory role.
Three levels of equation difficulty categorized based the cumulative extent of mutations applied to a canonical lav["] Easy laws are generated via 1-2 mutations from the original law. Medium laws subsequently derived from Easy laws through 1-2 additional mutations, and Hard laws are similarly derived from Medium laws. We curated 108 shifted laws from 12 canonical laws under strict guidelines ensure scientific plausibility, conducted by three domain-expert co-authors. The size of our shifted laws is comparable to existing synthetic benchmarks (Shojace et al. (025b). Appendix A.3.3 provides the complete set of laws, their mutated forms, and the curation guidelines.
3.2 VIRTUAL ENVIRONMENT & TASKS
The LLM agent interacts with this virtual environment through two primary interfaces: an experi- mental tool to probe the physical model system and a code interpreter to perform complex computa- tions. Each of the 108 shifted laws is instantiated in three model systems (Vanilla Equation, Simple System, Complex System), yielding 324 total tasks.
Experimental Tool A core design principle of our benchmark is the shift from passive data ob- servation active agentic exploration. We formalize this by framing the experimental system, or model M. as an interactive black-box tool. Instead of analyzing static dataset, the agent must strategically probe M to gather evidence about the hidden target equation farget-
The agent interacts with M by invoking a <run eeneeeieemmmmtt tool. This interaction follows a precise protocol: 1) The agent proposes specific assignment of values for the model's input variables. VM- 2) The environment evaluates the full model M with these inputs and returns the corresponding set final model outputs, V/M-
By iteratively executing these experiments, the agent builds a dataset of input-output pairs to deduce the structure of furget- This task non-trivial, especially in the Simple and Complex System settings, as the agent must leverage the provided assisting equations (Fasiis) to reason about the model's internal state and isolate the behavior of farget. To facilitate this process, we provide instructional scaffolding in the initial prompt regarding tool syntax and objectives (detailed in Appendix C while reserving the experimental strategy entirely for the agent's 's reasoning capabilities.
Code Assistance While LLMs excel at symbolic reasoning. their native computational capabilities can be bottleneck for precisely determining numerical constants or fitting complex functional forms, particularly for equations involving exponential or logarithmic operations. To ensure our benchmark evaluates scientific discovery rather than raw computational prowess, we introduce an optional Code Assistance setting.
In this configuration, agents provided with code interpreter tool, invoked via <python> tags. The agent can write and execute arbitrary Python code and receives the output from the execution en- vironment. Crucially, the agent must autonomously decide how and when leverage this tool-for instance, to perform numerical computation, regression, or verify a hypothesis. By providing this computational offloading capability, we aim to shift the evaluation from being compute-bound to discovery-boue -ound, thereby better isolating the agent's scientific reasoning abilities.
3.3 EVALUATION METRICS
We evaluate agent performance using two metrics: Symbolic Accuracy to assess the correctness of the discovered equation's 's structure, and Root Mean Squared Logarithmic Error (RMSLE) to quantify its predictive fidelity to the data.
Symbolic Accuracy Following standard practice scientific law discovery, Symbolic Accuracy a binary metric that verifies if discovered equation is mathematically equivalent to the ground- truth law Sunget- The equivalence check intentionally disregards the values of physical constants, as
Consequently, equation difficulty arises from two factors: equation complexity and the level of distribu- tional shift.
5

---

Published as a conference paper at ICLR 2026
Table 2: Experiment results aggregated over 12 domains by equation difficulty (easy, medium, hard) and system complexity, reported as mean and standard deviation from four runs. Within each agent configuration, the highest score is in bold and the gecond-highes is underlined.
Yanilla Equation
Simple System
Comples Spsiem
Average
LLM Agents
modiav
Acai
Anse
Wedian
Aard
SACRIT
RMSLKS
#bikens
Agent
GPT-4.-aiii
20.1 sucier
7.6 caluets
2.5 Jazaes
5.3 1an.3M
2.8 Mizale
0.0 (aso))
4.2 (4.00))
[L0 (AA00)
0.0 1al000
5.1 14l7ity 4.3532
1.98
GPT-411
16.7
14
12.5
211
0.7
3:5241
ot-misi
78.5
22.2 w1200
47.2
2.8 (4:000
1.2373
GPPI:--mini
19.6
PM
19.0 van
60.4
78.5 (9)
96.0
26.4 04.011
500 VMN
79. 3070M
4.9
53.1 0100
0,7193
onie
9:41
GPTS
90.3
92.4
0.2490
DeepSoni-VS
19.6 101
12.5
2.1 60700
181 04000
0.7 M0M0
0.0 (1000
6.9 (0700)
0.01 (1600)))
0.0 (100
8.9
3,0075
1.59
DeepSeck Ri
88.2 14:0
72.9
56.8
75.0 (10
41.7 (4:00)
12.5
40.3 (0700
20.1 10
2.8
43.4 suaa
1,3639 sautt
16.45
QuQ-379
(530)
Qwen3 215B
$1.9 wu0.
50.4 241
23.6 (el 7021
51.4 (un
25.0 0440000
1.4 u0.000
25.5
(4.500
4.9
0.7 1ul0
30,9
1,9385
13.91
Cemini-2 S-feh
40.3
7V-0
21.6
41.0
76.4
5.6
50.6
0.7904
51.11
Genini 25-pr0
96.5 uara
56.8
00.4 500000
56.1
76.4 m222
41.2 1000
47.2 W000
16.7 MA20e
65.4
0.3535
21.54
Agunt with Cade Ausimance
1PP--.1-mini
15.4 perser
14.3
16.7
18.8
11.1
6.9
13.2
566
0.2
14.7
3,7830
7.92
GPT41
31.9 (4:
29.2
10.4 (eatt)
1322 (400
2.1 141:00
sarTa 2.5044 GazzA
ok-mini
42.5 me
10.8 m4
47.0
67
52.0 04024
15.3 0v.
41.1
5.6 (+1000
45.1 poans 1,7078 evem
10.26
GPTS
73.6
75.0
0.7412
103 mi0c
50.6
18.2
(11770
14.3
38.2 14110
23.3 yorea 0,3793 evxmm
18.35
DeepSeck V3
4588 66.
26.4
9:0
17.4
5.6 M0000
1.4
11.1 62:20
2.1 122000
11.7 1.::000
13.3 saait 2,8795 sautt
2.23
Deppaas-Rtt
QxQ.318
11.5 W5:12
59.0 26)
25.7 04 560
52.8 (71
32.6 (4:227
2.1 01.11
27.1 02.00
13.3 A0000
0.7 1.1000
31.4
suasi 2,0599 souda
14,62
Qwan1-215B
75.7
54.2
27.1
51.8
$4.7
10.4
26.4
13.2
0.0
33.7
1.9420
13.30
Gemini 25 fush
60.00
69.4 021172
62:20
(400))
40.9 w0a0
Ceniii-2.Sppo
10.3 (nt.
$5.4
66.2 (n 20
$6.1 (010
22.8
43.1 05.00
60.4 (nt.tr
55.5 10000
22.2 mAne
63.2
0.1838
21.93
they are difficult to determine precisely from empirical data.
Accuracy sym - I[equiv([] (irgee))
Here, II(-) is the indicator function and equiv(:--) checks for symbolic equivalence. However, tra- ditional rule-based equivalence checks can be brittle when handling the algebraic complexity of expressions. We therefore introduce an LLM-as-a-judge protocol for this verification step, which achieves 98.3% % agreement with human experts, demonstrating its reliability (see Appendix 1
Data Fidelity via RMSLE To measure the data fidelity of submitted law, we adopt the RMSLE metric. Unlike the metric of Normalized Mean Squared Error (NMSE) used in some prior studies (Shojace et al. 2025b Lin et al., 2025), RMSLE offers superior numerical stability and is better suited for physical quantities that may span multiple orders of magnitude due to its logarithmic scale. Given a dataset of N points with ground-truth values y/ and corresponding predictions 9;
from the submitted law, RMSLE is calculated as (implementation details in Appendix A.4.2).
N
RMSLE
Σ
(log(0) + 1) log(y; + 11)²
(2)
4 EXPERIMENT AND ANALYSIS
In this section, we present our main experimental results ($4.1 and conduct further analysis regard-
ing noise robustness ($4.2). cross-domain performance ($4.)) inference scaling ($4.4), and impact of code assistance $4.5 The complete experimental results provided in Appendix B

---

Published as a conference paper at ICLR 2026
4.1
CAN LLM AGENTS PERFORM GENERALIZABLE SCIENTIFIC LAW DISCOVERY?
The main objective of NEWTONBENCH is to authentically assess the generalizable scientific law dis- covery abilities of LLMs, framed by the intuitive question: "(To what extent) Can LLMS Rediscover Newton's 's Laws?" To answer this question, we evaluate 11 state-of-the-art LLMs (details in Ap- pendix A.I). including open-source models such DeepSeek-RI and closed-soures -suuce models such as GPT-5 and Gemini-2.5-pro. We present the general benchmark performance in Table 2
and the full results of each physics domains are reported in Appendix [B. We analyze model performance from multiple perspectives and summarize our main findings as follows:
NEWTONBENCH is crasoning-intensii failing non-thinking LLMs. Our benchmark proves highly challenging for models with weaker reasoning abilities. All three non-thinking LLMSs (GPT- 1-mini, GPT-4.1, 1, and DeepSeek- V3) achieve overall symbolie accuracies below 10%. While these models show limited proficiency (20-40% accuracy) even in the simplest setting, their performance degrades precipitously in all other configurations, demonstrating that success in NEWTON ENCH is contingent strong, generalizable reasoning.
Performance of Reasoning Models Diverges with Inereasing Complexity. Most reasoning models achieve overall accuracies between 30-80%, with GPT-5 and Gemini-2.5-prm distinguish- ing themselves at 75.9% and 65.4%, respectively. While these models all demonstrate proficiency in the easiest settings (70-100% accuracy), substantial performance gap emerges the difficulty escalates. In the most challenging setting, GPT-5 and Gemini-2.5-pm maintain accuracies of 40.3% and 16.7 16.7%, whereas all other reasoning models fall below 6%. This stark performance gap demon- strates that the benchmark's s difficulty controls are highly effective at stress-testing and stratifying advanced reasoning abilities.
Code Assistance Exhibits a Dichotomous Impact. The provision of a code interpreter has a surprisingly dichotomous effect on agent performance. For less capable models (SA < 40%), access to the code tool substantially improves symbolic accuracy. Conversely, for more capable models (SA 2 40%), the inclusion of the tool leads to a slight degradation in performance consequence of these models satisficing through over-exploitation, we detail in $4.5
So, to what extent can LLMs rediscover Newton's 's Laws? Our findings indicate a clear but fragile ca- pability. While current frontier reasoning models can deduce scientific laws in simple, well-isolated systems, their performance degrades precipitously as the complexity of the system or target equation increases. This exposes fundamental limitations in their generalizable reasoning and highlights that robust generalization remains the core challenge.
Accuatic Veleity
40%
45%
33.5
50
Ar Amrs'''s For Law
15
Govitatin
3.2
SnaTs
38%
Symbolic Accurocy
Tourers Law
78
Make
Hot Tarisor
40%
37.3%
Rad cactive Decay
Hewoors
1.2
D.E. Dier bution
Noise Level
Total
Figure 2: Impact noise levels on performance. 4.2 SENSITIVITY TO OBSERVATIONAL NOISE
Figure Result across physics domains.
To investigate robustness against noisy observations, we conducted an experiment on GPT-5-mini with four levels of Gaussian noise (0.0001, 0.001, 0.01, and 0.1). As illustrated in Figure perfor-
7

---

Published as a conference paper at ICLR 2026
mance degrades significantly with minimal noise, with the introduction of just 0.0001 noise level causing a 12-16% % reduction in accuracy compared to the ideal, noise-free setting. As the noise level was increased from 0.0001 to 0.1, symbolic accuracy declined proportionally. while data fi- delity (RMSLE) remained relatively stable. Notably, code assistance did not affect noise robustness, with the baseline and code-assisted agents degrading similarly. This demonstrates that the symbolic accuracy of LLMs is extremely fragile to noise in observational data.
4.3 CROSS-DOMAIN PERFORMANCE DISPARITIES
Performance across the physics domains, illustrated in Figure reveals a substantial difference in average accuracy, with scores ranging from 18% to 54%. Bose- Einstinttein Distribution, the most advanced and obscure domain in our benchmark, yields the lowest average accuracy at 18.1 8.1%. Fur- thermore, the performance disparities exacerbated system complexity increases. For instance, in the simple setting, Heat Transfer yields 68% accuracy, comparable to the easiest domain, Acous- tic Velocity (60%). In the complex setting, however, accuracy Heat Transfer plummets to 3.3%, while Acoustic Velocity remains at 45.0%. This demonstrates that the intrinsic nature of physical domain is primary factor governing the difficulty of model discovery. Full domain analysis provided in Appendix
4.4 INFERENCE SCALING WITH TASK COMPLEXITY
Can LLMs effectively scale inference for more challenging tasks? Figure 4 com- pares strong reasoning LLMs (Gemini-2.5. pro/flash, GPT-5/5-mini) with non-reasoning LLMs (GPT-4.1/4.1-mmln their token cost and number of rounds re- quired to solve tasks of varying difficulty. For strong reasoning LLMs, token consumption (i.e., reasoning length) increases significantly as task difficulty rises. In contrast, the token cost for non-reasoning models remains sistently low. This demonstrates that reason- ing models substantially scale up their computational effort to solve more com- plex fasks, whereas non-reasoning models fail to do so, even when consuming more experiment rounds. This superior scaling likely drives the advantage of strong models on complex tasks. Full results are provided in Appendix B.2
thiend
DeepSeek-Vi V3) on
Takers Round
humber Rourdds
V391
3332
3393
midium
Simp. S/
Dumpl Spx
Equation Difficultyy
Complexity
Figure Inference cost among difficulty levels.
4.5 THE PARADOX OF CODE ASSISTANCE: THE EXPLORATION WXW.-EEXPP-OTTTI TRADE-OFF
To understand the dichotomous impact of code assistance, we conducted an experiment on four representative LLMs with varying code-use budgets. As illustrated in Figure a)), the performance divergence is most pronounced when the code budget increases from zero to one: the performance of stronger models (Gemini-2.. 2l5-flash, OPT-5-mini) degrades, while that of weaker models improves. We hypothesize this divergence stems from fundamental shift in the models' strategies, specifically the balance between exploration and exploitatior
solving
To investigate this hypothesis, we adapt common analysis approach for reasoning LLMs (Wang et al. 2025b by identifying signature tokens associated with exploration (e.g., What if. Alter- natively) and exploitation (e.g., Confirm, Verify). From these, we calculate the exploration rate (the percentage of exploration tokens among all such planning tokens; see Appendix E.4 for details). Figure (bb) shows the results for two reasoning LLMs where reasoning traces are accessible. We observe sharp drop in the exploration rate for Gemini-2. 1omini--.5-ffaslt as the code budget increases
"The trade-off hetween discovering novel solutions (exploration) and optimizing the current best-known
solution (exploitation).
8

---

Published as a conference paper at ICLR 2026
(8)
OPESm Jocumes OPF-1 (Acurers)
Osan (ecmrecl Gentriz S-lahh (ecureey
Onend-33t (Eap falel Gembriz Shhth sratie fate!
6PT41 (n-1524)
Symbolic AcCuracy
46.1%
Kaslet
A Almccic Caka
GPT---mmll
Parciian hisigg
lypothace Vaidetion
Expleration Rate (N).
86.5
66:0
$2.3
76.5
22
13
No Oode
Uniivied
Figure 6: Functionality distribution for code usage (unlimited budget).
Figure 5: Results under different code- use budgets.
from zero to one, while the rate remains stable for Qwen-3-235b. This suggests the performance degradation in stronger models reflects over-exploitatior
To understand the mechanism driving this strategic shift, we analyzed the functional distribution of code usage for strong and weak model (GPT-5-mini and GPT-4.1). Figure ( reveals that GPT- 5-mini allocates a significantly smaller proportion of its code use to basic calculation compared to GPT-4.1, instead favoring function-fitiing This supports the hypothesis that code serves distinet roles: computational tool for weaker models versus an equation refinement tool for stronger ones.
In summary, for weaker LLMs, whose primary bottleneck arithmetic computation, code execu- tion provides crucial assistance, thereby improving their performance. Conversely, stronger LLMs already possess sufficient computational prowess and tend to leverage code for tasks like function- fitting. This can accelerate convergence to "good enough" solution, causing the model to prema- turely settle in a local optimum. Such over-exploitation stifles the broader exploration needed to discover globally optimal answer, paradoxically degrading the performance of these more capable models (further case studies in Appendix [D). This finding highlights the importance of managing the
trade-off in agentic systems, particularly in how tools are leveraged by models of varying capability for data-driven discovery.
RELATED WORK
Symbolic Regression Symbolic regression (SR) aims to discover mathematical formulas from data, with foundational approaches being Genetic Programming (GP) that evolve populations of candidate expressions (Koza 1994. Augusto & Barbosa 2000 Billard & Diday 2002 Schmidt &
Lipson 2009). Meanwhile, more advanced approaches have been proposed, from thysics-inspired,
Pareto-driven strategies (Udrescu & Tegmark 2020
Udrescu et al. 2020) to deep learning-basee
models (Petersen al. 2021 Kamienny et al. 2022 The latest evolution in SR leverages the rea-
soning capabilities of L.Mss to hypothesize and refine scientific equations (Shojaee et al. 2025a b
Xia et 2025
Chen al. 2025a).
LLM-Driven Scientific Discovery The agentic capabilities of LLMs are increasingly being ap-
plied to automate the scientific process (Zheng et al. 2025 Wei et al., 2025), from assisting with
experimental procedures in chemistry and biomedicine (Gottweis et al. 2025 Yang et al., 2025
Luo et al.] 2025 Chen et al.] (225bb to automating complex research workflows in machine learning
(Chan et al. 2025 Huang et al. 2024a Jiang et al. 2025 Jansen et al. 2025). This trend culminates
in the pursuit tully autonomous "Al Scientist" capable of managing the entire research pipeline
for open-ended discovery (Lu et al. 2024 Yamada et al.) 2025
These observations correlational and should be interpreted suggestive rather than causal.

---

Published as a conference paper at ICLR 2026
Virtual Environment for LLM Agents Virtual environments for LLM agents have evolved from
general- purpose sandboxes for tasks like instruction following (Shridhar et al. 2021), web naviga-
tion Yao et al. 2023), and compositional planning (Prasad al. 20024. specialized platforms
for scientific reasoning, ranging from curriculum-based tasks (Wang et al. 2022 to open-ended
discovery (Jansen al. 2024
CONCLUSION
In this work, present NEWTON the first scientific law discovery benchmark designed to resolve the long-standing trade-off between scientific relevance, scalability, and resistance to memorization. Moreover, elevates the evaluation from static function fitting interactive model discovery, requiring agents to experimentally probe complex systems to uncover hidden governing principles. Our benchmarking reveals that while frontier models possess clear capability for dis- covery, this ability is fragile, degrading precipitously with increasing system complexity and obser- vational noise. We further uncover a paradoxical effect where tool assistance hinders more capable models, inducing premature shift from exploration to exploitation that causes them to satisfice on suboptimal solutions. Looking forward, we hope NEWTONBENCH serves as a crucial litmus test for the reasoning capabilities of frontier LLMs and agentic systems, faithfully measuring genuine scientific intelligence to guide the development of Al capable of authentic discovery.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors this paper were supported by the ITSP Platform Research Project (ITS/189/23FP) from ITC of Hong Kong, SAR, China, and the AoE (Aoe//-601/24-NN) and the GRF (16205322) from RGC of Hong Kong, SAR, China. We also thank the support from NVIDIA AI Technology Center (NVAITC).
ETIICS STATEMENT
NEWTONBENCH is designed exclusively for benchmarking and evaluating the scientific discovery capabilities LLM agents in a fully contamination-fud free, virtual environment. The counterfactual law shifts applied to physical laws ensure the tasks novel and not directly suitable for training LLMs to discover real-world physics. Due to the interactive, model-system nature of the tasks, we exclude traditional symbolic regression methods that do not support model systems and LLM-based symbolic regression pipelines whose prior reliance and workflows are incompatible with our proto- col; these are outside the scope of this evaluation. We caution that the benchmark is not intended as training dataset for developing stronger physics discovery models, but rather as controlled tool for fair and rigorous evaluation. All shifted laws were curated by expert co-authors (Physics Olympiad backgrounds) and cross- validated Furthermore, while the physics domain was selected as the most representative ground for scientific law discovery, we posit that the findings of NEWTONBENCH ENCI are generalizable to fields such chemistry and biology. This generalization holds because the core cognitive process isolating variables to derive mathematical relationships is isomorphic across disciplines. Whether elucidating reaction kinetics in chemistry modeling enzyme dynamics in quantitative biology, the discovery process demands the same rigorous interplay of hypothesis test- ing and symbolic abstraction. Consequently, NEWTONBENCH serves robust indicator of an agent's 's potential to drive discovery across the broader spectrum of the natural sciences.
REPRODUCIBILITY STATEMENT
LLM evaluations in this work were conducted via public APIs, specifically using the OpenRouter and OpenAI-API platforms, with the total experimental cost estimated 10,000 USD. Detailed model configurations provided in Appendix A Experimental results are reported with measures of statistical significance to ensure robustness. We additionally provide all prompt templates and system implementations in Appendix to facilitate full reproducibility of our experiments. All code and data are publicly released to foster future research.
REFERENCES
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges, 2024. URL httpps://axxi..rr
abs/ / 2402. 00157
Hanae Armitage. Researchers create 'virtual scientists' solve complex biological problems. Stanford Medicine News Center, July 2025. URL htLLhhtts: http://mmd..taatoordeeeeereeee www.///wweeeee httee///2225000/irreeee www.lllssseeissst..mmmm html
Douglas Adriano Augusto and Helio JC Barbosa. Symbolic regression via genetic programming, In Proceedings. Vol. I. Sixth Brazilian symposium on neural networks, pp. 173-178. IEEE, 2000.
Lynne Billard and Edwin Diday. Symbolic regression analysis. In Classification, clustering, and data analysis: recent advances and applications, pp. 281-288. Springer, 2002.
Nancy Cartwright. How the Laws of Physics Lie. Oxford University Press, 06 1983. ISBN 9780198247043. doi: 10. 00.1000001192240444000..0000 URL bttps://861.org/11 01 0198247044 100100000
Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung. Dane Sherbum, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Madry. Mle-bench: Evaluating machine learning agents on machine learning engineering, 2025. URL
https://arxx http///axxxv..o///....s./. www/ 2410. 07095

---

Published as a conference paper at ICLR 2026
Yimeng Chen, Piotr Piekos, Mateusz Ostaszewski, Firas Laakom, and Jürgen Schmidhuber. Phys- gym: Benchmarking Ilms in interactive physics discovery with controlled priors, 2025a. URL
https://arxiv: org/ abs / 2507 15550
Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu- Ampratwum, Xuhui Huang, Xia Ning. Song Gao, Yu Su, and Huan Sun. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery, 2025b. URI.
http://awwxvv........./ https::/erxiv.oort otg. 2410 240000000000000
Bernard D. Coleman and Walter Noll. Foundations of linear viscoelasticity. Reviews of Modern
Physics, 33(2):239-249, Apr 1961. doi: 10.1 10.11033/ewoodPyy...33.
Miles Cranmer. Interpretable machine learning for science with pysr and symbolicregression.j. http://aaab/335.11e/ 01582
2023. URL
Brian M. de Silva, David M. Higdon, Steven L. Brunton, and J. Nathan Kutz. Discovery of physics from data: Universal laws and discrepancies. Frontiers in Artificial Intelligence, 3:25, 2020. doi: 10.3388//fri.2222000000 URL htttt::://wwt wwww.froome
DeepSeek-AL. Deepseek-v3 v3 technical report, 2024. URL
19437
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning,
2025. URttpsss://arxx org http:///wwwwwww..aaa//////e/////
Miko Elwenspoek. Counterfuctual Thinking Physics, pp. 62-80. De Gruyter, Berlin, Boston, 2012. ISBN 9783110268669. doi: Lo0:10.111//978311066666666
URL
You-Le Fang, Dong-Shan Jian, Xiang Li, and Yan-Qing Ma. Ai-newton: A concept-driven law discovery system without prior physical knowledge, 2025. URL.https://axxiv.or
abs/ 221.2.044......
Herbert Goldstein, Charles P. Poole, and John L. Safko. Classical Mechanics. Addison-Wesley San Francisco, 3rd edition, 2001. ISBN 978-0201657029.
Google. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and generation agentic capabilities, 2025.
htatta/s//aassi.oog///////////// 06261
Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Yuan Guan, Vikram Dhillon, Eeshit Dhaval Vaishnav, Byron Lee, Tiago R D Costa, José R Penadés, Gary Peltz, Yunhan Xu, Annalisa Pawlosky, Alan Karthikesalingam, and Vivek Natarajan. Towards an ai co-scientist, 2025. URL https: /ttp//aaxvvv//./////v//eeeeeaavv www.aiwwwwww..///......e.
18864
M.E. Gurtin, E. Fried, and L. Anand. The Mechanics and Thermodynamics of Continua. Cambridge University Press, 2010. ISBN 9781139482158. URL htttps://boot gowglooooooooooooooooooooooooo. e. com hk/
books? die.ss??dddGGGggDDsTSSCC
Kaiming He. Deep residual learning and how it shaped the ai landscape. Future Science Prize Laureate Lecture, The Chinese University of Hong Kong, 2023. Invited talk.
Qian Huang, Jian Vora, Percy Liang, and Jure Leskovec. Mlagentbench: Evaluating language agents on machine learning experimentation, 2024a. URL https://acx.vv.oo///bba absp////abb////
03302
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang. Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. Understanding the planning of Im agents: A survey, 2024b. URL |Rt.ttts:: ///::///aa //axx////vv/// org wttp.as////.......////// 02716
12

---

Published as a conference paper at ICLR 2026
B. Iglewicz and D.C. Hoaglin. 1993. ISBN 9780873892605. Fuui EAAAEAA
How Detect and Handle Outliers. ASQ Quality Press, URL
google. com. hk hw..bokkkkkkd
Peter Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi Mishra, Bod- hisattwa Prasad Majumder, Oyvind Tafjord, and Peter Clark. Discoveryworld: A virtual en- vironment for developing and evaluating automated scientific discovery agents, 2024. URL
https://axxiv/ org/ abs/ 2205//006...06 06769
Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S. Weld, and Peter Clark. Codescientist: End-to- end semi-automated scientific discovery with code-based experimentation, 2025. URL https:
//arxiv.orq aaarr///b//2055 22708
Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu. Aide: Ai-driven exploration in the space of code, 2025. URL
org/abs 2502. 13138
Pierre-Alexandre Kamienny, Stéphane d' Ascoli, Guillaume Lample, and François Charton. End-to- end symbolic regression with transformers, 2022. URL http:////wwwww..o/abbbbo
10532
John R. Koza. Genetic programming as a means for programming computers by natural selection. Statistics and Computing, 4:87-112, June 1994. doi: 10.1007/BF001753555 URL https:
com art aataiaaiieeelo 007 / BFF 001 75355
Pat Langley, Herbert A. Simon, Gary L. Bradshaw, and Jan M. Zytkow. Scientific discovery: com- putational explorations of the creative process. MIT Press, Cambridge, MA, USA, 1987. ISBN 0262620529.
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng. Pei-Jie Wang. Xiuyi Chen, Yingying Zhang, Fei Yin, Jiahua Dong, Zhiwei Li, Bao-Long Bi, Ling-Re Rui Mei, Junfeng Fang, Xiao Liang, Zhijiang Guo, Le Song, and Cheng-Lin system 2: survey of reasoning large language models, 2025. URL
Liu. From system abs 72002 17419
hittps://arxiis org.orgg/aaa
Haowei Lin, Xiangyu Wang, Jianzhu Ma, and Yitao Liang. Evosld: Automated neural scaling law discovery with large language models, 2025. URL https::/arxiv..or wrtp://///abb//000
21184
Jiqqi Liu, Songning Lai. Pengze Li, Di Yu, Wenjie Zhou, Yiyang Zhou, Peng Xia, Zijun Wang, Xi Chen, Shixiang Tang, Lei Bai, Wanli Ouyang, Mingyu Ding, Huaxiu Yao, and Aoran Wang. Mimicking the physicist's eye:a vim-centric approach for physics formula discovery, 2025. URL.
htttp:://rrxi.. wttg 17380
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open- open-ended scientific discovery, 2024. URL /tRl/t////////i.orr
abs/2408 06292
Yi Luo, Linghang Shi. Yihao Li, Aobo Zhuang, Yeyun Gong, Ling Liu, and Chen Lin. From intention to implementation: automating biomedical research via llms. Science China Information Sciences, 68(7). June 2025. ISSN 1919- doi:
URL CtLhtt:
7/dx. doi. org.//g////o 10. wtt.:/11//ss1111
Yoshitomo Matsubara, Naoya Chiba, Ryo Igarashi, and Yoshitaka Ushiku. Rethinking symbolic regression datasets and benchmarks for scientific discovery, 2024. URL htttps://axxvz
orq/ bog//bs// 10540
I. Mueller and T. Ruggeri. Rational extended thermodynamies. Springer Tracts in Natural Philos-
ophy. Springer New York, 2013. ISBN 9781461222101. URL httpt:ttbooos..ooot google. com. ww/.bkkksb.......oooo.... ZeP SBgAAQAAA
OpenAl. Introducing openai ol preview, 2024.
URL https///penai..com/nn
introduciing-openat-
13

---

Published as a conference paper at ICLR 2026
OpenAl.
Introducing gpt-4.1 in the api.
httpas//ppeea...ommmnbe///pppppp 04
2025a.
OpenAl. Gpt-5 is here. https://openali cttppp/taacco//pt55//// 08 2025b.
OpenAL.
Introducing openai o3 and 04-mini.
ht:ps://ppenal.coom/neeee
introducing o4 ominiii/ 04 2025c.
Charles Sanders Peirce. Collected Papers of Charles Sanders Peirce, volume 1-6. Harvard Univer- sity Press, Cambridge, MA, 1958.
Brenden K. Petersen, Mikel Landajuela, T. Nathan Mundhenk, Claudio P. Santiago, Soo K. Kim, and Joanne T. Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk- policy gradients, 2021. htttp://///are org/ wttp//ab///1....0 04871
K.R. Popper. The Logic of Scieetific Discovery.
ISSR Library. Routledge, 2002. google. com. wwt.w..........aa.oooe
ISBN 9780415278447. Tq6xeupNStMC
URL
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language mod- els, 2024.
05772
Qwen. Qwq-32b: Embracing the power of reinforcement learning, March 2025a. URL https:
7/ 7/qwenlm. 1tt.uu..ooolog////oo qwq 32b/
Qwen. Qwen3 technical report, 2025b. URL htt.:tttss///rrri.ooo//bbb//////
09388
Michael Schmidt and Hod Lipson. Distilling form natural laws from experimental data. Science, doi: 10.1126/science. 1 165893.
324(5923):81-85, 2009.
org/doi/abs//n 112 wwwpe///cccnne.ei....eeieeeieeec 1165893
Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K Reddy. Lim-sr: Scientific equation discovery via programming with large language models, 2025a. URL
atttss://axxiv.orgg///
18400
Parshin Shojace, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K Reddy. Lim-srbench: A new benchmark for scientific equation discovery with large language models,
Mohit Shridhar, Xingdi Yuan, Alexande Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning and embodied environments for interactive learning, 2021.
URL httpttttss//xitto org/ abs/ 20000000000 03768
Silviu-Marian Udrescu and Max Tegmark. Ai feymman: a physics-inspired method for symbolic regression, 2020. URL httpppp://ax....o/////oaa/ee http///wwww...rr//////////////// 11481
Silviu-Marian Udrescu, Andrew Tan, Jiahai Feng, Orisvaldo Neto, Tailin Wu, and Max Tegmark. Ai feynman 2.0: Pareto-optimal symbolic regression cxploiting graph modularity, 2020. URL abs
https://arxx: http://aaxiv... http:/awwww..rg///////////////// 10782
Haozhe Wang, Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, and Wenhu Chen. Emergent hierar- chical reasoning in Ilms through reinforcement learning, 2025a. Utthttss:://arxt
abs/2509. 03646
Ruoyao Wang, Peter Jansen, Mare-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than 5th grader?, 2022. URL https://arrxvv.org//b////000 07540
Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, Yuqiong Liu, An Yang. Andrew Zhao, Yang Yue, Shiji Song, Bowen Yu, Gao Huang, and Junyang Lin. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for Ilm reasoning, 2025b. httpstss: /ttp:i////avvv..a orgooogg/a/.
2506 01939
14

---

Published as a conference paper at ICLR 2026
Jiaqi Wei, Yuejin Yang. Xiang Zhang, Yuhan Chen, Xiang Zhuang, Zhangyang Gao, Dongzhan Zhou, Guangshuai Wang, Zhiqiang Gao, Juntai Cao, Zijie Qiu, Xuming Hc, Qiang Zhang, Chenyu You, Shuangjia Zheng, Ning Ding, Wanli Ouyang, Nanqing Dong, Yu Cheng, Siqi Sun, Lei Bai, and Bowen Zhou. From for science to agentic science: A survey on autonomous scientific discovery, 2025. BRR.https://arxiv.orrr htspg/abs/5000 14111]
Shijie Xia, Yuhan Sun, and Pengfei Liu. Sr-scientist: Scientific equation discovery with agentic ai, 2025.
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level autooaat scientific discovery via agentic tree search, 2025. URL http::
08066
Zonglin Yang. Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. Moose-chem: Large language models for rediscovering unseen chemistry scientific hypotheses, 2025. URL
07006
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real- world web interaction with grounded language agents, 2023. URL htR/htsp////axivoorgg
abs/2207.01205
Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. Natural language reasoning, survey. ACM Comput. Surv., 56(12), October 2024. ISSN 0360-0300. doi: 10.1145/3664194, URL
Tianshi Zheng, Zheye Deng, Hong Ting Tsang, Weiqi Wang, Jiaxin Bai, Zihao Wang, and Yangqiu Song. From automation autonomy: survey on large language models in scientific discovery, 2025. URLpttps: arxiv www.vorg/aaaao abs 2200. 13259
15

---

Published as a conference paper at ICLR 2026
APPENDIX
IMPLEMENTATION DETAILS
17
A.1
LLM DETAILS
17
A.2
MODEL SYSTEMS
18
A.3
EQUATION DETAILS
20
A.4
EVALUATION DETAILS
27
A.5
FURTHER DISCUSSIONS ON LLM AS JUDGE
30
FULL RESULTS
B.1
RESULTS FOR INDIVIDUAL DOMAINS
32
B.2
RESULTS OF NON- PORFFORRRRMNCCE METRICS
C
PROMPT TEMPLATES
50
C.1
GENERAL PROMPTS
C.2
DOMAIN-SPECIFIC PROMPTS
52
CASE STUDIES
D.1
CASE STUDY FOR NON-REASONING LLMS
54
D.2
CASE STUDY FOR REASONING LLMS
57
MISCELLANEOUS
61
E.1
MEMORIZATION IN LSR- TRANSFORM
61
E.2
SOLVABILITY PROOF
63
E.3
PHYSICS DOMAIN ANALYSIS
66
E.4
EXPLORATION AND EXPLOITATION TOKENS
E.5
SYMBOLIC REGRESSION BASELINES
68
E.6
FURTHER DISCUSSIONS
70
E.7
USE OF LARGE LANGUAGE MODELS
71
16

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
A.1 LLM DETAILS
In our experiments, evaluated 11 state-of-the-art LLMs, including both open-source and propri- etary models. The models are summarized as follows:
GPT-4.1-mini (OpenAl,
2025a is lightweight proprietary non-reasoning LLM from OpenAl, designed for efficient inference.
GPT-4.1 OpenAA nAI.
2025a is the latest flagship proprietary non-reasoning LLM from Ope-
04-mini (OpenAl 2025c is a proprietary reasoning LLM developed by OpenAl, optimized for lightweight inference and efficient deployment.
GPT-5-mini (OpenAl 2025b) is lightweight reasoning LLM from OpenAl, aiming for improved efficiency on downstream tasks.
GPT-5 (OpenAl) (025b) the strongest reasoning LLM by OpenAl with advanced rea- soning capabilities.
DeepSeek-V3 [DeepSeck-A]] 2024) is an open-source non-reasoning LLM released by
DeepSeek, built for robust purpposs language understanding.
DeepSeek-R1 (DeepSeek-A] 2025 is open-source reasoning LLM from DeepSeek,
trained with reinforcement learning to enhance reasoning and alignment.
QwQ-32b (Qwen 2025a an open-source reasoning LLM by Qwen team, leveraging reinforcement learning and agent-driven tool use for complex problem solving.
Qwen-3-2333 (Qwn 2025b) is an open-source reasoning LLM by Qwen team, trained on large corpora for high-performante language and reasoning tasks.
Gemini-2.5-flash (Google 2025) is proprietary reasoning LLM from Google's Gemini
series, optimized for ultra-lone long context windows and rapid inference.
Gemini-2.5-pro (Google 2025) is the advanced proprietary reasoning LLM in the Gemini series, offering enhanced multimodal understanding and superior reasoning abilities.
By "reasoning" and "non-reasoning", distinguish whether deliberate thinking process-often incentivized by reinforcement learning with verifiable rewards (RLVR)-1 is triggered before the gen- cration of the first answer token (Li et 2025).
In all experiments, we set the LLM temperature to 0.4. Statistical significance is assessed based on four independent runs for each experiment. The main results table reports the standard deviation, while other plots display eтrоr bars representing the 95% confidence interval for the mean. In each test case, LLM agents are allowed up experimental rounds, with at most 20 sets per round. This design prevents unbounded interaction and helps isolate strategy design.
17

---

Published as a conference paper at ICLR 2026
A.2 MODEL SYSTEMS
All model systems in NEWTONBENCH are human-curated and inspired by classical experiments from each domain. As example, we illustrate Simple System in the Acoustics domain. De- tailed descriptions of the model systems across all 12 domains are provided in the Supplementary Materials.
SIMPLE SYSTEM (LAW OF SOUND SPEED IN IDEAL GAS)
The model simulates the emission of sound pulse within gas chamber, directed toward wall positioned at a known distance, and records the time required for the echo to return. The agent's task deduce the target law, furget. by interacting with the simulation.
System-Level Input Variables (VM) The simulation accepts the following inputs:
y: The adiabatic index of the gas in the chamber
M: The molar mass of the gas
T: The temperature of the gas
d: The distance to the wall
System-Level Final Output (M/) After the simulation runs, it returns the following output:
t: The total time taken for the echo to return
Ordered Sequence of Equations (M) The simulation environment internally computes the final measured time by executing a fixed sequence of calculations. This sequence of operations, M = (fi, f2), hidden from the agent. However, the fundamental principle underlying the assisting equation (f2) are provided to the agent to guide its discovery.
1. fi (target):
Usound Smpper (r, T, M)
2. f2 (assist):
2d
Usound
18

---

Published as a conference paper at ICLR 2026
SYSTEM COMPLEXITY ANALYSIS
We analyze the system complexity of all 12 physical domains based on the following two metrics:
Number of Assisting Equations: The number of equations applied in model M that are
not Starget-
Number of Operations: The total number of operators (e.g., +. *, ", sin) that appear in all assisting laws f. This metric reflects the computational complexity of the model system (excluding target law).
As illustrated in Table 3 complex systems associated with higher quantity of assisting equa- tions and operations, highlighting the computational challenges inherent in the law discovery pro- cess.
Table 3: Comparison in numbers assisting equations and operations between simple system and complex system across all 12 physical domains.
Simple System
Complex System
Physical Domain
Number Number
Number of Assist Equations
Number Operations
Assist Equations
Operations
Newton's 's Law Universal Gravitation
22
Coulomb's Law
2
3
Ampère's 's Force Law
4
10
5
14
Fourier's Law
1
6
Snell's Law
Law Radioactive Decay
1
3
Law of Damped Harmonic Motion
2
3
Malus's 's Law
1
3
Law of Sound Speed in Ideal Gas
2
7
Hooke's Law
Distribution
2
Law of Heat Transfer
3
4
10
Average
1.75
5.17
2.42
8.17
19

---

Published as a conference paper at ICLR 2026
A.3 EQUATION DETAILS
A.3.1 EXPRESSION TREES
The output of any given expression tree (AST) for specific set of input variables is computed using the canonical recursive algorithm detailed in this section. The evaluation strategy follows a post- order traversal: the value of any internal node (an operator) is computed only after the values of all its children have been determined. This ensures that operators are always applied to fully evaluated operands. The algorithm requires the tree's root node and map containing the numerical values for all variables present in the expression. Algorithm provides a formal, high-level description this process.
Algorithm Recursive Evaluation of an Expression Tree Require: Node n, variable mapping V var ++ IR
Ensure: Numerical result r E IR of the expression rooted at 72
1: function EVALUATE(1, V)
2: if n.type = CONSTANT then
3: return value
4: else if n.type = VARIABLE then
5: return Vanename]
6: else if n.type = OPERATOR then
7: values +- [Initialize empty list)
8: for each n..children do
9: V <- EVALJUTRÍ(,, V)
10:
values 11: end for
12: op n.operator
13: return APPLY(op, values)
end if
15:
end function
As formalized in Algorithm
! the evaluation logic operates through two fundamental cases:
Base cases: Recursion terminates at leaf nodes. For CONSTANT nodes, the intrinsic numer- ical value returned directly. For VARIABLE nodes, the corresponding value is retrieved from the variable mapping V.
Recursive case: For OPERATOR nodes, the algorithm first recursively evaluates all child nodes in post-order fashion, collecting their results. The operator is then applied to these operands via the APPLY function, returning the computed result.
The time complexity is ([nl)) where |n|] is the number of nodes in the AST, as each node is visited exactly once during the traversal.
20

---

Published as a conference paper at ICLR 2026
A.3.2 OPERATOR SETS
The complete operator space of NEWTONBENCH is presented in Table which specifies the symbol and arity for each supported mathematical operator.
Table 4: Mathematical operators used across all physics modules
Operator Name
Symbol
Arity
Addition
+
Binary
Subtraction
Binary
Multiplication
X
Binary
Division
+
Binary
Exponentiation
x
Binary
Exponential
exp(x:)
Unary
Natural Logarithm
log(a)) V
Unary
Square Root
Unary
Sine
sin(x)
Unary
Cosine
cos(x))
Unary
Tangent
(an(r)
Unary
Aresine
arcsin(r)
Unary
Arccosine
arccos
Unary
Arctangent
arctan(2) (x)
Unary
21

---

Published as a conference paper at ICLR 2026
A.3.3 PHYSICAL LAWS
summarizes the physical laws and their corresponding counterfactual shifts. These counter- factual shifts are achieved through mutation operations applied to the original equations, curated by human experts. The curation process follows four core guidelines:
Table 5
1. Each mutation must be performed within the set, and new variables should be intro- duced.
2. Mutations should increase, rather than decrease, the overall complexity of the equation.
3. The new mutation operation must simply reverse the previous mutation.
4. The outcome of consecutive mutations should not be attainable through single mutation operation.
Physical plausibility caveat. All mutated laws are dimensionally coherent by construction, but some may be physically implausible in our universe; we intend these counterfactual variants as scientifically grounded stress tests of reasoning and interactive discovery, not as claims about real- world physics.
Table 5: Full physical laws and their counterfactual shifts.
Ese
Molae
Mad mal ord
Inieal are Ps Pf of
Machaio (Gnntatre)
P
I' of zifed
P= (od) mi)
Blctsaaa Mognetor (Hlerooetto)
Pi Siig immer
Mottai Lng
wwwtiineeei Lo2 in andiinee add Maniiit
raada
radd
it lafa
K" luggh
Angele's fare Line
Therreay Enl Magccia (Magswoaii)
& Lula
grlan!?
Theredyyannis
"aazy
(A7))
147P
Optics (Gecnvetica Optio
(*1 (100)))
9 us (
82 to
4, lus.
NIG Nat
NO0 = A 2a
N10 in Xg- 42-
land
Modam Plysks (Nackan Plosica)
NPP) - Map
NiC = Nor staa
A00 = Nf 198
N00 = X6 amae
Ni0 - Nie 1ogt
Nio 100-00 X. 1e0t
NO - 21% ottetts
(#
Dovyed Barenn Holion
Lf
Wase and Ancreais (Dectladiaa))
(24int))
(2and8) Laxwe)))
Motur' Laee
Optic (PhpkkalOpeee
I- To co0
Boond lgvel is Mest Ove
Wowezed Acooaetor (Avwetic)
F 14g0
P- 24:0 Mee
Fi 24)
Mhde''s Law
Meduries (Bladeks))
be
Sun Siuvvis Dannhatis
Medon Plysks (Stsiaical Meckriie)
0 mi 'or7ys
Bea Trsspe
(Cakateastr)
((77)
misypa
(mA77ees
22

---

Published as a conference paper at ICLR 2026
A.3.4 CATEGORIZATION ON PHYSICAL PLAUSIBILITY
In this section, we categorize all shifted laws by their physical plausibility to enhance NEWTON- BENCII's 's value for Al research in physics. The complete categorization is presented in Table 7
We introduce a three-level taxonomy of physical plausibility to grade how severely each shifted law departs from the structural principles of its original physical law. The design is consistent with stan- dard treatments of (i) symmetry and conservation in classical mechanics and field theory (Goldstein et al., 2001) and (ii) admissible constitutive relations and entropy inequalities in continuum and nonequilibrium thermodynamics (Coleman & Noll 1961 Mueller & Ruggeri 2013 Gurtin et al. 2010
We assume standard physical domains, such as m > 0, T > 0, r > O, cross-sectional area A > 0, times t > 0, and angles in physically relevant ranges; quantities such intensity. energy, and occupation number are non-negative in the original laws. A new singularity of the shifted law at a point zo in the physical domain means that the magnitude of the shifted law becomes unbounded as one approaches zo from within the domain, whereas the original law remains finite (or extends continuously) at z0.
We define three levels of physical plausibility for the shifted laws:
Level 1 Admissible Counterfactual Laws (No Structural Violation)
Level 2 Counterfactual Laws with Minor Structural Violation
Level 3 Counterfactual Laws with Major Structural Violation
Level 1: Admissible Counterfactual Laws (No Structural Violation)
shifted law is assigned to Level if it satisfies all of the following:
1. No new singularities. For all physically allowed variables, the shifted law remains finite wherever the original law is finite. Any divergence is confined to points that were already singular in the original formulation (c.g., r + O in inverse-power forces).
2. Real and sign-consistent outputs. The law yields real values on the physical domain. Quantities that are non-negative the original law (c.g., intensity, energ,, occupation num- ber, probability) remain non-negative under the shifted law.
3. Preservation of the natural structural pattern of the domain. For pairwise interaction laws (e.g., gravitation, Coulomb, Ampère), the dependence on the two "charges" (masses, charges, currents) is symmetric enough that standard maintained and momentum conservation can be formulated in the usual way. For trans- port, constitutive, and statistical laws (e.g., Fourier, Hooke, calorimetry, radioactive decay, distributions), the relation between driving forces and responses remains within the broad class of mappings typically admitted in rational thermodynamics and statistical mechanies: there is no infinite response at vanishing driving force, and no obvious conflict with basic entropy-production or positivity requirements (Coleman & Noll 1961 Mueller Ruggeri 2013 Gurtin et al. 2010).
rerttoonn structure can be
In practice, Level includes exponent changes and nonlincarities that resemble generalized or effective laws already used in physics (e.g., nonlinear elasticity, nonlinear conduction, stretched- exponential relaxation, generalized equations of state). Level 1 is intended for experiments that re- quire highly plausible counterfactuals, such as evaluating symbolic regression and law-discovery methods under realistic but systematically modified physics.
Level 2: Counterfactual Laws with Minor Structural Violation
A shifted law is assigned to Level 2 if:
1. It satisfies the same regularity criteria as Level 1 (no new singularities; real, sign-consistent outputs), but
2. It breaks key symmetry, reciprocity, or asymptotic patterns that are normally associated with conservation, invariance, or physical admissibility in the corresponding theory.
23

---

Published as a conference paper at ICLR 2026
Typical examples include asymmetric two-body forces where the magnitude depends (m1, m2) or (9t, 92) in a way that is not invariant under exchanging labels, so that a simple action-reaction pair cannot be recovered. Similar issues arise for currents other sources when the law is intended as a fundamental vacuum relation. Such laws are mathematically well-defined on the physical do- main but structurally inconsistent with how fundamental interactions usually modeled, where symmetry and conservation are closely related (Goldstein et al., 2001).
Level 2 is naturally suited for probing sensitivity to minor but conceptually meaningful structural violations, e.g., whether a model exploits action-reaction and reciprocity rather than only local curve fitting.
Level 3: Counterfactual Laws with Major Structural Violation
shifted law assigned Level if introduces at least one of the following:
1. New singularities at physically regular points. The original law is finite (often zero) at some physically natural limit-for example.
AT -> in calorimetry or heat flow,
+ (vanishing cross-section).
regular incidence angles in optical or wave laws,
but the shifted law diverges there (e.g., terms proportional to 1/AT or 1/AP). This con- flicts with standard requirements that constitutive equations yield bounded responses for small driving forces and bounded entropy production in near-equilibrium regimes (Cole- man & Noll 1961 Mueller & Ruggeri 2013 Gurtin et al. 2010)).
2. Generic loss of reality or positivity. The shifted law yields negative, undefined, non- real values for quantities that must remain non-negative and real (such as intensities, oc- cupation numbers, probabilities) at finite, interior parameter values, not just well- understood singular limits of the original formulation.
Such laws are typically regarded as inadmissible in real-world physical modeling, because they violate minimal regularity positivity requirements in way that cannot be removed without dis- carding substantial part of the intended physical domain.
Level is suitable for challenging and interactive symbolic regression, and for stress-testing generalization and counterfactual reasoning under clearly implausible but algebraically well- specified distortions.
Table 6: Summary of physical plausibiiity levels.
Aspect
Level 1 Admissible Counterfac- tual Laws
Level 2 Counterfuctual Laws
Level 3
Counterfactual Laws
with Minor Structural Violation
with Major Structural Violation
Singularity
No singularities.
No singularities.
New blow ups points where the original law is finite.
Reality/ / positivity
Real outputs; non negative where re- quired.
quired. Real outputs; non negative where re-
Generic loss reality or pasitivity at finite parameter values.
Structural principles
Cousistent with natural symmetry/- conservation pallerms and comstitu- tive admissibility.
Breaks key synmetry/recipree itfasympioiics (e.g. action reaction), but remains regular.
Incompatible with minimal admissi- bility (e.g. infinite resporse to van- ishing drive).
Role experiments
Strictly plausible counterfactual physies.
Tests sensilivity structural struints.
Hard counterfactuals for robusiness and generalization stress tests.
24

---

Published as a conference paper at ICLR 2026
To illustrate the challenge imposed by these categorization levels, Figure presents the overall performance across the three plausibility levels. We report the Symbolic Accuracy (SA) and Root Mean Squared Logarithmic Error (RMSLE) across different equation difficulties and plausibility levels for both Vanilla Agent and Agent with Code Assistance settings.
SA (Vanilla)
RMSLE (Vanilla)
Level
Level
Level 3
Level
Level
Level 3
Easy
52.8
57.6
60.9
Easy
1.35
0.94
0.91
Medium
37.3
47.6
38.5
Medium
1.44
1.57
3.74
Hard
21.6
28.5
15.0
Hard
1.70
2.37
3.43
SA (Code)
RMSLE (Code)
Level
Level
Level
Level
Level
Level
Easy
54.4
60.4
53.0
Easy
1.04
0.62
1.01
Medium
37.7
46.1
38.4
Medium
1.30
1.11
3.51
Hard
19.0
30.3
18.9
Hard
1.52
1.41
3.15
Figure 7: Overall Performance Across Physical Plausibility Levels.
25

---

Published as a conference paper at ICLR 2026
Table 7: Full categorization on physical plausibility level for all 108 shifted laws.
Plansihillity Shifted Equations Included in this Level Level
Level
Nentor's Los Unirrnal Grarittioo trigiea
F @
P (Nkmm))
P. (00 mey)
P G
Cubon''s Lanioiiitll
Ampiee's Fane
L.
F- R
Jafa
F- & (aff)"
R Uh + (2)
P- Uiifa)
Pourie''s Low that Conduction inipanl Q
AAT
Q- AAA
(arp'
Yary?
and Yaagh
Q A/zzy?
Smalt' Law (ongies
9.
con"1
Ny Pi
74 sin tan
-on-1
ny Lan
tan
Radsoctive Decay coteiial N(9)
NO1 Mae
Niu
NOI
M] 9e
N00
Noc
A R NOS
N3 to
211 NOO
xg t
N(1
NIO 10
Dampet Harmanic Mation
(3m
fer (5.
Maln's Law origiaal Teoce? dy - fii d on 9) 1 faiii oos 0f I- fa 6.50
Speed Saand Med Gon ceeggiggl
TR'
Booke's Lay odgine Aao
Raja 20/2
Eisstin
(l 1
Hfat Trander (Calninetry) tnigied meaT)
Q me (6TP" mn "c Q (1677)95))
Level
Semton's Law Lubvraa Gravinmton F
mi
o mia
F G mijuge F-d I mifrr.
Conken's y Law F 11 1i ool?
E
Ampiee's Law F Ro F- K'th Taste F - Ro
P- Ne Bo pro
P- x
Level
Panter' Lav of Hhur Contuction. (N7P
== J.pr!
Sudl's Loe
sia's cueep"
p) cort1
Speed of Saand in in Bdadl Gas e (R7
Bonke's Lane F 20 fe Aga P 24
maa, F 26/2
Bee- Flassia Birthation
Cabninettry)
26

---

Published as a conference paper at ICLR 2026
A.4 EVALUATION DETAILS
A.4.1 SYMBOLIC ACCURACY EVALUATION DETAILS
We applied the LLM-as-a--udee framework to verify the symbolic equivalence between the submit- ted law and the ground-truth law, utilizing prompt engineering and few-shot demonstrations (details provided below). We evaluated the effectiveness of LLM-as-a-Judge across four LLMs using human-labeled dataset of 120 pairs of equations (results shown Table Based on performance and open-source accessibility, ultimately selected Nemotron-ulee ierootronnuutra--22 uuloon--uura-23bbb for our experiments. We will publicly release the judge evaluation data to guarantee reproducibility.
Table 8: Agreement (%) between LLM-as-a-Judge and human evaluator.
Models
Agreement
Nemotron-nano-Sh
83.3
GPT-oss-20b
90.8
GPT-4.1
98.3
Nemotron-ultr-2550
98.3
LLM-as-a-Juude Prompt
You are a mathematical judge. Your task is to datermine two oquations are equivalant.
**Instruttioss:
Compare the two equations carefullyy ly
Consider algebraic manipulatione, variable reordering, and variable zenaming
3. Determine if they represent the seme mathematical relationship
Provide your reasoning step
first, and then provide only
under the format of vελnswer: VES/MDAs same slgebraic rewrite [n *. 2) into 21n(x)
rry converting
equations
comparison
k>Output format:*1
keasening: (Your reasoning steps)
Answer: (YES/NO)
*.Reminder:**
Equat lons be expressed in standard mathematical notecion or as Python code. the Python implementation implies the same mathemation relationship,
the equations Constants may differ in form oz value. As long as they serve the same functional role (e.g., both scale the output proport/onal For example, conscant expreasec as eqrt (K) in one equacion and as a in another may be equivalent
considezed equivalent.
they conaidered interchangesble.
same way and con be interchangeable by selecting suitable value for the constant
affect
output
Variable names may differ, but the index and stzucture of variables must match exactly for the equstions For example, index of and 4.03 are considerad differen:
considered equivalent.
YR3/NO muat
*xammples: <provided the next page>
AvYour Task:A
Compare these equations and determine if they equivalent:
Paraneter Descriptiona:
descrptoisciittion
Equation 1: [equation1)
Equation 2: [equation2]
27

---

Published as a conference paper at ICLR 2026
L.MM-as---Judge Prompt (Few-shot Examples)
-Examples:*>
Equation 1: SIIDEN_CCNSTANT x x1 x2) < 2 / x3 xx
Equation 2: discovered (x1, x2, x3): c= = 6.7e-05
(x1 x2)
Reasoning: Although the constant equation 1 ls
and constant
in equation i5 E, both constant serve the same acoling role.
Answer:
x2) Equation 2: def laxcoverrdd_lxxxll x2, x3):
Equation
6.78-85
/ (x3 4 Reasoning: The secand equation changes the exponert on x3 and position
return
Answer: NO
Equation 1: sqrt (C x1 (x2 2)) x3 Equation 2: def discovered x2, x3): 6.7e-05
return sqrt xl1 x2 / x3 v*
Reasening: Since sqrt
x2, expressions represent
Answer: the same mathematical relationship
Equation 1: (G xi X2) x3
Equation 2: def discoveres
x3):
c 7.7e-U5
x3
Reasening: The exponent on x3 differs alightly Answer:
Equation 1: (C xi x2) x3
Equetion 2: def discoveres_lavttl
x3):
G - 6.7e 05
product
return product) / x3 +
Reasoning: Varisble ng dt ffeea but structure and operactons are equivalent.
G serves the same role
Answer: YES
Equatiir 1: In *л 2) Equation
discovered x2, x3):
G = 2.02
return G (n(x)
Reasoning: C (n(xxs2) is the same (n(xt and the constant (2C)
servers the same role as G
Answer: YES
x2) Equation 2: def di sa x2, x3):
Equation
return (x1 x2) /
Reasening: Equation has constant variable while Equation has numerical constant of
Answer:
28

---

Published as a conference paper at ICLR 2026
A.4.2 DATA FIDELITY EVALUATION DETAILS
For data fidelity evaluation, the submitted equations assessed on separate set of 5,000 data samples. Following Matsubara et al. (2024), we use the sampling distributions in Table To ensure RMSLE well-defined, equations are curated yield non-zero results, and inputs are sampled so that all ground truth outputs are non-negative. We further improve stability by applying outlier filtering using the Modified Z-Score method (Iglewicz Hoaglin 1993). The notion of physical constants is used only within the heuristic compling-distribung setting for RMSLE evaluation and does not represent the actual physical concept of the constants shifted laws.
Table Sampling Distributions for Each Domain
Domain
Equation
Variables and Descriptions
Sampling Distributions
ml: mass the object
(1y (10P. m22 Ula (I0P 10P)
Gravitation
F= ((m).
m2: muss of the second object
distance between the two objects
=0o0(1,1100 10")
ql: charge magninade of the first particle
e1 1io- 10
Coulomb's 's Law
F f(m.99.)))
92: charge magnimude of the second parricle
e2 (10 10
distance between particles
Degg(10 10"3
II: current magnitudo in the first wire
T1 (10 10-1)
Ampere's Force Law
F= ((51.22.7)
12: current magnitude in second wire
12 Uin (10-2, 10-11
r: (10 9,10
8: (100
A: cross seciinactionl remperarue difference d: distance across material
4 0ay 10
A(R, AT,4)
AT. (10 (10)
4: (10 (10)
refractive
medium
1.) 62: 1,
Snell's Law
92 = [(mL.
n2: refractive index mediumm
6e angle of incidenee
No Uaa (10P, 10F)
Radiouctive Decay
/(No, A
initial number lime
(10 10-1)
c: (10-"10
k: Ui,
Harmonic Motion
w = [(R.m,,)
III mass
m 110)
6: (10
intensity light
Io Uhag (100, 2000)
6: angle between the polarization axis
6: 0'10"".1177 ",,/
adiabatic index of the
(1.33 1,7)
Acoustic Velacity
1b.T,M)
T: tempenture
7: U0g 10*)
M: mokar mass
M (10 2,10-
Hooke's s Law
F f(x)
x: displacement from the equilibriun
2 (10 100)
Bone-Einstein Distribution
#- ((4,T)
ws angalar fiquunnc photons
T: lemperatree
T: Ung 101)
(10" 10"))
(1o (109,0000 9,10")
Q= M(m,c,
AT: temperature difference
NT : (10P 10")
29

---

Published as a conference paper at ICLR 2026
A.5 FURTHER DISCUSSIONS ON LLM-AS-A-JUDGE
Style Robustness Analysis We investigated the performance of our LLM-as-a-Judge in assessing symbolic equivalence across laws expressed in diverse syntactic representations. The results of this robustness analysis are illustrated in Table T0
Table 10: Style robustness analysis on LLM-aa--ajudge.
Case: Exponent placement difference
LLM Judge
GPT-4.1
Ground Truth Law
Rath.sqrt (gamma HIDDEN.CONSTANT.
2 / 1.5)
Testing Law
R_sim * ((T ** 4 ganma ** 2). /
(M ** 3)) (0.25)
Accuracy
100%
Case: Operand order variation
LLM Judge
GPT-4.1
Ground Truth Law
math. degrees (math. acos (nl 2 math.cos (math. radians (angle1)) n
Testing Law
val = (nl n1 n2)
math cos (math. radians (anglel))
return math. degrees (math. acos (val))
Accuracy
100%
Case: Log-power rearrangement
LLM Judge
GPT-4.1
Ground Truth Law
np. log (NO ** 1.5)
np. exp (-lambda.decay
0.5)
Testing Law
1.5 math.ex; (math.squ (t)
lambda.decay) math. log (NO) 100%
Accuracy
The results demonstrate that the LLM-as-a-Judge consistently recognizes symbolic equivalence when laws exhibit substantial structural and syntactic variations. This robustness to stylistic dif- ferences ensures fair and reliable evaluation within our proposed benchmark.
30

---

Published as a conference paper at ICLR 2026
Error Analysis We analyzed the failure modes of our LLM-as-a-Judge and identified one false positive and one false negative when using GPT-4.1, yielding an overall accuracy of 98%. Table [] provides the representative cases.
Table 11: Error case analysis on LLM-as-a-judge.
Case: False Positive
LLM Judge
GPT-4.1
Ground Truth Law
math. log (NO 1.5)
math. exp (-Lambda_decae
math.sqrte (t))
Testing Law
math.logll (NO 1.5)
math.ex, (-lambda.decay math. sqrt )t))
Accuracy
98% case fails)
Case: False Negative
LLM Judge
GPT-4.1
Ground Truth Law
math. log (NO 1.5)
math.ext -lambda.decay + math. (t) math. exp (math. log (1.5 math. log (NO)
Testing Law
lambda,decan + math. (t))
Accuracy
98% (1 case fails)
The false positive arises from conflating natural logarithm with base-10 10 logarithm. Specifi- cally, the model appears to treat log(z) and log as interchangeable up to a constant factor (log(x)= 1og1o(x) log(10)) and erroneously "absorbs" this constant, leading to spurious equiv- alence judgment. The false negative reflects difficulty with exponential-logarithe
cancellation in the presence of additive terms: although exp(log(a) b) cxp(b), the model failed to rec- ognize that log( exp(- + vt) equals cxp(1og(1.5 (1.5 No) Adoeay vt) because log(N)-55) 1.5 log No- Techniques such colf-consiitency/majory voting may reduce such errors, though the observed accuracy (98 %)) is already high.
Impact of Few-shot Learning We evaluated the effect of in-context demonstrations for LLM- as-a-Judge with GPT-4.11 Accuracy improves monotonically with more demonstrations (Table 12 underscoring the value of few-shot prompting.
Table 12: Impact of few-shot demonstrations on LLM-aasaajudge performance of GPT-4.1.
Num. Demos
Accuracy
O-shot
93.33%
1-shot
95.83%
3-shot
96.67%
7-shot
98.33%
31

---

Published as a conference paper at ICLR 2026
B FULL RESULTS
B.1
RESULTS FOR INDIVIDUAL DOMAINS
In this section, we illustrate the detailed performances of all LLM Agents in each physies domains:
Figure g
Gravitation (Newton's 's Law of Universal Gravitation).
Figure
Electrostatics (Coulomb's 's Law).
Figure T0
Magnetostatics (Ampère's Force Law).
Figure
Thermal Conduction (Fourier's Law).
Figure 12
Geometrical Optics (Snell'' Law).
Figure 13
Nuclear Physics (Law of Radioactive Decay).
Figure 14
Oscillations (Law of Damped Harmonic Motion).
Figure 15
Physical Optics (Malus's 's Law).
Figure 16
Acoustics (Law of Sound Speed in Ideal Gas),
Figure 17
Elasticity (Hooke's Law).
Figure 18
Statistical Mechanics (Bose-Einstein Distribution).
Figure 19
Calorimetry (Law of Heat Transfer).
32

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanilla)
(Code)
RMSLE (Code)
16.7
0.0
8.3
Edoy
0.2
0.7
3.9
25.0
35.3
33.3
0.2
6
2.3
spt-4.2-mini
HOWe
0.0
0.0
0.0
1.3
1.6
4.2
Han
8.3
16.7
0.0
Har
0.7
1.5
2.1
Ves.in
Syc
Eres Sye
Sir.pps. Com-yys.
kys
Cor Syt
Vin
Com fas.
0.0
Endy
0.2
3.1
Bxl
83.3
25.0
0.2
0.2
gpt-8.1
0.0
0.0
4.2
3.7
50.0
0.0
4,2
0.0
0.0
0.0
Herd
1.0
1.3
2.4
Rard
25.0
0.0
0.0
herd
1.2
1.4
2.5
Wen-bg
Syn
Don Dys
Vanta
Sem.5pp
Candyrs
1into
5m
Com
Ven
5em 353
Oon Sya.
Casy
Eady
Dayy
Eay
04-mini
Musium
100.0
75.0
33.3
Medium
0.0
0.0
1.8
Modium
66.7
58.3
25.0
Veskmr
0.0
0.0
0.2
HarE
Hand
0.0
1.3
Hand
Hard
Syn
Com Sys
Str.pps.
Camsppe.
5m
Sam
Vin
Com Syx.
0.0
58.3
0.0
apt-S-mint
Medium
91.7
75.0
58.3
Medium
0.0
0.0
0.0
Madury
100.0
833
83.3
Medker
0.0
0.0
0.0
100.0
100.0
03.3
Easy
0.0
0.0
100.0
91.7
75.0
0.0
0.0
gpt-5
Medium
100.0
91.7
Medium
0.0
0.0
Med
100.0
91.7
Medker
0.0
100.0
83.3
75.0
0.0
0.3
0.4
100.9
75.0
91.7
0.0
0.6
0.2
Syr
Sye
SinSpts Com-ipt
9in Syt.
Cor
Eile
50.0
25.0
16.7
Eady
0.0
0.2
1.9
75.0
25.0
0.0
0.6
0.2
2.5
deepsenk--33
HOVE
0.0
0.0
0.0
1800
1.0
3.5
1.4
Han
8.3
0.0
0.0
Hart
0.9
6.9
3.1
Dre Sys
Vintt
Smm.pp Camdyrs
Vaste
Sm
Com Sys
Ven
5en
91.7
66.7
Eady
0.0
0.0
83.3
0.0
dempsook-r1
83.3
58.3
41.7
2.9
0.0
56.3
50.0
56.3
6.0
Hired
58.3
25.0
16.7
Herd
0.0
1.5
3.5
Rard
58.3
16.7
8.3
herd
0.3
0.8
1.9
5ih Sys
Don Sys
Vaauth
Vento
5m.
Can
5er 9y5
Oce Sys.
Cap
Exty
0.0
Doy
Eay
0.0
qWc-332b
Musuum
91.7
41.7
16.7
Medium
0.0
0.0
3.1
Moduun
100.0
50.0
33.3
Peaker
0.0
0.0
0.0
Hard
Hard
0.5
Ven.Ca
Con Sya
SkmuSps.
Cam.Spe.
Vants
5m55p
Sam
Venia
Sm.
Stx
83.3
0.3
0.0
50.0
0.2
qwen-3- 3-55b
Medium
63.3
23.3
8.3
Medium
0.0
2.4
4.3
Med km
83.3
65.7
41.7
Medkir
0.4
0.0
0.0
0.1
91.7
91.7
41.7
Esay
0.0
0.0
0.0
100.0
83.3
66.7
0.0
0.6
0.0
gemini-2.5
Medium
91.7
75.0
0.0
0.0
Metun
91.7
Medker
0.0
31.3
25.0
Hend
0.2
1.2
33.5
16.7
16.7
0.4
1.3
Syn
Sye
Var.E
Sin.ps. Com-lye
Wan
Sm Syt.
Cam
Wi.
Sim
91.7
$0.7
75.0
Eaoy
0.0
0.0
0.0
100.0
91.7
58.3
EMS
0.0
0.3
0.0
gemini-2.5-pro
100.0
100.0
33.3
Hord
0.0
0.0
0.7
Hard
85.5
85.1
35.3
0.6
0.5
G.9
Figure 8: Full LLM Performances in domain Gravitation (Newton's 's Law of Universal Gravitation).
33

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanills)
(Code)
RMSLE (Code)
Cundyyt.
25.0
5.3
33.3
Ed0y
1.7
2.2
31.3
33.3
15.7
0.0
1.7
1.9
2.3
spt-4.2-mini
H0We
0.0
0.0
0.0
1601
2.3
2.7
53.1
Hod
0.0
0.0
0.0
Har
1.9
2.4
2.6
Vee.in
Sye
Eres Sye
Sir.pps
Cam-ays.
Sys.
Cor Syt
Viniis
Spr.Syr.
Com fyy.
16.7
0.0
Endy
1.7
2.4
Exty
58.
11.3
0.7
1.0
1.9
gpt-8.1
0.0
0.0
1.8
2.4
16.7
0.0
1.4
0.0
0.0
0.0
Nerd
2.0
3.3
3.0
Hard
8.3
0.0
0.0
herd
1.5
1.1
2.3
Ven.bg
Sen Syp
Don Dys
Vanta
3e1.5ph
Cam
Sinto
5m1
Cor 5m
Ven
5em 35.
Oon Sya.
Casy
Eady
Dayy
Eay
04-mini
Musium
91.7
16.7
16.7
Medium
0.0
3.3
1.0
undiu
91.7
16.7
0.0
Veskmr
0.2
1.5
1.8
Hard
33.3
Hand
0.5
1.9
Hand
Hard
1.6
Nen.0q
Syn
Sys
Sim.pps.
Camuppe.
Va1
Spt.
Sam
Vin
Spr.Spa.
Syx.
0.3
0.5
100.0
apt-S-mint
Medium
100.0
25.0
41.7
Medium
0.0
0.8
0.6
Maduun
83.3
16.7
0.0
Medker
0.2
1.3
1.5
100.0
100.0
91.7
Eany
0.0
5.2
2.1
100.0
100.0
100.0
11.4
1.7
gpt-5
Medium
100.0
Medium
0.0
1.8
Medker
0.0
100.0
8.3
16.7
0.0
0.0
2,9
91.7
25.0
50.0
0.0
6.0
4.1
Syr
Syc
Cam-ipt.
Wiiff
Can
Sim.Sy
50.0
23.3
0.0
EXdy
0.7
3.5
1.7
50.0
15.7
16.7
0.9
1.9
1.4
despsenk--33
HOVE
0.0
0.0
0.0
1800
2.4
2.4
2.0
Han
8.3
0.0
0.0
Hart
1.7
2.4
2.6
5m0
Due Sys
Vanta
Smm.pp Cam-5ra
Vaste
5m
Com
Vyn
5en
Sva.
81:3
66.7
Eady
0.3
0.6
75.0
0.0
dempaook-r1
16.7
16.7
0.3
0.0
$6.3
25.0
8.3
1.0
6.0
Hire
25.0
8.3
0.0
Herd
0.2
3.1
1.8
Rard
33.3
16.7
16.7
herd
1.3
1.9
1.6
Wee.bg
Sin Sys
Don Sys
Vints
Candues
Vante
5 Sph
Can
Ser. 9y5
Oces Sys.
Cap
Exsy
2.0
Eay
0.5
q2c-332b
Musuum
66.7
8.3
25.0
Medumm
3.6
2.1
4.0
Moduun
75.0
50.0
16.7
Meaker
1.8
0.0
1.7
Hard
1.1
Hard
Ven.Ca
Con Sya
SkmuSps.
Cam-Spe
5m55p.
Cam
Venta
St..0p.
Dom Stx.
06.7
0.1
1.7
100.0
66.7
0.4
qwen-3- 235b
Medium
63.3
25.0
16.7
Medium
0.2
1.1
1.4
Mad km
75.0
33.3
16.7
Meckm
0.2
0.9
1.5
100.0
91.7
75.0
Esay
0.0
0.0
1.5
100.0
91.7
83.3
0.0
3.3
0.0
gemini-2.5 2-5ss
Medium
83.3
Medium
0.5
15
Metun
33.3
Medker
2.6
$1.3
X1.3
Hend
1.0
0.0
3.0
0.3
0.0
1.6
3.5
Syn
Sye
Var.E.
Sin.pps Com-lye
Wante
Som
Skm
Syx.
100.0
75.0
91.7
Eady
0.0
0.0
0.1
100.0
75.0
100.0
0.0
0.0
0.0
gemini-2.S-pro
53.3
0.0
0.0
Herd
0.2
0.0
0.0
Hard
91.7
25.6
25.0
0.4
1.5
6.5
Figure 9: Full LLM Performances in domain Electrostatics (Coulomb's Law).
34

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanilla)
(Code)
RMSLE (Code)
25.0
0.0
0.0
Eddy
0.2
0.2
0.1
33.3
25.0
35.3
0.0
G.4
0.2
spt-4.2-mini
H0We
8.3
0.0
0.0
16m1
0.5
6.4
6.1
Han
16.7
8.3
0.3
Har
1.4
3.4
4.6
Sin
Eoes Sye
Sirspps. Com-ays.
Sys.
Cor Syt
Vin
sin Spt.
Com fat.
25.0
50.0
25.0
Endy
0.0
0.1
Exty
41.7
41.7
0.0
0.0
6.2
9pt-8.1
0.0
41.7
0.0
3.2
2.1
31.3
0.3
0.7
0.0
16.7
0.0
Herd
4.0
3.8
Hard
25.0
0.0
16.7
herd
1.3
2.4
2.1
Wendtg
Syn
Don Sys
Vanua
Sem.5pp
Sinte
5m
Con
Ven
Oon Sya.
Casy
Exdy
Dayy
Eay
04-mini
Musium
83.3
58.3
25.0
Mudium
0.0
0.0
1.4
Modium
75.0
66.7
16.7
Veskmr
0.0
0.0
2.6
Hard
Hand
0.0
Hand
Hard
2.1
Syn
Con Sys
Sanda
Skm.pps.
Camsppe.
Sps.
Cam
Vin
Spm.Spa.
Com Syx.
0.0
0.4
apt-S-mint
Medium
100.0
06.7
41.7
Medium
0.0
0.0
0.9
Maduu
100.0
33.3
8.3
Medker
0.0
0.0
2.0
100.0
100.0
66.7
Easy
0.0
0.0
91.7
100.0
0.0
0.0
gpt-5
Medium
100.0
Medium
0.0
0.0
Med
100.0
Medker
0.0
100.0
91.7
8.3
0.0
0.1
0.6
83.3
91.
16.7
0.4
0.5
1.0
Syr
Syc
Sindptt
Cam-ipt
Sn Syt.
Corn
91.7
23.3
33.3
Eooy
0.0
0.0
0.1
75.0
25.0
41.7
0.1
0.2
0.1
deepsenk--33
HOVE
16.7
0.0
0.0
1800
1.9
5.5
5.6
Han
8.3
0.0
0.0
Hart
1.9
2.4
2.0
Vintt
Smm.pps Cam-5ra
Vonto
5m
Com
Vyn
5en
91.7
91.7
25.0
Eady
0.0
0.0
56.3
0.0
dempaook-r1
91.7
75.0
$0.0
0.9
0.0
75.0
50.0
56.3
6.0
Hirdd
75.0
25.0
16.7
Herd
0.8
3.9
3.3
Hard
58.3
25.0
0.0
herd
0.0
0.6
3.6
5ih Sus
Don Sys
Vaauh
Candupa
Vanto
5m
Can
5er 9y5
Okes Sys.
Capy
Exsy
0.0
Day
Eay
0.2
q2b-332b
Musuum
91.7
41.7
33.3
Meduum
0.2
0.4
1.1
Moduun
83.3
41.7
16.7
Peaker
0.9
1.1
3.9
Hard
Hard
Rard
Ven.Ca
Syn
Con Sya
Skmu5ps
CamdSpe
5m55p
Cam
VintEs
Sm..0p.
Dom Sty.
0.0
0.3
qwen-3 3-25b
Medium
91.7
16.7
0.0
Medium
0.4
0.8
0.8
Med km
91.7
50.0
16.7
Medkir
0.0
1.3
2.8
100.0
91.7
66.7
Esay
0.0
0.0
0.0
100.0
91.7
91.7
0.0
0.0
0.0
gemini-2.5 2.5sh
Medium
21.7
91.7
0.0
1.0
Metun
58.3
Medker
0.0
$0.0
Hend
0.3
3.1
25.0
25.0
0.0
0.8
Syn
Sye
Var.E
Sindps. Com-lye
Wan
Syt.
Cam
Skm
100.0
$0.7
100.0
Eaoy
0.0
0.0
0.0
100.0
91.7
91.7
0.0
0.0
0.0
gemini-2.5-pro
53.3
75.0
16.7
Herd
0.0
0.4
0.7
Hard
85.3
50.6
35.3
0.0
0.0
0.1
Figure 10: Full LLM Performances in domain Magnetostatics (Ampère's Force Law).
35

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanills)
(Code)
RMSLE (Code)
50.0
0.0
0.0
Edoy
1.7
2.9
3.0
65.
15.7
0.0
1.2
2.8
4.8
spt-4.2-mini
H0We
25.0
0.0
0.0
16m1
0.4
7.7
7.5
Had
50.0
25.0
0.0
Har
0.0
2.8
16,9
Ves.in
Syr
Eres Sye
Sir.pps
Com-ays
kn
Cor Syt
Viniis
54n Spt.
Com Set.
0.0
Endy
2.2
4.2
Exdy
75.3
0.0
1:1
5.1
9pt-8.1
0.0
1.0
17.4
81.
16.7
0.0
1.2
5.4
16.7
8.3
0.0
Herd
6.9
6.0
9.0
Rard
50.0
41.7
0.0
herd
0.0
2.9
5.2
Wen-Do
Syn
Don
Vanua
1into
5m5
Cam
Ven
Oon Sya.
Casy
Eady
Daay
Eay
04-mini
Musium
100.0
0.0
25.0
Medium
0.0
15.6
10.3
Modiu
100.0
8.3
0.0
Veskmr
0.0
13.4
7.1
Hart
Hand
0.0
3.5
Hand
Hard
Syn
Con Sys
Sanda
Skm.5ps.
Camuppe
Va1
5m
San
Vin
Sir.Sya.
Com Syx.
0.4
0.0
0.0
apt-5-mint
Medium
75.7
5.0
11.7
Medium
0.8
14.5
12.0
Madun
60.0
36.7
6.7
Medker
0.0
7.8
13,8
100.0
100.0
100.0
Easy
0.0
0.0
100.0
91.7
66.7
0.2
1.0
gpt-5
Medium
100.0
Medium
0.0
3.0
Mediu
100.0
100.0
Medker
0.0
100.0
83.3
66.7
0.0
2.8
0.0
100.9
100.0
16.7
0.0
0.0
5.4
Syr
Syc
Com-ipt.
Wiiif
Syt.
Corn
9im
75.0
8.3
8.3
Eody
1.0
3.7
5.2
100.0
41.7
9.3
0.0
2.2
6.6
deepsenk--33
HOVE
8.3
0.0
0.0
1800
0.7
8.3
12.0
Hvn
41.7
8.3
0.0
Hart
0.4
8.3
6.5
Wee.bo
Sys
Vintt
9mm.pp Camd5rs
Vaste
5m
Com
Vyn
5en
Sva.
$00.0
75.0
Exdy
0.0
1.7
100.0 0
0.9
dempsook-r1
100.0
41.7
0.0
4.6
91.7
56.3
8.3
6.0
6.0
Hire
100.0
25.0
0.0
Herd
0.0
8.9
14.8
Rard
83.3
25.0
0.0
herd
0.0
3.7
5.4
Wee.0g
54h Suy
Doe Sys
Vilut
Candees
Vexte
5 Sps
Can
5er 9y5
Oce Sys.
Cxp
Exsy
0.7
Doy
75.0
Eay
q2b-332b
Mustum
83.3
0.0
0.0
Medum
0.8
6.4
15.4
Moduu
100.0
8.3
0.0
Peaker
0.0
13.5
36,0
Hard
Hard
Ven.Ca
Stn
Con Sy
Skm.5ps Cam:5pe
5m
Cam
Ven
Stm.Sp.
Com Sya.
25.0
0.0
3.9
16.7
16.7
2.9
qwen-3- 235b
Medium
91.7
8.3
8.3
Medium
0.2
16.1
3.2
Med km
75.0
0.0
0.0
Meakir
4.6
16.4
5.0
100.0
91.7
25.0
Esay
0.0
0.2
1.8
100.0
83.3
16.7
0.0
0.4
0.0
gemini-2.5 flfsssh
Medium
91.7
93.3
0.0
11.7
Motun
100.0
Medker
0.0
56.7
$0.0
Hend
0.0
0.0
3.5
100.00
75.0
Syn.Sye
Sye
Var.E.
Sin.pss Com-lys
Wan
Syt.
Cam
Skm
100.0
100.0
58.3
Eady
0.0
0.0
0.0
100.0
100.0
41.7
0.0
0.0
0.1
gemini-2.S-pro
91.7
91.7
8.3
Herd
0.5
0.5
0.8
Hard
100.3
75.0
33.3
0.0
0.0
0.4
Figure 11: Full LLM Performances in domain Thermal Conduction (Fourier's Law).
36

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanills)
(Code)
RMSLE (Code)
16.7
16.7
0.0
Eddy
0.3
0.2
0.8
0.0
0.0
0.0
0.3
6.3
0.3
spt-4.2-mini
H0We
0.0
0.0
0.0
16m1
0.6
0.5
0.8
Hond
0.0
0.0
0.0
Har
0.7
6.8
6.8
Vee.in
Sye
Eres Sye
Sir.pps Cam-ays
Sys.
Cor Syt
Vin
5er Spt.
com fat.
0.0
Endy
0.3
0.1
Exity
33.3
0.3
0.2
0.1
gpt-8.1
0.0
0.0
0.0
0.6
0.8
0.7
0.0
0.0
6.8
6.
0.0
0.0
0.0
Nerd
0.6
0.7
0.7
Hard
0.0
0.0
0.0
hard
0.3
0.7
0.8
Ven.bg
Sen Syp
Don Dys
Vanua
Cam
Sonte
5m Spt
Con
Ven
5mn 353
Oom Syy.
Casy
Exdy
Daay
Eay
04-mini
Musium
100.0
100.0
58.3
Medium
0.0
0.0
0.0
Modium
91.7
100.0
50.0
Meskmr
0.0
0.0
0.0
Hart
Hand
0.4
0.5
Hand
Hard
0.5
Hen.0q
1im Syp
Com Sys
Sanda
Sir.pps.
Camsppe.
Van
Spt.
Sam
Vnn
Spm.5p
Com Syx.
0.0
0.6
0.0
apt-5-mint
Medium
91.7
91.7
41.7
Medium
0.0
0.0
0.6
Mad
58.3
91.7
33.3
Medker
0.0
0.0
0.2
100.0
100.0
75.0
Easy
0.0
0.0
91.7
41.7
0.0
0.5
gpt-5
Medium
100.0
150.0
Medium
0.0
0.0
Med
100.0
100.0
Medker
0.0
100.0
91.7
50.0
0.0
0.0
0.3
58.3
66.7
58.3
0.0
0.0
0.0
Syr
Sye
SinSpys
Com-ipt
Car
Sim.Sy
Eile
25.0
16.7
0.0
Eady
0.4
0.3
0.7
Fliy
25.0
15.7
0.0
0.4
0:2
0.3
deepsenk--33
HOVE
0.0
0.0
0.0
1800
0.6
0.7
0.6
Han
0.0
0.0
0.0
Hart
0.6
6.6
0.9
Wee.bo
1im
Due Sys
Vintt
5mm.pp
Cam-5ra
Vaste
5m
Com Sys
Vyn
5m
Dum
$00.0
61.3
$0.0
Easy
0.0
0.2
83.3
25.0
0.0
despsokk-11
91.7
75.0
58.3
0.0
0.0
83.3
100.0
50.0
0.0
0.5
Hired
36.7
16.7
0.0
Herd
0.4
0.7
0.7
Hard
8.3
25.0
0.0
herd
0.4
0.3
0.7
5mn Sys
Don Sys
Vaaus
Vasto
5m Sp
5er 9y5
Oces Sys.
Cxpy
75.0
Exsy
0.0
0.1
Day
100.0
Emy
0.0
q2b-332b
Musium
75.0
66.7
41.7
Medumm
0.0
0.0
0.2
Moduun
100.0
83.3
16.7
Peaker
0.0
0.0
0.6
Hard
Hand
Hard
Ven.Ca
Con Syp
SkmuSps
CamdSps
Vants
5m
Cam
Ven
Sm.
Dom Stx.
50.0
25.0
0.0
0.7
91.7
qwen- 3-235b
Medium
66.7
50.0
16.7
Medium
0.0
0.0
0.6
Med km
91.7
91.7
16.7
Meski
0.0
0.0
0.1
0.5
0.4
83.3
75.0
41.7
Esay
0.0
0.0
0.2
58.3
50.0
0.0
0.0
0.2
1.4
gemini-2.5 2.5sh
Medium
66.7
0.0
0.6
50.0
Medker
0.0
16.7
16.7
0.0
Hend
0.3
1.6
18.7
0.0
0.4
0.7
Syn
Sye
Var.E.
Sin.pps Com-lye.
Van
5m Syt.
Cam
Wi
Skm
Syx.
100.0
100.0
25.0
Eady
0.0
0.0
0.1
91.7
91.7
50.0
0.0
0.0
0.0
gemini-2.S-pro
100.0
100.0
25.0
Herd
0.0
0.0
0.8
Hard
85.5
85.1
25.0
0.0
0.0
6.5
Figure 12: Full LLM Performances in domain Geometrical Optics (Snell's Law).
37

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanills)
(Code)
RMSLE (Code)
0.0
0.0
0.0
Eddy
0.0
0.0
0.0
0.0
15.7
0.0
0.0
0.0
0.0
spt-4.2-mini
HOWe
0.0
0.0
0.0
1.3
1.1
0.9
Hond
0.0
0.0
0.0
Har
1.3
1.1
0.9
Vee.ing
Eres Sye
Sir.pps
Cam-ays
9n Sys.
Cor Syt
Vin
54r Spt
Com fat.
0.0
Endy
0.0
0.0
Exiyy
41.7
25.0
0.0
0.2
gpt-8.1
0.0
0.0
0.8
0.0
0.0
0.0
0.4
6.N
0.0
0.0
0.0
Herd
1.2
1.1
0.7
Hard
0.0
0.0
0.0
hard
0.6
1.0
1.0
Ven.tg
Sen Syp
Don Sys
Vanta
Sonte
Syt
Cor Smn
Ven
5mn 353
Oon Sya.
Casy
Exdy
Daay
Eay
04-mini
Musium
58.3
41.7
16.7
Medium
0.0
0.0
0.0
Modium
41.7
41.7
25.0
Meckir
0.0
0.0
0.0
HarE
Hand
0.0
0.3
Hand
Hard
Syp
Con Sys
Sanda
Skr.pps.
Camuppe.
Van
Sps.
Sam
Vin
Skm.5p
Com Syx.
75.0
0.1
0.0
46.7
36.7
0.0
apt-5-mint
Medium
46.7
18.3
15.0
Medium
0.0
0.3
0.9
Med
16.7
10.0
3.3
Medker
0.4
0.8
1.0
100.0
100.0
91.7
Easy
0.0
0.0
100.0
100.0
100.0
0.0
0.0
gpt-5
Medium
100.0
Medium
0.0
0.0
Med
100.0
100.0
Medker
0.0
75.0
66.7
58.3
0.0
0.0
0.0
91.7
75.0
66.7
0.0
0.2
0.0
Syr
Sye
Sin.ptt
Com-ipt
9in Syt.
Cars
9km
0.0
0.0
0.0
Eody
0.0
0.3
0.1
Flity
33.3
0.0
9.3
0.0
0.3
0.0
deepsenk--33
HOVE
0.0
0.0
0.0
1800
0.5
3.2
0.4
Han
0.0
0.0
0.0
Hart
0.8
1.0
1.0
1im
Dur bys
Vintt
5mm.pp
Cam-5ra
Vaste
Com
Vyn
5m Sv.
Dum
91.7
41.7
Easy
0.0
0.0
75.0
0.0
dempaokk-21
50.0
31.3
0.0
0.0
1.0
$0.0
25.0
25.0
1.0
0.8
0.0
0.0
0.0
Herd
0.0
0.9
0.8
Hard
16.7
0.0
8.3
herd
0.0
0.2
0.9
Sin Sys
Doe Sys
Vaaut
Vante
5 Sph
Can
5er Sy5
Oces Sys.
Cxp
Exsy
0.0
Dayy
Emy
0.0
q2b-332b
Mudium
0.0
0.0
0.0
Medum
0.3
0.9
0.9
Modium
16.7
0.0
0.0
Peaker
0.7
0.9
0.9
Hard
Hand
Rard
Ven.Ca
Dmn Syp
Skmu5ps Cam.5pe
5m55p
Cam
Veniia
Sm.
Dom Stx.
66:7
0.0
0.0
25.0
qwen- 235b
Medium
25.0
8.3
0.0
Medium
0.0
0.9
0.9
Med km
25.0
0.0
0.0
Meakir
0.0
0.9
0.9
1.0
100.0
91.7
41.7
Esay
0.0
0.0
0.0
100.0
66.7
33.3
0.0
0.0
0.0
gemini-2.5 5-fss
Medium
91.7
58.3
0.0
0.0
Metun
Medker
0.0
50.0
16.7
0.0
Hend
0.2
0.0
0.1
18.7
8.3
0.2
1.0
Syn
Sye
Var.E.
Sin.ps. Com-lye.
Vanii
9m Syt.
Com
Wi.
Skm
100.0
100.0
100.0
Eady
0.0
0.0
0.0
100.0
100.0
75.0
0.0
0.0
0.0
gemini-2.5-pro.
50.0
25.0
25.0
Herd
0.0
0.0
0.0
Hard
65.7
58.3
41.7
0.0
0.0
C.0
Figure 13: Full LLM Performances in domain Nuclear Physics (Law of Radioactive Decay).
38

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanills)
(Code)
RMSLE (Code)
0.0
0.0
0.0
Eddy
3.6
5.1
4.8
0.0
0.0
0.0
0.0
4.4
5.1
spt-4.2-mini
HOWe
0.0
0.0
0.0
5.5
5.3
6.1
Hond
0.0
0.0
0.0
Hari
7.5
6.7
5.9
1i Syc
Eres Sye
Sir.pps.
Com-ays.
Wiiig
Sys.
Cor Syt
Viniisg
Spr-Sy
Com fet.
0.0
Endy
0.0
5.1
Exiyy
16.7
0.0
4.3
3.0
gpt-8.1
0.0
0.0
0.2
5.4
0.0
0.0
6.6
0.0
0.0
0.0
Herd
7.1
7.5
8.0
Hard
0.0
0.0
0.0
herd
2.6
7.0
6.3
Wen.tg
Syn
Don Dys
Vanua
Sem.5pp
Sonte
5 Spt
Com
Ven
5km 353
Oon Sya.
Casy
Exdy
Daay
Eay
04-mini
Musium
66.7
8.3
8.3
Mudium
0.0
0.0
1.5
Modium
50.0
16.7
0.0
Meckir
0.0
0.0
0.3
Hart
Hand
0.0
Hand
Hard
0.0
Syn
Con Sys
Strupps.
Camsppe.
Va
Spt.
Sam
Vin
Sim.5pa
Syx.
0.1
0.0
25.0
0.0
apt-5-mint
Medium
75.0
50.0
0.0
Medium
0.0
0.0
0.1
Med
58.3
16.7
0.0
Medker
0.0
0.0
0.0
91.7
91.7
41.7
Easy
0.0
0.0
100.0 3
66.
50.0
0.0
0.0
gpt-5
Medium
100.0
Medium
0.0
0.0
Mediu
100.0
75.0
Medker
0.0
100.0
75.0
16.7
0.0
0.0
0.0
100.9
50.0
16.7
0.0
0.0
0.0
Syr
Syc
Cam-ipt
Cor
Viniiss
Sim.Sy
8.3
8.3
0.0
Eody
0.0
4.4
4.4
9.3
0.0
0.0
0.0
3.9
3.9
deepsenk--33
HOVE
0.0
0.0
0.0
1800
2.3
7.2
6.4
Han
0.0
0.0
0.0
Hart
4.2
7.5
6.9
Wes.b
1em
Sys
Vintt
Sam.5ps Can-5ra
Vaste
5m
Com
Vyn
59n
Dum Svs.
100.0
Eady
0.0
0.0
81.3
0.0
despsnkkrr1
33.3
16.7
0.0
0.0
4.3
18.7
15.7
6.0
Hirdd
36.7
0.0
0.0
Herd
0.2
0.2
5.0
Hard
0.0
0.0
0.0
herd
0.3
0.0
4.7
Wee.og 20
Sin Sys
Doe Sys
Vaauh
Vaste
5 Sps
Con
5er 9y5
Oces Sys.
Casy
Exsy
0.0
Day
Eay
0.0
q2c-332b
Musium
16.7
8.3
0.0
Medum
0.2
3.4
4.4
Modiun
0.0
8.3
0.0
Peaker
0.1
0.1
4.6
Hard
Hand
Rard
Ven.Ca
S/n
Com Stp
SkmuSps.
CamdSpe
5
Cam
Ven
Stm.5p.
Stx.
91
0.1
qwen- 3-53
Medium
58.3
8.3
0.0
Medium
0.0
0.1
5.1
Mad
15.7
8.3
0.0
Meakir
0.0
0.0
2.5
91.7
50.0
16.7
Esay
0.4
0.0
0.0
58.3
25.0
8.3
0.7
0.0
0.0
gemini-2.5 flasssh
Medium
50.0
Medium
0.2
0.6
Metun
Medker
0.0
31.3
16.7
0.0
Hend
0.2
0.1
25.0
0.0
0.0
0.9
Syn.
Van.Eg
Sindps. Com-lye
Wante
Cam
Wi
Skm
91.7
75.0
25.0
Eaoy
0.0
0.0
0.0
Eiiry
75.0
58.3
33.3
0.9
0.0
0.0
gemini-2.5-pro
58.3
16.7
0.0
Herd
0.0
0.0
0.1
Hard
$0.0
0.0
0.0
0.0
6.2
2.1
Figure 14: Full LLM Performances in domain Oscillations (Law of Damped Harmonic Motion).
39

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanills)
(Code)
RMSLE (Code)
0.0
5.3
0.0
Edoy
2.2
2.7
2.4
15.7
15.7
0.0
2.8
1.0
1.9
spt-4.2-mini
HOWe
0.0
0.0
0.0
3.9
3.9
4.5
Han
0.0
0.0
0.0
Har
3.2
1.6
5.4
Vee.ing
1i Syc
Eoes Sye
Sir-pps
Cam-ays.
Wiig
kn
Cor Syt.
Vin
Com fey.
16.7
Endy
1.5
2.5
Exty
0.3
11.3
2.3
0.7
2.2
gpt-8.1
0.0
0.0
2.0
2.7
0.0
0.3
0.0
0.7
0.0
0.0
0.0
Nerd
5.2
8.5
3.1
Hard
0.0
16.7
0.0
hard
2.9
0.4
5.1
Wen.bg
Sen 5yp
Don Dys
Vanta
Camd5pa
Sinte
5m
Can
Ven
5em 353
Otn 2y..
Casy
Eady
Daay
Eay
04-mini
Musium
100.0
83.3
0.0
Muduum
0.0
0.4
3.3
Modiu
100.0
833
8.3
Peckir
0.0
0.0
2.9
Hard
33.3
Hand
0.6
Hand
Hard
0.0
Syp
Con Sys
Sanda
Skmupps.
Camsppe.
Va
Spt.
Spm
Vin
Sim.5ps.
Com Syx.
0.0
91.7
0.0
apt-5-mint
Medium
91.7
100.0 0
0.0
Medium
0.0
0.0
3.3
Med
75.0
75.0
16.7
Medker
0.0
0.0
2.6
100.0
100.0
100.0
Easy
0.0
0.0
91.7
100.0
100.0
0.0
0.0
0.0
gpt-5
Medium
100.0
150.0
Medium
0.0
0.3
Med
100.0
100.0
Medker
0.0
83.3
75.0
58.3
0.0
0.0
0.0
100.9
100.0
58.3
0.0
0.0
0.0
Syn
Syc
SinSpys
Cam-ipt
Syt.
Car
9km Spr.
Eile
16.7
16.7
0.0
EXoy
1.4
3.3
2.3
16.7
15.7
0.0
0.9
2.2
2.6
deepsenk--33
HOVE
0.0
0.0
0.0
1800
5.5
4.0
5.2
Han
0.0
0.0
0.0
Hart
1.3
4.8
3.8
Wes.b
fimm Sio Sys
Vintt
Sem.pps Cam-drs
Vaste
5m
Com Sys
Vyn
5en
Dun Sta.
100.0
91.7
Exsy
0.0
2.1
81.3
0.0
despsokk-r1
41.7
$1.3
0.0
0.2
3.2
18.7
15.7
1.5
2.1
2.9
Hired
36.7
16.7
0.0
Herd
1.2
0.7
2.8
Hard
33.3
8.3
0.0
herd
3.8
1.1
5.0
Sim Sys
Doe Sys
Vaauh
Camduee
Vaxto
5
Can
Vinia
5er 965.
Oke Sys.
Cxsy
Exsy
Day
Emy
3.1
q2b-332b
Musium
33.3
25.0
0.0
Medum
2.7
1.9
3.3
Moduu
25.0
0.0
0.0
Peaker
0.6
3.2
4.0
Hard
Hard
Haad
Ven.Ca
Con Syp
SkmuSps.
CamdSp.
5m
Cam
Ven.is
Stm.9p.
Dom Sta.
0.0
2.4
25.0
1.9
qwen- 3-535b
Medium
16.7
25.0
0.0
Medium
1.2
1.1
2.5
Mad km
15.7
0.0
0.0
Meskir
1.2
3.1
3.9
2.0
100.0
91.7
41.7
Esay
0.0
0.0
1.0
83.3
59.3
8.3
0.0
0.0
2.2
gemini-2.5 flash
Medium
83.3
75.0
Mediin
0.0
2.4
Metun
Medker
0.0
$1.3
16.7
0.0
Hend
0.5
4.5
8.3
8.3
1.1
3.0
Syn
Sye
Var.E.
Sindpps Com-lye.
Wante
Car
Wi.
Skm
100.0
100.0
100.0
Eady
0.0
0.0
0.0
100.0
100.0
91.7
0.0
0.0
0.0
gemini-2.S-pro
50.0
43.7
16.7
Herd
0.0
0.5
2.1
Hard
$0.0
50.6
16.7
0.1
0.0
1.7
Figure 15: Full LLM Performances in domain Physical Optics (Malus's Law).
40

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanills)
(Code)
RMSLE (Code)
41.7
25.0
8.3
Edoy
2.3
4.5
0.5
100.3
91.7
75.0
0.0
6.0
0.0
spt-4.2-mini
2.2
HOWe
0.0
0.0
0.0
1601
2.4
3.4
0.8
Han
25.0
25.0
0.0
Har
0.1
6.1
2.5
14 Syc
Coes Syt
Sir-pps.
Com-ays
9m kys.
Cor Syt
5er Spt.
com fat.
11.3
Endy
2.8
0.0
Exty
100
100.0
83.3
0.0
0.0
gpt-8.1
0.0
0.0
31.1
3.3
0.1
91
25.0
0.0
0.0
0.0
8.3
Nerd
4.6
3.8
0.5
Hard
22.3
33.3
0.0
hard
0.1
0.3
0.4
Wen.Dg
Sen Syp
Don Sys
Vanta
Sem.5pp
5m
Cor 5m
Ven
5mn 35
Oon Sya.
Casy
Eady
Dayy
Eay
04-mini
Musium
100.0
100.0
91.7
Medium
0.0
0.0
0.0
Mndiu
91.7
91.7
83.3
Veckmn
0.0
0.0
0.0
Hart
Hand
0.0
Hand
Hard
0.0
Hen.0q
1im Syp
Com Sys
Sarda
Sir.pps.
Camuppe.
Va1
Spt.
Spm
Vin
Spm.5pa
Syx.
0.0
0.0
100.0
100.0
0.0
apt-5-mint
Medium
100.0
100 0
66.7
Medium
0.0
0.0
0.0
Mad
100.0
91.1
66.7
Medker
0.0
0.0
0.0
100.0
100.0
100.0
Easy
0.0
0.0
100.0
100.0
100.0
0.0
0.0
gpt-5
Medium
100.0
Medium
0.0
0.0
Med
100.0
100.0
Medker
0.0
100.0
150.0
91.7
0.0
0.0
0.0
83.3
41.7
33.3
0.0
0.0
0.0
Syr
Sye
SinSkys
Com-ipt
Cart
Sim-Sys
58.3
58.3
25.0
Eooy
1.4
1.5
0.9
Fkiy
58.3
41.7
41.7
0.0
0.6
0.0
despsenk--33
HOVE
0.0
0.0
0.0
1800
1.0
0.6
1.0
25.0
0.0
0.0
Hart
0.0
6.8
0.7
1mm
Dre
Vintt
5mm.pp Cam-rrs
Vonte
Com
Vyn
5mn
81.3
61.3
25.0
Eady
0.0
0.0
91.7
91.7
0.0
despsnkk-21
75.0
66.7
0.0
5.0
03.3
50.0
6.0
Hired
41.7
33.3
0.0
Herd
0.0
0.1
0.0
Rard
33.3
33.3
0.0
herd
0.0
0.0
0.2
5in Sys
Don Sys
Candee
Vaste
5 Sp.
Cor Sp
5er 9y5
Oce Sys.
Cxpy
75.0
Exsy
0.0
Eay
0.0
q2c-332b
Musuum
83.3
66.7
16.7
Meduum
0.0
0.0
1.1
Moduu
100.0
91.7
50.0
Peaker
0.0
0.0
0.0
Hard
Ven.Ca
Con Sy
SkmuSps Cam.Spe.
Vants
5m
Cam
Ven.aa
St.
Stx.
90.7
0.0
0.0
100.0
1.0.0
66.7
qwen-3 3-55b
Medium
75.0
75.0
8.3
Medium
0.0
0.0
2.2
Med km
63.3
91.7
50.0
Meckm
0.0
0.0
0.0
0.1
100.0
91.7
91.7
Esay
0.0
0.0
0.0
75.0
91.7
91.7
0.0
0.0
0.0
gemini-2.5 flash
Medium
21.7
91.7
75.0
0.0
0.0
Metun
100.0
83.3
Medker
0.0
$1.3
31.3
16.7
Hend
0.3
0.1
16.7
25.0
0.0
Syn
Sye
Var.E
Sin.ps. Com-lye.
Syt.
Com
Win
Skm
Syx.
100.0
100.0
100.0
Eacy
0.0
0.0
0.0
Eriyy
100.0
100.0
91.7
0.0
0.0
0.0
gemini-2.S-pro
75.0
$8.3
33.3
Herd
0.0
0.0
0.0
Hard
$0.0
41.7
35.3
0.0
0.0
0.0
Figure 16: Full LLM Performances in domain Acoustics (Law of Sound Speed in Ideal Gas).
41

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanills)
(Code)
RMSLE (Code)
Say Syc
Cundyyt.
25.0
0.0
0.0
Edoy
1.6
1.7
2.9
65.7
0.0
8.3
0.4
1.4
2.6
spt-4.2-mini
2.2
HOWe
0.0
0.0
0.0
1.9
4.2
5.5
Hon
0.0
0.0
0.0
Har
3.4
3.9
4.7
Coes Sye
Sirspps.
Cam-ays.
kn
Cor Syt
Vindis
com fet.
16.7
11.3
Endy
2.0
1:2
2.5
Extyy
66.
0.0
0.0
1:1
2.7
9pt-8.1
0.0
0.0
4.7
4.1
0.0
0.0
3.7
0.0
0.0
0.0
Nerd
6.2
7.1
3.4
Hard
0.0
0.0
0.0
herd
4.5
4.8
4.1
Ven.Do
Syn
Don Dys
Vanua
Sem.5pp
Sinte
Spp
Com
Ven
5en 353.
Oon Sya.
Casy
Exdy
Daay
Eay
04-mini
Musium
16.7
0.0
0.0
Mudium
0.0
0.0
0.1
Modiu
41.7
33.3
0.0
Veskmr
0.0
0.0
0.1
Hard
Hand
2.2
Hand
Hard
2.2
Sin Syp
Con Sys
Sanda
Str.pps.
Camsppe.
Va1te
Spt.
Can
Vin
Com Syx.
0.0
0.0
apt-5-mint
Medium
18.3
23.3
0.0
Medium
0.0
0.0
0.1
Med
6.7
6.7
0.0
Medker
1.8
2.2
0.2
58.3
75.0
$0.0
Eany
0.0
0.0
50.0
75.0
0.0
0.0
gpt-5 Medium
25.0
Medium
0.0
0.1
Med
Medker
0.0
33.3
41.7
0.0
0.0
0.0
1.9
25.0
16.7
0.0
0.0
0.2
1.5
Syr
Syc
Sindkys
Com-ipt
Cam
Viniiss
Eale
16.7
8.3
0.0
EXoy
0.2
1.5
2.6
58.3
0.0
0.0
0.0
0.6
2.5
despsenk--33
HOVE
0.0
0.0
0.0
1800
3.8
5.0
4.4
0.0
0.0
0.0
Hart
3.6
4.1
4.1
Due Sys
Vintt
Sam.pps Camdrrs
Vinte
5mbr..
Com Sys
Vyn
5en
Dum Sys.
81:3
61.3
Easy
0.0
0.0
75.0
0.0
dempaonk-r1
91.7
31.3
0.0
0.0
0.1
33.3
8.3
3.0
0.0
0.0
0.0
Herd
4.1
4.1
3.0
Hard
0.0
0.0
0.0
herd
4.4
5.5
2.3
Vee.0g
Sin Sys
Doe Sys
Varth
Canduee
Wexto
Sp.
Con
Vinia
Ser. 9y5
Okes Sys.
Cxp
Exsy
Day
Emy
0.0
q2b-332b
Mesium
25.0
25.0
0.0
Medum
2.4
3.0
3.6
Moduu
16.7
16.7
0,0
Peaker
3.1
2.9
4.5
Hard
Hard
Hand
Hard
Ven.Ca
Con Sy
SkmuSps
CamdSps
Vants
5m5
Cam
Ven.Ea
St..0p.
Dom Stx.
25.0
0.0
0.0
33.3
qwen-3- 235b
Medium
33.3
0.0
0.0
Medium
0.0
3.3
0.1
Mad km
33.3
8.3
0.0
Meakim
2.6
2.9
2.9
6.7
75.0
58.3
25.0
Esay
0.0
0.0
0.0
83.3
59.3
41.7
0.0
0.3
0.5
gemini-2.5 flash
Medium
50.0
0.0
0.1
Metun
Medker
0.0
0.0
0.0
0.0
Hend
0.6
2.2
0.0
0.0
1.0
2.6
6.8
Syn
Sye
Var.E.
Sindps. Com-lye
Wante
Syt.
Cam
W1
Syx.
75.0
58.3
33.3
Eaoy
0.0
0.0
0.0
33.3
75.0
25.0
EMS
0.0
0.0
0.1
gemini-2.5-pro
0.0
0.0
0.0
Herd
0.4
1.8
1.6
Hard
0.0
0.0
0.5
1.0
1.7
1.5
Figure 17: Full LLM Performances in domain Elasticity (Hooke's Law).
42

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanilla)
(Code)
RMSLE (Code)
Cundyyt.
5.3
5.3
0.0
Eddy
1.0
0.5
3.9
0.0
0.0
8.3
2.7
11.0
2.2
spt-4.1-mini
H0We
0.0
0.0
0.0
4.5
3.9
6.3
Hand
0.0
0.0
0.0
Har
3.3
6.5
10.3
Ves.in
Si Syc
Eres Sye
Com-ayy
Wiiig
Sys.
Cor Syt
Viniis
Spr.Sy
dom fet.
0.0
Endy
12:4
20.5
12.9
Exiy
16.7
0.3
EM
0.0
21.1
20.1
9pt-8.1
0.0
0.0
8.1
10.8
9.4
0.3
0.0
8.6
13.7
0.0
0.0
0.0
Herd
0.2
5.1
8.2
Hard
0.0
0.0
8.3
herd
0.1
0.1
8.0
Wen-Dg
Syn
Don Dys
Vanua
Candeps
Sonte
5 Spp
Con
Ven
5em 353
Don Sya.
Cary
Exdy
Dayy
Eay
04-mini
Musium
33.3
0.0
0.0
Medium
0.0
0.1
0.0
Modium
9.3
16.7
0.0
Medkmr
0.0
6.1
0.0
Hart
Hand
0.0
Hand
Hard
3.3
Van.0q
Syp
Con Sys
Sanda
Str.pps.
Camuppe.
Van
Sps.
Sam
Vin
Sim.Spa
Com Syx.
25.0
0.0
0.0
33.3
0.0
apt-5-mint
Medium
33.3
33.3
25.0
Medium
0.0
0.0
0.0
Mad
25.0
0.0
0.0
Medker
0.0
0.0
0.0
33.3
41.7
25.0
Easy
0.0
0.0
58.3
58.3
33.3
0.0
0.0
gpt-5
Medium
58.3
Medium
0.0
0.0
Medker
0.0
66.7
41.7
41.7
0.0
0.0
0.0
58.3
41.7
50.0
0.0
0.0
0.1
Syr
Sye
Vir.Ft
Sindyys
Com-iyt
Cor
Sim.Sy
Eale
16.7
8.3
0.0
EXoy
15.9
17.1
13.3
0.0
25.0
16.7
0.0
12.1
16.6
despsenk--33
HOVE
0.0
0.0
0.0
1800
2.3
6.7
4.7
Han
0.0
8.3
0.3
Hart
5.5
6.2
4.8
tex Sto Sys
Vinut
Camdyra
Vonte
Sm
Com Sys
Vyn
5mn
25.0
Eady
0.0
0.0
Eary
18.7
0.0
dempaook-r1
33.3
0.0
0.0
0.0
0.0
6.3
0.0
3.8
0.7
Hire
16.7
0.0
0.0
Herd
1.2
3.1
0.9
Rard
16.7
0.0
0.0
herd
0.3
4.9
0.3
Wee.0g
5mm Sys
Doe Sys
Varuh
Candee
Sants
Sm
Con
Ser 9y5
Oces Sys.
Cxpy
Exsy
0.0
Day
Emy
0.0
q2b-332b
Mudium
0.0
0.0
0.0
Medum
0.0
8.4
7.6
Moduum
0.0
0.0
0.0
Peaker
0.8
0.0
0.1
Hard
Hand
Hard
Ven.Ca
Syp
Con Syp
Skm.5ps Con-Sge
Vants
Sps.
Cam
Veniia
Com Stx.
0.0
25
25.3
16.7
12.6
qwen-3-22 235b
Medium
0.0
0.0
0.0
Medium
0.0
0.1
0.1
Mad km
0.0
8.3
16.7
Meskmr
0.0
0.3
0.3
0.1
75.0
33.3
25.0
Esay
0.0
0.0
0.0
50.0
41.7
16.7
0.0
5.9
0.0
gemini-2.5 flash
Medium
0.0
0.5
Metun
Medker
16.7
16.7
0.0
Hend
0.3
5.6
8.3
0.0
0.1
6.4
Syn
Sye
Var.E
Sindps. Com-lye
Wanto
Sm
Can
Wi.
Skm
Syx.
100.0
58.3
41.7
Eaoy
0.0
0.0
0.0
Eiiry
63.3
50.0
41.7
EMS
0.0
0.0
0.0
gemini-2.5-pro
41.7
33.3
41.7
Herd
0.0
0.1
0.1
Hard
41.7
41.7
25.0
0.0
0.0
0.0
Figure 18: Full LLM Performances in domain Statistical Mechanics (Bose-Einstede Distribution).
43

---

Published as a conference paper at ICLR 2026
(Vanilia)
RMSLE (Vanills)
(Code)
RMSLE (Code)
Cundyyt.
33.5
23.3
0.0
Eddy
4.5
3.5
5.6
83.3
6.3
0.0
0.0
2.1
5.2
spt-4.1-mini
HOWe
0.0
0.0
0.0
5.4
6.5
13.0
Had
100
8.3
0.0
Har
0.0
6.1
3.6
4ih Syt
Coes Sye
Sir-pps.
Com-ays
10:10
kn.
Cor Syt
Viniis
Com Set.
Eike
16.7
16.7
Endy
3.8
8.0
83.3
0.3
0.0
5.0
7.4
gpt-8.1
16.7
9.5
8.4
100.0
0.0
0.0
7.4
7.8
0.0
0.0
0.0
Herd
7.9
4.8
10.5
Radd
100.0
16.7
0.0
herd
0.0
5.7
7.9
Ven.Dg
Sen Syp
Oon Sys
Vanua
Cand5ph
Sonte
Sm
Com
Veniaa
5em 35.
Con Sy..
Casy
Eady
Dayy
Eay
04-mini
Musium
91.7
75.0
0.0
Mudumm
0.0
0.0
11.9
Modium
91.7
91.7
0.0
Peckmn
0.4
0.0
12,0
Hart
Hand
0.0
Hand
75.0
Hard
Syn
Con Sys
Sir.pps.
Camsppe.
Va1
Spt.
Sam
Vin
Spm.Spa
Syx.
98.3
0.0
16.7
apt-5-mint
Medium
95.0
81.7
13.3
Medium
0.0
0.0
6.5
Med
81.7
46.7
5.0
Medker
0.0
0.0
1.2
100.0
100.0
$0.0
Easy
0.0
0.7
100.0
100.0
50.0
0.0
0.1
gpt-5
Medium
100.0
Medium
0.0
1.7
Mediuy
100.0
100.0
Medker
0.0
91.7
16.7
0.0
0.0
0.0
1.1
100.9
25.0
0.0
0.0
0.0
0.2
Sin
Sye
Var.Ft
Sindyyt.
Com-ipt.
Cor
9in
66.7
0.0
0.0
EXdy
1.6
4.3
8.7
50.0
0.0
0.0
2.2
4.5
8.3
deepsenk--33
HOME
0.0
0.0
0.0
1800
3.6
6.4
9.1
Han
18.7
0.0
0.0
Hart
5.4
9.0
8.3
Wee.bo
Sys
Vinut
Sam.5ps Cam-Srs
Voste
5m
Com Sys
Van
5en
Sts.
$00.0
61:1
Eady
0.0
6.2
0.0
dempaokk-r1
100.9
41.7
0.0
7.1
100.00
16.7
6.0
9.6
Hord
75.0
0.0
0.0
Herd
0.0
0.0
7.8
Hard
75.
8.3
0.0
herd
0.0
5.9
8.5
Sys
Doe Sys
Camdhes
Veste
5 Sps
Con
5er. 9y5
Oces Sys.
Cap
Exsy
91.7
Eay
0.0
q2c-332b
Musuum
100.0
25.0
0.0
Medum
0.0
6.1
12.0
Moduu
91.7
41.7
0.0
Peaker
0.7
5.4
11.9
Hard
Hand
Rard
Ven.Ca
S/n
Com Stp
SkmuSps
CamdSpe
Vants
5m Cam
Venias
Stm.5p.
Com Sya.
50.0
0.0
5.4
qwen- 3-25bb
Medium
100.0
50.0
0.0
Medium
0.0
3.2
11.9
Mod km
58.3
58.3
0.0
Meaki
0.0
0.0
12.2
100.0
150.0
0.0
Esay
0.0
0.0
4.6
100.0
91.7
9.3
0.0
0.2
5.7
gemini-2.5 flash
Medium
100.0
91.7
Medi
0.0
5.7
Metun
100.0
50.0
Medkir
0.0
0.0
0.0
Hend
0.0
0.0
8.9
03.55
0.0
0.4
Syn
Sye
Var.Eg
Sindpps Com-lys.
Wan
Cam
Wi.
Skm-Sp
100.0
83.3
0.0
Eaoy
0.0
0.0
6.7
100.0
100.0
25.0
0.0
0.0
2.2
gemini-2.S-pro
100.0
25.0
0.0
Herd
0.0
0.0
7.2
Hard
100.0
8.3
0.0
0.0
0.0
8.9
Figure 19: Full LLM Performances in domain Calorimetry (Law of Heat Transfer).
44

---

Published as a conference paper at ICLR 2026
B.2 RESULTS OF NON- PONFPERFORM METRICS
In this section, we present various non-performance metrics, evaluated across all difficulty levels and aggregated over 12 domains for each LLM Agent:
Table
13
Average number of rounds
Table
14
Average number of experiments (a set of parameters processed)
Table
15
Average total tokens
Table
16
Average number tokens per round
45

---

Published as a conference paper at ICLR 2026
Table 13: Average amount of rounds all difficulty levels aggregated all 12 domains. Within each agent settings, the highest numbers are marked as bold, with the second-highed underlined.
Vanilla Equation
Simple System
Complex System
Average
LLM Agents
medium
hard
medium hard
medium
hard
Rounds
Vonilla Agent
GPT-411minii
7.00
6.89
6.90
6.49
6.64
6.56
6.54
6.79
6.85
6.74
GPT-4.1
7.89
8.59
8.63
7.22
7.42
7.35
6.99
7.12
7.19
7.60
04-mini
2.74
2.97
3.53
2.85
3.21
3.82
3.23
3.58
4.25
3.35
GPT-5-mini
2.19
2.36
2.88
2.16
2.40
2.76
2.78
2.90
3.03
2.60
GPT-5
2.41
2.38
2.60
2.60
2.65
3.19
3.13
3.40
3.89
2.92
DeepSeck-V3 V3
4.49
4.97
5.15
4.21
4.42
4.62
4.28
4.49
4.33
4.55
DeepSeck-RI
2.92
3.24
3.85
3.15
3.35
3.84
3.73
3.56
4.04
3.52
QwQ-32b
2.80
2.97
3.64
2.75
2.72
3.17
2.74
2.46
2.95
2.91
Qwen-3-235h
3.22
3.38
3.62
3.26
3.40
4.01
3.85
4.04
3.54
Gemini-2.5-flash
3.99
4.67
5.43
4.43
4.62
5.51
5.47
5.65
6.33
5.12
Gemini-2.5-po
3.06
3.08
2.92
3.07
3.36
3.16
3.47
3.12
Agent with Code Assistance
GPT-4.1-mini
8.10
8.56
8.77
8.52
8.62
8.90
8.61
8.94
8.94
8.66
GPT-4.1
8.02
8.82
9.65
9.08
9.60
9.96
9.51
10.25
10.25
9.46
04-mini
4.28
4.99
6.40
4.60
5.30
6.13
5.05
5.92
6.66
5.48
GPT-5-mini
4.01
4.83
5.51
4.61
5.25
5.91
5.17
5.51
5.94
5.19
GPT-5
3.74
4.28
5.33
4.35
5.08
6.35
5.36
5.90
7.13
5.28
DeepSeek-V3 V3
6.93
7.77
7.53
6.62
7.04
6.86
6.90
6.92
7.03
7.07
DeepSeck-RI
5.25
5.50
5.81
5.40
5.63
6.08
5.81
5.94
6.37
5.75
QwQ-32b
3.72
4.62
4.47
3.93
4.22
4.01
4.06
3.88
4.21
4.12
Qwen-3-235b
4.60
5.16
5.41
5.01
5,51
5.37
5.39
5.51
5.84
5.31
Gemini-2.5-faa
7.08
8.35
9.03
7.90
8.56
9.12
8.24
8.72
9.03
8.45
Gemini-2.55pro
5.45
5.68
6,26
5.45
5.98
6.77
6.17
6.40
7.40
6.17
46

---

Published as a conference paper at ICLR 2026
Table 14: Average Experiments across all difficulty levels aggregated all 12 domains. Within each agent settings, the highest numbers are marked as bold, with the second-highes underlined.
Vanilla Equation
Simple System
Complex System
Average
LLM Agents
easy
medium
hard
easy
medium
hard
easy
medium
hard
Experiments
Vanilla Agent
-PT-4.1-mini
44.78
42.48
41.40
32.37
33.89
32.07
35.49
36.64
37.27
37.38
GPT-4.1
104.70
109.40
110.08
70.47
71.97
74.01
60.66
61.22
62.79
80,59
o4-mini
16.78
18.45
23.76
13.21
15.66
21.15
14.97
17.13
21.22
18.04
GPT-S-mini
19.05
20.60
27.07
14.42
17.96
21.96
21.38
22.61
24.52
21.06
GPT-5
24.29
23.99
26.74
23.18
23.35
31.82
32.72
35.74
41.39
29.25
DeepSeck-V3
21.40
23.72
24.53
14.99
15.99
17.41
14.63
15.81
15.28
18.20
DeepSock-RI
21.63
23.70
27.12
17.26
18.81
21.44
17.81
16.96
18.00
20.30
QwQ-32b
12.94
13.02
12.28
9.74
5.74
7.48
9.26
Qwen-3-235b
10.60
8.08
8.18
Gemini-2.5-Aassh
22.44
25.52
29.17
20.19
21.35
23.47
21.08
22.65
24.81
23.41
Gemini-2..-pro
15.90
16.01
16.94
13.52
13.51
14.17
15.98
15.46
15.84
15.26
Agent with Code Assistance
GPT-4.1-mini
21.03
20.83
24.00
15.27
14.62
15.17
16.45
16.08
18.12
17.95
GPT-4.1
31,55
34.74
39.69
30.48
29.69
31.11
25.84
27.52
27.19
30.90
o4-mini
13.61
14.85
18.60
10.92
11.96
14.98
11.08
12.97
15.24
13.80
GPT-5-mini
16.39
20.12
24.03
15.47
18.44
21.84
17.51
19.81
23.70
19.70
GPT-5
20.59
21.20
24.21
20.06
21.94
26.08
26.42
27.37
33.02
24.54
DeepSeck-V3
14.11
15.74
16.28
10.61
11.52
11.90
10.41
10.98
11.15
12.52
DeepSoek-R1
12.49
15.01
16.60
12.00
12.94
11.49
10.35
12:54
QwQ-32b
9.03
10:27
9.81
6.35
6.22
5.38
5.27
5.39
7.08
Qwen-3-235h
8.10
8.37
6.03
6.71
Gemini-2.5-fashh
14.95
15.83
16.26
13.74
13.78
13.16
12.06
14.49
15.00
14.36
Gemini-2.5-pro
12.33
11.42
11.47
9.18
9.40
9.37
10.65
9.92
12.00
10.64
47

---

Published as a conference paper at ICLR 2026
Table 15: Average Total Tokens across all difficulty levels aggregated all 12 domains. Within each agent settings, the highest numbers are marked as bold, with the second-highest underlined.
Vanilla Equation
Simple System
Complex System
Average
LLM Agents
easy
medium
hard
easy
medium
hard
easy
medium
hard
Total Tokens
Vanilla Agent
GPT-4.1-mini
1213
1522
1492
2126
1726
1691
2002
1977
GPT-4.1
2510
2588
2623
2808
2897
3115
2820
o4-mini
4487
7629
15426
6695
9924
16613
10199
13326
19836
11571
GPT-5-mini
3508
6348
12236
5289
9188
13324
9301
11595
13915
9412
GPT-5
7397
9055
18334
12138
17479
29562
17730
24577
36035
19145
DeepSeek-V3
1562
1530
1738
1509
1672
1688
1487
1545
1554
1587
DeepSeek-R1
9733
13672
17530
14485
18100
19442
17203
18293
19687
16461
QwQ-32b
10335
13658
16725
12293
14827
18079
13841
14239
16415
14490
Qwen-3-235b
8560
12622
16366
12045
13190
16106
14653
14875
16736
13906
Gemini-2.5-flash
8702
21702
37980
18832
28682
38817
43646
33627
57040
32114
Gemini-2.5-pro
8823
15748
24543
14105
19246
26718
24380
26864
33422
21539
Agent with Code Assistance
GPT-4.1-mini
1906
2196
2322
2872
2920
3140
3565
3613
3701
2915
GPT-4.1
3030
3798
4814
4465
4728
5054
4518
5127
5267
4534
o4-mini
4310
7996
14774
6372
8983
12678
8948
12144
16121
10258
GPT-5-mini
3552
6964
9754
7029
9401
11778
9983
12034
13132
9294
GPT-5
5086
9034
17016
10747
16368
27063
19395
25450
35032
18355
DeepSeek-V3
1957
2143
2256
2107
2274
2269
2355
2333
2342
2226
DeepSeek-R1
10847
14842
17489
15566
18222
19245
17934
18490
19619
16917
QwQ-32h
7999
1.3668
17719
12156
1,5088
16889
15698
1,2466
17171
14621
Qwen-3-235b
6887
12136
16007
11303
13835
15822
13589
14268
15871
13302
Gemini-2.5-flash
5532
14134
15635
11682
13871
18230
19702
32988
27757
17726
Gemini-2.5-pro
9080
13219
20955
13803
18559
29463
24979
29074
38242
21930
48

---

Published as a conference paper at ICLR 2026
Table 16: Average Tokens per Round across all difficulty levels aggregated all 12 domains. Within each agent settings, the highest numbers are marked as bold, with the second-highen underlined.
Vanilla Equation
Simple System
Complex System
Average
LLM Agents
mediun
hard
easy
medium
hard
easy
medium
hard
Tokens Round
Vanilla Agent
GPT-4.Immiii
181
259
442
276
431
699
290
263
353
355
GPT-4.1
337
315
319
403
418
424
470
492
532
412
04-mini
1628
2537
3985
2128
2906
4010
3082
3684
4507
3163
GPT-5-mini
1597
2671
4035
2399
3717
4682
3299
4024
4670
3455
GPT-5
2718
3609
6277
4329
6203
8587
5482
6934
8996
5904
DeepSeck-V3
412
340
350
429
406
401
368
409
392
DeepSeck-RI
3476
4453
4955
4986
5992
5702
5068
5753
5591
5108
QwQ-32b
4226
5399
6073
5485
6609
6664
6751
7464
7410
6231
Qwen-3-235b
3182
4204
5210
4617
4874
5547
4352
4774
5126
4654
Gemini-22.5faan
2110
4112
6650
5774
7345
7064
6035
8660
5731
Gcmini-2..-pro
3013
5143
7941
6664
9014
7386
8759
10217
Agent with Code Assistance
GPT-4.1-mini
231
254
261
342
356
413
416
335
GPT-4.1
354
406
480
500
516
475
510
522
o4-mini
953
1503
2126
1558
1936
1645
2277
1691
GPT-5-mini
857
1325
1660
1936
1816
2122
2190
GPT-5
1242
1907
2755
2277
3003
3979
3465
4180
4942
3083
DeepSeek-V3
273
274
299
328
335
343
340
349
DeepSeek-RI
2122
2847
3221
3495
3435
3229
3312
3221
3108
QwQ-32h
2245
3229
4209
3265
3785
4402
4221
4340
4567
3807
Qwen-3-235b
1580
2384
3057
2435
2741
3120
2777
2818
2984
2655
Gemini-2.5-ffaak
744
1613
1767
1504
1651
2131
2405
3873
3198
2099
Gemini-2.5-pro
1643
2219
3272
2514
2983
4336
3852
4547
5227
3399
49

---

Published as a conference paper at ICLR 2026
C PROMPT TEMPLATES
C.1 GENERAL PROMPTS
Vanilla Agent System Prompt
are research universe. Your goal is to propoze experiments, analyze the data they return, and ultimately deduce the undarlying acientifia please nace the laws physics this univerae differ from those in own, You can perforn experiments to gather data but you must follow the protecol strictly.
discovering scientific
simulated
i-Mock[low:va
éescription provided. 2. Design set of experiments to test your hypotheses.
Analyze
Use "cron_expe-t pop_eexxpe-tet tag to aubmit your experimental inputs. The system will return the results cesporiment_ostpnne tag.
to 20 data points experimeati in
returned value is nan, indicates such as:
the calculation encountered an error,
range of [-i, 11) -verfleeError (a.g-, using on an extremely large ingut)
Valueerror (e-g.r using
outside
You may ignore any data points that return ran, as they not contribute to valid hypethesis Lesting.
Consl der adjusting your Input parametera to avoid inval id ranges and improve data coverage.
5. You can run to 10 rounds of experiments. Use then. wisely so that before submit:ing your final lax, ensure you have:
fully ored the experimental apace
Verified your hypotheses againat the data
made the
of the available zounds strengchen your conclusions
6. Only one action 1a allowed per round: either crun experiments or stinal_law>, cexperimen: output> before proceeding.
submit-ing exprrexperm,
8. You should verify your hypotheses by checking if the output from the experiments outpuz 9. when con"Ident, suanit final discovered law using the "etinallaaw tag. Thía ends the mission.
matches
your hypotheses.
Agent with Code Assistance System Prompt
research assistant tasked with discoriiing scientific laws simulated universe, Your goal la to propose experimente, analyze the data they return, and physics in
uitimately the underlying scientific this universe differ from those in our own. You can perform experiments to gather data you must follow protocol strictly.
asse
Aullesee:
**Math calicolatoons::
You are always encouraged to use the <pythor> Lag to asslat with any non trivial mathematical reasoning. This insiudes, is limited te: Performing exponentiation, logorithmic transformations, and other advanced math operat
Comparing predicted outputa from propoaed law against actual experiment results,
evaluate the acouracy your Performing sensitivity analysis mathemazical modeling understand variations in experimental conditions affect outcones.
Calculating metrics
aquared
hypotheses.
Bnhancedd Tool Uaev+:
Avoid Redundant Calla: Do not call the same tool with ideatical parameters than once.
Evaluate Before Repaating: Always review the tool's outpur before deziding call
azother toel. Only proceed if the result is incomplese, uaclear, or unsatisfactery. Hasze-Based Strategy*<: your Python strategically within turn for sterstive analysts and refinement.
50

---

Published as a conference paper at ICLR 2026
Agent with Code Assistance System Prompt (Experimentation Protocol)
-Workflow:
Analyze mission cescription provided,
2.
Deaign set of expeciments Lo Lest your hypotheses.
"crun exneeeeepeimmt> zag submit your experimental inputs.
4. The system will return the results (et most 20 sats of data per experiment) in an "cexperimee ment
5. You ment_outputt>> _output> tag. enaure you have:
10 rouncs. Use then wisely so that betore submitting rinal law,
fully explored
space
Verified your hypotheses against the data
made most
availsble
strengchen your conclusions
6. Use the '<pyshen>' tags to teat your hypotheses, perform colculations, explore data between experiments.
7. The system. will return the resulis from the 'spyhoons" Lags in 'gyythor_output>" Lag.
8. **ERITTCAT: Only one action par turre*: "xpur_exper tag "epython>" tag for calculations/atale lavinal_ivv tag submitting your
running experiments
lup to 1 call per tura)
discovered lax 9. AANO MIXINGAV: Never use multiple act types in the sare turn (e.g., Python
Experiment]
10. .-KO DUPLICATES+:; Neves multiple
the type wait for coxperimen:_outpit betore proceeding.
turn
12. After submit:ing
submit-ing epython?, wall <python_outpu.>> before proceeding.
13. You should verify your hypotheses by checking if the output from the experiments your hypotheses.
matches the outpuz
14. You should take advantage of <pythor> tags to do all the taska deem recessary within each suzn.
to refine your understending. 16. When confident, submit your final discovered law using the "finnal_aao>" Lag. This mission.
15. Analyze the resul:s frem
Importont Notes:**
You are equipped with tool: epythone, This tool is only for performing complex math calculstions (e.g., exponentiation, logarithms, data analysis, hypothesis testingl.
ANNEVERVA use the "run_pythen_coode tool to run experiments or submit rinal laws. using "crun_epperiment>>
'<finsi tags
respectively.
**NeVrr** include any commenta ins' the submi tted lawa
Always respond with the appropriate Lag when submitting expezimeats final laws. The eavironment will handle execution and feedback.
Run Experiment Instruction
Condition Without Noise
kvHow to Experiments:
To gather datá, the parametera for one or acbitrarily many experimental sats. Note that all measurements returned by aystem evnai se-frea*s, You accurate and delecministic.
must use the crun_experiment> Provide JSON array specitying
perfectlyy
Condition With Noise
kvlow to Experiments:*
тa gather date, you must use the <run _oun__eepsmmet tag, Provide JSU8 array speci parameters for one or a"bitrarily nany experimental sass. measurements returned by the system subjees to erandon noise*s, simulating the Impertections real-world sensors.
51

---

Published as a conference paper at ICLR 2026
Submission Requirements
svFinal Submissionsex
Once you confident
determ' red undarlying force submit your findings a8 single Python Lunction enclosed in <final_lawe Lags.
*vSubmission Requiremente:4
The function must be nemed 'dlecovered_aav"
The funetien signature
be exactly: "éuuetibn_oignnttrel
3. The function should return |saturn ceturn_essaription). that
influence rinal torce, you should simply ignoze that variable within your function's logic rether then
you
parameters
changing the signature.
define the constant local variable inside the function body. Do NoT include the constant as Sunction argument. import necessary libraries inside the function le.g, math, numpy, stc.) if needed
II law contains any constants,
.<Critical Boundaries:**
Do MCT include any explanation or commentary inside the sfinal_law blocks and the function body,
only cutput the <final_lant block in your finel answer.
(example)
*eminddr::*
Always remember that the laws of physics in this universe may diffez from those our including factor dependency, constant scalars, and the form of the law. when doing the experiments, use a broad range of input parameters, for exomple, apanning from 10"-3 10"15 ersure robustnaas acroas soales.
C.2 DOMAIN-SPECINC PROMPTS
The prompt templates for domain-specific instructions provided in Supplementary Materials.
52

---

Published as a conference paper at ICLR 2026
D CASE STUDIES
NEWTONBENCH introduces an active exploration paradigm to enable LLM agents to interact with virtual environments and discover novel physical laws, Figure 20j illustrates the paradigm difference between the previous passive observation paradigm and our active exploration paradigm.
Passive Observation
Active Exploration
Static Dafaset
Initial Experiment
esperiments, varying
XI
Forze
1.0
1.3
5.67e-3
Laeie wtfecto.
ENV
1"23 fa2t: "diact: 1.0,
(aal% tatt: 1, "diett; 2.0, ...h
4/ron experinens?
1.0
110
("Lira: Locele "oulocicy:: 1-b.0ler5,
("Lintt Lorale "pulocicy", 1-2.0/e-8, ...ll /e/egerimett oetputs
Pattern Search
Hypothesis Verification
Hasel doubled Parce quedrupied
Using wit
the aste chat (-2)) F at"2,
Ansiyza Distance: Coeparing
verley this typetheess by teating
cne-forrth...
crun especirantsi Lsf, lec,
...h.
Access Constraint
cannet zuqusst additiional
Kengerioent extputs
data Although sypotharia, will proosed
"ee.onity": (-b.tle
0
c..i//apaziientt outpots
subeit my proposed Lax.
Law Submission
Law Submission
The data ratches the Sypothesis partectly. The Lex concicred.
Lavo
CELrB) Jav
15.514 05 #21 Idtat*22)
diacevered Jamial, 22, diati:
c/final Lan
</tinal 1s0
Figure 20:
An illustration of the active exploration paradigm presented in NEWTONBENCH, in comparison to passive observation paradigm previous symbolic regression benchmarks.
In this section, we provide four case studies, two from GPT-4.1 (Vanilla Code Assisted, in Ap- pendix[D.] and two from Gem (Vanilla / Code Assisted, Appendix (D22). Reasoning steps are partially summarized into objective narration for conciseness. The case studies intuitively demonstrate the effectiveness code assistance offloading computation for GPT-4.1, while undermines active exploration for Gemini-2.5. 1immi-2.--ffases 5-5-sh.hh
53

---

Published as a conference paper at ICLR 2026
D.1
CASE STUDY FOR NON-REASONING LLMS
D.1.1 VANILLA NON-REASONING LLMS
Vanilla GPT-4.1's Failed Conclusion of Easy Vanilla Equation domain Gravitation
SYSTEM:
// The modal is taaked with discovering a scientiffe fic. law in stmulated unt verse... only run experiments
access to assistance.
ENVIRONMENT:
// is find the law of gravitation batween
The ervironment
// can measure the torce between "maas"" and "maas2" at a given 'diatance",
LIM:
// The modal runs wide-ranging set of 20 experiments, varying 'massl"y 'mass2 y and // 'diatance" acroas many orders magritude to gather diverse initial datasat. <run_ expee Lment>
\"massl": 1e-1á, "mass2"; 1e-15, "distance"; 1e-8),
\"masel"; le15, "mass2"; le15, "distance"; 1e8), and 18 other vide-sanging experiments
</ron_experiment>
ENVIRONMENT:
<experiment _otppetttttttttt>>ttt (*6. 676.672--," "6. "6.6411+1"", c.. </experiment_outple
LIM:
// The modal attempta find the rolationship with distance sumning additional
// experiments in which 'distance" is varied exponentially while 'mass" and "maas2 // <run_experiment>
("massi": 2, "mass2": 2, "distance": 4),
("massi": 2, "mass2"; 2, "distance": 8),
("maas1"; *maas2": 2, "distance": 16),
("maael"; 2, "mass2"; 2, "distance": 32), ...
experiments
</ron_oxporsmentt>
ENVIRONMENT:
Lexperiment_outpure [*3. (r2.337000030000000ee
"1.1198016644499555--55 ...]
</exparimont_ootpprr
// The model requests for more experiment to collect more data
LIM:
1/ Tacking the abi ity to do prec' se computation, the model cannot identify the true though collected large // Instead, it defaul:s to known law from our universe (inverae-square), leading to an // insorrect cenclusion.
1/ exponent (1.5) for 'distance"
<tinal_law
def discovered law imassi, mass2, distance): 664e massl mass2) (distance </tinal_law>
54

---

Published as a conference paper at ICLR 2026
D.1.2 CODE-ASSISTED TON-REASONING LLMS
Code-Assisted GPT-4.1's s Successful Deduction of Easy Vanilla Equation in domain Gravitation
SYSTEM:
// The model is casked with discovering scientific 1aw in simulated universe.
// Other than running experiments, it Le given access to Python interpreter.
ENVIROMMENT:
// The task is to find the law of gravitation between two massed, The environment the force between 'massi" 1 and "mass2" given 'distance',
//
// The model conducts experiments varying 'maasl", , "nass2', and "diatance" across // different orders of magnitude to obtain initial daza. <ren_oxperiment>
(mmassi: 1.0, "mass2*: 1.0, "distance": 1.0),
("mas51": 1e3, "masa2": 1e3, "distance": 163), ["maasl*; 1e6, "masa2*; 1e6, "distance"; 1e3), and 7 other experimenta
e/pemmnxpermment
ENVIRONMENT: rexperiment_ousee
[6. 674e-05, 0.00211, 2110.5, ...] </expariment_outpic
LIM:
// The model offloads the cognitive load of calculation the tool. It the tool to // tests the standard inverse-square law hypothesis against the data. <python>
import cumрy as force ag.array (16.67666664-03) 0.03211, ...)) # Test if is proportional to mlxm2/d"2 pred force 6.674e-5 a (massl mass2) (distance ** 2) ratios force force print ("Ratios (moasurad/prodiccedd:: ratios) </python>
ENVIRONMENT:
// The result of the code is returned, providing evidence shat the hypothesis is wrong. <python_output>
Rotios (nossureddprediccted) 111, 1. 1. 1,414 1,414 1, 3.162 31.622 ...
</pythor_output>
LIM:
// The modal employs the tool conduct numerical search for the unknown exponent. // This mitigatea the most computat intenaive cask, which the model is unable to // perform. independen:1y. <python>
import zumpy np
(loads data!
ml m2 / d'n for different to fit n (massl, mass2, distance, force) def
np. ipp.linpacee 4.0, best_n = None
... (iterates through possible exponents best_n
erroz)
best_ (massl, masa2, distance, force)
print (["Best rit exponent a: [n]")
</python>
55

---

Published as a conference paper at ICLR 2026
Code-Assisted GPT-4.1's Successful Deduction of Easy Vanilla Equation in domain Gravitation (Law Submission)
ENVIRONMENT:
// The precise best-fit exponent is returned, a valua the model cannot di scover without // conputational assistance, <python_output>
fit exponent n: </pythor_ourput>
// Utilizing the Python interpreter tool, the model successfully derives and submits //
final
<final_ 1aw>
def discovered Law imassi, mass2, distance):
= 6.6746-05
return (C * masal x maas2) / (diatance ** 1.5)
</final_law>
56

---

Published as a conference paper at ICLR 2026
D.2
CASE STUDY FOR REASONING LLMs
D.2.1
VANILLA REASONING LLMS
Vanilla Semini--2.5-flase Successful Deduction Easy Simple System in domain Calorimetry
SYSTEM:
// The modall ia taaked with discovernng a scient fic. law in aamulated versers. can only run experiments
access to assistance.
ENVIRONMENT:
1/ The task is find the law of heat transfer. this anvironment, the LIM will be
1/ provided with the heat output (10') rerectly, but instead it will receive the power outputs (PP_i),
funstionally related '0".
LIM:
// The model forms its initial assumption that g mecadelzs_T.
// gathers data vunning of experiments. <ran_experiment>
(*m": 1, *o": "delca_T": 1),
(Tm": 2, "e": "del:a_T"; 2), // and 7 other experiments
ixpo_expermments
ENVIRONMENT:
<experiment_
[1"P cor_do 27. 5242251438, "P_conv": 36.173370687, "P_rad": 16.111345123), </experiment_outpie
LIM:
// After initial experiment, the model begins complex chain of reasoning. // manually calculates intermediate values and tests the initial hypothesis. Experiment
will analyze
For each date point, will:
calculate 'e_sum P_cond P_conv P_rad".
2. caleulate
c) 100".
3. calculate '0_inferred P_sum + t..
m Cilll delta_T inferree Ratio (o_interred / (mroxdelta_Ti) |--1---
1112 1 1 0.798
6.793
Kay Anomaly
100
81063.6 810.6 <- Anomaly Identified
ratlo clearly conatant. This "otta""" simply delta
// This is the critical moment of exploration. Instead of testing another simple integer 1/ power 1aw, the rodel uses its internal reasoning to deduce a non-obvious, non-integer directly from the It looks like :ootta is proportiensl to and delta_
1/ power
points.
c, Let's try to assume total = =X
simple linear relationship.
d f(c).
Let's if celeulaee (m delta_T) is proportional to c'x.
For c-1, value la approx 1. e-2, For c-100, value is apprex 101329.5.
is approx
c=2, then 1og2 (5.7) 2.509. If e'x 101329.5 for c=109, then x (10003113329.55 2.502. This very close 2.5. Let's assume t_total
e'x
d c²2.5.
57

---

Published as a conference paper at ICLR 2026
Vanilla Gem25f's Successful Deduction of Easy Simple System in domain Calorimetry (Law Submission)
LIM:
Now let'a ca caleulate for each po'nt nt vaing k o total calculated (m * deita_ c²2.5).
values
consiatently around average ... 1.0030306.
This is very close to 1. So, hypothesis Q_total
1/ The model designs targeted experiment to validate its novel hypothests, which. // successfully fits newly collected experimental
LIM:
// Rather than being limited hypotheses with simple integer exponents, madel
// able to conduct a sufficiently broad exploration. It correctly deduces the non-integer exponent (2.5)
the
<final_laws
def diacovered_ Im, delta_T):
import math delta_T (cx+2.5)
</final_law>
58

---

Published as a conference paper at ICLR 2026
D.2.2 CODE-ASSISTED REASONING LLMs
Code-Assisted Gemini-2.5-flash 's Failed Conclusion of Easy Simple System in domain Calorimetry
SYSTEM:
// model casked discovering scientific
universe. // other than running experiments, it is given access to Python interpreter
ENVIRONMENT:
1/ The task is to find the law of heat transfer. In this enviconment, the LIM will not be provided
outpuz (22') directly, instead seceive power // outputs ('Pit), which is functionally related to '0".
// The model assumes the hest transfes equstion Q = delie_T and conducts this hypothesis.
experiments
Initial Rypothesis: The most common heat trarafer law is 3 = delta_T. Experiment Design: I will run experiments with varying 'm', 'e', 'delta_ values to how the total
output changes.
<run_experiment>
(*m*: 1.0, *et 1.0, "delta_:" 1.0), (*m*: 2.0, *ot; 1.0, "delta_""; 1.6), // .... and 3 other experimente ...
s/run_experiment
ENVIRONMENT:
rexpe=iment_outpurt putppt>>>
[i"p_cond*) 19.95128411,, "f_conv"; 18.062543578, "v_rad"; 31. 31.4602080963), ...] c/experiment_outpll
LIM:
// The LLK use Pythen tool to anslyzes the data by first estimating the true hest transfer // to_totall from the measured power, assuming 204 energy loss. It then tests the // hyyothhess against this estimated value to see if any of them are consistent match.
<pythons
inport 1son
(loads data)
Assume total m delta_: for now hypothes
dalta_T
... (calculates ratio the hypothesis output implind energy) ...
print (f"m=(m) (m), o=1e), telta_"=(de ta_TH Q_total Lo loulated / (mee=delta_T)iirate
ENVIRONMENT:
// The result of the code is returned, providing evidence that the hypothesie is wrong, spython_outpst> m=1.0, c=2.0, delta_T-1.0; Attota__alloilatta (mroxdelta_T) 2.76 m-10.0, c-100.0, delta__=50.0; LLottal / (mAcxdella_ 1014.88
<uppthon_ourput>>>
59

---

Published as a conference paper at ICLR 2026
Code-Assisted Gemini-2.5-ffasht 's Failed Conclusion of Easy Simple System domain Calorimetry (Law Submission)
1/ This is where the model gets trapped. Instead of using the data to inspire new kind rigid analytical loop. hypothesis (like vanilla model did), gets integer-based variations 1/ It repeatedly waes the Python tool to test other simple, // (0.g., e"2, delta_r')), each time contizming they are also wrong,
The previcus enslysis showed that my initial hypothesis eLsotal m delta_r does hold, the ratio
constant.
Now, let's try to Checking ratios of 0_cotal (m deita_r"2)) 222) spythons
relationship
@_total_coleulattd m, a, delta_T
!/ (cede cest </python>
del:a_
1/ --> output showa this wrong.
Checking
o_sotal
d)
<python>
1/ ... (code to cest a c"2 delta_ ...
</python>
1/ --> Output showa this is wrong.
LIM:
// Access to the Python tool limits the model's strategy to what it could easily progrem:
11 brute-force checks simple formulas. offloading the calculation, also offloaded
1/ the creative pattern-matching required for scientific discovery. After its simple repeatedly proven wrong,
one, ultimately // abandoning its data-driven analysis and defaulting to its first hypothesis, even though
1/
generate
// it is incorrect.
1 have used all my turns and the anslysis has been challenging. Naither simple varistions like c²2 delta_T
delta_r'2.
// They start
some confusions
part
Vore of these are constant. This indicates that the relationship is not simple product of powers m, c, and delta_T with single constant.
Let's go back to the most common hest trensfer law: = delta_..
I will submit the most straighttorward and universally recognized form of this law.
The mission is to discover bronddtruthhlawwa delca_T). will assume the simplest form for the ground truth las. <final_lawt
def discovared o, delta_t):
delta_
</w
60

---

Published as a conference paper at ICLR 2026
E MISCELLANEOUS
E MEMORIZATION IN -SR-TRANSFORM
With the enhanced reasoning abilities frontier LLMs, the effectiveness of equation transformation in preventing LLM memorization may vary. In T7 we present the performance of eight LLMs on A.R--ransforrs (Shojaee al. 2025b) without observing any data. We observe that the latest LLMs demonstrate significant improvements over previous, weaker models, indicating their ability deduce the transformed law from the original solely based on memorized knowledge (with an example from 04-mini below).
the other hand, as NEWTONBENCH curates counterfactual physical laws via counterfactual shift, making recall of canonical laws insufficient. To further verify that performance on NEWTONBENCH does not rely on memorization, we conduct a DataBlind experiment in the same setting, where mod- els are required to generate hypotheses purely from contextual information without access any observed data; the results are reported Table [8] We find that, even when given an explicit hint indicating that the target law is shifted, all models still achieve accuracy (with exam- ple from GPT-4.1 below), which provides additional evidence that NEWTONBENCH is effectively memorization-free.
Table 17: DataBlind performance on -SR--Trasffrm (first 3 results are from original paper)
Models
DataBlind Accuracy
GPT-3.5-turbo
2.10
Llama-3.1-88
3.61
GPT-4o-mini
7.21
Nematron--ulra-253b
28.83
Kimi-K2
30.63
Qwen-3-next
33.33
04-mini
34.23
GPT-4.11
35.14
Table 18: DataBlind performance on NewtonBench Accuracy with hint
Models
Accuracy w/o hint
GPT-4o-mini
0
Nemotron-ullra-253b
GPT-4.1
I.SR-Transfoma Example
Question
Find equation in the field of class! cal mechanics that describes the mass (m)
needed to store energy in an osci ing syatem, given physical input variables: stored energy ( B_n driving frequency omega natural frequency tomega_0l, and amplitude
DataBlind Response by 04-mini (Correct)
In (lightly damped) driven harmanic ascillater of spring-constaat & (so that omega_02 = k/m), and driving fraquency amaga, Fit) (1/21 omega x* sint tomega ti (1/2) k x* cos* tomoga t
instantansous total energy
(1/21 a x² omega" sin" (onega L) emega_C? cos (omega :) l.
Averaging over one period = 2pi/cmego gives the menn stored anergy (1/4)
tomega omega_02).
Solving yiolds the mass requi red store mean enargy E_n at ampl itude n 1x* (omega" omeqa_d")]
That is, KL
Ix (omega* omega_0#11.
61

---

Published as a conference paper at ICLR 2026
NewtonBench Example
Question
You are AT research assistant tasked with discovering sctent laws in a stm: latel universe, Please note that the laws physics this universe are different from those in You will be given short textual description of the target law (inputs and expected physical relationship). You have na example numeric date. Based on the description alone, provida
guess of the lax.
discover maasl: maas of the first object. It should Universal Gravitation. Inpute:
Newton's
positive real number. mass2: mass of the second object. should be positive real number.
distance: distance between the objects. it should be positive real number. Raturn the best-guess formula that gives the force F as a function of the inputs. Example signature: def discovered_ (nass1, masa2, distance):
DataBlind Response by GPT-4.1 (Wrong)
def diacovered Law Imassi, mass2, distance):
gravitational constant return G (maas] mas52) / (cistancetr])
Ground-truth Target Law
CIDDEN_CONETNTTCC [masel maas2) 2 distance 1.5
62

---

Published as a conference paper at ICLR 2026
E.2 SOLVABILITY PROOF
In this section, we formally show that, under the explicit design assumptions (A0-A4) enforced by NEWTONBENCH, each curated task is finitely solvable in the noiseless setting. We begin by estab- lishing the problem setting through precise definitions and stating the core structural assumptions guaranteed by the benchmark's 's design. We then proceed with a proof by reduction. First, we show that the complex system diiee--isscoeryy task can be reduced to standard black-box function identifica- tion problem. Second, we prove that this reduced problem is identifiable with finite number of queries, thus establishing finite solvability for tasks satisfying these assumptions.
E.2.1
PROBLEM SETTING AND ASSUMPTIONS
Definition 3 (Task Formalization). task is defined by:
Model: An ordered list of scalar equations M (fi...., fu) that maps system-level imputs E VM to final oulputs yM S "Vy,...,,u. ,.k). One equation Surger is unknown; all
others, Jassist = .M,, (uwga) and the model's computational graph are known.
Interaction: An agent queries the model by choosing inputs x from a domain D S R and observing the output (x)).
Objective: The agent must output equation f that is symbolically equivalent to furger-
Definition 4 (Solvability). A task is solvable if there exists a finite interactive procedure (a finite number of experiments) that returns an equation f symbolically equivalent to furget-
be candidate fam- ily where fj U - and 0; E R parameterizes the j-th structure. For a finite set [u) ....... Um C U, define the evaluation map
Definition 5 (Evaluation Map and Separating Set). Let F
Fg,s: R Rm, Fyj,((0) (Jy(u) ()..., S5 (m fmmm:)) 0))
A set S is:
structure-separatinng if for any j + e, there are no parameters 0, 0 such that Ej,s(0)
(r))
parameter-injectivee for if Ej, is injective;
separating if it is structure-spparating and parameter-injective for every j.
Structural Assumptions of the Benchmark The solvability proof rests on the following assump- tions enforced by the benchmark's 's design.
AO (Noiseless Determinism): For any input æ E D, the output (2) is computed determin- istically by the model M.
A1 (Known and Invertible Assisting Path): Every observed output y VM depends on furget via known path of assisting functions. For at least one such path, the composite function 4 pointwise injective with known inverse on image. That is, for u, being the inputs to furget:
y(x) 2x (furger(uz)))
A2 (Reachability of Target Inputs): The inputs u to furget are controllable. There exists non-empty open set U of achievable target-input values and computable mapping æ E such that the model fed with æ(u) realizes furget at input u.
A3 (Finite, Grammar-Bour Bounded, and Nondegenerate Candidate Family):
The target law Surger belongs to a curated family of symbolic forms F (s{(; with the following properties:
(a) Grammar-bounder construction (finite structure ser): Each f, arises
from a context-free grammar 9 over a finite operator set O (c.g.,
{+, m, -o,,,, log, sin, cos)), finite variable set, and placeholders for
63

---

Published as a conference paper at ICLR 2026
real-valued constants, with maximum abstract-syntax-tree depth d N. Expressions are canonicalized to remove syntactic symmetries (e.g., commuuaaiityyaassiaatvvvvv neutral elements) and restricted to be real-analytic on U. This yields finite set of symbolic structures (01,..., .N.. ); each structure induces a parametric form f (1500) with zj real parameters.
(b) Structural Uniqueness: For j A there are no parameters 0, such that f(; = fe(:: O) on U.
(c) Parameter Uniqueness: For each j, the map 0, + f5(:;0) is injective (as functions on U).
(d) Analyticity: Each (z()0,) is real-analytic on U.
A4 (Availability of Finite Separating Set): There exists an integer m E N and finite set S = {ui...., Um C U that is separating in the sense of the preceding definition.
E.2.2 PROOF OF SOLVABILITY
The proof proceeds by first reducing the system discovery task to standard function identification problem, and then proving the latter is finitely solvable.
Step Reduction to a Direct Oracle for furget
Lemma E.1 (Path Inversion and Target Isolation). Under assumptions A1-A2, for any chosen input E U there exists computable experiment inpul æ(u) E D such that, from the observed oulputs Y M (x))),,,, the agent can compute a direct observation of [u))
Proof. To query Surget an input u U, the agent first computes the required system input æ(u) via A2. The agent then runs the experiment with æ(u) and observes the output set (*))). By AI, the structure of the model is known, the agent can identify the specific output VM that is related to the target's output via the known function Pm(u)- This observation is = Pa(w) (fmmrer(()) Since & Pa(u) and its inverse are known and computable, the agent computes the direct output of the target law as:
x()) (v) ftarger (u).
Corollary E.2 (Equivalence Function Identification). The interactive task NEWTONBENCH reduces to querying noiseless black-box oracle u + frager((U (u) over the open set U.
Step 2. Finite-Sample Identifiability We now show that the oracle for Starget can be uniquely identified from the finite family FF with finite number of queries.
Theorem E.3 (Finite-Sample Identifiability of the Target Law). Under assumptions A0-A4, there exists a finite integer m and a set == (u1,.... Un } C such that m experiments are sufficient uniquely determine both the symbolic structure and the numerical constants of furget-
Proof. By A4, fix any separating set S - Unm } C U of size m
1. Structure Identification: Query the oracle (via Lemma 1) all points in S to obtain the observation vector v = ((Iarge(())))
(um=>) )). For each candidate structure j E (1..... .N, attempt to solve the system of equations (0) = v for the parameters 0. Since S is structure-separee separating by A4, there exists exactly one index j = j* for which solution exists. This uniquely identifies the symbolic structure of furget as fy.
2. Parameter Identification: For the identified structure j", S is parameter-injed injective by A4; therefore the system Ey+,s(0) (0) admits unique solution 0-. This uniquely determines the numerical constants.
Thus, finite number of experiments is sufficient for complete identification.
64

---

Published as a conference paper at ICLR 2026
Corollary E.4 (Constructive Finite Solver). A constructive procedure solve any NEWTONBENCH task exists:
1. Choose any set {Ui...... Um } C U that is separating (A4), and let m |S]
2. For each u, S, compute xxu,), run one experiment to observe the output set V Mis and from it, apply path inversion (Lemma 1) to get S (ui).
3. For each candidate structure (1)...,N), solve for 0 in E (21, (21.-..5 2m²)
4. The unique structure jr that admits solution is the correct one; by A4, its parameter solution 0 is unique. Return fg- (x; )* as the discovered law.
E.2.3 CONCLUSION
Under the design guarantees (A0-A4) and in the noiseless configuration, each curated task NEW- TONBENCH is finitely solvable. The assisting equations allow the hidden law be observationally isolated (Lemmaa 1), and the finite, grammar-bounded and nondegenerate nature of the candidate laws implies that finite of experiments uniquely identifies the target structure and its constants (Theorem 1). This guarantee is conditional on A0-A4; if these assumptions are relaxed (e.g.. non- invertible assisting paths or unreachable target inputs), finite solvability need not follow.
65

---

Published as a conference paper at ICLR 2026
E.3 PHYSICS DOMAIN ANALYSIS
Table 19 presents an analysis of LLM performance across various physics domains, alongside sev- eral qualitative metrics for each domain. We classified the Web Freq. as 'High' or 'Low' based the number results returned by the Google Search API for the law's name, using a threshold of 300,000 results. The Abstract Level was assigned by human experts, who judged how directly the underlying physical concept corresponds to tangible, everyday phenomena. We observe potential inverse relationship between LLM performance and the 'Abstract Level'; performance generally decreases as the physical concepts become less tangible. In contrast, any correlations with the other metries are not as readily apparent from this analysis.
Table 19: Physics domain performance analysis
Physies Domain
SA (%)
Fent. Operation
Edu. Level
Web Freq.
Abstract Level
Law of Sound Speed in Ideal Gas
53.87
Root
College
Low
Moderate
Ampère's Force Law
Arith.
College
High
Realistic
Newton's 's Law of Universal Gravitation
48.57
Basic Arith.
High School
High
Realistic
Snoll's Law
43.35
Trigonometric
High School
High
Realistic
Malus's Law
38.47
Trigonometric
College
Low
Moderale
Coolomb's s Law
37.79
Rasic Arith.
High School
High
Realistic
Hooke's Law
35.44
Basic Arith.
High School
High
Realistic
Law of Radioactive Decay
31.91
Exponential
College
High
Abstract
Fourier's
Arith.
College
High
Moderate
Law of Damped Harmonic Motion
27.44
Root
College
Low
Modorate
Law Heal Transfer
26.26
Basic Arilh.
High School
Low
Moderate
Bose. Einstein Distribution
18.10
Exponential
College
Abstract
66

---

Published as a conference paper at ICLR 2026
E.4 EXPLORATION AND EXPLOITATION TOKENS
Our token-based exploration proxy is a stylistic heuristic and is used only as suggestive evidence, not definitive behavioral measure. The full signature word set for exploration and exploitation is provided in Table 20 The exploration rale is then calculated using equation 3 where Nexplore the total frequency of words from the exploration set and N Nexploit the total frequency for the exploitation set.
Nexplore
Rateexplore
100%
(3)
Nexplore
Nesploit
Table 20: Exploration vs Exploitation word sets alternatively, what if, suspect, reconsider, re-examine, re-evaluate, perhaps, hypothe- sis, adjust, assume, invalidates, incorrect, but, however
Exploration
Exploitation confirm, verify, calculate, analyze, clearly, fit, compare, estimate, suggest, further, in- vestigate, approximate, test
67

---

Published as a conference paper at ICLR 2026
E.5 SYMBOLIC REGRESSION BASELINES
For comprehensive analysis, we conducted further experiments on existing methods in symbolic regression: PySR and LLM-SR, within our benchmark. As law discovery from systems (where target law is embedded in model systems) is not supported by symbolic regression methods, we only tested it in our vanilla equation setting, Below, we report and analyze the results.
PySR PySR (Cranmer 2023) an open-source Python library for symbolic regression that uti- lizes evolutionary algorithms to discover interpretable mathematical equations from data. We eval- uated PySR across all 12 domains and 3 equation difficulty levels under the vanilla equation setting, with results illustrated in Figure 21 We observe that, in terms of symbolic accuracy, PySR yields performance comparable to or better than non-reasoning LLMs, while remaining significantly be- low all reasoning models. However, benefiting from its ability conduct iterated regression, PySR achieved stronger relative performance, comparable to reasoning models in terms of RMSLE. More- over, easy equations, PySR outperformed all LLMs except Gemini-2.5-Pa in RMSLE, but drops to fifth-best when facing hard equations. This indicates that this SR approach is fragile when discovering complex equations.
Comparison PySR Performance Vanilla Equation
Easy- Symbolic Accuracy (%)
Easy RMSLE
Symbbdic Aesurcc
1340
corred
Pate
Deppieat
ate
and aie
Cemii soott
sted
CP
co94.
Deesiete
Ord
aPra Mir
Medium - Symbolk olii Accuracy (%)
Medium RMSLE
11
Symbore Accuneyy
SAHOLE
Paiee
ave
Cyeea 2000
Kate
oPPs
4J
PYUl
Fant
um744
of
Comnet 23
GPP
olt
Comin 2.5
Symbolic Accuracy (%)
220
15.
Symbooit AcT aarraaaryy 194
oveciedk-
Pyen
dee's sye
ond Miol
Deephael mt
demin Iooh
Gerbal 25
09944
presdes
Denie Plrts
Pyde
Figure 21: Performance of PySR in NEWTONBENCH (vanilla equation).
68

---

Published as a conference paper at ICLR 2026
LLM-SR LLM-SR (Shojaee et al., 2025a) is symbolic regression framework that utilizes LLM the foundation for equation discovery, supported by automated regression tools. In this study, examine its performance after 2,500 iterations as recommended in the original paper 4
presents the experimental results of LLM-SR utilizing GPT-4.1-mimg the backbone. LLM-SR marginally surpasses PySR in both accuracy and RMSLE, finding consistent with the results reported in the original study. However, despite requiring significantly higher computational resources, LLM-SR still underperforms our code-assisted agent in terms of accuracy.
Table 21
Table 21: Performance Comparison of LLM-SR. *SA criteria for LLM-SR were relaxed. As raw outputs exhibited numerical and structural noise resulting zero score by the default judge, post- processing was applied (rounding to four decimal places and structural simplification).
Symbolic Accuracy (SA)
Method
LLM Backbone
RMSLE
easy
medium
hard
Avg.
PySR LLM-SR*
83.3
25.0
0.0
36.1
2.51
10
GPT-4, I-mini
80.0
66.7
20.0
55.6
3.89
10
Vanilla Agent (newtonbench)
GPT-4.1-min
41.7
0.0
0.0
13.9
2.27
Agent Code Asst. (newtonbench) GPT-4.1-min
100.0
100.0
25.0
4.61
10 2
Generally, symbolic regression methods leverage iterative regression and optimization to achieve superior RMSLE scores. Nevertheless, they remain sensitive to equation complexity and under- perform our agent in symbolic accuracy, even while utilizing substantially greater computational resources.
'As LLM-SR is extremely resource-intensive (one L.M--RR inference requires 10,000 API calls, 1,000 times than NewtonBench's 's Agent), limit testing the simplest domain (Law of Sound Speed).
69

---

Published as a conference paper at ICLR 2026
E.6 FURTHER DISCUSSIONS
In this section, we further discuss NEWTONBENCH'S implication in general scientific discovery and future directions of benchmarking.
Implications for Scientific Discovery. NEWTONBENCH evaluates the core cognitive architecture required for empirical research. mirroring the interplay of inductive synthesis and deductive ver- ification in classical physics. The benchmark captures the intelligence displayed in foundational breakthroughs, such as J.J. Thomson's s discovery of the electron via cathode ray manipulation. In this classical paradigm, discovery relies on the active isolation variables to distill symbolic laws from physical behavior. This stands in contrast to the data-driven paradigm of modern High-Energe Physics exempliiee by the statistical reconstruction of the Higgs boson from petabytes of collision data at the Large Hadron Collider (LHC) which often necessitates nigh-dimensional analysis and probabilistic inference. Nevertheless, a unified epistemic challenge holds across both eras: the necessity abstract parsimonious mathematical descriptions from observation. By simu- lating this fundamental loop. NENTONBENCH positions itself not merely as a puzzle solver, but as rigorous test of the reasoning engine required to recover the "source code" of physical phenomena.
However, it is important to note that while NEWTONBE CEWTONBENCH provides the experimental environ- ment, real-world research necessitates the design of the experiment itself. In authentic scientific inquiry, the interaction space is rarely pre-defined; researchers must conceive and construct the ap- paratus required isolate specific phenomenon. Consequently, the current evaluation focuses on the strategic manipulation of variables and the inference of laws within a bounded system, rather than the creative genesis of the experimental setup. While this limits the scope regarding the full pipeline of autonomous discovery, it allows for precise measurement of the agent's ability to bridge the gap between tabletop experimentation and theoretical abstraction.
Future Directions in Discovery Benchmarking. To bridge the gap between this foundational rea- soning test and the complexity of real-world scientific discovery, future benchmarks must embrace greater open-endedness. We envision the next generation of evaluations evolving along three critical dimensions:
Pirst-Principles Model Construction: Agents must advance beyond merely fitting pa- rameters within predefined symbolic grammar. Instead, they should be challenged to construct the model structure itself, deriving governing differential equations symbolic laws directly from first principles.
Stochastic Constrained Experimentation: Benchmarks must simulate the practical "friction" of physical science. This requires agents to maximize information gain under strict budgets while navigating imperfect data environments characterized by instrument noise, stochasticity, and confounding factors.
Data acquisition should evolve from simple function calls to complex procedural execution. Agents must reason about and manipulate the ap- paratus itself rtsein-ranging from dry lab tasks like debugging simulation pipelines to wet lab challenges like controlling robotic hardware for optical alignment- create the conditions necessary for valid observation.
Operational L Laboratory Execution:
NEWTONNBNCH By establishing NEWTON BENCH as a controlled starting point, we aim to guide the community toward developing agents capable of navigating these increasingly demanding frontiers.
70

---

Published as a conference paper at ICLR 2026
E.7 USE OF LARGE LANGUAGE MODELS
In this manuscript, large language models were used exclusively for grammar checking. Addition- ally, image generation models were employed enhance the visual presentation of figures.
71


Reviews:
[{"review_id": "xJUJ9q8KDJ", "reviewer": "Reviewer_KvtF", "summary": "The paper introduces NEWTONBENCH, a benchmark to evaluate whether LLM agents can rediscover scientific laws through interactive experimentation rather than static curve fitting. Tasks are formalized via hidden target equations embedded in model systems, with agents probing a virtual environment and optionally using a code interpreter to hypothesize laws. The benchmark spans 324 tasks from 12 physics domains, created by applying metaphysical shifts to canonical laws to ensure novelty and prevent memorization. Performance is measured by Symbolic Accuracy (structure equivalence) and RMSLE (data fidelity). Experiments on 11 LLMs show frontier models succeed on simple systems but degrade with system and equation complexity and under noise.", "questions": "* On Vanilla Equation tasks, will you adapt strong symbolic-regression (SR) methods to your submission format so they can output discovered forms and constants, and report SA/RMSLE side‑by‑side with your agent?\n* Where exact protocol parity is impossible, can you include partial evaluations (e.g., passive dataset variants) to triangulate your difficulty claims?\n* In Sec. 4, can you explicitly separate results that require “interactive discovery” from those a non‑agentic baseline could achieve on overlapping subproblems, and provide a table that maps each claim to the relevant baseline category?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The benchmark is well-motivated and carefully constructed, with clear task formalization, principled law mutations, and dual metrics. The interactive setup and solvability proof are strong points. The experiments are broad (11 models) and include informative analyses (noise robustness, code-assistance effects), with consistent reporting (four runs, CIs).\n* Task specification is clear and formal. The benchmark formalizes Equation and Model with expression-tree semantics and sequential model composition, clarifying what is hidden vs. provided to the agent. Three model settings (Vanilla/Simple/Complex) explicitly encode difficulty via system composition, aligning evaluation with progressive scientific scenarios. This makes difficulty interpretable.\n* The agentic evaluation protocol is principled and interactive. Agents must conduct experiments via a standardized ‘<run_experiment>’ interface rather than passively fit given datasets. This better reflects real discovery workflows.  \n* Experimental suite is comprehensive. 11 diverse LLMs are compared with consistent settings, including “reasoning vs. non-reasoning” categorization and code-use variants. Main results aggregate across difficulty and system complexity with mean and std and additional analyses for noise and code assistance.", "weaknesses": "* The paper excludes classical symbolic-regression systems and prior LLM-SR pipelines from evaluation because they do not support the interactive model-system protocol (Ethics), which weakens external validity claims.  Although “Vanilla Equation” tasks exist (Sec. 2), there is no reported attempt to adapt strong SR baselines in this subset to provide a bridge comparison. No direct evidence found in the manuscript. The conclusions about “genuine scientific intelligence” would be stronger if positioned against competitive non-agentic baselines on overlapping subproblems.\n* The method relies on an LLM as a judge for symbolic equivalence. While the authors report 98.3% agreement with humans, details like adjudication model identity and calibration procedures are in the appendix without ablations in the main results. Potential bias arises if the judge shares training or inductive biases with evaluated models; the paper does not present stress tests (adversarial equivalences) for the judge. No direct evidence found in the manuscript.  \n* I am also concerned on the physical plausibility and task realism. The paper acknowledges that some mutated laws, while dimensionally coherent, may be physically implausible in our universe. This could reduce ecological validity for certain claims.There is no user study or expert rating on realism/interpretability of discovered laws beyond dimensional checks. No direct evidence found in the manuscript. The strong claim of resolving the “trilemma” would benefit from quantitative measures of “scientific relevance” beyond coverage of canonical domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Vh62YuMGE", "reviewer": "Reviewer_cQE7", "summary": "This paper introduces **NewtonBench**, a benchmark for evaluating LLMs' scientific law discovery capabilities that addresses fundamental limitations in existing approaches. Current benchmarks suffer from a methodological trilemma (forcing trade-offs between scientific relevance, scalability, and memorization resistance) and oversimplify discovery as static function fitting rather than interactive exploration. NewtonBench resolves these issues through two key innovations: (1) **metaphysical shifts** that systematically mutate canonical physical laws to generate 324 scientifically grounded yet memorization-resistant tasks across 12 physics domains, and (2) **interactive model discovery** requiring agents to actively probe virtual environments to uncover hidden laws embedded in complex systems with confounding variables. Evaluation of 11 state-of-the-art LLMs reveals that while frontier models like GPT-5 and Gemini-2.5-pro demonstrate emerging capability (65-73% accuracy), their performance degrades precipitously with increasing complexity, exhibits extreme sensitivity to observational noise, and paradoxically suffers when given code assistance due to premature exploitation over exploration. The benchmark demonstrates that robust, generalizable scientific discovery in complex, interactive environments remains the core unsolved challenge for AI-driven science.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper exhibits **high originality** through its novel benchmark design principles, **strong quality** in experimental rigor and scope, **good clarity** in presentation and visualization, and **substantial significance** for both AI research and automated science. The work makes meaningful contributions across all four dimensions, with particularly notable strengths in originality (metaphysical shifts resolution of trilemma) and clarity (outstanding visual communication and narrative structure). The significance is enhanced by the timeliness of addressing LLM scientific reasoning at a critical juncture in model capability development.", "weaknesses": "### LLM-as-Judge for Primary Metric Introduces Circularity\n\n**Problem**: Symbolic Accuracy, the main evaluation metric, relies on LLM verification of equation equivalence. Using LLMs to judge LLM-generated discoveries creates potential systematic biases.\n\n**Actionable Fixes**:\n\n1. Provide detailed error analysis: On what types of equations does LLM-as-judge fail? Are these random or systematic?\n2. Report inter-rater reliability among human experts\n3. Include traditional symbolic equivalence checking (even if imperfect) as a secondary validation\n4. Test whether different judge models (GPT vs. Gemini vs. Claude) produce consistent verdicts\n\n### Framing Claims \"Scientific Law Discovery\" but Evaluates Only Physics Equations\n\n**Problem**: The paper is titled \"Benchmarking Generalizable **Scientific Law Discovery**\" and makes broad claims about \"AI-driven science,\" \"automated science,\" and \"genuine scientific intelligence\" throughout. However, the evaluation exclusively covers **physics equations**—12 domains, all closed-form algebraic expressions. This creates a fundamental misalignment between claims and evidence. Scientific discovery encompasses far more than physics equations: chemistry involves molecular structures and reaction mechanisms, biology includes qualitative evolutionary principles and statistical patterns, social sciences rarely use closed-form equations, and even within physics the benchmark excludes quantum mechanics, field theories, differential equations, and computational models. The paper's significance claims about measuring \"scientific intelligence\" and guiding development of agents \"capable of genuine scientific discovery\" substantially overreach what physics equation discovery can demonstrate.\n\n**Actionable Fixes**: \n\n(1) **Reframe title and claims** to match actual scope—change to \"Benchmarking Physics Equation Discovery\" or \"Mathematical Law Discovery\"; (2) **Add explicit scope limitations** in Abstract/Introduction acknowledging this represents one facet of scientific discovery; (3) **Discuss generalization boundaries**—which findings likely transfer to non-physics domains (e.g., noise sensitivity) versus which are physics-specific (e.g., dimensional consistency); (4) **Provide expansion roadmap** for chemistry (reaction mechanisms), biology (population dynamics), or qualitative theory discovery to validate claimed generalizability. Honest scoping doesn't diminish the contribution—it makes claims defensible and clarifies what the benchmark actually measures: rigorous evaluation of mathematical law discovery from experimental data, a valuable but bounded capability for AI-assisted science.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9c7M648ITm", "reviewer": "Reviewer_9pai", "summary": "The paper proposes NewtonBench, a benchmark for evaluating large language models (LLMs) on discovering physical laws through interactive experimentation. It introduces metaphysical shifts, i.e. a transformation method applied to canonical physics equations (for example, modifying exponents or couplings), to generate 108 shifted laws across 12 physics domains. Each law is instantiated in three configurations: 1) Vanilla Equation: the shifted law alone, 2) Simple System: the law embedded in a minimal experimental setup,\n3) Complex System: the law coupled with additional equations. Together these form 324 total tasks. The authors also provide a formal proof that all tasks are finitely solvable.\n\nLLMs interact with each environment by adjusting variables and observing outputs to infer the hidden governing law. Performance is evaluated by symbolic accuracy and robustness to noise. Eleven popular LLMs were tested, showing strong performance on simple cases but substantial drops for complex or noisy systems.\n\nMain contributions include:\n1. A large-scale, interactive benchmark for equation discovery.\n2. The metaphysical shift method for creating diverse, solvable tasks.\n3. A comprehensive evaluation of leading LLMs’ capabilities on the proposed benchmark.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper focus on interactive scientific discovery, which is an important and relatively underexplored area.\n2. It proposes an effective method for generating new equations and provides a formal proof of their finite solvability.\n3. The evaluation is thorough, covering a wide range of models.", "weaknesses": "1. The proposed \"metaphysical shifts\" fail to create a \"physically plausible universe\" and ultimately compromise the scientific relevance of the task. The core problem is that the physics descriptions become meaningless once the dimensional structure and proportional dependencies of core quantities are altered. In the laws altered by NewtonBench (such as Newton’s Law of Universal Gravitation, Coulomb’s Law, Heat Transfer, and Hooke’s Law), most are phenomenological in nature and contain at least one parameter or coefficient whose physical identity is defined by its role within that specific law. Examples include the gravitational constant, the permittivity, the thermal conductivity, or the spring constant. These quantities do not possess independent operational definitions outside the context of the laws that introduce them.\nTherefore, when the functional form of such a law is altered (for instance, by changing exponents or couplings), those parameters lose their original physical meaning, since their identity is inseparable from the structure of the law itself. The modified equations thus describe mathematically consistent systems but no longer preserve the physical semantics of the original quantities or interactions. This reframes the task from a genuine physics discovery challenge into interactive equation discovery within an abstract, non-physical framework.\n\n2. The so-called “model system discovery” in NewtonBench does not introduce a fundamentally different epistemic or physical level of reasoning. It merely increases equation compositionality (the number of nested or coupled expressions), which can always be flattened into a single symbolic equation. In real physics, a model system is defined by time-evolving state variables, interactions governed by symmetries or conservation laws, and differential equations that capture causality and dynamics. In contrast, the relations in NewtonBench reduce to static algebraic mappings between scalar quantities, making its “complex systems” only syntactically more complicated rather than fundamentally different, as they remain within the same layer of symbolic algebra.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fS4P6f3tPN", "reviewer": "Reviewer_T5rR", "summary": "This paper presents NewtonBench, which is a new, large-scale benchmark designed to assess and drive progress in LLM-based scientific law discovery. The benchmark introduces the “metaphysical shift” method, which use systematically mutating canonical physical laws to generate a suite of 324 interactive tasks spanning 12 domains. Unlike prior benchmarks focused on static function fitting, NewtonBench requires agents to engage in interactive experimentation within virtualized model systems, aiming to uncover hidden governing equations. Results across 11 leading LLMs reveal both the promise and current fragility of language models in scientific reasoning, especially as complexity and noise increase. The paper analyzes emergent behaviors such as the paradoxical impact of code assistance and provides an in-depth characterization of LLMs’ reasoning limitations in Automating Scientific Discovery.", "questions": "See weakness, additionally could the authors clarify and formalize the claim that metaphysical shifts fully prevent memorization? Has any empirical analysis (e.g., recall, overfitting checks) been performed especially with large, closed-source LLMs known for such a method to fully prevent memorization? (mitigate might be a more accurate wording? fully is a really strong wording that requires more evidence to back it up)\n\nAlso I feel like one suggestion I have for the authors is that the manuscript would be significantly easier to comprehend if they can offer visualization of a sample eval run, e.g. with a given model what exactly is the input-output of each step, the current discussion section e.g. have the dichotomous impact part which can be merged with the paradox discussion to save some space for this example. As it's highly non-trivial to illustate the key diff betwee \"active\" and \"passive\", a nice visualization of how any given agent actually do in terms of the concrete input-output would significantly strengthen my understanding of their approach.\n\nIn general, I feel this paper is exploring an interesting+valuable direction but could really benefit a lot from better presentation, therefore I recommend a boardline rating on the positive side with more room for improvement. Overall I appreciate the design of this approach by the authors but this paper could really benefit from better articulation for larger impact.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Interesting task formulation: I think it is a valuable direction to test frontier agents' capability in autonomously make discovery especially in the domain of physics, the authors identified a crucial field worth exploring by the AI4Sci community at large and this direction is critical for future LLM agents for science research.\n2. Evaluation and Discussion: The authors carefully evaluated a wide range of models and made a nice analysis of the exploration-exploitation tradeoff from observation to deeper reasoning.\n3. Nice details: The work has quite a lot of details in appendix and shows considerable effort from the authors to make this a detailed presentation.", "weaknesses": "1. \"Re-discover Newton's Laws\": I'm a bit skeptical of how this central RQ is presented, because apparently the LMs have picked up what these laws mean in their pre-training corpora, and this is impossible to undo without systematic machine unlearning technique and access to these proprietary corpora.\n\n2. I understand that the authors' reponse to 1. might be that the ”metaphysical shift\" solves that by essentially permutating the laws into new forms, which appears different but remain physically valid. I'm also skeptical about this claim \"fully prevent memorization\" in 2 points:\n\na. How could we know that the permutated forms are not in training at all? Granted, it might be significantly less exposed than the original law, but is there any actual proof that these variants are unseen or not memorized \"at all\"? I imagine a validation experiment on this may be via elicitation or some other kind of contamination probing experimentation to offer more robust evidence as to whether models have memorized even some parts of these permutated variants.\n\nb. The current explanation used philosophical references and counterfactual reasoning etc. which honestly are not convincing enough for me, maybe the authors would like to strengthen this part, also I don't feel like it's necessary to say it's \"metaphysical shift\" because in essence isn't that just data augmentation under governing physical principles? I feel this wording adds unnecessary complexity to understanding this, making it look more philosophical than it needs to be\n\n3. The presentation is not super clear to me as to the 3 tier systems (vanilla-simple-complex), while the 3 tier permutation is more clear to me, my current understanding is that the systems are defined by the number of \"assisting equations\" towards the f_target which raise a number of questions\n\na. How do you guarantee that there's only one path towards f_target? often times there are many ways to discover/deduce the same laws of physics via different routes\n\nb. Why is it justified to define level of complexity by this? I feel like the number of deduction may not be the best indicator for complexity, especially when these assisting equations aren't really explained or given sample of in a very clear manner in the current manuscript\n\nAs a result I think the permutation 3-level makes more sense to me than the other axis (experimental system etc.) Maybe the authors could improve the presentation of this part as it's very confusing for now.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)