Gravity-Bench-v1: A Benchmark on Gravitational Physics Discovery for Agents
Authors: Nolan Koblischke
Hyunseok Jang
Kristen Menou
Mohamad Ali-Dib
Venue: ICML 2025 poster (2025)
Keywords: Scientific Discovery
Benchmarking
Evaluations
Physics
Agents
Planning
Large Language Models
Primary area: deep_learning->large_language_models
TL;DR: Gravity-Bench evaluates scientific discovery capabilities of AI agents through gravitational dynamics simulations, challenging them to strategically collect observations and employ scientific reasoning, including in out-of-distribution scenarios.

Abstract: Modern science emerged from reasoning over repeatedly-observed planetary motions. We present Gravity-Bench-v1, an environment-based benchmark that challenges AI agents on tasks that parallel this historical development. Gravity-Bench-v1 evaluates agents on the discovery of physics concealed within a dynamic environment, using rigorous gravitational dynamics simulations. Gravity-Bench includes out-of-distribution cases, i.e. with physics that deviates from the real world, to evaluate true scientific generalization capabilities. Agents must plan to collect data within an experimental budget and must perform a dynamic form of data analysis and reasoning to solve tasks efficiently. Our benchmark admits an open-ended space of solutions. Reference solutions for each task are provided to calibrate AI performance against human expertise. Technically at an upper-undergraduate level, our benchmark proves challenging to baseline AI agents. Gravity-Bench-v1 and planned extensions should help map out AI progress towards scientific discovery capabilities.

A Benchmark on Gravitational Physics Discovery for Agents
Nolan Koblischke' Hyunseok Jang'
Kristen Menou
Mohamad Ali-Dib 2
Abstract
when it comes evaluating an Al agent's capacity for dis- covery under normal scientific conditions of uncertainty and novelty.
Modern science emerged from reasoning over repeatedly-observedd planetary motions. We present Gravity-Bench-v/ an environment-based benchmark that challenges Al agents on tasks that parallel this historical development. Gravity- Bench-v1 evaluates agents on the discovery of physics concealed within a dynamic environment, using rigorous gravitational dynamics simulations. Gravity-Bench includes i.e. with physies that deviates from the real world, to evaluate true scientific generalization capabil- ities. Agents must plan to collect data within an experimental budget and must perform a dy- namic form of data analysis and reasoning to solve tasks efficiently. Our benchmark admits an open- ended space of solutions. Reference solutions for each task are provided to calibrate Al perfor- mance against human expertise. Technically at an upper-undergraduuate level, our benchmark proves challenging to baseline AI agents. Cravity-Benche v1 and planned extensions should help map out Al progress towards scientific discovery capabilities.
To address this gap, we introduce Gravity-Bench-vi, a new benchmark specifically designed to evaluate the scientific reasoning and discovery capabilities of Al agents within controlled, physics- environment. This benchmark is inspired by the historical development of science (the two-body problem of gravitational dynamies) and leverages high-fidelity machine-precision simulation tools to build an environment where AI agents can interact with and explore faithful physics experiments.
cases,
In Gravity- Bench-B,, an Al agent not merely tasked with analyzing pre-collected data but it must engage in a fuller version of the scientific process. It must schedule observa- tions intelligently, within constrained budget, and make inferences based on the limited and accumulating data it collects. This setup allows for an assessment of the agent's 's ability to reason and make autonomous decisions under dynamically-shrinking uncertainty, as observational data accumulates.
Our benchmark admits open space of solutions, in the sense that the optimal planning for observations and algo- rithmic approach for quantitative answers are not priori known. We leverage this property to offer expert solutions with uniform sampling of observations (without planning) that we consider as strong expert baselines. Our benchmark opens the possibility for an AI agent to discover planning and/or reasoning approaches that best our expert solution, as discussed further below.
1. Introduction
The rapid evolution of artificial intelligence (AI) and ma- chine learning has led to significant advancements in various domains, particularly in natural language processing and computer vision. However, the design of AI agents for sci- entific research presents unique challenges, particularly in the context of autonomously discovering new natural phe- nomena. Traditional benchmarks, such as those focused on knowledge evaluation (Rein et al., 2023; Hendrycks al., 1a; Ting et al., 2024) general problem-solving capa- bilities (Clark et al., 2018; Zellers et al., 2019; Hendrycks et al., 2021b; Tian et al., 2024), fall short of what is needed
Gravity-Bench-v/ has a narrow scientific focus on the two- body gravitational dynamics problem. We are working on future expansions, including visual observation, extending to additional fields (e.g., electromagnetism)) incorporating observational effects such as measurement errors, and intro- ducing chree-dimensional orbital dynamics. While various extensions in task complexity, environment realism and across physics domains are possible, we believe that the performance gap demonstrated by current Al models, even within the current limited domain highlights the immediate value of evaluation setups like Gravity-Bench-y
Universsity of Toronto "New York University Abu Dhabi. Correspondence to: Nolan Koblischke
utall.utoronto.c.. Kristen Menou
<kristen.menov utoronto.ca>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In short, Gravity-Bench-vl challenges AI agents with

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
Gravity Bench
Baseline Agent
Binary Star Simulation
Task
Observe Tool X-
Simiulation Time
This system has an alternate law of gravitation
Observe star positions at [&,, l2. 410
FG X r (2+a)
with 100 observations max
Determine the offset at
Simulation Input
Python Tool
C'ru 0.03
Analyze da t f rame of star positions at requested times
Positions Over Time
Grader
Timo [s]
Star [n], Star [n]
268
(e111,,
Ckans
atruel
threshold
[Olcasel
Submit Answer
Correct/ Incorrect
Figure 1. Overview Gravity-Bench-y architecture and workflow. The binary star simulation environment (left, green) generates orbital trajectories based on input parameters, including out-of-distribtioon physics like modified gravity laws. An agent (right, purple) must solve physics discovery tasks by strategically collecting observations through the observe tool (limited to a budget of 100 observations). We evaluate against expert solutions based on full simulation data access but using uniform sampling of 100 observations as a baseline without planning. This design tests both scientific reasoning and intelligent observation planning capabilities.
tasks that mirror real-world scientific inquiry. pro- vides framework for evaluating their progress toward potential contributions to science, as well as their capabilities autonomous decision-making under un- certainty. Gravity-Bench-vl is available at https:// www.tb...../oooaoooooooooooooo and https://huggingffen http::/iinnaaaa.ecaaaaaaeaaaeeee cotdaassets/ Gravi www.ittyBench/rra.tttyeeeet
reflect the growing sophistication of AI foundation models in supporting, or enabling, various stages of the scientific cycle.
Many existing AI systems either treat scientific tasks in iso- lation, focus on specific optimizations emphasize pattern recognition (e.g., Reddy & Shojaee, 2024; Luo et al., 2025; Yuksekgonul et al., 2024; Ma et al., 2024a). Gravity-Bench diverges somewhat by framing discovery as a dynamic, it- erative process within a partially observable environment, simulating the challenges of real-world scientific inquiry. Agents in Gravity-Bench must actively explore to acquire hidden information and exploit collected data through rea- soning, embodying the interplay between observation and inference that underpins natural sciences. The rigorously- simulated nature of the Gravity-Bende environment also enables evaluation in out-of -itoo--distribution scenarios, testing generalization capabilities critical for robust scientific rea- soning.
2. Related Work
Advances in leveraging AI foundation models for scien- tific research and discovery encompass a wide spectrum of methodologies, reflecting the diversity of tasks underpin- ning the scientific method (Reddy& & Shojaee, 2024; Luo et al., 2025).
Specialized large language models (LLMMs,, such as Galac- tica or OpenScholar (Taylor et al., 2022; Asai et al., 2024: Sun et al., 2024), leverage domain-speciff training to im- prove literature analysis and information retrieval. Data- driven discovery has AI systems uncover patterns in exten- sive datasets, typically decoupling the data acquisition from the analysis (Majumder et al., 2024; Chen et al., 2024). Au- tomated statistical modeling focuses on deriving insights directly from existing data (Li et al., 2024). Workflow au- tomation frameworks have AI systems propose experiments and emulate research processes (Lu et al., 2024; Siegel et al., 2024; Ma et al., 2024a; Back et al., 2024; Ma et al., 2024b; Ghafarollahi & Buehler, 2024). Together, these methods
Existing benchmarks have explored virtual environments for data-driven discovery or symbolic regression (c.g.. Ma- jumder et al., 2024; Jansen et al., 2024; Udrescu & Tegmark, 2020; Guimerà et al., 2020; Lemos et al., 2023; Shojaee et al., 2025), though they often focus on rediscovery of known phenomena, solving textbook-style problems or fit- ting equations from pre-collected datasets. By contrast, Gravity- Gravity-Bench involves diverse dynamical scenarios requir- ing active observation scheduling, adaptive planning, and iterative scientific reasoning, mirroring the unpredictability
2

---

Gravity-Bench-vI: A Benchmark on Gravitational Physies Discovery for Agents
of real-world discovery processes. Gravity-Bench thus goes beyond curve-fitting or symbolic regression alone, as agents must proactively identify and integrate multiple quantities from strategically acquired data to solve tasks, emphasiz- ing scientific reasoning over memorization and encouraging hypothesis formulation that is novel yet rooted in the (simu- lated) environment being actively explored.
Modular partial observability: various observation pro- tocols can be adopted to sparsify in time the densely simulated data, making it possible to create environ- ments with varying levels of partial observability.
out-of-distributionn generalization: by enabling sim- ulation scenarios that do not occur in the real world, the engine also enables the evaluation of an agent's ability to handle novel situations and generalize be- yond its regular training data, a hallmark of scientific exploration.
Furthermore, the open-ended nature of Gravity-Benn -Bench tasks allows for diverse solution strategies, distinguishing it from tasks that emphasize solutions among preset answers. This characteristic favours exploratory reasoning and hypothesis generation in iterative cycles, rather than a more determinis- tic approach to measuring performance.
The observational protocol of the environment is an im- portant design choice, effectively decoupling the densely simulated data (in time) from the sparsely observable data. Here, for simplicity, we adopt two simple observational pro- tocols: full observability and partial observability with a finite observational budget. Within our simulated partially- observable environment,' planning and decision-makinn occurs through dynamic form of data collection, by obser- vational choices constrained by the environmental protocol (see Figure 1 for an overview of the benchmark). In a follow-up work, we also consider the addition of vision as an observational modality of the environment, enabling the evaluation of visual perception and reasoning as additional agent capabilities.
3. Benchmmak design
3.1. Environment Design
The core design principle behind our benchmark is the con- cept of rigorously-simullted partially-observable environ- ment.
Environments are preferred tools for evaluating agents, as they provide a dynamic setting to test capabilities, adaptabil- ity and generalization under controlled conditions. Many benchmark environments already exist in the literature, ad- dressing variety of domains and tasks, such as SWE- bench (Jimenez et al., 2024), RE-bench (Wijk et al., 2024), BrowserGym (Chezelles et al., 2024) or Aviary (Narayanan et al., 2024).
All simulations are implemented using Rebound (Rein & Liu, 2012: Tamayo et al., 2020), the current gold standard for gravitational few-body dynamics. Our standard Rebound simulation takes as input the stellar binary parameters (point masses, 3D positions, and 3D momentums), and any ad- ditional forces present. Rebound then solves Newton's ss gravity equations forward in time, for mostly 10 orbits. We save only the stars' Cartesian positions as function of time for the agent to access in solving the problem.
The engine driving our environment is a science-grade physics simulation tool. Using scientific simulation tools offers several advantages in the context of agentic bench- marks:
Focused subdomain expertise: The simulation targets a specific subset of physics/domain knowledge on which the agent is evaluated (here: 2-body gravita- tional physics). Implicit knowledge (c.g. Kepler's 3rd law) can be leveraged to solve some tasks more efficiently.
1.1.1. OBSERVATION PROTOCOL AND TOOL
In this version of Cravity-Benccl all orbits are in the (x.y) Cartesian plane by construction (i.e. z=0 at all times). This is closely related to the ideal 'face-on' geometry of real observations, where a binary's orbital plane coincides with the plane of the sky. Geometric projection effects, which are paramount in more realistic observations, will be addressed in a subsequent benchmark version.
Ground truth embedding: the enviromment encodes ground truth in the form of input simulation parame- ters. They impact the environment's dynamics, which is what is observable by the agent. The agent can be thus tasked to infer the hidden ground truth or to mea- sure/discover additional properties in more open open-ended tasks, mimicking natural scientific inquiry.
In Gravity- Bench-v1, we adopt two environment observa-
'Here, partial observability refers to the agent having only ac- cess to time-selected snapshots from the densely simulated data available in the environment. Even though our setup promotes planning under uncertainty and active information acquisition, con- cepts found in the POMDP literature (c.g.. Kaclbling et al., 1998; Bowyer, 2021), we note that our observation protocol assumes error-free observations. This idealization can be relaxed in fu- ture henchmark iterations additionally make the environment stochastic.
Limitless data generation: the simulation engine can generate virtually unlimited data for arbitrarily com- plex problems within the simulated scientific domain of interest, facilitating diverse and comprehensive eval- uations.
3

---

Cravity-Bench-vI: A Benchmark on Gravitational Physies Discovery for Agents
tion protocols, which are mediated by observation tool made available to the agent. In the first 'full-obs' protocol, the agent has access to the full dense set of simulation data.
In addition to standard Newtonian gravity, we introduce six scenarios that deviate from real-world physics. Three incorporate a drag force, requiring agents to infer the drag timescale from shrinking orbits, and three adopt a modified gravitational exponent with a force of gravity FG x r --22a)), where r is the separation between stars, and the task is to determine a (which is 0 in Newtonian gravity). We emphasize that our drag and modified gravity laws represent scenarios rarely, if ever, considered in stan- dard textbooks or physics literature two-body dynamics, hence they are out-of-distributin within the context of our benchmark and likely require compositional generalization from physics learned in other contexts.
In the second 'budget-obs', the agent is permitted a max- imum pre-determined number of observations, Nabs. con- strained to be within the time range covered by the dense simulation data. The agent free to choose which times to observe, 2 for up to 10 data points per observation-tool call, subject to maximum total of Nobs- In the 'budget-obs' protocol, the agent is therefore incentivized to be strate- gic within the observational budget allocated and to reason along as more observations are collected in several modular steps.
From our 16 two-body simulations, we design 50 tasks, 47 with numeric answers and 3 true/false. We match each task with multiple simulation variations for total of 206 task-simulation pairs.
In practice, the agent repeatedly queries the observation tool with series of observation times, and the tool returns the corrssonning data, appending them to previously-collecte observations."
4. Experiments
3.2. Scientific problems
Binary star systems modeled as point masses offer rich enough abstraction for our first benchmark to cover wide range of potential problems. In particular, this includes tasks that involve inferring hidden physical properties from limited observational data. For some tasks, the target val- ues are direct parameter inputs into the simulation such as component masses. Other tasks involve finding values that not direct inputs but can be derived from the simulation data, such as star's average distance from the center of mass, the fraction of time acceleration is below the mean, or the time it takes a star to travel 20% of its orbital path.
4.1. Evaluation details
For each task, we implement an expert solution based on only the data available to the Al agent and confirm the solu- tions agree with the simulation inputs if directly available for that task or a superior solution based on additional in- formation from the Rebound simulation (such as a built-in evaluator for orbital parameters).
These expert-reeerence solutions were produced by one PhD student and one research scientist with domain ex- pertise. Each of the two independently double-checked ev- ery answer, and the final values were cross-verified against Rebound's built-in orbital diagnostics whenever available.
We first design a diverse set of two-body gravitational simu- lations, illustrated in Figure 2. We deliberately incorporate symmetry-breaking strategies, such as displacing the cen- ter of mass from the system origin introducing uniform center-of-mas. drift (known as "proper motion"),, to mirror the messy realities encountered in genuine astronomical observations.
Under the constraint of an observation budget, the perfor- mance of our expert algorithmic solutions depends on the observation strategy. Therefore, we use the performance of a baseline strategy to set task-specific error thresholds that an AI answer must beat to be marked correct. Specifically, we define our baseline expert reference (expert-ref-NN. as the performance obtained using Nois observations equally spaced in time (without planning), since optimal observation strategies for all simulation-tase pairs are costly to develop. When evaluating Al agents with an observation budget of Nave = 100 (budget-obs- 100), we evaluate the Al agent performance against this 'expert-ref-100 1000 baseline.
We then design tasks to be solvable only through careful derivation requiring success at multiple intermediate steps. This aligns with the scientific process in reality. For exam- ple, to determine the total energy of the system, one must first find both stellar masses, which in turn require estimates of accelerations and separations.
Cubic interpolation between densely simulated timesteps pro- vides near continuous-time coverage with minimal numerical er- rors.
An answer is marked correct if its percentage error relative to our ground truth answers falls at or below the task-specific maximum permissible threshold. These thresholds account for the inherent difficulty of solving each task with lim- ited observations. They are set based on the performance gap between our expert solution using full simulation data (expert-ref) versus 100 uniform observations (expert-ref-
'The agent is allowed to observe at any point back in simulated time. An observation protocol that would more closely match observations in the real world would constrain any new observa- tions to occur only forward in time relative to reviouslyycollectee observations. We will explore additional observation protocols in future work.
4

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
10.1 a. M
10.1 Ma 5.6 Mo Unbound
Modified Gravity 1.97
10.1 Ma. 5.6 Ma
10.1 M 5.6 M
0,88 Ma. 0.63
Proper Motion
Modified Gravity 1.97 Modified Gravity 2.03
21.3 Mo. 3.
viher m)
y[ mi
m
y m]
y[ mi
y[1e13 ml
100%
-0.5
11.00
-7.5-5.0- -25
x nell m]
[1e12 ml
x [1e11mg ml
[1el1 ml
[lel1 ml
X [[ee12 m]
7.7 Mo. 4.9 Me
Simulation Progress
Me. 0.18 Ma Elliptical
Mo. M
Mo.
Drag == 8.3e8
Me.
9.6 3.1 Proper Motion
Elliptical, Single Orbit
Drag T= 1.7e9
Proper Motion
Drag T= 8.3e8
y (ie12 ml
2 mi
([ell mi
flea nu
m
[1e11 ml
Ttaal
o
-5
0
x [1lell m]
[leel m]
x [1e12 m]
x [1e12 ml
[1e12 m]
x [1e13 ml
9.6 M.. 3.1 M Proper Motion
9.5 a.. 3.1 M.
9.6 a.. 3.1 M Units: [AU, Ma, yr)
Example Tasks
Units: (cm, g, s)
9.6 3.1
What the total energy? What the orbital period? What is the mass of star2?
(1e12 ml
(Bei cm]
7.5t
[ie12 ml
7.5
y [AU]
5.0
5.0-
velocity start?
What is the distance at closest approach?
0%
2.5
specific angular momentum?
-2.5 0.0 [AU]
drag force exists, what its timescale?
x [1e12 m]
x [1e14 cm]
x [1e12 m]
Figure 2. Overview of the gravitational simulations used the benchmark. Each panel shows the orhital trajectories of a hinary star (Mo). The color gradient indicates simulation progress from (dark)
system in the x-y plane, with masses indicated in solar to end (light). Simulations include standard orbits, systems that are unbound, systems with modified gravity, systems with drag forces, systems with proper motion, etc. The benchmark also includes versions the same system represented different units, to evaluate unit handling. Sample questions that could be asked about these systems are shown.
Max velocity of starl
Closest Approach (Periastron)
100
100
100
100
Percent Error
(a)
Observations
Percent Error
(c)
Observations Attempted
Uniform
Attempted
Sampling
Threshold
50
50
50
50
GPT-4o
Claude 3.5
Sonnet
0
50
100
50
100
50
100
50
100
Observations Attempted
Observation Budget
Observations Attempted
Observation Budget
Figure Agent performance finding the maximum velocity of star under various observational budgets. (a) Percent error for each agent function of the total number of observations used, where each point represents an individual run. Uniformly sampling in time with a expert solution (red line) serves as a planning free baseline. Claude 3.5 Sonnet (blue) sometimes refines its observations enough achieve under 1% error, while GPT-4o (orange) shows less consistent improvement. (b) Observations attempted by each agent as a function of the max allocated observation budget. Points show individual runs, while lines with error bars show the mean and standard for that budget. While ideal approach would exploit all available observations (dashed line). both GPT-4o and Claude 3.5 Sonnet stop carly, often using fewer than half of the available observations for budgets above 30. This underutilization highlights in finding the periastron distance in single, highly elliptical orbit where the stars spend only 0.2% of the time within 5% of the closest approach. As discussed in Appendix D, an expertly planned solution can achieve error with observations, our safform-sampling baseline with 100 observations (without planning) performs poorly (70% error), do both Al agents. The horizontal dashed lines indicate the threshold by which the agents are marked in budget-obs-
lack of robust planning and answer verification. (c), (d) Percent
5

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
Determine the max velocity starl
Run 1: Claude 3.5 Sonnet
Run 2: Claude 3.5 Sonnet
help you calculate the maximum velocity for starl its
rill help you calculate the maximum velocity for starI its orbit. To do this, we'll need to:
orbit. To do this, we'll need Gather position data over time Calculate velocities from position differences Find the maximum velocity magnitude
1. Gather position data over the arbit
2. Calculate velocities between consecutive positions
3. Find maximum velocity
Let's by gathering observations apread time window get view of the orbit:
Let's start getting some observations spread window capture the orbital mation:
observe(O.0, 1.56e+9, 3.1e+9, ... 1.24e+10, 1.395e+10)
observe(O.0, 1.55e+9, 3.1e+ 1.2e+10 1.395e+10)
vxx' пр (nt-graddent(ff((" starl_xx] xx of[ "time" ]) ... dfl ufl'v_ "v_nennituude np.sqrt(df[ vx' vx']**2 dfl vyy [+42 df[ 'vz' [+2))
dt np.d:ff(df[ time' dx np.diff(dr[ (starrr[sssrrxx]]] vx dx/dt
Maximum velocity found far: m/s
v_total np.sqrt(vx**2 vy**2 vz**2) ...
Time points where velocity Is highest:
Maxinum velocity found so far: 16.266 m/s [Corresponding
v_nagnitude
velocity not printed)
1 1.395e+18
15.266
Let's get more detailed observations around the time where we found the highest velocity to make sure we don't miss the true maximum
Let's get more detailed observations around the times where we see higher velocities to make sure we don't miss the true maximum.
1ssserve11.3+1111 1.32++00, 1.34e+10, 1.46e+10, 148e+10)
observe(7.75e+4 2.325e- 1.3175e=+0, 14725e-10)
(Same code above) Time points where velocity 1s highest::
(Same code as above) Maxinun velocity found so far: 2745.81 m/s
time v_magnitude
1. 1.395++18 5437. 134
(progressively finer observations around max velocity)
(progressively finer observations around wrong time)
We have found that the maximum velocity of starl Is approximately 9080.41 m/s.
We have found that the maximum velocity for starl over its orbit is approximately 5135.48 m/s.
808041 m/s: Correct, 2% from value
5135.48 m/s: Incorrect, 45% from value
start Velocity (iiss)
10000
tell Tu aalon
10000
Observation Set
5000
5000
0.00
0.25
0.50
0.75
1.00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
1.25
1.50
Time [s]
1e10
Time [s]
1e10
Figure 4. Two observation-plannnn runs by Claude 3.5 Sonnet on the same task using 40 observations. The figure highlights how minor differences planning lead to drastically different outcomes. In each run, the agent collects position data in multiple steps, computes velocity from finite differences, and refines its search for the peak velocity. Top panels: Excerpts of the agent's 's traces including the planning, obscrvations, and code use. On the left, the agent systematically tracks the highest velocity times, and progressively refines its estimate. achieving final error of only 2%. On the right, however, the agent never accurately records peak velocityy times, and proceeds query intervals around finer time resolution as evidence of higher true velocities, rather than as improved measurement accuracy. Bottom: True velocity curves (gray) overlaid with the agent's 's observations (colored dots). Later queries appear brighter hues, showing how an intelligently planned approach can converge near correct velocity, while misplanned approach (right) fails to capture the velocity peak.
velocity times, resulting a error. It seems to misinterpret increasing velocity estimates from
6

---

Gravity-Bench-vI: A Benchmark on Gravitational Physies Discovery for Agents
Table Model performance. The Performance column shows the percentage of tasks each model solves within question question-specific error threshold. Under budget-obs- 100, a maximum of 100 observations can be requested, testing the ability to plan. The model is marked correct it within the threshold determined access all observations and is marked correct if within 5% of the correct answer. We also report total cost, total run time, and average observations used. Results averaged over three runs with standard errors shown.
each task (between 5% and 70%, see Section 4.1). Under full-obs, each model can
Score
Total Cost (S)
Total Time (min)
Mean Observations Used
Sequential Observations
100 Observation Budget
w4-mini-highh22255-4----11
49.4% + 2.6% %
81.23 + 1.68
2854.5 151.7
33.2 + 1.5
clande-3-5-somme--202444
21.5% + 2.5%
15.88 + 0.64
128.3 = 2.5
24.3 + 0.5
daudd-33-5haakkx222241002
16.1% + 2.3%
3.33 + 0.10
94,4 + 0.7
12.6 + 0.4
spt-4o-2022-11-22
15.5% + 2.1% %
9.60 1 0.12
63.5 + 1.6
12.2 1 0.7
gpL-40-miini22244-07-11
8.3% 1.5%
0.60 + 0.03
83.3 1 4.2
13.4 1 1.0
Full Table Access o4-mini-high--2225-0--16
73.9% + 2.4%
15.59 - 0.24
522.6 + 14.2
daude=35-xomme222224002
39.5% 3.2%
5.58 + 0.06
69.9 + 0.3
spt-4o-2024-11-20
36.1% + 3.2%
3.41 + 0.17
45.6 + 3.2
20241022
34.1% = 3.1%
1.46 0.03
63.3 - 1.3
gpt-4o-mini-200 -02--22024-07-18
26.7% = 2.8%
0.16 0.00
40.4 = 0.7
100): lexport_ref (xppettt0)pxper_ovvoooo (xplla__ Tasks where uni-
agent architectures. The baseline agent prompts are shown in Appendix F.
(fullobs)
form sampling achieves near-full-dee full-daalaa performance (e.g., orbital period estimation) receive strict thresholds (5%), while those where 100 uniform observations are insuffi- cient (e.g., maximum velocity measurement) allow larger margins (20%). For extreme cases like measuring the expo- nent of modified gravitational force, where expert-ref- 100 shows >> 1000% error, we set lenient but achievable thresh- olds (70%). To show this is achievable for this task, we find that an expert solution can reach within 1.7% eиror of the ground truth gravitational exponent with 70 elaborately planned observations (see Appendix C). For most problems, only the combination of strong algorithmic solution and observational strategy leads to a high quality answer.
4.3. Performance of baseline agent
We test OpenAl (OpenAI, 2024; 2025) and Anthropic (Anthropic, 2024) models in Table 1 04-mini-high achieves the highest performance among tested models with Claude 3.5 Sonnet in second. When constrained by 100- observation budget, each model shows a significant per- formance drop, suggesting observational planning remains challenging. The full-obs evaluation configuration requires less computation time than budget-obx- 100 because all ob- servations are immediately available to the agent.
Impressively, 04-mini-hin consistently solves multiple runs across all modified gravity tasks in full-obs, accurately esti- mating the exponent in scenarios with gravitational forces following FG x 222 This specific modification of gravity is rarely discussed in textbooks or literature in the two-body dynamics context. This suggests generalization to novel physical scenarios, although more evaluations in this regime are warranted. In budget-obs, it fails to consistently solve the ditto---ditributionn problems.
We also design our tasks to resist random guesswork. In the modified gravity case, the model must find the deviation from Newtonian gravity (f2+a) rather than the full exponent (r"), as 0.03 is much harder to estimate within 70% (correct range: [0.009, 0.051)) than 2.03 for example.
4.2. Baseline agent
Our observation protocol (Section 3.1.1) requires planning future observations based on existing observations, making single-step solutions unlikely succeed. Therefore, our benchmark is designed to evaluate AI systems that operate as agents that probe the enviromment and perform actions over multiple steps. We design a baseline agent around a ReAct-style scaffold (Yao et al., 2023). The agent can use our observe tool and a Python interpreter adapted from Langchain (Chase, 2022) with access to packages like numpy (Harris et al., 2020), scipy (Virtanen et al., 2020) and pandas (pandas development team, 2020) and receive outputs and exception tracebacks. While we evaluate this specific configuration, our benchmark supports arbitrary
The only other models to solve some OOD tasks were Claude 3.5 Sonnet, which consistently solves one modified gravity task and GPT-4o mini, which uncxpectedly solves single modified gravity task only once out of three runs. This might demonstrate the power of repeated sampling even of less capable models, discussed in Brown et al. (2024).
*We note that the Claude 4 models were released during the final stages of this work and thus was not included in our analy- sis (Anthropic, 2025).
7

---

Gravity-Bench-vI: A Benchmark on Gravitational Physies Discovery for Agents
4.4. Planning
higher velocity, though these initial velocity estimates are imprecise due the coarse time resolution. Upon detecting high-velocity region at a specific time, the agent imple- ments progressively finer temporal resolution around this interval. This iterative refinement approach achieves a fi- nal velocity measurement within 2% of the ground truth. Notably, the agent maintains a record of the highest veloc- ity magnitudes and corresponding times, enabling targeted subsequent observations in regions of interest.
An elaborately planned observational strategy is required to solve problems efficiently under the restriction of budget. This is most evident in problems such finding the max- imum velocity of star since velocity is highest only over the small fraction of an elliptical orbit when the two stars are closest to eachother. Uniformly sampling observations in time is not sufficient to determine this max velocity, as evidenced by the performance of our expert-ref-100 solution (~20% off).
In contrast, the run in the right panel fails to converge on the correct velocity. While attempting a similar strategy, the agent neglects to track the times of peak velocities. When the time resolution is refined, the velocity estimates ap- pear to increase, but this reflects improved measurement aceuracy rather than finding a truly higher-velocity region. Consequently, subsequent observations are mistakenly con- centrated in regions of lower velocity, and the agent reports velocity that deviates from the true maximum by approxi- mately 45%.
To investigate planning ability,, we provide the AI agents tasks, including finding the maximum velocity of star and finding the minimum separation of the two stars (the peri- astron), but vary the allowable max budget from 10 to 100 observations. Figure summarizes these runs by plotting the agent's 's error against the number of observations the agent decided to conduct. Elaborate planning is required to signifi- cantly outperform the expert-ref-100 100 baseline. This baseline achieves very high error (> 90%) with 10 observations to moderate error (20%) with 100 observations, reflecting the difficulty of pinpointing the specific max-velocity region with uniform temporal sampling (the minimum separation task shows similar results as discussed in Appendix D).
4.6. Failure modes
We observe that the models incorrectly assume symmetry in the system. For example, we observe that they often wrongly assume the center of mass is at the origin (0, O, 0), or they neglect that the system can have drift (see simu- lations labeled "proper motion" in Figure 2) when finding orbital properties that are critical to solving the problems, leading to incorrect answers.
A highly effective strategy would use substantial frac- tion of the budget to iteratively refine estimates around the time ranges suspected of containing the maximum velocity. However, we observe that all models tested request very few observations relative to their budgets. For budget-obs- 100, GPT-4o only uses 12 observations on average (Table 1) while Claude 3.5 Sonnet uses twice as many. This might explain Claude's higher performance but is still vast un- derutilization of the budget. Nonetheless, the experiment in Figure 3 shows that Claude 3.5 Sonnet solves this max velocity task to within 1 % due to elaborately observation planning.
Our findings reveal a tendency for Al models to take short- cuts rather than systematically derive intermediate quanti- ties. For example, they frequently bypass calculating the mass of the stars directly and instead assume value, such as gram, continue with the problem. As detailed in A Appendix E, we find that solutions containing such mass as- sumptions correlate strongly with incorrect answers across all models tested. Notably, GPT-4o makes mass assump- tion in 33% of incorrect solutions compared to 5% of correct solutions. Even Claude 3.5 Sonnet, which performs better overall, shows more than double the rate of mass assump- tions in incorrect responses versus correct ones.
We find that the agents stop further observation once a plau- sible solution is found rather than observing further to con- firm their answer (e.g. via denser sampling re-checking of other orbital regions). These experiments indicate that the tested models tend to prematurely rush to solutions and systematically underutilize their observational budget when working in the baseline agent framework, an issue we have not yet fully explained and explicitly identify as an impor- tant direction for future automated analysis of agent traces, given the extensive length and complexity of these traces.
5. Conclusion and Discussion
We introduced Gravity-Benche signed to evaluate Al agents in tasks that emulate the sci- entific discovery process, requiring iterative reasoning, dy- namic planning, and robust generalization. By challenging agents with both standard and out-of-distribution physics scenarios, the benchmark aims assess scientific reason- ing capabilities beyond memorized solutions. Our results demonstrate that while baseline AI models perform moder- ately well with the full table of observations, they struggle
novel benchmark de-
4.5. A case study on planning
Figure 4 presents two runs by Claude 3.5 Sonnet, finding the maximum velocity of a star with 40 observations. The left run demonstrates more effective strategy. It begins with a broad temporal sampling to identify regions with
8

---

Cravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
under constrained observational budgets, often failing to plan exploit available data effectively. These findings highlight current limitations in long-horizon reasoning and adaptive secision-making, which are important components for autonomous scientific discovery.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Looking ahead, Gravity-Bench-lic approaches have signifi- cant potential for growth as tools for advancing AI research in scientific reasoning. By expanding the benchmark to include incrementally more complex physics, one can aim to map out progress toward Al systems capable of gen- uine contributions to science. Additionally, this type of benchmarks with controlled environment and open-ended solution space may provide opportunities to characterize the robustness of autonomous agents in handling novel and uncertain scenarios, an issue connected safety. Finally, adapting environments like Gravity- Gravity-Bench-vl for reinforce- ment learning has the potential to serve as a stepping stone towards building Al agentic systems that not only analyze but also explore and innovate in the domain of scientific discovery.
References
Anthropic. Claude 3.5 somnet model card addendum, 2024. URL
com/
fabboc39aaabb1111222774448888888 Moode__arrd__Laadd____ddeeeee
Anthropic. Introducing claude https: /www. anthropic: cwme http://ww//lauudd4.... 2025. Ac- cessed: 2025-05-28.
Asai, A., He, J., Shao, R., Shi, W., Singh, A., Chang, J. C., Lo, K., Soldaini, L.., Feldman, S., D' D'arcy, M., Wadden, D., Latzke, M., Tian, M., Ji, P., Liu, S., Tong, H., Wu, B., Xiong, Y., Zettlemoyer, L.., Neubig, G., Weld, D., Downey, D., tau Yih, W., Koh, P. W., and Hajishirzi, H. Openscholar: Synthesizing scientific literature with retrieval- Ims, 2024. URL httts:://axxiy org abs / 241121111111 1. 14199.
Acknowledgements
Work in the Al Physics and Safety Lab at the University of Toronto is supported by the National Science and Engineer- ing Research Council of Canada, the Dunlap Institute of As- tronomy & Astrophysics (seed funding), Open Philanthropy, Anthropic (compute credits) and OpenAl (superalignment grant). MAD was supported by Tamkeen under the NYU Abu Dhabi Research Institute grant CASS.
Back, J., Jauhar, S. K., Cucerzan, S., and Hwang, S. J. Researchagent: Iterative research idea generation over scientific literature with large language models, 2024. URL https::ppp.. wttpxviivvvvm org/ /abs////////// 22444 07738.
Bowyer, C. M. Approximation methods for partially ob- served markov decision processes (pomdps), 2021. URL
We would furthermore like to acknowledge that this work was performed on land which for thousands of years has the Seneca
/ttps://arxiv.ort absp:///b//////// 13965.
been the traditional land of the and the Mississaugas of the Credit. Today this meeting place still the home to many indigenous people from across Turtle Island and we are grateful have the opportunity to work on this land.
Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL
224p////bb/////...o//// 2407 21787.
Chase, H. LangChain, October 2022. URL https://
Author Contributions
github w.thub.com/1ll
KM proposed the simulated observable environment ap- proach adopted in GBv1, the various task categories, guided the benchmark design throughout and wrote several sections. NK designed and wrote most of the benchmark code-base, specific tasks, and implemented all expert problem solutions. NK also wrote the bulk of the paper content and produced all figures. HJ contributed substantially to the code- base. MAD provided domain-knowledge validation on all the code, tasks and solutions, designed the table detailing all tasks and wrote several sections.
Chen, Z., Chen, S., Ning, Y., Zhang, Q., Wang, B., Yu, B., Li, Y., Liao, Z., Wei, C., Lu, Z., Dey, V., Xue, M., Baker, F. N., Burns, B., Adu-Ampratwum, D., Huang, X., Ning, X., Gao, S., Su, Y., and Sun, H. Scienceagent- bench: Toward rigorous assessment of language agents for data-driven scientific discovery, 2024. URL https: //arxiv. orgo abs////////// 2410 10. 05080.
Chezelles, T. L. S. D., Gasse, M., Drouin, A., Caccia, M., Boisvert, L., Thakkar, M., Marty, T., Assouel, R., Shayegan, S. O., Jang, L. K., Ld, X. H., Yoran, O., Kong. D., Xu, F. F., Reddy, S., Cappart, Q., Neubig, G., Salakhutdinov, R., Chapados, N., and Lacoste,

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
The browsergym ecosystem for web agent research, 2024. URL htttp:://axxi.....ttt wttp://www..rr///////////// 05467.
Kaelbling. L. P., Littman, M. L., and Cassandra, A. R. Planning and acting in partially observ- able stochastic domains. Artificial Intelligence, 101(1):99-134, 1998. ISSN 0004-3702. doi:
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL htttp:://axii...tt //aaxxxxvvooog.. abs/ 1803. 05457.
URL sitenc//azriicleaa////0440000080
Lemos, P., Jeffrey, N., Cranmer, M., Ho, S., and Battaglia, P. Rediscovering orbital mechanies with machine learning. Machine Learning: Science and Technology, 4(4):045002, December 2023. doi:
Eggleton, P. P. Aproximations to the radii Roche lobes. Astrophysical Journal, 268:368-366 May 1983. doi: 10.1086/160960.
Li, M. Y., Vajipey, V., Goodman, N. D., and Fox, E. B. Critical: Critic automation with language models, 2024. URL https::/p//.t.... //tp///axivvvvv/ org/ wtwg/ba/////.oooooorro.oo///eaae /22//2411111 000.411..99990 96590.
Ghafarollahi, A. and Buehler, M. Sciagents: Automating scientific discovery through multi-agent intelligent graph reasoning, 2024. URL https:: /ttppis///axxvvooo// org abs/ 05556.
Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery, 2024. URL https: / waaaiivv.rg////...vva.a........a abs 2088 06292.
Guimerà, R., Reichardt, I., Aguilar-Mogas, A., Massucci, F. A., Miranda, M., Pallarès, J., and Sales-Pardo M. A bayesian machine scientist to aid in the solution of challenging scientific problems. Science Advances, 6(5), January 2020. ISSN 2375-2548. doi: 10.1126/sciadv. aav6971. URL http:// /xx doi www.ooggg//0... 1126/ sciadv. aav6971.
Luo, Z., Yang, Z., Xu, Z.,, Yang, W., and Du, X. Llm4sr: A survey large language models for scientific re- search, 2025. URL https: http:///xxxvvvrio. 2501.04300 04306.
Ma, P., Wang, T.-H., Guo, M.. Sun, Z., Tenenbaum, J. B., Rus, D., Gan, C., and Matusik, Llm and simulation as bilevel optimizers: A new paradigm advance physical scientific discovery, 2024a. URL https: //arxiv.. org/ abs / 220/2245500000 0000050097..
Harris, R., Millman, J., der Walt, S. J., Gommers, R., Virtanen, P.. Cournapeau, D., Wieser, E., Taylor, Berg, S., Smith, N. J., Kern, R., Picus, M., Hoyer, S., van
Kerkwijk, M. H., Brett, M., Haldane, A., Fernández del Río, J., Wiebe, M., Peterson, P., Gérard-Warchant P., Sheppard, K., Reddy, T., Weckesser, W. Abbasi, H., Array programming
Ma, Y., Gou, Z., Hao, J., Xu, R., Wang, S., Pan, L., Yang, Y., Cao, Y., Sun, A., Awadalla. H., and Chen, W. Sci- agent: Tool-augmented language models for scientific reasoning, 2024b. URL https: http:///axvvvvvo /bs/ 2402. 11451.
Gohlke, C., and Oliphant, with NumPy. Nature, 585:357-362, 2020. doi: 10.1038/ 141586-020-2649-22
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding, 2021a. URL https: //arxiv. / abs/2009..0o 03300.
Majumder, P., Surana, H.,, Agarwal, D., Mishra, D., Meena, A., Prakhar, A., Vora, T., Khot, T., Sabhar- wal, A., and Clark, P. Discoverybench: Towards data- driven discovery with large language models, 2024. URL https: / arrxiv org/ /ttp:/207.0000
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song. D., and Steinhardt, J. Mcasuring math- ematical problem solving with the math dataset, 2021b. URL BtLhttt::/// //apiiii/////..v../////pap orgo/rgg//.... abs 2103. 03874.
Narayanan, S., Braza, J. D., Grifliths, R.-R., Ponnapati, M., Bou, A., Laurent, J., Kabeli, O., Wellawatte, G., Cox, S., Rodriques, S. G., and White, D. Aviary: training language agents on challenging scientific tasks, 2024. URL /ttpss://rrxe .rg. /atg//b////// 22/22411.222 21154.
Jansen, P., Côté, M.-A., Khot, T., Bransom, E., Mishra, B. D., Majumder, B. P., Tafjord, O., and Clark, P. Dis- coveryworld: A virtual environment for developing and evaluating automated scientific discovery agents, 2024. URL https: aaaii..iiivv///i///aaaaaaaaaavva org/ abs/ /406. 06769.
OpenAl. Hello GPT-4o, June 2024. URL https: ctttlaa.com/innxx//e/e/e 2025-01.
gpt 4o/. Accessed:
Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language models resolve real-world github issues?, 2024. URL org/abs/// https: /arxivv. org/ abs /2300 06770.
OpenAL. mini
Introducing openai o3 and o4- https://openaf: compoeaai.com.//ooooooommmoooome com hom/inders index/ ixtrodduiing--3--a--------- Accessed: 2025-05-24.
April 2025.
10

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
pandas development team, T. pandas-dew/pandas: Pan- das, February 2020. URL https::/doi..rrge 2281/zenodo.33
Peng, H. Scicode: A research coding benchmark curated by scientists, 2024.
Ting, Y.-S., Nguyen, T. D., Ghosal, T., Pan, R., Arora,
Reddy, C. K. and Shojace, Towards scientific discov- ery with generative ai: Progress, opportunities, and challenges. 2024. URL httph://aaaaat orgg / 2412.11422
H., Sun, Z., de Haan, T., Ramachandra, N., Wells,
A., Madireddy, S., and Accomazzi, A. Astromlab 1:
Who wins astronomy jeopardy!?, 2024. URL https: / wwg.as//..gg//a/.ae/a......
11194..
Rein, D., Hou, B. I.., Stickland, A. C.,, Petry, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https: / arxiv.. org/ abs/ / 2311.11111 12022.
Udrescu, S.-M. and Tegmark, M. Ai feynman: physies- inspired method for symbolic regression, 2020. URL https http://axxvvvvvo org abs 1905 11481.
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M.. Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kem, R., Larson, E., Carey, C. J., Polat, I., Feng, Y., Moore, R. W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020. doi: 10. 1038 URL https: / orgg 10. 10.1038//41122--01---8888-
Rein, H. and Liu, S. F. REBOUND: open-source multi- purpose N-body code for collisional dynamics. Astron- omy & Astrophysics, 537:A128, January 2012. doi: 10.1051/0004-6661220 118055.
Rein, H. and Tamayo, D. WHFAST: fast and unbiased implementation of a symplectic Wisdom-Holman inte- grator for long-term gravitational simulations. Monthly Notices of the Royal Astronomical Sociery, 4)2(6):376 388, September 2015. doi: 10. 10.109//mrs///v 1257.
Shojace, P., Meidani, K., Gupta, S., Farimani, A. B., and Reddy, C. K. Llm-sr: Scientific equation discovery via programming with large language models, 2025. URL org/ abs/ 2404. 18400.
Wijk, H., Lin, T., Becker, J., Jawhar, S., Parikh, N., Broadley, T., Chan, L., Chen, M., Clymer, J., Dhyani, J., Ericheva,
https:: //appii///rxvv../////
Siegel, Z. S., Kapoor, S., Nagdir, N., Stroebl, B., and Narayanan, A. Core-bench: Fostering the credibil- ity of published research through a computational re- producibility agent benchmark, 2024. URL https: //arxiv. org/ abs 2209 11363.
E., Garcia, K., Goodrich, B., Jurkovic, N., Kinniment, M. Lajko, A., Nix, S., Sato, I.., Saunders, W., Taran, M., West, B., and Barnes, E. Re-bench: Evaluating frontier ai r&d capabilities of language model agents against human experts, 2024. URL org/ abs 2411. 11441.11114..1111
Sun, L., Luo, D., Ma, D., Zhao, Z., Chen, B., Shen, Z., Zhu, S., Chen, L., Chen, X., and Yu, K. Scidfm: A large lan- guage model with -ixturs for science, 2024. URL htw.tttp//// http://aaxvvvv... org/ abspgg/bbb//000// 11...8840 18412.
Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and act- ing in language models, 2023. URL org/ abs 22200.1.1.........2 03629.
Tamayo, D., Rein, H., Shi, P., and Hernandez, D. M. REBOUNDx: library for adding conservative and dissipative forces otherwise symplectic N-body in- tegrations. Monthly Notices of the Royal Astronom- ical Society, 491(2):2885-290) 2200, January 2020. doi: 10.1093/mmna//st22870.
Yuksekgonul, M., Bianchi, F., Boen, J., Liu, S., Huang, 7., Guestrin, C., and Zou, J. Textgrad: Automatic "differen- tiation" " via text, 2024. URL https: pettps://axxiv.orr abs 22e./2406.00999
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence?, 2019. URL https: aaxii. org/ /bs/ 1905.07830.
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshom, A., Saravia, E., Poulton, A., Kerkez, V., and Stojnic, R. Galactica: A large language model for science, 2022. URL https: //tp///xxvvor///////// abs 2211, 09085.
Tian, M., Gao, I.., Zhang, S. D., Chen, X., Fan, C., Guo, X., Haas, R., Ji, P., Krongchon, K., Li, Y., Liu, S., Luo, D., Ma, Y., Tong. H., Trinh, K., Tian, C., Wang, Z., Wu, B., Xiong, Y., Yin, S., Zhu, M., Lieret, K., Lu, Y., Liu, G., Du, Y., Tao, T., Press, O., Callan, J., Huerta, E., and

---

Cravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
A. Rebound simulations details
All simulations are implemented using Rebound, a popular gravitational N-body integrator. The core of this framework the ordinary differential equation formulation and the numerical time-integrator, to achieve machine precision and minimize error build-up over time. For most problems we use WHF (Rein & Tamayo, 2015), an unbiased, machine precision, energy conserving integrator. The integration timestep is conservatively chosen to be one-five tne--ive-tue -hoousdhus of the system's orbital period. For problems where forces other than gravity are present, or the gravitational law has been modified, WHFast is not adequate. We then use IAS15, an adaptive time-step 15th-order integrator where errors are kept to below machine precision.
Our standard Rebound simulation takes as input the stellar binary parameters (point masses, positions, and momentums), in addition the integrator choice discussed above, and any additional forces present. Rebound then solves Newton's gravity equations forward in time, for 10 orbits", At densely-sampled timesteps, outputs the complete Cartesian and orbital elements for both stars. This detailed information is saved for reference part of the environment but is not provided to the agent. Instead, we separately save only the stars' Cartesian positions as function of time for the agent to access in solving the problem, modulo the observation protocol.
B. Description of the benchmark problems
Question
Solution
Comments
Jargestrgett_star
Calculate the separation magnitude of the two stars using their 3D positions. Compute the acceleration of star2 by taking the sec- ond derivative of its position with respect to time. Using the acceleration and separation magnitudes and d, calculate the mass of starl: M1 = 4 Repeat the process for G
The
separation array is given by d
magnitude
Determine the mass of the largest star.
V (x) x2 2 (y1 y2) + (2) )2 The gravitational acceleration given by a GM Fi = d' stellar masses are determined whenever needed, for the rest the problems.
d² as the force This illustrates how
G2M,M-MM
starl to calculate the mass star2. Finally, return the maximum of the two masses: largest_mass max(M M2).
max_veloo www.oeelllsstaeeeeee starl Calculate the maximum value of velocity for starl over the orbit.
Calculate the velocity magnitude array of starl by computing the first order time derivative of its 3D position. Return the maximum of the array.
The velocity magnitude array is given by
Vi = ((e=1)² + (uy1)² 2 + (U=1)² where 221 dra/dt. Final Answer MAX(V1). This illustrates how stellar velocities are de- termined whenever needed, for all the rest of the problems.
max max_aacceleration.se
Take the second order derivative of position with respect to time to get the acceleration array then calculate its magnitude. Return the maximum of the acceleration magnitude array.
a1 = (( (ay1)² + (az1)² where ax1 dri/ /dl2. This illustrates how stel- lar accelerations are determined whenever needed, for all the rest of the problems.
Calculate the maximum value of acceleration for starl.
'Except for Linear drag and Modi fied gravity
12

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
semi sumi_major_ax axis Determine the semi-me semi-major axis of the system's s orbit.
Calculate the total semi-major axis by find- ing the magnitude of the 3D stellar sepa- rations and averaging their extremes (min/- max).
The total semimajor axis is given by max(d))-mindd)) where is the separation ray. For other questions, you can calculate individual semi-major axes by weighting the total axis by each star's mass ratio to the total system mass. This illustrates how stellar semimajor axis are determined when- ever needed, for all the rest of the problems.
For verification, set up a Rebound Simula- tion. Add the two stars the simulation, specifying their masses, positions, and ve- locities. Compute the orbital elements for star2, with starl I as the primary. Extract and return the semi-major axis from the orbital parameters.
eccentricity Determine the eccentricity of the system's orbit.
Calculate an array of stellar separation mag- nitudes along with its maximum and min- imum values (fmaz and rmmnn) The eccen- tricity is given by (Imoæ Imin) (Imaz Emin). For verification, initialize a Rebound simulation, add the stars, and obtain the eccentricity from the simulation.
Orbital eccentricity is a dimensionless pa- rameter that measures the deviation of an orbit from being perfectly circular. This illustrates how stellar eccentricities are de- termined whenever needed, for all the rest of the problems.
semi www.omiioo.... .axis Determine the total semi- minor axis of the system's or- bit.
Calculate the total semimajor axis a and the eccentricity e. The semiminor axis b is given b a vI- e², For verifica- tion, set up Rebound Simulation. Add the two stars the simulation, specifying their masses, positions, and velocities. Compute the orbital elements for star2, with starl as the primary. Extract semi-major axis and eccentricity from the orbital parame- ters. Calculate and return the semi-minor axis.
This illustrates how stellar semiminor axis are determined whenever needed, for the rest of the problems.
period
Calculate the magnitude of stellar sepa- rations using their 3D positions. Use to identify the peaks (maximums) in the separation array, which correspond to the apoastron points. From these peaks, calculate the average period by taking the mean difference between consec- utive peak times. Return this value.
Apoastron is the point in the orbit of star or other object in a binary system where
Determine the orbital period of the system.
scipy.signal.fidd.eeeo
it is farthest from its companion. This il- lustrates how stellar periods are determined whenever needed, for the rest of the prob- lems.
13

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
fime_ffaction.eee acceleraticn
Calculate the orbital period. Find the in- dex corresponding to time just after one full period and use this to locate the time of the next pericenter passage. Isolate the data for a single orbit by selecting the rows where the time is between the pericenter passage and one full period later. Calculate the mean acceleration of starl for this or- bit. Identify the time intervals where the acceleration of starl is below the mean. To account for potential irregularities in time steps, sum the time differences where the acceleration is below the mean. Finally, compute the fraction of the total orbital pe- riod during which the acceleration is below the mean and return this value.
bello_mean .eaan
Calculate the fraction of time a single orbit where the ac- celeration of starl is below the mean acceleration.
2K+U
Calculate the stars' masses, velocities and separation magnitudes to compute the sys- tem's 's instantaneous kinetic (K) and poten- tial (U) energies. Average the kinetic and potential energies arrays, then return 2K+U. Calculate the stellar semimajor axis and ec- centricities. The apoastron is defined a(1 + e). For verification, initialize a Re- bound simulation and set its units. Add both stars to the simulation with their masses, positions, and velocities. Compute the or- bital parameters of one star relative the other using Rebound. Finally, return the apoastron, the maximum separation along the orbit.
From the Virial theorem we have 2K + U O, where K the total kinetic energy and U the gravitational potential energy. This should be satisfied close the machine preci- sion in the simulated environment.
Determine the quantitative value 2K for the sys- tem in joules.
apoastron
Determine the apoastron of the system's orbit.
area_swept.over aiver_tme_an tim Calculate, at apoastron, the rate of area swept per unit time by the imaginary line joining s to star2.
Calculate relative positions and velocities between the two stars using finite differ- ences then compute the specific angular mo- mentum vector and half its magnitude. For verification, initialize Rebound simula- tion, add the stars, and obtain the specific angular momentum from the simulation.
Kepler's 2nd law: the rate of area swept is half the specific angular momentum of the system given by h = v h + h2 + h? with h y-u.- - 2- where (x.y) is the orbital plane.
avg distance COM starl Calculate the time-averaged distance between starl and the COM over a single orbit.
Calculate the stellar masses along with the orbital period and the time of pericenter passage. Isolate single orbit by filtering the data between the pericenter passage and one period later. Calculate the distance be- tween starl and the COM during this orbit. Finally, integrate the distance over time and divide by total time to get the time-averaged distance. Return this value.
COM: center of mass. The dis- tance
between starl and the COM is given by
d (Fstal #COM) + (Maarl JCOM )2
with ICOM mi -ard +mg- Analogous mi 4.ma
equations can be written for yCOM and ZCOM-
14

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
bound Determine whether the sys- tem is gravitationally bound (True) or unbound (False).
Compute the velocity magnitudes for both stars. Calculate the gravitational potential energy (U) using the separation between the stars and their masses. Calculate the kinetic energy (K) using the squared veloe- ities and masses. Compute the mean ki- netic and potential energies over the dataset. Check whether the total energy (K + U) in- dicates bound system. Return True if the system is bound.
Unbound system: K ++U > 0, adopting the usual notation of zero potential energy at infinite separation.
shpler_third_im
Calculate the stellar masses, periods, and semimajor axis. Compute the ratio p2 /²/a from the data and compare it to the theo- 4n² retical value: +ma] If the percentage
Kepler's Third Law: the squares of the or- bital periods of the planets are directly pro- portional to the cubes of the semi-major axes of their orbits.
Determine Kepler's ' third law is satisfied. Answer: True if Kepler's third law is satisfied, and Answer: False if is not.
difference is less than 0. %6, return True.
linear_drag
Calculate the separation between the stars using their 3D coordinates. Identify the peaks (apoastron) and troughs (perias- tron) in the stellar separation data using www.siigall.iid.pec...
This system experiences drag given by ace;
-0,/T 8 for the i-direction, where ace is the accelera- tion, is the velocity, and timescale. Calculate the value of the coefficient of lin- ear drag, for the system.
Ensure both arrays (peaks and troughs) have the same length by truncating the longer one if necessary. Compute the semi-majon axis for each orbit by averaging the apoastron and periastron distances. Find the average time for each or- bit by averaging the times corresponding to the peaks and troughs. Define an exponen- tial decay function to model the semi-major axis a function of time: a(t) ao exp( (cp((2t//T) 2t/T). Use ccipy.optimize.cury fit to fit this model to the semi-majon axis data and extract т.
max.angular www.annll..elllccttyt..omo.oo starl Calculate the maximum value angular velocity for starl over the orbit.
Calculate the velocity vector v of starl and the magnitude of the 3D stellar separation r. The angular velocity vector of starl is then given by w X Finally, return the maximum of the resulting array.
This can be retrieved for verification from Rebound using the specific angular momen- tum h: w
max_momentumm.o starl Calculate the maximum lin- ear momentum for starl over the orbit.
Calculate the mass and the velocity magni- tude of starl. Calculate the linear momen- tum array as the mass x velocity. Return the maximum of the array.
modified_gravity. www.gitt-pwee..leeet law This system governed by an alternative law gravita- tion where the dependence (2+a) where alpha rep- resents the deviation from Newton's inverse square law. Calculate a.
Calculate the stellar separation magnitudes and the total acceleration one of the stars. Take the logarithm of both separation (r) and acceleration (a). Fit linear function to log(a) vs log(r), but only for: Separations above the median separation and data points without outliers (using MAD-based outlier removal). The slope of this fit gives us -(2 + a), so |slope| 2
example where the or- bits are effectively precessing. In reality F& x r- and thus other exponent values are unlikely to be present in pre-training data.
15

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
multiply _aass-period
Compute the average period P' of the sys- tem. Calculate and return the factor X us- ing Kepler's Third Law: = (2T
Determine the factor X by which the central mass should be multipled for the orbital period of the system be 21 days. You can assume the central mass is starl which is much more massive than star2.
orbital area
Calculate the stellar semimajor axis and eccentricities. Then calculate the semimi- nor axis and finally the area. For verifica- tion, set up a Rebound simulation object. Add the two stars to the simulation, specify- ing their masses, positions, and velocities. Retrieve the orbital elements of one star (with the other star as the primary). Ex- tract the semi-major axis a and eccentricity from the orbit. Calculate the semi-minor axis as = a x vI ea2 Compute and return the total area of the orbit.
The formula for the total area of an ellipse:
Determine the total area en- compassed by the orbit of the system.
A x a x b.
orbital area star2 Determine the total area compassed by the orbit of star2.
Same as above, but with A π 02 b2.
roche_lobe sabbus Determine the Roche lobe ra- dius of starl.
Start by calculating the stellar masses and total semimajor axis. The Roche lobe radius Rg of starl can be calculated as: RL1 0.49-g2/2 0.6.g%/8+]n(1+q"/ aio where is the mass ratio of the stars.
Approximate (1983).
formula from
Eggleton
specific_angular _angular momentum Determine the specific angu- lar momentum of the system.
Find relative positions and velocities be- tween two stars. Calculate the specific angular momentum by computing the cross product of relative position and velocity vectors, then find its average magnitude. For verification, set up Rebound simu- lation. Add both stars to the simulation with their masses, positions, and velocities. Retrieve the orbital parameters of star2 rela- tive starl and extract the specific angular momentum.
The specific angular momentum is h = Fx
16

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
snvel_time-orbeee Sipiitall-perrpppr Calculate the time needed for starl to travel 20% of the distance along its orbital path.
Calculate the orbital period and time of peri- center passage. Isolate the data correspond- ing to one complete orbit by selecting the rows between the pericenter passage and one full period later. Next, calculate the changes in the true anomaly and the radial distances and their differences. Use the for- mula for are length in polar coordinates to compute the are length for each time step:
ds = 12² (dz)² dv. Sum these are lengths to get the cumulative path distance along the orbit. The total perimeter of the orbital path is the last value of this cumula- tive sum. Set a target distance correspond- ing to 20% of the total perimeter, and find the index where the cumulative path dis- tance is closest to this target. Finally, return the time difference between this point and the time of pericenter passage.
virial. .hhoormm
Calculate the distance between the two stars using their 3D positions, and compute the gravitational potential energy for each time step.
Determine if the Virial The- orem is satisfied in this sys- tem. Answer True the Virial Theorem satisfied or False if it is not.
Compute the velocities of both stars by tak- ing the gradient of their positions with spect to time, and then calculate the kinetic energy for each star.
Compute the mean kinetic (K) and potential (U) energies over the entire orbit. Accord- ing to the Virial Theorem, for a system equilibrium, 2K + U O. Check if the absolute value of 2K U is within 0.1% of the absolute value of U. Return True if it is.
C. Choosing task-specific thresholds for budget-obs
We define task-specific thresholds to evaluate agent solutions derived from 100 observation budget. These thresholds based on the performance gap between our expert reference solution (using full simulation data) and the same solution but using a non-strategic observation approach with 100 uniformly sampled time points.
To establish these thresholds, we first calculate the percent difference between expert-ref- 100 and expert-reffullllob))) for each simulation-task pair. As shown in Figure 5, each black dot represents this difference for a specific pair, grouped by task. Red horizontal lines indicate our final thresholds, set to reflect typical performance for each task category.
Tasks that rely on global trends, such determining the orbital period, are easily solved by uniform sampling and therefore receive strict thresholds. In contrast, tasks that are harder to solve with only 100 observations receive more lenient thresholds. These might require, for example, precise knowledge of local features, such as finding the maximum velocity of a star.
For particularly challenging tasks where uniform sampling fails dramatically (e.g., modified gravity exponent estimation showing >1000% error), we set a reasonable threshold (e.g. 70%) and validate that it is achievable through expert observation strategies. For instance, we validate that modified gravity exponents can be determined to within 2% accuracy through
17

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
Expert solution performance: 100 uniformly spaced observations
Absolute percent difference
from full-obs baseline (%)
100
23 23.4%%
80
2877%
10%
15%
60
20%
Performance on simulation
40
>100%
Required threshold
20
Pro rofou bu
mas aeceterate stare (ra) aic travel tieee orei 20per pain modd graviss Joowe Jawe
mar velne KeU
mas pod valoo stare
stara
mas yocr tun stord
mi mo morhotta Protuuu arriiaa mor specitte swint Over come poo epo and yor orblits travel avd prdden ce pals
starr
stord
stard
dun Sip fuls
porer Saou Aphpmu
B yiu prg inntuuu yiui tou Ein sind coteee
pagae soree coriin unu
time fraction, a (onii at don
soung juan
ors certeiee
over tine
yeinu Junes
2o1n una
aae ynu biu
Ilios
Figure 5. Finding task-specific thresholds based on expert solutions performance without planning (expert-ref-100) Each black dot represents the absolute pereent difference (relative solution with access the full simulated data) for simulation-task pairs grouped by task, using 100 uniformly spaced observations. Scenarios yielding large differences (e.g., 30-100% or more) show that naive uniform sampling is inadequate planning strategy. Red horizontal lines mark the final success threshold we choose for each task, that for all the simulations that task is based on.
strategic informed sampling.
D. Another case study on planning
One task in Gravity-Bende -eeehhy-Benchhee involves finding the periastron distance, the closest approach between two stars in their orbit. This becomes especially difficult for one of our simulation with a single highly elliptical orbit, where the stars spend less than 2% of their orbital period near this closest point (Figure 2). Our uniform sampling baseline (expert-ref- 100) misses this narrow window completely, resulting in a 72% error. As shown in Figure 6, Al agents typically fixate on an early orbital phase containing close (but non-minimal) separation, missing the true periastron that occurs later in the orbit.
Expert-level solutions with strategic planning can solve this task through initial broad sampling identify potential regions that contain the minimum separation of stars, further analysis of the two closest regions: the start and end of the orbit, focused refinement on the late-orbit phase which shows tighter separations, and iterative sampling around the minimum. This approach achieves 2% accuracy using just 50 observations, compared to the 93% error from uniform sampling. We set a 5% success threshold for this task, reflecting both the expert solution's 's performance and the easier variants present in other simulations.
E. Mass assumptions
Table 3 highlights consistent pattern across evaluated models: solutions that shortcut the derivation and incorrectly assume stellar masses (e.g. setting both masses to 1.0) correlate strongly with incorrect answers. In every model we test, larger fraction of incorrect responses rely on these arbitrary mass assumptions than correct ones. GPT-4o shows the largest gap, indicating it is particularly prone to incorrectly assuming the stars have equal mass or a mass equal to gram or kg (what one might call a science hallucination). A small fraction of correct responses also contain mass assumptions, suggesting that occasionally these shorteuts coincidentally lead to the right solution, but this behavior avoids a principled approach to solve these problems.
18

---

Cravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
Claude 3.5 Sonnet
Expert Solution
Separation
1012
Separation
1012
Observation Set
[m]
[m]
10
10
Closest approach (periastron)
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Time [s]
1e8
Time [s]
1e8
Figure Comparison uniform and expert observation strategies for finding closest approach of the stars in highly elliptical single-orbit system. The plots show the separation of the stars over simulation time. The left panel (Claude 3.5 Sonnet) samples at sparse, uniform grid of times, missing
narrow periastron window near end and yielding a distance over 93% larger minimum. By contrast, the expert solution (right) first samples broadly, then zooms into the first and last parts of the orbit, then decides to continue refining observations the last part of the orbit. This adaptive strategy locates the periastron to within 2% accuracy using the same observation budget. Circles mark observations, with brighter hues indicating later observations.
true
Table Prevalence of unjustified assumptions in model solutions Analysis of 122 mass-dependent solutions shows AT models frequently assume stellar masses rather than calculating them. Percentages show how often these shortcuts appear in incorrect vs. correct solutions when using full observational data. Mass assumptions were identified by following patterns: center-of-man calculation df [' star2.x' /2" and explicit mass assignments: ("starl. iettarl.nn rlmmss 1.0", "star2 rmamsss = 1.0".
" "(áff starl.xx "ml m2". 1" 1.0", etc.). Results for other models are averages of three runs.
% of incorrect solutions
% of correct solutions that include mass assumption
that include mass assumption
Suuule---S-somme-20221000
24.6 + 2.6 (23/94)
9.8 + 2.5 (2/27)
claude-3-5-haiie h-w.-----hhai--224000000
25.9 + 3.6 (26/100)
18.5 + 2.7 (4/21)
gpt-4o-2024-11-20
33.0 4.8 (32/97)
9.3 = (2/25)
1ill--mmni-222--07-2
22.1 0.6 (22/99)
9.0 = 2.6 (2/22)
o4-minn-highh-2225-4444
28.6 6.5 (11/39)
12.1 + 0.5 (10/82)
19

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
F. Prompt Templates
20

---

Gravity-Bench-vI: A Benchmark on Gravitational Physics Discovery for Agents
Prompt for budget -ba 100
You are tasked with solving the follewing physics problem related to binary star system. You are provided observations of each star's position over time, (t, in units seconds and meters.
### Problem Description
Determine the tctal energy (K () for the system in joules. You must provide your answer in units of J.
HP Additional Instructions
To complete this task, you have access to the following tools and data:
1. An observational tool called 'Observe' that allows you observe the system at specific times of your choosing.
2. code interpreter that can exccute Python code.
When using 'Observe';
1. The "temus_reeeesteed" parameter shculd be 1ist that can contain any values in the time window [0.0, 7.21e+09] seconds. cannot request negative times. The upper limit for the time window was chosen to guarantee that the problem solvable with an appropriate sampling of observations using the total observational budget.
if it is in 2. You observe the system any time within the time window, the past compared to the last observation.
3. You can observe the system up to a total of 100 times and you can observe up to 10 times per observational request which is the maximum length of the
'times_reee requestedd list.
4. After each observation, the dataframe 'row_wise riss__results.df will be updated. It contains columns: time, stari_x, starl_y, stari_z, star2_x, star2_y, star?_z. You can access it using the code interpreter tool. For example, to access the first five rows, print www_wwiseressullee of (n=5))
when using the code interpreter:
1. Always use ()int() to display results.
2. Do not use read_csv or attempt to lead the Datairame, as it is already pre-loaded Important reminder: Repeated tool access is enabled until you have found the answer and have submitted it with the submittanswerr tool.
Prompt for full-obs
You are tasked with solving the following physics problem related to binary star system. You are provided observation of each star's position over time, (t,x, in units of seconds and meters.
### Problem Description
Detormine apoastron of the system' erbit. You must provide your answer in units of m.
### Additional Instructions
To complete this task, you have access to the following teels and data: 1. DataFrame 'df' containing columns: time, starl_x, starl_y, starl_2, star2_x, star2_y, star2_z.
2. A code interpreter with 'df' pre-icaded that can execute Python code.
when using the code interpreter:
1. Always use print() to display results.
2. Do not use read_csv or attempt to lead the DataFrame, as it is already pre-loaded Inportant reminder: Repeated tool access is enabled until you have found the answer and have submitted it with the subwit_answerr tool.
Figure 7. Exact agent prompts used in the two evaluation regimes. The first box applies when the agent is limited to budget of 100 observations (budget-obs -00), the second the agent receives the complete table observations (full-obs) Two example problem descriptions are shown.
21


Reviews:
[{"review_id": "8WmVFVcV0q", "reviewer": "Reviewer_VSix", "summary": "The paper introduces a new benchmark called Gravity-Bench-v1 to test the discovery potential of LLMs. The which the benchmark different 2-body star systems are simulated. These simulations include out of distribution parts where the gravitationally force have a different proportion to r by adding an alpha to r^{2 + $\\alpha$}. The simulation outputs start positions over time and the agents have access to them by an observation tool and a python based shell script.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s detailed consideration. They raised a concern about whether our simulations overall, and the modified law of gravity in particular, are truly out-of-distribution (OOD) tests for the AI models with solutions that cannot be found online (thus incorporated into the pretraining corpus). We argue that they are indeed OOD, on the following basis:\n\n1) A major OOD aspect to our benchmark is the methodology itself (that mimics the scientific method) the AI models need to follow to answer our problems. They need to:  \n   a) Understand both the given question and data.   \n   b) Plan their “observations” within the constraints of the budget. This is a physical reasoning task on its own as for example an eccentric star will spend very little time near its periastron.   \n   c) Reason through a usually multi-step roadmap to the solution.  \n   d) Execute the roadmap using the available tools.   \n2) The universally poor performance across the AI models (including o1) we test clearly indicates that these models are genuinely struggling with these reasoning challenges. The scores obtained (under 20% for budget-obs-100) show that our benchmark is far from saturation.  \n3) Unlike other benchmarks that rely on or mimic questions largely obtained from sources available on the internet, we specifically designed our own simulations entirely from scratch. It is highly unlikely that similar problems are available in online or offline sources, ensuring memorization from any source remains very unlikely.\n\nThese points above emphasize that our modified gravity law problem is OOD. Since it is not only that an identical question is unlikely to be found in any other source, the methodology to solve it (all the steps of point 1\\) makes it truly unique.\n\nTo conclude, even aside from the scenarios which we label 'out-of-distribution' (those with modified gravity and drag forces), our benchmark substantially contributes by evaluating dynamic planning and iterative reasoning under limited observational budgets, critical capabilities for scientific discovery yet rarely tested by existing benchmarks. This value is independent of whether one fully accepts our label of 'out-of-distribution' for the two scenarios where we go beyond the standard gravitational force.", "claims_and_evidence": "The claim of the paper is to provide a benchmark to test the scientific reasoning potential of LLMs. By including out-of-distribution physics tasks it should be tested if the model is not just memorizing the solutions. However, is this really the case? And can one show this? Just thinking about an irrotational vector field in three-dimensional space. In general the inverse-square law corresponds to the property that the divergence is zero outside the source. So a general way of this law in an n-dim. Euclidean space would be that the intensity \"I\" of the vector field falls off with the distance \"r\" following the inverse (n − 1). Already here is a lot of literature about this. Furthermore, there are non-euclidean formulations even for Newtonian Cosmology (https://arxiv.org/pdf/2002.10155). These are just some examples which show that besides the simple 3-d inverse-square law a lot of other laws exist where the physics is different. It’s also easy to imagine that r^{2 + $\\alpha$} can be given as an exam question in an undergraduate physics course. Also there are ton of alternatives formulations to general relativity or Modified Newtonian dynamics (MOND). Taking this beside there is another aspect: The models are closed source and are trained on data we don’t know. So how can the authors be sure that there approach is not in the data? It’s impossible to show this so making such a statement that r^{2 + $\\alpha$} is out of distribution is therefore impossible to proof and then all evidence in terms of out-of-distribution is meaningless.", "methods_and_evaluation_criteria": "The authors used closed source LLMs to evaluate them on out-of-distribution physics tasks. Since we don’t know the data the LLMs were trained on, the methodology does not make sense.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "No really search for alternatives to Newtonian Cosmology.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "--", "other_comments_or_suggestions": "--", "questions_for_authors": "--", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X7dGU88jBB", "reviewer": "Reviewer_f7pg", "summary": "This paper proposes a new benchmark, “Gravity-Bench-v1”, for evaluating the scientific discovery capabilities of AI agents. This benchmark is based on gravitational physics (in particular, the two-body problem), and measures the ability of agents to discover hidden physical laws in a dynamic environment. The distinctive feature of this benchmark is that the agents are given a limited observation budget (up to 100 observations), and they must plan their observations efficiently within this constraint, collect and analyze data, and solve the problem. The benchmark includes not only scenarios that follow the laws of physics in the real world, but also out-of-distribution cases (such as modified gravitational laws), to evaluate the true scientific generalization capabilities of AI. The evaluation experiments show that the latest AI models (such as o1, Claude 3.5 Sonnet, and GPT-4o) show moderate performance (up to 64%) when they have full access to data, but when their observation budgets are limited, their performance drops significantly (up to 21.5%). This suggests that current AI models have limitations in long-term planning and information gathering strategies.\n\n## Response After Rebuttal\n\nAfter carefully considering the authors' rebuttal, I maintain my evaluation as \"Weak reject (2)\". Here's why:\n\nThe authors provided detailed explanations regarding the PhD-level solutions, clarifying that they were verified by astrophysics experts. This point is appreciated.\n\nHowever, significant limitations still remain with this benchmark. Most notably, the current version is restricted only to two-body gravitational problems. For a benchmark claiming to evaluate scientific discovery across broader domains, this scope is extremely limited. While the authors mention plans for future extensions (adding noise, 3D extension, application to other fields), the narrowness of the current benchmark restricts the contribution of this paper.\n\nAdditionally, there remains a lack of detailed analysis regarding the observation budget usage patterns. The authors state that systematic exploration is difficult due to the approximately 1.3 million tokens generated per run, but at least a detailed analysis of representative cases would have provided insights into the agents' decision-making processes.\n\nRegarding the OOD (out-of-distribution) claims, while I understand the authors' explanation, demonstrating generalizability to broader scientific discovery tasks would require evidence beyond a single physics domain.\n\nIndeed, the fact that even the latest AI models achieve only 40.2% under budget constraints is interesting, but while this demonstrates the difficulty of the tasks, it doesn't fully validate the value of the benchmark design itself.\n\nOverall, while the fundamental idea behind GravityBench has merit and shows promise, the current version is too narrow in scope to recommend for acceptance at ICML. A future version implementing the extensions mentioned by the authors would likely result in a more comprehensive and valuable benchmark.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough review and valuable suggestions. Below, we address the main points:\n\n**(Response to point 1)** We appreciate the feedback with respect to adding additional details regarding the human solutions. They were developed by one 2nd year PhD student, one professor, and one research scientist (PhD+10), all in astrophysics. Each one of them double checked and verified every solution. We also confirmed that the human-ref results matched Rebound’s built-in measures of the orbital parameters when available.\n\nThese solutions were not developed to directly compare human and AI scores, but rather were made to serve as an empirical solution baseline to understand the difficulty of each question, which was necessary to set the threshold values an AI must exceed in order for it to get marked as “correct”. We will explicitly detail this in the revised manuscript.\n\n(**Response to 2,4,5**) We entirely agree with the recommendations for future improvements with respect to adding noise, moving to 3D, and branching to new fields. Publishing GravityBench is intentionally our first demonstration of a simulation-based scientific reasoning benchmark. We already are working on extensions including other areas of physics, like electromagnetism, and realistic observational errors. Developing these environments allow us to evaluate, and even train, AI agents on research-like tasks. Though even without these complexities, current models struggle at GravityBench, which demonstrates the benchmark's immediate value.\n\n(**Response to 3**) We agree that a deeper analysis of why agents underutilize their observation budget would be valuable. Although we began a manual analysis of agent traces, each run generated around 1.3 million tokens, making systematic exploration impractical. Therefore, we expect further analysis will require careful automation which will shed light on failure modes. The manual inspection we present suggests that agents prematurely rush toward solutions, rather than strategically and iteratively using their observation budget. We emphasize that GravityBench is self-contained in the sense that the reason for agent failures reflect limitations in reasoning and planning abilities rather than issues with the benchmark itself.\n\n**(Response to other comments)** Although we initially could not run o1 under all settings, we have now evaluated o3-mini-high, the current best-performing model, which still only achieves 40.2% under budget-obs-100. This performance demonstrates that our benchmark remains substantially challenging, as o3-mini-high still struggles with most questions, including those involving modified gravity or drag forces. These new o3-mini-high results will be provided in our camera-ready version.\n\nWe appreciate the comments on systematically verifying whether the shortcut of just assuming stellar masses leads causally to incorrect answers. However, we already know that this is indeed the case. Any solution which assumes stellar masses of 1 kilogram or equal masses is physically incorrect, which directly causes these solutions to fail. This is a direct causal relationship between the assumption and the solution rather than a correlation.\n\nWe acknowledge the point regarding limited references to literature on symbolic regression and AI-driven autonomous scientific discovery systems, and will expand our discussion to contextualize GravityBench with respect to these approaches. Regarding the connection to the POMDP literature, we are presently not entirely certain how \"partial observability\" in our benchmark relates with the standard interpretation used in idealized POMDP frameworks, as many of our problems involve regression rather than decision problems. We will clarify this relationship in the future, particularly as we introduce observational noise and uncertainty into our benchmark.", "claims_and_evidence": "The paper's claims are generally supported by clear and convincing evidence. The main claims are the proposal of a new benchmark for evaluating AI agents' scientific discovery capabilities and the challenges that current AI models face with this benchmark. These claims are supported by comprehensive experimental results using multiple models (e.g., o1, Claude 3.5 Sonnet, GPT-4o).\n\nIn particular, the data showing the difference in performance with and without the observation budget constraint, the analysis of the agent's strategy in the observation plan, and the detailed examination of the failure mode are convincing. In addition, the evaluation in the out-of-distribution task also supports the claim that it measures the true scientific generalization ability of AI.\n\nHowever, there is a lack of detailed explanation of the “PhD-level solution method”, and there is limited transparency regarding how this was actually implemented and how its accuracy was verified. In addition, the fact that the evaluation of o1 was partially limited due to budget constraints and that a comparison was not conducted under completely identical conditions for all models somewhat weakens the quality of the evidence.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are generally appropriate for the problem of scientific discovery. The use of Rebound, a scientific-grade physics simulation tool, provides a precise and reliable physical environment. In addition, by setting constraints on the observation budget, the decision-making process is modeled to be similar to that of actual scientific research.\n\nThe fact that the evaluation criteria include task-specific tolerance thresholds is appropriate. In particular, the fact that the thresholds are adjusted according to the difficulty of each task (e.g. 5% for easy tasks, 70% for difficult tasks) and are determined based on the performance of PhD-level solutions is reasonable.\n\nOne area for improvement is the potential to introduce noise and error into the observations, which would mimic a more realistic scientific environment. Also, the current system only handles trajectories restricted to the (x,y) plane, but it would be possible to increase the complexity and realism of the benchmark by extending it to more general 3D trajectories.", "theoretical_claims": "This paper makes mainly experimental contributions, and contains few assertions accompanied by rigorous theoretical proofs. However, the physical foundations, such as the description of physical laws (e.g., the modified gravitational law FG ∝ r^(-2+α)) and the virial theorem (2K + U = 0), are accurately stated.\n\nThe theoretical aspects mentioned in the paper are based on established principles and laws of physics, and there is no question about their accuracy. However, the paper itself does not provide new proof of these theories, but rather applies existing physical theories to create a benchmark environment.", "experimental_designs_or_analyses": "The experimental design is robust and well-constructed with a clear purpose. A total of 206 task-simulation pairs were created, enabling comprehensive evaluation. The fact that each model was evaluated multiple times (3 times) under the same conditions also increases the reliability of the results.\n\nHowever, there are some areas of concern. The limited number of evaluations of o1 (not evaluated under budget-obs-100) due to budget constraints makes a full comparison of all models difficult. Also, in the analysis of the observation planning capability, there is a lack of detailed analysis of why the agents do not use a large portion of the observation budget available to them. This could be useful information for a deeper understanding of the agents' behavior.\n\nThe analysis of failure modes, in particular the analysis of the correlation between the shortcut that assumes the mass of the star and incorrect answers, is very valuable. However, further verification of whether this correlation is a causal relationship would make the analysis more convincing.", "relation_to_broader_scientific_literature": "This research is appropriately positioned in the context of existing efforts to utilize AI in scientific research. In particular, it discusses the relationship with various approaches, such as LLM (Galactica, etc.) that specializes in literature analysis, data-driven discovery, automated statistical modeling, and workflow automation frameworks.\n\nThe Gravity-Bench is differentiated from existing approaches in that it views discovery as a dynamic and iterative process, and it emphasizes the interaction between exploration and inference in partially observable environments. In addition, while existing benchmarks focus on rediscovering known phenomena and textbook-style problem solving, the Gravity-Bench is differentiated by including a variety of dynamic scenarios that reflect the unpredictability of real-world discovery processes.\n\nWhat is particularly valuable is that this research recognizes the lack of AI benchmarks for scientific discovery and presents specific efforts to fill that gap.", "essential_references_not_discussed": "The paper covers a wide range of related research, but there is a lack of reference to some related literature.\n\nIn particular, a more detailed comparison of specific previous research on AI systems that focus on the discovery of physical phenomena may be beneficial. For example, there is a lack of comparison with AI that performs symbolic regression of physical laws (e.g. AI Feynman, BMS) and autonomous scientific discovery systems that combine experimental design and hypothesis generation.\n\nThere are also references to the literature on partially observable Markov decision processes (POMDPs), but there is limited specific comparison with previous research on the application of POMDPs in the context of scientific discovery. A detailed discussion of the relevance of previous research in this field would make the novelty and positioning of the proposed benchmark clearer.", "other_strengths_and_weaknesses": "The main strength of this paper is that it provides a concrete and rigorous benchmark for evaluating the complex process of scientific discovery. In particular, the design, which requires dynamic problem solving that combines observation planning and analysis, enables a realistic evaluation of the scientific capabilities of AI. In addition, the inclusion of out-of-distribution scenarios is also commendable, as it measures the true scientific reasoning capabilities of AI rather than an approach based on memorization.\n\nOne weakness is that the current version is limited to the twin-gravity problem, and it is unclear whether it can be generalized to a wider range of scientific discovery scenarios. In addition, the idealized observation conditions (lack of observation error and noise) do not fully reflect the complexity of real scientific observations.\n\nIn terms of originality, the combination of environmental benchmarking and scientific discovery evaluation is creative. In particular, the integration of scientific simulation and the concept of partial observability to create a new evaluation framework for scientific exploration is innovative.", "other_comments_or_suggestions": "Below are some suggestions for improving the paper:\n\n1. It would be good to add a detailed explanation of the “PhD-level solution” and increase the transparency of the implementation method and verification process.\n\n2. Introducing noise and measurement errors into the observations would allow for a more realistic imitation of a scientific discovery scenario.\n\n3. A detailed analysis of why the agents are not fully utilizing their observation budgets would be beneficial. This could lead to a better understanding of the agent's decision-making process and indicate directions for improvement.\n\n4. Currently, only trajectories limited to the (x, y) plane are handled, but in the future, it is hoped that the system will be extended to more general 3D trajectories.\n\n5. By providing more specific prospects for the possibility of extending the benchmark to a wider variety of scientific fields and phenomena, the future direction of the research will become clearer.", "questions_for_authors": "I would expect an answer to the concerns raised above, but have no additional questions otherwise.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "ctsWxc7uDZ", "reviewer": "Reviewer_XGag", "summary": "This paper introduces Gravity-Bench-v1, a novel benchmark for evaluating LLM agents on physics discovery tasks. The authors design an environment that simulates gravitational dynamics with high precision, where agents must strategically plan observations, analyze data, and reason to solve various tasks. The benchmark includes standard Newtonian physics as well as out-of-distribution scenarios (e.g., modified gravity laws) to test generalization capabilities beyond memorized knowledge in LLM agents. The work provides a rigorous framework for assessing scientific discovery capabilities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(Response to 1\\)** See our response to “points 2,4,5” of Reviewer f7pg.\n\n**(Response to 2\\)** See our response to “point 1” of Reviewer f7pg.\n\n(**Response to 3\\)** Regarding the prompting process and the manner in which the agents were informed of the budget, we adopted a minimalistic approach: the prompt merely described the observation tool, specified the budget constraints (a maximum of 100 observations, with up to 10 per request), and deliberately avoided suggesting any specific strategy. In the absence of explicit strategic guidance, it becomes increasingly evident that these AI models demonstrate an inherent tendency to prematurely converge on solutions while failing to make full use of the allocated observational budget. This behavior shows that the models often fail to approach the problem with the level of care expected in scientific problem-solving.\n\nThis approach was necessary for gathering baseline results, from which we can investigate if this behavior can be mitigated via prompting or other means. The full prompt is below and will be included in the Appendix of our camera-ready version.\n\n**(Response to 4\\)** We thank the reviewer for raising the point about iteratively re-calling the agent and how that would improve budget use and performance. In our current setup, agents decide when to stop and almost always do so before exhausting their observation budget. We intentionally did not force agents to use their full budget, as our goal was to assess whether they can autonomously reason about when additional data is required. We acknowledge the value of experimenting with force, i.e. increasing the number of iterations.\n\n(**Response to 5\\)** We thank the reviewer for pointing this out, we agree that a deeper analysis of how different models approach OOD scenarios is important. Due to the large size of the outputs (over 1.2 million tokens per run), we were unable to perform detailed manual inspection across all cases. However, we are currently developing automated tools to analyze these responses systematically.\n\n**\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\***  \nPrompt (Response to 3):  \nYou are tasked with solving the following physics problem related to a binary star system. You are provided observations of each star's position over time, (t,x,y,z), in units of seconds and meters.\n\n\\#\\#\\# Problem Description  \nDetermine the total energy (K \\+ U) for the system in joules.  \nYou must provide your answer in units of J.\n\n\\#\\#\\# Additional Instructions  \nTo complete this task, you have access to the following tools and data:  \n1\\. An observational tool called \\`Observe\\` that allows you observe the system at  \nspecific times of your choosing.  \n2\\. A code interpreter that can execute Python code.\n\nWhen using \\`Observe\\`:  \n1\\. The \\`times\\_requested\\` parameter should be a list that can contain any values in the time window \\[0.0, 7.21e+09\\] seconds. You cannot request negative times. The upper limit for the time window was chosen to guarantee that the problem is solvable with an appropriate sampling of observations using the total observational budget.  \n2\\. You can observe the system at any time within the time window, even if it is in the past compared to the last observation.  \n3\\. You can observe the system up to a total of 100 times and you can observe up to 10 times per observational request which is the maximum length of the \\`times\\_requested\\` list.  \n4\\. After each observation, the dataframe \\`row\\_wise\\_results.df\\` will be updated. It contains columns: time, star1\\_x, star1\\_y, star1\\_z, star2\\_x, star2\\_y, star2\\_z. You can access it using the code interpreter tool. For example, to access the first five rows, print(row\\_wise\\_results.df.head(n=5))  \n**\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\*\\***\nAnswers to direct questions:  \nQ1) Partial observations (budget-obs-100) require multiple sequential tool calls, as agents iteratively request and analyze observations, compared to full-obs where all observations are available immediately in a single step.\n\nQ2) We tested each model three times. However, we acknowledge additional runs would be valuable.  \n      \nQ3) We have not yet specifically analyzed whether the frequency of shortcuts differs between OOD cases and standard problems.", "claims_and_evidence": "- The paper's key research motivation that current benchmarks inadequately evaluate scientific discovery capabilities in llm agents is reasonable. I suggest authors to also provide compelling evidence of the limitations in existing benchmarks, showing the memorization vs discovery gap in previous works highlighted in sec 2.\n\n- The claim that current agents struggle with observation planning in a budget-constrained setting is convincingly demonstrated through the significant performance gap between full-observation and budget-constrained scenarios (Table 1). \n\n- I think the claim regarding PhD-level solutions as \"human reference\" requires more clarification. The paper lacks detailed information about how these solutions were developed, how many human experts were involved, and what their backgrounds were. This affects the validity of using these solutions as a reference point for evaluation.", "methods_and_evaluation_criteria": "The metric of setting task-specific error thresholds based on the performance gap between full-observation and partial-observation PhD-level solutions is reasonable and well-justified.\n\nHowever, the paper should provide more details about how agents are made aware of these budget constraints and what prompting strategies were used to encourage strategic observation planning. Also, it's worth noting the benchmark is still limited to two-body problems.", "theoretical_claims": "There are no formal theoretical proofs in the paper to verify.", "experimental_designs_or_analyses": "The experimental design is generally sound. However, some aspects of the experimental analysis can be improved:\n\n- The paper doesn't clearly explain why agents consistently use fewer observations than their budget allows (Figure 3). Is this a function of the prompting or an inherent limitation of the agents? How did authors explore this? \n\n- For error analysis, it would be valuable to also see how errors and number of observations selected evolve over successive steps/iterations in agents. \n\n- The findings about GPT-4o mini solving OOD tasks that GPT-4o fails on are intriguing but presented anecdotally rather than systematically analyzed. A more comprehensive analysis focused on these problems could be helpful to better understand the memorization vs discovery capabilities in different models.", "relation_to_broader_scientific_literature": "The paper effectively positions itself within the landscape of benchmarks for scientific tasks and data-driven discovery. \n\nThe authors appropriately distinguish their approach and difference from existing work by highlighting the dynamic, partially observable environment and the inclusion of OOD cases to test generalization rather than memorization.", "essential_references_not_discussed": "The issue of LLM memorization and limitation of current benchmarks for general data-driven reasoning as well as scientific discovery tasks is also discussed in some other works such as [1-3]. I would suggest authors to also consider citing these works in their related work section. \n\n[1] Shojaee et al., LLM-SR: Scientific Equation Discovery via Programming with Large Language Models, 2024\n\n[2] Cai et al., SciAssess: Benchmarking LLM Proficiency in Scientific Literature Analysis, 2024\n\n[2] Xie et al., On Memorization of Large Language Models in Logical Reasoning, 2024", "other_strengths_and_weaknesses": "**Strengths:**\n\n- The benchmark addresses a gap in evaluating scientific discovery capabilities beyond memorization.\n\n- The use of high-fidelity physics simulations provides a strong environment for testing science-focused agents.\n\n- The inclusion of OOD cases with modified laws and simulators is particularly valuable for assessing generalization in science.\n\n**Weaknesses:**\n\n- The scope is currently limited to two-body gravitational problems in physics, which represents a narrow slice of scientific discovery scenarios.\n\n- Insufficient details regarding the human reference evaluation settings, including the number of human experts involved and their evaluation protocol.\n\n- Lack of details regarding the prompts used for different experimental cases, particularly how agents make decisions about budget allocation.\n\n- The temporal dimension of budget utilization and error rate progression is missing from the analysis (improvement over steps, iterations in ReAct style agentic setting). Does increasing the number of iterations change the underutilization pattern observed?\n\n- There's limited analysis of how different models approach OOD scenarios compared to other problems in current reported results.", "other_comments_or_suggestions": "Minor Comment: I suggest adding annotation (a), (b), etc. for subplots in figures (e.g., in Figure 2 or 3) to ease referencing.", "questions_for_authors": "- In Table 1, why do the same models show lower computation time on full observations compared to partial observations?\n\n- Regarding the OOD results where GPT-4o-mini solved a modified gravity task while GPT-4o did not: Have you verified this finding across multiple runs to ensure it's not anomalous? A more comprehensive analysis could provide insights about the relation between model size and discovery.\n\n- In Section 4.6, you note that agents tend to take shortcuts rather than pursue systematic derivations. Have you analyzed whether the occurrence of these shortcuts differs between OOD cases and standard problems? Are shortcuts more or less common in novel problems?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a benchmark to test the scientific discovery abilities of LLMs by allowing them to make a limited number of observations in order to accurately predict the behavior of a 2 body star system simulation. The benchmark includes variations with modified laws of physics with the aim of being more out of distribution with respect to LLM pre-training data. The empirical results show that current SOTA LLMs struggle to plan to gather information effectively.

While the benchmark is limited to 2-body simulations, the current empirical evaluations demonstrate a limitation of the current models, and it is not in-general unreasonable that broader capabilities are first measured on a restricted domain. However, the authors should make sure that it is clear from the framing of the paper that this currently aims to measure this restricted subset of the general capability. Moreover, the authors should be sure the justification of all claims are made clear, such as those about "PhD-level" solutions.
Decision: Accept (poster)