LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Authors: Parshin Shojaee
Ngoc-Hieu Nguyen
Kazem Meidani
Amir Barati Farimani
Khoa D Doan
Chandan K. Reddy
Venue: ICML 2025 oral (2025)
Keywords: Benchmark
Scientific Discovery
Large Language Models
Symbolic Regression
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: We present LLM-SRBench, the first comprehensive benchmark for evaluating scientific equation discovery with LLMs, designed to rigorously assess discovery capabilities beyond memorization

Abstract: Scientific equation discovery is a fundamental task in the history of scientific progress, enabling the derivation of laws governing natural phenomena. Recently, Large Language Models (LLMs) have gained interest for this task due to their potential to leverage embedded scientific knowledge for hypothesis generation. However, evaluating the true discovery capabilities of these methods remains challenging, as existing benchmarks often rely on common equations that are susceptible to memorization by LLMs, leading to inflated performance metrics that do not reflect actual discovery. In this paper, we introduce LLM-SRBench, a comprehensive benchmark with 239 challenging problems across four scientific domains specifically designed to evaluate LLM-based scientific equation discovery methods while preventing trivial memorization. Our benchmark comprises two main categories: LSR-Transform, which transforms common physical models into less common mathematical representations to test reasoning beyond memorization, and LSR-Synth, which introduces synthetic, discovery-driven problems requiring data-driven reasoning. Through extensive evaluation of several state-of-the-art methods on LLM-SRBench, using both open and closed LLMs, we find that the best-performing system so far achieves only 31.5% symbolic accuracy.
These findings highlight the challenges of scientific equation discovery, positioning LLM-SRBench as a valuable resource for future research.

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Parshin Shojaee"
Ngoc-Hieu Nguyen*
Kazem Meidani 34 Amir Barati Farimani
Khoa D Doan
Chandan K Reddy'
Website:
com/ ceom/deeg deep- -ttpa//oooooooooooooooooooo-ot-e
Abstract
0.961
1.0
Scientific equation discovery has long been a cornerstone of scientific progress, enabling the
Numeric Error (NMSE)
Final Symbolic Error
0.741
derivation of laws governing natural phenomena. Recently, Large Language Models (LLMs) have gained interest for this task due to their poten- tial to leverage embedded scientific knowledge for hypothesis generation. However, diffi- cult to assess the true discovery capabilities of these methods because existing benchmarks of- ten use well-known equations. This makes them vulnerable memorization by LLMs and results in inflated performance metrics that do not re- flect genuine discovery. In this paper, we in- troduce LLM-SRBench, a comprehensive bench- mark with 239 challenging problems across four scientific domains specifically designed eval- uate LLM-based scientific equation discovery methods while preventing trivial memorization. Our benchmark comprises two main categories: ASR-Transform. which transforms common phys- ical models into less common mathematical repre- sentations to test reasoning beyond memorized forms, and LSR-Synth, which introduces syn- thetic, discovery-driven problems requiring data- driven reasoning. Through extensive evaluation of several state-of-the-art methods, using both open and closed LLMs, we find that the best- performing system so far achieves only 31.5% symbolic accuracy. These findings highlight the challenges of scientific equation discovery, posi- tioning L.LM-SRBend future research.
10
Feynman Benchmark LLM-SRBench (LSA-Tansform)
0.5
(LSR Syrth)
171
10-20
12
20
50
100
500
1000
of Search Candidates
Figure 1. Error analysis comparing simple LLM sampling (Llama- 3.1-8B) on 100 Feynman problems versus LLM-SRBench datasets (LSR-Transform and LSR-Synth). The sharp drops in numeric error curves and considerably lower symbolic error for Feynman problems suggest memorization rather than gradual discovery.
1. Introduction
Equation discovery, the process of uncovering symbolic mathematical expressions from observational data, has been cornersonn of scientific advancement. This task, also known as symbolic regression (SR), goes beyond mere data- driven predictive modeling by seeking interpretable math- ematical relations that reveal the underlying mechanisms of natural phenomena. When scientists derive mathemati- cal equations from empirical data, they gain more than just predictive power they obtain insights into fundamental physical principles, enable extrapolation beyond observed data, and facilitate knowledge transfer across scientific do- mains (Langley, 1981; Schmidt Lipson, 2009).
valuable resource for
Standard approaches to equation discovery have primar- ily relied on genetic programming (GP) and evolutionary algorithms (Cranmer, 2023; La Cava et al., 2021), which represent mathematical expressions trees and navigate the vast space of possible equations through evolutionary search techniques. However, these methods face two fundamental challenges. First, the NP-hard nature of equation discov- ery (Virgolin Pissis, 2022) makes their random mutation and crossover operations computationally prohibitive across
Equual contribution 'Vrgginia Tech VVinUniversity 'Carnegie Mellon University *Capital One. Correspondence to: Parshin Shojace <parshinshojaec@@ttee vt..iinsssojeeee...u........ vt.edu>..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LLM-SRBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
Goal Instruction
Typical Workflow
fect
Hypothesis
Discover the ma ml equationiaw that describes [output variable] based on given linput features).
Discovered mathemalical equation represented expressions, programs, etc.
LLM internal scientific knowledge
Use domain-spectt knowledge of (the scientific field] and provided data samples to find an equation thats scientfically valld and fits the data well
R Reasoning and planning
5 upporting explanatoos ceciaatooss/eaeeeeeeeeeee /reasoning
Programming
(owege*12 onega (nn2))
4 </((=*2
Scientific Context
Hypothesis Generation
eet cssatiant orogh OESSL poromals Energr-aass ratto nonasttedd by pormeeter
Problem description
Variable names and descriptions
combined frequency Lnptituae effacts deee unetat oergume venr cenguueee aaw2 ranerator devetiiator
Example:
Prompt Imput/eedbacc
expetion
musustis that deurides the aas (ir) seold to sustin leest veriatees
ittatiea stared fools
and asputhvoe (i).
Evaluation
Evaluation
Data Fidelity:
Data
Dama accuracy
Parameter ptmizaation
Outao--Dmain generalizaton
Eg
Symbolic Accuracy: Human expert/ evaluator Scientiic plausibilty interpretabiitty
47
12
23
15
12
iullsiio
Experiments
Statistical
28 36
Computational E Efficiency
Figure 2. Overview of the LLM-based Scientific Equation Discovery. The benchmark tasks (left) combine scientific context with numerical data. The discovery process (middle) iteratively leverages LLM's scientific knowledge and data-driven reasoning to generate hypotheses for underlying equations. Discovered hypotheses, represented as equation strings, trees, programs, then evaluated (right) using multiple metrics including data fidelity, symbolic accuracy, and computational efficiency.
vast search spaces. Second, unlike human scientists who leverage their domain knowledge and expertise to guide hypothesis formation, these approaches are mostly purely data-driven, and isolated from existing scientific knowledge. These limitations have motivated researchers to develop methods that incorporate scientific domain knowledge into the equation discovery process.
& Tegmark, 2020), and Strogatz dynamical systems (La Cava et al., 2016; Strogatz, 2018). A notable extension to this framework is SRSD (Matsubara et al., 2022), which en- hances the Feynman benchmark by incorporating physically meaningful sampling ranges for data points. However, these benchmarks exhibit significant limitations for the evaluation of LLM-based methods. Their problems mostly based on known physics equations from textbooks, which makes them often subject to memorization by LLMs.
Large Language Models (LLMs) have recently emerged as a promising solution to these challenges, offering a new paradigm for scientific equation discovery. LLMs, trained on vast corpora of scientific literature, possess extensive embedded scientific knowledge. This has sparked signif- icant interest in leveraging LLMs for scientific equation discovery, with several recent works demonstrating their po- tential (Shojaee et al., 2024b; Ma et al., 2024; Grayeli et al., 2024; Merler et al., 2024; Du et al., 2024; Reddy & Shojace, 2024; Zhang et al., 2024). These LLMM-baeed approaches have shown to enhance the equation hypothesis generation process by incorporating scientific priors, guiding the ex- ploration of equation search spaces more efficiently, and providing interpretable reasoning for the search process.
As noted by (Shojace et al., 2024b), LLMs frequently suc- ceed on these common equation discovery benchmarks through simple recitation based on variable names and prob- lem descriptions, rather than the actual process of data- driven discovery and reasoning. Our analysis (shown in Fig. 1) also confirms this finding the sudden drop in the numeric error curve within the first few iterations and signif- icantly lower symbolic error on Feynman problems indicate memorized solutions rather than a meaningful search to- wards discovery. To mitigate this issue, (Shojace al., 2024b; Ma al., 2024) have introduced handful of five custom-crafted problems designed to prevent memorization by manually modifying known physical models. While these efforts represent step forward, the small scale and limited diversity of these problem sets are insufficient to provide a comprehensive evaluation framework for emerg- ing LLM-based methods in scientific equation discovery. A more robust and systematic benchmark is needed to en- able standardized evaluation and foster the development of innovative methods in this emerging field.
Despite the promising potential of LLM-based equation dis- covery methods, their rigorous and robust evaluation still remains an open challenge. The current scientific equa- tion discovery benchmarks are primarily represented by SRBench (La Cava et al., 2021) and SRSD (Matsubara al., 2022). SRBench incorporates two key data groups for this purpose: the Feynman physics equations (Udrescu
2

---

LLM-SRBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
In this paper, we introduce LLM-SRBench, a new benchmark designed rigorously evaluate the capabili- ties of LLM-based scientific equation discovery methods. LLM-SRBench addresses the limitations of existing bench- marks by constructing problem sets that avoid trivial recita- tion while leveraging the scientific priors embedded in LLMs, simulating conditions akin to scientific discovery. The benchmark is structured around two main categories of problems, each targeting distinct aspects of equation discovery. The first category focuses on transforming com- mon scientific problems, such as those from the Feynman equations, into different mathematical representations of the same underlying physical problem. By symbolically altering input-output mappings and generating less common mathematical forms for the same problem, we challenge LLM-based equation discovery to go beyond memorization of the common forms. This approach is motivated by recent findings on the fragility of LLMS'' reasoning capabilities to unfamiliar representations of otherwise familiar problems (Mirzadeh et al., 2024; Xie et al., 2024; Wu et al., 2023). The second category extends the approach introduced by (Shojaee et al., 2024b), which combines known terms in the underlying equation with synthetic, novel terms to create problems that go beyond memorization and demand data- driven reasoning. We expand this idea into a comprehen- sive set of benchmark problems spanning diverse scientific domains. These problems incorporate carefully designed synthetic terms that are both novel and plausible. We fur- ther verify the solvability of the generated equations using numerical solvers, ensuring that the benchmark problems remain grounded in physical feasibility while presenting meaningful challenges for LLM- discovery methods.
We propose a novel benchmark design through alternative mathematical representations (.SR-Transform) and syn- thetic, discovery-driven problems (LSR-Synth) to ensure rigorous evaluation of scientific reasoning and discovery capabilities beyond L.M memorization.
Extensive experiments on state-of-the-ait methods reveal performance peaks at 31% highlighting the benchmark's challenging nature and its potential for future research.
2. LLM-SRBench
We introduce LLM- ALM-SRBench, a novel benchmark designed to evaluate LLM- -bd-dddddddd methods for data oata-driven scientific equation discovery. As shown in Fig. 2, in this benchmark, a "data-driven scientific equation discovery" task is defined as follows: Given a task dataset D, the corresponding scientific context C, the objective is to derive a hypothesis that repre- sents the underlying mathematical relations behind the data with high precision and scientific plausibility. This process resembles the iterative search and refinement undertaken by human scientists, where LLMs act as optimizers, proposing and refining hypotheses based on both scientific knowledge and empirical data.
2. LSR--rrasfors
This category is designed to evaluate whether LLM-based methods can discover equations in less common mathemati- cal forms, avoiding reliance on memorization of well-known representations. This approach is motivated by the observa- tion that LLMs often struggle with unfamiliar instantiations of otherwise familiar problems, as highlighted by recent studies on the fragility of LLM reasoning (Mirzadeh et al., 2024; Xie et al., 2024; Wu et al., 2023). By transforming existing benchmark problems into different mathematical representations, we challenge LLMs' capabilities in data- driven scientific equation discovery and reasoning.
LLM-SRBench comprises 111 problems in the first cate- gory L.SR-Transfone and 128 problems in the second category (LSR-Synth), spanning four scientific domains: chemistry (36), biology (24). physics (43), and material science (25). We comprehensively benchmark state-of-the- art L.LM-based scientific equation discovery methods with several LLM backbones on these datasets. Our experiments reveal several key insights into the capabilities and limi- tations of current based scientific equation discoy- ery methods. Results show that the best model can only solve 31.5% of problems on -ransforoorrm and 28.1% on LSR-Synth. This underscores the challenging nature of the tasks in LLM-SRBench and highlights its potential as a crit- ical evaluation foundation for future --LM-based scientific equation discovery methods. Overall, the contributions of this work are as follows:
We build on the Feynman (Udrescu & Tegmark, 2020) benchmark (current standard benchmark in scientific equa- tion discovery), which consists of 100 physies equations, and systematically transform these equations into alternative mathematical forms (examples in App. A. As demon- strated in Fig. 3(a), the transformation process involves seven key steps: 1) Equation Collection: We gather the original mathematical expressions, along with their input and output variables, and scientific problem descriptions from the Feynman benchmark. 2) Select Pivot Variable: For each equation, we choose an input feature to become the new target variable. 3) Prature-Target Transforma- tion: We transform the dataset by switching the roles of the selected input feature and the original target variable. 4) Symbolic Transformation: Using the SymPy library in Python on the parsed expressions, we solve each equa- tion with respect to the selected input variable, treating it
We introduce LLM-SRBench, the first comprehensive benchmark with 239 challenging problems across vari- ous scientific domains, designed to evaluate LLM- scientific equation discovery methods.
3

---

LLM--RBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
Chemisty Physis Bislogy Naterial Science
Equation Collection
Methematical Expressions ions Data Jnnutouutpttarriie Scientifi Descriptions
Feynman Benchmark
Select Scientific Problem
Ex: R Reaction Kinetics
Synthetic Benchmark
feature to become F new target varibll
Choose an input
(ux wi)
Select Plvot Variable
Given problem description, generate common and well-nown terms
kAC))epC R29
Known
Generation
2
3
Transformation
Feature-Target
Dx
xx
Xa
ly
Synthetic Term Generation
4 sin (s)dt))
Given problem descriptio, gonerate synthehic novel,
feasible terms $
W rito the symbolic transfomedd equalion for the variablee using SymPy
Symbolic Transformation
-kA (4(t)xxp(-ka) (aa(())))))
Solvability Check
( wr))
Check the equation solvable and rumerica ly stable
Solvability Check
Only retain symboli: forsformations that are analytically solvable
5
Novelty
Check k
thu eauRtian SEALL
Evaluate novetty of the equation using LLN
Transform the orig nal data samples for new equation Remove invalid samples under the equalion form
Dataset Refinement
samples for approved equetions NNmerrcaal solution differential equations
Datapoint Generation
Miea equetion is the plais clasytoo anchariic test dusorther max (es) cunaed eerticating ieactival leaat wariekted choen cennay (wi), and oap cheir (e).
Problem Reforrmuation
Expert Validation
Ittand express ions and their correspondig data samples are cmsss -hhckhed subject matter experts validale their pleusibility
New specificatio for the transformed problem using LLM
LSR-Synth
Figure 3. Data generation pipelines for the two dataset categories in LLM-SRBench. (a) ..SR-Tranfform converts Feynman problems into alternative mathematical forms through symbolic transformation and input-output role switching, and (b) LSR-Synth generates novel discovery-driven problems by combining known scientific terms in the underlying models with synthetic novel terms. Both pipelines include validation steps to ensure solvability and scientific plausibility.
as the new output and the original output variable input in the transformed equation. 5) Solvability Check: We retain only those transformations that are analytically solvable, ensuring the feasibility of the resulting equations. 6) Dataset Refinement: For the transformed equations with altered data domains (e.g., due square roots or denom- inators), filter the original Feynman dataset to ensure all data points fall within the valid domains of the new equations. 7) Problem Reformulation: Using LLM (GPT- 4o), we generate a new natural language specification for each transformed problem. During this data generation pro- cess, we constrain the transformed equations' complexity (measured by expression tree node count) to the range of original Feynman dataset distribution (full analysis in Fig. 8, App.A.1). This allows us focus on the semantic aspects of sercovery--secifficall the interplay between reasoning and memorization of the mathematical forms-rathe than conflating performance with the ability to handle syntac- tically complex and lengthy hypotheses. We also exclude transformed problems that L.M can solve through direct sampling without requiring access data.
equations. To support scientific knowledge-guide discov- ery, each task in LSR- -SR-Transform is supplemented with natural language description of the scientific problem and dataset, including variable names and their meanings. These descriptions are absent in the original Feynman benchmark but they are needed for LLMM-bsed scientific equation dis- covery methods to provide scientific context in prompts for knowledge-guide guided equation discovery by LLMs.
2.2. LSR-Synth
This category is designed to assess whether LLMs can dis- cover equations that incorporate new synthetic terms along- side known terms, requiring scientific as well as data-driven reasoning rather than reliance on memorization. The LSR- Synth dataset is motivated by the approach introduced in (Shojace et al., 2024b) for the handful of manually designed problems and systematically expands into a comprehen- sive set of benchmark problems across diverse scientific domains. By combining known terms with synthetic, novel terms, LLMs are challenged to demonstrate discovery capa- bilities in unobserved contexts, yet leverage their knowledge in the process. The LSR-Synth dataset spans four scientific domains: chemistry, biology, physics, and material science, focusing on key scientific problems, including reaction ki- neties in chemistry, population growth in biology, damped harmonic oscillators in physics, and stress-strain relation- ships in material science (examples in App. A.2).
This process yields 111 total transformed equations derived from the 100 original Feynman problems. Each transformed equation shares the same scientific context, problem descrip- tion, and variables as its original counterpart but presents a less common mathematical form to be discovered. The goal of Transfosformm is not to discover new equations but to evaluate whether LLM-based systems can validate discover- ies from non- trivial, data-driven transformations of known
The data generation process for LSR-Synth involves multi-
4

---

LLM-SRBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
ple steps, illustrated in Fig. 3(b), to ensure the creation of high-quality, challenging benchmark problems: 1) Select Scientific Problem: We select problems from different sci- entific domains, such as reaction kineties in chemistry or population dynamics in biology. 2) Known Term Gener- ation: Given the problem description, we prompt an LLM (GPT-40) to generate a list of common and well-known mathematical terms that typically appear in the underlying models. 3) Synthetic Term Generation: Similarly, we prompt the LLM to generate a list of diverse novel syn- thetic terms for given scientific problem, along with de- scriptions of the problem and variables. For example, in chemistry reaction kinetics, known terms for reaction rate (dA/d() based on concentration (A) and time (1) might in- clude first-order (-kA) and second-order kinetics (-kA3) or the exponential decay term -k exp (-kat), while syn- thetic terms could represent non-linear high-order saturation, e.g., h=12/(1 + AA or non-linear quantum tunneling ef- fects, e.g., KA exp (-2)/t². 4) Solvability Check: After sampling from the generated known and synthetic terms and combining them into a complete mathematical expres- sion, we verify the solvability of these expressions using numerical solvers such as solve_ive in Python. This step ensures that the expressions are feasible, providing basis for generating datapoints. 5) Novelty Check: In the context of each scientific problem and the complete expression, we evaluate the novelty of the new generated task using LLM (GPT-4o) as a novelty evaluator. This step to verify that the synthetic terms are novel in the pro- vided context and require data-driven reasoning rather than relying on established knowledge to be discovered. 6) Data- point Generation: For expressions that pass the solvability and novelty checks, we generate datapoints using numer- ical solvers based on the specified initial conditions and parameters. These datapoints are used to create the final task datasets. 7) Expert Validation: Finally, the filtered expressions, along with visualizations of their generated datapoints, are cross-checked by two subject matter experts to validate their plausibility. After these filtering steps, we finalize a candidate list of 128 problems across the four domains (36: chemistry; 24: biology; 43: physics; and 25; material seience). More detailed analysis of LLM-SRBench datasets are provided in App. A.
perspectives are critical, as equations may exhibit similar symbolic forms but differ numerically, vice versa.
Data Fidelity. We evaluate data-driven fidelity using two known metrics in equation discovery: (1) Acccuracy to tolerance т (Acc,) (Kamienny et al., 2022; Biggio et al., 2021), and Normalized Mean Squared Error (NMSE). These metrics are computed both in-domain test data and OOD data (when available) to assess generalization capacity, a crucial requirement for scientific equations.
Ui - y1
Acc, =1
max 13i5Noom
S T
Yi
1= (Ue y/6
Nosst
NMSE
-i=1
(y/
Symbolic Accuracy. We evaluate symbolic accuracy with model-based evaluation strategy using GPT-4o as eval- uator (prompt in App. B, Fig. 11). This approach addresses the limitations of current symbolic metrics like recovery rate in symbolic regression (La Cava et al., 2016), which are very sensitive to exact symbolic matches and fail to ac- count for mathematical equivalence, particularly in different hypothesis representations (e.g.. equation as strings, expres- sion trees, or Python programs). Here, GPT-4o evaluates mathematical equivalence by comparing the symbolic form of the predicted hypothesis versus the ground-truth equa- tion after removing parameters and constants. The ability of I LLMS to recognize semantic equivalence across differ- ent representations makes them particularly well-suited for evaluating LLM--baeed equation discovery methods, which often operate within a more diverse and open-ended hy- pothesis space. To validate this metric, two authors also independently evaluated symbolic equivalence on 130 sam- pled problems, finding 94.6% agreement between GPT-4o and human evaluators. App. B provides more details on the evaluation metrics.
3. Experiments
3.1. Experimental Setup
We benchmark state-of-the-art LLM-based scientific equation discovery methods using three LLM backbones: one open-source model (L1ama-3.1- 1..1--8-Inntrun and two proprietary models (GPT-4G-mini and GPT-3. 5 S-rurbo), Each discovery task takes as input the problem description, variables, the corresponding dataset, and an instruction specifying the task. The discovery methods then generate and refine equation hypotheses through LLMs. To ensure fair comparison, we standardize each of the methods to use 1k LLM calls per problem while maintaining their core algorithmic designs and hyperparameter settings. Detailed implementation specifics and prompts of each method are provided in App. C. We
2.3. Evaluation
Evaluating LLM--ased scientific equation discovery meth- ods introduces unique challenges due to the open-ended nature of the task and diverse symbolic representation of hy- potheses. A discovered equation can be assessed from two perspectives: (a) data fidelity, which measures how well the equation fits the observed and out-of-domain (OOD) data, and (b) symbolic accuracy, which evaluates the align- ment with ground-truth symbolic equation hypotheses. Both
5

---

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Table Comparison of different M---aaeed scientific equation discovery methods I.M.-SReench. Performance metrics include symbolic accuracy (SA), numeric precision (Acca.1) 1). and normalized mean squared error (NMSE). Bold values indicate best performance within each method, and underlined values show best overall performance across discovery methods.
LSR-Syuth
I TaR-Trasform
Chemistr
Biology
Physies
Material Science
SA 18ot
(
NMSEL
SA (R))
Ane
NMSEL
SA (8)t
(S0f
NMSR (R))
A00 NMSEL
Aoou(Rjt NMSE]
Direct Prompting (NottBiind)
Lluma-3. 3.1-8 AB--nnttacc
3.61
1.801
0.3697
0.0
0.0
0.0644
0.5481
0.0
0.0459
0.0
0.0
0.0826
GPT-3.5.ourb
2.10
1.801
0.3553
0.0
8.33
0.0023
0.0
4.16
0.5990
0.0
2.27
0.0274
0.0
0.0
0.0277
GPT-4o
7,21
6,306
0.2631
0.0
13.88
0.0221
4.16
0.4648
4,54
9.09
0.0647
0.0
0.0
0.0484
SGA
LIama-3JJ 3.1-8B Instract
2.70
0.909
0.3519
0.0
8.33
0.0458
0.0
0.2416
2.27
0.1549
0.0
12.12
0.0435
Gur--.5.turbo
0.0
0.909
0.3465
0.0
8.33
0.0071
0.0
8.33
0.1279
2.27
4.54
0.0249
0.0
28.10
0.0019
GPT-4o-min
9,91
8.11
0.2321
0.0
16.66
5.46e-4
4.16
12.51
0.0128
4.54
9.09
0,0511
0.0
36.11
6.02c-4
LaSR (Grayeli al., 2024)
Llama-3.1 --B--nstrcct
5.41
45.94
0.0021
0.0
27.77
2.77e-4
4.16
16.66
2.73e-4
4.54
25.02
0.0018
8.21
64.22
7.44e-5
GPT-3.5.ourbo
12.61
47.24
0.0015
0.0
38.89
1.51e-4
0.0
16.66
2.31e-4
6.81
22.71
0.0011
20.66
64.09
3.77e-5
GPT-4o-man mini
6.31
50.45
0.0011
2.77
38.92
9.11e-5
8.33
20.83
1.53e-4
31.81
9,94c-4
28.12
72.04
9.23e-6
LLM Sit (Sbojare
20241b)
3.1ma-3.1-BBB 3.1---tnttt
30.63
38.55
0.0101
8.33
66.66
8.01e-6
25.30
58.33
1.04e-6
6.97
34,09
1,23c-4
4.10
88.12
1.15e-7
GPT-3 S-turb
10.81
10.81
0.1449
0.0
50.22
2.87e-3
25.03
2.33e-5
25.12
8.84k-4
12,42
82.14
2.75e-8
GPT-4o-mini
31.53
39.64
0.0091
11.11
52.77
4.12e-6
16.66
29.16
3.06e-6
9.91
36.36
7.62e-5
20.24
88.28
3.21e-9
evaluate the following discovery methods:
(o)
(b)
39,10
LLM-SR (Shojace et al., 2024b), program search equa- tion discovery method that generates hypotheses of equa- tion skeleton as Python functions with the main idea of combining LLMs' scientific knowledge with multi-island evolutionary search guided by feedback from data.
Complecity
125
Complouty
LaSR (Grayeli et al., 2024), a concept learning equation discovery method that finds abstract textual concepts of mathematical relations from successful equation hypotheses with LLMs and uses these concepts to evolve new hypothe- ses through hybrid approach of evolutionary search (with PySR (Cranmer, 2023)) and I.L.-guided search.
LiV, Tanatore
10
30
50
SSymbolic Accuracy
Adfin
Figure 4. Performance comparison across equation complexity
levels for Feynman and LSR-Transforr datasets: (a) symbolic accuracy and (b) numeric precision (Acco.1) showing considerable performance gap between these two datasets at same complexity levels (averaged over all method-LLM, pairs).
SGA (Ma et al., 2024), a bilevel optimization equation dis- covery method that iteratively combines LLMs for discrete hypothesis generation of scientific laws and physical simu- lations in PyTorch for continuous parameter optimization with respect to data.
remains relatively low across both symbolic and numeric metrics, underscoring the fundamental challenges of this task. One key observation is the poor performance of direct prompting method (DataBlind), which only relies on LLMs' knowledge about the problem without access to data for data-driven refinement. This result underscores the neces- sity of combining LLM reasoning with observational data, as relying solely on prior knowledge proves insufficient for accurate equation discovery across different problems in LLMM-SRBench. We observe that on LSR-Transform data group, LaSR achieves the highest numerical accuracy, lead- ing in both Acco.1 and NMSE, while LLLM-SR with GPT.
Direct Prompting (DataBlind) serves as baseline for generating hypotheses purely from contextual information without access data. By not using data-driven reasoning and refinement in the hypothesis generation, this baseline helps to assess LLMs' memorization of the problem.
3.2. Main Results
Our experimental results (Table 1) reveals several key in- sights into the strengths and limitations of LLM- sci- entific equation discovery methods. Overall, performance
6

---

LLM--RBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
Direct
10
LIMSR
LaSR
SGA
10
MMSSE
O0D
10"7
10"4
10
Chemistry
Biology
Material Science
Physics
Figure 5. Detailed results of in-domain (ID) and out-of-dommin (OOD) performance using Normalized Mean Squared Error across various LSR-Synth scientific domains and L.L.- equation discovery methods (with GPT-4o-mini as LLM backhone).
4o-mini outperforms other methods in symbolic accuracy (~31%). This comparative advantage inverts in the LSR- Synth material science problems, where LaSR consistently yields better symbolic accuracy and LLM-SR achieves bet- ter numerical precision, suggesting that different equation discovery strategies may be better suited to different prob- lems.
Performance In-domain vs. OOD. Generalization to unseen data is a fundamental requirement for scientific laws and a critical aspect of equation discovery. A correct mathe- matical model of observations should not only fit observed data but also extrapolate accurately to out-of-domain (OOD) scenarios. However, current equation discovery benchmarks largely overlook this aspect. In this work, we advocate for explicit OOD assessment in scientific equation discovery by introducing held-out OOD test sets in benchmark. To systematically evaluate generalization beyond observed data, we generate dedicated OOD test sets for synthetic problems in the LSR-Synth category (see App. A for details on data generation). Fig. 5 provides a comparative analysis of ID vs. OOD results. As expected, all discovery methods exhibit higher NMSE in OOD settings, indicating degraded generalization compared to in-domain data. Among the evaluated methods, LLM-SR achieves the lowest NMSE across both ID and OOD settings, while direct prompting performs the worst. Also, we observe some domain-specific variations in generalization performance: the performance gap between ID and OOD is more pronounced in chemistry and biology problems compared
Another notable observation is the consistent outperfor- mance of models using GPT-4o-mini and Llama-3, 3.1-BB compared to those based on GPT-3.5-turbo. This may be due to improved reasoning architectures or better effective- ness of smaller, less opinionated models in the search and exploration needed for navigating space possible equa- tions. The lower performance on LSR-Synth compared to LSR-Transform tasks also indicates that the ability to find transformed variants of known problems does not necessar- ily extend to more challenging scenarios involving novel synthetic terms, where systematic data-driven exploration becomes essential.
3.3. Analysis
physies and material science, although the complexity of problems are designed to be similar, shown in Fig. 10. This suggests that dif- ferent scientific problems may pose distinct challenges for equation discovery methods, highlighting the need for future research to develop more robust approaches for different scientific disciplines.
LSR-Transform vs. Feynman datasets. We analyze the performance gap between Feynman and LSR-Transfome datasets across different equation complexity levels, mea- sured by the number of nodes in the corresponding expres- sion tree (La Cava et al., 2021). Fig. shows the aggregated average performance (over all methods and LLM back- bones) in terms of both symbolic accuracy (a) and numeric precision (b). plexity levels, -ra-Tranfform problems more challenging for current discovery methods than orig- inal Feynman problems. Also, this performance disparity persists even for simpler problems ([0-15) nodes), indicat- ing that the challenging nature of LSR- Transform problems for LLM- LLM--based scientific equation discovery methods is not necessarily due to the structural complexity.
be observed that even at the same com- substantially
OOD generalization and symbolic accuracy. We further analyzed the correlation between our proposed symbolic accuracy metric (Sec. 2.3) and data-dríven extrapolation performance in OOD settings (averaged over all L.R-S domains). As shown in Fig. 6, symbolic accuracy exhibits a strong positive correlation with numerical precision (Acco.1) on OOD data and a corresponding negative correlation with numerical error (NMSE). This strong correlation observed

---

LLM-SRBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
(b)
Meidani al., 2023). While these methods have advanced the field of automated symbolic function discovery from data, they mostly lack mechanisms to incorporate scientific domain knowledge into the discovery process.
Drrest
Symboolc Accuraty
Symbolcc Accuracy
LLMs for Equation Discovery. Recent work has lever- aged LLM-based symbolic regression to enhance scientific equation discovery through various approaches leveraging LLMs' knowledge. LLM-SR (Shojace et al., 2024b) utilizes LLMs' embedded scientific knowledge generate initial equation hypotheses in the form of Python programming functions, which are then refined through adaptive mutation and crossover operations with LLMs as evolutionary opti- mizers. In-Context Symbolic Regression (ICSR) (Merler et al., 2024) employs an iterative few-shot learning paradigm over expression candidates, using previously tested success- ful expressions along with their fitness scores to guide the generation of improved candidates. LaSR (Grayeli et al., 2024) alternates between hypothesis evolution, concept ab- straction, and concept iteration phases to build a learned library of scientific concepts for mathematical relations needed to find the equation for a given data. The learned concepts are then used with pure evolutionary search meth- ods (Cranmer, 2023) like PySR (Cranmer, 2023) as well as LLM-guided search to guide the equation hypothesis evolution. Scientific Generative Agent (SGA) (Ma et al., 2024) also implements a bilevel optimization framework for equation discovery where LLMs iteratively propose discrete hypotheses for scientific laws while physical simulations in PyTorch provide experimental validation and data-driven parameter optimization.
Acceu (OOD)
NMSE (OOD)
Figure 6. Correlation between symbolic accuracy and OOD performance across different equation discovery methods and LLM backbones: (a) symbolic accuracy vs. A showing positive correlation; (b) symbolic accuracy vs. normalized mean squared error showing negative correlation. Results are averaged over all ISSR-Sythh datasets.
between symbolic and OOD performance provides two key insights: First, it establishes OOD evaluation as a powerful approach for assessing the discovery of generalizable equa- tions-an aspect often underutilized symbolic regression research; second, it validates our -LM-dased symbolic eval- uation approach through its strong alignment with numeric generalization performance.
More detailed experimental results, including both qual- itative analyses of discovered equations and quantitative performance comparisons across scientific equation discov- ery methods and LLMs, are provided in App. D.
4. Related Work
Symbolic Regression Benchmarks. Symbolic regres- sion benchmarks can be broadly categorized into scientific saccovery-orientee and general-purpos mathematical dis- covery collections. The scientific equation discovery bench- marks are primarily represented by the SRBench (La Cava et al., 2021) and SRSD (Matsubara al., 2022) benchmarks. SRBench incorporates two key data groups for this purpose: the Feynman physics equations (Udrescu & Tegmark, 2020), and Strogatz. dynamical systems (La Cava et al., 2016; Stro- gatz, 2018). A notable extension to this framework is pre- sented in SRSD (Matsubara et al., 2022), which enhances the Feynman benchmark by incorporating physically mean- ingful sampling ranges for datapoints. The second category includes benchmarks like the Nguyen collection (Uy et al., 2011) and SRBench's 's black-box regression problems (La Cava et al., 2016) which include datasets without scien- tific contexts. However, these existing benchmarks are not well-suited for evaluating LLM-basse equation discovery methods. These general- purpose benchmarks focus on the data-driven discovery of abstract mathematical functions without scientific context, while the former scientific bench- marks consist of well-known equations likely memorized by LLMs, enabling success through recitation rather than
AI for Scientific Discovery. Recent advancements in AI for science highlight the ability of LLMs generate sci- entific hypotheses by leveraging their extensive knowledge and reasoning capabilities (Lu et al., 2024; Ji et al., 2024; Reddy & Shojaee, 2024). LLM agents, when augmented with external tools and scientific simulators, have shown promise in automated scientific data-driven analysis (Ma- jumder et al., 2024a). While recent benchmarks have been developed to evaluate LLMS and agents in hypothesis gen- eration and scientific question answering (Majumder al., 2024b; Chen et al., 2024), evaluation for equation discovery and symbolic regression- one of the core tasks in scientific discovery-remaine yet unexplored.
Symbolic Regression.
Symbolic regression approaches fall into three main categories: search-based based methods that explore equation spaces via evolutionary algorithms or re- inforcement learning (Schmidt& & Lipson, 2009; Cranmer, 2023; Petersen et al., 2021; Sun et al., 2023), learning-basse methods leveraging pre-trained Transformers on synthetic data (Biggio et al., 2021: Kamienny et al., 2022), and hybrid approaches that guide search using neural priors (Landajuela et al., 2022; Shojaee et al., 2024a; Mundhenk et al., 2021;
8

---

LLM-SRBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
scientific reasoning and discovery. Our work extends this line of research by focusing on scientific equation discovery with LLMs, designing the first comprehensive benchmark to assess discovery capabilities of LLM-based scientific equation discovery methods beyond memorization.
agents for data-driven scientific discovery, arXiv preprint arXiv:2410.05000, 2024.
Cranmer, M. Interpretable machine learning for science with pysr and symbolicregression. jl. arXiv preprint arXiv:2305.0158: 2023.
5. Conclusion
Du, M., Chen, Y., Wang, Z., Nie, L., and Zhang, D. Large language models for automatic equation discovery of nonlinear dynamics. Physics of Fluids, 36(9), 2024.
We introduce LLM-SRBench, the first comprehensive benchmark for LLM-driven scientific equation discovery, encompassing 239 tasks across two distinct categories: LSRR- Transform (111 problems derived from transformations of established physical models) and LSR-Synth (128 novel syn- thetic problems spanning four scientific disciplines). Our benchmark provides a standardized and multi-faceted eval- uation protocol for assessing scientific equation discovery with LLMs, accommodating diverse hypothesis representa- tions, including expression strings and programs. Extensive experiments with state-of-the-ar discovery methods and a peak
Grayeli, A., Sehgal, A., Costilla-Reyes, O., Cranmer, M., and Chaudhuri, S. Symbolic regression with a learned concept library. arXiv preprint (eXiv:2409009355 2024.
Ji, H., Wang, Q., Downey, D., and Hope, T. Scimon: Scien- tific inspiration machines optimized for novelty, In ACL Anthology: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 279-299, University of Illinois 2024.
various LLM backbones on SLLM-SRBenchshie performance of only 31%, highlighting the significant chal- lenges and open research opportunities in this domain. We envision that LLM-SRBench benchmark datasets and its evaluation protocol could serve as foundation for future research, driving progress in automated equation discov- ery and advancing our understanding of LLMs in symbolic reasoning needed in scientific discovery.
Kamiemny, P.-A., d' A'scolii, S., Lample, G., and Charton, F. End-to-end symbolic regression with transformers. In Advances in Neural Information Processing Systems, 2022.
La Cava, W., Danai, K. and Spector, L. Infer- ence of compact nonlinear dynamic models by epige- netic local search. Engineering Applications of Ar- rificial Intelligence, 55:292-306, 2016. ISSN 0952- 1976. doi: http://dooloog//000000iieggggppp
Impact Statement
The development and future adoption of L.Mh benchmark for evaluating LLM- scientific equation discovery has the potential to significantly impact the field of artificial intelligence for science and scientific discovery. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
004.
La Cava, W., Orzechowski. P., Burlacu, B., de Franca, F., Virgolin, M., Jin, Y., Kommenda, M., and Moore, J. Con- temporary symbolic regression methods and their relative performance. In Vanschoren, and Yeung, S. (eds.), Pro- ceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
Acknowledgments
This research was partially supported by the U.S. National Science Foundation (NSF) under Grant No. 2416728.
Landajuela, M., Lee, C., Yang, J., Glatt, R., Santiago, C. P., Aravena, I., Mundhenk, T. N., Mulcahy, G., and Petersen, B. K. A unified framework for deep symbolic regres- sion. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
References
Biggio, L., Bendinelli, T., Neitz, A., Lucchi, A., and Paras- candolo, G. Neural symbolic regression that scales. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Pruceedings of Machine Learning Research, pp. 936-945. PMLR, 18-24 Jul 2021.
Langley, P. Data-driven discovery of physical laws. Cogni- live Science 5(1):31-54, 1981.
Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint 2408:2400.00692 2024.
Chen, Z., Chen, S., Ning, Y., Zhang, Q. Wang, B., Yu, B., Li, Y., Liao, Z., Wei, C., Lu, Z., et al. Sci- enceagenthench: Toward rigorous assessment of language

---

LLM-SRBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
Ma, P., Wang, T.-H., Guo, M., Sun, Z., Tenenbaum, B., Rus, D., Gan, C., and Matusik, W. LLM and simula- tion as bilevel optimizers: A new paradigm to advance physical scientific discovery. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreview net forum? -i BcF sc 7P.
Schmidt, M. and Lipson, H. Distilling free-form natural laws from experimental data. Science Advance, 324 (5923):81-88 2009. ISSN 0036-8075. doi: 10.1126/ science. 1 165893.
Shojaee, P., Meidani, K., Barati Farimani, A., and Reddy, C. Transformer- planning for symbolic regression. Advances in Neural Information Processing Systems, 36, 2024a.
Majumder, B. P., Surana, H., Agarwal, D., Hazra, S., Sabhar- wal, A., and Clark, P. Data-driven discovery with large generative models. arXiv preprint arXiv:2402. 24X.:24022.13110 13610, 2024a.
Shojace, P., Meidani, K., Gupta, S., Farimani, A. B., and Reddy, C. K. Llm-sr: Scientific equation discovery via programming with large language models. arXiv preprint arXiv: 2tfi::4404.84000 18400, 2024b.
Majumder, B. P., Surana, H., Agarwal, D., Mishra, B. D., Meena, A., Prakhar, A., Vora, T., Khot, T., Sabharwal, A., and Clark, P. Discoverybench: Towards data-driven discovery with large language models. arXiv preprint arXiv:2407.01725. 2024b.
Strogatz. S. H. Nonlinear dynamics and chaos with student solutions manual: With applications physics, biology, chemistry, and engineering. CRC press, 2018.
Matsubara, Y.. Chiba, N., Igarashi, R., Tatsunori, T., and Ushiku, Y. Rethinking symbolic regression datasets and benchmarks for scientific discovery. arXiv preprint 2tt.:222061154400 2022.
Sun, F., Liu, Y., Wang, J.-X., and Sun, Symbolie physics learner: Discovering governing equations via monte carlo search. In The Eleventh International Conference on Learning Representations, 2023.
Mcidani, K., Shojace, P.. Reddy, C. K., and Farimani, A. B. Snip: Bridging mathematical symbolic and nu- merie realms with unified pre-training. In The Twelfth International Conference Learning Representations, 2023.
Udrescu, S.-M. and Tegmark, M. Ai feynman: A physics- inspired method for symbolic regression. Science Ad- vances, 6( 6(16)xcaay2631, 2020. doi: 10.1126/sciadv. aay2631.
Uy, N. Q. Hoai, N. X., O'Neill, M.. McKay, R. I., and Galván- -ónez, E. Semantically based crossover ge- netic programming: application to real-valued symbolic regression. Genetic Programming and Evolvable Ma- chines, 12:91-119, 2011.
Merler, M., Haitsiukevich, K., Dainese, N., and Marttinen, P. In-context symbolic regression: Leveraging large lan- guage models for function discovery. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 4: Student Research Work- shop), pp. 589 0899606,220 2024.
Virgolin, M. and Pissis, P. Symbolic regression is NP- hard. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
Mirzadeh, I., Alizadeh, K., Shahrokhi, H., Tuzel, O.,, Bengio, S., and Farajtabar, M. fism-symbolie Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.052 2024.
Wu, Z., Qiu, L., Ross, A.. Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. arXiv 2023.
Mundhenk, T. N., Landajuela, M., Glatt, R., Santiago, C. P., faissol, D., and Petersen, B. K. Symbolic regression via deep reinforcement learning enhanced genetic program- ming seeding. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Infor- mation Processing Systems, 2021.
preprint
Xie, C., Huang, Y., Zhang, C., Yu, D., Chen, X., Lin, B. Y., Li, B., Ghazi, B., and Kumar, R. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.2312. 2024.
Petersen, B. K., Larma, M. L., Mundhenk, T. N., Santi- ago, C. P., Kim, K., and Kim, T. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference Learning Representations, 2021.
Zhang, Y., Zheng, K., Liu, F., Zhang, Q. and Wang, Autoturb: Using large language models for automatic algebraic model discovery of turbulence closure. arXiv preprint arXiv:2410. 10657, 2024.
Reddy, C. K. and Shojaee, Towards scientific discov- ery with generative ai: Progress, opportunities, and chal- lenges. arXiv preprint 12Xiv:2412. /427, 2024.
10

---

LLM-SRBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
Appendix
A. Dataset Details
A.1. LSR-Transform
The LSR--rranform the first category of datasets in LLM-SRBench, designed to evaluate the ability of LLM-based scientific equation discovery methods in less common mathematical forms. This dataset challenges LLM- discovery methods to avoid reliance memorization of well-known representations and instead reason through unfamiliar instan- tiations of familiar problems. This approach is motivated by the observation that LLMs often struggle with unfamiliar instantiations of otherwise familiar problems, as highlighted by recent studies on the fragility of LLM reasoning (Mirzadeh et al., 2024). By transforming existing benchmark problems into alternative mathematical representations, L.SR-Transfor provides a rigorous testbed to evaluate how well LL-M-bsse discovery methods perform both (1) semantic scientific reasoning, which draws on LLMs' built-in scientific knowledge, and (2) data-driven reasoning, which utilizes experimental feedback for equation discovery. LSR- Traanssofo builds on the Feynman benchmark (Udrescu & Tegmark, 2020), widely used standard benchmark in scientific equation discovery and symbolic regression. The Feynman benchmark consists of 100 physics equations from Feynman Lecture Series', representing fundamental laws in physics. While the Feynman benchmark has been instrumental in evaluating symbolic regression methods, it primarily tests the ability to recover equations in their standard, well-known forms which are mostly memorized by LLMs. However, real-world scientific equation discovery often involves reasoning about unknown equations based on domain expertise and knowledge from literature as well as empirical data observations. To address this gap, L.SR-Transform transforms the original Feynman equations into less common alternative mathematical forms of the same physical problem by switching input-output variables and symbolically solving for the new target variables.
Transformed Problem Examples
Description Original Problem
Original Problem
48 (a0 m)))
energy (E) a harmonic oscillator system, given data on the mass of the object (ml, the angular frequency of the system lor), the natural angular frequency the system (w,). and displacement from equilbrium position (x).
Find mathematical expression that represents
E == 7 (co + tob)a
48 mx? o
4/ Pa cas(O)
Find mathemalical expression that represents the electric polentaal (X)) at point space due to an electric dipale, given dipole moment (nz). the angle between the dipole axis and the radius vector to the point 16), the distance from the dipcie to the point (r), and permittvity free space (f).
1
Pd (D)
cos(#) Axeel
V
4mm
y2
arccos( Amer"Vy Pd
No.
v Inc, +)
mathematical expression represents current (/) semiconductor diode, given data on the saturation current (ig). the elementary charge (q), the vollage across the diode (V), Bolizmann's constant (Ry), the absolute temperature (T).
q.V kg.T
aa.V exp (k..7)
exp
4.V Ng Int 1)
Figure 7. Examples of how LLM-SRBench L.SR-Transform) problems can be obtained from original Feynman benchmark problems.
Figure demonstrates the equation transformation process, showing examples of the original Feynman problems (along with their scientific deseriptions) and their potential transformed versions. These examples show the dataset's 's design for altering the mathematical representation of the same problem by analytically solving the equations with respect different input variables. For instance, the original harmonic oscillator energy equation E +m(w> + w²)x² is transformed into
symbolic representation of m F and w
4 mz?
w where the target variable switched from energy (F)
'nttps://space cattt.ddu/ooeee wwwooo///ttaar///////////eaa/eee www.:aaaaa.aiioooaaooaaaaa html

---

LLM--RBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
Feyrman Benchmark
LLM SRBench (LSR Transform)
025
Fraction of Datavet
0.20
0.2
0.30
0.6
0.00
5
15
25
Complexity
Figure Comparison expression complexity distributions between Feynman Benchmark and LLM-SRBench (LSR-Trai) Transform) datasets.
to mass (m) angular frequency (w). Similarly, in the electric potential equation V. -see
Pa cos (0) yd
is also transformed
4wcV. showcasing how the problem reformulated to solve for dipole moment (pa), and distance (r). These transformations introduce less-common mathematical representations that are simple but not trivial for LLMs to find from the problem description and data. By systematically altering the input-output relationships into new analytically solvable symbolic forms, LSR- Transform challenges models to reason through unfamiliar mathematical forms, testing their ability to generalize beyond memorized representations and leverage data-driven reasoning to find new forms.
into Pa
V cas cos and r= V Pa cos (0)
The transformed expressions generally exhibit higher complexity than the original physical laws in the Feynman benchmark. To maintain our focus evaluating semantic complexity (reasoning and memorization capabilities) rather than syntactic complexity and lengthy hypotheses, we deliberately filtered out LSR-transform expressions with significantly higher complexities from the dataset. This filtering ensures that the benchmark primarily challenges discovery models' ability to understand and conduct both scientific and data-driven reasoning rather than their capacity to model longer and more complex mathematical expressions. Figure 8 demonstrates the complexity distribution between the original Feynman Benchmark problems versus their transformed counterparts in L.R--Transfona Following (La Cava al., 2021), the complexity of each hypothesis (i.c., expression) is quantified as the number of nodes the expression tree representation of the equation. The expression tree is constructed by parsing the equation into its constituent unary and binary operators, variables, and constants.
Finally, we also exclude the transformed problems that LLM (L1 alaa--- 3. -Isttttnntttt can solve through direct sampling without requiring access to data. This process creates a dataset of 111 transformed equations, each sharing the same scientific context and variables as its original counterpart but presenting a less common mathematical form. The goal of T...R-TT -R--raanffrmm is not to discover new equations but to evaluate whether L.LM-based systems can guide discoveries from non-trivial, data-driven transformations of known equations.
Details of Filtering Process This section provides a comprehensive breakdown of the filtering steps applied during the LSSR-Tranffomm dataset generation, addressing the apparent reduction from 100 original Feynman problems to 111 transformed equations. The LSR-Transform dataset generation involves multiple filtering stages that significantly reduce the number of candidate problems. Starting from 100 original Feynman problems. the transformation process initially generates 471 candidate equations by selecting different pivot variables for each equation and performing ceature-targer transformations. This expansion reflects an average of approximately 4.7 transformed candidates per original problem, demonstrating the diversity introduced by considering multiple input variables as potential targets. The first major filtering occurs during the solvability check using SymPy's symbolic solver (Step 5 in Figure 3). which climinates 53 problems (11.3% of candidates) that cannot be analytically solved for the target variable. These typically include transcendental equations without closed-form solutions, high-degree polynomial equations where symbolic solutions become intractable, and equations involving complex multi-valued functions. After this stage, 418 problems remain. Notably, no equations are
12

---

LLM-SRBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
eliminated during dataset refinement (Step 6 in Figure 3). This stage focuses solely on filtering individual datapoints ensure they fall within the valid domains of the transformed equations (e.g., ensuring positive values under square roots, avoiding division by zero), while the equations themselves remain intact. The most significant reduction occurs during complexity filtering, where 307 problems (73.4% of remaining candidates) are eliminated, resulting in the final 111 problems. This filtering serves a crucial purpose: to ensure that the challenging nature of -SR-Transform stems from semantic complexity (reasoning about the scientific problem and unfamiliar mathematical forms) rather than syntactic complexity (handling lengthy expressions). Following La Cava et al. (?), complexity is measured as the number of nodes in the expression tree representation of each equation. Following this definition, we constrain the complexity distribution to match that of the original Feynman benchmark (Figure 8). In other words, transformed equations with complexity significantly exceeding the original Feynman distribution are exclude. These design choices maintains focus on testing reasoning capabilities while preserving analytical tractability and scientific diversity across physics domains. As demonstrated in Figure 8, even after filtering, ASR-Transform problems remain substantially more challenging than original Feynman problems at same levels of complexity.
A.2. LSR-Synth
The LSR-Synth is the second category of datasets in LLM-SRBench which is a collection of synthetic problems designed to benchmark the performance of LLMs in scientific equation discovery. This dataset is particularly focused on generating plausible yet challenging equation discovery problems that span multiple scientific domains, including chemistry, physics, biology, and material science. The problems in -yyth are constructed by combining known terms, which are well- established in the scientifie literature, with synthetic terms that introduce novel and plausible variations to the equations.
Figure 9 provides examples of problems from the LSR-Synth. These examples demonstrate the dataset's design, which combines well-established mathematical and scientific expressions with novel. domain-specific variations to create chal- lenging models that address the trivial LLM memorization. Each equation is composed of both known and synthetic terms (highlighted in red). Known terms are terms that are commonly found in scientific equations and are well-documented in the literature for that domain and specific problem. For example, terms like A() and -Co A(1) are typical in chemistry reactions as the first-order and second-order kinetics. These terms are included to ensure that the problems remain grounded in the established scientific context, providing foundation for the LLM- -ased methods to build upon for equation discovery related to each scientific problem. On the other hand, synthetic terms are introduced to create novel variations in the problems to avoid trivial LLM memorization. For instance, terms like sin ( (VA(t) and cos (log (A(t) + 1)) in chemistry reaction kinetics are designed to challenge the LLM-based discovery models by introducing non-linearities and interactions that are not commonly seen in standard models. These terms are critical for testing the ability of LLM-based equation discovery models to generalize beyond memorization of standard known formulations and discover new patterns from data-driven reasoning and refinement. The combination of known and synthetic terms in LyR-Synth creates a dataset that both challenging and representative of established scientific problems. This approach enables rigorous evaluation of models' capabilities in interpreting and discovering complex scientific equations, striking a balance between domain familiarity and innovative data-driven reasoning. To generate these known and synthetic terms across various domains, we leverage LLM (GPT-40) by providing problem domain context and descriptions, prompting it to generate candidate terms. These suggested terms and equations are then filtered based on solvability and novelty criteria, followed by domain expert validation.
Figure 10 provides an analysis of the complexity of the problems in the LSR-Synth dataset. Similar to Figure 8, complexity quantified as the number of nodes in the expression tree. This figure highlights the diverse nature of the -SR-Synth dataset, with complexity levels ranging from simple expressions to highly complex ones. By spanning wide range of domains (chemistry, physics, biology, and material science) and hypothesis complexities, LSR-Synth serves as a comprehensive dataset for evaluating the capabilities of LLMs in scientific equation discovery.
Once the structure of equations is generated, their parameters (coefficients) are sampled randomly from specified and scientifically valid ranges, and then data are generated through different solution methods depending on the domain. For dynamical systems (chemical reactions, population dynamics, and physical oscillators), we employ numerical integration using SciPy's solve.ivp with the RK45 method, while static relationships (material stress-strain) are evaluated directly over predetermined input ranges. For each domain, we generate 5000 evenly spaced samples. In dynamical systems, these samples span the time interval E [0, 60). while for material stress-strain relationships, the samples cover strain e E [0, 0.6] and temperature T E [273, (2)3773J K. To evaluate out-of ooft-----istiittinnn (OOD) generalization, for Lime-dependent systems, we designate the last 500 time points as the out-of-domain (OOD) test set, with the remaining 4500 points used for in-domain (ID) training and validation. Similarly, for the stress-strain domain, the OOD test set comprises the last 500 points based
13

---

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Example LSR-Synth Problems with Known Synthetic Terms
-C)-A()) s. ((A())
Chemistry:
Raction rate with respect to Time and Concentration
-G,A G. exp((( Cy. (().A((
Known Terms Synthetic Terms
C.A(t)) ((()
C2 (os(log(A()) 1))
r(1 P())
PG) +r. aP()+ +1
P(
Blology:
Kg
Growth rate with respect to Time and Population size
P(t)
r.P()) + r
.()) + BB(()s)((u))
Ko
(1 P())
P())
(-1 P()
+r ( exp(-x())))
Kg
Hg
(g.si((() - (() - Cg.x. exp(-Ix((D
Physics:
Acceleration with respect (g.si(()) - Cxx(()²² - Sin (x(1) x(t) C.. miee Displacement, and Velocity
Ggsin(() - C,,+(() - Cg.)iixx((()))))._.))))))))))))) (()) Cg.xxt)P (() ((xxxx)))x((I)
Co (1 - G. To)). C2.x((( (7-ppp(((--1000 T
Material Science: Stress with respect Strain
(1 C. T.)) C2(T
To)log(e
Temperature
G, G
Co G. r.)) Cg. expl
Co expl (? T,))..
Figure Examples of L.LM-SBBench L.SR-Synth) problems with known and synthetic terms across different domains. Fach problem presents target equation as the hypothesis to discovered which composed of known terms and synthetic terms (in blue).
on temperature values, maintaining consistent evaluation framework across all domains. The data generation process incorporates the same quality control criteria used in equation generation. Generated solutions must satisfy: (1) solvability within specified numerical tolerance, (2) mcaningful physical behavior (avoiding divergence or constant solutions), and (3) uniqueness compared to existing solutions (using RMSE thresholds). These criteria ensure that the final dataset contains diverse, physically meaningful, and numerically stable solutions suitable for benchmarking equation discovery methods.
B. Evaluation Details
B.1. Data Fidelity
We evaluate the data-driven performance of discovered equations through multiple complementary metrics focusing on both predictive accuracy and generalization capability. The primary metrics include Accuracy to Tolerance (Accz), and Normalized Mean Squared Error (NMSE). The Acc, metric provides binary assessment of prediction accuracy based on point-wist wise relative error. An equation is considered accurate if the maximum relative error across all tolerance 7. Formally:
0i - 3/3
Acc, =1
max 151SN
<T
Vi
where Ú: represents the predicted value, yi is the true value, and Niett is the number of test samples. The indicator function 1(-) returns if the condition is satisfied and 0 otherwise. This metric is particularly useful for cases where maintaining a consistent level of accuracy across all predictions is crucial, as identifies equations that might have occasional but significant deviations from the true values. NMSE also provides a continuous measure of the overall prediction quality, normalized by the scale of the true values:
14

---

LeM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
6.12
LSA Syett Bomaits
Chemistry
Biology
6,00
Physics Naterial Science
Fraction Dataset
5.08
6.06
24
10-34
15:39
25-24
25-26
30:30
25-38
42-41
Complexity
Figure 10. Distribution of problem complexity in LLM--LMSEEE SRBench (LSR-S) L.SR-Synth) datasets across scientific domains.
Σ Newss (1i y₁) 1= New (y/ -2
NMSE
This normalization makes the metric scale-invariant, allowing meaningful comparisons across different datasets and equation types. The NMSE ranges from to 0o, where indicates perfect prediction. Unlike Acc,, NMSE provides a more nuanced view of model performance by considering the magnitude of prediction errors across all test points rather than just their maximum relative enror. Beyond standard predictive metrics, we also place particular emphasis evaluation of out-of-distribution (OOD) generalization, critical requirement for scientific equations. For datasets in LSR-Synth which have been generated synthetically. we evaluate the discovered hypotheses on held-out OOD test sets to also assess the extrapolation capabilities. The performance gap between in-domain and OOD test sets (ANMSE and AAce,) provides valuable insights into the generalizability the discovered equations.
B.2. Symbolic Accuracy
We introduce a novel evaluation methodology for equation discovery that leverages LLM (GPT-40) as an evaluator for assessing mathematical equivalence between predicted and gold equation hypotheses. Traditional metrics in symbolic regression, such as recovery rate (La Cava al., 2016), exact match, normalized tree edit distance (Matsubara et al., 2022), often fail capture the true semantic equivalence of mathematical expressions, especially when dealing with different representation formats algebraically equivalent forms. Our approach employs GPT-4o automated evaluator, capable of analyzing symbolic equivalence across diverse representation formats including equation strings, expression trees, and executable programs. The evaluation process begins by pre-procce processing the hypotheses by (1) removing additional information (such as natural language comments in the case of programs), and (2) replacing constants with placeholder parameter vectors, focusing solely on logical structure and mathematical relations. To assess the reliability of this LLM- based symbolic evaluation approach equation discovery, we conducted a human evaluation study. Two of the authors independently assessed mathematical symbolic equivalence on a of 130 randomly sampled problems. The validation study revealed 94.6% agreement rate between GPT-4o and human evaluators, where agreement rate is calculated the percentage of cases where both LLM and human evaluators made the same judgment about the mathematical equivalence between predicted and ground truth equations (123 out of 130).
Figure 11 provides the prompt used for our GPT-4o based evaluation of the mathematical symbolic equivalence between the generated hypothesis (in the form of program or expression) against the ground truth equation. In this setting, the GPT-4o first articulates its mathematical reasoning before making an equivalence binary assessment.
15

---

LLM--RBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Given ground truth mathematical expression A and the hypothesis B, determine there exist any constant that would make the hypothesis equivalent to the given ground truth expression.
Question:
Let's think step by step. Explain your reasoning and then provide the final answer as:
"reasoning": "Brief step-by-step analysis", "answer":
"Yes/No"
(A): "sqet (E2 [Bsrepsilon) (p)
(A):
E4*E/m- emega"2 *x221//x
(B j: Hypothesis as Program
(B): Hypothesis as Expression
def equation(tf, epsilon, paraes)
(4*B/m)
EF_ Frspponon #p.roxdnun(EF apsilon, 1e 18)
internediate _eslll tt_epsilon parans
denominator
C2
omegaga
demoninetor
C3 omega
aget (4*B/m) /x
radius longth pparans [1] Intemeodiate_resuun result)
parans radius Longth
LLM (GPT-4o) udgement
Human udgement
LLM (GPT-4o) udgement
Human J udgement
R easoning:
Reasoning:
"The expressions match if params[0] params(1) and params[2] = 1, as this aligns both the scalar and conslant factors appropriately.
"The ground fruth expression contains a quadratic term omega inside square root, which cannot expressed as linear combination of the terms in hypothesis B.
Answer: Yes
Answer: No
Figure 11. Symbolic assessment in equation discovery with GPT-4o as evaluator
C. Implementation Details
For a comprehensive evaluation, we implement four state-of-the-art LLM-guided scientific equation discovery baselines, each tested on LLM-SRBench datasets with three different LLM backbones: an open-source model (1. a.llma-3.. -I--B-Innttrutt and two closed-source models (GPT-3. 3-5-turbo and GPT-4o-mini).
C.1. Parameters
Table 2 presents the key implementation details for each discovery agentic method. We adopt most of the hyperparameters from the original implementation for these methods. We have only changed some hyperparameters different baselines that affect the number of LLM calls the search framework. This is to make sure have fair comparison across bascline discovery frameworks with same access budget to LLMs. In experiments, all baseline frameworks have k calls to LLMs (per problem) through the discovery process.
C.2. Prompts
C.2.1. LLM-SR
We use the default prompts from LLM-SR's (Shojaee et al., 2024b) public code repo (https://github. http--/ssooli--mmmeeeettttttteee LLMM SR), which includes:
1. Instruction prompt.
helpful assistant teaked with discavezing nathenatioal function scientific ayscena. the "oquation" function belos, considering the physical neanine zolationshipa iaputs.
2. Evaluation specification prompt.
impert DUmPy ED
Hiiitialize parane-era
patsant [1.0] NPARAMS
diel)
Evaluate equation on obaervations.
16

---

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Table Implementation details of M.-M-baeed scientific equation discovery methods.
Method
Parameters
Direct Prompting (DataBlind)
Temperature
equation program hypotheses sampled from LLM initial prompt
No access data for data-driven refinement
Time limit T 30s per program hypothesis execution,
BFGS optimizer from Scipy for parumeter optimization of equation skeletons
SGA (Ma ul., 2024)
Torch-based implementation model and torch. nn. Module
Mean square loss data-driven feedback agentic search
Adam optimizer PyTorch for differential parameter optimization of equation skeletons
LaSR (Grayeli 2024)
Iterations 25
Cycles per ileration 550
Populations = 10
Population size 33
Maximum size = 30
Operators: +, +. / exp. log, sqrt, sin, cos, tan, cnsh
LLM weights: mutate =0.005, llm.crossover =0.005, llm-gen.random =0.005
Top-K K 20 concepts from library
Default configuration PySR for parameter optimization
LLM-SR (Shojuee et ul., 2024b)
Temperature r 0.8
Batch size == equation programs per prompt
e parallel evaluators
Time limit T 30s per program hypothesis.
Memory
islands for population diversity through search
à in-context examples per prompt
Maximum 10 parameters per equation skeleton
BFGS optimizer from Seipy for parameter optimization of equation skeletons
load data observat-lon
impula, palputa lapuls"], date ousputaf;
inputs
Oplinixw paraneteca besed data
ncipy.optinii import loas (parens]:
y_peed uquet lon (xX, parama]
return rp.near
outputis) 2)
past_partial Lambda pesamas losa (parana)
result min:mine close, partialy [1-0]xwwx w000 UPARANG, net hode BFGSS
Return evaluacion score
opt imi params reeu te.
losa reauls. Fun
if "p. return
elees return -loes
3. Equation example specification Python programming function.
444 Punctiio Exanples
equation_yD ( (SINEUT_VAR .... SINPOT_VAR(X., parans):
m*n Hallamatiica Cunction lon for [BOUTFOT_VAR_DER
Arga:
[INPUT_VAR(O]: numpy artzy soproscating obscrvations (CINFUT_WAR_DESCIO)
SIMPUT_VAR(N): A nunpy areay zopresencing observations of [EINEUT_WAR_EESCINN)
pazans: Artey
paraneters
Roturn: A cumpy array soprosenting (BOITET__VARRDES) as the resuit of asplying the mathenatical Zunetion to the inpals.
nen
Loa example lagic as Functian body
def equel ion l (ETNEUT_LAARRAAA VAR[O), .... SENDOT_VAREY, parans) Equation example
#44 Punct fon to be completed
17

---

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
set equatioo ISINFUT VOR .... 4INPUT VAR(N), paremst: Improvement version of equation_vo and equaticn_yl
C.2.2. LASR
We use the default prompts from LaSR's (Grayeli al., 2024) public code repository (https://github.com LcRegression ieeereeessoo.eeeeeeeesisooooooo j1), which includes:
tri
1. The LLMINIT prompt, which is used in an -aagmnngeetted initialization operation.
2. LLMMUTATION prompt is used to mutate expression based of concepts.
3. LLMCROSSOVER prompt used to construct a new expression from the crossover two sampled expressions based on a set of concepts.
4. LLM Concept Abstraction prompt in CONCEPTABSTRACTION function, which extracts natural language concept from current trends of hypotheses at each iteration.
5. LLM Concept Evolution prompt in CONCEPTEVOLUTION function, which creates a new concept that follows a of ideas in the current library.
In the following, we provide examples of these prompts.
LLMINIT prompt.
<Systen pronpt>
You are helpful assistant that proposes met hemar icel expression by fol Tawing three provided suggest lone. expression consict tae tollowing variebles: Hyariabless). All constants will reprasenzed with symbol expresssinn
only operators: 1-egeratorai).
<Ueer prompt>
Suggestion 1: Rassumpl 1)
Euggestion 2: [[assum2]]
Suggestion 31 ((ssoupp))
Fropose exprersions would approprista eugesstions. Provide commentary enumerates proposed expressione tollowing tormat:
secisions. *s1gson (*oxpri", "expe2",
(331"
LLMMUTATION prompt.
<systom pronpt>
You are a helpful assistant that matatea nathanatical expreasion by following faw provided sugeastions. You will be given three scggestions and single reference expressicn to mutate. expression consict
expression only tollowing varisbles: Hvariablest). All will reprosenzed with symbol oparators: [(peraterra)..
<User proepL>
[[assupp]];
Euggestion 2: [[assump2];
Supgestion 3: [(asaupp]):
Reference Exprassion: [[xxpz]]
Propose (18I) expcessiona
appropriate given sugguationa relozences. Provide commntary
each of your decisions. End with a I30N 1ist that enumerates the proposed expressiens following this format: *>13oon
[fexpr1*,
"xxps2",
3. LLMCROSSOVER prompt.
<Byaten prospt>
You helpful aesi stant that recombines two nathenst ieal expressione by following few provided expreasions swaurbine.
You
thiww suggestiona
expresaion must consist the following variables: [ivrrabllss). constanta will represenced with ayabol C. Fach expression xill only use these operators: Legeratoraal.
18

---

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
<Usce prompt>
Euggestion 1: [
Euogestion 2: [[aaaump2]]
Eupgestion 3: ((ssumpp)
Roterence Expressior 1f (eepri1)
(1ex212)
Propose (181) expressions that would be appropriste given the suggestions and reterences, Frovide short comentory for each of your dacisions. End with 2308 11st that enumerates the proposed expressiens following this format: >sooonn
M"exx2", "exps2",
LLM Concept Abstraction prompt.
cnysten pronpt
You are helpful assistant that hypoclasioes about the underlying axaung luns that generated a 1ist of goud and bad nathenat feall expreseions detaille ways, My oitimate is diecover assumptions generated the mechenatical expressiors.
leall expressions mathenstical structure, and any relation to paysica. concepts. Noto that capital 0 represents an arostrery constant
good expreseions
<llerr prompt>
Expression 1: Hggexp2111
Good Expression 2: Hggepp2111
Good Expressioo 3: Higexp33) 3
Expression [g
Expression (ggepe511
Bad Expression 11 [Ibexpr1:]
Rad Expressior (1bexpr21]
Expression [1bexpx3]]
Bad Expression 41 [Ieexpra:]
Red Repressior 5t (1bexpr51)
would appropriace given expressione, Frovide s3ort commentary for each your cncisiios. Do not calk about togl es related simplicity or complexity the expressions. went ideas that are unigue and iateresting enzuga to anaze the world's best nathematiciars. End with JSCN 1ist that enumerates the proposed tsgoon
Fropoce HNN hyposheses
tolloxing tornat:
["hypi",
"hyp?",
"N1"
LLM Concept Evolution prompt.
<Byston prospt>
You Insight ful assistant skilled in logicel ressoning and deducsion lan, Your task is analyse set of ideas and infer nontrivial conclusions loginally follos Lhen. The ultimate propertien of the hidden expreastone, Pocus providing logicel concluetons that are unique, interest ing, and profoard.
underlying principles
<User prompt>
2: [1idwa23]
Idea 3: [11dea35]
Tdea d1ldea4:1
[lidwa5)]
Resed on these ddess, deduuee (1N)) logical conclusions or hypot beses that directly ly fol low from t then. Provide a brief explanation lon Fos each conclusion, highligittin Ube luginal connect Loas lons belvewn the ideass. Aveid discnss Lopics related simplicisy complexity expressione, following formet:
the proposed
joon
i*conclusion
*Conclusion
CConclusion [[ME)*
C.2.3. SGA
The following prompts are used in our implementation of SGA (Ma et al., 2024) for scientific equation discovery tasks, following the original implementation SGA's public code repository (https: /tt::://giii /github. com coom/PingchunnM//Se. which includes:
System prompt for task.
19

---

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
You axe an Latelligent AI assistant for coding and scientidic equation discovery. You tasked with discovering mathenstical runction structures scientiric nake suze you underatand chan.
Follow the usez's requirenents carefully
Koop your answers short and to the point.
To not provide any intornation that is not requasted. document use Hacksown zormat your solation.
code
explain the behiad
You are very taniliar xith Python and FyTorch.
not use any external Libraries other than tha libreries used it. the examples.
Code formatting prompt for scientific equation discovery task.
** PyTorch Tips
When working wish benxors, always use PyTorch's operators (such ds 'tarch.exp', "torchh.os "sorch.agg r..)
optinal pertoraance, 2. Tn Pytoreh, operator input argunence must tensors, not Floats,
encure compatiniiiey
** Code: Rempirements
only library lowed is PyToreh,
provided by
exerples.
2. lnnctate the size of the tensor as comment after each tensor operation. For extmpee, t [2, 3, 3)
3, Separste the code into paremetere that can be tuned with differentiable optimization and the symbolic expression.
represented PyTorch code, beflne their the struzture and tunction signatures zolow:
ve in the
proposed codo must strictly
hpython
impert torch
import toroh.cn 8s
class Symoolictiuuciionn (nn, Nodulel:
det init Icclt, 1SPARAM IXPARIN
Defina trainable continueus perancters cifferentiable optinization. Tentativoly iritialize the porameters the detault values args.
Args:
super() . init_
ESARAN_INIT
forvard(sclt, (SINEUT VARCABLESS) forsh.Iensor: CERGWAO_FINCTTOOOLRRRO
Solution Bagcirementa
Analyze step-by-step what the potential problem in the previcus iterations based on the teadback. Think about why optimize. Bocus
aive advics scientific equation. Start this sectisn with "Ht Analysis", Amalyzo all iscrations individually, and start the subswction for each Lteration with *Hat Lseration R", ahaze Bl atands for the index. Benanber to analyze every iteration in the bistory.
the reaults previous Steratiens niamatched
formulstion
Think atep by s.ep shat you awed analysis auagesis apacifis fanctional ferm or conscralnts, think about hox these will be incorparated into the svabolic equation, Think about how to separate your algoritan into continuous paraneter part and symbolic expression model aseado-usd code, writien in great detail. Remerbes update
a.ep
this iveration, Thiak about what is needed improow pes Curmance,
Describe the your teainable paranetars based on provious optinisations. Start this section with *12 acep-by-Scap Plan".
default valuea
3. output the code in single code block pyyyhno 18s with detailed commenta in the code block. Do not add cods block. Start sectica "Hs code".
trailing commens before
Context prompt for each scientific problem.
* Context
The objective is to construct a mathenstical expression that accurately maps input varishles to a target output based on a provided dataset. The task involves filling in eode block to datine symbolic expression nodel that ninimizes the difference
predicted
grourd-truth autputs. another ror generating or moditying
block derines symoolic structure at the expression.
with functions:
porameters withir the expressica Footback is provided in the form of netrics messuring the arror between the nadel's predictions and the ground-truth values, as well as quidanze on structural improvements to the cymbolic expression.
The expression. ropresents [ROUI2UT_VAB_pasS given data (SINPOTS_DESC);
D. A Aditiional Results and Analysis
Detailed Numeric Accuracy Analysis. While Table presents median Normalized Mean Squared Error for each method- LLM combination across LLM-SRBent datasets, Figure 12 provides a more comprehensive view of error distributions across all samples. These box plots illustrate performance variations across datasets from two perspectives:
20

---

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
comparing different equation discovery methods with GPT-4o-mini as the LLM backbone, and examining different LLM backhones when using LLM-SR method. The substantial variance in NMSE performance across samples reflects the diverse complexity inherent in our benchmark-stemmiin from both the varying mathematical transformations in LSR- Transfraof and the different combinations of known and synthetic terms in LSR-Synt datasets. Notably, the relative difficulty of datasets varies across methods and L.M backbones, suggesting that different methods and L.MMs possess distinct capabilities in terms of leveraging domain knowledge, reasoning, and generating novel hypotheses.
LSR: Transform
Chemittry
Biology
Physics
Material Science
10-1-
b0000
10
10"4
10-7
10-1
NMSSE
10
20
10
10-13
10
Direct Prompting
5GA
LaSit
LLM-SR
Liema 3
CPT1-355
GFT-96 min
SED Mathad (with GPI-4p mini)
LLM Backbone (with LLM-SR)
Figure 12. Normalized Mean Squared Error (NMSE) of discovered equations in various domains of LLM SRBench with respect to (left) different equation discovery methods using GPT-4omini LLM backbone, and (right) different LLM backbones using LLM-SR method
Symbolic Accuracy and Generalization. For scientific equation discovery methods, both symbolic accuracy and out-of- domain generalization serve as crucial evaluation metrics, reflecting the methods' ability uncover true governing equations. Figure 13 examines the relationship between these metrics, plotting symbolic accuracy against both OOD accuracy and OOD NMSE across all Bechhod-LMMddomiil combinations in LSR- The strong correlation observed between symbolic and OOD performance yields two important insights: first, it establishes OOD evaluation as powerful metric for assessing the discovery of generalizable equations, an approach historically underutilized in symbolic regression; second, it validates our LLM-based symbolic evaluation approach through its strong alignment with numeric generalization performance.
Qualitative Analysis of Outputs. To provide deeper insights into the behavior of different discovery methods, Figure 14 illustrates their final discovered hypotheses on biological population growth problem (BPGO) using Llama-3. 1-BB as the LLM backbone. Direct Prompting (Figure 14(a)) generates equations that capture basic population dynamics, demonstrating LLMs' ability to propose scientifically plausible structures. SGA's solution (Figure 14(b)) successfully incorporates one of the common population growth terms while exploring additional structural components. LaSR (Figure 14(c)) discovers an equation structure that combines multiple interaction terms, though it differs from established scientific formulations. LLM-SR (Figure 14(d)) combines both standard population dynamics terms and synthetic components in its solution. These examples demonstrate the diverse approaches methods take in balancing scientific interpretability with mathematical expressiveness when discovering equation structures.
E. Discussion and Future Directions
Our findings from LLM-SRBench reveal several key insights that inform the design of future LLMs for scientific discovery applications. Scientific equation discovery remains challenging problem for LLMs, requiring complex interplay of domain knowledge, search capabilities with data-dríven feedback, and mathematical manipulation skills. Our results demonstrate that this problem poses significant challenges for L.-M-based discovery frameworks across different model architectures, suggesting that current approaches may be fundamentally limited in their ability to perform genuine scientific discovery.
This work questions the current evaluation paradigm for equation discovery in emerging L.LM-based techniques.
21

---

LLM--RBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
SGA, turrb.5.turbo
LLMSR, GPT-3.5 turbo LLMSR, GPT-4e-mini
LaSR, GPT-3.5
Direct, GPT-3.5-turbo
SGA, GPT-to-mini
LaSR, GPT-to-mini
Direct, GPT-to-mini
SGA, Uaam--311-8b
LLMSR, Llama-3.1-80
La5R, Liama-3.1-80
Direct, Llama-3.1-80
Bollogy
20
Chemattry
20
Physics
% Symbolic Accuracy
10-
% Symbolic Accuracy
10
10
60
80
100
20
10-7
1211
AcCo.1 (OOD)
NMSE (OOD)
Figure 13. Symbolic Accuracy versus OOD performance all domains, methods, and backbone LLM pairs.
demonstrate that existing benchmarks for this task are susceptible memorization and inadequate for evaluating these techniques' true scientific discovery capabilities. Motivated by these limitations, designed LLM-SRBench address the memorization issue through two key innovations: synthetic imaginary scenarios (I-SR-Synth category) that are not based on existing scientific knowledge and require data-driven discovery tools for solution, and transformed equations L.SR-Transform category) that convert common forms of scientifically known equations into less familiar formulations. The LSR-Synth category targets genuine innovation in LLM-based discovery techniques by climinating the possibility of problems are difficult to recite from memory and require reasoning over hypothesis generation steps, making them suitable candidates for evaluating recently emerging LLM-based scientific discovery agents. While the mathematical transformations in -SR-Transform meaningfulness varies considerably across contexts. Many transformations correspond to legitimate physics problems from the Feynman Lecture Series collection and represent alternative problem formulations with practical significance. For example, in the Harmonic Oscillator Energy problem, the original formulation E = mm( + wf)x²² expresses energy as function of system parameters, while the transformed version m (==++ determines the mass required for given energy storage. This transformation maintains scientific meaning by addressing the engineering question of what mass needed to store a specific amount of energy in oscillating system, and such inversions are common in engineering design problems where system parameters must be determined to achieve desired performance characteristics. Similarly, pp cos(O) IneV.
recalling memorized equations, while
algebraically valid, their scientific
4E
the Electric Potential problem transforms from V. (distance for given potential), addressing the practical question of determining measurement distances electrostatic experiments or sensor design.
cas(O) (potential at point due dipole)
However, all transformations maintain clear physical interpretability. Some result in equations where the target variable appears in complex functional forms that may not correspond to natural physical questions, such as solving for angular frequency in oscillatory systems yielding expressions involving square roots of differences that lack intuitive physical meaning. Additionally, certain transformations may obscure natural causal suutiinnsips-ttasfoommii causes acceleration" to "acceleration determines force" maintains mathematical validity but may not reflect underlying physical causality. The L.SR-Transfors category represents a deliberate balance between mathematical rigor and physical meaningfulness by constraining the complexity of transformed problems match original problems, focusing on semantic rather than syntactic challenges in scientific equation discovery, while maintaining the original scientific context and variable meanings to ensure that underlying physics remains relevant even when mathematical formulation changes. The varying scientific meaningfulness of transformations reflects broader challenges in automated scientific discovery that warrant future investigation. Automated discovery systems must incorporate mechanisms to evaluate not only data-driven correctness but also scientific plausibility and interpretability of generated hypotheses, as mathematical validity alone is insufficient for meaningful scientific contribution. The most effective approach to scientific equation discovery likely involves close collaboration between Al systems, which excel at exploring vast hypothesis spaces, and human domain scientists, who can
from "force
22

---

LLM-SRBench: A New Benchmark for Scientifie Equation Discovery with Large Language Models
scientific meaningfulness and guide discovery directions based on deep contextual understanding. Future equation discovery methods could improve by incorporating literature retrieval tools to build grounding foundations for scientific context and domain knowledge, helping to prioritize discoveries that are mathematically valid, data-consistent, novel, and scientifically meaningful. The field needs evaluation frameworks that assess not just mathematical correctness but also scientific novelty, interpretability. and practical applicability of discovered equations, moving beyond narrow accuracy metrics toward a more comprehensive understanding of what constitutes valuable scientific discovery in the age of LLMs with their vast scientific knowledge.
F. Comparison with Standard (non-LLM) Symbolic Regression Baselines
To further validate the utility of LLM-SRBench and demonstrate the advantages of LLM-based approaches, we conducted additional experiments comparing LLM-based methods with traditional symbolic regression techniques that do not in- corporate domain knowledge. We evaluated PySR (Cranmer, 2023), a state-of-the symbolic regression method based on genetic programming, on all LLM-SRBench datasets. PySR operates purely on numerical data points without access to the scientific context, variable descriptions, or domain knowledge that LLM-based methods can leverage in discovery process. We used PySR's default configuration with the same computational budget (equivalent number of evaluations) the LLM-based methods to ensure fair comparison. Table 3 presents the performance comparison between the best- rest-performing LLM--ased method from Table 1 and PySR across all LLM--R datasets. The results reveal several key insights about the complementary strengths and limitations of non-LLM versus LLM-based approaches in equation discovery.
PySR demonstrates competitive and sometimes even better numerical accuracy (Acco.1) across all datasets. However, PySR consistently shows significantly lower symbolic accuracy, particularly struggling with non-physics domains where it achieves 0% symbolic accuracy on chemistry, biology, and material science datasets. The performance gap is most pronounced in problems that require specialized scientific knowledge. While PySR fit mathematical patterns in the data, it lacks the scientific intuition discover equations that align with established physical principles or domain-spectin terminology. Interestingly, PySR shows relatively better performance on physics problems, achieving modest symbolic accuracy of 4.54% on LSR-Synth Physics and 8. % on Transfoonf (which is based on Feynman physics equations). This suggests that physics problems may contain mathematical patterns that are more aligned with the dictionary design in PySR. So they can be discovered better through the data-driven search pipeline designed in PySR. These findings strengthen the motivation for LLM-based scientific equation discovery and demonstrate that LLM--BBench successfully captures challenges in equation discovery that traditional symbolic regression methods cannot adequately address through numerical data-driven optimization alone.
Table 3. Performance comparison between M.M--a methods and state-of-the-ar non-LLM symbolic regression baseline PySR on LLM-SRBBench SA = Symbolic Accuracy (%), Acco. 1 = Accuracy to tolerance 0.1 (%).
Dataset
LLM-SR (best)
LaSR (best)
SGA (best)
PySR SA Acco.1
(Metric)
SA Acco.1
SA/ / Acco.1
SA/ / Acco.1
-SR-Transform
31.53/39.66 39.64
12.61 50.45
9.91 / 8.11
8.11 / 56.76
LSR-Synth Chemistry
11.11 / 66.66
2.77 / 38.92
0// 16.66
0/41.67
LSR-Synth Biology
25.30 / 58.33
8.33 / 20.83
4.16 12.51
0/
LSR-Synth Physics
9.91 / 36.36
9.91 / 31.81
4.54 9.09
4.54 / 29.55
LSR-Synth Material Science
20.24 / 88.28
28.12 72.04
0/ 36.11
6/.
Table 4: LSR-Synth mathematical equations for each scientific domain.
Domain
Equation ID
Equation
Chemistry
CKR1
-kA(t)² + k k, +)
CKR2
-kA(t)2 - kA(t) + hi (os((o((A()) + +1)
CKR3
-kA(t) + Kin, cos(log(A()) 1))
Continued on next page
23

---

LLM--RBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Table 4 continued from previous page
Domain
Equation ID
Equation
CKR4
-k((t)² - k. A(t) (xp(-k,t)) + kw cos(log(A(()) +))
CKR5
kA(1)2 + K, ((1) log(7l+))]1)
CKR6
-k((A(t) + Ay)(t)
CKR7
-kA(t) exp(- + km sin A(t))
CKR8
-kA(t)) exp( k,t) + kw cos (1og( (A(t) 1))
CKR9
-kA(1)² A((1) ki sin (log( A(l) + +1)
CKR10
-ky A(t) + w cos(log( ((t) + 1))
CKR11
-kA(1)² + K. sin logg A()) + 1))
CKR12
=kA(t) + km sin A(t))
CKR13
-kA(t) exp( k,t) + Kt sin(log(A(t) + 1))
CKR 14
-kA(t) + K.p A(n)
CKR15
-k/A A(t) kA((t) exp(-k,t) + kp (it(w)(t)
CKR16 16
-ky A(1) A(())) exp( hh,t + by sin (log( A(L) + 1))
CKR17
-kA(t) ky A(t)00s
CKR18
-kA(t) exp( k.t) + A(t)003
CKR19
-kA(t)² + Kp sin(w
CKR20
-kA(1)2 - K A(1)
+ Ky (in(log((A)) + 1))
CKR21
-kA(t) exp( k. + (tn(wA(())
CKR22
-kA(1) exp k1,)) + A()) + 1)
CKR23
-kA(t)² - k(tt) exp( A((())))AAA -
CKR24
-ky A(t) + Ko sin(w
CKR25
-ky A(t) 2 + A(t)
CKR26
-kA(t) sin (A()) 1))
CKR27
-kA(t)² 4: A(1) exp km sin( A(t))
CKR28
-kA(1)² (xpA((()xpp(kxx))) A(1)0.3
CKR29
-kA(t) exp k,t) + (g)(())) +
CKR30
-kA(t) kt(t) exp k.,t +=A)) + +)
CKR31
-kA(t) Att exp( k. + K, sin(log(A(t) + 1))
CKR32
/v A(t) kA(()) A. cos (log A(t) +1)
CKR33
-A(()) k. A.A(() exp () + k A(1)0.33
CKR34
-ky Akt((t) kA(t)2 + kr sinlog(A((())) 1))
CKR35
-kA(t)2 + A(t 0.33
CKR36
-kA(t) A(t) log(nt + 1)
Continued on next page
24

---

LLM--RBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Table 4 continued from previous page
Domain
Equation ID
Equation
Biology
BPG1
r(1 =
BPG2
BPG3
BPG4
r(-1+ 712--170000000010/////0000001110
BPG5
r(1 P0//6//000-777////=000-00-------
BPG6
r(1 P()/ P100///000000000//00/0000)0))11
BPG7
+ P0 +rP(()
BPG8
r(-1
BPG9
r(1 P +Pt(t)
BPG10
BPG11
rP(1)0.33 rP(()
BPG12
r(1 (())(()/)))()) r + +xP()))x(((1))
BPG13
3P(I)sin(wt) P
BPG14
r(1 - 11 eep(==a--ppppepe- P(O)))
r(-1+ +114--
BPG15
BPG16
rP(t)0.33 + +Pt(t) exp(()))
BPG17
r|1++ - P
P rP(()
BPG18
BPG19
3P(1)sin(ww)) +r(1 ((0
BPG20
r(1 (L-P((/)))))))))+(()))
BPG21
P(O)))
BPG22
r(-1+ P=14-P000001-PP//55000)00-070101
BPG23
r(1 (10-cpp((-P()))p)))-P0)pppppppp-
BPG24
r(1 10-
Physics
PO1
Posin(() B (sin((())) -6) (1x((1) exp(-Ix((0)
PO2
Fosin(() (1)(() - -1) exp( (x)()(x((t)
PO3
-av(t)3 p(1 - x(t)))u((t) w1z(() (x)p(
PO4
Fo sin(())) - - (inn()())) (3v(t)
PO5
Fo sin(() - av(()3 (7 / 0.33 1)x(() wzr(()
PO6
-psin(u(t)) - 28r((t) w/ (7 (t) 0.33 1)x((t) wfr(()3 w
PO7
-Blog(1o(1)) -33r(() (1)
POS
-av(t)3- - (l)(() 0.33 wt)
Continued on next page
25

---

LLM--RBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Table 4 continued from previous page
Domain
Equation ID
Equation
PO9
- (t)
PO10
Fo sin(()) - 111 x(t) (((() - w8())o(())03 + 1)x(t) - wpx((t)
PO11
Fo sin(t) - w0( 1)x(()) - wBx(()) uBx(11)
PO12
-Bsin(v(()) 1)x(() wfz(1)3
PO13
Fo sin(() av(() (lu(t)) wh( + 1)x(()) wfz((t)
PO14
Fo sin(())) (11 (()))((())))))))))))))())))))))) - - (0(7)/(0))a + (1x(()
PO15
Fo sin(t) - 3 log(( + - 3 sin()(t)) 2Bv(() (110))p(11---1011011111
- x())xx))))u(())
PO16
Fu sin(t) wi
(t) wfx(t) wxx((t)xp(( exp(-Ix<tt))
PO17
Fo sin(t) B sin()(())))())) (in w1)x(1)
PO18
Po sin(() ( (in((()))))()) 28v(() wf=((1)
PO19
- (tt)
PO20
-2Bo(tt -(x((())((((((((()
PO21
-av(t)² 1) 23v(t) - 1))((t)
PO22
Fo sin((()) /3 (in
PO23
-2Bo(() 8 - 1(1 100 wf-((13²
PO24
Fo sin(t) 8 log([u(()) ( 1) (x(1)
PO25
Fo sin(l) - cvv(1)3 - 3 [og([u(1))+
PO26
Fo sin(t) sin()(t))
PO27
Fu sin(t) B log (()()) + 1) 23v(t) - wfzr(t²
PO28
Fo (tn) au(()3 23v((t) 3 exp(( (ex((11o(0))0())
PO29
-2 (1) 0.33 + - w3r(())))))) - wā
PO30
-u(1 x(()))u((()))))))))))))))))))() - -g) -
1)x(1)
PO31
PO32
-wb())u(())u0s) + 1)x(()) - -f
PO33
Fosin(t) av))) 3 (exp(-lu wfx(t)3
PO34
-2,B((1) (n1 - ((()))=(()) - + ()x(()) wg
PO35
-28b((t) - (11 o(1))))(()) w8(7/u(11) + 1)x(1)
PO36
Fo sin(()) B (in((())) - ug(7/o(1) 0.33 1)x((l)
PO37
Fo sin(t) 8 eexp(=lx(()))er
PO38
Fo sin(()) av())3 - 23v(t) ( + 1)x(t)
PO39
- -Bsin(u(t)) - 11(1 -0)))=((((())))))))) (xp())))
PO40
Posin(() - -ox(1)3 -
3 -Bexp(((x(()))(()) -11 - ((1)))((())
PO41
Fo sin(t) (l)(()) ( 1)x(1) wfx(()3 wfr(()
Continued on next page
26

---

LLM--RBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Table 4 continued from previous page
Domain
Equation ID
Equation
PO42
-u(1 =
- - 23u(())
PO43
Fo au(()) (()
PO44
Fo - (Bin(())-Bsnn(((()))))))))) 2 1d(1 (0x0))))))))((((0)))))))
Material
MatScil
Eue(-ar(T - To)
n n[TTTTTTT To)
MatSci2
He KcN exp(-Q/(RT)) -nssin(T
MatSci3
He + +n(T exxx-))
MatSci4
He + Kc (cpp(-Q/RRT) + (T - To)
MatSci5
Ege² + + To) log(e +)
MatSció
Ebe(-ar(1 To) + 1) + KE exp( exp(-Q/(RIT) EMn(TT -T)
MatSci7
Eac(-ar(TT (T) + 1)
(T - To)²
MatSci8
He - BT - To) + n log(( 1)
MatSci9
Boc(-ar(T) - To) + + c To) He B(T To) + +
MatSci10
MatSeil1
He KE (x
+ en
MatSci12
Ke exp(- + (T To)
MatSci13
Eac(-ar(T - To) + + Kc exp( (Rp(-Q/(RT)
crexp(-11 -T- To)?)
MatSci14
-B(T - To) -7
MatSci15
-B(T To) (
MatSci16
Ege(-arT( Tn) + 1) + (
MatSci17
Eoe² + en( 2
MatScil8
Eae( Foc(-ar(T - To) + 1) B(T To) + n(T TT) log(c + 1)
MatSci19
He + n( (T) sin(c)
MatSei20
Eg²² ((T -T) To) + en To)
MatSei21
Enc² + enniin(T - To)
MatSci22
KeN exp(-Q/(RTT)) 3(T To) + (T log(e 1)
MatSci23
+ -T) - KcN exp(-Q/(RT)) + en sin TT)
He² + 7 - TT) sin(e)
MatSci24
MatSei25
Ege² Ege(-ar(T To) 1) + 17TT To) log(e 1)
27

---

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
dP
P 96.9069
Ground Truth:
0.9540 dt
P 0.9540Pi
Predicted: Direct (Llama-3.1-8b)
def equation ap.ndaaray, P: .ndarra, params: nndaray) >> ndarray: Mathematical funation for Population growth rate
Azgs:
t: A numpy array representing observations of Time. numpy array representing observations Population time Array of numeric constants parameters to be optimised
numpy array representing Population growth rate the result of applying the ma tetthheatine tical Function to the inputs.
output
params [0] P # linear term based on population size
params [1]
A aquared birth or [3] Linear and linear term
[2]
params [4]
output
b) Predicted: SGA (Llama-3.1-8b)
Symbolicequation Module)
def init (self, float 0.5, K: float = 100.0, time scale: float 0.05):
Define trainable continuous parameters for difffernntaable optimization. Tentatively initialize the parameters
default values args.
Args:
I (float): growth rate. (Eloat): carrying capacity.
time scale (float): time-dapandent confficient.
super () init ()
self.r r nn. Ia.Paaameter ter (torch.
Ceerch.tensor tensor (r, dtypestorch. float)) (1,)
(K, dtypentorch, float)) (1,)
self. time scale Parameter (torch. tensor (time scale, dtype=torch. float))
(1,)
(torch. tensor dtypestorch, float))
def forward t: torch. Tensor, P: torch. torch.Tensor) >> torch. Fensor: torch.Fensor: thamaaticaaal function for Population growth rate
Args: Time
2: Population at time t.
Return:
dt: Population growth
nen
# (B,)
dP dt self.r torch, exp (-self, time scae__c p (1. self. # (B,)
dP
28

---

LeM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
c) Predicted: LaSR (Llama-3.1-8b)
dP
(910.311- ((P.(P+VP)) + log ((t + 4.6396) (- (-0.66095)) sin 00.95572
9910.3
d) Predicted:: LLM-SR (Llama-3.1-80)
def equation
ndarray,
ndarray) ndarray:
Population growth logistic growth model, decay,
and periodicity.
Arga:
A numpy array zepresenting observations of Time.
A numpy array zepresenting observations of Population time b.
params: Array of numeric constants parameters optimised
tau)
Re neturns:
A numpy array xepresenting Population growth rate as the result applying the thematical function the inputs.
une
Apply transformation tions to inputs (if needed)
up
Normalize
P / np max (P) Normalize population
Compute the rates dB dt params [0] P (1 B) params [1] params [2] p**parama [3]
logistio growth model
Compute the periodicity factor periodic_factee factor parems [4] np, sin (paramss (t parans [6]) params [?])
Combine growth rate and periodicity
dt with wewtttitttteeeteee pariodicity dt periodic factor
return dP dt fet_wwtt_pprrddde
Figure 14. Example of output hypotheses from different LLM scientific equation discovery methods for BPGO problem in LSR-Synth biology domain.
29

---

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Ground Truth:
0.19 A.19 0.19 cos (log(A(t) + 1))
a) Predicted: Direct (Llama-3.1-B))
def equation(t: ap.ndarray, .ndarra,, params: .p.ndarray) nddrray: Mathenaicaal function for change concentration (Gzowth, Reaction oz Dacay)
Args:
E: numpy array zepresenting observations of Time.
numpy array xepresenting observations of Concentration
params: Array of numeric constants or parameters to be optimized.
paramss(0]
constant, 0.1 min*-1)
[1]: Co (initial concentration, e.g., 10) for decay or growth
models
Raturn:
array representing Rate of change of concentration as the result of applying the nathematical function inputs. equation implemented under assumption that x*A represents linear dacay, growth.
output params [0] parama [1] params [2]
return output
b) Predicted: SGA (Llama-3.1-4)
class SymbolicRquation (an. Module)
def init (self, k: float 9.00e-02, a_ float 9. 60e-01 A float 60e-01):
Define trainable continuous paraneters for differentiable optimization. Tentatively initialize the paranaters with the default values in
k (float): dime-deeendent reaction rate constant.
(float): initial
A (float): time-dopandant change in concentration.
super() ()
init_())
self, k an Parameter (torch. tensor (k, dtypertorch,
self, Parameter (torch. tansor (A_t, dtypestorch. float)) (2,)
(A_0, dtypestorch, float)) (2,) solf alf.Att an. Paramator (torch. tansor
def (sel,, t: torch, Tensor, A: torch, Tensor) thanatical function Rate of change concentration chenistry reaction kinetics
terch.Tensor) >> torch.Tensor:
Arge:
Tine. (B, 1)
concentration
(B, 1)
Return:
dt: Rate of change of concentration in chemistry reaction kinetics.
# dt self.k self.A.0 self tt (B,
return
30

---

LeM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
c) Predicted: LaSR (Llama-3.1-8b)
(-0.00404022(.33222)) (44))000((1111144444400000000100
d) Predicted: LLM-SR (Llama-3.1 3.1-bb)
def equation .g.ddarray, ndarray, rp.adraray) ndarray: Mathematical kinetics
change concentration chemistzy
Args:
t: A numpy array zepresenting observations of Time. numpy array zepresenting observations of Concentration time parems: Array of numeric constants or parameters to be optimized
Return:
numpy array representing kinetics the result of applying the mathematical. function to the inputs. nnn
change
chemistry
output params[O] A params [1] A+*2 + parana [2] A++3 + params [3] + params return output
Figure Example output hypotheses from different LLM scientific equation discovery methods for CKR2 problem LSR-Synth chemistry domain.
31

---

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Ground Truth:
(-0.05 (31..99 + 0.223 321.119 + 3.63 (7 316.19)
a) Predicted: Direct (Llama-3.1-88)
def equation (epsilon: .ndarry, np.ndarray, params: up. .ddarray ap nddrray::
Ma Mathhmatien tical function for Stress
Args:
epsilon: à numpy array representing observations Strain. 7: A numpy arrey representing observations of Temperature pazams: Array of
paranaters be optimized
numpy array representing Stress as the result of applying the mathematical function inputs.
reference temperature
signa_ 0 parans[O]
alpha params [1]
params [2]
linear apsilon,
[3] apsilon
multiplicative _ot
apsilon_ term = params [4] opsilon
linear temperature
parana [5]
output sigma 0 [1 alpha (I reference temperaturel) + epsilon
linear cectrr_ppsloo__. tern mulipplicativeeessee return output
term linear linear_temperee temperature term
b) Predicted: SGA (Llama-3.1-8b)
BymbolicRquation Module)
c_o: 9.83e-01, c_1: 9.86e-01, e_2:
9. 0.8
73-733,
-1.11e-02, 63e-02):
trainable continuous paramatera differentiable optimization.
Tentatively initialize the parameters with the default values in
Arge:
0.0 (fleat): confficient.
(float): confficient.
c_2 (float): confficient.
(float): coefficient quadratic
(flost): coefficient for exponential term.
(float): coefficient non-linear relationship
and
Temperature.
.0.
super ()
init ()
self Farameter (torch. tensor 0, dtype=torch. fleat)) #
self.c_ an. Parameter (torch. tensor 1, dtypentorch.
# (1,)
(torch,
2, dtypentorch. float))
(2,)
self nn. on.Paramtter (torch. tensor (a, dtype-torch. float)) # (1,)
dtype-torch. float)) (1,)
self.d nn. Parameter (torch. tensor (d, dtypestorch. float)) # (1,)
def fsew,, epsilon: torch, Tensor, T: torch, Tensor) -> torch. Tensor::
on "n"atthenticcal function
Args:
opsilon: Strain. $ (B, 1)
Tomperature. (B,
naturn:
sigma: Stress. [B, 1)
[B, 1)
sigma self.c.0 selt.c torch. exch.e epsilon
self torch. cos (self.d epsilon zet return sigma
32

---

LeM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
c) Predicted: LaSR (Llama-3.1-8)) 1-8b)
e (-712.36)+ + ( 1.37002 2.2798 (T (T- -)))) +6.125 -15078 sin(log(T))
d) Predicted: LLM-SR (Jlama331--8))
equation (epsilon: ap. adarrey ap. ap.ndarray, params: ap.ndarray) ap.ndarray: Ka Wathematical tical function for Stress
Args:
opsilon: numpy array representing observations Strain.
2: numpy array zepresenting observations of Temperature. parameters optimized
params: Array
A numpy array representing Stress the result of applying the mathematical
function to inputs.
output params [0]
(epsilon (1/params (4]) 1)
params [1]
(epsilon * (1/params($1) 1)
params [2]
(epsilon (1/params 1) np. exp (paramas spsilon)
params [5]
epsilon params [6] T params
return output
Figure 16. Example of output hypotheses from different LLM scientific equation discovery methods for MatSciO problem LSR-Synth material science domain.
33

---

LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
Ground Truth:
Fo sin(t) B sin(v(t)) - w/ xzt)² wp x<(t) [x(1]
a) Predicted: Direct (Llama-3,1-80) 3--8b)
def equation(x: (x: np.ndarray, t: np ndarzay, v: op. nddrraa,,, params: np. mp.ndarray) >> ndarray:
Mathematical function for Acceleration
Arga:
x: numpy array representing observations
t: numpy array zepresenting observations of Time. numpy acray representing observations Velocity
params: Array of numeric constants paraneters to be optimized
Re turn:
numpy array representing Acceleration as the result of applying the ma thematical function to the inputs.
Since acceleration is Force/nass and Porce is masstvelocity change, we represent this equation as
dv/dt
representation format,
position it will correspond
to
d"2x/dt"2
simple numeric analogy 4th order approximation involving high
degree dependency functions:
output (params [0] params *x*a3 params *x*x2 parane [3] parans (4]) parans [S] hy**2 params[6]*5 + params [ (x***2 +
output
b) Predicted: SGA (Llama-3.1-8))
class SymbolicRquation Module)
daf init (self, c_0: float 9.50a-01, c_1: float 9.50a-01, e e_2: float 9. 9.50e-01, c3 float 1.05e+00):
Define trainable continuous paramaters for differentiable optinization. Tentatively initialize the parameters with the default values args.
Arga:
0.0 (float):
(float) coefficient.
2_2 (float) coefficient.
(float): coefficient.
super
()
self.c fif.c.. 0 nn. an.Parameter (torch. (o 0, dtypestorch. (1,)
an.Parameter (torch. tensor atypestorch. float)) (1,)
self, c 2 nn. Parameter Parac (torch, tensor
Shhemtorch.flabbe (2,)
self c nn. Paramaatt (torch. tensor(e_3, dtypestorch.floatt (1,)
def forward torch, Tensor, t: torch. Pensor, torch, Tensor)
.. "atthenatiial function for Acceleration Nenl-linear Harmonic Oscillator
x: Position at time t. (B, t: Time. (B, }
v: Volocity at time t. (B,
Return:
dv dt: Acceleration Nonl-linear x, dv dt = self.a_0 torch.expi (-self.o x) self. torch. 0_3 return dt
Oscillator. (B,
Non- zelationship
cos cos (self, c_
34

---

LeM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models
c) Predicted: LasR (Llama-3.1-80)
(a+4)) Lattrade wag 0
amc)
d) Predicted: LLM-SR LLlama-3.1-80)
equation mp..ndrrra, t: up.adarray, V: np.ndarray, parana: np.ndarray) ndarray:
unn Mathematical
Arge:
x: array representing observations of Pesition at time t.
t: numpy array zepresenting observations Time.
v: numpy array zepresenting observations of Velocity at time t,
parems: : Array of numeric constants or parameters be optimized
Return:
numpy array representing Acceleration result of applying the mathematical function to the inputs.
Since acceleration is Forca/nass and Force is masstvolocity change,
represent this
a dv/dt or in our representation format, given our position x it will correspond
d'2x/dt"2 or by simple numeric analogy 4th order approximation involving high degree dependency functions:
output (parama[0] *x**4 params params[1] [1]+**+3 + [2] params [3] *x params(4)
parans [y]*2 params[6] [6]*t params [7] (x**2
return output
Figure 17. Example of output hypotheses from different LLM scientific equation discovery methods for POO problem in LSR-Synth physics domain.
35


Reviews:
[{"review_id": "FCK4rRhgq3", "reviewer": "Reviewer_rJd1", "summary": "This paper introduces LLM-SRBench, a benchmark designed to evaluate Large Language Models' capabilities in scientific equation discovery. The authors address a limitation in existing benchmarks: they primarily consist of well-known equations from textbooks that LLMs may have memorized during training, potentially leading to performance metrics that reflect recitation rather than discovery abilities.\nLLM-SRBench comprises 239 problems across two categories:\n\nLSR-Transform (111 problems): Transforms Feynman physics equations into alternative mathematical forms by changing which variable is solved for, challenging LLMs to discover less familiar representations of known physical relationships.\nLSR-Synth (128 problems): Creates problems across chemistry, biology, physics, and material science by combining established scientific terms with synthetic terms, requiring models to employ both scientific reasoning and data-driven discovery.\n\nThe authors evaluate several LLM-based equation discovery methods (Direct Prompting, SGA, LaSR, LLM-SR) using various LLM backbones (Llama-3.1-8B, GPT-3.5-turbo, GPT-4o-mini). Their findings show that the best-performing method achieves 31.5% symbolic accuracy, indicating challenges in scientific equation discovery.\nThe paper also presents an evaluation methodology that considers both data fidelity (numeric accuracy) and symbolic correctness, including an LLM-based approach for assessing mathematical equivalence across different representations. The authors note a correlation between symbolic accuracy and out-of-domain generalization.\nThis work provides a testbed for current methods and may contribute to advancing scientific equation discovery research.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your time and expertise to review our submission. Please find our responses below. \n\n\n> * there's no systematic exploration of whether different methods have domain-specific advantages.\n\nWe agree this is an important consideration, but it falls outside our study's scope. Our benchmark aims to help community towards building better general LLM-based equation discovery agents applicable across domains. Domain-specific variations mostly come from the LLM's knowledge rather than the agentic discovery framework itself. Further experiments with different LLM backbones would be helpful to study this question in more depth.\n\n\n> Do you have either intuition or insight into specific criteria for when numeric performance and symbolic accuracy metrics diverge?\n\n\nThank you for the thoughtful question. This is indeed a common challenge in equation discovery. Equations with good numeric accuracy may differ significantly in symbolic manner. Conversely, equations with similar symbolic structures can also exhibit significantly different numeric behaviors due to constant/coefficient variations affecting function behavior. This is why our benchmark incorporates both numeric and symbolic metrics for comprehensive evaluation.\n\n\n\n> The benchmark design assumes that transforming equations (LSR-Transform) and combining known with synthetic terms (LSR-Synth) effectively models scientific discovery. How do you justify that these approaches genuinely reflect how scientists discover new equations in practice, rather than simply creating mathematically challenging problems? Would performance on these tasks predict success in real scientific discovery?\n\nThanks for raising this important question. Whether performance on these tasks predicts success in real scientific discovery remains an open research question. But we think that our benchmark can be a good starting point to guide models towards that direction, particularly in the task of scientific equation discovery. \nTo ensure our benchmark reflects scientific discovery rather than just creating mathematically challenging problems, we implemented filtering steps in the design of benchmark (detailed in Section 2, Figures 4, 8, and 10) to avoid lengthy problems with excessive mathematical and syntax complexity.", "claims_and_evidence": "The paper convincingly asserts that LLM memorization is a factor in existing benchmarks, with memorization demonstrated through error curves; the authors leave open the possibility of alternative explanations.  They further demonstrate the benchmark's difficulty with a convincing performance evaluation across methods and SOTA-standard LLMs.", "methods_and_evaluation_criteria": "The paper utilizes a two-pronged approach for dataset creation: transforming known equations through variable substitution and combining known scientific terms with synthetic elements. The benchmark spans four scientific domains with varying distribution of problems across domains. The evaluation employs dual metrics focusing on data fidelity (numeric accuracy/error) and symbolic accuracy, with separate assessments for in-domain and out-of-domain generalization. The authors introduce an LLM-based approach for evaluating symbolic equivalence, validated through comparison with human judgments. The paper evaluates various LLM-based scientific equation discovery methods but does not include non-LLM symbolic regression baselines for comparison. Questions remain about whether the synthetic problems authentically model scientific discovery processes rather than testing mathematical manipulation skills.", "theoretical_claims": "This paper is empirical in nature.  No significant theoretical claims are made.", "experimental_designs_or_analyses": "The paper provides appropriately comprehensive coverage by standardizing methods to 1k LLM calls while preserving core algorithmic structures, evaluating multiple methods (Direct Prompting, SGA, LaSR, LLM-SR) across different LLM implementations. Their comparison of performance across equivalent complexity levels demonstrates that challenge stems from semantic transformation rather than structural complexity. The correlation analysis between symbolic accuracy and OOD performance validates their evaluation metrics. However, the design lacks ablation studies to isolate which components drive performance differences, and while domain performance variations are noted, there's no systematic exploration of whether different methods have domain-specific advantages.", "relation_to_broader_scientific_literature": "While I am not an expert in this field, the paper appears to effectively situate itself within several research domains, connecting to existing symbolic regression benchmarks while addressing their limitations for LLM evaluation", "essential_references_not_discussed": "I am not aware of any essential references that are not discussed.", "other_strengths_and_weaknesses": "This paper is very well written and seems to explain the nuances of the field well, deserving to be a part of the published record.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Do you have either intuition or  insight into specific criteria for when numeric performance and symbolic accuracy metrics diverge?  \n\nThe benchmark design assumes that transforming equations (LSR-Transform) and combining known with synthetic terms (LSR-Synth) effectively models scientific discovery. How do you justify that these approaches genuinely reflect how scientists discover new equations in practice, rather than simply creating mathematically challenging problems? Would performance on these tasks predict success in real scientific discovery?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rEF4EAhccW", "reviewer": "Reviewer_Yany", "summary": "This paper introduces LLM-SRBench, a benchmark designed to evaluate LLMs on scientific equation discovery tasks. The authors identify a key problem: existing benchmarks like Feynman equations can be solved by LLMs through memorization rather than actual discovery. To address this, they develop two benchmark categories: (1) LSR-Transform, which transforms common equations into less familiar mathematical forms, and (2) LSR-Synth, which creates novel synthetic equations that combine known scientific terms with plausible synthetic components. The benchmark spans 239 challenging problems across chemistry, biology, physics, and material science domains. They show that state-of-the-art methods achieve only 31.5% symbolic accuracy, highlighting significant challenges in this field.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your time and expertise to review our submission. Please find our responses below.\n\n> * However, it would be good to include more detailed case studies or error analyses to better understand which types of equations or mathematical patterns pose the greatest challenges for current methods.\n> * While showing that current methods struggle, there's limited analysis of why they fail or which specific reasoning capabilities are lacking\n\nThank you for the thoughtful suggestion. We examined the failure cases during rebutal period and noticed that LLM-based models usually don't fail due to the syntax function errors (such as missing variables or dependencies). Instead, most errors come from incorrect terms or combination of terms within the equations, reflecting mistakes due to semantic aspects of discovery for a given scientific problem. We could not identify a uniform pattern of specific mathematical relations that consistently challenge current discovery methods. We believe this is because LLM-based equation discovery involves a complex interplay of domain knowledge, search capabilities with data-driven feedback, and mathematical manipulation skills. To thoroughly study failure case patterns would require investigating all these dimensions. We agree with the reviewer on the importance of this research direction and consider it a promising avenue for future work.\n\n\n> The correlation between symbolic accuracy and OOD performance is intriguing. Does this relationship hold equally across all scientific domains, or are there areas where symbolic understanding is less predictive of generalization?\n\nThank you for this thoughtful question. We have actually explored the correlation between symbolic accuracy and OOD performance across different domains (detailed in Figure 13 Appendix). Our results demonstrate that this positive correlation holds across all four domains. Symbolic understanding appears to be a reliable predictor of generalization capability in equation discovery.\n\n\n\n> How sensitive is performance on your benchmark to the quality or quantity of provided data? In real scientific scenarios, data is often limited or noisy - have you explored this dimension?\n\nWe have not yet explored the impact of noise or data limitations in our current benchmark generation process. Our primary motivation was to develop a benchmark that helps the community build better general LLM-based equation discovery agents for scientific domains, addressing limitations in existing benchmarks for emerging LLM-based techniques. We fully agree with the reviewer on the significance of noise and limited data in real-world scientific discovery scenarios. This is one of the directions we're considering for future benchmark enhancements.\n\n\n> Your benchmark focuses on discovering equations given data and context. Have you considered extending it to evaluate LLMs' abilities to generate plausible hypotheses before seeing complete datasets, which is another important aspect of scientific discovery?\n\nWe would like to clarify that our benchmark evaluates SOTA methods that already incorporate a two-phase approach: hypothesis generation followed by data-driven validation. In the first phase, LLMs generate plausible equation hypotheses without seeing the data, which are then refined based on how well they fit the data. We agree that evaluating pure hypothesis generation capabilities is an important dimension of scientific discovery. While our present focus has been on end-to-end data-driven scientific equation discovery, extending the benchmark to explicitly measure hypothesis quality before the data validation is also a valuable direction for future work. We also think that combining mathematical derivation processes with data-driven reasoning represents an interesting avenue for future research that has not yet been thoroughly explored in current methods.\n\n\n> The examples of output hypotheses in Figure 14 provide good qualitative insights\n\nAs some reviewers have higlighted this example helpful, we plan to include more hypothesis examples from different domains in the revision of paper.\n\n\n> * Could benefit from more discussion of how its findings might influence future LLM designs to improve scientific reasoning\n> * Would be nice to see more discussion around if some of the equation transformations in LSR-Transform, while mathematically valid, are meaningful from a scientific perspective\n\nThank you for the suggestion. We will make sure to add more discussion regarding these points in the camera-ready version.", "claims_and_evidence": "The paper's main claims are well-supported by the evidence presented. The authors convincingly demonstrate the memorization issue through Figure 1, showing how performance on standard Feynman problems exhibits patterns consistent with memorization (sharp error drops and lower symbolic error rates) rather than actual reasoning and discovery. The experiments across multiple methods and LLM backbones provide strong evidence that their benchmark is substantially more challenging than existing ones. The performance analysis across different scientific domains, complexity levels, and generalization capabilities is thorough and supports their conclusions.", "methods_and_evaluation_criteria": "The methods for creating the benchmark are well-designed and appropriate. The LSR-Transform approach uses rigorous symbolic transformation while maintaining appropriate complexity and ensuring analytical solvability. The LSR-Synth methodology includes careful verification of both solvability (using numerical solvers) and scientific plausibility (through expert validation).\n\nThe evaluation metrics are comprehensive, including both numeric performance (accuracy to tolerance, NMSE) and symbolic accuracy. I particularly appreciate the validation of their GPT-4o-based symbolic evaluation against human experts (94.6% agreement), which strengthens confidence in their assessment approach. The inclusion of out-of-domain evaluation is also valuable for assessing true scientific understanding.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "The experimental design is comprehensive and well-executed. The authors evaluate four different methods (Direct Prompting, SGA, LaSR, LLM-SR) using three different LLM backbones (Llama-3.1-8B-Instruct, GPT-4o-mini, GPT-3.5-turbo) with standardized conditions (1,000 LLM calls per problem).\n\nI found the analyses of performance across equation complexity levels (Figure 4) and in-domain versus out-of-domain generalization (Figure 5) particularly informative. The correlation between symbolic accuracy and OOD performance (Figure 6) is an interesting finding that validates their evaluation approach.\n\nHowever, it would be good to include more detailed case studies or error analyses to better understand which types of equations or mathematical patterns pose the greatest challenges for current methods.", "relation_to_broader_scientific_literature": "The paper effectively places itself within the broader literature on symbolic regression, scientific equation discovery, and LLM reasoning capabilities. It builds upon previous benchmarks (SRBench, SRSD) while addressing their limitations for LLM evaluation. The connection to literature on LLM reasoning fragility with unfamiliar representations (Mirzadeh et al., 2024; Xie et al., 2024) provides solid theoretical grounding for their approach.", "essential_references_not_discussed": "None that I'm aware of.", "other_strengths_and_weaknesses": "Strengths:\n- The benchmark addresses a gap in evaluating LLMs on scientific discovery vs. memorization\n- The evaluation across multiple methods, backbones, and domains is thorough\n- The correlation between symbolic accuracy and OOD performance is an interesting finding\n- The examples of output hypotheses in Figure 14 provide good qualitative insights\n\nWeaknesses:\n- While showing that current methods struggle, there's limited analysis of why they fail or which specific reasoning capabilities are lacking\n- The synthetic problems, while validated for plausibility, may not perfectly capture real scientific discovery challenges\n- Could benefit from more discussion of how its findings might influence future LLM designs to improve scientific reasoning\n- Would be nice to see more discussion around if some of the equation transformations in LSR-Transform, while mathematically valid, are meaningful from a scientific perspective", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Have you identified specific patterns or characteristics in the equations where current methods consistently fail?\n2. How sensitive is performance on your benchmark to the quality or quantity of provided data? In real scientific scenarios, data is often limited or noisy - have you explored this dimension?\n3. The correlation between symbolic accuracy and OOD performance is intriguing. Does this relationship hold equally across all scientific domains, or are there areas where symbolic understanding is less predictive of generalization?\n4. Your benchmark focuses on discovering equations given data and context. Have you considered extending it to evaluate LLMs' abilities to generate plausible hypotheses before seeing complete datasets, which is another important aspect of scientific discovery?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Himnf9pVAJ", "reviewer": "Reviewer_vLgS", "summary": "The paper introduces LLM-SRBench, a novel benchmark designed to evaluate the capabilities of LLMs in scientific equation discovery. The key motivation behind the benchmark is to prevent trivial memorization by LLMs, which has been a limitation in existing equation discovery benchmarks. The benchmark consists of 239 equation discovery tasks across 4 scientific domains (physics, chemistry, biology, and material science) and is structured into two different categories:\n \nLSR-Transform: Reformulates well-known physical equations, from the Feynman benchmark, into less common mathematical forms, making it more difficult for LLMs to rely on memorization.\n \nLSR-Synth: Introduces synthetic equations that combine known scientific terms with novel, discovery-driven components, requiring reasoning and data-driven inference.\n \nThe paper evaluates multiple SOTA LLM-based equation discovery methods, including LLM-SR, LaSR, SGA, and direct prompting baselines, using both open-source (Llama-3.1-8B) and closed-source (GPT-4o-mini, GPT-3.5) models. To evaluate model performance, the paper also proposes a novel symbolic accuracy metric, using GPT-4o as an evaluator. Symbolic accuracy correlates with out-of-domain (OOD) performance, suggesting that correctly identified symbolic structures contribute to better generalization. The best-performing method achieves only 31.5% symbolic accuracy on LSR-Transform, highlighting the difficulty of the task. Direct prompting (without data-driven reasoning) performs poorly, showing the necessity of data feedback in scientific equation discovery. LLMs struggle with OOD generalization, indicating that discovered equations often fail to extrapolate beyond the training data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your time and expertise to review our submission. Please find our responses below. \n\n> A systematic classification of failure cases across different domains would provide valuable insights. Examining where and how models fail—whether due to misidentified variables, incorrect functional forms, or numerical instability\n\nThank you for the constructive suggestion. We examined the failure cases during rebutal period and noticed that LLM-based models usually don't fail due to the syntax function errors (such as missing variables or dependencies). Instead, most errors come from incorrect terms or combination of terms within the equations, reflecting mistakes due to semantic aspects of discovery for a given scientific problem. We could not identify a uniform pattern of specific mathematical relations that consistently challenge current discovery methods. We believe this is because LLM-based equation discovery involves a complex interplay of domain knowledge, search capabilities with data-driven feedback, and mathematical manipulation skills. To thoroughly study failure case patterns would require investigating all these dimensions. We agree with the reviewer on the importance of this research direction for future work.\n\n\n**Representative Failure Examples** Thank you for this suggestion. We will include examples of failure cases for different methods across domains in the camera-ready version.\n\n\n> The paper provides evidence that generalization performance varies across scientific domains, ... However, it does not analyze why this variation occurs. \n\nThanks for the thoughtful question. We have explored problem complexity across domains (Figure 10 in Appendix shows the distribution by complexity). While biology and physics problems have similar and slightly higher complexity than other domains, we observe a smaller ID-OOD gap for physics but a larger one for biology in the results. This suggests factors beyond mere complexity are at play (like LLM domain knowledge).\nThe reviewer raises an interesting point that would require domain-specific analysis beyond our study's scope. Our primary goal is to develop a benchmark that helps the community build better general LLM-based equation discovery agents applicable across domains. More specialized applications would require domain-focused analyses\n\n\n> Novelty Check...In the LSR-Synth pipeline, the authors use GPT-4o as a novelty evaluator to determine whether a generated equation is distinct from known scientific expressions. However, LLMs may incorrectly classify an equation as novel despite it existing in prior literature. .. this does not guarantee that the novelty check is reliable.\n\nWe cannot theoretically guarantee whether GPT-4o evaluation of novelty is reliable, but evaluating novelty is a non trivial task and we think that LLMs (with their vast knowledge about literature) could be a helpful tool for novelty assessment. Our empirical evidence also supports the effectiveness of this approach: the consistently lower performance of discovery methods on LSR-Synth datasets suggests these problems aren't trivial and do contain novel elements (if they were simply known equations, LLM-based discovery models would likely recall them from embedded knowledge and solve them easily). We agree that more rigorous novelty evaluation involving domain experts and specialized scientific literature retrieval tools would enhance the design of benchmark problems. However, this extension would require significant domain knowledge and human expert review which is not feasible for the rebuttal period but valuable for future work.\n\n\n**Dataset Size Reduction in LSR-Transform** Here is the detailed breakdown of our filtering process in LSR-Transform: After transformation steps in Figure 3 (Step 4), we obtained 471 transformed problems from the 100 original Feynman problems; During the solvability check with sympy (Step 5), 53 problems were discarded, leaving 418 problems; In dataset refinement (Step 6), we only filtered datapoints to ensure they were within the domain of the new transformed equations, without eliminating any equations at this stage. We then filtered out 307 problems due to significantly higher complexity compared to original problems, resulting in the final 111 problems in LSR-Transform. This is to ensure that the challenging nature of LSR-Transform stems from semantic aspects of discovery rather than from syntactically more complex or lengthy problems (as shown in Figures 4 and 8). We will clarify this process further in the revised version of paper.\n\n\n**Additional Qualitative Examples of Outputs** As some reviewers have higlighted examples of Figure 14 helpful, we plan to include more hypothesis examples from different domains in the revision of paper.\n\n---\nWe hope that most of the reviewer’s concerns have been addressed. We’d be happy to engage in further discussions.", "claims_and_evidence": "Supported claims:\n \nThe following claims are well-supported by empirical evidence provided in the paper:\n \n1) The authors demonstrate that existing equation discovery benchmarks allow LLMs to rely on memorization rather than true discovery by showing significant performance drops when problems are reformulated (Figure 1).\n2) The claim that LLM-SRBench is more challenging is well-supported by the low symbolic accuracy scores (Table 1).\n3) The evidence for poor OOD generalization is strong, as models show significantly higher NMSE on OOD test sets (Figure 5), reinforcing the difficulty of extrapolation.\n \n \nClaims That Need Stronger Justification:\n \nThe paper provides evidence that generalization performance varies across scientific domains, with chemistry and biology showing a larger performance gap between in-domain (ID) and OOD cases compared to physics and material science. However, the authors do not analyze why this variation occurs. While it is possible that some domains inherently pose more complex challenges for equation discovery, another explanation could be that the dataset design itself contributes to the difficulty, either through the structure of equations, the nature of data distributions, or the frequency of certain mathematical patterns.\n \nA systematic classification of failure cases across different domains would provide valuable insights. Examining where and how models fail—whether due to misidentified variables, incorrect functional forms, or numerical instability—could help determine whether LLMs struggle more in certain disciplines due to intrinsic domain complexity or due to dataset-specific biases. Table 1 and Figure 5 already indicate that different models perform differently across domains, but the paper does not include concrete examples of incorrect outputs or categorize failure modes. A deeper analysis, including sample failure cases and their classification by domain and model type, could reveal whether generalization failures follow a consistent pattern (e.g., do chemistry problems frequently lead to overfitting, while physics problems lead to missing terms?).\n \nTo strengthen this point, the authors should consider adding:\n \n- Representative examples of failure cases across domains (e.g., incorrect equations generated by benchmark models).\n- A classification of common failure types (e.g., missing terms, incorrect exponents, incorrect dependencies).\n- A discussion on whether these errors stem from domain-specific challenges or dataset characteristics (e.g., certain equation structures in some domains being more prone to errors).", "methods_and_evaluation_criteria": "The paper benchmarks state-of-the-art LLM-based scientific equation discovery methods using three LLM backbones (Llama-3.1-8B, GPT-3.5, and GPT-4o) and evaluates their performance on LLM-SRBench. The evaluation criteria, including novel symbolic accuracy, numerical precision, NMSE, and OOD generalization, are well-aligned with the problem of equation discovery. Additionally, the symbolic accuracy evaluation using GPT-4o is verified against human expert judgments, ensuring reliability in assessing equation correctness.\n \nBy comparing LLM-SRBench with existing benchmarks, the authors effectively demonstrate its increased difficulty and reduced sensitivity to memorization. The models evaluated include LLM-SR, LaSR, SGA, and Direct Prompting (DataBlind), providing a comprehensive comparison of different equation discovery approaches.", "theoretical_claims": "No theoretical justification is required in this paper, as it primarily focuses on empirical benchmarking rather than formal proofs. The evaluation metrics used, including symbolic accuracy, NMSE, and OOD generalization, are well-defined and appropriate for assessing scientific equation discovery. No issues were found with their formulation or application.", "experimental_designs_or_analyses": "The experimental design is sound, with clear methodology and appropriate evaluation metrics. The hyperparameters used for different methods (LLM-SR, LaSR, SGA, and Direct Prompting) are detailed in the appendix, ensuring reproducibility. Each step in the LSR-Transform and LSR-Synth pipelines is well-documented, and the authors provide code detailing the dataset generation and evaluation procedures, further supporting transparency. No major issues were found in the experimental design.", "relation_to_broader_scientific_literature": "The paper builds on prior work in symbolic regression (e.g., AI Feynman, PySR) and LLM-based scientific discovery, addressing the issue of memorization in existing benchmarks. It extends benchmarks like SRBench and SRSD by introducing LSR-Transform and LSR-Synth, which focus on reasoning beyond recall. The study aligns with recent advances in LLM-guided symbolic regression (e.g., LLM-SR, LaSR, SGA) and contributes a evaluation framework to test equation discovery in a more challenging and diverse setting. By systematically testing LLMs on equation discovery tasks requiring reasoning over data rather than recall, this work helps advance our understanding of LLMs' ability to generalize mathematical structures, a crucial step toward automated scientific discovery, as demonstrated across four domains: physics, chemistry, biology, and material science.", "essential_references_not_discussed": "No, the paper appropriately cites and discusses the relevant prior work in symbolic regression, LLM-based scientific discovery, and existing benchmarks to the best of my knowledge.", "other_strengths_and_weaknesses": "The strengths and weaknesses of the paper are thoroughly discussed in the \"Claims And Evidence,\" \"Methods And Evaluation Criteria,\" and \"Questions For Authors\" sections above. These include the paper’s well-structured benchmark, strong experimental design, and clear evaluation criteria, along with areas needing further justification, such as failure case analysis, novelty verification.\n \nThe LSR-Transform dataset is built upon the Feynman benchmark dataset, but the authors introduce transformations and natural language specifications for each reformulated problem, making it more challenging and reducing reliance on memorization. The LSR-Synth dataset aims to incorporate novel synthetic terms into existing equations, creating discovery-driven problems that go beyond simple equation recall. This approach has the potential to be extended beyond the 4 scientific domains used in the paper, making it a valuable resource for broader scientific applications.", "other_comments_or_suggestions": "None.", "questions_for_authors": "•  Domain-Specific Generalization Performance\nThe paper provides evidence that generalization performance varies across scientific domains, with chemistry and biology showing a larger performance gap between in-domain (ID) and out-of-domain (OOD) cases compared to physics and material science. However, it does not analyze why this variation occurs. Could the observed difficulty stem from intrinsic domain complexity, or might it be influenced by dataset-specific factors such as equation structure, data distribution, or frequency of certain mathematical patterns? Systematic classification of failure cases across different domains could help clarify whether these challenges are due to intrinsic scientific difficulty or dataset artifacts? Specifically:\n1) Can the authors provide representative examples of failure cases across domains?\n2) Would a classification of common failure types (e.g., missing terms, incorrect exponents, incorrect dependencies) help reveal domain-specific trends?\n3) Could the authors discuss whether the failure modes observed in different disciplines stem from the nature of scientific equations themselves or from dataset characteristics?\nClarifying this aspect would help determine whether performance disparities across domains are inherent or dataset-driven. Please refer to the \"Claims That Need Stronger Justification\" section for detailed concerns.\n•  Novelty Check in LSR-Synth Pipeline\nIn the LSR-Synth pipeline, the authors use GPT-4o as a novelty evaluator to determine whether a generated equation is distinct from known scientific expressions. However, LLMs may incorrectly classify an equation as novel despite it existing in prior literature. Since model performance on this dataset is poor, it suggests the problems are challenging, but this does not guarantee that the novelty check is reliable.\n4) Could the authors justify why asking GPT-4o alone is sufficient for novelty evaluation? Would a secondary human expert review or detailed literature analysis improve the reliability of the novelty check?\n5) Given that novelty assessment is crucial for ensuring LSR-Synth problems do not introduce trivial cases, how can the authors ensure that LLM-generated novelty assessments are not biased or incorrect?\n•  Dataset Size Reduction in LSR-Transform\nIn Section 2.1 LSR-Transform, the authors state:\n\"This process yields 111 total transformed equations derived from the 100 original Feynman problems.\"\nThe dataset generation pipeline includes steps to increase diversity by selecting a new target variable (Step 2: Select Pivot Variable) and switching the roles of input-output variables (Step 3: Feature-Target Transformation). However, despite this expansion, only 111 transformed equations remain from an original set of 100 Feynman equations.\n6) Is this reduction due to eliminations in Step 5 (Solvability Check) or Step 6 (Dataset Refinement)? Could the authors provide a breakdown of the proportion of equations discarded at each stage to better understand why the final dataset size remains close to the original? Understanding this filtering process would clarify how much of the dataset reduction is due to analytical intractability versus imposed constraints on the transformed dataset.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "No"}}, {"review_id": "QQcFpcAM2l", "reviewer": "Reviewer_tSk5", "summary": "The paper introduces a benchmark for scientific equation discovery, where the model are able to use both input/output values along with a problem description in human language, into constructing an equation that describes the data well. The model tested on this benchmark would be measured by the accuracy of the recovered equation and also the semantic accuracy of the equation. From results, normal LLMs are able to perform poorly on this task.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your time and expertise to review our submission. Please find our responses below.\n> it would be interesting to see how traditional symbolic regression methods would fare on the dataset... which would make the need for LLM-based discovery stronger, and the dataset more useful.\n\nThanks for the thoughtful suggestion and constructive feedback. Previous work on LLM-based scientific equation discovery (LLM-SR, LaSR, and SGA) has provided evidence for the benefits of these approaches compared to traditional methods. However, we agree that evaluating traditional non-LLM methods on our benchmark would further demonstrate these advantages.\nIn response to your suggestion, we conducted experiments with PySR (a state-of-the-art non-LLM baseline) on our benchmark datasets during the rebuttal period. The results (shown in table below) demonstrate that PySR often achieves competitive numeric accuracy but considerably lower symbolic accuracy, particularly on non-physics datasets. This confirms that while traditional methods can fit data numerically, they struggle with symbolic understanding due to lack of domain knowledge. We also think that these findings further motivate the value of LLM-based techniques for scientific equation discovery. Thank you again for the constructive feedback. We will surely include this analysis in the final version.\n\n| Dataset (Metric) | LLM-SR (best) | LaSR (best) | SGA (best) | PySR |\n|---------|---------------|-------------|------------|------|\n| LSR-Transform (SA / Acc0.1) | 31.53 / 39.64 | 12.61 / 50.45 | 9.91 / 8.11 | 8.11 / 56.76 |\n| LSR-Synth Chemistry (SA / Acc0.1) | 11.11 / 66.66 | 2.77 / 38.92 | 0 / 16.66 | 0 / 41.67 |\n| LSR-Synth Biology (SA / Acc0.1) | 25.30 / 58.33 | 8.33 / 20.83 | 4.16 / 12.51 | 0 / 25.0 |\n| LSR-Synth Physics (SA / Acc0.1) | 9.91 / 36.36 | 9.91 / 31.81 | 4.54 / 9.09 | 4.54 / 29.55 |\n| LSR-Synth MatSci (SA / Acc0.1) | 20.24 / 88.28 | 28.12 / 72.04 | 0 / 36.11 | 0 / 68.0 |\n\n**Incoporation of noise** Thank you for the comment. We have not explored the imacpt of noise in the our benchmark's data generation. The main motivation of this benchmark was to help community towards building better general LLM-based equation discovery agents that can be leveraged in scientific domains, addressing the challenges of current benchmarks for the emerging LLM-based techniques.\nWe fully agree with the reviewer on the importance of noise in real-world scientific discovery scenarios. This is one of the directions we're considering for future benchmark enhancements.\n\n**R2 vs Accuracy to Tolerance** Thanks for raising this important question. Based on our analysis (and some of the previous works [Kamienny et al., 2022; Biggio et al., 2021]), R2 is not a good metric for the evaluation of symbolic regression, particularly if we have large-scale synthetic data with very different output scales. \n- R2 can be easily saturated by normalized mean patterns (similar to NMSE), often missing prediction nuances at different scales. This is evidenced by the consistently high R2 scores (above 0.999) achieved by most recent methods in benchmarks like SRBench.\n- The accuracy-to-tolerance metric (defined in Section 2.3) aggregates point-wise normalized metrics for each datapoint rather than normalizing mean aggregated scores. It also implements a tolerance threshold for the worst point-wise normalized distance, making evaluation more robust to function behavior nuances (e.g., sudden changes or spikes)\nAs the goal of symbolic regression is to learn correct underlying mathematical relations, we should care more about these nuances of function numeric behavior as well as the symbolic mathematical alignment which makes accuracy to tolerance a better metric for numeric precision assessment. \n\n**Evaluation of Equation Semantic Similarity with LLMs vs more classical methods** Thank you for the comment. We will surely add more details on equation semantic similarity evaluation in the updated version.\nRegarding the question, we think that restricting the outputs of the equations to specific forms for LLM-based equation discovery won’t be a good idea since one of the main benefits of LLMs is their capabilities in programming and code generation which opens new more flexible representations for equation discovery that was not possible with the previous expression tree based methods and their corresponding symbolic evaluations. \nWhile evaluating semantic similarity is non-trivial, our empirical analysis and human study show that LLMs are usually strong at recognizing equation equivalence across different styles and representations. We also think that correlation between symbolic accuracy (computed from GPT-4o) and generalization (computed from OOD data) shown in Figure 6 further support LLMs effectiveness as semantic symbolic evaluators for equation discovery.\n\n---\nWe hope that most of the reviewer’s concerns have been addressed. We’d be happy to engage in further discussions.", "claims_and_evidence": "I think based on the paper, it can be shown that LLMs are still unable to properly incorporate scientific information into its predictions, which I think is a fair conclusion from the trial experiments ran with the benchmarks.", "methods_and_evaluation_criteria": "I think the dataset is quite unique, and it is definitely a good start to other works in this area. Some general comments I have with the proposed dataset are as follows:\n\n- A point that would make the dataset more realistic and fitting of real world scenarios would be incorporation of some observation noise, which is typical when actual experiments are conducted. This could potentially be done through curation of some real experiment data, or can simply be addition of some noise into the training set.\n\n- I notice that the preliminary results are mostly for LLM evaluators, which make sense since the benchmark also include domain knowledge in human language. Despite this, it would be interesting to see how traditional symbolic regression methods would fare on the dataset, even if they are not able to incorporate the domain knowledge. Based on what the metrics from the authors, they may fare well in terms of accuracy but less so in semantic similarity, which would make the need for LLM-based discovery stronger, and the dataset more useful.\n\nAlso some comments regarding evaluation criteria:\n\n- I'm not fully convinced with using LLMs to evaluate semantic similarity, also despite the 94.6% figure quoted by the authors whose experimental details are not so clear (e.g., how can we be sure the 130 test cases cover all cases that can be output by an LLM?). I understand that it can be hard to determine semantic similarity, but maybe more classical methods would be applicable if the output form of the equations can be more restrictive (which may be against what the authors want though).\n\n- From what I understand, typical SR papers uses R2 score as the evaluation criteria. I'm wondring why this paper does not also include that.", "theoretical_claims": "There are none.", "experimental_designs_or_analyses": "The results seem okay, but would also benefit from some kind of confidence/score variance value as well. Some issues are with evaluation criteria, which I describe above.", "relation_to_broader_scientific_literature": "I think the paper would have great relevance, since it can be a good step to involve scientific reasoning into modelling data. Even though there are still ways that the benchmark could be improved to be more realistic with the scientific uses, or to incorporate even more equations from more domains within each subfields, but overall it may still be a good start for work in this direction.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Most of the questions/suggestions I have additionally can be found in previous sections, and are related to some aspects of the evaluation criteria, and with the realism of the dataset that may be improved.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **(a) Summary**
This paper presents LLM-SRBench, a new benchmark designed to rigorously assess scientific equation discovery capabilities of large language models (LLMs) while mitigating memorization effects. It introduces two components: (1) LSR-Transform, which reformulates familiar physics equations to challenge recall-based strategies; and (2) LSR-Synth, which introduces synthetic equations that require novel hypothesis generation across four scientific domains. The benchmark evaluates both symbolic and numeric accuracy and highlights current limitations in LLMs' generalization and symbolic reasoning. The best-performing method achieves only 31.5% symbolic accuracy, underscoring the difficulty of the task and motivating further work in this direction.

**(b) Strengths**
The reviewers agree on several key strengths of the paper:
* Novel benchmark design:
rJd1, Yany, and vLgS commend the benchmark’s effort to eliminate memorization by introducing reformulated and synthetic equations.
* Thorough evaluation:
Yany and vLgS highlight the comprehensive experiments using multiple LLMs (GPT-4o-mini, LLaMA, GPT-3.5) and methods (LLM-SR, LaSR, SGA, etc.).
* Sound methodology:
All reviewers appreciate the dual evaluation approach using symbolic accuracy and numeric fidelity. Yany and vLgS find the use of GPT-4o for symbolic equivalence evaluation well-justified and empirically validated.
* Contribution to broader research:
tSk5 and rJd1 praise the benchmark’s relevance to advancing scientific discovery capabilities in LLMs and symbolic regression.

**(c) Weaknesses**
Despite the paper’s merits, reviewers identified several areas for improvement:
* Evaluation Limitations:
tSk5 questions the sole reliance on GPT-4o for symbolic similarity evaluation and suggests including more classical evaluation techniques (e.g., R²), though the authors rebut this with empirical justification.
* Novelty validation:
vLgS raises concerns about GPT-4o being the only novelty evaluator in LSR-Synth, recommending secondary human or literature-based verification.
* Lack of failure analysis:
vLgS and Yany request deeper insight into failure patterns—such as domain-specific trends, frequent error types, and case studies—though the authors plan to include this in the camera-ready version.
* Generalization gaps:
vLgS and Yany observe that performance disparities across domains (e.g., chemistry vs. physics) lack sufficient causal analysis, though complexity statistics were later provided.
* Missing baselines:
tSk5 recommends comparing with traditional symbolic regression methods. The authors address this by adding PySR results showing poor symbolic accuracy despite good numeric fit, strengthening the LLM motivation.
* Data realism:
tSk5 also notes the benchmark could benefit from incorporating noise or real-world data to better reflect experimental uncertainty, which the authors acknowledge as future work.

**(d) Decision**
Overall, the reviewers are positive about the paper, recognizing its timely contribution and high-quality benchmark design. While several concerns were raised, the authors addressed them constructively during the rebuttal, adding ablations, baselines (e.g., PySR), and deeper analyses. Remaining weaknesses, such as GPT-based novelty evaluation and limited failure breakdowns, are acknowledged by the authors as areas for future development. Given its novelty, empirical rigor, and relevance to the growing field of LLM-driven scientific reasoning, I recommend acceptance.

**Additional Comments on Reviewer Discussion**
* rJd1: Read the rebuttal, appreciated the authors’ clarifications, and maintained a positive stance (Strong Accept).
* Yany: Engaged with the rebuttal and acknowledged the authors’ responses, especially around symbolic accuracy and domain-wise correlations.
* vLgS: Initially raised significant concerns about novelty and domain differences, but found the rebuttal clarifying and helpful, though GPT-based novelty checks remain a lingering issue.
* tSk5: Provided constructive evaluation-focused suggestions; acknowledged the rebuttal and appreciated the added comparisons with classical SR methods.
Decision: Accept (oral)