Robust Equation Structure Learning with Adaptive Refinement
Authors: Yunlun Li
Sinno Jialin Pan
Venue: ICLR 2026 Poster (2026)
Keywords: Symbolic Regression
Genetic Programming
Equation Discovery
Large Language Model
AI for Science
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
TL;DR: We introduce, RESTART, a symbolic regression framework combines LLM-based hypothesis generation with explicit structure refinement.

Abstract: Symbolic regression (SR) aims to automate scientific discovery, but often truncates the hypothetico–deductive cycle, focusing on hypothesis and experiment while lacking systematic analysis. We introduce RESTART, a framework that closes this loop by adding a principled analysis stage to diagnose and correct structural errors. RESTART features two core mechanisms: a short-term refinement process that uses boosting to identify unexplained signals and guide an LLM toward targeted corrections, and a long-term structure library that distills successful refinements into reusable code snippets for cumulative knowledge. On LLM-SRBench across Physics, Biology, and Materials Science, RESTART achieves lower error and higher accuracy than state-of-the-art baselines. It also generalizes robustly, recovering near-exact functional forms on out-of-distribution data, representing a significant advance toward fully automated scientific discovery.

Published as a conference paper at ICLR 2026
ROBUST EQUATION STRUCTURE LEARNING WITH ADAPTIVE REFINEMENT
Yunlun Li
Sinno Jialin Pan
Department of Computer Science and Engineering The Chinese University of Hong Kong, Hong Kong yunlun. 1ie1 ink. cwwk. edu. hk, sinnopanBcuhk.edd edu. .kk
ABSTRACT
Symbolic regression (SR) aims automate scientific discovery, but often trun- cates the hypothetico-ddeuucive cycle, focusing on hypothesis and experiment while lacking systematic analysis. We introduce RESTART, framework that closes this loop by adding principled analysis stage to diagnose and correct structural errors. RESTART features two core mechanisms: short-term refine- ment process that uses boosting to identify unexplained signals and guide an LLM toward targeted corrections, and long-term structure library that distills success- ful refinements into reusable code snippets for cumulative knowledge. On LLM- SRBench across Physics, Biology, and Materials Science, RESTART achieves lower error and higher accuracy than state-of-the-ard baselines. It also general- izes robustly, recovering near-exact functional forms on out-of-distributtin data, representing significant advance toward fully automated scientific discovery.
INTRODUCTION
Scientific progress often unfolds through a simple yet profound loop (bottom left of Figure D scientist proposes model, tests it against observations, identifies its shortcomings, and refines the hypothesis for next iteration. This iterative cycle powered dis- coveries from Kepler's laws of planetary motion Newton's 's formulation of classical mechanics, and remains the foundation of modern science (Nola & Sankey, 2014: Li et al., 2021). Automating this process is the goal of symbolic regression (SR), which seeks to recover human- -iterpriiterreetable mathematical expressions from data. By discovering equations that both fit observations and remain human-readable, SR has the potential to accelerate scientific insight across domains ranging from physics to biology (Sun et al., 2023) Cranmer et al.) 2020) (Shi et al. (023). However, SR is NP- hard (Udrescu & Tegmark, (200)), as the search space of expressions grows combinatorially with expression length and operator set, making efficient exploration critical.
Existing SR methods can be broadly categorized into search-based and mapping-bused approaches (Shojace et al. 2025a). Search-based methods such genetic programming (GP) (Koza & Poli, 2005 Dubčáková 2011 Cranmer 2023 Mundhenk et al. (022)) evolve candidate populations via mutation and crossover, but rely heavily random exploration and often revisit similar regions of the search space, leading to slow convergence. Mapping-basee based methods employ autoregressive
models such Transformers (Vaswani al., 2017, Kamienny al.) 2023 Biggio al.) 2021) trained on large synthetic datasets to directly map numerical data to symbolic expressions. They can produce strong single-shot hypotheses, but lack an explicit mechanism for refining hypotheses based on observed eтror patterns, making them brittle in outtoo--distriuttion (OOD) settings. Recent work has begun to incorporate Large Language Models (LLMs) into SR, leveraging their capacity for symbolic reasoning and natural language priors (Shojaee et al. 2025a Grayeli et al. 2024 Ma et al. 20244 Wang al. 2025). However, most such methods still instantiate only the first two steps of the scientific loop hy hypothesize and saperiment-witheeee turns observed errors into explicit guidance for iterative refinement.
principled analysis mechanism that
We introduce RESTART? (Robust Equation STructure learning with Adaptive RefinemenT), novel framework that explicitly closes the hypootesiie-epppeinnen-aaaaaaa loop for SR. RESTART
The code can be found at http:://tthhhccomm/yyylnnnAAAAAs

---

Published as a conference paper at ICLR 2026
starts with mapping- based initializer to generate strong first hypothesis, then uses LLM to iteratively refine it under a two-level guidance mechanism as follows:
Short-term Guidance (Targeted Refinement): After each test, we learn an exploration function to model what the currently learned equation fails to explain, providing highly specific, localized feedback to steer the next hypothesis.
Long-term Guidance (Cumulative Knowledge); We maintain a persistent structure li- brary to store validated refinements. Using an improvement-gatee admission policy, ensure only enabling efficient knowledge reuse.
refinements are retained, preventing library bloat and
Together, these components form a complete scientific cycle, emulating how human scientists itera- tively refine their models. The empirical results show that our method converges faster, and discovers more accurate and parsimonious equations. Our contributions are summarized as follows:
We propose RESTART, framework that operationalizes the entire scientific discovery cycle. Inspired by how human scientists refine hypotheses, RESTART employs a short- term guidance mechanism to learn data-driven exploration function for localized refine- ment. Concurrently, long-term structure library cumulatively stores only performance- improving refinements, enabling reusable knowledge across iterations.
Extensive empirical studies LLM-SRRench (Shojaee et al. (2025b) show that RESTART achieves lower error and higher recovery accuracy than state-of-the-art baselines, including GP-based, mapping-based and LLM-based methods.
2 PROBLEM FORMULATION
Symbolic Regression (SR). Given a dataset D = {(x,, 1 with x; E Rd and y; E R, SR seeks a function f Rd -E E that minimizes a loss function L, which is typically the empirical mean squared error (MSE). The discovered should be concise and human- inteaminnterrettblee. Some works additionally penalize expression complexity when selecting the final function (Cranmer 2023).
LLM-guided equation proposal. Let P be the space of prompts that encode the task description and a few-shot set of equation exemplars. These exemplars are concrete equation candidates previ- ously discovered during the search and stored in the exemplar buffer (see Section[4.4ffo details). Let T be the space of candidate equation templates, and an equation template whose symbolic operations are fixed while the numeric constants are left unspecified. Later, we fit these constants to data, turning the template into an executable equation. In practice, we represent each template as a short Python code snippet. Conditioned on p P, an LLM induces a distribution 4p(-) over T. Given l: ii.d. samples
T is an equation form
Up: i= 1..... k. we define the expected best-of-k: loss as
(P( Bryngp 1=1 min L(T;) 1515K
where L(T;) is the fitted loss for template Ty after parameter optimization, and best-of-k denotes selecting the loss of the best candidate among & independent templates sampled from qp- Concep- tually, one may seek a prompt p², such that
p arg min (p) PEP
(2)
To this end, finding a better prompt is way to optimize the final result. However, a static few-shot prompt biases the LLM's search distribution, 9p. towards regions merely syntactically similar to the exemplars, which may not correspond to regions of low empirical loss. Instead, we construct prompts adaptively based on the evolving exemplar buffer, which can seen as a form of meta- optimization over that gradually shifts 4p toward the template with lower empirical loss.
3 RELATED WORK
Search-based methods explore equation space via stochastic operators. Genetic programming (GP) (Koza & Poli 2005 Dubēáková 2011. Cranmer] 2023) evolves expression trees through
2

---

Published as a conference paper at ICLR 2026
crossover and mutation, while RL-based methods cast SR sequential decision process (Petersen et al. 2021 Xu et al. 2024).. However, both suffer from the combinatorial explosion of the search space leading to slow convergence, hyperparameter sensitivity, and overly complex expressions LLa Cava et al. 2021 Holt et al. 2023). Critically, they require evaluating hundreds of thousands of candidates, making them computationally expensive.
LLM-based search and refinement methods utilize LLMs to encode strong priors and symbolic reasoning into search-based SR (Shojaee et al. 2025a; Wang et al. 2025), enabling more efficient exploration of the hypothesis space. In these approaches, candidate equations are represented as executable code, evaluated on data, and then fed back the LLM, which is prompted to produce improved versions. Some methods further augment LLM prompting with error-related signals (c.g., loss values residuals) provide a weak signal (Ma et 2024, Wang al. 2025). More recent frameworks, such as (Grayeli et al. 2024, Wang et al. 2025). ) also maintain a dynamic concept li- brary that summarizes validated patterns and leverages them to bias future generations toward more promising regions of the search space. Unlike prior work that samples many positive and nega- tive hypotheses and abstracts them into natural-language concepts, RESTART distills structure only from high-value generations and stores it as executable code, yielding more efficient and directly actionable guidance.
Mapping-based methods treat SR as supervised sequence prediction, training autoregressive models on large synthetic corpora (Petersen et al.] 2021 Biggio et al.] 202T; Kamienny et al. 2022 Li et al. 2023). They deliver strong single-pass hypotheses but are brittle under distribu- tion shift. Several works finetune mapping-basee based SR models with RL objectives (Landajuela et al.) 2022 Kamienny
al. (023) to improve generalization via error-driven updates. However, this approach is highly sample-inefficent and gradient updates on large language models are computa- tionally prohibitive, making impractical LLM scale. This motivates coupling mapping- -based initialization with guided, iterative refinement-precee precisely what our framework achieves.
4
METHODOLOGY
Our proposed RESTART implements the complete cetpsssteeeeeeeieeendenttt----ee by unifying a powerful LLM-based generator with a principled refinement mechanism, shown in Its three core components are:
cycle
Figure I
Informative Initialization: The search begins with a strong hypothesis from a mapping- based estimator, preserving nonlinear structures that linear models often miss.
The unexplained signal from each experiment is explicitly modeled as an error-aware subproblem. The solution to this subproblem directly guides the L.LM's next hypothesis.
Targeted Refinement:
Cumulative Knowledge Retention: Successful, boosting-driven revisions are distilled into reusable structure library enabling long-ten term knowledge accumulation across iterations.
Unlike prior work that uses static few-shot prompts, RESTART adaptively constructs prompts by incorporating both short-term feedback and long-term knowledge.
4.1 INITIALIZATION VIA TRANSFORMER
Rather than initializing the search with a simple linear model (Shojaee et al.) 2025a), which only captures additive effects of the input variables, we initialize with the transformer-based estimator E2E (Kamienny et al. (0022). E2E directly maps the input data to a symbolic expression as a prior. This data-dependent initialization yields stronger initial hypothesis fo that already encodes salient nonlincarities, such as polynomial, trigonometric, and exponential functions, as well as interaction terms between multiple variables.
4.2 INSPECTING SYMBOLIC SUBPROBLEMS
At iteration t = 0, let ft Rd + R denote the current symbolic hypothesis. We consider a symbolic function class 9 consisting of exploration functions g Rd+11 -> R that take as input both
3

---

Published as a conference paper at ICLR 2026
Hypothesize
Experiment
Teat All Hypotheses
Generated Hypotheses
candionn penama
panamat))
peparemnaly coimtajt, penamar
parame
df seuaion
equ equstori..
Execute
Respond
MSE_jss Jass 2.14
Equation
Structure
Prampt
like
Inttial Hypothests
Code
Dwinition
Boosiing Equation
Coda
LLM
Loss
Finees Soore
f(x) x
Equation
Structure 2
Subproblem Solving
Solver
8ti 2 a
Structure
Exempler
Hypothesis
alffx)).)-f(xxxsinnn
f(x)-2x1
Structure
Equation
ser eqp.aarr..
Bocsting
Current Hypothesis
Structure
Prev_Joss
Our
Equation
Short Term Structure Analysis
Anays
Long Term Structure Library
Analysis
Figure 1: Overview of the RESTART framework. The framework follows three main steps of the scientific cycle (illustrated at bottom left): (i) Hypothesis stage: We first generate an informative initial hypothesis using symbolic regression solver. In subsequent iterations. RESTART adaptively constructs prompts from exemplars and the structure library, and queries LLM to generate sev- eral hypotheses f(z) as executable Python functions. (ii) Experiment stage: RESTART exccutes each fzz) on the dataset (X, y), computes the loss, and optimizes its parameters (constants) to min- imize the error. (iii) Analysis stage: RESTART formulates subproblem (xx), xc) targeted refinement, solves it using symbolic regression solver, and adds both f(x) and (f)) to the exemplar buffer. It then computes fitness Sti if $tt a, use the LLM summarize the structure that led to the improvement and store it in the structure library for knowledge accumula- tion.
the prediction fı(x) at iteration and the original features x:
gt +
arg min geg
((g(f((x),x),
(3)
where L(g(-), y) denotes the loss between predictions g(.) and targets The objective in Eq. makes the exploration function St boosting-style correction the hypothesis toward the target Because takes both fx(x) and x as inputs, the symbolic form of ge (x)).x), x) describes how the current prediction should adjusted as a function of current value and the features in order to better match the data.
The backend for solving Eq. 3 is pluggable; viable options include KAN (Liu et al. 2025). RL- based methods (Petersen et al. 2021), mapping-based methods (Kamienny et al., 2022). The choice of backend does not alter the formulation (see Appendix B.3 for details). In our experiments, adopt pre-trained Transformer (Kamienny et al.
2022) balance speed and accuracy. Finally,
we store the tuple S, = (fe, (x)((x,, (x), gt. L(g ( ( x) X y)) in the exemplar buffer B, to guide subsequent prompt construction.
4.3 RETAINING VALIDATED IMPROVEMENTS
While structure analysis identifies hypothesis's shortcomings, applying these insights indiscrimi- nately risks overfitting to short-term noise, inflating complexity, and relying too heavily on approx- imate equations. To mitigate this, retain only those boosting-driven updates that demonstrably

---

Published as a conference paper at ICLR 2026
reduce the task loss. This process involves two steps: (i) gating modifications with a fitness score, and (ii) distilling eligible modifications into reusable structures.
Step 1: Evidence and gating. To ensure only meaningful modifications are retained, we gate updates using comprehensive fitness score, Stit- This score holistically evaluates a candidate equa- tion's s improvement by considering both relative and absolute loss reduction compared to exemplars. Given new hypothesis fnew (x) with loss nnw = f( (x)) and the exemplar hypothesis fuase (2) with the lowest loss bhase [(fvase(x)) from the current prompt, we compute,
base
new
R
and
base
neww
base
where is the relative improvement ratio and A is the absolute improvement. To compare values across different scales, normalize them to range of [0, 1) as,
Sy = 1 - kR and Sa = 1- (1+
Here, 8r captures the proportional gain, with k hyperparameter that rescales its sensitivity, while Sa captures the absolute gain in a saturating form, preventing extremely large absolute improvements from dominating the score. Then combine these scores using weighted average, Sti. 100 (w,3, + Sa), where w, + wa = 1 (see Appendix B.1 for more details). This design recognizes both large absolute gains on high-error problems and significant relative gains on low-error ones.
Step 2: Structure formation. After scoring each candidate modification - with Sht. given threshold a, retain those with 8üt 2 and refer to them as high-value. For any high-value modification, the associated structural change often captures information about the ground-truth equation, which may help explain the high fitness score and benefit subsequent iterations. Ac- cordingly, we prompt the LLM with (Se, ((),:,((((())........)))))())<)) y), L(Se+1 (x). to summarize the salient change as
C (name, desc.
where name is a canonical identifier, desc is a brief textual description, and h is small Python code snippet (e.g., sin. (x) implementing the structure. Representing each structure as code snippet aligns with our design: the structure a distilled summary of the observed modification and can directly guide subsequent generations. We update the structure library C' by inserting (C++1, $6t) and, whenever another entry shares the same name, merging them by (i) taking the set union their code snippets h, (ii) assigning to the merged entry the highest Să observed across its instances, and (iii) keeping the earliest desc among those instances. We enforce a capacity constraint |C| < K and, if exceeded, evict the lowest-scoring entries until the constraint is satisfied, thereby limiting redundancy while keeping the library compact and effective.
4.4 ADAPTIVE PROMPT CONSTRUCTION
At each iteration, the prompt for generating new hypotheses fi+1,j integrates three complementary information sources: (i) few-shot exemplars, (ii) boosting equation summaries, and (iii) validated candidate structures. This design couples short-term feedback with long-term, reusable knowledge.
Few-shot exemplars. We maintain an exemplar buffer B, as populations with multiple islands, suggested by (Shojaee al. (2025a) and Grayeli et al. (2024) (see Appendix [.]] for more details).
For iteration 1+1,, we select n exemplars { 1n C B, via MSE-weighted random sampling to balance quality and diversity. Each exemplar includes the fitted equation and compact summary of its associated boosting equation, which highlights the residual error. This provides immediate context on what has been learned and what remains unexplained.
Structure snippets. To inject higher-level, reusable guidance, we sample m structures from the library C, with probability proportional their validation score. Each structure provides descrip- tive name, a short definition, and a symbolic code sketch. While boosting equations offer instance- specific, short-term cues, the structure library encodes cumulative, and long-term refinements.
5

---

Published as a conference paper at ICLR 2026
Adaptive Hypothesis Generation Prompt
1. Task Definition: The prompt begins by setting the LLM's role, overall goal, and the expected output format.
You helpful assistant
Complete the 'equation' function below, Find the mathenatical function skeleton that represents
def equation_vd (x, params) ...
Analysis for current equation (f)
2. Few-Shot Exem- plar:
current equation E (X) with MSE:
the boosting equation (f found be:
Provides a concrete
g_0
example of past at- tempt, including the crucial boosting equa- tion analysis in doc- string.
with MSE: ..r
(Rest of the exemplars)
def equation_v] (x, parama):
Improved version of 'equation_vj-11,
3. Iterative Context & Final Instruction: The final section pro- vides the most recent context, including high- scoring structures, and gives the final instruc- tion for generating the next equation.
Top scoring structures for the problem: Namel, fit: definition: ...
functions:
Consider previous versions of the equation with the corresponding boosting equation function and structures (if any) for the problem
ear
Figure 2: The annotated structure of our adaptive prompt. The prompt is dynamically assembled in three main parts: (1) A fixed task definition setting the goal. (2) A series of few-shot exemplars showing past attempts and their boosting equation analyses. (3) The final iterative context, which includes reusable structure library and the instruction for generating the next hypothesis.
Prompt assembly. The final prompt is constructed from fixed template populated with: (i) the task definition and variable specifications, (ii) the n exemplar-hoosting equation pairs, and (iii) the high-scoring structure snippets (see Figure 2 Conditioned this prompt Pt+1, the generator LLM g autoregressively samples k candidate symbolic expressions. Each candidate is emitted as Python function with tunable constants (e.g., def (x, params)) ). We then optimize these constants using BFGS solver to minimize the training loss, yielding a set of fitted hypotheses (f+t15]. Each new hypothesis, along with its loss and boosting analysis, appended to the buffer Bi+1. ensuring subsequent prompts reflect the evolving understanding. This approach tightly inte- grates short-term error signals with long-term structural motifs. Though the per-hypothesis analysis adds computational overhead, it provides precise feedback that significantly improves sampling ef- ficiency. Our experiments show that this leads to faster convergence and superior solutions without substantial increase in total runtime when using the E2E (Kamienny et al. 2022 backend.
5 EXPERIMENTS
We present comprehensive evaluation of proposed framework, RESTART, designed to val- idate the effectiveness of its end-to-end crnoohsszze-xpprimeeee analyze loop. Our experiments demonstrate that its principled feedback mechanisms achieve consistently improved performance.

---

Published as a conference paper at ICLR 2026
LSR- Transfrroo
Biology
Material Science
Physies
NMSE
ACC
NMSE
ACC
NMSE
ACC
NMSE
ACC
DSR
0.472.1775
46.24..4225
0.206..II
16.67
Lwove
36.0014x.
0.104gam
27.27
EZE
2,098 -098
57.6624083
0.578=0400
12.50=33.7
0.008=000
76.00+3551
0.425
20.45+3464
PySR
0.175 200
72.07=44.5
0.003-0012
54.17=3050
0.057=0292
72.00+4533
0.004
73.86+3633
LLMDirect
0.355
32.88
0.454_.031
16.67
0.012
58. 38.001-17.1
0.099
36. 36.3644904
0.1601
72.32.4257
0.016
L.003
22000
0.002: aue
SGA
0.374
47.39.44.46
0.975=2257
12.50
4.021 #19.066
48.00=44..9
34.09245.46
RESTART
0.157=9.40
85.23+35m
Table 1: Performance comparison on datasets within training distribution. RESTART achieves the best or near-best performance across all datasets. For LLM--base approaches, the backbone model is Qwen3-8B (Yang et al. 2025).
5.1 EXPERIMENTAL SETUP
LLM-SRBench (Shojaee et al. 2025b is comprehensive benchmark designed to evaluate LLM- based scientific equation discovery methods beyond simple memorization. We focus on two key categories from this benchmark: i) LSR-Transfone which contains 111 problems derived from established physical models; ii) LSR-Synth, which includes 93 problems that combine known sci- entific terms with novel, plausible synthetic terms from biology, physics, and materials science domains (see details in Appendix
To contextualize the performance of RESTART, we compare it against representative baselines from major symbolic regression paradigms: PySR (Cranmer 2023) (GP-based), DSR (Petersen
al. 202T) (RL-based), E2E (Kamienny al. 2022 (mapping- based),, LLMDirect (Shojaee al.
2025b)
(LLM baseline), SGA Ma et al. 20244)
(LLM optimization), and LLMSR (Shojace et al.
2025ap (Current SOTA LLM-based SR). For LLM- based methods, we use Qwen3-8B (Yang al.
2025) as the default LLM backbone, unless stated otherwise. Hyperparameters follow the authors
public settings where available (see Appendix foror details).
5.1.1 EVALUATION METRICS
Following (Shojaee et al. 2025a b). we evaluate numeric fit quality using the normalized mean
squared error (NMSE): NMSE
Σ (y1-1) 7) where yı denotes the true target for test input æ;.
ds the corresponding model prediction, = No. Σx' yi the mean of the test targets, and Niet the number of test samples. To ensure numerical cubbility-patticnn
when exponential or logarithmic clip the NMSE at maximum of 100. This clipping prevents exploding errors and division-by-near-aan effects from skewing the averages, enabling more robust cross-task task comparisons.
terms cause extreme values the denominator approaches
Following common practice SR benchmarks (Shojaee et al. 2025a b), we report the Accuracy to tolerance (Acc,), which measures whether the discovered equation matches the ground-truth
expression across the entire input domain: Acc, = 1 mmxxi
0:-36 < with r 0.1 %
unless stated otherwise. This metric evaluates generalization by requiring uniformly low relative error across all data points, rather than just low average error, which can mask overfitting. To ensure robust performance estimates and mitigate the influence of pathological outliers, we follow (Kamienny et al. 2022 Biggio et al. 2021) by reporting results over the best 95% of predictions, discarding the worst 5% based on relative error.
5.2 MAIN EXPERIMENTAL. RESULTS
Research Question 1: Does RESTART discover accurate symbolic expressions from data? Quantitative results Table I show that RESTART achieves superior performance, with lower NMSE and higher accuracy than all baseline methods across nearly every benchmark. This estab- lishes its effectiveness in discovering accurate symbolic expressions. A focused analysis on the three regurs most challenging tasks from the Biology dataset shown in Figure reveals that RESTART's relative advantage is greatest under difficult conditions, particularly for problems BPG12 and BPG20. The minimal NMSE values suggest successful identification of the underlying equation's structure.

---

Published as a conference paper at ICLR 2026
6911
1400
RRD
(a) In- Distttiibuuion
(b) Oft-off ofDtstribbutii
Figure 3: NMSE of RESTART on four challenging problems under (a) in-distribution and (b) out- of-distribution settings. both settings, RESTART not only performs well on problems solvable by baselines but also yields notable improvements these complex tasks.
Biology
Material Science
Physies
ACC
NMSE
NMSE
ACC
DSR
29.900=43.34
36.67
7.348:30664
84.00-37:42
5.715
25.00:43.00
E2E
39.465:44..0 ow
4.17:t0041
1.395:2.:12
94.00-21.6
25.310:
21.59:1644
PySR
8.875#26.612
47.50:249.45
4.115::.0.00
96,00=20000
7.151==23081
62.50=43.30
LLMDirect
68.661=44906
12.50a33.75
1.704:33801
84.00-33.52
28
13.64:34.71
LLMSR
1.667211..m nM
45.83
2.0844.0..9
96.00
14,808 14 400
65.91
SGA
63.701 =455944
8.33
12.072-229997
78.00
33.4100:44299
14.77
RESTART
5.087:114.70
52.08
0.075
100.00
8.167
71.59
Table 2: Performance comparison 3 datasets (OOD results only). NMSE values clipped at a maximum 100. For LLM- based approaches, the backbone model is Qwen3-8B (Yang et al, 2025
Figure 4: Visualizing equations on challenging tasks. Each panel shows the ground -rout equation (black) and the outputs from LLMSR (red) and RESTART (blue). We select four challenging equations from the Biology dataset and evaluate the discovered equations over both in-distribution (blue shaded) and diitt--iisttiuttion (pink shaded) regions. RESTART not only matches the ground truth within the training range but also generalizes well to unseen regions, closely following the true functional shape.
Qualitative analysis confirms this: baseline methods frequently omit key terms, whereas RESTART's adaptive refinement iteratively incorporates them. This highlights the core benefit of our short-term guidance, where the exploration function provides data-driven signals for pre- cise corrections. For example, RESTART correctly identified the harmonic interaction in BPG12 as shown in Figure 4 a component missed by LLMSR. (An analysis of the refinement process is provided in Appendix
Research Question Does RESTART recover equations close the ground truth? On out- of-distribution (OOD) splits, RESTART achieves the highest accuracy across all domains as shown T This demonstrates that our method not only fits the training data well but also discovers equations that generalize effectively to unseen data.
Figure visualizes the equations discovered on a subset of challenging tasks. For problems like BPG18 and BPG20, baselines such as LLMSR may achieve low training error, but their solutions visibly diverge outside the training region. This divergence is often caused by overly complex terms that reduce local emror but distort the global function shape. In contrast, RESTART 's short- term guidance mechanism explicitly identifies missing operators or interactions, which prevents the model from fitting noise and yields robust equations. Furthermore, the improvement-gate
8

---

Published as a conference paper at ICLR 2026
(a) Search Trajectory
(b) Ablation Study (ID)
(c) Ablation Study (OOD)
Figure 5: (a) The median search trajectories on four challenging equations; (b) Ablation study on BPG12 under ID setting: (c) Ablation study on BPG12 12 under OOD setting.
structure library retains only validated components, preventing equation bloat and ensuring the final expressions are both concise and physically meaningful.
Research Question 3: Is RESTART computationally efficient? Although guidance mecha- nisms introduce overhead for subproblem solving and structure summarization, RESTART is signif- icantly more efficient discovering accurate equations. As shown in Figure RESTART's NMSE decreases sharply within the first 100 iterations, rapidly surpassing the performance of LLMSR. This swift convergence demonstrates that the cost of targeted analysis is substantially offset by major gain in search efficiency. By guiding the search toward promising regions instead of exploring blindly, RESTART drastically reduces the number of ineffective hypotheses evaluated.
Furthermore, Figure 3 shows that RESTART discovers better equations than other baselines using only 25% of the iterations. This underscores its ability to find strong. generalizable solutions with fraction of the computational budget, proving that our approach achieves a superior trade-off off be- tween analysis overhead and overall search effectiveness.
5.2.1 ABLATION STUDY
Research Question 4: What is the individual contribution of each key component: initializa- tion, short-term guidance, and long-term memory? We conduct an ablation study to evaluate the contributions of RESTART along three axes: (i) removing key components from RESTART, (ii) evaluating different L.M backhones, and (iii) using alternative subproblem solvers. The re- sults, shown in Figure 5b and [cc] confirm that all three components critical for peak perfor- mance. Removing any one- initializzaionn short-term analysis, or long-term retention-causes noticeable performance drop, validating the necessity of the full loop. Additionally, we evaluated an alternative variant of RESTART, termed Additive, whose pur- pose to test the behavior of the framework when the boosting step is restricted to fitting the classical additive residual. In this variant, the exploration function is defined as g IR" - IR and is fit as g(x) = y - fe(x). After solving this subproblem, we prompt the LLM with (fr, (x), y). St. (()(xx), y y-ffill fx))))
that the model receives both the current hypothesis and the residual- based signal for comparison with the full RESTART formulation. From Figures 55 and 5cc we observe substantial increase in NMSE under the Additive variant. This highlights that performance improvements in RESTART arise primarily from structural refinement rather than simply filling the numerical residual gap.
Furthermore, while the choice of LLM backbones affects absolute performance (with stronger mod- els yielding higher accuracy, as seen in Table 3), RESTART consistently outperforms other L.MM- based methods regardless of the foundation model. This indicates that its advantage stems from the algorithmic design rather than a specific model's capabilities.
Finally, as shown in Table we observe a trade-off in subproblem solvers: more expressive solvers can propose better structures at a higher computational cost. However, results Tables [ and show that traditional solver alone is insufficient. Optimal symbolic recovery-achiee its integration with RESTART 's iterative hypothesis refinement
both
accuracy and and guidance mechanisms. Details are provided in the Appendix B.3

---

Published as a conference paper at ICLR 2026
5.2.2
CASE STUDY
Research Question 5: How does RESTART perform on a real-world scientific task? Here, we consider a real-world scientific discovery task. Defining and measuring particle speed in the clas- sically forbidden region is crucial for testing Bohmian mechanics and understanding microscopic transport. Using coupled waveguides, [Sharoglazova et al. (2025) showed that the measured nondi- rectional speed follows / m, where is the particle speed, m the particle mass, and A the energy offset inside the step (A = Vo + fug)). This relation describes how speed increases with | in the forbidden region and is mimror-yymmetric in A. While the equation appears elementary,
the actual physical law used in the experiment is:
2 224-1.0021100 76634x 10-22 1000, which is
m
more challenging. It involves (i) nested nonlinear operators (absolute value within a square root), (ii) variall interaction through division, and (iii) mrggemmgnttude unit-conversion constants embedded inside nonlinear transformations, which makes constant recovery and operator ordering difficult for search-based SR.
To assess RESTART in a real-world scenario, we re- examine this physical law from only 46 experimen- tal data points, including basic unit conversions (c.g.. meV-J). As shown in Figure 6a and Figure 6b RESTART successfully rediscovers the relationship and produces two key expressions: RESTART-1 achieves the best fit by recovering the analytic scal- ing while adding compact, interpretable corrections (e.g., parameterized scaling and energy- energy-dependent terms) that may account for known systematics in the speed extraction process (the parabolic build-up fit can overestimate v by up to ~ 6.7% in the forbid- den regime (Sharoglazoss) lazova et al. 2025);; RESTART-2 more closely adheres to the published formula, sacri- ficing small amount of fit quality for greater gener- alizability and interpretability. These results demon- strate that RESTART highly data-efficient and prior- aware. Through short-term refinement, it can gener- ate targeted, testable corrections that help bridge the- ory and experiment, even with scarce, noisy data. To further validate RESTART 's capability scientific discovery, we report more challenging task in Ap- pendix G
Rt²
(equuttion)
2|
0.9642
1.000
RESTART-1
0.9827
0.9699
RESTART-2
0.9672
0.9985
LLMSR
0.9688
0.9958
LLMDIRECT
0.0
0.0
PYSR
0.8984
0.0
(a) Energy results table
4,200
Coeed temat
RESTAAT RESTAT
2,000
LLM5
1,002
030 nrge
Tair
-0.4
-0.2
0.9
0.2
0.4
bnergy (mav)
(b) Energy figure
6 DISCUSSION AND CONCLUSION
Figure 6: (a) Comparison two RESTART- discovered equations with the reported equa-
We presented RESTART, symbolic regres- sion framework that completes the hypothe- size-experiment-analyye cycle by combining LLM- guided hypothesis generation with structural analysis and persistent structure library. Our experiments demonstrate that the explicit analysis stage is critical: solving the subproblem ((f x) and using it guide the LLM with structural refinements, together with the structure library, leads to a more efficient and accurate search process. The physics case study further illustrates the potential symbolic regression for real scientific tasks.
tion = V /2 (Sharoglazova et al. 2025
m and basclines. (data)) denotes the R com- puted against the measured ground-truth data points, whereas St2 (equation) denotes the It² from points sampled via the reported equation. Both discovered equations achieve a higher It on experimental data while remaining consis- tent with the reported equation, suggesting plausible correction. (b) Visualization of three equations: RESTART-1 fits the data well but slightly diverges from the reported equation in the OOD range.
Remaining challenges include the computational cost of repeated LLM queries and the inability to directly verify the correctness of each stored structure. Future work will explore integrating auxiliary models to pre-fill filter structures and candidate hypotheses, as well as incorporating domain constraints to improve plausibility without increasing computational overhead.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The research work described in this paper was conducted the JC STEM Lab of Machine Learning and Symbolie Reasoning funded by The Hong Kong Jockey Club Charities Trust.
REFERENCES
Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurélien Lucchi, and Giambattista Parascan- dolo. Neural symbolic regression that scales. In International Conference on Machine Learning, 2021.
Miles Cranmer. Interpretable machine learning for science with PySR and symbollcregrrssionnji. arXiv preprint arXiv:2305.0758) 2023.
Miles Cranmer, Alvaro Sanchez Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. In Advances in Neural Information Processing Systems, 2020.
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Cheng- gang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. I. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li. Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang. Peng Zhang. Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang. Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shut- ing Pan. T. Wang. Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang. Wenjun Gao, Wenqin Yu, Wentao Zhang. Q. Li, Xiangyue Jin, Xianzu Wang. Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang. Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin. Y. K. Li, Y. Q. Wang. Y. X. Wei. Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang. Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijum Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. DeepSeek- -3 technical report. arXiv preprint urXiv:2412. 19437. 2024.
Renáta Duběáková. Eureqa: software review. Genetic Programming and Evolvable Machines, 12: 173-178, 2011.
Arya Grayeli, Atharva Sehgal, Omar Costilla-Reyes, Miles Cranmer, and Swarat Chaudhuri. Sym- bolic regression with learned concept library. In Advances in Neural Information Processing Systems, 2024.
Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Deep generative symbolic regression. In International Conference Learning Representations, 2023.
Kamienny, Stéphane d' Ascolli Guillaume Lample, and Francois Charton. End- to-end Symbolic Regression with Transformers. In Advances in Neural Information Processing Systems, 2022.
11

---

Published as a conference paper at ICLR 2026
MeereeAAexandie Kamienny, Guillaume Lample, Sylvain Lamprier, and Marco Virgolin. Deep Gen- erative Symbolic Regression with Monte- Conte-Carlo--e Carlo-Tree-Seeee SreecSS In International Conference on Ma- chine Learning. 2023.
John R. Koza and Riccardo Poli. Genetic Programming. Search Methodologies: Introductory Tutorials in Optimization and Decision Support Techniques, pp. 127-164. Springer US, 2005.
William La Cava, Bogdan Burlacu, Marco Virgolin, Michael Kommenda, Patryk Orzechowski, Fabrício Olivetti de França, Ying Jin, and Jason H Moore. Contemporary symbolic regression methods and their relative performance. In Advances in Neural Information Processing Systems, 2021.
Mikel Landajuela, Chak Shing Lee, Jiachen Yang, Ruben Glatt, Claudio P Santiago, Ignacio Ar- avena, Terrell Mundhenk, Garrett Mulcahy, and Brenden K Petersen. A unified framework for deep symbolic regression. In Advances in Neural Information Processing Systems, 2022.
Wenqiang Li, Weijun Li, Linjun Sun, Min Wu, Lina Yu, Jingyi Liu, Yanjie Li, and Songsong Tian. Transformer-basee model for symbolic regression via joint supervised learning. In International Conference Learning Representations, 2023.
Zelong Li, Jianchao Ji, and Yongfeng Zhang. From Kepler to Newton: Explainable ai for science. arXiv preprint arXiv:2111.122100 2021.
Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruchle, James Halverson, Marin Soljacic, Thomas Y. Hou, and Max Tegmark. KAN: Kolmogorov-Amoll networks. In International Conference on Learning Representations, 2025.
Pingchuan Ma. Tsun-Hsuan Wang, Minghao Guo, Zhiqing Sun, Joshua B. Tenenbaum, Danicla Rus, Chuang Gan, and Wojciech Matusik. LLM and simulation as bilevel optimizers: A new paradigm to advance physical scientific discovery. In International Conference Machine Learning. 2024.
Michael de la Maza and Bruce Tidor. An analysis of selection procedures with particular attention paid to proportional and boltzmann selection. In International Conference Genetic Algorithums, 1993.
Terrell Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P Santiago, Daniel Faissol, and Bren- den K Petersen. Symbolic Regression via Deep Reinforcement Learning Enhanced Genetic Pro- gramming Seeding. In Advances in Neural Information Processing Systems, 2021.
Robert Nola and Howard Sankey. Theories of scientific method: an introduction, Routledge, 2014.
Brenden K Petersen, Mikel Landajuela, T Nathan Mundhenk, Claudio P Santiago, Soo K Kim, and Joanne Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference Learning Representations, 2021.
Bernardino Rorere-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nature, 62577995):468 475. January 2024. ISSN 1476-4687.
Juan Salcedo-Gallo, Michiel Burgelman, Vincent Flynn, Alexander S. Carney, Majd Hamdan, Tunmay Gerg, Daniel C. Smallwood, Lorenza Viola, and Mattias Fitzpatrick. Demonstration of tunable non- non-Hermitian nonlinear microwave dimer. Nature Communications, 16(1):7193, August 2025. ISSN 2041-1723.
Violetta Sharoglazova, Marius Puplauskis, Charlie Mattschas, Chris Toebes, and Jan Klaers. En- ergy-speed relationship of quantum particles challenges Bohmian mechanics. Nature, 643(8070): 67-72, July 2025. ISSN 1476-4687.
Hongzhi Shi, Jingtao Ding, Yufan Cao, Quamming Yao, Li Liu, and Yong Li. Learning symbolic models for graph-structured physical mechanism. In International Conference on Learning Rep- resentations, 2023.
12

---

Published as a conference paper at ICLR 2026
Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K. Reddy. LLM-SR: Scientific equation discovery via programming with large language models. In Inter- national Conference Learning Representations, 2025a.
Parshin Shojace, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K. Reddy. LLM-SRBench: A new benchmark for scientific equation discovery with large language models. In International Conference on Machine Learning. 2025b.
Fangzheng Sun, Yang Liu, Jian-Xun Wang, and Hao Sun. Symbolic physics learner: Discover- ing governing equations via monte carlo tree search. In International Conference on Learning Representations, 2023.
Silviu-Marian Udrescu and Max Tegmark. Al Feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Runxiang Wang, Boxiao Wang, Kai Li, Yifan Zhang, and Jian Cheng. DrSR: LLM based scientific equation discovery with dual reasoning from data and experience. arXiv preprint arXiv:2506.04282. 2025.
Yilong Xu, Yang Liu, and Hao Sun. Reinforcement symbolic regression machine. In International Conference Learning Representations, 2024.
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo. Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint arXiv:2505, 093888, 2025.
13

---

Published as a conference paper at ICLR 2026
A LLM USAGE
We used LLMs to assist in the preparation of this manuscript. Specifically, LLMs were used for polishing the language, improving readability, and restructuring sentences for clarity. All conceptual contributions, experimental design, analyses, and results were produced and verified by the authors. The authors carefully reviewed and edited all model outputs before inclusion in the final manuseript.
B METHOD DETAILS
B.1 IMPLEMENTATION AND EXPERIMENT DETAILS
We implement RESTART using a population exemplar buffer, following the island-based
search paradigm Shojaee et al. 2025a Romera-Paredeer et al. 2024 Cranmer 2023). We maintain M 10 disjoint islands, each storing a diverse population of candidate hypotheses (ff(z)) to promote exploration. At each iteration, we first select an island Z uniformly at random, and then sample kexemplar exemplars from this island to assemble the prompt.
Within each island, we group hypotheses according to their loss T.(f( Groups are sampled proportionally to a Boltzmann distribution (Maza & Tidor 1993) over their losses:
cxp(LLf(=))/3e))
B. = To 1 u mod N
Pt EjEz (T(((T(((==)))0))
N
where denotes the number of hypotheses in island I, To 0.1 is initial temperature, and N = 30000 controls the annealing schedule. This scheme adaptively lowers the temperature as the island becomes saturated, gradually shifting from exploration exploitation. Within each group,
exemplars sampled with length- based probability, p!t" exp
len( where len(f.) is Sp
the program length and 3p = 1 is hyperparameter that encourages shorter and more interpretable expressions. Newly generated hypotheses inserted back into the same island for future iterations.
Additionally, experiments were conducted on two hardware setups: workstation with 2 RTX 4090 GPUs and server with 8xL40S GPUs.
Improvement-Gatte Scoring. To combine absolute and relative improvement signals, we define Sät w. A + w, R and w. w, 1, where A is the absolute improvement and R is the relative improvement. The weights wa and w, are determined piecewise according to the logarithm of the absolute improvement Atog log10 (A)) We then set:
If Alog 2 1: We = 0.7, wr 0.3.
0 < Atog 1: W. 0.5, w, 0.5.
If - < Alog < 0: W. 0.2, w, = 0.8.
If Alog 2: Wa 0.0, wr 1.0.
This piecewise schedule ensures that large absolute improvements are weighted more heavily, whereas very small absolute improvements rely primarily on relative gains be considered signifi- cant. After computing the fitted score Stit, we regard an improvement as valid if stt 2 a, a = 40.0, and trigger the structure distillation process. To control memory and maintain efficiency, we cap the maximum number of structures stored in the library K = 20.
B.2 LLM BACKBONE
We compare Qwen3 YYag et al. 2025) backbones of varying scales and DeepSeek v3.1 (DeepSeek- AI et al., 2024) to assess how the generative prior affects symbolic search performance. Across most backbones, our method consistently outperforms LLM-SR, demonstrating the effectiveness of the structure analysis and targeted refinement. Stronger and reasoning-enablee models yield more accurate candidate programs, leading to improved equation discovery, but they also incur substantially higher computation cost. This comparison highlights clear eecuracy-compute trade- off: smaller backbones offer a good balance of speed and accuracy, whereas reasoning or larger backbones can further boost performance when additional compute budget is available.
14

---

Published as a conference paper at ICLR 2026
Qwen3-1B
Qwen3-8B
Qwen3-8B-thin
00D
OOD
OOD
LLMSR
0.49983
0.69452
0.20248
0.37592
0.90352
1.22674
RESTART
0.45502
0.77328
0.02254
0.01442
0,00801
0.01103
Owen3
328
Qwen Flash
DS Chat
ID
OOD
1D
OOD
1D
000
LLMSR
4.12=110M
3.09x10-1
0,01499 1.76x000
0.02082
0,02884
0,04199
1.34110
1,46x10"
3
1.0x10""
7
Table 3: Comparison of LLMSR and RESTART across different LLM backbones on BPG18.
OOD
Average Evaluation Time
LLM
0.02683
0.01481
6,418
Polynomial
0.47034
0.65663
0.580
KAN
0.03668
0.01963
27.651
DSO
0.01486
0.00849
59.91
PySR 0.00122
0.00096
14.153
E2E
0.02254
0.01442
6.587
Table 4: Comparison different symbolic backhones BPG18, along with the average time for the exper- iment and analysis stages per iteration (primarily subproblem solving). Note that the average time includes invalid evaluations, which contribute values close to zero.
B.3 SUBPROBLEM SOLVER BACKEND
Table provides a detailed comparison of candidate backends for solving the symbolic subprob- lem defined in Eq. (3). We evaluate representatives from mapping- -basid methods (E2E), search- based methods (PySR, Polynomial, DSR, KAN), a simple polynomial regression baseline, and direct L.L.-based solver that is prompted with the dataset {(f(x), X, y)} to generate new equa- tions. This LLM-based solver differs from the Llmdirect baseline in that it is data-aware rather than data-blinded. This comparison isolates the contribution the subproblem solver from the rest of the RESTART pipeline and quantifies its impact on both accuracy and runtime.
Importantly, the choice of backend does not change the formulation of Eq. (3); it only affects the quality and efficiency of the discovered refinement Jt- This volver-agnosti design keeps the frame- work modular and allows it to benefit from future advances in symbolic regression without altering its overall search procedure.
Overall, we observe a clear comprae trade-off: more expressive search-based solvers (e.g.. PySR, DSR) achieve lower NMSE but incur significantly higher computation cost, whereas fast mapping-based methods (E2E) sacrifice some accuracy in exchange for much higher through- put. Consistent with the results in Tables | and optimal equations. The best performance is obtained when these solvers are embedded within the iterative bypothesize-experiner experiment--aallyee cycle of RESTART, where they provide targeted refine- ments rather than attempting to solve the entire problem from scratch.
single solver alone insufficient recover
C RESULT DISTRIBUTION
Following the setup in LLM-SRBench (Shojaee et al.) 2025b), report box plots (Figure 7) that illustrate performance variations across the LLM-SRBench datasets for different methods, highlight- ing the distribution of the experimental results.
15

---

Published as a conference paper at ICLR 2026
107
100
10-
10
50
10a4
10
LSM-tranal om
Physics
Material Science
Bidlogy
RESTART
5PA
LUMOR
LLMDING
E30
OSO
Figure 7: Distribution of in-distribution NMSE across all benchmark problems
D BASELINE METHOD DETAIL
D.1 DSR
DSR Petersen al. 2021) formulates symbolic expression generation sequential decision pro- cess in which policy network constructs expressions token by token. DSR employs a risk-seeking policy-gradient objective that emphasizes learning from top-reward trajectories. Additionally, DSR trains a new model for each dataset. The hyperparameters are presented in Figure & and are used as the default settings unless otherwise specified.
D.2 E2E
Kaamienny et al. 2022) employs Transformer encoder-decoder that maps tokenized data directly to a symbolic expression string. The model supervised on synthetic corpora to learn correspondences from data distributions to formula structures. During inference, autoregressive decoding with beam search outputs parseable expressions, followed by parameter fitting for numeric accuracy. The hyperparameters presented in Figure and are used as the default settings unless otherwise specified.
E
D.3 PYSR
P Cranmer 2023) uses population- genetic programming framework that represents candidate models symbolic expression trees composed of primitive operators. It iteratively ap- plies mutation, crossover, and island migration to evolve diverse candidates. multi-objective criterion balances data-fit and expression complexity and maintains Pareto front of solutions. The hyperparameters are presented in Figure TO and are used as the default settings unless otherwise specified.
D.4 LLMDIRECT
As suggested in Shojaee et al. (2025b), we include a data-blind zero-shot LLM baselind that prompts to produce syntactically valid equations purely from its pretrained knowledge, without ditioning on the dataset. relies on built-in mathematical priors and general reasoning to propose
Im-srbench
16

---

Published as a conference paper at ICLR 2026
der_params:
task:
task_type: "regression"
function_set: ["add", "sub", "mul", "div", "sin", "cos", "exp",
log"]
metric: "inv_nrmse"
metric_params: [1.0]
extra_retric_test: null
extra_Metricctest__pe [1
threshold: 1.0e-12
protected: false
reward_ncise: 0.0
"="
normalize_variince:: false
@eclsloh.tre__trreiiid...stt []
training:
n_samples: 1000000
batch_size 10000
epsilen: 0.05
1_cores_batch: 50
policy_optimizer::
Learning_rater 0.0005
entrapy_weigtt 0.03
entropy_gamma: 0.7
prior:
length:
min_: 4
max_ 64
"on": true
repeat:
tokons: "const"
min_: null
max_:
"on": true
inverse:
"on": true
trig:
"on": true
const:
"on": true
ne_inputs:
"on": true soft_length:
loc: 10
scale:
"on": true
domain_range:
"on" : false
Figure 8: DSR hyperparameter configuration.
o2c_params:
model path: "c2c.pt"
max_input_points: 200
n_trees_to_refinee 10
rescale: true
max maa__num_sampless 2000
Figure E2E hyperparameter configuration.
17

---

Published as a conference paper at ICLR 2026
pasaass
niterations: 40
maxsize: 30
populations: 15
population_size:
ncycles_per_iteratine 550
binary_operators: ["+", "*", n_", "/", "C"]
unary_operators: ["cos", "exp", "sin", "sqrt", "log"]
batching: true
batch_size: 5000
constraints:
**"; [-1, 20]
"exp": 20
"log": 20
"sqrt":
"sin": 10
cos" 10
Figure PySR Hyperparameter Configuration
diverse symbolic forms. The hyperparameters are presented in Figure 11 and are used the default settings unless otherwise specified.
11n rect_params:
samples per prompt: 5
Figure MyAMDDDrachhyeeppraaaaaa configuration.
D.5 SGA
SGAB 6 (M et al. 2024) adopts bilevel setup in which an LLM-based agent proposes symbolic structures while a separate optimizer fits the continuous parameters of each proposal. It also uses two different temperatures for LLM generation separate exploitation and exploration. We follow the default hyperparameter settings suggested by Shojaee et al. (2025b) without additional modifi- cations.
D.6 LLMSR
Shojaee et al. 2025a) is a sfate-of-thh-art LLM-driven evolutionary method that generates equation candidates code and refines them through LLM queries. It stores generated equations in populations; at each round, LLMSR randomly selects one population and samples equations to construct new prompt. The hyperparameters are presented in Figure 12 and are used the default settings unless otherwise specified.
LLMSR
Limsr_params:
num: 1000
per_ pe___prrrrrr::: num num_islands: 10
semples_per_prong
Figure 12: LLMSR hyperparameter configuration.
18

---

Published as a conference paper at ICLR 2026
E LLM-SRBENCH
LLM-SRBench (Shojaee et al. 2025b) is a benchmark designed for the systematic evaluation of symbolic regression (SR) methods, with a particular focus on leveraging large language models (LLMs) for scientific discovery. Unlike previous benchmarks such as SRBench (La Cava et al.) 2021), where LLMs may simply recall the ground-truth equations from memory- thhmory-thus bypassing the core purpose of the symbolic regression task-LLM- LLLM-SBBench is constructed to be challenging, diverse, and representative of real-world scientific inference problems. Its design encourages LLMs to move beyond rote memorization and to demonstrate genuine reasoning and equation-discovery capabilities. The benchmark consists of two major categories:
LSR- ASR-Transform:
This category contains 111 problems derived from established physical models such as kinematics, dynamics, and thermodynamics. Each problem is generated by systematically transforming known governing equations, for example by switching input and output variables, rearranging terms, adding noise to simulate experimental condi- tions. This construction requires models to reason about the underlying functional relation- ships rather than merely recall canonical forms, thus providing a robust test of model's capability for true equation discovery.
2. LSR-Synth: This category focuses on evaluating generalization. It includes problems that combine well-known scientific quantities with synthetic but physically plausible terms, ensuring that the solutions cannot be solved by rote memorization. The tasks span three scientific domains: biology, physics, and materials science. The presence of novel terms forces SR models to infer correct functional forms based on data patterns, simulating real- world scenarios where researchers encounter new variables unmeasured effects.
Each problem in LLM-SRBench provides training and testing data sampled from the ground-truth function, as well as standardized evaluation metrics such as normalized mean squared error (NMSE). Together, these tasks cover both interpolation and extrapolation settings, making ALM--RBBnccl comprehensive and challenging suite for benchmarking LLM-based SR algorithms.
F AVERAGE TRAJECTORY
we reported the average NMSE trajectories for four biology equations. Note that because NMSE values differ on log scale, the mean can be dominated by the higher-NMSE trajectories. For example, if there are trajectories- one with NMSE 1 x 10-1 and nine with NMSE 1 x 10-9 the average NMSE will still be above 1 x 10-5, which is substantially higher than most of the trajectories.
As shown in Figure 13
G CASE STUDY
Here we report the discovered RESTART-1 and RESTART-2 in Figures 14 and 15
Method
R (equaaton
RESTART
0.9771
PYSR
0.5325
LLMSR
0.1737
LLMDDRECT
0.00
Table 5: Performance on the nonlinear microwave- equation.
Additional Case Study. To further evaluate RESTART on a challenging real scientific discovery setting beyond low-order algebraic laws, add second study based nonlinear Hermitian microwave dimer model from recent experimental physics (Salcedo-Galde et al. 2025):
G//2 (C3 + sin(o/Ca) Cs Ca
(, G)
C7
where C; constants. The result is shown in Table and Figure 16
19

---

Published as a conference paper at ICLR 2026
Step NMSE on (BPG 2 18
BPG12
DPG16
10a1
10
20"1
1822
10
RESTART
RESTART
LJPBR
730
4M
630
990
1000
200
A03
660
800
1050
Skep
Stap
RPG7
RPG70
RESTART
30"
10-
10
1802
10"7
30"1
12a
10a
20: 13
LLMSR
200
Figure 13: Average NMSE vs. Step on 4 Biology Equations, with shaded standard deviation.
H RESTART SEARCH PROCESS
Figure 17 compares the population growth rate equations discovered by RESTART and the base- line LLMSR. RESTART successfully recovers both key components of the ground-truth BPG12 equation: (i) the harmonic oscillation term 0.877 P sin(0.567t) capturing seasonal fluctuations, and (ii) the logistic growth term 0.701(1 P/65.75 75) with a realistic carrying capacity. In contrast, LLMSR produces a much noisier expression with only a weak logistic component and no harmonic interaction, yielding lower R. on both ID and OOD data.
To understand why RESTART succeeds, we inspect the prompt compptreesonss pair in Figure [8 Fig- 19 The prompt contains not only the current equations but also the top-ranked structures, two of which-Harmonic Oscill 1illlal ator Interaction on and Seas sonal Variat iirtiiiin with Dampi somping-are ng-are precisely the building blocks needed for the ground-truth solution.
This targeted guidance allows the LLM to synthesize a new hypothesis, equat ion_v2, that directly combines the harmonic interaction and logistic terms. As result, the MSE dramatically decreases from 52.35 (for exemplar equation_ 11) to 2.67 x 10-9, effectively achieving near-perfect symbolic recovery. This case highlights how RESTART's iterative refinement and structure guided prompting transform an initially rough approximation into a scientifically valid and highly accurate equation.
I PROMPT
Figure 2 illustrates the design of our adaptive prompt, which is dynamically assembled at each itera- tion of the hypothesize- experiment-anaal cycle. Unlike static few-shot prompting, our approach builds the prompt from three components that evolve over time:
Task Definition: A fixed preamble that sets the LLM's role, clarifies the objective (finding executable mathematical function), and specifies the variable description for given task.
Few-Shot Exemplars: One or more previous hypotheses are inserted, along with their mean squared error, and the corresponding boosting equation analysis. These exemplars serve as in-context refinement guidance.
Accumulated Knowledge: Finally, we include the structures from the structure library reusable building blocks directly inserted into the comment section for next generation.
20

---

Published as a conference paper at ICLR 2026
def RESTART-1 (Delta: np.ndarryy, mass: np. np.ndarray, parans: np. sd.rrarray ap. ap.naaraa:
Step 1: Convert Delta from mev to Joules Delta_d Delta 1.602176634e-21 $ mev - 1.602176634e-21 J
Step Base velocity using WKB approximation (m/s) velocity_base np.sqrt (2 np.abs (Delta_J) mass)
4 Step 3: Convert velocity to km/s velocity_km_per__ per_s base 1000
Step 4: Apply parameterized sca acaling velocity_km__er__ km__y_km______ =- params[O) [0]
Step Phase-modulated nonlinear corrections
+ Phase-dependent sine and cosine corrections
velocity_km_per_e +- params [1] (0.001 * Delta)
velocity_kn_per_e += params [2] np.cos (0.001 Delta)
Hyperbolic tangent correction with tunable phase phase_tanh 0.1 0.05 velocity_km_perr_
velocity_km_per_e Baciity_km_______ +- params np.tanh (phase_tanh)
Nenlinear energy-dependent correction
energy_tern velocity_km_pea km__tty___________ += params [4] energy_term
(0.5 + 0.1 Delta)
Additional nonlinear transformation stabilize variance This term includes tunable energy-dependent phase and amplitude
params [5] Delta + params [6]
velecity _m___e_____________ += params [7] sqrt (90.0 0.9 np.abs (2.0 30.0 np. co.cos (phase_energy)))
$ Optional: Add nonlinear feedback term to stabilizo variance
$ This is an advanced term that can be used for more complex models if (params) 8:
feedback
term term params 88 np. (1 np. (Delta)) (1 np .abs (Delta))
velocity_km per_s += feedback term
return velocity_km_per.
Figure 14: Discovered RESTART-1
21

---

Published as a conference paper at ICLR 2026
def RESTART (Delta: ap.ndarray, mass: Ep..darray, parans: np. ad.rrra
-> np. odanrayyy::
Constants
hbar 1.0545718e--8
eV_to_ = 1. 602176634ee11 # eV = 1.602e-19 J km_per_m. 1e-3 km 1000
$ Convert Delta from meV to Joules
Delta_J = Delta 1e-3 eV_to_ # mev to eV to J
Extract tunable parameters decay_rate params [0] 4 8
amplitude - params [1] 4 A
oscillation freq params [2] #
phase shift params
quadratic_coefe coeff params [4] E
energy_scaling = params [5] S
velocity_scaling = params [6] # K
offset params [7]
+ Compute the exponential decay factor
sqrt_ np. sgrsqrt (np.abs (Delta_ī))
decay_factor np.exp (-decay_rate sqrt_Delta_J)
Compute the oscillatory modulation oscillation no.cos (oscillation freq x Delta__ + phase_shiie shift
Compute the quadratic correction tern guadratic_coeff * (Delta_J ** 2)
quadratic_tera
& Compute the energy scaling factor energy_term. 1 energy_scaling Delta_J
$ Compute the velocity of probability flow velocity =
ampl implitude: decay_factor oscillation quadratic_term energy_term
(sqrt_Deita_ Jeita_J np. sqrt (mass)) velocity km per_er_m
+ offset
return velocity
Figure 15: Discovered RESTART-2
22

---

Published as a conference paper at ICLR 2026
Ground PySR
3.5
1.5
25
1.0
Ground Amplhuube
2.0
0.5
0.0
Phase (rediars)
Gain (d8)
Figure 16: 3D comparison of amplitude trajectory under different methods.
def RESTART BPCT_BBG1222 params):
T, a, omega, phi, theta, gamma, alpha, beta params
r_t - (1 + np.sin (omega + phi)) np.exp (-gamma
beta)
K_t = (1 + np.sin (omega + theta))
harmonic (np.sin (omega phi) P) np. (-ganna beta)
growth rate r_t P (1 K t) alpha harmonic beta return growth_rate
def
(t, P, params)
к0, alpha, beta, eps, omega, eta, gamma, zeta, phi params - - K0 eta
logistic
time_lineat alpha tt
seasonal eps np.sin (omega t) nonlinear = gamma t ** 2
delayed P np. roll.11(F, (P, int (phi)) ); delayed :int (phi)
feedback zeta
(1 - K) sp.sin (omega (t phi)) (1
0.1 + op.sin (omega + t))
dF_ct = legistic + time_linear beta seasonal + nonlinear
feedback
return dP_dt
Figure 17: Comparison of RESTART and LLMSR discovered functions for BPG12. The ground equation is f (t, = 0.877 P sin(0.567)) + 0.701 (1 P/65.75)P. two-term structure combining a sinusoidal modulation and logistic growth term. RESTART correctly recovers this structure. while LLMSR introduces additional linear, quadratic, and delayed-feedback components that deviate from the form.
23

---

Published as a conference paper at ICLR 2026
import numpy as np
MAX_NPARAMS 10
params [1.0] +MAX_NPARAMS
def equation v0 v.....):
.*.
Analysis for current (E)
Given (X, y), Current equation (f) has MSE: 52.35
Boosted equation candidate: (1(2.25 (-1.50 + 0. 0.0236+F)
...) ... with MSE: 72.82
def equation_vl(i v...).
**n
Improved version of 'equation_vO', Current equation (f) has MSB: 52.35 Boosted equation candidate: ((10.994 (0.078 10044*cureet__ppc.
1) ...) ...
with MSE: 770.39
Top scoring conceptss for the problem:
Harmonic Osci 1lator Interaction (fit: 95/100)
Damped Harmonic Interaction (fit: 88/100)
Seascnal Variation with Darping (fit: 85/1 85/100)
Figure 18: Partial prompt content used in the key iteration. It includes current loss, boosted residual equation, and top-ranked structural concepts.
def equation_ye (t:: nd.ndarray, P: nd.ndarray, parama: np.ndarray) >> np.ndarray:
Improved version of the population growth zate equation that incorporates:
Damped harmonic oscillation for seasonal variation
Logistic growth with time-dependent carrying capacity
Expenential damping
Harmonic interaction between population and dime-dependent parameters
theta, gamma, alpha, beta params r_t =rr (1 a ap.sin(onee
r, omega, phi,
phi)) ap.exp (-gamma
beta)
K_t - K (1 A np.sin (omega t theta)) inp.sin
barmenic_interaation gamma t beta)
phi) P) np.exp
growth rate rrt
K.t) alpha
harmonic interaction harmonic_interactiioo return growth_ratt rate
beta
Figure Partial L-geeraerteed response (equation_v2) that recovers the pear-ground-trrg truth BPG12 structure.
24

---

Published as a conference paper at ICLR 2026
This step adapts the search to accumulated knowledge and focuses exploration promis- ing directions.
This dynamic construction allows RESTART to incorporate both short-term error feedback (via boosting equations) and long-term knowledge (via the structure library) into each prompt.
From the optimization perspective introduced in Section 2 this adaptive assembly can be viewed as an approximate solution Eq. equation 2 instead of using fixed prompt p that may bias 9p(-) ward syntactically similar but suboptimal regions of the hypothesis space, we iteratively update by injecting exemplar feedback and discovered structures. This optimizaion procedure progres- sively shifts the induced distribution qp toward regions of lower empirical loss, thereby increasing the probability that the best-of-k sample achieves smaller loss ((T).
25


Reviews:
[{"review_id": "HD0z5Ag6CQ", "reviewer": "Reviewer_NT3Q", "summary": "This paper presents RESTART (Robust Equation Structural Learning for Symbolic and Causal Discovery), a unified framework aimed at closing the loop of scientific discovery by coupling symbolic regression, robust structural learning, and long-term knowledge accumulation. \nThe authors evaluate RESTART on LLM-SRBench across Physics, Biology, and Materials Science domains, as well as synthetic causal-discovery settings with noisy or adversarial perturbations. Results show that RESTART outperforms state-of-the-art GP-, RL-, and LLM-based symbolic regression systems in both in-domain accuracy and out-of-distribution (OOD) generalization, while remaining more robust to noise and spurious correlations.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The integration of symbolic regression with robust structural learning is good. The paper operationalizes the full scientific-discovery cycle, something most prior works treat only partially. The two-level guidance mechanism (boosting + structure library) provides a compelling framework for iterative theory refinement.\n2. Experiments span multiple scientific domains, several baseline categories (GP, RL, mapping, LLM), and realistic noise regimes. Ablations and the physics case study demonstrate how each component contributes to performance and stability. The OOD gains are particularly meaningful for genuine scientific discovery.", "weaknesses": "1. While the integration is original, the constituent elements (boosting-based residual learning, NOTEARS-style differentiable structure learning, structure libraries) each draw heavily from prior art. Clarifying what RESTART fundamentally adds beyond combining them would strengthen the contribution.\n2. The two-layer optimization (boosting subproblems + adversarial robustness) roughly doubles compute. Table 4 lists per-solver times but lacks an end-to-end comparison normalized by total wall-clock or FLOPs. Claims about “25 % fewer iterations’’ are anecdotal without systematic cost analysis.\n3.. RESTART introduces many tunables (fitness-gating, adaptive weights, divergence parameters, temperature schedules). The paper does not analyze sensitivity or provide heuristics for new domains.\n4. The paper highlights success stories but gives little insight into when the approach fails.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gPiFkdVfio", "reviewer": "Reviewer_PQgs", "summary": "The paper “Robust Equation Structure Learning with Adaptive Refinement (RESTART)” presents a novel framework for Symbolic Regression (SR) that explicitly models the full loop of hypothesis → experiment → analysis → refinement using large language models (LLMs).\n\nWhile prior SR methods have typically lacked a distinct analysis stage, RESTART introduces a refinement mechanism driven by error analysis, making it the core of its contribution.\n\nThe system consists of three main modules:\n1. Informative Initialization – Uses existing mapping-based estimators (e.g., E2E) to generate initial hypotheses.\n1. Targeted Refinement (short-term guidance) – Analyzes residuals and guides the LLM to make localized improvements.\n1. Long-term Structure Library (long-term guidance) – Stores reusable symbolic structures as executable code snippets.\n\nExperiments on LLM-SRBench (covering physics, biology, and materials science) show that RESTART achieves 20–30% reduction in NMSE, +5–10% improvement in Symbolic Accuracy, and better OOD generalization than baseline SR systems.\n\nA case study also demonstrates the rediscovery (with correction) of a physical law reported in a Nature paper (July 2025 issue), illustrating RESTART’s potential for real-world scientific discovery.", "questions": "1. How does the inclusion of the Analyze phase quantitatively improve Symbolic Accuracy (beyond NMSE)?\n1. Can you guarantee that the LLMs used were not trained on the *Nature (2025)* article or its equation?\n1. How is the quality of the Structure Library maintained—how do you prevent incorrect or redundant structures from accumulating?\n1. Do you plan to include structural metrics (e.g., symbolic edit distance, tree isomorphism rate) in future evaluations?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Formalizing the “Analyze” stage in SR\n\n   * RESTART defines “analysis” as an explicit symbolic sub-task ( g_t(f(x), x) ) that learns residual errors and guides refinement in a *boosting-like* manner.\n   * The LLM is not just a generator but an *autonomous analyst*, producing targeted corrections based on error structure—a clear conceptual advance.\n\n2. Two-level guidance design (short-term and long-term)\n\n   * The model separates immediate error-based refinement from cumulative structure learning.\n   * Unlike multi-agent or evolutionary systems, RESTART combines *directionality* (via residual analysis) with *knowledge reuse* (via a library).\n\n3. Comprehensive baseline comparison\n\n   * Evaluated against search-based (e.g., GP), mapping-based, and LLM-based SR methods (LLM-SR, LaSR, PySR, AI-Feynman).\n   * RESTART shows consistent improvements in both ID and OOD conditions, supporting its analytic-guided hypothesis refinement.\n\n4. Empirical validation through case study\n\n   * Successfully rediscovered a quantum particle energy–velocity relation from *Nature* (2025/7/3).\n   * This demonstration links RESTART’s symbolic reasoning to genuine scientific modeling tasks.", "weaknesses": "1. Ablation study limited to NMSE only\n\n   * Figures 4b and 4c report NMSE-based comparisons only.\n   * The analysis does not include *Symbolic Accuracy* (structural correctness of equations), which is crucial for validating RESTART’s core claim—enhanced symbolic understanding.\n\n2. Contribution of “Analyze” phase not isolated for structure accuracy\n\n   * While NMSE degradation is shown when omitting the Targeted Refinement module, the improvement in symbolic structure accuracy is not quantified.\n\n3. Potential data leakage concerns\n\n   * The *Nature* case study uses data from July 2025, but the paper does not ensure that the LLMs used (GPT-4o, Qwen3, DeepSeek) were trained before that publication.\n   * Without such guarantees, the rediscovery results risk reflecting memorization rather than inference.\n\n4. Lack of theoretical grounding\n\n   * The residual decomposition function ( g_t ) lacks convergence analysis or formal justification.\n   * No quantitative discussion on search space reduction induced by the Analyze stage.\n\n5. Unaddressed SR-specific pitfalls\n\n   * Generated equations can still be overly complex (spurious terms).\n   * The paper does not explain how the Structure Library maintains the validity or consistency of stored code snippets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hEqv8GVM3O", "reviewer": "Reviewer_NteJ", "summary": "This paper proposes RESTART, a robust large-language-model-based framework for symbolic equation discovery that integrates principled short-term and long-term analytical guidance into the discovery process. The authors evaluate RESTART on 239 problems from the recent LLM-SRBench benchmark and report substantial improvements over the state-of-the-art baseline LLM-SR. The paper is generally well-motivated, clearly written, and supported by extensive experimental evaluation and insightful analyses.", "questions": "check weaknesses section", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Although individual ideas (boosting-style refinement, structure library) are not novel alone, their integration into the discovery framework is novel and seem to provide considerable performance improvement. \n- Experiments on a wide range of datasets from LLM-SRBench are thorough and demonstrate consistent gains over state-of-the-art baselines.\n- The paper includes comprehensive ablation studies and a meaningful real-world physics case study which strengthen their evaluation.\n- The paper is generally very well written and easy to follow.", "weaknesses": "- I noticed that authors are only reporting results on 93 out of 128 LSR-Synth from the llm-srbench benchmark. why is that and what criteria is behind this selection of 93 out of 128 problems?\n\n- In pg 6, Acc₀.₁ is described as the symbolic accuracy from [1], but it is actually a stricter numeric metric rather than the symbolic accuracy (SA), which in [1] is computed via an LLM-as-judge for symbolic equivalence. I would suggest authors to also report SA in their results (see fig. 11 in [1]).\n\n- The analysis in case study of sec 5.2.2 on real experimental data is insightful, however, with the current report of results it's not clear what are the symbolic forms of equation discovered by RESTART and how's the advantage of these new hypothesis compared to the ones from LLM-SR baseline. Providing symbolic forms similar to Figure 12 for some examples or at least this case study could be helpful to better understand the problem.\n\n- Some abrupt jumps in Fig. 4a suggest high variance in the observed results. Are these from a single run or averaged over multiple runs? Can you also report the confidence of these curves across different runs? \n\n- Authors have not shared to the anonymous version of code in their submission. It would be good if authors can also share their code if paper is accepted so that research community can also build on it.\n\n**Minor Comment:**\n- It would be helpful to include the LLM backbone model in the captions of Tables 1 and 2 for clarity.\n\n[1] LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models, ICML 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i7QZvvTark", "reviewer": "Reviewer_bYpV", "summary": "This paper introduces RESTART, an LLM-based symbolic regression framework that combines LLM-based hypothesis generation with several guidance mechanisms: (1) efficient initialization, (2) short-term refinement via boosting-style residual analysis, and (3) long-term knowledge retention through a structure library. Experiments on LLM-SRBench show better performance compared to baselines including LLMSR, with good efficiency gains.", "questions": "1. Most SR backends like E2E don't condition on f_t(x) form. Is g actually receiving f_t(x) or just fitting x to the residual? What exactly does each subproblem backend receive?\n\n2. How does the method handle cases where the missing structure isn't additively separable (e.g., f*(x) = h(x) · k(x))? Can you provide benchmark examples and show if performance degrades on such problems? Intuitively, the residual model would not work well in such cases and would suggest functions that are not appropriate. This might guide the framework towards predicting more complex linear forms (potentially with nonlinear basis functions). To be clear, even with this limitation, the additional component could be valuable in many use cases, but it would be helpful to discuss this in the paper and provide analysis on the effect on final discovered forms.\n\n3. Can you report symbolic accuracy metrics (e.g., symbolic recovery like in LLM-SRBench) alongside numeric error? Can you show performance distributions across all problems rather than just aggregated means?\n\n4. For the case study: what was the exact prompt used? How do baselines like LLM-SR perform on the same problem? The problem is published in 2025, after the cutoff date of many LLMs, but how do you rule out that the LLM is recalling or deriving the simple equation form rather than discovering it?\n\n5. What does \"Additive\" mean in Figure 4? Why are the ablations conducted only on 2 problems, and how are they selected? Is there an ablation on the fitness score design (absolute, relative, and weighted s_fit)?\n\nSuggestions:\n* I would suggest authors to bring examples of prompts and how the two mechanisms work to the main paper for clarity. \n* I was wondering if the authors could provide the code for reproducibility and evaluation.\n* minor: some typos exist in the paper (for example: \"Libarary -> \"Library\" in Figure 1)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The framework makes intuitive sense: fitting symbolic functions to residuals provides data-driven feedback rather than generic error signals, and the structure library of motifs offers a principled way to accumulate good patterns.\n* Good evaluation setup and empirical results on LLM-SRBench problems. The efficiency analysis showing comparable performance with 25% of iterations is valuable. Figures are well-designed and good ablation studies are provided in the main paper and the appendix.\n* The writing is clear, literature coverage is reasonable, and the appendix provides useful implementation details. The problem motivation around completing the scientific discovery cycle is well-articulated.", "weaknesses": "* Limited novelty: The contribution consists of fairly straightforward additions to the LLM-SR framework: pre-trained initialization, residual fitting for guidance, and a pattern library. While sensible, these feel incremental: residual analysis resembles gradient boosting adapted to SR, and structure libraries conceptually overlap with recent concept library approaches, though implemented differently.\n* Residual fitting assumption: The boosting formulation fits g(x) to the residual (y - f_t(x)), which appears to assume the missing component is additively separable from the current hypothesis. This may not hold when the true structure involves multiplicative or other nonlinear compositions with respect to f_t(x). See questions for details.\n* Evaluation gaps: Only numeric accuracy metrics are reported (NMSE, Acc_threshold), not symbolic correctness. Variation across problems isn't visualized beyond 4 selected examples. The case study lacks baseline comparisons and doesn't provide the prompt used. Several design choices (fitness scoring, ablation scope) need better justification. See questions for specifics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BqI5gfx7q1", "reviewer": "Reviewer_APP1", "summary": "The paper proposes RESTART (Robust Equation STructure learning with Adaptive RefinemenT), a symbolic regression (SR) framework that explicitly closes the hypothesize → experiment → analyze loop. \n\nIn experiments, RESTART reports lower NMSE and higher symbolic accuracy than existing methods.", "questions": "Please try to address the concerns raised in the previous section and provide a revision of the paper during the rebuttal.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method achieves better performance than existing LLM-based symbolic regression systems.\n\n- The modular design that separates initialization, refinement, and structure retention could enable future extensibility.", "weaknesses": "Overall, this paper suffers from jargon-heavy and somewhat hand-wavy writing. Many parts need clearer, more concrete explanations and consistent notation. Below are detailed comments for revision:\n\nLine 82: The term “structure proposal” is unclear. Please define explicitly what constitutes a structure—e.g., is it an equation template, code snippet, or symbolic expression tree?\n\nLine 83: Define “equation exemplars.” Are these example equations used in few-shot prompts, or previously discovered hypotheses stored in a buffer?\nLine 84: The notation for ( q_p ) and ( \\tau_{1:k} ) is confusing. Replace ( \\tau_{1:k} \\sim q_p^k ) with the clearer and standard form ( \\tau_i \\sim q_p,\\ i=1,\\ldots,k ). Avoid the unconventional ( q_p^k ) notation.\n\nLine 85: Explain “best-of-k” clearly—does it refer to selecting the lowest-loss sample among ( k ) generated equations?\n\nLine 146: Remove the period in the section title. Also, “EFFICIENT INITIALIZATION” is vague—rename it to something more descriptive, such as “Initialization via Mapping-Based Estimator.”\n\nLine 150: Specify what nonlinear and cross-variable dependencies are being modeled. Clarify what “using another estimator” means here—does the initialization merely provide a stronger approximation, or is it computationally more efficient?\n\nLine 158: Equation (3) is syntactically incorrect. Replace ( g_t \\in \\arg\\min_g L(g(f_t(x),x)) ) with the correct assignment form ( g_t \\gets \\arg\\min_g L(g(f_t(x),x)) ). Additionally, clearly define ( t ) (iteration index) and ( g_t ) (the symbolic residual or exploration function). The following explanation seems incoherent—please rewrite it with precise mathematical meaning rather than generative phrasing.\n\nLine 202: Clarify what ( L ) denotes (loss function) and what ( r^{f_t} ) refers to (is it the residual, ( y - f_t(x) )?).\n\nLine 226: Variable names are inconsistent—change ( S_r ) → ( s_r ) and ( S_a ) → ( s_a ) to match prior definitions.\n\nLine 231: Define what you mean by a “high-value structure.” Also, specify what the symbol h represents in the tuple ( (name, desc, h) ).\n\nLine 295: Explain all symbols in the NMSE equation: ( y ) (true target), ( \\bar{y} ) (mean of targets), and ( \\hat{y} ) (predicted output). Introduce them before using the formula.\n\nLine 359: The font size of Table 2 is excessively large. Remove unnecessary LaTeX sizing commands and ensure uniform table formatting.\n\nSection 5.2.2: The example expression ( \\sqrt{c_1 |x_1| / x_2} ) does not appear particularly challenging; classic search-based SR methods can likely recover it easily. Provide a stronger justification for why this case is nontrivial.\n\nLine 450: In the expression “( i\\Delta = E - V_0 + \\hbar J_0 ),” remove the leading i—it appears to be a typographical error.\n\nFigure 5a: Define both ( R^2(\\text{data}) ) and ( R^2(\\text{equation}) ) explicitly. Clarify how each is computed (e.g., is ( R^2(\\text{data}) ) the fit to measured data and ( R^2(\\text{equation}) ) the symbolic agreement with the analytical form?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)