Symbolic Regression with a Learned Concept Library
Authors: Arya Grayeli
Atharva Sehgal
Omar Costilla Reyes
Miles Cranmer
Swarat Chaudhuri
Venue: NeurIPS 2024 poster (2024)
Keywords: symbolic regression
genetic programming
program synthesis
generative modelling
foundation models
Primary area: machine_learning_for_physical_sciences

Abstract: We present a novel method for symbolic regression (SR), the task of searching for compact programmatic hypotheses that best explain a dataset. The problem is commonly solved using genetic algorithms; we show that we can enhance such methods by inducing a library of abstract textual concepts. Our algorithm, called LaSR, 
uses zero-shot queries to a large language model (LLM) to discover and evolve concepts occurring in known high-performing hypotheses. We discover new hypotheses using a mix of standard evolutionary steps and LLM-guided steps (obtained through zero-shot LLM queries) conditioned on discovered concepts. Once discovered,  hypotheses are used in a new round of concept abstraction and evolution. We validate LaSR on the Feynman equations, a popular SR benchmark, 
as well as a set of synthetic tasks. On these benchmarks, LaSR substantially outperforms a variety of state-of-the-art SR approaches based on deep learning and evolutionary algorithms. Moreover, we show that LASR can be used to discover a new and powerful scaling law for LLMs.

Symbolic Regression with a Learned Concept Library
Arya Grayeli" UT Austin, Foundry Technologies
Atharva Sehgal* UT Austin
Omar Costilla-Reyes MIT
Miles Cranmer University of Cambridge
Swarat Chaudhuri
UT Austin
Abstract
We present a novel method for symbolic regression (SR), the task of searching for compact programmatic hypotheses that best explain a dataset. The problem is com- monly solved using genetic algorithms; we show that we can enhance such methods by inducing library of abstract textual concepts. Our algorithm, called L.SRR, uses zero-shot queries large language model (LLM) discover and evolve abstract concepts occurring in known high-performing hypotheses. We discover new hypotheses using mix of standard evolutionary steps and LLM-guided steps (obtained through zero-shot LLM queries) conditioned on discovered concepts. Once discovered, hypotheses evolution. We validate LASR on the Feynman equations, a popular SR benchmark, well as set of synthetic tasks. On these benchmarks, LASR substantially outperforms a variety of state-of-the-art SR approaches based on deep learning and evolutionary algorithms. Moreover, we show that LASR can be used to discover new and powerful scaling law for LLMs.
used in
round of concept abstraction and
Introduction
Symbolic regression (SR) 33 is the task of finding succinct programmatic hypotheses written in flexible, domain--peccfic programming language that best explain a dataset. Initially proposed in the 1970s, SR has recently emerged as prominent approach to automated scientific discovery, with applications in domains from astrophysics 30 12
to chemistry 2 22
to medicine [51.
Computational complexity is fundamental challenge in SR, as the space of hypotheses that an SR algorithm must search is discrete and exponential. Previous work has approached this challenge using methods like genetic programming [0], neural-guide search [1] 43), deep reinforcement learning [3 and hybrid algorithms 28 However, new tools to enhance the scalability of SR remain a critical need for applications in SR and scientific discovery.
In this paper, show that abstraction and knowledge-diie directed discovery can be powerful principles in building such scaling tools in SR. State-of-the-a genetic algorithms for SR [10] evolve pools of candidate hypotheses using random mutation and crossover operations. By contrast, human scientist does not just randomly mutate their explanations of data. Instead, they synthesize background knowledge and empirical observations into abstract concepts, then these concepts derive new explanations. We show that zero-shot queries to large language models (LLMs) can be used to implement such discovery process on top of standard SR algorithm.
Concretely, we present a new method for symbolic regression, called LASR, that discovers library of abstract, reusable and interpretable textual concepts and uses it to accelerate SR. LASR alternates between three phases: (i) concept-directed hypothesis evolution, where standard genetic operations
"qqual contribution; Correspondence atharvas@utexas.com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Hypothesis Evolution
Hypothesis
Concept Library
Best Hypothesis per Population *
Dataset
II
π
Symbolic Evolution
LLM Evolution
π 2
II3
10 operations
* T
Concept Evolution
Concept Abstraction
"exponential growth/decay"
Hypochesi Evalitio
To
Io
Concept Library
"Depend's temperature"
C3
C4
Specification Synthesis
Concept Evolution
LLM Concept Crossover
C1
"exponential growth/decay"
C4
"Boltzmann Distribution"
An overview of LASR. LASR iteratively refines library of interpretable textual concepts which are used to bias the search for hypotheses for scientific discovery tasks. This involves three distinct phases: (Top) finding optimal hypotheses within a concept-directed hypothesis evolution, (Right) leveraging the optimal hypotheses to find new concept abstractions, and (Left) iterating learned concepts to discover new concepts to accelerate hypothesis evolution. LASR introduces an orthogonal direction of improvement current symbolic regression algorithms [TO] (in gray).
Figure I:
over hypotheses are interleaved with LLM-guided mutation and crossover operations conditioned known library concepts; (ii) the L.M--baeed abstraction of patterns in known high-performing hypotheses into new concepts; and (iii) the LLM-directed evolution of concepts into more succinct and general forms. Together, these three steps form an open-ended alternating maximization loop that combines evolutionary exploration with the exploitation of the LLM's background knowledge and in-context learning ability.
We experimentally compare LASR on Feynman Equations 26 a popular SR benchmark which the goal is to discover 100 equations from the Feynman Lectures in Physies against several state- of-the-art genetic and deep learning approaches. LASR can discover 66 of the 100 target equations, while the best existing approach could be attributed to test set leakage, we compare LASR with a state-of-the-art genetic approach on suite of synthetic benchmarks. We show that LASR substantially outperforms the bascline. Finally, we support the contribution of LASR to the research community by evaluating the methodology in case study where the goal is find new scaling laws for LLMs.
solve 59. To address the concern that LASR's performance
In summary, the contributions of this paper are as follows:
We pose the problem of discovering an open-ended, reusable concept library that accelerate solutions to downstream SR tasks.
We present LASR, a method for combining zero-shot LLM queries and standard evolutionary operations to simultaneously induce concept library and gerehperformin hypotheses. LASR's 's strategy using LLMs to accelerate evolutionary algorithms may have future applications in settings beyond SR.
We offer promising experimental results, including demonstration that LASR outperforms state- of-the-art algorithms in standard SR tasks and synthetic domains, as well as a case study that uses LASR to discover novel LLM scaling law.

---

2 Problem Formulation
Symbolic Regression. We formulate symbolic regression (SR) program synthesis [6] problem. The inputs to this problem include language L of programmatic hypotheses and a dataset = {(x;, of input-output examples. The syntax of L is described by context-free grommar 24 The grammar allows each hypothesis π to be represented using set mathematical operators (e.g..
addition, multiplication, trigonometric functions) that facilitate the composition of simpler hypotheses into more complex ones. We abstractly define the fitness of a hypothesis т as the likelihood Pc (T) that it generates D.
In order to prevent finding non-useful solutions, we impose a prior probability distribution Pc( over hypotheses that penalizes syntactically complex hypotheses. We now pose SR as the task of finding a hypothesis π* that maximizes the fitness while minimizing syntactic complexity. The
problem can be expressed as maximum posteriori (MAP) estimation problem [15]
π = arg max Pe ( = arg max (PI
PC(
optimization
regularization
Recent work leverages large language models (LLMs) for program synthesis N 19 32 Large
language models (LLMs) approach program synthesis as a token prediction problem, directly approx- imating the likelihood of programs by training internet-scale datasets. That is.
Pe (D) ze PLLM( (+) (C), desc(D)).
(2)
where (I)) and (C) are, respectively, textual representations of and specification of the syntax of L, and the task description (D) is a few-shot serialization of a subset of the examples in D.
Symbolic Regression with Latent Concept Libraries. Classical symbolic regression typically assumes no prior knowledge or intuition about the problem. In contrast, human scientific discovery often leverages empirical patterns [52) and intuitions derived from previously observed data. For example, recognizing a 'power law relationship between variables' has led to the formulation of fundamental empirical laws across various fields, such the Arrhenius equation in Chemistry, the Rydberg formula in Physics, Zipf's s law in Linguistics, and Moore's 's law in Computer Science
We model such empirical patterns as natural-languaaee concepts drawn from latent concept library C. We frame the relationship between the concept library and programs as a Hierarchical Bayesian model consisting of: (i) prior p(C) representing the natural distribution over concept libraries; (ii) a model P C) that quantifies the likelihood of various hypotheses for a given concept library C; and (iii) the previously mentioned fitness function PC (D x) for programs и. We assume that the distributions p(C) and (T C) can be approximated using LLMs. That is, we can prompt an LLM to generate interesting concepts, and we can prompt an LLM with set of concepts to generate loken-sequent representations of hypotheses that adhere to the concepts. Now we state the problem of symbolic regression with latent concept learning as one of simultaneously inducing an optimal concept library and an optimal programmatic hypothesis:
arg m(x C(D) arg max s,C
(D)m)
p(T[C)
((C)
#.C
By execution
By LLM By LLM
3 Method
LASR performs two-stage evolution over natural- languall concepts and programmatic hypotheses. The two stages follow an alternating maximization strategy shown in Figure [ (1) Hypothesis evolution: We fix the set of concepts and focus on maximizing the hypotheses" fitness to the dataset, and (2) Concept abstraction and evolution: We leverage the best hypotheses found induce a new library of concepts.
In the rest of this section, we first describe PySR, the SR algorithm [10 that LASR extends. Next, we show how to modify this algorithm into one guided by natural-language concepts. Finally, we show how these concepts can be naturally extracted and evolved into new concepts. The full LASR
algorithm is presented in Algorithm I and visualized in Figure 2 LASR is built in Julia with an
additional Python interface and uses an source, optimized framework for LLM inference 25
Artifacts available athttps://trishhull github. 1o/lasr-web
3

---

Inftiolration
(Optional)
Hypotheais Populations
Concept Library
C1
Poreto Frontier
Expression Search Operotions
Concept Library Operations
(c400 (tecaioon)
(X3 sterations)
Munetion
Roxdus
Concept Extraction
Erssueer
Masiton
Sprtek::
Concept
Data Fitress
LLM
Sample Concepts
Figure 2: A single step of LASR. LASR induces multiple hypothesis populations that are evolved using scalable evolutionary algorithm. Concept guidance is provided by randomly replacing symbolic operations with concept-directe LLM operations with probability p. After each iteration, the top-performing programs are summarized into natural language concepts, which are evolved to form new concepts that are sampled to guide the search in the next iteration.
Base Algorithm: PySR. LASR builds on PySR [10, scalable, parallelizable genetic search algorithm for SR. The search in PySR maintains multiple populations (II,...., [1 of hypotheses, with each hypothesis represented procedure INITIALIZEPPPULAAIONN,, PySR creates a new expression at random to insert into population. After running this step. PySR SRCYCLF, which evolve these populations in parallel, simplifies and optimizes the constants of the resulting hypotheses, and then migrates top-performing hypotheses between populations.
expression tree. In its initialization step, captured by
genetic search, encapsulated in procedure
Like other evolutionary algorithms, the search PySR uses symbolic mutation and crossover operations. The mutation step is broken into many categories, each with distinct weighting, to either mutate a constant, mutate an operator, add node (append, prepend, insert), delete a subtree of an expression tree, simplify the tree, initialize new tree, or do nothing. One of these operations is randomly selected at each call to a mutation request, and each operation executes itself at random but within user- gpor-provdded constraints. For example, deleting subtree is done by choosing random leaf node such as a feature or constant. The crossover
node to replace with a vandomly-generate step involves swapping random subtrees of two expressions in population.
LLM-guided Hypothesis Evolution. LASR speeds up PySR by injecting natural language priors into its search procedure. To do this, we modify the INITIALIEEOPULAATONS procedure to use an LLM-augmented initialization operation, and the SRCYCLE routine to use ILM-augmented versions of its symbolic mutation and crossover operations. The altered procedures are named LLMINIT, LLMMUTATE, and LLMCROSSOVER, respectively. These operations do not replace their standard genetic counterparts. Instead, we introduce a hyperparameter that, with a fixed probability, substitutes the standard genetic operation with the LLM-based operation. This enables "doping" each population with a program that respects the language priors, while ensuring that we do not bottleneck the local exploration of the search space.
The LLM-guided operations follow the same base format: they sample multiple concepts from the concept library, concatenate these concepts with the task-specific variable names and language operations, and append a specialized prompt for each task. We employ zero-shot prompts (sce Appendix A.2 for more details) to avoid sampling biases. In further detail:
LLMINIT:
The LLMINIT function takes an initial of concepts and uses them to initialize the populations for the evolutionary search step. The initial set of concepts can either be instantiated from an optional set of user-provided "hints" or generated by the LLM.
For mutation within a population, we sample a set of concepts from the concept library C, and construct a prompt that uses this set of concepts to mutate an expression x; into тт The prompt to the L.MM takes inspiration from the standard genetic mutation operation, and asks it to mutate the expression given the concepts sampled from the library.
LLMMUTATE:

---

Algorithm Pseudocode for LASR. LASR takes as input an optional set of user-provided hints Co. dataset of input-output pairs of high- nigh-dimensional data D, and four hyperparameters: the number of iterations I, the number of populations K, the number of steps for concept evolution M, and the mixture probability of using LLM-based or GP-based evolutionary operators p. LASR produces two artifacts: the evolved library of concepts C and the expression with the highest fitness score x².
I: function LASR(Co, D y:)) K, M.
C INTTAALZZECONEEXLLBBARVVE
Add (optional) user hints to library.
3: (I11,... 11x INTIIALIZAPPPUAATINNICC, K)
4: for in range(I) de for i in range(K) do
5:
6:
IT, SRCYCLE(T,, P,C,p) JF EXTRRCTPPRETOOFOONTEEEERR
Interleaved Symbolic + LLM Search Includes positive negative programs
7:
1,),D)
C - CONEPPAASSSTCCCCIOO(....
9:
for in range(M)
10:
C CONCEPTEVOLLUTIONS
11:
T < BESTEXPRESSION(FF
Based on dataset loss and program complexity
12:
return C, T²
LLMCROSSOVER: The LLMCROSSOVER function also samples a set of I concepts from the concept library along with two hypotheses π. and ボj to construct new expression TR, which reuses sub-expression trees from the two hypotheses while respecting the sampled concepts. Our implementation is inspired by prior work (40
see Figure
Concept A Abstraction. After each iteration of symbolic regression, we use a function EXTRACT- PARETOFRONTIER to collect: (i) the hypotheses, across all populations, that are Pareto-optimal with respect to the criteria of syntactic simplicity and dataset loss; (ii) the hypotheses with the worst loss across all populations. The resulting set of hypotheses F { TT ... "I Th captures the trends that were most helpful and most detrimental to performance during hypothesis search. Now we use the CONCEPTARSTE ACTION function, which uses zero-shot prompt extract natural language concept c* that summarizes the positive trends while eschewing negative trends. This concept is subsequently added to the concept library. The prompt for the function is presented in Figure 5
Concept Evolution. Each concept in C represents trends that were useful at previous state in the search process. After adding new concepts into the library, we use a function CONCEPTEVOLUTION to evolve the library to include new ideas that logically follow from the ideas in the current library. The implementation of this function follows that of the LLMCROSSOVER operation in that we are using multiple concepts reference to generate new ones, with the key distinction that, unlike in the LLMCROSSOVER operation, the fitness of each generated concept here is difficult to quantify. Thus, include all the generated responses in the concept library. While these concepts may sometimes be inaccurate, they increase the evolutionary algorithm's exploration ability.
4 Experiments
We demonstrate the effectiveness of LASR on multiple tasks integral to scientific discovery. First, we evaluate LASR's performance on the Feynman Equation datasct, a widely adopted scientific discovery benchmark, under variety of ablations and additional priors. Second, measure the effect of data leakage by evaluating LASR's performance on a procedurally generated synthetic dataset of challenging equations. Finally, we conduct a case study using LASR to discover LLM scaling laws with data from the BIG-Bench evaluation suite [16].
LASR's 's main focus is to serve as a practical toolkit for scientists. Therefore, our evaluation primarily targets slightly noisy environments, using exact solution rate to gauge performance rather than statistical similarity measures like correlation R22, which are less relevant to scientific discovery applications. Additional experiments characterizing the practical behavior of LASR are in Appendix A.5 Information regarding compute usage is in Appendix A.3.I
5

---

GPlearn
AFP
AFP-FE
DSR
uDSR
AlFeynman
PySR
LaSR
20/100
24/100
26/100
23/100
40/100
38/100
59/100
72/100
Table 1: Results on 100 Feynman equations from [49]. We report exact match solve rate for all
models. LASR achieves the best exact match solve rate using the same hyperparameters as PySR.
LASR (Llama3-8B)
LASR (GPT-3.5)
Type Solve
PySR
p = 1%
p 5%
p 10%
p= 1%
Exact Solve
59/100
67/100
69/100
71/100
72/100
Almost Solve
7/100
5/100
6/100
2/100
3/100
Close 16/100
9/100
12/100
12/100
10/100
Not Close
18/100
19/100
13/100
16/100
15/100
Table 2: Evaluation results on Feynman dataset by cascading LASR's s LLM backbone (1lmma3-8b, gpt-3.5-turby and changing the probability of calling the model (p = [0.01, 0.05, 0.10]) in the order of increasing concept guidance. LASR outperforms PySR even with minimal concept guidance using an open-source LLM.
4.1 Comparison against baselines in the Feynman Equation Dataset
Dataset: The Feynman Equations dataset is a well established benchmark for Symbolic Regression algorithms [49]. This dataset consists of 100 physics equations extracted from the Feynman lectures on Physics. We compare against performance reported on SRBench [26]: a continuously updated compare against GPlearn, 28 38) Within this subset PySR
public benchmark of SR methods on many datasets. Specifically,
AFP, AFP-FE, DSR, uDSR, PySR, and Al Feynman 42 represents an ablation of model without the L.M genetic operations and the concept evolution (Section 3.
47
49
We evaluate on slightly noisy version of this dataset in order to simulate experimental errors common in scientific discovery domains. More details are presented Appendix [A.4.1
Setup: We instantiate LASR using gpt-3. 3pt-33.5turte torttuuroo-110 4 as the backbone LLM and calling it with p 0.01 for 40 iterations, and compare results with PySR which uses the same default hyperparameters. For the other baselines, we use the numbers reported in SRBench with one exception being uDSR (28), for which couldn't find any benchmarking numbers. For this method, we derive the exact solve rate from (37
Results: We showcase results in T We draw three observations from this experiment. First, LASR achieves a higher exact solve rate than all other basclines. Second, both PySR and LASR outperform the other baselines by wide margin, indicating that scalable and efficient synthesis is imperative to practical scientific discovery algorithms. Finally, and most notably, a subset of the equations LASR finds could not be derived with any of the previous methods.
4.2 Cascading Experiments
LASR's performance is inherently bottlenecked by the reasoning capabilities of the backbone LLMs and the frequency of their invocation in each iteration. To evaluate the effect of the backbone LLM on LASR's performance,
instantiate model cascade over two of LASR's hyperparameters: the backbone model (11ama3-8b 17). gpt-3. 3- 1ww.5-tuuoo-ff11 -0125) and the probability p with which we call that model in the evolution step (p [1 11, 5%, 10%]) [).
Setup: Our cascade operates as a tournament. We start LASR with the configuration that provides 1%) and progressively increase the value of p and then the backbone model. Each subsequent model only evaluated on the problems that the previous model could not solve. We compare this against PySR's performance on the Feynman equation dataset. To ensure a fair comparison, we cascade PySR using the same procedure but find it does not solve any additional equations. For this experiment, we tag each equation with qualitative rating comparing the equation to the ground truth form (Exact Solve, Almost Solve, Close, and Not Close). An in-depth discussion this metric is presented in Section A.7
the least language guidance (1lama3-8)
Results: Our results are presented in We draw two key observations from these results. First, LASR outperforms PySR even with minimal concept guidance (11ama3-8b 8b at p 1%). Second,

---

Liama3 0.1%
PySR
Llama3 0.1% Hints
Liama3 0.1% No Vars
Llama3 0.1% No Concept Library
Llama3 0.1% No Concept Evolution
Equations Solved under MSE
100
80-
40
15
20
25
Iterations
Iterations
Figure 3: Evaluation results for ablations/extensions of LASR. (Left): We ablate three components of LASR: Concept Evolution, Concept Library, and variable names and evaluate their MSE solve rate performance on the Feynman dataset over 40 iterations. We find that each component contributes to accelerating search different stages in the search process. (Right): We extend LASR by providing an initial concept library Co in the form of user provided hints. We find that natural language hints significantly increases the speed of solving equations.
increasing the backbone model size and the mixture probability substantially enhances LASR's performance, indicating that as the language reasoning capabilities of LLMs improve, so will our performance.
4.3 Ablation Experiments
We conduct ablations on the use of Concept Evolution (skip phase three from Figure Concept Library (skip phase two and three), variable names, and user hints. BigureBishowe how these ablations affect performance over 40 iterations. We designate an equation as "solved" if, after iterations, the MSE of our predicted equation is less than 10-11 This metric differs from 'Exact Solved' as defined in the prior experiments: an equation can be "exactly solved' yet have an MSE higher than 10-11 due the noise floor in the target variables, and equation have low loss but not be exact match. We observe from the results that: (1) Removing variable names results a substantial performance drop,
lose semantic meaning provided by variables (for instance, observing could suggest employing trigonometric functions on 0). (2) Learning a concept library enables faster convergence solutions. Without the concept library, task convergence is slower, and widens under higher concept guidance conditions (p > 0.1%).
4.4 Qualitative Analysis and User Hints
The concept library provides interpretable window into evolutionary search process. To showcase the concepts learned by LASR, we take a sample equation from the Feynman dataset, the electric field of a dipole E 3pa cos Amrrr
and comment on the libraries learned at various intervals. We rudimentary concepts emerge in the second iteration:
"The presence of busic trigonometric functions funccttoooiiiii like sin in the good expressions contributes to their quality, indicating connection to physical concepts such as waveforms periodic phenomena."
And, in subsequent iterations, the concepts become even more refined:
"The good mathematical expressions exhibit a balance between mathematical operations such as multiplication, division, and trigonometric functions, which are known to have physical interpretations and relevance various scientific phenomena."
This iterative refinement helps LASR consistently maintain high-quality concepts, allowing it to converge to an exact match within 40 iterations. By contrast, PySR and the concept library ablations fail to converge on an exact match solution, returning equations that while low-loss involve many extra terms absent from the ground truth. This reinforces our hypothesis that injecting semantic meaning into the search process not only improves search efficiency, but also regularizes against

---

complex equations - as the LLM-generated concepts help filter out irrelevant terms. A deeper qualitative analysis is in Appendix A.
Extending LASR with Hints: A benefit of LASR is that its search can be initialized with a set of user-specified, natural-language "hints." To evaluate this capability, we generate hints for each equation based on variations of the chapter title of the Feynman lecture that the equation belongs to. We intentionally keep the hints vague see if knowledge about just the general field is sufficient in improving LASR's 's performance. We showcase results in Figure 3 We observe a noticeable boost in performance from injecting these hints, even for our weakest performing model, indicating that even minimal user input can substantially enhance LASR's 's effectiveness in discovering equations.
4.5
Data Leakage Validation
An important consideration in using LLMs for existing SR problems is the possibility that the L.M was exposed to the hold-out problems in the validation set, presenting an unfair advantage to LLMs trained on massive datasets. Intuitively, LASR generates its own concepts which conditioned on suboptimal programs, which are unlikely to be within the LLM's memorized responses. To validate this, we generate dataset of 41 synthetic equations that are engineered to deviate from common physical and mathematical structures and have arbitrary variables. For example, one such equation is 0.782zx+0.536
y 22e (log -2ge
We find that PySR struggles to solve equations with these characteristies (given 400 iterations). Hence, solving such equations hinges on the language guidance components.
We run LASR with Llama3-8B at 0.1%. We then compare our synthesized program's 's test set R² with that of PySR's s. We justify using correlation instead of exact-match as we are not motivated by the application of results for scientific discov- ery in this experiment. Our results are summarized in Table 3 and show that LASR's concept-guided synthesis still pro- vides a considerable performance boost compared to PySR demonstrating that LASR can outperform PySR even when data leaking is not possible.
PySR
LaSR
ILlama3-8B. 0.1%)
0.070
0.913
Table 3: Evaluation results of data leakage. We present the test set n? of PySR and of LASR on syn- thetic symbolic regression dataset. Higher R² is better.
4.6 Using LASR to discover LLM Scaling Laws
So far, we have demonstrated that LASR can discover equations that are practical but already known (Feynman Dataset) and equations that are novel but aren't practical (Synthetic Dataset). To investigate LASR's utility in finding novel and practical empirical trends, we investigate whether LASR can discover novel LLM scaling laws on the BigBench dataset [16]. More details on this experiment are presented in Section A.6
Traditionally, to identify an LLM scaling law, practitioners must first manually posit a "skeleton equation" with a fixed set of known variables and unknown free parameters, and then optimize the unknown parameters based on a dataset of model hyperparameters and resulting dataset fitness 23 I 5. Instead of starting with predefined equation, we use LASR to discover the skeleton equation that best fits various subsets of the BigBench dataset.
Setup. BigBench contains 204 tasks with scored responses from 55 language models trained with different hyperparameters. We evaluate on the subset of tasks where the preferred metric 'Multiple choice grade' (53,812 samples). Our goal is to find the equation that best predicts the test score given the model hyperparameters and the dataset hyperparameters. We run LASR with 3840 populations of 200 candidates each for 7 hours (overnight). The runtime of LASR is comparable to other SR algorithms for this experiment as the slowest operation isn't 't generating candidate equations but rather optimizing and evaluating candidate equations.
Results. LASR discovers the following scaling law on the subset of BigBench:
score
|troin_sttee steps f
#ebote
+ E
(4)
8

---

Scaling Law Skeleton
MSE Loss
Free Parameters
Equation
0.03598 = 0.00265
3
Chinchilla 23
0.03610 = 0. 0.00268
5
Modihed Chinchilla
0.03598 0.00264
5
Residual Term Only
0.09324 = 0.01992
1
Table 4: Preliminary results on evaluating different LLM scaling laws. We measure MSE loss on a held out subset of BigBench [6.. The equation discovered with LASR performs as well as the Chinchilla equation [23] on BigBench while using less free parameters. The residual term skeleton equation (score = E) also performs well.
where score the MCQ grade, train_ train_steps is the number training steps for the model, and #shots is the number of in-context examples provided during inference. The fitted free parameters are presented Equation 5
Qualitative Evaluation: Equation describes an empirical relationship between training hyperparam- eters (training steps) and inference hyperparameters (number of shots). asserts that increasing the number of shots exponentially increases the model's performance for low-resource models, while having diminishing gains as the number of training steps of the model increase. This observation consistent with work in scaling test-time compute [46].
As the output artifacts of LASR are interpretable, we can integrate this empirical relationship between training steps and number of shots into known scaling laws. Specifically, we can augment the chinchilla scaling law follows:
A
B
score
3 +
F.
(Chinchilla 23
(train_stepa batch (
(#params)
A
B
score (train_steps batch_ size)
a Hehotss
(#params)* 3
(Modified Chinchilla)
Quantitative Evaluation: We fit the free parameters of each equation on the training set (43, 049 samples) and measure the MSE loss between the actual grade and the predicted grade on the validation (10, 763 samples). The results are presented in Table[4 We find that the Equation 44s performance, as well as modified Chinchilla's 's performance, is competitive with that of Chinchilla in predicting the MCQ grade. However, the horizontal line score E demonstrates acceptable performance as well. We believe increasing the scale of these preliminary experiments (with richer datasets or longer search horizon) will lead additional empirical findings.
Related Work
Symbolic Regression. The field of SR started in the 1970s 18 29 and has recently become Two algorithmic themes here are:
prominent approach to Al-for-science 33
Non-parametric Algorithms: Most work on SR focuses improving search efficiency using heuris- ties or parallelization. Specifically, PySR [00] builds a multi- population evolutionary algorithm that incorporates various preexisting heuristics [39], and introduces novel ones such as simulated annealing, an evolve-simplify--piiiii
loop, and an adaptive parsimony metric. PySR has been successfully applied to study problems in domains such as cosmology [12), international economics [50], and climate modeling (20).
LASR extends PySR to enable the discovery of latent concepts.
Parametric Algorithms: Recent work in SR and program synthesis has often used neural networks accelerate search +
34 13 35]. The interplay between the neural and the symbolic components in these works can be abstracted into two categories: (1) leveraging LLMs to induce
40
38
28
program scaffolds [34] (4. (35), and (2) learning neural policy accelerate search [8 28
13.
Funsearch 40 uses a pretrained LLM to implement a mutation operator on a database of executable programs under fixed specification to find super-optimized programs in extremal combinatorics. LASR is a generalization of FunSearch: while FunSearch conditions program generation on a static "spec- ification" (analogous to concept library), discover the concept library in the course of the
We highlight two methods from the first category: Funsearch [40] and LLM-SR
45

---

generalize to higher-compute algorithm. We do not compare against FunSearch due to resource constraints. As for LLM-SR EST. leverages pretrained LLM for generating program sketches [36]. The sketch parameters are optimized and cached in database, which is in turn used to generate new sketches. Our work is an orthogonal direction of improvement. It is technically possible to "plug" the L.MMSR framework (or other LLM--ased search algorithms (35) into LASR and use our generated concepts to guide the lower-level search component.
The second category includes methods like DSR 38 which, just like LASR, frame SR as sequence modeling problem. However, the search in LASR leverages learned concept library and the language and code biases in LLMs, instead of relying on amortization alone.
Program Synthesis with Foundation Models. Recent work in program synthesis models program generation as sequence prediction problem. Under this paradigm, the DSL and the input-output specification is serialized in the prompt and a gone-eeenration foundation model [1 7 4 leveraged to autoregressively generate candidate programs. This approach has been impactful in many areas including spreadsheet formula prediction [3] g, competitive programming [32], and visual program-
ming [48, 21 9 LASR is similar to work in this area in that the LLM Mutate, LLM Crossover, and
LLM Initialization functions all follow the sequence prediction paradigm to synthesize mathematical equations, relying on guidance from the concept library.
Program Synthesis with Library Learning. Deploying classical program synthesizers in new domain often necessitate hand-engineering DSLs to enable scalable synthesis. This severely limits the generality and practicality such methods. An emerging direction of research called library learning attempts to learn the DSL and the programs simultaneously [15 This is typically framed as a hierarchical Bayesian optimization problem over the space of programs and the space of library functions that generate those programs. Notably, [19] uses LLM guidance
3
19
27 53
14
44 [54]
to assist in program induction and in nuto-documenting learned library modules and [53] considers learning programs under latent distribution over the space of natural language and the space of the DSL. LASR shares similar problem formulation to these works, but optimizes over the space of programs and over the space of natural language descriptions of these programs.
6
Conclusion
We have presented LASR, a framework that uses zero-shot queries to an LLM to induce abstract, reusable concepts that can be used accelerate SR. We have shown that LASR outperforms state- of-the-art approaches on the standard Feynman equation task. We have also used the algorithm to discover a novel scaling law for LLMs.
key benefit of LASR is that its capabilities are ultimately bottlenecked by those of the underlying LLM. LLMs rapidly gaining capability and getting cheaper, and future versions of LASR should be able to tap into this progress.
Many directions of research remain open. First, our strategy of accelerating evolutionary search with LLM-based concept induction may be applicable beyond the SR setting. Future research should explore such applications. Second, while our approach here was entirely based on in-context learning, it is worth exploring if finetuning improves the performance of the LLM. Finally, we evaluated the learned concept library exclusively the downstream SR task. However, the library may also be valuable in other tasks such as clustering or explanation synthesis. Exploring these other tasks is an attractive topic for future work.
Limitations. The current instantiation of LASR has several limitations. First, it cannot guarantee that the concepts it learns correct or insightful. Even concept that leads to strong performance in downstream SR tasks may do so because of quirks of the model and data, and end up misleading scientists using the method in discovery process. Also, we do not currently have way to ensure that the learned concepts are mutually consistent. Finally, our evaluation here was constrained by our compute budgets for LLMs and search. Whether the trends regimes remains to be seen.
10

---

Acknowledgements: This research was partially supported by the NSF Expeditions in Computing Award #CCF-1918651. the NSF National AI Institute for Foundations of Machine Learning (IFML), and ARO award #W91 INF-21-1-000) We thank Foundry Technologies for providing substantial computational resources for our experiments.

---

References
[1] Ibrahim Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[2] Rohit Batra, Le Song, and Rampi Ramprasad. Emerging materials intelligence ecosystems propelled by machine learning. Nature Reviews Materials, 6(8):655-678 2021.
[3] Matthew Bowers, Theo X Olausson, Lionel Wong, Gabriel Grand, Joshua B Tenenbaum. Kevin Ellis, and Armando Solar-Lezama. Top-down synthesis for library learning. Proceedings of the ACM on Programming Languages, 7(P0PL):1182-2213 2023.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[5] Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws, In The Eleventh International Conference on Learning Representations, 2023.
[6] Swarat Chaudhuri, Kevin Ellis, Oleksandr Polozov, Rishabh Singh, Armando Solar-Lezama, Yisong Yue, et al. Neurosymbolic programming. Foundations and Trends@ in Programming Languages, 7(3):158-2444 2021.
[7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[8] Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, and Denny Zhou. Spreadsheetcoder: Formula prediction from semi-structured context. In International Conference on Machine Learning, pages 1661-1672. PMLR, 2021.
[9] Mia Chiquier, Utkarsh Mall, and Carl Vondrick. Evolving interpretable visual classifiers with large language models, 2024.
[10] Miles Cranmer. Interpretable machine learning for science with pysr and symbolicregression, arXiv preprint arXiv:2305.001558 2023.
[11) Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. In Neural Information Processing Systems, 2020.
[12] Benjamin L Davis and Zehao Jin. Discovery of planar black hole mass scaling relation for spiral galaxies. The Astrophysical Journal Letters, 956(1):66))) 1)222, 2023.
[13] Jacob Devlin, Jonathan Uesato, Surya Bhupatiraju, Rishabh Singh, Abdel-rahman Mohamed, and Pushmeet Kohli. Robustfill: Neural program learning under noisy I/O, In ICML, 2017.
[14] Kevin Ellis, Lucas Morales, Mathias Sablé-Meyer, Armando Solar-Lezama, and Josh Tenen- baum. Learning libraries of subroutines for neurally Bayesian program induction. In Advances in Neural Information Processing Systems, pages 7805-7815 2018.
[15] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning. arXiv preprint arXiv:2006.088 200.20000...8,,, 2020.
[16] Aarohi Srivastava et al. Beyond the imitation game: Quantifying and extrapolating the capabili- ties of language models. Transactions on Machine Learning Research, 2023.
[17] Abhimanyu Dubey et al. The Iaaaa 3 herd of models, 2024.
[18] Donald Gerwin. Information processing, data inferences, and scientific generalization. Behay- ioral Science, 1955)3314-325, 1974.
12

---

[19] Gabriel Grand, Lionel Wong, Matthew Bowers, Theo X Olausson, Muxin Liu, Joshua B Tenenbaum, and Jacob Andreas. Lilo: Learning interpretable libraries by compressing and documenting code. arXiv preprint 1rXiv22310.19791, 2023.
[20] Arthur Grundner, Tom Beucler, Pierre Gentine, and Veronika Eyring. Data-driven equation discovery of a cloud cover parameterization. Journal of Advances in Modeling Earth Systems, (6))):22233M0003733, 2024.
[21] Tanmay Gupta and Aniruddha Kembhavi. Visual programmingg Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953-14962, 2023.
[22] Alberto Hernandez, Adarsh Balasubramanian, Fenglin Yuan, Simon AM Mason, and Tim Mueller. Fast, accurate, and transferable many-body interatomic potentials by symbolic regres- sion. npj Computational Materials, 5(1):112, 2019.
[23] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 2203::2033.5556, 2022.
[24] John E. Hoperoft, Rajeev Motwani, and Jeffrey D. Ullman. Introduction to automata theory, languages, and compulation, 3rd Edition. Pearson international edition. Addison- Addison-Wesley, 2007.
[25] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[26] William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabrício Olivetti de França, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason Moore. Contemporary symbolic regression methods and their relative performance. arXiv preprint arXiv:2107.14351, 2021.
[27] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):
2015.
[28] Mikel Landajuela, Chak Shing Lee, Jiachen Yang, Ruben Glatt, Claudio P Santiago, Ignacio Aravena, Terrell Mundhenk, Garrett Mulcahy, and Brenden K Petersen. A unified framework for deep symbolic regression. Advances in Neural Information Processing Systems, 35:33985- 33998. 2022.
[29] Pat Langley. Bacon: Conference on Artificial Intelligence, 1977.
A production system that discovers empirical laws. In International Joint
[30] Pablo Lemos, Niall Jeffrey, Miles Cranmer, Shirley Ho, and Peter Battaglia. Rediscover- ing orbital mechanies with machine learning. Machine Learning: Science and Technology. 0440004 2023.
[31] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.0616/ 2023.
[32] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 37866624) 2022.
[33] Nour Makke and Sanjay Chawla. Interpretable scientific discovery with symbolic regression: review. Artificial Intelligence Review, 57(1):2, 2024.
[34] Matteo Merler, Nicola Dainese, and Katsiaryna Haitsiukevich. In-context symbolic regression: Leveraging language models for function discovery. arXiv preprint arXiv:2404. /9094, 2024.
[35] Elliot Meyerson, Mark J. Nelson, Herbie Bradley, Adam Gaier, Arash Moradi, Amy K. Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompting, 2024.
13

---

[36] Vijayaraghavan Murali, Letao Qi. Swarat Chaudhuri, and Chris Jermaine. Neural sketch learning for conditional program generation. ICLR, 2018.
[37] Deep Symbolic Optimization Organization. Srbench symbolic solution. /mmges//
https:
//github. www http://ooooor//e-e-esse/seeeeeee http:///ooorrreeeepppe/b/oooooao
Subbenchhsymbooicsoooluo swlution. png Accessed: 2024-05-22.
[38] Brenden K Petersen, Mikel Landajuela, T Nathan Mundhenk, Claudio P Santiago, Soo K Kim, and Joanne I Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. arXiv preprint arXiv: 1e1.99922299111 I, 2019.
[39] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780-4789, 2019.
[40] Bernardino Komera-Paredes Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang. Omar Fawzi, et al. Mathematical discoveries from program search with large language models. Nature, 625(7995):468 475.8 2024.
[41] Michael Schmidt and Hod Lipson. Distilling frree-form natural laws from experimental data. Science, 324(5923)881-3) 2009.
[42] Michael D. Schmidt and Hod Lipson. Age-fitness pareto optimization. In Annual Conference on Genetic and Evolutionary Computation, 2010.
[43] Ameesh Shah, Eric Zhan, Jennifer J Sun, Abhinav Verma, Yisong Yue, and Swarat Chaudhuri. Learning differentiable programs with admissible neural heuristics. In Advances in Neural Information Processing Systems, 2020.
[44] Richard Shin, Miltiadis Allamanis, Mare Brockschmidt, and Oleksandr Polozov. Program synthesis and semantic parsing with learned code idioms. In Advances in Neural Information Processing Systems, pages 10825-10835 2019.
[45] Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan Reddy, Lim-sr: Scientific equation discovery via programming with large language models. arXiv preprint urXiv:2404. 18400, 2024.
[46] Charlic Snell, Jachoon Lec, Kelvin Xu, and Aviral Kumar. Scaling I1m test-time compute optimally can be more effective than scaling model parameters, 2024.
[47] Trevor Stephens. gpleam: Genetic programming in python, with scikit-learn inspired api, 2024. Accessed: 2024-05-22.
[48] Didac Surís. Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08128, 2023.
[49] Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(1 6(16):eaay6631, 2020.
[50] Sergiy Verstyuk and Michael Douglas. Machine learning the gravity equation for international trade. Available at SSRN 4053795, 2022.
[51] Marco Virgolin, Ziyuan Wang, Tanja Alderliesten, and Peter AN Bosman. Machine learning for the prediction of pseudorealistic pediatric abdominal phantoms for radiation dose reconstruction. Journal of Medical Imaging, 7(4):046501-046500 2020.
[52] Eugene P Wigner. The unreasonable effectiveness of mathematics in the natural sciences. In Mathematics and science, pages 291-306. World Scientific, 1990.
[53] Catherine Wong, Kevin M Ellis, Joshua Tenenbaum, and Jacob Andreas. Leveraging language to learn program abstractions and search heuristics. In International conference on machine learning, pages 1193-1 1204. PMLR, 2021.
[54] Eric Zelikman, Qian Huang, Gabriel Poesia, Noah Goodman, and Nick Haber. Parsel: Al- gorithmic reasoning with language models by composing decompositions, Thirty-seventh Conference Neural Information Processing Systems, 2023.
14

---

A Appendix
A.1 Broader Societal Impacts
We have presented LASR:
a symbolic regression framework that leverages concept guidance to accelerate symbolic regression. We hope that LASR helps accelerate the search for empirical laws in the broader scientific community. In this section, we discuss the broader societal impacts and ethical considerations of our work.
Potential for Misuse: As with other ML techniques, symbolic regression can be leveraged by bad actors to inflict societal harm. Our experiments show that LASR accelerates the search for empirical laws from raw observations. In our setting, we are restricted to observations about physical phenomena. However, a malicious actor could misuse LASR to find patterns in datasets that violate personal rights.
Privacy Concerns: As mentioned before. LASR enables finding patterns in raw observations. We hope that LASR is leveraged by scientists to explain physical phenomena. However, it is possible to use such models to learn behavioral profiles without the active knowledge or explicit consent of the subjects.
Bias and Fairness: LASR generates two artifacts: a hypothesis that maximizes fitness function (represented as equation) and library of concepts that helped discover that hypothesis. LASR ensures fairness and lack of bias in the generated equation as long as the fitness function is free of biases well. However, we leverage foundation models induce our library of concepts which could be trained on biased data which may reflect in our concept library. Furthermore, we cannot directly evaluate the efficacy of the concept library and its factual correctness. This doesn't I affect equation generation since equations are quantitatively evaluated. However, a human analyzing the concepts LASR learns might misinterpret trends that the model picks up on.
A.2
LLM Prompts
Note that in the prompts in Figure Figure 5 and Figure we refer to our hypothesis as expressions and the concepts as hypotheses and suggestions. This prompting style was found to work best for the LLM.
A.3
Implementation Details
A.3.1
Compute Usage
We run all experiments on a server node with 8xA100 GPUs with 80 GB of VRAM each. However, experiments can be reproduced with GPU with 16 GB of VRAM. We
able to LASR on a laptop utilizing a quantized model hosted locally P Moreover, certain models are hosted on external servers (such as GPUs. For this project, we chose to run alama3-88 8b using VLLM [25]. However, our framework is compatible with any LLM inference framework that allows hosting an OpenAl compliant RESTful server. For reference, each iteration makes around 60, 000 calls. Each call to the LLM is just under 1000 tokens. This gives an upper bound on total compute of 60, 000, 000 tokens per iteration y 100%. Hence, running our model at 1% for 40 iterations would result in just under 25M tokens for each equation (around hours on a single A A100).
0025) which allows running LASR on machines without
A.3.2 Concept Sampling
In order to determine which concepts from the concept library we sample for the LLM Hypothesis Evolution, randomly choose the top-K most recent concepts in the library. This ensures that use the latest concepts, which are generally reflective of more informed hypotheses, and thus better to use. In practice, we K 20. Additionally, for Concept Evolution, we exclude the top-K most recent concepts from being used, and rather use older concepts. This is motivated by the desire not have the concept library converge on a few ideas, rather want diversity of thought. Our concepts are intended to be longer lasting than the hypotheses that generated them, similar to how observational data comes and goes, but the conclusions from them are more persistent.
"TheBloke/Mistral. -tt.rrrrattt---stuuttt------ 2-UUF using 1lama. cpp
15

---

helpful assistant recombines mathematical expressions by following few provided suggestions. will be given three suggestions and two expressions recombine.
(System) Header
An expression must consist of the following variables: ((variables) All constants will be represented with the symbol C. Each expression will only use these operators: ((operators)).
Suggestion [{assump1) }}
Suggestion {{assump2))
Suggestion {{assumpa})
(User)
Expression {(expr))>
Crossover
Expression 2: ((expr2))
Prompt
Propose ((N) expressions that would be appropriate given the suggestions and expressions. Provide short commentary for each of your decisions. End with a JSON list that enumerates the proposed expressions following this format:
(User)
j'soon
["expr1".
JSON
"expr2",
Formatting
(X)))
Instructions
Based the pravided suggestions and expressions, propose following five recombined expressions:
1. ''log(exp(C strt C)) + C): This expression combines the exponential function from Expression 2 with logarithmic function from Expression term taken from Expression 2 and added to the exponential function, while the C' term taken from Expression 1.
sqrt(theta
"(sin(C sqrt(theta)) (theta 1og(theta theta)) :
expression combines the sine function from Expression with the power operation from the same expression. The 'theta log (theta theta)) term is taken from Expression 1, and the sin(C sqrt(theta)) term also from Expression The term is added the end.
(Assistant) Generated Explanation
'exp sqrt(thetal sin(C (errttheta)) ): This expression combines the exponential function from Expression with the sine function from Expression Both the exponential and sine functions taken from separate expressions and multiplied together.
sqrt(theta C):: This expression combines the logarithmic function from Expression with the power operation from the same expression, The "(teea (theta / theta)) term taken from Expression and the sqrt(theta C)" term is taken from Expression 2. The + C' term is added at the end.
'log( theta log(theta / theta))
5. "(sin(C sqrt sqrt(theta)) (theta log(theta theta)) C) This expression combines the power operation from Expression with the sine function from the same expression. The (theta log(theta / theta)) term taken from Expression and the sin(C sqrt(theta)) 'term also from Expression The term is added at the end.
Here the list of proposed expressions JSON format:
(Assistant) Formatted
"jsson
[eIog sqrt(theta C)) C)"
"(sin(C sqrt(theta)) (theta log(theta / theta)) + C)",
JSON
"exp(C sqrt(theta)) sin(C sqrt sqrt(theta))",
Summary
"log(theta log theta)) C sqrt(theta C)", "(sin(C sqrt(theta)) A (theta log(theta / theta)) []"
Figure 4: LLMCROSSOVER prompt with an example output. LLMMUTATION and LLMINIT follow the same structure but with slightly different wording and with one and no reference expressions, respectively. Variables within double braces are replaced with the instance specific arguments. These prompts are available in prompts/ *, txt in the linked repository.
16

---

(System)
You are helpful assistant that hypothesizes about the underlying assumptions that generated list of good and bad mathematical expressions detailed ways, My ultimate goal is discover what assumptions generated the observed good mathematical expressions and excludes the bad mathematical expressions, Focus more the good expressions, their mathernatical structure, and any relation to physical concepts. Note that capital C represents an arbicrary constant.
Domain
Specific
Header
Good Expression 1: {(gexpr))
Good Expression 2: {geexpr))
Expression {{gexpa)
Good Expression 4: ((gexpr4))
Good Expression 5: ((gexpr5))
(User)
Bad Expression 1: ((bexpr1))
Concept
Bad Expression ((bxppr))
Abstraction
Bad Expression 3: (bexpr3))
Expression ((bexpr4))
Prompt
Bad Expression ((bexpr5))
Propose ((N) expressions that would be appropriate given the suggestions and expressions. Provide short commentary for each your decisions. with JSON enumeertts proposed expressions following this format:
"json
(User)
("hyp1".
JSON
"hyp2",
Formatting
"Nyp((N)))
Instructions
Hypothesis 1: The good expressions involve complex exponential or trigonometric functions with nested power operations, potentially indicating link to physical phenomena that exhibit exponential growth or oscillatory behavior.
The presence sine and cosine functions in the good expressions suggests a connection to periodic phenomena in real-world applications, such as wave functions harmonic oscillations.
Hypothesis 2:
(Assistant)
Synthesized Concepts with Explanation
Hypothesis 3: inclusion arbitrary constants the good expressions may represent parameters in physical models, implying that the expressions are modeling dynamic systems or relationships between variables.
Hypothesis 4: The good expressions involve repeated application of matherntical operations (e.g.. exponentiation, multiplication, and addition) in layered manner, indicating depth of complexity that mirrors intricate physical systems natural processes.
Hypothesis 5: The mathematical relationships within the good expressions exhibit balance between simplicity and complexity, suggesting that they capture fundamental principles or underlying structures that govern physical phenomena.
Here the proposed expressions in JSON format:
'Jjson
['The good expressions involve complex exponential trigonometric functions with nested power operations, potentially indicating link to physical phenomena that
exponential growth or oscillatory behavior."
"The presence of sine and cosine functions in the good expressions suggests connection to periodic phenonena real-world applications, such as wave functions harmonic oscillations.",
(Assistant)
Formatted
"The inclusion of arbitrary constants in the good expressions may represent parameters in physical models, implying that expressions modeling dynamic systems or relationships between variables."
JSON
Summary
"The good expressions involve repeated application of mathenatical operations (e.g., exponentiation, multiplication, and addition) in layered manner, indicating a depth of complexity that mirrars intricate physical systems or natural processes."
"The mathenatical relationships within the good expressions exhibit a balance between simplicity and complexity, suggesting that they capture fundamental principles or underlying structures that govern physical phenomena."
Figure 5: LLM Concept Abstraction prompt with an example output. The LLM Concept Crossover function follows a similar structure, with modified task description for crossover on concepts.
17

---

helpful assistant that merges and refines ideas about a set hidden mathematical expression in new, interesting, and diverse ways. ultimate goal to discover the underlying properties these hidden expressions. The resulting ideas should be nontrivial conclusion given the previous ideas.
(System) Header
Idea 1: ((idea1)
[(idea2))
(User)
Idea 3: ((idea3)
Idea 4: ((idea4)}
Concept
Idea ((idea5))
Abstraction
Propose hypotheses would be appropriate given ideas, Provide short commentary for each of your decisions. Do not talk about topics related to the simplicity complexity of the expressions. want ideas that are unique and interesting enough to amaze world's best mathematicians. End with JSON list that enumerates the proposed hypotheses following this format:
Prompt
"jsson ("hyp1" "hyp2",
(User) JSON
Formatting
"hyp((N))"
Instructions
Hypothesis 1: Temperature Modulated Exponential Decay in Dynamical Systems, This hypothesis posits the exponential growth decay system is inversely dependent temperature. temperature increases, the decay rate system's state energy function decreases, leading slower approach to equilibrium. Conversely, as temperature decreases, the system's decay rate increases, leading to faster stabilization This relationship suggests that temperature acts as modulating factor the dynamical system's 's behavior, potentially introducing non-linear effects.
(Assistant)
Hypothesis 2: Temperature Driven Phase Transitions Dynamical Systems. this hypothesis, the state energy function dynamical system undergoes phase transitions depending temperature changes. These transitions can be modeled by exponential functions that describe the shift from one state to another, suggesting that critical points the temperature spectrum trigger significant changes system's properties biological
Synthesized Concepts
explain phenamena processes that are highly sensitive to temmeeepan perature variations.
changes
with Explanation
Emperature-Dependeer Dynamical Systems. Here, the hypothesis suggests that state energy function of dynamical system fundamentally linked to entropy, which is itself inversely dependent on temperature. The dynamical system's behavior can be described through entropy-based energy function that exhibits exponential growth decay. This approach merges thermodynamics with dynamical systems theory, proposing that changes in entropy, driven temperature variations, dictate the system's evolution over time.
Hypothesis Entropy-Based Energy Functions
Here the list proposed expressions JSDN format:
(Assistant)
json
Formatted
"Temperature Modulated Exponential Decay Dynamical Systems", "Temperature-Dráven Phase Transitions in Dynamical Systems", "Entropy-Based Energy Functions in Temperature-Dependent Dynanical Systems"
JSON Summary
Figure 6: LLM Concept Evolution prompt with an example output. The LLM Concept Evolution prompt follows similar structure to the concept abstraction and LLM operation prompts with slight modifications.
18

---

niterations=40,
ncyclesperiteratiion555,
pepulations=15,
poputation_size-3] maxsize=30,
binary_operatorsm ("4", "*", "=", "/", "*"],
unary_operatorr==""xp""" "log", "sqrt"" "sin", "cos"l, weightt_raddmmze=0..
mesteed ("tttee_conttrrints="sss""""" {"sin": 8, "cos": 0),
"cos": {"sin": B, "cos": 0),
"exp": ("exp":0, "log": 0),
"Log": ("exp": 0, "Log": 0),
"sart": ("sqrt": 0)),
constraint==("sin's 10, "cos": 10, "exp": 20, "log": 20, "sqrt": 20,
"pow": (-1, 26)),
Figure 7: The PySR hyperparameters used all experiments. Whenever possible, we use the default
PySR parameters.
A.3.3 Hyperparameters
Figure showcases the hyperparameters used for all our experiments. Wherever possible, we use the default PySR parameters. Additionally, LASR introduces three new hyperparameters: (1) % of LLM calls, (2) List user hints, and (3) a dictionary of parameters pertaining to backend LLM communication. Following other methods in SRBench, operators for solving the Feynman equations, excluding special operators like aresin and arctan. These operators seldom required, and removing them speeds up the search process. We generally set the number of iterations to 40. However, certain experiments may demand more or less iterations.
utilize only subset of the necessary
A.4 Dataset Details
A.4.1 Feynman Equations
Dataset:
The Feynman Equation dataset widely adopted benchmark for scientific discovery [4). The dataset consists of 100 physics equations extracted from the Feynman lectures on Physics. Each equation
the form y = [(1,, (22,...) The number of input variables ranges from two to ten, and the dataset provides 100,000 samples for each equation. We compare against publically available methods benchmarked on SRBench [26]. SRBench a continuously updated benchmark which catalogs the performance of various methods the Feynman dataset as well other symbolic regression problems. Specifically, we compare against GPlearn, AFP, AFP-FE, DSR, uDSR, PySR, and the original AI Feynman algorithm 42 represents an ablation of our model without the LLM genetic operations and the concept evolution slightly noisy version of this dataset in order to simulate experimental
28
38
Within this subset, notably, PySR
(Section 3 We evaluate errors common in scientific discovery domains. Specifically, we compare numbers against those reported in and reproduced by SRBench with target noise of 0.001.
Methodology: For the Feynman dataset, we took the equations and the bounds which each variable was sampled at and generated our dataset. Then, we added additional noise of 0.001 to our target variable, following the noise formula detailed in the Appendix A.4 of [26), as well as additional random noise variables with arbitrary names to force the model for proper feature selection. We then evaluate exact matches by looking truth equation. For the ablation graphs, we used the PySR hyperparameter "early_stor stt__sttp__ooo__ cocopppooniiion"" to check if there is "solution" after N iterations.
the predicted equation symbolically simplifies into the ground
A.4.2 Synthetic Dataset
For the synthetic dataset, we ran script that generates uncommon mathematical hypotheses that satisfy our constraints at random. Then, we ran PySR for 400 iterations and found all the equations that PySR performed poorly in, i.c. MSE loss greater than while having complexity less than 20.
19

---

Metric
PySR (106 Iterations, 10 hour timeout)
LaSR (40 iterations)
Exact Solve
59 + 3/100
72/100
Table 5:
An asymmetric comparison of PySR and LASR on the Feynman equations dataset (A.5.1) We run PySR for 10 hours per equation (thresholded to 10% iterations) and compare the exact solve rate with that of LASR run for 40 iterations. We find that PySR is able to discover three more equations, but LASR still substantially outperforms PySR.
For these 41 remaining equations, we then compared LASR and PySR after 20 iterations using the average of their test set 1t² for each hypothesis.
A.5
Additional Experiments
This section highlights additional experiments on various benchmarks to further characterize LASR's performance.
A.5.1 Asymmetric comparison with PySR
A common concern with evaluating genetic optimization algorithms w.r. number of iterations is that the 'hard stop' after certain number of iterations might yield populations that haven't fully converged to their optimal values.
To account for this discrepancy in performance, we conduct asymmetric comparison with PySR on the Feynman Equations dataset. Specifically, we allow PySR to run uninterrupted for 10 hours per equation, with the maximum number of iterations set x 105. We compare the results with that of LASR run interrupted for 40 iterations per equation (hence the asymmetric comparison). The results detailed in Table
Overall, we find that, despite running PySR substantially longer than LASR, LASR still substantially outperforms PySR. This is because PySR, like other evolutionary algorithms, is susceptible to falling in local minima and converges to this local minima extremely fast. This indicates that supplementing 'local search' strategies with LLM guidance is useful for symbolic regression.
A.5.2 Subset of equations discovered by LaSR on the Feynman Dataset
Equation Nomber Reference
Ground Truth Equation
Discovered Equation
Equation (1.5.20)
pix) pou
(100522251 Manisy
0.62112635
Fiuation (T.11.18)
h a,b, + nuyy a.
(raye) + ((onsp)) + (nsgs))
([(ve((at) (iqfo 2.991080 10-1)) 1.374113445 10-*) -r.] con(us) 1.573005)
Equation 51 (1.50.26)
(ealf)) Kicooust ccoe wor).
1.3085427 0.287302222
Eguation 21 18.4)
R 101 mrng
(mtrgsetimura) tay sayiiing
Equation 63 (II. 11.20)
Pol
Equation 96 16)
saaasssuss me 000011104044) J. 1-0011250258
Equation 57 1.bb))
Ey
cO 0.19281955)
(1h000tamoo (220021177
Table 6: A subset of equations discovered by LASR on the Feynman Equations dataset (over PYSR). The equations presented in the form discovered by LASR, and usually reduce to the ground truth equations after some simplification steps. Note that there are minor discrepancies in the variable names between the ground truth equations found in the online Feynman Lectures httt:s::/www.tte. http///www.ymmmaaeecrrrue..ee caltech, edu) and those in our Feynman Equations dataset.
20

---

Ground Truth Equation
Discovered Equation
C-log(ya) + (Vtn + y4 + /Uz)
(y+++C+mm) (92 + C)) + +11
exp
C-y+
(ui (C)C)) ( (v vs))
C (Wi (cos((y) 1 (C) 10)0)1111)))
V/
C
(os(ons(ys))
(uf +C) (Vexp(un) -C) -C)
v (ep(())) y1 (cos(y1) + 22n1)
C
(vi (vE-oxp(eeootv))))
44 ))
exp (log(U4 + 3/2 111))
g/ ((y22 cos(y22) 1/ (U's + 1) 1/2)
(cos v + v/ C)
(ys y2) (1yvs C*- 1/2 1/1 y2) C)
((y2 vs) (ws (3/2 (ys + 1)) + y1) C ys)
(VDz exp (C + / vva)) ((ys us) log(vs))
((V -c) (ou Cus C)) sin sin (vs+t10 +c)) llg
(xx1)))xxp(/112
C V
1/2
(Vr +C)
Vo
(va CDV
exp(x))
W2
cos(C 11))
81
sin
((1 C) (sin (3/4 + +C) y1) (yb y1 C))
V1
/exp(C . ys)
C-
exp(log(Ua) V22)
Us: (C2p://24440 24 2/2
exp Vus + 24) + (B1 (2) C)
Table 7: Qualitative evaluation of LASR on the synthetic equations dataset (A.5.)). We attempt to recover the ground truth equation from slightly noisy dataset generated from the ground truth equations. None of the algorithms we tested (including LASR) are able to recover the ground truth equations, underscoring the challenge of exact symbolic match this dataset. A study on the !t2 performance is presented in Table
A.5.3 Qualitative comparison of Synthetic Dataset equations
In T we reported R² test set performance of LASR and PySR on the synthetic dataset. this section, WC qualitatively compare the equations discovered by LASR and PySR on the procedurally generated dataset of equations. None of the algorithms recover the ground truth form, but we find that LASR's is equations fit much better to the ground truth data than PySR's equations. These equations are presented Table 3
A.5.4 Stochasticity of LASR and PySR
A common concern with using LLM's for scientific discovery is that the equations could be obtained due to dataset memorization rather than reasoning and learning on the equations form. To explore this further, run LASR with the same hyperparameters twice with different seeds. We expect LASR to behave like PySR (or any stochastic evolutionary algorithm) and produce two syntactically different programs that achieve similar data fitness score. However, if the algorithm purely relies on training set memorization, we expect the model to find the same equation form in both experiments. We present results in Table
Overall, both equations fit well to the underlying dataset and reduce to the ground truth. Yet, despite using the same hyperparameters, the functional forms exhibit sharp differences. This further reinforces our hypothesis that LASR's 's performance is not simply the result of regurgitated memorized responses.
21

---

Discovered Equation (LaSR Llama 3b 1%; same params)
Loss
Complexity
1,8
2 x 10-6
16
0.000219-
No. 200050 Loin:e) (-8
1.372847
2 x 10-6
1-(-117.252995 0.001495)
Table 8: Performance of LASR successive runs with the same hyperparameters on Equation 53 in the Feynman equations dataset (A.5.4). This equation defines the heat flow from a point source (an inverse square law with ground truth formulation: h $ा/2z We evaluate LASR by running twice with identical hyperparameters but different random seeds to assess whether the model has memorized the equation's 's form. In both runs, LASR consistently discovers perfforrroriin solutions. Moreover, the resulting functional forms show significant variation between runs, supporting the hypothesis that LASR's performance is not rooted in memorization.
A.6 Using LASR to find an LLM Scaling Law
So far, we have used LASR to discover equations that are already well-established in prior work. In this section, we investigate LASR's utility making novel empirical discoveries. Specifically, we investigate whether LASR can discover novel scaling laws for LLMs.
Motivation: Traditionally, identify an LLM scaling law, practitioners must first manually posit a skeleton equation with fixed set of known variables and unknown free parameters, and then optimize the unknown parameters based on a dataset of model hyperparameters and resulting dataset fitness Instead of starting with predefined equation, we use LASR to discover the skeleton equation that best fits various subsets of the BigBench dataset. Removing the need to manually posit skeleton equation simplifies the methodology for finding scaling laws in many ways. First, removes human preconceptions about the expected relationships between hyperparameters, Second, it increases the number of variables and the type of variables human practitioners can jointly reason about. Finally, enables positing equations of much higher complexity and variable interdependence than otherwise possible.
23 5
Dataset: BigBench is collaborative benchmark intended to probe large language models and ex- trapolate their future capabilities [16]. It contains 204 tasks drawing upon problems from linguistics, cognitive science, math, physics, biology, software development, etc. For each task, Bigbench mea- sures the performance on many model families (OpenAl's GPT models, Google's dense transformer architectures, and Switch-style sparse transformers). Each BigBench task is evaluated on a preferred metric (chosen by dataset creators). In our experiments, we consider subset of tasks where the preferred metric is 'multiple choice grade.' This subset contains the highest diversity of tasks and around 53,812 total data points.
Methodology: We run LASR with 3840 populations (parallelized across 16 cores) with each population evolving 200 candidates evaluated with the Zygote autodifferentiation backend. We ran LASR overnight (for 7 hours). BigBench necessitates optimizing matrix of parameters rather than singular scalar constants. This shifts the compute bottleneck from generating pool of candidates (in which LASR is slower than PySR) to evaluating pool of candidates (which is equally slow for both algorithms). We also impose regularization constraints to accelerate the optimization procedure and avoid degenerate equations. Specifically, we expect a monotonically increasing performance trend with respect to the number of training steps.
Results: LASR discovers the following scaling law on the subset of BigBench:
-0.0248235
score
#abota + 0.360124
(5)
prrin_ssepss 116050.90
Where score is the MCQ grade, train_stepp is the number of training steps for the model, and #shots is the number of in-context examples provided during inference. LASR maintains Pareto frontier of the simplicity-aceuracy tradeoff between various best performing programs. We present all the programs in the pareto frontier of this experiment in Table
22

---

Complexity
Equation
A
B
6
A+
(-C)
8
A+
D
training wtwpssttpp.pp.ppoooco "0n.
10
A+
Sraining-Attg Atop
13
A+E
training steps+
16
A+G
(+ (app))
H
(trainingg_eree stops
17
E+
training s steps SOatErR (age)1
Cool lroining steps)
19
A+M
P)
(training stops Q)
21
A+M
(training.atape
25
A+G
training stapa K+(
H+ooot Awam Achadding sarss))
R
J 5T
26
Seriiinngsteppa-- K+(
H.+cout 2E0 Ebeddin param
.T
(CRINIR sleps)
28
A+G
(training_stapa K (
perts)
J J-
Froosttuunng Ju4po)
30
A+G
training.oteps K-
paroea) J-
32
(A+G)-
training.stepp K
Food Aontttttttll...o param 1+ [##oossralttngg.oog
Table 9: Pareto frontier of LASR for the BigBench experiment (Section A.6 Equation is shown as the equation with complexity 10 in this table.
Qualitative Evaluation: Equation has some interesting properties. First, it describes empiri- cal relationship between training typerparameters (training steps) and inference hyperparameters (number of shots). It asserts that increasing the number of shots exponentially increases the model's performance for low-resource models while having diminishing gains as the number of training steps of the model increase. Second, Equation Jonly requires three free parameters (Chinchilla (23 required five free parameters). This might allow LASR 's equation to better generalize to different LLM operation regimes. However, the equation has some issues well. If the number of shots is an even number, the exponentially increasing trend reflects over the value A and turns into an exponentially decreasing trend. A manual inspection of the training data reveals the majority of samples had an odd number of shots (1, ....) which causes the generalization error.
Quantitative Evaluation: We compare Equation 4 Chinchilla [23 Modified Chinchilla and single free parameter equation on their ability to explain the score of a held out validation set 10,763 data points. We fit the free parameters of each equation to the training set data and measure the MSE loss between the actual grade and the predicted grade on the validation set. The results are presented in Table 4 Overall, we find that the Equation 4 performance, well modified Chinchill'ss ' performance, is competitive with that of Chinchilla's 's in predicting the MCQ grade.
We hope our preliminary results provide further insight into the practical value of LASR in addressing real-world challenges, including those in machine learning
A.7 Metries for Cascading experiment
For the cascading experiment, we aim to evaluate the progression of different configuration towards solving equations. The quantitative metric used in the SRBench comparison experiment, Exact Solve, does not allow for such fine-grained analysis. Therefore, we categorize the synthesized equations into four buckets: Exact Solve, Almost Solve, Close, and Not Close. Exact Solve is quantitatively evaluated using symbolic match. equation tagged as 'Almost Solve' if the dataset loss small, but the generated equation has an extra term lacks one term. A Close equation captures the
23

---

general structure of the solution (such as a square root nested in an exponential) but not more than that, and Not Close includes all equations that are far from the solution.
A.8 Further Qualitative Analysis
LASR generates two artifacts: the best fit program, and the library of natural language concept that helped find that program. These artifacts provide unique window into the inner workings of LASR. This section goes over a qualitative study how LASR and PySR go about discovering Coulomb's law F inr2c from data. Both methods are able to find answer to this equation. However, their approach to finding the best fit equation as well as the form of the equation they discover differs significantly.
Setup: Coulomb's law is equation #10 the Feynman equation dataset. It describes how the force between two point charges changes with respect to the distance between the charges, the magnitudes of the charges, and the permittivity of free space constant. The corresponding data for this equation has target noise of 0.001 to simulate experimental errors.
By analyzing the form of the equation and relationships between variables in Coulomb's law, we can uncover several interesting properties: First, observe that this is an inverse square law (The force F varies inversely with the square of the distance r between the charged particles). Second, notice that the F is directly proportional to the magnitude of the charges Q1 and 42. Third, observe that the resultant force symmetric with respect to the magnitude of the charged particles (i.e.: The magnitude of the F doesn't It change if the magnitude of the charged particles is swapped).
PySR Solution: PySR finds the following solution to this equation:
02-3.382
stnl eEpLLFT exp(CC)
0.00T.
/0.712
0.087 /e
0.191
F
This equation has a complexity of 26 and achieves loss of 2.191505 x 10-12 12 on the dataset. Obtaining a simplification of this solution is rather painstaking.
LASR's Solution: LASR finds the following solution to this equation. We also present three steps of simplification:
F
Q1
0.07957782
r
1.9181 61N166661100
Q1
(Substitute constant)
r
19181133 10-5
6
4m
Q192
1
(Simplify denominator)
r 1.9181636x 10
4m
9192 1
(Negligible.
1.9181636x10 0)
r (r) 4m
42
This equation has a complexity of 15 and achieves a much lower loss of 4.6709058 x 10-14 on the accompanying dataset. We can see with just three steps of simplification how this equation might be reduced to the ground truth.
Let's examine some essential concepts from various iterations in the search process. Keep in mind that an LLM operates on tokens in each concept. Consequently, even small relevant substrings can positively influence future LLM inference calls, despite full concepts appearing verbose humans.
1. Iteration The good mathematical expressions exhibit a clear and coherent relationship be- tween the variables involved, with a focus power functions and trigonometric cinirnottttrifffiiionsss that can be easily related to physical concepts.
2. Iteration 6 The good mathematical expressions exhibit a certain level of symmetry or regularity their form, possibly reflecting underlying patterns or relationships between the variables and constants.
24

---

3. Iteration 24: The good mathematical expressions have a clear and consistent structure involving the variables ql. q2, epsilon, C, and with a specific pattern of division and multiplication.
25

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant
information is Not Available.
Please provide a short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction spell out the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer this question will not be perceived well by the reviewers.
The claims made should match thcoretical and experimental results, and reflect how much the results be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the conclusion section.
26

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., the approach was only tested on few datasets with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide a link to an open source implementation of LASR in Julia.
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reprodueible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary either make possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
model (e.g., large language model), then there should
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e-g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Same answer as above. We also include README start the codebase. Eventually, we'd like to merge our contributions into larger repository (PySR).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NewrIPS code and data submission guidelines (https
/ /nipss.cc/pulicc/uudde http:///c/ggiieee//ooouuuubiaias wwwp///ooeesssiiiiooooe ssisssoiccyccyiiioooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
28

---

Providing much information as possible in supplemental material (appended to the paper) is recommended. but including URLs data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e-g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We'v ve put best efforts in ensuring the research work is comprehensive enough that all experiments can be replicated. Nevertheless, for any missing details, we refer the reader to our open sourced repository and happy make changes enhance comprehensiveness.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our main evaluation metric is exact match of synthesized equation w.r.t reference equation. This statistic doesn't t permit a level of granularity at which we can show error bars. An important source of uncertainty is backbone LLM. We' ve painstakingly engineered our experiments to use local LLMs at the cost of modest performance to enable reproducibility. However, certain experiments necessitate proprietary models whose internal details may change.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-gimm error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
29

---

Answer: [Yes]
Justification: We mention our experimental setup details as well as other experimental setups we have successfully tried.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect. with the NeurlPS Code of Ethics httpp:::/earrrs..ccppbbi///////s
Answer: [Yes]
Justification: Yes. We've made sure our submission follows with the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g., if there is special considera-
tion due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We talk about broader impacts in the appendix.
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e-g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
30

---

Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our contributions do produce artifacts like trained neural network parame- ters that can be misused.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our codebase is built on top of many prior works in SR (PySR, Prompting- Tools.jl, and SymmbolicRerresionn. We ve attributed them appropriately in accordance with their respective licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release all code and provide a README of how to instantiate the codebase. We hope merge tool into existing state-of-the-an symbolic regression framework, which necessitates comprehensible organization as well.
Guidelines:
The answer NA means that the paper does not release assets.
31

---

Researchers should communicate the details of the dataset/code/mmodl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "67KvUEeSkS", "reviewer": "Reviewer_1Xi9", "summary": "This paper introduces LASR, a symbolic regression framework that enhances PySR by incorporating Large Language Models (LLMs) to discover and evolve \"concepts\" from high-performing equations. These concepts are then used to guide the search process. LASR is evaluated on the Feynman equations dataset and a set of synthetic tasks, showing improved performance over existing symbolic regression baselines.", "questions": "* Could you provide the 7 additional equations that LASR solves beyond PySR as well as the forms found by LASR? Are these 7 equations consistent across runs? What have been specific contributions of the LLM for them?\n\n* How can you ensure that the LLM is not implicitly using its prior knowledge to generate simple Feynman physics equations? The ablation study shows that variable names and hints improve performance, which raises concerns. For instance, variable names like \"g\" for gravity or \"Kb\" for the Boltzmann constant could trigger the LLM to suggest the well-known relevant equations.\n\n* Have you conducted ablation studies that remove all physics-related terms, variable names, and any mention of physics or internal knowledge from the prompts? This would help (to some extent) isolate the LLM's ability to learn purely from data patterns.\n\n* For equations solved by both LASR and PySR, could you provide the number of iterations required to obtain the correct form using each method?\n\n* Could you clarify what constitutes an \"iteration\" in your experiments? There are discrepancies between Figure 1 (10^6 iterations), the main experiments (40 iterations), and the synthetic data experiments (400 iterations). Could you explain these differences and provide results for more iterations (maybe around 10^6 iterations), particularly for the synthetic dataset? Additionally, could you clarify how the concept library operations frequency (every 400 iteration based on Figure 2), and LLM usage frequency (p=1%) translate to these small iteration counts? For example, for 40 iterations, how many time LLM operations and concept library operations will be used?\n\n* For the data leakage validation, why were custom synthetic equations used instead of existing datasets like black-box datasets in SRBench? This would provide a more standardized comparison and better demonstrate LASR's capabilities on real-world problems.\n\n* Could you provide a computation time comparison between LASR and PySR for experiments with more iterations, to assess the computational trade-offs of using LLMs?\n\n* Could you provide examples of \"hints\" that are provided in experiments of Feynman equations?\n\n* Based on Table 2, increasing the value of p seems to improve performance. Have you considered analyzing higher values of p, especially for GPT-3.5? Could you explain why higher values were not explored?", "limitations": "See above.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The idea of integrating LLMs into symbolic regression to learn and use abstract concepts in terms of natural language is interesting. The methodology is well-structured  with clear explanations of the algorithm components.", "weaknesses": "My main concerns are regarding the evaluation. Experimental design and analysis are insufficient to convincingly demonstrate the method's advantages over existing approaches; specifically: There are serious concerns about potential data leakage and unfair advantage when using LLMs on well-known, simple physics equations that may be part of LLM training data. While there is a section on the data leakage validation, its limited evaluation scope and lack of comprehensive analysis on more complex and real-world datasets, makes it difficult to assess the true capabilities and generalizability of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work! We address questions inline:\n\n> **Could you provide the 7 additional equations that LASR solves ...?** \n\nWe’ve added a table containing these equations and the ground truth equation to the global PDF for figures.\n\n> **Are these 7 equations consistent across runs?**\n\nYes, LaSR discovers these equations consistently across runs.\n\n>  **What have been specific contributions of the LLM for them?**\n\nDue to the nature of evolutionary algorithms, it is extremely challenging to trace how each operation (whether it be an LLM-based operation or a symbolic operation) interacts with other operations to evolve a candidate pool. Intuitively, the LLM operations help introduce new candidates in the population that respect the natural language hypotheses (whether learned or user provided). LLM-biased candidates that are a good fit are retained by the evolutionary search, while bad candidates are discarded. \n\nWe refer the reviewer to the qualitative study in the Appendix of a Feynman equation task, where we compare the equation LaSR finds with the equation PySR – an ablation of LaSR without the LLM operations – finds. Notably, we find that LaSR’s equation is more interpretable, yet is very far from the common formatting of the equation (available on the internet and, possibly, in the training dataset of the LLM).\n\n> **How can you ensure that the LLM is not implicitly using its prior knowledge ...**.\n\nWe refer the reviewer to Figure 3 (Left) where the ablation of LaSR *without the variable names and the hints* still outperforms PySR.\n\nWe find that more language guidance consistently improves LaSR’s performance (by adding variable names, by adding hints, by using a larger model, or by calling the LLM more frequently). \n\nFurthermore, In Appendix A.4.1, we analyze the equations discovered by PySR and LaSR from data corresponding to Coulomb's law. If LaSR’s performance was rooted in regurgitating memorized data, LaSR’s equation would correspond to the common formatting of Coulomb's law (F= (1/(4 * \\pi * epsilon)) q_1.q_2/r^2). Instead, we observe that LaSR’s discovered equation displays a very different format, and needs at least five simplification steps to transform to the commonly seen equation.\n\n> **Have you conducted ablation studies...**\n\nYes. Figure 3 (Left) depicts an ablation of LaSR without any physics knowledge (variable names or hints). This ablation still outperforms PySR. Also, one of our prompts contains the term “physical concepts.” We’ve verified that inclusion/exclusion of this phrase has no impact on LaSR’s performance.\n\n> **For equations solved by both LASR and PySR, could you provide the number of iterations required to obtain the correct form using each method?**\n\nFigure 3 tracks the number of equations solved after each iteration for PySR, LaSR, and various ablations of LaSR. Our experiments are fully reproducible, and we plan on releasing our logs with our codebase, which should highlight more details.\n\n> **Could you clarify what constitutes an \"iteration\" ... There are discrepancies between ... operations will be used?**\n\nAn iteration is one full run of the SR cycle, and the concept crossover steps (this is best expressed in Algorithm 1). $10^6$ in Figure 1 is the total number of *operation calls per iteration*. As mentioned in Appendix 3.1, LaSR makes $60000$ operation calls per iteration, $60000$ calls/iteration * $40$ iterations = $2.4 \\times 10^6$ calls. We have updated Figure 1 to say “$10^6$ operations” instead of “$10^6$ iterations.” Thank you for pointing this out!\n\nWe cannot run LaSR (or PySR) for $10^6$ iterations, as that would take roughly $10^6$ to $10^7$ minutes on an 8 x NVIDIA A100-80GB server node. As mentioned in Appendix 3.1, assuming $p=0.01$, we will make $60000$ calls per iteration so $0.01 * 60000 = 600$ LLM operation calls (roughly) per iteration.\n\n>  **For the data leakage validation, ...  black-box datasets in SRBench?**\n\nGreat question! We wanted to compare PySR’s performance and LaSR’s performance on a set of unknown problems with ground truth equations available, which is why we chose to procedurally generate our own equations. The black box datasets in SRBench do not have ground truth equations.\n\n> **Could you provide a computation time comparison?**\n\nOur goal is to present experimental results that can be replicated reliably by the broader community. As such, we root our experiments in the number of iterations instead of wall time.\n\nThis is because the wall time is heavily dependent on the LLM backend, the number of tokens each query uses, and the network quality. For instance, for `gpt-3.5-turbo` we cannot reliably measure the runtime as the backend is slower during periods of high traffic. Even for local models, vLLM's query scheduling is non-deterministic, so the wall time can be substantially different across experiments.\n\nRegardless, we were measuring around one second per query using vLLM. Since we make 60k calls per iteration (Appendix A.3.1) the amount of time for p=1% would be around 600 seconds / 10 minutes per iteration. PySR roughly takes 30 seconds per iteration. We expect performance to improve as we increase CPU/GPU parallelism, run LaSR on dedicated backend hardware, and as the hardware and software to run LLM inference improves.\n\n> **(...) examples of \"hints\" in Feynman equations?**\n\nAs mentioned in Section 4.4, we use the title of the chapter the equation appears in in the Feynman Lectures on Physics. For instance, the Shockley diode equation is introduced in the “Semiconductors” chapter of the Feynman lectures. Hence, the hint would be “Semiconductors.”\n\n> **Based on Table 2, (...) Could you explain why higher values were not explored?**\n\nAs we mention in the limitations sections, our evaluation was constrained by our compute budgets for LLMs and search. Whether these trends hold for higher compute regimes remains to be seen.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vdLYs1ZufQ", "reviewer": "Reviewer_2Egm", "summary": "This work focus on symbolic regression. They enhaned the traditional method like genetic algorithms by inducting a library of abstract textual concepts. The algorithm, called LASR, uses zero-shot queries to a large language model to discover and evolve concepts occurring in known high-performing model to discover and evolve concepts occurring in known high-performing hypotheses. The algorithm can be seen as a kind of hybird of evolutionary algorithm and LLMs. Through experiments, LASR substantially outperforms a variety of state-of-the-art SR approaches.", "questions": "1. What's the operation of LLMINIT? If we do not have any prior knowledge, then what will we prompt LLMs to do the initialization? Is it the same when we nothing more to know in advance.\n2. The initilization, mutation and crossover operation in LASR hybrid traditional evolutionary algorithms and LLMs, so what's the intuition to do this? Because there are several works that directly replace genetic operation with LLMs operation, such as (https://arxiv.org/abs/2401.02051). I am curious about have you ever tried to use merely LLMs for the whole process, that would be an interesting topic.\n3. Is LASR capable to resolve high-dimentional symbolic regression tasks? Because current research about symbolic regression struggle to due with high-dimensional settings.\n4. Have you ever tried to compared the work KAN (https://arxiv.org/abs/2404.19756), which can also be used in symbolic regression and report a great results.", "limitations": "See weakness and question parts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This work proposed to introduce a concept library in symbolic regression, which is really similar to how human works, so the idea make sense. For the introduction of the library, this work also leverage abstrction or understanding ability of LLMs to design three phrases process, concept evolution, hypothesis evolution and concept abstraction. The design mix the strenghes of evolutionary algorithms and LLMs. \n2. The experimental results are good comparing to those other baselines of learning-based or evolutionary-based.", "weaknesses": "1. Introdution of LLMs would inevitably raise the cost for the task comparing those traditional algorithms.\n2. (this could be a question) The introdution of concept library seems to not making sense in all cases, imaging that we are facing some totally unknown black-box environment, the backbone function could be something random or out of the knowledge we have. In this setting, the traditional evolutionary algorithms might perform better because they do not have this kind of knowledge as the constrant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work! We address questions inline:\n\n>  **Introduction (...) algorithms.**\n\nWe agree that LLMs might require additional compute, which can raise the cost. However, this is a worthwhile tradeoff for scientific discovery for a couple of reasons:\n\n- The expected reward of finding even one additional equation is extremely high. As LaSR consistently outperforms PySR, the additional cost of running LaSR is worth the tradeoff. \n- We are optimistic that the cost of running LaSR will further decrease – not increase – as local language models become faster, smaller, more accurate, and cheaper in the near future `[1]`. We are even able to run LaSR on a laptop with the latest quantized models!\n- LaSR usually considers a *larger* space of equations than a traditional symbolic method in the same number of iterations. Furthermore, LaSR produces two artifacts: an equation of best fit as well as a library of concepts the method finds relevant to discovering that equation. These provide invaluable context information to a scientist.\n\n> **(...) black-box environment, the backbone (...) traditional evolutionary algorithms might perform better because they do not have this kind of knowledge as the constraint.**\n\nThis is exactly the setting of our Synthetic dataset experiment. We procedurally generate novel synthetic equations, and their respective data points, that lie outside the distribution of known equations, and find that LaSR is able to solve more equations than PySR – an ablation of LaSR without language guidance.\n\nThe reason LaSR works in such scenarios is because LaSR is a *neurosymbolic* approach which uses a classical evolutionary search as the backbone and then uses the LLM to direct this search. In the synthetic domain, the novel synthetic blackbox equations generated through the evolutionary search still exhibit abstract mathematical trends, and LLMs can discover these trends, and then use these trends to guide further search.\n\n> **What's the operation of LLMINIT? If we do not have any prior knowledge, then what will we prompt LLMs to do the initialization? Is it the same when we nothing more to know in advance.**\n\nGreat question! In the absence of prior knowledge / hints, we fall back to the symbolic initialization function. The full prompt is available in the linked repository.\n\n> **Have you ever tried to use merely LLMs for the whole process, that would be an interesting topic.**\n\nThank you for the reference! Using an LLM for the whole process would not be ideal in cases where we do not have much prior concrete knowledge about the problem. For instance, in a completely black box setting, in LaSR, the LLM helps guide the evolutionary search using abstract concepts extracted by the classical evolutionary operations while exploring the “local” search space.\n\nNevertheless, LaSR paves the way to try out exciting techniques from the prompt programming `[2]` / prompt tuning community for solving scientific discovery problems.\n\n>  **high-dimensional symbolic regression tasks**\n\nFor high dimensionality problems, the SR community relies on the model distillation paradigm `[3]`. Here, we first fit a sparse GNN (or any large parametrized network with a sparsity constraint) to the data and then distill each network subcomponent into an independent equation. As LaSR is a drop-in replacement for PySR, a practitioner can use LaSR with this methodology to learn equations in high-dimensional tasks with almost no code changes. We leave this for future work.\n\n> **Comparison with KAN**\n\nGreat suggestion! KAN’s `[4]` take a different, orthogonal approach as compared to our work for symbolic regression. Symbolic regression with KAN’s relies on a two-step model distillation strategy similar to `[3]`. First, they fit a sparse KAN to the dataset and then they use a “search” component to find the closest fit symbolic function for each learned activation (the search component here is iterative greedy refinement).\n\nConsecutively, the closest direct comparison of LaSR with KAN’s would be to measure the efficacy of greedy iterative refinement compared to LaSR for model distillation. This is an exciting direction for future work!\n\n`[1]`: https://arxiv.org/abs/2407.21783\n\n`[2]`: https://github.com/stanfordnlp/dspy\n\n`[3]`: https://arxiv.org/abs/2006.11287\n\n`[4]`: https://arxiv.org/pdf/2404.19756", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c8SdixJUsu", "reviewer": "Reviewer_mCEz", "summary": "This paper introduces a method that learns a library of concepts (natural language description of helpful and unhelpful concepts) as a means of guiding genetic search for symbolic regression. The core idea is that such concepts can be used to bias genetic operations through an LLM.\n\nThe method was evaluated on the 100 Feynman equations and on a synthetic dataset. The paper also includes ablation studies on the various components of the system.", "questions": "1. In line 83, shouldn't it be $P(C)$ instead of $P_C$?\n\n2. In the mutation operator, it is stated that for each deleted subtree, one adds a single leaf node (lines 111-112). Is this correct? Why not add an entire new subtree?\n\n3. In the for-loop in line 4 for Algorithm 1, shouldn't it be $I$ instead of $N$?\n\n4. Why not provide a figure in the appendix to describe Concept Evolution (similar to figures 4 and 5)?\n\n5. How does that cascade work for PySR? (the one that didn't work)\n\n6. Why use MSE and not the exact correct metric in Section 4.3?\n\n7. Why is the difference between PySR and LaSR so large in Table 3, but somewhat small in Table 1? \n\n7. Weren't the sentences in Section 4.4 swapped? To me, the first sentence looks more refined than the second. This is because it mentions \"waveforms or periodic phenomena,\" while the second only talks about \"scientific phenomena.\" Perhaps the argument in that section needs some rethinking?\n\n8. In the prompt shown in Figure 5, how much is the excerpt \"relation to physical concepts\" contributing to the quality of the concepts?", "limitations": "The paper lists all limitations I could think of, either in the last section \"limitations\" or throughout the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a creative form of using LLM to speed up the search of genetic algorithms for symbolic regression. Instead of simply storing a library of programs, as GA algorithms do, the algorithm also stores a library of natural language concepts. Such concepts can be seen as abstractions of the population of programs encountered in search. Given in natural language, the abstractions can be used to drive the search as an LLM can be used to generate programs based on the description. Such a creative approach!\n\nThe idea presented in this paper is general and can be more broadly applied to other problems. I can already see how I could use a similar approach in my own research!\n\nAnother strength of the approach is the author's care with data contamination. Initially, I was skeptical of the approach as it uses LLMs to solve problems whose solutions are available online. The authors then explain that the way the LLM is used is unlikely to allow it to simply retrieve the solution from its training data. The explanation makes perfect sense since the LLM is used to extract concepts from programs the GA generates, and they can't encode the solution available online. In addition to this explanation, the authors also included an experiment on synthetic data showing the advantages of the learned concepts over the search alone. Nicely done!\n\nI also enjoyed the fact that the system is built on top of PySR, which is a very efficient system for symbolic regression. This eliminates the possibility that all the gains the LLM provides could be easily washed with clever engineering. The current results already show that clever engineering alone is outperformed by the system.", "weaknesses": "The paper also has a few weaknesses.\n\n**Claims that need to be fixed**\n\nSome of the claims in the paper are a bit strong and I suggest toning them down. While the leakage explanation the authors provided is reasonable, I would be careful in claiming state-of-the-art performance. When writing \"LASR achieves a higher exact solve rate than all other baselines,\" it is worth mentioning the possibility of leakage.\n\nAnother claim that seems to be incorrect is the following: \"LaSR's increasing the backbone model size and the mixture probability significantly enhances.\" I think the authors meant to write \"substantially\" and not \"significantly\" as there is no statistical test involved. I would also explain why these results are substantially better as the number of problems solved isn't much larger. The explanation I gave to myself is that solving each of these equations is very difficult, so solving one new equation is already quite an achievement.\n\nAnother claim that needs to be adjusted: \"demonstrating that LaSR's performance gains are not rooted in memorized responses.\" The experiments on the synthetic dataset do not demonstrate this. The experiment with the synthetic dataset is almost independent of the experiment with the 100 Feynman equations. What the experiment demonstrates is that LaSR can outperform PySR even when data leakage is not possible.\n\n**Discovering what is known**\n\nPerhaps an unfair criticism of the paper is that the method it introduces is used to discover things we already know. I understand that the bar would be way too high and it would be unhealthy to the research area if we required the discovery of new things with the presentation of novel approaches to scientific discovery. So I do not make this criticism as a means of arguing for rejecting the paper (I think the paper should be accepted), but more as a reflection of what the community has been pursuing. The hope is that systems such as PySR and LaSR will eventually be used to make actual discoveries.\n\n**Lack of Running Time**\n\nI missed the running time of LaSR and PySR in the paper. How do they compare with LaSR making calls to an LLM?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work! We address all questions inline:\n\n> **Claims that need to be fixed.** \n\nWe would be happy to adjust these claims along the lines you suggest. Thank you for pointing this out!\n\n> **In line 83, shouldn't it be P(C) instead of P_C?**\n\nGood catch! Fixed.\n\n> **In the mutation operator, it is stated that for each deleted subtree, one adds a single leaf node (lines 111-112).**\n\nThis isn’t correct. We’ve updated the explanation for `delete_subtree` on line 111. In PySR’s implementation, the delete_subtree function randomly selects a single node from the subtree, deletes that node, and replaces the deleted node with one of its children (selected randomly).\n\n>  **In the for-loop in line 4 for Algorithm 1, shouldn't it be I instead of N?**\n\nYes, it should be. Fixed in the latest manuscript.\n\n> **Why not provide a figure in the appendix to describe Concept Evolution (similar to figures 4 and 5)?**\n\nThank you for the suggestion! We added a figure for concept evolution in the appendix. The prompt should be available in the provided repository.\n\n> **How does that cascade work for PySR? (the one that didn't work)**\n\nThe PySR cascade operates in a similar fashion to the LaSR cascade. We run PySR for 40 iterations, prune equations that are below a threshold solution, and repeat the process the same number of times as we do for LaSR. Pruning and rerunning PySR should have no effect on the performance but we do this anyway for completeness.\n\n>  **Why use MSE and not the exact correct metric in Section 4.3?**\n\nDue to rounding errors of the data, waiting for a MSE error of exactly zero (an exact match) is not ideal. \n\n> **Why is the difference between PySR and LaSR so large in Table 3?**\n\nThank you for pointing this out. Randomly generated synthetic equations wouldn’t be informative about the capabilities of LaSR or PySR. To properly verify the utility of language guidance, we generated the synthetic dataset equations to be within the upper limit of PySR’s performance. To construct this dataset, we first generated equations which, anecdotally, contain many characteristics that PySR struggles with. Then, we verified PySR’s performance on this dataset for 400 iterations to ensure there weren’t trivial bugs in the equations (such as the equation evaluating to zero). This left us with 41 equations, for which we report LaSR’s performance with the same hyperparameters. We’ve edited the synthetic dataset section to clarify the data construction process, which explains PySR’s poor performance. \n\n> **Weren't the sentences in Section 4.4 swapped? To me, the first sentence looks more refined than the second. This is because it mentions \"waveforms or periodic phenomena,\" while the second only talks about \"scientific phenomena.\"**\n\nThe second sentence is, in fact, refining upon the first sentence in some ways. For instance, the second sentence mentions more mathematical operations (multiplications, division, and trigonometric operations) than the first sentence.\n\nThis is a common challenge with using LLMs: some parts of the generated concept undergo more refinement than other parts do. And it’s hard to quantitatively contrast two natural language concepts. Nevertheless, this additional generation flexibility is advantageous in many situations because it enables the LLM to abstract similar parts of the sentence between iterations.\n\n> **In the prompt shown in Figure 5, how much is the excerpt \"relation to physical concepts\" contributing to the quality of the concepts?**\n\nWe didn’t find any performance difference including/excluding that phrase from the prompt. In our experience, the instruction fine tuned models are most sensitive to the initial instructions, the formatting instructions, and the bullet point list of concepts/equations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VcsXuIzPDy", "reviewer": "Reviewer_CCz9", "summary": "This paper proposes a way to incorporate LLM prompting to improve symbolic regression. They uses PySR, a standard symbolic regression library, as their base SR algorithm. Then they add LLM prompts to different SR algorithm steps: population mutation, crossover, and initialization. They replace the PySR implementation with LLM prompted implementations 1% of the time. The LLM prompts are based on identifying high level concepts and prompting the LLM to do its operation using the concept as a suggestion to follow. High level concepts are tracked using an abstraction prompt given high performing expressions, and are evolved using LLM prompting as well. The authors show that augmenting PySR with LLM concept-based prompting solves around 7/100 addition tasks from the AI Feynman SR benchmark, and show that PySR + LLM (LaSR) performs better on a synthetic task designed to test for data leakage.", "questions": "See weaknesses section. \n\nSome other suggestions:\n- fix backwards quotations marks", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Well written, great figures\n- The framework for integrating LLM-based SR into PySR is well-designed, and could in principle work for other prompting approaches besides the concept-based SR. The LLM-mutation, crossover, and initialization steps could be replaced with other LLM based techniques. Cool!\n- Using LLM's to learn concepts to guide SR is a well-motivated choice of prompting technique, given the importance of high level concepts for human equation discovery\n- Based on the literature review comparing LaSR to two other LLM SR tools, LaSR seems like an original contribution.\n- Given existing literature on program synthesis with library learning, LaSR is a great approach that bridges the gap a bit between SR and program synthesis, as done with modern tools.\n- LaSR is also a good contribution to the growing body of work on library learning and its benefits for search. It is very similar to LiLO, which is built off DreamCoder, but applied to symbolic regression.\n- The authors have a strong analysis of LLM incorporation based on (1) algorithmic cost in terms of millions of tokens per iteration, and (2) comparing GPT 3.5 with open source llama 8b on the results.", "weaknesses": "- It's not clear how valuable the concept abstraction approach is compared to some \"baseline\" of simple LLM prompting. For example, one baseline could just use a single concept \"This expression is a good formula for symbolic regression\" or something like that, and see how it compares. This could perhaps be a direction for future work: try a bunch of simple prompting strategies for combining LLM's with PySR, and report how well each of them work. \n- I'm not sure what to make of the synthetic dataset. In particular, PySR works so well on AI Feynman alone, but works very poorly without the LLM addition on this synthetic dataset. Why does LaSR work so much better than PySR here, but not help as much on AI Feynman? One hypothesis is that AI Feynman has a lot of easy tasks, and the synthetic benchmark only has tasks right on the edge of what PySR can discover, which LLM incorporation helps push over the edge. Another pessimistic take is that the synthetic benchmark is designed with LaSR in mind. While this still eliminates worries on data leakage, an explanation here would help understand better how LaSR is being helpful. \n\nI'd like to emphasize that including answers to these questions (both of which could suggest negative results) in the paper would not decrease my review score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work! We address your questions inline:\n\n> **It's not clear how valuable the concept abstraction approach is compared to some \"baseline\" of simple LLM prompting. For example, one baseline could just use a single concept \"This expression is a good formula for symbolic regression\" or something like that, and see how it compares. This could perhaps be a direction for future work: try a bunch of simple prompting strategies for combining LLM's with PySR, and report how well each of them work.**\n\nGood suggestion! This strategy – fixing a static (but highly domain specific) prompt – usually causes the LLM to regurgitate low performing programs and requires a bigger, slower model to work properly. One of our ablations is very close to the suggested experiment: in Figure 3 Left, we compare against LaSR with no concept evolution (in this case, “the fixed prompt” is just the system prompt) . Nevertheless, LaSR paves the way to try out exciting techniques from the prompt programming `[1]` / prompt tuning community for solving scientific discovery problems.\n\n**I'm not sure what to make of the synthetic dataset. In particular, PySR works so well on AI Feynman alone, but works very poorly without the LLM addition on this synthetic dataset. Why does LaSR work so much better than PySR here, but not help as much on AI Feynman? One hypothesis is that AI Feynman has a lot of easy tasks, and the synthetic benchmark only has tasks right on the edge of what PySR can discover, which LLM incorporation helps push over the edge.**\n\nThank you for pointing this out. Randomly generated synthetic equations wouldn’t be informative about the capabilities of LaSR or PySR. To properly verify the utility of language guidance, we generated the synthetic dataset equations to be within the upper limit of PySR’s performance. To construct this dataset, we first generated equations which, anecdotally, contain many characteristics that PySR struggles with. Then, we verified PySR’s performance on this dataset for 400 iterations to ensure there weren’t trivial bugs in the equations (such as the equation evaluating to zero). This left us with 41 equations, for which we report LaSR’s performance with the same hyperparameters. \n\n**Another pessimistic take is that the synthetic benchmark is designed with LaSR in mind. While this still eliminates worries on data leakage, an explanation here would help understand better how LaSR is being helpful.**\n\nThank you for pointing this out. As mentioned before, randomly generated synthetic equations wouldn’t be informative about the capabilities of LaSR or PySR. We deliberately want to pick equations that would be hard for PySR to properly verify the utility of language guidance. We’ve edited the synthetic dataset section to clarify the data construction process, which explains PySR’s poor performance. \n\n**Discovering what is known…**\n\nOur experiments on synthetic data suggest that we can use LaSR to make discoveries even without significant prior domain knowledge. In addition, since the submission, we have been actively exploring ways to use LaSR in novel discovery tasks. For example, we currently have exciting preliminary results on automatically discovering novel LLM scaling laws using LaSR. Such real-world case studies require deep collaboration with domain experts and are hence outside the scope of the present foundational effort. However, we anticipate many such results in the future. \n\n**Lack of Running Time**\n\nOur goal is to present experimental results that can be replicated reliably by the broader community. As such, we root our experiments in the number of iterations instead of wall time.\n\nThis is because the wall time is heavily dependent on the LLM backend, the number of tokens each query uses, and the network quality. For instance, for `gpt3.5` we cannot reliably measure the runtime as the backend is slower during periods of high traffic. Even for local models, vLLM's query scheduling is non-deterministic so the wall time will be substantially different across experiments.\n\nRegardless, we were measuring around one second per query using vLLM. Since we make 60k calls per iteration (Appendix A.3.1) the amount of time for p=1% would be around 600 seconds / 10 minutes per iteration. PySR roughly takes 30 seconds per iteration. We expect performance to improve as we increase CPU/GPU parallelism, and as the hardware to run LLM inference improves.\n\n`[1]`: https://github.com/stanfordnlp/dspy", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We’d like to thank the reviewers for their thoughtful comments and suggestions. We've incorporated many suggested changes in our manuscript and will update the PDF whenenver the portal opens up next. 

Multiple reviewers expressed concerns about data leakage from LLMs. While this is a valid concern for all LLM-based approaches, we have strong evidence against data leakage:

- **Synthetic Data experiments:** The synthetic dataset contains 41 novel and challenging equations that do not exist in any LLM training dataset. We find that LaSR outperforms PySR on this dataset; strongly reinforcing that LaSR’s performance gains are not rooted in memorization or data leakage. 
- **Ablation without common identifiers:** Figure 3 (Left) showcases that LaSR, in the absence of common identifiers like variable names, still outperforms PySR.
- **Evidence from qualitative evaluation:** In Appendix A.4.1, we analyze the equations discovered by PySR and LaSR from data corresponding to Coulomb's law. If LaSR’s performance was rooted in regurgitating memorized data, LaSR’s equation would correspond to the common formatting of Coulomb's law ($F=\frac{1}{4\pi\epsilon_0} \frac{q_1 q_2}{r^2}$). Instead, we observe that LaSR’s discovered equation displays a very different format, and needs at least five simplification steps to transform to the commonly seen equation.

We elaborate on these points, and address the reviewers' other concerns, in the individual responses below.

Meta-review: The Reviewers agree that the method is well-motivated, and aligns well with the goal of developing a system that reflects how humans discover useful equations. Unlike prior work that uses LLMs to generate expressions for symbolic regression, the new approach, LaSR, is integrated w/ an off-the-shelf classical evolutionary SR method, so it is able to exploit the advantages of both paradigms. The method works as expected, performing well empirically, and the paper can be accepted along these lines.

The main concern with this kind of paper is information leakage, since the LLM has certainly been exposed to all the Feynman equations in its training data. The authors have taken substantial steps to address this concern, including: demonstrating the approach on a previously unpublished set of synthetically generated equations, demonstrating that the approach discovers different functional forms of the same equation over different runs, and running ablations w/ a priori knowledge removed.

This does not completely eliminate concerns of data leakage, but I would argue that completely eliminating this concern is not possible at this time. With LLM-based approaches, even knowledge of black-box problems and best known solutions so far could bias the model; and, at a higher level, knowledge of a stochastic synthetic equation generator could have a similar impact. I would argue that existing methods including PySR implicitly have tons of prior knowledge baked in, since they were developed with the benchmark in mind of rediscovering known physical equations. Overall, establishing completely "fair" comparison of SR methods remains an open problem that is outside the scope of this work. That said, I have four recommendations for clarifying the contribution:

(1) In the paper, articulate the challenges of "fair" comparison as fully as possible.

(2) I encourage the authors to follow the reviewer advice to include results on black-box SRBench problems in the final version. Although the authors say they are interested in scenarios where the ground truth is known, in a real application of such a method there would be no ground truth. These experiments could also show a "real new discovery" if they find equations on the Pareto front of known best equations. Similarly, if LaSR leads to another "new discovery" before the final version, e.g., in scaling laws as the authors have suggested, I encourage them to add it to the paper.

(3) The result that different trials produce different functional forms of the same equation should be clearly highlighted in the paper, e.g., in a table showing differing discovered solutions across different trials for multiple problems. 

(4) Clearly explain the precise motivation and construction of the synthetic domain.

One interesting result in the paper was that the LLM could be used to generate only 1% of expressions and still yield notable benefits. I would really like to see the other extreme, i.e., how well does the method do at 100% LLM? If it's possible to add these experiments to the final version, it would better contextualize the method compared to existing LLM-only methods, e.g., LMX [1], the first evolutionary approach to use LLMs to equations for SR. The Related Work section would generally benefit from discussions of such earlier approaches. As is, the reader could reach the incorrect conclusion that LaSR is the first method to generate SR equations w/ LLMs.

[1] Meyerson et al. "Language Model Crossover: Variation through Few-Shot Prompting" arXiv:2302.12170
Decision: Accept (poster)