Symbolic Regression with a Learned Concept Library

Authors: arya  grayeli
atharva  sehgal
omar  costilla
reyes  mit
miles  cranmer
swarat  chaudhuri
u t austin
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 16:45:47

Abstract: We present a novel method for symbolic regression (SR), the task of searching for compact programmatic hypotheses that best explain a dataset. The problem is commonly solved using genetic algorithms; we show that we can enhance such methods by inducing a library of abstract textual concepts. Our algorithm, called LaSR, 
uses zero-shot queries to a large language model (LLM) to discover and evolve concepts occurring in known high-performing hypotheses. We discover new hypotheses using a mix of standard evolutionary steps and LLM-guided steps (obtained through zero-shot LLM queries) conditioned on discovered concepts. Once discovered,  hypotheses are used in a new round of concept abstraction and evolution. We validate LaSR on the Feynman equations, a popular SR benchmark, 
as well as a set of synthetic tasks. On these benchmarks, LaSR substantially outperforms a variety of state-of-the-art SR approaches based on deep learning and evolutionary algorithms. Moreover, we show that LASR can be used to discover a new and powerful scaling law for LLMs.

Main paper: 
 introduction: 
 Symbolic regression (SR) [33] is the task of finding succinct programmatic hypotheses -written in a flexible, domain-specific programming language -that best explain a dataset. Initially proposed in the 1970s, SR has recently emerged as a prominent approach to automated scientific discovery, with applications in domains from astrophysics [30, 12] to chemistry [2, 22] to medicine [51] .Computational complexity is a fundamental challenge in SR, as the space of hypotheses that an SR algorithm must search is discrete and exponential. Previous work has approached this challenge using methods like genetic programming [41, 10] , neural-guided search [11, 43] , deep reinforcement learning [38] and hybrid algorithms [28] . However, new tools to enhance the scalability of SR remain a critical need for applications in SR and scientific discovery.In this paper, we show that abstraction and knowledge-directed discovery can be powerful principles in building such scaling tools in SR. State-of-the-art genetic algorithms for SR [10] evolve pools of candidate hypotheses using random mutation and crossover operations. By contrast, a human scientist does not just randomly mutate their explanations of data. Instead, they synthesize background knowledge and empirical observations into abstract concepts, then use these concepts to derive new explanations. We show that zero-shot queries to large language models (LLMs) can be used to implement such a discovery process on top of a standard SR algorithm.Concretely, we present a new method for symbolic regression, called LASR, that discovers a library of abstract, reusable and interpretable textual concepts and uses it to accelerate SR. LASR alternates between three phases: (i) concept-directed hypothesis evolution, where standard genetic operations Figure 1 : An overview of LASR. LASR iteratively refines a library of interpretable textual concepts which are used to bias the search for hypotheses for scientific discovery tasks. This involves three distinct phases: (Top) finding optimal hypotheses within a concept-directed hypothesis evolution, (Right) leveraging the optimal hypotheses to find new concept abstractions, and (Left) iterating on learned concepts to discover new concepts to accelerate hypothesis evolution. LASR introduces an orthogonal direction of improvement over current symbolic regression algorithms [10] (in gray). over hypotheses are interleaved with LLM-guided mutation and crossover operations conditioned on known library concepts; (ii) the LLM-based abstraction of patterns in known high-performing hypotheses into new concepts; and (iii) the LLM-directed evolution of concepts into more succinct and general forms. Together, these three steps form an open-ended alternating maximization loop that combines evolutionary exploration with the exploitation of the LLM's background knowledge and in-context learning ability.We experimentally compare LASR on Feynman Equations [26] -a popular SR benchmark in which the goal is to discover 100 equations from the Feynman Lectures in Physics -against several stateof-the-art genetic and deep learning approaches. LASR can discover 66 of the 100 target equations, while the best existing approach can solve 59. To address the concern that LASR's performance could be attributed to test set leakage, we compare LASR with a state-of-the-art genetic approach on a suite of synthetic benchmarks. We show that LASR substantially outperforms the baseline. Finally, we support the contribution of LASR to the research community by evaluating the methodology in a case study where the goal is to find new scaling laws for LLMs.In summary, the contributions of this paper are as follows:• We pose the problem of discovering an open-ended, reusable concept library that can accelerate solutions to downstream SR tasks.• We present LASR, a method for combining zero-shot LLM queries and standard evolutionary operations to simultaneously induce a concept library and high-performing hypotheses. LASR's strategy of using LLMs to accelerate evolutionary algorithms may have future applications in settings beyond SR.• We offer promising experimental results, including a demonstration that LASR outperforms stateof-the-art algorithms in standard SR tasks and synthetic domains, as well as a case study that uses LASR to discover a novel LLM scaling law.2 Problem Formulation Symbolic Regression. We formulate symbolic regression (SR) as a program synthesis [6] problem. The inputs to this problem include a language L of programmatic hypotheses and a dataset D := {(x i , y i )} N i=1 of input-output examples. The syntax of L is described by a context-free grammar [24] . The grammar allows each hypothesis π to be represented using a set of mathematical operators (e.g., addition, multiplication, trigonometric functions) that facilitate the composition of simpler hypotheses into more complex ones. We abstractly define the fitness of a hypothesis π as the likelihood p L (D | π) that it generates D.In order to prevent finding non-useful solutions, we impose a prior probability distribution p L (π) over hypotheses π that penalizes syntactically complex hypotheses. We now pose SR as the task of finding a hypothesis π ⋆ that maximizes the fitness while minimizing syntactic complexity. The problem can be expressed as a maximum a posteriori (MAP) estimation problem [15] :EQUATIONRecent work leverages large language models (LLMs) for program synthesis [8, 19, 32] . Large language models (LLMs) approach program synthesis as a token prediction problem, directly approximating the likelihood of programs by training on internet-scale datasets. That is, p L (π|D) ≈ p LLM (⟨π⟩ | ⟨L⟩, desc(D)),(2) where ⟨π⟩ and ⟨L⟩ are, respectively, textual representations of π and a specification of the syntax of L, and the task description desc(D) is a few-shot serialization of a subset of the examples in D.Symbolic Regression with Latent Concept Libraries. Classical symbolic regression typically assumes no prior knowledge or intuition about the problem. In contrast, human scientific discovery often leverages empirical patterns [52] and intuitions derived from previously observed data. For example, recognizing a 'power law relationship between variables' has led to the formulation of fundamental empirical laws across various fields, such as the Arrhenius equation in Chemistry, the Rydberg formula in Physics, Zipf's law in Linguistics, and Moore's law in Computer Science.We model such empirical patterns as natural-language concepts drawn from a latent concept library C. We frame the relationship between the concept library and programs as a Hierarchical Bayesian model consisting of: (i) a prior p(C) representing the natural distribution over concept libraries; (ii) a model p L (π | C) that quantifies the likelihood of various hypotheses for a given concept library C; and (iii) the previously mentioned fitness function p L (D | π) for programs π. We assume that the distributions p(C) and p L (π | C) can be approximated using LLMs. That is, we can prompt an LLM to generate interesting concepts, and we can prompt an LLM with a set of concepts to generate token-sequence representations of hypotheses that adhere to the concepts. Now we state the problem of symbolic regression with latent concept learning as one of simultaneously inducing an optimal concept library and an optimal programmatic hypothesis:arg max π,C p(π, C|D) = arg max π,C p(D|π)By executionBy LLMBy LLM(3)The good mathematical expressions exhibit a clear and coherent relationship between the variables involved, with a focus on power functions and trigonometric functions that can be easily related to physical concepts. 
iteration 6: 
 The good mathematical expressions exhibit a certain level of symmetry or regularity in their form, possibly reflecting underlying patterns or relationships between the variables and constants. 
method: 
 LASR performs a two-stage evolution over natural-language concepts and programmatic hypotheses. The two stages follow an alternating maximization strategy shown in Figure 1 : (1) Hypothesis evolution: We fix the set of concepts and focus on maximizing the hypotheses' fitness to the dataset, and (2) Concept abstraction and evolution: We leverage the best hypotheses found to induce a new library of concepts.In the rest of this section, we first describe PySR, the SR algorithm [10] that LASR extends. Next, we show how to modify this algorithm into one guided by natural-language concepts. Finally, we show how these concepts can be naturally extracted and evolved into new concepts. The full LASR algorithm is presented in Algorithm 1 and visualized in Figure 2 . LASR is built in Julia with an additional Python interface 2 and uses an open-source, optimized framework for LLM inference [25] . Like other evolutionary algorithms, the search in PySR uses symbolic mutation and crossover operations. The mutation step is broken into many categories, each with distinct weighting, to either mutate a constant, mutate an operator, add a node (append, prepend, insert), delete a subtree of an expression tree, simplify the tree, initialize a new tree, or do nothing. One of these operations is randomly selected at each call to a mutation request, and each operation executes itself at random but within user-provided constraints. For example, deleting a subtree is done by choosing a random node to replace with a randomly-generated leaf node such as a feature or constant. The crossover step involves swapping random subtrees of two expressions in a population.LLM-guided Hypothesis Evolution. LASR speeds up PySR by injecting natural language priors into its search procedure. To do this, we modify the INITIALIZEPOPULATIONS procedure to use an LLM-augmented initialization operation, and the SRCYCLE routine to use LLM-augmented versions of its symbolic mutation and crossover operations. The altered procedures are named LLMINIT, LLMMUTATE, and LLMCROSSOVER, respectively. These operations do not replace their standard genetic counterparts. Instead, we introduce a hyperparameter p that, with a fixed probability, substitutes the standard genetic operation with the LLM-based operation. This enables "doping" each population with a program that respects the language priors, while ensuring that we do not bottleneck the local exploration of the search space.The LLM-guided operations follow the same base format: they sample multiple concepts from the concept library, concatenate these concepts with the task-specific variable names and language operations, and append a specialized prompt for each task. We employ zero-shot prompts (see Appendix A.2 for more details) to avoid sampling biases. In further detail:• LLMINIT: The LLMINIT function takes an initial set of concepts and uses them to initialize the populations for the evolutionary search step. The initial set of concepts can either be instantiated from an optional set of user-provided "hints" or generated by the LLM.• LLMMUTATE: For mutation within a population, we sample a set of l concepts from the concept library C, and construct a prompt that uses this set of concepts to mutate an expression π i into π j . The prompt to the LLM takes inspiration from the standard genetic mutation operation, and asks it to mutate the expression given the concepts sampled from the library.Algorithm 1 Pseudocode for LASR. LASR takes as input an optional set of user-provided hints C 0 , a dataset of input-output pairs of high-dimensional data D, and four hyperparameters: the number of iterations I, the number of populations K, the number of steps for concept evolution M , and the mixture probability of using LLM-based or GP-based evolutionary operators p. LASR produces two artifacts: the evolved library of concepts C and the expression with the highest fitness score π ⋆ . for i in range(K) do 6:Πi ← SRCYCLE(Πi, D, C, p) ▷ Interleaved Symbolic + LLM Search 7:F ← EXTRACTPARETOFRONTIER({Π1 . . . ΠK }, D) ▷ Includes positive + negative programs 8:C ← C ∪ CONCEPTABSTRACTION(F, C) 9:for _ in range(M ) do 10:C ← CONCEPTEVOLUTION(C) 11: π ⋆ ← BESTEXPRESSION(F)▷ Based on dataset loss and program complexity 12: return C, π ⋆• LLMCROSSOVER: The LLMCROSSOVER function also samples a set of l concepts from the concept library along with two hypotheses π i and π j to construct a new expression π k , which reuses sub-expression trees from the two hypotheses while respecting the sampled concepts. Our implementation is inspired by prior work [40] -see Figure 4 .Concept Abstraction. After each iteration of symbolic regression, we use a function EXTRACT-PARETOFRONTIER to collect: (i) the hypotheses, across all populations, that are Pareto-optimal with respect to the criteria of syntactic simplicity and dataset loss; (ii) the hypotheses with the worst loss across all populations. The resulting set of hypotheses F = {π ⋆ 1 ., π ⋆ a . . . π - 1 ., π - b } captures the trends that were most helpful and most detrimental to performance during hypothesis search. Now we use the CONCEPTABSTRACTION function, which uses a zero-shot prompt to extract a natural language concept c ⋆ that summarizes the positive trends while eschewing negative trends. This concept is subsequently added to the concept library. The prompt for the function is presented in Figure 5 .Concept Evolution. Each concept in C represents trends that were useful at a previous state in the search process. After adding new concepts into the library, we use a function CONCEPTEVOLUTION to evolve the library to include new ideas that logically follow from the ideas in the current library. The implementation of this function follows that of the LLMCROSSOVER operation in that we are using multiple concepts as a reference to generate new ones, with the key distinction that, unlike in the LLMCROSSOVER operation, the fitness of each generated concept here is difficult to quantify. Thus, we include all the generated responses in the concept library. While these concepts may sometimes be inaccurate, they increase the evolutionary algorithm's exploration ability.The good mathematical expressions have a clear and consistent structure involving the variables q1, q2, epsilon, C, and r, with a specific pattern of division and multiplication.A.9 Subset of equations discovered by LaSRGuidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: No theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 We demonstrate the effectiveness of LASR on multiple tasks integral to scientific discovery. First, we evaluate LASR's performance on the Feynman Equation dataset, a widely adopted scientific discovery benchmark, under a variety of ablations and additional priors. Second, we measure the effect of data leakage by evaluating LASR's performance on a procedurally generated synthetic dataset of challenging equations. Finally, we conduct a case study using LASR to discover LLM scaling laws with data from the BIG-Bench evaluation suite [16] .LASR's main focus is to serve as a practical toolkit for scientists. Therefore, our evaluation primarily targets slightly noisy environments, using exact solution rate to gauge performance rather than statistical similarity measures like correlation R 2 , which are less relevant to scientific discovery applications. Additional experiments characterizing the practical behavior of LASR are in Appendix A.5. Information regarding compute usage is in Appendix A. 3 Dataset: The Feynman Equations dataset is a well established benchmark for Symbolic Regression algorithms [49] . This dataset consists of 100 physics equations extracted from the Feynman lectures on Physics. We compare against performance reported on SRBench [26] : a continuously updated public benchmark of SR methods on many datasets. Specifically, we compare against GPlearn, AFP, AFP-FE, DSR, uDSR, PySR, and AI Feynman [42, 47, 49, 28, 38] . Within this subset PySR represents an ablation of our model without the LLM genetic operations and the concept evolution (Section 3). We evaluate on a slightly noisy version of this dataset in order to simulate experimental errors common in scientific discovery domains. More details are presented in Appendix A.4.1.Setup: We instantiate LASR using gpt-3.5-turbo-0125 [4] as the backbone LLM and calling it with p = 0.01 for 40 iterations, and compare our results with PySR which uses the same default hyperparameters. For the other baselines, we use the numbers reported in SRBench with one exception being uDSR [28] , for which we couldn't find any benchmarking numbers. For this method, we derive the exact solve rate from [37] .Results: We showcase results in Table 1 . We draw three observations from this experiment. First, LASR achieves a higher exact solve rate than all other baselines. Second, both PySR and LASR outperform the other baselines by a wide margin, indicating that scalable and efficient synthesis is imperative to practical scientific discovery algorithms. Finally, and most notably, a subset of the equations LASR finds could not be derived with any of the previous methods.LASR's performance is inherently bottlenecked by the reasoning capabilities of the backbone LLMs and the frequency of their invocation in each iteration. To evaluate the effect of the backbone LLM on LASR's performance, we instantiate a model cascade over two of LASR's hyperparameters: the backbone model (llama3-8b [17] , gpt-3.5-turbo-0125) and the probability p with which we call that model in the evolution step (p = [1%, 5%, 10%]).Setup: Our cascade operates as a tournament. We start LASR with the configuration that provides the least language guidance (llama3-8b at p = 1%) and progressively increase the value of p and then the backbone model. Each subsequent model is only evaluated on the problems that the previous model could not solve. We compare this against PySR's performance on the Feynman equation dataset. To ensure a fair comparison, we cascade PySR using the same procedure but find it does not solve any additional equations. For this experiment, we tag each equation with a qualitative rating comparing the equation to the ground truth form (Exact Solve, Almost Solve, Close, and Not Close).An in-depth discussion on this metric is presented in Section A.7.Results: Our results are presented in 2. We draw two key observations from these results. First, LASR outperforms PySR even with minimal concept guidance (llama3-8b at p = 1% We extend LASR by providing an initial concept library C 0 in the form of user provided hints. We find that natural language hints significantly increases the speed of solving equations.increasing the backbone model size and the mixture probability substantially enhances LASR's performance, indicating that as the language reasoning capabilities of LLMs improve, so will our performance.We conduct ablations on the use of Concept Evolution (skip phase three from Figure 1 ), Concept Library (skip phase two and three), variable names, and user hints. Figure 3 shows how these ablations affect performance over 40 iterations. We designate an equation as "solved" if, after N iterations, the MSE of our predicted equation is less than 10 -11 . This metric differs from 'Exact Solved' as defined in the prior experiments: an equation can be 'exactly solved' yet have an MSE higher than 10 -11 due to the noise floor in the target variables, and an equation can have low loss but not be an exact match. We observe from the results that: (1) Removing variable names results in a substantial performance drop, as we lose semantic meaning provided by variables (for instance, observing θ could suggest employing trigonometric functions on θ). (2) Learning a concept library enables faster convergence to solutions. Without the concept library, task convergence is slower, and widens under higher concept guidance conditions (p > 0.1%).The concept library provides an interpretable window into our evolutionary search process. To showcase the concepts learned by LASR, we take a sample equation from the Feynman dataset, the electric field of a dipoleE f = 3p d cos θ sin θ 4πϵr 3and comment on the libraries learned at various intervals. We see rudimentary concepts emerge in the second iteration:"The presence of basic trigonometric functions like sin in the good expressions contributes to their quality, indicating a connection to physical concepts such as waveforms or periodic phenomena."And, in subsequent iterations, the concepts become even more refined:"The good mathematical expressions exhibit a balance between mathematical operations such as multiplication, division, and trigonometric functions, which are known to have physical interpretations and relevance in various scientific phenomena."This iterative refinement helps LASR consistently maintain high-quality concepts, allowing it to converge to an exact match within 40 iterations. By contrast, PySR and the concept library ablations fail to converge on an exact match solution, returning equations that -while low-loss -involve many extra terms absent from the ground truth. This reinforces our hypothesis that injecting semantic meaning into the search process not only improves search efficiency, but also regularizes against complex equations -as the LLM-generated concepts help filter out irrelevant terms. A deeper qualitative analysis is in Appendix A.8.Extending LASR with Hints: A benefit of LASR is that its search can be initialized with a set of user-specified, natural-language "hints." To evaluate this capability, we generate hints for each equation based on variations of the chapter title of the Feynman lecture that the equation belongs to. We intentionally keep the hints vague to see if knowledge about just the general field is sufficient in improving LASR's performance. We showcase results in Figure 3 . We observe a noticeable boost in performance from injecting these hints, even for our weakest performing model, indicating that even minimal user input can substantially enhance LASR's effectiveness in discovering equations.An important consideration in using LLMs for existing SR problems is the possibility that the LLM was exposed to the hold-out problems in the validation set, presenting an unfair advantage to LLMs trained on massive datasets. Intuitively, LASR generates its own concepts which are conditioned on suboptimal programs, which are unlikely to be within the LLM's memorized responses. To validate this, we generate a dataset of 41 synthetic equations that are engineered to deviate from common physical and mathematical structures and have arbitrary variables. For example, one such equation is y = 0.782x3+0.536x2e x 1 (log x2-x2e cos x 1 ). We find that PySR struggles to solve equations with these characteristics (given 400 iterations). Hence, solving such equations hinges on the language guidance components.LaSR (Llama3-8B, 0.1%) 0.070 0.913 Table 3 : Evaluation results of data leakage. We present the test set R 2 of PySR and of LASR on a synthetic symbolic regression dataset. Higher R 2 is better.We run LASR with Llama3-8B at 0.1%. We then compare our synthesized program's test set R 2 with that of PySR's. We justify using correlation instead of exact-match as we are not motivated by the application of results for scientific discovery in this experiment. Our results are summarized in Table 3 and show that LASR's concept-guided synthesis still provides a considerable performance boost compared to PySRdemonstrating that LASR can outperform PySR even when data leaking is not possible.So far, we have demonstrated that LASR can discover equations that are practical but already known (Feynman Dataset) and equations that are novel but aren't practical (Synthetic Dataset). To investigate LASR's utility in finding novel and practical empirical trends, we investigate whether LASR can discover novel LLM scaling laws on the BigBench dataset [16] . More details on this experiment are presented in Section A.6.Traditionally, to identify an LLM scaling law, practitioners must first manually posit a "skeleton equation" with a fixed set of known variables and unknown free parameters, and then optimize the unknown parameters based on a dataset of model hyperparameters and resulting dataset fitness [23, 1, 5] . Instead of starting with a predefined equation, we use LASR to discover the skeleton equation that best fits various subsets of the BigBench dataset.Setup. BigBench contains 204 tasks with scored responses from 55 language models trained with different hyperparameters. We evaluate on the subset of tasks where the preferred metric is 'Multiple choice grade' (53,812 samples). Our goal is to find the equation that best predicts the test score given the model hyperparameters and the dataset hyperparameters. We run LASR with 3840 populations of 200 candidates each for 7 hours (overnight). The runtime of LASR is comparable to other SR algorithms for this experiment as the slowest operation isn't generating candidate equations but rather optimizing and evaluating candidate equations.Results. LASR discovers the following scaling law on the subset of BigBench: [46] . However, if the number of shots is an even number, the exponentially increasing trend reflects over the line score = E and turns into an exponentially decreasing trend. A manual inspection of the training data reveals the majority of samples had an odd number of shots (1, 3, ...) which causes this generalization error.EQUATIONAs the output artifacts of LASR are interpretable, we can integrate this empirical relationship between training steps and number of shots into known scaling laws. Specifically, we can augment the chinchilla scaling law as follows:score = A (train_steps • batch_size) α + B (#params) β + E (Chinchilla [23]) score = A (train_steps • batch_size) α•#shots + B (#params) β + E (Modified Chinchilla)Quantitative Evaluation: We fit the free parameters of each equation on the training set (43, 049 samples) and measure the MSE loss between the actual grade and the predicted grade on the validation set (10, 763 samples). The results are presented in Table 4 . We find that the Equation 4's performance, as well as modified Chinchilla's performance, is competitive with that of Chinchilla's in predicting the MCQ grade. However, the horizontal line score = E demonstrates acceptable performance as well. We believe increasing the scale of these preliminary experiments (with richer datasets or longer search horizon) will lead to additional empirical findings.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide a link to an open source implementation of LASR in Julia. 
related work: 
 Symbolic Regression. The field of SR started in the 1970s [18, 29] and has recently become a prominent approach to AI-for-science [33, 34, 40] . Two algorithmic themes here are:Non-parametric Algorithms: Most work on SR focuses on improving search efficiency using heuristics or parallelization. Specifically, PySR [10] builds a multi-population evolutionary algorithm that incorporates various preexisting heuristics [39] , and introduces novel ones such as simulated annealing, an evolve-simplify-optimize loop, and an adaptive parsimony metric. PySR has been successfully applied to study problems in domains such as cosmology [12] , international economics [50] , and climate modeling [20] . LASR extends PySR to enable the discovery of latent concepts.Parametric Algorithms: Recent work in SR and program synthesis has often used neural networks to accelerate search [43, 40, 38, 28, 34, 13, 35] . The interplay between the neural and the symbolic components in these works can be abstracted into two categories: (1) leveraging LLMs to induce program scaffolds [34, 40, 35] , and (2) learning a neural policy to accelerate search [38, 28, 43, 13] .We highlight two methods from the first category: Funsearch [40] and LLM-SR [45] . Funsearch [40] uses a pretrained LLM to implement a mutation operator on a database of executable programs under a fixed specification to find super-optimized programs in extremal combinatorics. LASR is a generalization of FunSearch: while FunSearch conditions program generation on a static "specification" (analogous to our concept library), we discover the concept library in the course of the algorithm. We do not compare against FunSearch due to resource constraints. As for LLM-SR [45] , it leverages a pretrained LLM for generating program sketches [36] . The sketch parameters are optimized and cached in a database, which is in turn used to generate new sketches. Our work is an orthogonal direction of improvement. It is technically possible to "plug" the LLM-SR framework (or other LLM-based search algorithms [35] ) into LASR and use our generated concepts to guide the lower-level search component.The second category includes methods like DSR [38] , which, just like LASR, frame SR as a sequence modeling problem. However, the search in LASR leverages a learned concept library and the language and code biases in LLMs, instead of relying on amortization alone.Program Synthesis with Foundation Models. Recent work in program synthesis models program generation as a sequence prediction problem. Under this paradigm, the DSL and the input-output specification is serialized in the prompt and a code-generation foundation model [31, 7, 4 ] is leveraged to autoregressively generate candidate programs. This approach has been impactful in many areas including spreadsheet formula prediction [13, 8] , competitive programming [32] , and visual programming [48, 21, 9] . LASR is similar to work in this area in that the LLM Mutate, LLM Crossover, and LLM Initialization functions all follow the sequence prediction paradigm to synthesize mathematical equations, relying on guidance from the concept library.Program Synthesis with Library Learning. Deploying classical program synthesizers in a new domain often necessitate hand-engineering DSLs to enable scalable synthesis. This severely limits the generality and practicality of such methods. An emerging direction of research -called library learning -attempts to learn the DSL and the programs simultaneously [15, 3, 19, 27, 53, 14, 44, 54] . This is typically framed as a hierarchical Bayesian optimization problem over the space of programs and the space of library functions that generate those programs. Notably, [19] uses LLM guidance to assist in program induction and in auto-documenting learned library modules and [53] considers learning programs under a latent distribution over the space of natural language and the space of the DSL. LASR shares a similar problem formulation to these works, but optimizes over the space of programs and over the space of natural language descriptions of these programs.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: Same answer as above. We also include a README to start the codebase. Eventually, we'd like to merge our contributions into a larger repository (PySR).Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 We have presented LASR, a framework that uses zero-shot queries to an LLM to induce abstract, reusable concepts that can be used to accelerate SR. We have shown that LASR outperforms stateof-the-art approaches on the standard Feynman equation task. We have also used the algorithm to discover a novel scaling law for LLMs.A key benefit of LASR is that its capabilities are ultimately bottlenecked by those of the underlying LLM. LLMs are rapidly gaining capability and getting cheaper, and future versions of LASR should be able to tap into this progress.Many directions of research remain open. First, our strategy of accelerating evolutionary search with LLM-based concept induction may be applicable beyond the SR setting. Future research should explore such applications. Second, while our approach here was entirely based on in-context learning, it is worth exploring if finetuning improves the performance of the LLM. Finally, we evaluated the learned concept library exclusively on the downstream SR task. However, the library may also be valuable in other tasks such as clustering or explanation synthesis. Exploring these other tasks is an attractive topic for future work.Limitations. The current instantiation of LASR has several limitations. First, it cannot guarantee that the concepts it learns are correct or insightful. Even a concept that leads to strong performance in downstream SR tasks may do so because of quirks of the model and data, and end up misleading scientists using the method in a discovery process. Also, we do not currently have a way to ensure that the learned concepts are mutually consistent. Finally, our evaluation here was constrained by our compute budgets for LLMs and search. Whether the trends we see generalize to higher-compute regimes remains to be seen.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We've put our best efforts in ensuring the research work is comprehensive enough that all experiments can be replicated. Nevertheless, for any missing details, we refer the reader to our open sourced repository and are happy to make changes to enhance comprehensiveness.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Our main evaluation metric is exact match of a synthesized equation w.r.t a reference equation. This statistic doesn't permit a level of granularity at which we can show error bars. An important source of uncertainty is our backbone LLM. We've painstaikingly engineered our experiments to use local LLMs at the cost of modest performance to enable reproducibility. However, certain experiments necessitate proprietary models whose internal details may change.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We mention our experimental setup details as well as other experimental setups we have successfully tried. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: Yes. We've made sure our submission follows with the NeurIPS code of ethics. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We talk about broader impacts in the appendix. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Broader Societal ImpactsWe have presented LASR: a symbolic regression framework that leverages concept guidance to accelerate symbolic regression. We hope that LASR helps accelerate the search for empirical laws in the broader scientific community. In this section, we discuss the broader societal impacts and ethical considerations of our work.Potential for Misuse: As with other ML techniques, symbolic regression can be leveraged by bad actors to inflict societal harm. Our experiments show that LASR accelerates the search for empirical laws from raw observations. In our setting, we are restricted to observations about physical phenomena. However, a malicious could misuse LASR to find patterns in datasets that violate personal rights.Privacy Concerns: As mentioned before, LASR enables finding patterns in raw observations. We hope that LASR is leveraged by scientists to explain physical phenomena. However, it is possible to use such models to learn behavioral profiles without the active knowledge or explicit consent of the subjects.Bias and Fairness: LASR generates two artifacts: a hypothesis that maximizes a fitness function (represented as an equation) and a library of concepts that helped discover that hypothesis. LASR ensures fairness and lack of bias in the generated equation as long as the fitness function is free of biases as well. However, we leverage foundation models to induce our library of concepts which could be trained on biased data which may reflect in our concept library. Furthermore, we cannot directly evaluate the efficacy of the concept library and its factual correctness. This doesn't affect equation generation -since equations are quantitatively evaluated. However, a human analyzing the concepts LASR learns might misinterpret trends that the model picks up on.Note that in the prompts in Figure 4 , Figure 5 , and Figure 6 , we refer to our hypothesis as expressions and the concepts as hypotheses and suggestions. This prompting style was found to work best for the LLM.We run all experiments on a server node with 8xA100 GPUs with 80 GB of VRAM each. However, our experiments can be reproduced with a GPU with 16 GB of VRAM. We were even able to run LASR on a laptop utilizing a quantized model hosted locally 3 . Moreover, certain models are hosted on external servers (such as gpt-3-turbo-0125) which allows running LASR on machines without GPUs. For this project, we chose to run llama3-8b using vLLM [25] . However, our framework is compatible with any LLM inference framework that allows hosting an OpenAI compliant RESTful server. For reference, each iteration makes around 60, 000 calls. Each call to the LLM is just under 1000 tokens. This gives an upper bound on total compute of 60, 000, 000 tokens per iteration if p = 100%. Hence, running our model at p = 1% for 40 iterations would result in just under 25M tokens for each equation (around 4 hours on a single A100).In order to determine which concepts from the concept library we sample for the LLM Hypothesis Evolution, we randomly choose the top-K most recent concepts in the library. This ensures that we use the latest concepts, which are generally reflective of more informed hypotheses, and thus better to use. In practice, we set K = 20. Additionally, for Concept Evolution, we exclude the top-K most recent concepts from being used, and rather use older concepts. This is motivated by the desire to not have the concept library converge on a few ideas, rather we want diversity of thought. Our concepts are intended to be longer lasting than the hypotheses that generated them, similar to how observational data comes and goes, but the conclusions from them are more persistent.You are a helpful assistant that recombines two mathematical expressions by following a few provided suggestions. You will be given three suggestions and two expressions to recombine.An expression must consist of the following variables: {{variables}}. All constants will be represented with the symbol C. Each expression will only use these operators: {{operators}}. igure 4: LLMCROSSOVER prompt with an example output. LLMMUTATION and LLMINIT follow the same structure but with slightly different wording and with one and no reference expressions, respectively. Variables within double braces are replaced with the instance specific arguments. These prompts are available in prompts/*.txt in the linked repository.You are a helpful assistant that hypothesizes about the underlying assumptions that generated a list of good and bad mathematical expressions in detailed ways. My ultimate goal is to discover what assumptions generated the observed good mathematical expressions and excludes the bad mathematical expressions. Focus more on the good expressions, their mathematical structure, and any relation to physical concepts. Note that capital C represents an arbitrary constant. Propose {{N}} expressions that would be appropriate given the suggestions and expressions. Provide short commentary for each of your decisions. End with a JSON list that enumerates the proposed expressions following this format: Hypothesis 1: The good expressions involve complex exponential or trigonometric functions with nested power operations, potentially indicating a link to physical phenomena that exhibit exponential growth or oscillatory behavior.The presence of sine and cosine functions in the good expressions suggests a connection to periodic phenomena in real-world applications, such as wave functions or harmonic oscillations.The inclusion of arbitrary constants in the good expressions may represent parameters in physical models, implying that the expressions are modeling dynamic systems or relationships between variables.The good expressions involve repeated application of mathematical operations (e.g., exponentiation, multiplication, and addition) in a layered manner, indicating a depth of complexity that mirrors intricate physical systems or natural processes.The mathematical relationships within the good expressions exhibit a balance between simplicity and complexity, suggesting that they capture fundamental principles or underlying structures that govern physical phenomena.Here is the list of proposed expressions in JSON format: You are a helpful assistant that merges and refines ideas about a set of hidden mathematical expression in new, interesting, and diverse ways. My ultimate goal is to discover the underlying properties of these hidden expressions. The resulting ideas should be a nontrivial conclusion given the previous ideas. Propose {{N}} hypotheses that would be appropriate given the ideas. Provide short commentary for each of your decisions. Do not talk about topics related to the simplicity or complexity of the expressions. I want ideas that are unique and interesting enough to amaze the world's best mathematicians. End with a JSON list that enumerates the proposed hypotheses following this format:Domain(System)Header (User) Concept Abstraction Prompt ```json ["hyp1", "hyp2", ... "hyp{{N}}" ] ``(Hypothesis 1: Temperature Modulated Exponential Decay in Dynamical Systems. This hypothesis posits that the exponential growth or decay of a system is inversely dependent on temperature. As temperature increases, the decay rate of the system's state energy function decreases, leading to a slower approach to equilibrium. Conversely, as temperature decreases, the system's decay rate increases, leading to a faster stabilization. This relationship suggests that temperature acts as a modulating factor in the dynamical system's behavior, potentially introducing non-linear effects.Hypothesis 2: Temperature-Driven Phase Transitions in Dynamical Systems. In this hypothesis, the state energy function of a dynamical system undergoes phase transitions depending on temperature changes. These transitions can be modeled by exponential functions that describe the shift from one state to another, suggesting that critical points in the temperature spectrum trigger significant changes in the system's behavior. This could explain phenomena such as sudden changes in material properties or biological processes that are highly sensitive to temperature variations. Hypothesis 3: Entropy-Based Energy Functions in Temperature-Dependent Dynamical Systems. Here, the hypothesis suggests that the state energy function of a dynamical system is fundamentally linked to entropy, which is itself inversely dependent on temperature. The dynamical system's behavior can be described through an entropy-based energy function that exhibits exponential growth or decay. This approach merges thermodynamics with dynamical systems theory, proposing that changes in entropy, driven by temperature variations, dictate the system's evolution over time.Here is the list of proposed expressions in JSON format: 3) a dictionary of parameters pertaining to backend LLM communication. Following other methods in SRBench, we utilize only a subset of the necessary operators for solving the Feynman equations, excluding special operators like arcsin and arctan. These operators are seldom required, and removing them speeds up the search process. We generally set the number of iterations to 40. However, certain experiments may demand more or less iterations.Dataset: The Feynman Equation dataset is a widely adopted benchmark for scientific discovery [49] . The dataset consists of 100 physics equations extracted from the Feynman lectures on Physics. Each equation is in the form y = f (x 1 , x 2 , . . . ). The number of input variables ranges from two to ten, and the dataset provides 100,000 samples for each equation. We compare against publically available methods benchmarked on SRBench [26] . SRBench is a continuously updated benchmark which catalogs the performance of various methods on the Feynman dataset as well as other symbolic regression problems. Specifically, we compare against GPlearn, AFP, AFP-FE, DSR, uDSR, PySR, and the original AI Feynman algorithm [42, 47, 49, 28, 38] . Within this subset, notably, PySR represents an ablation of our model without the LLM genetic operations and the concept evolution (Section 3). We evaluate on a slightly noisy version of this dataset in order to simulate experimental errors common in scientific discovery domains. Specifically, we compare numbers against those reported in and reproduced by SRBench with a target noise of 0.001.Methodology: For the Feynman dataset, we took the equations and the bounds at which each variable was sampled at and generated our dataset. Then, we added additional noise of 0.001 to our target variable, following the noise formula detailed in the Appendix A.4 of [26] , as well as additional random noise variables with arbitrary names to force the model for proper feature selection. We then evaluate exact matches by looking at if the predicted equation symbolically simplifies into the ground truth equation. For the ablation graphs, we used the PySR hyperparameter "early_stop_condition" to check if there is a "solution" after N iterations.For the synthetic dataset, we ran a script that generates uncommon mathematical hypotheses that satisfy our constraints at random. Then, we ran PySR for 400 iterations and found all the equations that PySR performed poorly in, i.e. MSE loss greater than 1, while having a complexity less than 20. We run PySR for 10 hours per equation (thresholded to 10 6 iterations) and compare the exact solve rate with that of LASR run for 40 iterations. We find that PySR is able to discover three more equations, but LASR still substantially outperforms PySR.For these 41 remaining equations, we then compared LASR and PySR after 20 iterations using the average of their test set R 2 for each hypothesis.This section highlights additional experiments on various benchmarks to further characterize LASR's performance.A common concern with evaluating genetic optimization algorithms w.r.t number of iterations is that the 'hard stop' after a certain number of iterations might yield populations that haven't fully converged to their optimal values.To account for this discrepancy in performance, we conduct an asymmetric comparison with PySR on the Feynman Equations dataset. Specifically, we allow PySR to run uninterrupted for 10 hours per equation, with the maximum number of iterations set to 10 6 . We compare the results with that of LASR run interrupted for 40 iterations per equation (hence the asymmetric comparison). The results are detailed in Table 5 .Overall, we find that, despite running PySR substantially longer than LASR, LASR still substantially outperforms PySR. This is because PySR, like other evolutionary algorithms, is susceptible to falling in local minima and converges to this local minima extremely fast. This is why supplementing "local" search strategies with LLM guidance is useful.In Table 3 , we reported R 2 test set performance of LASR and PySR on the synthetic dataset. In this section, we qualitatively compare the equations discovered by LASR and PySR on the procedurally generated dataset of equations. None of the algorithms recover the ground truth form, but we find that LASR's equations fit much better to the ground truth data than PySR's equations. These equations are presented in Table 3 .A common concern with using LLM's for scientific discovery is that the equations could be obtained due to dataset memorization rather than pure reasoning and learning on the equations form. To explore this further, we run LASR with the same hyperparameters twice (with different seeds). If the LLM purely relies on memorization, we expect the model to find the same equation form in both experiments. We present results in Table 7 .Overall, both equations fit well to the underlying dataset and reduce to the ground truth. Yet, despite using the same hyperparameters, the functional forms exhibit sharp differences. This further reinforces our hypothesis that LASR's performance is not simply the result of regurgitated memorized responses.So far, we have used LASR to discover equations that are already well-established in prior work. In this section, we investigate LASR's utility in making novel empirical discoveries. Specifically, we investigate whether LASR can discover novel scaling laws for LLMs.Predicted Equationexp C-log(y2) C-y4 + √ y 1 + y 4 + √ y 2 (y4+C+y1) C • (y 2 + C) C + y 1 (y1+C)+( √ y3•(y1•(C-y3)•y3))cos(cos(y1))y 3 • C-(y1•(y3-(cos((y1+C)•C)•C)+C)) C -C exp(y 1 ) • y 1 • (cos(y 1 ) + 2y 1 ) y C 1 + C • exp(y 1 ) -C + C - (y 2 1 •exp(cos(y1))) C + y 4 -C exp (log(y 4 + y 2 • y 1 )) • cos(y 2 ) - √ y 1 + y3 C -C y 1 • ((y 2 • cos(y 2 )) -y 4 -(y 3 + 1) • y 2 ) -1 (y 3 + y 2 ) • (((y 3 + C) • y 2 + y 1 + y 3 ) -C) ((y 2 + y 3 ) • (y 3 + (y 2 • (y 3 + 1)) + 1 + y 1 ) -C + y 3 √ y 2 + exp C + √ y 2 • ((y 4 • y 3 ) • log(y 3 )) √ y C 2 -C •(y4•C•(y3-C))-sin(y C 2 +C) •log(y3) C + C (C•y3)•exp( √ y2-y1) C/(y2+y1) y 2 • C- √ y C 2 √ y C 1 +C y 1 • y 3 -C (y2+C)• √ y1 C • exp(y1) y1 C-y2 sin C y 1 +C C -cos(C -y 1 ) y 1 • y 2 1 C cos(C-y 4 -C) / exp(C • y 5 ) ((y 1 -C) • (sin(y 4 + C) • y 1 ) • (y 5 + y 1 -C)) • C -C exp(log(y 3 ) • √ y 2 ) - exp (y 3 + y 4 ) + (y 1 • y 2 ) • C y3•(y2/C) C +C-y4-y2 -y2Table 6 : Qualitative evaluation of LASR on the synthetic equations dataset (A.5.2). We attempt to recover the ground truth equation from a slightly noisy dataset generated from the ground truth equations. None of the algorithms we tested (including LASR) are able to recover the ground truth equations, underscoring the challenge of exact symbolic match on this dataset. A study on the R 2 performance is presented in Table 3 .Discovered Equation (LaSR -Llama 3b 1%; same params) Loss ComplexityEQUATIONEQUATIONTable 7 : Performance of LASR on successive runs with the same hyperparameters on Equation 53in the Feynman equations dataset (A.5.3). This equation defines the heat flow from a point source (an inverse square law with ground truth formulation: h = P 4πR 2 ). We evaluate LASR by running it twice with identical hyperparameters but different random seeds to assess whether the model has memorized the equation's form. In both runs, LASR consistently discovers high-performing solutions. Moreover, the resulting functional forms show significant variation between runs, supporting the hypothesis that LASR's success is not rooted in memorization.Motivation: Traditionally, to identify an LLM scaling law, practitioners must first manually posit a "skeleton equation" with a fixed set of known variables and unknown free parameters, and then optimize the unknown parameters based on a dataset of model hyperparameters and resulting dataset fitness [23, 1, 5] . Instead of starting with a predefined equation, we use LASR to discover the skeleton equation that best fits various subsets of the BigBench dataset. Removing the need to manually posit a skeleton equation simplifies the methodology for finding scaling laws in many ways. First, it removes human preconceptions about the expected relationships between hyperparameters. Second, it increases the number of variables and the type of variables human practitioners can jointly reason about. Finally, it enables positing equations of much higher complexity and variable interdependence than otherwise possible.Dataset: BigBench is a collaborative benchmark intended to probe large language models and extrapolate their future capabilities [16] . It contains 204 tasks drawing upon problems from linguistics, cognitive science, math, physics, biology, software development, etc. For each task, Bigbench measures the performance on many model families (OpenAI's GPT models, Google's dense transformer architectures, and Switch-style sparse transformers). Each BigBench task is evaluated on a preferred metric (chosen by dataset creators). In our experiments, we consider a subset of tasks where the preferred metric is 'multiple choice grade.' This subset contains the highest diversity of tasks and around 53,812 total data points.Methodology: We run LASR with 3840 populations (parallelized across 16 cores) with each population evolving 200 candidates evaluated with the Zygote autodifferentiation backend. We ran LASR overnight (for 7 hours). BigBench necessitates optimizing a matrix of parameters rather than singular scalar constants. This shifts the compute bottleneck from generating a pool of candidates (in which LASR is slower than PySR) to evaluating a pool of candidates (which is equally slow for both algorithms).Results: LASR discovers the following scaling law on the subset of BigBench: score = -0.015540 train_steps -127140.499EQUATIONWhere score is the MCQ grade, train_steps is the number of training steps for the model, and #shots is the number of in-context examples provided during inference.Qualitative Evaluation: Equation 4 has some interesting properties. First, it describes an empirical relationship between training hyperparameters (training steps) and inference hyperparameters (number of shots). It asserts that increasing the number of shots exponentially increases the model's performance for low-resource models while having diminishing gains as the number of training steps of the model increase. Second, Equation 4 only requires three free parameters (Chinchilla [23] required five free parameters). This might allow LASR 's equation to better generalize to different LLM operation regimes. However, the equation has some issues as well. If the number of shots is an even number, the exponentially increasing trend reflects over the value of A and turns into an exponentially decreasing trend. A manual inspection of the training data reveals the majority of samples had an odd number of shots (1, 3, ...) which causes the generalization error.Quantitative Evaluation: We compare Equation 4, Chinchilla [23] , Modified Chinchilla, and a single free parameter equation on their ability to explain the score of a held out validation set of 10,763 data points. We fit the free parameters of each equation to the training set data and measure the MSE loss between the actual grade and the predicted grade on the validation set. The results are presented in Table 4 . Overall, we find that the Equation 4's performance, as well as modified Chinchilla's performance, is competitive with that of Chinchilla's in predicting the MCQ grade.We hope our preliminary results provide further insight into the practical value of LASR in addressing real-world challenges, including those in machine learningFor the cascading experiment, we aim to evaluate the progression of different configuration towards solving equations. The quantitative metric used in the SRBench comparison experiment, Exact Solve, does not allow for such fine-grained analysis. Therefore, we categorize the synthesized equations into four buckets: Exact Solve, Almost Solve, Close, and Not Close. Exact Solve is quantitatively evaluated using a symbolic match. An equation is tagged as 'Almost Solve' if the dataset loss is small, but the generated equation has an extra term or lacks one term. A Close equation captures the general structure of the solution (such as a square root nested in an exponential) but not more than that, and Not Close includes all equations that are far from the solution.A.8 Further Qualitative Analysis LASR generates two artifacts: the best fit program, and the library of natural language concept that helped find that program. These artifacts provide a unique window into the inner workings of LASR. This section goes over a qualitative study of how LASR and PySR go about discovering Coulomb's law F = q1q2 4πr 2 ϵ from data. Both methods are able to find an answer to this equation. However, their approach to finding the best fit equation as well as the form of the equation they discover differs significantly.Setup: Coulomb's law is equation #10 in the Feynman equation dataset. It describes how the force between two point charges changes with respect to the distance between the charges, the magnitudes of the charges, and the permittivity of free space constant. The corresponding data for this equation has a target noise of 0.001 to simulate experimental errors.By analyzing the form of the equation and relationships between variables in Coulomb's law, we can uncover several interesting properties: First, observe that this is an inverse square law (The force F varies inversely with the square of the distance r between the charged particles). Second, notice that the F is directly proportional to the magnitude of the charges q 1 and q 2 . Third, observe that the resultant force is symmetric with respect to the magnitude of the charged particles (i.e.: The magnitude of the F doesn't change if the magnitude of the charged particles is swapped).PySR Solution: PySR finds the following solution to this equation: Obtaining a simplification of this solution is rather painstaking.F = q2•3.LASR's Solution: LASR finds the following solution to this equation. We also present three steps of simplification:F = q 1 r q2 r + 1.9181636×10 -5 q2 ϵ • 0.07957782 = q 1 r q2 r + 1.9181636×10 -5 q2 ϵ • 1 4π (Substitute constant) = q 1 q 2 r r + 1.9181636×10 -5 q2 ϵ • 1 4π (Simplify denominator) ≈ q 1 q 2 r (r) ϵ • 1 4π (Negligible. 1.9181636×10 -5 q2 ≈ 0)This equation has a complexity of 15 and achieves a much lower loss of 4.6709058 × 10 -14 on the accompanying dataset. We can see with just three steps of simplification how this equation might be reduced to the ground truth.Let's examine some essential concepts from various iterations in the search process. Keep in mind that an LLM operates on tokens in each concept. Consequently, even small relevant substrings can positively influence future LLM inference calls, despite full concepts appearing verbose to humans. • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Artifacts available at https://trishullab.github.io/lasr-webTheBloke/Mistral-7B-Instruct-v0.2-GGUF using llama.cpp 


Decision: The Reviewers agree that the method is well-motivated, and aligns well with the goal of developing a system that reflects how humans discover useful equations. Unlike prior work that uses LLMs to generate expressions for symbolic regression, the new approach, LaSR, is integrated w/ an off-the-shelf classical evolutionary SR method, so it is able to exploit the advantages of both paradigms. The method works as expected, performing well empirically, and the paper can be accepted along these lines.

The main concern with this kind of paper is information leakage, since the LLM has certainly been exposed to all the Feynman equations in its training data. The authors have taken substantial steps to address this concern, including: demonstrating the approach on a previously unpublished set of synthetically generated equations, demonstrating that the approach discovers different functional forms of the same equation over different runs, and running ablations w/ a priori knowledge removed.

This does not completely eliminate concerns of data leakage, but I would argue that completely eliminating this concern is not possible at this time. With LLM-based approaches, even knowledge of black-box problems and best known solutions so far could bias the model; and, at a higher level, knowledge of a stochastic synthetic equation generator could have a similar impact. I would argue that existing methods including PySR implicitly have tons of prior knowledge baked in, since they were developed with the benchmark in mind of rediscovering known physical equations. Overall, establishing completely "fair" comparison of SR methods remains an open problem that is outside the scope of this work. That said, I have four recommendations for clarifying the contribution:

(1) In the paper, articulate the challenges of "fair" comparison as fully as possible.

(2) I encourage the authors to follow the reviewer advice to include results on black-box SRBench problems in the final version. Although the authors say they are interested in scenarios where the ground truth is known, in a real application of such a method there would be no ground truth. These experiments could also show a "real new discovery" if they find equations on the Pareto front of known best equations. Similarly, if LaSR leads to another "new discovery" before the final version, e.g., in scaling laws as the authors have suggested, I encourage them to add it to the paper.

(3) The result that different trials produce different functional forms of the same equation should be clearly highlighted in the paper, e.g., in a table showing differing discovered solutions across different trials for multiple problems. 

(4) Clearly explain the precise motivation and construction of the synthetic domain.

One interesting result in the paper was that the LLM could be used to generate only 1% of expressions and still yield notable benefits. I would really like to see the other extreme, i.e., how well does the method do at 100% LLM? If it's possible to add these experiments to the final version, it would better contextualize the method compared to existing LLM-only methods, e.g., LMX [1], the first evolutionary approach to use LLMs to equations for SR. The Related Work section would generally benefit from discussions of such earlier approaches. As is, the reader could reach the incorrect conclusion that LaSR is the first method to generate SR equations w/ LLMs.

[1] Meyerson et al. "Language Model Crossover: Variation through Few-Shot Prompting" arXiv:2302.12170

Reviews: review: paper_summary: paper_summary: This paper introduces LASR, a symbolic regression framework that enhances PySR by incorporating Large Language Models (LLMs) to discover and evolve "concepts" from high-performing equations. These concepts are then used to guide the search process. LASR is evaluated on the Feynman equations dataset and a set of synthetic tasks, showing improved performance over existing symbolic regression baselines.
strength_weakness: strength_weakness: My main concerns are regarding the evaluation. Experimental design and analysis are insufficient to convincingly demonstrate the method's advantages over existing approaches; specifically: There are serious concerns about potential data leakage and unfair advantage when using LLMs on well-known, simple physics equations that may be part of LLM training data. While there is a section on the data leakage validation, its limited evaluation scope and lack of comprehensive analysis on more complex and real-world datasets, makes it difficult to assess the true capabilities and generalizability of the method.
questions: questions: * Could you provide the 7 additional equations that LASR solves beyond PySR as well as the forms found by LASR? Are these 7 equations consistent across runs? What have been specific contributions of the LLM for them?

* How can you ensure that the LLM is not implicitly using its prior knowledge to generate simple Feynman physics equations? The ablation study shows that variable names and hints improve performance, which raises concerns. For instance, variable names like "g" for gravity or "Kb" for the Boltzmann constant could trigger the LLM to suggest the well-known relevant equations.

* Have you conducted ablation studies that remove all physics-related terms, variable names, and any mention of physics or internal knowledge from the prompts? This would help (to some extent) isolate the LLM's ability to learn purely from data patterns.

* For equations solved by both LASR and PySR, could you provide the number of iterations required to obtain the correct form using each method?

* Could you clarify what constitutes an "iteration" in your experiments? There are discrepancies between Figure 1 (10^6 iterations), the main experiments (40 iterations), and the synthetic data experiments (400 iterations). Could you explain these differences and provide results for more iterations (maybe around 10^6 iterations), particularly for the synthetic dataset? Additionally, could you clarify how the concept library operations frequency (every 400 iteration based on Figure 2), and LLM usage frequency (p=1%) translate to these small iteration counts? For example, for 40 iterations, how many time LLM operations and concept library operations will be used?

* For the data leakage validation, why were custom synthetic equations used instead of existing datasets like black-box datasets in SRBench? This would provide a more standardized comparison and better demonstrate LASR's capabilities on real-world problems.

* Could you provide a computation time comparison between LASR and PySR for experiments with more iterations, to assess the computational trade-offs of using LLMs?

* Could you provide examples of "hints" that are provided in experiments of Feynman equations?

* Based on Table 2, increasing the value of p seems to improve performance. Have you considered analyzing higher values of p, especially for GPT-3.5? Could you explain why higher values were not explored?
limitations: limitations: See above.
paper_summary: paper_summary: This work focus on symbolic regression. They enhaned the traditional method like genetic algorithms by inducting a library of abstract textual concepts. The algorithm, called LASR, uses zero-shot queries to a large language model to discover and evolve concepts occurring in known high-performing model to discover and evolve concepts occurring in known high-performing hypotheses. The algorithm can be seen as a kind of hybird of evolutionary algorithm and LLMs. Through experiments, LASR substantially outperforms a variety of state-of-the-art SR approaches.
strength_weakness: strength_weakness: 1. Introdution of LLMs would inevitably raise the cost for the task comparing those traditional algorithms.
2. (this could be a question) The introdution of concept library seems to not making sense in all cases, imaging that we are facing some totally unknown black-box environment, the backbone function could be something random or out of the knowledge we have. In this setting, the traditional evolutionary algorithms might perform better because they do not have this kind of knowledge as the constrant.
questions: questions: 1. What's the operation of LLMINIT? If we do not have any prior knowledge, then what will we prompt LLMs to do the initialization? Is it the same when we nothing more to know in advance.
2. The initilization, mutation and crossover operation in LASR hybrid traditional evolutionary algorithms and LLMs, so what's the intuition to do this? Because there are several works that directly replace genetic operation with LLMs operation, such as (https://arxiv.org/abs/2401.02051). I am curious about have you ever tried to use merely LLMs for the whole process, that would be an interesting topic.
3. Is LASR capable to resolve high-dimentional symbolic regression tasks? Because current research about symbolic regression struggle to due with high-dimensional settings.
4. Have you ever tried to compared the work KAN (https://arxiv.org/abs/2404.19756), which can also be used in symbolic regression and report a great results.
limitations: limitations: See weakness and question parts.
paper_summary: paper_summary: This paper introduces a method that learns a library of concepts (natural language description of helpful and unhelpful concepts) as a means of guiding genetic search for symbolic regression. The core idea is that such concepts can be used to bias genetic operations through an LLM.

The method was evaluated on the 100 Feynman equations and on a synthetic dataset. The paper also includes ablation studies on the various components of the system.
strength_weakness: strength_weakness: The paper also has a few weaknesses.

**Claims that need to be fixed**

Some of the claims in the paper are a bit strong and I suggest toning them down. While the leakage explanation the authors provided is reasonable, I would be careful in claiming state-of-the-art performance. When writing "LASR achieves a higher exact solve rate than all other baselines," it is worth mentioning the possibility of leakage.

Another claim that seems to be incorrect is the following: "LaSR's increasing the backbone model size and the mixture probability significantly enhances." I think the authors meant to write "substantially" and not "significantly" as there is no statistical test involved. I would also explain why these results are substantially better as the number of problems solved isn't much larger. The explanation I gave to myself is that solving each of these equations is very difficult, so solving one new equation is already quite an achievement.

Another claim that needs to be adjusted: "demonstrating that LaSR's performance gains are not rooted in memorized responses." The experiments on the synthetic dataset do not demonstrate this. The experiment with the synthetic dataset is almost independent of the experiment with the 100 Feynman equations. What the experiment demonstrates is that LaSR can outperform PySR even when data leakage is not possible.

**Discovering what is known**

Perhaps an unfair criticism of the paper is that the method it introduces is used to discover things we already know. I understand that the bar would be way too high and it would be unhealthy to the research area if we required the discovery of new things with the presentation of novel approaches to scientific discovery. So I do not make this criticism as a means of arguing for rejecting the paper (I think the paper should be accepted), but more as a reflection of what the community has been pursuing. The hope is that systems such as PySR and LaSR will eventually be used to make actual discoveries.

**Lack of Running Time**

I missed the running time of LaSR and PySR in the paper. How do they compare with LaSR making calls to an LLM?
questions: questions: 1. In line 83, shouldn't it be $P(C)$ instead of $P_C$?

2. In the mutation operator, it is stated that for each deleted subtree, one adds a single leaf node (lines 111-112). Is this correct? Why not add an entire new subtree?

3. In the for-loop in line 4 for Algorithm 1, shouldn't it be $I$ instead of $N$?

4. Why not provide a figure in the appendix to describe Concept Evolution (similar to figures 4 and 5)?

5. How does that cascade work for PySR? (the one that didn't work)

6. Why use MSE and not the exact correct metric in Section 4.3?

7. Why is the difference between PySR and LaSR so large in Table 3, but somewhat small in Table 1? 

7. Weren't the sentences in Section 4.4 swapped? To me, the first sentence looks more refined than the second. This is because it mentions "waveforms or periodic phenomena," while the second only talks about "scientific phenomena." Perhaps the argument in that section needs some rethinking?

8. In the prompt shown in Figure 5, how much is the excerpt "relation to physical concepts" contributing to the quality of the concepts?
limitations: limitations: The paper lists all limitations I could think of, either in the last section "limitations" or throughout the paper.
paper_summary: paper_summary: This paper proposes a way to incorporate LLM prompting to improve symbolic regression. They uses PySR, a standard symbolic regression library, as their base SR algorithm. Then they add LLM prompts to different SR algorithm steps: population mutation, crossover, and initialization. They replace the PySR implementation with LLM prompted implementations 1% of the time. The LLM prompts are based on identifying high level concepts and prompting the LLM to do its operation using the concept as a suggestion to follow. High level concepts are tracked using an abstraction prompt given high performing expressions, and are evolved using LLM prompting as well. The authors show that augmenting PySR with LLM concept-based prompting solves around 7/100 addition tasks from the AI Feynman SR benchmark, and show that PySR + LLM (LaSR) performs better on a synthetic task designed to test for data leakage.
strength_weakness: strength_weakness: - It's not clear how valuable the concept abstraction approach is compared to some "baseline" of simple LLM prompting. For example, one baseline could just use a single concept "This expression is a good formula for symbolic regression" or something like that, and see how it compares. This could perhaps be a direction for future work: try a bunch of simple prompting strategies for combining LLM's with PySR, and report how well each of them work. 
- I'm not sure what to make of the synthetic dataset. In particular, PySR works so well on AI Feynman alone, but works very poorly without the LLM addition on this synthetic dataset. Why does LaSR work so much better than PySR here, but not help as much on AI Feynman? One hypothesis is that AI Feynman has a lot of easy tasks, and the synthetic benchmark only has tasks right on the edge of what PySR can discover, which LLM incorporation helps push over the edge. Another pessimistic take is that the synthetic benchmark is designed with LaSR in mind. While this still eliminates worries on data leakage, an explanation here would help understand better how LaSR is being helpful. 

I'd like to emphasize that including answers to these questions (both of which could suggest negative results) in the paper would not decrease my review score.
questions: questions: See weaknesses section. 

Some other suggestions:
- fix backwards quotations marks
limitations: limitations: N/A

Comments: title: Response to the authors
Response to Authors
Slight corrections.
comment: Thank you for answering all my questions and fixing all the small issues.
Thank you for engaging with us and helping us improve this work! Apolgies for the delay! PySR experiments took a while. We respond to your points inline:

> ground truth equations

We’re happy to include the LaSR’s discovered equation and the ground truth equations in the broader artifact release. We did not report on exact match for ground truth equations because neither the equations discovered by PySR nor those by LaSR were close enough to regard as “exact matches.” Instead, we relied on correlation metrics used in the SR community specifically for this circumstance. As shown in Table 3, we find that LaSR’s equations are far closer to the ground truth data as compared to PySR’s equations.

> Comparison on wall time instead of # iterations

In our opinion, comparing w.r.t # iterations is a fair experimental strategy for several reasons:

1. In practice, evaluating an equation itself is more expensive than the LLM queries, and thus the # iterations becomes directly correlated to time. For example, you might want to use the discovered expression inside a scientific simulation, which can be very expensive to evaluate. For such problems, the wall clock time is a function of the # iterations alone.

2. As mentioned in our original rebuttal, evaluating on wall clock time is less reliable and reproducible than evaluating on the # iterations. 

3. Regardless, LaSR’s experiments took roughly 10 hours on an 8xA100 80GB server node. We ran PySR for 10 hours (capped at 1 million iterations) on the Feynman equations dataset and report results here:

   | Feynman Equations | PySR ($1000000$ Iterations, 10 hour timeout) | LaSR ($40$ iterations) |
   | ----------------- | -------------------------------------------- | ---------------------- |
   | Exact Solve       | 59 + 3/100                                   | 59 + 7/100             |

   Overall, we find that, despite running PySR substantially longer than LaSR, LaSR still substantially outperforms PySR. This is because PySR, like other evolutionary algorithms, is susceptible to falling in local minima and converges to this local minima pretty fast. This is why supplementing such “local” search strategies with LLM guidance is useful.


> The improvement of LaSR looks marginal

In scientific discovery, finding even one additional equation can be highly valuable. We are encouraged by the fact that LaSR's ablations still outperform the current state-of-the-art algorithm (PySR) under the same conditions.

> Differing  performance for MSE metric and exact match metric

Exact match is rooted in the symbolic equations sketch discovered, while MSE is based on how well the symbolic equation fits to the dataset. The latter is susceptible to optimization errors, especially since these optimization methods do not guarantee finding a global minima (we use BFGS for all experiments). Also, we evaluate on a challenging version of the Feynman dataset where we add random noise to each equation. This further exacerbates parameter fitting errors.

The MSE threshold metric is necessary for gauging performance per iteration because evolutionary algorithms constantly evolve a pool of candidates and the MSE threshold allows localizing when the best performing candidates start appearing in the candidate pool. 

> Concerns about memorization / equation consistency

There is a slight miscommunication here. LaSR consistently discovers solution equations for these seven problems. However, the discovered equations do not necessarily follow the same format. For instance, Equation 53 in the Feynman equations dataset defines the heat flow from a point source (an inverse square law with ground truth formulation: $h = \frac{P}{4\pi R^2}$). LaSR finds the following solutions in successive runs with the same hyperparameters:

| Discovered Equation (LaSR - Llama 3b 1%; same params)        | Equation Loss      | Equation Complexity |
| ------------------------------------------------------------ | ------------------ | ------------------- |
| $h = \frac{2.8841133}{0.000219 - \left( r \cdot \left(\frac{-36.240696 - (\sin(r) \cdot 0.002057)}{P}\right) \cdot r \right)}$ | $2 \times 10^{-6}$ | $16$                |
| $h = \frac{P}{r \cdot (r \cdot 17.252695 - 0.001495)} \cdot 1.372847 $ | $2 \times 10^{-6}$ | $11$                |

Both equations fit well to the underlying dataset and reduce to the ground truth. Yet, despite using the same hyperparameters, the functional forms exhibit sharp differences. 

> The observation ... part of its inference

We are encouraged that LaSR's equations differ significantly in format from the known ground truth equations. It is highly unlikely that the discovered equations are available verbatim online, further reinforcing our hypothesis that LaSR’s performance is not simply the result of  regurgitated memorized responses.

> Discussion of LaSR ...  physical concepts

We’re happy to add relevant discussions to the manuscript. Thank you for the excellent suggestions!
Thank you for your responses. Some of my question have been answered. But I still have some concerns including:

1. For the lack of experiments on SRBench black-box datasets, the authors mention that they wanted to experiment on datasets with known ground-truth equations. However, first, the authors have not reported on the function recovery for this dataset. More importantly, LaSR is eventually compared based on fitness (R2) in sec 4.5 and table 3, which could have been the case for SRBench black-box datasets.

2. I understand that using LLMs can be time-consuming, and this is in no ways a downside. However, I find comparing LaSR and PySR on same number of iterations (and with only 40 or 400 iterations in Fig.3 and table 3) not fair, since they require very different computation and wall-clock time. Also, since PySR is a GP-based method and lacks prior knowledge, it would be reasonable for them to need more number of iterations to converge to the same level of performance. So it would be more reasonable to report the results after all the methods have converged (or at least same computation/wall-clock time), and not necessarily at the same iteration. 

3. In the current results shown in Fig 3, without the additional memorization hints (variable name, hints, or mention of physical concepts), the improvement of LaSR (which in this ablation could be potentially attributed to the LLM role in learning good vs. bad forms) looks marginal. Also, this plot shows the results on MSE and not the exact match as reported in table 1. So it is not shown that how many equation are discovered out of 100 in this condition. Based on figure 3, PySR has solved ~41 equations after 40 iterations on MSE metric but based on table 1, it has solved 59 equations based on exact match. This discrepancy in the metrics and the final results of LaSR in this condition should be explained.

Other concerns and comments:

As an additional note, the authors mention that in table 1, the 7 improved equations are consistent across all runs. As both initialization of GP methods and the responses of LLMs involve some level of variance, this would increase the concern that these 7 enhanced equations could have in fact obtained due to the memorization rather than pure reasoning and learning on the equation forms. 

The observation on the form of discovered equations is interesting and worth discussing, however, not generating the exact known form of equations does not guarantee that the model has not used its internal memorization as a part of its inference.  I would suggest authors to include a discussion in the paper on the results of LaSR without any variable name and even a notion of "physical concepts".
Thanks for your response! All of my concerns have been addressed. Good work and I will maintain my score.
Thank you for pointing this out. Apologies for any confusion this might have caused!
Due to a formatting error, it seems part of our response to your questions ended up in our response to Reviewer CCz9. We copy the relevant portions verbatim in this comment:

> **Discovering what is known…**

Our experiments on synthetic data suggest that we can use LaSR to make discoveries even without significant prior domain knowledge. In addition, since the submission, we have been actively exploring ways to use LaSR in novel discovery tasks. For example, we currently have exciting preliminary results on automatically discovering novel LLM scaling laws using LaSR. Such real-world case studies require deep collaboration with domain experts and are hence outside the scope of the present foundational effort. However, we anticipate many such results in the future. 

> **Lack of Running Time**

Our goal is to present experimental results that can be replicated reliably by the broader community. As such, we root our experiments in the number of iterations instead of wall time.

This is because the wall time is heavily dependent on the LLM backend, the number of tokens each query uses, and the network quality. For instance, for `gpt3.5` we cannot reliably measure the runtime as the backend is slower during periods of high traffic. Even for local models, vLLM's query scheduling is non-deterministic so the wall time will be substantially different across experiments.

Regardless, we were measuring around one second per query using vLLM. Since we make 60k calls per iteration (Appendix A.3.1) the amount of time for p=1% would be around 600 seconds / 10 minutes per iteration. PySR roughly takes 30 seconds per iteration. We expect performance to improve as we increase CPU/GPU parallelism, and as the hardware to run LLM inference improves.

`[1]`: https://github.com/stanfordnlp/dspy
I think your comment on runtime and discovering what is known belong in the response to a different review than mine.

Thanks for your response. I will keep my score.