A Unified Framework for Deep Symbolic Regression
Authors: Mikel Landajuela
Chak Lee
Jiachen Yang
Ruben Glatt
Claudio P. Santiago
Ignacio Aravena
Terrell N. Mundhenk
Garrett Mulcahy
Brenden K. Petersen
Venue: NeurIPS 2022 Accept (2022)
Keywords: symbolic regression
reinforcement learning
combinatorial optimization

Abstract: The last few years have witnessed a surge in methods for symbolic regression, from advances in traditional evolutionary approaches to novel deep learning-based systems. Individual works typically focus on advancing the state-of-the-art for one particular class of solution strategies, and there have been few attempts to investigate the benefits of hybridizing or integrating multiple strategies. In this work, we identify five classes of symbolic regression solution strategies---recursive problem simplification, neural-guided search, large-scale pre-training, genetic programming, and linear models---and propose a strategy to hybridize them into a single modular, unified symbolic regression framework. Based on empirical evaluation using SRBench, a new community tool for benchmarking symbolic regression methods, our unified framework achieves state-of-the-art performance in its ability to (1) symbolically recover analytical expressions, (2) fit datasets with high accuracy, and (3) balance accuracy-complexity trade-offs, across 252 ground-truth and black-box benchmark problems, in both noiseless settings and across various noise levels. Finally, we provide practical use case-based guidance for constructing hybrid symbolic regression algorithms, supported by extensive, combinatorial ablation studies.

A Unified Framework for Deep Symbolic Regression
Mikel Landajuela landajuelalai@lnnl. gov
Chak Shing Lee keee029@1lnl. gov
Jiachen Yang
yang40ellnl. gov gov
Ruben Glatt glatt18linl.go
Claudio Santiago
Nathan Mundhenk mundhenk1 @11n1 gov
prata@linl. gov
Ignacio Aravena Travenasolis@@lnn) gov
Garrett Mulcahy mulcahy4@llnl.ge gov
Brenden Petersen" bpël1nl. gov
Computational Engineering Division Lawrence Livermore National Laboratory Livermore, CA 94550
Abstract
The last few years have witnessed a surge in methods for symbolic regression, from advances in traditional evolutionary approaches novel deep learning-base based systems. Individual works typically focus on advancing the state-of-the-an for particular class of solution strategies, and there have been few attempts investigate the benefits of hybridizing or integrating multiple strategies. In this work, identify five classes of symbolic regression solution wrategies-recursivv problem simplification, neural-guided search, large-scale pre-training, genetic programming, and linear models-and propose strategy to hybridize them into a single modular, unified symbolic regression framework. Based on empirical evaluation using SRBench, new community tool for benchmarking symbolic regression methods, our unified framework achieves sfate-of fiefthh-ar art performance in its ability (1) symbolically recover analytical expressions, (2) fit datasets with high accuracy, and (3) balance sccracy-complexy ground-truth and black-box benchmark problems, both noiseless settings and across various noise levels. Finally, we provide practical use case- case-based guidance for constructing hybrid symbolie regression algorithms, supported by extensive, combinatorial ablation studies.
trade-offs, across 252
Introduction
simplng Ann
(X,y) v)
Solve P1
con bine
(X,y)
Symbolic regression (SR) aims to
(m)
identify tractable mathematical ex- pression best fit dataset. Specif- ically, given the set of pairs {(x:
(X, 9)
Solve Pm
encode de
tion
evolve ve
( Fi,a), 31) J SR seeks mathematical function f R TR. such that the residual [x) y minimized, where the matrix X
(X, y)
A
TRNN
TGP
cont
Controller
Encoder
optimize
train
gaxd has entries X., Zijj the vec- tor y E R" has entries y: and we
Solve P;
TRNN U TGP
Figure 1: Unitied deep symbolic regression. The five integrated solution strategies are color-coded: AIF, DSR, LSPT, GP, LM.
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

assume wiint-wise application of real-valued functions over vectors and matrices, i.e., f(X) = (f(1 E R (we keep this convention throughout the paper for convenience).
Since the introduction of SR as an application of genetic programming (GP) (Koza, 1994), many classes of methods especiilly machine learning-based methods in recent years have been proposed. These methods include recursive problem simplification (Udrescu et al. 2020), neural-guided search (Petersen et al.)
2021a), large-scale pre-training for problem generalization (Biggio et al. 202T Kamienny et al.
2022 Vastl et al. 2022). sparse linear regression using nonlinear basis functions (Brunton et al. 2016). and combination of evolutionary search with learning (Mundhenk et al.
2021). Each of these independent advances exploits different information (e.g., graph modularity or constraints) to offer key capabilities for various aspects of the SR problem (e.g., search or generalization), but the specificity of each work resulted in design choices that pose limitations in other facets of the problem. This motivates an investigation of whether disparate solution strategies are complementary to one another and how they can be effectively combined. In this paper, we carefully rewire the five aforementioned solution strategies into a unified SR framework and show that they are indeed complementary.
Based the SRBench pipeline for benchmarking SR algorithms bhttps://githei Brbench), our unified deep symbolic regression (uDSR) framework achieves a new state-of-the-ar in SR: for ground-truth problems, uDSR ranks highest in both symbolic- and accuracy- based solution rates for both noiseless settings and for multiple noise levels; for black-box problems, uDSR falls on the eccuracy-complexxs Pareto frontier alongside three other non- non-dominated algorithms.
com/cavalat
Our main contributions in this work include: (1) a modular, unified framework for SR that integrates five disparate SR solution strategies (illustrated in Figure D (2) novel LINEAR token that identifies sparse coefficients of a linear combination of basis functions and can be used by almost any SR algorithm; (3) achieving new state-of-the-art for SR on SRBench in its metrics for symbolic recovery, solution accuracy, and accuracy vs complexity trade-offs; and (4) practical recommendations for designing SR algorithms, driven by extensive and combinatorial ablation studies that show the complementary benefits of these previously disjoint SR solution strategies.
2 Background
We describe each of the five solution strategies framework unifies. For each strategy, we focus on a key capability and a key limitatior summarized in Table I which motivates our unification that captures all capabilities and mitigates all limitations via the inherent synergy among strategies. We encourage readers to read the original works for additional background.
AT Feynman. AI Feynman 2.0 (AIF) (Udrescu et al. 2020) approaches SR via recursive problem simplification; it is the only known algorithm to date to employ this solution strategy. Specifically, AIF exploits knowledge of physics and the given training data, (X, y). by identifying simplifying properties (c.g., multiplicative separability) of the underlying functional form. Applying a simplifica- tion yields sub-problems (or one sub-problem) of lower dimensionality, Simplifications are applied recursively until all sub-problems (i.e., leaf nodes of the recursion tree) are I-dimensional. For each sub sub-problem, the paper employs combination of brute force search plus polynomial fit (BF/PF). Finally, solutions to the sub-pra -robllrssems are then combined back into a final solution to the original problem. While AIF positions itself standalone SR algorithm, it is more appropriately understood as a SR problem simplification algorithm, as pointed out in prior work (Petersen et al. 2021a Biggio al. 2021). From this general perspective, the use of BF/PF is simply design choice, orthogonal to the core methodology, and may be enhanced by integration with more advanced search methods for solving sub-problems.
Deep symbolic regression. Deep symbolic regression (DSR) Petersen et al. 202 leverages the effectiveness of deep reinforcement learning for combinatorial search in SR. DSR employs neural network controller with parameters to represent a distribution p(T 0) over mathematical expressions [T....., Tr represented by the sequence of "tokens" T; (mathematical operators, input variables, or constants) in the pre-order (depth- firstt then left-to-right) traversal of the corresponding expression
"e clarify that the key limitations are not necessarily issues with the algorithms themselves: often, they are conscious design decisions by the authors. For example, in the context of physics-basee based SR problems, AIF's use of BF/PF only strengthens their claim that recursive simplification alone is often sufficient to solve the problem. This work considers wider context of challenging SR problems, in which these be viewed as limitations.

---

tree (see Koza 1994) for details). For instance, 7 = [+, 11, corresponds to expression f() = exp(x2). Expressions are sampled from the controller autoregressivelyy i.e.
p(T = Ie pf f), and evaluated based on fitness R(T, (X. to the (X,y) dataset.
The controller then trained to maximize the reinforcement learning objective:
J(0) == Er-p(r10) [R(T, (X,y))).
(I)
using a risk-seeking policy gradient that aims to maximize best-case performance. Autoregressive sampling affords the abilitty to easily incorporate prior knowledge by directly modifying likelihoods of individual tokens Tis,/1 0) in situ, for example by precluding expressions with nested trigonometric operators. This dramatically reduces the search space, which has been shown to greatly improve the ability to recover ground-truti expressions (Petersen et al.) 2021b). Whereas DSR focuses discrete search and pruning the combinatorial search space, it does not exploit the (X, y) data. In fact, the (X, y) data itself only shows up in the reward computation in I which is a weak, aggregate learning signal. Thus, DSR trains from scratch for each new SR problem and cannot generalize to new problems.
Large-scale pre-training. Large-scale pre-training (LSPT) algorithms (Biggio et al. 202 Kamienny al. 2022, Vastl et al. 2022) tackle the problem of generulization in SR: they seek a single model that, once trained, produces distribution over solutions to any SR problem. To accomplish this, the algorithm "Neural Symbolic Regression that Scales" (NeSymReS) (Biggio et al. (02T) uses a set transformer (Lee et al. 2019) to learn encoding of the (X, y) data. A second transformer network then maps the encoding to distribution over expressions using autoregressive sampling. The model is trained end-to-end via supervised learning, using a large dataset of SR problems (input data (X, y) and corresponding ground truth label 7) produced by a programmatic expression generator (Lample) and Charton 2019). Given (X.y) y) at test time, beam search is used to find eigh-likelihoo expressions. LSPT approaches claim that pre-training alone is sufficient to solve many SR problems. However, the lack of fine-tuning during testing means that LSPT algorithms may not be optimized for new out-of-distribution SR problems.
Genetic programming. Genetic programming (GP) refers to well-cstablished class of evolutionary algorithms commonly applied to SR in which a set or "population" " of mathematical expressions is maintained and "evolved"" over time using genetic operators such mutation, crossover, and selection. We refer the reader to Langdon and Poli (013) for an overview of GP. In this work, we consider only standard GP (as defined in Chapter 7 of Bäck al. (018);; however, integration of GP into the unified framework applies to virtually any GP method. GP excels at exploring the search space to find complex expressions, but does not employ a neural network or other parameterized model to learn over time. This limitation was addressed in Mundhenk et al. (2021), where GP was combined with a learned expression generator, but not in concert with other methods considered here.
Linear models. Linear models (LM) are a mainstay of statistics and machine learning MMurhhy 2022). LM methods seek expressions of the form y = Σ 3:0; = $ where $b == (di,.... dx) comprises library of A user-defined basis functions Φ; RR - R. and , = (BI, Bz) a vector of sealar coefficients that is often learned using least squares minimization. To avoid over-fitting, the vector / can be alternatively optimized using sparsity inducing optimizers. As an example, the algorithm "Sparse Identification of Nonlinear Dynamical systems" (SINDy) (Brunton et al. 2016) is a simple, computationally expedient, and popular LM method for SR. particularly for learning systems of differential equations. SINDy identifies sparse representations by using LASSO [Tbssiiran] 1996). While effective at quickly learning sparse expressions, the restriction to a linear combination of pre-defined basis functions results in massively reduced search space, precluding such functional forms as rationals and compositions. In our unified framework, we abstract the entire process of learning linear model into one single discrete token, thereby restoring the nonlinear, combinatorial search space while retaining the ability to quickly learn sparse linear subcomponents.
3 Related Work
In addition to AIF, DSR, LSPT, GP, and LM described above, we describe several other related SR solution strategies that are not part of our unified framework. EQL* presents a unique deep-le framework for SR that employs neural networks whose activation functions are elementary operators, enabling end-to-end differentiable training (Sahoo et al. 2018). Grammar VAE learns to encode expression trees in continuous space and uses Bayesian optimization to optimize them in latent
3

---

Table I: Key capabiliiiess and limitations of the five unified solution strategies.
Solution strategy
Label
Key capability
Key limitation
Feynman
ATF
Exploits (X, 3) data to sim-
Underlying SR algorithm resorts to simple BF/PF.
plify a SR problem into lower- dimensional sub-problems.
Deep symbolic
DSR
Neural network learns over time, with the ability to incorporate in situ constraints.
Does not exploit the (X, y) data, which only appears in the (weak) reward function.
regression
Large-scale pre-training
LSPT
Leverages big data, learning from many other problems by condi- tioning on the (X, y) data.
Limited to fixed numbers of di- mensions, and does not fine-tune the search a particular SR prob- lem.
Genetic programming
GP
Rapidly explores the search space via genetic operators. Quickly learn sparse coefficients linear combination basis functions.
Does not employ a neural network learn over time. Operate on dramatically duced search space.
Linear models
LM
space (Kusner et al. 2017). Both EQL* R and GrammarVAE impose specific representations of the expressions (either by constraining the search space or by projecting it into a latent space) and thus do not facilitate straightforward integration with the solution strategies in our framework. The idea of pre-training models for use in neural-guide search, used by LSPT approaches and our unified framework, is inspired by Bello al. (0117. who apply similar method the traveling salesman problem.
Previous work on hybrid approaches to SR mainly focus on improving a single aspect of one approach, such as alleviating the challenge of high dimensional data by extracting features for GP (Tcke and Bongard 2013), or improving the starting population of GP iterations (Mundhenk et al. 2021 Often, the motivation for combining separate methods comes from specific problem settings, such as the case of datasets with missing values (Al-Helali al., 2018) or noisy high-dimensional measurements in systems with known physical constraints (Reinbold et al. (02T). In contrast, the unification in our work serves enhance multiple general capabilities SR, such as generalization, combinatorial search and optimization, and tinlletooken abstraction of function spaces.
Our empirical analysis heavily leverages SRBench, reproducible and open-source platform for benchmarking new SR methods against 14 established methods on accuracy and complexity for 122 real-world datasets and exact symbolic correctness for 130 ground-truth problems with varying noise levels LLa Cava et al. 2021 Bench has seen rapid adoption by recent works that use it to evaluate improvements in accuracy, exact solution rate, and solution complexity (Kamienny et al.
2022 Virgolin and Bosman 2022 Zhang et al., 2022).
Lastly, we note similarity between this work and Rainbow, work that combines several deep reinforcement learning methods into a unified algorithm and performs extensive ablation studies (Hessel et al. 2018). Analogous to Rainbow's 's use of Deep Q Networks (DQN) base framework into which it integrates several improvements, this work uses DSR as a base framework into which we integrate AIF, LSPT, GP, and L.M.
Methods
Our overall unification strategy is to carefully rewire the solution strategies identified above as con- nected but non-overlapping modules in an algorithmic framework that leverages their key capabilities. This is visualized in Figure [ (see also Figure [2] Appendix L for an illustrative example of expression discovery under uDSR). In an initial offline stage, pre-training a parametric controller model on large dataset facilitates generalization to test problem instances (LSPT). In the online stage, given specific problem instance, recursive problem simplification module (AIF) exploits modularity of mathematical expressions to produce sub-problems of lower dimensionality for the main trunk of uDSR. Within the trunk, a Recurrent Neural Network (RNN) continually learns over many iterations (DSR) and provides good candidates to seed the starting population for genetic programming (GP); high fitness populations produced by GP are combined with elite candidates for

---

controller parameter updates via risk-seeking policy gradient. Finally, the RNN is permitted the use of powerful symbolic token that encapsulates an entire space spanned by basis functions (LM).
Below, we deseribe the integration of each module in detail. Algorithm provides high-level pseudocode for uDSR. More detailed pseudocode is available in Algorithm of Appendix
Table 2: Feature compatibility matrix for uDSR and the five integrated solution strategies.
Recursive
problam
simpliication
Search space pruning
Leverages
big data
Evolutionary
search
Lincer
constaut
uptimization
Nanlincar
constant optimization
Neural guided starc
ATF
DSR
LSPT
GP
LM
dSR
Beginning with DSR. For DSR, we begin with the method proposed by Petersen al. a0) including two improvements to exploration (soft length prior and hierarchical entropy regularizer) introduced in follow-up works (Landajuela al. 202 la (b. DSR includes number of in situ priors and constraints to the search space, which help bias and/or prune the search, and have been shown to greatly improve performance (Landajuela et al, 2021b). We provide a complete list DSR priors and constraints in Appendix B including several new constraints required to unify all five methods (described below). DSR also includes a CONST token that represents an arbitrary floating- point constant, whose value is optimized using a nonlinear optimizer, c.g., L-BFGS-B B (Zhu et al. 1997).
Integrating AIF via recursive simplifications. Our key insight to integrate AIF (Udrescu al. 2020) is that its recursive problem simplification steps can be viewed as orthogonal to the underlying SR algorithm. While AIF resorts BF/PF, we propose using a hybrid of the other four integrated algorithms (which we continue to describe in subsequent paragraphs). To integrate AIF, we consider the starting SR problem as Pi, a d-dimensional SR problem. One application of AIF's simplification step produces either one sub-problem P2, with dimensionality d2 < d. or two sub-problems: Pg and as well metadata M containing information
P3. with dimensions d2 and d3 such that d2 + d3 required to recombine sub-solutions back to the full solution of PI. Recursive simplifications are applied until sub-problems at the leaf nodes of the recursion tree are each 1-dimensional. For each sub-problent P, (including the starting problem, P1), we apply our hybrid SR algorithm to its corresponding dataset, (X, y):- Finally, using the metadata from each simplification step, solutions from all sub-puobeee bobb-roblems are recombined to form single, final solution to the original problem, Pi.
Integrating LM via the novel LINEAR token. Our design goal for integrating LM is retain the ability to quickly learn linear subcomponents of an expression, while still operating in a general nonlinear functional space. That is, while LM finds expressions of the form g(x) (xx))
seek to search more generic expressions of the form f(x) F(x, g(x)), where F is nonlinear function subject to certain constraints (see below), while still using linear methods determine the coefficients of (zx). We accomplish this by proposing to represent g(x) as a special type of terminal token, which we call LINEAR. LINEAR serves as a placeholder for a LM-like solution, Tx where the values of the coefficients 8 are determined only when traversal has terminated. Thus, abstract the entire process of learning a linear model into a single discrete token. In our approach, the function F(x, w)) must admit expression tree with single argument token for and there must exist G(x, v) such that y Fxx, ((r, Vx, y Rd R.
Given an expression far) F(x, ())) with F as described above, let F( denote the view of f as a function of 3 only. Naively, one could solve a nonlinear optimization problem to determine the values of just DSR and Biggio et al. (2021) optimize CONST tokens. However, nonlinear optimization is computationally expensive, scales poorly with more coefficients, and challenging to induce sparsity. Instead, consider the following convex optimization problem, for which there
5

---

are efficient sparse linear solvers: arg min L(3) == [X) J (.)
(2)
BER*
We attain this formulation by solving the equation y F(X, LINEAR) for LINEAR, which gives LINEAR = = y). Since this requires taking the inverse of all ancestor operators of LINEAR- binary operators are reduced to unary operators by lexical closure, elaborated more programmaticalyy below- we use J- (v) =- G(X, v) to denote the view of G' as a function of only.
Equation exhibits attractive property that enables LINEAR to exactly recover non-linear expressions with linear subcomponents. When LINEAR is chosen at the correct position in a traversal, in the sense that the true computational graph contains linear combination of basis functions in the
corresponding subgraph, we have that / (v) exactly equals that linear combination. For example, given (X, y) data generated from a ground-trute expression (z) sin( +epp 3.442142)
and a basis containing all monomials up to degree 3, the traversal T = SIN, 21, EXP, LINEAR]
yields/ f (v) = log (y- sin(x1)) )). and the corresponding solution to exactly recovers S*
Programmatically, we solve 2 using the expression tree for r and the following procedure (illustrated
First, partially execute the tree by evaluating all nodes that are not direct ancestors of LINEAR. Second, redefine binary operators as unary operators by fixing (or "binding") the one pre-computed argument (this is called partial application a type of lexical closure- in computer science). Third, successively apply the root node's 's inverse operation on the y data until LINEAR is the only token in the tree, resulting in J (v). Finally, perform a linear fit on using the transformed data, (X, (v))-
in Figure O of Appendix C
pscudocode provided in Algorithm 2 Appendix AA:
This procedure for abstracting linear model into a single operator exhibits several complications: (1) it only works if there is exactly one LINEAR token, (2) it does not work in the presence of other placeholder tokens, e.g., the CONST token used in DSR, and (3) it does not work if LINEAR has non-invertible ancestors (e.g., SIN), as = F(X, LINEAR) might not have a unique solution on LINEAR. Fortunately, seamlessly incorporating these restrictions into the search space is precisely what the DSR algorithm excels at. To this end, we enable LINEAR into our framework by introducing three constraints to the search space (further detailed in Appendix B): (1) there can be most one LINEAR token; (2) the LINEAR token and CONST token are mutually exclusive- -ne cannot appear if the other is present; and (3) the LINEAR token cannot be the descendant of a non-invertible unary token, e.g., SIN. In this work, we consider the set of basis functions & given by the monomials of degree up to D and solve using a sparsity-inducing least square method (see Appendix C for details).
Lastly, we observe that standard LM regression is restored when F(x, w) w. In that case, we recover the function fxx (T)) with trivial traversal [L.NEAR]] Thus, our method expands the search space of LM by allowing nonlinear function compositions. For instance, F(x, w))
leads to f(x) . which is outside the search space in standard LM regression.
1
Integrating GP via population seeding. To our knowledge, hybridization of DSR and GP is the only combination of the five methods that has been explored to date. To summarize, as in Mundhenk et al. (021). we hybridize DSR with GP by abstracting the GP algorithm as an inner optimization loop within DSR. Each time DSR samples batch of expressions, we use this batch as the starting population for an inner-loop GP algorithm. The top samples at the end of GP are combined with the DSR batch for training the neural network. This process leverages GP's ability to rapidly explore the search space, while using DSR's 's persistent neural network to mitigate the disadvantages of GP's non-parametric nature Mundhennk et al. 2021).
Integrating LSPT via supervised or reinforcement learning. We integrate LSPT by merging the RNN used in DSR with a set transformer in an encoder-decoder fashion. Our encoder architecture closely mimics that of NeSymReS (Biggio et al. 2021). The set transformer encodes given dataset (X, y) into a latent representation Ag- This representation is passed as initial state to the RNN, which decodes it into an expression т. The resulting architecture models the distribution p(r r 0, vb (X, v)). where 0 and v are the parameters of the RNN and set transformer, respectively (see Figure D]. As in Biggio et al. (2021), we pre-train the model end-to-end on many SR problems using supervised learning, using ground truth expressions as labels, resulting in parameters 0*, g/a However, different from existing LSPT approaches, given a new dataset (X', g²), we propose to search for a symbolic

---

Algorithm/ Unified deep symbolic regression. Color key: ATE, LSPT, DSR, GP, LM
input Symbolic regression problem consisting of tabular data (X,9) output Best fitting expression T*
parameters Pre-trained controller and encoder parameters (0", , objective function J(0, v))
1: Ply..., Pm 4- AIF recursively simplity P into sub-problems
2: for each sub-problem P. and corresponding dataset (X, do
<- load pre-trained encoder and controller parameters 4: while budget not exceeded do
3: 0= 0", _/="" ,0''=="="''
5:
TRNN sample N expressions from p(r]0, ,,, (X,
f Tup
evolve population by performing S generations of GP
7:
T f combine TRNN and top expressions from Top
8:
T + compute coefficients of LINEAR tokens
9:
A < train controller T according to [(0, sp²)
10:
( + store best expression for problem P.
7"
< use AIF combine solutions 7(1) .....T (m) into final solution
12: return T"
expression by fine-tuning p(r|0, x*, (X), using reinforcement learning, with initial conditions 8 0"
The pre-training phase requires large dataset comprising SR problems, each defined by set of points (X. y) along with true expression т. To scale to arbitrary dataset size, we seek a problem instance generator pe (T, v)) = pa ((X, ())()))((T)......)))))))))))>))) from which we can sample expressions and corresponding (X,y) data. Full details of the sampling process used by Pg are provided in Appendix D Briefly, we leverage the fact that the distribution induced by DSR includes prior (Petersen et al. 2021b). To generate expressions, Pg = p(T(0 = 0) directly samples from this prior, ignoring emissions from the RNN. Thus. the algorithm bootstraps "imagines" expressions that it then uses for training data, rather than relying on an external expression generator as in Biggio al. 2021); Kamienny et al. 2022)) Vastl et al. 2002 After sampling pg (g((X, y)(r) is obtained following the strategy in Biggio et al. (0021).
In this work, we explore two different objectives for model pre-training. First, following Biggio et al. 2021 we consider the supervised learning (SL) objective:
JSL (0, vb) == (Xw)) [logp (Tyl0, bb, (X, v))]
During fine-tuning, ground truth labels will not be available. To mitigate an objective mismatch be- tween pre-training and fine-tuning objectives, we alternatively explore pre-training à la reinforcement learning (RL) (Bello et al. 2017). Specifically, we consider:
(R,, v) = Exx.w)/rrg [Br<p(rie,w.(xxu)) [R(T, x),))))]-
(3)
In practice, this objective is maximized using novel formulation of the risk-seeking policy gradient for multitask learning (see Appendix [ for details).
Experiments
Pre-training. We pre-train four models: two trained using SL. and two trained using RL. each with and without the LINEAR token. Details of the pre-training setup can be found in Appendix [
Experimental setup using SRBench. We empirically assess uDSR using SRBench La Cava et al. 202I), an ooen-source and reproducible pipeline for benchmarking SR algorithms. SRBench features 130 problems with hidden ground-truth analytic solutions and 122 real-world datasets with no known analytic model ("black-box"" problems) from the PMLB database (Olson et al.) 2017). By running uDSR through the SRBench pipeline, enable direct comparison to its curated results of 14 contemporary SR methods. The 14 baselines are described in Appendix
For all experiments, we use minimal set of tokens: +. X, +, SIN, COS, EXP, LOG, SQRT, 1.0, CONST, and (except for the appropriate ablations) LINEAR. For simplicity, our choice of $ (basis functions for LINEAR) includes only polynomial terms up to degree 3. Hyperparameters are listed
7

---

uDSR
uDSR
AIF
Operon
AFP_F FE
GP-GOMEA
DSR
SBP-GP
AFP
MRGP
GP (gplearn)
AIF
GP-GOMEA
ITEA
EPLEX
EPLEX
AFP
Ne
FEAT
Operon
GP (gpleam)
SINDY
ITEA
SBP-GP
Target Noise
NeSymReS
Target Noise
BSR
0.0
0.0
FEAT
SINDY
FFX
BSR
0.01
MRGP
0.1
FFX
0.1
20
40
60
80
20
40
60
80
100
Symbolic Solution (%)
Accuracy Solution (%)
Figure 2: AKBench--generaee comparisons of symbolic (left) and accuracy-based (right) solution rates for uDSR and 16 baseline SR methods, averaged across 130 ground-truth SR problems and four different noise levels.
Table Appendix F When possible, we reused the hyperparameters of the published methods most simila ar each uDSR component.
We ran uDSR via SRBench on the 252 datasets with 10 random seeds each. Each run was performed for up to 2,000,000 expression evaluations per sub- cub-problem, until 24 hr walltime limit was reached. For ground-truth problems, SRBench computes two metrics, each averaged across the 130 problems: (1) symbolic solution rate, based on symbolic equivalence using SymPy (Meurer et al. 2017) and (2) accuracy solution rate, defined achieving an R² 0.999 on held-out test data. For black-box problems, SRBench reports accuracy (R² on held-out test data) vs complexity (number of nodes in the SymPy-parsed expression tree).
Performance results. Figure 2 shows that uDSR outperforms all other bench- marked methods both in symbolic solu- tion (by large margin) and accuracy solu- tion rates. Figure 3 shows that uDSR falls on the Pareto frontier, alongside Operon (Kommenda et al. (000)) (at higher ac- curacy and complexity). and DSR and SINDy (at lower accuracy and complex- ity).
RandomForent
MRGP
LGBM
Complexity Rank
AIF
15
SRP-GP
KomelRidge
PEAT
ITFA
Notably, the previously published SR- Bench results had no clear winner across the three categories of (1) symbolic solu- tion, (2) accuracy solution, and (3) Pareto efficiency. The top performers for sym- bolic solution rate (namely, AIF) dif- fered from those for accuracy solution rate (namely, Operon and MRGP). Only Operon appeared as top performer in two categories. However, in this work, that uDSR is highest in all three categories.
EPLEX
10
AFP
GP GP-GOMEA
GDBR
NeSymRee
5
[spleamn
DSR
SINDY
R² Test Rank
Figure 3: BeBBench-generater comparison of test It2 (on held- data) vs complexity uDSR and 22 baseline regres- sion/SR methods, averaged across 122 black-box SR problems.
6 Ablations and Discussion
Ablations. Since uDSR integrates many disparate components, is critical assess the relative contributions of each component in various contexts. To identify synergies, anti-synergies, and

---

1.0
Symbolic success
Accuracy Success
0.8
0.6
DSR
0.2
6P
AIF
05R
1SPT
Figure 4: Combinatorial ablations of uDSR components on 130 SRBench ground-trun problems. Indicators below bars denote which components enabled for each ablation. v: Component enabled. Component disabled. S: Pre-training enabled using SI.. R: Pre-training enabled using RI.. #: Number of enabled components. Error bars represent standard error across random seeds per problem. Colored labels are referred to in the main text. Ablation 'D" is equivalent to Mundhenk et al. 2021 Ablation 'DSR' is equivalent to Landajuela
et al. 2021b).
diminishing returns among the integrated methods, we perform a large-scale scale study of all combinations of the five integrated methods (standard "leave-one-out" style ablations in which we begin with the full uDSR and ablate one component at time do not provide this full picture). There are 48 total combinations: AIF E [on, off] x GP E [on, off) x LINEAR E [on, off] x pre-training E [SL,, RL, off] x DSR [on, off). We define ablating each component as follows: For AIP, "off" means that we do not perform recursive "off" means we do not perform inner-loop GP algorithms between batches. For LINEAR, "off" means we exclude the LINEAR token from the library. For pre-training, SL and RL refer to whether we use the model pre-trained using SL or RL; "off" means we do not use any pre-trained model, and the controller architecture does not include the set transformer component, For DSR, "off" means the learning rate for the neural network is set zero; notably, the controller is still used for sampling, including all priors and constraints. For each ablation, we run on the 130 ground-truth SRBench problems with 10 random seeds each (48 x 130 10 62, 400 total runs), using a maximum of 500,000 expression evaluations per sub-problem.
only the root problem is considered. For GP,
Figure shows the symbolic solution rate and accuracy solution rate computed by SRBench, with labels for select ablations we refer to in the subsequent discussion. Generally, performance increases with the number of enabled components, N. For fixed N, GP and LM tend to yield the largest marginal performance gains. Refer to Figure D in Appendix for a visualization of the marginal performance gains yielded by each component.
Recommendations. The full uDSR algorithm, using all components and RL pre-training achieves the best symbolic and accuracy success SRBench. Nonetheless, we believe that the choice of SR algorithm should be guided by use case. Thus, based on the extensive ablation studies presented above, provide our recommendations for the five integrated solution strategies.
AIF's recursive simplifications have the attractive property of providing strict improvement over the base SR algorithm, since the root sub-problen in the sub-problem tree is simply the original problem; sub-problems further down the tree only have the ability to improve the end result. This property comes at the expense of incurring roughly m-fold increase in computational cost, where m is the number of identified sub-problems. Empirically, our ablations confirm the strict improvement (c.g.. compare 'A' with 'DSR' in Figure g. though performance gains exhibit diminishing returns as the number of integrated methods increases. Thus, we suggest inclusion of AIF for use cases in which computational cost is not an important bottleneck. Finally, we note that we only considered AIF without its dimensionality analysis component, as this is really a pre- pre-prceessing step orthogonal to the SR algorithm. For real-world problems with physical units, we highly recommend that users first perform dimensional analysis on their SR problem.
The best results were obtained using RL pre-training with all other components enabled. However, in general, L.PTT across the ablations yielded mixed results. For example, the only significant anti-synergy among all components occurs between SL pre-training and GP (compare 'B1' and 'B2'

---

in Figure 4.. To gain further insight into which use cases may benefit from LSPT, we inspected the initial rewards for each pre-training modality before fine-tuning. As shown in Figure of Appendix D both SI. and RI pre-trained models yield higher reward than randomly initialized models. This is
especially the case for RI. pre-training, where the traces of the initial expressions (see Figure g of
Appendix D appear very close to the real data. Thus, pre-training can be desirable for use cases with very low budgets (e.g., zero-shot learning), a point reinforced empirically by Biggio et al. 202T). However, shown in Figure 7f Appendix D pre-trained models may have tendencies to become too "sharp" (i.e., exhibiting low entropy), resulting in reduced exploration that can hinder performance during fine-tuning We conclude that to be effective in combination with other solution strategies in SR. pre-trained models must strike the delicate balance between high initial reward and sufficient initial diversity.
Both the LINEAR token and the inclusion of our GP component almost always provide large improve- ments in both symbolic solution and accuracy solution (e.g., compare 'C" and 'D' with 'DSR' in Figure F. They also do not incur significant additional computational cost given a fixed budget of expressions (see Table T of Appendix H.. Thus, we highly recommend their inclusion for most use cases. A limitation of the LINEAR token, and to a lesser extent the use GP, is that it is inefficient in improving accuracy relative to its complexity, which we show in Figure of of Appendix |] Thus, users may not want LINEAR or GP for use cases in which they seek a low-complexity expression.
Conclusion
We introduce a modular framework for SR, which we carefully integrate five different SR solution strategies in attempt to maximize the benefits of each and cover each other's 's weaknesses. Our unification strategy focuses on abstraction of the constituent strategies: using DSR as a baseline, we abstract AIF as algorithmic wrapper, LSPT as a pre-training step. GP an inner-optimization loop, and LM as a single optimizable token. Using the SRBench pipeline for benchmarking SR algorithms, uDSR demonstrates state-of- of-thooft performance across 252 benchmark problems. For future work, we leave open the questions of using more advanced modules for GP, such as GP-GOMEA (Virgolin et al. 2017), or controller architectures (such transformers as alternatives to the RNN), well as the possible benefits from hyperparameter tuning individual modules. We hope that future work in SR considers additional strategies for hybridizing and/or abstracting SR solution strategies.
Acknowledgments and Disclosure of Funding
We thank Livermore Computing at Lawrence Livermore National Laboratory (LLNL) for the compu- tational resources that enabled this work. Funding was provided by the LLNL Laboratory Directed Research and Development project 19-DR-003. We thank the Computational Engineering Directorate and the Data Science Institute LLNL for additional support. This work was performed under the auspices of the U.S. Department of Energy by LLNI. under contract DE-AC52-07NA27:444 L.LNI-CONF-835375.
References
Abolafia, D. A., Norouzi, M., Shen, J., Zhao, R., and Le, Q. V. (2018). Neural program synthesis with priority queue training. arXiv preprint (rEXiv11001. 80.00326..
Al-Helali, B., Chen, Q. Xue, B., and Zhang, M. (2018). A hybrid gp-knn imputation for symbolic regression with missing values. In Australasian Joint Conference on Artificial Intelligence, pages 345-357. Springer.
Arnaldo, I., Krawiec, K. and O'Reilly, U.-M. (2014). Multiple regression genetic programming. In Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, pages 879 886.
Bäck, T., Fogel, D. B., and Michalewicz, Z. (2018). Evolutionary computation 1: Basic algorithms and operators. CRC press.
Bello, I., Pham, H., Le, Q. V. Norouzi, M., and Bengio, S. (2017). Neural combinatorial optimization with reinforcement learning.
10

---

Biggio, L., Bendinelli, T., Neitz, A., Lucchi, A., and Parascandolo, G. (2021). Neural symbolic regression that scales. In International Conference on Machine Learning, pages
PMLR.
Brunton, S. L., Proctor, J. L., and Kutz, J. N. (2016). Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 1 113(15):3932-399)
de França, F. O. and Aldeia, G. S. I. (2021). Interaction-traassformatioin evolutionary algorithm for symbolic regression. Evolutionary computation, 29())367-390.
Fortin, F.-A., De Rainville, F.-M., Gardner, M.-A. G., Parizeau, M., and Gagné, C. (2012). Deap: Evolutionary algorithms made easy. The Journal of Machine Learning Research, 13(1:21111-2755.
Friedman, J., Hastie, T., and Tibshirani, R. (2010). Regularization paths for generalized linear models via coordinate descent. Journal of statistical software, 33(1):1.
Hessel, M., Modayil. J., Van Hasselt, H.. Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. (2018). Rainbow: Combining improvements in deep reinforcement learning. In Thirty--econd AAAI conference on artificial intelligence.
leke, I. and Bongard, J. C. (2013). Improving genetic programming based symbolic regression using deterministic machine learning. In 2013 IEEE Congress on Evolutionary Computation, pages 1763-1770. IEEE.
Jin, Y., Fu, W., Kang, J.. Guo, J., and Guo, J. (2019). Bayesian symbolic regression. arXiv preprint arXiv:9900088991
Kamienny, P.-A., d' Ascoli, S., Lample, G., and Charton, F. (2022). End-to-end symbolic regression with transformers. arXiv preprint arXiv:2204. 10532..
Karpathy, A. and Fei-Fei, L. (2015). Deep vesual-semantic alignments for generating image descrip- tions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
Kim, J. T., Landajuela, M., and Petersen, B. K. (2021). Distilling wikipedia mathematical knowledge into neural network models. arXiv preprint (10:::20000003000
Kommenda, M., Burlacu, B., Kronberger, G., and Affenzeller, M. (2020). Parameter identification for symbolic regression using nonlincar least squares. Genetic Programming and Evolvable Machines, 21(3):47--01.
Koza, J. R. (1994). Genetic programming as a means for programming computers by natural selection. Statistics and computing, 4(2):87-112.
Kusner, M. J., Paige, B., and Hernández-Lobato J. M. (2017). Grammar variational autoencoder. In International conference machine learning, pages 1945-1954. PMLR.
La Cava, W., Helmuth, T., Spector, L., and Moore, J. H. (2019). A probabilistic and multi-objective analysis of lexicase selection and g-lexicase selection. Evolutionary Computation, 27(3):377-444 402.
La Cava, W., Orzechowski, P., Burlacu, B., de Franca, F. O., Virgolin, M., Jin, Y., Kommenda, M., and Moore, J. H. (2021). Contemporary symbolic regression methods and their relative performance. In Thirty-fific fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
La Cava, W., Singh, T. R., Taggart, J., Suri, S., and Moore, J. H. (2018). Learning concise representa- tions for regression by evolving networks of trees. arXiv preprint arXiv:1807.0099:
Lample. G. and Charton, (2019). Deep learning for symbolic mathematics. International Conference on Learning Representations.
Landajuela, M., Petersen, B. K., Kim, S., Santiago, C. P., Glatt, R., Mundhenk, N., Pettit, J. F. and Faissol, D. (2021a). Discovering symbolic policies with deep reinforcement learning. In International Conference on Machine Learning, pages 5979-5989 PMLR.

---

Landajuela, M., Petersen, B. K., Kim, S. K., Santiago, C. P.. Glatt, R., Mundhenk, T. N., Pettit, F., and Faissol, D. M. (2021b). Improving exploration in policy gradient search: Application to symbolic optimization. arXiv preprint arXiv:2107.09158
Langdon, W. B. and Poli, R. (2013). Foundations of genetic programming. Springer Science & Business Media.
Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W. (2019). Set transformer: A framework for attention based permutation-invant neural networks. In International Conference Machine Learning, pages 3744-3753. PMLR.
Luke, and Panait, L. (2001), survey and comparison of tree generation algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2008) pages 81-88. Citeseer.
McConaghy, (2011). Ffx: Fast, scalable, deterministic symbolic regression technology. In Genetic Programming Theory and Practice IX, pages 235-260..20 Springer.
Meurer, A., Smith, C. P., Paprocki, M., Čertík, O., Kirpichev, S. B., Rocklin, M., Kumar, A., Ivanov, S., Moore, J. K., Singh, S., et al. (2017). Sympy: symbolic computing in python. PeerJ Computer Science, 3:c103.
Mundhenk, T., Landajuela. M., Glatt, R., Santiago, C. P., Petersen, B. K., et al. (2021). Symbolic regression via deep reinforcement learning enhanced genetic programming seeding. Advances in Neural Information Processing Systems, 34:24912-24999
Murphy, K. P. (2022). Probabilistic Machine Learning: An introduction. MIT Press.
Olson, R. S., La Cava, W., Orzechowski, P., Urbanowiez, J., and Moore, J. H. (2017). Pmlb: large benchmark suite for machine learning evaluation and comparison. BioData mining, 10(1):1-13.
Pawlak, T. P., Wieloch, B., and Krawiec, K. (2014). Semantic backpropagation for designing search operators in genetic programming. IEEE Transactions on Evolutionary Computation, 19(3):326- 340.
Petersen, B. K., Landajuela, M., Mundhenk, T. N., Santiago, C. P., Kim, S. K., and Kim, J. T. (2021a). Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference Learning Representations.
Petersen, B. K., Santiago, C., and Landajuela, M. (2021b). Incorporating domain knowledge into neural-guided search via in situ priors and constraints. In 8th ICML Workshop Automated Machine Learning (AutoML).
Popova, M., Shvets, M., Oliva, J., and Isayev, O. (2019). Molecularrn: Generating realistic molecular graphs with optimized properties. arXiv preprint arXiv:1905.13372
Reinbold, P. A., Kageorge, L. M., Schatz, M. F., and Grigoriev, R. O. (2021). Robust learning from noisy, incomplete, high-dimensional experimental data via physically constrained symbolic regression. Nature communications, 12(1):1-8.
Sahoo, S., Lampert, C., and Martius, G. (2018). Learning equations for extrapolation and control. In International Conference on Machine Learning, pages 4442 4450. PMLR.
Schmidt, M. and Lipson, H. (2009). Distilling free-form natural laws from experimental data. science, 324((923):81--8)
Schmidt, M. and Lipson, H. (2011). Age-fitness pareto optimization. In Genetic programming theory and practice VIII, pages 129-146. Springer.
Schmidt, M. D. and Lipson, H. (2008). Coevolution of fitness predictors. IEEE Transactions on Evolutionary Computation, 7266)773-7749
Tibshirani, (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288.
12

---

Udrescu, S.-M., Tan, A., Feng, J., Neto. O., Wu, T., and Tegmark, M. (2020). Ai feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. Advances in Neural Information Processing Systems, 33:4860-4877
Vastl, M., Kulhánek, J., Kubalík, J., Demerr E., and Babuška, R. (2022). Symformer: End-to-end symbolic regression using transformer-base architecture. arXiv preprint arXiv:2205.15764
Virgolin, M., Alderliesten, T., and Bosman, P. A. (2019). Linear scaling with and within semantic backpropagation- based genetic programming for symbolic regression. In Proceedings of the genetic and evolutionary computation conference, pages 1084-1092.
Virgolin, M., Alderliesten, T., Witteveen, C., and Bosman, P. A. (2017). Scalable genetic program- ming by gene-pool optimal mixing and input-space entropy-based building- bhilding-block learning. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1041-1048.
Virgolin, M., Alderliesten, T., Witteveen, C., and Bosman, P. A. (2021). Improving model-based genetic programming for symbolic regression of small expressions. Evolutionary computation, 29(2):211237..
Virgolin, M. and Bosman, P. A. (2022). Coefficient mutation in the gene-pood optimal mixing cvolutionary algorithm for symbolic regression. arXiv preprint arXiv:2204.121::
Zhang, H., Zhou, A., Qian, H., and Zhang, H. (2022). Ps-tree: A piecewise symbolic regression tree.
Swarm and Evolutionary Computation, 71:101061.
Zhu, C., Byrd, R. H., Lu, P., and Nocedal, (1997). Algorithm 778: L-bfgs-b: Fortran subroutines
for large-scale cound-constrained optimization. ACM Transactions on mathematical software (TOMS), 2354)5550-560
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Several limitations are discussed in Section d
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to repro- duce the main experimental results (either in the supplemental ma- terial or as a URL)? [Yes] uDSR source code is provided at https://github conm/brendennttersen//dee desp-symbolic-oppmiiiaee
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Hyperparameters used and hyperparameter selection are described in Section andd Appendix F
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] All figures and tables with results include error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Computational resources used are described in Appendix
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Software and dataset assets are described and cited in Appendix J Our benchmarks use the PMLB database, cited in Section
(b) Did you mention the license of the assets? [Yes] Asset licences are described in Appendix J
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "vwvaipFHOaq", "reviewer": "Reviewer_CaGz", "summary": "This paper tackles a symbolic regression (SR). SR is a task to explain observed data using mathematical expressions. The authors state that there are five individual methods and the proposed method combines them. The experimental results on SRBench, a benchmark for SR, demonstrate that the proposed method performs better than the existing methods.", "questions": "The reviewer would like the authors to solve the three weaknesses above.\n", "limitations": "There are some descriptions of technical limitations, not societal ones.\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strengths\n- Combining the individual methods for SR is interesting. The view is also helpful in understanding the current trend of the SR methods.\n- The overall architecture and experimental procedure are well described in the main text and supplementary material.\n- The experimental result shows that the proposed method, uDSR achieves better performance than the existing ones.\n\n#### Weaknesses\n- Since the proposed method is a combination of the existing methods, its technical novelty is limited.\n- There is another method that combines several types of SR methods [a] that is shown to outperform current state-of-the-art methods. Some comparisons to uDSR were necessary.\n- Figure 4 is great in terms of ablation study, but it seems that the combination of all five modules achieves statistically significant improvements over the other variations.\n\n[a] Mundhenk et al. Symbolic Regression via Neural-Guided Genetic Programming Population Seeding. NeurIPS, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ZUPDH0X-fcX", "reviewer": "Reviewer_xSLH", "summary": "This work proposes a new symbolic regression method that unifies 5 existing symbolic regression methods and assesses the approach using SRBench a recently proposed benchmark framework for symbolic regression. It also discusses contribution of each component to the unified framework through  comprehensive ablation studies (48 combinations of the components used in the framework) and provides some recommendations regarding the component choice based on the ablation studies.\n", "questions": "- Use of the inverse function (L223) does not look like a good option in practice, but how can the author assure that it can be generalized and applied to real-world problems? e.g., what if $y - \\sin(x_1) \\leq 0$ in $\\bar{f}^{-1}(y) = \\log(y - \\sin(x_1))$\n- How many of the 252 datasets could the proposed method not finish the inference within the 24 hours walltime?\n- How did this study obtain the results of the 14 baseline methods? If it just refers to the SRBench paper and shows the copy of the results, the authors should clearly describe so in the paper and explain whether or not the noise-injected datasets (Fig. 2) are identical to those in the SRBench paper. If the authors reran the baselines (, which would be preferred), the authors should mention so and explain computing resource used for the baselines.\n- “the highest Pareto ranking,” (L302) should be clarified in the paper.\n- If the authors can provide (pre)training time and inference time for each configuration in Fig. 4, it will be interesting and support some of the statements such as L351-352.\n- “For real-world problems with physical units,” (L354-355) what datasets in the authors’ mind for instance? i.e., If none of the datasets used in this study is categorized into it, how did the authors find the dimensional analysis on the SR problem recommended?\n- Table 2 makes the reviewer think if SINDy is contributing to the framework as its only one property is already covered by AI Feynman.\n\n# Misc\n- Figure 1 should be placed around Section 4, where the figure is first referred.\n- **NSRS** should be replaced with **NeSymReS** to respect the name used in the original study (Biggio et al., 2021)\n- “where F is a nonlinear function subject to certain constraints (see below)” Please specify where it can be found.\n- The repeated use of “thesis” in this paper sounds weird. Maybe it means either “statement”, “claim”, or “argument”?", "limitations": "While the authors state in the checklist that several limitations are discussed in Section 6, the reviewer found only one limitation explicitly mentioned (L374-375). The reviewer did not find any potential negative societal impact of this work discussed in this work.\nAs explained in **Weaknesses** above, the main limitations of this work are the evaluation metrics (not well explained or justified either) and datasets (looks overlapping the ranges of values in generated datasets used for pretraining).\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, this study provides interesting findings of symbolic regression, and the reviewer found that the study may be beneficial for the community while there are some things to be addressed/clarified.\n\n# Strengths\n1. From the motivations described and ablation studies in the paper, the proposed, unified framework for deep symbolic regression seems carefully designed, and it is not like just a combination of random existing methods.\n2. The ablation study to discuss which component of the proposed framework contributes to the improvement seems very comprehensive. The community may also learn from recommendations based on the ablation study.\n3. Although the reviewer has some concerns in the SRBench-based evaluation in this study (see **Weaknesses**), it seems that the proposed framework provides improved performance for datasets used in SRBench, comparing to existing symbolic regression methods.\n\n\n# Weaknesses\n1. Some of the evaluation metrics used in this study (symbolic/accuracy solution rate, $R^2$, accuracy solution) should be clarified and justified in the paper (i.e., the paper should be self-contained). While these metrics are used in SRBench, the reviewer is concerned how essential for SR each of these metrics is.\n2. It seems that NeSymReS (referred to as NSRS in this study) and SINDy are both missed in evaluation as baselines while those are components of the proposed method and the baseline results for the other components (AIF, DSR, and GP) are provided.\n3. The reviewer is afraid that the choices of ranges of variables in the instance generation for pretraining are very heuristic and potentially leaked from the datasets used for evaluation in this study. According to Algorithm 3 in Appendix, the ranges of sampled values for variables seem overlapping those of Feynman datasets used in this study e.g., most of them are within range of 1 to 10 (See FeynmanEquations.csv and BonusEquations.csv available at [the original dataset provider](https://space.mit.edu/home/tegmark/aifeynman.html))", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KM_ku0caz6l", "reviewer": "Reviewer_PqgV", "summary": "This paper provides a holistic review of symbolic regression approaches, and picked 5 most popular ones for algorithm amalgamation to construct a new symbolic regression algorithm. The new symbolic regression algorithm is claimed to maximize the benefits of each and cover each other’s weaknesses. \n", "questions": "\nBelow are some questions and suggestion regarding algorithm description and evaluation.\n\nA concrete example is missing. Though the authors showed example formulas in line 221 and introduced the benchmark, it is important for the symbolic regression paper to introduce at least one example distilled symbolic rule, so as to make it easier for the reader to link what the authors are describing to which parts in the final distilled algorithm. Especially for this hybridizing work, could the author examplify which part in the symbolic rule is generate by which sub-problem? Especially the CINDy part, as how the vector based algorithm connects to symbolic optimization remains unclear to me, and could be made clear if explained via examples.\n\nHowever, it did not show the possible downside of hybridizing different algorithms. Algorithm combination in a chain shaped form is known demonstrate more variance and less bias, so will the combined algorithm underperform the one single algorithm? (\"If certain single algorithm is good enough, why combining with the other worse algorithm won't downgrade it?\") \n\nThough the author explained what the \"on\" and \"off\" mean in line 320, it is still unclear how different is the hybridized algorithm will be from some original algorithm, if all other sub-algorithms are turned off. Based on the results in fig 4, it seems that one single algorithm achieved ~ 0.6, which seems a bit lower than the results reported in the original papers. The ablation study focus on checking the components of the proposed algorithm, while the critical concern regarding this algorithm amalgamation model is what if one choose single sub-algorithm, and how does it compare with the hybridyzed algorithm. It is encouraged that the author evaluate more explicitely on how does the hybridized compared to single ones from the explanation side, and evaluate the result they got compared to the originally reported ones.\n\n\nThe workflow of the proposed hybridyzation is clearly explained both in Algorithm 1 and Fig 1. But the integration of the strength of the NSRS remains unclear to me. It seems that the NSRS has been downgraded to only generate expression with one specific (X,Y) problem. Is the proposed algorithm still possible to generalize across new problems?\n", "limitations": "Based on my limited assessment, this work stays free of possible negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThis paper studies an important problem, symbolic regression. The algorithm does not contain novel parts, but it offer novel way for us to understand the symbolic regression algorith especially what is the core parts and how could we combine them. The writting of this paper is clear enough. This paper provides thorough ablation studies, which showed different combinatorial ablations of uDSR components.\n\nHowever, certain perspectives remains to be improved, including demonstration, and results explanation. I've put the corresponding suggestions and questions in the following, and would raise my assessment if they are properly addressed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept