Symbolic Regression via Deep Reinforcement Learning Enhanced Genetic Programming Seeding
Authors: Terrell N. Mundhenk
Mikel Landajuela
Ruben Glatt
Claudio P. Santiago
Daniel faissol
Brenden K. Petersen
Venue: NeurIPS 2021 Poster (2021)
Keywords: symbolic regression
genetic programming
neural-guided search
reinforcement learning

Abstract: Symbolic regression is the process of identifying mathematical expressions that fit observed output from a black-box process. It is a discrete optimization problem generally believed to be NP-hard. Prior approaches to solving the problem include neural-guided search (e.g. using reinforcement learning) and genetic programming. In this work, we introduce a hybrid neural-guided/genetic programming approach to symbolic regression and other combinatorial optimization problems. We propose a neural-guided component used to seed the starting population of a random restart genetic programming component, gradually learning better starting populations. On a number of common benchmark tasks to recover underlying expressions from a dataset, our method recovers 65% more expressions than a recently published top-performing model using the same experimental setup. We demonstrate that running many genetic programming generations without interdependence on the neural-guided component performs better for symbolic regression than alternative formulations where the two are more strongly coupled. Finally, we introduce a new set of 22 symbolic regression benchmark problems with increased difficulty over existing benchmarks. Source code is provided at www.github.com/brendenpetersen/deep-symbolic-optimization.

Symbolic Regression via Neural-Guided Genetic Programming Population Seeding
Nathan Mundhenk mundhenk1e @lln1.gom
Mikel Landajuela landajuelalai@1 gov
Ruben Glatt glatt10linl.gov
Claudio P. Santiago prataelinl.go gov
Daniel M. Faissol faissol1@1inl. gov
Brenden K. Petersen
bp@lin1.go gov
Computational Engineering Division Lawrence Livermore National Laboratory Livermore, CA 94550
Abstract
Symbolic regression is the process of identifying mathematical expressions that observed output from black-box process. It is discrete optimization problem generally believed to be NP-hard. Prior approaches to solving the problem include neural-guided search (e.g. using reinforcement learning) and genetic program- ming. this work, we introduce a hybrid neural- giirell-uuidd//eeeete programming approach to symbolic regression and other combinatorial optimization problems. We propose neural-guides component used to seed the starting population of a random restart genetic programming component, gradually learning better starting populations. On number of common benchmark tasks to recover underlying expressions from a dataset, our method recovers 65% more expressions than a recently published top-performing model using the same experimental setup. We demonstrate that running many genetic programming generations without interdependence on the neural-guided component performs better for symbolic regression than alternative formulations where the two more strongly coupled. Finally, we introduce new set 22 symbolic regression benchmark problems with increased difficulty over existing benchmarks. Source code is provided at www giw.gitee thub. cwm/ cam//rreendeepeteeeee//eeee aateepp-yymbllic--ppiiiiaaaat
Introduction
Symbolic regression involves searching the space of mathematical expressions to fit a dataset using equations which are potentially easier to interpret than, for example, neural networks. A key difference compared polynomial or neural network- regression is that we seek to illuminate the true underlying process that generated the data. Thus, the process of symbolic regression is analogous to how a physicist may derive set of fundamental expressions to describe natural process. For example, Tycho Brahe meticulously mapped the motion of planets through the sky, but it was Johannes Kepler who later created the expressions for the laws that described their motion. Given a dataset (X,y), v). where each point has inputs X; R" and response yi IR, symbolic regression aims to identify a function T -> that best fits the dataset, where the functional form of f short closed-form mathematical expression.
The space of mathematical expressions is structurally discrete (in functional form) but continuous in parameter space (e.g. floating-point constants). The search space grows exponentially with the length of the expression, rendering symbolic regression challenging machine learning problem. It generally believed to be NP-hard [Lu et 2016); however, no formal proof exists. Given the
35th Conference on Neural Information Processing Systems (NeurIPS 2021))

---

Use starting population
Sample N
Perform S generations
RNN
T
GP
T
t
e
Train (e.g. using VPG, RSPG, PQT)
Extract top M from all generations
T
T
Combine
T
T
T
t
t
Figure 1: Method overview. A parameterized sequence generator (e.g. RNN) generates N samples, ie. expressions for symbolic regression. These samples are used the starting population for GP component. GP then runs S generations. The top M samples from GP are extracted, combined with the N samples from the RNN, and used to train the RNN, e.g. using VPG, RSPG, or PQT. Since GP is stateless, runs in a random restart-like fashion each time the RNN is sampled.
large, combinatorial search space, traditional approaches symbolic regression commonly utilize
evolutionary algorithms, especially genetic programming (GP) Koza 1992 Schmidt and Lipson] 2009 Fortin et al. 2012 Bäck et al. 20188. GP-based symbolic regression operates by maintaining population of mathematical expression "individuals" that are "evolved" using evolutionary operations like selection, crossover, and mutation. A fitness function acts to improve the population over many generations.
Neural networks can also be leveraged for symbolic regression Kusner et al., 2017 Sahoo et al.
2018 Udrescu and Tegmark| [2220]. Recently, it has been proposed to solve symbolic regression using neural-guided search [Petersen et al. 202T Landajuela al. 2021b). This approach works by using a recurrent neural network (RNN) to stochastically emit batches of expressions as a sequence of mathematical operators "tokens." Expressions strategy is used to improve the quality of generated formulas. PPtersen et al.) (2021] propose a risk-seeking policy gradient strategy, which filters out the lesser performers and returns "elite set" of expressions to the RNN each step of training. Constraints can be employed to prune the search space, preventing nonsensical or extraneous statements in generated expressions. For instance, inversions (c.g. log (e") and nested trigonometric functions (e.g. sin + cos(x:)) can be avoided. The performance attained using neural-guided search outperformed GP-based approaches, including commonly used commercial software.
evaluated for goodness of fit and training
Genetic programming and neural- neural-guides guided search are mechanistically dissimilar, yet both have proven to be effective solutions to symbolic regression. Might it be possible to combine the two approaches leverage each of their strengths? The population of individual solutions in GP is structurally the same as a batch of samples emitted by the RNN in neural- -uided search. In principle, they can formally be interfaced by allowing GP individuals to flow to the RNN training and vice versa.
For reinforcement learning-ban based training objectives like the used in Petersen al. 2021 this way of coupling creates an out-of de-disriibution problem on the RNN side. Standard off off-policy methods, like importance sampling [Glynn and Iglehart 1989], do not apply here since the GP distribution is intractable. Note, however, that the mechanistic interpretation of the policy gradient as maximizing the log likelihood of individuals proportional to their fitness is still valid. On the other hand, for training alternatives based on maximum likelihood estimation over a selected subset of samples. like the cross-entropy method [De Boer et al.) 2005) or priority queue training [Abolafia et al. [018)) there assumptions about samples being "on-policy." and thus they can be applied seamlessly. In this work, we explore three different ways of training the RNN: two reinforcement learning-baas training methods (without off- off-policy correction) and the priority queue training method.

---

2
Related Work
Deep learning for symbolic regression. Several recent approaches leverage deep learning for symbolic regression. AI Feynman [Udrescu and Tegmark [020] proposes problem-simplificatin tool for symbolic regression. They use neural networks to identify simplifying properties in a dataset (e.g. multiplicative separability, translational symmetry), which they exploit to recursively define simplified sub-problems that can then be tackled using any symbolic regression algorithm. In Grammar VAE, Kusner et al. (2017) develop a generative model for discrete objects that adhere to pre- pre-specified grammar, then optimize them in latent space. They demonstrate this can be used for symbolic regression; however, the method struggles to exactly recover expressions, and the generated expressions are not always syntactically valid. (Sahoo et al. (2018) develop symbolic regression framework using neural networks whose activation functions are symbolic operators. While this approach enables end-to-end differentiable system, back-propagatie wwwpraaaaaiiaaaaa through activation functions like division or logarithm requires the authors to make several simplifications to the search space, ultimately precluding learning certain simple classes of expressions like vi or sin(r/v).
Genetic programming/polie gradient hybrids. The overall idea of combining GP with gradient- based methods in general predates the deep learning era [Igel and Kreutz
1999
Topchy and Punch
2001 Zhhang and Smart 2004 Montastruc et al.) 2004] Wierstra et al. 2008 and dissatisfaction
with deep reinforcement learning (DRL) is not entirely new [Kurenkov 2018 J. Recently, several
approaches have combined GP and reinforcement learning (RL). In these works, RL used to alter or augment the GP process, e.g. by adjusting the probabilities of performing different genetic operations. Alternatively, GP is used to augment the creation or operation of a neural network [Such et al. 2018
Chang et al.) [2018] Gangwani and Peng 2018 Chen et al.) [019 Stanley et al.) | Real et al. 2019 Mikkulainen et (019 et al.] 2019 Tian al.] 2020] Chen et al.) |2020a bj.
Sample population interchanging methods. We identify a broad class of existing methods that can be characterized by using samples from a sequence generator and interchanging those samples with a GP population. The sequence generator may be any discrete distribution or generative process that creates a sequence of tokens, e.g. a recurrent neural network or transformer. Samples from the sequence generator are then treated as interchangeable with a population generated by GP. Thus, sequence generator samples can be inserted into the population of GP, and GP samples can be used to update the sequence generator. In some cases the sequence generator may not have learnable parameters.
Within this class of methods, Pourchot and Sigaud 2019) and Khadka and Tumer 2018 solve physical continuous control problems like the inverted pendulum moon rover with fungible neural network controller. Their approaches are similar to neuro- eeuro-evolution (Stanley and Miikkulainen 2002 Floreano et al.) 2008 Lüders et al.) [017 Risi and Togelius 2017]. as both techniques are based on using deep deterministic policy gradients (DDPG) [Lillicrap et al. (016) and utilize a synchronous shared pool of actors between GP and RL components. Khadka and Tumer 2018] introduced the ERL method that pools RL and GP actor samples into cyelic replay buffer. This single buffer is drawn to either seed a GP step or help update the actor-critic model. Both GP and RL work synchronously, and the population is interchangeable. Pourchot and Sigaud [2019] introduce CEM-RL which solves the same family of problems in a similar way. The most notable difference is that the shared population is not fed directly into GP, but rather it is used to update a distribution from which a population is drawn.
Most closely related to our work, Ahn et al. 2020 develop genetic expert-guided learning (GEGL). In GEGL, samples are generated using stochastic RNN-based sequence generator and added to maximum reward priority queue (MRPQ). A GP component then applies mutation and/or crossover on each item in the MRPQ. Notably, only one generation of evolutionary operators is applied to each sample in the MRPQ. As a consequence, there is no notion of selection operator. In contrast, demonstrate that the selection operator and performing multiple generations of evolution is crucial to maximize the benefits of GP. After performing one generation of GP, the resulting population is then added to a second MRPQ. Samples from the union of the GP-based MRPQ and RNN-based MRPQ are then used to train the RNN. In contrast, we find that priority queues are not necessary and may prevent sufficient exploration. As with ERL and CEM-RL, GEGL's evolutionary and training steps are one-to-one, meaning that each training step is followed by exactly one GP generation. Additionally, GP pulls from a persistent memory of sample populations, which it shares with the neural network component. This strong coupling makes the evolutionary component much more interdependent on the neural network.
3

---

3 Methods
Our overall algorithm comprises two main components: (1) a sequence generator (with learnable parameters) and (2) genetic programming component. In the sections below, we first discuss pre- liminaries, then describe each component individually, and finally describe how the two components interact.
Preliminaries. Any mathematical expression f can be represented by an algebraic expression tree, where internal nodes are operators (e.g. x, xini and terminal nodes are input variables (e.g. 2) or constants [Petersen et al. 2021]. We refer to r = [TI...., T/r as the pre-order traversal of such correspondence between
an expression tree. Notably, there is one-to-on its pre-order traversal (see Petersen et al. [021] for details). Each T; is an operator, input variable, or constant selected from a library of possible tokens, e.g. [+. x, +, sin, cos, exp, log, x/. A pre- order traversal 7 can be instantiated into a corresponding mathematical expression f and evaluated based on its fitness to a dataset. Specifically, we consider the metric normalized root-mean-square error (NRMSE), defined as follows. Given pre-order traversal 7 and a dataset of (X,y) pairs of
expression tree and
size N, with X E IR. and y E R. we define NRMSE(T) = + Ei= (y, - (x,))² where f: R" + R is the instantiated mathematical expression represented by т, and og is the standard deviation of yy Finally, the fitness or reward function is given by R(T) 1/(1 + (RMSEE(T)) We use the terms "fitness" and "reward" interchangeably, with the former being more typical in the context of GP and the latter more common in the context of RL.
Sequence generator. The sequence generator is parameterized distribution over mathematical expressions, p() Typically, a model is chosen such that the likelihood of an expression is tractable with respect to parameters 0, allowing backpropagation of differentiable loss function. A common choice of model is an autoregressive RNN, in which the likelihood of the ith token (denoted T;) is conditionally independent given the preceding tokens Ti..... ((--1). That is, (Ttaa 0)
p( 0). We follow the sequence generator used in Petersen et al. [2021]: an autoregressive RNN comprising single-layer LSTM with 32 hidden nodes. For notational simplicity, hereafter assume the use of an RNN as the sequence generator.
The sequence generator is typically trained using reinforcement learning or related approach. Under this perspective, the sequence generator can be viewed as a reinforcement learning policy, which we seek to optimize by sampling a batch of N expressions T, evaluating each expression under a reward function R(T). and performing gradient descent loss function. In this work, explore three methods for training the RNN:
Vanilla policy gradient (VPG): Using the well-known REINFORCE rule (Williams 1992], training is performed over the batch T, yielding the loss function: C() ETET(((TT) V log p(r)0), where is a baseline term or control variate, e.g. an exponentially-weighte moving average (EWMA) of rewards.
Risk-seeking policy gradient (RSPG): Petersen et al. [222]] develop an alternative to VPG intended to optimize for best-case instead of average reward: L(0) = T Σrer(R(T)
R log (() R(F)>R, where s is a hyperparameter controlling the degree of risk- seeking and R, is the empirical (1 -)) quantile of the rewards of T.
Priority queue training (PQT): Abolafia et al. [2018) introduce a non-RL approach also intended to focus on optimizing best-case performance. Samples from each batch are added to a persistent maximum reward priority queue (MRPQ), and training is performed over sam- ples in the MRPQ using a supervised learning objective: [(()= = x 2 TEMPPO Ve log P(T(0), where & is the size of the MRPQ
Note that our method agnostic to the training procedure; additional procedures may be considered, for example the cross-entropy method (De Boer et al.) 2005), which is closely related to PQT. In our formulation, include, as is common, an additional term in the loss function proportional to the entropy of the distribution at each position along the sequence [Bello et al.) 2016 Abolafia et al. 2018 Petersen et al. 2021 Landajuela et al. 2021b].
Genetic programming. Genetic programming (GP) begins with set ("population") of expression trees ("individuals"), denoted TGP. In standard GP. these individuals are randomly generated; how- ever, discuss in the subsequent section how this differs in our algorithm. A single iteration or

---

Algorithm Neural-guided genetic programming population seeding
input batch/population size N; number of GP generations per RNN training update S; constraints
Ωi crossover probability Pe; mutation probability Pm: tournament size k; GP sample selection
size M (must be I .N)
input Loss function L()) for training the RNN, including corresponding hyperparameters (e-g.
EWMA coefficient for VPG, risk factor for RSPG, priority queue size for PQT)
output Best sample T'
1: Initialize reward function T R
2: Initialize RNN distribution over expressions, p(-)0.9)
3: Initialize GP operation, GP(Pe, Pm, k, R) = : T
4: T" & null
5: while total samples below budget do
6: TRNN (r(1) p(-10, [
Sample batch of size N b Seed GP starting population with RNN batch
7:
TGP t- TRNN
8:
for 1...., S do
T(P + (TaP
9:
D Apply GP operations
Tuain f Top-M T (0)
UTT U... U... T(T
10:
D. Filter best M samples from GP
11:
TrAin f- Tuain U TRRN
P Join RNN and best GP samples D Compute rewards Train the RNN (e.g. using PQT) Update best sample
12:
R (R(T) V E TTimin
0 Vo((O)
if' max R > R((*)) then T* + (arg R)
15:
return
"generation" of a GP algorithm consists of several "evolutionary operations" that directly alter the current population. A mutation operator introduces random variations an individual, for example by replacing one subtree with another randomly generated subtree. A crossover operator involves exchanging content between two individuals, for example by swapping one random subtree of one individual with another random subtree of another individual. A selection operator is used to select which individuals from the current population persist onto the next population. A common selection operator is tournament selection [Koza 1992], in which set of candidate individuals are randomly sampled from the population, and the individual with the highest fitness is selected. In each generation of GP, each individual has a probability of undergoing mutation and probability of undergoing crossover; selection is performed until the new generation's 's population has the same size as the current generation's 's population. GP does not have an explicit objective function does the sequence generator, but it does tend to move the population toward higher fitness [Such et al., 2018).
For the GP component, we begin with a standard formulation from DEAP (Fortin et al. 2012 and introduce several key changes:
1. Typically only uniform mutation is used. Instead, we select among uniform, node replace- ment, insertion, or shrink mutation with equal probability.
2. We incorporate constraints from PPtersen et al. [2021] (for example, constraining nested trigonometric functions, e.g. sin[1 + (x))). If genetic operation would result in an individual that violates any constraint, that genetic operation instead reverted. That is, the child expression instead becomes a copy of the parent expression. This procedure ensures that all individuals satisfy all constraints for all generations.
3. The sample population is never initialized randomly. Rather, the initial population is always seeded by samples from the RNN. (See next section for details.)
Neural-guided genetic programming population seeding, While both fully baseb ap- proach (i.e. deep symbolic regression [Petersen al. (02T)) and fully Ga-based approach (i.e. GP-based symbolic regression Koza 1992]) vnvolve generating "batches" or "populations" of expres- sions, they arrive at their expressions very differently. Namely, we observe that the RNN can generate expressions "from scratch" given only parameters #: in contrast, GP requires an extant population to operate on. More specifically, given parameters 0, the RNN can be used to sample a batch of expressions, TRNN; contrast, generation of GP begins with a population of expressions T(() and application of one generation of GP produces new, augmented population 7(6+1)
5

---

Table I: Recovery rate of several algorithms the Nguyen benchmark problem across 100 independent training runs, Results our algorithm are obtained using PQT; slightly lower recovery rates were obtained using VPG and RSPG training (see Table 3 for comparisons).
Recovery rate (%)
Benchmark
Expression
Ours
DSR
PQT
VPG
GP
Eureqa
Nguyen-1
x + x² + x
100
100
100
96
100
100
Nguyen-2
+ xd + x tx
100
100
99
47
97
100
Nguyen-3
5 + x + x.3 10+0 +xx2 +x
100
100
86
4
100
95
Nguyen-4
2 + xd + x² +x² x²
100
100
93
1
100
70
N
sin(x²) cos(x) [r)
100
72
73
5
45
73
6
sin(x) + sin( +x²))
100
100
98
100
91
100
Nguyen-7
log(2 + 1) + log(2" +1
97
35
41
3
0
85
8
100
96
21
5
5
0
9
sin(x) + (
100
100
100
100
100
100
Nguyen-10 10
( cos
100
100
91
99
76
64
Nguyen-11
x
100
100
100
100
7
100
Nguyen-12
x² - r + 2
0
0
0
0
0
0
Average
91.4
83.6
75.2
46.7
60.1
73.9
Thus, we propose that natural point at which to interface neural-guided search and GP the starting population of GP, TT(o) Specifically, we propose to use the most recent batch of samples from the RNN directly as the starting population for GP: T(P) TRNN. From there, we can perform generations of GP, resulting in final GP population, T Finally, set" of the top-performing GP samples, and include these samples in the gradient update for neural guided search (e.g. VPG, RSPG, PQT). This process constitutes one step of our algorithm, and is repeated until a maximum number of total expression evaluations reached. Thus, GP acts as an inner optimization loop, within an outer gradient-based loop of neural- soural-guided search.
sub-select "elite
Note that the GP process is restarted for each new batch of samples from the RNN; thus, the process similar to GP with random restarts; the key difference for our proposed algorithm is that the GP starting population upon each restart changes the RNN learns via an objective function. Thus, from the perspective of GP, the RNN provides increasingly better starting populations. Empirically, we found that this hybrid approach also allows for larger learning rates relative to pure neural-guided search.
We hypothesize that this hybrid approach of neural-guided search and GP will leverage the strengths of each individual approach. Whereas GP is stateless and there is no learning step, neural-guided search is stateful (via RNN parameters 0) and learns from data via well-defined, differentiable loss function. Whereas neural- heural-guided search is known to easily get stuck in local optima, GP can produce large variations in the population, resulting in "fresh" samples that are essentially -iitt---ditiiittinnn from RNN-induced distribution. We discuss this further in Discussion.
We provide pseudocode for our algorithm in Algorithm [ and illustrate it in Figure focuses the task of symbolic regression, our approach applies to any symbolic optimization task with black-box reward function.
| While this work
Results
We used two popular benchmark problem sets to compare our technique to other methods: Nguyen [Uy al. 2014] and the R rationals [Krawiec and Pawlak 2013). A symbolic regression benchmark problem is defined by a ground truth expression, set sampled points from the expression, and set of allowable tokens. Additionally, we introduce new benchmark problem set with this work, which we call Livermore. The impetus for introducing a new benchmark problem was that our algorithm achieves nearly perfect scores on Nguyen, so we designed a benchmark problem set with a large include variants Nguyen- R-1* R-2* and R-3* which
range of problem difficulty. Finally, use the same expression and set of tokens, but increase the domain over which sampled points are taken. All benchmarks are described in Appendix Table 9 Hyperparameters are shown in Appendix

---

Table 2: Recovery rate of several algorithms on the Nguyen, R, and Livermore benchmark problem sets across 25 independent training runs. 95% confidence intervals are obtained from the standard error between mean recovery on 37 unique benchmark problems. Recovery rates on individual benchmark problems are shown Appendix Table
Recovery rate (%)
All
Nguyen
R
Livermore
Ours
74.92
92.33
33.33
71.09
GEGL [Ahn et al. 2020
64.11
86.00
33.33
56.36
Random Restart GP (L.c. GP only)
63.57
88.67
2.67
58.18
DSR (i.e. RNN only) [Petersen et al. 2021
45.19
83.58
0.00
30.41
95% confidence interval
11.5
11.7
+2.81
1..2
Table 11 For all algorithms, we tuned hyperparameters using Nguyen-7 and R-3*. None of the Livermore benchmarks were used for hyperparameter tuning.
We follow the experimental procedure of Petersen et al. [20T]] unless otherwise noted. For all benchmark problems, we run each algorithm multiple times using different random number seed, Experiments were conducted on 36 core, 2.1 GHz, Intel Xeon E5-2695 workstations. We run each benchmark single core, which take average of 4.4 minutes per run the Nguyen benchmarks. Runtimes on individual Nguyen benchmarks are shown in Appendix Table 6
Our primary empirical performance metric is "recovery rate," defined as the fraction of independent training runs in which an algorithm discovers expression that is symbolically equivalent to the ground truth expression within a maximum of 2 million candidate expressions. The ground truth expression is used to determine whether the best found candidate was correctly recovered. Table T shows recovery rates on each of the N Yguyen benchmark problems compared with DSR, PQT, VPG, GP, and Eureqa [Schmidt and Lipson 2009]. We note that DSR stands as the prior top performer on this set [Petersen et 2021). As we see, the recovery rate for our algorithm is 9.3% higher than the previous leader, DSR.
We next compare against a recent method called GEGL [Ahn et al. 2020] using the Nguyen, R, and I. Livermore benchmark problem sets. GEGL is hybrid RNN/GP algorithm originally demonstrated for designing small molecules that fit set of desired parameters. Many details of the open-source implementation are tied to the particular problem of molecular design (e.g. the mutation operators are specific to molecules), so we adapted their method to symbolic regression (i.e. using the same genetic operators as our method). In addition to GEGL, we compare against a "GP only" and "RNN only" version of our method, which are the most critical ablations. "GP only" does not use an RNN, and each GP population is seeded using randomly generated expressions; this is essentially GP with random restarts. "RNN only" does not use GP, essentially reducing DSR [Petersen et al. 2021) For these experiments, we use the variations Nguyen- 12*, R-1", R-2", and R-3". Table 2 shows that our method ties with GEGL on the R benchmark set and outperforms the other three methods overall. also outperforms all other approaches on the other benchmark sets. is interesting to note how well random restart GP does on its own, outperforming DSR and on par with GEGL. Results for individual benchmarks are shown in Appendix T
Results on additional benchmark problem sets are found Appendix Tables 7 and g Finally, in Appendix Table 13 we see an additional improvement in recovery rates by repeating our experiments for the Nguyen, R, and Livermore benchmarks using the soft length prior and hierarchical entropy regularizer from Landajuela et al. 2021a which were developed concurrently with this work.
5 Discussion
Intuition. We provide two-sided perspective as why we believe our hybrid method outperforms its constituent components so strongly. First, from the RNN perspective, we believe GP helps by providing "fresh" new samples that help escape local optima. NN-based discrete distributions often concentrate their probability mass on a relatively small portion of the search space, resulting in premature convergence to locally optimal solutions. In contrast, the evolutionary operations of GP generate new individuals that may fall well outside the RNN's concentrated region of the search space. Second, from the perspective of GP, the RNN helps by learning good starting populations. With a good starting set, GP can easily identify excellent solutions within few generations. Further,
7

---

80
Recovery rate (%)
40
0
1
15 25
50
GP generations per RNN training step
Figure 2: Recovery rate of method from 25 independent runs the Nguyen, R, and Livermore benchmark problem sets as a function number of GP generations per RNN training step. Error bars represent the 95% confidence interval. The generations reduces to DSR.
Table Recovery rates for various ablations of our algorithm (sorted by overall performance) from 25 independent runs on the Nguyen, R, and Livermore benchmark problem sets.
Recovery rate (%)
All
Nguyen
R
Livermore
Trainer PQT
74.92
92.33
33.33
71.09
No RL samples to train RNN (all off-policy)
74.81
93.00
29.33
71.09
Trainer VPG
74.27
93.67
28.00
70.00
Entropy weight=
73.95
92.00
30.67
70.00
Trainer RSPG
73.95
92.67
39.33
69.82
(A): Uniform mutation only
72.65
91.67
25.33
68.73
(B): No inv/trig constraint
71.35
91.33
32.00
65.82
(A) and (B)
68.11
91.33
21.33
61.82
No GP samples to train RNN (all on-policy)
66.27
89.00
24.00
59.64
(C): Trainer learning rate =
65.95
90.67
17.33
59.09
Random Restart (GP only)
63.57
88.67
2.67
58.18
(D): No RI. seeds to GP
63.24
89.00
2.67
$7.45
(C) and (D)
63.14
88.67
2.67
57.45
DSR (RL only) [Petersen et al. 2021
45.19
83.58
0.00
30.41
95% Confidence Interval
11,54
#1.76
+2281
+1.32
the RNN may act to keep GP from moving too quickly in a suboptimal direction and may have effect analogous to using a trust region. Each time GP restarts, it falls back to the current probability mass of the training RNN.
GP generations per RNN training step. An important hyperparameter in method is S, the number of GP generations to perform per RNN training step. We hypothesized that S proportional to expression length. That is, the larger expression is, the more evolutionary operations should be required to transform any arbitrary seed individual into the target expression. Our initial estimate was that 25 GP steps would be needed per each RNN training step. Figure shows post-hoc analysis of how performance varies depending on how many GP steps do between each RNN training step. The optimal number of steps between implementation is 30 tokens. However, each GP step can make many changes at time, number needed GP steps to be less than maximum possible hamming distance. As such, we can see the similarity between the optimal number of GP steps and the length of expressions.
and 25. We note that the hard limit set by our
expect
Ablations. We several post-hoc ablations determine the contribution of various design choices to our method's performance. Table [ note that the choice of RNN training procedure (i.e. VPG, RSPG, or PQT) does not make large difference; while PQT outperforms VPG and RSPG, the difference falls within the 95% confidence interval. Interestingly, removing the entropy regularizer by setting the weight to zero made a small difference in performance. Other works have found the entropy regularizer to be extremely important [Abolafia et al.] 20018) Petersen et al. 2002 Landajucla et al. 2021aa. We hypothesize that entropy less important in hybrid approach because the GP-produced samples provide the RNN with
recovery rate for several ablations. We first

---

Table 4: Minimum normalized mean-square error (NMSE) from 25 independent runs on three
challenging benchmark problems.
Minimum NMSE
Ours
DSR
Nguyen-12
1.41 x 10-4
2.01 x 10-2
R-1
3.55
10-5
2.74 x 10-3
R-2
2.83
10-4
5.31 x 10-3
sufficient exploration. Using different types of mutation in the GP component with equal probability slightly improves performance: we hypothesize that this increases sample diversity. Including the constraints proposed by Petersen et al. 2021] also improves performance.
We performed several ablations designed to determine the relative importance of the RNN versus GP components. Completely removing the RNN leads to standalone GP algorithm running with random restarts. This actually performs quite well compared to GP without random restarts. This is not surprising since random- restarts has been shown to help GP by others (Houck et al.] 1996 Ghannadian et al. 1996, Fukunagal 2010). Similarly, we can set the RNN learning rate zero. This ablation essentially reduces to a version of GP with random restarts, in which the starting population based randomly initialized RNN; this performs similarly GP with random restarts.
We also explore a fully "off-policy" ablation, in which only GP samples are considered in the training step (RNN samples are excluded). Notably, this does not significantly affect performance, as the GP samples (having undergone many generations of GP refinement) RNN samples. Similarly, we explore a fully "on-policy" version, which only RNN samples are considered in the training step (GP samples
generally superior to the
excluded). In this case, there is no feedback between GP and the RNN; the only role of GP is to provide strong candidates that may be selected as the best final expression. This ablation results in large drop in performance, suggesting that it is important that the RNN be trained on samples from the GP.
Finally, in Appendix Table 12 we show additional results when various hyperparameters from Petersen et al. [2021] or Fortin et al. 2012 are restored to their original values.
The primary limiting factor to approach is that there are still some expressions which cannot be completely recovered. While it may be a matter of finding the right set of hyperparameters, we have not as yet recovered benchmarks Nguyen- 12, R-1, or R-2. While we have not yet solved three benchmarks, Table shows that we still significantly outperform DSR in terms of NMSE. Other challenging benchmarks include R-3, which we recover about 1 in 25 runs, and Livermore-7 and I Livermore-8, which we recover about once in every few hundred runs.
Linitations.
6 Conclusion and Future Work
We introduce a hybrid approach to solve symbolic regression and other symbolic optimization problems using coural-guided search to generate starting populations for generic programming component running in a random restart-like fashion. The results are state-of-the-ar on a series of benchmark symbolic regression tasks.
Our future research will focus on methods to mitigate or resolve the off-policy issue of policy gradient-based training methods, by either correcting the weights on the gradient computation or considering alternative optimization objectives. We will also further examine why Nguyen-12, R-1, and R-2 seem intractable and see if we can come solution. We also plan doing in-depth analysis of our algorithm with noisy data.
Acknowledgements
We thank the L.NLL Hypothesis Testing team, Kate Svyatets, and Miles the cat for their helpful comments, feedback. and support. This work was performed under the auspices of the U.S. Depart- ment of Energy by Lawrence Livermore National Laboratory under contract DIS-AC52-07NA227344 Lawrence I. Livermore National Security, LLC. LLNI-CONNF-8200 and was supported by the L.NII- LDRD Program under Projects 19-DR-003 and 21-S1-001. The authors have no competing interests to report.

---

References
Daniel A Abolatia, Mohammad Norouzi, Jonathan Shen, Rui Zhao, and Quoc V Le. Neural program synthesis with priority queue training, arXiv preprint orXiv:/80.0033 2018.
Sungsoo Ahn, Junsu Kim, Hankook Lee, and Jinwoo Shin. Guiding deep molecular optimization with genetic exploration. In Proceeding of the Conference on Neural Information Processing Systems (NeurIPS), 2020.
Thomas Bäck, David B Fogel, and Zbigniew Michalewicz. Evolutionary Computation 1: Busic Algorithms and Operators. CRC press, 2018.
Irwan Bello, Hieu Pham, Quoc V Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. arXiv preprint 1rX99:161..0994, 2016.
Simyung Chang, John Yang. Jaeseok Choi, and Nojun Kwak. Genetic-gated networks for deep reinforcement learning. In Proceeding of the Conference on Neural Information Processing
Systems (NeurIPS), 2018.
Diqi Chen, Yizhou Wang, and Wen Gao. Combining gradient-based method and an evolution strategy for multi-objective reinforcement learning. Applied Intelligence, 50, 2020a.
Qiong Chen, Mengxing Huang, Qiannan Xu, Hao Wang, and Jinghui Wang. Reinforcement learning-
based genetic algorithm in optimizing multidimensional data discretization scheme. Mathematical Problems in Engineering, 2020b.
Yukang Chen, Gaofeng Meng, Qian Zhang, Shiming Xiang, Chang Huang, Lisen Mu, and Xinggang Wang. Renas: Reinforced evolutionary neural architecture search. Proceeding of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Pieter-Tjert Tjerk De Boer, Dirk P Kroese, Shie Mannor, and Reuven Y Rubinstein. A tutorial on the cross-entropy method. Annals of operations research, 134(1):19-67, 2005.
Dario Floreano, Peter Dürr, and Claudio Mattiussi. Neurocvolution: from architectures to learning. Evolutionary Intelligence, 1:47-62, 2008
Félix-Antoine Fortin, François-Micht De Rainville, Marc-André Gardner, Mare Parizeau, and Christian Gagné. DEAP: Evolutionary algorithms made easy. Journal of Machine Learning Research, 13:2171-2175, jul 2012.
Alex S. Fukunaga. Restart scheduling for genetic algorithms. In Proceeding of the 5th International Conference on Parallel Problem Solving from Nature, 2010.
Tanmay Gangwani and Jian Peng, Policy optimization by genetic distillation. In Proceeding of the International Conference on Learning Representations (ICLR), 2018
Farzad Ghannadian, Cecil Alford, and Ron Shonkwiler. Application of random restart to genetic algorithms. Intelligent Systems, 95:81-102, 1996.
Peter W Glynn and Donald Iglehart. Importance sampling for stochastic simulations. Management science, 35(1 1): 367-1392, 1989.
Christopher R. Houck, Jeffrey A. Joines, and Michael G. Kay. Comparison of genetic algorithms, random restart, and two-opt switching for solving large location-alloeeee problems. Computers and Operations Research, 23:587-596, 1996.
Christian Igel and Martin Kreutz. Using fitness distributions to improve the evolution of learning structures. Proceedings of the 1999 Congress on Evolutionary Computation- CECC9 (Cat. No. 99T118406), volume 3, pages 1902-1909. IEEE, 1999.
Ying Jin, Weilin Fu, Jian Kang, Jiadong Guo, and Jian Guo. Bayesian symbolic regression. arXiv preprint arxiv: 19.000892.000 2019.
Shauharda Khadka and Kagan Tumer. Irolution-guides policy gradient in reinforcement learning. In Proceeding of the Conference on Neural Information Processing Systems (NeurIPS), 2018.
10

---

John R Koza. Genetic Programming: On the Programming of Computers by Means of Natural Selection, volume MIT press, 1992.
Krzysztof Krawiec and Tomasz Pawlak. Approximating geometric crossover by semantic backpropa- gation. In Proceeding of the fifteenth annual conference on Genetic and evolutionary computation (GECCO), 2013.
Andrey Kurenkov. Reinforcement learning's foundational flaw. The Gradient, 2018
Matt J Kusner, Brooks Paige, and José Miguel Hernández-Lobato. Grammar variational autoencoder. In Proceedings of the 34th Proceeding of the International Conference on Machine Learning (ICML)-Volume) 70, pages 1954. JMLR. org, 2017.
Mikel Landajuela, Brenden K Petersen, Soo K Kim, Claudio P Santiago, Ruben Glatt, T Nathan Mundhenk, Jacob F Pettit, and Daniel M Faissol. Improving exploration in policy gradient search: Application to symbolic optimization. arXiv preprint arXiv:2107.09158, 2021a.
Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep rein- forcement learning. In International Conference on Machine Learning, pages 3979-5989 PMLR, 2021b.
Timothy Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Proceeding of the International Conference on Learning Representations (ICLR), 2016.
Qiang Lu, Jun Ren, and Zhiguang Wang. Using genetic programming with prior formula knowledge to solve symbolic regression problem. Computational intelligence and neuroscience, 2016.
Benno Lüders, Mikkel Schläger, Aleksandra Korach, and Sebastian Risi. Continual and one-shot learning through neural networks with dynamic external memory. In In European Conference the Applications of Evolutionary Computation, pages 886-901, 2017.
Risto Müikkulannenl, Jason Liang, Elliot Meyerson, Aditya Rawal, Dan Fink, OlivierFrancon, Bala Raju, Hormoz Shahrzad, Arshak Navruzyan, Nigel Duffy, and Babak Hodjat. Evolving deep neural networks. Artificial Intelligence in the Age of Neural Networks and Brain Computing, 2019.
Ludovic Montastrue, Catherine Azzaro-Pantel, Luc Pibouleau, and Serge Domenech. Use of ge- netic algorithms and gradient based optimization techniques for calcium phosphate precipitation. Chemical Engineering and Processing: Process Intensification, 43810): 2004. ISSN 0255-2701. doi: /ttp://rccc///////552/0000000000 Com/scienae
12.002. URL http:::://www..cieeeee.s wcw.ecediieeccttt
Brenden K Petersen, Mikel Landajuela, T Nathan Mundhenk, Claudio P Santiago, Soo K Kim, and Joanne T Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk- seeking policy gradients. Proceeding of the International Conference Learning Representations (ICLR), 2021.
Aloïs Pourchot and Olivier Sigaud. Cem-rl: Combining evolutionary and gradient-based methods for policy search. In Proceeding of the International Conference on Learning Representations (ICLR), 2019.
Esteban Real, Chen Liang, David R. So, and Quoc V. Leg. Automl-zero: Evolving machine learning algorithms from scratch, In Proceeding of the International Conference on Machine Learning (ICML), 2019.
Sebastian Risi and Julian Togelius. Neuroevolution in games: State of the art and open challenges. IEEE Transactions on Computational Intelligence and Al in Games, 9:25 2017.
Subham S Sahoo, Christoph H Lampert, and Georg Martius. Learning equations for extrapolation and control. In Proceedings of the 35th International Conference Machine Learning (ICML), 2018.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data, science, 324(5923):81-88 2009.

---

Adarsh Sehgal, Hung La, Sushil Louis, and Hai Nguyen. Deep reinforcement learning using genetic algorithm for parameter optimization. In Proceeding of the IEEE International Conference on Robolic Computing (ICRC), 2019.
Kenneth O. Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topolo- gies. Evolutionary computation,, (0:99-127, 2002.
Kenneth O. Stanley, Jeff Clune, Joel Lehmanl, and Risto Mikkullinen. Designing neural networks through neuroevolution. Nature Machine Intelligence, 2019.
Felipe Petroski Such, Vashisht Madhavan, Edoardo Conti, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Deep neuroevolution: Genetic algorithms are competitive alternative for training deep neural networks for reinforcement learning. In www.ii::12..006000
2018.
Yuan Tian, Qin Wangl, Zhiwu Huang, Wen Li, Dengxin Dai, Minghao Yang. Jun Wang. and Olga Fink. Off-policy reinforcement learning for efficient and effective gan architecture search. In Proceeding of the European Computer Vision Conference (ECCV), 2020.
Alexander Topchy and William Punch. Faster genetic programming based local gradient search of numeric leaf values. In Proceeding of the third anmual conference on Genetic and evolutionary computation (GECCO), 2001.
Leonardo Trujillo. Luis Muñoz, Edgar Galván-López, and Sara Silva. neat genetic programming: Controlling bloat naturally. Information Sciences, 333:21-43,, 2016.
Silviu-Marian Udrescu and Max Tegmark. Ai feynman: physics-inspired method for symbolic regression. Science Advances, 6(16), 2020.
Nguyen Quang Uy, Nguyen Xuan Hoai, Michael O'Neill, R.I. McKay, and Edgar Galvan- Lopez. Cemantically-based crossover genetic programming: Application realvalued symbolic regres- sion. Genetic Programming and Evolvable Machines, 12:91-119, 2014.
Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Natural evolution strategies. In Proceeding the IEEE Congress Evolutionary Computation, 2008.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):22--256, 1992.
Mengjie Zhang and Will Smart. Genetic programming with gradient descent search for multiclass object classification. In Maarten Keijzer, Una-May O' d'Reilly, Simon Lucas, Ernesto Costa, and Terence Soule, editors, Genetic Programmingg pages 399-408, Berlin, Heidelberg, 2004. Springer Berlin Heidelberg, ISBN 978-3-540-24650-3.
12


Reviews:
[{"review_id": "zkfyPQnftnp", "reviewer": "Reviewer_DKkf", "summary": "The manuscript proposes a method to solve symbolic regression problems.  Recurrent  neural networks (RNNs)  are trained using policy-gradient reinforcement learning to find initial populations for genetic programming (GP).\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The overall approach is mainly a combination of existing methods. Thus, I regard the originality as limited.\nI did not find obvious flaws or misleading statements in the manuscript. \nIt is interesting that GP alone performs so well (table 2).\n\nSome part were not clear to me:\nlines 193-195: Are the individuals which violate the constraints removed (i.e., get lethal fitness) or repaired (e.g., ln(e^x) is simply replaced by x)? \n\nlines 54-59:  The authors could elaborate what they exactly refer to with \"out-of-distribution\", \"off-policy\", and \"on-policy\" in this context. What are the two policies? Would the authors regard the RNN to be on-policy and, if yes, why is this a problem?\n\nlines 231-234: I am not fully sure that I understand the fundamental difference between PQT and RSPG: PQT takes the best from T_rl and T_gp combines while RSPG only removes the worst solutions from T_rl?\n\nline 303: The entropy term comes out of the blue - where is it used?\n\nLiterature embedding:\n\nlines 87-89: the idea of combining GP and gradient-based optimization of the numerical values goes back to:\nC. Igel. M. Kreutz. Using Fitness Distributions to Improve the Evolution of Learning Structures. Congress on Evolutionary Computation (CEC 99), pp. 1902-1909, IEEE Press, 1999 [see section 5]\n\n\nMinor comments: \n\nline 27: \", is\" -> \"is\"\n\nline 48: \"the problem\" -> \"the general problem\" ?\n\nline 419: first name abbreviations\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In my evaluation, no additional discussion of potential negative societal impact of the presented work is necessary.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "nH68YMPzsHJ", "reviewer": "Reviewer_ELHf", "summary": "This paper proposes to combine the policy gradient method with the genetic algorithm method to solve the symbolic regression problem. Basically, an RNN trained using policy gradient computes candidate equations, and these equations are served as \"seeds\" for the genetic algorithm. The output of the genetic algorithm is used to train the RNN through policy gradient. Then this cycle can forward. The authors test their method on various datasets and achieve the best performance among all the baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:  This work is somehow a novel combination of \"Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients Petersen et al. 2021\" and \" Guiding deep molecular optimization with genetic exploration Ahn et al. 2020\". The related work section is sound.\n\nQuality: The idea of combining reinforcement learning (RL) and genetic algorithm is quite simple. Use the results computed by the RL policy to guide the genetic algorithm, and the improved results from the genetic algorithm can be used to improve the RL policy. Also, several methods to improve this cycle are studied and explained.  In the merit of simple, the performance looks good.\n\nClarity: It would be better if the authors can improve the writing. It is a little bit hard for me to follow the paper. Too many details are mentioned in the main text and hard to capture the high-level idea. Also, for the experiment section, it would be better if the author can mark your results at least in bold.\n\nSignificance: The program this paper study is significant.\n\nQuestions:\n\n1. What's the difference between DSR in Tabel1 and DSR (RL only) in Tabel 2? If there are differences, why use two different baselines? And why not run all the baselines on all the datasets?\n\n2. In terms of the fairness of comparison. I think you run 100 times for each method seems not fair. It would be better if you set a time budget and let all the methods run under the budget.\n\n3. It seems that Nguyen-10 and 11 are more complicated than 12. But all the methods fail on 12. Do you have some intuitions?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A.", "review_text": "", "extra_scores": {"ethical_concerns": "NO", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "gZZkYj_CByW", "reviewer": "Reviewer_wnag", "summary": "This paper proposes a hybrid method of policy gradient-based reinforcement learning and genetic programming for symbolic regression. The solutions provided by the policy-gradient-based deep reinforcement learner (DRL) are used as the population in genetic programming. The experimental result using benchmark functions for symbolic regression shows that the proposed method is superior to the existing symbolic regression methods in most cases.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The hybrid method of a policy gradient method and genetic programming (GP) is interesting. However, the combination method proposed in this paper seems to be simple. In the proposed method, the sampled solutions from a deep neural network are used as the initial population of GP iterations. In addition, the generated solutions by GP are injected into the samples used for updating the neural network weights via policy gradient.\n\nThe samples generated by GP iterations are used for updating neural network weights by policy gradient update. The policy gradient formula shown in section 3 is the Monte-Carlo approximation of the true gradient under the current policy. If we inject the samples generated by GP to calculate the policy gradient, the estimation of the policy gradient is biased. It should be carefully noted the justification of the proposed method.\n\nThe reviewer cannot understand the reason for the performance improvement by the proposed method. How does genetic programming contribute to improving the recovery rate? The authors should elaborate on the reason for the performance improvement.\n\nIn [Petersen et al., 2021], other benchmark symbolic regression problems are also examined, such as Jin-X and Neat-X. How about the performance of the proposed method on these benchmarks?\n\nDoes \"GP-Meld\" mean \"DREGS\" in Table 4?\n\n\n-----\n[Update after rebuttal]\nThe authors reply to my request properly. I agree with the clear performance improvement by the proposed method as the strength of this paper. SO, I would raise my score. However, the novelty is limited because the proposed method is a simple heuristic combination of the existing methods.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The reviewer thinks that the limitation is fairly stated.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "bRLJxvejZ-0", "reviewer": "Reviewer_Ug8S", "summary": "The paper describes a method to enhance the performance of Genetic Programming (GP) for symbolic regression by iteratively enhancing initial populations for 'random restart' GP implementations. The method works through a generative recurrent network (RNN) and an Reinforcement Learning (RL) approach, in which the RNN gets rewarded when outputs a set of initial math expressions that help the GP to achieve better solutions after each run. The GP and the RL algorithm work cooperatively in an iterative fashion where for every GP run (25 cycles) the RL rewards the RNN. Authors provide an extensive battery of tests in several benchmarking datasets, as well as proposing their own set of problems. Their method achieves better results than all other methods they compare against, including a vanilla GP.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and structured. Language is clear at all times; math is not overused. \n\nThe method is very well described and presented. I really like the simplicity behind it, and the authors do a good work explaining the motivations and intuitions behind it.\n\nThe contribution, however, can be arguably minor, because they are taking two solid methods and just gluing them together; nevertheless, it works, so I'm guessing authors will have to defend the relevance of their contribution to the other reviewers.\n\nI personally consider it worthy for acceptance at NeurIPS; there are, however, some details that I consider that need attention, before I can give a higher evaluation for the paper:\n\n- I reviewed closely your experimental design, and I noticed that you use extremely small sample points sets (10~20 points); I wonder how results change if you provide more sample points, let's say, in the range of hundreds or thousands, to the different methods tested.\n\n- If I understood correctly, your 'primitives' (as called in GP literature) is {+,−,×,÷,sin,cos,exp,log,x} (from your appendix). So, there are certain problems, such as Nguyen-8 and Nguyen-11 where the GP has no chance to find the correct expression, because it would be simply too difficult, or even impossible, to build it given those primitives.... why or how did you expect the GP to find those expression?? For example, express to me x^y using only the set of functions mentioned above. But more importantly, how did your method manage to find such functions?\n\n- The most important element that I find missing in your draft, is a bit of discussion on how or why you think your method works, at a low level analysis. I do understand the perks and capabilities of RL, but my question is... what exactly is doing the RL that enhances that much a vanilla GP. My bet is that is pruning the primitives set, such that the GP has a smaller and better defined search space, but I'd really like to hear your thoughts on this.\n\nNotice how your method could help immensely to understand on why GP sometimes work, sometimes doesn't, i.e. what makes a problem difficult for GP, which has been the million dollar question in GP research, but you need to provide a bit of discussion in such regard.\n\n[UPDATED RECOMMENDATION]\nAuthors properly addressed all my questions and concerns. I've no doubt on the effectiveness and relevance of their method. The paper is well written, and my main objection (lack of discussion on how the method works at a low level) has been attended. Therefore, I see no real reason for this work for not being accepted.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors do not provide any discussion regarding potential societal impacts of their work; nevertheless, considering it is a highly theoretical work, a purely algorithmic contribution, I do not consider such analysis a requirement for this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received review scores with a high variance: 5,6,6,9. The paper was actively discussed, both with the authors and in private.

Reviewer Ug8S (score: 9) championed the paper for its simplicity and strong results. Yet, the review text and the arguments given (e.g., "I see no real reason for this work for not being accepted.") do not sound like a score of 9 at NeurIPS, more like a 6 or 7).

Negative points in the reviews and the discussion were the lack of originality (the paper heuristically combines existing methods: RL in order to sample the initial population of evolutionary methods, to solve symbolic regression problems).

A brief note about the NeurIPS checklist: the authors' responses to the questions are very casual (e.g. a simple "Yes" would be preferable over "It would be a bad idea not to"), as if these points are obviously satisfied; however, the paper did *not* attach the code and as the code repo merely gave "www.anonymous.submission". That does *not* count as the code being available; it should be available for reviewers to check in an anonymized repo or attached as supplementary material.

Overall, in view of the strong results and the simplicity of the method, I am leaning towards acceptance as a poster.

Decision: Accept (Poster)