Rethinking Symbolic Regression: Morphology and Adaptability in the Context of Evolutionary Algorithms
Authors: Kei Sen Fong
Shelvia Wongso
Mehul Motani
Venue: ICLR 2023 poster (2023)

Abstract: Symbolic Regression (SR) is the well-studied problem of finding closed-form analytical expressions that describe the relationship between variables in a measurement dataset. In this paper, we rethink SR from two perspectives: morphology and adaptability. Morphology: Current SR algorithms typically use several man-made heuristics to influence the morphology (or structure) of the expressions in the search space. These man-made heuristics may introduce unintentional bias and data leakage, especially with the relatively few equation-recovery benchmark problems available for evaluating SR approaches. To address this, we formulate a novel minimalistic approach, based on constructing a depth-aware mathematical language model trained on terminal walks of expression trees, as a replacement to these heuristics. Adaptability: Current SR algorithms tend to select expressions based on only a single fitness function (e.g., MSE on the training set). We promote the use of an adaptability framework in evolutionary SR which uses fitness functions that alternate across generations. This leads to robust expressions that perform well on the training set and are close to the true functional form. We demonstrate this by alternating fitness functions that quantify faithfulness to values (via MSE) and empirical derivatives (via a novel theoretically justified fitness metric coined MSEDI). Proof-of-concept: We combine these ideas into a minimalistic evolutionary SR algorithm that outperforms all benchmark and state of-the-art SR algorithms in problems with unknown constants added, which we claim are more reflective of SR performance for real-world applications. Our claim is then strengthened by reproducing the superior performance on real-world regression datasets from SRBench. For researchers interested in equation-recovery problems, we also propose a set of conventions that can be used to promote fairness in comparison across SR methods and to reduce unintentional bias.

Published as a conference paper at ICLR 2023
RETHINKING SYMBOLIC REGRESSION: MORPHOLOGY AND ADAPTABILITY FOR EVOLUTIONARY ALGORITTMS
Kei Sen Fong Shelvia Wongso , and Mehul Motani 1.2
Department of Electrical and Computer Engineering, National University of Singapore
2 N.I Institute for Health, Institute for Digital Medicine (WisDM), Institute of Data Science, National University of Singapore
[fongkeisen, shelvia w/eu. nus edu, cotani@nus.edu.
ABSTRACT
Symbolic Regression (SR) is the well-studied problem of finding closed-form an- alytical expressions that describe the relationship between variables in measure- ment dataset. In this paper, we rethink SR from two perspectives: morphology and adaptability. Morphology: Current SR algorithms typically use several man- made heuristics influence the morphology (or structure) of the expressions the search space. These man-made heuristics may introduce unintentional bias and data leakage, especially with the relatively few equation-recovery benchmark problems available for evaluating SR approaches. To address this, we formulate novel minimalistic approach, based on constructing a depth-aware mathematical language model trained terminal walks of expression trees, replacement these heuristies. Adaptability: Current SR algorithms tend to select expressions based only single fitness function (e.g., MSE the training set). We promote the use of an adaptability framework in evolutionary SR which uses fitness func- tions that alternate across generations. This leads to robust expressions that per- form well on the training set and are close to the true functional form. We demon- strate this by alternating fitness functions that quantify faithfulness values (via MSE) and empirical derivatives (via a novel theoretically justified fitness metric coined MSEDI). Proof-of-concee
We combine these ideas into a minimalistic evolutionary SR algorithm that outperforms suite of benchmark and state oh-the- art SR algorithms in problems with unknown constants added, which we claim are more reflective of SR performance for real-world applications. Our claim is then strengthened by reproducing the superior performance on real-world regression datasets from SRBench. For researchers interested in equation-recom lems, we also propose a set of conventions that can be used to promote fairness in comparison across SR methods and to reduce unintentional bias.
prob-
INTRODUCTION
Important discoveries rarely come in the form of large black-box models; they often appear sim- ple, elegant, and concise expressions. The field of applying machine learning to generate such mathematical expressions is known as Symbolic Regression (SR). The expressions obtained from SR come in a compact and human-readabee form that has fewer parameters than black-box models. These expressions allow for useful scientific insights by mere inspection. This property has led SR to be gradually recognized as a first-class algorithm in various scientific fields, including Physics (Udrescu & Tegmark 2020), Material Sciences (Wang et al. 2019 Sun al. 2019) and Knowledge Engineering (Martinez-Gil. & used SR is genetic programming (GP) (Koza, 1992). GP generates populations of candidate ex- pressions and evolves the best expressions (selected via fitness function) across generations through evolutionary operations such as selection, crossover, and mutation. In this paper, we rethink GP-SR from two svolutionary-insphnn perspectives: morphology and adaptability.
2020) in recent years. The most common technique

---

Published as a conference paper at ICLR 2023
Morphology. SR algorithms have traditionally used several man-made heuristics to influence the morphology of expressions. One method is to introduce rules (Worm & Chiu 2013). constraints (Petersen et al. 2019 Bladek & Krawiec 2019) and rule-based simplifications (Zhng al. 2006),
with the objective of removing redundant operations and author-defined senseless expression. An
example is disallowing nested trigonometric functions (e.g. sén(1 cos(x))) Another method is to
assign complexity scores to each elementary operations (Loftis et al., 2020 Korns) (2013), intending to suppress the appearance of rare operations that are given a high author- casboor--ssgneddd complexity score. However, these man-made heuristics may introduce unintentional bias and data leakage, exacerbated by the small quantity of benchmark problems in SR (Orzechowski al., 2018). With the success of deep learning and its applications to SR, there exists substantial motivation to utilize deep learning to generate potential morphologies of candidate expressions in SR (Petersen et al. 2019) Mundhenk et al. 202T). Such technique also comes with the benefit of being easily transferable to a problem with a different set of clementary operations. In this regard, we first show how current SR methods are reliant on these man man-made heuristics and highlight the potential drawbacks. Then, in this paper, we show how using our neural network pre-trained on Physics equations (which we later introduce as TW-MLM) improves the performance of GP-SR even in the absence of such man-made heuristics.
Adaptability. SR algorithms tend to evaluate a candidate expression based on its faithfulness to empirical values. Some common fitness functions are Mean Absolute Error (MAE), Mean Squared Error (MSE) and Normalized Root Mean Square Error (NRMSE) (Mundhenk et al. 2021), among other measurements. We propose that SR should create a variety of characterizattio measures be- yond the existing ones that measure faithfulness to empirical values, While previous work have sug- gested alternative characteristics in dealing with time-series data (Schmidt & Lipson 2010 2009), it is not easily transferable to SR in general and was not theoretically derived. In this paper, we propose to quantify the faithfulness of candidate expression's 's empirical derivative to the ground (Bateson] 2017). we adopt an adaptability framework that
truth. Motivated by evolutionary-theon changes the fitness functions across generations. This process makes harder for pseudo-equations to survive through the different fitness functions and easier for ground truth equation to survive across the generations. The additional benefit of such a method lies in the increased utility of the eventual expression. If the intention of the user is to take the derivative of the eventual equation, then a measure of faithfulness to empirical derivatives as fitness would assist that objective. In this paper, alternate between different fitness functions to improve performance of GP-SR. Specifically, alternate between MSE and newly defined fitness function we term MSEDI.
Proof-of-concept. We combine these ideas to demonstrate a proof-of-concept (foreshadowed in Figure through minimalistic evolutionary SR algorithm that outperforms all methods (including state-of-the-arr SR) in problems with unknown constants (i.c.. Jin * (Jin al., 2019 and outper- forms many benchmark models in problems without unknown constants (i.c.. Nguyen* * (Uy et al.) 2011) and R* (Krawiec & Pawlak 2013). We contend that performance on datasets with unknown constants are more indicative of SR performance in real-world applications that including naturally occurring processes such as scaling. Our claim is then strengthened by reproducing this superior performance on real-world regression datasets from SRBench (La Cava al. 2021
To accommodate future research SR using real-life datasets, we propose extra SR con- ventions for synthetic datasets in line with the "relevancy" criteria for results on benchmarks problem to correlate well with results on real- world applications (MeDermott et al. 2012).
New candishee exgresiooss every generation
Firso 7HRMEM
From GP
TW-MLM 1
Fitcess Plnctton Evalinttion
Exsiuiionay
Altenate betwoen MSE and MISDDI
Opentions
The remainder of this paper is organized as follows: Section X explains related work and mechanisms to improve SR, focusing on recent deep learning work. Section 3 describes our proposed methodology and mechanisms. Sec- tion g
MLM Datasel
Pre-training
Genetic Programming
Figure 1: Integrated proof-of-concept schematic.
combines all our proposed mechanisms with the vanilla GP approach and compares the per- formance with state-of-the-ar commercial and traditional SR approaches. Reflections and future work are given in Section 5
The main contributions of this paper are as follows:
2

---

Published as a conference paper at ICLR 2023
1. We propose set of conventions and best practices that reduce the risk of human bias in the context of using SR on real-life datasets.
2. We develop predictive model called TW-MLM that learns the morphology of expressions through a mathematical language model, which is then used to generate candidate expressions for SR. TW-MLM serves as an alternative man-made heuristics that is less prone human bias and easily transferable to new problems.
3. We propose a method for alternating fitness functions inspired by adaptability in evolution theory to tackle the problem of multi-objective optimization in SR. We do this by alternating between
MSE and our novel theoretically justified metric called MSEDI.
4. We develop an integrated proof-of-concee by combining our ideas and performed extensive testing on both synthetic and real-world datasets (from SRBench).
2 RELATED WORK
Deep Learning for SR. Deep learning has seen much success in SR. including Deep Symbolic Regression (DSR) (Petersen et al. 2019), which uses reinforcement learning to train a recurrent neural network (RNN) to complete expression tree. Deep Symbolic Optimization Neural-Guide Genetic Programming (DSO-NGGP) is an improved method that combines a similar RNN trained by reinforcement learning coupled with GP (Mundhenk al.] (021). The authors suggest that GP helps to produce large variations in the population of candidate equations when stuck at a local optimum. However, these methods are used in conjunction with several man-made heuristics to be effective. Other works include Al-Feynman, which utilizes neural networks to discover simplifying properties such as symmetry and separability to reduce the number of variables (Udrescu & Tegmark| 2020).
Mathematical Language Model. Symbolic mathematics has been successfully addressed as both a machine translation problem and a next word prediction problem (Lample Charton| 2019 Kim
et al. 2021]. In particular, it has been reported that by adding pre-trained seq2seq model as a supplementary mechanism to neural-guided SR, performance has improved (Kim et al. [002]) The consensus is that mathematics should be viewed as a language with complex rules. Most useful mathematical expressions are not only short and concise, they also tend to obey a variety of hidden
rules, such as avoiding nested trigonometric functions (e.g. sin(1 + cos(z)). In this context, mathematical language model can be developed to learn such rules implicitly.
Derivatives in SR. The usage of derivatives in SR has been hinted in previous works for time- series data (Schmidt & Lipson 2010 2009). In their work, given N data samples from two time- dependent variables, zrt) and (tt, the system's 's derivatives are Az Ay = x'/y/' where x' and y' represent the empirical derivatives of x and y with respect to time. These values are then compared against the derivative obtained from the candidate expressions, /xz/ōy,, through mean logarithmic error: + Σ log(1 - / - /0yl)) However, the theoretical derivation is not explored in their paper. Recent works in SR also include model discrimination by incorporating prior infor- mation on the sign of the first derivative based on physical knowledge (Engle & Sahinidis (021). In paper, develop theoretical basis in line with existing assumptions for modelling errors to quantify faithfulness to derivatives for general equations which do not necessitate time-dependency.
Oulti-Objective Genetic Programming (MOGP). Several general approaches exist for MOGP (Konk et af.] 2006). A possible approach is to set all but one objective as constraints. However, the process of selecting values for constraints and which objectives to set as constraints is arbitrary. Another approach is to output a Pareto optimal set instead of a single individual expression to reflect the trade-off's between the different objectives. However, in the context of SR, the ground truth expression would be the best performer for both objectives, rather than forming a trade-off.
Benchmarks Methods. For comparison, we include traditional SR, state-of-the tort--rrttt-t (SOTA), com- mercial algorithms, and random search. The methods selected report the lifetime population size of expressions to enable fair comparison of recovery rates. The benchmark methods are: (i) DSR (Pe- tersen et al. 2019): Previous SOTA method that pioneered usage of reinforcement learning in SR; (ii) DSO-NGGP (Mundhenk et al. 202I): Current SOTA method in SR that is hybrid of DSR and GP: (iii) GPLeam (Stephens] 2016): Python framework for standard GP-SR (Kozal 1992). which has seen wide usage as a generic SR method (Pankratius et al., 2018 Ferreira et al. 2019); (iv) Tur-
ingBot BouringBot 2020): Commercial SR product based on simulated annealing. which has been shown to be competitive among top commercial SR algorithms (Ashok et al. 202I). (v) Random
Search: Generate expressions random without evolution.
3

---

Published as a conference paper at ICLR 2023
3 METHODOLOGY AND MECHANISMS
Here, we first propose revised conventions to be used for SR experiments targeted at resolving the flaws and criticism of current SR metrics, which are independent of our SR algorithm. We then introduce our novel methods of controlling morphology of expressions and promoting adaptability in evolution, and justify these methods individually through preliminary results and ablation studies.
3.1 PROPOSED CONVENTIONS AND BEST PRACTICES
We propose and justify conventions for SR experiments, adhering to the criteria discussed in the call for better benchmarking to be done so that SR experiments
correlate better with results on real- world applications (McDermott et al. 2012)). In many recent papers, recovery rate has been utilized as the primary metric for evaluating SR and is defined "the fraction of independent training runs in which an algorithm discovers an expression that is symbolically equivalent to the ground truth expression within maximum of 2 million candidate expressions" (Mundhenk et al.) [202 Petersen iet al. 2019 Kim et al., 2021 Larma et al.) (02I). To this end, the conventions we propose will be Tocused on improving the effectiveness of recovery rate as an evaluation metric for SR experiments.
Fixed set of primitive functions. In contrast to previous SR experiments that use a varying primitive function set depending on the equation, we propose to use a fixed set of primitive functions across all datasets and all methods. This is necessary step towards using SR on real-life datasets since we are blind to the underlying primitive functions in real-world scenarios. In our paper, we use fixed primitive function set {add, mul, sub, div, sin, cos, arcsin, log, erp, pow] for all datasets and methods, selected from the dataset used to train our mathematical language model.
We also propose new measure, top-1 approximate recovery rate, that more reflective of performance on real-life datasets compared to the exact recovery rate defined in the first paragraph of Section 3.1 Top-/ means taking only the best scoring expression in the strictest sense, as consistent with how SR is used for real-life data (Abdellaoui & Mchrkanoon 2021 Phukoetphim et al., 2016 Barmpalexis et al. 201T). In other words, we only assess one best equation per experimental run. We define an approximate recovery to be when the r-squared value of expression (touted as the best error measure for SR (Keijzer) (0044) over the entire sampling domain of the dataset is more than 99%. This is consistent with calls from the GP community to discourage measuring exact recovery on synthetic datasets as they usually do not correlate well with performance on real-life applications (McDermott et al. 2012). The most obvious drawbact bawk of an exact recovery rate is that for real-life datasets, checking mathematical equivalence since there will not be any accompanying ground truth equation for comparison. In our paper, we present top-1 approximate recovery rate as our primary metric, but we also include the results for top-/ exact recovery rate in brackets for comparison.
Top-1 Approximate Recovery.
impossible to measure the true recovery rate by
Selecting appropriate lifetime population size. To guard against setting a lifetime population that is too high, we also propose to benchmark against random method (generate the entire lifetime pop- ulation in one generation without any genetic operation) and to ensure the metric across all methods is not saturated at the selected population size. This is done to elicit meaningful conclusions from the results. Previous works have used an arbitrary lifetime population size of million (Mundhenk) et al. 2021 Petersen al. 2019 Kim et al. 2021 Larma et al., 2021), but is difficult to compare results across the different methods since many equations have near 100% recovery, even for the worst-performing methods. Additionally, using an arbitrary value may create unintended bias in results [Bergstra & Bengio, 2012). To these ends, we reduce the lifetime population size to 10000 following 2 observations we made prior to our main experiments. First, the new size, the top-1 approximate recovery rate is never saturated across all methods, allowing us to compare the perfor- mance of each method across various equations. Second, we evaluate fully random search, which is implemented in practice by setting GP with only 1 generation at full population size, and find that the performance across most of the equations is near O. This gives us higher confidence to report that SR methods with positive performance do recover equations by pure chance.
Datasets with unknown constants are more relevant. We also recommend testing on datasets which include unknown constants, such as Jin dataset, since we find that the performance of meth- ods varies drastically with and without unknown constants. In addition, it is of practical interest to consider datasets with unknown constants since. it common in real-life relations between vari- ables, such as feature scaling (Udrescu & Tegmark 2020).. For example, we have f = e 002 / real-world physics equation from Al-Feynman database (see Appendix Table 7).
4

---

Published as a conference paper at ICLR 2023
Table 1: Top-1 Approximate / Top-1 Exact Recovery Rates (%) of current methods across Jin (with unknown constants) and Nguyen datasets (without unknown constants). Results were averaged over 100 runs per equation per method.
DSO-NGGP
GPLEARN
TURINGBOT
DSO-NGGP
GPLEARN
TURINGBOT
1
47/0
29/0
1/0
NGUYEN*-1
95/75
3/1
59/5
J
23/0
NGUYEN
34/0
13
12/0
16/0
0/0
NGUYEN
81/12
15/2
28/0
J
21/0
81/5
19/0
15
7/5
14/3
1/0
NGUYEN
10/3
0/0
8/0
J
0/0
NGUYEN
39/0
NGUYEN 9-7
99/2
57/0
98/0
NGUYEN
39/10
-AVRRAEE
11.33/0.83
17.17/0.50
0.50/0.00
NGUYEN AVERRAEE 68,25/23.38
19.00/2.00
40.50/1.88
In our experimental results recorded in Table 1 we observe that between TuringBot and GPLearn, ToringBot has the poorer performance in Jin dataset that has unknown constants. However, Tur- ingBot has the superior performance in Nguyen* * dataset that does not contain unknown constants. A similar observation can be made between GPLearn and DSO-NGGP where the relative perfor- mance is reversed depending on the presence of unknown constants. This phenomenon is due the difference the frequency of appearance of constants and the presence and extent of constants op- timization in each method. For datasets with no unknown constants, the more the algorithm utilizes and optimizes constants, the more likely that equations with morphology that are dissimilar to the true equation can become top candidates. This can be seen as inhibiting the evolutionary process, where the spots for expressions to undergo evolution are instead taken up by these psuedo-equations (expressions that perform well on the training set in terms of MSE but are not close to the ground truth equation in its functional form).
In the context of rial-life datasets, we argue that is justifiable to assume that unknown constants will appear frequently in naturally occurring processes such scaling. The performance of methods on datasets without unknown constants would then be less reflective of the performance on real-life datasets. We thus recommend that SR experiments favor datasets with unknown constants and assert that results obtained from such datasets are more reflective of real-life application.
3.2 MORPHOLOGY OF EXPRESSIONS
Previous works have been done to influence the morphology of candidate expressions using deep learning methods (Petersen et al. 2019 Bladek & Krawiec 2019 Udrescu & Tegmark 2020). However, these methods have never been fully independent, and have instead been an addition to another method or utilize man-made heuristics such as those mentioned in earlier sections. We also find that these methods can be heavily reliant on the heuristics. For instance, DSR performance drops sharply with the removal of in-situ constraints and complexity scores, with top-l approximate recovery rate decreasing to 33% the original value and top-1 exact recovery rate decreasing to 10% the original value. Furthermore, these heuristics increase the likelihood for the algorithm to be biased towards certain form of expressions, allowing for an implicit data leakage. Finally, extremely difficult task form such man-made rules, and it is made even harder when the discovery such rules needs to be repeated from scratch when the primitive function space is distinctively different, such as when changing from ordinary algebra boolean algebra. Thus, we aim to propose a method that is free of such man-made heuristics. Here, we outline a standalone method of generating candidate expressions that can be used independently.
Terminal walks representation. Instead of using the prefix representation of expressions as in- put to seq2seq model done in other SR methods, we take inspiration from random walks used in node2vec (Grover & Leskovec, 2016) and generate terminal walks from expression trees to re- flect the hierarchical nature of expressions. A single terminal walk refers to the collection of nodes traversed from the root of the expression tree to either variable node or constant node. These terminal walks will then be treated as sentences. The benefit of such a method over prefix repre- sentation is that the distances between operations in terminal walks are reflective of the distances between operations in expression tree. On the other hand, in prefix representation, the operations that appear to be faraway may instead be near in the expression tree. For example, for the equation sin(1 + cz)( ). the nested cos is 2 tokens away in both the terminal walk (sin, add, cos, x) and in the expression tree form. However, it is tokens away in the prefix notation.
5

---

Published as a conference paper at ICLR 2023
Table 2: Top-1 Approximate/ / Top-1 Exact Recovery Rates (in percentage) of GP and TW-MLM-GP across Nguyen*, R R$ and Jin * dataset. Results were averaged over 100 runs per equation per method.
TW-MLM-GP (RNN)
TW-MLM-GP (TRANSFORMER)
GP
NGUYEN R
24.13/7.25
23.51/7.31
19.00/2.00
19.33/0.00
18.67/0.00
9.67/0.00
JIN
21.33/0.50
17.67/0.50
17.17/0.50
Building language model to replace heuristics. We then treat the collection of all terminal walks as a corpus of sentences to train a RNN as commonly practiced in sext-word-predictin natural language processing tasks (Barman & Boruah 2018). For paper, we used embedding layer, long short-term memory layer and a dense layer sequentially to train a lightweight RNN. This RNN will be our mathematical language model for our GP algorithm, which we coin as terminal walks mathematical language model (TW-MLM). Candidate expressions are then generated by randomly selecting an operation from uniform distribution, then for every incomplete link in the tree, an incomplete terminal walk is generated and fed into the TW-MLM. The TW-MLM then outputs a probability distribution that is used to select the next node to complete the tree. This process repeats until the tree has no incomplete links.
Ablation: TW-MLM to improve recovery rates. In this ablation experiment, we use the baseline GP algorithm Koza 1992) implemented in GPLearn for comparison since the other competitive SR methods impose man-made heuristics which would complicate the insights drawn from experimen- tal results. For training the TW-MLM, we use the set of 2023 terminal walks generated from 100 Physics equations (Udrescu & Tegmark 2020). These equations are suitable as they contain widely accepted expressions used in real-life scenarios, in contrast to the other SR datasets. When tokeniz- ing the equations, variables are represented as a single "variable token". When TW-MLM generates new equation for a numerical dataset, this token is replaced by randomly selected variable.
Throughout this paper, we use a lightweight RNN, comprising of a single emhedding layer, a single long short-term memory layer and single dense layer. We experiment with replacing the RNN with a transformer, which we find to perform worse. As seen in Table 2 our experiments show that with just the sole usage of our TW-MLM to generate candidate expressions, we can drastically improve both top-/ exact recovery rate and top-/ approximate recovery rate across all 3 datasets. The intuition for the improvement of results that the TW-MLM learns the intrinsic patterns which makes equations human-readable, which promotes the GP algorithm to explore a search space with a high concentration of hemablre equations.
3.3 ADAPTARILITY IN EVOLUTION
Borrowing the idea of adaptability from the theory of evolution (Bateson 2017), we argue that a candidate expression close to the ground truth will survive through multitude of fitness functions, whereas pseudo-expression may perform deceptively well for one but perform worse for the other fitness measures. In this context, the challenge for SR is to find suitable secondary fitness functions that measure characteristics beyond the primary fitness function, i.c., faithfulness to empirical values via the MSE One must also optimize for both fitness functions and we do this by alternating between the primary and secondary fitness functions.
Additionally, we note that one benefit of the simple closed-form analytical expression found by SR is that it allows the user to apply traditional mathematical tools on the expression, such as derivatives. In this paper, we explore the derivative as part of the secondary fitness functions, e.g., faithfulness to empirical derivatives. We describe two natural approaches to doing this, mean squared error of derivatives and mean squared error of difference, and argue that the former is superior to the latter for our application.
Mean Squared Error (MSE) of Derivative (MSEDE). MSE is commonly used as a fitness and optimisation function, and yields the Maximum Likelihood Estimate (MLE): ÔMSE = arg ming Im 2.) MSE rewards expressions that faithful to the values in the datasets. Likewise, we can reward expressions for being faithful to empirical derivatives. We can compute the MSE of the empirical derivatives from the dataset and candidate expressions. Consider N pairs

---

Published as a conference paper at ICLR 2023
of values (2,, U4), where = 1,2..... ...,, N, sorted based in ascending values of x. We define the empirical derivative as Als -
Az; 34 m Then, MMEEDE = arg ming E (Au A
Am;
Ar,
Mean Squared Error of Difference (MSEDI), Here, we develop a new fitness measure derived from theoretical basis that is consistent with the traditional error modelling framework used in MSE: Given measurements = f(z) C + e, where a an arbitrary constant and e ~ NN0 o), aim to find i = (o(),, such that go (x) is close approximation to (x:) We derive the MLE of parameters 0 by considering empirical derivatives.
NOO,, (A We note that the difference between derivatives obtained from dataset and candidate expression follows Gaussian distribution as well, which are independent to each other under the set of odd-valued or even-valued The total log likelihood across both sets should be similar by symmetry. We choose evaluate the total V2a the log likelihood of log likelihood across all i instead of picking one set. By letting 0; Ar;' Ay:
When (x) R f(x) C, Av Al AZ, Az.
5-1-5 Az. A2.
particular Az;
Ay: is In(Pr( Av: (0) Am;
In(v (n(v) 20 Anr
Air ABr Ar;
The total log likelihood across all i is thus,
N Σ
(- -I In( (n(,)) 20
Ali
Aú;
(I)
Ar.
Az,
The parameters 0 of go(x)) are obtained by maximizing the total log likelihood in 1 to obtain:
Ayi
Aúi
0 MSEDD = arg max
In(v In(V2ToO)
(2)
20
Ar;
Ar.
N
Agi
U/
arg max
(3)
20
Ar
Az;
arg min
(Ay: Aū:)²
(4)
MSEDI more relevant for real-world compared to MSEDE. However, we discover that MSEDE a potentially harmful fitness function when dealing real-life datasets since overfits selected set of noisy derivative values. Our experiments that include noise yield that using MSEDE led to no recovery across 100 repetitions of experimentation, while MSEDI was successful. We observe that random error, contributes to large noise in empirical derivatives, especially when Ar; is small. In other words, MSEDE can viewed as a weighted version of MSEDI, with the weights being (An.)7 The MSEDE function thus encourages over-fitting to values which are heavily weighted.
1. which is problematic when Ar, is O, which is com- Additionally, MSEDE relies on the value Ar, mon in real-world datasets with duplicated x values. Thus MSEDI better fulfills the criteria of relevancy to real-life problem (White al. 2013), providing an additional justification to choose our theoretically derived MSEDI.
Multi-objective optimization using MSEDI and MSE. As discussed Section 2 it is difficult to create a general method to optimize for both fitness functions using traditional methods. For instance, using a weighted combination do work for some equations one fitness will dominate, effectively only optimizing for one fitness in those cases. Instead, using the idea of adaptability, we utilize MSEDI
secondary test with the intention to filter away ggeudo-expressionn as described earlier in Section 3.3 Our experiments show the addition of this adaptability mechanism (using MSEDI as fitness function) once every 5 generations improves the exact recovery by more than double while maintaining a similar approximate recovery rate.
Ablation: Escaping local optima with and without MSEDI. In addition to acting as a secondary fitness function to remove pseudo-expresssions, we also find that MSEDI functions by helping GP escape poor local optima. Among the experiments that do not recover the equation, we save the state of the GP and run it for an additional 5 generations of MSE. If the top-1 equation remains the gennrtttonn same (which occurred for 72.82% of experiments), we rollback the state of the GP and run just generation of MSEDI. The percentage of experiments in which this generation of MSEDI helps

---

Published as a conference paper at ICLR 2023
to change the top equation are 19.82%, 15.58%, 44.33% for the Nguyen*, R* and Jin datasets respectively. Thus, from the results we know that MSEDI helps to escape poor local optima which would have otherwise stagnated.
4 INTEGRATED PROOF-OF-CONCEE
Combining the ideas described above, we added TW-MLM and the adaptability mechanism, that alternates MSE and MSEDI as fitness function, GP with constant optimization and using the primitive function set {add, mul, sub, div, sin, cos, aresin, log, exp, pow) as discussed in Section The interfacing of each component can be visualized in a simple schematic (Figure D
3.1
Algorithm T outlines our proof-of-concept. TW-MLM is trained once on ML.M dataset and utilized to generate candidate expres- sions at the start of every generation GP. GP then evaluates and filters from this set of expressions together with candidates derived from the previous generation through fit- ness functions that alternates between gen- eration. In method, use MSE base fitness function and switch to MSEDI once every 5 generations (decided based on hyper-paramette tuning on a smaller-sized experiment). We then conduct series of periments test the proposed approach on both synthetic and real-world datasets.
Algorithm 1: Outline of our Integrated Proof-of- Concept
while current generation < generations do
Fill up population of equations by generating
equations from TW-MLM;
current generation then
Evaluate fitness of equations using
MSEDI
else
Evaluate fitness of equations using MSE end
Evolve and select equations based on evolutionary operations in GP (e.g.. crossover, subtree mutation, hoist mutation, point mutation, reproduction)
Synthetic Datasets Experiments. We com- pare with the benchmarks methods and ran- dom search outlined in Section We choose synthetic datasets with unknown constants (i.e., Jin and without unknown constants (i.e., Nguyen and R*). Each equation in Table 6 (Appendix) is used to conduct 100 experiments per method. In Table and Table 4 we tabulate the performance on the three synthetic datasets (which contain 17 equations be recovered).
end
Synthetic Datasets Performance Comparison. Our method shows overall competitive perfor- mance with state-of-the-art. On equations with unknown constants, shown in the Jin dataset in T our method outperforms all other methods by a large margin, in both top-1 approximate re- covery rate and top-l exact recovery rate. On equations without unknown constants (e.g..Ngure Nguyen our method outperforms all except DSO-NGGP for almost all of the equations, as shown in Table Since Jin # included unknown constants that allows for relations that are reflective of real-life
4
Table Synthetic Dataset: Top-1 Approximate / Top-1 I Exact Recovery Rates (%) of methods for Jin dataset. Results were averaged over 100 experiments per equation per method.
OURS
RANDOM SEARCH
TURINGBOT
GPLEARN
DSR
DSO-NGGP
JIN
JIN
78/3
0/0
1/0
29/0
10/0
47/0
JIN*22
58/25
0/0
1/0
23/0
0/0
2/0
JIN
53/7
0/0
0/0
16/0
0/0
12/0
JIN"-4
31/10
0/0
21/0
JIN9.5
21/20
14/3
JIN
0/0
0/0
0/0
0/0
0/0
0/0
AVERAGE
40.17/10.83
0.17/0.00
0.50/0.00
17.16/0.50
1.67/0.00
11.33/0.83
Table 4: Synthetic Dataset: Top-1 Approximate / Top-1 Exact Recovery Rates (%) of methods for Nguyen* and R* datasets. Results were averaged over 100 experiments per equation per method.
OURS
RandOM Search
TuRiNGBor
GPLEARN
DSR
DSO-NGGP
SOUYEN*-AVERAN
42.13/18.38
17.63/0.00
40.50/1.88
19.00/2.00
30.63/7
68.25/23.38
AVERAGE
21.00/0.00
0.67/0.00
6.00/0.00
5.33/0.00
0.67/0.00
23.67/0.00
8

---

Published as a conference paper at ICLR 2023
Table 5: Real- World Dataset (from SRBench); Percentage of experiments where our method outper- forms DSO-NGGP in r-squared value. Results are shown for 100 experiments per dataset.
NAME OF DATASET
1027
1028,SWD
1029_LEV
1030
STROGATZ. SHEARFLOW]
192.YINEYARD-C SMALL
OUTPERPORMANCE
57%
66%
88%
94%
73%
78%
variables such as feature scaling (Udrescu & Tegmark 2020), it can be argued that the results from Jin * hold more practical value. In this sense, our model clearly outperformed all models as seen in Table 3
Real-World Datasets (SRBench) Evaluation. We then evaluated our method against DSO-NGGP on 6 real-worke world datasets from SRBench (La Cava et al. (021), with the results tabulated in Table S Unlike traditional SR datasets, real-world datasets do not have an accompanying ground truth equa- tion. We present the percentage of experiments in which our method outperformed DSO-NGGP in terms of r-squared value. Our method consistently outperforms DSO-NGGP, further corroborating previous results and strengthening the claim that SR datasets with unknown constants are more reflective of worldw dataset performance.
Intuition and Ablation. The function of MW-MMLM in our method is to implicitly learn the rules and constraints about the morphology of human- readable equations that are previously developed based on human judgment. This increases the likelihood of our method to venture into the search space containing expressions that are consistent with existing widely-accepted human-readable equations. In this sense, TW-MLM acts as a guide in the vast search space of possible equations. Though man- made heuristics such as rules and constraints exist to fulfill these roles, it is difficult to find, difficult to express and not easily transferable to new problems. On the other hand, TW-MLM finds these rules and constraints implicitly and the method can be casily transferred to a new problem. The adaptability mechanism (by alternating between MSE and MSEDI fitness function) then synergizes with TW-MLM and GP by increasing the difficulty for pseudo-equation to survive each generation, as shown in the ablation study in Section 3.2 We also observed throught the ablation study in Section 3.3 how MSEDI helps to escape from poor optima points in GP. These mechanisms added to GP allow us to create a model that performs competitively with sate-of-thee-n
5 REFLECTIONS
Summary. In this paper, we demonstrate an efficient proof-of-concee -f-nccept that incorporates two new independent mechanisms into genetic programming baased SR: (a) Terminal walks mathematical lan- guage model (TW-ML.M and (b) Adaptability via alternating fitness functions (i.e., between MSE and our novel theoretically justified metric called MSEDI). Through these simple modifications, we are able to obtain competitive results with respect diverse range of methods, outperforming all, including state-of-the-arr, when datasets with unknown constants are involved. We then reproduce this outperformance on real-world datasets. We also state and justify the conventions can promote consistency, comparability and relevance to real-world problems. Ultimately, we hope that SR can demonstrate more competitive real-world results among other machine learning meth- ods, given its natural advantage in both interpretability and explainability. The code for this paper is available at:
that
And/AaattailiitySR
Limitations. Our method performs much better on equations with unknown constants (Jin in contrast equations without unknown constants (Nguyen*). The reason behind this observation can be attributed to the difference in frequency of unknown constants during the generation of candidate expressions. Our TW-MLM trained on 100 Al-Feynman equations which has the frequent use of constants. This means that the TW-MLM has to learn to include constants into candidate expressions frequently, increasing the chances of over-optimization of constants candidate expressions.
Future Work. (i) The performance of our method on datasets without unknown constants may suffer due to over-optimization of constants in candidate expressions. To reduce the chances of this, SR can alternate between randomly generating constants and optimizing for constants. This is similar to the way we alternate between fitness functions in this paper. (ii) The search space of SR grows rapidly with an increasing number of variables, making real-world datasets with many variables a difficult problem for SR. We are currently working on an iterative greedy approach to multi- vartile SR. dealing with additional variable per SR-run to address the issues that SR faces with time complexity.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This research is supported by A*STAR, CISCO Systems (USA) Pte. Ltd and the National Univer- sity of Singapore under its Cisco-NUS Accelerated Digital Economy Corporate Laboratory (Award 12100)00002. Additionally, we would like thank the members of the Kent-Ridge Al research group at the National University of Singapore for helpful feedback and interesting discussions on this work.
REFERENCES
Ismail Alaoui Abdellaoui and Siamak Mehrkanoon. Symbolic regression for scientific discovery: an application to wind speed forecasting. arXiv preprint arXiv:2102.10570, 2021.
Dhananjay Ashok, Joseph Scott, Sebastian J Wetzel, Maysum Panju, and Vijay Ganesh, Logic guided genetic algorithms (student abstract). In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 15753-15754, 2021.
Partha Pratim Barman and Abhijit Boruah, rnn based approach for next word prediction in samese phonetic transcription. Procedia computer science, 143:117-123, 2018.
Panagiotis Barmpalexis, Kyriakos Kachrimanis, Athanasios Tsakonas, and Emmanouel Geor- garakis. Symbolic regression via genetic programming in the optimization of a controlled release pharmaceutical formulation. Chemometrics and Intelligent Laboratory Systems, 107(1):75-82, 2011.
Patrick Bateson, Adaptability and evolution. Interface Focus, 765):20160126, 2017.
James Bergstra and Yoshua Bengio. Random search for machine learning research, 13(2), 2012.
optimization. Journal of
Iwo Bladek and Krzysztof Krawiec. Solving symbolic regression problems with formal constraints. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 977-984, 2019.
Marissa Engle and Nikolaos V Sahinidis. Deterministic symbolic regression with derivative infor- mation: General methodology and application equations of state. AIChE Journal, pp. c17457, 2021.
Jimena Ferreira, Martín Pedemonte, and Ana I Torres. A genetic programming approach for con- struction of surrogate models. In Computer Aided Chemical Engineering, volume 47, pp. 451- 456. Elsevier, 2019.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. Proceedings of the 22nd ACM SIGKDI international conference Knowledge discovery and data mining, pp. 855-864, 2016.
Ying Jin, Weilin Fu, Jian Kang, Jiadong Guo, and Jian Guo. Bayesian symbolic regression. arXiv preprint 1rXiv:/910.000 08892, 2019.
Maarten Keijzer. Scaled symbolic regression. Genetic Programming and Evolvable Machines, 5(3): 259-269, 2004.
Joanne T Kim, Mikel Landajuela Larma, and Brenden K Petersen. Distilling wikipedia mathematical knowledge into neural network models. arXiv preprint arXiv:2104.059330 2021.
Abdullah Konak, David Coit, and Alice E Smith. Multi-objective optimization using genetic
algorithms: tutorial. Reliability engineering system safety, 9199)99221 2006.
Michael F Korns. A baseline symbolic regression algorithm. In Genetic Programming Theory and Practice X, pp. 117-137. Springer, 2013.
John Koza. Genetic programming. on the programming of computers by means of natural selec- tion. Complex adaptive systems, 1992.
10

---

Published as a conference paper at ICLR 2023
Krzysztof Krawiec and Tomasz Pawlak. Approximating geometric crossover by semantic backprop- agation. In Proceedings of the 15th annual conference on Genetic and evolutionary computation, pp. 941-948, 2013.
William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabrício Olivetti França, Marco Vir- golin, Ying Jin, Michael Kommenda, and Jason H Moore. Contemporary symbolic regression methods and their relative performance. arXiv preprint 4rXiv:2107.14351, 2021.
Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXiv preprint arXiv:1912.01412.. 2019.
Mikel Landajuela Larma, Brenden K Petersen, Soo K Kim, Claudio P Santiago, Ruben Glatt, T Nathan Mundhenk, Jacob F Pettit, and Daniel M Faissol. Improving exploration in policy gradient search: Application to symbolic optimization. arXiv preprint arXiv:2107.091 2021.
Christian Loftis, Kunpeng Yuan, Yong Zhao, Ming Hu, and Jianjun Hu. Lattice thermal conductivity prediction using symbolic regression and machine learning. The Journal of Physical Chemistry A, (25(():4435-455)) 450, 2020.
Jorge Martinez-Gin Gil and Jose Coaave--ozzzaez...o A novel method based on symbolic regression for interpretable semantic similarity measurement, Expert Systems with Applications, 160:113663, 2020.
James McDermott, David R White, Sean Luke, Luca Manzoni, Mauro Castelli, Leonardo Van- neschi, Wojciech Jaskowski, Krzysztof Krawiec, Robin Harper, Kenneth De Jong, et al. Genetic programming needs better benchmarks. In Proceedings of the 14th annual conference on Genetic and evolutionary computation, pp. 791-798, 2012.
T Nathan Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P Santiago, Daniel M Faissol, and Brenden K Petersen. Symbolic regression via neural-guided genetic programming population seeding, arXiv preprint /rXiv:2 /.00:55/ 2021.
Patryk Orzechowski, William La Cava, and Jason H Moore. Where now? large benchmark study of recent symbolic regression methods. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1183-1190, 2018.
Victor Pankratius, Cody Millard Rude, Guillaume Rongier, and Thomas Herring. Towards deriv- ing theories from data: Explorations of model inference in geophysics. In AGU Fall Meeting Abstructs, volume 2018, pp. IN13A-01,
Brenden K Petersen, Mikel Landajuela Larma, T Nathan Mundhenk, Claudio P Santiago, Soo K Kim, and Joanne T Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. arXiv preprint (wXXiv:912.0489. 2019.
Phanida Phukoetphim, Asaad Shamseldin, and Keith Adams. Multimodel approach using neural networks and symbolic regression to combine the estimated discharges of rainfall-runoff models. Journal of Hydrologic Engineering, 21(8)04411002, 2016.
Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. science, 324(5923):81-855 2009.
Michael Schmidt and Hod Lipson. Symbolic regression of implicit equations. In Genetic Program- ming Theory and Practice VII, pp. 73-85. Springer, 2010.
Trevor Stephens. Genetic programming in python, with scikit-learn inspired api: gplearn 2016.
Sheng Sun, Runhai Ouyang, Bochao Zhang, and Tong- Yi Zhang. Data-driven discovery of formulas by symbolic regression. MRS Bulletin, 44(7):559- -64,, 2019.
TuringBot. Symbolic regression software. 2020.
Silviu-Marian Udrescu and Max Tegmark. Al Feynman: A physics-inspired method for symbolic regression. Science Advances, 6( 6(16))eaay2631, 2020.
11

---

Published as a conference paper at ICLR 2023
Nguyen Quang Uy Nguyen Xuan Hoai, Michael O'Neill, Robert I McKay, and Edgar Galván- López. Semantically-basee crossover in genetic programming: application to real-valued sym- bolic regression. Genetic Programming and Evolvable Machines, 12(2):91-119, 2011.
Yiqun Wang, Nicholas Wagner, and James M Rondinelli. Symbolic regression in materials science. MRS Communications, 93)::793-805, 2019.
David R White, James McDermott, Mauro Castelli, Luca Manzoni, Brian W Goldman, Gabriel Kronberger, Wojciech Jaškowski, Una-May O'Reilly, and Sean Luke. Better gp benchmarks: community survey results and proposals. Genetic Programming and Evolvable Machines, 14(1): 3-29, 2013.
Tony Worm and Kenneth Chiu. Prioritized grammar enumeration: symbolic regression by dynamic programming. In Proceedings of the 15th annual conference Genetic and evolutionary com- putation, pp. 1021-1028, 2013.
Mengjie Zhang, Phillip Wong, and Dongping Qian. Online program simplification in genetic pro- gramming. In Asia-Pacific Conference on Simulated Evolution and Learning, pp. 592-600 600. Springer, 2006.
12

---

Published as a conference paper at ICLR 2023
A
APPENDIX
Table 6: Symbolic regression dataset specifications. Input variable is denoted by U(a, b, c) denotes с random points uniformly sampled between a and b for x. Equations were selected or modified to have the same number of variables throughout maintain the same search space size allow for meaningful comparison.
NAME
EXPRESSION
SAMPLING RANGE
NGUYEN+-1
x²
x² f
U(-1,1,20)
NGUYEN*-2
+
x + x² +x
U(-1,1,20)
NGUYEN*-3
5
+ x x² +x² +x++2 +
U(-1,1,20)
44
. sin (x*) cos(x) -1
+ x² + x. +
U(-1,1120)
N5UYEN*-5
U(-1,1,20)
N-UYEN#-6
sin(2) sin (x xr
U(-1,1,20)
NGUYEN+-7
log(z + 1) + log (I +1)
U(0,2,20)
NGUYEN
Vx
U(0,4,20)
R+-1
(x+11)
U(-1,1,20)
8" -9r444
R*-2
U(-1,1,20)
R*-3
U(-1,1,20)
27 ++ + +2+1
JIN
2.5r
1.32 + 0.5z² 1.7x
U(-3,3, 100)
JIN
8.0r + 8.0s. 15.0
U(-3,3, 100)
JIN
0.7z - 1.7x
U(-3,3, 100)
JIN*-4
ex5) 5.0 cos(2)
U(-3,3, 100)
JIN
6.0 (n)))n cos(x)
U(-3,3, 100)
JIN
1.35z² + 5.5 ((x 1.0)3)
U(-3,3, 100)
13

---

Published as a conference paper at ICLR 2023
Table 7: 100 Al-Feynman Physics Equations Udrescu & Tegmark (0220)
Feynman
Equation
eq.
1.6.20a
f = e 12 2
f = e 2n //220²2
1.620
00-0)²2
/v2502
1.6.00b
25
188.14
d (22
+ (1/2 - 31)
19.18
F
Gmyuigg
(x2-21) 21) )'
(y2 31 +(32-21) 21)
I.10.7
m
VI-
mi
I,11,19
A T1 + D2l/2 + 1393
I.2.11
F nN
I.12.2
F 9192
I,2.44
Ej A
H61
1,12.5
F = Q2 Fuf
112.11
F 9 (FB + Br sin 0)
113,4
K = 3m (v² + u² + w²
L13.12
U = Gm m2
1
L,4.3
mgz
I,1.4
U Appring 2
2 ut
L15,3x
1
1 u2 /02
1 uar e
L15.3t
/lu2/ol o2 may
1,15110
p
/
I.16.6
a+0 U1 = Itue/c2 e
L18.4
ne +12
118.12
rF sin O
1.8.16
= mrv sin
I.24.6
E mm (w² + wg) x
I.25.13
Ve
I.26.2
0 = arcsin (n sin 02)
1,27.6
f1
J. 2q
1.29.4
1.29.16
I
V 1 + 22 22 22 cos (#1 0z)
sin inn0 /2)
L.0033
1, Imo siu (8/2)
1.3.5
0 = arcsin
q²a"
L32.5
P
6reed
I.32.17
P 1cEE3
(8Tr² /3)
w+/(w> w8)²)
I,34,8
qr
&
1.34.10
T-ofc
1.34.14
1to/c
WD
I.34,27
E hw
L37.4
I. I + I, TT,,T cos 8
1.38.12
r 1
E 3 mqq
1.39.10
PF V
1.39.11
E = 4- --IPEV PEV 7 P Vinaz "
1.39.22
I.40.1
n = Age
14

---

Published as a conference paper at ICLR 2023
Feynman eq.
Equation
1.41.66
Ri Irad
I43.16
1 = Maviri
1.43.31
D Mcki T
L43.43
9-1 Kip
I.4.4
nkyT In
L47.23
IPE
P
1.4.200
E
me² v 11
L50.26
cOS (w))
(wt)
x T3 Ti Ti)A
I1,2.42
I1.3.24
FF I
IL.4.33
Ve
I1.6.11
V. Are PAE
1
Pa
11.6.15a
E
x
y²
Ang 24
1L.6.1b
E 4
cOs 0 sin 0
IL8.77
E 4mred Fiden
I1.8.31
I1.00.9
I1.11.
I
m(u0 -wa)
qE,
1 pat, cos
II111.1
no
n.P E,
IL.11.0
P.
na T
I1.11.2
P. I I-no//3 EE
IL111.8
0 = 1 (no/3)
IL.13.17
B T
II.13.23
Pe VC
IL.1333
8 Vi-o2 Je
Reg
I1.15.4
E PM B cOs 0
I1.15.5
= EaEg COS
I1.21.32
V. 4
V/O)
I1.24.17
k W
43
11.27.16
FE ECL
11.27.18
Eden L
1I.34.2a
I 2 9Er
11,34.2
P =
7
IL.34.11
W 9-QB
II.34.29a
μM BJ.
qh
11.34.29b
E
I1.35.8
11
exp (m. (/C +exp())eeexeeeee (k,2))
rhg
I1.35.21
M UpplAs Lanh
BM
R
II.36.38
f Um
Hm M
KgT
II.37.1
E HM (1 + x B
II.38.3
F Ax
11.38.14
11 2(1+a)
1II.4.32
15

---

Published as a conference paper at ICLR 2023
Feynman eq.
Equation
III.4.33
E
Aw Pu
111.7.38
w 2pa
h
) 2
III.8.54
By sin F
111.9.52
Py h
pa Et sin((w (w))))))/))
(ww-wa)t
10.19
PM B B + B
111.42.43
1, nh
111.1.18
2Ed
14
1 1o
E F
III.5.11
E = 2U (1 cos kd))
II1.5.14
m
ZEa
A²
III15.27
k nd
III.7.37
f = B(1 a cos 0)
II1.9.551
E 2 n
111.11.20
16


Reviews:
[{"review_id": "uYTCkHKmXZ", "reviewer": "Reviewer_vhnA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents two improving methods for symbolic regression (SR). The first method introduces the pre-trained recurrent neural network (RNN) model using the physics equation library to generate the candidate solutions. The second method is a novel loss function called mean squared error of difference (MSEDI). The MSEDI is alternately used with normal MSE in the proposed method. The experimental evaluation using benchmark datasets demonstrates that the proposed method outperformed existing SR methods in several problems.", "summary_of_the_review": "Although the concept of the proposed approach seems to be interesting, the experimental evaluation is weak to validate its effectiveness. Moreover, the algorithm description should be improved.\n", "strength_and_weaknesses": "**Strengths**\n* Two improving methods for symbolic regression are introduced. The experimental results show that the proposed method is superior to other SR methods, especially on the functions with unknown constants.\n\n**Weaknesses**\n* As only the concept-level explanation of the proposed method is given, it is hard to understand and re-implement the detailed algorithm. For instance, the training procedure of RNN and hyperparameter settings of the proposed method are omitted.\n* This paper presents two improving methods: pre-trained TW-MLM and MSEDI. However, it is not clear the contribution of each method to the performance gain. The ablation study should be conducted.\n* The proposed method uses the pre-trained model on physics equations. A simple baseline method using the physics equation dataset should be considered, e.g., incorporating the same physics equations into the initial population in genetic programming.\n* MSEDI assumes a scalar input variable. It seems hard to apply the proposed method to multi-dimensional input problems.\n\n**Comments**\n* Why didn't the authors use all datasets in SRBench? The selection reason for the 6 datasets should be clarified.\n* The physics equations in Table 7 include various different input variables, while the target functions in the experiments are composed of a single variable. The reviewer could not understand how to handle such a variable mismatch.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ObQBGvR4tAL", "reviewer": "Reviewer_yTKT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an evolutionary algorithm for symbolic regression combining genetic programming and a mathematical language model. The authors propose reasonable conventions that they use in their evaluation and suggest to be used by the community in general. To generate examples for training a lightweight RNN mathematical language model they propose to use a terminal instead of prefix representation. To solve the task of SR they also propose a method for alternating fitness functions for multi optimization. They evaluate their approach on synthetic datasets with and without unknown constants and compare it to a selection of SR approaches. Additionally they compare their approach to DSO-NGGP on a real-world datasets from SRBench.", "summary_of_the_review": "Overall a good, well written paper. The narrative can be slightly toned down. While the suggestions for evaluation conventions may be ultimately of benefit to the community a broader review and discussion is needed on this topic. While it is good to propose conventions and learn from the best practices, here the proposed is an evaluation frame used to demonstrate the performance of the presented approach. I believe that it is unnecessary to introduce novel terminology for concepts that are well known and understood such as morphology for structure of the model and adaptability for objective criteria. The authors themselves establish 1-to-1 relationship between current terminology and their new terminology as early as in the abstract and the introduction section.\n\nRegarding the proposed top-1 approximate recovery, i agree that perhaps the best scoring solution should be taken into account when comparing methods. This, however, removes the ability to evaluate one major performance aspect of SR methods - the consistency of recovery of expressions. A good SR method should consistently produce (symbolically or goodness-of-fit) good solutions across runs. Additionally, using only a goodness-of-fit criterion (r-squared), without considering symbolic equivalency can lead to overfitted expressions and/or expressions with spurious variable interactions that can't be easily explained. This in turn also reduces the compactness and interpretability of the solutions, which is a major upside of SR methods over other black-box methods. At least the parsimony of generated expressions should not be underestimated especially in the context of scientific discovery. I accept that this might not be exactly the case and I have misunderstood the proposed convention. In that case it would be good that authors explain this concept additionally and remove concerns by providing an empirical demonstration at least in the scope of their evaluation.\n\nRegarding the proposed selection of appropriate lifetime population size, this applies mostly to population-based algorithms. Additionally, the authors propose a selection of arbitrary population size or a heuristic for selecting a population size based on the performance of a  random search (may be out of scope for this work, but the authors should take a look at Bergrstra and Bengio, JMLR, 2012 on random search). Making the selection of arbitrary lifetime population size a convention can lead to biased evaluation, enabling selection of sizes where preferred approaches show favorable results. This also casts a bit of doubt on the presented evaluation of the approach. In fact, as it is the case in related work, instead of comparison based on a single point, a more general picture of the performance of an approach can be obtained by looking at the performance as a function of the number of evaluations instead. This would strengthen the performance claims of the proposed approach and as a convention it would also be more inclusive to other, non-evolutionary approaches.\n\n", "strength_and_weaknesses": "Strengths:\n- Well argumented\n- Well evaluated\n- Contributions both to methods for solving SR and evaluation/benchmarking of SR methods\nWeaknesses:\n- Ambitious goal of setting conventions\n- Conventions are \n- Unnecessary introduction of novel terminology", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C5VATUBUvo", "reviewer": "Reviewer_Kqo2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper combines ideas in language modeling and the theory of evolution to develop a novel approach to symbolic regression. By incorporating the terminal walk in language modeling, the proposed method is argued and shown to be able to reduce the reliance on human heuristics for constraints on the learned expression. By incorporating adaptivity from the theory of evolution, the proposed method is shown to be able to accommodate various learning targets, and perform better especially in practical situations when the true expression contains unknown parameters. The ideas are clearly stated and related literatures are properly compared to. The arguments are also backed by numerical experiments. ", "summary_of_the_review": "I find this to be a nice paper. It presents a novel approach that combines ideas from language models and evolution theory to solve symbolic regression. It properly relates to the literature, clearly states the benefits and intuitions of using these ideas, and supports these arguments by numerical experiments, which further identify the importance and challenges of accounting for unknown parameters. However, the authors need to be careful about the theoretical justification, and should provide reproduction codes when possible. ", "strength_and_weaknesses": "Strengths:\n\n1. A novel combination of existing techniques. \n\nThis paper presents a novel combination of existing techniques in other areas for solving the symbolic regression problem. Although these ideas already exist, the important point is to identify their roles for symbolic regression and properly combine them to achieve better performance in symbolic regression. \n\n2. Clearly stated ideas. \n\nThe ideas in the proposed approach are clearly stated. In particular, I appreciate that they are properly related to their original literature, and why it makes sense to use these techniques is well explained. \n\n3. Supporting experiments. \n\nAnother strength of this paper is the experiments that support the arguments and deliver practical messages. For example, it is good to realize that different methods may perform differently with and without unknown constants, and recognize the importance of unknown constants in practice. \n\n\n\nWeaknesses:\n\n1. Theoretical justification. \n\nThe only theoretical derivation on page 7 looks suspicious and I am not convinced. To be specific, even if the gaussian model is correct, the likelihood function is incorrect: $\\epsilon_{i+1}-\\epsilon_i$ are not i.i.d. random variables, hence the likelihood function cannot be a product over $i=1,\\dots,n$. Although MSEDI performs well in the experiments, the authors still need to think carefully about how to justify this criterion. \n\n2. Reproducibility. \n\nAs this paper is mostly an empirical study, the current version falls short due to the lack of open-source reproduction codes or libraries. I understand that this may be restricted in the blinded version. But at least it should be provided later on. \n\n3. The choice of RNN architecture. (question)\n\nThis point is more of a question: how do you choose the RNN architecture for your language model? Did you do some selection based on their performance, or are they commonly used in the area of language models? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9iDvLunkT18", "reviewer": "Reviewer_jSK3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors create a novel approach for symbolic regression using two different mechanisms. The first is a math language model, and the second is an adaptable strategy for alternating fitness functions during evolution. They experiment with their approach in synthetic and real-world data and achieve state-of-the-art results.", "summary_of_the_review": "- The work presented here is interesting and novel. There is margin for minor and major improvements in this work. Specially regarding the reproducibility of their training procedure and some more intuition on selecting some parts of the main contribution.", "strength_and_weaknesses": "Strength:\n\n- The experimental setup of the authors is very strong, showing different baselines and good results across different modalities of datasets.\n- The ablations studies also look sound, with some really interesting findings.\n- This work is also well written with no major mistakes that I could find, coupled with the right information when needed.\n\nWeaknesses:\n\n- (minor) I think this work would benefit a lot from a more classic structure. The authors also used their method section (Section 3) to present some ablations and results. This causes some confusion and makes the actual method not that clear. Perhaps moving the ablations afterward to the method’s actual result would clarify things. Also, the “conclusion” section, or as the authors called Reflection, seems a little too extensive for this work (will go into this later).\n- (major) The authors motivate their work on the premise that deep learning methods are black-box models that remove the possibility of insights when looking into SR. However, later in the introduction the authors admit that using deep learning models is quite good for generalization propose. Their method also uses an RNN as the backbone, which to me seems that would introduce the problem of black-box models into SR again.\n- (minor) Since we are using language models, I wish there were details on why not using Transformers/BERT or similar.\n- (medium) The authors give no instructions on how to reproduce their work. Yes, the intuitions are there, but what is the structure of the LSTM, embedding, and dense layer they used? How many neurons are we talking about? How many hidden layers does the LSTM have? Is it a bi-lstm? Furthermore, how was this LM trained? Which optimizer? Which LR? The authors should have answered all these questions in their work, and by not doing so harm their reproducibility.\n- (minor) the authors abbreviate twice MSE, MSEDE, and MSEDI\n- (medium) I wish there were more in-depth explanations about the results. The authors invest a long time in the ablations with intuition and provide none in the result section of this work.\n- (medium) I wish the authors had added the methods with human heuristics in their comparison. It would be ideal to have an ablation study on the trade-off from creating heuristics manually (and perhaps adding bias) and machine-made.\n- (medium) If there is a concern regarding bias in man-made heuristics, there should be an ablation regarding potential biases when using language models.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster