Neural-Guided Symbolic Regression with Asymptotic Constraints

1 INTRODUCTION . The long standing problem of symbolic regression tries to search expressions in large space that fit given data points ( Koza & Koza , 1992 ; Schmidt & Lipson , 2009 ) . These mathematical expressions are much more like discovered mathematical laws that have been an essential part of the natural sciences for centuries . Since the size of the search space increases exponentially with the length of expressions , current search methods can only scale to find expressions of limited length . Moreover , current symbolic regression techniques fail to exploit a key value of mathematical expressions that has traditionally been well used by natural scientists . Symbolically derivable properties such as bounds , limits , and derivatives can provide significant guidance to finding an appropriate expression . In this work , we consider one such property corresponding to the behavior of the unknown function as it approaches 0 and∞ . Many expressions have a defined leading polynomial power in these limits . For examples when x→∞ , 2x2 + 5x has a leading power of 2 ( because the expression behaves like x2 ) and 1/x2 + 1/x has a leading power of −1 . We call these properties “ asymptotic constraints ” because this kind of property is known a priori for some physical systems before the detailed law is derived . For example , most materials have a heat capacity proportional to T 3 at low temperatures T and the gravitational field of planets ( at distance r ) should behave as 1/r as r →∞ . Asymptotic constraints not only provide more information about the expression , leading to better extrapolation , but also constrain the search in the desired semantic subspace , making the search more tractable in much larger space . These constraints can not be simply incorporated using syntactic restrictions over the grammar of expressions . We present a system to effectively use asymptotic constraints for symbolic regression , which has two main parts . The first is a conditional production rule generating neural network ( NN ) of the desired polynomial leading powers that generates production rules to construct novel expressions ( both syntactically and semantically ) and , more surprisingly , generalize to leading powers not in the training set . The second part is a NeuralGuided Monte Carlo Tree Search ( NG-MCTS ) that uses this NN to probabilistically guide the search at every step to find expressions that fit a set of data points . Finally , we provide an extensive empirical evaluation of the system compared to several strong baseline techniques . We examine both the NG-MCTS and conditional production rule generating NN alone . In sharp contrast to almost all previous symbolic regression work , we evaluate our technique on thousands of target expressions and show that NG-MCTS can successfully find the target expressions in a much larger fraction of cases ( 71 % ) than other methods ( 23 % ) with search space sizes of more than 1050 expressions . In summary , this paper makes the following key contributions : 1 ) We identify asymptotic constraints as important additional information for symbolic regression tasks . 2 ) We develop a conditional production rule generating NN to learn a distribution over ( syntactically-valid ) production rules conditioned on the asymptotic constraints . 3 ) We develop the NG-MCTS algorithm that uses the conditional production rule generating NN to efficiently guide the MCTS in large space . 4 ) We extensively evaluate our production rule generating NN to demonstrate generalization for leading powers , and show that the NG-MCTS algorithm significantly outperforms previous techniques on thousands of tasks . 2 PROBLEM DEFINITION . In order to demonstrate how prior knowledge can be incorporated into symbolic regression , we construct a symbolic space using a context-free grammar G : O → S S → S ‘ + ’ T | S ‘ − ’ T | S ‘ * ’ T | S ‘ / ’ T | T T → ‘ ( ’ S ‘ ) ’ | ‘ x ’ | ‘ 1 ’ ( 1 ) This expression space covers a rich family of rational expressions , and the size of the space can be further parameterized by a bound on the maximum sequence length . For an expression f ( x ) , the leading power at x0 is defined as Px→x0 [ f ] = p s.t . limx→x0 f ( x ) /x p = non-zero constant . In this paper , we consider the leading powers at x0 ∈ { 0 , ∞ } as additional specification . Let S ( G , k ) denote the space of all expressions in the Grammar G with a maximum sequence length k. Conventional symbolic regression searches for a desired expression f ( x ) in the space of expressions S ( G , k ) that conforms to a set of data points { ( x , f ( x ) ) |x ∈ Dtrain } , i.e . find a g ( x ) ∈ S ( G , k ) : φ ( g ( x ) , Dtrain ) , where φ denotes the acceptance criterion , usually root mean square error ( RMSE ) . With the additional specification of leading powers c ( 0 ) and c ( ∞ ) at 0 and∞ , the problem becomes : find a g ( x ) ∈ S ( G , k ) : φ ( g ( x ) , Dtrain ) ∧ ( Px→0 [ g ] = c ( 0 ) ) ∧ ( Px→∞ [ g ] = c ( ∞ ) ) . 3 CONDITIONAL PRODUCTION RULE GENERATING NEURAL NETWORK . It is difficult to directly incorporate the asymptotic constraints as syntactic restrictions over the grammar . We , therefore , develop a neural architecture that learns to generate production rule in the grammar conditioned on the given asymptotic constraints . Figure 1 ( a ) and ( c ) show an example of how an expression is parsed as a parse tree by the grammar defined in Eq . ( 1 ) . The parse tree in Figure 1 ( c ) can be serialized into a production rule sequence r1 , . . . , rL by a preorder traversal ( Figure 1 ( d ) ) , where L denotes the length of the production rule sequence . Figure 1 ( b ) shows the leading powers of the exemplary expression in Figure 1 ( a ) . The conditional distribution of an expression is parameterized as a sequential model pθ ( f |c ( 0 ) , c ( ∞ ) ) = L−1∏ t=1 pθ ( rt+1|r1 , . . . , rt , c ( 0 ) , c ( ∞ ) ) . ( 2 ) We build a NN ( as shown in Figure 1 ( e ) ) to predict the next production rule rt+1 from a partial sequence r1 , . . . , rt and conditions c ( 0 ) , c ( ∞ ) . During training , each expression in the training set is first parsed as a production rule sequence . Then a partial sequence of length t ∈ { 1 , . . . , L− 1 } is sampled randomly as the input and the ( t + 1 ) -th production rule is selected as the output ( see blue and orange text in Figure 1 ( d ) ) . Each production rule of the partial sequence is represented as an embedding vector of size 10 . The conditions are concatenated with each embedding vector . This sequence of embedding vectors are fed into a bidirectional Gated Recurrent Units ( GRU ) ( Cho et al. , 2014 ) with 1000 units . A softmax layer is applied to the final output of GRU to obtain the raw probability distribution over the next production rules in Eq . ( 1 ) . Note that not all the production rules are grammatically valid as the next production rule for a given partial sequence . The partial sequence is equivalent to a partial parse tree . The next production rule expands the leftmost non-terminal symbol in the partial parse tree . For the partial sequence colored in blue in Figure 1 ( d ) , the next production rule expands non-terminal symbol T , which constrains the next production rule to only those with left-hand-side symbol T . We use a stack to keep track of non-terminal symbols in a partial sequence as described in GVAE ( Kusner et al. , 2017 ) . A mask of valid production rules is computed from the input partial sequence . This mask is applied to the raw probability distribution and the result is normalized to 1 as the output probability distribution . The training loss is calculated as the cross entropy between the output probability distribution and the next target production rule . It is trained from partial sequences sampled from expressions in the training set using validation loss for early stopping.1 4 NEURAL-GUIDED MONTE CARLO TREE SEARCH . We now briefly describe the NG-MCTS algorithm that uses the conditional production rule generating NN to guide the symbolic regression search . The discrepancy between the best found expression g ( x ) and the desired f ( x ) is evaluated on data points and leading powers . The error on data points is measured by RMSE ∆g { · } = √∑ x∈D { · } ( f ( x ) − g ( x ) ) 2/|D { · } | on training points Dtrain : { 1.2 , 1.6 , 2.0 , 2.4 , 2.8 } , points in interpolation region Dinterpolation : { 1.4 , 1.8 , 2.2 , 2.6 } and points in extrapolation region Dextrapolation : { 5 , 6 , 7 , 8 , 9 } . The error on leading powers is measured by sum of absolute errors at 0 and∞ , ∆P [ g ] = |Px→0 [ f ] − Px→0 [ g ] | + |Px→∞ [ f ] − Px→∞ [ g ] | . The default choice of objective function for symbolic regression algorithms is ∆gtrain alone . With additional leading powers constraint , the objective function can be defined as ∆gtrain +∆P [ g ] , which minimizes both the RMSE on the training points and the absolute difference of the leading powers . Most symbolic regression algorithms are based on EA ( Schmidt & Lipson , 2009 ) , where it is nontrivial to incorporate our conditional production rule generating NN to guide the generation strategy in a step-by-step manner , as the mutation and cross-over operators perform transformations on fully completed expressions . However , it is possible to incorporate a probability distribution over expressions in many heuristic search algorithms such as Monte Carlo Tree Search ( MCTS ) . MCTS is a heuristic search algorithm that has been shown to perform exceedingly well in problems with large combinatorial space , such as mastering the game of Go ( Silver et al. , 2016 ) and planning chemical syntheses ( Segler et al. , 2018 ) . In MCTS for symbolic regression , a partial parse tree sequence r1 , . . . , rt can be defined as a state st and the next production rule is a set of actions { a } . In the selection step , we use a variant of the PUCT algorithm ( Silver et al. , 2016 ; Rosin , 2011 ) for exploration . For MCTS , the prior probability distribution p ( ai|st ) is uniform among all valid actions . We develop NG-MCTS by incorporating the conditional production rule generating NN into MCTS for symbolic regression . Figure 1 ( f ) presents a visual overview of NG-MCTS . In particular , the prior 1Model implemented in TensorFlow ( Abadi et al. , 2016 ) and available in submitted materials . ‡ Evaluated on the subset of holdout sets . Hard are unsolved expressions in the subset ( Appendix H ) . probability distribution p ( ai|st , c ( 0 ) , c ( ∞ ) ) is computed by our conditional production rule generating NN on the partial sequence and the desired conditions . We run MCTS for 500 simulations for each desired expression f ( x ) . The exploration strength is set to 50 and the production rule sequence length limit is set to 100 . The total number of expressions in this combinatorial space is 3× 1093 . We run evolutionary algorithm ( EA ) ( Appendix G ) and grammar variational autoencoder ( GVAE ) ( Kusner et al. , 2017 ) ( Appendix H ) with comparable computational setup for comparison .

Summary: This paper considers a task of symbolic regression (SR) when additional information called 'asymptotic constraints' on the target expression is given. For example, when the groundtruth expression is 3 x^2 + 5 x, it behaves like x^2 in the limit x -> infinity, thus the power '2' is given as additional information 'asymptotic constraint'. In the paper's setting, for SR with univariate groundtruth functions f(x), 'asymptotic constraints' for x-> infinity and x -> 0 are given. For this situation, the paper proposes a method called NG-MCTS with an RNN-based generator and MCTS guided by it to consider asymptotic constraints. In SR, a learner is asked to acquire an explicit symbolic expression \hat{f}(x) from a given set of datapoints {x_i, f(x_i)}, but unlike the parameter optimization aspect of a standard supervised ML setting, the problem is essentially combinatorial optimization over exponentially large space of symbolic expressions of a given context-free grammar (CFG). For a given symbolic space with a prespecified CFG, extensive experimental evaluations are performed and demonstrated significant performance gain over existing alternatives based on EA. Also, quantitative evaluations about extrapolative performance and detailed evaluation of the RNN generator are also reported.