Title: Transformer-based Planning for Symbolic Regression
Venue: NeurIPS 2023
Accepted: yes
Keywords: Symbolic Regression

Transformers

Planning

Deep Learning
Abstract: Symbolic regression (SR) is a challenging task in machine learning that involves finding a mathematical expression for a function based on its values. Recent advancements in SR have demonstrated the effectiveness of pre-trained transformer models in generating equations as sequences, leveraging large-scale pre-training on synthetic datasets and offering notable advantages in terms of inference time over classical Genetic Programming (GP) methods. However, these models primarily rely on supervised pre-training objectives borrowed from text generation and overlook equation discovery goals like accuracy and complexity. To address this, we propose TPSR, a Transformer-based Planning strategy for Symbolic Regression that incorporates Monte Carlo Tree Search planning algorithm into the transformer decoding process. Unlike conventional decoding strategies, TPSR enables the integration of non-differentiable equation verification feedback, such as fitting accuracy and complexity, as external sources of knowledge into the transformer equation generation process. Extensive experiments on various datasets show that our approach outperforms state-of-the-art methods, enhancing the model's fitting-complexity trade-off, extrapolation abilities, and robustness to noise.

Transformer-based Planning for Symbolic Regression

Abstract

Symbolic regression (SR) is a challenging task in machine learning that involves finding a mathematical expression for a function based on its values. Recent advancements in SR have demonstrated the effectiveness of pre-trained transformer models in generating equations as sequences, leveraging large-scale pre-training on synthetic datasets and offering notable advantages in terms of inference time over classical Genetic Programming (GP) methods. However, these models primarily rely on supervised pre-training objectives borrowed from text generation and overlook equation discovery goals like accuracy and complexity. To address this, we propose TPSR, a T ransformer-based P lanning strategy for S ymbolic R egression that incorporates Monte Carlo Tree Search planning algorithm into the transformer decoding process. Unlike conventional decoding strategies, TPSR enables the integration of non-differentiable equation verification feedback, such as fitting accuracy and complexity, as external sources of knowledge into the transformer equation generation process. Extensive experiments on various datasets show that our approach outperforms state-of-the-art methods, enhancing the model's fitting-complexity trade-off, extrapolation abilities, and robustness to noise 1 .

1 Introduction

Symbolic regression (SR) is a powerful method to discover mathematical expressions for governing equations of complex systems and to describe data patterns in an interpretable symbolic form. It finds extensive applications in science and engineering, enabling the modeling of physical phenomena in various domains such as molecular dynamics, fluid dynamics, and cosmology [1-6]. Symbolic representations provide valuable insights into complex systems, facilitating a better understanding, prediction, and control of these systems through the design of accurate, generalizable, and efficient models [7-9]. SR models establish the functional relationship between independent and target variables by mapping them to mathematical equations. The input data can be obtained from simulations, experimental measurements, or real-world observations. Symbolic regression, however, poses several challenges, including the combinatorial nature of the large optimization search space, vulnerability to the quality of input data, and the difficulty of striking a balance between model fitting, complexity, and generalization performance [10, 11].

Symbolic regression encompasses a wide range of methods, spanning different categories. Traditional approaches, such as Genetic Programming (GP), use a heuristic population-based search strategy where each individual represents a potential solution to the problem [12, 13]. Though GP algorithms are capable of finding solutions for nonlinear and complex problems, they are typically slow to converge due to the vast functional search space. Also, as they need to start the search from scratch for each dataset, they tend to be computationally expensive, prone to overfitting, and sensitive to the

DL-based SR

GP-based SR

ML methods

20

XGB

AdaBoost

RandomForest

OMLP

?

MRGP*

20

SBP-GP

MRGP

AIFeynman*

FEAT

FFX

Q

ComplexityRank

*TPSR(入=0.1）Q

*TPSR(A=0)

SBP-GP*

KernelRidge

FFX*

ComplexityRank

15

Operon

TPSR (A= 0)

*TPSR(A=0.5）O

*TPSR(入=1）

*E2E

Q

ITEA*

TPSR(A=0.1)

EPLEX

10

TPSR(A=0.5）

E2E

O

10

FEAT*

Operon*

EPLEX*

TPSR(A=1)

AFP_FE

GP-GOMEA

AFP

GP-GOMEA*

AFP*

OQAFP_FE*

5

ITEA

BSR

O

gplearn*

BSR*

Linear

gplearn

DSR

MedianR²Rank

10

15

20

0

0

5

10

15

20

(a)Black-boxdataset

(b)Feynman dataset

choice of parameters [14]. Recent works in SR have shown promising results by using pre-trained transformers [15] for generating equations as sequences of tokens. These models leverage the prior knowledge learned through large-scale pre-training and can generate equations with a single forward pass, leading to considerably faster inference time compared to the GP-based methods [16-19]. However, one of the limitations of these models is that they focus on the supervised language model pre-training objective borrowed from text generation, i.e., they are trained solely with the token-level cross-entropy loss, which can result in equations that may exhibit high token-level similarities but are suboptimal with respect to equation discovery objectives such as fitting accuracy and complexity which are critical in this task. To mitigate this issue, beam search [20, 21] or sampling [22] approaches have been employed as decoding strategies to propose multiple candidate equations for a given dataset, and then select the optimal candidate equation based on the fitting accuracy after optimizing for constants. Nonetheless, both beam search and sampling decoding strategies primarily rely on the pretrained transformer's logits and next token probabilities, and therefore do not receive any performance feedback during the generation of equation candidates.

To consider the equation discovery objectives in the transformer generation process and still benefit from the pre-trained model logits, we propose TPSR, a T ransformer-based P lanning strategy for S ymbolic R egression. TPSR leverages a lookahead planning algorithm, using Monte Carlo Tree Search (MCTS) as a decoding strategy on top of pre-trained transformer SR models to guide equation sequence generation. TPSR significantly improves performance of the discovered equations by considering verification feedback during the generation process and still remains considerably faster than GP-based models which do not leverage the pre-training priors and learn expressions for each dataset from scratch. Notably, our approach is model-agnostic and can be applied to any pre-trained SR model, enabling optimization of generated equation sequences for non-differentiable objectives that may encompass combinations of fitting accuracy, complexity, and equation forms. Additionally, we incorporate different caching mechanisms to reduce the overall inference time. Our experimental results demonstrate that applying TPSR on top of the pre-trained E2E SR model [18] significantly enhances its performance across various benchmark datasets. As depicted in Fig. 1, TPSR achieves a strong balance between fitting accuracy and model complexity compared to other leading baselines. It also effectively drives the E2E model towards the optimal trade-off, represented by the first Pareto front. The major contributions of this work are summarized below:

Proposing TPSR, a new method that combines pre-trained transformer SR models with Monte Carlo Tree Search (MCTS) lookahead planning to optimize the generation of equation sequences while considering non-differentiable performance feedback.

Developing a new reward function that balances equation fitting accuracy and complexity to optimize the generated equations for an effective trade-off.

Demonstrating that TPSR consistently outperforms state-of-the-art baselines across various SR benchmark datasets, generating equations with higher fitting accuracy while maintaining lower complexity to avoid non-parsimonious solutions. TPSR still achieves considerably faster inference time than GP-based models which do not use pre-trained priors.

Showcasing the extrapolation and noise robustness of TPSR compared to the baseline and conducting an ablation study to investigate the impact of various model components.

2 Related Work

Symbolic Regression without Learned Priors. Genetic Programming (GP) algorithms are typically employed for single-instance SR, aiming to find the best-fit equation for a "single" dataset at hand [12]. Recently, alternative neural network-based search algorithms have been explored, including deep reinforcement learning (RL) [14, 23, 24], combinations of GP and RL [25], and Monte Carlo Tree Search (MCTS) as a standalone framework [26]. Despite their successes, all these methods lack the benefits of prior knowledge learned from large-scale pre-training. Consequently, they are slow during inference as they need to restart the search from scratch for new datasets.

Pre-trained Transformers for Symbolic Regression. In recent years, pre-trained transformers have shown remarkable performance in natural language and programming language tasks [2729]. This success has inspired researchers to develop pre-trained transformer models for SR [1619, 30]. For example, Biggio et al. [16] introduced a Neural Symbolic Regression model that scales (NeSymReS) with the amount of synthetic training data and generates equation skeletons where all the numerical constants are represented by a single token ' C '. Kamienny et al. [18] proposed an end-to-end transformer SR framework that predicts the complete equation form along with its constants. More recent works [30, 31] introduced unified frameworks that include a transformerbased pre-training stage as the prior for subsequent RL or GP optimization steps. While GP and RL methods have to start anew for each problem, the transformer approaches rely on synthetic data and the power of large-scale pre-trained priors to generate equations in a single forward pass. However, these models are pre-trained on token-level language modeling loss function and thus can perform suboptimal for other equation discovery objectives critical in SR such as fitting accuracy to the observed data as well as equation's complexity. Our model, TPSR, utilizes lookahead planning to guide the generation of equations towards better performance by employing these objectives as feedback during the transformer generation process.

Planning in Sequence Generation. Recently, planning algorithms have been utilized in NLP tasks to optimize text output for specific objectives, such as controlling generated text to meet certain constraints like non-toxicity or conveying certain emotions [32-34]. Recent advances in programming language models developed in code generation have also yielded promising techniques that could be adapted for SR, as they share several vital similarities with each other. Both involve generating sequences of symbols for a given input and typically require optimizing the generated sequences for specific criteria which is different from the pre-trianing objective. For code generation, this may involve optimizing objectives like code compilability, readability, or passing test cases [35-37]. Similarly, in SR, the focus may be on equation-specific sequence-level objectives such as fitting accuracy or minimizing complexity. Motivated by these successes, we develop an approach that combines MCTS planning with pre-trained transformer SR models for improved equation discovery.

3 Methodology

3.1 Preliminaries

In SR, the main goal is to find a symbolic expression for the unknown function f ( · ) mapping the d -dimensional input x ∈ R d to the target variable y = f ( x ) ∈ R . Given a dataset of n observations D = ( x i , y i ) n i =1 , SR methods try to generate an equation ˜ f ( · ) such that y i ≈ ˜ f ( x i ) for all i ∈ N n . Also, the proposed equation is desired to generalize well and to effectively balance the fitting accuracy and complexity. The transformer SR models are trained on a large-scale dataset comprising equation instances paired with their corresponding observations, { ( D 1 , f 1 ( · )) . . . ( D M , f M ( · )) } , where M is the dataset size (number of paired samples). During inference, the trained model directly generates the equation ˜ f ( · ) as a sequence of tokens in an autoregressive manner. An effective way to represent the expression tree of equations in a sequence is to use prefix notation as in [38]. For embedding the

(1)

...

y1

(1)

...

xi

(d)！

iy1

x(1)

...

xi

(d)

1

iy

：

：

：

！:

：

xn

x(1)

Xn

(d)

yn

xn

x(1)

xn

(d)！

iyn

ux

(1)

Xn

(d)

1

iyn

Dataset

Embedder

Embedder

Embedder

f(x) =[+,cos,x,×,2,x]

Encoder

Encoder

BeamSearch

orSampling

Encoder

Decoder

<start>

Sorting

<start>

MCTS-GuidedDecoder

Reward

Decoder

f(x)

L

Optimization

×

Decoder

Fitting (y,y)

Token-levelCrossEntropyLoss

Complexity

y=f(x)

(a) Training

(b)Inference withBeam Search/Sampling

(c)InferencewithMCTS

observations, we adopt the pre-trained SR model backbone from [18]. Notably, given the potential for large input sequences with tokenized numeric data and the quadratic complexity of transformers, the method introduced in [18] deploys a linear embedder module to map tokenized inputs to a singular embedding space before introducing them to the transformer encoder and decoder. Subsequent to embedding, these models encode the input observations and then pass the encoded representation along with the masked tokens to decode the equation sequence. To train the model, token-level cross-entropy loss with teacher forcing is employed to learn the distribution of next token prediction conditioned on the encoded dataset and the current state of sequence (Fig. 2(a)).

Achieving a good fitting performance from the model's predicted sequence demands generating accurate constants in the equation. To address this, the generated skeleton or equation can undergo a round of optimization to estimate their constants using nonlinear methods, such as Broyden-Fletcher-Goldfarb-Shanno algorithm (BFGS) [39]. Previous works [18, 16] employ beam search and sampling strategies for transformer decoding in combination with constant optimization to propose several candidate equations. Subsequently, they use fitting metrics such as R 2 to order these candidates and output the final equation with the best performance (Fig. 2(b)). Transformer models utilizing beam search or sampling decoding strategies can generate multiple high-likelihood equation sequences, but their generation process is based on logits obtained from model parameters pre-trained with token-matching loss relative to the reference equation. As a result, such models lack the capability to receive verification feedback during generation and optimize sequence for equation discovery objectives such as fitting or complexity of equations.

3.2 MCTS-Guided Equation Generation

To generate equations that are both better-fitting and less-complex, it is crucial to incorporate feedback into the equation generation process. To achieve this, we utilize Monte Carlo Tree Search (MCTS) lookahead planning during inference, guiding the decoder towards optimal solutions for fitting and complexity objectives (as shown in Fig. 2(c)). The MCTS-guided transformer decoding explores different possibilities, identifying the most promising paths based on the objectives.

We frame the SR equation generation task as a Markov Decision Process (MDP) where state s represents the current sequence at generation step (token) t . If s has not reached the terminal state (i.e., the < EOS > token), we select the next token from the vocabulary as action a , updating state s ′ by concatenating s and a . Upon reaching the terminal state, the reward r is computed and used to update the decoding model. MCTS represents states as nodes and actions as edges within a tree structure, navigating state-space from the root node (i.e., initial state) to reach terminal states with maximum rewards. MCTS balances exploration and exploitation, considering nodes that lead to higher quality equations (i.e., higher Q-values) and under-explored nodes (i.e., those with fewer visits). During the generation process of the transformer, we utilize the MCTS algorithm iteratively to conduct lookahead planning and determine the next token. However, the large search-space requires more than the sole application of MCTS to discover high-quality equations. We need to effectively share information between the pre-trained transformer model and MCTS for better generations. To achieve this, we incorporate the probabilities of the next-token that are acquired through the pre-trained

Repeat

Selection

Expansion

Evaluation

Backpropagation

(<start>

<start>

<start>

(<start>

×

十

×

+

+

×

+,x1

+,x1

(+,sin)

+,x1

+,sin)

+,x1

(+,sin)

+,sin, x2

+,sin,x

(1)

1y1

ExtractTop-K

：

：

1:

Tokens

(1)

Embedder

Generate

f:[+, sin, x2,.,x1]

Reward (f)

Decoder

Sequencewith

Encoder

BeamSearch

transformer SR models into the MCTS planning process. This incorporation helps to enhance the search process, leading to more efficient and effective results. The key steps of MCTS for transformer decoding in SR models, as depicted in Fig. 3, are as follows:

Selection. The Upper Confidence Bound for Trees [40] criterion is employed to select actions (i.e., next tokens) for fully extended nodes in the search tree, balancing exploration and exploitation. We use the P-UCB heuristic [41] as

where Q ( s, a ) is the maximum return for action a in state s across all simulations, promoting the exploitation of the optimal child node. The second term encourages exploration of less-visited children, with N ( s ) as state s 's visit count and s ′ as the subsequent state. P θ ( a | s ) is the probability of the next token a given the partial sequence state s from pre-trained transformer model parameterized by θ . The exploration-exploitation trade-off is adjusted by hyperparameter β . Lastly, the next token action maximizes the P-UCB: Select( s ) = arg max a P -UCB( s, a ) .

Expansion. In the expansion stage, after selecting a node that is not fully expanded, a new child (next token) for the current state is explored. Random expansion of the node from the vocabulary, however, might result in an invalid equation (that does not comply with the prefix notation) and makes the search process very time-consuming. Therefore, given partial equations, only top -k most likely choices of the next token are considered as the possible children of the node for expansion. In other words, we are restricting the actions to be only from the top -k high-likelihood options which are retrieved from the pre-trained transformer SR model's logits. These options are then ordered to determine the sequence in which the children will be expanded.

Evaluation. To evaluate the newly expanded nodes, we perform simulations to complete the equation sequence. This is necessary because the new state may still be a partial equation and performance feedback can only be obtained at the end of the sequence when the equation generation is completed. In MCTS, it is common to employ random actions during the simulation stage. Nevertheless, random action selection for equation generation, much like during expansion, suffers from certain drawbacks in terms of time and the possibility of generating invalid equations. Consequently, the pre-trained transformer SR model is invoked again, this time utilizing beam search with a beam size of b , to generate complete equation candidates based on the current state. The beam size b determines the number of complete equations to be generated from the current partial equation. Following these simulations, the highest reward among all the complete equation candidates is assigned to the new node value.

Backpropagation. After generating a complete equation ˜ f ( · ) , the corresponding reward r ( ˜ f ( · )) can be computed. The highest reward among all simulations is then assigned to the new node, which recursively backpropagates its estimated value to its parents until it reaches the root of the tree. This

update process involves updating the Q values of all state-action pairs, denoted as s ′ and a ′ , along the trajectory in the tree to reach the root. Specifically, for each state-action pair, the Q value is updated by taking the maximum of the current Q value and the new value r : Q ( s ′ , a ′ ) ← max( Q ( s ′ , a ′ ) , r ) . More details on TPSR, including its steps and implementation can be found in Appendix C.

3.3 Reward Definition

We define a numerical reward r ∈ R to evaluate complete equation candidate ˜ f ( · ) , promoting fitting accuracy and regulating complexity. After optimizing constants in the complete sequence, we compute the reward. We first calculate the normalized mean squared error (NMSE) between ground-truth target variable y and predicted target variable ˜ y = ˜ f ( x ) , and formulate the reward as:

where l represents equation complexity as the sequence length in prefix notation [18, 42, 16]; L denotes the model's maximum sequence length; and λ is a hyperparameter balancing fitting and complexity reward terms. Higher λ values favor less complex equations, encouraging best-fitting and penalizing non-parsimonious solutions. NMSE is calculated as ( 1 n ∥ ∥ y -˜ f ( x ) ∥ ∥ 2 2 ) / ( 1 n ∥ ∥ y ∥ ∥ 2 2 + ϵ ) , where ϵ is a small constant to prevent numerical instability.

3.4 Efficient Implementation with Caching

During MCTS evaluation, the transformer model generates complete equation sequences from a given state, constructing implicit tree structures for beam search and computing top -k next tokens for visited states. These computations are required in future MCTS iterations, so we employ two caching mechanisms, top -k caching and sequence caching , to reduce redundancy and improve efficiency. Top -k caching stores computed top -k values for given states. For example, in Fig. 4, when evaluating state s = [+ , sin] in MCTS iteration t , top -k tokens are computed for s and subsequent visited states, such as [+ , sin , x 2 ] . State and top -k value pairs are cached for future use, avoiding re-

Iterationt

1

Iteration t+1

Evaluation

1

Expansion

Evaluation

1

<start>

1

<start>

<start>

1

1

+

×

+

×

+

×

一

+,x

+,sin

一

+,x1

+,sin

+,x

+,sin

1

+,sin, x2

+,sin,x

=

(+,sin,x2)

(+,sin,x2

Sequence

：

Cache

Top-k

Cache

f：[+,sin,x2,…,x]

f:[+, sin,x,….,x]

dundant token retrieval. Sequence caching caches complete equations generated with the provided beam size. If a state matches a stored equation partially, the cached equation can be used directly in future iterations, bypassing iterative sequence generation. Both caching strategies are designed to enhance efficiency without compromising performance. More details are provided in Appendix C.

4 Experiments

In this section, we present our experimental results that evaluate the effectiveness and efficiency of TPSR. While the proposed decoding strategy is generally model-agnostic, here we showcase the results of using TPSR for the end-to-end (E2E) pre-trained SR transformer backbone [18], as E2E is the state-of-the-art open-source pre-trained SR model with publicly accessible model weights. Additional results of using TPSR with the NeSymReS pre-trained SR backbone [16] can be found in Appendix D.4. We evaluate our framework by answering the following research questions ( RQs ):

RQ1. Does TPSR perform better than other decoding strategies (beam search/sampling) and competing baseline methods over standard SR benchmark datasets?

RQ2. Does TPSR provide better extrapolation and robustness to noise?

RQ3. Are TPSR's caching mechanisms effective in reducing computation time?

RQ4. What is the role of individual components in TPSR's overall performance gain?

4.1 Datasets

We evaluate TPSR and various baseline methods on standard SR benchmark datasets from Penn Machine Learning Benchmark (PMLB) [43] studied in SRBench [42], as well as In-domain Synthetic Data generated based on [38, 18]. The benchmark datasets include 119 equations from Feynman Lectures on Physics database series 2 [44], 14 symbolic regression problems from the ODE-Strogatz database 3 [45], and 57 Black-box 4 regression problems without known underlying equations. We limit the datasets to those with continuous features and input dimension d ≤ 10 , as the transformer SR model [18] is pre-trained with d max = 10 . The In-domain Synthetic Data consists of 400 validation functions with different levels of difficulty and number of input points. This data is referred to as "in-domain" because the validation functions and their corresponding input points are generated using the same approach as the data on which the backbone transformer model [18] is pre-trained. More details on each of these datasets are provided in Appendix A.

4.2 Evaluation Metrics

We evaluate our model using the following three metrics: R 2 score [42], accuracy to tolerance ω [16, 46], and equation complexity [18, 42].

where R 2 measures fitting performance with ¯ y as the mean of y in test set, Acc ω evaluates equation precision based on tolerance threshold ω , and equation complexity is determined by the number of nodes in the expression tree T of the generated equation ˜ f ( · ) . Following [18, 42], we set R 2 = 0 for rare pathological examples and discard the worst 5% predictions for Acc ω to reduce outlier sensitivity.

4.3 (RQ1) Effectiveness of TPSR

Table 1 presents the performance comparison results of TPSR with the baseline decoding strategies on the SRBench benchmark and the In-domain synthetic dataset. For the E2E baseline, we use the settings reported in [18], including beam/sample size of C = 10 candidates, and the refinement of all the candidates K = 10 . For our model, we use the width of tree search as k max = 3 , number of rollouts r = 3 , and simulation beam size b = 1 as the default setting. For PMLB datasets that contain more than 200 points, we follow [18] and use B bags of data, each containing N = 200 points, due to the limitation that the baseline method is pre-trained with N ≤ 200 data points. In the baseline method [18], a total of BC candidates are generated ( C candidates for B bags), which are then sorted and refined to generate the best equation. However, for TPSR, since we need to train an MCTS for each bag, we use an iterative decoding approach, starting with the first bag and continuing

0.0

0.2

0.4

0.6

0.8

1.0

SBP-GP

Operon

TPSR

(

λ

=0)

TPSR

(

λ

=0.1)

GP-GOMEA

TPSR

(

λ

=0.5)

TPSR

(

λ

=1)

EPLEX

E2E

MRGP

AFP_FE

AIFeynman

FEAT

AFP

gplearn

FFX

DSR

ITEA

BSR

R

2 >0.99)

10 1

10 2

10 3

10 4

10 5

Model Complexity

10 1

10 2

10 3

10 4

Inference Time (s)

Target Noise

0.0

0.001

0.01

0.1

-0.5

0.0

0.5

1.0

*TPSR

(

λ

=0.1)

*TPSR

(

λ

=0)

*Operon

*TPSR

(

λ

=0.5)

*TPSR

(

λ

=1)

*SBP-GP

*FEAT

*EPLEX

*E2E

XGB

*GP-GOMEA

AdaBoost

RandomForest

*ITEA

*AFP_FE

*AFP

*FFX

KernelRidge

*DSR

*MRGP

*gplearn

MLP

Linear

*BSR

*AIFeynman

Median

R

2

10 2

10 4

Model Complexity

10 0

10 2

10 4

Inference Time (s)

with subsequent bags until a criterion ( R 2 > 0 . 99 ) is met or we use a maximum of B = 10 bags. To ensure a fair comparison, we use B = 10 for the E2E baseline method as well. In this table, we demonstrate the results of our proposed framework, TPSR, with varying values of the λ parameter that controls the trade-off between fitting performance and complexity in the hybrid reward function defined in Eq. (2). For a detailed comparison of the experimental settings across different approaches, refer to Table 2 in Appendix B.

As shown in Table 1, when λ = 0 , the framework generates complex equations that overoptimize for fitting performance. However, as we increase λ , the framework generates less complex equations with a slight reduction in fitting performance. Notably, even for large values of λ , such as λ = 1 , the fitting performance of TPSR significantly outperforms that of the baseline methods. Based on the results, we recommend a default setting of λ = 0 . 1 as it offers a balanced trade-off between complexity and accuracy, while also mitigating potential overfitting (as detailed in Appendix D.1). These findings demonstrate the superiority of TPSR over the baseline methods in terms of fitting performance across all datasets, while generating equations with comparable or reduced complexity than those generated by the baseline methods. Table 1 shows that TPSR exhibits a more significant gap in fitting performance when compared to E2E baselines on SRBench datasets, while this gap is smaller for In-domain datasets (even performing slightly worse on Acc ω for larger λ = 0 . 5 , 1 ). This is due to the In-domain dataset being generated using the same approach as the E2E pre-training data, resulting in the E2E model's superior performance on this synthetic dataset. Furthermore, qualitative comparisons of TPSR with baseline symbolic and black-box regression models demonstrate the

TPSR (A=0.0)

TPSR (A=0.1)

TPSR (A=0.5)

TPSR (A=1.0)

J1

E2E

Interpolation

0.95

0.60

Extrapolation

0.90

0.55

9

0.85

0

V

0.50

2

0.80

0.45

0.75

A

0.70

0.40

0.65

0.35

0.60

Train

1

2

4

8

16

0.0

0.001

0.01

0.1

b

Target NoiseY

(a)

(b)

superior performance of TPSR in learning the underlying equation and out-of-domain extrapolation (see Appendix D.3).

Fig. 5 presents a detailed comparison of our proposed TPSR with the baseline E2E transformer model and all the SRBench baselines on the PMLB Feynman and Black-box datasets. This figure illustrates the relative position of each algorithm with respect to (1) fitting performance, (2) model complexity, and (3) inference time. The results indicate that transformer-based planning in the TPSR significantly enhances the performance of E2E and outperforms even the state-of-the-art GP baselines, achieving the highest fitting performance on the black-box datasets. This is achieved while the complexity of the generated equations in TPSR is not greater than that of E2E, and shows a great fitting-complexity-time balance compared to other SR algorithms. The pareto plots provided in Fig. 1 and Appendix D.2 also demonstrate the effectiveness of TPSR in balancing fitting-complexity as well as fitting-time compared to all other SRBench baselines. Our TPSR effectively pushes this balanced performance to the first pareto front for both the Feynman and Black-box datasets. Moreover, it is important to note that, while the inference time of TPSR is longer than the baseline E2E transformer model, it still has significantly lower inference time than RL or GP-based SRBench baselines. Further results on the SRBench and In-domain datasets are provided in Appendix D.

4.4 (RQ2) Extrapolation and Robustness

The ability to extrapolate well is inherently linked to the quality of the equations discovered through symbolic regression. To investigate the extrapolation performance of TPSR to out-of-training regions, we normalize the input test data points to different scales ( σ ) instead of unit variance (used for training points) as per [18]. Fig. 6(a) depicts the average performance of TPSR compared to E2E with sampling decoding on the training data as well as testing data in scales of σ ∈ { 1 , 2 , 4 , 8 , 16 } for the In-domain Synthetic dataset. Also, we investigate the effect of different complexity controlling levels ( λ ∈ { 0 , 0 . 1 , 0 . 5 , 1 . 0 } ) on the extrapolation performance. It can be observed that, while λ = 0 (i.e., no complexity regularization) achieves the best fitting accuracy on the training data, it has a sub-par performance for σ > 8 . This can be due to the overfitting issue when the symbolic model is much more complex than the true function, similar to the common overfitting issue in ML models. Results highlight the importance of controlling complexity in the extrapolation of identified equations. For values of λ > 0 , the overfitting issue is mitigated as the generated equations become less complex. However, very high values of λ (e.g., λ = 1 ) mostly result in poor accuracy performance. The flexibility of TPSR for allowing different values of λ to balance fitting and complexity for a given task is crucial for optimal performance. Fig. 6(b) also presents the robustness of TPSR with different λ levels compared to the E2E transformer baseline on the Feynman dataset. The results indicate that MCTS-guided lookahead planning can offer robust performance with a smaller drop in accuracy compared to the baseline in the presence of noise.

4.5 Ablation Study

In this section, we investigate the effect of different MCTS parameters and caching mechanisms on the performance of TPSR by conducting ablation experiments on the Feynman datasets.

TPSR (varying rollout)

TPSR (varying Kmax)

TPSR (varying b)

→ E2E

180

0.96

160

9

0.94

140

9

0

0.92

(sec.)

120

A

2

0.90

Time

100

80

0.88

60

0.86

40

0.84

20

0.82

No

Seq.

Seq.+ TopK

0

50

100

150

200

250

300

350

400

Caching Methods

#of Generated Candidates

(a)

(b)

(RQ3) Caching Mechanisms. In Fig. 7(a), we illustrate the effectiveness of the sequence and top -k caching mechanisms in reducing the total inference time of TPSR ( λ = 0 . 1 ). Our experiments show that sequence caching has more effect in dropping the inference time as it replaces the time-consuming sequence generation process. Overall, these two mechanisms can reduce the total inference time by around 28% .

(RQ4) Search Parameters. Fig. 7(b) shows the accuracy performance against the number of generated equations during the decoding process for both TPSR ( λ = 0 . 1 ) and the baseline E2E with sampling decoding. In this figure, the 'number of generated equation candidates' represents the total number of complete equation sequences generated by each method. Specifically, this refers to the sample size in the E2E with sampling decoding, and the function calls of the beam search sub-routine multiplied by beam size b in TPSR. The results show that under the same number of generated equation candidates, TPSR significantly outperforms the E2E baseline. This is primarily attributed to the fact that the E2E baseline is deprived of any performance feedback during the equation generation process. We report the results for variants of TPSR with different MCTS parameters. We assess the performance with varying number of rollouts, r = { 1 , 3 , 6 , 9 } , number of beams in simulations, b = { 1 , 3 } , and the maximum number of possible expansions at each state, k max = { 2 , 3 , 4 } . The default setting of TPSR parameters are b = 1 , k max = 3 , and r = 3 . Results indicate that increasing r , k max , and b all contribute to the better performance of TPSR, with the most significant improvement observed when increasing r . This is because more rollouts provide model with more opportunities to learn from trials and learn better values.

5 Conclusion

In this work, we propose TPSR, a model-agnostic decoding strategy for symbolic regression that leverages the power of pre-trained SR transformer models combined with MCTS lookahead planning, and outperforms the existing methods in generating equations with superior fitting-complexity-time trade-off. We demonstrate the flexibility of TPSR in controlling discovered equation complexity without fine-tuning the pre-trained model. We also show that TPSR performs 100 x faster than state-of-the-art genetic algorithms by leveraging the pre-trained priors. Additional results show that better expressions obtained with lookahead planning can further improve model performance in terms of noise robustness and extrapolation to unseen data. Future research could focus on enhancing the adaptability of feedback-based expression generation mechanisms, potentially by modulating the flexibility of MCTS or transformer model weights, and the integration of MCTS with the training or fine-tuning of transformer SR models. Furthermore, employing parallelization and distributed computing could potentially improve planning efficiency.

References

He Ma, Arunachalam Narayanaswamy, Patrick Riley, and Li Li. Evolving symbolic density functionals. Science Advances , 8(36):eabq0279, 2022.

Yiqun Wang, Nicholas Wagner, and James M. Rondinelli. Symbolic regression in materials science. MRS Communications , 9(3):793-805, 2019.

Samuel H. Rudy, Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Data-driven discovery of partial differential equations. Science Advances , 3(4):e1602614, 2017.

Kazem Meidani and Amir Barati Farimani. Data-driven identification of 2d partial differential equations using extracted physical features. Computer Methods in Applied Mechanics and Engineering , 381:113831, 2021.

Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences , 113(15):3932-3937, 2016.

Miles Cranmer, Alvaro Sanchez-Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS'20, Red Hook, NY, USA, 2020.

E. Kaiser, J. N. Kutz, and S. L. Brunton. Sparse identification of nonlinear dynamics for model predictive control in the low-data limit. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences , 474(2219):20180335, 2018.

Kazem Meidani and Amir Barati Farimani. Identification of parametric dynamical systems using integer programming. Expert Systems with Applications , 219:119622, 2023.

Yoshitomo Matsubara, Naoya Chiba, Ryo Igarashi, Taniai Tatsunori, and Yoshitaka Ushiku. Rethinking symbolic regression datasets and benchmarks for scientific discovery. arXiv preprint arXiv:2206.10540 , 2022.

Marco Virgolin and Solon P Pissis. Symbolic regression is NP-hard. Transactions on Machine Learning Research , 2022.

Haoran Li, Yang Weng, and Hanghang Tong. CoNSole: Convex neural symbolic learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science Advance , 324(5923):81-85, 2009.

Bogdan Burlacu, Gabriel Kronberger, and Michael Kommenda. Operon c++: An efficient genetic programming framework for symbolic regression. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion , GECCO '20, page 1562-1570, New York, NY, USA, 2020. Association for Computing Machinery.

Brenden K Petersen, Mikel Landajuela Larma, Terrell N. Mundhenk, Claudio Prata Santiago, Soo Kyung Kim, and Joanne Taery Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations , 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30, 2017.

Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo. Neural symbolic regression that scales. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 936-945. PMLR, 18-24 Jul 2021.

Mojtaba Valipour, Bowen You, Maysum Panju, and Ali Ghodsi. Symbolicgpt: A generative transformer model for symbolic regression. arXiv preprint arXiv:2106.14131 , 2021.

Pierre-Alexandre Kamienny, Stéphane d'Ascoli, Guillaume Lample, and Francois Charton. End-to-end symbolic regression with transformers. In Advances in Neural Information Processing Systems , 2022.

Martin Vastl, Jonáš Kulhánek, Jirí Kubalík, Erik Derner, and Robert Babuška. Symformer: End-to-end symbolic regression using transformer-based architecture. arXiv preprint arXiv:2205.15764 , 2022.

Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711 , 2012.

Sam Wiseman and Alexander M. Rush. Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing , pages 1296-1306, Austin, Texas, November 2016. Association for Computational Linguistics.

Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 889-898, Melbourne, Australia, jul 2018. Association for Computational Linguistics.

Allan Costa, Rumen Dangovski, Owen Dugan, Samuel Kim, Pawan Goyal, Marin Soljaˇ ci´ c, and Joseph Jacobson. Fast neural models for symbolic regression at scale. arXiv preprint arXiv:2007.10784 , 2020.

Mikel Landajuela, Brenden K Petersen, Sookyung Kim, Claudio P Santiago, Ruben Glatt, Nathan Mundhenk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep reinforcement learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 5979-5989. PMLR, 18-24 Jul 2021.

Terrell N. Mundhenk, Mikel Landajuela, Ruben Glatt, Claudio P. Santiago, Daniel faissol, and Brenden K. Petersen. Symbolic regression via deep reinforcement learning enhanced genetic programming seeding. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Fangzheng Sun, Yang Liu, Jian-Xun Wang, and Hao Sun. Symbolic physics learner: Discovering governing equations via monte carlo tree search. In The Eleventh International Conference on Learning Representations , 2023.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 1877-1901. Curran Associates, Inc., 2020.

Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 8696-8708, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

Mikel Landajuela, Chak Lee, Jiachen Yang, Ruben Glatt, Claudio P. Santiago, Ignacio Aravena, Terrell N. Mundhenk, Garrett Mulcahy, and Brenden K. Petersen. A unified framework for deep symbolic regression. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022.

Samuel Holt, Zhaozhi Qian, and Mihaela van der Schaar. Deep generative symbolic regression. In The Eleventh International Conference on Learning Representations , 2023.

Thomas Scialom, Paul-Alexis Dray, Jacopo Staiano, sylvain lamprier, and Benjamin Piwowarski. To beam or not to beam: That is a question of cooperation for language GANs. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021.

Rémi Leblond, Jean-Baptiste Alayrac, Laurent Sifre, Miruna Pislar, Lespiau Jean-Baptiste, Ioannis Antonoglou, Karen Simonyan, and Oriol Vinyals. Machine translation decoding beyond beam search. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 8410-8434, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

Antoine Chaffin, Vincent Claveau, and Ewa Kijak. PPL-MCTS: Constrained textual generation through discriminator-guided MCTS decoding. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 29532967, Seattle, United States, July 2022. Association for Computational Linguistics.

Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In International Conference on Learning Representations , 2019.

Parshin Shojaee, Aneesh Jain, Sindhu Tipirneni, and Chandan K Reddy. Execution-based code generation using deep reinforcement learning. arXiv preprint arXiv:2301.13816 , 2023.

Shun Zhang, Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. Planning with large language models for code generation. In International Conference on Learning Representations , 2023.

Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations , 2020.

Roger Fletcher. Practical Methods of Optimization . John Wiley & Sons, New York, NY, USA, second edition, 1987.

Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In Johannes Fürnkranz, Tobias Scheffer, and Myra Spiliopoulou, editors, Machine Learning: ECML 2006 , pages 282-293, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabricio de Franca, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason Moore. Contemporary symbolic regression methods and their relative performance. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , volume 1, 2021.

Randal S. Olson, William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz, and Jason H. Moore. Pmlb: a large benchmark suite for machine learning evaluation and comparison. BioData Mining , 10(1):36, Dec 2017.

Silviu-Marian Udrescu, Andrew Tan, Jiahai Feng, Orisvaldo Neto, Tailin Wu, and Max Tegmark. Ai feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 4860-4871. Curran Associates, Inc., 2020.

William La Cava, Kourosh Danai, and Lee Spector. Inference of compact nonlinear dynamic models by epigenetic local search. Engineering Applications of Artificial Intelligence , 55:292-306, 2016.

Stéphane D'Ascoli, Pierre-Alexandre Kamienny, Guillaume Lample, and Francois Charton. Deep symbolic regression for recurrence prediction. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 4520-4536. PMLR, 17-23 Jul 2022.

Pierre-Alexandre Kamienny, Guillaume Lample, Sylvain Lamprier, and Marco Virgolin. Deep generative symbolic regression with Monte-Carlo-tree-search. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 15655-15668. PMLR, 23-29 Jul 2023.

Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics , 29:1189-1232, 2000.

Peer reviews:
comments: SUMMARY:
The paper introduces TPSR, a Transformer-based Planning strategy for Symbolic Regression. TPSR incorporates Monte Carlo Tree Search into the transformer decoding process, enabling the integration of non-differentiable feedback such as accuracy and complexity. Experimental results show that TPSR outperforms existing methods in terms of fitting-complexity trade-off, extrapolation abilities, and robustness to noise.

STRENGTHS:
- The paper is well written and easy to understand.
- The idea of enhancing large scale pre-trained Transformers with improved search capablities is very promising in the context of symbolic regression
- The model shows good performance both compared to the E2E baseline and the GP methods.

WEAKNESSES:
- My main concern is about the novelty of the approach. A very similar idea has been recently investigated in [1] where the authors also combine MCTS with pre-trained Transformers. I would be grateful if the authors could clarify any eventual differences between the two approaches.
- The impact of $\lambda$ seems quite significant in your experiements. However, it is not clear to me how one should select it in practice.


[1] Kamienny, Pierre-Alexandre, Guillaume Lample, and Marco Virgolin. "Deep Generative Symbolic Regression with Monte-Carlo-Tree-Search." (2023).

QUESTIONS:
Please refer to the weakness part above.

LIMITATIONS:
n/a
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-02

comments: SUMMARY:
Authors propose a transformer-based planning (using MCTS) strategy to solve symbol regression task. Different from traditional decoding method, the new method is able to integrate non-differentiable feedback into the transformer-based process of equation generation. Experiments demonstrate the significent performance.

STRENGTHS:
Distilling symbolic equation from noisy data is intractable. Recent progress is achieved by training neural networks to generate candidate symbolic expressions, which is really promising.  

This work combines the Monte Carlo Tree Search and pretrained transformer-based symbol regression model for equation generation. Compared with Genetic programming method, the new approach not only leverages pre-trained priors, but also considers feedbacks during the generation process.

WEAKNESSES:
There is not much initiality in the new method. It demonstrates a new application for a combination of two existing methods.

QUESTIONS:
Are there experiments to show changes of performances, if we change the selection set of mathematical operators and symbols?
 
Can out-of distribution data be identified, and used for the promotion of the symbolic regression process?

It might not be diffiuclt for symbol regression method to find laws, such as f=ma. Could it find E = m c^2?

LIMITATIONS:
Monte Carlo Tree Search is statistical, and Pre-trained transformer is trained through data, the integration of the two methods is still within the traditional paradigm of machine learning, so, may not work well for out-of distribution data.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to incorporate  Monte Carlo Tree Search (MCTS) on top of pretrained transformer-based SR models to guide equation sequence generation. This is to address the challenges where existing methods purely rely on the pretrained transformer’s output and without accounting for external performance requirement. In MCTS, the authors develop a reward function to encourage the balance between fitting accuracy and regulating complexity for the SR generation. Also, the caching tricks are employed to improve the implementation efficiency. SR benchmark datasets are used to demonstrate the improved performance of the proposed method over the state-of-the-art.\n\nSTRENGTHS:\nIncluding performance feedback in the pipeline of generation of SR equation generation from pre-trained transformer-based SR models is well-motivated.  To achieve so, this paper proposes including MCTS as the decoder in this pipeline, and imparting    the external requirement, via a reward function in MCTS, to eventually improve the performance of equation generation. The extensive experiments and baseline comparison clearly show the effectiveness of the proposed method, in terms of fitting-complexity trade-off, extrapolation abilities, and robustness to noise. \n\nThe presentation of techniques is clear, and the evaluation in my opinion is solid. Overall, this paper makes a good contribution in the SR field.\n\nWEAKNESSES:\nI only have two comments:\n\n- In Equ. (1), how to select $\\beta(s)$? It would be better to show its effect on the performance in the ablation study as well.\n- Currently, the method still relies on a pre-trained transformer SR model. The authors could give some perspective about how (or if it is possible) the MCTS can also be incorporated in the transformer training (or fine-tuning) process.\n-A typo in line 151: trnasformer--> transformer\n\nQUESTIONS:\nSee my comments in the above section.\n\nLIMITATIONS:\nSee my comments in the above section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission proposes a neural network-based approach to symbolic regression (SR), namely generating equations as sequences. It leverages the power of pretrained SR transformer models and the MCTS algorithm to tradeoff the fitting accuracy and equation complexity. Experimental results on the SRBench and the In-domain Synthetic datasets demonstrate that the proposed approach outperforms the backbone E2E transformer model.\n\nSTRENGTHS:\nSoundness:\nThe techniques employed in the proposed approach are sound. The approach is able to use any non-differentiable target function to guide the training of a neural model for symbolic regression. Experiments demonstrate that it outperforms a state-of-the-art transformer model which is used as the backbone in the proposed approach, indicating that the implementation of the proposed approach is likely to be correct.\n\nPresentation:\nThe submission is in general well written and organized, easy to follow.\n\nWEAKNESSES:\nPresentation:\nThere is a minor issue on the term single-instance symbolic regression introduced in Related Work. According to the description about the therein algorithms GP, RL, GP+RL and MCTS, the difference between them and the proposed approach mainly lies in not employing pretrained knowledge. Thus, the term single-instance is strange and cannot tell the true difference from the proposed approach.\n\nContribution:\nThe proposed approach seems to be a combination of the E2E transformer model [18] and the MCTS framework [26]. Although the transformer model can be replaced with other neural network-based models, the contributions beyond [18] and [26] are not significant. Moreover, the current evaluation cannot confirm that either the proposed approach is a general framework for enhancing any neural network-based model for symbolic regression, or the approach achieves the truly state-of-the-art performance. For the former confirmation, the authors need to compare multiple implementations having different backbone models with the original backbone models. For the latter confirmation, the authors need to compare the proposed approach with more state-of-the-art solutions such as [30] and [31].\n\nQUESTIONS:\nWhy the algorithms GP, RL, GP+RL and MCTS used for symbolic regression are called single-instance?\n\nLIMITATIONS:\nAs far as I can see, the authors have adequately addressed the limitations through sufficient discussions in the supplemented material.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to combine pretrained Symbolic Regression models with MCTS procedure to improve SR performance without finetuning the pretrained models. Experiments are conducted to demonstrate the improved performance the proposal.\n\nSTRENGTHS:\n1. Proposed a new MCTS based decoding procedure to improve pretrained SR models performance without finetuning the models.\n2. The paper is clearly written and easy to follow in texts. \nThe experiments clearly demonstrates the performance improving over the baselines and the E2E models' decoding.\n\nWEAKNESSES:\n1. A few key building blocks needs to be summarized from the literature to be a self-contained paper, e.g. how the datasets are embedded. \n2. The methodology contribution is minor as  only MCTS is introduced on top of SR models although experimental performance improvement is observed. Although this is a valuable contribution it might not meet the bar for NEURIPS.\n\nQUESTIONS:\n1. Please provide more statistics of equation (1)'s terms. For example, would N(s) be mostly 1s in your setting? Please be more specific about what \"visit count\" means exactly. Will the sub-routine beam search generated sequence be accounted as a visit? Will cache hit be counted as visit during beam search sub-routine? \n2. Figure 7,  please clarify how you count the number of generated candidates. Are the sub-routine beam search generated equations are counted? Will this difference render the results differently? \n\n\nMinors:\n\t1. In the abstract, \"GP-based methods\" should use the full name of GP for broader readers' convenience. \n\n\t2. Line 48, there should be an indent space at the beginning of the sentence. \n\n\t3. In section 3, to be self-contained, please succinctly re-iterate the key components of the underlying SR pretrained models, e.g. dataset embedding.\n\n\t4. Line 188, please briefly explain why this is still a Q function in standard MDP framework. \n\n\t5. Section 4, please be specific whether E2E and TPSR are using exactly the same experiment settings except the MCTS and beam search difference. If possible, a table in the supplemental materials comparing the experiment settings across different approaches might be helpful.\n\n\t6. Line 244, in the equation of R^2, is \\bar{y} the average values of y in N_{test}? Please clarify.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}