Transformer-based Planning for Symbolic Regression
Authors: Parshin Shojaee
Kazem Meidani
Amir Barati Farimani
Chandan K. Reddy
Venue: NeurIPS 2023 poster (2023)
Keywords: Symbolic Regression
Transformers
Planning
Deep Learning
TL;DR: We introduce a novel transformer-based planning framework for symbolic regression by leveraging priors of large-scale pretrained models and incorporating lookahead planning.

Abstract: Symbolic regression (SR) is a challenging task in machine learning that involves finding a mathematical expression for a function based on its values. Recent advancements in SR have demonstrated the effectiveness of pre-trained transformer models in generating equations as sequences, leveraging large-scale pre-training on synthetic datasets and offering notable advantages in terms of inference time over classical Genetic Programming (GP) methods. However, these models primarily rely on supervised pre-training objectives borrowed from text generation and overlook equation discovery goals like accuracy and complexity. To address this, we propose TPSR, a Transformer-based Planning strategy for Symbolic Regression that incorporates Monte Carlo Tree Search planning algorithm into the transformer decoding process. Unlike conventional decoding strategies, TPSR enables the integration of non-differentiable equation verification feedback, such as fitting accuracy and complexity, as external sources of knowledge into the transformer equation generation process. Extensive experiments on various datasets show that our approach outperforms state-of-the-art methods, enhancing the model's fitting-complexity trade-off, extrapolation abilities, and robustness to noise.

Transformer-based Planning for Symbolic Regression
Parshin Shojaee" Kazem Meidani
Amir Barati Farimani 2,3 Chandan K. Reddy'
Department of Computer Science, Virginia Tech
Department of Mechanical Engineering, Carnegie Mellon University Machine Learning Department, Carnegie Mellon University
Abstract
Symbolic regression (SR) a challenging task in machine learning that involves finding mathematical expression for a function based on its values. Recent advancements in SR have demonstrated the effectiveness pre-trained transformer models in generating equations as sequences, leveraging large-scale pre-training on synthetic datasets and offering notable advantages in terms of inference time over classical Genetic Programming (GP) methods. However, these models primarily rely on supervised pre-training objectives borrowed from text generation and overlook equation discovery goals like accuracy and complexity. To address this, we propose TPSR, a Transformer-base basedd Planning strategy for Symbolic Regression that incorporates Monte Carlo Tree Search planning algorithm into the transformer decoding process. Unlike conventional decoding strategies, TPSR enables the integration of non-differentiable equation verification feedback, such fitting accuracy and complexity, as external sources of knowledge into the transformer equation generation process. Extensive experiments various datasets show that our approach outperforms state-of-the-arl methods, enhancing the model's citping-oomplextty trade-off, extrapolation abilities, and robustness noise
Introduction
Symbolic regression (SR) is a powerful method to discover mathematical expressions for governing equations of complex systems and to describe data patterns in interpretable symbolic form. It finds extensive applications in science and engineering, enabling the modeling of physical phenomena in various domains such molecular dynamics, fluid dynamics, and cosmology (1-6). Symbolic representations provide valuable insights into complex systems, facilitating a better understanding, prediction, and control of these systems through the design of accurate, generalizable, and efficient models 97. SR models establish the functional relationship between independent and target vari- ables by mapping them to mathematical equations. The input data can be obtained from simulations, experimental measurements, or real-world observations. Symbolic regression, however, poses several challenges, including the combinatorial nature of the large optimization search space, vulnerability the quality of input data, and the difficulty of striking a balance between model fitting, complexity, and generalization performance [10
Symbolic regression encompasses a wide range of methods, spanning different categories. Traditional approaches, such Genetic Programming (GP),
heuristic population-be based search strategy where each individual represents potential solution to the problem [12] [3]. Though GP algorithms capable of finding solutions for nonlinear and complex problems, they are typically slow to converge due to the vast functional search space. Also, as they need to start the search from scratch for each dataset, they tend to be computationally expensive, prone to overfitting, and sensitive the
"qqual contribution. Contact email: parshinshojnee@@v.ddu 'The codes are available at: https:///gtthub.com/tee www.sayll...aaaataatccoeeeeeoeee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

DL DL-based SR
GP-based SR
ML methods
Sandomoonsst
NTOP
SUP
TTPSR (1-300 30
O TTXX
Comp Rank
Compleily
TPSk 0.5500
SPPSR
IEAA
16
TPSR DE
D.3 TPSA
0.5
EPLEX
E26
TPSR
(AP
GOMEA
ITEA
relesm
DSRY
ArTeymma
CSR
10 an (a) Black-box dataset
Accuracy (b) Feynman dataset
Figure 1: Pareto plot comparing the rankings of all methods in terms of the R² performance and identified equation complexity for (a) SRBench Black-box datasets and (b) Feynman datasets. Our results with Transformer-based Planning (TPSR) applied on top of E2E transformer SR model improves its average accuracy both data groups while maintaining similar range of equation complexity. TPSR can successfully reach the first Pareto-front which is better than E2E baseline on both data groups. Connecting lines denote Pareto dominance rankings, colors denote the families of models, and "+" indicates SR versus ML methods in Black-box datasets.
choice of parameters 14] Recent works in SR have shown promising results by using pre-trained transformers [1]] for generating equations as sequences of tokens. These models leverage the prior knowledge learned through large-scale pre-training and can generate equations with single forward pass, leading to considerably faster inference time compared to the GP-based methods [9] However, one of the limitations of these models that they focus on the supervised language model pre-training objective borrowed from text generation, i.e., they are trained solely with the token-level cross-entropy loss, which can result in equations that may exhibit high token-level similarities but are suboptimal with respect to equation discovery objectives such as fitting accuracy and complexity which are critical in this task. To mitigate this issue, beam search [0 or sampling [22] approaches have been employed as decoding strategies to propose multiple candidate equations for given dataset, and then select the optimal candidate equation based on the fitting accuracy after optimizing for constants. Nonetheless, both beam search and sampling decoding strategies primarily rely the pre- trained transformer's ' logits and next token probabilities, and therefore do not receive any performance feedback during the generation of equation candidates.
To consider the equation discovery objectives in the transformer generation process and still benefit from the pre-trained model logits, we propose TPSR, Transformer-based Planning strategy for Symbolic Regression. TPSR leverages lookahead planning algorithm, using Monte Carlo Tree Search (MCTS) decoding strategy top of pre-trained transformer SR models to guide equation sequence generation. TPSR significantly improves performance of the discovered equations by considering verification feedback during the generation process and still remains considerably faster than GP-based models which do not leverage the pre-training priors and learn expressions for each dataset from scratch. Notably, our approach is model-agnostic and can be applied to any pre- traitrdined SR model, enabling optimization of generated equation sequences for non-differentiable objectives that may encompass combinations of fitting accuracy, complexity, and equation forms. Additionally,
we incorporate different caching mechanisms to reduce the overall inference time. Our experimental results demonstrate that applying TPSR on top of the pre-trained E2E SR model [ significantly
enhances its performance across various benchmark datasets. As depicted in Fig. I. TPSR achieves a
strong balance between fitting accuracy and model complexity compared other leading baselines.
It also effectively drives the E2E model towards the optimal trade-off, represented by the first Pareto front. The major contributions of this work are summarized below:
Proposing TPSR, a new method that combines pre-trained transformer SR models with Monte Carlo Tree Search (MCTS) lookahead planning optimize the generation of equation sequenees while considering non-differentiable performance feedhack.
Developing a new reward function that balances equation fitting accuracy and complexity to optimize the generated equations for effective trade-off.

---

Demonstrating that TPSR consistently outperforms state-of- tfate-ofthhe-ar baselines across various SR benchmark datasets, generating equations with higher fitting accuracy while maintaining lower complexity to avoid non-parsimonious solutions. TPSR still achieves considerably faster inference time than GP-based models which do not use pre-trained priors.
Showeasing the extrapolation and noise robustness of TPSR compared to the baseline and conduct- ing an ablation study to investigate the impact of various model components.
2 Related Work
Symbolic Regression without Learned Priors. Genetic Programming (GP) algorithms are typ- ically employed for single-instance SR, aiming to find the best-fit equation for a "single" dataset at hand [12]. Recently, alternative neural network-based search algorithms have been explored,
including deep reinforcement learning (RL) 14 23 241, combinations of GP and RL (25] and
Monte Carlo Tree Search (MCTS) as a standalone framework [26). Despite their successes, all these methods lack the benefits of prior knowledge Iearned from large-scale pre-training. Consequently, they are slow during inference as they need to restart the search from scratch for new datasets.
Pre-trained Transformers for Symbolic Regression. In recent years, pre-trained transformers have shown remarkable performance in natural language and programming language tasks [27 29 This success has inspired researchers to develop pre-trained transformer models for SR 16
19 30 For example, Biggio al. [16] introduced a Neural Symbolic Regression model that scales (NeSymReS) with the amount of synthetic training data and generates equation skeletons where all the numerical constants represented by single token "C"". Kamienny al. [18] proposed an end-to-end transformer SR framework that predicts the complete equation form along with its constants. More recent works 330 31) introduced unified frameworks that include a transformer- based pre-training stage as the prior for subsequent RL or GP optimization steps. While GP and RL methods have start anew for each problem, the transformer approaches rely on synthetic data and the power of large-scale pre-trained priors to generate equations in a single forward pass. However, these models are pre-trained on token-level language modeling loss function and thus can perform suboptimal for other equation discovery objectives critical in SR such as fitting accuracy to the observed data as well as equation's 's complexity. Our model, TPSR, utilizes lookahead planning to guide the generation of equations towards better performance by employing these objectives as feedback during the transformer generation process.
Planning in Sequence Generation. Recently, planning algorithms have been utilized in NLP tasks to optimize text output for specific objectives, such as controlling generated text to meet certain constraints like non-toxicity or conveying certain emotions (32434]. Recent advances in programming language models developed in code generation have also yielded promising techniques that could be adapted for SR, as they share several vital similarities with each other. Both involve generating sequences of symbols for given input and typically require optimizing the generated sequences for specific criteria which is different from the pre-trianing objective. For code generation, this may involve optimizing objectives like code compilability, readability, passing test cases 37). Similarly, in SR, the focus may be on equation-specine accuracy minimizing complexity. Motivated by these successes, develop an approach that combines MCTS planning with pre-trained transformer SR models for improved equation discovery.
sequence-level objectives such as fitting
3 Methodology
3.1 Preliminaries
In SR, the main goal is to find a symbolic expression for the unknown function f()) mapping the d-dimensional input x E Rd to the target variable y = f()) R. Given a dataset of observations
D = (z,, 3/: ) SR methods try to generate an equation f(-) such that y f(x;) for all e N,, Also, the proposed equation is desired to generalize well and to effectively balance the fitting accuracy and complexity. The transformer SR models are trained on a large-scale dataset comprising equation instances paired with their corresponding observations, {(D), (-)) (DM: (-)), where M is the dataset size (number of paired samples). During inference, the trained model directly generates the equation f(-) as a sequence of tokens in an autoregressive manner. An effective way to represent the expression tree of equations in sequence is to use prefix notation as in (38). For embedding the
3

---

Delewt
Embedder
End Embeeider
Endiacater
Search Samplig
<start>
Sorting
MICTS-Gulded Decoder
Decoder
Decoder
Decoder
Foo
Fitting (v,
Optimication
X
Toker-level
Complexity
y= f()
(a) Training
[b] Inforence with Beam Search/ Sampling
[c] Inference with MCTS
Figure 2:
An overview of our proposed method with MCTS guided decoding at inference compared to the concurrent works with beam search/sampling decoding strategy.
observations, adopt the pre-trained SR model backbone from (18), Notably, given the potential for large input sequences with tokenized numeric data and the quadratic complexity of transformers, the method introduced in [1]] deploys linear embedder module to map tokenized inputs singular embedding space before introducing them to the transformer encoder and decoder. Subsequent to embedding, these models encode the input observations and then pass the encoded representation along with the masked tokens to decode the equation sequence. To train the model, token-level cross-entropy loss with teacher forcing is employed learn the distribution of next token prediction conditioned on the encoded dataset and the current state of sequence (Fig. 2(
Achieving good fitting performance from the model's predicted sequence demands generating accurate constants in the equation. To address this, the generated skeleton or equation can un- dergo round of optimization Ben-Fletcher-Gooleee
estimate their constants using nonlinear methods, such as Broy- Shanno algorithm (BFGS) [39]. Previous works [18] [6] employ beam search and sampling strategies for transformer decoding in combination with constant optimization to propose several candidate equations. Subsequently, they use fitting metrics such as R22 to order these candidates and output the final equation with the best performance (Fig. (b)). Transformer models utilizing beam search or sampling decoding strategies can generate multiple high-likelihood equation sequences, but their generation process based on logits obtained from model parameters pre-trained with ooken-matching loss relative to the reference equation. As result, such models lack the capability to receive verification feedback during generation and optimize sequence for equation discovery objectives such as fitting or complexity of equations.
3.2 MCTS-Guided Equation Generation
To generate equations that are both better-fitting and less-complex, is crucial to incorporate feedback into the equation generation process. To achieve this, we utilize Monte Carlo Tree Search (MCTS) lookahead planning during inference, guiding the decoder towards optimal solutions for fitting and complexity objectives (as shown in Fig. [])). The MCTS-guided guided transformer decoding explores different possibilities, identifying the most promising paths based on the objectives.
We frame the SR equation generation task as a Markov Decision Process (MDP) where state represents the current sequence generation step (token) If has not reached the terminal state (i.e., the <EOS> token), we select the next token from the vocabulary as action a, updating state s' by concatenating and Upon reaching the terminal state, the reward is computed and used to update the decoding model. MCTS represents states as nodes and actions as edges within a tree structure, navigating state-space from the root node (i.e., initial state) to reach terminal states with maximum rewards. MCTS balances exploration and exploitation, considering nodes that lead to higher quality equations (i.e., higher Q-values) and under-explored nodes (i.e., those with fewer visits). During the generation process of the transformer, we utilize the MCTS algorithm iteratively to conduct lookahead planning and determine the next token. However, the large search-space requires more than the sole application of MCTS to discover high-quality equations. We need to effectively share information between the pre-trained transformer model and MCTS for better generations. To achieve this, we incorporate the probabilities of the next-token that are acquired through the pre- pre-trained

---

Repeat
Expansion
Evaluation
Backpropagation
swant
ealar
calar
Salar
sin
H.sin,fs K2
+ Sin.x
Top-K
Tokens
Reward (f)
Enthetder
Decoder
Generate
f: sin, Xp
Frender
Beam Search
Figure 3: Overview of TPSR'ss 's key steps: Selection, Expansion, Evaluation, and Backpropagation. MCTS-guided lookahead planning in decoding interacts with the pre-trained transformer SR model in the expansion and evaluation steps employing the transformer top-k sampling and beam search, respectively. The designed reward is used to guide the backpropagation.
transformer SR models into the MCTS planning process. This incorporation helps to enhance the search process, leading to more efficient and effective results. The key steps of MCTS for transformer decoding in SR models, as depicted in Fig. are as follows:
Selection. The Upper Confidence Bound for Trees 40 criterion is employed to select actions (i.c.. next tokens) for fully extended nodes in the search tree, balancing exploration and exploitation. We use the P-UCB heuristic [1 as
In (N())) 1 1+ N(s))
P-UCB(s, =
Q(s, (((c)()))=(((s)))))))))))))))))) (
(I)
where Q(s, a) is the maximum return for action a in state across all simulations, promoting the exploitation of the optimal child node. The second term encourages exploration of less-visited children, with ((x) as state s's visit count and s' as the subsequent state. Po(als) is the probability of the next token a given the partial sequence state s from pre-trained transformer model parameterized trade-off is adjusted by hyperparameter 3. Lastly, the next token action maximizes the P-UCB: Select arg max, P-UCB(s, (s.a..
by 0. The exploration-exploiinee
Expansion. In the expansion stage, after selecting a node that not fully expanded, new child (next token) for the current state is explored. Random expansion of the node from the vocabulary, however, might result in an invalid equation (that does not comply with the prefix notation) and makes the search process very tone-consumnng Therefore, given partial equations, only top-k most likely choices of the next token are considered as the possible children of the node for expansion. In other words, we are restricting the actions to be only from the lop-k: high-likelihood options which are retrieved from the pre-trained transformer SR model's logits. These options are then ordered to determine the sequence in which the children will be expanded.
Evaluation. To evaluate the newly expanded nodes, we perform simulations to complete the equation sequence. This is necessary because the new state may still be partial equation and performance feedback can only be obtained at the end of the sequence when the equation generation is completed. In MCTS, it common to employ random actions during the simulation stage. Nevertheless, random action selection for equation generation, much like during expansion, suffers from certain drawbacks in terms of time and the possibility generating invalid equations. Consequently, the pre-trained transformer SR model is invoked again, this time utilizing beam search with beam size of b, to generate complete equation candidates based on the current state. The beam size b determines the number of complete equations to be generated from the current partial equation. Following these simulations, the highest reward among all the complete equation candidates is assigned to the new node value.
Backpropagation. After generating a complete equation f(-), the corresponding reward r(f(.) can be computed. The highest reward among all simulations is then assigned to the new node, which recursively backpropagates its estimated value to its parents until it reaches the root of the tree. This
5

---

update process involves updating the Q values of all state-action pairs, denoted as s' and a', along the trajectory in the tree to reach the root. Specifically,, for each state-action pair, the Q value is updated by taking the maximum of the current Q value and the new value r: Q(s', a') +- max (Q(s',a)) a'); r).
More details on TPSR, including its steps and implementation can be found in Appendix
3.3 Reward Definition
We define a numerical reward r E I evaluate complete equation candidate f(-). promoting fitiigg accuracy and regulating complexity. After optimizing constants in the complete sequence, we compute the reward. We first calculate the normalized mean squared error (NMSE) between ground-truth target variable and predicted target variable ij = f(x), and formulate the reward as:
1 r(f() x. + NMSE(Y, f(x)) + Aexp( 1(())) L
(2)
where 1 represents equation complexity as the sequence length in prefix notation
18
42 16 L
denotes the model's maximum sequence length; and À is hyperparameter balancing fitting and complexity reward terms. Higher À values favor less complex equations, encouraging best-fitting
and penalizing non con-parsimooouus solutions. NMSE is calculated as (+ y f(x) I| e). where E is a small constant to prevent numerical instability.
3.4 Efficient Implementation with Caching
During MCTS evaluation, the transformer model generates complete equation sequences from given state, constructing implicit tree structures for beam search and computing lop-k next tokens for visited states. These computations are required in future MCTS iterations, so we employ two caching mechanisms, lop-k caching and sequence caching, to reduce redundancy and improve effi- ciency. Top-k caching stores computed top-k val- ues for given states. For example, Fig. when evaluating state s = [+, sin] in MCTS iteration f, top-k tokens are computed for s and subsequent visited states, such as [+ sin, .22. State and top-k value pairs are cached for future use, avoiding re- dundant token retrieval. Sequence caching caches complete equations generated with the provided beam size. If a state matches a stored equation partially, the cached equation can be used directly in future iterations, bypassing iterative sequence generation. Both caching strategies are designed to enhance efficiency without compromising performance. More details are provided in Appendix C
lleration
lteration
Evelusion
Exuarsion
Evalation
Top-k Cache
Cutche
Figure 4: An illustration of caching mechanisms in TPSR.
Experiments
In this section, we present our experimental results that evaluate the effectiveness and efficiency of TPSR. While the proposed decoding strategy is generally model-agnostic, here we showcase the results of using TPSR for the end-to-end (E2E) pre-trained SR transformer backbone (18), as E2E the state-of-the-art open-souree pre-trained SR model with publicly accessible model weights. Additional results of using TPSR with the NeSymReS pre-trained SR backbone [16] can be found in Appendix D.4 We evaluate our framework by answering the following research questions (RQs):
RQL. Does TPSR perform better than other decoding strategies (beam search/sampling) and competing baseline methods over standard SR benchmark datasets?
RQ2. Does TPSR provide better extrapolation and robustness to noise?
RQ3. Are TPSR's caching mechanisms effective in reducing computation time?
RQ4. What is the role of individual components in TPSR's overall performance gain?

---

Table 1: Performance of TPSR compared with beam search and sampling decoding strategies on the SRBench 42 and In-domain Synthetic [8] datasets.
Data Gruup
Model
Feynman
Strogatz
Black-box
At 0.99
Complexity
1 1P > 0.99
Complexity
7 W
Complexxtt
E2F+Beam
0.815
54.19
0.357
53:21
0.847
83.61
E2E Sampling
0.848
50.73
0.357
50.14
0.864
82.78
SRBench
TPSR (A=0)
0.952
84.42
0.928
82.78
0.938
129.85
TPSR ()=0.1)
0.949
57.22
0.785
56.14
0.945
95.71
TPSR (A=0.5)
0.924
50.01
0,714
47.02
0.931
82.58
TPSR (A=1)
0,916
47.24
0.571
43,42
0,924
79.43
Data Group
1 R2 0.90
1
1 Acro.u
Artp.01
Aorpuoo
Complexity
E2E+Beum Sampinng
0.657
0,782
0.461
0.298
0.2
38.37
0.640
0,794
0,472
0.332
0.208
39.82
In-domain
TPSR (A=D)
0,702
0,828
0.550
0.416
0.333
67.11
TPSR ()=0.1)
0.708
0.833
0,514
0.326
0.213
40.31
TPSR (A=0.5)
0.697
0,830
0,459
0,274
0,184
36.55
TPSR (A=1)
0.691
0.827
0.439
0.271
35.67
4.1
Datasets
We evaluate TPSR and various baseline methods on standard SR benchmark datasets from Penn Machine Learning Benchmark (PMLB)
[3] studied in SRBench [42. as well as In-domain Synthetic Data generated based on [38] 18]. The benchmark datasets include 119 equations from Feynman Lectures Physics database series 44 14 symbolic regression problems from the ODE- ODE-Strogatz database 3
45). and 57 Black-box regression problems without known underlying equations. We limit the datasets those with continuous features and input dimension S 10, the transformer SR model [8] is pre-trained with dmaz = 10. The In-domain Syutheric Data consists of 400 validation functions with different levels of difficulty and number of input points. This data is referred as "in-domain" because the validation functions and their corresponding input points are generated using the same approach as the data which the backbone transformer model [18] is pre-trained. More details on each of these datasets are provided in Appendix A
4.2 Evaluation Metrics
We evaluate our model using the following three metries: R² score 42 accuracy to tolerance W 16 46), and equation complexity [8 42).
Σx ENerre (y: 0:)2 Σ Musr (y=- 7)2
R²
Acc =1 max
Us Vi
S w),
Complexity -
y/4
where Rt measures fitting performance with as the mean of in test set, Acc,, evaluates equation precision based on tolerance threshold w, and equation complexity is determined by the number of nodes in the expression tree of the generated equation f(-). Following [88 42], we set !t2 0 for rare pathological examples and discard the worst 5% predictions for Acc., reduce outlier sensitivity.
4.3
(RQ1) Effectiveness of TPSR
Table presents the performance comparison results of TPSR with the baseline decoding strategies on the SRBench benchmark and the In-domain synthetic dataset. For the E2E baseline, we use the settings reported in (18), including beam/sample size of C 10 candidates, and the refinement of all the candidates K 10. For our model, we use the width of tree search as kmaz = 3, number of rollouts 3, and simulation beam size b 1 as the default setting. For PMLB datasets that contain more than 200 points, we follow [18 and use B bags of data, each containing N = 200 points, due the limitation that the baseline method is pre-trained with N I 200 data points. In the baseline method [18], a total of BC¹ candidates are generated (C' candidates for B bags), which are then sorted and refined to generate the best equation. However, for TPSR, since we need train an MCTS for each bag, we use an iterative decoding approach, starting with the first bag and continuing
html

---

Accuracy (R*50.99 0.99)
Complexity
Inference Time
10--2
Target
0.0
TPSR 11
0.003
0.01
GP-C0MFA
0.1
TP5R (.5
TPSR
Alaytman
mm
ITEA
85R
Median R?
Madel Complexity
Inference Time (s)
(1
"TpAR (1-0.))
"TPSR 21=2)
108P-5P
1ith GOMFA
Adadcool
Randennone
corredldy
1058
*6hoyman
-0.5
0.0
0.5
1.0 102 10a 100 102 10a
Figure 5: Performance comparison of TPSR and SRBench algorithms in terms of Accuracy- Complexity-Time
Feyuman (top) and Black-box (bottom) datasets. For Feynman dataset, al- gorithms are sorted based on mean accuracy defined as the ratio of solutions with R² > 0.99 on test set under various noise levels. and for Black-bo box dataset, the algorithms are sorted based on the median R² score on test set. TPSR demonstrates a strong balance of performance with relatively low model complexity and lower inference time compared to GP-based algorithms. The error bars represent the 95% confidence interval and "*" refers to SR methods for Black-box dataset.
with subsequent bags until a criterion (R² > 0.99) is met or we use maximum of B 10 bags. To ensure fair comparison, we use B = 10 for the E2E baseline method as well. In this table, we demonstrate the results of our proposed framework, TPSR, with varying values of the À parameter that controls the trade-off between fitting performance and complexity in the hybrid reward function For detailed comparison of the experimental settings across different approaches, refer to Table in Appendix
defined in Eq.
As shown in Table when À A 0, the framework generates complex equations that overoptimize for fitting performance. However, as we increase X, the framework generates less complex equations with a slight reduction in fitting performance. Notably, even for large values of X, such X = the fitting performance of TPSR significantly outperforms that of the baseline methods. Based on the results, we recommend a default setting of > 0.1 as offers a balanced trade-off between complexity and accuracy, while also mitigating potential overfitting (as detailed in Appendix D.I). These findings demonstrate the superiority of TPSR over the baseline methods in terms of fitting performance across all datasets, while generating equations with comparable or reduced complexity than those generated by the baseline methods. Table [ shows that TPSR exhibits a more significant gap in fitting performance when compared to E2E baselines on SRBench datasets, while this gap is smaller for In-domain datasets (even performing slightly worse on Acc., for larger is due to the In-domain dataset being generated using the same approach as the E2E pre-training data, resulting the E2E model's 's superior performance on this synthetic dataset. Furthermore, qualitative comparisons of TPSR with baseline symbolic and black-box regression models demonstrate the
0.5, 1). This
8

---

TPSR (A=0.0) 0.0)
TPSR (A=0.1)
TPSR (A=0.5)
TPSR A==1.0)
E2E
ineposttinn
0.00
0.90
0.55
Avg. Acc. (R 0.999
0.85
ACCo3
2.50
0.45
0.75
0.70
0.15
0.60
0.001
Target Noise Y
(a)
(b)
Figure 6: TPSR with E {0, 0.1, 0.5, 1} compared E2E for (a) Extrapolation performance
where in-domain accuracy is shown for different input sampling variance (σ)), and (b) Robustness to noise, where mean accuracy (R2 > 0.99) shown for various target noise levels (7).
superior performance of TPSR in learning the underlying equation and out-of-domain extrapolation (see Appendix D.3).
Fig. presents a detailed comparison of our proposed TPSR with the baseline E2E transformer model and all the SRBench baselines on the PMLB Feynman and Black-box datasets. This figure illustrates the relative position of each algorithm with respect to (1) fitting performance, (2) model complexity, and (3) inference time. The results indicate that transformer-based planning in the TPSR significantly enhances the performance of E2E and outperforms even the state-of-the-ard GP baselines, achieving the highest fitting performance on the black-box datasets. This is achieved while the complexity of the generated equations in TPSR is not greater than that of E2E, and shows a great balance compared other SR algorithms.. The pareto plots provided in Fig. [] as well
and Appendix D.2 also demonstrate the effectiveness of TPSR in balancing fitting-tinn time compared to all other SRBench baselines. Our TPSR effectively pushes this balanced performance to the first pareto front for both the Feynman and Black-box datasets. Moreover, is important to note that, while the inference time of TPSR is longer than the baseline E2E transformer model, it still has significantly lower inference time than RL or GP-based SRBench baselines. Further results on the BRBehch and In-domain datasets are provided in Appendix D
4.4 (RQ2) Extrapolation and Robustness
The ability to extrapolate well is inherently linked the quality of the equations discovered through symbolic regression. To investigate the extrapolation performance of TPSR to out-of-training regions, we normalize the input test data points to different scales (a) instead of unit variance (used for training points) as per [18]. Fig. (]a) depicts the average performance of TPSR compared to E2E with sampling decoding on the training data as well as testing data in scales of E {1, 2, 4, 8, 16) for the In-domain Synthetic dataset. Also, we investigate the effect of different complexity controlling levels (i.c.,
(A E 0.1, 0.5, I.0}) on the extrapolation performance. can be observed that, while no complexity regularization) achieves the best fitting accuracy on the training data, it has sub-pan performance for > 8. This can be due to the overfitting issue when the symbolic model is much more complex than the true function, similar to the common overfitting issue in ML models. Results highlight the importance of controlling complexity in the extrapolation of identified equations. For values of À > 0, the overfitting issue is mitigated However, very high values of X (e.g.. x = 1) mostly result in poor accuracy performance. The flexibility of TPSR for allowing different values of À to balance fitting and complexity for given task is crucial for optimal performance. Fig. b(b) also presents the robustness TPSR with different levels compared to the E2E transformer baseline on the Feynman dataset. The results indicate that MCTS- -cTS-e lookahead planning can offer robust performance with a smaller drop in accuracy compared to the baseline the presence of noise.
the generated equations become less complex.
4.5 Ablation Study
In this section, we investigate the effect of different MCTS parameters and caching mechanisms on the performance of TPSR by conducting ablation experiments on the Feynman datasets.

---

Avg.Acc (R2 (899)
140
0.92
Time (sec.)
120
100
20
0.09
Caching Methods
Generated Candidates
(a)
(b)
Figure Ablation study on the modules and parameters of TPSR. (a) Effect caching mechanisms:
Sequence caching and top-k caching improve the inference time of TPSR () 0.1). (b) Efficiency and parameters of TPSR: Average accuracy of TPSR (varying model parameters), and baseline E2E (varying sampling size) across number of generated candidates.
(RQ3) Caching Mechanisms. Fig. a), we illustrate the effectiveness of the sequence and top-k caching mechanisms in reducing the total inference time of TPSR (A 0.1). Our experiments show that sequence caching has more effect dropping the inference time as it replaces the time-consuming sequence generation process. Overall, these two mechanisms can reduce the total inference time by around 28%.
(RQ4) Search Parameters. Fig. b)) shows the accuracy performance against the number of generated equations during the decoding process for both TPSR (A 0.1) and the baseline E2E with sampling decoding. In this figure, the 'number of generated equation candidates" represents the total number of complete equation sequences generated by each method. Specifically, this refers to the sample size in the E2E with sampling decoding, and the function calls of the beam search sub-routine multiplied by beam size in TPSR. The results show that under the same number of generated equation candidates, TPSR significantly outperforms the E2E baseline. This is primarily attributed to the fact that the E2E baseline is deprived of any performance feedback during the equation generation process. We report the results for variants of TPSR with different MCTS parameters. We assess the performance with varying number rollouts, r = (1.3, 6, 9}, number beams in simulations, b (1,3), and the maximum number of possible expansions at each state, kmex (2,3, 4). The default setting of TPSR parameters are b = 1, kmar = 3. and = 3. Results indicate that increasing kmazz and b all contribute the better performance of TPSR, with the most significant improvement observed when increasing r. This is because more rollouts provide model with more opportunities learn from trials and learn better values.
Conclusion
In this work, we propose TPSR, a model-agnostic decoding strategy for symbolic regression that leverages the power of pre-trained SR transformer models combined with MCTS lookahead planning, and outperforms the existing methods in generating equations with superior cotiing-ooplexxty---ee trade-off. We demonstrate the flexibility of TPSR in controlling discovered equation complexity without fine-tuning the pre-trained model. We also show that TPSR performs 100x faster than state-of-the-arf genetic algorithms by leveraging the pre-trained priors. Additional results show that better expressions obtained with lookahead planning can further improve model performance in terms of noise robustness and extrapolation unseen data. Future research could focus on enhancing the adaptability of feedback- based expression generation mechanisms, potentially by modulating the flexibility MCTS or transformer model weights, and the integration of MCTS with the training or fine-tume funning of transformer SR models. Furthermore, employing parallelization and distributed computing could potentially improve planning efficiency.
10

---

References
[1] He Ma, Arunachalam Narayanaswamy. Patrick Riley, and Li Li. Evolving symbolic density functionals. Science Advances, 8(36)eebb0229, 2022.
[2] Yiqun Wang, Nicholas Wagner, and James M. Rondinelli. Symbolic regression materials science. MRS Communications, 9(3):793-888 805, 2019.
[3] Samuel H. Rudy, Steven L.. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Data-driven discovery of partial differential equations. Science Advances, 3(4)x1602614, 2017.
[4] Kazem Meidani and Amir Barati Farimani. Data-driven identification of 2d partial differential equations using extracted physical features. Computer Methods in Applied Mechanics and Engineering, 381:113831, 2021.
[5] Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 11333))3332--333, 2016.
[6] Miles Cranmer, Alvaro Gonzaez-G Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020.
[7] E. Kaiser, J. N. Kutz, and S. L. Brunton. Sparse identification of nonlinear dynamics for model predictive control in the low-data limit. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, (74(211):20180333 2018.
[8] Kazem Meidani and Amir Barati Farimani. Identification parametric dynamical systems using integer programming. Expert Systems with Applications, 219:1 19622,, 2023.
[9] Yoshitomo Matsubara, Naoya Chiba, Ryo Igarashi, Taniai Tatsunori, and Yoshitaka Ushiku. Rethinking symbolic regression datasets and benchmarks for scientific discovery. arXiv preprint arXiv:2206. 10540, 2022.
[10] Marco Virgolin and Solon P Pissis. Symbolic regression is NP-hard, Transactions on Machine Learning Research, 2022.
[11] Haoran Li. Yang Weng, and Hanghang Tong. CoNSole: Convex neural symbolic learinn.. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022.
[12] Michael Schmidt and Hod Lipson. Distilling free-form natural laws from experimental data. Science Advance, 324(5923):8 118-5885)88885,0 2009.
[13] Bogdan Burlacu, Gabriel Kronherger, and Michael Kommenda. Operon c++: An efficient genetic pro- gramming framework for symbolic regression. In Proceedings of the 2020 Genetic and Evolutionary Compusation Conference Companion, GECCO '20, page 1562-1570 New York, NY, USA, 2020. Associa- tion for Computing Machinery.
[14] Brenden Petersen, Mikel Landajuela Larma, Terrell N. Mundhenk. Claudio Prata Santiago, Soo Kyung Kim, and Joanne Taery Kim. Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In International Conference on Learning Representations, 2021.
[15] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
[16] Luca Biggio, Tommaso Bendinelli, Alexander Neitz, Aurelien Lucchi, and Giambattista Parascandolo. Neural symbolic regression that scales. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 1339 of Proceedings of Machine Learning Research, pages 936-945. PMLR, 18-24 Jul 2021.
[17] Mojtaba Valipour, Bowen You, Maysum Panju, and Ali Ghodsi. Symbolicgpt: A generative transformer model for symbolic regression. arXiv preprint arXiv:2106, 14131, 2021.
Kamienny. Stéphane d' A'scol,,, Guillaume Lample, and Francois Charton. End-to-ena [18] symbolic regression with transformers. In Advances in Neural Information Processing Systems, 2022.

---

[19] Martin Vastl, Jonáš Kulhánek, Jirí Kubalik, Erik Derner, and Robert Babuška. Symformer: End-to-end symbolic regression using transformer-basee architecture. arXiv preprint arXiv:2205. 2022.
[20] Alex Graves. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711 2012.
[21] Sam Wiseman and Alexander M. Rush. Sequence-to-sequenee learning as beam-scarch optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1296-1306, Austin, Texas, November 2016. Association for Computational Linguistics.
[22] Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 889-898, Melbourn,, Australia, jul 2018. Association for Computational Linguistics.
[23] Allan Costa, Rumen Dangovski, Owen Dugan, Samuel Kim, Pawan Goyal, Marin Soljačić, and Joseph Jacobson. Fast neural models for symbolic regression at scale. arXiv preprint arXiv:2007. 10784, 2020.
[24] Mikel Landajuela, Brenden Petersen, Sookyung Kim, Claudio Santiago, Ruben Glatt, Nathan Mund- henk, Jacob F Pettit, and Daniel Faissol. Discovering symbolic policies with deep reinforcement learning. In Marina Meila and Tong Zhang, editors. Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 5979-5989. PMLR, 18-24 Jul 2021.
[25] Terrell N. Mundhenk, Mikel Landajucla, Ruben Glatt, Claudio P. Santiago, Daniel faissol, and Brenden K. Petersen. Symbolic regression via deep reinforcement learning enhanced genetic programming seeding. In A. Beygelzimer, Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[26] Fangzheng Sun, Yang Liu, Jian-Xun Wang, and Hao Sun. Symbolic physics learner: Discovering governing equations via monte carlo tree search. In The Eleventh International Conference on Leurning Representations, 2023.
[27] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties.
[28] Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss,, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[29] Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. CodeT5: Identifier-wware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[30] Mikel Landajuela, Chak Lee, Jiachen Yang, Ruben Glatt, Claudio P. Santiago, Ignacio Aravena, Terrell N. Mundhenk, Gareet Mulcahy, and Brenden K. Petersen. A unified framework for deep symbolic regression. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[31] Samuel Holt, Zhaozhi Qian, and Mihacla van der Schaar. Deep generative symbolic regression. In The Eleventh International Conference on Learning Representations, 2023.
[32] Thomas Scialom, Paul-Alexis Dray, Jacopo Staiano, sylvain lamprier, and Benjamin Piwowarski. To heam not to beam: That is question of cooperation for language GANs. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[33] Rémi Leblond, Jean-Baptiste Alayrac. Laurent Sifre, Miruna Pislar, Lespiau Jean-Baptiste, Ioannis Antonoglou, Karen Simonyan, and Oriol Vinyals. Machine translation decoding beyond beam search. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8410 8434, Online and Punta Cana, Dominican Republic. November 2021. Association for Computational Linguistics.
12

---

[34] Antoine Chaffin, Vincent Claveau, and Ewa Kijak. PPL-MCTS: Constrained textual generation through discriminator-guine MCTS decoding. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2953- 2967, Seattle, United States, July 2022, Association for Computational Linguistics.
[35] Xinyun Chen, Chang Liu, and Dawn Song. Execution-gure guidedd neural program synthesis. In International Conference on Learning Representations. 2019.
[36] Parshin Shojace, Aneesh Jain, Sindhu Tipimeni, and Chandan Reddy. Execution- based code generation using deep reinforcement learning. arXiv preprint arXiv:2301.38116 2023.
[37] Shun Zhang. Zhenfang Chen, Yikang Shen, Mingyu Ding, Joshua B. Tenenbaum, and Chuang Gan. Planning with large language models for code generation. In International Conference on Learning Representations, 2023.
[38] Guillaume Lample and François Charton. Deep learning for symbolic mathematics. In International Conference on Learning Representations. 2020.
[39] Roger Fletcher. Practical Methods of Optimization. John Wiley & Sons, New York, NY, USA, second edition, 1987.
[40] Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In Johannes Fürkrranz, Tobias Scheffer, and Myra Spiliopoulou. editors, Machine Learning: ECML 2006, pages 282-293, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg.
[41] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot. Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap. Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi. and go through self self-play. Science, 362(6419):1 140-11444 2018.
[42] William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabricio Franca, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason Moore. Contemporary symbolic regression methods and their relative performance. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
[43] Randal S. Olson, William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz, and Jason H. Moore. Pmll: large benchmark suite for machine learning evaluation and comparison. BioData Mining, 10(1):36, Dec 2017.
[44] Silviu- Marian Udrescu, Andrew Tan, Jiahai Feng. Orisvaldo Neto, Tailin Wu, and Max Tegmark. Ai feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. In H. Larochelle, M. Ran- zato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems. volume 33, pages 4860-4871. Curran Associates, Inc., 2020.
[45] William La Cava, Kourosh Danai, and Lee Spector. Inference of compact nonlinear dynamic models by epigenetic local search. Engineering Applications of Artificial Intelligence, 55:292-306, 2016.
[46] Stéphane D' Ascoli, Pierre- Alexandee Kamienny, Guillaume Lample, and Francois Charton. Deep symbolic regression for recurrence prediction. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4520 4536. PMLR, 17 17-23 Jul 2022.
[47]
Kamienny. Guillaume Lample. Sylvain Lamprier. and Marco Virgolin. Deep generative symbolic regression with SaatteCCrl--trr-ssarcde
Andreas Krause. Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages PMLR, 23-29 Jul 2023.
[48] Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. Annals of Statistics, 29:1 1232 2000.
13


Reviews:
[{"review_id": "VaKvEqJos8", "reviewer": "Reviewer_i1kS", "summary": "This paper proposed to combine pretrained Symbolic Regression models with MCTS procedure to improve SR performance without finetuning the pretrained models. Experiments are conducted to demonstrate the improved performance the proposal.", "questions": "1. Please provide more statistics of equation (1)'s terms. For example, would N(s) be mostly 1s in your setting? Please be more specific about what \"visit count\" means exactly. Will the sub-routine beam search generated sequence be accounted as a visit? Will cache hit be counted as visit during beam search sub-routine? \n2. Figure 7,  please clarify how you count the number of generated candidates. Are the sub-routine beam search generated equations are counted? Will this difference render the results differently? \n\n\nMinors:\n\t1. In the abstract, \"GP-based methods\" should use the full name of GP for broader readers' convenience. \n\n\t2. Line 48, there should be an indent space at the beginning of the sentence. \n\n\t3. In section 3, to be self-contained, please succinctly re-iterate the key components of the underlying SR pretrained models, e.g. dataset embedding.\n\n\t4. Line 188, please briefly explain why this is still a Q function in standard MDP framework. \n\n\t5. Section 4, please be specific whether E2E and TPSR are using exactly the same experiment settings except the MCTS and beam search difference. If possible, a table in the supplemental materials comparing the experiment settings across different approaches might be helpful.\n\n\t6. Line 244, in the equation of R^2, is \\bar{y} the average values of y in N_{test}? Please clarify.", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Proposed a new MCTS based decoding procedure to improve pretrained SR models performance without finetuning the models.\n2. The paper is clearly written and easy to follow in texts. \nThe experiments clearly demonstrates the performance improving over the baselines and the E2E models' decoding.", "weaknesses": "1. A few key building blocks needs to be summarized from the literature to be a self-contained paper, e.g. how the datasets are embedded. \n2. The methodology contribution is minor as  only MCTS is introduced on top of SR models although experimental performance improvement is observed. Although this is a valuable contribution it might not meet the bar for NEURIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and thoughtful questions. Please find our answers below.\n\n**Summary of Pretrained SR Details.** We adopted the pretrained SR model backbone from [18], using its embedding module to embed data points, and leveraging Transformers encoder and decoder modules for representation and expression generation. Given the potential for large input sequences with tokenized numeric data, and the quadratic complexity of Transformers, [18] introduced a linear embedder module to map inputs to a single embedding space before feeding them to the Transformers encoder. We left out these details because of page limits and relied on references. However, we agree on the importance of clarity. We'll make sure to include a summary of these key points in our updated paper.\n\n**Methodology Contribution.** We hope the clarifications provided below address your concern about contribution.\n\nIntegration is Not Simply Concatenation: While we leverage both MCTS and pretrained SR models, our contribution isn't a straightforward 'stacking' of the two. The fusion demands a meticulous redesign, modification, and the introduction of new methods to counter the distinct challenges of such integration. This distinction, we believe, is an innovation in itself and is elaborated upon in Appendix C.2 and C.3, with visual insights provided in Fig. 8 and 9.\n\nDifferentiating from Related Works: As highlighted in our related works, other works have used MCTS and LLMs for Planning in NLP. A notable example, [34], merges MCTS with a pre-trained discriminator. \nThis discriminator can assess both partial and complete states, streamlining its combination with MCTS to determine a node's (or state's) value. Contrarily, our SR equation generation model only allows evaluations and feedback upon equation completion, complicating and increasing the cost of the planning process. We'll need to generate complete equations using beam search simulations and design caching mechanisms to reduce repetitive generation calls to the pretrained SR model. \n\nWe want to again emphasize the following contributions: \n\n* We are the first to combine MCTS as a planning-based decoding module with pretrained SR models for the task of equation generation. TPSR not only achieves significantly better performance than the backbone pretrained model but also holds competitive performance compared to the established baselines. \n* We have designed the interfaces between these two components (pretrained SR model and planning search) effectively and dealt with unique challenges of making the framework computationally more efficient.\n* We have also showcased the versatility of TPSR across different equation-specific goals, from fitting accuracy to complexity. Notably, TPSR allows for optimizing equation learning based on varying objectives without necessitating finetuning of the large pretrained SR models.\n\n**Number of Visits Clarification.** The term $N(s)$ in equation (1) represents the \"visit count\" of state $s$, indicating the number of times that state $s$ has been encountered in the tree search during decoding. In our experiments, the value of $N(s)$ varies greatly depending on the complexity of the symbolic regression task and the state $s$ at hand. For simpler problems, or at early stages of the tree search, $N(s)$ might indeed be closer to 1, as states would not have been explored as thoroughly. For more complex problems, or deeper into the search, $N(s)$ can increase significantly as the same state may be visited multiple times in search. In our MCTS setting, a \"visit\" means that a state-action pair $(s,a)$ has been passed through during the tree search, and the corresponding child state $s'$ has been added to the tree. Sequences that are generated as part of the beam search sub-routine of simulations in the evaluation stage of MCTS are not directly considered as visits to the nodes corresponding to these sequences. Instead, they serve the purpose of completing the partial equation to allow for feedback computation. As for cache hits, they are also not counted as visits. The reason is that caching in this context is used to save computation by storing previously computed values, and a cache hit simply means retrieving a stored value rather than performing a new visit. \n\n**Number of Generated Candidates.** It's important to note that the visit count is just used in the selection step of the search to promote exploration (as explained in equation (1) ), while the number of generated equation candidates (shown in Fig. 7 as budget), refers to the total number of complete equation sequences that have been generated by each method, i.e., the sample size in the E2E with sampling decoding baseline, and the number of function calls of beam search sub-routine in TPSR (refer to Alg. 1), excluding instances where cached sequences were identified and utilized through sequence caching. \nThanks for raising these insightful questions! We will make sure to include clarification on these points in the main paper.\n\n**Response to Minor Comments:**\nWe will address each of the points you raised in the updated version. Answering some of the questions:\n* **Line 188:** The term $Q$ function here represents the value associated with each node/state $s$ when action $a$ is taken (leading to node $s'$). If the node is part of the trajectory to the current state's root, this value is updated after the backpropagation step.\n* **Section 4:** We assure that both E2E and TPSR use the same experimental settings, with the only difference being the MCTS and beam search/sampling implementations. Some details of the settings are already included in Lines 251-261. We agree that adding a table would make it more clear. We'll make sure to include it in the appendix of the updated version. \n* **Line 244:** You are correct. The reported $R^2$ is on the test set, therefore, $\\bar{y}$ represents the average values of $y$ in $N_{test}$. We will add a brief note to clarify this."}, {"review_id": "P2e0tV8eRJ", "reviewer": "Reviewer_zZmW", "summary": "This submission proposes a neural network-based approach to symbolic regression (SR), namely generating equations as sequences. It leverages the power of pretrained SR transformer models and the MCTS algorithm to tradeoff the fitting accuracy and equation complexity. Experimental results on the SRBench and the In-domain Synthetic datasets demonstrate that the proposed approach outperforms the backbone E2E transformer model.", "questions": "Why the algorithms GP, RL, GP+RL and MCTS used for symbolic regression are called single-instance?", "limitations": "As far as I can see, the authors have adequately addressed the limitations through sufficient discussions in the supplemented material.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Soundness:\nThe techniques employed in the proposed approach are sound. The approach is able to use any non-differentiable target function to guide the training of a neural model for symbolic regression. Experiments demonstrate that it outperforms a state-of-the-art transformer model which is used as the backbone in the proposed approach, indicating that the implementation of the proposed approach is likely to be correct.\n\nPresentation:\nThe submission is in general well written and organized, easy to follow.", "weaknesses": "Presentation:\nThere is a minor issue on the term single-instance symbolic regression introduced in Related Work. According to the description about the therein algorithms GP, RL, GP+RL and MCTS, the difference between them and the proposed approach mainly lies in not employing pretrained knowledge. Thus, the term single-instance is strange and cannot tell the true difference from the proposed approach.\n\nContribution:\nThe proposed approach seems to be a combination of the E2E transformer model [18] and the MCTS framework [26]. Although the transformer model can be replaced with other neural network-based models, the contributions beyond [18] and [26] are not significant. Moreover, the current evaluation cannot confirm that either the proposed approach is a general framework for enhancing any neural network-based model for symbolic regression, or the approach achieves the truly state-of-the-art performance. For the former confirmation, the authors need to compare multiple implementations having different backbone models with the original backbone models. For the latter confirmation, the authors need to compare the proposed approach with more state-of-the-art solutions such as [30] and [31].\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review of our submission. We appreciate your positive feedback and would like to address your concerns.\n\n**Terminology: Single-Instance SR.** We acknowledge that the distinction might not be clearly conveyed by the term \"single-instance SR\" itself. The intention behind using this term was to highlight that algorithms like GP and RL for SR typically focus on finding the best-fit equation for a \"single\" dataset at hand, without leveraging pretrained knowledge from a large set of datasets. However, we understand that this term may not fully capture the differentiation from the methods discussed in the next paragraph of Related Work which use pretrained knowledge for SR. To enhance clarity, we will revise this and use \"symbolic regression without learned priors\" instead of \"single-instance symbolic regression\". We hope that this clarification addresses your concern. If you have any further suggestions or insights, we would appreciate your input.\n\n**Contribution.** While we integrate both pretrained SR models and MCTS, our contribution goes beyond merely combining two existing techniques. The MCTS planning employed in our work takes inspiration from similar methods used in NLP, such as [34], where MCTS is applied for text generation. However, utilizing MCTS planning for equation generation in SR introduces significant differences from its application in NLP. Incorporating the MCTS planning algorithm into the pretrained SR backbone demands a thorough redesign and the introduction of novel techniques to effectively address the distinctive challenges posed by this integration. We firmly believe that this differentiation constitutes an innovative aspect in itself (refer to Appendix C.2 and C.3, as well as Figures 8 and 9). \n\nWe want to again emphasize the following contributions: \n\n* We are the first to combine MCTS as a planning-based decoding module with pretrained SR models for the task of equation generation. TPSR not only achieves significantly better performance than the backbone pretrained model but also holds competitive performance compared to the established baselines. \n* We have designed the interfaces between these two components (pretrained SR model and planning search) effectively and dealt with unique challenges of making the framework computationally more efficient.\n* We have also showcased the versatility of TPSR across different objectives, from fitting accuracy to complexity. Notably, TPSR allows for optimizing equation learning based on varying objectives without necessitating finetuning of the large pretrained SR models.\n\n**Model-Agnostic and SOTA Confirmation.** \nIn the current version of the manuscript, we have explicitly indicated that our framework is model-agnostic and holds the potential to enhance sequence generation in a variety of pretrained SR models. This includes both existing SR models and potential future models that might exhibit greater capabilities. However, we originally provided the results of using TPSR only on the E2E backbone [18], as E2E is the SOTA pretrained SR model with open source code, and publicly accessible model's weights and logits. In response to your valuable suggestions, we explored integrating TPSR with other pretrained SR backbones to illustrate its model-agnostic enhancement capabilities. Consequently, we integrated our TPSR planning strategy with \"Neural Symbolic Regression that Scales\" (NeSymReS) by Biggio et al. [16], a pioneering work that proposes large-scale pretraining for SR. However, NeSymReS does have some limitations, including its acceptance of datasets with a maximum of three dimensions. Also, this approach predicts equation skeletons and requires a more complex constant optimization process. Despite these challenges, we evaluated both NeSymReS and its TPSR-enhanced version using a dataset comprising 52 Feynman equations with a dimensionality of $d \\leq 3$. Results are provided in Table 1 of the response PDF (please check global response), showing that TPSR has significantly improved the fitting accuracy without changing the average complexity of the equations when $\\lambda$=0.1 and with a slight increase when $\\lambda=0$. We will include these results in the appendix and refer the readers to that section when discussing the model-agnostic feature of our framework. We would like to remark that due to the limitation of NeSymReS to very low-dimension problems, it cannot be evaluated on the SRBench datasets for which we have provided our main results.\n\nBesides confirming that our TPSR framework is indeed model-agnostic and can be used as a planning module for future models, we have shown in our manuscript that the current TPSR model applied on E2E performs SOTA in some benchmarks and is a competitor in others. In line with your suggestion, we also investigated a comparison with the mentioned SOTA works [30] and [31]. We would like to note that the code for [31] was released two months after the NeurIPS submission deadline. Additionally, while the code of [30] was partially released approximately a month before our submission (as part of the DSO package), we observed that at least two out of the five main components of this model (pretrained weights and AI-Feynman), along with some details, were not included in their current release. We tried our best to conduct new experiments to compare our results with [30], utilizing their current code version. Results (shown in Table 2 of the attached PDF (please check global response)) represent that TPSR outperforms [30] in black-box datasets, and performs competitively in the Feynman dataset. However, we would like to note that since the current results of [30] are evaluated without some of the main components of the model, we think it is not a fair comparison to be included in the main results of our paper. Also, we again would like to emphasize that being model-agnostic is a much more important aspect of this work as the SR models are rapidly improving."}, {"review_id": "rFcLXuixY9", "reviewer": "Reviewer_6B7y", "summary": "This paper proposes to incorporate  Monte Carlo Tree Search (MCTS) on top of pretrained transformer-based SR models to guide equation sequence generation. This is to address the challenges where existing methods purely rely on the pretrained transformer’s output and without accounting for external performance requirement. In MCTS, the authors develop a reward function to encourage the balance between fitting accuracy and regulating complexity for the SR generation. Also, the caching tricks are employed to improve the implementation efficiency. SR benchmark datasets are used to demonstrate the improved performance of the proposed method over the state-of-the-art.", "questions": "See my comments in the above section.", "limitations": "See my comments in the above section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Including performance feedback in the pipeline of generation of SR equation generation from pre-trained transformer-based SR models is well-motivated.  To achieve so, this paper proposes including MCTS as the decoder in this pipeline, and imparting    the external requirement, via a reward function in MCTS, to eventually improve the performance of equation generation. The extensive experiments and baseline comparison clearly show the effectiveness of the proposed method, in terms of fitting-complexity trade-off, extrapolation abilities, and robustness to noise. \n\nThe presentation of techniques is clear, and the evaluation in my opinion is solid. Overall, this paper makes a good contribution in the SR field.", "weaknesses": "I only have two comments:\n\n- In Equ. (1), how to select $\\beta(s)$? It would be better to show its effect on the performance in the ablation study as well.\n- Currently, the method still relies on a pre-trained transformer SR model. The authors could give some perspective about how (or if it is possible) the MCTS can also be incorporated in the transformer training (or fine-tuning) process.\n-A typo in line 151: trnasformer--> transformer", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of our motivation and contribution. The typo that you raised has been corrected and the response to your comments is provided below.\n\n--- \n> * In Equ. (1), how to select $\\beta(s)$? It would be better to show its effect on the performance in the ablation study as well.\n>\nIn response to your comment, we conducted ablation study with different values of $\\beta(s)$ on the Feynman SRbench datasets. Our findings, illustrated in Fig. 1 of the response PDF (please check global response), show: (1) small values of $\\beta(s)$ (e.g., $\\beta(s)$=0) offer lower performance, probably due to limited exploration. (2) large values of $\\beta(s)$ (e.g., $\\beta(s)$=100) also affect performance negatively due to excessive exploration compared to exploitation. (3) Optimal results are seen for $\\beta(s)$ between 0.1 and 10. Also, Fig. 1(b) shows that as $\\beta(s)$ increases, more equation sequence candidates emerge, signaling more exploration. However, after $\\beta(s)$>0.1, the candidate count doesn't grow much, possibly due to repetitive sequences frequently activating caching mechanisms. We will make sure to include this ablation study in the updated version of paper.\n\n\n--- \n> * Currently, the method still relies on a pre-trained transformer SR model. The authors could give some perspective about how (or if it is possible) the MCTS can also be incorporated in the transformer training (or fine-tuning) process. \n> \n\nWe thank the reviewer for raising this very interesting question! Incorporating MCTS into the training/fine-tuning of the transformer SR models is indeed an intriguing direction for future work (we have also briefly mentioned this in the conclusion section (Lines 361-362)). To train/fine-tune transformer SR models with non-differetiale equation-specific feedback, one way is to employ deep reinforcement learning techniques such as policy gradient so that we can backpropagate to update model weights. However, if we want to use the search ability of MCTS instead of RL and policy gradient methods, following these directions might help: **(1)** Utilizing the transformer SR model as both the policy and value network. This actor-critic process can predict action probabilities, hence guiding MCTS to more promising search paths. Additionally, it can estimate the value of specific states, assisting MCTS during reward backpropagation. **(2)** Leveraging trajectories generated from MCTS, or its rollouts, to enrich the training set for the transformer model. These trajectories, particularly the novel solutions discovered during MCTS explorations, can expose the model to non-obvious equation formulations it might not have encountered in traditional training sets. **(3)** Crafting a co-adaptation feedback loop between MCTS and the transformer model where MCTS and the transformer model parameters can be adjusted iteratively based on each component's performance feedback.\n\nWe would like to point out that these potential directions will certainly involve intricate challenges and may demand rigorous experimentation to ascertain their efficacy. We acknowledge the constructive nature of your comment, and it further strengthens our resolve to delve into this direction in future works."}, {"review_id": "7zXl7X3ocp", "reviewer": "Reviewer_QJmi", "summary": "Authors propose a transformer-based planning (using MCTS) strategy to solve symbol regression task. Different from traditional decoding method, the new method is able to integrate non-differentiable feedback into the transformer-based process of equation generation. Experiments demonstrate the significent performance. ", "questions": "Are there experiments to show changes of performances, if we change the selection set of mathematical operators and symbols?\n \nCan out-of distribution data be identified, and used for the promotion of the symbolic regression process?\n\nIt might not be diffiuclt for symbol regression method to find laws, such as f=ma. Could it find E = m c^2? ", "limitations": "Monte Carlo Tree Search is statistical, and Pre-trained transformer is trained through data, the integration of the two methods is still within the traditional paradigm of machine learning, so, may not work well for out-of distribution data. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Distilling symbolic equation from noisy data is intractable. Recent progress is achieved by training neural networks to generate candidate symbolic expressions, which is really promising.  \n\nThis work combines the Monte Carlo Tree Search and pretrained transformer-based symbol regression model for equation generation. Compared with Genetic programming method, the new approach not only leverages pre-trained priors, but also considers feedbacks during the generation process. ", "weaknesses": "There is not much initiality in the new method. It demonstrates a new application for a combination of two existing methods. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful comments and questions on our paper. We appreciate your positive remarks regarding the importance of this study.\n\n---  \n> * Are there experiments to show changes of performances, if we change the selection set of mathematical operators and symbols?\n>\n\nIn symbolic regression, the choice of mathematical operators and symbols significantly impacts the equations obtained. Our work proposes a transformer-based decoding strategy with MCTS, which requires adhering to the vocabulary of operators defined by the pre-trained transformer SR model (As mentioned in Appendix E, limitations, line 768). This constraint ensures compatibility with the pre-trained model but may restrict the set of available mathematical symbols.\n\nGenerally, the selection of mathematical symbols in symbolic regression involves a trade-off between expressivity and problem complexity. Larger vocabularies provide greater expressivity, allowing the method to represent more diverse equations. However, this increase in expressivity can also enlarge the search space, making the problem more complex. To strike a balance, most recent symbolic regression works use common mathematical operators that are prevalent in benchmark problems and scientific datasets, such as the Feynman dataset.\n\n\n---\n> * Can out-of distribution data be identified, and used for the promotion of the symbolic regression process?\n>\n\nWe believe that this is a very important question in symbolic regression, especially regarding pretrained SR models. We would like to discuss this from two aspects.\n\nFirst, we would like to discuss the comparison of our TPSR with pretrained models. Pretrained symbolic regression methods, in contrast with search methods, are trained with a large set of synthetic equations, originating from a distribution $\\Omega$. In fact, these equations are generated using an expression generator with specific settings, and the points are sampled in specific ways. Therefore, equations and/or datasets that are not generated from the same generator can be considered out-of-distribution. In our work, we evaluate our TPSR model and the pretrained E2E model on both in-domain equations from the same distribution of data, as well as SRBench equations which are considered out-of-distribution compared to the training samples. As shown in Table 1 of the main paper,  using lookahead planning in TPSR significantly improves the performance of the pretrained E2E model on out-of-distribution SRBench datasets. We have also observed that the performance improvement gap between TPSR and E2E is higher for SRBench datasets compared to this gap in in-domain datasets (also discussed in lines 271-275).\n\nSecond, from a broader perspective, since pretrained models have learned parameters conditioned on the input datasets and equations (in comparison to search methods), while they have the advantage of leveraging priors learned from large-scale data, they are limited in handling datasets that are very far from the training distribution and discovering very different equation forms from what was generated during the training. TPSR can help to remedy this issue by searching and lookahead planning in the decoding stage; however, it is still limited to the degree that it can be applied to out-of-distribution datasets. This is because of the dependency of TPSR on the fixed priors of the pretrained SR model. We believe that improving SR models for this purpose is an exciting line of research that should be considered for future works, as also mentioned in our conclusion section (lines 360-362). Some potential ideas include fine-tuning the weights of the SR model using non-differentiable rewards for the new out-of-distribution datasets to improve its performance.\n\n---\n> * It might not be diffiuclt for symbol regression method to find laws, such as f=ma. Could it find E = m c^2?\n>\n\nThank you for raising this thought-provoking question. We believe that this question can be examined from various angles, and we have tried to address your concerns below.\n\nSymbolic regression methods operate by fitting equations to datasets of observations (features X and corresponding y). When considering the applicability of these methods, it's important to acknowledge that the original benchmark datasets, including well-known cases like the Feynman dataset, often do not span extreme ranges of values for X. As an example, Equation I.48.20 from the Feynman dataset ($e = \\frac{mc^2}{\\sqrt{1-\\frac{v^2}{c^2}}}$) samples values in the range of $U(1,5)$ for $m$, $U(3,10)$ for $c$, and $U(1,2)$ for $v$. This is while, for instance, the value of $c$, representing the speed of light, could be considered a constant with a significantly higher value of $2.998*10^8$ which is far from the simplified covered range in Feynman dataset. In fact, when operating under simplified assumptions and utilizing large quantities of synthetic observations, it is possible to recover complex equation forms using advanced symbolic regression models. However, real-world scenarios present greater complexity due to factors such as diversity, ranges, and precision of observations.\n\nBesides the range of observations, situations like the equation $E=mc^2$ pose a challenge when assuming $c$ to be a constant. In such cases, identifying $c$ requires additional constraints, such as employing dimensional analysis. Notably, due to the inherent constant nature of $c$, even if a diverse dataset encompassing various value ranges for $m$, $c$, and $E$ is employed, the relationship between $m$ and $e$ would exhibit a linear correlation. Accordingly, we recognize that there are inherent limitations in scientific discovery when using contemporary symbolic regression models. Addressing these limitations offers an exciting avenue for future advancements in the field.\n\nWe hope our response addresses your concern. Please do let us know if we have correctly interpreted your concern or if further clarification is needed."}, {"review_id": "YhNK4NnzSv", "reviewer": "Reviewer_ijKU", "summary": "The paper introduces TPSR, a Transformer-based Planning strategy for Symbolic Regression. TPSR incorporates Monte Carlo Tree Search into the transformer decoding process, enabling the integration of non-differentiable feedback such as accuracy and complexity. Experimental results show that TPSR outperforms existing methods in terms of fitting-complexity trade-off, extrapolation abilities, and robustness to noise.", "questions": "Please refer to the weakness part above.", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to understand.\n- The idea of enhancing large scale pre-trained Transformers with improved search capablities is very promising in the context of symbolic regression\n- The model shows good performance both compared to the E2E baseline and the GP methods.", "weaknesses": "- My main concern is about the novelty of the approach. A very similar idea has been recently investigated in [1] where the authors also combine MCTS with pre-trained Transformers. I would be grateful if the authors could clarify any eventual differences between the two approaches.\n- The impact of $\\lambda$ seems quite significant in your experiements. However, it is not clear to me how one should select it in practice.\n\n\n[1] Kamienny, Pierre-Alexandre, Guillaume Lample, and Marco Virgolin. \"Deep Generative Symbolic Regression with Monte-Carlo-Tree-Search.\" (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable feedback on our work. We appreciate your positive comments on the clarity and potential of our work in symbolic regression.\n\n--- \n> * My main concern is about the novelty of the approach. A very similar idea has been recently investigated in [1] where the authors also combine MCTS with pre-trained Transformers. I would be grateful if the authors could clarify any eventual differences between the two approaches.\n>  \n\nWe understand your concern about the novelty of our approach and its potential similarity to the work [1]. Allow us to clarify the distinct differences between our Transformer-based Planning for Symbolic Regression (TPSR) and the DGSR-MCTS approach:\n\n* **General Approach:** The general mechanism used for generating the equations is different in DGSR-MCTS and TPSR. DGSR-MCTS exploits a pretrained mutation policy M to generate the expression by following a series of mutations from an empty expression (root). This is while TPSR follows the seq2seq approach of E2E to generate the expression token-by-token. Consequently, TPSR uses the pretrained E2E as its backbone but DGSR-MCTS pretrains the mutation policy from scratch. \n\n* **Definition of MCTS and Search Strategy:** One fundamental distinction lies in the definition and application of Monte-Carlo Tree Search (MCTS). In DGSR-MCTS, the search tree consists of full mathematical equations, with each node representing a distinct equation and edges corresponding to mutations between equations. In contrast, our TPSR employs MCTS as a decoding strategy in the context of the transformer model. Each node in the search tree of TPSR represents the current state of generated tokens, potentially forming non-complete sequences, with edges corresponding to mathematical operators or variables. As a result, the search tree of DGSR-MCTS with \"n\" nodes includes \"n\" different equations, while the TPSR search tree includes intermediate decoding sequences, and completed equations only exist at the terminal nodes. This distinction inherently leads to major differences in selection, expansion, and back-propagation mechanisms within the MCTS algorithm.\n\n* **Parameter Update and Learning:** DGSR-MCTS utilizes MCTS to update and learn the distribution of mutations for a group of out-of-distribution datasets. The approach involves fine-tuning an actor-critic-like model to adjust the pre-trained model on a group of symbolic regression instances. On the other hand, TPSR uses the pre-trained transformer's learned distribution to guide the expansion during the search process, without updating any specific parameters for in-domain or out-of-domain equations (without fine-tuning). Consequently, the same settings and pre-trained model are applied to both in-domain and out-of-domain equations in TPSR.\n\n* **Computation Time:** Another notable difference is the computational requirements of the two approaches. DGSR-MCTS involves pre-training a mutation policy, a critic network, and performing fine-tuning stages for these networks, leading to significantly higher computation time (a limit of 24hrs and 500K evaluations as stated in their original paper). In contrast, TPSR has substantially lower computation time and the number of evaluations, typically in the order of $10^2$ equations, taking approximately $10^2$ seconds (as shown in Fig. 6 and 7 of the main paper). This renders TPSR more suitable for applications where fast yet accurate equation discovery is critical.\n\n[1] Pierre-Alexandre Kamienny, Guillaume Lample, Sylvain Lamprier, and Marco Virgolin. \"Deep Generative Symbolic Regression with Monte-Carlo-Tree-Search.\" (2023).\n\n---\n> * The impact of $\\lambda$ seems quite significant in your experiements. However, it is not clear to me how one should select it in practice.\n> \n\nIndeed, the impact of $\\lambda$ as the complexity regularizer is significant in most cases. We evaluated and discussed different values of $\\lambda$ as it can affect the trade-off between accuracy and complexity, as shown in Table 1 and Fig. 10 (Appendix D.1). The appropriate choice of this hyperparameter may depend on the specific use case, where the balance between finding an accurate function and sacrificing complexity, versus emphasizing interpretability and equation simplicity over relative accuracy, becomes relevant. However, we agree that proposing default settings for hyperparameters would be beneficial for general use. Based on our results, particularly Fig. 10 in Appendix D.1, we conclude that setting $\\lambda = 0.1$ can achieve high accuracy while reducing complexity and avoiding overfitting (please also check Fig. 6). We will make sure to include this discussion in the updated manuscript."}]

Author rebuttal:
We sincerely thank all the reviewers for dedicating their time and expertise to review our manuscript. Please refer to the attached PDF where we have included the Tables and Figures referenced in the subsequent responses. We hope our clarifications address your concerns and look forward to further discussions.



Meta-review: The paper attempts to improve symbolic regression. In this regards, the authors leverage MCTS in the transformer decoding process to tradeoff the fitting accuracy and equation complexity. All the reviews were mostly on positive side, but with some concerns about novelty of the approach. We thank the authors and reviewers for actively engaging during the discussion phase to improve the paper. Author response resolved many concerns of the reviewers and consequently many reviewers increased their scores. Please update the final version of the paper with the clarifications asked by reviewers, guidance on selection of $\lambda$, and an updated contribution list.
Decision: Accept (poster)