Title: Transformer-based model for symbolic regression via joint supervised learning
Authors: Wenqiang Li
Weijun Li
Linjun Sun
Min Wu
Lina Yu
Jingyi Liu
Yanjie Li
Songsong Tian
Venue: ICLR 2023

Abstract: Symbolic regression (SR) is an important technique for discovering hidden mathematical expressions from observed data. Transformer-based approaches have been widely used for machine translation due to their high performance, and are recently highly expected to be used for SR. They input the data points, then output the expression skeleton, and finally optimize the coefficients. However, recent transformer-based methods for SR focus more attention on large scale training data and ignore the ill-posed problem: the lack of sufficient supervision, i.e., expressions that may be completely different have the same supervision because of their same skeleton, which makes it challenging to deal with data that may be from the same expression skeleton but with different coefficients. Therefore, we present a transformer-based model for SR with the ability to alleviate this problem. Specifically, we leverage a feature extractor based on pure residual MLP networks to obtain more information about data points. Furthermore, the core idea is that we propose a joint learning mechanism combining supervised contrastive learning, which makes features of data points from expressions with the same skeleton more similar so as to effectively alleviates the ill-posed problem. The benchmark results show that the proposed method is up to 25% higher with respect to the recovery rate of skeletons than typical transformer-based methods. Moreover, our method outperforms state-of-the-art SR methods based on reinforcement learning and genetic programming in terms of the coefficient of determination ($R^2$).


Review discussion:
This paper is motivated by a practical issue. I like the idea of using the supervision of the whole expression skeleton and the supervision of the preorder traversal of its expression tree in contrastive learning (maybe with data augmentation) to deal with the case that different instances of the same skeleton can have very different shapes, and instances of very different skeletons can be very close.
The authors did extensive experiments, the presented algorithm performs very well on the benchmark dataset.
The paper has covered a wide range of related work.
The paper is easy to follow and straightforward. The novelty is not significant as it is a basic combination of contrastive learning and Transformers. The implementation is not hard but it works well.
NO.
NA
Dear reviewer jvHV,
Many thanks for your valuable time to review our paper! We're happy that you appreciate our idea and acknowledge the significant improvement we achieved compared with other baseline methods.
We agree with you. A good algorithm should be uncomplicated and effective. Our proposed joint supervised learning mechanism effectively alleviates the ill-posed problem existing in other all data-driven symbolic regression methods. To our knowledge, we are the first to address this problem by combining contrastive learning. We believe that our proposed joint supervised learning mechanism is beneficial for future data-driven symbolic regression methods.
Thank you again for your positive feedback!
Best regards, Authors of Paper6520
Strength:
1. The problem is interesting.
2. The proposed framework achieves good results.
Weakness:
1. It would be better if the authors could further clarify the potential application of symbolic regression.
2. It is unclear to me why the authors adopt a point cloud-like feature extractor (like PointMLP). It seems that the tackled problem is a special regression problem, so why not use 1D convolution or directly use a transformer to handle it?
Also, there are some works tackling tabular data, like TabNet. The authors should further clarify the motivation of network design.
3. Since the proposed framework was trained and evaluated on the same synthetic dataset. There may be overfitting problems. The authors should discuss this point.
The whole paper is well-written and proposes a new solution for an interesting problem.
NO.
Dear Reviewer dp8W,
Thank you for your positive feedback! We are glad that you acknowledge that the problem is interesting and the framework we propose performs well. We will elaborately address your concerns and questions as follows.
>W1: It would be better if the authors could further clarify the potential application of symbolic regression.
Thank you for your valuable suggestion! We accept your suggestion and it will make our paper better. Our method shows great potential in recovering some of the laws of physics. As reported in section4.6, our model performs well in recovering Feynman physics equations. Due to the limitation of the main text, **in Appendix E we have added a description of the current successful application of symbolic regression and the details of recovering the Feynman equations in our method.**
>W2: It is unclear to me why the authors adopt a point cloud-like feature extractor (like PointMLP). It seems that the tackled problem is a special regression problem, so why not use 1D convolution or directly use a transformer to handle it?
Also, there are some works tackling tabular data, like TabNet. The authors should further clarify the motivation of network design.
Thank you for your questions. Let me clarify the motivation for our network design step by step.
- Symbolic regression aims to discover the underlying mathematical expression that **contains symbols and coefficients** and best fits the observed data, rather than a simple regression problem such as least squares or polynomial fitting. In our work, we regard the discovery process as a translation process, exploring the best mapping from data points' space to mathematical expressions' space.
- **We believe that learning a good representation of the expression is the key to data-driven symbolic regression methods.** Given the input data points $\mathcal{D}=\{ (\pmb{x_i}, y_i) \}_{i=1}^n \in\mathbb{R}^{n\times(d+1)}$, we use the point cloud-like feature extractor for two main reasons: (i) the conversation from the data point $\mathcal{D}$ to the latent representation should not be sensitive to the order in which the points of the dataset are given, because the optimal equation should not depend on the order in which the points are listed. Note that the input to a symbolic regression instance is a set of data points, rather than a sequence.
Thus, **the feature of the data points should be similarly order-invariant**. (ii) the conversation should not strictly depend on the number of points in the input data. The reason is that the data points provided may have varying sizes, thus we do not want our method to be restricted to cases with a ﬁxed number of input points. **The above two crucial properties coincide with the point cloud**.
Therefore, we treat data points sampled from the mathematical equation as the point cloud and use the point cloud-like feature extractor (i.e. PointMLP) to extract the features of data points. As described in Section4.5, the PointMLP-based feature extractor we used performs better than other methods.
- The 1D convolution and transformer you mentioned have shown promising results on sequence feature extraction. TabNet is used to process tabular data by using sequential attention to choose data features. The reason we do not use these is that our input is not an ordered sequence or tabular data but a set of points like the point cloud. In our method, given the feature extracted from PointMLP, we use the GPT based on the transformer decoder to autoregressively generate expressions.
>W3: Since the proposed framework was trained and evaluated on the same synthetic dataset. There may be overfitting problems. The authors should discuss this point.
We are sorry that we did not make this point clear in the paper, which caused you this concern. **The test set we use is different from the training set.** Most benchmark test sets for evaluation are not seen during training. At test time, the inputs to the model, i.e., the data points from the expression are resampled to avoid potential overfitting problems, which is different from the inputs the model sees during training. We have clearly clarified this point in Section4.2 of the revised version and highlighted the changes in blue.
Again, thank you for your suggestions and comments. We hope our responses have provided you with a clearer picture of our work and have addressed your concerns. If you still need more clarifications to increase the scores, please let us know.
Best regards, Authors of Paper6520
Dear Reviewer dp8W:
We would like to thank you for your positive and constructive comments, which are very helpful and make our paper stronger! We have posted point-to-point replies to each question/comment raised by you and uploaded the revised version of our paper (with track changes marked in blue). We sincerely hope that we could address your concerns. We would appreciate it if you would like to consider updating the score and increasing your confidence. We will also be happy to answer any further questions you may have during the discussion.
Best regards,
Authors of Paper6520
Thank you for your reply. The responses have addressed my concern. I will keep my score and recommend accepting this paper.
Dear Reviewer dp8W:
We sincerely thank you for your careful and detailed reading of our paper and responses. We are glad that our responses address your concern. Your suggestions have been very helpful for us to improve the paper. Thank you again for strongly supporting our paper!
Best regards,
Authors of Paper6520
**Strengths**
* Introducing the idea of feature extractor and contrastive learning into the transformer-based symbolic regression model is convincing.
Although each component of the proposed method was previously proposed in the context of other tasks, incorporating such methods into the SR model seems to be novel.
* The numerical experiment supports the effectiveness of the proposed method.
**Weaknesses**
* In the experiment, the same dataset is used for training and testing. I am not confident whether such a setting is reasonable. From the machine learning perspective, the obtained model should be evaluated on test datasets.
**Comments**
* It would be nice if the proposed method was evaluated using recent benchmark datasets such as SRBench (https://github.com/cavalab/srbench).
* In Figure 2, the redder color might indicate the cosine similarity is closer to **0**.
* How is the computational cost of the proposed method compared to other SR methods?
* This paper is well-written. The motivation and contribution are easily understood.
* Although the components of the proposed method are existing ideas, introducing such promising techniques into the transformer-based SR model is novel.
* The code is provided and the detailed setting of the experiment is reported.
NO.
Dear Reviewer eWWS,
Many thanks for your positive feedback! We are pleased that you found our paper novel, well-written, and clear. Here we will carefully address your concerns and questions in detail.
>weakness: In the experiment, the same dataset is used for training and testing. I am not confident whether such a setting is reasonable. From the machine learning perspective, the obtained model should be evaluated on test datasets.
We are sorry that we did not make this point clear in the paper. **The test set we use is different from the training set.** We use a total of six benchmarks for evaluation, most of which were not seen during training. To avoid potential overfitting problems, at test time, the inputs to the model, i.e., the data points from the expression, are resampled, which is different from the inputs the model sees during training. **We have clearly clarified this point in Section4.2 of the revised version and highlighted it in blue.**
>C1: It would be nice if the proposed method was evaluated using recent benchmark datasets such as SRBench (https://github.com/cavalab/srbench).
Thank you for your suggestion！ SRBench[1] is a good living benchmark for symbolic regression. It mainly contains three benchmarks: ODE-Strogatz[2] problem set, Feynman[3] datasets, and black-box regression datasets from PMLB[4], among which Feynman datasets have been included in our test. SRbench's other expressions are mostly high-dimensional, up to 1000 variables, and are therefore currently more suitable for comparing non-data-driven methods, such as GP[5] and DSO[6], which are processed or trained from scratch for each new dataset. In our method, because the training set we used contains expressions of up to two variables, our method and other data-driven baseline methods we chose (i.
e. SymbolicGPT and NeSymReS) do not yet support testing expressions of higher dimensions. In future work, we can train with data of high-dimensional expressions to enable our method to infer expressions of more variables, but not the focus of our work now. We have used five popular public benchmarks in our work and a more challenging test set SSDNC for data-driven methods, so we believe that the combined result on the total six datasets is strong enough to support all the major claims. Thank you again for your suggestion and we will consider it in our future work.
>C2: In Figure 2, the redder color might indicate the cosine similarity is closer to 0.
Thank you for your careful review. We have corrected this clerical error in the revised version.
>C3: How is the computational cost of the proposed method compared to other SR methods?
Thank you for your question. In general, our method is comparable in computational cost to other data-driven SR methods, and they are all smaller than GP and DSO. More importantly, our and other data-driven SR methods have an obvious advantage over methods like GP and DSO in inference speed. GP and DSO need to train a model from scratch on each new dataset whereas our model only needs to be trained once. We evaluate our method, GP, and DSO in terms of computational time at inference. As shown in the below table, we report the average runtimes (on the same device) across the Nguyen benchmark, with early stopping if the benchmark expression is recovered.
| **Method** | **Runtime (sec)** |
|-----------------|-------------------|
| Ours        | 25.39             |
| DSO        | 324.65            |
| GP           | 475.63            |
Thanks again for your appreciation and constructive comments on our work!
Best regards, Authors of Paper6520
**References:**
[1] William La Cava, Patryk Orzechowski, Bogdan Burlacu, Fabricio Olivetti de Franca, Marco Virgolin, Ying Jin, Michael Kommenda, and Jason H Moore. Contemporary symbolic regression methods and their relative performance. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[2] Steven H Strogatz. Nonlinear dynamics and chaos: with applications to physics, biology, chemistry, and engineering. CRC press, 2018.
[3] Silviu-Marian Udrescu and Max Tegmark. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020.
[4] Joseph D Romano, Trang T Le, William La Cava, John T Gregg, Daniel J Goldberg, Praneel Chakraborty, Natasha L Ray, Daniel Himmelstein, Weixuan Fu, and Jason H Moore. Pmlb v1. 0: an open-source dataset collection for benchmarking machine learning methods. Bioinformatics, 38 (3):878–880, 2022.
[5] John R Koza. Genetic programming as a means for programming computers by natural selection.
Statistics and computing, 4(2):87–112, 1994.
[6] T. N. Mundhenk, M. Landajuela, R. Glatt, C. P. Santiago, D. M. Faissol, and B. K. Petersen. Symbolic regression via neural-guided genetic programming population seeding. 2021.
Thank you for your reply. The responses are convincing me. I will keep my score and recommend accepting this paper.
Dear Reviewer eWWS:
We sincerely thank you for your careful and detailed reading of our paper and responses. We are happy that our responses are convincing you.  Thank you again for strongly supporting our paper!
Best regards,
Authors of Paper6520
Dear Reviewers, Area Chairs, and Program Chairs,
We thank all reviewers for their positive feedback and suggestions. We are pleased that all reviewers have acknowledged the contributions of our work and recognized its value.
Besides, we are glad to see that the paper is considered to have the following strengths:
- well-written and easy to follow (Reviewer eWWS, dp8W, jvHV)
- clear motivation and contributions (Reviewer eWWS, jvHV)
- comprehensive experimental evaluation and good performance (Reviewer eWWS, dp8W, jvHV)
- the method is novel (Reviewer eWWS)
- the problem is interesting (Reviewer dp8W)
- cover a wide range of related work (Reviewer jvHV)
We again thank all the reviewers for their positive comments. Here, we summarize the motivation, novelty, and contributions of this work again.
- **Motivation**. In recent years, data-driven symbolic regression (SR) methods have shown promising results in terms of inference speed compared to traditional GP-based methods.
However, these methods focus more attention on using large scale training data and ignore a key problem: the ill-posed problem, i.e. expressions that may be completely different have the same supervision because of their same skeleton, which makes it challenging to deal with data that may be from the same expression skeleton but with different coefficients.
Specifically, previous methods only use the single traversal of the binary tree of the expression skeleton (coefficients are replaced by the constant placeholder) as supervision information and only use the cross-entropy as the loss function for training. This supervision is evidently far from adequate for mathematical expressions of which symbols and coefficients are strongly coupled. The experimental results show that the ill-posed problem seriously affects the accuracy of the model in reasoning about the expressions from observed data points, so we explore a more efficient data-driven SR method.
- **Novelty and Contributions**. Please note that we propose a new framework that aims to efficiently alleviate the crucial ill-posed problem existing in data-driven SR methods to promote the development of SR. The encoder of the data-driven SR model should not only be able to **(a)** extract more efficient features of the data points, but also **(b)** represent the expression on a high level such that the transformer decoder only prints the representation as a sequence of symbols. For **(a)**, we leverage a point cloud-like feature extractor based on pure residual MLP networks to extract more efficient features of the observed data points. For **(b)**, we cleverly combine contrastive learning (CL) techniques to achieve this. We propose a joint supervised learning mechanism that is our core idea.
Specifically, we use whole expression skeletons as complementary supervision and increase the inter-class distance (for expressions with different skeletons) and decrease the intra-class distance (for expressions with the same skeleton but different coefficients), which makes the output feature of the feature extractor more efficient in the representation of expressions. We demonstrate the effectiveness of our method through extensive experiments of our model with/without CL. To our knowledge, we are the first to raise this problem and solve it by combining contrastive learning. Our method achieves up to 25% higher than other state-of-the-art methods in terms of the recovery rate of the expression skeleton, and achieves the highest accuracy on both the five popular public datasets and the synthetic SSDNC dataset.
We responded to the individual comments of each reviewer.
Additionally, we highlight the key updates of the revised version marked in blue for the convenience of the reviewers (we will remove the coloring in the final version). The updates are listed below:
- **In Section 4.2**, we have clarified that the test set is different from the training set. (Reviewer eWWS, dp8W)
- **In the caption of Figure 1**, we have corrected a clerical error. (Reviewer eWWS)
- **In Appendix E**, we have cited several current successful applications of symbolic regression in different fields, as well as added the details of our method to recover the Feynman physical equations so as to clarify the potential applications of the proposed technique and method design. (Reviewer dp8W)
Finally, we thank all reviewers again for their thoughtful reviews and for spending time writing the reviews. Please let us know if have any recommendations for further improving the paper and strengthening support.
Best regards, Authors of Paper6520

Meta-review:
role: PC
data: metareview:_summary,_strengths_and_weaknesses: All reviewers agree that this work is well-written and easy to follow with clear motivation and contributions. The experimental experiments support the effectiveness of the proposed method.
justification_for_why_not_higher_score: The contribution of this work is a solid accept but has not reached the bar for spotlight.
justification_for_why_not_lower_score: All reviewers agree to accept the paper.

Decision: Accept: poster