Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a symbolic regression model using Transformers. It enables to pre-train the model using datasets from the other domain, which is helpful when the target domain dataset is relatively small. Moreover, this approach performs the parameter refinement with BFGS to further enhance the performance. Compared to the state-of-the-art approaches, which are mainly genetic programming based, this approach achieved not superior but competitive performance on benchmark tasks, and showed significantly smaller inference time.

strengths_and_weaknesses: Clarity

The motivation of each algorithmic component is described well. However, as I am not an expert in SR, it was hard for me to understand this paper in details. Probably it is because of my lack of the common knowledge in the SR community. For example, how is it pretrained? I couldn’t find any statement about the setting for the pretraining. Or is the method written in Section 2.1 for the pretraining? If so how is the post-training performed? The computational time written in 166 is for what? It should depends on the dataset, right? It was also not clear for me how the refinement of the parameter is performed in the end-to-end manner. 

Significance

Based on the authors’ statement, this approach is the first one that reaches competitive performance to GP-based approaches. Although the performance is not superior, the advantage of using pre-training over GP-based approach is promising.

The author’s claim the advantages of the proposed approach over the top performance approaches in its inference time. However, it was not evaluated whether this comes from the trick described in Section 2.2 or it is intrinsic to the approach using Transformer. 

questions: Please see the comment above.
limitations: Although the checklist says the code is available, I couldn't find the pointer to the code in the main text. It would always be nice to have a pointer to the code, even if it is included in the supplementary. 

Algorithmic description of the proposed approach would help readers understand.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a transformer-based approach to perform end-to-end symbolic regression, that is, predicting simultaneously the function and the values of numerical constants. The trained model is augmented with several tricks at inference, including refinement, scaling, and bagging, etc. Experimental results show that the accuracy approaches to the state-of-the-art GP approach and the inference time is significantly reduced by several orders of magnitude.
strengths_and_weaknesses: Symbolic regression is well studied with various machine learning technique. In contrast to addressing the problem in a two-step manner, the proposed approach directly predicts the function along with the value of constants. Such an end-to-end model is not surprisingly new. The key innovation is the employed tricks for improving the performance of trained model, and the employed techniques are technically sound.

As the accuracy is still lower than state-of-the-art GP approaches, the main benefit of the proposed approach is the reduced inference time. However, I am not sure whether inference time is the key factor of symbolic regression. It would be helpful to introduce more background on the importance of inference time.

Besides, more comparison with other end-to-end symbolic regression approach and ablation studies would be quite helpful to demonstrate the effectiveness of the proposed approach.
questions: 1. Could you make a further comparison with [29] instead of just mentioning “[29] infers one-dimensional recurrence relations”? It seems that both of them have the same architecture and similar tasks.

2. Could you please provide more ablation studies to demonstrate the effects of: (1) mixture of distributions and different distribution shapes; (2) input points scaling at inference time.

3. Could you please compare with other deep learning symbolic methods like [5] in experiments? It seems that [5] performs well on both running time and accuracy.
limitations: Please see questions listed above.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper provide a novel sequence prediction based solution to symbolic regression (SR). Compared to search-based SR methods, the proposed method is more efficient in inference and generalize to new tasks effortlessly. Compared to privious sequence based SR methods, the proposed E2E is able to scale to larger dimensions.
strengths_and_weaknesses: 
Symbolic regression is one challenging, theoretically and practically important task that incorperates pattern recognition, rule based reasoning, sequence modeling, discrete optimization, and beyond. 

Compared to GP based SR, the sequence prediction based SR scales to novel unseen problems effortlessly. Compared to previous works in the sequence prediction based SR such as symbolicGPT and NSRS, this work scales better, capable to deal with > 3 dimensional problems and is with better accuracy. This paper claims that using the proposed dataset generation scheme, the pre-trained E2E SR could generalize to both in-distribution and OOD problems.

In the formulation of sequence based SR, the most critical issue is how to sample the pre-training tasks such that the model learned on the generated samples could capture symbolic knowledge, while not overfit to the pre-training dataset. To address this, the authors provided a principled dataset generation workflow, and showed via experiments that E2E could generalize to OOD. The author also provided in the supples additional series of experiments on whether the E2E is memorize the data.


The authors provided implementations which helps the reproducibility.

Though being a new SR paper that pushes the SOTA, this paper upscales the dimension from (<=3) only to (<10), as the author mentioned in line 287. Hence the method still remains improvable in scaling to larger dimensions.

Other limitations dwell in the evaluation of the pre-training data generation procedure, which is critical to this sequence prediction type of SR formulation, but is under evaluated.

I haven't checked the provided implementations carefully. If the results are fully reproducible and more evaluations could be given to pre-training data generation on OOD, based on my limited assessment, this paper could be given high merit.

questions: One important issue for the sequence based SR is that people need to know to what extent the model is learned to overfit the pretraining data. The authors provided OOD evaluations in the exp section  and the dataset generation controlling params in the supples, which is helpful to this question. However, it would be better to 1) expliclitely show how different is the OOD symbolic samples differ from the pre-training data generation params? Better demonstrate this with concrete examples of OOD symbolic ground truth and  pretraining examples. And 2) it is helpful but not mandatory to: show that modifying the pre-training dataset distribution could positively contribute to but not greatly impact the OOD result.


As a new method targeting at sequence based SR, it is better to compare with prior works in this domain, which are symbolicGPT and NSRS. I am aware that official implementations seems to be not yet available, hence I'm not posting this as a must-have. However, since the E2E SR and symbolicGPT are similar in many ways, it would still be beneficial if we could see the simulated results: one way is to alter the E2E implementation so as to only predict skeletons, then apply coeff optimization to un-officially simulate the symbolicGPT.
limitations: This work does not explicitely show potential negative societal impact.
review_title: null

Meta-review: metareview: The paper proposes a transformer-based approach to perform end-to-end symbolic regression. All three reviewers seem to agree on the usefulness of the proposed approach to reduce inference time.  As pointed out by Reviewer Hxn5, although the performance is not superior, the advantage of using pre-training over GP-based approach is promising. The discussion phase has allowed covering important criticisms and the autors have included in the appendix a discussion on inference time (App. G), an extended comparison with other DL skeleton approaches (App. H), as well as ablation studies (App. E), commiting in turn to integrate the latter as well as possible to the main paper in the camera ready version. Furthermore it was stressed in the discussion by one of the referees that the implementation released in the supplementals is of high quality and bug-free, which should be of help to the future research community once open-sourced. For all these reasons, I am recommending the paper to be accepted. 
confidence: Less certain
Decision: Accept