Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents two improving methods for symbolic regression (SR). The first method introduces the pre-trained recurrent neural network (RNN) model using the physics equation library to generate the candidate solutions. The second method is a novel loss function called mean squared error of difference (MSEDI). The MSEDI is alternately used with normal MSE in the proposed method. The experimental evaluation using benchmark datasets demonstrates that the proposed method outperformed existing SR methods in several problems.
strength_and_weaknesses: **Strengths**
* Two improving methods for symbolic regression are introduced. The experimental results show that the proposed method is superior to other SR methods, especially on the functions with unknown constants.

**Weaknesses**
* As only the concept-level explanation of the proposed method is given, it is hard to understand and re-implement the detailed algorithm. For instance, the training procedure of RNN and hyperparameter settings of the proposed method are omitted.
* This paper presents two improving methods: pre-trained TW-MLM and MSEDI. However, it is not clear the contribution of each method to the performance gain. The ablation study should be conducted.
* The proposed method uses the pre-trained model on physics equations. A simple baseline method using the physics equation dataset should be considered, e.g., incorporating the same physics equations into the initial population in genetic programming.
* MSEDI assumes a scalar input variable. It seems hard to apply the proposed method to multi-dimensional input problems.

**Comments**
* Why didn't the authors use all datasets in SRBench? The selection reason for the 6 datasets should be clarified.
* The physics equations in Table 7 include various different input variables, while the target functions in the experiments are composed of a single variable. The reviewer could not understand how to handle such a variable mismatch.
clarity,_quality,_novelty_and_reproducibility: * The paper is not well organized. The reviewer feels that it is hard to understand the main focus of this paper and the detailed algorithm of the proposed method.
* The experimental evaluation is not convincing. The ablation study should be done, and the hyperparameter sensitivity of the proposed method should be checked.
* The code is not provided, and it is hard to understand the detailed algorithm of the proposed method.

summary_of_the_review: Although the concept of the proposed approach seems to be interesting, the experimental evaluation is weak to validate its effectiveness. Moreover, the algorithm description should be improved.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors create a novel approach for symbolic regression using two different mechanisms. The first is a math language model, and the second is an adaptable strategy for alternating fitness functions during evolution. They experiment with their approach in synthetic and real-world data and achieve state-of-the-art results.
strength_and_weaknesses: Strength:

- The experimental setup of the authors is very strong, showing different baselines and good results across different modalities of datasets.
- The ablations studies also look sound, with some really interesting findings.
- This work is also well written with no major mistakes that I could find, coupled with the right information when needed.

Weaknesses:

- (minor) I think this work would benefit a lot from a more classic structure. The authors also used their method section (Section 3) to present some ablations and results. This causes some confusion and makes the actual method not that clear. Perhaps moving the ablations afterward to the method’s actual result would clarify things. Also, the “conclusion” section, or as the authors called Reflection, seems a little too extensive for this work (will go into this later).
- (major) The authors motivate their work on the premise that deep learning methods are black-box models that remove the possibility of insights when looking into SR. However, later in the introduction the authors admit that using deep learning models is quite good for generalization propose. Their method also uses an RNN as the backbone, which to me seems that would introduce the problem of black-box models into SR again.
- (minor) Since we are using language models, I wish there were details on why not using Transformers/BERT or similar.
- (medium) The authors give no instructions on how to reproduce their work. Yes, the intuitions are there, but what is the structure of the LSTM, embedding, and dense layer they used? How many neurons are we talking about? How many hidden layers does the LSTM have? Is it a bi-lstm? Furthermore, how was this LM trained? Which optimizer? Which LR? The authors should have answered all these questions in their work, and by not doing so harm their reproducibility.
- (minor) the authors abbreviate twice MSE, MSEDE, and MSEDI
- (medium) I wish there were more in-depth explanations about the results. The authors invest a long time in the ablations with intuition and provide none in the result section of this work.
- (medium) I wish the authors had added the methods with human heuristics in their comparison. It would be ideal to have an ablation study on the trade-off from creating heuristics manually (and perhaps adding bias) and machine-made.
- (medium) If there is a concern regarding bias in man-made heuristics, there should be an ablation regarding potential biases when using language models.
clarity,_quality,_novelty_and_reproducibility: - The authors are clear about their method and indeed present a novel approach. However, the lack of information regarding the Language Model training procedure hurts reproducibility.
summary_of_the_review: - The work presented here is interesting and novel. There is margin for minor and major improvements in this work. Specially regarding the reproducibility of their training procedure and some more intuition on selecting some parts of the main contribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose an evolutionary algorithm for symbolic regression combining genetic programming and a mathematical language model. The authors propose reasonable conventions that they use in their evaluation and suggest to be used by the community in general. To generate examples for training a lightweight RNN mathematical language model they propose to use a terminal instead of prefix representation. To solve the task of SR they also propose a method for alternating fitness functions for multi optimization. They evaluate their approach on synthetic datasets with and without unknown constants and compare it to a selection of SR approaches. Additionally they compare their approach to DSO-NGGP on a real-world datasets from SRBench.
strength_and_weaknesses: Strengths:
- Well argumented
- Well evaluated
- Contributions both to methods for solving SR and evaluation/benchmarking of SR methods
Weaknesses:
- Ambitious goal of setting conventions
- Conventions are 
- Unnecessary introduction of novel terminology
clarity,_quality,_novelty_and_reproducibility: The paper is well written and well structured. The authors provide clear treatment of the area of contribution and positioning of the proposed approach. The novelty lies in the proposed mathematical language model and its combination with standard GP in a multi optimization context. The methods are well described and can be potentially reproduced.


summary_of_the_review: Overall a good, well written paper. The narrative can be slightly toned down. While the suggestions for evaluation conventions may be ultimately of benefit to the community a broader review and discussion is needed on this topic. While it is good to propose conventions and learn from the best practices, here the proposed is an evaluation frame used to demonstrate the performance of the presented approach. I believe that it is unnecessary to introduce novel terminology for concepts that are well known and understood such as morphology for structure of the model and adaptability for objective criteria. The authors themselves establish 1-to-1 relationship between current terminology and their new terminology as early as in the abstract and the introduction section.

Regarding the proposed top-1 approximate recovery, i agree that perhaps the best scoring solution should be taken into account when comparing methods. This, however, removes the ability to evaluate one major performance aspect of SR methods - the consistency of recovery of expressions. A good SR method should consistently produce (symbolically or goodness-of-fit) good solutions across runs. Additionally, using only a goodness-of-fit criterion (r-squared), without considering symbolic equivalency can lead to overfitted expressions and/or expressions with spurious variable interactions that can't be easily explained. This in turn also reduces the compactness and interpretability of the solutions, which is a major upside of SR methods over other black-box methods. At least the parsimony of generated expressions should not be underestimated especially in the context of scientific discovery. I accept that this might not be exactly the case and I have misunderstood the proposed convention. In that case it would be good that authors explain this concept additionally and remove concerns by providing an empirical demonstration at least in the scope of their evaluation.

Regarding the proposed selection of appropriate lifetime population size, this applies mostly to population-based algorithms. Additionally, the authors propose a selection of arbitrary population size or a heuristic for selecting a population size based on the performance of a  random search (may be out of scope for this work, but the authors should take a look at Bergrstra and Bengio, JMLR, 2012 on random search). Making the selection of arbitrary lifetime population size a convention can lead to biased evaluation, enabling selection of sizes where preferred approaches show favorable results. This also casts a bit of doubt on the presented evaluation of the approach. In fact, as it is the case in related work, instead of comparison based on a single point, a more general picture of the performance of an approach can be obtained by looking at the performance as a function of the number of evaluations instead. This would strengthen the performance claims of the proposed approach and as a convention it would also be more inclusive to other, non-evolutionary approaches.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper combines ideas in language modeling and the theory of evolution to develop a novel approach to symbolic regression. By incorporating the terminal walk in language modeling, the proposed method is argued and shown to be able to reduce the reliance on human heuristics for constraints on the learned expression. By incorporating adaptivity from the theory of evolution, the proposed method is shown to be able to accommodate various learning targets, and perform better especially in practical situations when the true expression contains unknown parameters. The ideas are clearly stated and related literatures are properly compared to. The arguments are also backed by numerical experiments. 
strength_and_weaknesses: Strengths:

1. A novel combination of existing techniques. 

This paper presents a novel combination of existing techniques in other areas for solving the symbolic regression problem. Although these ideas already exist, the important point is to identify their roles for symbolic regression and properly combine them to achieve better performance in symbolic regression. 

2. Clearly stated ideas. 

The ideas in the proposed approach are clearly stated. In particular, I appreciate that they are properly related to their original literature, and why it makes sense to use these techniques is well explained. 

3. Supporting experiments. 

Another strength of this paper is the experiments that support the arguments and deliver practical messages. For example, it is good to realize that different methods may perform differently with and without unknown constants, and recognize the importance of unknown constants in practice. 



Weaknesses:

1. Theoretical justification. 

The only theoretical derivation on page 7 looks suspicious and I am not convinced. To be specific, even if the gaussian model is correct, the likelihood function is incorrect: $\epsilon_{i+1}-\epsilon_i$ are not i.i.d. random variables, hence the likelihood function cannot be a product over $i=1,\dots,n$. Although MSEDI performs well in the experiments, the authors still need to think carefully about how to justify this criterion. 

2. Reproducibility. 

As this paper is mostly an empirical study, the current version falls short due to the lack of open-source reproduction codes or libraries. I understand that this may be restricted in the blinded version. But at least it should be provided later on. 

3. The choice of RNN architecture. (question)

This point is more of a question: how do you choose the RNN architecture for your language model? Did you do some selection based on their performance, or are they commonly used in the area of language models? 
clarity,_quality,_novelty_and_reproducibility: Quality: 

This paper is of good quality overall as an empirical study. It is well motivated, clearly written, and well supported by experiments. However, the theoretical justification is not solid, as I stated in the weakness part. 

Clarity: 

This paper is clearly written and properly related to the literature. 

Originality: 

This paper combines two existing ideas in other fields, but the combination itself is normal. 
summary_of_the_review: I find this to be a nice paper. It presents a novel approach that combines ideas from language models and evolution theory to solve symbolic regression. It properly relates to the literature, clearly states the benefits and intuitions of using these ideas, and supports these arguments by numerical experiments, which further identify the importance and challenges of accounting for unknown parameters. However, the authors need to be careful about the theoretical justification, and should provide reproduction codes when possible.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This interesting paper presents a new approach to symbolic regression (genetic programming to find expressions that conform to datasets). The two innovations is using a language model instead of heuristics to get equations of the right shape (biasing the search space towards equations that look like known equations) and alternating fitness functions between generations to increase adaptability. I find both of these ideas intriguing (with the language model idea probably being the more novel one) and it is satisfying to see that these modifications work and bring strong performance. I think this paper is a clear accept.
justification_for_why_not_higher_score: The problem is not a huge research topic, so might have a limited audience. On the other hand, it is a good paper.
justification_for_why_not_lower_score: Clearly novel, good results, well written.
Decision: Accept: poster