Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper presents a novel approach to the problem of symbolic regression, where the goal is to learn relationships between variables in the form of mathematical expressions. This is clearly a very relevant task towards constructing explainable AI systems.

The proposed approach in this paper is based on the generation of mathematical expression with recurrent neural networks, by exploiting background knowledge about the form of the expressions to impose constraints on the generated examples.

The RNN is trained by maximizing a risk-seeking policy gradient that aims to increase best-case performance. The key idea here is to increase the reward of the top-epsilon fraction of samples from the distribution, without taking into account the samples that fall below such threshold.

The technique is sound and novel, and it provides a significant contribution to this research area. The positioning of the paper with respect to the state-of-the-art in the field is highly accurate.

A very solid experimental evaluation is carried out on several benchmarks, comparing the proposed approach with state-of-the-art systems for the same task, including commercial software such as Eureqa and Wolfram. The analysis includes an ablation study and experiments conducted with different amounts of training data. The proposed methodology is shown to perform better than all the competitors.

Overall, I consider the paper to be a strong contribution for ICLR.

* In the experiments with different levels of noise in data, why was Gaussian noise added to the dependent variable only, and not also to the independent variables?

- Pag. 4, "is not allowed.While" -> "is not allowed. While"
- Pag. 5, "but in practice has high variance" -> "but in practice it has high variance"



review_title: Recurrent neural networks trained in a reinforcement learning framework for symbolic regression

final_score: rating: 7: Good paper, accept
review_content: #### Summary

This paper presents a symbolic regression algorithm which uses policy gradient to learn a distribution over the space of mathematical expression structures. The distribution is represented by an RNN, and the on-policy sampling is also realised by this RNN. To prune the massive sample space, the authors include several constraints according to some prior domain knowledge. Furthermore, instead of optimising the RNN directly with policy gradient, the proposed algorithm also makes use of the risk-constrained method that emphasises the risks above a given percentile criteria. The proposed approach has achieved the best performance on several benchmark symbolic regression datasets.

#### Pros

+ This paper is motivated by an interesting problem. I like the idea of using domain knowledge to introduce hierarchical constraints, which seems to be effective in this task.
+ The authors have done extensive experiments, the presented algorithm performs well on the benchmark dataset.
+ This paper has covered a wide range of related work.

#### Cons

- The main contribution of this paper is introducing the heuristic constraints to prune the expression search space and applying existing reinforcement learning techniques (e.g., using RNN to represent the policy of sampling a grammar; the risk-constrained reinforcement learning) to the symbolic regression task. It looks more incremental rather than a novel contribution.
- The number of experiments in the main article is small, and many experimental details are hidden in the supplementary, I think the authors should re-organise the paper and prompt some of the results in the appendix to the main article.
- One of the biggest problem for the on-policy reinforcement learning is its time complexity since it requires a large sample for estimating the expectation of risk. Considering that symbolic regression is a hard problem with highly complicated underlying distribution over the expression structures, I wonder how efficient is the proposed approach comparing to the other baseline methods. I think the authors should include the results of running time of all the compared approaches.
- The experimental setting is questionable for some compared methods. In Appendix E, the authors report the results of several SOTA symbolic regression approaches. However, as the authors stated, the experimental results are different because the choice of datasets and primitive functions are different. I have noticed that this is a re-submission from ICLR 2020, and reviewers were pointing out the problem of lacking comparison one year ago. I think one year should be enough for carrying out experiments of all these SOTA approaches under the same experimental setting. Still, there is no such result in the current submission.

#### Recommendation

I think this paper proposes a nice approach to tackle the symbolic regression problem. Although the presented algorithm seems to be a combination of existing approaches, its performance on benchmark data sets looks quite good. However, the experiments are still weak because this paper lacks a fair comparison (e.g., using same library and the same set of problems), and my biggest concern of the proposed approach is its time complexity. I think the authors should address these problems to improve this work.

--------------------------------------------------------------------------
#### Update after rebuttal
The authors have explained the reasons for the comparing experiments in Appendix E and updated the result of time complexity, which I think worth including in this paper.
review_title: Accept

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary

The paper uses RNN trained by risk seeking RL objective to predict mathematical expression that generated the target dataset. In the second step placeholders for constants in sampled expression are optimized again by gradient optimizer to maximize the reward function. The decoder RNN uses heuristics that make it easier to generate valid math expressions.


Strong points

+ The paper is clearly written and potentially relevant to larger audience.
+ The proposed method is relatively easy to implement and it leads to good empirical results.


Recommendation

I recommend acceptance of this paper since the proposed technique is highly competitive, easy to implement and well presented.


Questions

* How does compute used by GP to get results in Tab 1 compare to policy gradient based methods? My perspective is that all the techniques should find the correct solution in the limit with enough computational resources (under assumption that they can escape local optima). Therefore making sure that all the methods had roughly similar budgets is important.

* In figure 2E mean reward for standard PG is going up even after 2M training steps, would it be possible to run the experiment longer to see where the max would be once mean reward converges? (Similarly in figure 8 in tasks Nguyen 3,5 and 8 mean of standard GP is still going up.) It is great to see that risk seeking PG converges much faster on these tasks, however knowing what is the limit for standard PG would be also interesting.

* In section 3.1 you say that generated expressions had to be between 4 and 30 symbols long, however tasks 8 and 11 (sqrt(x) and x^y) have shorter descriptions. What am I missing?


Possible improvements

* Using the set of problems from AI Feynman would make empirical evaluation stronger. However the same can be said for AI Feynman system and Nguyen dataset.

* Beyond scope of this paper: At the moment a new sequential model is learned for each task and it learns about that task only through the reward function. What if the model can "see" the dataset first by reading it through n-dimensional CNN (or RNN) that would produce "dataset embedding" that will be later used to condition the sampling RNN. In the same way that image captioning model is conditioned on image embedding. Generating infinite training dataset (with random expressions) should be trivial, that is one advantage over image captioning with limited data.

Typos

standard standard -> standard
review_title: Review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The authors introduce a novel method for inferring a simple algebraic expression for an output in terms of an input. The method outputs a sequence tokens of the algebraic expression as represented by the pre-order traversal of an expression tree. For each token of the sequence, a recurrent neural network outputs a probability distribution on possible tokens, from which the token is sampled. Some notable contributions:

1. The authors introduce a novel reinforcement learning objective that optimizes the quality of the best examples instead of optimizing the average quality.

2. At each point in the sequence, their recurrent neural network takes as input the sibling and parent nodes of the expression tree, instead of the previous token in the sequence.

3. Simple constraints can be introduced in their approach by zeroing out the sampling probability for tokens that don't meet the constraints. (For example, they disallow redundant expressions like log(exp(x))).

The authors do a comprehensive comparison with other methods (in which their work performs very favorably) as well as an ablation analysis to confirm that each of their innovations is helpful (which I was very pleased to see).

Well done.
review_title: Well-written, well-executed, well-suited, well done

Meta-review: This paper proposes an approach of generating mathematical expressions with a recurrent neural network, which is trained with risk-seeking policy gradient to maximize the quality of best examples rather than average examples.  The proposed approach also enables easily incorporating domain knowledge or constraints to avoid illegal or redundant expressions.  In extensive experiments, the proposed method is shown to significantly outperform strong baselines, including commercial software.  All of the reviewers find the work interesting and relevant, and there are no major concerns or issues after discussion.  The topic is also of interest to a wide range of audience in the ICLR community.
Decision: Accept (Oral)