Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a hybrid method of policy gradient-based reinforcement learning and genetic programming for symbolic regression. The solutions provided by the policy-gradient-based deep reinforcement learner (DRL) are used as the population in genetic programming. The experimental result using benchmark functions for symbolic regression shows that the proposed method is superior to the existing symbolic regression methods in most cases.
main_review: The hybrid method of a policy gradient method and genetic programming (GP) is interesting. However, the combination method proposed in this paper seems to be simple. In the proposed method, the sampled solutions from a deep neural network are used as the initial population of GP iterations. In addition, the generated solutions by GP are injected into the samples used for updating the neural network weights via policy gradient.

The samples generated by GP iterations are used for updating neural network weights by policy gradient update. The policy gradient formula shown in section 3 is the Monte-Carlo approximation of the true gradient under the current policy. If we inject the samples generated by GP to calculate the policy gradient, the estimation of the policy gradient is biased. It should be carefully noted the justification of the proposed method.

The reviewer cannot understand the reason for the performance improvement by the proposed method. How does genetic programming contribute to improving the recovery rate? The authors should elaborate on the reason for the performance improvement.

In [Petersen et al., 2021], other benchmark symbolic regression problems are also examined, such as Jin-X and Neat-X. How about the performance of the proposed method on these benchmarks?

Does "GP-Meld" mean "DREGS" in Table 4?


-----
[Update after rebuttal]
The authors reply to my request properly. I agree with the clear performance improvement by the proposed method as the strength of this paper. SO, I would raise my score. However, the novelty is limited because the proposed method is a simple heuristic combination of the existing methods.
limitations_and_societal_impact: The reviewer thinks that the limitation is fairly stated.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper describes a method to enhance the performance of Genetic Programming (GP) for symbolic regression by iteratively enhancing initial populations for 'random restart' GP implementations. The method works through a generative recurrent network (RNN) and an Reinforcement Learning (RL) approach, in which the RNN gets rewarded when outputs a set of initial math expressions that help the GP to achieve better solutions after each run. The GP and the RL algorithm work cooperatively in an iterative fashion where for every GP run (25 cycles) the RL rewards the RNN. Authors provide an extensive battery of tests in several benchmarking datasets, as well as proposing their own set of problems. Their method achieves better results than all other methods they compare against, including a vanilla GP.
main_review: The paper is very well written and structured. Language is clear at all times; math is not overused. 

The method is very well described and presented. I really like the simplicity behind it, and the authors do a good work explaining the motivations and intuitions behind it.

The contribution, however, can be arguably minor, because they are taking two solid methods and just gluing them together; nevertheless, it works, so I'm guessing authors will have to defend the relevance of their contribution to the other reviewers.

I personally consider it worthy for acceptance at NeurIPS; there are, however, some details that I consider that need attention, before I can give a higher evaluation for the paper:

- I reviewed closely your experimental design, and I noticed that you use extremely small sample points sets (10~20 points); I wonder how results change if you provide more sample points, let's say, in the range of hundreds or thousands, to the different methods tested.

- If I understood correctly, your 'primitives' (as called in GP literature) is {+,−,×,÷,sin,cos,exp,log,x} (from your appendix). So, there are certain problems, such as Nguyen-8 and Nguyen-11 where the GP has no chance to find the correct expression, because it would be simply too difficult, or even impossible, to build it given those primitives.... why or how did you expect the GP to find those expression?? For example, express to me x^y using only the set of functions mentioned above. But more importantly, how did your method manage to find such functions?

- The most important element that I find missing in your draft, is a bit of discussion on how or why you think your method works, at a low level analysis. I do understand the perks and capabilities of RL, but my question is... what exactly is doing the RL that enhances that much a vanilla GP. My bet is that is pruning the primitives set, such that the GP has a smaller and better defined search space, but I'd really like to hear your thoughts on this.

Notice how your method could help immensely to understand on why GP sometimes work, sometimes doesn't, i.e. what makes a problem difficult for GP, which has been the million dollar question in GP research, but you need to provide a bit of discussion in such regard.

[UPDATED RECOMMENDATION]
Authors properly addressed all my questions and concerns. I've no doubt on the effectiveness and relevance of their method. The paper is well written, and my main objection (lack of discussion on how the method works at a low level) has been attended. Therefore, I see no real reason for this work for not being accepted.
limitations_and_societal_impact: Authors do not provide any discussion regarding potential societal impacts of their work; nevertheless, considering it is a highly theoretical work, a purely algorithmic contribution, I do not consider such analysis a requirement for this paper.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes to combine the policy gradient method with the genetic algorithm method to solve the symbolic regression problem. Basically, an RNN trained using policy gradient computes candidate equations, and these equations are served as "seeds" for the genetic algorithm. The output of the genetic algorithm is used to train the RNN through policy gradient. Then this cycle can forward. The authors test their method on various datasets and achieve the best performance among all the baselines.
main_review: Originality:  This work is somehow a novel combination of "Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients Petersen et al. 2021" and " Guiding deep molecular optimization with genetic exploration Ahn et al. 2020". The related work section is sound.

Quality: The idea of combining reinforcement learning (RL) and genetic algorithm is quite simple. Use the results computed by the RL policy to guide the genetic algorithm, and the improved results from the genetic algorithm can be used to improve the RL policy. Also, several methods to improve this cycle are studied and explained.  In the merit of simple, the performance looks good.

Clarity: It would be better if the authors can improve the writing. It is a little bit hard for me to follow the paper. Too many details are mentioned in the main text and hard to capture the high-level idea. Also, for the experiment section, it would be better if the author can mark your results at least in bold.

Significance: The program this paper study is significant.

Questions:

1. What's the difference between DSR in Tabel1 and DSR (RL only) in Tabel 2? If there are differences, why use two different baselines? And why not run all the baselines on all the datasets?

2. In terms of the fairness of comparison. I think you run 100 times for each method seems not fair. It would be better if you set a time budget and let all the methods run under the budget.

3. It seems that Nguyen-10 and 11 are more complicated than 12. But all the methods fail on 12. Do you have some intuitions?
limitations_and_societal_impact: N/A.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The manuscript proposes a method to solve symbolic regression problems.  Recurrent  neural networks (RNNs)  are trained using policy-gradient reinforcement learning to find initial populations for genetic programming (GP).


main_review: The overall approach is mainly a combination of existing methods. Thus, I regard the originality as limited.
I did not find obvious flaws or misleading statements in the manuscript. 
It is interesting that GP alone performs so well (table 2).

Some part were not clear to me:
lines 193-195: Are the individuals which violate the constraints removed (i.e., get lethal fitness) or repaired (e.g., ln(e^x) is simply replaced by x)? 

lines 54-59:  The authors could elaborate what they exactly refer to with "out-of-distribution", "off-policy", and "on-policy" in this context. What are the two policies? Would the authors regard the RNN to be on-policy and, if yes, why is this a problem?

lines 231-234: I am not fully sure that I understand the fundamental difference between PQT and RSPG: PQT takes the best from T_rl and T_gp combines while RSPG only removes the worst solutions from T_rl?

line 303: The entropy term comes out of the blue - where is it used?

Literature embedding:

lines 87-89: the idea of combining GP and gradient-based optimization of the numerical values goes back to:
C. Igel. M. Kreutz. Using Fitness Distributions to Improve the Evolution of Learning Structures. Congress on Evolutionary Computation (CEC 99), pp. 1902-1909, IEEE Press, 1999 [see section 5]


Minor comments: 

line 27: ", is" -> "is"

line 48: "the problem" -> "the general problem" ?

line 419: first name abbreviations


limitations_and_societal_impact: In my evaluation, no additional discussion of potential negative societal impact of the presented work is necessary.
review_title: null

Meta-review: This paper received review scores with a high variance: 5,6,6,9. The paper was actively discussed, both with the authors and in private.

Reviewer Ug8S (score: 9) championed the paper for its simplicity and strong results. Yet, the review text and the arguments given (e.g., "I see no real reason for this work for not being accepted.") do not sound like a score of 9 at NeurIPS, more like a 6 or 7).

Negative points in the reviews and the discussion were the lack of originality (the paper heuristically combines existing methods: RL in order to sample the initial population of evolutionary methods, to solve symbolic regression problems).

A brief note about the NeurIPS checklist: the authors' responses to the questions are very casual (e.g. a simple "Yes" would be preferable over "It would be a bad idea not to"), as if these points are obviously satisfied; however, the paper did *not* attach the code and as the code repo merely gave "www.anonymous.submission". That does *not* count as the code being available; it should be available for reviewers to check in an anonymized repo or attached as supplementary material.

Overall, in view of the strong results and the simplicity of the method, I am leaning towards acceptance as a poster.

Decision: Accept (Poster)