Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper introduces Supe-RL that intermingles off-policy reinforcement learning with periodic beam search operations. The method makes a greedy selection between the rl-solution that it had and the best produced by the beam search using Polyak update to update the incumbent rl-solution if the one suggested by beam search is better. Experiments in robotics simulation benchmarks show that Supe-RL improves over prior hybrid methods. 

An extensive review of related literature is included that is extremely helpful in situating the work amongst the broader field. Visuals like Figure 1 also really hep the user in understanding the core concepts presented. 
 
The paper could be clearer with more copy-editing and technical rigor. Some technical terms are used loosely and ancillary claims made without justification which serve to confuse the reader. For example, the author claims (Page 1) that gradient-free approaches have poor generalization skills. I am not quite sure if this is correct as gradient-free EA/GA has been an industry standard as a general-purpose black-box optimizer. 

The author also claims that ERL uses an ES. I have read the ERL paper in the past and from memory know this to be incorrect. Verifying with the ERL paper, it seems to use an EA, not an ES. This is crucial as an ES is a finite-difference approximation of the gradient at heart and would be rather redundant with a gradient-based learner. EA on the other hand belongs to an entirely different family of algorithms and resembles natural evolution more closely. 

The author also uses the word redundancy in a confusing way. First, the author claims that redundancy in the population leads to diverse experience (page 4). I am not sure how this would work. Robustness is certainly a contributing factor to redundancy but diversity? I think the author is trying to convey that a population allows for a spread-out search across multiple promising directions/points in the search space concurrently thus helping diversity. I am not sure if redundancy is the correct word to describe this feature.
Similarly, the author mentions enriching the (replay) buffer with new redundant experiences (Page 4 Section 3)? That seems like an oxymoron.
Page 5 reads “Finally, results in Section 4 considers the redundancy offered by the population to introduce part of the experiences of the genetic evaluation into the same prioritized buffer of the drla” – I am not quite sure how redundant experiences would add value?

The author refers to Polyak updates as soft updates and makes it a central component of the method’s novelty. However, these kinds of updates with a target network is a fairly standard tool in most modern policy gradient algorithms i.e., Soft-Actor Critic (SAC). The use of the word “soft update” is also a bit confusing as I thought this indicated the use of a modified Bellman backup with a Boltzmann policy representation (similar to Soft Q-learning) at first. It is unfortunate that the namespace is overloaded.

If my understanding is correct, the mutated solution (neural network policy) from a round of EA directly replaces the DRL solution? This is generally a risky thing to do. Gradient-based methods often rely on the weights to lie within specific distributions (ranges of magnitudes) for effective gradient computation and update. This is particularly true if the network uses non-linear activation functions that can easily saturate (dying ReLU since the authors use ReLU). However, EAs have no such requirement and the weights can often blow up to big numbers particularly for a non-linear function approximation. Is there a safeguard to avoid this scenario?
I believe the nuances behind sensitivities of the optimizer (SGD and ADAM) that the author mentions (plus the fact that Polyak updates help) are symptoms of this fundamental issue (transferring policy weights with a nonlinear function approximator directly from a GA to a gradient-based learner).

Overall, the authors present an interesting idea. However, the novelty is marginal compared to earlier methods and far too many technical liberties are taken with the manuscript that make it difficult to absorb as a reader. 


review_title: Interesting idea - reservations about the technical rigor of the manuscript

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces another combination of an Evolutionary Strategy (ES) with RL, which basically consists of running an RL agent which every GE episodes alternates to run an evolutionary algorithm iteration based on the current policy as the parent of the population. If none of the individuals generated are better than the parent, the RL continues as it was before the genetic operations, otherwise the RL agent parameters are soft updated towards the value of the best individual.
The paper introduces a simple but effective approach which tries to obtain the benefits of both the ES and RL methods, making sure that the genetic operations do not  have a negative impact in the RL process. The contribution is interesting and fair, although the authors give considerable attention to the navigation problems used for validation,  which environments are also presented in this paper in the sections of the method and the results. Whereas in the title, abstract and first two sections the paper is completely focused only in the proposed hybrid learning method.
- Why not focusing on the more conventional/standard tasks (approached in the appendices) for evaluating the learning methods in the main paper, and leaving the proposed navigation tasks for the appendices, for a reader it would be more intuitive when analysing results of deeply well known studied problems. With this, the paper could focus more on the method, because for instance, Section 3 is dedicated considerably to stress the navigation tasks, while the method is indeed more general and not specialized in this kind of tasks. On the other hand, the content of Appendix E would be more valuable in the main paper.
- In Section 3, it is mentioned "Such mutations act in a similar fashion of a dropout layer, biasing the DRL agent to perform better than the evolutionary population in the long term". It is not clear why it is better, what is the foundation of this statement? Authors could add references or present a deeper discussion about it. If the reason is given based on the experiments of this work, this statement fits better the experiments section, or conclusions.
- It is important to stress more that the method is especially intended for scenarios in which it is possible to parallelize the learning process. In cases of learning with physical systems, it is not possible to compute the genetic operators and the population evaluation in parallel, this could significantly increase the convergence time, which could be unfeasible. 

- For SGRainbow in Section 3.1.1 it is mentioned "...as GRainbow, required to tune an SGD optimizer for this scenario, decaying its initial learning rate from 0.1 to 0.001 based on the current success rate of the model". How is this decay computed? it seems it is a function of the success rate, however for tasks that do not have this metric, how could this be done?
- Why the statistics are taken with only 5 runs in Section 4? this does not seem to be enough, additionally the tasks do not look extremely complex to think that running more experiments is unfeasible (this is also why analysing main results based on the standard tasks is more straightforward).
- Regarding the results of Table 1, it is not clear what are exactly time and number of steps, since normally they represent the same information. It is said steps have to do with trajectory length,  but what does length mean? it could be either trajectory duration, or distance of the trajectory.
- In general the results look positive and show an improvement over the baselines and previous methods combining the two worlds. It would be interesting to mark in the learning curves, the times in which the genetic operation was executed, in order to observe its effect through the convergence.

review_title:  

final_score: rating: 6: Marginally above acceptance threshold
review_content: Aiming at exploiting the benefits of population based policy optimization and policy gradient, this paper proposes a novel framework that combines these two techniques. The authors claim that the previously proposed frameworks that combines evolutionary approaches and policy gradient approaches uses actor critic approaches in the policy gradient part, and that it is important to be able to incorporate value-based methods instead of actor critic approaches since the value based approach sometimes outperforms the actor critic approaches. The proposed framework is designed to be capable of combining evolutionary policy search approaches with ANY deep reinforcement learning algorithms. The framework is simple and relatively easy to combine with different deep reinforcement learning algorithms. However, in two instantiation of the proposed framework presented in this paper, the authors tune the details for each of these two. For one cases the authors obviously exploiting the structure of the combined reinforcement learning algorithm (existence of the target network and update the target network only), which is not possible in general. 

The weakest point of this paper is the evaluation. The above mentioned two instantiations are evaluated only on a single task for each. It is definitely not sufficient to evaluate the generality of the framework. Moreover, the proposed approach is not compared to existing frameworks combining evolutionary approaches and policy gradients. The only existing approach is ERL, where its actor critic part is replaced so as to have the same RL approach as the proposed framework. Such a replacement is fine to evaluate the goodness of the framework itself. However, since ERL is not designed to be combined with other approaches than policy gradient, it is easy to imagine (and also mentioned in the introduction) that the performance will drop severely. To show the advantage of the proposed framework, the authors should compare the performance with the original ERL, and also more recent variants such as CEM-RL and CERL. 








review_title: Evaluation is insufficient to reveal the advantage of the proposed approach over existing ones.

Meta-review: This paper proposes a deep reinforcement learning algorithm Supe-RL that combines model free RL with genetic updates. The idea is to periodically mutate and evaluate the actor and greedily choose the best performing child, and incorporate it in the main actor via Polyak averaging on a target policy network. The algorithm can be in principle combined with any gradient based deep RL method. Supe-RL was demonstrated by combining it with Rainbow and PPO and evaluated in navigation tasks as well as standard MuJoCo benchmarks.

Overall, the reviewers found the idea interesting and to have value to the RL community. The reviewers raised some questions regarding technical rigor, evaluations, and the choice of base DL algorithms. As is, I find this a slightly above borderline submission, and thus recommend acceptance. However, I would encourage the authors to test their method also with a state-of-the-art off-policy algorithms, such as TD3 or SAC, in continuous domains, to better calibrate its overall performance.
Decision: Accept (Poster)