Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper offers a novel population-based evolutionary training framework for off-policy RL algorithms based on the idea of combining both Evolutionary Algorithm (EA) and Deep Reinforcement Learning (DRL) into a single training framework due to the recent promising results when combining and exploiting the advantages of both EQ and DRL methods.

The paper first addresses two current limitations of hybrid approaches:
 - How to randomly perturb the policy in a meaningful and controlled manner?
 - Perturbed networks may end up being similar and reduce the overall diversity.

The paper suggests solving these two problems by proposing to combine: 
Behavior-Regularized Perturbation (BRP) to randomly perturb a policy network within a specified behavior divergence range online and 
Behavior-Targeted Training (BTT) to randomly assign a sampled target behavior divergence and inject it into the actor training process.

The proposed framework shows the benefits over baselines.
strengths_and_weaknesses: Strengths:
- well-written, clear, and easy to follow
- results are good compared to baselines

Weaknesses
- computational expensive
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Can you provide more reasons and explanations on why BRP adopts the Euclidean norm and not other distance measures?

Post rebuttal update

The authors answered my question with related references, which convinced me, so I kept my score weak accept.


issues: - computational expensive and the author proposes parallel implementation which shows how solve the problem.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work presents a direct policy search framework called BEL (Behavior-aware Evolutionary Learning) based on the evolution strategy (ES) approach. The main idea here is to include a policy-gradient step in the generation of the off-spring policies at each generation of the evolution strategy. To ensure that policies do not deviate too much from the parent policy distribution, a regularization step is added. The paper claims that BEL has superior sample efficiency and stability compared to existing methods, and BEL produces diverse agents in sparse reward scenarios. The paper also claims that due to the parallel nature of the ES approach, BEL is a competitive method to train policies for real-world robots. 
strengths_and_weaknesses: Although the paper picks up an important problem, that is, sample efficient and stable training of policies, it is hard to see how their approach is solving it. From the beginning of the paper itself, many things are either unclear or unexplained and in several places, the authors simply introduce terms/notations, etc. out of nowhere. For instance, from the title of the paper and the abstract, we get the impression that the paper is about maintaining behaviorally diverse policies during the policy search process. But nowhere in the paper do the authors explain what they call a "behavior". Overall the paper lacks a clear intuition behind why their approach works better than the state-of-the-art algorithms like TD3, SAC, etc in standard mujoco tasks. This is a big claim. So it needs clear justification. 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The paper needs major revision. The paper lacks clear intuition to justify their big claims. Many things are either unclear or unexplained and in several places, the authors simply introduce terms/notations, etc. out of nowhere. Experiments are not convincing enough.

### Post rebbutal update

The authors have addressed my concerns convincingly and improved relevant sections of the paper. Thus I have updated my recommendation to weak accept.
issues: In the abstract: "The main idea is to train a population of behaviorally diverse policies..." => Since behavioral diversity is a key element of the paper, spend some time to explain it clearly in the introduction itself, preferably with a example.

Why BEL is sample efficient? What is the intuition behind it? Generally evolutionary strategies are less sample efficient. 

- The last paragraph of the introduction is quite vague. There are many unexplained terms introduced here, e.g., behaviour divergence range, behavior, target behavior divergence etc. A concrete example will make this paragrapgh more clear.

- Fig 1 is more complicated than the text. Why not use a sequntial step by step flow diagram than connecting multiple blocks to each other without any clear start or end. Also, a longer caption of a few sentences would also be useful for readers.

- Eq 6: missiong subscript $\theta$ on the left side of the equation.

- Line 139: What is $\mathbf{S}$ ? 

- The texts on the plots and figures are extremely small and illegible. 

- Fig 2(d): What is RUT in the legend?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents Behavior-aware Evolutionary Learning (BEL), which combines off-policy reinforcement learning with evolutionary strategies (ES) for policy optimization. BEL consists of two parts. The first part is Behavior-Regularized Perturbation (BRP), which improves standard gaussian-based perturbations for ES algorithms by considering the sensitivity of the policy's output to parameter value changes. The second part is Behavior-Targeted Training, which ensures that the fine-tuned policies after RL still have desired diversity. With the combination of BRP and BTT, BEL outperforms baselines in achieved reward. The authors further conducted ablation studies to verify design choices.
strengths_and_weaknesses: Strengths:
* Nice incorporation of ES and gradient-based RL

Overall, I think this paper presents an effective method to combine Evolutionary Strategies (ES) with gradient-based Reinforcement Learning. The use of TD3 improves the quality of the random perturbations generated by ES. Moreover, TD3 is an off-policy algorithm, which means that it does not require additional environment interaction to train on. Lastly, the incorporation of BRP and BTT makes the algorithm more effective by ensuring that perturbation happens at desired scale, and the fine-tuned policy remains at desired diversity.

* Well-written and easy-to-understand

The paper is well-organized and easy to follow. Although the paper presents solid ideas with very detailed experiments, it does not feel difficult to read, and the contents flow naturally. I do have a few minor comments about certain details, but the overall writing is great.

* Well-designed experiments

The experiments are well designed and justifies the performance, as well as the detailed design choices of the algorithm.

Weaknesses:
* Less related to robotics
The paper presents a general RL algorithm with evaluation on standard Mujoco benchmarks. While the algorithm should be easily adapted to robotics tasks, the evaluation is mostly in simulation. Overall, I think this paper is clearly relevant to robotics. However, additional robot experiments would be desired for a conference like CoRL.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think it’s a nice paper that should be accepted. The only thing that holds me back from a “strong accept” is its lack of real-robot experiments. However, even at the current shape, I think this paper shows sufficient relevance to robotics and deserves to be accepted.
issues: I would like to point out some issues about Fig.2 in the paper. Overall, this figure is pretty difficult to understand due to small font sizes and lack of explanation. A couple of more detailed comments:

Fig.2(a) should probably be a standalone figure and appear earlier in the paper (close to methods section).

Fig.2(b) and (c): the axis labels are too small. In addition, they are referred to as 3(a) and 3(b) in the text (Line 166).

Fig.2(e): I’m having difficulty understanding this figure. Looks like the bottom plots (without BTT) is covering the state space better than the top plots (with BTT)? Why are they in different colors?

Other than these, I do not have specific comments about the writing of the paper. It would be great if the authors could show more robotics-relevant experiments in the future as well.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: In the paper "Efficient and Stable Off-policy Training via Behavior-aware Evolutionary Learning", the authors present a novel algorithm called Behavior-aware Evolutionary Learning (BEL). This algorithm enhances an evolutionary strategy algorithm with two different mutation operators that are taking into account the impact of the mutations on the "behaviour" of the policy and encourages diverse behaviours during each generation. 
The paper proposes multiple experiments to visualise better the effects of their algorithm as well as three ablation studies. Finally, the authors demonstrate the higher performance of their algorithm against state-of-the-art algorithms in six different environments. 
strengths_and_weaknesses: The paper is overall very well written and easy to understand. The algorithm is easy to understand and is well motivated. The experimental evaluations and ablations demonstrate the benefits of the proposed implementation. 

I have three minor issues with the paper: 

- What the paper calls "behavioural" diversity is what we would usually call phenotypic diversity in evolutionary robotics (ER), i.e., changes in the policy outputs. Behavioural diversity in ER usually refers to changes in the robot's behaviour, which includes not only changes in the policy outputs but also how these changes interact with the environment. There is for instance a large field of ER called Quality-Diversity (QD) which focuses on promoting behavioural diversity in Policy Search. QD uses "Behavioural descriptors" for this purpose, and I was expecting some sort of connection between the proposed "behaviour-aware" aspect of the algorithm and behavioural descriptors. It was only after reading half of the paper that I realised the terminology mismatch. If this could be clarified in the paper to avoid confusion that would be great. 

- The figures could be significantly improved: the titles, legends, and ticks are barely readable. This should be addressed. 

- I was unable to find the details on how the weight linear combination is performed. I assume this is relatively trivial (i.e., the weight being proportional to the fitness), but if the paper could specify this it would be great. 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Apart from the three comments raised above, which could be improved during the rebuttal, the paper is very interesting and would be a nice addition at the conference. 
issues: See above.
review_title: null

Meta-review: metareview: Please check the comments of the reviewers in detail.

### Strengths
- the paper is well written and easy to follow
- experimental results (in simulation) are promising (better than SAC, TD3, and CEM-RL)

### Weaknesses
- motivation: the paper needs to explain better why diversity is likely to help 
- definitions: the paper needs to explain better what the authors call a behavior and how they calculate a difference between behaviors
- weak statistics: only 10 replicates of the experiments, no statistical analysis (e.g. Wilcoxon test)
- weakly relevant to robotics: this is generic "RL for basic control" (Mujoco experiments) and does not address any specific challenge of current robotics
- the figures should be improved for readability
- several typos that should be easy to fix

Additional related litterature (not mentioned by the reviewers):
- behavioral sensitivity: Lehman, J., Chen, J., Clune, J., & Stanley, K. O. (2018, July). Safe mutations for deep and recurrent neural networks through output gradients. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 117-124).
- targeting behaviors: Baranes, A., & Oudeyer, P. Y. (2013). Active learning of inverse models with intrinsically motivated goal exploration in robots. Robotics and Autonomous Systems, 61(1), 49-73.

## Post-rebuttal update
I would like to thank the authors for their effort in improving the paper, which has been substantially improved.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)