Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a Diversity Policy Gradient for Quality-Diversity optimization method. The approach solves a problem of finding a collection of well performing policies in continuous control environments. Rather than searching for a single optimal solution, the method aims to find a diverse set of high-performing solutions. As a result, the paper solves exploration and control problems with improved sample-efficiency over its evolutionary competitors.

main_review: The paper is well-written, clear, and easy to follow. It solves an interesting problem with improved sample-efficiency over other state-of-the-art algorithms. The authors considered a good set of experiments. The results and limitations are nicely discussed.

Minor comment: you may want to rewrite lines 170 and 171 in the Related Work section.

limitations_and_societal_impact: Limitations are very well discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces Diversity Policy Gradient (DPG), an algorithm designed to discover diverse neural policies to solve RL problems in a sample efficient manner. The authors adopt a MAP-Elites scheme to select and evolve neural policies however with the additional twist of updating the policies not just by their quality but also according to their novelty from other known solutions. They show their results on several RL challenges where their algorithm successfully avoids local minima and outperforms other known algorithms such as TD3. 
main_review: While I'm somewhat familiar with QD algorithms, I'm not a domain expert, therefore I'm unable to judge the significance/originality of this paper. I found the idea of using QD to evolve neural policies interesting and to me it was novel. 

I found the paper was generally clearly written, and in fact I appreciated the latter parts of the paper where the authors guided the reader to some relevant questions that the paper aims to address. Overall, I found the paper methodologically sound (e.g. ablation studies, benchmarks, and tasks all seemed reasonable and well-motivated), and the communication was quite clear. 

From my perspective this is a well-executed paper, of interest to NeurIPS community.

Minor comment:
- There seems to be a leftover comment in the text in line 170 ("Not clear in it's current form...")

----
Questions:

It was not clear to me how the behavior descriptor and state descriptor extraction functions behave and if the choice of these functions were impactful in whether the algorithm works properly. I would appreciate further clarification in the paper. Are these affine or euclidean maps? Does it matter?

I'm also wondering if these mappings preserve some notion of proximity of solutions, and in some sense allow for "transfer" of good trajectory level ideas between policies implicitly through the grid-level mutation-selection balance? If not, do the authors have ideas about whether this information can be exploited?

I wonder if there is a better update schedule than random diversity and quality updates. The authors mention joint updates performed worse. It's not necessary, but I suggest I authors experiment with annealing/oscillating approaches to DPG/QPG ratios for the population, with a larger period than a single update. My main concern is that perhaps QPG ascent benefits from consecutive updates to find better solutions faster, and the current approach "destroys" the gains from the gradient, leading to sample inefficiency. 
limitations_and_societal_impact: Societal impact is limited in it's current state, though I find that the approach *could* have useful implications for enabling different policies that while equally performant, may have beneficial secondary aspects like ethical superiority. 

The main limitation of the study in my view is that doesn't help prescribe how to pick among multiple neural policies, and does not investigate a comparison between found solutions, i.e. in what sense is "diversity" in solutions more useful than just increasing the chance of finding very good ones.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a Quality-Diversity and Policy Gradient (QD-PG) algorithm that deals with exploration-exploitation trade-off in the optimization problem. The authors validate this algorithm through several benchmark tasks in RL.  The results demonstrate that QD-PG produces competitive solutions especially in the deceptive tasks. The motivation of this paper is clear and the comparison experiments are conducted extensively. 
main_review: Although the idea of combining diversity and quality is not novel, the way of combining QD and PG in this paper is interesting. The cost is the extra evaluation plus the definition of a good metric for measuring the diversity. Some comments need to be clarified:
1. How are the two critics (diversity and quality) updated? In the paper, it is not clearly stated. I believe that the update of the ciritic plays an important role for the performance of this algorithm.
2. Is QD-PG more sample efficient than its QD competitors? For the explanation of this research question, can the authors give some quantative results on the TD3 as well?
3. How many replicates of experiments in Table 1?
4. Can you explain why QD-PG SUM fails to learn the ANT-TRAP?
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper proposes a novel method to learn a set of diverse and high-quality policies. Ablation studies are sufficient to verify that the results are the consequence of the algorithm components. All reviewers gave positive scores. 

However, this is not the first work on learning diverse policies. For examples, learning diverse policies were commonly demanded in game AI [Generating Behavior-Diverse Game AIs with Evolutionary Multi-Objective Deep Reinforcement Learning. IJCAI 2020], and skill discovery also requires diverse sub-policies [Learning Diversity is All You Need: Learning Skills without a Reward Function. ICLR 2019]. The major weakness is that the previous studies on learning diverse policies/skill are all ignored. 

Another weakness is that there is no theoretical guarantee on the convergence of the algorithm and on the level of diversity we can expect. This does not mean that a theoretical guarantee is a must. However, the algorithm design is heuristic. The outcome of the algorithm might be undesired. A discussion on the rationality of the algorithm on how much the diversity can be is expected.

Overall I think this paper contains novelty and contributions, but it should have done a complete survey of related work, contacted experiments comparing other diversity encourage methods, and discussed in-depth on the convergence property.
Decision: Reject