Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
Authors: Thomas PIERROT
Valentin Macé
Geoffrey Cideron
Karim Beguir
Antoine Cully
Olivier Sigaud
Nicolas Perrin
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Deep Learning
Quality Diversity
Evolutionary Computing
Optimization
Exploration

Abstract: A fascinating aspect of nature lies in its ability to produce a large and diverse collection of organisms that are all high-performing in their niche. By contrast, most AI algorithms focus on finding a single efficient solution to a given problem. Aiming for diversity in addition to performance is a convenient way to deal with the exploration-exploitation trade-off that plays a central role in learning. It also allows for increased robustness when the returned collection contains several working solutions to the considered problem, making it well-suited for real applications such as robotics. Quality-Diversity (QD) methods are evolutionary algorithms designed for this purpose. This paper proposes a novel algorithm, QD-PG, which combines the strength of Policy Gradient algorithms and Quality Diversity approaches to produce a collection of diverse and high-performing neural policies in continuous control environments. The main contribution of this work is the introduction of a Diversity Policy Gradient (DPG) that exploits information at the time-step level to thrive policies towards more diversity in a sample efficient manner. Specifically, QD-PG selects neural controllers from a MAP-Elites grid and uses two gradient-based mutation operators to improve both quality and diversity, resulting in stable population updates.  Our results demonstrate that QD-PG produces collections of diverse solutions that solve challenging exploration and control problems while being two orders of magnitude more sample efficient than its evolutionary competitors.

Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization
Anonymous Author(s)
Affiliation
Address
email
Abstract
A fascinating aspect of nature lies in its ability produce large and diverse collection of organisms that are all nigh-performing in their niche. By contrast, most AI algorithms focus finding single efficient solution to given problem. Aiming for diversity in addition to performance is a convenient way to deal with the exploration-exploitaiie trade-off that plays central role in learning. It also allows for increased robustness when the returned collection contains several working solutions the considered problem, making it well-suited for real applications such as robotics. for this purpose. This paper proposes novel algorithm, QD-PG, which combines the strength of Policy Gradient algorithms and Quality Diversity approaches to produce a collection of diverse and high-performe perfhrrrffrmgggggg neural policies continuous control environments. The main contribution of this work is the introduction of a Diversity Policy Gradient (DPG) that exploits information at the time-step level thrive policies towards more diversity in sample- cample-efficient manner. Specifically, QD-PG selects neural controllers from a MAP-Elites grid and uses two gradient- based mutation operators to improve both quality and diversity, resulting in stable population updates. Our results demonstrate that QD-PG generates collections of di- verse solutions that solve challenging exploration and control problems while being two orders of magnitude more sample-efficient than its evolutionary competitors.
2
4
6
(QD) methods are evolutionary algorithms designed
10
11
13
14
15
17
18
19
Introduction
20
Natural evolution has the fascinating ability to produce diverse organisms that are all well adapted to their respective niche. Inspired by this ability to produce a tremendous diversity of living systems, Quality-Diversit (QD) is a new family of optimization algorithms that aims at searching for 24 collection of both diverse and nigh-performing solutions (Pugh et al.) 2016 Cully & Demiris, 2017).
21
23
25 While classic optimization methods focus on finding a single efficient solution, QD optimization aims to cover the range of possible solution types and to return the best solution for each type. This process sometimes referred to as "illumination" " in opposition to optimization, as reveals (or illuminates) search space of interest often called the behavior descriptor space (Mouret & Clune 2013).
27
29 The principal advantage of QD approaches resides in their intrinsic capacity to deliver large and
30 diverse set of working alternatives when single solution fails (Cully et al.) 2015). By producing collection of solutions instead unique one, QD algorithms allow to obtain different ways to solve single problem, leading to greater robustness, which can help to reduce the reality gap when applied
31
33 to roboties (Koos et al., 2012). Diversity seeking is the core component that allows QD algorithms to generate large collections of diverse solutions. By encouraging the emergence of novel behaviors in 35 the population without focusing on performance alone, diversity seeking algorithms explore regions of the behavior descriptor space that are unreachable for conventional algorithms (Doncieux et al.
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

E
16
The agent robot rewarded for running forward fast as possible. Following the reward signal without further exploration leads the agent into the trap, which corresponds to poor local minimum. QD-PG produces collection of solutions that are diverse and performnf,,,, allowing to find several working alternatives to solve a deceptive control problem.
Figure I:
37
Another benefit of QD is its ability to solve hard exploration problems where the reward signal sparse or deceptive, and on which standard optimization techniques are ineffective (Colas et al.) 2020). This ability can be interpreted as a direct consequence of the structured search for diversity in the behavior descriptor space.
2019).
38
39
40
41
Quality-Diversity algorithms build on black-box optimization methods such as evolutionary algo- rithms evolve population of solutions (Cully & Demiris 2017). Historically, they rely on random mutations to explore small search spaces but struggle when facing higher-dimensional problems. As result, they often scale poorly problems where neural networks with many parameters provide state-of-the-art results (Colas et al., 2020).
42
43
44
45
Building large and efficient controllers that work with continuous actions has been a long-standing goal in Artificial Intelligence and in particular in robotics. Deep reinforcement learning (RL), and especially Policy Gradient (PG) methods have proven efficient training such large controllers (Schulman et al. 2017 Lillicrap et al. 2015 Fujimoto et al. 2018 Haarojja et al. 2018). One the keys this success lies in the fact that PG methods exploit the structure of the objective function when the problem can be formalized as Markov Decision Process (MDP), leading to substantial gains in sample efficiency. Moreover, they also exploit the analytical structure of the controller when known, which allows the sample complexity of these methods to be independent of parameter space dimensionality (Vemula et al.) 2019). In real-world applications, these gains turn out to be critical when interacting with the environment is expensive. PG methods usually rely on simple exploration mechanisms, like adding Gaussian noise (Fujimoto et al. 2018) or maximizing entropy (Haarnoja) et al. 2018) to explore the action space, which happens to be insufficient in hard exploration tasks
47
49
51
52
53
54
55
56
57
58
where the reward signal is sparse or deceptive (Colas al. 2018 Nasiriany et al. 2019).
59
Successful attempts have been made to combine evolutionary methods and reinforcement learning (Kaadka et al. 2019 Khadka & Tumer 2018 Pourchot & Sigaud] 2018 [Shi et al. 2020). However, all these techniques only focus on building high-perforning solutions and do not explicitly encourage diversity within the population. In this regard, they fail when confronted with hard exploration problems. To address these problems, one needs to seek both high-performing solutions and diversity within them.
51
82
63
64
65
Contributions
66
In this work, we introduce the idea of a diversity policy gradient (DPG) that thrives solutions towards more diversity. We show that the DPG can be used in combination with the standard policy gradient,
67
68 dubbed quality policy gradient (QPG), to produce high-performing and diverse solutions. Our
69 algorithm, called QD-PG, builds on MAP-Elites (Mouret & Clune 2015), demonstrates remarkable sample efficiency brought by off-policy PG methods, and produces collections of good solutions
71 in single run (see Figure Tİ. We compare QD-PG to state-of-thhe-ar RL algorithms and to several evolutionary methods known as Evolution Strategies (ESs) augmented with a diversity objective,

---

73 namely the NS-ES family (Conti et al. 2018) and the ME-ES ES algorithm (Colas et al. 2020). We
74 show that QD-PG generates collections robust solutions hard exploration problems while RL algorithms struggle to produce single one, and that QD-PG is two orders of magnitude more sample efficient than the best of its evolutionary competitors.
75
76
Background
77
78
Problem statement
We consider an MDP (S.A.R, where is the state space, A the action space, R S A
80 the reward function, : x A S the dynamics transition function and y a discount factor. policy, To S - A parameterized by E θ, which is called a solution to the problem. We say that a solution is 83 highy-performing if the expectation over the sum of rewards is high when using TT.. The fitness of solution measures its performance F: θ + IR where F(O)- = E 2r're.
81 We assume that both S and A are continuous and consider a controller,
82
84
To characterize the novelty of solution w.r.t J other solutions, in QD methods, we introduce behavior descriptor (BD) space B, a behavior descriptor extraction function & θ - B, and define a distance metric |I-Ils over B. The novelty n @ O - RR+ of a solution w.r.t. a list of solutions Σ, IK((0), [(0,)]lss. In other words, we quantify the novelty of a solution w.r.t. a list of solutions as the sum of distances between its behavior descriptor and the behavior descriptors of all solutions of the list. We also use the distance ||-1|x to characterize
86
(0)) j=1..... is defined as (0, (0.))=...))))
88
89
81 the diversity of a set of K solutions [0s)x=1.....K We formally define diversity d: @ -> IR+ as
d
Σ min ((())).((E))l
A
92 meaning that a set of solutions diverse if the solutions are distant with respect to each other in the sense of ||-I|
93
94
The MAP-Elites algorithm
95 MAP-Elites (Mouret& & Clune 2013) is a simple yet state-of-the-art QD algorithm that has been successfully applied to wide range of challenging problems such as robot damage recovery (Cully et al. (015), molecular robotic control (Cazenille et al. (019)) and game design (Alvarez et al.) 2019).
96
97
98
In MAP-Elites the behavior descriptor space B is discretized into grid of cells, also called niches, with the aim filling each cell with a high-performing solution. The algorithm starts with an empty grid and initial random set of K solutions that are evaluated and added to the grid by following simple insertion rules. the cell corresponding to the behavior descriptors of a solution is empty, then
99
101
102
the solution is added to this cell. If there is already a solution in the cell, the new solution replaces it only if it has greater fitness. At each iteration, P existing solutions are sampled uniformly from the grid and randomly mutated to create P new solutions. These new solutions are then evaluated and added to the grid following the same insertion rules. This cycle is repeated until convergence or for a given budget of iterations.
103
105
106
107
Though MAP-Elites is a compelling and efficient method, suffers from a low sample efficiency as it relies on random mutations. Recently. Colas al. (2020) tackled this problem by updating the solutions through an Evolution Strategy known as the Cross- Pooss-Entrooyy method. Notably, they showed that MAP-Elites could be scaled with their method to address complex MUJoCo control environments at the cost of very large computational resources. In this study, we propose to harness policy gradients (QPG and DPG) to build more sample-efficient MAP-Elites approach.
109
111
3 Key Principle: Diversity Policy Gradient
114 Let us assume that we have a MAP-Elites grid containing K solutions (01....,0p, 0x). To increase 115 diversity in the grid using the DPG, we need to update one sampled solution 0 from the grid using 116 gradient ascent. To do so, we aim to compute the gradient of the population diversity w.r.t. where diversity is defined in Equation I As the K solutions are independent, order does not matter and
3

---

118
we can consider optimizing arbitrarily 0 = 81. To compute the gradient of d w.r.t. 81, we need
119
to separate the terms that depend on 0 from the others. The terms that depend on 8, correspond
120
to the distance of 01 to its nearest neighbor, which we define as 02, and to the distances of 01 to
the Os for which 01 is the nearest neighbor. We can arbitrarily index them from 3 to 1 thus:
121
d( [0x ((8x)==.....k)) = و (l((())) E(9y)ll= + M, where M- Eie(t.....) min {((x)))
122
Right
Only the first term of the sum depends on 0 0r. Furthermore, observe that this term equals the novelty of solution 81 w.n.t the list (0g)25)su- Therefore, the gradient of diversity w.r.t. 01 is
124
Ve, d( (9x k) = Ve, n(01, (03) (0))23)s.). That is, we can increase the diversity of the population by increasing the novelty of 01 w.r.t the list (0, (0))25is3' In practice, we replace this list by list of nearest neighbors of 81, as this is easier to compute and the elements of (0y)25is) tend to be among the nearest neighbors of 01.
125
128
128
129
Under this form, the diversity gradient cannot benefit from the variance reduction methods in the RL literature to efficiently compute policy gradients Sutton al. (1999). To this end, we need to express as a gradient over the expectation of a sum of scalar quantities obtained by policy TT, at each step when interacting with the environment. Therefore, to build DPG, we need information about the novelty of solution at the time step level. To do so, we introduce a novel space D, dubbed state descriptor space and state descriptor extraction function &p S + D. We assume DD and B have the solution, we now define the novelty of a state s w.n.t J
131
133
135
same dimension. Similarly to the novelty
other states (=1......... J as n. : S X S + R such that n(s, (8)) (s)))=1.....) Σラ [[v(s), (85))]l,,
136
where ||-I|p is distance metric over D.
138
Now, we need to link novelty defined at the time step level to novelty defined at the solution level. We define the novelty of state w.r.t. set of solutions. We say that state is novel w.r.t. some solutions
139
140
the state is novel w.r.t. to the states visited by these solutions. More formally:
(00) (0g)
Σ Birey (())))).
(2)
While we adopt this definition in this paper, one might as well consider other definitions where, for 142 instance, a state is compared to states that have been visited at the same time step during another episode. In this context, if the following relation is satisfied:
BITE, Σ n(ss, (0))25is)) n(01, (0,0),(0))2c) (0))23/su)
(3)
144
then we can compute the DPG of d w.r.t. 0, as
DPG V 0:
= Vo, Era, (n(s)(0))sss)))
(4)
145
This expression corresponds to the classical policy gradient setting where and where the corresponding reward signal, here dubbed diversity reward, is computed as r = no(s, (0z) 25)zs)- Therefore, this gradient can be computed using any PG estimation technique replacing the environ- ment reward by the diversity reward r..
146
148
149
B) enforces a relation between B and D and between extraction functions vp and &. In practice, it may be hard to define the behavior descriptor and state descriptor of solution that satisfy this relation while being meaningful to the problem at hand and tractable. But strict equality is not necessary. suffices that an increase on the left-hand side implies an increase on the right-hand side
Equation
150
151
152
153
so that we can still update 0 using 4 Furthermore, when this is not the case, the diversity gradient update might not result in an increase of diversity in the behavior descriptor space, but in that case the MAP-Elites insertion rule will remove the corresponding solution. We show in Section that we can define descriptors that do not satisfy the above relation all the time, but still give satisfactory results.
154
196
Remark: 0g can appear twice in the list (0) )25)5)

---

157
4
Related Work
158
distinguishing feature of our approach that we combine diversity seeking at the level of trajectories using behavior descriptors and diversity seeking in the state space using state descriptors. The former is used by MAP-Elites to select solutions from the grid and contributes structural bias towards diversity, whereas the latter is used during policy gradient steps in the RL part, see Figure 2b We organize the literature review below according to this split between two types of diversity seeking mechanisms.
160
162
183
Q search in the solution space
164
Simultaneously maximizing diversity and performance is the central goal of QD methods (Pugh et al. 2016: Cully & Demiris, (017). Among the various possible combinations offered by the QD framework, Novelty Search with Local Competition (NSLC) (Lehman & Stanley 201 Tb) and
165
166
187
MAP-Elites (Mouret & Clune,,201 (015) are the two most popular algorithms. NSLC builds on the Novelty
168
Search (NS) algorithm (Lehman & Stanley, 201 and maintains an unstructured archive of solutions
169
selected for their local performance while MAP-Elites uniformly samples individuals from a structured
170
grid that discretizes the BD space. Not clear in its current form. I suggest: "QD-PG uses the standard grid of MAP-Elites. However, we also show in Appendix | that QD-PG can be used with alternative archive structures.
171
172
179
With the objective of improving their data-efficiency, QD-ES algorithms that combine QD and ESs, such NSR-ES and NSRA-ES, have been applied to challenging continuous control environments in Conti et al. (2018). But, as outlined in Colas et al. (2020). they suffer from poor sample efficiency and the diversity and environment reward functions could be mixed in more efficient way. In that respect, the most closely related work w.r.t. ours is ME-ES (Colas et al. 2020). The ME-ES algorithm also optimizes quality and diversity using MAP-Elites and two ES
174
175
178
177
178
Using these methods was shown to be critically more efficient than iopulation-based GA algorithms (Salimans et al.) 2017). but our results show that they are still less sample efficient than off- off-policy deep RL methods, as they
179
180
181
do not leverage the analytical computation of the policy gradient at the time step level. To the best of our knowledge, no QD or ES algorithm use an explicit critic for both performance and diversity.
182
183
resulting in even higher data-efficiency.
184
Q search in the state or action spaces
185
Seeking for diversity in the space of states or actions is generally framed into the RL framework. This is the case of algorithms maintaining a population of RL agents for exploration without an explicit diversity criterion (Jaderberg et al. 2017) or algorithms explicitly looking for diversity but in the
186
187
action space rather than in the state space like ARAC (Doan al. 2019). P3S-TD3 (Jung al. 2020 and DVD (Parker-Holder et al. 2020).
189
190
An exception is Stanton & Clune (2166) who define a notion of intra-life novelty that is similar to our state novelty defined in Section 3 However, their novelty relies on skills rather than states. Our
191
192
work is also related to algorithms using RL mechanisms to search for diversity only (Eysenbach et al., 2018, Pong et al., 2019 Lee et al., 2019 Islam et al., 2019). These methods have proven useful in sparse reward situations, but they are inherently limited when the reward signal can orient exploration, as they ignore it. Other works sequentially combine diversity seeking and RL. The GEP-PG algorithm Colas et al. (2018) combines a diversity seeking component, namely Goal Exploration Processes Forestier et al. 2017) and the DDPG deep RL algorithm (Lillicrap et al.) 2015). This sequential is also present in CO-EEXPLORE (Ecoffet et al. 2019). Again, this approach is limited when the reward signal can help driving the exploration process to efficient solutions. These sequential approaches first look for diversity in the behavior descriptor
193
195
196
197
combination of exploration- wwwpatiinn-tee-eepoottttt
199
200
201
space, then optimize performance in the state action space, whereas we do so simultaneously in the behavior deseriptor space and in the state space.
202
203
To the best of our knowledge, QD-PG is the first algorithm optimizing both diversity and performance in the solution and in the state space, using a sample-efficient policy gradient computation method
205
for the latter.
5 Methods
206
207
Our full algorithm called QD-PG, its pseudo code is given in Appendix A and its architecture is depicted in Figure 2 QD-PG is iterative algorithm based MAP-Elites that replaces random
5

---

209
mutations with policy gradient updates. As we consider a continuous action space and want to
210
improve sample efficiency by using an off-policy policy gradient method, we rely on the Twin Delayed Deterministic Policy Gradient (TD3) algorithm (Fujimoto et al. 2018). See Appendix for a detailed description of TD3.
212
(8, ((9))
Evaluation
Replay Buffer
plas
ng
Slnde Deaer plara
Population
MAP-Elites
Half of the solutions are updated for diversity
Diversity PG
Diversity Cntcc Q
Quality
Quality Q?
Wlen
updited quality
Sampling Solutions
(a)
Figure 2: (a): The RI. part of QD-PG operates at the time step level while the QD part operates at the controller level, considering the MDP black box. (b) One QD-PG iteration consists of three phases: 1) A new population of solutions is sampled from the MAP-Elites grid. 2) These solutions are updated by an off-policy RL agent: half of the solutions are optimized for quality and the other half for diversity. The RL agent leverages one shared critic for each objective. 3) The newly obtained solutions evaluated in the environment. Transitions stored in replay buffer while the updated solutions, their final scores and behavior descriptors are stored in the MAP-Elites grid.
QD-PG maintains three permanent structures. In the QD part, MAP-Elites grid stores the most novel and performing solutions. the RL part, a replay buffer contains all transitions collected when evaluating solutions and archive A stores all state descriptors obtained so far. QD-PG starts with an initial population of random solutions, evaluates them and inserts them into the MAP-Elites grid. At each iteration, solutions are sampled from the grid, copied, and updated. The updated solutions then evaluated through one rollout in the environment and inserted into the grid according to insertion rules. Transitions collected during evaluation are stored in the replay buffer, and state descriptors are stored in the archive A. Note that these state descriptors are first filtered to avoid insertion in the archive of multiple state descriptors that are too close to each other.
214
216
217
218
219
220
221
222
During the update step, half the population is updated with QPG ascent and the other half with DPG ascent. The choice of whether agent is updated for quality diversity is random, meaning that it can be updated for quality and later for diversity if selected again. To justify this design, we show in 224
225
Section that updating consecutively for quality and diversity outperforms updating based on joint criteria. Both gradients are computed from batches of transitions sampled from the replay buffer. The
226
227
QPG is computed usual from rewards whereas for DPG, we get fresh novelty rewards as
D r
Σ (1b(s)), (b;)))
where (8j){=1..... are the J nearest neighbors of state 8; in the archive A. Diversity rewards must be recomputed each update because A changes during training. Following Equation (2). diversity rewards should be computed as the sum of the distances between the descriptor of s, and the descriptors of all the states visited by list of / solutions. In practice, consider the / nearest neighbors of st. This choice simplifies the algorithm and faster and works well in practice.
230
232
233
TD3 relies on a parameterized critic to reduce the variance of its policy gradient estimate. In QD-PG, we maintain two parameterized crities Q and Q? respectively dubbed diversity and quality crities, every time a policy gradient is computed, QD-PG also updates the corresponding critic. In fact, as TD3, we use pairs of critics and target crities fight the overestimation bias. We share the critic
234
296
parameters among the population in Pourchot & Sigaud (2018). Reasons for doing come from

---

238
the fact that diversity is not stationary. as it depends on the current population. If each agent had its own diversity critic, since an agent may not be selected for large number of generations before being selected again, its critic would convey an outdated picture of the evolving diversity. We tried this solution, and it failed. side benefit of critic sharing is that both critics become accurate faster as they combine experience from all agents. Additional details on QD-PG implementation are available in Appendix
239
241
242
243
Experiments
244
In this section, we intend to answer the following matters: 1. Can QD-PG produce collections of diverse and high-performing neural policies and what are the advantages to do so? 2. Is QD-PG more sample efficient than its QD competitors? 3. To what extent are the considered benchmarks difficult for classical policy gradients methods? 4. What is the usefulness of the different components of QD-PG?
246
248
249
250
Environments
251
We asses QD-PG capabilities in continuous control environments that exhibit high dimensional observation and action spaces as well as strong exploration difficulties. Two types reward signals, dubbed sparse and deceptive, are known to be particularly difficult for classical RL methods. These rewards appear in many applications such as robotics or combinatorial optimization. Sparse rewards are obtained if given condition is specified, leading to majority of null rewards and to credit assignment difficulties. Deceptive rewards are dense signals, i.e., they are non-zero at each time step
252
253
254
255
256
but can mislead the search process to some local optimum. In such problems, good approach to the -expoootooonoooiiiiiionnnn trade-off is essential. The agent should learn when to ignore the reward signal and explore to avoid local minima and when to follow it to increase its return. Deceptive environments constitute a natural choice to highlight QD efficiency to balance exploration and exploitation. In this study, we consider three OpenAl Gym environments based on the MuJoCo physics engine that all cxhibit strong deceptive rewards (illustrated in Appendix 5). Such environments have been widely
258
280
262
263
used in previous works (Parker-Holder et al. [2200] Colas al. 2020 Frans et al.) 2018 [hi et al.)
264
2020) for their deceptive nature, a characteristic that is absent of more widespread continuous control
285
environments like HALFCHEETAH-VE HOPPER-V2 or still ANT-V2.
266
In the POINT-MAZE AZE environment, an agent represented as a green sphere must find the exit of the maze depicted in Figure represented as a red sphere. An observation contains the agent position at time 4, and an action corresponds to position increments along the x and y axes. The reward is expressed as the negative Euclidean distance between the center of gravity of the agent and the exit center. The trajectory length cannot exceed 200 steps.
268
270
271
The ANT-MAZE environment is modified from OpenAl Gym ANT-V2 (Brockman et al., 2016) and also used in (Colas et al.) 2020 Frans et al., 2018). In ANT-MAZE, a four-legged ant has to reach a goal zone located in the lower right part of the maze (colored in green in Figure 4b). Its initial position is sampled in small circle located in the maze's 's extreme bottom left. As TOINT-MAZE the reward is expressed as the negative Euclidean distance between the ant and the center of the goal zone. Maze walls are organized so that following the gradient of the reward function drives the ant into a dead-end. In A ANT-MAZE, the final performance is defined as the maximum reward received during an episode. The environment is considered solved when an agent obtains a score superior to
272
273
274
276
277
278
10, corresponding reaching the goal zone. An episode consists of 3000 time steps, this horizon is three times larger than in usual MUJoCo environments, making this environment particularly challenging for RL based methods (Vemula al. 2019).
280
282
Finally, the ANT-TRAP environment also derives from VNT-V2 and is inspired from (Colas et al.. 2020 Marker-Holder et al. 2020). In ANT-TRAP, the four-legged ant initially appears in front of a trap and must bypass to run as fast as possible in the forward direction (see Figure(4c) as in ANT-V2,
283
284
the reward is computed as the ant velocity on the x-axis. The trap consists of three walls forming dead-end directly in front of the ant, leading to a strong deceptive reward. In this environment, the
286
trajectory length cannot exceed 1000 steps. As opposed to POINT-MAZE and ANT-MAZE, where the
288
objective is to reach the exit area, there is no unique way to solve ANT-TRAP and we expect a QD algorithm to generate various effective solutions depicted in Figure
7

---

290
Baselines and Ablations
QD-PG is compared to three types of methods. First, to answer question 2, we compare QD-PG to family of QD baselines, namely ME-ES, NSR-FS, and NSRA-FS (Colas et al. 2020). Appendix E. recaps the properties of all these methods. Second, answer question 3, we compare QD-PG to
292
288
294
family of policy gradient baselines. Soft Actor Critic (SAC) (Haarnoja et al. 2018) and the Twin
295
Delayed Deep Deterministic policy gradient (TD3) (Fujimoto et al. (0188) are continuous control
296
algorithms achieving state-of-the-art results on MuJoCo benchmarks. Random Network Distillation
297
(RND) [Burda et al. 2018) is a curiosity-driven RL agent (Schulman et al. (017) which was shown to perform well in hard exploration settings. CEM-RL RL (Pourchot & Sigaud 2018) mixes Cross- Fross-Entropy
298
299
Methods (CEM) and RL to evolve population of agents to maximize quality and obtains state- of- results MuJoCo benchmarks. Finally, to answer question 4, we propose to investigate the following matters: Can we replace alternating quality and diversity updates by single update that optimizes for the sum of both criteria? Are quality gradients updates alone enough to fill the
300
301
302
303
MAP- Alitee grid? Are diversity gradients updates alone enough to do so? Consequently, we consider the following ablations of QD-PG: QD-PG SUM computes gradient to optimize the sum of the quality and diversity rewards, D-PG applies only diversity gradients to the solutions, and Q-PG applies only quality gradients, but both D-PG and Q-PG still use QD selection (see Appendix E.T.
305
307
We compare QD-PG to its ablations and RL competitors in all enviromments and show results in Table La Detailed results including graphic charts and coverage maps are given in Appendix and more details about the evaluation procedure are given in Appendix E.
308
309
Results
310
1. Can QD-PG produce collections of neural policies and what are the ad advantages to do so?
312
Table La presents QD-PG performances. In all environments, our algorithm manages to find working solutions that avoid local minima and reach the overall objective. In addition to its exploration capabilities, QD-PG generates collections of high performing solutions in a single run. During the ANT-TRAP experiment, the final collection of solutions returned by QD-PG contained, among others. 5 solutions that were within 10% performance margin from the best one. As illustrated in Figure] [ these agents typically differ in their gaits and preferred trajectories to circumvent the trap.
313
314
315
316
318
Generating a collection of diverse solutions comes with the benefit of having a repertoire of diverse solutions that can be used as alternatives when the MDP changes (Cully et al., 2015). We show that QD-PG is more robust than conventional policy gradient methods by changing the re- ward signal of the ANT-MAZE environment. We replace the original goal in the bottom right part of the maze (see Figure 3) with a new randomly located goal in the maze. Instead of running QD-PG to optimize for this new objec- tive, we run a Bayesian optimization process to quickly find a good solution among the ones already stored in the grid. With a budget of only 20 solutions to be tested during the Bayesian optimization process, we are able to quickly recover good solution for the new objective. We repeat this experiment 100 times, each time with a different ran- dom goal, and obtain average performance of - 110 with a standard deviation of 9. In other words, 20 interaction episodes (corresponding to 60.000 time steps) suffice for the adaptation process to find a solution that performs well for the new objective without the need to re-train agents. More detailed results can be found in Appendix E.3
319
320
321
322
323
324
325
326
Figure 3: QD-PG produces a collection diverse solutions. In ANT-MAZE, even after setting new randomly located goals, the MAP-Elites grid still contains solutions that are suited for the new ob- jectives.
328
330
332
333
334
336
2. Is it more sample efficient than its QD competitors?
338
Table db compares QD-PG to Deep Neuroevolution algorithms with diversity seeking component in terms of sample efficiency. QD-PG runs on 10 CPU cores for 2 days while its competitors used 1000
339
310 CPU cores for the same duration. Nonetheless, QD-PG matches the asymptotic performance of ME-ES using two orders of magnitude fewer samples, explaining the lower resource requirements.
Videos of QD-PG agents are available at: https://sites. wwwp:///w....//////////a//aaaapp
8

---

Table Results for all environments. Final Perf. is the minimum distance to the goal in ANT-MAZE and the episode return in TOINT-MAZE and ANT-TRAP. The Ratio to ours column compares the sample efficiency of a method to QD-PG.
(a) Comparison to ablations and PG baselines.
(b) Comparison to evolutionary competitors.
Final Perf. (= st)
ANT-MA2E
Algorilium
POINT-MAZE
ANT-MAZE
ANT-TRAP
Algorillum
Final Perf.
Sleps to goul
Ralio to ours
QD-FG
24(+0)
-7(77)
1541(+86)
QD-PG
-(177
1.15
QD-PG SUM
-25(+1))
-5(43)
1018(46)
CEM-RL
-2((+0)
00
D-PG
+37)
-2(+0)
1016(48)
ME-CS
-5(+1)
2.1e10
209
Q-PG
-128(+0)
-26(+0)
1175(=70)
NSR-ES
-26(+0)
CBM-RL
-312(+1)
-6
931(+22)
NSRA ES
-2(11)
2.1e10
182
-127(+1)
-39(11)
1019(+21]
TD3
-1)
-26(10)
1131(17)
RND
-35(+10)
-27(#1)
978(+61)
We see three reasons for the improved sample efficiency of QD-PG: 1) QD-PG leverages replay buffer and can re-use each sample several times. 2) QD-PG leverages novelty at the state level and can exploit all collected transitions to maximize quality and diversity. For instance, in ANT-MAZE. a trajectory brings 3000 samples to QD-PG while standard QD methods would consider it a unique sample. 3) PG exploits the analytical gradient between the neural network weights and the resulting policy action distribution and estimates only the impact of the distribution on the return. By contrast, standard QD methods directly estimate the impact on the return of randomly modifying the weights.
345
345
347
348
349
To what extent the considered benchmarks are difficult for policy gradients methods?
350
Table Ia compares QD-PG to state-of-the-an policy gradient algorithms and validates that classical policy gradient methods fail to find optimal solutions in deceptive environments. TD3 quickly converges to local minima of performance resulting from being attracted in dead-ends by the deceptive gradients. While we may expect SAC to better explore due to entropy regularization, it also converges to that same local minima in ANT-TRAP and POINT-MAZE. Besides, despite its exploration mechanism based on CEM, CEM-RL also quickly converges to local optima in all benchmarks, confirming the need for a dedicated diversity seeking component. RND, which adds an exploration bonus used as an intrinsic reward (see Appendix for more details), also demonstrates performances inferior to QD- PG in all environments but manages to solve POINT- MAZE. In ANT-MAZE and ANT-TRAI as shown in Appendix G.2 RND extensively explores the BD space but fails to obtain high returns.
351
352
353
354
355
356
357
358
359
380
4. What is the usefulness of the different components of QD-PG?
The ablation study in Table la shows that when maximising quality only, Q-PG fails due to the deceptive nature of the reward and when maximizing diversity only, D-PG sufficiently explores to solve the problem in both POINT-MAZE and ANT-MAZE but requires more steps and finds lower- performing solutions. When optimizing simultancously for quality and diversity, QD-PG SUM fails to learn in ANT-TRAP and manages solve the task in ANT-MAZE but requires more samples than QD-GG. We hypothesize that quality and diversity rewards may give rise to conflicting gradients. For instance, the beginning of training in ART-TRAP, the quality reward drives the ant forward whereas the diversity reward drives it back to escape the trap and explore the environment. Therefore, both rewards cancel each other, preventing any learning. This study validates the usefulness of QD-PG components: 1) optimizing for diversity is required to overcome the deceptive nature of the reward; 2) adding quality optimization provides better asymptotic performance; 3) it better to disentangle quality and diversity updates.
382
364
365
366
358
369
370
371
372
373
8
Conclusion
374
This paper is the first to introduce a diversity gradient to explore diversity both at the state and skill levels. Based on this component we proposed novel algorithm, QD-PG, inspired from the Quality-Diversipy literature, that produces collections of diverse and high-performing neural policies in sample-efficient manner. We showed experimentally that QD-PG generates several solutions that achieve high returns in challenging exploration problems. Finally, we demonstrated that in few interactions with the environment, QD-PG finds alternative solutions that still obtain good performance when the MDP changes.
375
376
378
380

---

References
382
Alvarez, A., Dahlskog, S., Font, J., and Togelius, J. Empowering quality diversity in dungeon design with interactive constrained map-elites, In 2019 IEEE Conference Games (CoG), pp. 1-8. IEEE, 2019.
384
385
Brockman, G., Cheung, V., Pettersson, L.,, Schneider, J., Schulman, J., Tang, J., and Zaremba, W.
Openai gym. arXiv preprint arXiv: 1606.0114)) 2016.
387
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. XrXiv preprint arXiv:1810. 1xX.:18100.1894,, 2018.
388
Cazenille, L., Bredeche, N., and Aubert-Kato, N. Exploring self-assembling behaviors in a swarm of
390
bio-micro-robots using surrogate-asisted map-elites. arXiv preprint (rXiv:19100002:3 2019.
391
Colas, C., Sigaud, O., and Oudeyer, P.-Y. GEP-PG: Decoupling exploration and exploitation in deep
reinforcement learning algorithms. arXiv preprint arXiv:/802.05054. 2018.
Colas, C.,, Madhavan, V., Huizinga, J., and Clune, J. Scaling map-elites to deep neuroevolution. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 67-75, 2020.
394
395
Conti, E., Madhavan, V., Such, F. P., Lehman, J., Stanley, K., and Clune, J. Improving exploration in evolution strategies for deep reinforcement learning via population of novelty-sceking agents, In Advances in neural information processing systems, pp. 5027-5038, 2018.
396
397
Cully, A. and Demiris, Y. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245-259, 2017.
399
400
Cully, A., Clune, J., Tarapore, D., and Mouret, J.-B. Robots that can adapt like animals. Nature, 521 (7553):503- 507, 2015.
Doan, T., Mazoure, B., Durand, A., Pineau, J., and Hjelm, R. D. Attraction-repulsion actor-critic for continuous control reinforcement learning, arXiv preprint arXiv:190900544. 2019.
403
404
Doncieux, S., Laflaquière, A., and Coninx, A. Novelty search: theoretical perspective. In
405
Proceedings of the Genetic and Evolutionary Computation Conference, pp. 99-106, 2019.
406
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, J. Go-explore: a new approach for hard-exploration problems. arXiv preprint 19.1:::901.1090000 2019.
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without a reward function. arXiv preprint / 2018.
409
410 Forestier, S., Mollard, Y., and Oudeyer, P.-Y. Intrinsically motivated goal exploration processes with
automatic curriculum learning. arXiv preprint arXiv:1708 02090, 2017.
412
Frans, K., Ho, J.. Chen, X., Abbeel, P., and Schulman, J. Meta learning shared hierarchies. Proc. of ICLR, 2018.
413
Fujimoto, S., Van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic methods. urXiv preprint arXiv: (Tfv:/80000000 09477, 2018.
415
416
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A.,
Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05000 2018.
418
419
Islam, R., Ahmed, Z., and Precup, D. Marginalized state distribution entropy regularization policy optimization. arXiv preprint arXiv:1912.05/222 2019.
420
421
Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki. W. M., Donahue, J.. Razavi, A., Vinyals, O..
422
Green, T., Dunning, L. Simonyan, K., et al. Population- based training of neural networks. arXiv preprint arxiv: 1.09846, 2017.
423
Jung, W., Park, G., and Sung, Y. Population-guidee parallel policy search for reinforcement learning. In International Conference on Learning Representations, 2020.
424
10

---

428
Khadka, S. and Tumer, K. Evolution-guice guided policy gradient in reinforcement learning, In Neural
427
Information Processing Systems, 2018.
428
Khadka, S., Majumdar, S., Miret, S., Tumer, E., Nassar, T., Dwiel, Z., Liu, Y., and Tumer, K.
429
Collaborative evolutionary reinforcement learning. arXiv preprint arXiv: 1905.0060 2019.
430
Koos, S., Mouret, J..B., and Doncieux, S. The transferability approach: Crossing the reality gap in
431
evolutionary robotics. IEEE Transactions on Evolutionary Computatioo, 17(1):122-145, 2012.
Lee, L.,, Eysenbach, B., Parisotto, E., Xing, E., Levine, S., and Salakhutdinov, R. Efficient exploration via state marginal matching. arXiv preprint
433
2019.
434
Lehman, J. and Stanley, K. O. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189 223, 2011a.
I Lehman, J. and Stanley, K. O. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation. pp. 21 211-211 1-218. 2011b.
437
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wienstra, D. Continuous control with deep reinforcement learning. arXiv preprint 1509./:500.022 2015.
440
441
Mouret, J.-B. and Clune, J. Illuminating search spaces by mapping elites. arXiv preprint
442
arXiv: 044999 2015.
443
Nasiriany, S., Pong. V. H., Lin, S., and Levine, S. Planning with goal-conditioned policies. arXiv preprint liv: 100453,,8333 2019.
444
445
Porker-Holder, J., Pacchiano, A., Choromanski, K., and Roberts, S. Effective diversity in population-
446
based reinforcement learning. In Neural Information Processing Systems, 2020.
447
Pong, V. H., Dalal, M., Lin, S., Nair, A., Bahl, S., and Levine, S. Skew-fit: State-covering self- supervised reinforcement learning. arXiv preprint #xXv::903.00399. 2019.
Pourchot, A. and Sigaud, O. Cem-rl: Combining evolutionary and gradient-based methods for policy search. arXiv preprint arXiv:1810.01222 2018
450
451
Pugh, J. K., Soros, L. B., and Stanley, K. O. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016.
453
Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever, I. Evolution strategies as a scalable alternative to reinforcement learning. arXiv preprint arXiv: 000..7703..666 2017.
454
455
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensiona continuous control using generalized advantage estimation. arXiv preprint 15t.ii::100..244: 2015.
456
457
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization
458
algorithms. arXiv preprint urXiv:1707. 06000, 2017.
Shi, I.,, Li, S., Zheng, Q., Yao, M., and Pan, G. Efficient novelty search through deep reinforcement learning. IEEE Access, 8: 1818099-12888 2020.
460
461
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy
gradient algorithms. In Proceedings of the 30th International Conference in Machine Learning,
463
2014.
464
Stanton, C. and Clune, J. Curiosity search: producing generalists by encouraging individuals to
continually explore and acquire skills throughout their lifetime. PloS one, 110)):00122235, 2016.
Sutton, R. S., McAllester, D. A., Singh. S. P., Mansour, Y., et al. Policy gradient methods for reinforcement learning with function approximation. In NIPs, volume 99, pp. 1057-1063. Citeseer, 1999.
467
Vemula, A., Sun, W., and Bagnell, J. Contrasting exploration in parameter and action space: A zeroth- order optimization perspective. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2926-2935 PMLR, 2019.
470

---

Checklist
472
473
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
475
contributions and scope? [Yes]
476
(b) Did you descrihe the limitations of your work? [Yes]
477
(c) Did you discuss any potential negative societal impacts of your work? [No] We believe
478
that this work, in itself, is not prone to have any negative societal impact.
479
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
482
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
483
(b) Did you include complete proofs of all theoretical results? [N/A]
494
3. If you ran experiments..
485
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] The code and instructions to run it are available in the supplementary materials.
487
488
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Implementation details, hardware details and hyperparameters are presented in Appendix C.
489
490
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Yes, we report mean and variance for all experiments, both graphically and in result tables.
432
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] Computational details are provided in Appedix C.
495
496
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
498
(a) If your work uses existing assets, did you cite the creators? sourced RL environments.
[Yes] We use open
499
500
(b) Did you mention the license of the assets? [Yes] We cite the Mujoco physics engine, for which we have licenses.
502
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] We provide appendices, source code and a demonstration website.
503
504
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
505
506
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
509
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We did not used crowdsourcing or conducted research with human subjects.
510
511
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] This work did not involve research with human subjects
513
515
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] This work did not involve research with human subjects
516
517
12


Reviews:
[{"review_id": "bn69bOrtZR", "reviewer": "Reviewer_TBUP", "summary": "This paper proposes a Quality-Diversity and Policy Gradient (QD-PG) algorithm that deals with exploration-exploitation trade-off in the optimization problem. The authors validate this algorithm through several benchmark tasks in RL.  The results demonstrate that QD-PG produces competitive solutions especially in the deceptive tasks. The motivation of this paper is clear and the comparison experiments are conducted extensively. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Although the idea of combining diversity and quality is not novel, the way of combining QD and PG in this paper is interesting. The cost is the extra evaluation plus the definition of a good metric for measuring the diversity. Some comments need to be clarified:\n1. How are the two critics (diversity and quality) updated? In the paper, it is not clearly stated. I believe that the update of the ciritic plays an important role for the performance of this algorithm.\n2. Is QD-PG more sample efficient than its QD competitors? For the explanation of this research question, can the authors give some quantative results on the TD3 as well?\n3. How many replicates of experiments in Table 1?\n4. Can you explain why QD-PG SUM fails to learn the ANT-TRAP?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 "}, "extra_text": {}}, {"review_id": "N6f8uUlM7qQ", "reviewer": "Reviewer_76VX", "summary": "This paper introduces Diversity Policy Gradient (DPG), an algorithm designed to discover diverse neural policies to solve RL problems in a sample efficient manner. The authors adopt a MAP-Elites scheme to select and evolve neural policies however with the additional twist of updating the policies not just by their quality but also according to their novelty from other known solutions. They show their results on several RL challenges where their algorithm successfully avoids local minima and outperforms other known algorithms such as TD3. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I'm somewhat familiar with QD algorithms, I'm not a domain expert, therefore I'm unable to judge the significance/originality of this paper. I found the idea of using QD to evolve neural policies interesting and to me it was novel. \n\nI found the paper was generally clearly written, and in fact I appreciated the latter parts of the paper where the authors guided the reader to some relevant questions that the paper aims to address. Overall, I found the paper methodologically sound (e.g. ablation studies, benchmarks, and tasks all seemed reasonable and well-motivated), and the communication was quite clear. \n\nFrom my perspective this is a well-executed paper, of interest to NeurIPS community.\n\nMinor comment:\n- There seems to be a leftover comment in the text in line 170 (\"Not clear in it's current form...\")\n\n----\nQuestions:\n\nIt was not clear to me how the behavior descriptor and state descriptor extraction functions behave and if the choice of these functions were impactful in whether the algorithm works properly. I would appreciate further clarification in the paper. Are these affine or euclidean maps? Does it matter?\n\nI'm also wondering if these mappings preserve some notion of proximity of solutions, and in some sense allow for \"transfer\" of good trajectory level ideas between policies implicitly through the grid-level mutation-selection balance? If not, do the authors have ideas about whether this information can be exploited?\n\nI wonder if there is a better update schedule than random diversity and quality updates. The authors mention joint updates performed worse. It's not necessary, but I suggest I authors experiment with annealing/oscillating approaches to DPG/QPG ratios for the population, with a larger period than a single update. My main concern is that perhaps QPG ascent benefits from consecutive updates to find better solutions faster, and the current approach \"destroys\" the gains from the gradient, leading to sample inefficiency. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Societal impact is limited in it's current state, though I find that the approach *could* have useful implications for enabling different policies that while equally performant, may have beneficial secondary aspects like ethical superiority. \n\nThe main limitation of the study in my view is that doesn't help prescribe how to pick among multiple neural policies, and does not investigate a comparison between found solutions, i.e. in what sense is \"diversity\" in solutions more useful than just increasing the chance of finding very good ones. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "K0Z-jswqwC", "reviewer": "Reviewer_QfRA", "summary": "The paper proposes a Diversity Policy Gradient for Quality-Diversity optimization method. The approach solves a problem of finding a collection of well performing policies in continuous control environments. Rather than searching for a single optimal solution, the method aims to find a diverse set of high-performing solutions. As a result, the paper solves exploration and control problems with improved sample-efficiency over its evolutionary competitors.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written, clear, and easy to follow. It solves an interesting problem with improved sample-efficiency over other state-of-the-art algorithms. The authors considered a good set of experiments. The results and limitations are nicely discussed.\n\nMinor comment: you may want to rewrite lines 170 and 171 in the Related Work section.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are very well discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel method to learn a set of diverse and high-quality policies. Ablation studies are sufficient to verify that the results are the consequence of the algorithm components. All reviewers gave positive scores. 

However, this is not the first work on learning diverse policies. For examples, learning diverse policies were commonly demanded in game AI [Generating Behavior-Diverse Game AIs with Evolutionary Multi-Objective Deep Reinforcement Learning. IJCAI 2020], and skill discovery also requires diverse sub-policies [Learning Diversity is All You Need: Learning Skills without a Reward Function. ICLR 2019]. The major weakness is that the previous studies on learning diverse policies/skill are all ignored. 

Another weakness is that there is no theoretical guarantee on the convergence of the algorithm and on the level of diversity we can expect. This does not mean that a theoretical guarantee is a must. However, the algorithm design is heuristic. The outcome of the algorithm might be undesired. A discussion on the rationality of the algorithm on how much the diversity can be is expected.

Overall I think this paper contains novelty and contributions, but it should have done a complete survey of related work, contacted experiments comparing other diversity encourage methods, and discussed in-depth on the convergence property.
Decision: Reject