Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization

Abstract:
A fascinating aspect of nature lies in its ability to produce a large and diverse 1 collection of organisms that are all high-performing in their niche. By contrast, 2 most AI algorithms focus on finding a single efficient solution to a given problem. 3 Aiming for diversity in addition to performance is a convenient way to deal with the 4 exploration-exploitation trade-off that plays a central role in learning. It also allows 5 for increased robustness when the returned collection contains several working 6 solutions to the considered problem, making it well-suited for real applications such 7 as robotics. Quality-Diversity (QD) methods are evolutionary algorithms designed 8 for this purpose. This paper proposes a novel algorithm, QD-PG, which combines 9 the strength of Policy Gradient algorithms and Quality Diversity approaches to 10 produce a collection of diverse and high-performing neural policies in continuous 11 control environments. The main contribution of this work is the introduction of a 12 Diversity Policy Gradient (DPG) that exploits information at the time-step level to 13 thrive policies towards more diversity in a sample-efficient manner. Specifically, 14 QD-PG selects neural controllers from a MAP-Elites grid and uses two gradient15 based mutation operators to improve both quality and diversity, resulting in stable 16 population updates. Our results demonstrate that QD-PG generates collections of di17 verse solutions that solve challenging exploration and control problems while being 18 two orders of magnitude more sample-efficient than its evolutionary competitors. 19

A fascinating aspect of nature lies in its ability to produce a large and diverse1 collection of organisms that are all high-performing in their niche. By contrast,2 most AI algorithms focus on finding a single efficient solution to a given problem.3 Aiming for diversity in addition to performance is a convenient way to deal with the4 exploration-exploitation trade-off that plays a central role in learning. It also allows5 for increased robustness when the returned collection contains several working6 solutions to the considered problem, making it well-suited for real applications such7 as robotics. Quality-Diversity (QD) methods are evolutionary algorithms designed8 for this purpose. This paper proposes a novel algorithm, QD-PG, which combines9 the strength of Policy Gradient algorithms and Quality Diversity approaches to10 produce a collection of diverse and high-performing neural policies in continuous11 control environments. The main contribution of this work is the introduction of a12 Diversity Policy Gradient (DPG) that exploits information at the time-step level to13 thrive policies towards more diversity in a sample-efficient manner. Specifically,14 QD-PG selects neural controllers from a MAP-Elites grid and uses two gradient-15 based mutation operators to improve both quality and diversity, resulting in stable16 population updates. Our results demonstrate that QD-PG generates collections of di-17 verse solutions that solve challenging exploration and control problems while being18 two orders of magnitude more sample-efficient than its evolutionary competitors.19
1 Introduction20
Natural evolution has the fascinating ability to produce diverse organisms that are all well adapted to21 their respective niche. Inspired by this ability to produce a tremendous diversity of living systems,22 Quality-Diversity (QD) is a new family of optimization algorithms that aims at searching for a23 collection of both diverse and high-performing solutions (Pugh et al., 2016; Cully & Demiris, 2017).24 While classic optimization methods focus on finding a single efficient solution, QD optimization aims25 to cover the range of possible solution types and to return the best solution for each type. This process26 is sometimes referred to as “illumination" in opposition to optimization, as it reveals (or illuminates)27 a search space of interest often called the behavior descriptor space (Mouret & Clune, 2015).28
The principal advantage of QD approaches resides in their intrinsic capacity to deliver a large and29 diverse set of working alternatives when a single solution fails (Cully et al., 2015). By producing a30 collection of solutions instead of a unique one, QD algorithms allow to obtain different ways to solve31 a single problem, leading to greater robustness, which can help to reduce the reality gap when applied32 to robotics (Koos et al., 2012). Diversity seeking is the core component that allows QD algorithms to33 generate large collections of diverse solutions. By encouraging the emergence of novel behaviors in34 the population without focusing on performance alone, diversity seeking algorithms explore regions35 of the behavior descriptor space that are unreachable for conventional algorithms (Doncieux et al.,36
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
2019). Another benefit of QD is its ability to solve hard exploration problems where the reward signal37 is sparse or deceptive, and on which standard optimization techniques are ineffective (Colas et al.,38 2020). This ability can be interpreted as a direct consequence of the structured search for diversity in39 the behavior descriptor space.40
Quality-Diversity algorithms build on black-box optimization methods such as evolutionary algo-41 rithms to evolve a population of solutions (Cully & Demiris, 2017). Historically, they rely on random42 mutations to explore small search spaces but struggle when facing higher-dimensional problems. As43 a result, they often scale poorly to problems where neural networks with many parameters provide44 state-of-the-art results (Colas et al., 2020).45
Building large and efficient controllers that work with continuous actions has been a long-standing46 goal in Artificial Intelligence and in particular in robotics. Deep reinforcement learning (RL), and47 especially Policy Gradient (PG) methods have proven efficient at training such large controllers48 (Schulman et al., 2017; Lillicrap et al., 2015; Fujimoto et al., 2018; Haarnoja et al., 2018). One of the49 keys to this success lies in the fact that PG methods exploit the structure of the objective function50 when the problem can be formalized as a Markov Decision Process (MDP), leading to substantial51 gains in sample efficiency. Moreover, they also exploit the analytical structure of the controller when52 known, which allows the sample complexity of these methods to be independent of parameter space53 dimensionality (Vemula et al., 2019). In real-world applications, these gains turn out to be critical54 when interacting with the environment is expensive. PG methods usually rely on simple exploration55 mechanisms, like adding Gaussian noise (Fujimoto et al., 2018) or maximizing entropy (Haarnoja56 et al., 2018) to explore the action space, which happens to be insufficient in hard exploration tasks57 where the reward signal is sparse or deceptive (Colas et al., 2018; Nasiriany et al., 2019).58
Successful attempts have been made to combine evolutionary methods and reinforcement learning59 (Khadka et al., 2019; Khadka & Tumer, 2018; Pourchot & Sigaud, 2018; Shi et al., 2020). However,60 all these techniques only focus on building high-performing solutions and do not explicitly encourage61 diversity within the population. In this regard, they fail when confronted with hard exploration62 problems. To address these problems, one needs to seek both high-performing solutions and diversity63 within them.64
Contributions65
In this work, we introduce the idea of a diversity policy gradient (DPG) that thrives solutions towards66 more diversity. We show that the DPG can be used in combination with the standard policy gradient,67 dubbed quality policy gradient (QPG), to produce high-performing and diverse solutions. Our68 algorithm, called QD-PG, builds on MAP-Elites (Mouret & Clune, 2015), demonstrates remarkable69 sample efficiency brought by off-policy PG methods, and produces collections of good solutions70 in a single run (see Figure 1). We compare QD-PG to state-of-the-art RL algorithms and to several71 evolutionary methods known as Evolution Strategies (ESs) augmented with a diversity objective,72
namely the NS-ES family (Conti et al., 2018) and the ME-ES algorithm (Colas et al., 2020). We73 show that QD-PG generates collections of robust solutions in hard exploration problems while RL74 algorithms struggle to produce a single one, and that QD-PG is two orders of magnitude more sample75 efficient than the best of its evolutionary competitors.76
2 Background77
Problem statement78
We consider an MDP (S,A,R, T , γ) where S is the state space,A the action space,R : S ×A → R79 the reward function, T : S × A → S the dynamics transition function and γ a discount factor.80 We assume that both S and A are continuous and consider a controller, or policy, πθ : S → A81 parameterized by θ ∈ Θ, which is called a solution to the problem. We say that a solution θ is82 highy-performing if the expectation over the sum of rewards is high when using πθ. The fitness of a83 solution measures its performance F : Θ→ R where F (θ) = Eπθ ∑ t γtrt.84
To characterize the novelty of a solution w.r.t. J other solutions, as in QD methods, we introduce a85 behavior descriptor (BD) space B, a behavior descriptor extraction function ξ : Θ→ B, and define a86 distance metric ||.||B over B. The novelty n : Θ×ΘJ → R+ of a solution θ w.r.t. a list of solutions87 (θj)j=1,...,J is defined as n ( θ, (θj)j=1,...,J ) = ∑ j ||ξ(θ), ξ(θj)||B. In other words, we quantify the88 novelty of a solution w.r.t. a list of J solutions as the sum of distances between its behavior descriptor89 and the behavior descriptors of all solutions of the list. We also use the distance ||.||B to characterize90 the diversity of a set of K solutions {θk}k=1,...,K . We formally define diversity d : ΘK → R+ as91
d ({θk}k=1,...,K) = K∑ i=1 min k 6=i ||ξ(θi), ξ(θk)||B, (1)
meaning that a set of solutions is diverse if the solutions are distant with respect to each other in the92 sense of ||.||B.93
The MAP-Elites algorithm94
MAP-Elites (Mouret & Clune, 2015) is a simple yet state-of-the-art QD algorithm that has been95 successfully applied to a wide range of challenging problems such as robot damage recovery (Cully96 et al., 2015), molecular robotic control (Cazenille et al., 2019) and game design (Alvarez et al., 2019).97 In MAP-Elites, the behavior descriptor space B is discretized into a grid of cells, also called niches,98 with the aim of filling each cell with a high-performing solution. The algorithm starts with an empty99 grid and an initial random set of K solutions that are evaluated and added to the grid by following100 simple insertion rules. If the cell corresponding to the behavior descriptors of a solution is empty, then101 the solution is added to this cell. If there is already a solution in the cell, the new solution replaces it102 only if it has greater fitness. At each iteration, P existing solutions are sampled uniformly from the103 grid and randomly mutated to create P new solutions. These new solutions are then evaluated and104 added to the grid following the same insertion rules. This cycle is repeated until convergence or for a105 given budget of iterations.106
Though MAP-Elites is a compelling and efficient method, it suffers from a low sample efficiency107 as it relies on random mutations. Recently, Colas et al. (2020) tackled this problem by updating108 the solutions through an Evolution Strategy known as the Cross-Entropy method. Notably, they109 showed that MAP-Elites could be scaled with their method to address complex MUJOCO control110 environments at the cost of very large computational resources. In this study, we propose to harness111 policy gradients (QPG and DPG) to build a more sample-efficient MAP-Elites approach.112
3 Key Principle: Diversity Policy Gradient113
Let us assume that we have a MAP-Elites grid containing K solutions (θ1, . . . , θK). To increase114 diversity in the grid using the DPG, we need to update one sampled solution θ from the grid using115 gradient ascent. To do so, we aim to compute the gradient of the population diversity w.r.t. θ, where116 diversity is defined in Equation (1). As the K solutions are independent, order does not matter and117
we can consider optimizing arbitrarily θ = θ1. To compute the gradient of d w.r.t. θ1, we need118 to separate the terms that depend on θ1 from the others. The terms that depend on θ1 correspond119 to the distance of θ1 to its nearest neighbor, which we define as θ2, and to the distances of θ1 to120 the θs for which θ1 is the nearest neighbor. We can arbitrarily index them from 3 to J1, thus:121 d({θk}k=1,...,K) = ∑J j=2 ||ξ(θ1), ξ(θj)||B + M , where M = ∑ i 6∈{1,...,J}min
k 6=i ||ξ(θi), ξ(θk)||B.122 Only the first term of the sum depends on θ = θ1. Furthermore, we observe that this term equals the123 novelty of solution θ1 w.r.t. the list (θj)2≤j≤J . Therefore, the gradient of diversity w.r.t. θ1 is124
∇θ1d({θk}k=1,...,K) = ∇θ1n(θ1, (θj)2≤j≤J). That is, we can increase the diversity of the population125 by increasing the novelty of θ1 w.r.t. the list (θj)2≤j≤J . In practice, we replace this list by a list of126 nearest neighbors of θ1, as this is easier to compute and the elements of (θj)2≤j≤J tend to be among127 the nearest neighbors of θ1.128
Under this form, the diversity gradient cannot benefit from the variance reduction methods in the RL129 literature to efficiently compute policy gradients Sutton et al. (1999). To this end, we need to express130 it as a gradient over the expectation of a sum of scalar quantities obtained by policy πθ1 at each step131 when interacting with the environment. Therefore, to build a DPG, we need information about the132 novelty of a solution at the time step level. To do so, we introduce a novel space D, dubbed state133 descriptor space and a state descriptor extraction function ψ : S → D. We assume D and B have the134 same dimension. Similarly to the novelty of a solution, we now define the novelty of a state s w.r.t. J135 other states (sj)j=1,...,J as n : S × S J → R such that n(s, (sj)j=1,...,J) = ∑J j=1 ||ψ(s), ψ(sj)||D,136 where ||.||D is a distance metric over D.137 Now, we need to link novelty defined at the time step level to novelty defined at the solution level. We138 define the novelty of a state w.r.t. a set of solutions. We say that a state is novel w.r.t. some solutions139 if the state is novel w.r.t. to the states visited by these solutions. More formally:140
n(s, (θj)j=1,...,J) = J∑ j=1 Eπθj ∑ t ||ψ(s), ψ(st)||D. (2)
While we adopt this definition in this paper, one might as well consider other definitions where, for141 instance, a state is compared to states that have been visited at the same time step during another142 episode. In this context, if the following relation is satisfied:143
Eπθ1 ∑ t n(st, (θj)2≤j≤J) = n(θ1, (θj)2≤j≤J), (3)
then we can compute the DPG of d w.r.t. θ1 as144
∇DPGθ1 = ∇θ1Eπθ1 ∑ t n(st, (θj)2≤j≤J). (4)
This expression corresponds to the classical policy gradient setting where γ = 1 and where the145 corresponding reward signal, here dubbed diversity reward, is computed as rDt = n(st, (θj) 2≤j≤J).146 Therefore, this gradient can be computed using any PG estimation technique replacing the environ-147 ment reward by the diversity reward rDt .148
Equation (3) enforces a relation between B and D and between extraction functions ψ and ξ. In149 practice, it may be hard to define the behavior descriptor and state descriptor of a solution that satisfy150 this relation while being meaningful to the problem at hand and tractable. But a strict equality is not151 necessary. It suffices that an increase on the left-hand side implies an increase on the right-hand side152 so that we can still update θ1 using (4). Furthermore, when this is not the case, the diversity gradient153 update might not result in an increase of diversity in the behavior descriptor space, but in that case the154 MAP-Elites insertion rule will remove the corresponding solution. We show in Section 6 that we can155 define descriptors that do not satisfy the above relation all the time, but still give satisfactory results.156
1Remark: θ2 can appear twice in the list (θj)2≤j≤J
4 Related Work157
A distinguishing feature of our approach is that we combine diversity seeking at the level of trajectories158 using behavior descriptors and diversity seeking in the state space using state descriptors. The former159 is used by MAP-Elites to select solutions from the grid and contributes structural bias towards diversity,160 whereas the latter is used during policy gradient steps in the RL part, see Figure 2b. We organize the161 literature review below according to this split between two types of diversity seeking mechanisms.162
QD search in the solution space163
Simultaneously maximizing diversity and performance is the central goal of QD methods (Pugh164 et al., 2016; Cully & Demiris, 2017). Among the various possible combinations offered by the165 QD framework, Novelty Search with Local Competition (NSLC) (Lehman & Stanley, 2011b) and166 MAP-Elites (Mouret & Clune, 2015) are the two most popular algorithms. NSLC builds on the Novelty167 Search (NS) algorithm (Lehman & Stanley, 2011a) and maintains an unstructured archive of solutions168 selected for their local performance while MAP-Elites uniformly samples individuals from a structured169 grid that discretizes the BD space. Not clear in its current form. I suggest: "QD-PG uses the standard170 grid of MAP-Elites. However, we also show in Appendix F that QD-PG can be used with alternative171 archive structures.172
With the objective of improving their data-efficiency, QD-ES algorithms that combine QD and ESs,173 such as NSR-ES and NSRA-ES, have been applied to challenging continuous control environments in174 Conti et al. (2018). But, as outlined in Colas et al. (2020), they suffer from poor sample efficiency175 and the diversity and environment reward functions could be mixed in a more efficient way. In that176 respect, the most closely related work w.r.t. ours is ME-ES (Colas et al., 2020). The ME-ES algorithm177 also optimizes quality and diversity using MAP-Elites and two ES populations. Using these methods178 was shown to be critically more efficient than population-based GA algorithms (Salimans et al., 2017),179 but our results show that they are still less sample efficient than off-policy deep RL methods, as they180 do not leverage the analytical computation of the policy gradient at the time step level. To the best181 of our knowledge, no QD or ES algorithm use an explicit critic for both performance and diversity,182 resulting in even higher data-efficiency.183
QD search in the state or action spaces184
Seeking for diversity in the space of states or actions is generally framed into the RL framework. This185 is the case of algorithms maintaining a population of RL agents for exploration without an explicit186 diversity criterion (Jaderberg et al., 2017) or algorithms explicitly looking for diversity but in the187 action space rather than in the state space like ARAC (Doan et al., 2019), P3S-TD3 (Jung et al., 2020)188 and DvD (Parker-Holder et al., 2020).189
An exception is Stanton & Clune (2016) who define a notion of intra-life novelty that is similar to190 our state novelty defined in Section 3. However, their novelty relies on skills rather than states. Our191 work is also related to algorithms using RL mechanisms to search for diversity only (Eysenbach et al.,192 2018; Pong et al., 2019; Lee et al., 2019; Islam et al., 2019). These methods have proven useful in193 sparse reward situations, but they are inherently limited when the reward signal can orient exploration,194 as they ignore it. Other works sequentially combine diversity seeking and RL. The GEP-PG algorithm195 Colas et al. (2018) combines a diversity seeking component, namely Goal Exploration Processes196 (Forestier et al., 2017) and the DDPG deep RL algorithm (Lillicrap et al., 2015). This sequential197 combination of exploration-then-exploitation is also present in GO-EXPLORE (Ecoffet et al., 2019).198 Again, this approach is limited when the reward signal can help driving the exploration process to199 efficient solutions. These sequential approaches first look for diversity in the behavior descriptor200 space, then optimize performance in the state action space, whereas we do so simultaneously in the201 behavior descriptor space and in the state space.202
To the best of our knowledge, QD-PG is the first algorithm optimizing both diversity and performance203 in the solution and in the state space, using a sample-efficient policy gradient computation method204 for the latter.205
5 Methods206
Our full algorithm is called QD-PG, its pseudo code is given in Appendix A and its architecture is207 depicted in Figure 2. QD-PG is an iterative algorithm based on MAP-Elites that replaces random208
mutations with policy gradient updates. As we consider a continuous action space and want to209 improve sample efficiency by using an off-policy policy gradient method, we rely on the Twin210 Delayed Deterministic Policy Gradient (TD3) algorithm (Fujimoto et al., 2018). See Appendix B for211 a detailed description of TD3.212
QD-PG maintains three permanent structures. In the QD part, a MAP-Elites grid stores the most novel213 and performing solutions. In the RL part, a replay buffer contains all transitions collected when214 evaluating solutions and an archive A stores all state descriptors obtained so far. QD-PG starts with an215 initial population of random solutions, evaluates them and inserts them into the MAP-Elites grid. At216 each iteration, solutions are sampled from the grid, copied, and updated. The updated solutions are217 then evaluated through one rollout in the environment and inserted into the grid according to insertion218 rules. Transitions collected during evaluation are stored in the replay buffer, and state descriptors219 are stored in the archive A. Note that these state descriptors are first filtered to avoid insertion in the220 archive of multiple state descriptors that are too close to each other.221
During the update step, half the population is updated with QPG ascent and the other half with DPG222 ascent. The choice of whether an agent is updated for quality or diversity is random, meaning that it223 can be updated for quality and later for diversity if selected again. To justify this design, we show in224 Section 6 that updating consecutively for quality and diversity outperforms updating based on joint225 criteria. Both gradients are computed from batches of transitions sampled from the replay buffer. The226 QPG is computed as usual from rewards whereas for DPG, we get fresh novelty rewards as227
rDt = J∑ j=1 ||ψ(st), ψ(sj)||D, (5)
where (sj)j=1,...,J are the J nearest neighbors of state st in the archive A. Diversity rewards228 must be recomputed at each update because A changes during training. Following Equation (2),229 diversity rewards should be computed as the sum of the distances between the descriptor of st and230 the descriptors of all the states visited by a list of J solutions. In practice, we consider the J nearest231 neighbors of st. This choice simplifies the algorithm and is faster and works well in practice.232
TD3 relies on a parameterized critic to reduce the variance of its policy gradient estimate. In QD-PG,233 we maintain two parameterized critics QDw and Q Q v , respectively dubbed diversity and quality critics,234 every time a policy gradient is computed, QD-PG also updates the corresponding critic. In fact, as235 in TD3, we use pairs of critics and target critics to fight the overestimation bias. We share the critic236 parameters among the population as in Pourchot & Sigaud (2018). Reasons for doing so come from237
the fact that diversity is not stationary, as it depends on the current population. If each agent had238 its own diversity critic, since an agent may not be selected for a large number of generations before239 being selected again, its critic would convey an outdated picture of the evolving diversity. We tried240 this solution, and it failed. A side benefit of critic sharing is that both critics become accurate faster as241 they combine experience from all agents. Additional details on QD-PG implementation are available242 in Appendix C.243
6 Experiments244
In this section, we intend to answer the following matters: 1. Can QD-PG produce collections of245 diverse and high-performing neural policies and what are the advantages to do so? 2. Is QD-PG more246 sample efficient than its QD competitors? 3. To what extent are the considered benchmarks difficult247 for classical policy gradients methods? 4. What is the usefulness of the different components of248 QD-PG?249
Environments250
We asses QD-PG capabilities in continuous control environments that exhibit high dimensional251 observation and action spaces as well as strong exploration difficulties. Two types of reward signals,252 dubbed sparse and deceptive, are known to be particularly difficult for classical RL methods. These253 rewards appear in many applications such as robotics or combinatorial optimization. Sparse rewards254 are obtained if a given condition is specified, leading to a majority of null rewards and to credit255 assignment difficulties. Deceptive rewards are dense signals, i.e., they are non-zero at each time step256 but can mislead the search process to some local optimum. In such problems, a good approach to the257 exploration-exploitation trade-off is essential. The agent should learn when to ignore the reward signal258 and explore to avoid local minima and when to follow it to increase its return. Deceptive environments259 constitute a natural choice to highlight QD efficiency to balance exploration and exploitation. In this260 study, we consider three OpenAI Gym environments based on the MUJOCO physics engine that all261 exhibit strong deceptive rewards (illustrated in Appendix 5). Such environments have been widely262 used in previous works (Parker-Holder et al., 2020; Colas et al., 2020; Frans et al., 2018; Shi et al.,263 2020) for their deceptive nature, a characteristic that is absent of more widespread continuous control264 environments like HALFCHEETAH-V2, HOPPER-V2 or still ANT-V2.265
In the POINT-MAZE environment, an agent represented as a green sphere must find the exit of the266 maze depicted in Figure 4a, represented as a red sphere. An observation contains the agent position267 at time t, and an action corresponds to position increments along the x and y axes. The reward is268 expressed as the negative Euclidean distance between the center of gravity of the agent and the exit269 center. The trajectory length cannot exceed 200 steps.270
The ANT-MAZE environment is modified from OpenAI Gym ANT-V2 (Brockman et al., 2016) and271 also used in (Colas et al., 2020; Frans et al., 2018). In ANT-MAZE, a four-legged ant has to reach272 a goal zone located in the lower right part of the maze (colored in green in Figure 4b). Its initial273 position is sampled in a small circle located in the maze’s extreme bottom left. As in POINT-MAZE,274 the reward is expressed as the negative Euclidean distance between the ant and the center of the goal275 zone. Maze walls are organized so that following the gradient of the reward function drives the ant276 into a dead-end. In ANT-MAZE, the final performance is defined as the maximum reward received277 during an episode. The environment is considered solved when an agent obtains a score superior to278 −10, corresponding to reaching the goal zone. An episode consists of 3000 time steps, this horizon279 is three times larger than in usual MUJOCO environments, making this environment particularly280 challenging for RL based methods (Vemula et al., 2019).281
Finally, the ANT-TRAP environment also derives from ANT-V2 and is inspired from (Colas et al.,282 2020; Parker-Holder et al., 2020). In ANT-TRAP, the four-legged ant initially appears in front of a trap283 and must bypass it to run as fast as possible in the forward direction (see Figure 4c), as in ANT-V2,284 the reward is computed as the ant velocity on the x-axis. The trap consists of three walls forming a285 dead-end directly in front of the ant, leading to a strong deceptive reward. In this environment, the286 trajectory length cannot exceed 1000 steps. As opposed to POINT-MAZE and ANT-MAZE, where the287 objective is to reach the exit area, there is no unique way to solve ANT-TRAP and we expect a QD288 algorithm to generate various effective solutions as depicted in Figure 1.289
Baselines and Ablations290
QD-PG is compared to three types of methods. First, to answer question 2, we compare QD-PG to a291 family of QD baselines, namely ME-ES, NSR-ES, and NSRA-ES (Colas et al., 2020). Appendix E.1292 recaps the properties of all these methods. Second, to answer question 3, we compare QD-PG to a293 family of policy gradient baselines. Soft Actor Critic (SAC) (Haarnoja et al., 2018) and the Twin294 Delayed Deep Deterministic policy gradient (TD3) (Fujimoto et al., 2018) are continuous control295 algorithms achieving state-of-the-art results on MUJOCO benchmarks. Random Network Distillation296 (RND) (Burda et al., 2018) is a curiosity-driven RL agent (Schulman et al., 2017) which was shown to297 perform well in hard exploration settings. CEM-RL (Pourchot & Sigaud, 2018) mixes Cross-Entropy298 Methods (CEM) and RL to evolve a population of agents to maximize quality and obtains state-299 of-the-art results MUJOCO benchmarks. Finally, to answer question 4, we propose to investigate300 the following matters: Can we replace alternating quality and diversity updates by a single update301 that optimizes for the sum of both criteria? Are quality gradients updates alone enough to fill the302 MAP-Elites grid? Are diversity gradients updates alone enough to do so? Consequently, we consider303 the following ablations of QD-PG: QD-PG SUM computes a gradient to optimize the sum of the quality304 and diversity rewards, D-PG applies only diversity gradients to the solutions, and Q-PG applies only305 quality gradients, but both D-PG and Q-PG still use QD selection (see Appendix E.1).306
We compare QD-PG to its ablations and RL competitors in all environments and show results in307 Table 1a. Detailed results including graphic charts and coverage maps are given in Appendix E and308 more details about the evaluation procedure are given in Appendix E.1.309
7 Results310
1. Can QD-PG produce collections of neural policies and what are the advantages to do so?311
Table 1a presents QD-PG performances. In all environments, our algorithm manages to find working312 solutions that avoid local minima and reach the overall objective. In addition to its exploration313 capabilities, QD-PG generates collections of high performing solutions in a single run. During the314 ANT-TRAP experiment, the final collection of solutions returned by QD-PG contained, among others,315 5 solutions that were within a 10% performance margin from the best one. As illustrated in Figure 1,316 these agents typically differ in their gaits and preferred trajectories to circumvent the trap.317
Generating a collection of diverse solutions comes with318 the benefit of having a repertoire of diverse solutions that319 can be used as alternatives when the MDP changes (Cully320 et al., 2015). We show that QD-PG is more robust than321 conventional policy gradient methods by changing the re-322 ward signal of the ANT-MAZE environment. We replace323 the original goal in the bottom right part of the maze (see324 Figure 3) with a new randomly located goal in the maze.325 Instead of running QD-PG to optimize for this new objec-326 tive, we run a Bayesian optimization process to quickly327 find a good solution among the ones already stored in the328 grid. With a budget of only 20 solutions to be tested during329 the Bayesian optimization process, we are able to quickly330 recover a good solution for the new objective. We repeat331 this experiment 100 times, each time with a different ran-332 dom goal, and obtain an average performance of −10 with333
a standard deviation of 9. In other words, 20 interaction episodes (corresponding to 60.000 time334 steps) suffice for the adaptation process to find a solution that performs well for the new objective335 without the need to re-train agents. More detailed results can be found in Appendix E.3. 2336
2. Is it more sample efficient than its QD competitors?337
Table 1b compares QD-PG to Deep Neuroevolution algorithms with a diversity seeking component in338 terms of sample efficiency. QD-PG runs on 10 CPU cores for 2 days while its competitors used 1000339 CPU cores for the same duration. Nonetheless, QD-PG matches the asymptotic performance of ME-ES340 using two orders of magnitude fewer samples, explaining the lower resource requirements.341
2Videos of QD-PG agents are available at: https://sites.google.com/view/qd-pg
We see three reasons for the improved sample efficiency of QD-PG: 1) QD-PG leverages a replay342 buffer and can re-use each sample several times. 2) QD-PG leverages novelty at the state level and343 can exploit all collected transitions to maximize quality and diversity. For instance, in ANT-MAZE,344 a trajectory brings 3000 samples to QD-PG while standard QD methods would consider it a unique345 sample. 3) PG exploits the analytical gradient between the neural network weights and the resulting346 policy action distribution and estimates only the impact of the distribution on the return. By contrast,347 standard QD methods directly estimate the impact on the return of randomly modifying the weights.348
3. To what extent the considered benchmarks are difficult for policy gradients methods?349
Table 1a compares QD-PG to state-of-the-art policy gradient algorithms and validates that classical350 policy gradient methods fail to find optimal solutions in deceptive environments. TD3 quickly351 converges to local minima of performance resulting from being attracted in dead-ends by the deceptive352 gradients. While we may expect SAC to better explore due to entropy regularization, it also converges353 to that same local minima in ANT-TRAP and POINT-MAZE. Besides, despite its exploration mechanism354 based on CEM, CEM-RL also quickly converges to local optima in all benchmarks, confirming the355 need for a dedicated diversity seeking component. RND, which adds an exploration bonus used as356 an intrinsic reward (see Appendix G for more details), also demonstrates performances inferior to357 QD-PG in all environments but manages to solve POINT-MAZE. In ANT-MAZE and ANT-TRAP, as358 shown in Appendix G.2, RND extensively explores the BD space but fails to obtain high returns.359
4. What is the usefulness of the different components of QD-PG ?360
The ablation study in Table 1a shows that when maximising quality only, Q-PG fails due to the361 deceptive nature of the reward and when maximizing diversity only, D-PG sufficiently explores to362 solve the problem in both POINT-MAZE and ANT-MAZE but requires more steps and finds lower-363 performing solutions. When optimizing simultaneously for quality and diversity, QD-PG SUM fails364 to learn in ANT-TRAP and manages to solve the task in ANT-MAZE but requires more samples than365 QD-PG. We hypothesize that quality and diversity rewards may give rise to conflicting gradients. For366 instance, at the beginning of training in ANT-TRAP, the quality reward drives the ant forward whereas367 the diversity reward drives it back to escape the trap and explore the environment. Therefore, both368 rewards cancel each other, preventing any learning. This study validates the usefulness of QD-PG369 components: 1) optimizing for diversity is required to overcome the deceptive nature of the reward;370 2) adding quality optimization provides better asymptotic performance; 3) it is better to disentangle371 quality and diversity updates.372
8 Conclusion373
This paper is the first to introduce a diversity gradient to explore diversity both at the state and374 skill levels. Based on this component we proposed a novel algorithm, QD-PG, inspired from the375 Quality-Diversity literature, that produces collections of diverse and high-performing neural policies376 in a sample-efficient manner. We showed experimentally that QD-PG generates several solutions377 that achieve high returns in challenging exploration problems. Finally, we demonstrated that in378 a few interactions with the environment, QD-PG finds alternative solutions that still obtain good379 performance when the MDP changes.380
References381 Alvarez, A., Dahlskog, S., Font, J., and Togelius, J. Empowering quality diversity in dungeon design382 with interactive constrained map-elites. In 2019 IEEE Conference on Games (CoG), pp. 1–8. IEEE,383 2019.384
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.385 Openai gym. arXiv preprint arXiv:1606.01540, 2016.386
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation.387 arXiv preprint arXiv:1810.12894, 2018.388
Cazenille, L., Bredeche, N., and Aubert-Kato, N. Exploring self-assembling behaviors in a swarm of389 bio-micro-robots using surrogate-assisted map-elites. arXiv preprint arXiv:1910.00230, 2019.390
Colas, C., Sigaud, O., and Oudeyer, P.-Y. GEP-PG: Decoupling exploration and exploitation in deep391 reinforcement learning algorithms. arXiv preprint arXiv:1802.05054, 2018.392
Colas, C., Madhavan, V., Huizinga, J., and Clune, J. Scaling map-elites to deep neuroevolution. In393 Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 67–75, 2020.394
Conti, E., Madhavan, V., Such, F. P., Lehman, J., Stanley, K., and Clune, J. Improving exploration in395 evolution strategies for deep reinforcement learning via a population of novelty-seeking agents. In396 Advances in neural information processing systems, pp. 5027–5038, 2018.397
Cully, A. and Demiris, Y. Quality and diversity optimization: A unifying modular framework. IEEE398 Transactions on Evolutionary Computation, 22(2):245–259, 2017.399
Cully, A., Clune, J., Tarapore, D., and Mouret, J.-B. Robots that can adapt like animals. Nature, 521400 (7553):503–507, 2015.401
Doan, T., Mazoure, B., Durand, A., Pineau, J., and Hjelm, R. D. Attraction-repulsion actor-critic for402 continuous control reinforcement learning. arXiv preprint arXiv:1909.07543, 2019.403
Doncieux, S., Laflaquière, A., and Coninx, A. Novelty search: a theoretical perspective. In404 Proceedings of the Genetic and Evolutionary Computation Conference, pp. 99–106, 2019.405
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, J. Go-explore: a new approach for406 hard-exploration problems. arXiv preprint arXiv:1901.10995, 2019.407
Eysenbach, B., Gupta, A., Ibarz, J., and Levine, S. Diversity is all you need: Learning skills without408 a reward function. arXiv preprint arXiv:1802.06070, 2018.409
Forestier, S., Mollard, Y., and Oudeyer, P.-Y. Intrinsically motivated goal exploration processes with410 automatic curriculum learning. arXiv preprint arXiv:1708.02190, 2017.411
Frans, K., Ho, J., Chen, X., Abbeel, P., and Schulman, J. Meta learning shared hierarchies. Proc. of412 ICLR, 2018.413
Fujimoto, S., Van Hoof, H., and Meger, D. Addressing function approximation error in actor-critic414 methods. arXiv preprint arXiv:1802.09477, 2018.415
Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A.,416 Abbeel, P., et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905,417 2018.418
Islam, R., Ahmed, Z., and Precup, D. Marginalized state distribution entropy regularization in policy419 optimization. arXiv preprint arXiv:1912.05128, 2019.420
Jaderberg, M., Dalibard, V., Osindero, S., Czarnecki, W. M., Donahue, J., Razavi, A., Vinyals, O.,421 Green, T., Dunning, I., Simonyan, K., et al. Population-based training of neural networks. arXiv422 preprint arXiv:1711.09846, 2017.423
Jung, W., Park, G., and Sung, Y. Population-guided parallel policy search for reinforcement learning.424 In International Conference on Learning Representations, 2020.425
Khadka, S. and Tumer, K. Evolution-guided policy gradient in reinforcement learning. In Neural426 Information Processing Systems, 2018.427
Khadka, S., Majumdar, S., Miret, S., Tumer, E., Nassar, T., Dwiel, Z., Liu, Y., and Tumer, K.428 Collaborative evolutionary reinforcement learning. arXiv preprint arXiv:1905.00976, 2019.429
Koos, S., Mouret, J.-B., and Doncieux, S. The transferability approach: Crossing the reality gap in430 evolutionary robotics. IEEE Transactions on Evolutionary Computation, 17(1):122–145, 2012.431
Lee, L., Eysenbach, B., Parisotto, E., Xing, E., Levine, S., and Salakhutdinov, R. Efficient exploration432 via state marginal matching. arXiv preprint arXiv:1906.05274, 2019.433
Lehman, J. and Stanley, K. O. Abandoning objectives: Evolution through the search for novelty434 alone. Evolutionary computation, 19(2):189–223, 2011a.435
Lehman, J. and Stanley, K. O. Evolving a diversity of virtual creatures through novelty search and436 local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary437 computation, pp. 211–218, 2011b.438
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D.439 Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015.440
Mouret, J.-B. and Clune, J. Illuminating search spaces by mapping elites. arXiv preprint441 arXiv:1504.04909, 2015.442
Nasiriany, S., Pong, V. H., Lin, S., and Levine, S. Planning with goal-conditioned policies. arXiv443 preprint arXiv:1911.08453, 2019.444
Parker-Holder, J., Pacchiano, A., Choromanski, K., and Roberts, S. Effective diversity in population-445 based reinforcement learning. In Neural Information Processing Systems, 2020.446
Pong, V. H., Dalal, M., Lin, S., Nair, A., Bahl, S., and Levine, S. Skew-fit: State-covering self-447 supervised reinforcement learning. arXiv preprint arXiv:1903.03698, 2019.448
Pourchot, A. and Sigaud, O. Cem-rl: Combining evolutionary and gradient-based methods for policy449 search. arXiv preprint arXiv:1810.01222, 2018.450
Pugh, J. K., Soros, L. B., and Stanley, K. O. Quality diversity: A new frontier for evolutionary451 computation. Frontiers in Robotics and AI, 3:40, 2016.452
Salimans, T., Ho, J., Chen, X., Sidor, S., and Sutskever, I. Evolution strategies as a scalable alternative453 to reinforcement learning. arXiv preprint arXiv:1703.03864, 2017.454
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control455 using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.456
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization457 algorithms. arXiv preprint arXiv:1707.06347, 2017.458
Shi, L., Li, S., Zheng, Q., Yao, M., and Pan, G. Efficient novelty search through deep reinforcement459 learning. IEEE Access, 8:128809–128818, 2020.460
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy461 gradient algorithms. In Proceedings of the 30th International Conference in Machine Learning,462 2014.463
Stanton, C. and Clune, J. Curiosity search: producing generalists by encouraging individuals to464 continually explore and acquire skills throughout their lifetime. PloS one, 11(9):e0162235, 2016.465
Sutton, R. S., McAllester, D. A., Singh, S. P., Mansour, Y., et al. Policy gradient methods for466 reinforcement learning with function approximation. In NIPs, volume 99, pp. 1057–1063. Citeseer,467 1999.468
Vemula, A., Sun, W., and Bagnell, J. Contrasting exploration in parameter and action space: A zeroth-469 order optimization perspective. In The 22nd International Conference on Artificial Intelligence470 and Statistics, pp. 2926–2935. PMLR, 2019.471
Checklist472
1. For all authors...473 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s474 contributions and scope? [Yes]475 (b) Did you describe the limitations of your work? [Yes]476 (c) Did you discuss any potential negative societal impacts of your work? [No] We believe477 that this work, in itself, is not prone to have any negative societal impact.478 (d) Have you read the ethics review guidelines and ensured that your paper conforms to479 them? [Yes]480 2. If you are including theoretical results...481
(a) Did you state the full set of assumptions of all theoretical results? [N/A]482 (b) Did you include complete proofs of all theoretical results? [N/A]483
3. If you ran experiments...484 (a) Did you include the code, data, and instructions needed to reproduce the main experi-485 mental results (either in the supplemental material or as a URL)? [Yes] The code and486 instructions to run it are available in the supplementary materials.487
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were488 chosen)? [Yes] Implementation details, hardware details and hyperparameters489 are presented in Appendix C.490
(c) Did you report error bars (e.g., with respect to the random seed after running experi-491 ments multiple times)? [Yes] Yes, we report mean and variance for all experiments,492 both graphically and in result tables.493
(d) Did you include the total amount of compute and the type of resources used (e.g.,494 type of GPUs, internal cluster, or cloud provider)? [Yes] Computational details are495 provided in Appedix C.496
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...497 (a) If your work uses existing assets, did you cite the creators? [Yes] We use open498 sourced RL environments.499 (b) Did you mention the license of the assets? [Yes] We cite the Mujoco physics engine,500 for which we have licenses.501 (c) Did you include any new assets either in the supplemental material or as a URL? [Yes]502 We provide appendices, source code and a demonstration website.503 (d) Did you discuss whether and how consent was obtained from people whose data you’re504 using/curating? [N/A]505 (e) Did you discuss whether the data you are using/curating contains personally identifiable506 information or offensive content? [N/A]507 5. If you used crowdsourcing or conducted research with human subjects...508
(a) Did you include the full text of instructions given to participants and screenshots, if509 applicable? [N/A] We did not used crowdsourcing or conducted research with510 human subjects.511
(b) Did you describe any potential participant risks, with links to Institutional Review512 Board (IRB) approvals, if applicable? [N/A] This work did not involve research513 with human subjects514
(c) Did you include the estimated hourly wage paid to participants and the total amount515 spent on participant compensation? [N/A] This work did not involve research with516 human subjects517

Review summaries:
The paper proposes a Diversity Policy Gradient for Quality-Diversity optimization method. The approach solves a problem of finding a collection of well performing policies in continuous control environments. Rather than searching for a single optimal solution, the method aims to find a diverse set of high-performing solutions. As a result, the paper solves exploration and control problems with improved sample-efficiency over its evolutionary competitors. 

This paper introduces Diversity Policy Gradient (DPG), an algorithm designed to discover diverse neural policies to solve RL problems in a sample efficient manner. The authors adopt a MAP-Elites scheme to select and evolve neural policies however with the additional twist of updating the policies not just by their quality but also according to their novelty from other known solutions. They show their results on several RL challenges where their algorithm successfully avoids local minima and outperforms other known algorithms such as TD3. 

This paper proposes a Quality-Diversity and Policy Gradient (QD-PG) algorithm that deals with exploration-exploitation trade-off in the optimization problem. The authors validate this algorithm through several benchmark tasks in RL.  The results demonstrate that QD-PG produces competitive solutions especially in the deceptive tasks. The motivation of this paper is clear and the comparison experiments are conducted extensively.