Sample efficient Quality Diversity for neural continuous control

1 INTRODUCTION . Natural evolution has the fascinating ability to produce organisms that are all high-performing in their respective niche . Inspired by this ability to produce a tremendous diversity of living systems within one run , Quality-Diversity ( QD ) is a new family of optimization algorithms that aim at searching for a collection of both diverse and high-performing solutions ( Pugh et al. , 2016 ) . While classic optimization methods focus on finding a single efficient solution , the role of QD optimization is to cover the range of possible solution types and to return the best solution for each type . This process is sometimes referred to as ” illumination ” in opposition to optimization , as the goal of these algorithms is to reveal ( or illuminate ) a search space of interest ( Mouret & Clune , 2015 ) . QD approaches generally build on black-box optimization methods such as evolutionary algorithms to optimize a population of solutions ( Cully & Demiris , 2017 ) . These algorithms often rely on random mutations to explore small search spaces but struggle when confronted to higher-dimensional problems . As a result , QD approaches often scale poorly in large and continuous sequential decision problems , where using controllers with many parameters such as deep neural networks is mandatory ( Colas et al. , 2020 ) . Besides , while evolutionary methods are the most valuable when the policy gradient can not be applied safely ( Cully et al. , 2015 ) , in policy search problem that can be formalized as a Markov Decision Process ( MDP ) , Policy Gradient ( PG ) methods can exploit the analytical structure of neural networks to more efficiently optimize their parameters . Therefore , it makes sense to exploit these properties when the Markov assumption holds and the controller is a neural network . From the deep reinforcement learning ( RL ) perspective , the focus on sparse or deceptive rewards led to realize that maximizing diversity independently from rewards might be a good exploration strategy ( Lehman & Stanley , 2011a ; Colas et al. , 2018 ; Eysenbach et al. , 2018 ) . More recently , it was established that if one can define a small behavior space or outcome space corresponding to what matters to determine success , maximizing diversity in this space might be the optimal strategy to find a sparse reward ( Doncieux et al. , 2019 ) . In this work , we are the first to combine QD methods with PG methods . From one side , our aim is to strongly improve the sample efficiency of QD methods to get neural controllers solving continuous action space MDPs . From the other side , it is to strongly improve the exploration capabilities of deep RL methods in the context of sparse rewards or deceptive gradients problems , such as avoid traps and dead-ends in navigation tasks . We build on off-policy PG methods to propose a new mutation operator that takes into account the Markovian nature of the problem and analytically exploits the known structure of the neural controller . Our QD-RL algorithm falls within the QD framework described by Cully & Demiris ( 2017 ) and takes advantage of its powerful exploration capabilities , but also demonstrates remarkable sample efficiency brought by off-policy RL methods . We compare QD-RL to several recent algorithms that also combine a diversity objective and a return maximization method , namely the NS-ES family ( Conti et al. , 2018 ) and the ME-ES algorithm ( Colas et al. , 2020 ) and show that QD-RL is two orders of magnitude more sample efficient . 2 PROBLEM STATEMENT . We consider the general context of a fully observable Markov Decision Problem ( MDP ) ( S , A , T , R , γ , ρ0 ) where S is the state space , A is the action space , T : S × A → S is the transition function , R : S ×A → R is the reward function , γ is a discount factor and ρ0 is the initial state distribution . We aim to find a set of parameters θ of a parameterized policy πθ : S → A so as to maximize the objective function J ( θ ) = Eτ ∑ t γ trt where τ is a trajectory obtained from πθ starting from state s0 ∼ ρ0 and rt is the reward obtained along this trajectory at time t. We define the Q-value for policy π , Qπ : S × A → R as Qπ ( s , a ) = Eτ ∑ t γ trt , where τ is a trajectory obtained from πθ starting from s and performing initial action a. QD aims at evolving a set of solutions θ that are both diverse and high performing . To measure diversity , we first define a Behavior Descriptor ( BD ) space , which characterizes solutions in functional terms , in addition to their score J ( θ ) . We note bdθ the BD of a solution θ . The solution BD space is often designed using relevant features of the task . For instance , in robot navigation , a relevant BD is the final position of the robot . In robot locomotion , it may rather be the position and/or velocity of the robot center of gravity at specific times . From BDs , we define the diversity ( or novelty ) of a solution as measuring the difference between its BD and those of the solutions obtained so far . Additionally , we define a state Behavior Descriptor , or state BD , noted bdt . It is a set of relevant features extracted from a state . From state BDs , we define the BD of a solution θ as a function of all state BDs encountered by policy πθ when interacting with the environment , as illustrated in Figure 1a . More formally , we note bdθ = IEτ [ fbd ( { bd1 , . . . , bdT } ) ] , where T is the trajectory length and fbd is an aggregation function . For instance , fbd can average over state BDs or return only the last state BD of the trajectory . If we consider again robot navigation , a state BD bdt may represent the position of the robot at time t and the solution BD bdθ may be the final position of the robot . With state BDs , we measure the novelty of a state relatively to all other seen states . The way we compute diversity at the solution and the state levels is explained in Section 4 . 3 RELATED WORK . A distinguishing feature of our approach is that we combine diversity seeking at the level of trajectories using solution BDs bdθ and diversity seeking in the state space using state BDs bdt . The former is used to select agents from the archive in the QD part of the architecture , whereas the latter is used during policy gradient steps in the RL part , see Figure 1b . We organize the literature review below according to this split between two types of diversity seeking mechanisms . Besides , some families of methods are related to our work in a lesser extent . This is the case of algorithms combining evolutionary approaches and deep RL such as CEM-RL ( Pourchot & Sigaud , 2018 ) , ERL ( Khadka & Tumer , 2018 ) and CERL ( Khadka et al. , 2019 ) , algorithms maintaining a population of RL agents for exploration without an explicit diversity criterion ( Jaderberg et al. , 2017 ) or algorithms explicitly looking for diversity but in the action space rather than in the state space like ARAC ( Doan et al. , 2019 ) , P3S-TD3 ( Jung et al. , 2020 ) and DvD ( Parker-Holder et al. , 2020 ) . We include CEM-RL as one of our baselines . Seeking for diversity and performance in the space of solutions Simultaneously maximizing diversity and performance is the central goal of QD methods ( Pugh et al. , 2016 ; Cully & Demiris , 2017 ) . Among the various possible combinations offered by the QD framework , Novelty Search with Local Competition ( NSLC ) ( Lehman & Stanley , 2011b ) and MAP-Elites ( ME ) ( Mouret & Clune , 2015 ) are the two most popular algorithms . In ME , the BD space is discretized into a grid of solution bins . Each bin is a niche of the BD space and the selection mechanism samples individuals uniformly from all bins . On the other hand , NSLC builds on the Novelty Search ( NS ) algorithm ( Lehman & Stanley , 2011a ) and maintains an unstructured archive of solutions selected for their local performance . Cully & Mouret ( 2013 ) augment the NSLC archive by replacing solutions when they outperform the already stored ones . In QD-RL , we build on the standard ME approach and on an augmented version of the NSLC algorithm where the population is selected from a global quality-diversity Pareto front inspired from Cully & Demiris ( 2017 ) . Relying on these components , algorithms such as QD-ES and NSR-ES have been applied to challenging continuous control environments in Conti et al . ( 2018 ) . But , as outlined in Colas et al . ( 2020 ) , these approaches are not sample efficient and the diversity and environment reward functions could be mixed in a more efficient way . In that respect , the most closely related work w.r.t . ours is Colas et al . ( 2020 ) . The ME-ES algorithm also optimizes both quality and diversity , using an archive , two ES populations and the MAP-ELITES approach . Using such distributional ES methods has been shown to be critically more efficient than population-based GA algorithms ( Salimans et al. , 2017 ) , but our results show that they are still less sample efficient than off-policy deep RL methods as they do not leverage the policy gradient . Finally , similarly to us , the authors of Shi et al . ( 2020 ) try to combine novelty search and deep RL by defining behavior descriptors and using an NS part on top of a deep RL part in their architecture . In contrast with our work , the transfer from novel behaviors to reward efficient behaviors is obtained through goal-conditioned policies , where the RL part uses goals corresponding to outcomes found by the most novel agents in the population . But the deep RL part does not contain a diversity seeking mechanism in the state space . Seeking for diversity and performance in the state space Seeking for diversity in the space of states or actions is generally framed into the RL framework . An exception is Stanton & Clune ( 2016 ) who define a notion of intra-life novelty that is similar to our state novelty described in Section 2 . But their novelty relies on skills rather than states . Our work is also related to algorithms using RL mechanisms to search for diversity only , such as Eysenbach et al . ( 2018 ) ; Pong et al . ( 2019 ) ; Lee et al . ( 2019 ) ; Islam et al . ( 2019 ) . These methods have proven useful in the sparse reward case , but they are inherently limited when the reward signal can orient exploration , as they ignore it . Other works sequentially combine diversity seeking and RL . The GEP-PG algorithm Colas et al . ( 2018 ) combines a diversity seeking component , namely Goal Exploration Processes ( Forestier et al. , 2017 ) and the DDPG deep RL algorithm ( Lillicrap et al. , 2015 ) . This sequential combination of exploration-then-exploitation is also present in GO-EXPLORE ( Ecoffet et al. , 2019 ) and in PBCS ( Matheron et al. , 2020 ) . Again , this approach is limited when the reward signal can help orienting the exploration process towards a satisfactory solution . These sequential approaches first look for diversity in the space of trajectories , then optimize performance in the state action space , whereas we do so simultaneously in the space of trajectories and in the state space . Thus , as far as we know , QD-RL is the first algorithm optimizing both diversity and performance in the solution and in the state space , using a sample efficient off-policy deep RL method for the latter .

Summary: This paper is addressing the problem of hard exploration / escaping local minima in continuous control, by optimizing a population of agents for both environment reward and diversity, using both off-policy RL and Quality-Diversity (QD) optimization. Each agent is individually optimized using off-policy RL for either environment reward or diversity, and at a population level, individuals are selected using a QD method such as Map-Elites. On a hard exploration problem, Ant-Maze (Colas et al 2020), the proposed method is shown to be significantly more data-efficient compared to several other SOTA ES methods (ME-ES (Colas et al, 2020), NS-ES (Conti et al 2018), NSR-ES (Conti et al 2020)).