Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery
Authors: Felix Chalumeau
Raphael Boige
Bryan Lim
Valentin Macé
Maxime Allard
Arthur Flajolet
Antoine Cully
Thomas PIERROT
Venue: ICLR 2023 notable top 25% (2023)

Abstract: Deep Reinforcement Learning (RL) has emerged as a powerful paradigm for training neural policies to solve complex control tasks. However, these policies tend to be overfit to the exact specifications of the task and environment they were trained on, and thus do not perform well when conditions deviate slightly or when composed hierarchically to solve even more complex tasks. Recent work has shown that training a mixture of policies, as opposed to a single one, that are driven to explore different regions of the state-action space can address this shortcoming by generating a diverse set of behaviors, referred to as skills, that can be collectively used to great effect in adaptation tasks or for hierarchical planning. This is typically realized by including a diversity term - often derived from information theory - in the objective function optimized by RL. However these approaches often require careful hyperparameter tuning to be effective. In this work, we demonstrate that less widely-used neuroevolution methods, specifically Quality Diversity (QD), are a competitive alternative to information-theory-augmented RL for skill discovery. Through an extensive empirical evaluation comparing eight state-of-the-art algorithms (four flagship algorithms from each line of work) on the basis of (i) metrics directly evaluating the skills' diversity, (ii) the skills' performance on adaptation tasks, and (iii) the skills' performance when used as primitives for hierarchical planning; QD methods are found to provide equal, and sometimes improved, performance whilst being less sensitive to hyperparameters and more scalable. As no single method is found to provide near-optimal performance across all environments, there is a rich scope for further research which we support by proposing future directions and providing optimized open-source implementations.

Published as a conference paper at ICLR 2023
NEUROEVOLUTION IS A COMPETITIVE ALTERNATIVE TO REINFORCEMENT LEARNING FOR SKILL DISCOVERY
Felix Chalumeau" Raphael Boige*! Bryan Lim?
Valentin Macé'
Maxime Allard? Arthur Flajolet! Antoine Cullyf2
Thomas Pierrott
"InstaDeep IImpeiil College
ch.lluuaaa,, r. r.boge, v.mace, alfajtte,, J.pierrot]@instadeepp Jinstadeep.comm (bw1116, m.allard20, a.cully)@ic.ac.uk
ABSTRACT
Deep Reinforcement Learning (RL) has emerged powerful paradigm for train- ing neural policies to solve complex control tasks. However, these policies tend be overfit to the exact specifications of the task and environment they were trained on, and thus do not perform well when conditions deviate slightly or when composed hierarchically to solve even more complex tasks. Recent work has shown that training mixture of policies, opposed to single one, that are driven explore different regions of the state-action space can address this shortcoming by generating diverse set of behaviors, referred to skills, that can be collectively used to great effect in adaptation tasks or for hierarchical planning. This is typically realized by including diversity term often derived from information theory in the objective function optimized by RL. However these approaches often require careful hyperparameter tuning be effective. In this work, demonstrate that less widely-used neuroevolution methods, specifically Quality Diversity (QD), are competitive alternative to information-theom thhhoryaaggeeeeeeee augmented RL for skill discovery. Through extensive empirical evaluation comparing eight state-of-the-art algo- rithms (four flagship algorithms from each line of work) on the basis of (i) metrics directly evaluating the skills' diversity, (ii) the skills' performance on adaptation tasks, and (iii) the skills' performance when used as primitives for hierarchical planning: QD methods are found to provide equal, and sometimes improved, per- formance whilst being less sensitive to hyperparameters and more scalable. As no single method is found to provide opar-optimal performance across all environ- ments, there is a rich scope for further research which we support by proposing future directions and providing optimized open-source implementations.
INTRODUCTION
In the past decade, Reinforcement Learning (RL) has shown great promise at tackling sequential decision making problems in generic fashion, leading breakthroughs in many fields such as games (Silver et al. 2017). robotics (Andrychowicz et al. 2020). and control in industrial settings (De grave al. 2022). However, neural policies trained with RL algorithms tend to be over-tuned the exact specifications of the tasks and environments they were trained on. Even minor disturbances the environment, to the starting state, or the task definition can incur significant loss of performance (Kumar et al.) 20200 Pinto et al. 2017). A standard approach to improve generalization introduce more variations during training (Tobin al. 2017) but this assumes we can foresee all possibilities, which is not always true in the real world. Even in settings where this is feasible, intro- ducing a wide spectrum of variations will make the problem harder to solve and the resulting policy may not perform as well in the nominal case. Another approach consists in co-training an adversarial agent whose task to perturb the environments so as to minimize the policy's performance (Pinto et al. 2017). However, adversarial methods are notoriously unstable in Deep Learning (Arjovsky & Bottou 2017) and can also compromise performance in the nominal scenario.
"Equal Contribution
1 Equal Supervision

---

Published as a conference paper at ICLR 2023
To improve robustness without explicitly identifying all possible variations, jointly training multiple policies to solve the same task in diverse ways has emerged as a promising line of work in the RL literature (Kumar et al. 2020). To motivate the approach, consider the problem of learning a policy to control the joints of a legged robot with the goal of running as fast as possible. Any damage to the robot legs might affect an optimal policy's ability make the robot run fast, if at all. Yet, many of the slightly sub-optimal policies to the original problem (e.g. a policy making the robot hop using only one leg) would perform equally well in this perturbed setting. Two seemingly-opposs main lines of work have been pursued to maximize both performance and diversity in a collection of policies.
RL-rooted approaches (Bysenbach et al,, 2019; Sharma et al., 2019 Kumar et al.) 2020) introduce a generated latent variable and parametrize the policy to be function of the state as well as this latent variable. At training time, the latent variable drawn from a static distribution and fed as an input alongside the state to the policy, effectively defining mixture of policies. To encourage diversity among these policies, a term derived from information theory that depends both on the policy parameters and the latent variable is added to the objective function (hereinafter referred to as fitness function). This term is typically formulated as the mutual information between the latent variable and subset of the policy's trajectory, possibly conditioned on observations from the past.
randomly-generated
Neuroevolution rooted approaches instead stem from the subfield of Quality Diversity (QD) opti- mization (Pugh et al. 2016 Cully & Demiris 2017 Chatzilygeroudis et al. 202I) and combine the tools developed in this space with RI. algorithms to get the best of both worlds (Nilsson & Cully 2021: Pierrot et al.) )022). QD optimization aims at generating and maintaining large and diverse collections of solutions, as opposed to a single optimal solution in Optimization Theory, by imitating the natural evolution of individuals competing for resources in their respective niches. In comparison to traditional Evolutionary Strategies, QD algorithms explicitly use mapping from solution to a vector space, referred as behavior descriptor space, to characterize solutions and maintain data structure, a repertoire, filled with high- figh-performing solutions that cover this space as much as possible.
Evolutionary Strategies (possibly hybridized with RL algorithms) have proven to be competitive alternative to RL algorithms for many common sequential- making problems (Pierrot et al., 2022, [Salimans et al. 2017). Hence, natural to believe that QD algorithms could also be competitive with high-performing policies in similar settings. Yet, QD approaches remain neglected skill-discovery studies (Kumar et al. 2020), perhaps because they lack the sample- sumple-efficiency of state-of offe-oof-the-rt RL algorithms, sometimes requiring two orders of magnitude more interactions with the environment to solve task (Pierrot et al. 2022). While this is significant shorteoming for real-world applications that cannot be accurately described by a computational model, simulators are readily available for many applications. Additionally, when the simulator and the algorithm are implemented using modern vectorized frameworks such as JAX (Bradbury et al. 2018) and BRAX (Freeman et al. [2021] evolutionary approaches are competitive with RL approaches in terms of total training time on an accelerator in spite of the low sample-efficiems of these methods (Lim et al. 2022).
RL approaches generate diverse populations of
Our contributions are the following. (1.) We provide extensive experimental evidence that QD methods are competitive with RL ones for skill discovery in terms of performance given fixed compute and training time budgets and hyperparameter sensitivity. Specifically, using environments taken from the QD and RI. literature, we compare eight state-of-the-art skill-discovery methods from the RT. and QD world on the basis of (i) metrics directly evaluating the skills' diversity, (ii) the skills' performance on adaptation tasks, and (iii) the skills' performance when used as primitives for hierarchical planning. (2.) We open source efficient implementations of all environments and algorithms] based on the QDax library Armed with these, running any of the experiments, some of which require hundreds of millions of environments steps, takes only 2 hours on a single affordable accelerator. (3.) We provide a detailed analysis of the strengths and weaknesses of all methods, show that no single method outperforms all others on all environments, and we identify future research directions.
2 PRELIMINARIES AND PROBLEM STATEMENT
We consider sequential decision making problems formulated Markov Decision Processes (MDPs) and defined by (S, AA, R, P,,), where S is the state space, A is the action space, 7 E [0, 1] is the discount factor, R : 8 x A > IR is the reward signal and P S x A - S is the transition function.
2

---

Published as a conference paper at ICLR 2023
Unidirectionl Environments
Omvidraactioal
Adoptation Experiments
Gravity Change: Unidirectional
Ant-Uni
Halfcheetah-Uni
Ant-Omnl
Trap Enviranments
Maze Environments
Heerarchical Learning
Motor Unidirectional environments
Ant-Trap
Point-Mazz
Hurdles in Halfcheetah
Target Location Change: AntMaze
Figure I: Illustrations of environments used for benchmarking (left) and perturbations applied to them for adaptation and hierarchical learning experiments (right). UNIDIRECTIONAL environments, policies are trained to make the robots run forward with diverse gaits. In the OMNIDIRECTIONAL environment, the goal is to make a legged robot move on the 2D plane while minimizing the control energy. In ANT-TRAP, robot learns to in the forward direction while avoiding trap. In ANT-MAZE and POINT-MAZE, a robot must reach a target position in a maze. Once trained over these tasks, policies evaluated perturbed versions of the tasks and environments.
Policies are assumed to be implemented by neural networks. We consider two cases. (1. A mixture of policies is conditioned on a latent variable Z, in which case it is denoted by Ty SSx Z + ((A) where f denotes the neural network weights and (LA) is the space of distributions over the action space. The collection of policies II is obtained by enumerating all possible values for the latent variable. (2.) Policies share the same neural network architecture but have their own sets of parameters, In this case, II corresponds We denote the trajectory of a single policy т in the environment by T Ω. The expected sum of rewards of policy n, referred as its fitness, is F( = E,,,, (R)).
the collection of N policies Tæ, : + D(A), iE [1, N].
In this work, compared to traditional RL, we consider a second objective: the policies in the collection II, referred to as skills, must not only be high-performing but also diverse as whole. The notion of diversity can be defined in various ways. In the RI. literature, diversity is often measured by looking at the volume spanned by components of the trajectories induced by the policies, such as the sequences of visited states Holder et al. 2020). In the QD literature, diversity is measured in a so-called behavior descriptor space & using mapping & S + & that is either hand-engineered progressively refined in an unsupervised fashion. This approach is more general special case the first one but also allows to leverage expert knowledge on the task at hand, which is often available. We use the notion of behavior descriptor space in order quantitatively compare the diversity of policies generated by RI. and QD methods. Specifically, we define bounded behavior descriptor space for each MDP, that may differ from the ones used internally by QD methods, which is discretized into discrete tessellation of cells. Policies from II are inserted into the tessellation as function of their behavior descriptors which enables to use diversity metrics developed in the QD literature. This process is made precise in Section 5.2
includes
DIFFERENT SOLUTIONS TO THE SAME PROBLEM
Information- www.tttthorrreeeeeeee RL methods. Most methods aiming maximize diversity the RL
literature, such as DIAYN (Eysenbach et al. 2019) and DADS (Sharma et al. 2019), fall into the class of candnt-conditooned policies introduced in Section 2 They strive to maximize the Mutual Information (MI) between latent variable z, which is drawn randomly from a pre-specified distribution, and some components the trajectories induced by the conditioned policy. The MI maximization is approximately carried out through the introduction of a discriminator function z q(2[s) given a state E S (resp. a dynamics function 2 + q(5++1 88. 2) given the current state s, and the next state st+1) for DIAYN (resp. DADS) trained separately to differentiate the latent variables that lead to visit different states. The latent-conditonne policy is optimized using an off off-policy model-free RL algorithm by taking the reward the discriminator value, referred to intrinsic reward.
3

---

Published as a conference paper at ICLR 2023
DIAYN and DADS unsupervised algorithms as they do not aim to maximize a reward function associated with a specifie task. Recent work in MI- Mi-based RL, namely SMERL (Kumar et al. 2020). has extended DIAYN and DADS to supervised settings where both diversity and performance w.r.t
reward function should be optimized. They formulate the problem as a Constrained MDP (CMDP) where the objective is maximize the diversity of mixture of policies, while also constraining the learned policies to be close to optimality w.r.t. the reward function. By solving these CMDPs, they obtain a collection of policies that are both diverse and high-performing. SMERL can use either DIAYN or DADS internally to generate collections of policies, two variants we refer to as SMERL (DIAYN) and SMERL (DADS). An alternative approach to SMERL simply consists in summing the task reward and the intrinsic reward and using the original DIAYN and DADS algorithms (Kumar et al. 2020
Osa et al. 2022 Gaya et al., (021). This approach is very similar to Hausman et al. (2018). We refer to these variants as
and DADS REWARDD.. For clarity's sake, we only report the performance of the original DIAYN (resp. DADS) algorithm in the Appendix as it turns out to be significantly outperformed by DIAYN+REWARD (resp. DADS+REWARD) in our experiments.
QD methods fall into the second category of policies introduced in Section 2 These methods explicitly evolve and maintain discrete repertoire of independent policies. QD algorithms tradition- ally rely on Evolutionary algorithms to incrementally update policies almost independently. Such techniques have the advantage of being easily parallelizable and can be used with non-differentiabll policies and controllers (Tang et al. 2020). They have been shown to be viable alternatives to RI. algorithms (Salimans et al. 2017).
MAP-ELITES is one of the most widespread QD algorithms. This method assumes that the behavior descriptor space has been discretized into finite tessellation of cells, which together define repertoire where policies are stored. This algorithm proceeds by iterations. At each iteration, a batch of policies sampled uniformly from the repertoire and copied. Mutations and crossover operations are applied to them to obtain a new batch of policies which are evaluated in the environment to compute their fitnesses and behavior descriptors. For any of these policies, the cell in the repertoire corresponding to its behavior descriptor is empty, then it is copied into this cell. Otherwise, the new policy replaces the current incumbent only if its fitness is higher than the current incumbent's fitness.
The exact choice of mutation and crossover operators significantly affects the performance of QD algorithms. Choices that rely exclusively on random perturbations typically require a significant number of iterations. To remedy this problem, several works have combined first-order optimization methods with Evolutionary algorithms to guide the parameter search (Colas et al. 2020; Pierrot et al.. 2022). In particular, PGA-MAP-EL. on the MAP-ELITES framework to select, mutate, and replace policies in the repertoire but further introduces policy-gradient mutation operator which is used alongside the genetic operator. As part of every MAP-ELITES iteration, PGA-MAP-ELIN and stores it into replay buffer in order to train a pair of critics using the TD3 algorithm (Fujimoto et al. 2018 These critics are used to compute the policy-gradient updates applied to the policies.
(Nilsson & Cully] 2021) is a hybrid algorithm that builds
collects the experience of all evaluated policies
While MAP-ELITES and PGA- MMA--MAPPPIEE---- ELITES are provided with a pre-specified behavior descriptor space. it is defined in an unsupervised fashion in recent extension to MAP-ELITES dubbed AURORA (Cully) 2019. Grillotti & Cully, 2022a). This is achieved by training an autoencoder to reconstruct the trajectories collected in the environments and defining the behavior space as the latent space of the autoencoder. In this paper, we introduce a new unsupervised method which extends both AURORA and DEGA-MAPELLIEES This method, referred to PGA-AURORA, use the same policy-gradient mutation operator as PGA-MAP-ELITES and the same unsupervised technique as AURORA to define the behavior descriptor space.
4 RELATED WORK
Historically, QD algorithms originated from Evolutionary algorithms, which also implement selection, mutation, and evaluation of populations of solutions. Novelty Search (NS) (Lehman & Stanley aa first introduced the idea of searching for novelty alone. In a sense, NS is similar to DIAYN and DADS as it is an exploration-onfy algorithm. Novelty Search with Local Competition (NSLC) (Lehman & Stanley 2011bb and MAP-ELITES (Mouret & Clunel 2015) are two of the most popular QD algorithms. NSLC builds off the NS algorithm and, in contrast to MAP-ELITES, it maintains an unstructured repertoire of solutions that selected for their performance when compared to similar
4

---

Published as a conference paper at ICLR 2023
individuals. Relying on the same QD and Evolutionary Strategies (ES), such as NSR-ES and NSRA-ES, have been successfully applied to challenging continuous control environments (Conti et al. 2018 Aiming for improved sample efficiency. Colas et al. (020)) developed ME-ES which optimizes both for quality and diversity using MAP-ELITES, and two ES populations. Similarly, CMA-ME (Fontaine et al. (2220) uses a covariance matrix adaptation technique to model and update a population distribution to sample individuals from and also maintains a population of varied emitters to generate solutions. More recently, some authors proposed to incorporate a volicy-gradiem component in the evolutionary framework (Nilsson the authors of QD-PG (Pierrot et a 2022) introduced a diversity policy gradient to make diversity information available at the timestep level.
strategies, QD-ES algorithms, which combine
Cully 2021 Pierrot et al. 2022] Tjanaka et al.) 2022). Inspired by
Multiple methods from the RI. literature build an intrinsic reward with MI to learn diverse set of skills. They are similar in spirit to DIAYN and can be adapted, along the same lines as what is done for SMERL, to use extrinsic rewards. Gregor et al. (2016) originally introduced MI for skill discovery. Most recent efforts in the skill-discovery RL literature focus on ways to extend DIAYN and DADS to improve the state coverage of the learned skills by using contrastive learning (Laskin al. 2022), adding constraints to the skills (Hansen et al. | et al. (021), or changing the loss function (Durugkar et al. 2021). Nevertheless, Zahavy al. (2021) introduced original approach where the intrinsic rewards are defined based on successors features rather than MT. We choose DIAYN, DADS, and SMERL as basclines in our experiments as they are the most widely used RL methods for skill discovery at the time of writing. There also exist methods based on tools other than MI that strive evolve diverse population of RL agents, such as Zhou et al. (2022) Parker-
et al. (0200); Zhang et al. 22019); Masood & Doshi Velez (2019). but these studies mostly consider diversity as means to increase performance w.r.t. the rewards, whereas DIAYN, SMERL, and DADS are evaluated on pure diversity, on adaptation tasks, or in hierarchical learning settings.
5 EXPERIMENTS
In this section, we present the frameworks used to compare the methods listed in Section 3 and analyze the results. First, in Section [.I we introduce the environments and tasks upon which the frameworks are built. In Section 5.2, we leverage tools and metrics from the QD literature to directly evaluate the diversity of the solutions found by each method in the process of solving the tasks. In Section 5.3 we investigate how diversity translates into robustness with few-shot adaptation experiments inspired from the RL literature. Finally, in Section 5.4] we evaluate the policies found by each method when used as primitives in a hierarchical learning setting.
5.1 ENVIRONMENTS
In order not to favor any method in the study, we use three different categories of environments and tasks: (i) continuous control locomotion tasks from the RL literature, (ii) exploration tasks with deceptive rewards from the QD literature, and (iii) hybrid tasks mixing both types as a middle ground. All environments are illustrated Figure [
Low-dimensionall exploration task with deceptive rewards. Following several works from both the skill-discovery RL and QD communities (Eysenbach al. 2019 Kamienny et al. 2021 Campos et al. 2020), we consider a simple navigation environment where a point is controlled to navigate on 2D plane a specified target position. To turn the task into a challenging exploration problem, we introduce a maze through which the point must navigate to reach the target position. We refer to this task as POINT- MAZE.
High-dimensional locomotion tasks. We consider three locomotion tasks, namely ANT, HALFCHEE- TAH and WALKER, often used the RI. community to evaluate algorithms, including in RI. works focused on diversity-seeking strategies (Nilsson & Cully) 2021 Eysenbach et al. 2019: Fujimoto ct al. 2018). In these environments, the problem is to move legged robots by applying torques to their joints through actuators. These tasks are challenging for evolutionary algorithms: they typically require orders of magnitude more environment interactions than RL methods to evolve competitive policies (Colas et al. 2020 Pierrot et al., (022). We define two subsets of tasks for these environments. The first subset, referred to as UNIDIRECTIONAL locomotion tasks and with the reward signal defined as the x-axis velocity minus the energy consumption, includes ANT-UNI, WALKER- -NRI,
5

---

Published as a conference paper at ICLR 2023
SMERL(DIAYN)
SMERLDODDS)
DEDS-R
DIAYN REWARD
PGA-ME
PGA-AURORA
MAP-EIite
AntTrop
Anttaase
15090
PointMarz
Donn
Walker24 Uni
Maloheocth Uni
Figure 2: Visualizations of repertoires post training. Each repertoire divides the behavior descriptor space into 1024 cells. Cells are left blank when no policy with the corresponding behavior descriptor has been found and are otherwise colored as a function of the best fitness achieved across policies with matching behavior descriptor. For ANT-TRAP, ANT-MAZE, POINT-MAZE, and ANT-OMNI, the behavior descriptor is the (x. y) position reached at the end of an episode: these plots highlight explored areas of the 2D plane. For HALFCHEETAH-UN. and WALKER-UND the descriptor is the fraction of time the robots' feet were touching the ground during an episode (one dimension per foot).
and HALFCHEETAH-UNT The second subset, referred to as OMNIDIRECTIONAL locomotion tasks and with the reward signal defined as the opposite of the energy consumption, includes ANT-OMNI.
Mixing locomotion and exploration tasks. Following Pierrot et al. (0022;; Parker-Holder et al. (2020): Frans al, (2018): Shi et al. (2020), we consider two additional hybrid tasks built upon ANT that cxhibit both continuous control and exploration challenges, making them particularly difficult even for statc-of-tha art evolutionary and RL algorithms. In ANT-TRAP, the ant has to run forward while initially facing trap consisted of three walls that prevents it from moving forward along the x-axis direction indefinitely. In AAT-MAZE, the ant is initially placed in a maze and the goal is to reach a specified target position.
5.2 EXPERIMENTS DIRECTLY EVALUATING DIVERSITY
On the one hand, metrics quantifying the amount of diversity in a collection of policies are lacking in the MI literature. Success inducing diversity is instead typically assessed indirectly through performance on adaptation or hierarchical learning experiments where either the tasks are modified
or new ones are introduced (Eysenbach al., 2019 Kumar al. 2020 Sharmaa et al. 2019). On the other hand, diversity metrics were introduced very early in the QD literature and have been refined over time since QD algorithms are primarily assessed on this basis. Hence, we use metrics developed in the QD literature and extend them to MI RL approaches to be able compare the methods on the same ground. Noting that almost all QD metrics crucially rely on the definition a behavior descriptor space (to characterize solutions) and the availability of a repertoire of solutions indexed with respect to this space for all methods, we specify behavior descriptors for all tasks and extend the RL methods.
concept of repertoires
Behavior descriptor spaces. For POINT-MAZE, ARAP, AAT-MAZE, and ANT-OMNI, following
several prior and RI. works (Pierrot al. 2022 Hansen al. 2021) the behavior descriptor of

---

Published as a conference paper at ICLR 2023
Table I: Maximum fitness and QD score at the end of the training phase (median over 5 seeds). Additional statistics (mean, standard deviation, and interquartile mean) are included in the Appendix.
ANT-TRAP ANT-MAZE
POINT-MAZE
ANT-UNI
ANT-OMNI
WALKER-UNI
HALFCHECTAII-UN
Maximum fitnsss
(MEAT((DAYY)
c.
c3.88
el
1.71
2.49 e2
1.19
1.89
SMERL(DADS)
4.48
-7.62
-8.10c
1.61
2,49
1.17c3
c3
MADS-REWARD
c.28
-3.81
-8.10
c.24
2.38
6.30
1.89
DIAYN+REWARD
9,42
-3.13e3
el,00el
c334
2.29
1.00
2.13
PGA-MAP DAA-MA-FLIIESS
9.77
.9.32
-2,30
e.45
2,49
1.22 e3
1.75
PGA-AURORA
c.72
.9.27
-6.eel
2.49
1,05
e37
MAP-OLITES
3,81 c2
-1.01 04
-2.cll
4.25
2,49
6.87 c2
1.23
AURORA
4.05 c2
-1.10
-2.35
5.05
2.49 e2
5,71
1.21
QD score
SMERL(DIAYN)
1.02
c5
38.9 04
6.20
c,95
3.01
7.92 c5
(DADS(
9.88
8.22
3.57 c3
0.83 e5
4.15
2.69 c5
7.51 c5
DADS-REWARD
e,42
e4
2,88
3.05
3.89
8.31 e5
DIAYN+REWARD
e.27
9.18e5
4.04
1.29
e.00
4.92
1.68 c6
PGA-MAP-ELITES
c5
2.74
c.92
e.
c,58
8,23
2.98
PGA-AURORA
c.40
c.59
2.83 c5
e.
c.69
5.10
2.39 c6
ELITES
e.
2,98
4.22
9.08 e5
3.04
6,45
3.05
3,54
2.18
c.58
e.82
4,00
4.59
2.45
policy is defined the (x, y) position of the robot's center of gravity at the end of evaluation episode. For UNIDIRECTIONAL tasks, the i-th component the behavior descriptor vector is defined the proportion of time during which the i-th foot of the robot is in contact with the ground in evaluation episode. This is simple but effective way to capture the gait robot as demonstrated Cully et (2015) where it used help robots adapt after having some of their legs damaged.
We stress that, in our experiments, MAP-ELITES and PGA-MAP-ELITES make use of the same behavior descriptors used to compute the QD metrics, which induces bias in the analysis. This bias does not affect AURORA and PGA-AURORA as these methods build their own behavior descriptors in an unsupervised fashion. Nevertheless, for fairness, we embed the choice of the behavior descriptor space into MI RI methods whenever possible. Hence, in ANT-MAZE, ANT-TRAP, ANT-OMNI and POINT-MAZE, MI RI. algorithms are provided with the prior that diversity w.r.t. the (æ, y) position of the robot is what needs to be optimized. Implementation details are deferred to the Appendix.
Passive repertoires for RL methods. The MI methods considered in this paper do not maintain repertoire to store diverse solutions. For fairness, and since increasing the size of the discrete latent space to match the size of the QD repertoires makes these methods unstable (Eysenbach et al., 2019), we fill repertoire on the side during training with all the intermediate policies found by these methods, similarly to what is done internally in QD methods. The repertoires of size 1024 are identically defined across methods using Centroidal Voronoi Tessellations (Vassiliades et al. 2017).
QD Metrics. Armed with behavior descriptor spaces for all environments and repertoires for all methods, compute three metrics that often used in the QD community to track the performance of a collection of solutions: (i) the maximum fitmess of the repertoire, computed as the maximum fitness attained across the solutions stored in the repertoire, (ii) the coverage of the repertoire, computed as the number of cells (behavior niches) that have been filled, and (iii) the QD score. computed as the sum of fitnesses attained by the solutions stored the repertoire.
Results and analysis. All methods are trained for exactly two hours on Quadro RTX 4000 GPU for each environment described in Section 5.1 Each experiment repeated with 5 seeds and the final repertoire is evaluated using the aforementioned QD metrics. The maximum fitness and QD score are reported in Table [ Out of the 5 seeds, the best in terms of QD score final repertoires are depicted on Figure 2 for all methods and environments.
Helf-heolahh
Werker
Figure 3: Distribution of QD scores obtained
over range of hyperparameters on UNIDI- RECTIONAL. tasks and ANT-OMNT.
First, observe that, setting aside ANT-OMNI where DADS+REWARD has managed to largely explore the environment, methods outperform the RL methods for all environments and tasks if base the comparison on QD scores and coverage only. This a particularly desirable property in ANT-MAZE (resp. ANT-TRAP) because reaching high coverage of the repertoire means that the
7

---

Published as a conference paper at ICLR 2023
Walker2D Uni
Ant Uni
HalfCheetah Uni
AntMaze
Leg dysfunction
1040
1000
1240
000
600
420
12
0.
10
45
0.0
13
Gravity update
Lo02
1500
1200
11000
500
420
Gravity coefficlent 45
30
2.5
30
A3
0.0
0.5
1.0
Gravity cooficiont
Gravity coomiciont
Max fitness
SMERL (DIAYN)
SMERL (DADS) --- REWD- - REWARD --- PPGA-ME - MAP-Elites
AURORA -- -GAAAUURDAA
Figure 4: Maximum fitness of the methods when confronted to (left) a perturbed environment and (right) modified task (median and interquartile range on 5 seeds, additional statistics are in the Appendix). We observe varying resilience levels across methods. In particular, PGA-MAP-FI.T and PGA-AURORA adapt better to extreme environment perturbations in UNIDIRECTIONAL environments.
method has been able to explore the maze (resp. get past the trap) effectively given that the behavior descriptor vector is defined as the final (I, v) position of the robot.
Second, note that no single method outperforms all others in terms of maximum fitness. PGA-MAP- ELITES ranks first on MAINT-MAZE and WALKER- but is always outperformed by an MI RL method in the others. Additional observations are included in the Appendix.
While the excellent performance of REWARD on ANT-OMNI is impressive achievement in itself, we run an additional set of experiments to investigate whether this may be the result of a lucky guess the exact choice of hyperparameters. Specifically, we run the same experiments on ANT-OMNI and UNIDIRECTIONAL environments multiple times for DADS+REWARD and MAP- ELITES, sampling from a grid of reasonable hyperparameters values for both methods (exact details are deferred to the Appendix). The medians and quartiles of the QD-score distributions obtained are shown on Figure 3 We observe that the performance of DADS+REWARD is very sensitive to the exact choice of hyperparameters and that the median performance of MAP-ELITES is always significantly better than the median performance of DADS+REWARD on all tasks.
5.3 FEW-SHOT ADAPTATION EXPERIMENTS
While metrics borrowed from the QD literature provide first basis for comparison, they suffer from an inherent bias in favor of QD methods as discussed in Section 3.2 Following prior work that analyze the resilience of RL methods to changes in the environment or task definition (Kumar al. 2020), we design two series of few-shot adaptation experiments that do not favor any particular method a priori. Few-shot here refers the fact that no re-training is allowed upon modification of the task or the environment, only evaluations of pre-computed policies are allowed. Throughout this section, MAP-ELITES and PGA-MAP-ELITES use the behavior descriptor spaces introduced in Section 5.2
Environment adaptation experiments. In the first series of experiments, we modify a property of the UNIDIRECTIONAL environments after training, as illustrated on Figure [ First, we vary the gravity coefficient in a neighborhood grid of size 20 around the default value (i.c. 1). We jointly refer to these experiments as the gravity-update setting. Second, we vary the actuators input-to-torque coefficients for whole leg of the robot in a neighborhood grid of size 20 around the default value (i.e. 1). We jointly refer to these experiments as the leg-dysfunction setting.
Task adaptation experiments. In the second series of experiments, we modify the task of the ANT-MAZE environment by moving the target position after training, which directly modifies the reward signal. New target positions are sampled uniformly the 2D plane delimited by the maze.
For both series of experiments, we train all methods on the nominal versions of the environments for exactly two hours and re-evaluate the final output of the training phase the modified environment or task. Specifically, for each method, we enumerate all their learned skills, evaluate them one hundred times in the modified environment, and pick the one that performs best in terms of median fitness across evaluations. Each experiment is repeated with 5 different seeds and the maximal fitnesses obtained are reported on Figure 4 for the environment and task adaptation experiments.
8

---

Published as a conference paper at ICLR 2023
Results and analysis. First, note that, on the environment adaptation experiments, PGA- SAA--MPP-LITTEE and PGA-AURORA are performing on par with MI RL methods in the case of small perturbations, i.e. when the coefficient is perturbed by less than 50%. Methods that do not leverage policy gradients (i.e. MAP-ELITES and AURORA) are outperformed on all tasks in this regime.
Second, QD methods more resilient than MI RL when confronted with significant perturbations. For instance, in the gravity-update setting, the relative performance loss is limited to 40% for PGA- MAP-ELITES when the gravity coefficient divided by 4 compared to at least 60% for MI RL methods. As a testament of the resilience of QD methods, note that the best performing method is a QD one for 10 out of the 12 most extreme scenarios. Similar observations hold for ANT-MAZE where "GA-MAP-ELITES and PGA-AUROR, perform as well as DIAYN+REWARD and DADS+REWARD.
Third, it is remarkable that PGA-AURORA and AURORA perform almost as well as their QD supervised counterparts in these experiments. This shows that unsupervised behavior descriptor learning is really effective for QD methods. Furthermore, observe that PGA- AURORA adapt with similar loss fitness than MI RL methods (recall that all of these methods have access to the exact same priors).
5.4 HIERARCHICAL LEARNING EXPERIMENTS
To further assess the skills discovered by the methods during training on nominal environment, we introduce a hierarchical learning setting inspired from the RL literature (Eysenbach et al. 2019). In HALFCHEETAH--HEE HURDD,, hurdles are spaced out along the x-axis direction in front of the robot's initial position in HCHHETH see Figure [ The robot has to jump over them to move forward, the reward definition remaining unchanged. Just like in Eysenbach et al. 2019), we train PPO meta-controller (Schulman et al. 2017) to choose a skill, which is to be unrolled in the environment for the next ten consecutive timesteps, among the ones learned by given skill-discovery method during training on the nominal version of the environment, see Section [52 The meta meta-controller is
trained for 200 million policy interactions, corresponding to 20 million actions of the meta-controller.
Results and analysis. The performance of the meta-controller is shown on Figure 5 for various choices of skill-discovery methods used to generate primitives. By visualizing the meta policies in the environment, we observe that PPO is able to learn a meta-controller that consistently manages to jump over multiple consecutive hurdles when it trained with primitives derived from "GA-MAP-ELITE and DIAYN+REWARD, which achieve similar performance on this experiment. Surprisingly, all other skill-discovery methods fail to provide primitives that can be composed to make the robot jump over hurdles.
Halfcheetah Hurdles
000
00
0.5
10
20
23
30
Num Stops
SMERL (DUAYN)
ME
SMERL (DADS)
MAP ElRes
6 DISCUSSION AND CONCLUSION
PCA AURORA
Figure 5: Fitness of the meta-
Our experiments show that QD methods are competitive with MI RL methods when the goal is to find set of diverse and high- performing policies in continuous control and exploration en- vironments. MI RL methods are task-agnostic and data-efficient but struggle to generate large collections of policies. Moreover, they are limited in the set of de- scriptors priors they can accommodate, QD methods are less data-efficient but can better leverage the vectorization capabilities of modern libraries such as JAX. Moreover, they are significantly less sensitive to the choice of hyperparameters than MI RL methods and their ability to evolve large sets of diverse solutions is a precious tool for hierarchical learning and adaptation applications when the task definition changes or when confronted with extreme environment perturbations. Our benchmarks however show that QD methods do not outperform MI RI. methods and vice-versa which opens the door to promising work directions. We believe that MI RI methods could make use of skill repertoires to improve their memory capacity thereby increasing the pool of available options for downstream applications. Conversely, QD methods leveraging gradients, such as PGA-MAP-ELITES could benefit from the introduction of intrinsic rewards described in the MI RL literature to guide the search for diversity. Ultimately. we believe that combining the coverage capacity of a repertoire with the data-efficiency of gradient-based techniques key to develop robust, diverse, and high-performing repertoires of solutions for challenging continuous control.
controller function of environ- ment steps for all types of skills primitives (median on seeds).

---

Published as a conference paper at ICLR 2023
REFERENCES
OpenAl:
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In International Conference on Learning Representations, 2017.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http:// /tttp://githu... cww.//oog///////ooooooooo/
Victor Campos, Alexander Trott, Caiming Xiong. Richard Socher, Xavier Giró-i Nieto, and Jordi Tor- res. Explore, discover and learn: Unsupervised discovery of state-covering skills. International Conference on Machine Learning, pp. 1317-1327, 2020.
Konstantinos Chatzilygeroudis, Antoine Cully, Vassilis Vassiliades, and Jean- -ap-tttt Mouret. Quality-diversity optimization: a novel branch of stochastic optimization. Black Box Op- timization, Machine Learning, and No-Free Lunch Theorems, pp. 109-135, 2021.
Cédric Colas, Vashisht Madhavan, Joost Huizinga, and Jeff Clune. Scaling map-elites to deep neuroevolution. In Proceedings of the Genetic and Evolutionary Computation Conference, pp.
67-75, 2020.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via a population of novelty-sceking agents. In Advances in Neural Information Processing Systems, pp. 5027-5038, 2018
Proceedings of the Genetic and Evolutionary Computation Conference, pp. 81-89, 2019. Antoine Cully. Autonomous skill discovery with diverrityy and unsupervised descriptors. In
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245-25 2017.
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiss Mouret. Robots that adapt like animals. Nature, 521(7553):503-507, 2015.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897):41 419, 2022.
Gaurav Dixit, Everardo Gonzalez, and Kagan Tumer. Diversifying behaviors for learning in asymmet- ric multiagent systems. Proceedings of the Genetic and Evolutionary Computation Conference, pp. 350-358, 2022.
Ishan Durugkar, Steven Hansen, Stephen Spencer, and Volodymyr Mnih. Wasserstein distance maximizing intrinsic control. arXiv preprint arXiv:21 10.2111.5550 15331, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity you need: Learning skills without reward function. In International Conference Learning Representations, 2019.
Matthew C Fontaine, Julian Togelius, Stefanos Nikolaidis. and Amy K Hoover. Covariance matrix adaptation for the rapid illumination behavior space. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 94- 2020.
Kevin Frans, Jonathan Ho. Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In International Conference Learning Representations, 2018.
10

---

Published as a conference paper at ICLR 2023
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint 11Xiv:2106.13281,, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference Machine Learning, pp. 1587-1596, 2018.
Jean-Baptiste Gaya, Laure Soulier, and Ludovic Denoyer. Learning a subspace of policies for online adaptation in reinforcement learning. arXiv preprint arXiv:21 2021.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. In International Conference Learning Representations Workshop Track Proceedings, 2016.
Luca Grillotti and Antoine Cully. Unsupervised behavior discovery with quality-diversity optimiza- tion. IEEE Transactions on Evolutionarr Computation, 26(6):153--1522 2022a.
Luca Grillotti and Antoine Cully. seteaanceggidder unsupervised discovery of abilities with quality- diversity algorithms. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 77-85, 2022b.
Steven Hansen, Guillaume Desjardins, Kate Baumli, David Warde-Farley, Nicolas Heess, Simon Osindero, and Volodymyr Mnih. Entropic desired dynamics for intrinsic control. In Advances in Neural Information Processing Systems, volume 34, pp. 11436-11448 2021.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. International Conference on Learning Representations, 2018.
fierre-Alcxandr Kamienny, Jean Tarbouriech, Sylvain Lamprier, Alessandro Lazaric, and Ludovic Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. arXiv preprint arXiv:2110.14455. 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33:8198-8211 2020.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cic: Contrastive intrinsic control for unsupervised skill discovery. arXiv preprint arXiv:2202.00161, 2022.
Joel Lehman and Kenneth Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189-22) 123-223, 201 la.
Joel Lehman and Kenneth Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 211-218, 201
Bryan Lim, Maxime Allard, Luca Grillotti, and Antoine Cully. Accelerated quality-diversity for robotics through massive parallelism. arXiv preprint arXiv:2202.01258. 2022.
Muhammad A Masood and Finale Veshi-Velez. Diversity-inducing policy gradient: Using maximum mean discrepancy to find set of diverse policies. In International Joint Conference on Artificial Intelligence, 2019.
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv: 1504009990 2015.
Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 866-876 875, 2021.
11

---

Published as a conference paper at ICLR 2023
Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Discovering diverse solutions in deep reinforcement learning by maximizing state- state-action mutual information. Neural Networks, 152:90-104, 2022.
Jack Parker-Holder, Aldo Pacchiano, Krzysztof Choromanski, and Stephen Roberts. Effective diversity in population-baser reinforcement learning. In Neural Information Processing Systems, 2020.
Thomas Pierrot, Valentin Macé, Félix Chalumeau, Arthur Flajolet, Geoffrey Cideron, Karim Beguir, Antoine Cully, Olivier Sigaud, and Nicolas Gelrin-Gilbert. Diversity policy gradient for sample effi- cient quality-diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1075-1083, 2022.
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce- ment learning. In International Conference on Machine Learning, pp. 2817-2826, 2017.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint arXiv:/703.0386- 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprini arXiv: 1rXXvv1907.0167. 2019.
Longxiang Shi, Shijian Li, Qian Zheng, Min Yao, and Gang Pan. Efficient novelty search through deep reinforcement learning. IEEE Access, 11112889-1288111 2020.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815. 2017.
Yujin Tang, Duong Nguyen, and David Ha. Neuroevolution of self-interpretable agents. In Proceed- ings of the Genetic and Evolutionary Computation Conference, pp. 414-424, 2020.
Bryon Tjanaka, Matthew C Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradients for differentiable quality diversity in reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 1102-1111, 2022.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 23-30, 2017.
Vassiis Vassiliades and Jean- SaantBapiiste Mouret. Discovering the elite hypervolume by leveraging interspecies correlation. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 149-156, 2018.
Vassilis Vassiliades, Konstantinos Chatzilygeroudis, and lean-Baptiste Mouret. Using centroidal voronoi tessellations scale up the multidimensional archive of phenotypic elites algorithm. IEEE Transactions on Evolutionary Computation, 22(4):623-633 2017.
Tom Zahavy, Andre Barreto, Daniel J Mankowitz, Shaobo Hou, Brendan O' D'onoggue, Iurii Kemaev, and Satinder Baveja Singh. Discovering set of policies for the worst case reward. In International Conference on Learning Representations, 2021.
Yunbo Zhang, Wenhao Yu, and Greg Turk. Learning novel policies for tasks. In International Conference Machine Learning, pp. 7483-7492, 2019.
Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via reward- switccinn policy optimization. In International Conference on Learning Representations, 2022.
12

---

Published as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
In this section we detail the implementation choices that have been made to compare the algorithms. All our implementations are based on the framework (Bradbury al. (2018) and are specifically designed to run an hardware accelerator, especially with the accelerator-compatible environments of BRAX (Freeman et al. 2021.
Table Identification of components shared across methods.
Algorithm
Genetic Mutation
Map-Elltes-baseed
PG update
SAC-based
TD3-based
SMERL(DIAYN)
(MERL(DADS)
X
X
REWARR
DIAYN+REWARD
X
X
V
PGA-MAP-ELITES
V
X
PGA-AU AORRA
X
MAP-ELITES
X
AURORA
X
X
X
Algorithm
Unsupervised divensity
SMERL
DADS diversity
DIAYN diversity
SMERL(DIAYN)
X
REWARD
X
V
X
X
PGA-MAP-ELITES
X
X
PGA-AURORA
X
X
X
MAP ELITES
X
X
X
X
A.1 ALGORITHMS UNDER STUDY
MAP-Elites Our implementation of MAP-ELITES takes advantage of the just-in-time compilation and vectorization capacity of JAX, similarly to the implementation presented in QDAX (Lim et al.) 2022). All environments are implemented in BRAX (itself constructed JAX), which enables to simulate multiple copies of the same environment in parallel, on the same device. Hence, we can simultaneously unroll thousands of episodes and evaluate as many policies. In addition, we put focus on implementing the mutation and insertion steps of MAP-ELITES to be compatible as well with the just-in-ting compilation paradigm of JAX. These design choices enable parallelization and lead an overall speedup of several orders of magnitude compared to previous sequential implementations. Finally, we opt for the Iso+LineDD variation /Vassiliades Mouret, 2018) for the updates of population.
PGA-MAP-Elites We provide here the first implementation of PGA-MAP-ELITES in JAX. This implementation benefits from the same vectorization strategies as MAP-ELITES which yields similar speedups. The major difference between these two algorithms lies in the policy variation step: rather than applying the Iso+LineDD mutation all the time, PGA-MAP-EL
uses a mix of the latter variation and policy gradient (PG) variation. As in Nilsson & Cully (221)) this PG variation consists in updating the selected policies to follow a TD3 critic trained with transitions from a shared replay buffer.
DIAYN Our implementation closely follows the original implementation of Eysenbach et al. (2019). For the latent skill variable, we rely on categorical prior with dictionary size [2] and we sample with probability p(z) [ For a given transition (8, 2) from the replay buffer, the reward is
computed online as follows:
log(g(z(s)) log(p(=))
(1)
The discriminator q(2(s) is parameterized neural network that outputs softmax distribution and is trained as a classifier to predict the skill & from the current state s. The discriminator is updated the same frequency as the policy and the critic. However, one distinction with the original
13

---

Published as a conference paper at ICLR 2023
implementation of DIAYN stems from the way we take advantage of the JAX's parallelization capacity: we sample multiple skills and simultaneously generate trajectories by unrolling the skill-conditioned policy on identical copies of the environment. This way, we collect data from different skills at the same time, and accelerate the training of the discriminator.
DIAYN+REWARD and SMERL(DAAYN) DIAYN+REWARD is supervised RL method that consists in incorporating signal from the environment into the DIAYN algorithm. At each timestep, sum the environment reward and the reward of DIAYN (multtipiee by scaling coefficient), Equation 3 Thus, we expect to discover diverse and high-performing skills. SMERL(DIAYN) rather formulates the problem in constrained manner: diversity should be maximized only when a policy c-close to the return of an optimal policy, é being a margin set as an hyperparameter. To implement this constraint, we maintain for each transition in the replay buffer the return of its associated episode. Only transitions from an e-optimal episode are updated to add the diversity reward.
DADS Our implementation of DADS builds on the same basis as the original implementation. The prior chosen for is the same as in DIAYN and the reward is defined as follows:
= (og(g(((+11e))))) 2)) log(p(s))
(2)
where p(s) is obtained by marginalizing over the (discrete) distribution of z. Following the work from Sharma et al. (219)) we define the skill-dynamics model as a neural network which outputs Gaussian Mixture distribution, with 4 experts and an identity covariance matrix. We also predict As = 8t41 St instead of 8441 and we exclude 8t from the input, resulting in a skill-dynamies network of the form (As]z). We maintain the running average and the standard deviation of the target As to normalize the output of the skill-dynamics network. Our implementation trains the skill-dynamics network offline, similarly to the training of DIAYN'S discriminator, with parameters updated at the same frequency as the policy and the critic. Finally we take advantage of the batching capability of JAX along the same lines as for DIAYN.
DADS+ and SMERL(ADSS) The summation method of DIAYN+REWARD (resp. DI- WNN+REWARD) can be easily extended with any MI method that provides diversity reward. It naturally leads to the method DADS+REWARD (resp. (DEDL(DADS)).
AURORA AURORA is QD algorithm introduced in Cully 2019). In this algorithm, the population is stored in an unstructured archive instead of a structured grid similarly to what is done in MAP-ELITES. The main difference of AURORA compared to usual QD methods is that the behavior descriptor is learned instead of being manually defined by the user. To do so, an auto-encoder learns to encode and reconstruct trajectories collected in the environment. The latent space defines the behavior space. In order to determine the behavior descriptor of a policy, its trajectory in the environment is given input to the encoder. The output of the encoder the behavior descriptor of the policy. The auto-encoder is trained following simple geometric schedule (with decreasing frequency). At each training step, the auto-encoder learns to reconstruct the trajectories of the policies that are stored in the archive. Then, the behavior descriptors are computed with the new weights of the encoder and the unstructured archive is updated accordingly. The genetic mechanism used to generate offspring in our study is the same as MAP-ELITES; use the Iso+LineDD variation.
PGA-AURORA PGRORU new algorithm that we introduce in this paper. It has com- ponents inspired by AURORA and -AA--MA-ELLTESS It uses the unsupervised behavior descriptor definition from AURORA, as well as its unstructured archive. But instead of using purely genetic mutations like AURORA, AURRAU uses the policy-gradient mutation introduced in PGA-MAP- ELITES NNilsson& & Cully, 202I). This baseline is interesting because it has exactly the same prior knowledge as the MI RL methods considered in this benchmark: it does not rely the definition of behavior descriptor but a prior can be injected. In our experiments, the population of the unstructured archive is the same for the other QD methods and the latent space size is set to 5. The auto-encoder uses LSTM units to encode and decode the given trajectories. The implementation of PGA- AURORA is also fully implemented in JAX.
14

---

Published as a conference paper at ICLR 2023
A.2 PASSIVE REPERTOIRE: MEASURING DIVERSITY FOR
POLICIES
Skill-discovery RL methods considered in this paper do not maintain a repertoire to store diverse solutions. Instead, policies are indexed by the possible values that the discrete latent variable can take for DIAYN, DADS, and SMERL. For these methods, a possible approach to introduce the same type of repertoire used by QD methods is to fill one in an ad-hoc fashion (i.e. post training) by enumerating all possible values for the latent variable and evaluating the policy conditioned on this value each time. However, this puts these methods a clear disadvantage if the cardinality of the latent space is small compared to the size of the repertoire given that most QD metrics are increasing functions of the number of non-empty cells. We could increase the size of the discrete latent space to match the size of the repertoire but skill-discovery RL methods become unstable when the size of the latent space is large (Eysenbach et al. 2019). Instead, we choose to fill the repertoire during training with all the intermediate policies found by skill-discoven RL methods, in the same fashion what is done internally in QD methods. Specifically, every 100,000 training step, we evaluate the fitness and behavior descriptors of all of the latent-conditioned policies and insert them in the repertoire whenever possible, following the same insertion rules as the behavior descriptor value is empty if the solution stored previously this cell has lower fitness. We stress that, in our experiments, the QD methods make use of the same behavior descriptors used to compute the QD metrics, which induces a bias in the analysis. To make the study less biased, we embed the choice of the behavior descriptor space into skill-discovery RL methods whenever possible. This is possible for OMNIDIRECTIONAL environments where we can choose a discriminator function for all skill skill-discovery methods that is conditioned on the behavior descriptor as the latter is just function of the current state. This however not possible for UNIDIRECTIONAL environments.
the cell corresponding to
A.3 DETAILS ON QD-SCORE COMPUTATION
As commonly done in the QD literature, we add an offset to the fitnesses when computing the QD score to guarantee that it is an increasing function of the coverage. The repertoires, which are initialized identically across methods, of size 1024 and are constructed using Centroidal Voronoi Tessellations (Vassiliades et al. 2017).
A.4 ENVIRONMENTS AND MODIFICATIONS
To define our environments, we use the BRAX physics simulator, which is provided in an open- source package fully implemented in JAX Freeman et al. (202T). BRAX builds environments from configurations that describe the elements to simulate. Specifically, the properties of the body parts of the agent, its joints, and its actuators, as well as the types of contacts that need to be taken into account during the simulations and a set of physic constants (including friction and gravity) need to be specified in the configuration.
Implementations for the base environments mentioned in section 5.1 (i.e. ANT, WALKER, and HALFCEETAII) are provided in the BRAX package. The tasks derived from these environments (i.e. UNIDIRECTIONAL ANT-OMNI, ANT-MAZE, ANT-TRAP) can be implemented with minor code changes. In UNIDIRECTIONAL, we pass along the feet/ground contact information from the simulator to the user to be able to compute the behavior descriptors. In ANT-OMNI, we do the same for the (x, y) position of the center of gravity of the robot. In ANT-TRAP. three walls are added in front of the ant create the trap and the (x, y) position of the center of gravity is made available. Similarly, we have created maze with simulated walls to generate the ANT-MAZE task.
In POINT-MAZE, actions are bounded 2-dimensional vectors corresponding to the (z, increments that can be added to the current position. The observation received at each timestep is the current point's ' position on the 2D plane. The reward signal is defined as the negative euclidean distance between the current position and the target position.
In ANT-TRAP, the reward signal is defined as the x-axis velocity minus the energy consumption. In ANT-MAZE, the reward signal is defined as the negative euclidean distance between the ant's center of gravity and the target position.
For adaption experiments. we took inspiration from both the QD and the RL literature. The experiment consisting in introducing leg damage has been studied in Cully et al. (2013) and and Kumar et al. (2020). Experiments that involve changing the gravity property have been used in Gaya al. (2021) A major
15

---

Published as a conference paper at ICLR 2023
difference in this work lies in the fact that we study significantly larger range of perturbations in each setting, which enables us to get a sense of the adaptation capabilities each method against both small and significant perturbations. The adaptation task for ANT-MAZE is also inspired from prior work in the QD literature (Pierrot et al. 2022).
In order create variations of the environment in adaptation tasks, apply wrappers modify a specific entry of the configuration. For instance, to change the gravity value, we (1) retrieve the configuration of the environment, (2) fetch the value of the gravity constant, (3) multiply it by given input and (4) set the new value in the configuration. Similarly, we change the behavior of the actuator of a specific joint by updating the "actuator "actuatoorstreenthh" entry in the BRAX configuration. In order to create the leg- rog-dysfunction adaptation task, we pick a multiplicative factor and multiply the "actuator_ "actuator_strength of all the joints that link bodies of the concerned leg by this factor. given action input will result in different update of the joint angle in the resulting physical simulation compared to the original simulation. completely ineffective. At the higher extreme of the range (i.e. 4.5), the joint swings much faster than in the nominal case for a given input command.
the lower extreme of the range (i.e. 0), this makes the actuator
Below we detail several elements to change in the BRAX configurations in order to replicate our adaptation tasks. We also provide the code to replicate those.
Leg dysfunction in WALKER- UNNI: the leg impacted by the dysfunction is the left leg of the WALKER. The impacted joints in BRAX configuration are called Befigh_left_jeee "leg_left_joing and "foot_left_joing
Leg dysfunction in ANT-UNI: the leg impacted by the dysfunction in the ant corresponds to the following joints in BRAX configuration: "$ Torso_Ane 4" and "Aux 4 Body 13".
Leg dysfunction HALFCHEETAH UNI: the leg impacted by the dysfunction is the front leg of the HALFCHEETAH. The impacted joints in the BRAX configuration are called "fthigh", "fshin" and "ffoot".
Gravity update in all the UNIDTRECTIONAL tasks: the gravity parameter of the environment is usually -9.8 by default. For each environment, we multiply by a range of 20 values, to get values ranging from -2.45 (low gravity) to -490 (high gravity).
Target position update in ANT-MAZE: we sampled 10 positions uniformly inside the maze (i.e. in |-3, 38)²). The values used in our final experiments are reported on Table
For the hierarchical learning task, we have added the (x, y)-sisttion of the robot to the observation because is needed by the meta-controller take relevant actions.
Run number
x position
position
Training
35.0
0.0
Adaptation 1
11.74
35.16
Adaptation 2
-233
14.86
Adaptation
32.48
32.37
Adaptation 4
29.52
17.46
Adaptation
12.46
7.05
Adaptation 6
17.36
4.45
Adaptation
8.38
25.17
Adaptation 8
24.02
34.36
Adaptation 9
21.75
14.12
Adaptation 10
19.51
14.58
Table New target positions in MAZE adaptation task. During the training phase, the position is set to (35, 0) but varies during adaptation. The 10 different values used during our adaptation experiments are reported.
For the SAARCHHETAHHHHUDDAAA we add obstacles in front the halfcheetah in the x-direction. To do so, we add capsule body to the Brax simulation configuration and include collisions between the
16

---

Published as a conference paper at ICLR 2023
robot's body and the capsules. Those hurdles placed every three meters, have length (y-axis) of 1 meter, a height (z-axis) of 0.25 meter and radius of 0.25 meter.
B HYPERPARAMETERS
In this section, we document the various hyperparameters that have been used in the study and the selection process that led us to pick these values.
Hyperparameter
Value
Policy learning rate
0.0003
Critic learning rate
0.0003
Environment batch size
200
Batch size
256
Discount factor
0.99
Entropy coefficient
0.1
Policy hidden layers size
256, 256
Critic hidden layers size
256, 256
DIAYN
Discriminator learning rate Diseriminator hidden layers size Number of discrete skills
0.0003
[256, 256] 5
DADS
Skill-dynamicss learning rate Skill-dynamics hidden layers size
0.0003 [256, 256] 5
Number of discrete skills
Table 4: Hyperparameters for DIAYN and DADS. Most parameters are inspired from the original papers that introduced the methods.
Skill-discovery MI RL methods. Throughout our evaluation, use the same hyperparameter values for shared parameters across all methods and environments. Table X details the choice of hyperparameters for DIAYN and DADS, which both use SAC internally. This set of hyperparameters values, derived from Osa et al. (2022) and Kumar et (2020), yields good performance across all environments for both methods. However, some environmennt-ependeen parameters require more care. In particular, the global reward multiplier of the environment is set to 1.0 for all environments except for HALFCHEETAH-UNI (resp. ANT-UNI), where it set to 5.0 (resp. 10.0), following observations reported in BRAX (Freeman et al.) 2021). One crucial parameter choice for DIAYN+REWARD and REDS+REWARD (and by extension (MERL((IAYY) and (DADI) is the diversity reward scale, 3, that affects the diversity term in the weighted sum:
environment +,9r diversity
We 3 = 2.0 it has proved favor the emergence of diverse and high-performing skills all environments our experiments, setting aside A ANT-MAZE, where we set , = 3.0 to adjust to the larger scale of the environment's rewards, and ANT-OMNI, where we set 4.0 to compensate for the fact that the environment's 's reward drives the agent to learn static strategies by penalizing the use of the motors. The same values of 3 were used for SMERL. To select the optimal SMERL return, similarly to Kumar et al. (2020). we evaluate a SAC agent on 5 seeds and select the median of the best-return policy the target. The margin-to-optiman return is set as 10% of the target return. For all DIAYN and DADS variants, we select discrete skill distribution of size |Z| 5. following choice made SMERL (Kumar et al. 2020). It often hard learn bigger numbers of distinct skills (Eysenbach et al. 2019).
Finally, the value for environment batch size was set to 200. To select it, we tried a range of values similarly to what was done in BRAX paper (Freeman al. 2202I). In the end, 200 was the highest value that sped up the training as much as possible while not perturbing the learning dynamics.
17

---

Published as a conference paper at ICLR 2023
Hyper-parameter
Value
Environment batch size
100
Policy learning rate
0.001
Critic learning rate
0.0003
Policy hidden layers size
[256, 256
Critic hidden layers size
[256, 256]
Policy noise
0.2
Noise clip
0.5
Discount
0.99
Reward scaling
1.0
Policy gradient proportion
50%
Critic training steps
300
Policy training steps
100
Iso sigma
0.005
Line sigma
0.05
Table 6: Hyperparameters for PGA-MAP-ELITES Most parameters values are standard in the literature on MAP-ELITES and TD3 for the tasks considered this paper. For fairness, we used the same architecture for all methods under study whenever possible. The number of policy gradient steps is increased compared to the original implementation as this has proved to improve performance.
QD methods For PGA-MAP-ELITES, we took the hyperparameter values used in the original implementation (Nilsson & Cully 2021 ). see Table [6] setting aside the number of PG steps applied to a policy when is updated by a policy gradient variation. Instead of the value 10 used in the original paper, we opted for a value 100 which yielded better performances in practice in our experiments.
Hyperparameter
Value
Environment batch size
1000
Policy hidden layers size Iso sigma
[256, 256]
0.005
Line sigma
0.05
Table 5: Hyperparameters for MAP-ELITES.
Note that pick different value for the vironment batch size for MAP-ELITES and PGA-MAP-ELITES. This hyperparameter corresponds to the number of policies that evolved and evaluated in parallel each step of the algorithm. For MAP-ELITES, empirical evidence from a previous study, namely Lim et al. (2022), suggests that large environment batch size dramatically speeds up the training phase without impacting the performance. Hence we chose to batch 1000 environments in parallel in order to leverage as much as possible the vectorization capabilities of JAX.
However, there is no such empirical evidence for PGA-MAP-ELITES Specifically, since some elements are shared across the policies trained in parallel (e.g. the replay buffer of the underlying TD3), scaling up the environment batch size may negatively impact the training. As matter of fact, multiplying by 10 the environment batch size for PGA-MAP-ELITES will not result in an 10-fold increase of the number of environment steps collected in the same amount time (which would be the reason, we decided to keep the exact same value of environment batch size in the original PGA-MAP-ELITES study (i.e. 100).
Hyperparameter
Value
Latent space size
5
Archive initial value
0.2
Table 7: AURORA-Specific hyperparameters for AURORA. The other hyperparameters are similar to MAP-ELITES.
for MAP-ELITES! For this
18

---

Published as a conference paper at ICLR 2023
C EVOLUTION OF QD METRICS DURING TRAINING
In this section, we report some additional results which shed light on the evolution of performance during training w.r.t. the QD metries described in Section [ In the main body of this paper, the final performances (QD score and maximum fitness) achieved by all methods after two hours of training are reported in Table while visualizations of the best final grids obtained for each method are also available in Figure 2 In this section, we plot the evolution of the QD metrics during training for all methods Figure G Additionally. Table & reports the number of environment steps carried out during training, averaged across the 8 evaluated environments, for all methods.
In our study, we chose to express the train- ing budget in terms of time instead of en- vironment steps. As hardware accelerators are getting more efficient, we believe that the intrinsic capacity of an algorithm to make the most of the available hardware is an important property to assess. Fur- thermore, with the development of highly accurate simulators, many real-world ap- plications can be simulated, which enables to break down the training phase in two steps: (1) training in simulation (where the number of environment interactions is ir- relevant, only the time budget is), and (2) sim-to-real training; rather than a single training phase in the real world.
Algorithm
Env. steps
(DIANN)
1.0 x 107
DIAYN+REWARD
1.0 X 107
(MEDL(DADS)
8.5 106
DADS+REWARD
8.5 x 106
PGA-MAP-ELITES
1.8 x 10
MAP-ELITES
1.6 10
AURORA
1.2 X 10
PGA-AURORA
1.1 x 10
Table 8: Number of training steps carried out (on av- erage) during training by the various methods under study.
Since sample-efficiency is critical for (2),
we also provide plots showing the evolution of the QD metries w.r.t. the number of enviromment interactions during training. We plot the evolution of QD metrics for all QD (resp. skill-discovery) methods as a function of the number of environment steps, with maximum horizon of 1.5 x 10 (resp. 10") steps, Figure (resp. Figure 8). Figure highlights the gain in sample efficiency yielded by the PG variation in PGA-MAP-ELAN compared to MAP-ELITES (although MAP-ELITES performs well time scale, Figure 6
We also provide the standard deviation of the maximum fitness and QD-score reported in Table i Table Ø
Table 9: Maximum fitness and QD score at the end of the training phase (median over 5 seeds). The standard deviation is reported.
ANT-IRAP
ANT-MAZE
POINI-MAZE
ANI-UNI
ANT-UMNI
WALKER-UNI
HALPCHEE
Maximum fitness
SMERL(DIAYN)
(.49 0.01)x2
-7.88 0.4
(_0.0 (12.57ll
2.49 (1..000e
1.19 (10006)e3
1.89 (_LO 14je3
SMERL(DADS)
4.48 (2:002212
8.10 (1:3.1%el
L61 (=503
2.49 (1:0000e2
L.17220.14)e)
1.92 (203
DADS-REWARD
(20,4144
7.81 (10.49%e3
8.10 (1.3200ll
1.24 (20 521e3
2.
6,30 (10.48)22
L89 (=0,000))
(+0.511)
(x:255iel
(=0.2111
2.29 (+0.04e2
(40. 13jc3
(=0.27it
PIA-MAP-BLIIES
(a2.22222
.932 (+40-3565
145
2,49
0=5001ee
AGA-AURORA
.927 (H0.06633
(12.3el
1.44 (:0
2 (1:000022
(40. 16j63
1.47 291e3
MAP-ELITES
188 (+0.11)e2
-10.1 (+0.05e4
-2.5 (+H.05eel
4.25 (-0.2
2.49 (+0.0002
6.87
700.270p2
1,23 (=006)63
AURORA
4.05 LL0.5222
L1000
1l0.
5.05 (10.26)x2
2.49 (1.0.00)2
5.71 (11.34)22
121 (100081e)
0.17jes
7.84
38.9 (143
6:20
1.38164
1.95
(+1.00e
3.01 (40 2015
7.92 23je5
SMERLÉBADS)
(38 0.56ie4
3.57 6.841e3
1.171e5
4.15
RADS-REWARD
2:43 0.80je5
9.12 (: 76je5
0:4 7,56)e4
2.88
1,7755
5,05 1.
3.89 41)e5
8.31 (=3.91h55
DIYNNGRWWRRD
(+ 0.621e5
9.18 (+ 1.1s
4.4
1.600
1.20
33)
4.00 30
4.92 (+4.35)e5
1.68 (-0 18166
POA-MAP-ELITES ELITES
7,89 (12,7188
2.74 (. 0 50)e6
3.92 (.12 0.033e
9.19
0.09
1.58 (1..011 17
8.23 (11042he5
2.98 (_
RGA-AURORA
4.40
1.59 (:0 15j66
2.88 (305
7.33 (:00 19i55
1.60 (100 47x55
5.10 (4)65
2.39 061e6
4.53 0.54)e5
295 (1ie6
4.22 0.01le5
9.08 (10 201e5
3,04 (10.74yes
6.45 (30,3
3,05 (=0.000
1..111
1.18
(_10.22e55
(100.79e5
(111.6)e5
(_0.64
19

---

Published as a conference paper at ICLR 2023
102
Max fitness
102
Coverage
105
QD score
1.00
PointMaze
-0.6
-1.2
0.50
0.25
-2.4
0.00
4000
1.25
AntTrap
1.00
0.50
15
0.25
164
166
3.2
-0.75
45
AntMaze
11.00
-1.25
-1.50
0.8
-1.75
103
166
1.00
16
Ant Uni
0.75
1.
0.50
40
0.25
04
0.00
Ant Omni
30
2.0
20
163
165
HalfCheetah Uni Walker2D Uni
80
20
103
166
2.0
100
3.2
2.4
100
25
0.8
2500
5000
2500
5000
5000
Time (s)
Time (s)
Time (s)
SMERL (DIAYN)
DADS+REWARD DIAYN REWARD
SMERL (DADS)
MAP-Elites
PGA-AURORA
Figure 6: Evolution of the maximum fitness, coverage and QD during the training phase (2 hours), for all methods under study.
20

---

Published as a conference paper at ICLR 2023
163
Max fitness
1e1
Coverage
165
QD score
1.0-
6.0
AntTrap
06
3.0
04
1.5
0.0
164
1e6
45
AntMaze
-1.2
30
15
-1.6
0.0
PointMaze
75
50
-1.5
25
-.0
163
165
80
Ant Uni
1.-
0.
20
187
165
Ant Omni
24
2.0
2.40
18
2.34
2.26
0.5
0.0
1e5
HalfCheetah Uni Walker2D Uni
1.25
80
1.00
60
80
0.50
20
163
100
16
75
50
08
25
0.8
04
0.0
0.5
100
1.5
0.0
0.5
1.0
1.5
0.0
0.5
1.0
1.5
Timesteps
1e8
Timesteps
1e8
Timesteps
1e8
PGA-ME
MAP-Elites
AURORA
PGA-AURORA
Figure 7: Evolution of the maximum fitness, coverage and QD score along environment interactions, during the training phase. Comparing QD algorithms MAP-ELITES, AURORA, EGA-MAP-ELITES and PGA-AURORA on 1.5 x 10 timesteps.
D ADDITIONAL VISUALIZATIONS OF THE TRAINING PHASE
In this section, we provide additional visualizations of the training phase. Figure 11 display nine intermediate repertoires obtained by MAP-ELITES, PGA-MAP-ELITE and DADS+REWARD various
21

---

Published as a conference paper at ICLR 2023
1n3
Max fitness
1el
Coverage
1e5
QD score
1.5
2.0
AntTrap
1.00
1.5
0.75
12
0.5
0.6
0.25
184
166
1.00
AntMaze
0.75
-1.00
-1.25
0.50
-1.50
3
0.25
-177
0.00
182
164
-0.5
PointMaze
7.5
5.0
-15
2.5
-2.5
0.3
103
165
3.2
Ant Uni
24
0.8
0.4
0.0
122
165
Ant Omni
2.4
30
2.
2.0
10
08
1.8
163
165
HalfCheetah Uni Walker2D Uni
1.2
0.9
45
50-
0.0
0.3
15
15
00
163
166
2.0
16
45
1.5
12
1.0
30
08
04
Timesteps
1e6
Timesteos
1e6
Timesteps
1e6
SMERL (DIAYN)
SMERL (DADS)
DADS+REWARD
Figure 8: Evolution of the maximum fitness, coverage and QD score along environment interactions, during the training phase. Comparing skill-discovery algorithms DIAYN+REWARD, SDERL( DADS+REWARD and SMERL(DADS) on 10" timesteps.
points during training, evenly distributed over the first hour of training. These visualizations shed some light on the mechanisms through which these repertoires are filled during training.
22

---

Published as a conference paper at ICLR 2023
For instance, observe that the use of policy gra- dient in MAA--MAP-LLLEEE focuses the search toward solutions that move alongside the x-axis direction (re- call that only x-axis movements are rewarded in the locomotion environments) whereas the first check- point for MAP-ELITES shows solutions that can go everywhere alongside the y-axis even though no so- lution has been able to hit the trap yet. MAP-ELITES struggles to further fill the repertoire (after 20M steps, the repertoire has almost reached its final state), whereas
continuously improves the repertoire coverage to finally reach (nearly) 100% after two hours of training, see Figure 2 Finally, note that DADS+REWARD rapidly reach high-returns behaviors and focus on improving them through the training. However, it neglects intermediate policies that are sub-optimal but diversity-carrying.
Figure 9: Visualization of final skills' trajec- tories (40 random seeds per skill) learned by REWSRD ANT-OMNI.
We provide further comparison in the way the reper- toire are filled by the different methods. Figure[12] 12a highlights the capability of MAP-ELITES to fill the whole descriptor space, but it struggles getting excel- lent policies whereas SMERL(DIAYN) lacks coverage and can only evolve a small population of elite poli- cies. On ANT-OMNI, the reward signal corresponds to the robot control cost, which hence gives incentives for minimal movement.
Nevertheless, we can see on Figure 12b how PGA- MAP-ELITES and DADS+REWARD eventually explore the environment. PGA-MAP-ELITES uses the diver- gent search of genetic mutations to gradually find exploratory behaviors and although it fails to go far from the origin, it is able to build a dense set of be- haviors. On the other hand, DADS+REWARD uses the diversity rewards to discover far region of the behavior space through directed skills. Both methods have efficient strategies to keep exploring even if the environment rewards discourage them do so.
Visualizations of the final skills' trajectories (40 random seeds per skill) learned by DIAYN+REWARD (left) and DADS+REWARD (right) on ANT-TRAP.
Figure 10:
While we have extended the concept of repertoires to skill-discovery methods in Section 5.2 to be able to compare all methods with the same metrics, an alternate way to assess the diversity achieved by the skill-discovery methods is to directly visualize the trajectories of the learned skills. On Figure T0 we show the trajectories of the skills learned by DIAYN+REWARE and REDSRD the end of the training phase. On this particular seed, note that REWARD has been able to learn skills that are significantly distinct from one another, with one them able to sidestep the trap and run far along the x-axis direction. Conversely, DADS+REWARD could hardly learn 5 distinct skills but managed to find two significantly distinct skills that go around the trap on opposite sides. shows
Nevertheless, Figure shows how RADS+REWARD is able to learn skills going in 5 perfectly distinct directions in ANT-OMNI.
E ADDITIONAL OBSERVATIONS FOR THE TRAINING PHASE
is worth mentioning that all skill-discovery RL methods achieve higher fitnesses than the QD methods on ANT-MAZE even though they fail to fully explore the behavior descriptor space, seen on Figure 2. Amusingly, watching the policies trained by these methods interact with the simulator reveals that they all have found way to jump above wall of the maze, thereby directly reaching the target position without having to solve the maze. Although the task was not designed to be solved this way, it is interesting
that some algorithms were able to find an original way solve the
23

---

Published as a conference paper at ICLR 2023
MAP-Elites
PGA ME
DADS REWARD
Figure 11: Intermediate repertoires of MAP-Elites PGA-MAP-Elites and DADS+REWARD AntTrap. Visualizations were generated at regular intervals from the start training (top) to halfway through the training phase (i.e. at the 1h mark) (bottom).
task. Running pure optimization RL algorithm with mechanism for diversity does not find this way to jump above the maze.
It is interesting to see that Skill Discovery RL methods get best performance on 4 tasks collectively, but only get one best fitness each. No method has a consistent best performance on the tasks.
For the UNIDIRECTIONAL tasks, we observe that the skill-discovery methods have been able to illuminate an important portion of the space even though they are not explicitly looking for diversity w.r.t. the behavior descriptors. However, note that they have not been able to find solutions at the edges of the repertoire, whose cells can only be occupied by solutions consisting in walking on only one feet or in asymmetric manner which we expect to be more robust perturbations.
24

---

Published as a conference paper at ICLR 2023
MAP-Elites
SMERL (DIAYN)
PGA-ME
DADS + RREAARD
(a) MAP-Elites and (MERL(DIAYN) on HalfChectah-Un
(b) PGA-MAP-EII and DADS+REWARD on Ant- mmmim
Figure 12: Intermediate repertoires. Visualizations were generated regular intervals from the start of training (top) to halfway through the training phase (i.e. at the 1h mark) (bottom).
Lastly, note that MAP-ELITES outperforms all other methods w.n.. the QD score for out of the tasks while achieving similar coverage than PGA-MAP-ELITES, which means that it has been able to find performrformin solutions. This is a surprising fact in itself because this method does not leverage back- propagaopan through the neural policies is expected to be less efficient than the
25

---

Published as a conference paper at ICLR 2023
other methods. This is testament to the efficiency of the vectorized JAX implementation that enables MAP-ELITES to iterate over large populations of policies very quickly.
F ADDITIONAL RESULTS FOR THE ADAPTATION EXPERIMENTS
In this section, we provide additional results for the environment adaptation experiments described in Section 5.33 Specifically, we plot the relative change in fitness, referred to as fitness gain, compared to the nominal scenario (where the environment unchanged) with the raw fitness reported on Figure in the main body of the paper. Observe on Figure [3 that the fitness gain is always 0 when the gravity (resp. actuator) coefficient is 1, this is by definition because the environment is unmodified in this case. We expect the fitness gain to be negative in most cases, because agents should perform worse environments they never trained can see that, in some cases (small gravity values or highly sensitive actuators), some methods find policies that can make the best of these changes and actually perform better than on the environment they were trained on.
Figure 13 This is to be compared
Nevertheless,
Figure 13 also provides insights into the sensitivity of the various methods to environmental changes. HAP-ELITES, while lagging behind in terms of raw performance compared to other methods, is particularly resilient in the face of environment perturbations. For instance, for the ANT-UNI task, the performance of MAP-ELITES drops by at most 20% on the full range of gravity values used for evaluation, whereas the fitnesses of most methods drop by at least 70% in the most extreme conditions on both ends of the spectrum.
Walker2D Uni
Ant Uni
HalfCheetah Uni
Leg dysfunction
320
Fitness gain 1%
Fitmass gain (%)
Fitness cain 1%)
60
Actuator confficient
45
Actuator Boeeie
Actustor coefficient
Gravity update
Fimees (%5
Fitresz gain 1S)
Pirees: gaing 190
-03
-200
-1.0
25
Lirevity coefficient
Gravity coefficrett
Gravity coeffictent
SMERL (DWAYN) SMERL (DADS)
REWAR
PGA-ME
AURORA PGA AURORA
REWARD
MAP Elites
Figure 13: Relative change in fitness, referred to fimess gain, on the adaptation tasks. Comparing the algorithms' performances on modified environments their own performances on the nominal environments offers another perspective on the resilience capabilities of the methods under study.
G COMPARING SUPERVISED SKILL-DISCOVERY METHODS
This paper uses four methods inspired from the skill-discovery literature: DIAYN+REWARD and DADS+REWARD sum their diversity terms with the extrinsic reward from the environment, whereas AMERL(DDAYE) and (DADS) optimize the diversity reward only if the skill is already close to be optimal w.r.t. the extrinsic reward. We can draw three conclusions about the relative merits of each approach from the results obtained in this paper.
First, during the training phase, we can see, on Table as well as on Figure that SMERL methods outperform humm"based methods most environments w.r.t. fitness, with the notable exception of ANT-TRAP where DADS+REWARD outperforms any other method by a significant margin. On the contrary,
that "sum"-based" based methods achieved higher scores than the SMERL methods on this benchmark. These results seem coherent with the fact that SMERL prioritizes optimizing for quality over optimizing for diversity. Nevertheless, this choice also impacts the
26

---

Published as a conference paper at ICLR 2023
exploration capabilities of SMERL methods which, in turn, affects their fitness performances some environments that require a good amount of exploration to be solved, such as ANT-TRAP. SMERL methods fail to find way around the trap in ANT-TRAP whereas the "sum"-baset" methods eventually find one. Even if this improved exploration capability does not systematically lead to improved fitness in all environments, it helps -based methods attain higher QD scores.
Second, in the adaptation experiments, SMERL methods outperform "sum"-based methods as long as the perturbations remain small but do not necessarily perform better in extreme scenarios. On ANT-UNI for instance, SMERL(DIAYN) outperforms DIAYN+REWARIS by a significant margin (more than 20% of maximum fitness, see Figure 4) when the actuator coefficient is perturbed by less than 25%, but for perturbations greater than 60%, DIAYN+REWARD eventually outperform (for smaller values) or draw (for bigger values) (DTAN, We can observe similar tendencies on the gravity update experiment.
Third, "sum"- based methods seem to suffer from higher variances than SMERL methods. The performance of DADS+REWARD the adaptation tasks. We also observed a significantly higher variance of DADS+REWARD and DIAYN+REWARD on several environments during the training phase, for all three metrics reported on Figure
ANT-UNI (see Figure 4 is example of this tendency on
Although the benchmark carried out in this paper confirms the benefits of SMERL (Kumar et al. 2020) for adaptation tasks, it also shows that there is room for further improvements for these methods, particularly in settings where the task or the environment is significantly perturbed after training.
H HYPERPARAMETERS SENSITIVITY
In this section, we detail the hyperparameeeer-snsitttty experiment presented in Section whose re- sults are shown Figure 3 Recall that this experiment motivated by the fact that DADS+ achieves impressive results on the ANT-OMNI task, covering a significant portion of the behavioral descriptor space, and outperforms all QD methods- let alone other skill-discovery methods, see 2 Although it is not entirely surprising that DADS-related methods perform better than DIAYN- inspired methods, we were surprised see such discrepancy in the results and we suspected that the results would be rather sensitive to the exact choice of hyperparameters, For this reason, as described in Section [..2. we decided to run a study on the hyperparameter sensitivy of DADS+REWARD ANT-OMNI, comparing it to MAP-ELITES, the second most performing method in terms of QD score. To do so, we selected, for both methods, hyperparameters that impact the updates of the policies and we ran a grid search on them to compare the distributions of QD scores. For DADS + REWWRRD, we selected the hyperparameters that impact the reward signal, as they are key to trade off diversity for fitness, and used three different values for each hyperparameter, within an acceptable range, based on the literature. We picked the diversity reward scaling value (referred to as / in Section B) in the set (0.1, 1,03. We picked the entropy temperature coefficient a in the set {0.1, 0.5. 1), it directly impacts the randomness of skills. For MAP-ELITES, the distribution of evolved policies is highly dependant on the exact definition of the genetic variations used subroutines. Hence, decided to run grid search on the iso (resp. line) sigma values of the Iso+LineDD variation (Vassiliades & Mouret 2018) used in MAP-ELITES. Iso sigma parameter were selected in the set (0.001, 0.01, 0.1} and line sigma were selected in the set {0.01, 0.1, 1.0]. For each algorithm, this yields 9 different possibilities. Each possibility is evaluated on 5 different seeds. The results are reported on Figure
on
In order to strengthen this study, we decided to run the same experiments on the three UNIDIREC- TIONAL tasks as well: ANT-UNI, WALKER-UME UNI and HALFCHEETAH-UNS We present the results on all the four environment on Figure 14
On HALFCHEFTAH-UN, we can see that MAP-ELITES being very consistent over the set of hyperparameters, with high median and a very low variance, where DADS+REWARD has a low median and a rather high variance. On the WALKER-UNI, both methods have a similar variance over the hyperparameters but MAP-ELITES is consistently performing better. On ANT-UNI, MAP-ELITES has a significantly higher variance but it's worse case still outperforms DADS+REWARD'S s best QD score on the task. Those results shows that MAP-ELITES is easy to off-the-shelf as it tends to have a smaller variance and as its worse case performance are usually decent. This makes this family algorithm very convenient for industrial use.
27

---

Published as a conference paper at ICLR 2023
Ant omni
Halfcheetah uni
Ant uni
Walker2d uni
1.00
0.75
0.50
0.25
0.00
MAPEEitteDDADS-SU MAP-Elites DADS-SUUM
MAP-Elites DADS-SUNNUUUU
MAP-Elites DADS-SUM
grid of hyperparameters ANT-OMNI, HALFCHEETAH-UN., ANT-UNI and WALKER-UND QD scores are normalized with the maximal values obtained each task. Each point represent one set of hyper-parameters. Main represent the 0.25 and 0.75 quantiles, the horizontal har represents the median and the light color hoxes represent the 0.125 and 0.875 quantiles.
Figure 14: QD obtained when averaging
I UNSUPERVISED BEHAVIOR DISCOVERY IN QUALITY DIVERSITY METHODS
The two Quality Diversity methods presented in the paper, MAP-ELITES and PGA-MAP-ELITES are using predefined behavior descriptor, which suppose some prior knowledge about the task. Skill-discovery RL methods often use prior too (for instance, the xy position of the center of gravity, which is what we did in this study as well), they can be applied without it if needed.
Nevertheless, recent works from the QD community introduce methods to learn the behavior descriptor in an unsupervised manner (Cully) 2019; Grillotii & Cully) 2022a). One of these methods, called AURORA, stores the trajectories of the policies of its population, and
Variational Autocncoder (VAE) (Kingma & Welling 2013) to encode and try to reconstruct the trajectory. The encoder is then used encode any policy's trajectory into the latent space of the VAE.
Several other recent works build on this approach and use other information to learn automatically the behavior descriptor from data. RUDA (Grillotti Cully 20226) builds AURORA but bias the behavior descriptor towards behaviors that are helpful in an adaptation tasks. A similar approach is explored in the multi-agent setting by Dixit et al. (2022) evolve population of agents where the diversity improves the collaboration in common task.
In order validate the interest of those methods, decided to implement AURORA and compare its performance with MAP-ELITES in several training and adaptation tasks, to see the potential loss of performance induced by this autonomous way learn the behavior descriptor. We do not claim any exhaustive study of this method and let this for future work, but want to mention their existence and give an idea of its performance on some of our tasks.
The authors of AURORA were able to provide us JAX implementation that worked on the ANT robots. To better assess the interest those methods, we chose a task were no prior was given: ANT-UNI. The data given to the VAE is made of trajectories in the environment, with the full observed state. The proportion of contact time is not given to the algorithm. To the best of our knowledge, AURORA had never been trained used for adaptation
UNIDIRECTIONAL task.
We decided to go further and use the unsupervised mechanism of AURORA to handle the diversity of the population of (GA-MAP-ELITES This variant, that refer to PGA-AURORA, is the first attempt to use gradient mechanism inside AURORA.
The results of AURORA and PGA-AURORA compared to all other methods is reported in Figure 4
Our results show that the unsupervised behavior discovery method introduced by AURORA resulted in very small performance loss compared to MAP-ELITES and that combining it with PGA-MEP-ELITES even show improved adaptation ability in the whole adaptation range on ANT-UNI with the gravity update and 80% of the adaptation range on the leg dysfunction task.
28

---

Published as a conference paper at ICLR 2023
J IMPACT OF THE PRIOR AND OF THE ENVIRONMENT REWARD SIGNAL ON SKILL DISCOVERY METHODS
Impact of the prior The impact of prior in skill discovery methods has already been studied in the literature, for instance in the paper that introduced DADS (Sharma et al. 2019). Nevertheless, we wanted to illustrate this effect on an example taken from one of the tasks of our study, ANT-OMNI.
Our setting is different from the setting used in Sharma et al. (019) because the algorithms are modified to take into account the reward of the environment. In this task, the reward corresponds to the opposite of the control cost.
To illustrate the effect of the prior, we took the skill discovery methods that performed the best on ANT-OMNI with the prior: DIAYN+REWARD We removed the prior, and trained the methods again on five seeds. Hence, instead of searching for diversity in the (a,, y)- space, skills are looking for diversity in the entire state space of the ANT robot.
Ant Omni
and
Coverage
2.
On Figure [5| we report the evolution of the coverage of all methods along time. On Fig- 17] we show the final repertoire and skills of the median obtained for DIAYN+REWARD (resp. DADS+REWARD) in terms of coverage.
1
2000
Time (5)
RFWARR (na priar! DIANN-REWARD (no prtor)
RFWAR (prior)
DIAYN REWARD (priar)
The evolution of coverage on Figure 15 shows that the final value typically divided by four for DADS+REWARD and decreased by 40% for DIAYN+REWARD. The final repertoire and skills, Figure 17 illustrate how consistently the methods are able to navigate in the environment with or
Effect of the prior on skill discovery methods in ANT-OMNI. Evolution of the coverage along time. The given prior is the (z, position of the ant's center of gravity.
Figure 15:
without the prior. The impact of the prior would be smaller in environments were the extrinsic reward can help to explore but it would still have significant impact.
Table 10: Maximum fitness and QD score at the end of the training phase (median over 5 seeds).
ANT-TRAP
ANT-MAZE
ANT-UNI
ANT-OMN
WALKER-UNI
HALFCHEETAH UNI
Maximum fitness
(MERL(DIAYN)
4.49 (10.01)22
-7.88
(=0,4))33
1.71 (1:0.99)e3
2.49 (1:0.00)22
1.19 (=000))3
1,89 (100.144 14)e3
SMERL(DADS)
(+00022c2
-7.62 (-0.3)c33
1.61 (+:0.1803
2.49 (+:0.00e2
14)
1.92 (#0.21
DADS-REWARD
1.28 (1:0.11e3
-7.11 (20.9))3
1.24 (2)332)ee3
2.38 (10.11122
(1:0.33e3
DIAYN+REWARD
9.42 (+:3433c2
(0.51ce)
1.34 (+:3221e33
2.29 (#0.04%c2
1.00 (3),1
2.13 (+:0.27633
DIAYN
2.46 (10009912
(2.0004
2.19
2.11 (1:024922
(20.20)e2
(10061)e2
DADS
2.46 (43.84)22
9-16
2.18
(+:072
2.11 (#0068)e2
3.50(=0.74)c2
4.04 (#:143)22
QD score
SMERI (DIAYN)
1.02 17)e5
4+0
6.20
(+-1.38)44
1.95 (+1,70)e3
3.01 (+0.20)e55
7.92 (4-2.23)e5
SMERL(DADS)
9.88 (1.0.66)e
8.22 (1.0.7)e55
0.83 (1.1.7)e5
4.15 (11.11)e3
2.69 (10.381e55
7.51 1.66
REDS-R
2.42 (++0800je5
(+0.766)
2.88 (+1107)es
3.05 (+1.22)e5
3.89 (-0041))55
8.31 (4-3.91)e5
DIAYN+REWARD
2.27 (1.0.2))55
9.18 (10016)e
1.29 (1.0.39)e5
4.00 (10.39944
4.92 (11 (1.5.5
1.68 (110.18te6
5.80 (+1 (47)44
9.35 (+2283)e4
7.46
(++ 29)
4.84 54-4.50)e4
1.02 (+0.20)e5
4,00 (+0.500)e5
DADS
1.37 (10..4)e5
6.99 (12.32255
1.13 (100.9995
3.50 (100.6)e55
2.67 (10.62)e5
7.97 (11.13)e5
Impact of the environment reward signal Throughout our study, we considered Skill Discovery methods extended to a supervised setting where they take into account the environment reward signal either by summation or with the SMERL method. We advocate that evaluating DADS and DIAYN without this additional supervision would be unfair, since other methods explicitly optimize for the environment's 's reward. To validate this point, we train DADS and DIAYN in a fully unsupervised setting and measure their performance. In Table I0 we report the maximum fitness and the QD-score obtained by these methods after 2-hours of training. Except on ANT-OMNI (where the environment reward is a penalty for moving), the best score is always attained by a supervised method, in terms of maximum fitness well QD-score.
29

---

Published as a conference paper at ICLR 2023
To further emphasize the importance of the environment reward signal, adaptation experiments with DADS and DIAYN and evaluate the robustness the learned policies on perturbed enviromments. On Figure I6 we report the adaptation results, and compare DADS and DIAYN their supervised counterparts DIAYN+REWARD. SMERI. (MERI.(DIANN), DADS+REWA how the performance of DIAYN and DADS plummet without the access to the environment reward signal. It confirms the intuition that without any environement reward signal or finetuning, DADS and DIAYN fail consistently learn perrfrrrf policies and adapt the face of perturbed environment's s dynamics.
and SMERI (DADS).. It striking
Walker2D Uni
Ant Uni
HalfCheetah Uni
Leg dysfunction
1500
1000
800
500
500
000
1.5
20
45
0.0
1.
2.0
45
15
20
45
Gravity update
1200
1520
800-
Fitreess
500
520
Grevity coefficient
Gravity conffcient
30
45
15
Gravity
3.0
45
SMEAL (DIAYN)
SMERL
DADS
DIAYN
Figure 16: Effect of the environment reward signal the algorithms DADS and DIAYN adaptation tasks. Results are the median and interquartile range over 5 random seeds.
(a) w/o prior
(b) w/ prior
(c) w/o prior
(d) w/ prior
(c) w/o prior
(f) w/ prior
w/o prior
(h) w/ prior
Figure 17: Repertoire and skills obtained by DADS+REWARD (left, (a-b-e-f)) and DIAYN+REWARD (right, (c-d-g-h)) in ANT-OMNI with or without the prior. Using prior narrows the space where the method is trying find diversity. When no prior is given, the entire state space is considered. In ANT-OMNI, the prior is the (z, y) position of the ant's center of gravity.
K ADDITIONAL STATISTICAL ANALYSIS OF THE RESULTS
In this section, we provide additional data concerning the results in Table I Figure and Figure
In Figure 18 we report the performance on various adaptation and hierarchical tasks, similarly to Figure 4 and Figure 5 but using IQM aggregation rather than median aggregation. We observe almost
30

---

Published as a conference paper at ICLR 2023
identical results for all benchmarks, setting aside PGA-AURORA (resp. DIAYN+REWARD) that was ranked third (resp. second) for the ANT-MAZE adaptation task and is now ranked second (resp. third).
include additional metries corresponding to mean, median and IQM of the maximum fitness and the QD score over the 5 seeds used in the training phase, whereas Table [ only includes median for readability purpose. Note that the best method for each environment is almost always the same irrespective of the exact statistic used.
In Table 11
Walker2D Uni
Ant Uni
HalfCheetah Uni
Leg dysfunction
1937
2000
162
200
1a
2.
Gravity update
199-
1565
1500-
g10ce
302
cooffiiintt
Craviyy
S4ERL (C1A/))
- SVEAL (DAES)
REVARD
PGA-ME
NAP-llmm
AURORA
FGA ALBOR
AntMaze
Halfchestah Hurdles
PGA-ME
PGA--URORA
-DIAYN + REWARD
Fitness
-20
-MAP.Eiros
-80
-100
-SMERL (DIAYN)
-SMERL (DADS)
Num Steas
-AURORA
SMERL (DVAYN)
PGA-ME
(DADS)
0.0
0.5
1.0
DADS-REWARD
AURORA
Max fitness
DIAAR
PGA-AURORA
Figure 18: Maximum fitness of the methods when facing perturbed environment (top) modified task (bottom-left) and fitness of the PPO meta-controller for various choices of skill-discovery methods used to generate primitives (bottom-right). IQM aggregation used instead of median aggregation.
Table 11: Final Max fitness and score (training phase). Mean median IQM (5 seeds).
ANE-TRAP
ANT MAZE
POINT MAZE
ANT-UNI
ANT-OMNI
WALKER-US UNI
KALFCHESTAR UNI
Maximum fitnce
SMERI (DIAYN)
4,4044 400444404440-0 c0
- (41-- 88L7 03
1.391108-55 30
1,50 1,711.999 .9
2.492.4992449 2.49 c2
1.181, 1911 18
1871 27118901
SMIRI (DADS)
4.4814
43
-1.66--762.7.666
09.88889961155 cl
1.591 1.59166111 c3
14912.992.40
1.101 1711
1991 0011 06
DADS-RIWARD
1.17112811.30
-7.65-781177
3.788809 el
1131 2411
16 35
6.12K 2065
1
DIYYN-RDWARD
7.929-221811
3.521-4488-560
1.4613440 44.1..01.00 43
2,2542.200
9.661 0000 000
2.622122..05
KGA-IMAP-ELITTS
8 8099 770 28
< 15-9 9269 24
-1.661-226--2255
466 A5I
1.492.99..0 c2
741177511
PSA-AURORA
7064 7944
-4.84-6.111
c0
191 1411
44
2.492.89
0.5901 091
1 4701
MAP-ILLTTSS
3.778888887777
-1 00---01--10440
2247---48---447 47
4.3542254.27
24492.4924499
6.4K 686
1,2711 2011
AURORA
3.964.0944055
4.119-1.10-1100
2.300-2-35-22
3l
5.365005522.
2.492.499.49
5 7115
1,220.2110.10 19
aror
SMERLODIANO
LOGIL
7.6947.477.71.0
4.703.894.22d
6.776.196.66 000.66666400
2.521.952237
3.078.011.06 e5
7.577,928.11
(DADS)
LOU00991
5.184.577.15
1258.320.50
3.594.193.78
2.782.692.0 ct
7.00.510..8
DADS-EIWARR
2.340 422.
2.518.228..0 228.
4,244 191
2.552 980.890.
DIAYNA REWARD
2.512 271
9,229
3.7000043.5
1 2011
5,334
4.7234.9244.2
1.771
PGA-A CA--MEA-ELITES
5.2377 13337.897.577 25
2.731
1.933923.95
9.199,198.18
1.891.881.77
8.888228.11 c5
23902 98112.9.77....
POA AUB-OA
4.414 20LL-8 48
2.632.592.50
2 98922992.755
7.3-7337.77
1.60116991000
5,005 103.07 c5
2,402
MAP-IELTTIS
4.444 5344
2.981.0
4.234 224.2222222
9.949.0994 06..99006-4
6.58
3.0553.0613.00 3.8813.065.
AURORA
3,4933 $41.76 75..7664
1,8912 (N1
1.1344554
6. 155.82%..0 c5
A27140034445
4.4614
2,450,4512
31

---

Published as a conference paper at ICLR 2023
L ALGORITHMS PSEUDOCODES
We give the pseudocodes for all the algorithms used our study. The main difference compared to the pseudocodes found the original papers is that the termination of the algorithm governed time condition. As a reminder, details of our implementations can also be found in Appendix A.
Algorithm 1: MAP-Elites
Given:
Sample size N, time allocation max_time
MAP-ELITES repertoire M
initial policies (((,N)
// Main loop
current_time
while current_ting -itnntttttteeee-eeeee-t-eee max_maiiinn time
if current time > then
// Sampling ing and mutation Sample N policies h-m.N from the repertoire M
Update the parameters (0:) the policies with genetic updates
end
// Eval Evaauaaton and insertion in repertcire Evaluate fitness hehavior descriptor the new policies (Tg b=gNN
Update current time Add the updated policies in the repertoire when relevant
end
Algorithm 2: PGA-MAP-Elites
Given:
Sample size N. time allocation max time
MAP-ELITES repertoire M. Replay Buffer B
N initial policies (1,.0) and critic Q.
// Main loop
current time +-
while current time max time do
if current_time then
// Saamp ing and mutation Sample policies )==,), in repertoire M Sample batches of transitions in replay buffer B Update half the policies using the critic Q.
Update the other half with genetic updates
// Train the critic according to TD3 rule Update the critic Qu
end
// Evaluatio i on and insertion in repertcire Evaluate the new policies and store collected transitions in buffer B Update current time Add the new policies the repertoire MI when relevant;
end
32

---

Published as a conference paper at ICLR 2023
Algorithm 3: AURORA
Given:
Sample size N, time allocation max_time time
Behavior extraction model &e. training schedule train times
MAP-ELITES repertoire M, N initial policies (#TO (1,N1
// Main loop current time t- 0
while current time
time do if current_time then
// Samp1 ng and mutation Sample N policies (=,,)_11,, from the repertoire M Update the parameters (0: (0,) of the policies with genetic updates
// Eva ivrluatiion and insertion in repertcire Evaluate fitness and behavior descriptor of the new policies [Tū J=1,N
Undate current_ting time Add the updated policies in the repertoire M when relevant
// Training the behavior extraction model current time train times then
Update &a to reconstruct trajectories of policies in M Re-evaluate fitness and behavior descriptor of the stored policies Update repertoire M. with new behavior descriptors
end end
Algorithm 4: PGA-AURORA
Given:
Sample size N, time allocation max time
Behavior extraction model &e. training schedule train times
MAP-ELITES repertoire M, N initial policies (me, )=
Replay Buffer B and critic Q+
// Main loop
current
while current_ting time < max time do
if current_time > then
// [Sampl ing and mutation
Sample N policies {т0, J==1, repertoire MI Sample batches transitions in replay buffer Update half the policies using the critic Qu
Update the other half with genetic updates
// Train Update the critic Q+
critic according to TD3 rule
end
// Evaluation and insertion
repertcire
Evaluate the new policies and store collected transitions in
Update current time
Add the updated policies in the repertoire M when relevant;
// Training the behavior extraction medel current_ time train times then
Update &o to reconstruct trajectories of policies in M
Re-evaluate fitness and behavior descriptor of the stored policies
Update repertoire Mf with new hehavior descriptors
end end
33

---

Published as a conference paper at ICLR 2023
Algorithm DewaN+
Given:
Time allocation max_ san_ttme, steps per episode mrisodelengnn
Discriminator model qo. Replay Buffer B3
Diversity reward scale
Skill-conditinned policy my and critic Qu, skill prior distribution p(z);
// Main 1oop
current time
while gurrent_time max_ jan__imeede do Sample skill ~ p(2) and initial state 80 for + l...episode__lnntth do
// Data collection in the environment Sample action according current skill
Take a step in environment
// Compute new rewa rds
Sample batch of transitions from buffer B Compute the diversity rewards with the discriminator Tdiversity = log (=(22804+11 logp(=)
Update reward by combining extrinsic diversity rewards: it Ta B Vaiverity
// Update iiscriminator, policy and critic
Update policy parameters 0 to maximize collected rewards
Update discriminator parameters á maximize skill/state likelihood Update critic parameters to estimate state/action value
end end
Algorithm 6: SMERL(DIAYN)
Given:
Time allocation max time, steps per episode episode length
Discriminator model qo, Replay Buffer B
Diversity reward scale 3, reward target and threshold
Bkill-conditione policy and critic Q., skill prior distribution p(2);
// Main loop
current_time + 0
while current_timg tirrentttttmeeee max_maaamiiaaa s___timmeeee do Sample skill ~ p(2) and initial state 80 for 4- L...episode_length do
// Data collection in the environment Sample action according to current skill
Take step in the environment
// Computec new rewards
Sample batch of transitions from buffer
Compute the diversity rewards with the discriminator Tdiversiyy log (2( log p(=)
Compute of extrinsic rewards during the episode associated each transition Σ, Pe
Compute threshold condition IK R> R
Update reward by combining extrinsic and diversity rewards: ISmeri. = F1 + Fdiversity
// "Jddate inssrimintoor, policy and critic
Update policy parameters maximize collected rewards
Update discriminator parameters ø to maximize skill/state likelihood
Update critic parameters to estimate statc/action value
end
end
34

---

Published as a conference paper at ICLR 2023
Algorithm DADS+Reward
Given:
Time allocation max_time, steps per episode mrisod_lengre
Skill-dynamic model qa. Replay Buffer B
Diversity reward scale
Skill-conditinned policy my and critic Qu, skill prior distribution p(z);
// Main 1oop
current time
while purrent_time max_ jan__imeede do Sample skill ~ p(2) and initial state 80 for + l...episode__lnntth do
// Data collection in the environment Sample action according current skill
Take a step in environment
// Compute new rewa rds
Sample batch of transitions from buffer B Compute the diversity rewards with the skill-dynamies raivesityy log(g(sc+11))) 18,, 2)) log(p(s))
Update reward by combining extrinsic diversity rewards: I' Fa Vaivenity
// Update inseriminator, policy and critic
Update policy parameters 0 to maximize collected rewards
Update skill-dynamics model parameters to maximize dynamics prediction accuracy Update critic parameters to estimate state/action value
end end
Algorithm 8: SMERL(DADS)
Given:
Time allocation max time, steps per episode episode length
Skill-dynamics model
Diversity reward scale 3, reward target and threshold
ckill-conditioned policy and critic Qu, skill prior distribution p(2);
// Main loop
current_time + 0
while current_timg sevrentttmmmeee max_maaamiiaia s___timmeeee do Sample skill ~ p(2) and initial state 80 for 4- L...episode_length do
// Data collection in the environment Sample action according to current skill
Take step in the environment
// Computec new rewards
Sample batch of transitions from buffer
Compute the diversity rewards with the skill-dynamics Favesity log(g(s++1 (sc, 2)) - log(p(s))
Compute of extrinsic rewards during the episode associated each transition Σ. re
Compute threshold condition IK R> R
Update reward by combining extrinsic and diversity rewards: ISMEri. = F1 + Fdiversity
// "Jddate discrin iiscriminaoor, policy and critic
Update policy parameters maximize collected rewards Update skill-dynamics model parameters to maximize dynamics prediction accuracy Update critic parameters to estimate statc/action value
end end
35


Reviews:
[{"review_id": "ZANbP3kl5h2", "reviewer": "Reviewer_yJjm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper compares RL methods with information theory inspired diversity rewards and quality diversity methods for skill discovery on a large set of environments implemented in jax/brax (and thus able to take advantage of accelerators). It compares methods in terms of diversity metrics, performance on adaptation tasks and usefulness for hierarchical planning and finds that QD methods are competitive and sometimes perform better and potentially less sensitive to hyperparameters. ", "summary_of_the_review": "Nice paper presenting an empirical comparison between RL-based skill discovery methods and QD methods. I think the paper can be improved in some ways but presents interesting empirical results. ", "strength_and_weaknesses": "Strengths:\n* extensive careful empirical evaluation with interesting results\n\nWeaknesses:\n* comparing with the same time budget introduces dependencies on hardware, and implementation quality that are difficult to assess. I would at least report roughly how many environment interactions each methods processes to give readers an indication\n- Related to the first point I’m unclear why RL methods are less able to leverage accelerators (i.e. more env interactions). Perhaps the authors could provide their intuitions on this point. Is this because recent methods have been developed/tuned in a different regime?\n- Unclear if any of the methods in this paper could scale to something that seems less toy (e.g. dexterous manipulation with a robotic hand or humanoid locomotion)\n- I appreciate that the paper is has a lot results but having a limited understanding of QD methods I might have benefitted from more details.\n- Consider citing Hausman et al. (https://openreview.net/forum?id=rk07ZXZRb) which is pretty similar to DIAYN + reward (and predates DIAYN).\n- Figures are all a bit small and hard to read. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "TZ75ltaBGZ", "reviewer": "Reviewer_hXTi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes and integrates the usage of Quality-Diversity methods for skill discovery in RL. While prior methods largely rely on maximum-information (MI) RL, this paper makes a case for QD as an alternative approach when used with fast environment simulators. The authors provide a large number of experiments which show that QD-based methods are competitive with the MI RL approach and should be researched in more dept as future work.", "summary_of_the_review": "Interesting work that combines QD and skill discovery in RL. Extensive experiments and thorough analysis of prior work are included in the manuscript. I have raised several questions and some concerns that I’d like to be answered by the authors during the rebuttal. ", "strength_and_weaknesses": "## Strengths\n\n- The paper takes on an exciting direction, using QD for skill discovery. While the MI-RL line of work has shown great results in recent years, QD methods can provide an alternative, given the diversity element that is being explicitly optimized for.\n- I like how the paper positions itself neutrally to both types of approaches, namely QD and MI-RL. After presenting all the experimental results, it comes to the conclusion that no method is optimal for all environments and lays down avenues for future work. It also accepts the pros and cons of both approaches, and analyses all methods without any biases, as far as I could tell.\n- The paper includes extensive empirical evaluations and analysis, including experiments directly evaluating diversity, few-shot adaption and hierarchical learning. The list of baselines used in the experiments is fair, as far as I’m concerned.\n\n## Weaknesses\n\n- One weakness of the work is that it always provides the median of the metric of interest over the seeds (e.g. Table 1, Figure 4). This can provide unfair results if there are outliers. I suggest that the authors provide several metrics of interest, such as mean, IQM, etc. [rliable]([https://github.com/google-research/rliable](https://github.com/google-research/rliable)) is a great tool for this.\n- I didn’t quite understand why haven’t the authors include rewardless DIAYN and DADS results in their experiments. Throughout Section 5, they only use DIAYN/DADS+REWARDS as baselines.\n- QD-based methods are obviously more expensive computationally, given they store separate weights for each policy/skill. The authors argue that this is okay if the simulator is very fast. I have no arguments against it, but it would be great to know how much more compute-heavy QD-based methods are. The authors only mention that everything is within 2 hours of training, but there can be a big difference within this range.\n- Minor: please use the correct citation styles of prior work. See the ICLR instruction template for information on where to use \\citep and \\citet.\n\nNote: My score is not final and can change (in either direction) based on the authors' responses or comments of other reviewers.\n\n# Post rebuttal update\n\nI have increased my score based on the responses from the authors.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5dzX4bDnue", "reviewer": "Reviewer_rzaa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a comparison and benchmark of Evolutionary strategies and rl based skill discovery approaches. The authors provide a thorough literature review on the most recent approach in both lines of research and compare them in light of different metrics, how they generalise to changing environments (adaptive gravity, motor failure and different target locations), and how they can be use in hierarchical planning.", "summary_of_the_review": "I recommend this paper to be accepted since it provides a comparison of approaches in MI RL and quality diversity approaches for skill discovery. They empirically analyse how different methods on each side fare in terms of metrics for diversity and in terms of few shot generalisation.", "strength_and_weaknesses": "Strengths:This paper provides a comprehensive and important benchmark connecting RL based skill discovery with Quality Diversity approaches. It provides three settings for comparison using metrics, using few shot generalisation on adaptive environments, and using primitives for hierarchical planning. \nWeaknesses:\nThe metrics provided in Table 1 and figure 2 seem to be biased towards QD methods since RL based on MI does not naturally have or optimise for QD score, can the authors provide additional metrics that are perhaps less bias?\nIn the hierarchical learning experiments can the authors clarify the number of seeds used? And are there other environments that can be used for this setting?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0iOHDO2oEx", "reviewer": "Reviewer_UYjV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "At a high level, this paper is an empirical study of a collection of RL algorithms over a collection of environments.\nThe algorithms are reimplemented versions of:\n* Multiple Quality Diversity (QD) methods, which are evolutionary and do not use exact backpropagation.\n* Information Theory-Augmented RL methods, which involve adding additional mutual information terms + latent vectors inside a gradient-based RL pipeline.\n\nThe environments consist of:\n* Unidirectional (Single task) \n* Omnidirectional (Multi-task)\n* Adaptation\n* Additional Exploration (Trap/Maze/etc)\n\nall of which are based on Continuous Control. \n\n", "summary_of_the_review": "I currently lean reject, because of the lack of a compelling purpose or analysis. Currently as written, from a devil's advocate point of view, the paper can easily appear to be a collection of somewhat random algorithms applied on a collection of random benchmarks - the paper needs to have stronger analysis and focus on a compelling purpose for providing these experimental results.\n\nEDIT: I have raised my score to a 5 in light of the authors' rebuttals. I think the writing currently is still not convincing enough to a non-expert such as myself, and I urge the authors to improve on this front.", "strength_and_weaknesses": "# Strengths\n* Comprehensive algorithm list and benchmarking. While my knowledge on the diversity methods is limited, it seems that the provided open source codebase could be a useful tool for future research, due to its extensiveness. From a glance at the Appendix, there also seem to be many ablation studies.\n* The overarching goal (i.e. analyze different quality diversity methods and how useful they are) is somewhat solid. The main issue is with its execution, which I provide below.\n\n# Weaknesses\n* Lack of a **main punchline**: The current paper acts as an empirical study across previously invented algorithms, rather than inventing an method of its own. This is completely fine, but such a paper's primary focus will then be to provide deep experimental conclusions unknown previously. Currently, the experimental conclusions are not very insightful. A few examples: \n    * Section 6: \"Our benchmarks however show that no single method outperforms all others, which opens the door to promising work directions\" is too vague and can be said of any method in any field.\n    * Section 5.2: \"We observe that the performance of DADS+REWARD is very sensitive to the exact choice of hyperparameters\" - Many RL algorithms are sensitive to hyperparameters; why is this an interesting observation?\n    * Section 3: Are any of the methods mentioned here similar or different at a deeper, mathematical level? Currently all of the methods described use mostly words, which makes it hard to determine what components contribute to which behaviors (e.g. without needing to check references, how is MAP-ELITES different from PGA-MAP-ELITES?). It would be far more effective if the paper discussed the mathematical components which are common to certain algorithms, and show experimentally if these components contribute to certain behaviors.\n\n* Factual/Empirical Inaccuracies\n    * Section 5.1 \"These [standard locomotion] tasks are challenging for evolutionary algorithms that do not leverage backpropagation through the neural policies\" - The word \"challenging\" is too harsh here. ARS [1] has been consistently shown to beat PPO in standard locomotion tasks for a variety of reasons, such as the use of deterministic, linear policies.\n    * In Table 4 of the Appendix, I see that a MLP of layer sizes (256, 256) was used. Was this used for all evolutionary methods? The issue is that the performance of evolutionary methods are greatly affected by the problem dimension (i.e. parameter count in this case). It's already been shown that linear policies suffice and are actually sometimes the best [1] for many continuous control tasks. Since the difference in parameter count between a linear policy (order of 100's) and a MLP (20K+) is huge, this can be a very big confounding factor.\n\n\n[1] Simple random search provides a competitive approach to reinforcement learning (https://arxiv.org/abs/1803.07055)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%