Efficient Exploration using Model-Based Quality-Diversity with Gradients
Authors: Bryan Lim
Manon Flageat
Antoine Cully
Venue: Submitted to ICLR 2023 (2023)
Keywords: Quality-Diversity
Exploration
Reinforcement Learning

Abstract: Exploration is a key challenge in Reinforcement Learning, especially in long-horizon, deceptive and sparse-reward environments. For such applications, population-based approaches have proven effective. Methods such as Quality-Diversity deals with this by encouraging novel solutions and producing a diversity of behaviours. However, these methods are driven by either undirected sampling (i.e. mutations) or use approximated gradients (i.e. Evolution Strategies) in the parameter space, which makes them highly sample-inefficient. In this paper, we propose a model-based Quality-Diversity approach, relying on gradients and learning in imagination. Our approach optimizes all members of a population simultaneously to maintain both performance and diversity efficiently by leveraging the effectiveness of QD algorithms as good data generators to train deep models. We demonstrate that it maintains the divergent search capabilities of population-based approaches while significantly improving their sample efficiency (5 times faster) and quality of solutions (2 times more performant).

Under review as a conference paper at ICLR 2023
EFFICIENT EXPLORATION USING MODEL-BASED QUALITY-DIVERSITY WITH GRADIENTS
Anonymous authors
Paper under double-blind review
ABSTRACT
Exploration key challenge in Reinforcement Learning, especially in long- horizon, deceptive and sparse-reward environments. For such applications, population- approaches have proven effective. Methods such as Quality- Diversity deals with this by encouraging novel solutions and producing diver- sity of behaviours. However, these methods are driven by either undirected sam- pling (i.e. mutations) use approximated gradients (i.c. Evolution Strategies) in the parameter space, which makes them highly sample-inefficinnt. In this paper, we propose a model-based Quality-Diversiin approach. It extends existing QD methods to use gradients for efficient exploitation and leverage perturbations imagination for efficient exploration. Our approach optimizes all members of population simultaneously to maintain both performance and diversity efficiently by leveraging the effectiveness of QD algorithms good data generators to train deep models. We demonstrate that it maintains the divergent search capabilities of population-base based approaches on tasks with deceptive rewards while significantly improving their sample efficiency and quality of solutions.
INTRODUCTION
Reinforcement Learning (RL) has demonstrated tremendous abilities to learn challenging tasks across a range of applications (Mnih et al.) 2013) Silver et al. 2016 Akkaaa et al.) 2019). However, they generally struggle with exploration as the agent can only gather data by interacting with the environment. On the other hand, population based learning methods have shown to be very effective approaches (Jaderberg et al. 2017; Vinyals et al. 2019: Ecoffet et al.) 202I; Wang al. 2020). In contrast to single agent learning, training population of agents allow diverse behaviors and data to be collected. This results in exploration that can better handle sparse and deceptive rewards Ecoffet et al.
(022)) as well as alleviate catastrophic forgetting (Conti et al. 2018).
An effective way to use the population of agents for exploration are novelty search methods (Lehman & Stanley 2011a| la Conti et al. 2018) where the novelty of the behaviors of new agents is measured with respect to the population. This novelty measure is then used place of the conventional task
reward similar to curiosity and intrinsic motivation approaches (Oudeyer al. 2007| Bellemare
ct al.] 2016; Pathak et al. 2017). Quality-Diversity (QD) Pugh et al. (016): Cully et al. (2015); Chatzilygeroudis et al. (202)) extends this but also optimizes all members of the population on the task reward while maintaining the diversity through novelty. Beyond exploration, the creativity involved in finding various ways to solve a problem/task (i.c. the QD problem) an interesting aspect of general intelligence that is also associated with adaptability. For instance, discovering diverse walking gaits can enable rapid adaptation to damage (Cully et al. 2015).
However, a drawback of conventional population based approaches is the large amounts of samples and evaluations required, usually the order of millions. Some methods that utilize Evolutionary Strategies (ES) and more recently MAP-Eliter Mouret & Clunc (2015) (a common QD algorithm). sidestep this issue as they can parallelize and scale better with compute (Salimans al. 2017 Conti et al. 2018 Lim et al.) 2022p than their Deep RL counterparts, resulting in faster wall-clock times. Despite this, they still come at cost of many samples. One of the main reasons for this lies in the underlying optimization operators. QD methods generally rely undirected search methods such as aobjectiveagnnssee favor creativity and exploration. More directed search such as ES has also been used (Colas et al.) 2020) but relies on a large number of such perturbations (rthhusands) to approximate a single step of natural gradient to direct the improvement of solutions.
random perturbations (Mouret & Clune 2015 Vassiliades & Mouret 2018) to

---

Under review as a conference paper at ICLR 2023
Addition
Reward f
Descriptor
Perturbation
Population
Dynamics Model
Learning Behaviors in Imagination
Critic
Add tion
Population of Policies e
The GDA-QD algorithm can be summarized follows: (1) the current population θ copied in é, (2) Ô is used to perform multiple steps of QD optimization fully in imagination using the dynamics model, (3) the critic update sampled policies learned in imagination with policy- gradient updates; they are concatenated with (4) policies sampled from the resulting population learned in imagination Ô, (5) these concatenated batch of policies are evaluated in the environment and used to update the real population of policies for the next optimization loop; the transitions collected in the environment are then used to train the dynamics model and the critic.
Figure 1:
In this paper, we introduce an extended version of Dynamics-Aware QD IDA-QD-sse as well as Gradient and Dynamics Aware QD (GDA-QD), new model-based QD method to perform sample- efficient exploration in RL. GDA-QD optimizes an entire population diverse policies through QD process in imagination using learned dynamics model. Additionally, GDA-QD augments the conventional QD optimization operators with policy gradient updates using critic network to obtain more performant population. Beyond the effective exploration capabilities of QD methods, they are also excellent data generators. We leverage this idea to harvest a diversity of transitions to train the dynamics model and the critic. Thus, GDA-QD combine the powerful function-approximation capabilities of deep neural networks with the directed-search abilities of gradient-based learning and the creativity of population- -basd approuches. We demonstrate that successfully outperforms both Deep RI. and QD baselines in hard-exploration task. GDA-QD exceeds the performance of baseline QD algorithms by ~ 1.5 times, and can reach the same results in times less samples.
PRELIMINARIES
2.1 REINFORCEMENT LEARNING
Reinforcement Learning (RL) is commonly formalised Markov Decision Process (MDP) (ut-- ton & Barto 2018) represented by the tuple (S,A, R), where S and A are the set states and actions. P[ St, ar) the probability of transition from state 8; St+11 given an action ar. where St. 8t41 E S and E A. The reward function defines the reward obtained at each timestep r; r(se de, S141) when transitioning from state 8, to 8141 under action 0₃- An agent acting in the environment selects its next action based on the current state st by following a policy To s The conventional objective in RL is then to optimize the parameters of policy ne. such that maximizes the expected cumulative reward R(() = Σt= Te over the entire episode trajectory T:
J(TO) ENNno [R(r)]
2

---

Under review as a conference paper at ICLR 2023
Algorithm 1: DA-QD-ext and GDA-QD (highlighted lines are specific to GDA-QD)
Inputs: num iterations, N num imagined iterations, Peradient prop. of gradient-updatee
policies, and Pmodel
Peradient (for DA-QD DA-QD-ext Poradient 0 and Pmodel
Initialisation: On init init.populationn.gg do imit.dynamiis.modee Q2 init.critic()
for j = -,..., do θ c x 0/// // copy , in 6
// Optimiae ze populat in in maginationn
for imaginnatonn
_,., N do
0 random (undom_selection(())
i + perturb(0)
F F(0), d(0) evaluate evvaluateimmaginationn....
6, - update-popplle powpeepioollaiiiipiiia upd@ttepooulatinnn(,,((((,((( F(O),
10
Onew + get les_aastadddd((()) // get Last policies added to θ
11
12
moodet select new P Pmodel)
13
Ogradient apply -gg.diedddnnngggg Q select(( Pgradient
i final & (model graddient) // concatenate Omoded and gradinat
F()), d(0) < evaluate finat) // evaluate to get reward and descriptor d 15
24+1 update_ update_populante popplattion() F(0), d(0))
q update_ soddat_ddnamiis_moone (g+). Qu <- (
return ,
The transition probabilities P(8++ 1s,, (s,a;)) ax) of the environment are usually assumed to be unknown. Model-based RI. (Wang et al. 2019hp methods learn a parametric model Po 8e, ae) typically using supervised learning, from data collected when interacting in the environment. Policies are then trained using transitions obtained by rolling out the model.
2.2 QUALITY DIVERSITY
(QD) Pugh al. 2016 Cully & Demiris 2017 are diversity-seekin population-
based approaches to learning. QD methods maintain diversity of policies in the population θ while maximizing the performance of each policy 0 θ The population usually contains thousands of policies. QD considers an objective function F(0) acting on the parameters of the policy 6. Addi- tionally, QD also considers a behavior descriptor d(0) that characterizes the behavior induced by policy. d(0) is used to maintain solutions in their behavioral niche guarantee the population diver- sity and that there is no two solutions in the population with similar behavior descriptor d().. When applying QD to a RL problem, they are defined follows, where d(T) is the behavior descriptor of a given trajectory T:
F(0) J ((TT) = Eronn [R(T) and d(0) EToTo [d(T)]
Similar Evolutionary Strategies (ES) Salimans et al. 2017). QD methods operate on entire episodes and hence both the objective and the descriptor can be computed simultaneously for any parameter vector 0. QD methods then aim to maximize:
max QD-Score(O)
Σ F(0) BE
By maintaining population of both diverse and nigh-performing policies, QD uses the exist- ing parameters in the population stepping stones (gguyen et al., 2016 Wang al. 2019a) in the optimization. At each iteration i, random set of policies are sampled from the current popu- lation G, perturbed and evaluated in the environment. Based these perturbed policies might replace existing ones or fill in a new niche in the population. This incrementally improves the QD-Score (eroree()) and encourages creativity and exploration.
the results of these evaluations,
METHOD
In this section, introduce two new model-based QD methods: DA-QD-ext and GDA-QD.

---

Under review as a conference paper at ICLR 2023
Figure 2: Deceptive reward tasks in the form of PointMaze, AntTrap and AntMaze environments.
3.1 DA-QD-EXT: LEARNING AND SIEVING IN IMAGINATION
QD approaches are conventionally driven by random perturbations of the parameters of solutions. This undirected and divergent process gives QD its exploration capabilities but its major drawback is the number of samples it requires. This is especially evident for nigh-dimensional optimiza- tion problems such as optimizing deep neural networks where usually more directed gradient based methods are used. In this work, inspired by Lim et al. (202T), we perform the perturbation process in imagination, relying on learned dynamies model of the environment to reduce the number of environment interactions when evaluating such perturbed policies.
At each iteration j of the algorithm, the current population O is first "copied" into imagination θy. The policies of this provisional population T Ô, are then perturbed, and evaluated in imagination using the rollouts of the dynamics model 9ø- Both the objective F(Tg) and the descriptor d( of the policies be obtained from the state information present in the rollouts. Using this process, G undergoes multiple steps of QD optimization in imagination. The resulting policies Tq E Q that are added to the provisional population é, during learning imagination are then evaluated in the environment and used to update the population θ if they improve the QD-Score of the population. The updated population θ is then used as a start for the next iteration the algorithm. This process of performing QD in imagination acts as sieve and filters out perturbed solutions that not likely to improve the quality and diversity of the population, hence increasing the sample efficiency.
Following Chua et al. (2018) use probabilistic bootstrap ensemble of models qs capture uncertainties. Each model in the ensemble is a probabilistic model which predicts parameters of Gaussian distribution N (S a4), Es (st,,, ($))a)) az)) which
then sample from, capturing the
aleatoric uncertainty. This model-based QD method corresponds to DA-QD [im et al. 2021 However, optimize high-dimensional closed loop neural network policies in complex exploration domains and hence, refer to it as DA-QD-ext.
3.2 GDA-QD: INCORPORATING GRADIENTS IN QUVEITY-DIVERSITY
As mentioned above, the random perturbations driving QD approaches prove inefficient when ap- plied to high-dimensional search spaces such as the parameters of deep neural networks (Colas et al.] 2020). To deal with this, we augment the usual perturbation operator with policy gradient informa- tion as done by Nilsson & Cully (00II. The policy gradient can be more intuitively thought of as more directed perturbation, hence being more efficient optimization update procedure. To apply policy gradients to a populationx, we maintain a critic network Q2 which approximates the action-
value function Q(s,, a.) = ET= 7 Tc+*++1 St. at and gives the expected return from being in state 8, and following action dt. The critic allows to gradually improve any policy in the di- rection maximizing the expected return by computing policy-grade gradient in Equation 4 approximated over batch of transitions. We train Qp using the same procedure TD3 (Fujimoto al. 2018).
Vo, ((,)) Ex [V Vo TV (s) V as)
(4)
In DA-QD-ext explained in Section 3.1 the population of policies θ, is copied in imagination θ to undergo multiple steps of QD optimization and returns B policies to be evaluated. In GDA-QD, we combine the efficient parameter based perturbation in imagination from DA-QD-ext with more directed policy gradient updates explained above. The former is critical for efficient exploration,
4

---

Under review as a conference paper at ICLR 2023
ME-ES
Novelty Search-ES
Objective based HL
Intr. Motivation
Diversity based
D4-OD-Ext [Ours)
MAP-Eiltes
NS Reward-ES
TD3
SMERL
GDA-QD (Ours)
Adaptive
SAC
DIAYN
Figure 3: Final population of methods in the AntTrap environment (top left). Each policy in the population is represented with a dot in the final position manages to reach within the episode. The color of the square around each policy indicates its total reward, the lighter the better. The obstacle clearly appears on this plot, as the empty area in the middle. Our approach GDA-QD does not get stuck in the trap and clearly outperforms baselines given the same number of evaluations.
while the latter critical for efficient exploitation. To achieve this, a proportion Pmodel of B is sampled to be evaluated directly while another Pgradient using the policy gradient operators before being evaluated. In the following, we use Pmodel = 0.9. We study the impact of this value and illustrate the complementary properties of these two generation-procee proceeurrss in Appendix We demonstrate that both parameter update procedures are essential guarantee performance. Figure and Algorithm provides summary of this algorithm.
Pmodet is sampled to be improved
key design choice in GDA-QD for simplicity is that only parameter-basse baseed perturbations are ap- plied in imagination, while the policy gradient updates not applied in imagination. This to clearly make a separation between efficient exploration and efficient exploitation (through gradient optimization). Applying policy gradient updates in imagination could potentially further improve GDA-QD at the cost of training the critic more often in imagination. We leave this for future work.
3.3
QUALITY-DIVERSITYY AS DATA GENERATORS FOR DEEP MODELS
key quality of QD algorithms that they are excellent generating diverse and high-quality data (Ecoffet et al. 1 Grillotti & Cully 2022 In work, the search for diversity of high- performing behaviors when optimizing the population results a diverse dataset of transitions. We leverage this property train deep models which require and often excel when provided with such data. As commonly done, the diverse dataset transitions is stored in a replay buffer and used to train (1) the dynamics model 4s and (2) the critic Q Both these models are suitable candidates as the training of the critic Q is off-policy and can trained using transitions collected from any behavioral policy. Additionally, training the dynamics model go is supervised learning problem which would benefit from large and diverse dataset of transitions.
We found this property be especially important to our method. Specific to the training of the dynamics model, GDA-QD does not use transitions produce by gradient-optimized policies to train the model. This was found empirically in our studies as we observed the transitions produced by the gradien--optimized policies induced a shift in distribution that prove detrimental for the training of the dynamics models. Results of this study are detailed later Section 4.3 It is important to recognize that the transitions used to train the dynamics model are not merely just from policies that have been randomly perturbed but population of policies that have undergone multiple steps of QD optimization in imagination and are expected to improve the population. This results in diverse and high performing dataset of transitions that GDA-QD uses to further train its models.
EXPERIMENTS
We aim to evaluate our method by answering four main questions: (1) Can we scale model-based QD approaches RI. domains and Neuroevolution? (2) Does GDA-QD results in more performant
5

---

Under review as a conference paper at ICLR 2023
POINTMAZE Cov
ANTTRAP Cov
ANTMAZE
QD-SCORE
MAX-REW
QD-SCORE
MAX-REW
QD-SCORE
Cov
MAX-REW
TD3
-126.38
189.52
1.05
SAC
1226188
204.68
1.06
FS
0.46
0.52
-126.85
2.97
2.91
200.95
18.6
10.88
0.97
DIAYN
-67.98
-6.15
0.20
SMERL
:38.29
171.81
1.06
NS-ES
0.93
1.8
-147880
10.12
28.76
-13.14
45.45
42,6
1.23
ME
93,74
99.92
25.36
42.44
43.44
218.77
56.71
37.68
1.29
PGA-ME
93.06
99.92
-24.06
47,82
47.08
274.52
62.56
39.66
1.48
NSR-ES
1.07
1.24
-126.85
6.36
6.26
196.30
22.89
14.24
1.02
NSRA-ES
1.44
1.78
-126.85
14.18
29.32
170.38
46.5
43.56
1.30
ME-ES
21.0
30.52
-62.05
12.70
21.92
157.42
33.48
1.15
DA-QD=oXt
96.67
99.92
-24.81
50,33
51.0
196.66
70.94
43.7
1.51
GDA-QD
1.87
Table 1: Final QD-Score (% of maximum value), Coverage (%) and Max-Total-Reward reached by all algorithms all considered tasks. Each experiment is replicated 15 times, we report in the table the median value across runs. In the algorithms name, ME stands for MAP-Elites.
and sample-efficient learning than traditional QD approaches and simple model-based QD? (3) What is the importance of the policy-gradien, perturbation in the performance of GDA-QD? (4) How does the data-generation capabilities of GDA-QD enforce efficient learning?
4.1 EXPERIMENTAL SETUP
Tasks and Environments: We focus on tasks considered in literature hard exploration problems:
PointMaze Lehman & Stanley 201 1b
Parker-Holder et al. (2020). AntTrap (Conti et al. 2018
Colas al. 2020 Cideron al. 2020
Parker-Holder et al.] 2020p and AntMaze (Colas al. 2020
Cideron et al. 2020 Salter et al.
2022
(see Fig. 2 The reward in these tasks is deceptive making
start, we consider a simple PointMaze environment where a 2-dimensional point agent is given reward corresponding to the distance to the goal in the maze. The AntTrap and AntMaze are higher dimensional continuous control tasks where 8-DoF Ant robot learns how to walk, aiming go beyond the trap in AntTrap and to reach the goal in AntMaze. In AntTrap, the robot gets increasing rewards for going as fast as possible while minimizing energy-usage. In the AntMaze tasks, the reward is the distance to the goal. This reward definition for all the tasks considered makes them deceptive. The descriptor d(Te) used in all the tasks defined as the x-y position the end of the trajectory (æT, yT).
exploration and diversity critical when solving them.
Baselines: Across our experiments, we consider the following baselines:
MAP-Elites: the most-commonly used QD algorithm (Mouret & Clune
2015
PGA-MAP-Elites: (Nilsson & Cully 2021 augments MAP-Elites with a policy-gradient based update operator.
OpenAl ES: Evolution Strategy (Salimans et al. 2017 relying on natural gradient approximation.
Novelty Search ES: We compare against NS-ES Conti et al. 2018) as an intrinsic motivation
baseline. This uses the OpenAl-ES algorithm but with novelty reward instead of task reward..
QD-ES Algorithms: QD-ES algorithms consider both quality and novelty during optimization. We use NSR-ES, NSRA-ES (Conti et al. 2018). and MAP-Flites-ESs (ME-ES) (Colas et al., 2020)
NSR-ES and NSRA-ES build on NS-ES by including the task reward term as weighted sum with
the novelty reward term. ME-ES mixes MAP-Elites with OpenAl ES Colaa et al. 2020p.
Single Policy Deep RL Algorithms:
2018 and SAC Haarnoja We consider TD3 Fujimoto et al.
al. 2018). SAC entropy regularized and is popular choice for greater exploration.
Mutual Information RL Algorithms: We also consider DIAYN Eysenbach et al. (2018) and
SMERL Kumar al. (2020) which are also diversity seeking algorithms. To ensure the compar- isons are lair with descriptor based methods, we use the x-y prior when running these algorithms. DIAYN is purely unsupervised and does consider the task rewards. SMERL considers the task reward during optimization
For fairness, as the single policy baselines (TD3, SAC, DIAYN, SMERL) do not rely on a popula- tion, complete-episode evaluations, only reports its final value in the results dotted

---

Under review as a conference paper at ICLR 2023
PointMaze
AntTrap
AntMaze
100
80
100
80
80
60
60
40
QD
20
400
2.00
-50
Maximumm-ooo www.ium--rrraaaoo Reward
-100
1.75
1.50
-200
200
1.25
-20
1.00
-300
100
0.75
0.50
-400
0
-450
0.00
10000200000000000000000000
1.0
Evaluations
Evaluations
Evaluations
DA-QD-ext
ME
NSRA-ES
NSR-ES
SMERL
TD3
Figure 4: QD-Score (top) Bax-Total-Rewae (bottom) of all algorithms the AntTrap (left), PointMaze (centre) and AntMaze (right) tasks plotted against number of evaluations. Each experi- ment replicated 15 times, the solid line corresponds to the median over replications and the shaded area to the first and third quartiles.
line. For more qualitative comparison of these algorithms, we also collect the trajectories of the policies throughout the learning process and plot them as part of a population. To ensure the com- parability of algorithms using ES (OpenAI-ES, NS-ES, NSR-ES, NSRA-ES, ME-ES) with other approaches, we consider every estimate-evaluation as one sample, making these algorithms highly sample-inefficient.
Metrics: We consider two metrics to assess the performance of GDA-QD:
QD-Score: defined in Section 2.2 It quantifies the diversity and quality of the overall population and allows to compare population- based methods.
Max-Total-Rens Reward:: the total reward of the best individual of the current population. This metrics allows comparison with single-policy methods such as RL baselines.
Implementation and hyperparameters: Our source code available at CO-BE-RELEASED ! includes containerised environment to replicate our experiments. The methods presented in this paper as well as all our baselines are based on the implementation of MAP-Elites in the QDax open-source library (Lim al. 2022). using the Brax simulator Freeman et al. (202T). All hyperpa- rameters and implementation details used in our algorithms and for model training can be found in the Appendix
4.2 RESULTS
The results of experiments summarized in Figure 4 and Table [ We also display visual- ization of the final populations of policies for each algorithm in Figure 3 (AntTrap) and Appendix C (PointMaze and AntMaze).
Figure shows that both our proposed model-based versions, DA-QD-ext and GDA-QD, signifi- cantly outperform all baselines in terms of sample efficiency and final performance. This demon-
TThe code will be made available after the review period

---

Under review as a conference paper at ICLR 2023
QD-Score
Coverage
Max- Max-Total Reward
1e6
400
1.5
1,0
40
200
0.5
100
0.0
0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0 1e6
0.2
0.4
Evaluations
1e6
Evaluations
Evaluations
1e6
10% pg
10% pg
pg
50% pg
50% pg filtering QD transitions
no filtering
filtering pg transitions
no filtering
filtering pg transitions
Figure 5: QD-Score (left), Coverage (middle) and Max-Total-Rewant (right) on the AntTrap task of GDA-QD with different proportion of volicy-gradient generated policies (pg) and different tran- sitions filtering schemes. The solid line corresponds to the median over replications and the shaded area to the first and third quartiles.
strates that we can scale model-based QD methods deep neuroevolution. The performance of DA-QD-ext suggests that learning diverse behaviors in imagination using the dynamics model is a simple but effective approach to save samples yet maintain the divergent search capabilities of the random perturbations. However, we can see that the maximum total reward obtained by DA-QD-ext seems to stagnate and increase slowly, especially when compared to GDA-QD and PGA-MAP- Elites. This can be explained by the absence of reward maximizing optimization updates such as the policy gradient updates present in GDA-QD and PGA-MAP-Fllies Hence, GDA-QD is shown to get the "best of both worlds" by benefitting from the QD in imagination well as the policy gradient updates. This is evident in its performance across the QD-Score and max. return metrics.
In terms of baselines, F show that all objective-only baselines: SAC, TD3 and ES, struggle with deceptive rewards. They all get stuck into the traps present in AntTrap and AntMaze. Diversity Based RL algorihtms such DIAYN struggle with no task reward signal while SMERL manages to reach the final goal only in the simpler PointMaze. The single-agent baselines (TD3, SAC, DIAYN, and SMERL) are plotted as horizontal lines to represent the max performance obtained by the agent. This is done to allow comparison as they do not rely on a population, making the Coverage and QD-Score metries not relevant, and also performs policies-updates within episodes. The poor per- formance of ES-based algorithms, in particular ME-ES, can be attributed to the number of samples (~ hundreds or thousands) required just to approximate a single gradient step. It is important to note that these methods commonly do not consider the notion of evaluations and generally evaluate the algorithms versus time or number of generations as they are suited to be heavily parallelized across clusters of CPUs. Despite this limitation, intrinsically-motivaeedd ES baselines NS-ES and its QD variant NSRA-ES manage to get good coverage on AntTrap and AntMaze. However, they struggle to discover high-perfoor perhfrrrfor solutions within the given evaluation budget. We provide visualization of the adaptive mechanism of NSRA-ES in Appendix
4.3 IMPORTANCE OF QD FOR TRAINING DEEP MODELS
To investigate the data generation abilities of QD, we compare the performance of the algorithm when the dynamics model is trained on different data distributions based the policies rolled out in the environment. In our case, we have two main types of data generators: policies obtained thought the QD process in imagination (giving Mmodet), and policies perturbed also using policy- gradients (giving D gradient). We run an ablation where we train the dynamics model on transitions collected by either Dinodet Dgradient, or a mixture of both Emodee+ggaddiee corresponding transitions. Figure 5) shows the performance curves when running this ablation on the AntTrap task. We first test the effect of training the model with D model when Paradient = 0.1. We notice minor difference in which experiments that train only on Dmodel perform better than when not using any filter (i.e. D D ). To enable a fair comparison in terms of the number of transitions added to the replay buffer when attempting do the converse (i.e. using D prradient), ) we use a Pmoddt = Poradient = 0.5. We observe better performance when the dynamics model is only trained with Dmodel. This is compared to training on D modde graddient where the difference
by filtering out the
8

---

Under review as a conference paper at ICLR 2023
is minimal as seen previously. However, performance significantly drops when training just on Dgradient- We hypothesize that this is due to bias in the transitions obtained through the policies perturbed by policy gradient resulting in skewed dataset while the transitions given by policies obtained from QD in imagination provides a diverse and high-quality dataset.
RELATED WORK
Searching for Diversity. Prior studies have shown the importance of maintaining a diverse set of solutions to solve problem. Novelty-search approaches (Lehman & Stanley, 2011a) inspired from evolutionary computation optimize for the novelty of solutions defined by behavioral character- ization with respect to population instead of the optimization objective. QD methods (Cully & Demiris, 2017; Mouret& & Clune 2015) extend this approach by also considering the objective, aim- ing to find both diverse and sesh-eeformmig population of solutions. Similarly, our work builds on QD approaches with the aim to maintain diversity of high-perfomnn nigh-performing solutions. Other approaches
to searching for diversity also exist in the RL community. Unsupervised RL methods (Eysenbach) mutual information maximization objective to learn a diversity of behaviors in a skill-conditioned policies. Similar to QD, Kumar et al. (2020); Zahavy et al. (022) have proposed to extend these unsupervised RL approaches by integrating ob-
ct al.
2018; Sharma al. 2020) commonly
jectives using constrained Markov decision processes. While these approaches typically focus on maintaining dozen of different policies, our algorithm discovers thousands of independent diverse policies.
Neuroevolution seeks to evolve neural networks through biologically-inen inspired methods such as evo- lutionary algorithms and have interesting properties unavailable to common gradient-base meth- ods (Stanley et al. 2019). However, a limitation in the neuroevolution domain is the dimensionality of the search-space, that quickly limits the effectiveness of random perturbations. Some methods have overcome this limitation by using indirect encoding methods (Clune et al,, 2009) or through natural approximated gradients (Salimans et al. 2017). Our work hybridizes neuroevolution with deep reinforcement learning methods. Similar to Nilsson & Cully (202I). we use policy-gradiens as directed perturbations to effectively maneuver the high-dimensional search space but, sig- nificantly improve the sample efficiency and performance by also augmenting these operators with model-based methods.
As the perturbations commonly used in QD are sample inefficient, prior work has sought the use of data-driven models to alleviate this. SAII. Gaier et al. (2018) introduced the use of surrogate model in the form of a Gaussian Process model to predict the objective. As Gaussian processes generally only work well on low-dimensional data, more recent methods have explored the use of deep networks (Keller et al., [0200] Lim et al.) [02T] Zhang et al., 2022 as forms surrogate models to predict both the objective and descriptors. We utilize the model- based QD framework from Lim et al. (022)) which first introduced the idea of maintaining an imagined population and also builds on model-based RL methods (Wang et al.) (019b) where a dynamics model is used. Critically, this work has only been applied to low-dimensional open-loop policies. To the best of our knowledge, our work is the first model-based QD algorithm that scales to the more complex deep neuroevolution domain where optimize closed loop RL policies.
Model-based
6 CONCLUSION AND FUTURE WORK
In this paper, introduce a novel model-based Quality- method, GDA-QD, which opti- mizes population of diverse policies to explore more efficiently. To the best of our knowledge, this approach is the first basel-based QD algorithm scaling to neuroevolution to optimize deep neural network policies. We leverage a key property of QD algorithms as effective data generators to train deep models in the form of dynamics model and critic. In turn, these models help to significantly improve the sample efficiency and final performance of the QD algorithm. The dynamics model used to learn and sieve policies in imagination while the critic is used to apply policy gradient updates to sampled policies. Our experiments show that GDA-QD outperforms a range of Deep RL and QD baselines on hard exploration task containing deceptive rewards. Overall, we demonstrate some of the powerful synergies that arise between population- based learning and deep learning approuches. In future work. we hope to extend our work to more complex domains through the use of latent dynamics models (Ha & Schmidhuber 2018 Hafner et al. 2019.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Iige Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz. Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint 1rXiv:1910.07113. 2019.
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information pro- cessing systems, 29, 2016.
Konstantinos Chatzilygeroudis. Antoine Cully, Vassilis Vassiliades, and Jean-Baptiste Mouret. Quality-diversity optimization: novel branch of stochastic optimization. Black Box Opti- mization, Machine Learning, and No-Free Lunch Theorems, pp. 109-135. Springer. 2021.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. arXiv preprint arXiv: 121 2018.
Geoffrey Cideron, Thomas Pierrot, Nicolas Perrin, Karim Beguir, and Olivier Sigaud. Qd-rl: Effi cient mixing of quality and diversity in reinforcement learning. arXiv preprint arXiv:2006.088: 08505, 2020.
Jeff Clune, Benjamin E Beckmann, Charles Ofria, and Robert T Pennock. Evolving coordinated quadruped gaits with the hyperneat generative encoding. In 2009 iEEE congress on evolutionary computation, pp. 2764-2771. IEEE, 2009.
Cédric Colas, Vashisht Madhavan, Joost Huizinga, and Jeff Clune. Scaling map-elites to deep neu- roevolution. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp- 67-75, 2020.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff
Clune. Improving exploration in evolution strategies for deep reinforcement learning via a popu- lation of sovelty-seeking agents. Advances in neural information processing systems, 31, 2018.
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular frame- work. IEEE Transactions on Evolutionary Computation, 22(2):245 11944522599990 2017.
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptis Mouret. Robots that can adapt like animals. Nature, 521(7553):503-507, 2015.
Adrien Ecoffet, Joost Huizinga. Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):5800-888 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprint arXiv: 2018.
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax a differentiable physics engine for large scale rigid body simulation, 2021. URL http:
// httbub..com/gooole///aaaa
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-
critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Adam Gaier, Alexander Asteroth, and Caanttaptiste Mouret. Data-efficient design exploration through surrogate-assisted illumination. Evolutionary computation, 26(3):381-410 2018.
Luca Grillotti and Antoine Cully. Unsupervised behaviour discovery with quality-diverse optimi- sation. IEEE Transactions on Evolutionary Computation, 2022.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018
Tuomas Haanojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. PMLR, 2018.
10

---

Under review as a conference paper at ICLR 2023
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojeiech M Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based train- ing of neural networks. arXiv preprint arXiv: 1.09846, 2017.
Leon Keller, Daniel Tanneberg, Svenja Stark, and Jan Peters. Model-based quality-diversity search for efficient robot learning. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 9675-9680. IEEE, 2020.
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems,
2020.
Joel Lehman and Kenneth O Stanley, Evolving diversity of virtual creatures through novelty search and local competition. Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp. 211-11221 1-218, 2011a.
Joel Lehman and Kenneth Stanley. Novelty search and the problem with objectives. In Genetic programming theory and practice IX, pp. 37-56. Springer, 201 lb.
2021. 2021. Bryan Lim, Luca Grillotti, Lorenzo Bernasconi, and Antoine Cully. Dynamics-aware quality- diversity for efficient learning of skill repertoires. arXiv preprint arXiv:2109.08522,
Bryan Lim, Maxime Allard, Luca Grillotti, and Antoine Cully. Accelerated quality-diversity for robotics through massive parallelism. arXiv preprint arXiv: 220.:22002.12558 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 (177540):559-533 -3333, 2015.
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint orXiv: 2015.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding innovation engines: Automated cre- ativity and improved stochastic optimization via deep learning. Evolutionary computation, 24(3): 545-572, 2016.
Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 866-875, 2021.
Pierre Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for au- tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-266, 2007.
Jack Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33:18050-18062. 2020.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2781 PMLR, 2017.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolu- tionary computation. Frontiers in Robotics and AI, 3:40, 2016.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative reinforcement learning. arXiv preprint 1r03iv/773.03366 2017.
Sasha Salter, Markus Wulfmeier, Dhruva Tirumala, Nicolas Heess, Martin Riedmiller, Raia Hadsell, and Dushyant Rao. Mo2: Model-based offline options. arXiv preprint arXiv:2209.01997, 2022.

---

Under review as a conference paper at ICLR 2023
Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu. Emergent real-world robotic skills via unsupervised off policy reinforcement learning. arXiv preprint arXiv:2004.12977 2020.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieserr, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
Kenneth Stanley, Jeff Clune, Joel Lehman, and Risto Mikkkulannen. Designing neural networks through neuroevolution. Nature Machine Intelligence, 1(1):24-35, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Vassilis Vassiliades and lean-Baptiste Mouret. Discovering the Elite Hypervolume by Leveraging Interspecies Correlation. In GECCO 2018 Genetic and Evolutionary Computation Conference, Kyoto, Japan, July 2018. doi: 10.1 10.11455/205455332055022 URL https: httss//hhh..iiia......
0100111777 1a1-01166733
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350-354 2019.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Poet: open-ended coevolution of environments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 142-151, 2019a.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. International Conference on Machine Learning, pp. 9940- 9951. PMLR, 2020.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. arXiv preprint arXiv: 907.02057 2019b.
Tom Zahavy, Yannick Schroecker, Feryal Behbahani, Kate Baumli, Sebastian Flennerhag, Shaobo Hou, and Satinder Singh. Discovering policies with domino: Diversity optimization maintaining near optimality. arXiv preprint arXiv:2205. 13321, 2022.
Yulun Zhang, Matthew Fontaine, Amy Hoover, and Stefanos Nikolaidis. Deep surrogate as- sisted map-elites for automated hearthstone deckbuilding. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 158-167, 2022.
12


Reviews:
[{"review_id": "xDzRurxsfF", "reviewer": "Reviewer_TxSP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents two new algorithms, DA-QD and GDA-QD, for training a large population of diverse reinforcement learning agents for continuous control. These algorithms combine the benefits of surrogate-assisted quality-diversity (QD) and off-policy RL. Specifically, DA-QD maintains two MAP-Elites buffers, one real and one \"imagined,\" both along two trajectory-based behavioral descriptors. At each iteration of the algorithm, the real MAP archive is copied into the imagined MAP archive, and MAP-Elites with gaussian perturbations is applied to the imagined population based on rollouts in the learned dynamics model. The final, new elites after this step are then the candidates to be evaluated on the real environment. These real rollouts are then used to update the real MAP archive as well as update the learned dynamics model. GDA-QD is the same algorithm with the exception that only some proportion $p$ of real rollouts is directly evaluated for inclusion in the real archive, while a proportion $1-p$ is updated with TD3. They show on a continuous control task with a local minimum, Ant-Trap, that GDA-QD's use of a learned dynamics model leads to much higher returns than variants of MAP-Elites with ES and policy-gradient updates, as well as TD3.", "summary_of_the_review": "This paper presents an incremental extension of a model-assisted QD method by extending it with TD3 policy gradient updates. However, given that incremental nature of the algorithmic contribution, the work should feature a more comprehensive experimental setting. Moreover, the current environment used in the experiments is simply an exploration problem and thus does not provide a useful benchmark for QD. Based on these issues, I cannot recommend this paper for acceptance.", "strength_and_weaknesses": "### Strengths\n- The background concepts are clearly motivated and communicated.\n- Combination of TD3 with DA-QD seems to be a novel algorithmic configuration.\n- The presentation of solution populations for each method in Figure 3 is well done.\n\n### Weaknesses\n\n**Experimental setting seems inappropriate for benchmarking QD**\n\nMost importantly, the method is based on QD, but the experimental setting is an exploration problem that does not necessarily require QD. I recommend using a different experimental setting that focuses on the benefits of QD over exploration. In particular, QD inherently maintains a population of solutions, making such methods advantageous for multi-modal optimization. Thus, an experimental setting that requires learning solutions for a multi-modal solution space, or that requires adapting to changes in the environment, e.g. if the ant's legs can become damaged, would be much more appropriate than Ant-Trap, which is simply an exploration problem solvable with existing exploration methods in RL.\n\nRegarding the Ant-Trap environment in particular: Deep RL has dealt with such problems using exploration methods, such as entropy regularization and intrinsic motivation. Moreover, there exist efficient quality-diversity methods in deep RL, such as DIAYN, DOMiNO, and DvD, which do not require maintaining two populations or the use of a dynamics model. Thus, at a minimum, it seems important for this work to compare to the following:\n- SAC [1], which learns a maximum entropy policy.\n- DIAYN [2].\n- A basic intrinsic motivation method (e.g. RND [3]).\n\nDOMiNO [4] and DvD [5] both require pre-specifying a small population compared to the MAP-Elites methods used in this work, so it may be possible to find settings where their use of relatively smaller populations proves a disadvantage or makes them inapplicable.\n\nThere also seems to be another important baseline missing: The current baselines control for the use of dynamics model (PGA-MAP-Elites), the use of dynamics model + TD3 (MAP-Elites), as well as the use of dynamics model + MAP-Elites (TD3), in addition to comparing to ES baselines, but these ES baselines do not seem directly relevant to the study. Importantly, the study seems to be missing a comparison to TD3 with a dynamics model (control for the use of MAP elites).\n\nLastly, the comparison to vanilla TD3 seems unfair, as the DA-QD methods maintain large populations, while TD3 is effectively a population of size 1. A fair comparison would be to run a population of TD3 agents (e.g. set to the size of the batch size used for MAP-Elites updates), and compare QD-score, coverage, and max total reward with respect to this population.\n\n**Additional weaknesses**\n- The paper's presentation of the DA-QD algorithm seems to obfuscate the originality of contributions. It is not clear from the writing whether the DA-QD algorithm itself is a new contribution or simply taken from Lim et al, 2021. This is because the introduction states the paper introduces an extension of DA-QD called DA-QD-ext, but then refers to the algorithm as simply DA-QD in the rest of the paper. Which contributions are new should be clearly stated in the paper. \n- DA-QD and GDA-QD seem to require a lot more evaluations than TD3 due to maintaining two large populations inside of the real and imagined MAP-Elites archives. It would be useful to see a comparison of number of steps needed.\n- The paper does not motivate the key design choice of only performing gaussian perturbations in the model. It is not clear why PG-based updates are not used when updating the imagined population.\n\n### References\n- [1] Haarnoja, Tuomas, et al. \"Soft actor-critic algorithms and applications.\" arXiv preprint arXiv:1812.05905 (2018).\n- [2] Burda, Yuri, et al. \"Exploration by random network distillation.\" arXiv preprint arXiv:1810.12894 (2018).\n- [3] Eysenbach, Benjamin, et al. \"Diversity is all you need: Learning skills without a reward function.\" arXiv preprint arXiv:1802.06070 (2018).\n- [4] Zahavy, Tom, et al. \"Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality.\" arXiv preprint arXiv:2205.13521 (2022).\n- [5] Parker-Holder, Jack, et al. \"Effective diversity in population based reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 18050-18062.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "wo64qsIj5io", "reviewer": "Reviewer_tyZo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors introduce a model-based quality diversity (QD) approach, which aims to both solve a problem whilst inducing diverse solutions. To do this, the authors perform several rounds of quality diversity optimization over their population inside the world model, and further augment a subset of the population with a policy gradient from the critic post-hoc, with the aim of guiding solutions towards those that solve the underlying MDP (i.e., the quality part of QD).\n\nThey show that by augmenting QD approaches with a model, they achieve improved sample efficiency (by performing the usual costly rollouts inside a world model), and also strong asymptotic performance. They also show findings regarding the impact of world model data distribution and policy gradient proportion on final performance.", "summary_of_the_review": "Overall the idea of combining model-based approaches with QD algorithms is nice, and this work lends further support to this direction of research. However I have issues regarding the presentation, analysis and experiments which means I am not comfortable recommending acceptance at this point. I'd be happy to raise my score if the authors are able to address these. \n\n---\n\nThe authors largely addressed my concerns, hence I've increased my score.", "strength_and_weaknesses": "Strengths:\n* The core idea of augmenting QD with a model makes a lot of sense, whereby costly real-world samples can be avoided by sampling imaginary trajectories instead.\n* Results appear promising on the evaluated environment.\n\nWeaknesses:\n* I found the paper rather difficult to read, and generally not that well written. I caveat that this may be a consequence of my lack of experience with QD literature, but I found myself requiring several read throughs of the paper before I felt I fully understood what was going on. To give some examples of things I found difficult to parse:\n    * In the algorithm box, many variables and functions aren't explained (e.g., $p_{model}$, $q_\\phi$, last_added). While they are introduced somewhere in the main body, I found myself having to flit between the body and box quite frequently.\n    * Similarly, I found section 3.2 hard to parse and rather terse, specifically the final paragraph. There is quite a bit of terminology introduced and they all seem to interact with each other here. Some repetition, or a clearer approach would be appreciated.\n    * The behavior descriptor is mentioned in a few places, but never defined mathematically; it would have been useful to define this either in the appendix or when introduced. It was also not clear to me how the descriptor was combined with the objective function.\n    * \"This means that we investigate the performance of the algorithm when the dynamics model is trained on different data distributions based on the policies rolled out in the environment.\". It's not entirely clear to me what the authors are trying to say here? It's seems rather imprecise (what are the different distributions? Why these different distributions (i.e., pg v.s. non-pg)?), and made it hard for me to understand the motivation of this section.\n* The quality of the experiments isn't good enough; only one environment is presented, and only 3 seeds. While I appreciate population based methods might be expensive to run, given the lack of environments, it seems reasonable to expect more seeds.\n* The result of filtering out the PG trajectories for model-training seems counterintuitive to me, why is it the case that the performance is lower when including more data? The only explanation given in the paper is that the data are \"skewed\", and I think this could merit more investigation as it's counter intuitive. I think something showing MSE/Log-likelihood on a held out test set of dynamics could shed light here, or perhaps t-SNE/traces of imagined trajectories.\n* I'd like some clarity on\n    * What is greedy learning rate in the appendix?\n    * What horizon was used in the model rollouts?\n    * What is the sample efficiency of TD3 v.s. QD algorithms in terms of environment steps?\n* Nits:\n    * The TD3 and OpenAI ES citations are mixed up\n    * In Algorithm 1, typo in `evaluate_imgination`\n    * withing -> within", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jiGUrENNXe5", "reviewer": "Reviewer_FZRr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper considers the case of population-based approaches for exploration in reinforcement learning, in particular the Quality-Diversity approach, an evolutionary algorithm to create collections of high performing solutions. The authors propose two algorithm variations of QD which take into consideration the learned dynamics model and the policy gradient. The resulting policy is evaluated on the ant-trap problem, a hard exploration problem with a deceptive reward, where the short term optimization of the reward leads the agent into a trap. The resulting algorithm exceeds the performance and is more sample efficient that the baseline QD approach.", "summary_of_the_review": "The paper proposes an approach that integrates the quality-diversity approach with reinforcement learning. The proposed approach outperforms other QD approaches on the ant-trap problem, but it is unclear how it generalizes further. \n---\nThe authors had made several additions to the paper during the discussion session, which convinced me to raise my score. ", "strength_and_weaknesses": "+ Clear description of the justification and the proposed technique. \n+ Interesting hybridization of reinforcement learning and neuroevolution. \n+ Performance increase over other QD approaches over the chosen experimental problem. \n- The system positions itself within the QD community, with most of the baseline algorithms coming from the same community. \n- The system is only evaluated on the ant trap problem, which is hand engineered to trigger a particular kind of behavior, of the ant being trapped in the obstacle. It is unclear whether the proposed approach would generalize to any other type of environments. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "79q2ikzIwB", "reviewer": "Reviewer_QkBZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a Quality-Diversity approach that bootstraps the sample efficiency with a learned dynamics model. The authors demonstrate that the model-based QD method outperforms both Deep RL and QD baselines in a hard-exploration task.\nThe idea is simple and straight, but the writing of the paper should be improved since the reviewer hardly follows the motivation and the components in Figure1. Besides, the experimental results are quite limited to a simple environment. The paper pays much content to the ablation study, which is trivial.", "summary_of_the_review": "see above", "strength_and_weaknesses": "Strength:\n\nThe idea is simple and straight, the experiment results seem significantly improves the sample efficiency and quality of solutions to the RL QD problem.\n\nWeaknesses:\n\n(1) It is not clear how the dynamics model is built. Although the author has mentioned that they apply a probabilistic bootstrap ensemble of models, the reviewer is still confused about the method, goal, and other details for training the dynamics model.\n\n(2) The experimental results are limited to a simple AntTrap environment. More comparisons of other environments are expected.\n\n(3) The training curve in Figure 2 and Figure 4 should take the overhead of training dynamics models into account since they perform some level of feature extraction and generalization. A fair comparison would be setting the total training time as the x-axis when using the same hardware resources.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject