Accelerated Quality-Diversity through Massive Parallelism
Authors: Bryan Lim
Maxime Allard
Luca Grillotti
Antoine Cully
Venue: Accepted by TMLR ()

Abstract: Quality-Diversity (QD) optimization algorithms are a well-known approach to generate large collections of diverse and high-quality solutions. However, derived from evolutionary computation, QD algorithms are population-based methods which are known to be data-inefficient and requires large amounts of computational resources. This makes QD algorithms slow when used in applications where solution evaluations are computationally costly. A common approach to speed up QD algorithms is to evaluate solutions in parallel, for instance by using physical simulators in robotics. Yet, this approach is limited to several dozen of parallel evaluations as most physics simulators can only be parallelized more with a greater number of CPUs. With recent advances in simulators that run on accelerators, thousands of evaluations can now be performed in parallel on single GPU/TPU. In this paper, we present QDax, an accelerated implementation of MAP-Elites which leverages massive parallelism on accelerators to make QD algorithms more accessible. We show that QD algorithms are ideal candidates to take advantage of progress in hardware acceleration. We demonstrate that QD algorithms can scale with massive parallelism to be run at interactive timescales without any significant effect on the performance. Results across standard optimization functions and four neuroevolution benchmark environments shows that experiment runtimes are reduced by two factors of magnitudes, turning days of computation into minutes. More surprising, we observe that reducing the number of generations by two orders of magnitude, and thus having significantly shorter lineage does not impact the performance of QD algorithms. These results show that QD can now benefit from hardware acceleration, which contributed significantly to the bloom of deep learning.

Published in Transactions on Machine Learning Research (01/2023)
Accelerated Quality-Diversity through Massive Parallelism
Bryan Lim
Deportment of Computing Imperial College London
Maxime Allard
m. m.allant200@mpperial.cc
Department of Computing Imperial College London
Luca Grillotti
Deportment of Computing Imperial College London
Antoine Cully
a. a.cully@iiperial.cm
Department of Computing Imperial College London
Reviewed on OpenReview: https: // wttp://veeevvveeeeeem/ee net/ forum? id= #nNITCJyfI
Abstract
Quality-Diversigg (QD) optimization algorithms are a well-known approach to generate large collections of diverse and high-quality solutions. However, derived from evolutionary computation, QD algorithms are population-based methods which are known to be data- inefficient and require large amounts of computational resources. This makes QD algorithms slow when used in applications where solution evaluations are computationally costly. A common approach to speed up QD algorithms to evaluate solutions in parallel, for instance by using physical simulators robotics. Yet, this approach limited to several dozen of parallel evaluations as most physics simulators can only be parallelized more with a greater number of CPUs. With recent. advances in simulators that run on accelerators, thousands of evaluations can now be performed in parallel single GPU/TPU. In this paper, present QDax, an accelerated implementation of MAP- Elitess which leverages massive parallelism on accelerators to make QD algorit hms more accessible. We show that QD algorithms are ideal candidates to take advantage of progress in hardware acceleration. We demonstrate that QD algorithms can scale with massive parallelism to run at interactive timescales without any significant effect on the performance. Results across standard optimization functions and four neuroevolution benchmark environments show that experiment runtimes reduced by two factors of magnitudes, turning days of computation into minutes. More surprising, observe that reducing the number of generations by two orders of magnitude, and thus having significantly shorter lineage does not impact the performance of QD algorithms. These results show that QD can now benefit from hardware acceleration, which contributed significantly to the bloom of deep learning.
1 Introduction
Quality-Diverse (QD) algorithms (uugh el al. 2016 Cully & Demiris, 2017 Chatzilygeroudis el. al.
2021
have recently shown to be an increasingly useful tool across a wide variety of fields such as robotics Cully
ct al. 2015 Chatzilygeroudis et al. 2018 reinforcement learning (RL) (Ecoffet et al. 2021), engincering
design optimization Gaaier et al. 2018), latent space exploration for image generation (Fontaine & Nikolaidis
2021 and video game design Gravina et al. 2019 Fontaine et. al. 2020a Farle et al. 2021). Instead of

---

Published in Transactions on Machine Learning Research (01/2023)
optimizing for single solution like in conventional optimization, QD optimization searches for population of high-performing and diverse solutions.
Adopting the QD optimization framework has many benefits. The diversity of solutions found in QD enables rapid adaptation and robustness to unknown environments (Cully et al. 2015 Chatzilygerondis et al. 2018
Kaushik et al. 2020). Additionally, QD algorithms are also powerful exploration algorithms. They have been shown be effective in solving sparse-reward hard-exploration tasks and achieved state-of-the-art results on previously unsolved RL benchmarks (Ecoffet et all, 2021). This is result of the diversity of solutions present acting as stepping stones (Clune] 2019 during the optimization process. QD algorithms are also useful in the design of more open-ended algorithms (Stanley et al., 2017 Stanley 2019: Clunc 2019) which endlessly generate their own novel learning opporunities. They have been used in pioneering work for open-ended RL with environment generation (Wang et al. 2019: 2020). Lastly, QD can also be used as effective data generators for RL tasks. The motivation for this arises from the availability of large amounts of data which resulted in the success of modern machine learning. The carly breakthroughs supervised learning in computer vision have come from the availability of large diverse labelled datasels (Deng al., 2009 Barbu et al. 2019). The more recent successes in unsupervised learning and pre-training of large models have similarly come from methods that can leverage even larger and more diverse datasets that are unlabelled and can be more easily obtained by scraping the web (Devlin et al. 2018 Brown et al. 2020). As Gu et al. |Gu et al. 2021 highlighted, more efficient data generation strategies and algorithms are needed to obtain similar successes in R..
Addressing the computational scalability of QD algorithms (focus of this work) offers a hopeful path towards open-ended algorithms that endlessly generates its own challenges and solutions to these challenges. Likewise, they can also play a significant role in this more data-centric view of RL by generating diverse and high-quality datasets of policies and trajectories both with supervision and in an unsupervised setting (Cully 2019 Paolo
et al. 2020
The main bottleneck faced by QD algorithms are the large number of evaluations required that is on the order of millions. When using QD in the field of Reinforcement Learning (RL) for robotics, this issue is miligated by performing these evaluations in physical simulators such as Bullet (Coumans &
GPU/TIU
Bai 2016 2020), DART (Lee et al. 2018), and Mu- JoCo Todorov et al. 2012). However, these simu- lators have mainly been developed to run on CPUs. Methods like MP1 can be used to parallelise over multiple machines, but this requires a more sophis- ticated infrastructure (i.e., multiple machines) and adds some network communccttio overhead which can add significant runtime to the algorithm. Additionally, the number of simulations that can be performed in parallel can only scale with the number of CPU cores available. Hence, both the lack of scalability coupled with the large number of evaluations required generally make the evaluation process of evolutionary based algorithms like QD, for robotics take days on modern 32-core CPUs. Our work builds on the advances and availability of hardware accelorators, high-performance Makoviychuk et. al.
Epocis
spodi
Figure 1: QDax uses massive parallelism on hardware accelerators like GPUs/TPUs to speed up runtime of QD algorithms by orders of magnitude
programming frameworks (Bradbury et. al. 2018) and simulators (Freeman et. al. 2021 2021 that support these devices to scale QD algorithms.
Historically, significant breakthroughs in algorithms have come from major advances in computing hardware. Most notably, the use of Graphic Processing Units (GPUs) to perform large vector and matrix computation enabled significant order-of- maguuttde speedups in training deep neural networks. This brought about modern
deep-learning systems which have revolutionized computer vision (Krizhevsky et al. 2012 He et al. 2016 Redmon al. 2016), natural language processing (Hochreiter & Schmidhuber 1997 Vaswani et al. 2017) and even biology (Jumper et al. 2021]. Even in the current of deep learning, significant network architectural
breakthroughs (Vaswani et al. 2017 were made possible and are shown to scale with larger datasets and
more computation (Devlin et al.
2018 Shoeybi et al. 2019 Brown et al. 2020: Smith et al. 2022
2

---

Published in Transactions on Machine Learning Research (01/2023)
Our goal in this paper is to bring the benefits of advances in compute and hardware acceleration to QD algorithms (Fig. D. The key contributions of this work are: (1) We show that massive parallelization with QD using large batch sizes siguificantly speeds up run-time of QD algorithms at no loss in final performance, turning hours hours/days of computation into minutcs.
(2) We demonstrate that contrary to prior beliefs, the number of iterations (generations) of the QD algorithm is not critical, given a large batch size. We observe this across optimization tasks and a range of QD-RL tasks. (3) We release QDax, an open-source accelcrated Python framework for Quality-Diversity algorithms (MAP-Elites) which enables massive parallelization on a single machine. This makes QD algorithms more accessible to a wider range of practitioners and researchers. The source code of QDax is available at
2 Related Work
Quality-Diversin QD algorithms were derived from interests in divergent search methods (Lehman & Stanley 2011a) and behavioural diversity [Mouret & Doncieux, 2009 2012) in evolutionary algorithms and hybridizing such methods with the notion of fitness and reward (Tehman & Stanley 2011b). While QD algorithms are promising solutions to robotics and RI, they remain computationally expensive and take long time to converge due to high sample-complextty The move towards more complex environments with high-dimensional state and action spaces, coupled with the millions of evaluations required for the algorithm to converge, makes these algorithms even more inacoessible to regular hardware devicos. Progress has been made towards lowering the sample complexity of QD algorithms and can generally be categorized into two separate approaches. The first approach is to leverage the efficiency of other optimization methods such as evolution strategies (Colas et al.
2020; Fontaine et al. 2020b Cully 2020: Wang et al. 2021 ) and Pierrot ct al.
policy-gradients (Nilsson & Cully 2021
2021). The other line of work known as model-based Lim al. 2021h, reduces the number of evaluations required through the use of surrogate models to provide prediction of the descriptor and objective.
quality-diversijy (Gaier et al.) 2018 Keller al.]
2020
Our work takes a separate approach orthogonal to sample-effeienny and focuses improvement on the runtime of QD algorithms instead by leveraging the batch size at each iteration. Additionally, despite algorithmic innovations that improve sample-eficiency, most QD implementations still rely on evaluations being distributed over large compute systems. These often give impressive results (Colas et al. 2020 Fontaine et al.j 2019) but such resources are mainly inaccessible to most researchers and yet still take significant amount of time to obtain. Our work aims to make QD algorithms more accessible by running quickly on more commonly available accelerators, such as cloud available GPUs.
Hardware Acceleration for Machine Learning. Machine Learning, and more specifically Deep Learning methods, have benefited from specialised hardware accelerators that can parallelize operations. In the mid-2000's rescarchers started using GPUs to train neural networks (Stcinkrau et al. 2003) because of their high degree of parallelism and high memory bandwidth. After the introduction of general purpose GPUs, the use of specialized GPU compatible code for Deep Learning methods (Raina et al. 2009
Ciresan et al.)
2012 enabled deep neural networks to be trained a few orders of magnitude quicker than previously
CPUs Lecun et al. 2015). Very quickly, frameworks such Torch (Collobert et al. 2011
Tensorflow (Abadi
et al. 2016), PyTorch (Paszke et al.] 2019) or more recently JAX (Bradbury et al. 2018 were developed to
run numerical computations on GPUs or other specialized hardware.
These frameworks have led to tremendous progress in deep learning. In other sub-fields such as deep reinforcement learning (DRL.) or robotics, the parallelization has happened on the level of neural networks. However, RL algorithms need lot of data and require interaction with the environment to obtain it. Such methods suffer from slow data collection process the physical simulators used to collect data were mainly developed for CPUs, which results in a lack of scalable parallelism and data transfer overhead between devices. More recently, new rigid body physics simulators that can leverage GPUs and run thousands of simulations in parallel have been developed. Brax (Freeman et al., 2021) and IsaacGym (Makoviychuk et al.) 2021 are examples of these new types simulators. Gradient- based DRL methods can benefit from this massive parallelism as this would dircctly correspond to estimating the gradicnts more accurately through collection of larger amounts of data in a faster amount of time (Rudin et al. 2021) at each optimization step. Recent work (Gu et al. 2021 Rudin et al. 2021) show that control policies can be trained with DRL algorithms like
3

---

Published in Transactions on Machine Learning Research (01/2023)
PPO (Schulman et al. 2017) and DIAYN [Eysenbach al. 2018) in minutes single GPU. However, unlike gradient-based methods, it is unclear until now how evolutionary and iopulation-basee approaches like QD would benefit from this massive parallelism since the implications of massive batch sizes has not been studied to the best of our knowledge. Our work studics the impact of massive parallelization on QD algorithms and its limitations.
3 Problem Statement
Quality-Diversity Problem. The Quality-Diversity (QD) problem Pugh et al. 2016): Chatzilygeroudis (2021) is an optimization problem which consists of scarching for set of colutions A that (1) are locally optimal, and (2) exhibit diverse features. QD problems are characterized by two components: (1) an objective function to maximize f O + R. and (2) a descriptor function D S R". That descriptor function used to differentiate between solutions; takes as input a solution 0 E θ, and computes a descriptive low-dimensional feature vector.
ct al. 2 : Fontainc & Nikolaidis
d θ + D
The goal of QD algorithms is to return an archive of solutions A satisfying the following condition: for each achievable descriptor E D, there exists solution A, E A such that d(O,,c) and f(OA.c) maximizes the objective function with the same descriptor [f(0) 0 E θ A d(0) = c}. However, the descriptor space 'D is usually continuous. This would require storing an infinite amount of solutions in the set. QD algorithms
alleviate this problem by considering a tesselation of the descriptor space into cells (Mouret Clune 2015
Cully al. 2013 Vassiliades al. 2017): (cell:)iez and keeping only a single solution per cell. QD algorithms aim finding set of policy parameters (0,)şET maximizing the QD-Score Pugh et al. (0166) defined follows (where f(-) is assumed non-negative without loss of generality):
maximize (0))ses
QD-Score = I(0,, such that Vj E J. d(0,) E Ecelly
(1)
JEJ
Thus, maximizing the QD-Score is equivalent to maximizing the number of cells containing a policy from the archive, while also maximizing the objective function in each cell.
Quality-Diversity for Neurocvolationn-aseed RL. QD algorithms can also be used on RL problems, modeled as Markov Decision Processes As,,, where S is the states set, A denotes the set of actions, p is a probability transition function, and is state-dependent reward function. The return R is defined as the sum of rewards: R = E, rt- In a standard RL setting, the goal is to find policy To maximizing the expected return.
The QD for Reinforcement Learning (QD-RL) problem Nilsson & Cully 2021 Tjankka et al 2022 is
defined as QD problem in which the goal is find a set of diverse policy parameters A (= (0)))e leading to diverse high-performing behaviours (Fig. 2 In the QD-RL context, the objective function matches the expected return /(0,) E [R(())]; the descriptor function d(-) characterizes the state-trajeclory of policy
me, d(O,) E ())) (with denoting the state-trajectory $1:T). The QD-Score maximize (formula can then be expressed as follows, where all expected returns should be non-negative:
QD-Score = ΣE E [1(10:]
such that Vj E J, E [())( E cell,
maximize (0)bet
(2)
JEJ
4 Background: MAP-Elites
MAP-Elites Mouret & Clune (015)) a well-known QD algorithm which considers a descriptor space discretized into grid cells (Fig. 3 th column). At the start, an archive created and initialized by ovaluating random solutions. Then, at every subsequent iteration, MAP-Elites (i) generates new candidato solutions, (ii) evaluates their return and descriptor and (iii) attempls to add them to the archive A. The iterations done until reach a total budget И of evaluations. During step (i), solutions are selected uniformly from the archive A, and undergo variations obtain new batch of solutions B. In all our experiments, the iso-line variation operator (Vassiliades & Mouret 2018) (Appendix Algo.
4

---

Published in Transactions on Machine Learning Research (01/2023)
Then (ii), the solutions in the sampled batch B (0 are evaluated to obtain their respective returns (R(())) [[[NN and descriptors (d(0))) Finally (iii); each solution 0₃ E A is placed in its corresponding cell in the behavioural grid according to its descriptor d(Oz). If the cell is empty, the solution is added to the archive. If the cell is already occupied by another solution, the solution with the highest. return is kept, while the other is discarded. A pseudo-code for MAP-Elites is presented in Algo.
We use MAP-Elites to study and show how QD algorithms can be scaled through parallclization. We leave other variants and enhancements of QD algorithms which use learned descriptors (Cully 2019 Paolo et. al. 2020; Miao et al., 2022] and different optimization strategies such policy gradients (Nilsson & Cully 202 I
Pierrot et al. 2021 Tjanaka et al. 2022 and evolutionary strategies (Coola et al. 2020 Fontaine et al, 2020b) for future work; we expect these variants to only improve performance on tasks, and benefit from the same contributions and insights of this work.
5 Leveraging Hardware Acceleration for Quality-Diversity
In population-basel methods, new solutions are the result of older solutions that have under- gone variations throughout as described in Algo. 1 The number of itera- tions/ I of a method commonly depends on the total number of evaluations (i.e. computational budget) И for the optimization algorithm and the batch size Ng. For a fixed computation budget, a large batch size NB would result. in a lower number of iterations I and vice versa. At each iteration, a single solution can undergo a variation and each variation is learning step to converge towards an optimal solution. It
Algorithm MAP-Elites (Ng: Batch size)
iterative process
1:
for iteration E [1,7] do
2:
if first iteration then
3:
B F random solutions else
4:
5:
8- select solutions from archive A
6:
= =0,,) + VARLATION(
7:
for j E Na] do
8:
run episode of " get return R(0;) and traj. (0,))
9:
cell + get grid cell of descriptor (0,))
10:
fcell + get content of cell
11:
if feall is None then
12:
Add A to cell
13:
else if R((0)) R((ecu) then
follows that the number of iterations I defines
14:
Replace Qzell with 0y cell else
the maximum number of learning steps a solu-
15:
Discard 0,
tion can take. In the extreme case of Ng= = H, the method simply reduces to single random variation to a random sample of parameters.
return archive .A
AAyy variation operator can be used to obtain new solutions. We
use the iso-line variation (see Algo 2 Appendix. A.2
Based on this, an initial thought would be that the performance of population-basee methods would be negatively impacted by a hcavily reduced number of learning stops and require more itcrations to find good performing solutions. The iterations of the algorithm are a sequential operation and cannot be parallelized. However, the evaluation of solutions at each iteration can be massively parallelized by increasing the batch size NB which suits modern hardware accelerators. We investigate the effect of large Ng on population- basedd methods (more specifically QD algorithms) by ablating exponentially increasing Ng which was relatively unexplored in the literature.
Conventional QD algorithms parallelize evaluations by utilizing multiple CPU cores, where each CPU separately runs an instance of the simulation to evaluate a solution. For robotics experiments, we utilize Brax (Freeman et al. 202T), a differentiable physics engine in Python which enables massively parallel rigid body simulations. By leveraging a GPU/TPU, utilizing this simulator allows us to massively parallelize the evaluations in the QD loop which is the major bottleneck of QD algorithms. To provide sense of scale, QD algorithms normally run on the order of several dozens of evaluations in parallel with Ng ~ 102 due to limited CPUs while Brax can simulate over 10,000 solutions in parallel allowing QDax to have Ng ~ 10 Brax is built on top of the JAX (Bradbury et al. 2018 programming framework, which provides an API to run accelerated code across any number of hardware acceleration devices such as CPU, GPU or TPC.
Beyond massive parallclization, acceleration of our implementation is also enabled with code compatible with just-in-time (JIT) compilation and fully on-device computation. Another key benefit of JAX is the JIT compilation which allows JAX to make full use of its Accelerated Linear Algebra (XTA).. We provide
5

---

Published in Transactions on Machine Learning Research (01/2023)
implementation details with static archives that makes QDax compatible with JIT in the Appendix. Lastly, another bottleneck which slowed the algorithm down was the data transfer and marshalling across devices. To address this issue, we carefully consider data structures and place all of them on-device. QDax places the JIT-compiled QD algorithm components on the same device. This enables the entire QD algorithm to be run without interaction with the CPU.
A
B
C
A
D
E
Figure 2: QD algorithms can discover diverse solutions. A, B,C: Uni-directional Walker, Ant and Humanoid task respectively discover diverse gaits for moving forward, D: Omni-directional Ant task discovers ways in every direction, E: Ant Trap task with deceptive rewards.
6 Experiments
Our experiments aim to answer the following questions: (1) How does massive parallelization affect perfor- mance of Diverty- (MAP-Elites) algorithms? (2) How significant is the number of iterations /lenrnilennnng steps in QD algorithms? (3) What magnitude of speed-up does massive parallelization offer over existing implementations? (4) How does this differ across different hardware accelerators?
6.1 Domains
Rastrigin and Sphere. Rastrigin and Sphere are standard optimization functions commonly used as benchmark domains in optimization (Hansen et al. 2010 2021 and QD-liteaatuee (Fontaine et al., 2020b Fontaine & Nikolaidis 2021). We optimize More details regarding the objective function and descriptors are provided in the Appendix. We select these simple domains to demonstrate that the phenomena of large batch sizes applies generally to QD algorithms and not just to certain domains.
100 dimensional parameter space bounded between and
Planar Arm The planar arm is a simple low-dimensional control task used in QD literature (Cully et al. 2015 Vassiliades & Mouret] 2018 Fontaine & Nikolaidis 2021 where the goal is to find the inverse kinematic solution (joint positions) of planar robotic arm for every reachable position of the end effector. As the arm is redundant, the objective ff for each solution is minimize the variance of the joint angles (i.c. smooth solutions) while the descriptor corresponds to the z-y position of the end effector obtained via forward kinematics. For experiments,
7-DoF arm.

---

Published in Transactions on Machine Learning Research (01/2023)
Continuous Control QD-RL. We perform experiments on three different QD-RL benchmark tasks (uully
et al. 2015 Conti et al. 2018 Nilsson & Cully 2021 Tjanaka et al. 2022 omni-directionall robot locomotion, uni- oni-directional robot locomotion and a deceptive reward trap task. In the omni-directional task, the goal is to discover locomotion skills to move efficiently in every direction. descriptor functions is defined as the final z-y positions of the center of mass of the robot at the end of the episode while the objective / is defined as sum of a survival reward and torque cost. In contrast, the goal in the uni-directional tasks is to find a collection of diverse gaits to walk forward as fast as possible. In this task, the descriptor function is defined as the average time over the entire episode that each leg is in contact with the ground. For each foot i, the contact with the ground C₁ is logged as a Boolean (1: contact, O: no-contact) at each time step t. The descriptor function of this task was used to allow robots to recover quickly from mechanical damage (Cully et al. 2015). The objective f of this task is a sum of the forward velocity, survival reward and lorque cost. Full equation details can be found in the Appendix B In the Trap tasks, the environment contains a trap 2 right in front of the ant. The goal is to learn to move forward as fast as possible. If naively done using purely objective based algs the ant will get stuck in the trap. The objective I of this task is a sum of the forward velocity, survival reward and torque cost while the descriptor is the final z-y position of the robot at the end of the episode.
We use the Hopper, Walker2D, Ant and Humanoid gym locomotion enviromments made available on Brax (Freeman et al. 2021) on these tasks. In total, we report results on a combination of six tasks and Aut, Uni-directional Hopper, Walker, Ant and Humanoid, and Ant Trap. Fig. 2 illustrates examples of the types behaviors discovered from these tasks. We use fully connected neural network controllers with two hidden layers of size 64 and tanh output activation functions as policies across all QD-RL environments and tasks.
environments; Omni-direction
6.2 Effects of massive parallelization on QD algorithms
To evaluate the effect of the batchsize Ng. we run the algorithm for a fixed number of evaluations. We use 5 million evaluations for all QD-RT environments and 20 million ovaluations for rastrigin and sphere. We evaluate the performance of the different batch sizes using the QD-score. The QD-score (Pugh et all 2016 aims to capture both performance and diversity in a single metric. This metric is computed as the sum of objective values of all solutions in the archive (Eqn. []. We plot this metric with respect to three separate factors: number of evaluations, number of iterations and total runtime. Other commonly used metries in QD literature such as the best objective value and coverage can be found in the Appendix. We use single A100 GPU to perform our experiments.
Fig. g shows the performance curves for QD-score and more importantly the diffcrences when plot against the number of evaluations, iterations and total runtime. A key observation in the first column is that the metrics converge to the same final score after the fixed number of evaluations regardless of the batch size used. The Wilcoxon Rank-Sum Test for the final QD score across all the different batches results in p-values p>0.05 after applying the Bonferroni Correction. This shows that we do not observe statistically significant differences between the different batch sizes. Therefore, larger batch sizes and massive parallelism do not, negatively impact the final performance of the algorithm. However, an important observation is that the larger batch sizes have a trend to be slower in terms of number of evaluations to converge. This can be expected as a larger number of evaluations are performed per iteration at larger batch sizes. Conversely, in some cases (Ant-Uni), a larger batch size can even have a positive impact on the QD-score. Given this result, the third column of Fig. then demonstrates the substantial speed-up in total runtime of the algorithm obtained from using larger batch sizes with massive parallelism while obtaining the same performances. We can obtain similar results but in the order of minutes instead of hours. An expected observation we see when comparing the plots in the second and third column of Fig. B is how the total runtime is proportional to the number of iterations. As we are able to increase the evaluation throughput at each iteration through the parallelization, it takes a similar amounts of time to evaluate both smaller and larger batch sizes. The speed-up in total runtime of the algorithm by inereasing batch size eventually disappears as we reach the limitations of the hardware. This corresponds to the results presented in the the next section 6.3 (Fig. where we see the number eval/s plateauing.
7

---

Published in Transactions on Machine Learning Research (01/2023)
QD-score vs Evaluations
GD-Score lterations
SD-Sc vs Runtime
Final Archivee
06
400
Rastrig m
-5-
37-
000
130
15-
-30
5:2
3.1
1.20
Sphere
-20
0.48
348
3.22
500
535
20
504
334
-40
500
0.0
0.5
1.4
15
20
20000
6000
00300
00000
15
100
50
02
24
46
-0.025
Am
04
0.000
02
05
-0.50
105
-12
166
1.5
700
00
3.3
300
02
0
5000
30000
1900
20000
3000
4003
8000
0.
02
04
0.
3250
Humanaid
1.
750
02
230
08
1.55
Trra
200
256
344 1024
4096
16384
32768
65536
131072
Figure 3: Performance metrics across domains. The plots show the QD-Score against the number of evaluations, iterations and run-times for each batch size. The rightmost column shows the final archives. The Ant and Hopper Uni results are presented in the Appendix. The bold lines and shaded areas represent the median and interquartile range over 10 replications respectively.
8

---

Published in Transactions on Machine Learning Research (01/2023)
Batch Size
Black-box
Simple Control
Neuroevolution for QD-RL
Rastrigin
Sphere
Arm
HopperUni
WalkerUni
AntUIni
HumanoidUni
AntOmni
AntTrap
256
55025
65330
7761
185
8587
14951
11183
7272
1339
1,024
12720
16327
1770
63
1432
3412
3608
1690
562
4,096
2920
4086
451
15
254
734
470
400
119
16,384
832
1053
104
4
02
177
182
98
63
32,768
455
535
29
85
101
49
65.536
240
281
34
2
21
45
56
33
18
131,072
147
151
23
13
26
21
16
Table 1: Number of iterations needed to reach threshold QD-score (minimum QD-score across all batch sizes)
for the range of batch sizes. The medians over the 10 replications are reported.
The most surprising result is that the number of iterations and thus learning steps of the algorithm do not signifi- cantly affect the performance of the algorithm when increasing batch sizes NB are used (Fig. 3 second column). In the case of the QD-RL domains, we observe that using Ng = 131,072 which runs for a total of only I= 39 provides the similar performance as when run with NB 256 and I = 19,532. This is true for all the problem domains presented. To evaluate this more concretely, Table D shows the iterations needed to reach threshold QD-score. The threshold QD-score is the minimum QD-score reached across all the batch sizes. The results in Fig. | clearly show that larger batch sizes require significantly less iterations to achieve the same QD-score.
Given a fixed number of evaluations, a larger batch size would imply a lower number of iterations. This can also be observed in the second column of Fig. 3 Therefore, our results show that a larger batch size with less iterations has no negative impact on QD algorithms and can significantly speed-up the runtime of QD algorithms using parallelization for fixed evaluation budget. The Nierations/eeannnng steps remain important part of QD algorithms as new solutions that have been recently discovered and added to the archive A from previous iteration can be selected stepping stones to form good future solutions. This is particularly evident in more complex tasks (Ant Trap) where there is an exploration bottleneck in which it can be observed that large batch sizes struggle due to the lack of stepping stones from a low number of iterations. However, our experiments show that iterations are not as important as long as the number of evaluations remains identical. This is a critical observation as while iterations cannot be performed in parallel, evaluations can be, which enables the use of massive parallelization.
This is in contrast to other population- passed approaches such evolutionary strategies (Salimans et al. 2017]. where we observe that massively large batch sizes can be detrimental when we run similar batch size ablation experiments (see Appendix C). We hypothesize that the reason for this is because in conventional ovolutionary strategics (ES), there is a single optimization "thread" going on. The population is mainly used to provide an empirical approximation of the gradient. Increasing the size of the population helps to obtain a better estimate, but after a certain size, there are no benefits. On the contrary, with larger batch sizes, the algorithms will waste evaluations. Conversely, in Quality-Diversity algorithms (MAP-Elites in particular), each of the thousands cells of the container can be seen as an independent optimization "thread". This enables MAP-Elites to greatly benefit from extra-large batch sizes.
6.3 Evaluation throughput and runtime speed of QD algorithms
We also evaluate the effect that increasing batch sizes have on the evaluation throughput of QD algorithms. We start with a batch size Ng of 64 and double from this value until we reach a plateau and observe a drop in performance in throughput. In our experiments, a maximum batch size of 131,072 is used.
The number of evaluations per second (eval/s) is used to quantify this throughput. The eval/s metric is computed by running the algorithm for fixed number of generations (also referred to as iterations) N (100 in our experiments). We divide the corresponding batch size Ng representing the number of evaluations performed in this iteration and divide this value by the time it lakes to perform one iteration 4n- We use the final average value of this metric across the entire run: eval/s = N Σm= Nz. While the evaluations T, per sccond can be an indication of the improvement in throughput from this implementation, we ultimately care about running the entire algorithm faster. To do this, we evaluate the ability to speed up the total runtime of QD algorithms. In this case, we run the algorithm to fixed number of evaluations (1 million),
9

---

Published in Transactions on Machine Learning Research (01/2023)
Implementation
Simulator
Resources
Eval/a
Batch size
Runtime (s)
Batch size
QDx (Ours)
Brax
GPU A100
30,846 11,031
65,636
69
65,536
QDx (Ours)
Brax
GPU 2080
8,192
117
8,192
pyribs
PyBullet
32 CUU-coess
8,192
7,234
4,096
pymapelites
PyBullot
32 CUU-coees
185
8,192
6,509
16,384
Sferesy2
DART
32 CPU-cores
1,190
512
1,243
32,768
Table 2: Maximum throughput of evaluations per second and fastest run time obtained and their corresponding batch sizes across implementations. The medians over the 10 replications are reported.
usually done in QD literature. Running for fixed number of iterations would be unfair comparison the experiments with smaller batch sizes would have much less evaluations performed in total.
For this experiment, we consider the Ant Omni- directional task. We compare against common im- plementations of MAP- and open-source simula- tors which utilize parallelism across CPU threads (see Table All baseline algorithms used simulations with fixed number of timesteps (100 in our experi- ments). We compare against both Python and C++ implementations as basclines. Pymapclites (Mouret & Clune| 2015) is a simple reference implementa- tion from the authors of MAP-Elites that was made be casily transformed for individual purposes. Pyribs (Tjanaka al.) 2021) is a more recent QD op- timization library maintained by the authors of CMA-
Average Evalts
30
Ful Runtime
10a
10'
102
Batch Size
Batch Size
QDax-A100 (Ours)
pyribs pymapelites
sferes
QDax-2080 (Ours)
Figure 4: Average number of eval/s and full runtime of algorithm across batch sizes and implementations. Note the log scales on both axes to make distinction between batch sizes clearer.
ME (Fontaine et al. 2020b). In both Python imple- mentations, evaluations are parallelised on cach core using the multiprocessing Python package. Lastly, Sferesy2 (Mouret & Doncieux 2010) is an optimized, multi-core and wight C++ framework for evo- lutionary computation, which includes QD implementations. It relies on template- based programming to achieve optimized execution speeds. The multi-core distribution for parallel evaluations is handled by Intel Threading Building Blocks (TBB) library. For simulators, we PyBullet (Coumans & Bai 2016 2020) in our Python baselines and Dynamic Animation and Robotics Toolkit (DART) (Lee et al. 2018) for the C++ baseline. We recognize that this comparison is not perfect and there could exist more optimized combinations of implementations and simulators but believe the baselines selected gives a good representation of what is commonly used in most works Cully & Mouret (2013) Nilsson & Cully 2021
jjanaka. ct al. (2022)
We also test our implementation on two different GPU devices, a more accessible RTX2080 local device and perforr-performacce A100 on Google Cloud. We only consider a single GPU device at each time. QDax was also tested and can be used to perform experiments across distributed GPU and TPU devices but omit these results for simplicity.
Fig. (Left) clearly shows that QDax has the ability to scale to much larger batch sizes which results in a higher throughput of evaluations. It is important to note the log scale on both axes to appreciate the magnitude of the differences. For QDax implementations (blue and orange), the number evaluations per second scales as the batch size used increases. This value eventually plateaus once we reach the limit of the device. On the other hand, all the baseline implementations scales to a significantly lower extent. These results can be expected as evaluations using simulators which run CPUs are limited each CPU core runs separate instance of the simulation. Therefore, given only fixed number of CPU cores, these baselines would not scale as the batch size is increased. Scaling is only possible by incrcasing the number of CPUs used in parallel which is only possible with large distributed system with thousands of CPUs in the network. QDax can reach up to a maximum of 30,000 evaluations per second on an A100 GPU compared to maximum of 1,200 (C++) or 200 (Python) evaluations per second in the basclines (scc Table 2) This 30 100 times increase in throughput, turning computation on the order of days to minutes. The negligible differences between the pyribs (green) and pymapelites (red) results show that the major bottleneck is indeed
10

---

Published in Transactions on Machine Learning Research (01/2023)
the evaluations and simulator used, as both of these baselines use the PyBullet simulator. The performance of the Sferesv2 (purple) implementation can be attributed to its highly optimized C++ code. However, the same lack of scalability is also observed when the batchsize is increased. When looking at run time of the algorithm for a fixed number of ovaluations on Fig. ! (Right), we can see the effect of the larger throughput of evaluations at each iteration reflected in the decreasing run-time when larger batch sizes are used. We can run a QD algorithm with million evaluations in just slightly over a minute (See Table 2 when using a batch size of 65,536 compared to over 100 minutes taken by Python baselines.
Our results also show that this scaling through massive parallelism is only limited by the hardware available. The experiments on both the RTX2080 (orange) and A100 (blue) show similar trends and increases in both evaluations per second and total runtime. The 2080 plateaus at a batch size of 8,192 capable of 11,000 eval/s while the higher-end A100 plateaus later at a batch size of 65,536 completing 30,000 eval/s.
7 Limitations and Future Work
In this paper, we presented QDax, an implementation of MAP-Elites that utilizes massive parallelization on accelerators that reduce the runtime of QD algorithms to interactive timescales on the order of minutes instead of hours days. We evaluate QDax across range QD tasks and show that the performance of QD algorithms are maintained despite the significant. speed-up that comes with the massive parallelism. Despite the apparent importance of itcrations in QD algorithms, we show that when large batch sizes are used, a heavily reduced number of iteralions and hence learning steps, provides similar results thus greatly accelerating the runtime of these QD algorihtms. This is observed on all the QD problems we considered, ranging from black-box optimization problems to high-dimensional RL tasks.
Despite reaping the benefits of hardware in order to significantly accelerate the algorithm, there are some limitations that could arise in the future when scaling up this work. As the archive stores the parameters of the entire population among other things, the memory of the device becomes an issue preventing larger networks with more parameters and higher dimensional inputs from being used. This issue showed up at the 072 in the humanoid environment which had a significantly larger observation space
largest batch size of 131,072 of close to 300 dimensions. Similarly, this memory limitation also prevent larger archives with more cells from being used (i.e. larger population size). However, the experiments in this paper do not do use anything less than what is commonly used in QD literature.
QDax is a general framework and tool that is useful for accelerating population-base learning, including QD algorithms, over wide range of problem settings. From the code provided in the supplementary materials and the experiments shown, we demonstrate that QDax can be used for standard black-box function optimization tasks, simple low-dimensional control tasks and more complex nourocvolution tasks such as in QD-RL. Other than the MAP-Elites (ME) algorithm used in this paper, the QDax framework can be used to implement and accommodate more complex QD algorithms such as ME-ES (Colas et al.) 2020), PGA-ME NNissoon & Cully] 2021), CMA-ME (Fontaine et al. 2020b), Differentiable QD (Fontaine & Nikolaidis 2021 and more. However, while we expect the findings of the paper in terms of the relationship between the batch size and iterations to hold for these aigorttims, the time performance might suffer due to additional computation required such as gradient steps or matrix inversions by these more sophisticated algorithms.
Through this work, we hope the increased accessibility of QDax can help bring ideas from an emerging field of optimization to accelerate progress in machines learning. We also hope to see new algorithmic innovations that will leverage the massive parallelization to improve performance of QD algorithms.
Broader Impact Statement
Accessibility was a key consideration and motivation of this work. Our work turns the execution of QD algorithms from what took days/wecks on large CPU clusters to only minutes on single casily accessible and free cloud GPU/TPU. We hope that this will prevent limitations of access of these algorithms to wider and more diverse communities of people, particularly from emerging and developing economies and beyond only well-resoureed rescarch groups and enterpriscs. We believe this increase in accessibility has the potential to open up novel applications of QD in new domains. On the other hand, a key scientific takeaway from our work was also that QD algorithms were able to scale with more powerful and modern hardware through
11

---

Published in Transactions on Machine Learning Research (01/2023)
massive parallelization. We recognize that this could also open up the possibility of "buying' results (Schwartz et al. 2020 by utilizing more compute.
We are also aware that algorithms that can scale well with compute come at the risk of carbon footprint cost. Evident from the bloom of deep learning and more recently large language models, computational demands only continue to increase with models that scale. Interestingly, our work shows that given the same machine, we can decrease computation time by two orders of magnitude with no loss in performance of the algorithm which can aid in reducing environmental impact. Nonetheless, we highlight research the environmental and carbon impact of Al and recommendations to be accountable in minimizing its effects (Dobbe & Whittaker 2019: Dhar 2020). A common step towards reducing AI's climate impact is to increase transparency on energy consumption and carbon emissions from computational resources used. We report the estimated emissions used in our experiments in the Appendix. Estimations were conducted using the Machine Learning Impact calculator
Lacoste et al. 2019
Acknowledgmentt
This work was supported by the Engineering and Physical Sciences Research Council (EPSRC) grant EP EP/V006673/1 project REcoVEDS and by Google with GCP credits.
References
Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16), pp. 265-283, 2016. pdf
URL https://wwat usenix. httg:/syttto///lllee///oorrrrere
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Danny Gutfreund, Joshua Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. 2019.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+-NumPy programs, 2018. URL /t/pp/////gtttuu.../ooo/////////
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. arxiv preprint ar vrXic:2005.14165 2020.
Konstantinos Chatailygeroudis, Vassilis Vassiliades, and Japtisae Mouret, Reset-free trial-and-error learning for robot damage recovery. Robotics and Autonomous Systems, 100:236-222 2500, 2018
Konstantinos Chatzilygerouddis, Antoine Cully, Vassilis Vassiliades, and -can-Bap Mouret. Quality- diversity optimization: novel branch of stochastic optimization. In Black Boz Optimization, Machine Learning, and No-Free Lunch Theorems, pp. 109-135. Springer, 2021.
Dan C. Ciresan, Ueli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for image 1542
classification. CoRR, abs/ 120/22222.27 2012. URL
Jeff Clune. Ai-gas: Ai-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint arxiv:190s 1098, 2019.
Cédric Colas, Vashisht Madhavan, Joost Huizinga, and Jeff Clune. Scaling map-elites to deep neurocvolution.
In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, pp. 67-75, 2020.
Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: A matlab-like environment for machine learning. 2011. URL
12

---

Published in Transactions on Machine Learning Research (01/2023)
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via population of novelty- seeking agents. Advances in neural information processing systems, 31, 2018.
Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybuulet.. org.) 2016-2020.
Antoine Cully, Autonomous skill discovery with quality-diversity and unsupervised descriptors. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 81-89, 2019.
Antoine Cully. Multi-emitter map-elites: Improving quality, diversity and convergence speed with heteroge- neous sets of emitters. arXiv preprint arXiv:2007.05352, 2020.
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Compulation, 22(2):245 259, 2017.
Antoine Cully and Jean-Baptiste Mouret. Behavioral repertoire learning in robotics. In Proceedings of the 15th annual conference on Genetic and evolutionary computation, pp. 175-182, 2013.
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. Nature, 521(7553):500 507, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference compuler vision and pollern recognition, pp. 248 255. leee, 2009.
Jacob Devlin, Ming-Wa- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. orXiv preprint arXiv:1810.0440 04805, 2018.
Payal Dhar. The carbon impact of artificial intelligence. Nature Machine Intelligence, 2(8):423 425, 2020.
R. Dobbe and M. Whittaker. Ai and climate change: How they're connected, and what we can do about it. 2019.
Sam Earle, Justin Snider, Matthew C Fontaine, Stefanos Nikolaidis, and Julian Togelius. Illminatiig diverse neural cellular automata for level generation. arXiv preprint arXiv:2109. 05489, 2021.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune. First return, then explore. Nature, 590(7847):580 586, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.0607 2018.
Matthew Fontaine and Stefanos Nikolaidis. Differentiable quality diversity. Advances in Neural Information Processing Systems, 34, 2021.
Matthew C Fontaine, Scott Lee, Lisa Soros, Fernando de Mesentier Silva, Julian Togelius, and Amy K Hoover. Mapping hearthstone deck spaces through map-elites with sliding boundaries. In Proceedings of The Genetic and Evolutionary Computation Conference, pp. 161-169, 2019.
Matthew C Fontaine, Ruilin Liu, Ahmod Khalifa, Jignesh Modi, Julian Togelius, Amy K Hoover, and Stefanos Nikolaidis. Mummiattiig mario scenes in the latent space of generative adversarial network. aiv preprinl arXiv:2007.0561, 2020a.
Matthew Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K Hoover. Covariance matrix adaptation for the rapid illumination of behavior space. In Proceedings of the 2020 genetic and evolutionory computation conference, pp. 94-102, 2020b.
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax a differentiable physics engine for large scale rigid body simulation, 2021. URL http://github.com google/brax
13

---

Published in Transactions on Machine Learning Research (01/2023)
Adam Gaier, Alexander Asteroth, and Jean-Baptiste Mouret. Data-efficient design exploration through surrogate-assisted illumination. Beolutionary compulation, 26(3):381-410, 2018.
Daniele Gravina, Ahmed Khalifa, Antonios Liapis, Julian Togelius, and Georgios N Yannakakis. Procedural content generation through quality diversity, In 2019 IEEE Conference on Games (CoG), pp. 1 8. LEEE, 2019.
Shixiang Shane Gu, Manfred Diaz, Daniel C Freeman, Hiroki Furuta, Seyed Kamyar Seyed Ghasemipour, An- ton Raichuk, Byron David, Erik Frey, Erwin Coumans, and Olivier Bachem. Braxlines: Fast and interactive toolkit for rl-driven behavior engineering beyond reward maximization. arXiv preprint arXiv:21 10. 10.22.00..66000 04686, 2021.
Nikolaus Hansen, Anne Auger, Raymond Ros, Steffen Finck, and Petr Posik. Comparing Results of 31 Algorithms from the Black-Box Optimization Benchmarking BBOB-2009. In ACM-GECCO Genetic and Brolutionary Compulation Conference, Portland, United States, July 2010. URL https://hal
archives- frrssff///aa www.aaa//oa5//777 pp. 1689-1696.
Nikolaus Hansen, Anne Auger, Raymond Ros, Olaf Mersmann, Tea Tušar, and Dimo Brockhoff. Coco: A platform for comparing continuous optimizers in a black-box setting. Optimization Methods and Software, 36(1):114-144, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 98):11335-)780, 1997.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídck, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andrew J Ballard, Andrew Cowie, Bernardino Cooeeaa-Paredes Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Sebastian Bodenstein, David Silver, Oriol Vinyals, Andrew W Senior, Koray Kavukeuoglu, Pushmect Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583 589, 2021. doi: 10.
Rituraj Kaushik, Pierre Desreumaux, and Jean-Baptiste Mouret. Adaptive prior selection for repertoire-based online adaptation in robotics. Frontiers in Robotics and AI, 6:151, 2020.
Leon Keller, Daniel Tanneberg, Svenja Stark, and Jan Peters. Model-based quality-diversity search for efficient robot learning. Xiv preprint ar arXiv:2008.044 045889 2020.
Alex Krizhevsky, Nya Sutskever, and Geoffrey F. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097 1105, 2012.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. arXiv preprint arXie:1910.0999 097000 2019.
Yann Lecun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature 2015 521:7553, 521:436 444, 5 2015. ISSN 1476-4687. doi: no.ture/naturel 4539.. URL https::/www.naatt www.we...atr...mmm com/ htt.b//reccco/rrrrrcre////atrrss
Jeongseok Lee, Michael X. Grey, Sehoon Ha, Tobias Kunz, Sumit Jain, Yuting Ye, Siddhartha S. Srinivasa, Mike Stilman, and C. Karen Liu. DART: Dynamic animation and robotics toolkit. The Journal of Open Source Software, 3(22):500, Feb 2018. doi: 10.21105/Joss.006 005000. URL lnttps://doi. orgg 2010000/ 00500
Jocl Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Frolutionary computation, 19(2):189- 122-22,,,,2, 201 la.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Joel Lehman and Kenneth O Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the 19th annual conference on Genelic and evolulionary compulation, pp. 211-2222111111211 218, 2011b.
Bryan Lim, Luca Grillotti, Lorenzo Bernasconi, and Antoine Cully, Dynamics-awart quality-diversio for efficient learning of skill repertoires. arXiv preprint S1:::109.08522 2021.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning. CoRR, abs/2108.10470 2021. URL https://arxiv.org
abs/210 2108.100777
Jiayu Miao, Tianze Zhou, Kun Shao, Ming Zhou, Weinan Zhang, Jianye Hao, Yong Yu, and Jun Wang. Promoting quality and diversity in population-based reinforcement learning via hierarchical trajectory space exploration. In 2022 International Conference on Robotics and Automation (ICRA), pp. 7544 7550. IEEE, 2022.
J.BB. Mouret and S. Doncicux. SFERESv2: Evolvin' in the multi-core world. In Proc. of Congress on Evolulionary Compulation (CEC), pp. 4079 -086, 2010.
J-B Mouret and Stéphane Doncieux. Encouraging behavioral diversity in evolutionary robotics: An empirical study. Bvolutionary compulation, (00(1)91-133)) 2012.
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. ar arXiv:1504.04900 2015.
arXiv preprinl
Jean- Captiset Mouret and Stéphane Doncieux. Overcoming the bootstrap problem in evolutionary robotics using behavioral diversity. In 2009 IEFE Congress on Fvolutionary Compulation, pp. 1161-1168. IEFE, 2009.
Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 866-875, 2021.
Giuseppe Paolo, Alban Laflaquiere, Alexandre Coninx, and Stephane Doncieux. Unsupervised learning and exploration of reachable outcome space. In 2020 IEEE International Conference on Robotics and Aulomation (ICRA), pp. 2379 2385. IEEE, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adamn Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- serghppeffomancc deep learning library. CoRR, abs/1912.01703, 2019. URL httt:///axiiv http://arxii.or/rabb//aaa// abtg/22bs/191...... 01703
Thomas Pierrot, Valentin Macé, Geoffrey Cidcron, Karim Beguir, Antoine Cully, Olivier Sigand, and Nicolas Perrin. Diversity policy gradient for sample efficient quality-diversity optimization. 2021.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: new frontier for evolutionary computation. Frontiers in Robotics and AT, 3:40, 2016.
Rajat Raina, Anand Madhavan, and Andrew Y. Ng. Large-scale deep unsupervised learning using graphics processors. In Proceedings of the 26th Annual International Conference on Machine Learning. ICML '09, pp. New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: (l.1145/155374..0
URL http::///oi.
1145/1553374 1553486
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and paltern recognittin,, pp. 779 779-788,11 2016.
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. arxiv preprint arXiv:2109. 11978, 2021.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint or Xin: 101000000000 008.03864 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint WxXi::/077.03447 2017.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63 (12):54 63, 2020.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv: S1fi:1999.088 9999000000, 2019.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yaz- dani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train megatron-turing nlg 530b, large-scale generative language model, 2022.
Kenneth O Stanley. Why open-endedness matiers. Artificial life, 25(3):232 235, 2019.
Kenneth O Stanley, Joel Lehman, and Lisa Soros. Open-endedness: The last grand challenge you've 'e never heard of. While open-endedness could be a force for discovering intelligence, it could also be component of Al itself, 2017.
Dave Steinkrau, Patrice Y. Simard, and lan Buck. Using gpus for machine learning algorithms. In Proceedings of the Eighth International Conference on Document Analysis and Recognition, ICDAR '05, pp. 1115-1119, USA, 2005. TEEE Computer Society. ISBN 0769524206. doi: 10.1109//CIDAR22005555 URL
https://doi.org wttg//0.1199//0O. 1109/ICDAR. 20000 251
Bryon Tjanaka, Matthew C. Fontaine, Yulun Zhang, Sam Sommerer, Nathan Dennler, and Stefanos Nikolaidis. pyribs: A bare-bones python library for quality diversity optimization. httpss///giihhb..co//crrrosss pyribs 2021.
Bryon Tjanaka, Matthew Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradients for differentiable quality diversity in reinforcement learning. arXiv preprint 02t::22202006666, 2022.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Vassilis Vassiliades and Jean- Caan-Baptitte Mouret. Discovering the Elite Hypervolume by Leveraging Interspecies Correlation. In GECCO 2018 Genetic and Evolutionary Computation Conference, Kyoto, Japan, July 2018. doi: 00055522055555322555555
http://hhaaaaaaaaaaaaaaaaaaaaaaa httamiaafffhaa-01747775755 URL https: httphhhhl.niii..mmmm www.iiiiaaaar....
Vassilis Vassiliades, Konstantinos Chatzilygeroudds, and Jean-Bapliste Mouret. Using centroidal voronoi tessellations to scale up the multidimensional archive of phenotypic elites algorithm. IEEE Transactions on Evolutionary Computation,
(1))))636330 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advanees in neural information processing systems, pp. 5998 6008, 2017.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Poet: open-ended coevolution of environments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 142-151, 2019.
Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced poet: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning, pp. 9040-9900000 -9551. PMT.R, 2020.
16

---

Published in Transactions on Machine Learning Research (01/2023)
Yutong Wang, Ke Xue, and Chao Qian. Exolutionary diversity optimization with clustering- basedd selection for reinforcement, learning. In International Conference on Learning Represenlations, 2021.
Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. Natural evolution strategies. In 2008 IEFE Congress Euolutionary Compulation (IEEE World Congress Compulational Intelligence), pp- 3381-3387, 2008. doi: D0.1109/CEC.2008.000
17


Reviews:
[{"review_id": "uaL2uAfAfE", "reviewer": "Reviewer_xwDj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n\nThe observation of smaller batches and small iterations vs large iterations and smaller batches perhaps indicated that the sampling of even small batches is varied enough to preserve the convergence. In addition the divergence criteria can be strong enough so that the sampling for smaller batches can be still have high enough distinct candidate solutions. \n\nSoftware scalability\n\nWeakness\n\nI have an issue with the word batch being used here. In general while related to neural networks and in particular when related to neural network the batch represents a set of inputs fed to a network. The main specific point about the batch is that the output or the error form each of the inputs in the batch is then summed and averaged. After which a single error is propagated through the network. So one batch of N samples generates one error E. In the present work this watch confusing to me because the authors refers to batch of solutions but do not average their error or evolution parameters as averaged from all the samples in the batch. While for tracking purpose the QD-score is used it is not used in the algorithm directly. I think a different word should be used here. If this observation is not correct please provide a more detailed explanation of how the batch with a single evaluation is used for updating the solutions in the archive. \n\nThe difference between results requires more discussion. As the authors pointed out the results of large batches accelerate certain problems and slow down others. The acceleration of certain problems could mean the search space is small enough or that the landscape of the problem space is well suited for the QD algorithms. But this should be explained and analyzed in details. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper the authors introduce an study on a subclass of evolutionary algorithms, notably the Quality-Diversity optimizations. The study focuses on hardware acceleration, parallelization and quality related changes due to highly parallel executions. The paper is well written, with appropriate background and previous literature setup. The result of this paper shows an interesting result and that is because of the highly parallel nature of the QD optimization larger parallel problem segmentation can accelerate the solution convergence but without performance improvement. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Figure 3 is first referenced. Figures should be referenced in order of appearance in the manuscript. Please change this. In addition Figure 1 and 2 are never referenced. \n\n", "broader_impact_concerns": "There are no any ethical issues that should be of concern. In addition the authors present a proper broader impact section related to the environmental impact due to larger parallelization and thus more power requiring hardware components. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nJBcsRDD0l", "reviewer": "Reviewer_JCVC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The paper is well organized and clearly written with hypothesis outlined and experiments conducted to support the hypothesis.\n* The method and problem settings are generally well described and the results are generally well presented.\n* The insight provided, such as the one that MAP Elites can converge with less generations on larger batch sizes, is potentially valuable to other researcher in QD algorithms.\n\n**Weaknesses**\n\n* It's unclear how QDax might be used for other algorithms (i.e. not MAP Elites) or other problems settings, such as the ones not presented in this paper. It would have been nice to see a discussion of this in the limitations section.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a framework (QDax) for massively parallel evaluations for Quality-Diversity (QD) algorithms and investigates how massively parallel evaluations affect the performance of QD algorithms, both in performance and compute cost. The authors first introduce the broader goal of their method, which is to enable parallel fitness evaluations in QD algorithms leveraging parallel computing in GPUs and TPUs, and introduce related work pertaining to QD algorithms and hardware acceleration. Next, the authors describe the general problem setting of QD optimization and related metrics and outline the MAP Elites algorithm, which serves as the basis of their experiments. Subsequently, the authors describe their implementation of massively parallel implementation of MAP Elites by utilizing JAX based libraries for parallel fitness evaluations of different solutions.\n\nIn their experiments, the authors show how large batches of evaluations do not hinder, and in some cases potentially speed up, the convergence of MAP Elites. This is indicated by the smaller of generations required for convergence in the plots for Figure 3. Additionally, the authors how the wall clock time of experiments is significantly reduced using their method (Table 1) and discuss the limitations of their work as well as future work. The results and data presented generally confirm the authors' stated hypothesis.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add how new solutions $\\pi$ are created in Algorithm 1 for greater clarity.\n2. Add table summarizing batch size and generations to convergence related to the results in Figure 3. This would help present the data related to less generations required for MAP Elites convergence in a more cohesive way.\n3. A discussion on how portable the QDax framework is to other settings (algorithms and optimization problems).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PoxbUSrOuE", "reviewer": "Reviewer_Lvqf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Overall the paper is clear and easy to read and understand. It is well structured and nicely illustrated. \n- The experimental evaluation demonstrates the acceleration of the proposed methods and the source code is released, which is good.\n- The observation that massive parallelism does not affect the final performance of the algorithm when given the fixed number of evaluations is interesting, which means that linear acceleration is possible within the limit of device.\n\n**Weaknesses**\n\nThe contribution of the paper is limited. There have been previous works on parallelism of the QD algorithm which used simulators running on CPUs, while this work applies a simulator from reinforcement learning running on GPUs/TPUs named Brax to the QD algorithm, which was not proposed by the authors but only applied, causing limited contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "To speed up runtime of QD algorithms, this paper parallelizes the evaluation of solutions, which is performed in physical simulators when using QD in the field of Reinforcement Learning (RL) for robotics. Unlike previous QD parallel frameworks that use simulators that run on CPUs, this article proposes QDax, which uses simulators that run on GPUs/TPUs, allowing the QD algorithm to take advantage of the hardware acceleration for massive parallelism. The authors also observe through several QD-RL tasks that larger batch sizes and massive parallelism do not negatively impact the final performance of the algorithm when given the fixed number of evaluations, which is interesting.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I think conducting experiments on more tasks would make the results more convincing, thus strengthens the work.\n\n- The paper misses some recent work in the QD-RL field, such as [1] and [2], adding the reference of which will also strengthen the paper.\n\n[1] Evolutionary Diversity Optimization with Clustering-based Selection for Reinforcement Learning. In: ICLR, 2022.\n[2 Promoting Quality and Diversity in Population-based Reinforcement Learning via Hierarchical Trajectory Space Exploration. In: ICRA, 2022.\n", "broader_impact_concerns": "The authors have made a comprehensive statement in the Broader Impact Statement section.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents, QDax, which leverges massive parallelism on accelerators to speed Quality-Diversity (QD) algorithms, and contributes an open source library for parallelizing QD on a single machine. All reviewers agree, and I concur, that the paper would be a welcome contribution to the TMLR. The authors addressed all the reviewers comments.
Decision: Accept (poster)