Quality-Similar Diversity via Population Based Reinforcement Learning
Authors: Shuang Wu
Jian Yao
Haobo Fu
Ye Tian
Chao Qian
Yaodong Yang
QIANG FU
Yang Wei
Venue: ICLR 2023 poster (2023)
Keywords: quality diversity
reinforcement learning
user-defined
population

Abstract: Diversity is a growing research topic in Reinforcement Learning (RL). Previous research on diversity has mainly focused on promoting diversity to encourage exploration and thereby improve quality (the cumulative reward), maximizing diversity subject to quality constraints, or jointly maximizing quality and diversity, known as the quality-diversity problem. In this work, we present the quality-similar diversity problem that features diversity among policies of similar qualities. In contrast to task-agnostic diversity, we focus on task-specific diversity defined by a set of user-specified Behavior Descriptors (BDs). A BD is a scalar function of a trajectory (e.g., the fire action rate for an Atari game), which delivers the type of diversity the user prefers. To derive the gradient of the user-specified diversity with respect to a policy, which is not trivially available, we introduce a set of BD estimators and connect it with the classical policy gradient theorem. Based on the diversity gradient, we develop a population-based RL algorithm to adaptively and efficiently optimize the population diversity at multiple quality levels throughout training. Extensive results on MuJoCo and Atari demonstrate that our algorithm significantly outperforms previous methods in terms of generating user-specified diverse policies across different quality levels.

Published as a conference paper at ICLR 2023
QUALITY-SIMILAN DIVERSITY VIA POPULATION BASED REINFORCEMENT LEARNING
Shuang Wu' Jian Yao'r, Haobo Fu' Ye Tian', Chao Qian", Yaodong Yang²: Qiang Fu?, Wei Yang*
Tencent AT Lab, Shenzhen, China State Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China SPeking University, Beijing, China
ABSTRACT
Diversity is growing research topic in Reinforcement Learning (RL). Previous research on diversity has mainly focused on promoting diversity to encourage exploration and thereby improve quality (the cumulative reward), maximizing diversity subject to quality constraints, or jointly maximizing quality and diversity, known as the quality-diversity problem. In this work, we present the quality-similet diversity problem that features diversity among policies of similar qualities. In contrast task- ausk-agnostic diversity, we focus on task task-specific diversity defined by a set of user-specified Behavior Descriptors (BDs). A BD is scalar function of a trajectory (e.g., the fire action rate for an Atari game), which delivers the type of diversity the user prefers. To derive the gradient of the user-specified diversity with respect to a policy, which is not trivially available, we introduce a set of BD estimators and connect it with the classical policy gradient theorem. Based the diversity gradient, we develop a population- oopulation-based RL algorithm to adaptively and efficiently optimize the population diversity at multiple quality levels throughout training. Extensive results on MuloCo and Atari demonstrate that our al algorithm significantly outperforms previous methods in terms of generating user-specified diverse policies across different quality levels (see Atari and MuJoCo videos).
INTRODUCTION
Existing research on policy diversity in deep Reinforcement Learning (RL) can be generally divided
into three categories, according to the role diversity plays. The first category (Hong et al. 2018
Eysenbach et al.
2018 Conti et al.) 2018 Parker-Holder al. 2020 Kumar et al.
2020
Peng et al.
2020] Tang et al. 2020 Han & Sung [221]] [henghhao et al.) 2021 McKee et al.
2022 focuses on
maximizing the final quality (the cumulative reward) of policy, and policy diversity only serves as a means to better fulfill this goal via improving the efficiency of exploration. Therefore, the diversity measure is preferred to be aask-agnostic as the knowledge of what type of task -secifcc diversity benefits the quality may not be accessible in most cases. The second category (Masood Doshi- Velez 2019 Zhang et al., 2019; Sun al., 2020; Ghasemi al.) [02T] Zahavy al.
2021 et al. 2022) is concerned with constrained optimization problems, where either diversity is optimized subject to quality constraints or vice-versa. Again, existing methods this category task-agnostic diversity, thereby the obtained diversity is often explained in hindsight, i.c., it is unknown what type of policy diversity to expect until the optimization is finished. The third category optimizes quality and diversity simultaneously. which is usually known as the -iality-Diversity (QD) method (Cully et al., 2015 2015) Mouret & Clune 2013 (Pugh et al. 2016))
have mainly focused
Colas et al. 2020: Fontaine & Nikolaidis 2021] Nilsson & Cully] 2021 Pierrot et al. 2022 Wang
et al.] 2021 Tjanaka et al. 2022). In contrast to task-agnostic diversity, most QD methods focus on task-specific diversity, where users are allowed to specify set of interested Behavior Descriptors (BDs). A BD is a scalar function of trajectory (i.e., the whole game episode) and thus does not have an analytical function form with respect to a single policy or state. Therefore, the gradient of a BD with respect to policy is not trivially available, and this extends to the diversity measure
"Equal contribution. Correspondence to: Haobo Fu wtabboff@@eencet.comm

---

Published as a conference paper at ICLR 2023
defined on multiple BDs. As a result, previous QD methods (Cully et al. 2015 Mouret & Clune
Pugh et al., 2016) rely on black-box optimization techniques, such as evolutionary algorithms.
to evolve a population of diverse policies. Some recent QD methods (Colas et al. 2020 Fontaine
& Nikolaidis 2021 Nilsson & Cully 2021 Pierrot et al. 2022; Tjanaka et al. 2022) try to inject
gradient information into the evolutionary optimization process.
In this work, we formulate the Quality-Simile Diversity (QSD) problem where the objective is to produce set of diverse policies at multiple quality levels. We propose new QD metric called the QSD score that clusters policies of similar qualities, and the diversity is evaluated at each quality level. In QSD problems, diverse policies of non-optimal qualities are also preferred, which directly meet practical needs in some real-world Al applications. For example, in the field of game AI (Zhang et al. 2021 Fu et al. 2021). it is often desirable to provide diverse accompanying Als whose qualities are matched to a beginner, an amateur, and master, respectively. Besides, measuring the diversity between a beginner and a master would be of little interest. The QSD problem also connects with adaptive curricula (Wang et al., 2019 Team et al.) 2021 Parker-Holder et al. 2022), where the environment gradually increases curriculum fumic levels from simple to complex. Optimizing the intermediate diversity non-optimal quality levels helps faster and better convergence of the agent's 's capabilities than training directly at a complex curriculum level.
Moreover, the ability to generate task specifc diversity is superior and supplementary to task-agnostic diversity when the user has a clear preference for the type of diversity in practice. For example, diverse hand gestures
of no interest if the user only needs gait diversity in robot locomotion tasks. Hence, in this work, we optimize an explicit diversity measure function defined on several user-specified BDs, as opposed to the non-differentiable cell coverage percentage in most QD methods. To the best of our knowledge, none of existing methods has obtained the exact gradient of a user-specified BD (defined on trajectories) with respect to a policy, nor has any derived an unbiased estimation of this gradient using state-action samples. In particular, the diversity gradient is approximated by generating samples in the policy parameter space (Colas et al.. 2020; Taanaka et al., 2022). or simply assumed in Fontaine & Nikolaidis (2021), which might not hold in many real-world situations. A set
of "state" BDs (essentially a type of intrinsic reward) are introduced in Pierrot et al. 2022). expecting that a positive correlation between state and trajectory BDs might suffice.
To fill this gap, we propose a set of BD estimators that predict the corresponding BD value for the current policy. Equipped with these BD estimators, we build the policy gradient theorem (Sutton jet al. 1999; [Silver et al.] 2014) to derive the gradient of user-specified BDs with respect to a policy for discrete continuous actions. Based the population-based training (PBT) (Jaderberg et al.) 2017), we develop an RI diversity algorithm, named QSD-PBT, that leverages diversity gradient and adaptively adjusts diversity loss to preserve similar qualities of the population. QSD-PBT efficiently optimizes the diversity of multiple quality levels in a single run and outperforms previous methods in terms of the QSD score in both MuJoCo and Atari environments. Meanwhile, QSD-PBT demonstrates strong abilities in achieving user-specified diversity by discovering visually distinct policies across a variety of environments. To summarize, the contributions of this work are as follows:
We formulate the Quality-Similat Diversity (QSD) problem and propose a new performance metric.
We derive the gradient of user-specified BDs defined on trajectories with respect to a policy.
We develop a population-base RL algorithm that efficiently optimizes the diversity of multiple quality levels in a single run.
PROBLEM DEFINITION
We focus the episodic Markov Decision Processes (MDPs), which can be defined by a tuple (S, A,T, TTr, 7). S and A stand for the state space and action space respectively. T S x A + S is the environment transition function, and r S x A + IR is the expected reward function. policy o (s)) maps a state to a probability distribution over AA. A trajectory 7 is a state-action sequence [Sp. ag. $1, a1, .... srl, which is obtained by executing a policy from the initial step 0 to the terminal step T in the environment. The objective of RL is to find policy π that maximizes its expected cumulative rewards (also known as the quality in this work: J(x) = (E===[[((]]] where R(T) = E 7rr(se, az) is the return of a trajectory, and E [0, 1] is the discount factor. The state value function (s) E " at) = =] measures the quality following from state
2

---

Published as a conference paper at ICLR 2023
In particular, we define (T(sT) = 0. The Q-function QT = E| rsss ar)ls 8, a2 a] measures the quality following л after taking action a in state s.
Behavior descriptor. We would like to maximize a task-specific diversity measure at different quality levels in this work. Hence, users are allowed to specify a set of BDs, (T( (1 S i i L) that reflect the type of interested policy diversity. (T( is evaluation function of trajectory that is finite and easy to be implemented. For instance, a BD could be the ratio between left and right movements in the trajectory of Atari MsPacMan. The trajectory BD is a general form and can be simplified to state or action BD when the user is only interested in certain states or actions of the environment, e.g., the terminal position MsPacMan. Accordingly, the BD value of policy me (parameterized using 0) is defined as (Ta) = En We denote all the BD values of policy To by ( (o)
(TT), B2( (To)].
Diversity measure function. Since BD values of policy form an L-dimensional vector, we need function that measures the overall diversity of the population as scalar: Div(II) := [(B(TT. ), ..., B)) where II { [1 < j S N denotes set of policies. The choice of the f should follow these properties: (1) f should be bounded and non-negative for easy implementation; (2) Since we do not define the order of an agent in the population, J should be invariant of any permutation of the policies; (3) f should be differentiable such that we can derive its gradient. Two recommended measure functions detailed in Section C.2] the mean of all wair-wise euclidean distances, and the Determinantal Point Process (DPP) (Parker- et al. 2020).
Quality diversity score. Note that the quality of a policy for task is usually real-valued and oie-iinesio obtaining a diverse set of policies at every possible quality level would require an infinite number of policies. Hence, we approximately evaluate an algorithm's 's QSD performance by partitioning the obtained quality range into M disjoint intervals, and only the diversity of policies within the same quality interval is evaluated by f. The QSD score is defined as follows:
M
QSD score =
(iv))
where I. denotes the of policies obtained throughout training with qualities in the m-th interval.
POPULATION-BASED RL FOR QUALITY-SIMILA DIVERSITY
The gradient of the user-specified BD with respect to policy is not tractable since it is defined trajectory. Hence, a possible solution to our QSD problem would be adapting derivative- methods, such conventional QD methods (Mouret & Clune 2015). However, derivative-free methods often scale poorly with large-scale neural networks that are necessary to handle complex inputs, such as the image feature in Atari. Note that the form in our definition of user-specified BD is similar to the quality, it would be feasible to directly derive the diversity gradient using the policy gradient theorem (Sutton et al. 1999). Given the diversity gradient, there exist two choices how to obtain
diverse population: sequential training (Zhang et al. 2 et al. 2021: Zhou et al. 2022) and
population- based training (Jaeeberg al. 2017 Jung al.] 2019; Parker-Holder Holder al. 2020). The
population- based training is more appropriate in our case, because the diversity measure is defined on a population that can not be factorized into incremental diversity settings as Zhang et al. (2019) Zahavy et al. (2021) I): Zhou et al.) (2022). Based on the above discussion, we develop an efficient population-bast based RL algorithm, named QSD-PBT, for optimizing user-specified diversity BDs across different quality levels. Each component of QSD-PBT is elaborated in the following.
3.1 DERIVING THE DIVERSITY GRADIENT
Using the chain rule, we can write the gradient of the diversity of a population II= with respect to the policy 0, (without loss of generality, we use 0 hereafter) as:
= (ma, (1 I S N)
@Dv 00
0f
(B,
(B,(TT)
00
1=1
The partial derivative of f with respect to (() is easily obtained, as long as f is an explicit diversity measure function, such as the mean pair-wise distance or the determinant in DPP. Note
3

---

Published as a conference paper at ICLR 2023
that in (TT) E, (T)] (T) is scalar function evaluating the i-th user-specified BD with respect to a trajectory т. Following the policy gradient theorem (Sutton et al. 1999), we have:
(
I bi(r)Vo log (o((a)))))
= ET
00
The gradient in Equation is preferred be estimated by samples [st. oe, st+1, (r) in practice. To this end, a set of state and state-action BD estimators parameterized by Ø; are introduced:
B; (To.e-1, 863 0b) T mo [b)(
(4)
QB 1,, ax; Φi) ET+ [b)(
(5)
where The denotes segment of a trajectory starting from time step to j: Tajj [st, a, .... Sy, az). It
worth noting that (T) can not be factorized into sum of quantities at each time step. For this reason,
VB St) at state S depends on the whole historical state-action sequence T0:-11 Similar
to the advantage in RL, we define the BD advantage Ag, (To--1, St, a1) = Q (To.-1. ae)
VB. St). Since VB, (To:-1, st) is aetion-independent the gradient in Equation can be estimated using a mini-batch of samples {(Tor-1)> St ,at A ) as:
(B,(TT)
K
1 Vo log To (k) ( (K)),
00
K
where the sampled BD advantage A can be estimated using the conventional methods, e-g.. B:
Generalized Advantage Estimator (GAE) (Schulman et al., 2016), Direct Advantage Estimation (Pan al. (02I). The above derives the diversity gradient for the discrete-action policy. Based on the deterministic policy gradient theorem (Silver et al.) 2014), have the following proposition for deterministic policy (T())) in continuous action space.
Proposition 1. In deterministic policy and continuous action space case, the derivative of (r with respect to policy parameters 0 is:
(Bi(TO) 00
Σ
( p(so 81) VoNo V QT, (To:-1,, S1, ae)la so. dso:t. (S..
(=1)
B0st
where Jan. and dsoa are short for Jon --- and dsgdsı...ds respectively, and pls; - {5)
p(s;) I p(s+ (k (0(sk))- Using mini natc- of samples (Tmple 8t (k) (k) } we can
develop an unbiased estimator of the gradient above:
(Bi(TT) 00
K
V P
To B.
(k) (8) (02-11 84 ( (000
(8)
K
k=1
The proof of proposition Lis in Appendix A. Based on the proposition, at each training step, we sample mini--batch {(T0r) (b) a, 1== to estimate the gradient. It is worth noting that the trajectory 70:-11 may be a quite long sequence, e.g., more than 10k state-action pairs in Atari games. Hence, a feature extractor needed encode the trajectory. As for training BD estimators, We use the final BD value of a trajectory as the target and apply the mean squared error loss function. More implementation details are provided in Appendix C.4
3.2 OVERALL TRAINING SCHEME
In the QSD problem, we focus on maximizing the diversity at different quality levels. Previous methods applied to QD problems such as the constrained optimization subject to some quality constraints are inconsistent with our objective. Another possible solution is combine the quality loss and the diversity loss with a coefficient and adjust the coefficient using some online learning algorithms, such as bandits Marker-Holder et al.) (2020). However, it is not obvious what the online learning objective should be and how to adapt the coefficient optimally. To solve the QSD problem, quality level.
start from dealing with the sub-problem of QSD, i.e., maximizing diversity
4

---

Published as a conference paper at ICLR 2023
Optimizing diversity at one quality level. Considering population with N policies, the diversity and the average quality of the population are (II( and N Σ J(Ta,) respectively. Each sub- problem corresponds to optimizing combined quality loss and diversity loss with fixed coefficient X, saying the target weight A Instead of training with Anc from start to end, we let À start from a large initial value A and gradually decay to Aso the end of training. Specifically, at each training step t, the combined loss for the population is (IT)
N ((o,)) XTIiv(II), where we require A(+1 X and lim 0 Ase. This is motivated by a general observation that exploration is important in population-basee RL training, and a larger X focuses more on the diversity and thus helps exploration. Apart from encouraging exploration throughout training, our decaying method preserves the convergence property under some assumptions, which is proved in the following.
Proposition 2. Let f(x) : R" + R. g(x) R" - R be Lipschitz smooth, convex, bounded functions with derivative bounded, and a positive decreasing series { converges to X. Denote hy(x)= = [(x) + >zg(z)) Then using the gradient descent algorithm and choosing a proper step size, the algorithm will converge to the global minimum of (xx + >g(x))
The proof is in Appendix A.2 We can see that the decaying method converges in convex and Lipschitz smooth cases. In non-convex cases, we validate it experimentally in Section and Appendix [.I
Preserving the quality similarity. Constraining only the average quality of the population at certain levels does not guarantee similar qualities in the population. To encourage quality similarity among agents in a population, we distribute A, to each agent in the population, denoted as x()) for the j-th agent, and adapt each 2!" during training. The loss function for the j-th agent (policy) is I, (no,) = (m),) X ( DiV( Specifically, policies with better qualities should focus more on diversity optimization, and vice-versa. Following the discussion above, the scheme of x,5) throughout the training process is designed as x!5) = Ao + (Au - )xx)exp(-E max, n(5)
) where Xo is an R
initial coefficient. to indicates the preferred decay step, and f is the current training step. R2(") is the evaluation of the current quality of the j-th policy in the population. We assume that the quality of any policy is non-negative and upper bounded. As a result, we have lim,- A
Aoo-
Optimizing diversity across multiple quality levels. Once target Ax is designated, the objective function (TI) corresponds to maximizing diversity at a certain quality level. Since our goal in QSD is obtain policy diversity multiple quality levels, a straightforward way is optimize the objective function (IT) multiple times with a set of target weights (A h==1 Alternatively, we could obtain policy diversity at multiple quality levels in single run. Given two target weights Aoo,1 >> A when we have finished the optimization with A we can continue the optimization from Ao 1 to solve the quality level targeted by 2 rather than re-initializing the model and training from the initial Ag. This can be extended to multiple target weights, if À is decayed slowly enough from a large value Ao to 0. practice, we find much more efficient than independent training with h=1- As result, we apply single training with À decaying from À to 0 to integrally solve the QSD problem, which gives the overall training loss for the j-th policy as
multiple target values (Aoc. )
_.)( = (T(,, X Div(II),
with X150 = Ao exp( max, R to
We employ PPO (Schulman et al. 2017 as the backbone
for discrete actions. As for scenarios with continuous action space, the TD3 (Fujimoto et al. 2018) algorithm is applied as the backbone. We term our diversity optimization along with population-basee RL backbones as QSD-PBT. The pseudocode is given in Appendix C.5 and the chdee is open-sourced.
4 EXPERIMENTS
The effectiveness of QSD-PBT is validated on both MuJoCo (Brockman et al. 2016) continuous control tasks and Atari games (Bellemare et al. 2013) with discrete action spaces. For the diversity measure, we use the mean of pisr Euclidean distance between the BD vectors of two policies. For the specification of BDs, built-in joint torques as BDs, similar to Barker-Holdde et al. (2020). For Atari games, we estimate the advantage by the widely-adopted GAE method and follow the PPO Schulman et al. (2017)
follow common practices in MuJoCo tasks by incorporating the
5

---

Published as a conference paper at ICLR 2023
implementation that normalizes the advantage to make the training more robust. We design general- proposed BDs from the perspective of human players for the five Atari games. It is worth noting that the design of all the BDs does not unfairly favor any particular algorithm. Hence, the comparison among different methods is fair and unbiased (also illustrated in Appendix [B.3). We compare
QSD-PBT with independent PPO (Schulman et al.) (017) (driven only by the quality). two QD-style
methods: QD-PG (Pierrot et al., 2022) (diversity gradient by 'state" BDs) and EDO-CS (Wang al. (02T) (diversity gradient by evolution strategies), and two population-based RL algorithms: PBT (Jadeberg et al., (017) (driven only by the quality) and DvD (Parker- et al. 2020) (driven
by both the quality
quattllity and task-agnostic diversity).
For each environment, the range of quality is firstly estimated by training a state-of th-tte- quality- driven method (TD3 for MuJoCo tasks and PPO for Atari games) and then partitioned into M = 10 disjoint intervals of equal scope. The highest quality Rmaz achieved by TD3 or PPO is considered as the 'optimal' quality and shared by all the comparing methods. To calculate the QSD score for each method, we save a number of candidate policies within each quality interval during training. The time overhead for each method is estimated by the number training steps till the average quality of the population reaches a near-optimal quality (0.9Rmax). Error bars plotted in the figures or standard deviations presented in the tables are obtained using 5 independent runs. The population size N is 8 for MuloCo experiments and 10 for Atari experiments. Other implementation details are presented in Appendix C Additional results that demonstrate the versatility of QSD-PBT for different user-specified BDs and diversity measure functions are included in Appendix B.3 and B.1.2
4.1 MuJoCo TASKS
Three tasks from the MuJoCo environment selected: Humanoid, Hopper, and Walker2d. We define two types of BDs: (1) Scoring speed is the final score divided by the number of time steps used, indicating how fast the agent scores. (2) Built-in joint torques are the average torques applied to the hinge joints over a trajectory, which have a dimension of 17, 3, and respectively.
Table The QSD scores on MuloCo tasks. #step denotes the time overhead of each method.
MuloCo Tasks
QD-PG
EDO-CS
PBT
DvD-TD3
QSD-PBT (Ours)
score 0%
1.78 + 0.12
1.56 1 0.13
1.20 + 0.20
3.02 + 0.10
3.45 + 0.07
Humanoid
score 60%
0.40 + 0.02
0.62 + 0.03
0.30 + 0.15
1.13 + 0.02
1.39 + 0.03
Rmox: 5.5k
#step (k)
760 + 160
782 176
120 1
107 +
90 +
score 0%
0.59 + 0.01
0.72 0.09
0.63 + 0.03
0.80 + 0.02
1.26 0.08
Hopper
score > 60%
0.08 + 0.01
0.18 + 0.02
0.17 + 0.01
0.24 + 0.01
0.26 + 0.01
Rmax: 3k
#step (k)
222 31
200 + 27
111 7
208 27
1.0 +: 19
score > 0%
0.91
0.86 + 0.10
0.78 + 0.03
1.68 + 0.16
2.28 + 0.05
Walker2d
score 60%
0.07+0001
0.13 0.06
0.19 + 0.01
0.47 + 0.04
0.79 0.02
Rmani 5k
#step (k) 350 +
471 + 102
155 =
191 +
186 + 14
Table T shows each method's QSD score, where we also report the QSD score calculated using only intervals with high quality (above 60% of Rmarx) PBT gets the lowest 3 out of 6 QSD scores, indicating that methods considering only quality result in considerable degradation of diversity. QD-PG gets trapped in terms of quality, which results in low QSD score. EDO-CS performs slightly better than QD-PG, which is consistent with the experimental results in Wang et al. (202T). QD-PG and EDO-CS are much more time-consuming than other RL-style methods. The performance of DDD-DDDD (Parker-Holder et al., (020) comes as the second best, and QSD-PBT consistently achieves the highest QSD scores. Similar conclusions can be arrived in Figure where we plot the quality-diversity curves throughout training. One important reason for the superiority of QSD-PBT is the unbiased diversity gradient with respect to the user-specified diversity and the estimation using samples in the state-action space, while other methods are either biased (QD-PG using "state' BDs and DvD-TD3 using task- aask-agnostic BDs) or estimated using samples in the policy parameter space (EDO-CS). Another reason is that the diversity is better exploited within each quality interval due to the adaptive diversity loss, which is further studied in Appendix B.T Additional illustration of the diversity different quality levels is in Figure 8

---

Published as a conference paper at ICLR 2023
Humanaid
Hopper
Waler2d
POT Dud Da
PGT Dud0 100
PER DIOTDD
Ghuen
0.15
0.15
3d00 Cuultt
3003
1038
20.0
Figure 1: The quality similar diversity across 10 quality intervals on MuJoCo tasks.
4.2
ATARI GAMES
We select five Atari games. MsPacMan and RiverRaid are PvE (i.c., single-agent) games where players obtain scores by achieving pre-designed goals. FishingDerby, IceHockey, and Boxing are PvP (i.e., multiagent) games where the final score is the score difference between the player and a built-in AI competitor. Five
BDs are constructed from the perspective of human players: (2) Fire rate is the frequency of fire action in episode. (3) Action continuity measures the continuity of agent's actions by counting the number of action changes, which is actually an Al version of APM (actions per minute, widely applied to human players). For example, this BD for the action sequence -oap---o--lfe--ff--ooooo is 2. (4) Two orientation BDs (Ieft right and up (own) measure the preference of moving directions.
(1) Game time is the number of time steps in an episode.
Table 2:
The QSD scores on Atari games. #step denotes the time overhead of each method.
Atari Games
PPO
PBT
DvD-PPO
QSD-PBT (Ours)
score > 0%
1.28 + 0.13
1.14 + 0.09
1,40 + 0.05
2.47 + 0.06
MsPacMan
score 60%
0.53 + 0.06
0.42 0.07
0.56 + 0.04
1.02 + 0.04
Rmax: 15k
#step (k)
456 + 86
269 +
415 + 52
902 + 108
PyF
score 0%
0.92 0.12
0.84 + 0.08
1.32 0.12
2.66 0.13
RiverRaid
score > 60%
0.43 + 0.08
0.38 + 0.05
0.60 = 0.05
1.32 + 0.04
Rmax: 15k
#step (k)
204 31
16
352 144
734 183
score > 0%
1.60 0.17
1.55
2.32 = 0.08
3.64 + 0.12
FishingDerby
score 60%
0.78 0.09
0.76 0.06
1.01 = 0.06
1.68 0.05
Rmass: 48
#step (k)
262 + 138
1
168 + 87
242 + 25
score 0%
1.68 0.16
1.14
2.74 0.30
4.81 0.27
IeeHockey
PvP
score > 60%
0.81 0.05
0.00
1.13 + 0.12
1.92 + 0.08
Rmes: 40
#step (k)
S51 1 100
520
443 42
404 1 61
score > 0%
0.56 0.08
0.69 0.07
0.92 = 0.28
1,28 1 0.17
Boxing
score 60%
0.24 - 0.07
0.05
0.40 = 0.13
0.59 0.09
Rmar: 100
#step (k)
24 +
23 21
2
27 : 3
In Atari games we find that the quality of QD-PG and EDO-CS algorithms reach much lower levels than other PPO-based methods, therefore we have not included them for comparison and introduced PPO baseline. The results are summarized in Table and Figure 2 most of which are consistent with that in the MuJoCo experiment. The exploitation strategy PBT accelerates the training of quality at considerable cost of diversity. DyD-PPO performs relatively well, even though it optimizes the agnosiii diversity defined on action probabilities. Yet. the BDs for Atari games are more macroscopical than those for MuloCo tasks, the improvement of DvD-PPO over PBT is smaller in Atari games than that in MuJoCo tasks. Due to our diversity gradient and the adaptive diversity loss, QSD-PBT algorithm consistently achieves the highest QSD scores, though with more time overhead in MsPacMan and RiverRaid due to a relatively large diversity loss coefficient at the beginning.
We further demonstrate the ability of QBD-PBT in generating user-specified diversity, using the two orientation diversity BDs in MsPacMan, as shown in BigureBannd in the video In this game, there shorteuts connecting the left and right ends of the maze, which allows agent return to the middle of the maze via consistent (c.g., always left) moving directions. The orientation preference macroscopic BD that is significantly different from any task task-agnostic diversity investigated in
7

---

Published as a conference paper at ICLR 2023
Vaiaccddrr
FranrgDatty
Icatfockyy
Boeeg
0.
PRT
BuB Peo (Cum)
BdBl PND OCDFSTICCa
+ OSDIESTIONN
PBT IND Peo
PDT
950
RkC Queli
Quety
Quality
Quality
Quelity
Figure 2: The quality similar diversity across 10 quality intervals on Atari games.
previous literature. As shown Figure QSD-PBT achieves diverse (in a user-specified way) and visually distinct policies with good quality while other comparing methods perform unsatisfactorily (including DvD-PPO which optimizes a aask-agnottic diversity).
0.8
PPO
PBT
DvO-PPQ
OSD-PET (Ours)
shortcuss
rightright
0.2
Prefanieleemo
0.0
-0.
rereer bottom
-0.4
4470
5180
Figure 3: Visualization of orientation diversity in MsPacMan.
5
RELATED WORK
Diversity as means to improve quality. A diversity-driven approach for exploration (Hong et al. 2018) was proposed by adding a distance regularization between the current policy and a previous policy. Unsupervised learning of diverse policies was studied in (Eysenbach et al. (2018) to serve as an effective pretraining mechanism for downstream RL tasks. Novelty search was hybridized with the OpenAl ES to improve exploration in sparse or deceptive deep RL tasks (Conti et al. 2018). population determinant diversity measure PaarkerHHolde et al. 2020) was proposed to improve exploration. Diverse opponent policies have large influence the online performance of opponent modelling Fu et al. (2022).
Diverse behaviors were learned in order to effectively generalize varying environments that different from training (Kumar et al. 2020). A diversity-regulariied collaborative exploration strategy was proposed in Peng al. (2020). Reward randomization (Tang et al. 2020) was employed to discover diverse policies in multi-agent games with the aim of improving the final policy quality. Trajectory diversity was maximized for better zero-shot coordination in collaborative multi-agent environment (Lupu et al.
202I). Diversity was studied in multi- open-ended learning (Liu et al. 2021 Perez-Nieves al. 202T) to improve the final exploitability (another definition of quality).
Maximizing diversity subject to high quality constraints or vice-versa. A maximum mean discrepancy regularizer was proposed to produce a set of near-optimal policies having different distributions of trajectories (Masood & Doshi- Velez 2019). A two-objective update technique (Zhang (ct al. 2019) was developed for sequentially obtaining set of novel policies, each which solves a given task in the meantime executing distinct action sequences. A method based on the Trank-Wolff algorithm (Frank& & Wolfe 1956) was introduced to compute a set of diverse and near-optimal policies (Ghasemi et al. (2021). A set of diverse policies in the space of successor features (Barreto et al.) 2017) were sequentially obtained by solving a line of constrained MDPs (Zahavy et al., 2021), where an intrinsic diversity reward was maximized subject to a quality constraint. A reward-switching
8

---

Published as a conference paper at ICLR 2023
technique was recently proposed (Zhou al. 2022) to discover a diverse set of high-quality policies by sequentially solving a novelty-constraimg optimization problem for the current policy.
Maximizing both quality and diversity via QD-style methods. QD algorithms (Pugh et al. 2016 Cully & Demins 2017) are a type of evolutionary algorithms, where the goal is to evolve a set of diverse and high-quality solutions in single run. A representative QD algorithm is MAP-Elin Elites
(Cully et al. 2013 Mouret & Clune 2013). In order to scale to large policy neural networks, several
recent attempts have been made to combine QD with policy gradient (Cideron et al. 2020: Nilsson & Cully 202 Pierrot et al., 2022) or ES (Colas et al. 2020 Wang et al. 20211. Evolutionary multi-
objective deep reinforcement learning was employed (Shen al. 2020) to generate behavior-diverse game AIs. A kernel-
method with Stein variational gradient descent was proposed (Gangwani al., 2020) for training set of QD policies. Assuming that both the quality and the BD are fully differentiable (which is generally not true in the RI. settings), new MAP-Elites algorithm was developed in |oontaine & Nikolaidis (202T). subsequent work (Tjanaka et al.] (022)) extended differentiable QD (Fontaine & Nikolaidis 2021 to RL settings, where the quality and BD gradients were estimated using TD3 and ES respectively.
6 DISCUSSION AND CONCLUSION
In this work, motivated by the practical need in generating task-specific diverse policies of similar qualities, we formulate the QSD problem and new performance metric called the QSD score. For the first time, we derive the gradient of the user-defined diversity measure with respect to a policy and approximate the gradient using samples in the state action space (as opposed to the policy parameter space). Based on our diversity gradient, an efficient population-base then developed and has demonstrated strong performance, in terms of maximizing user-specified diversity across different quality levels, on both MuJoCo and Atari.
RI algorithm (i.e., QSD-PBT)
BD definition. We are aware in some real-world situations it might be non-trivial to express the user- intended diversity via a set of BDs but relatively easy via a similarity function Sim(TI: T)) between two trajectories. Sim [i, Tj) more general than BDs, since we can derive Sim Tj) using BDs but the opposite is not true. We look forward to extending QSD-PBT to such and more general situations.
BD estimator. In order to estimate our diversity gradient, we need train the state BD (Equation or state-action BD estimators (Equation 5). Note that both BD estimators depend on previous state-action sequence To.1-1, which might cause problems in very long trajectories. It is currently handled by applying LSTM models (in MuJoCo) or sufficient statistics (in Atari). More advanced techniques, such as attention (Vaswani et al. 2017), may be needed for more accurate estimation.
Population-based training. We use a separate neural network for each agent in the population, which increases the overall training time and memory overhead linearly with the population size N. A more efficient approach may be sharing most of the feature extraction part, e.g., the 3-layer convolution model in DQN, for all agents and building separate policy and value heads for each agent in the population. This implementation is connected with the ensemble and multi-task learning (An) et al. 2021 Flet-Berliac & Preux] 2019). From experience in previous research, parameter sharing accelerates training convergence yet in our case degrades the diversity of PPO and PBT, since they are optimized without explicit diversity objectives. For fair comparisons, parameter sharing is not implemented in this paper but is recommended in practice.
Quality and diversity optimization. In this paper, we break the QSD problem into set of sub- problems (each corresponding to maximizing diversity at a certain quality level) and solve them integrally in single run, and the resulting algorithm is termed QSD-PBT. In spite of achieving better empirical performance than other baselines, whether QSD-PBT is the most efficient method in optimizing the QSD score defined in Equation is unclear. One possible solution is combining the strength of our diversity gradient (Equation 6 and g) with existing QD-style methods. Another is regarding the QSD problem as a Pareto front optimization problem and applying continuous exploration (Ma et al. 2020).
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We thank Ke Xue (from Nanjing University) and Peng Yang (from Southern University of Science and Technology) for their helpful discussions. Peng Yang and Chao Qian are sponsored by the CCF-Tencent Open Research Fund (CCF- -enneet RAGR202201 10). Yaodong Yang is sponsored by the CCF-Tencent Open Research Fund (CCF-Tencent RAGR20220109), We are grateful the anonymous reviewers for their insightful feedback.
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 84:7436-7447, 2021.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint
2016.
Li Chenghao, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, 2021.
Geoffrey Cideron, Thomas Pierrot, Nicolas Perrin, Karim Beguir, and Olivier Sigaud. Qd-rl: Efficient mixing of quality and diversity in reinforcement learning. arXiv preprint arXiv:2006.08505, 2020.
Cédric Colas, Vashisht Madhavan, Joost Huizinga, and Jeff Clune. Scaling map-elites to deep neuroevolution. In Proceedings of the 2020 Genetic and Evolutionary Computation Conference,
pp. 67-76,77770. 2020.
Edoardo Conti, Vashisht Madhavan, Felipe Petroski Such, Joel Lehman, Kenneth Stanley, and Jeff Clune. Improving exploration in evolution strategies for deep reinforcement learning via population of novelty-seeking agents. Advances in neural information processing systems, 31, 2018.
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245 259, 2017.
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean- Cean-Baptiste Mouret. Robots that adapt like animals. Nature, 521(7553):503-507 2015.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2018
Jiajun Fan, Changnan Xiao, and Yue Huang. Gdi: Rethinking what makes reinforcement learning different from supervised learning, arXiv preprint arXiv:2106.0622 2021.
Yannis Flet-Berliac and Philippe Preux. Merl: Multi-head reinforcement learning. arXiv preprint arXiv:1909.11939 2019.
Matthew Fontaine and Stefanos Nikolaidis. Differentiable quality diversity. Advances in Neural Information Processing Systems, 34, 2021.
Marguerite Frank and Philip Wolfe. An algorithm for quadratic programming. Naval research logistics quarterly, 3(1-2):95-110, 1956.
10

---

Published as a conference paper at ICLR 2023
Haobo Fu, Weiming Liu, Shuang Wu, Yijia Wang, Tao Yang, Kai Li, Junliang Xing, Bin Li, Bo Ma, Qiang Fu, et al. Actor-critic policy optimization in a large-scale imperfect-informate information game. In International Conference Learning Representations, 2021.
Haobo Fu, Ye Tian, Hongxiang Yu, Weiming Liu, Shuang Wu, Jiechao Xiong, Ying Wen, Kai Li, Junliang Xing, Qiang Fu, et al. Greedy when sure and conservative when uncertain about the opponents. In International Conference on Machine Learning, pp. 6829-6848. PMLR, 2022.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Tanmay Gangwani, Jian Peng, and Yuan Zhou. Harnessing distribution ratio estimators for learning agents with quality and diversity. In 2020 Conference on Robot Learning (CoRL 2020). 2020.
Mahsa Ghasemi, Evan Scope Crafts, Bo Zhao, and Ufuk Topeu. Multiple plans are better than one: Diverse stochastic planning. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 31, pp. 140-148, 2021.
Seungyul Han and Youngchul Sung. Diversity actor-critic: Sample-awan aampee-aware entropy regularization for sample-efficien exploration. In International Conference on Machine Learning, pp. 4018-4029. PMLR, 2021.
Zhang-Wei Hong, Tzu-Yun Shann, Shih-Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Yi Lee. Diversity-driven exploration strategy for deep reinforcement learning, Advances in Neural Information Processing Systems, 31, 2018.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint 1711v:171 2017.
Whiyoung Jung, Giseung Park, and Youngchul Sung. Population-guided parallel policy search for reinforcement learning. In International Conference on Learning Representations, 2019.
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33:8198 15:8198-8222 1111999811,,. 8210, 2020.
Xiangyu Liu, Hangtian Jia, Ying Wen, Yujing Hu, Yingfeng Chen, Changjie Fan, Zhipeng Hu, and Yaodong Yang. Towards unifying behavioral and response diversity for open-ended learning in zero-sum games. Advances in Neural Information Processing Systems, 34:941-952, 2021.
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International Conference Machine Learning, pp. 7204-7213. PMLR, 2021.
Pingehuan Ma, Tao Du, and Wojeiech Matusik. Efficient continuous pareto exploration multi-task PMLR, 2020.
learning. In International Conference Machine Learning, pp.
Muhammad A Masood and Finale Velei- Diversity-inducing policy gradient: Using maximum mean discrepancy to find set of diverse policies. arXiv preprint arXiv:1906.0000 2019.
Kevin R McKee, Joel Z Leibo, Charlic Beattic, and Richard Everett. Quantifying the effects of environment and population diversity in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems, 3661):1-16, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540)-29-5533 2015.
Jean- Ccan-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint 1:Xi::1504.04909 2015.
Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 866-875. 2021.
11

---

Published as a conference paper at ICLR 2023
Hsiao-Ru Pan, Nico Gürtler, Alexander Neitz, and Bernhard Schölkopf. Direct advantage estimation. arXiv preprint arXiv:2109.060% 2021.
Jack Parker-Holder Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33:1 8050-8800,, 2020.
Jack Parker-Holder, Minqi Jiang. Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprini MXXi::203.013000 2022.
Zhenghao Peng, Hao Sun, and Bolei Zhou, Non-local policy optimization via riveessty-reeullarized collaborative exploration. arXiv preprint arXiv:2006.07781, 2020.
Nicolas Perez-Nieves, Yaodong Yang, Oliver Slumbers, David H Mguni, Ying Wen, and Jun Wang. Modelling behavioural diversity for learning in open-ended games. In International Conference on Machine Learning, pp. 8514 8524. PMLR, 2021.
Thomas Pierrot, Valentin Macé, Felix Chalumeau, Arthur Flajolet, Geoffrey Cideron, Karim Beguir, Antoine Cully, Olivier Sigaud, and Nicolas Perrin-Gilbert. Diversity policy gradient for sample efficient quality-diversin optimization. In ICLR Workshop on Agens Learning in -pen-Endedssss, 2022.
Justin K Pugh, Lisa Soros, and Kenneth O Stanley, Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In International Confer- ence on Learning Representations, (ICLR), 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 110.:0777.044 2017.
Ruimin Shen, Yan Zheng, Jianye Hao, Zhaopeng Meng, Yingfeng Chen, Changjie Fan, and Yang Liu. Generating behavior-divers. game ais with evolutionary multi-objective deep reinforcement learning. IJCAI, pp. 3371-3377, 2020.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference machine learning, pp. 387-395. PMLR, 2014.
Hao Sun, Zhenghao Peng, Bo Dai, Jian Guo, Dahua Lin, and Bolei Zhou. Novel policy seeking with constrained optimization. arXiv preprint arXiv:2005.10696 2020.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang. Fei Fang, Simon Shaolei Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. In International Conference on Learning Representations, 2020.
Open Ended Learning Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg, Michael Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12888, 2021.
Bryon Tjanaka, Matthew C Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradi- ents for differentiable quality diversity in reinforcement learning. arXiv preprint vrXiv:2202.03666 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2023
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Poet: open-ended coevolution of environments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 142-151, 2019.
Yutong Wang, Ke Xue, and Chao Qian. Evolutionary diversity optimization with clustering-based selection for reinforcement learning. In International Conference on Learning Representations, 2021.
Tom Zahavy, Brendan O'Donoghue, Andre Barreto, Sebastian Flennerhag, Volodymyr Mnih, and Satinder Singh. Discovering diverse nearly optimal policies with successor features. In ICML 2021 Workshop on Unsupervised Reinforcement Leurning. 2021.
Yunbo Zhang, Wenhao Yu, and Greg Turk. Learning novel policies for tasks. International Conference Machine Learning, pp. 1483-74492 PMLR, 2019.
Yunsheng Zhang, Dong Yan, Bei Shi, Haobo Fu, Qiang Fu, Hang Su, Jun Zhu, and Ning Chen. Combining tree search and action prediction for state-of-the performance in doudizhu. In IJCAI, pp. 3413-3419, 2021.
Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via reward-switchig policy optimization. arXiv preprint arXhv:2204.022-6 2022.
13

---

Published as a conference paper at ICLR 2023
A THEORY ANALYSIS
A.1 PROOF OF PROPOSITION
Proof. We prove the correctness of Equation 7 and that Estimation 8 an unbiased estimation the gradient in Equation 7 with a constant scale. We denote segmentation of a trajectory starting from
time step j as Taj [i,, ai; ..., 8j. aj], with specially T-1 == Ø and Tu == [so, on). Recall the definitions of state BD estimator and state-action BD estimator:
Te 1, 8t): Ea [b((T)),
(10)
Q To: St, a+) E [T)]
(11)
We consider determinate policy for continuous actions, and the output of the policy (8()) is scalar, i.., O₂ (st)). As a result, the following two equations hold between the state BD estimator and the state-action BD estimator:
Q Bg (70:-11,8,,0) = p( (o())) 7l (To., 86+ 1d+1)1
(12)
VB, (To.c-1, Q3, (0: (T())))
(13)
where p(se+1|se, (T()s) is the transition probability from 8t 8141 after taking the action To (o(s)) We use p' (s; + 8j) to represent the probability p(s,) I (($k+1lss, (sx))) We omit the index for clarity in the following. Note that the BD of a policy can be written as:
(r(Tg) f PB (T-1,,1111_000 (T-1, To(5)))
(14)
and the gradient is:
(
p(so) Vo (T-1, 80, To (so))
(15)
08
80
Further, we have:
VoQ (T-1, (0, To(o))) = Vo / p(si (so, To(+o)) (TO, 81 )d8i
T (oo 81 s1sss / p(s1|80: mo(sm) (To, 81)ds1
VoTo (So) Voo an) lao= (allso..allaoottttt
=VOTTo /
Ven Vg. (To, (1)dss / p(sa|so. To (s) (To. $1 dss
=VoTo Vo (0) / p( no (o(s)) (To, 81, (0(8)))
(16)
Equation shows the relation between the gradient of the current state-action BD estimator and that of the next state-action BD estimator. Generalizing this result, we have
of To B. (Tot-1, St, To(s)) = -VoTo(st 2 (To:--1, St, ax) a==
p(8441 18441/8e:00 To (Tod, 8141, (1(4,)))())))) (17)
8041
14

---

Published as a conference paper at ICLR 2023
We apply Equation 17 to Equation 13
recursively:
(
00
+
No dssdds
p(so + $1 V To. $1 no
8011
p(so) VOTo 50 V Q 2
T- s0. =0)
80
p(50 7 81)
V V. Q7 (To 81, a1 (1) al)
50:1
p(50 82 V (T.11 S2. To dso:2
5022
T Σ
1=0 / p(so- + 8t)) (OTO(+)) Va. QH (Tox-1, 8t, (o) dsaa
(18)
where fay. is short for Sso L. J..
S., and dso:: is short for dsydss...ds; The first equality is according
to Equation I6 and the successive equality is according to Equation 17
Proof. We further prove that with a constant scale, Equation Bis an unbiased estimation of Equation 7 Note that in practice we obtain sampled trajectories by executing policy x0)), and the samples in the trajectories are added into a replay buffer. As a result, a sample (st, az, To.--1) is added into the replay buffer with probability p(so) 1[k-0 ( To (sx)), i.e., p(50 + st). In other words, the probability that the sample (St, Ge, T0:r-1) is in the current mini-batch proportional to p(so st).
which we denote by p(ao-se) (C' is a constant). The expectation of the sampled gradient is:
E V VTTTT((S (5t) V ae 2%, (Tot-1, St, (x)))
VoTo(s:) Va, Q (To--1, St, ae)a p(so - 81) dsod
Σ
C
t= Spse
1 (B(()))
(19)
=c 00
where the last equality is according to Equation 18
A.2 PROOF OF PROPOSITION
Proof. Denote h(x) = fx) + >g(x). By assumption, we have h(x)) is Lipschitz smooth, i.e., there Vh(2>) Vh(ii) s L|[ - vll. Vz,y ER"
exists L >>0 0, s.t.
(20)
So,
Ib(y) - h(x)) (y-)) = / Vh(r t(y (()y-)))* x))T(y
x)dt - Vh(x) =x] x)
|/v/(z + t(y z))² vh(x)=11 l1y z||df
< Jo
tLy =1
L
lyv xi| 2
(21)
2
15

---

Published as a conference paper at ICLR 2023
iee,
L M(y) h(x) S (y 2x) + (V=))
y11?
(22)
Since g(x) is bounded, we assume g(x) 0. Otherwise, let C' min, g(x), rewrite (2) f()) + (g(x)) -C) + C and perform the following analysis on hy (#) = f (xx + (g(x))
(note that both h4 (2: neew and (2r) have the same derivatives and minimums). Hence, for each fixed
I, the series {/r (z) t=0 is decreasing to h(x).
Consider optimizing algorithm which starts zo, and, each time step i, updates xe using the gradient descent I++1 It (xx)) By Equation 22 we have,
L he(x141) - hx( = Vhe(a,) (x141)))((++111111)) - xz)
- |1 22
2
|-n|\V/(=>>|11 + Lnp
2 [1)h1()))11
(23)
Choosing 7) I: we have, he( I he(x)). and since (441(1141) I (2(+1),, we obtain ( S (z,). Hence lim,-soo (x+) exists (h(r) is bounded). It is easy to prove that (1x(2)) converge to h(x) uniformly, hence, lim (x4))) lime h(x)). We denote M lim h(z)).
We claim that tim Vh(2;) = O, hence M min, M(x). We prove the claim by contradiction. Otherwise, there exists 60 satisfies that for any T. 3to T, s.t.| Vh(rt,)) )1 >>>>>>>>>>> eg.
Similar to Equation 23
M(++1) - h(20)) & Vh(aa) ((((1))((E(1111111 - x4) + || - xell
< (Vhh(a))><ha(a + Laz
2 [[
( Laj
(VVh)(E,)) Vh(x;) (ha((t)))) +
(x)))
=<((h((z,)) - Vh(2,)) - - [/o(22))1
(24)
In the last equation, we choose 1 +. Since li Vhr(z) Vh(x), have lim;- (V+, (h(z:))(Vh((rr)
S lim (xr)) Vh(x:)]l [/Vh+(
Since whatever how large the T is, we can find to > s.t. ||Vh(z, > so we can find [h(Ito+1) - h((x)))= 4eo. which is contradict to the convergence of {h(zz)) (Cauchy principle of convergence).
Hence we prove the claim.
Finally, since h(()) is convex, the algorithm will converge to global minimum of h(x).
B ADDITIONAL EXPERIMENTAL RESULTS
B. ABLATION STUDY
B.1.1 THE ADAPTIVE DIVERSITY LOSS
We demonstrate the effects of adaptive diversity loss constrained by the population quality described
in Section 3.2 The Atari game FishingDerby is used for the ablation study, and three training settings are compared: (1) The baseline setting is the diversity loss function with a fixed coefficient X (2) In the decay setting, we apply exponential decay of the coefficient Xoo with decay step lg- (3) In the adaptive setting, we apply an adaptive coefficiens (t, RR,)) that considers both the relative quality of the agent and the current training steps according to Section 3.2 The hyperparameters for this
ablation study are detailed in Table 10
The experimental results the ablation study are shown in Figure 4 For the training setting of fixed coefficient (0), it is worth noting that the qualities of some agents get trapped into local optima,
16

---

Published as a conference paper at ICLR 2023
80
other agents
0.4
Fsed
Decayed
40
40
Adaptive
Quality
agent 9
Quality
Diversity
0.3
4gent
Fixed
-80
agam
-80
Decayed
0.1
100
150
200
80
100
150
200
10
20
30
40
Slepe
Sieps
Qualty
Figure 4: Comparison of three diversity training settings: the fixed diversity loss (fixed), the decayed diversity loss (decayed), and the adaptive diversity loss with quality constraint (adaptive). Left: typical training curves of the qualities of population of 10 agents with the fixed setting. Middle: training curves of the qualities population of 10 agents for the three settings. Right: the QSD performance for the three settings.
as shown in Figure 4 (left), which hinders the optimization of diversity among policies of similar qualities. Moreover, as shown (midlle,, the qualities of agents improve slowly for both the fixed and the decay training settings, which may have negative effect maximizing diversity among policies of near-optimal qualities, e.g., quality levels above 40 this game. By introducing the adaptive quality constraint, we observe a much faster improvement of the population quality in Figure 4 (middle). The overall QSD performances (right). We observe that the quality constraint optimization has slightly better QSD scores higher quality levels (>60%) although it improves the quality at the cost of diversity degradation at low quality levels. We recommend applying the adaptive diversity loss balanced by the population quality in practice.
the three settings are shown in Figure 4
We further investigate the effect of the hyperparameter decay rate to and initial Ao in the QSD-PBT algorithm. In the Atari game FishingDerby, we first fix the initial Ao to 0.1 and vary the decay rate ty from 20k 500k training steps, then fix the decay rate to 500k and vary the initial Xg from 0.05 to 0.4. Ablation results are shown in Figure 5] where computational/ting overheads and QSD scores are measured the same way as in Table and Table 2 From the experimental results, we can conclude that the initial λo significantly affects the training convergence and demonstrates a trade-off between the final QSI) score and the training overhead in large range (from 0.05 to 0.3). When Å is too large (Ao 0.4), QSD-PBT can not reach the optimal quality within acceptable time overhead (500k steps), then the partial sum of the quality intervals is reported and thereby the QSD score degrades. Compared with Å QSD-PBT algorithm is less sensitive the decay rate to, higher decay rates (20 and 50) lead to similar results as lower initial An while a suitable decay rate (200) would both maintain the QSD score and reduce the time overheads.
Decay rate (K)
0.3
+
Initial Ao
5.0
0.2
4.5
0SD sooe
4.0
0.1
3.5
500
200
0.4
3.0
100
0.05
2.5
20.
50
200
250
300
350
400
450
500
Time overhead measured by steps (K)
Figure 5: The final QSD scores and the time overheads under different settings of decay rate to and initial λu-
17

---

Published as a conference paper at ICLR 2023
B.1.2 OTHER DIVERSITY MEASURE
In this section, we provide more analysis of the measure function. Note that the measure function is in need in our algorithm to measure the diversity among the policies, so that our algorithm can train to improve the diversity. In the main experiment, we choose the MSE function be the measure function. We note that our method theoretically can be applied to any measure function as long as the measure function is differentiable. We provide one experiment for example below. However, the choice of the measure function absolutely will affect the performance of the algorithm, and we discuss this later.
We claim that QSD-PBT can be applied to any diversity measure as long as the diversity measure differentiable with respect to the BDs. We here provide additional experimental results using different (from the main experiments) diversity measure, i.e., the determinant of DPP, using the MuJoCo task Hopper.
Table 3: The QSD scores on the MuJoCo task Hopper using the determinant of DPP as the diversity measure. #step denotes the number of training steps.
MuJoCo Task
EDO-CS
PBT
DvD-TD3
QSD-PBT (Ours)
score > 0%
0.95 0.43
0.80 + 0.14
2.32 = 0.96
9.49 + 2.83
Hopper
score > 60%
0.10 + 0.04
0.09 0.03
0.31 + 0.09
0.45 + 0.12
Rman: 3k
#step (k) 200 + 27
111 7
208 + 27
170 10
From the results presented in Table 3 we can conclude that QSD-PBT still outperforms other methods when the diversity measure is the determinant of DPP. The results are consistent with that in Table ] when the pair-wise distance employed as the diversity measure. One reason is that QSD-PBT optimizes the user-defined diversity objective directly, while the other methods optimize a task- agnostic diversity measure (PBT, DvD) or quality within different cells of the BD space (EDO-CS). Besides, QSD scores within intervals above 60% of the maximum qualities in Table 3 drop faster compared the results on the pair-wise distance diversity measure in Table The reason that the determinant of DPP is much more sensitive to the changes in BDs, in comparison with the mean pair-wise distance measure. Therefore, we recommend the mean pair-wise Euclidean distance as the first choice for users and experiments in this paper.
The choice of measure function will have an impact on the training overheads from two aspects:
(L) The measure function itself may have high computational complexity, e.g.. O(r²)) in DPP, where n is the population size. However, n is small in practice (8 in MuJoCo and 10 in Atari) and the complexity is negligible compared neural network models. For example, when increase population size n from 10 to 100, we do not observe any decrease of GPU sample speed during training.
(2) Different measure functions will affect the convergence of QSD-PBT and hence affect the computational sources. In Figure we provide training curves of MSE and DPP measure functions in MuloCo tasks. From the figure, we can see the convergence rate of DPP is slightly slower than MSE. The reason is that MSE calculates the pairwise distance of BDs among all policies in the population and averages them by equal weights, while the DPP is more sensitive to the most similar policies and unstable. For example, the DPP of population that contains two same policies in it will be zero regardless of how diverse the other policies are in the population.
B.2 RESULTS ON RANDOMLY DESIGNED BDs
Since the trajectory BD can be technically anything, the design principles in this paper are focused on generality and practicality. The BDs in previous experiments include various forms, e.g., action BD (MuJoCo joint torques), state BD (Atari game time), and trajectory BD (Atari left/right preference). However, we find that particular choices of BD still favor certain algorithms. For example, BD defined on actions favors the DvD algorithm since directly optimizes the KL distance of agents' actions. Hence the improvement of DvD over PBT is more significant in the MuJoCo results (Figure than in the Atari results (Figure 2). Instead of defining meaningful and explicit BDs, we further investigate the performance of QSD-PBT on implicit BDs that would less favor certain algorithms. Since can
18

---

Published as a conference paper at ICLR 2023
Humanoid
Hopper
Walker2d
0000
NSE
DPP
5000
5000
4000
3000
5000
2000
1000
1000
100
150
2000
50
1000
150
200
50
100
150
200
tmeestoppk
tmmcstopik)
timestepiki
Figure 6: Comparison of the training curve between MSE and DPP measure function. Lines and shadings represent the mean and 1-std range of the qualities among the policies the population.
only get states pixel images from the Atari environment, we focus on actions and the implicit BD is designed as the randomly weighted action rate:
by (T) :. a, w; [(wt I. K
m = 1; w > 0,
(25)
where weight w; is sampled from the normal distribution and normalized by the softmax function,
K 6, action rates a 6-dimension vector [fire, noop, up, down, left, right], the calculation of action rates is in Table [ We generate five random BDs, i.e., the weights form 5 x 6 matrix, and provide additional training in the Atari game FishingDerby, the results are shown in Figure) Besides, QSD-PBT checkpoints trained on generally designed BDs in Section 2 are cross-evas eosss-evalatted by random BDs, marked as "CE". From the experiment, the results are consistent with Figure [ and Figure QSD-PBT CE is trained with the BD fire rate and the diversity of DvD-PPO is defined by action pairs, therefore both of them favor certain random BDs and perform slightly better than PPO and PBT. Comparing QSD-PBT and QSD-PBT CE, conclude that our algorithm is capable of "what you will get is what you have defined" since allows for the direct computation of the diversity gradient.
0.50
PPO
PBT
0.45
DxD-PPO
0.40
Diversity
0.35
0.30
0.25
0.20
0.15
0.10
10
20
30
40
Quality
Figure 7: The quality similar diversity across 10 quality intervals on random BDs, "CE" means cross-evaluation on these BDs.
B.3 DIVERSITY RESULTS ON EACH INDIVIDUAL BD
We claim that QSD-PBT can generate user-intended diversity as long as a diversity measure function and set of user-defined BDs are provided. We illustrate this ability of QSD-PBT in the main text using orientation BDs in Figure We here provide additional results using the four Atari games for each of the five BDs defined for the Atari experiments.
Results in Table 2 and Figure 2 are further decomposed into each individual BD in Figure 9 where plot the QSD for single B.D at time, i.e., B(To) [B, (To)). After being normalized by
19

---

Published as a conference paper at ICLR 2023
scale factors in Table g the five BDs have similar ranges. Overall from Figure 9 that QSD-PBT outperforms previous methods in terms of the QSD performance for each of the five BDs defined for the Atari experiments in most cases.
conclude
We observe that some user-defined BDs are highly correlated with the quality, e.g., the game time in PvE-style games (MsPacMan and RiverRaid), where QSD-PBT achieves only slight improvements compared with other methods. How to adaptively handle the correlation between a BD and the quality remains an interesting question. Besides, for the left_right BD in the game RiverRaid, the river course is narrow (shown in supplementary videos), and there is no shortcut (as in the game MsPacMan) that connects the left and right areas. Therefore, left and right actions are restricted to have similar numbers, and the resulting diversity scores are small for all the methods in this game.
B.4 ADDITIONAL VISUALIZATIONS
We illustrate the diversity at different quality levels in Figure R using the Humanoid-v2 task. In this task, an agent can be rewarded by standing or walking, and walking at faster speeds often leads to higher rewards. In Figure M(a), agents at a lower quality level (4000-5000) learn to stand and balance in various poses, e.g., with open or closed legs and different hand gestures. In contrast, agents at a higher quality level (5000-6000) prefer various walking gaits, including ambling, striding, and mincing, which is demonstrated in Figure bb).
[a1] standing. open legs, speed 0.3
(aZ) standing. legs, speed
(83) standing, closed speed 0.2
(o4) standing, closed legs, speed 0.4
(a) polkces 4000050000 qualiy level
[b1] ambling, bending backward, speed 1.0
(b2) ambling, bending forward, speed 1.7
[b3] stridin, speed
(b4) mincing. speed 1.8
1b] Diversity of policles at 5000-6000 quality level
Figure 8: Visualization of diversity at different quality levels on Humanoid-v2.
Additional visualizations of diverse policies generated by QSD-PBT are provided in Atari and MuloCo videos The results demonstrate that QSD-PBT effective in both maximizing diversity across different quality levels and generating user-intended diversity across different user-defined BDs.
For the MuJoCo videos, we focus on demonstrating the diversity at different quality levels. For the Humanoid task, agents at a high quality level (5000-6000) prefer various walking gaits, e.g., ambling, striding, and mincing. Agents at a low quality level (4000-5000) only learn to stand still and balance in various poses. For the Walker2d task, agents at a high quality level (5000-6000) learn to run or only walk with small steps just hop. For
stride, while agents at low quality level (4000-5000) the Hopper task, agents at a high quality level (3000-4000) prefer fast hopping with different poses, while agents low quality level (2000-3000) only learn to stand or jump with small steps.
20

---

Published as a conference paper at ICLR 2023
McPacMMan
RiverRaid
FistingDerby
lceHockey
Boxing
0.25
0.10
0.20
0.15
0.08
0.15
0.10
0.10
0.04
0.0
0.00
0.14
0.32
0.4
04-
0.24
0.20
0.76
0.02
0.20
0.6
PPO
0.11
0.15
PBT
DO--PD
0.00
0.16
GBD-P8T (Oun)
0.3
035
0.25
0.20
0.00
0.35
0.30
0.15
0.25
0.00
0.15
0.10
0.0
0
0.10
0.05
0.00
0.02
0.00
0.25
0.15
6.8
0.20
0.10
0.10
0.10
0.02
10:0
01
0.05
3-
0.06
8:
0.05
0.21
1200010000
Figure 9: Detailed diversity results for individual BDs in Atari games
For the Atari videos, focus demonstrating the diversity of the five user-defined BDs. Aside from a video version of Figure in MsPacMan, the orientation BDs (left_ tiiirtrghhttttt and ap_down)> are further visualized in FishingDerby, where PPO agents always fish the up and left area since it an optimal and casy-to-explore solution. The diversity on the other three BDs (game time, fire rate, and action continuity) are demonstrated in IceHockey and RiverRaid. In the game IceHockey, agents get scores by various strategies, and three of them are displayed: (1) Dribble, run, and shoot (fire) in the normal style. (2) Stand still, defend, and shoot away. (3) Swing left and right without shooting, In the game RiverRaid, agents can speed up slow down by up down actions, and therefore the total game time differs. Our agents can generate diverse behaviors by considering multiple aspects during driving, e.g., the speed, obstacles, and the remaining fuel.
B.5 TRADE-OFF BETWEEN QUALITY AND DIVERSITY
Generally speaking, the quality and the diversity are trade-off, higher quality tends to result in lower diversity. But as shown in Figure [ Figure 2 and Figure since environments and BDs are so varied, it is hard to clarify the accurate correlation at each quality level and we need to analyze case- ba-cee. Usually it depends on (1) the correlation
the BDs and the quality, and (2) the exploration space and complexity of the environment. Here observe the difference between Figure [and Figure 2 In MuJoCo tasks, the QSD curve shows the quality and diversity trade-off, while in Atari games, the trade-off disappears.
21

---

Published as a conference paper at ICLR 2023
We conclude with two major reasons: (1) The definition of BDs. For MuJoCo tasks, we define most of the BDs on joint torques, which are highly correlated to the pose control the robots and the resulting score. While for Atari games, correlations with the quality, e.g., the left or right preference in Figure 3 does not influence the score since most games are mirror symmetric. (2) The exploration space and complexity of the environment. Compared with MuJoCo tasks, Atari games are more complex and have more exploration space. Take the game RiverRaid for example, narrow and the agent can only go straight. When it reaches certain scores, the agent will enter into new scenes that have much broader course. The scene switching provides more exploration and diversity space, resulting in the improvement of diversity along with quality.
define relatively coarse- boure--raained BDs that have lower
lower quality level (<3000), the initial river course is
C IMPLEMENTATION DETAILS C.I DESCRIPTIONS OF THE BDs
For MuJoCo tasks, the BDs are defined on the scoring speed and the built-in joint torques respectively, The joint torques are the actions MuJoCo tasks. The action spaces for each MuJoCo task are shown in Table 6 and more details number of frames used to obtain the BD of Scoring speed. For each joint in a trajectory, we sum the actions, i.e., the torques, applied to it and divide the by the number of frames to obtain a joint BD. As a result, the number of BDs for each MuloCo task is the number of joints in that task plus one, i... the Scoring speed. We normalize each BD the range of [0, 1.0]. An overview of all the BDs for each task in the MuJoCo experiment is presented in Table
be found here For trajectory, we divide its total score by the
Table 4:
Action space for Hopper.
Num
Description (Torque applied to different joints)
Range
1
the thigh rotor
2
the leg rotor
[-1,1]
3
the foot rotor
Table 5: Action space for Humanoid.
Num
Description (Torque applied to different joints) the hinge in the y-coordinate of the abdomen
Range
the hinge in the z-coordinate of the abdomen
[-0.4,0.4]
the hinge in the x-coordinate of the abdomen the rotor between torso/abdomen and the right hip (x-coordinate) the rotor between torso/abdomen and the right hip (z-coordinate)
6
the rotor between torso/abdomen and the right hip (y-coordinate) the rotor between the right hip/thigh and the right shin
7
8
the rotor between torso/abdomen and the left hip (x-coordinate)
9
the rotor between torso/abdomen and the left hip (z-coordinate)
10
the rotor between torso/abdomen and the left hip (y-coordinate) the rotor between the left hip/thigh and the left shin
[-0.4,0.4]
11
12
the rotor between the torso and right upper arm (coordinate -1)
13
the rotor between the torso and right upper arm (coordinate -2)
14
and right lower the rotor between the torso and left upper arm (coordinate -1)
the rotor between the right upper
15
16
the rotor between the torso and left upper arm (coordinate -2)
17 the rotor between the left upper arm and left lower arm
For Atari games, we define a common set of BDs for each game. There are 5 different BDs in total. The design of these BDs are motivated by covering different ways in which diverse and meaningful
22

---

Published as a conference paper at ICLR 2023
Table 6: Action space for Walker2d.
Num
Description (Torque applied different joints)
Range
1
the right thigh rotor
2
the right leg rotor
3
the right foot rotor
[-1.1]
4
the left thigh rotor
5
the left leg rotor
the left foot rotor 6
Table 7: trajectory.
An overview of all BDs for each MuJoCo task. #frame is the total number of frames in
BD
Definition
Normalization
Scoring speed
tutal_scon score
# #frame
Joint torques in Hopper (dim 3)
actions
2++11
#trame
2
Joint torques in Humanoid (dim 17)
actions
2=4-0.44
ftrame
0.8
Joint torques in Walker2d (dim 6)
actions
x+1
#trame
policies could possibly differ. We normalize each BD to similar range. An overview of all the BDs in the Atari experiment is presented Table g
Table 8: An overview of all BDs for Atari games. # indicates the total number of frames, action changes, or specific actions in trajectory.
BD
Definition
Scale factor
Game time (minute)
# frame
900
Fire rate (per second)
fire
frame
3
Action continuity
log
15 15.Hartion change
1
# #trame
15
Left_ right
log
# left
max #right.
13
Up_down
log
#up
max #down.
15
C.2 THE DIVERSITY MEASURES
We use the mean wise distance as the diversity measure for the main experimental results. For population of policies IT with size N, IT = (To, [1 $ j & N). the mean pair-wise distance is defined as:
N-1
2
Σ
Σ
[BT,))) - (mm,,) lll2
(26)
N(N 1)
j=+t1
where B(TT) is the vector of the BD values of a policy defined in Section
As stated in the main text, our method QSD-PBT can be applied to any explicit diversity measure as long as the measure differentiable with respect to B(TT). Later in Appendix B...2. we present results using another diversity measure, i.e., the determinant of DPP. The DPP determinant of == = (To, | < < N) is defined as:
(B())(((())))B( 1)
(27)
[1B(Fo, ) B M1
where K. is a given kernel function. We set K. exp(
in case.
23

---

Published as a conference paper at ICLR 2023
C.3 THE CALCULATION OF THE QSD SCORE
In practice, the number of policies obtained throughout training with qualities lying in the same interval can be much larger than N, which is the population size of II used for evaluating the diversity measure (Iv(IT). To make an efficient and fair comparison across different methods in terms of the QSD score, for each quality interval we sample population II of size .N for 100 times. The way we sample II is by alternating the training index of the policy till N policies are obtained, because policies with the same training index tend to have similar BI values. The training index of a policy denotes which agent in the population the policy comes from during population- -based training. We calculate the diversity measure for each sampled II and use the sample mean of the 100 evaluations as the diversity measure for the corresponding quality interval, which is then aggregated according to Equation I to produce the QSD score.
C.4 HYPERPARAMETERS AND DETAILS OF THE COMPARED BASELINE METHODS
For MuJoCo tasks, we compare QSD-PBT three population-based training algorithms: EDO-CS, PBT, and DvD. EDO-CS uses the ES for optimizing the quality and a selection mechanism to induce diversity among the defined BDs. PBT uses TD3 as the backbone and mostly focuses on the quality of each policy in the population. The Perturb exploration and Truncation selection exploitation strategies are adopted every 10k training steps for our setting of PBT. The DvD-TD3 in the original paper is implemented. DvD-TD3 optimizes a combined loss of quality and task-agnostic diversity measure defined on state action probabilities. All the hyperparameters for each method are listed in Table 9 except for EDO-CS and QD-PG, for which we implement with the architecture and hyperparameters suggested in their papers. For QSD-PBT, we use the LSTM as a feature exactor in BD estimators. We pass the trajectory state and trajectory action to an LSTM, respectively, then concatenate the feature and pass it to the MLP. The 'state' BD we used for reproducing QD-PG is the action (a real-valued vector, which has dimension of 17, 3, and 6 respectively in the three MuJoCo tasks) in a state. We believe this is a fair setting for QD-PG, because the trajectory BD we defined here is the average torques (i.e., actions) applied to the hinge joints over a trajectory.
For Atari games, we replace the baseline method EDO-CS with PPO for training efficiency consider- ation, where a population of N independent PPO agents are trained. As Atari games have discrete actions and high-dimensionas image inputs, all the methods employ the DQN (Mnih et al. 2015) architecture as the backbone model. We find it effective to provide statistics from the beginning of game to the current state (the number of frames, the number of fire, left, right, up, down actions and the number of action changes) as sufficient encoding of trajectory 70:j- Instead of the LSTT applied in MuloCo experiments, these additional features are applied in the DQN model to better estimate BD defined in Table S. Accordingly, we use PPO in the implementation of both DvD and PBT. All the hyperparameters for each method are listed in Table TO
C.5 PSEUDOCODE OF QSD-PBT
The developed QSD-PBT is in general a population-based RL algorithm. In QSD-PBT, we employ parallel actors and learners to simultancously train a population of N agents. The policy of each agent is trained using the gradient of the loss function defined in Equation 9 Meanwhile, the policy value functions or functions are trained using the mean squared error, and so are the state BD estimators or the state-action BD estimators of each agent. Moreover, QSD-PBT maintains a running average estimation each agent's quality, each agent's BDs, and the population's 'ss mean quality. The pseudocode of QSD-PBT is given in Algorithm T Note that, without loss of clarity, we present QSD-PBT with TD3 and QBD with PPO in one algorithm.
24

---

Published as a conference paper at ICLR 2023
Table 9: Hyperparameters for MuJoCo tasks.
Hyperparameters Shared
Value
Population size (N)
Replay buffer size
10 x N
Mini-batch size
1024 x
Optimizer
Adam
Learning rate
3e-4
Discount factor (7)
0.99
Target network update rate (r)
0.005
Std of gaussian exploration noise
0.1
Delayed policy update
Policy noise
0.1
Noise clip
0.5
Policy network
MLP (state_diim) dll_imm226-----aattii dim)
Critic network
MLP (state siaat_dim+action 2 dim-256-256611
BD Estimator
LSTM(128) + MIP (256-256-BI__d)
PBT
Exploitation interval
every 10k steps
Truncation selection ratio
0.25
DvD-TD3
DvD loss factor
0.5
DvD embedding mini-batch size
256
QSD-PBT
Adaptive diversity loss (0)
0.1 (Hopper, Humanoid), 0.5 (Walker2d)
Adaptive diversity loss to
x 10a
Algorithm 1: QSD-PBT with TD3 or PPO bascline
Initialize network parameters for population of N agents: (0,,00,..,00
Start multiple actors and learners in parallel. Actors: while true do
Fetch the latest model from the learners, and run multiple game episodes.
Generate RL training samples in the form: (a,, St, re], and the encoding result of
To:t-1, and send them to the replay buffer.
For each agent, calculate the average BD results B(TT) and the quality It using results in multiple games, and send R, B(xo), B(TTO) to monitor.
Learners:
for / E 1,2,,, do
Fetch current BD results B(To), ((Ta), and qualities H of all agents from the monitor. Calculate the diversity measure function (Iv(
for n E N. do
Caleulate the diversity gradient of Div(II) at B.
Fetch a batch of samples for agent n from the replay buffer.
Calculate value loss and policy loss according TD3 or PPO algorithm.
Calculate mean-square loss for BD estimators V= by B(To..).
Calculate diversity loss according to Equations
6.63
Update critic parameters On using gradients on value loss and BD estimator loss.
Update actor parameters O,, using gradients
Lrotat (To..) in Equation 9
25

---

Published as a conference paper at ICLR 2023
Table 10: Hyperparameters for Atari games.
Hyperparameters
Value
Shared (PPO)
Population size (N)
10
Number of actors
700 (CPUs)
Number of learners
8 (GPUs)
Frame skip
4
Reward shaping
log(abs(r) +1) (2-10)20) (r<o)) Fanet al. 2021
Replay buffer size
50000 x
Mini-batch size
256 x N
Optimizer
Adam
Learning rate
le-4
Discount factor (7)
0.99
GAE parameter (X)
0.95
PPO clipping ratio
0.2
Value function coefficient
0.5
PBT
Exploitation interval
every 20k steps 0.2
Truncation selection ratio
DvD-PPO
DvD loss factor
0.1
DvD embedding mini-batch size
512
QSD-PBT
Adaptive diversity loss (0)
0.1
Adaptive diversity loss to
105
26


Reviews:
[{"review_id": "x8gA4_mlTBE", "reviewer": "Reviewer_7pVo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the topic of diversity in reinforcement learning. The authors propose the quality similar diversity problem, whose goal is finding diverse policies within multiple quality levels. In particular, the paper considers task-specific diversity where the user defines certain behavior descriptors (BDs) that are a function of trajectories and capture the type of diversity the user prefers. Given this problem statement, the authors present methods for estimating BDs via the policy gradient theorem. The gradient is then combined with a population-based training via an adaptive diversity loss. This leads to an RL algorithm that optimizes the quality-similar population diversity of a set of policies. Empirical evaluations show improved performance over prior methods w.r.t. diversity of policies across multiple quality levels. ", "summary_of_the_review": "The presented method appears to be useful, practical, simple, and well-motivated. The authors do a good job explaining their method, justifying their design choices, and providing intuition. There are no major weaknesses that would justify rejection. ", "strength_and_weaknesses": "**Strengths**\n- The problem statement is interesting and distinct from diversity problems studied in prior work. In particular, this paper aims at finding a set of diverse policies in a hierarchy of quality levels. This is useful in certain AI applications such as games and appears related to curriculum learning tasks. The problem formulation in this paper can be of independent interest to other topics in RL as well.\n- Besides the quality-similar diversity problem statement, another technical contribution of this paper is noting that an exact gradient of the user-specified BD based on trajectories can be computed directly using the policy gradient theorem. Thus, one can obtain an unbiased sample-based approximation to the gradient and utilize techniques developed for policy gradient methods.\n- Guidelines for adaptive diversity loss and handling multiple quality levels are useful.\n- The paper presents a thorough review and categorization of prior diversity methods.\n- The paper is well-written and well-organized. The authors provide a solid motivation behind their work and discuss applications and intuition behind their design choices throughout the paper.\n- The empirical evaluations show strong performance compared to prior methods in both continuous and discrete tasks. \n\n**Weaknesses** \n\n- The setting considered in the paper requires multiple design choices, which brings the practicality of the method into question. Apart from designing rewards, one needs to design a set of diversity design functions that depend on trajectories (which include a large number of states and actions) and may be difficult to elicit desired behavior. Another design element is the diversity measure function (an example is given which is the determinant of a kernel matrix.)\n- In Section 3.2, the authors mention a relationship between large $\\lambda$, which determines the strength of diversity, and optimistic initial values used for exploration. This connection is unclear and the two seem rather distinct from my perspective.\n- Despite the authors discussing this in Section 4 and Appendix, It is still unclear to me whether particular choices of BD would favor certain algorithms. Since BDs can be technically anything, it would be good to also empirically compare the performance of algorithms given random BDs.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ltglWBVI-x", "reviewer": "Reviewer_bTHx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses task-specific diversity problem, which exhibits in policies with similar expected return (what the authors called quality). The task specificity is framed by a set of scalar functions of a trajectory defined by the user, which are called Behavior Descriptors (BDs). A BD captures some property of a given trajectory, which is preferred by the user, e.g., number of times that a self-driving agent shifts gear (say the less the better). The authors propose an estimator for each BD, based on which they compute the gradient of BD w.r.t the policy's parameters. This gradient is then used to augment standard policy grad in a tunable manner, the result of which is a class of algorithms, called population-based RL algs.", "summary_of_the_review": "- Definition of BD needs more explanation. Do you need some restrictions such as being bounded (or finite)? \n\n- First paragraph of page 3: what do you mean by a measure function? Do you mean a function satisfying to be non-negative, zero for the empty set, and countable additive? If yes, I would probably make it explicit and simply discuss what these properties mean in the case of $f$. If no, then the definition of $f$ does not require it to be non-negative. I am wondering if $f$ can also be negative, then the QSD score can be zero for some given M-partition even if all $\\mathcal{B}_i$ are strictly non-zero. This may make the definition counterintuitive. \n\n- Page 3, first paragraph: depending on the definition of $f$, its evaluation might become a computational bottleneck. Some explanation on computational complexity would be apropos. \n\n- Section 3, Lines 5-6: “Note that both the user-specified BD and the quality of a policy are defined on a trajectory” -> not correct according to your definition of quality. The return (not the quality) is defined for a trajectory. \n\n- Page 4, first paragraph: suggestion: for the BD advantage function, you can observe that its expectation over $\\pi_{\\theta}$ is zero, which simply follows from your definition of value functions. Hence, in practice, it might be quite beneficial to subtract the mean (taken over mini-batch) from BD advantage to make the implementation more robust. \n\n- Moreover, for computation of advantage functions, I strongly recommend the following rather than GAE:\n“Direct Advantage Estimation”, https://arxiv.org/abs/2109.06093\n\n- Section 3.2: For maximizing diversity, section 4 of the following reference also seems relevant:\nhttps://arxiv.org/abs/2110.01548 \n(They have an ensemble of Q networks, while you have a set of policy networks.)\n\n\nMinors:\n\n- Last paragraph of Introduction: the acronym PBT used before it is defined. \n\n- First paragraph of section 2: your reward function is indeed an expected reward as it only defines over state-action and not the full transition ($r(s,a) = \\mathbb{E}_{s’} ~r(s,a,s’)$).\n\n- First paragraph of section 2: $T$ is not defined. More importantly, it looks like that you are considering episodic MDPs (also evident from allowing $\\gamma=1$). If so, make it explicit. Moreover, it is not clear whether you have termination at $t=T$ or not. If yes, you need to also define terminal states (whose value is zero by definition). \n\n- Section 3, 3rd line -> I would say “conventional” rater than “traditional”, since the reference is too recent to be a tradition. ", "strength_and_weaknesses": "Strengths:\n\n1. The main objective (to produce a set of diverse policies at multiple quality levels) seems useful and interesting to investigate. \n2. Formulation of the Quality-Similar Diversity (QSD) is novel and seems to achieve superior results. \n\nWeaknesses:\n\n1. The paper lacks some analysis of sensitivity as well as computational overhead. In particular, it is not clear how the rate with which $\\lambda_i$'s are changed will impact convergence and stability of the algorithm as a whole. Additionally, the choice of $f$ can have significant impact on wall-clock time and computational requirements. \n2. The exposition of concepts can be improved as it lacks accuracy at points, which causes confusion (see the comments below). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lHVN9vZZXEo", "reviewer": "Reviewer_rdoE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the question of policy diversity in RL, with the specific goal of optimizing the diversity of obtained policies _within_ different quality levels (as one would, for example, want different AI difficulty levels within video games). \n\nTo do so, the authors first propose a framework that describes this specific task, which they refer to as QSD (quality-similar diversity), with an associated QSD score. This score is meant to apply to diversity metrics built upon user-specified behavior descriptors (BDs), as opposed to diversity metrics that are learned as part of the optimization process. \n\nSecond, the authors show that one can obtain unbiased estimates of the gradient of such a user-specified BD with respect to the policy (and thus, an estimate of the diversity metric's gradient). \n\nFinally, the authors use population-based algorithms to optimize the QSD score, showing across a variety of continuous and discrete tasks that their proposed algorithm (QSD-PBT) outperforms a variety of RL algorithms with different approaches to diversity optimization.", "summary_of_the_review": "This paper addresses the problem of learning diverse policies of differing quality. This paper is well-written, with theoretical results that may be of independent interest, and a novel population-based algorithm that is experimentally verified across a variety of tasks. Although I have a few remaining questions about the experimental evaluation, this is overall an interesting work.", "strength_and_weaknesses": "# Strengths\n- Clarity: this paper is very clearly written; the overall phrasing of the problem, literature review, and description of the methodology are all very clear.\n- The experiments verify the usefulness of the proposed approach, and compare to a wide range of baselines.\n- The main theoretical results on estimating the diversity gradient may be of independent interest.\n- The details in the final QSD-PBT algorithm are investigated with a corresponding ablation study which confirms the discussion in the main paper.\n\n# Weaknesses\n- The one section I found could benefit from some clarification is $\\S$ 3.3; I found the description of the mechanism by which the different $\\lambda_i$ are obtained somewhat unclear. Am I correct in understanding that for each stratification level, all $\\lambda_0$ are equal (set to a high value which presumably is not particularly influential)? How does this tie into the reference to $\\lambda_{\\infty, 1}$ and $\\lambda_{\\infty, 2}$?\n- One potential weakness is that all evaluations of QSD-PBT use the introduced QSD-score metric. I'd be interested in whether the authors considered other ways of combining the stratified diversity terms of (1) (for example, one could consider the worst diversity score across all intervals, although the resulting optimization difficulties may be sufficient to discard this out of hand). \n\n# Questions\n- Maybe I missed this result: do you report the highest quality discovered by each algorithm (regardless of diversity)? On a related note, how are the quality intervals defined for Figure 1: are they predetermined by the range of possible rewards, or are they chosen based on the highest reward over all approaches $R_{max}$?\n- Figure 1 shows that there is a trade-off between quality and diversity; however, this trade-off disappears in most of Figure 3. Do you know what this could be due to?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster