Diversifying Policy Behaviors with Extrinsic Behavioral Curiosity
Authors: Zhenglin Wan
Xingrui Yu
David Mark Bossens
Yueming Lyu
Qing Guo
Flint Xiaofeng Fan
Yew-Soon Ong
Ivor Tsang
Venue: ICML 2025 poster (2025)
Keywords: Quality Diversity
Reinforcement Learning
Imitation Learning
Primary area: reinforcement_learning->inverse
TL;DR: We designed a novel Extrinsic Behavior Curiosity module to significantly improve the efficiency and performance of Quality Diversity Policy Learning algorithms.

Abstract: Imitation learning (IL) has shown promise in various applications (e.g. robot locomotion) but is often limited to learning a single expert policy, constraining behavior diversity and robustness in unpredictable real-world scenarios. To address this, we introduce Quality Diversity Inverse Reinforcement Learning (QD-IRL), a novel framework that integrates quality-diversity optimization with IRL methods, enabling agents to learn diverse behaviors from limited demonstrations. This work introduces Extrinsic Behavioral Curiosity (EBC), which allows agents to receive additional curiosity rewards from an external critic based on how novel the behaviors are with respect to a large behavioral archive. To validate the effectiveness of EBC in exploring diverse locomotion behaviors, we evaluate our method on multiple robot locomotion tasks. EBC improves the performance of QD-IRL instances with GAIL, VAIL, and DiffAIL across all included environments by up to 185\%, 42\%, and 150\%, even surpassing expert performance by 20\% in Humanoid. Furthermore, we demonstrate that EBC is applicable to Gradient-Arborescence-based Quality Diversity Reinforcement Learning  (QD-RL) algorithms, where it substantially improves performance and provides a generic technique for learning behavioral diverse policies. The source code of this work is provided at https://github.com/vanzll/EBC.

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
Zhenglin Wan
Xingrui Yu 22 David Mark Bossens Yueming Lyu 23
Qing Guo Flint Xiaofeng Fan
Yew-Soon Ong 234 Ivor W. Tsang 234
Abstract
1. Introduction
Imitation learning (IL.) enables intelligent systems to quickly learn complex tasks by learning from demonstrations, which is particularly useful when manually designing reward function is difficult. IL has been applied to many real-world scenarios such as autonomous driving (Bojarski, 2016), robotic manipulation (Zhu et al., 2018), surgical skill learn- ing (Gao et al., 2014), and drone control (Ross et al., 2013). The concept of II. relies on the idea that experts can show- case desired behaviors, when they are unable to directly code them into a pre-defined program. This makes IL ap- plicable to any system requiring autonomous behavior that mirrors expertise (Zare et al., 2024). IL can be divided into two paradigms: behavior cloning (BC), which mimics ex- pert's s exact state-action pairs via sneervisdd-eaarinng and inverse reinforcement learning (IRL), which infers a reward function from expert demonstration and uses reinforcement learning (RL) learn the policy.
Imitation learning (IL) has shown promise in var- ious applications (e.g. robot locomotion) but is often limited to learning single expert policy, constraining behavior diversity and robustness in unpredictable real-world scenarios. To address this, we introduce Quality Diversity Inverse Re- inforcement Learning (QD-IRL), a novel frame- work that integrates quality-diversity optimiza- tion with IRL methods, enabling agents to learn diverse behaviors from limited demonstrations. This work introduces Extrinsic Behavioral Cu- riosity (EBC), which allows agents to receive ad- ditional curiosity rewards from an external critic based on how novel the behaviors are with respect to a large behavioral archive. validate the effec- tiveness of EBC in exploring diverse behaviors, we evaluate method on multiple robot loco- motion tasks. EBC improves the performance of QD-IRL instances with GAIL,, VAIL, and DiffAlL. across all included environments by up to 185%, 42%, and 150%, even surpassing expert perfor- mance by 20% in Humanoid. Furthermore, we demonstrate that EBC is applicable Gradient- Arborescence-based Quality Diversity Reinforce- ment Learning (QD-RL) algorithms, where it sub- stantially improves performance and provides a generic technique for learning behavioral diverse policies. The source code of this work is provided at http:///ituub..oommaallf/f///
However, traditional IL methods tend to replicate only the specific strategies demonstrated by the expert. If the expert demonstrations cover a narrow range of scenarios, the II. may struggle when faced with new or unseen situations. Additionally, IL faces challenges in stochastic environments where outcomes are uncertain or highly variable. Since the expert's actions may not capture all possible states or con- tingencies, IL often struggles to learn an optimal strategy for every scenario (Zare et al., 2024). These limitations are further exacerbated when the demonstration data is limited, as the IL. algorithm will only learn specific expert behavior patterns. Hence, traditional IL methods are significantly constrained due to the lack of ability learn diverse behav- ior patterns to adapt stochastic and dynamic environments. Moreover, since BC is much more susceptive to the afore- mentioned challenges (Ross et al., 201 1), this work will mainly focus on IRL as our IL paradigm.
"Equal contribution Scchoo of Data Science, The Chinese University of Hong Kong, Shenzhen, China ²CFAR, Agency for Science, Technology and Research, Singapore $IHPC, Agency for Science, Technology and Research, Singapore 'College of Comput- ing and Data Science, Nanyang Technological University (NTU), Singapore. Correspondence to: Xingrui Yu <yu_xingrui cfarr.- star star.edu.ss>.
On the other hand, the Quality Diversity (QD) algorithm is an emerging optimization paradigm that designed to find set of diverse solutions to optimization problems while maximizing each solution's 's fitness value (fitness refers to the problem's 's objective) (Pugh al., 2016). For instance, QD algorithms can generate diverse human faces resem- bling "Elon Musk" with various features, such as different
Proceedings of the 42nd International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
eye colors (Fontaine & Nikolaidis, 2021a). In robot control, the diverse behaviors trained by QD algorithms excels al- lows adaptation in changing environments (Tjanaka et al., 2022). For example, if an agent's ss leg is damaged, it can adapt by switching to a policy that uses the other undam- aged leg to hop forward due to the diverse behaviors stored in the QD archives (Fontaine & Nikolaidis, 202 Natu- rally, one valuable question is raised: can we design a novel IRL framewokk that can combine the respective strengths of traditional IRL and QD algorithms, enabling the agent to learn a broad set of high-performing policies from limited demonstrations? This framework, which we call Quality Diversity Inverse Reinforcement Learning (QD-IRL), leads to a key challenge of combining QD with gradient-based policy optimization methods: the gradient-based policy op- timization methods (such as Proximal Policy Optimization (Schulman et al., 2017)) tend converge to the policies with high quality, i.e. high cumulative reward, while ignoring behavioral diversity. While the challenge exists for QD-RL too, it will be exacerbated in QD-IRL since the reward func- tion learned by IRL methods will be much more localized, largely inhibiting balanced exploration (Yu et al., 2025).
prove that EBC can similarly be applied in QD-RL, signif- icantly improving the QD performance by simply adding the bonus into the reward function. We summarize our contributions as follows:
We propose a QD-IRL framework, combining the advan- tages of QD and IRL while addressing the key limitation of IL for various applications.
We identify the key challenges of combining QD and gra- dient based policy optimization algorithms. To address this challenge, we introduced Extrinsic Behavioral Curios- ity (EBC) to encourage behavior-level exploration, and theoretically proved the effectiveness of our method.
Our framework can potentially enhance any IRI. applica- tion that requires learning diverse policies, and can also significantly improve existing SOTA QD-RL algorithm, thus providing generie framework for future research of learning behevioral-diverse policies.
2. Preliminaries and Related Work
To address this challenge, we introduce Extrinsic Behav- ioral Curiosity (EBC). Inspired by the concept of curiosity in psychology, where humans seek out novel and diverse experiences for the sake of learning, EBC encourages ex- ploring diverse behavioral patterns. Specifically, when the agent identifies a new behavior pattern, a reward bonus will be added to the current reward function. Due to the nature of gradient-based policy optimization, the reward bonus will prevent stagnation at local optima and promote balanced exploration. After one sub-region of behavioral patterns is unlocked, the reward bonus of this region is reduced to 0 for further exploitation. Distinet from Intrinsic Curiosity Module (ICM) (Pathak et al., 2017), EBC adopts a higher- level curiosity, owing its name to two defining features. First, EBC promotes the exploration of unseen behavior pat- terns rather than unseen states. Second, the curiosity reward comes from an extrinsic source; that is, our approach allows an external critic to judge whether or not a behavior is novel or not based on a QD archive stored externally to the agent. Similar ideas have been explored in the work of (Yu et al., 2025), which employs a reward bonus to foster behavioral exploration. However, whereas Yu et al. work compute single-step bonus for each (s, a) pair, our work introduces an episode-level bonus which aligns closely with episode- based measure computation. To validate our framework, we conduct experiments with limited expert demonstrations across various environments. Notably, our framework is po- tentially capable of enhancing any IRL. method for QD tasks. reaches or even surpasses expert performance in terms of both QD-Score and coverage in the Walker2d and challeng- ing Humanoid environments. Meanwhile, we empirically
2.1. Quality Diversity Reinforcement Learning
Distinct from traditional policy optimization which aims to find a single policy to maximize the cumulative reward, Quality Diversity Reinforcement Learning (QD-RL) aims to find a set of high-performing and diverse policies. Parametrizing policies based on 0 E ² C tive function (i.e., fitness) is defined as f O > R, which is the cumulative reward in RL context. The diversity is based on a k-dimensional measure function m : TR." -> R. The goal is to find policies O O for each local region in the behavior space B m (0). In grid-based algorithms A-P-Elites, one discretizes B into M cells (local regions) and forms policy archive eá where each cell i = 1,..., .M represents a small hypercube |a, by] within a multi-dimensional grid of the behavior space. By filling new cells and replacing existing solutions if they are outperformed, the algorithm gradu- ally evolves an archive of high-quality and diverse policies (Chatzilygeroudis al., 2021; Pugh et al., 2016; Mouret & Clune, 2015a;b). Based on MAP-Elites, Co- variance Matrix Adaptation MAP-Eliter (CMA-ME) (Fontaine et al., 2020) further adopts covariance matrix adaptation as the evolution algorithm to find new high-performing solutions more effectively and efficiently. In such algorithms, QD-RL can be treated equivalently to solving M ootii--optimization problems of the form
RR. the objec-
max
f(0)
s.t. m (0)
[a. bi].
where i E (1,2,...,,M) and each constraint corresponds to one cell in the behavior space. Because traditional policy-
gradient algorithms will simply optimize the policy towards
2

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
the highest cumulative reward, they ignore valuable behav- ioral diversity. This problem is also referred to as the "be- havioral overfitting" issue (Yu et al., 2025), which is one of the key challenges to QD-RL.
erties for non-stationary problems such as the QD objective (Eq. 2).
Concise paragraphs struggle to adequately summarize the extensive QD-famlly. and understanding the rationale be- hind CMA-MEGA and PPGA requires effort. We recom- mend readers to explore prior works in depth (Batra et al., 2023; Fontaine & Nikolaidis, 2021a) or refer to Appendix A.2 and A.1 for further algorithmic details.
The majority of QD-RL works focus on the paradigm of Differentiable Quality Diversity (DQD), which makes use of policy gradient techniques to guide the search for quality diversity. One seminal algorithm in this regard is the Co- variance Matrix Adaptation MAP-Elites via Gradient Ar- borescence (CMA-MEGA; (Fontaine & Nikolaidis, 2021b)), which maximizes the QD objective
2.2. Imitation Learning
In Imitation learning (IL) (Zare et al., 2024), an agent learns high-performing policies from demonstration data. A tradi- tional approach to solve this challenge is Behavior Cloning (BC), which uses supervised learning to learn the policy from demonstrations, a technique which unfortunately suf- fers from severe error accumulation (Ross et al., 2011) 1). More recent techniques include inverse reinforcement learn- ing (IRL), where one seeks to learn a reward function from the demonstrations and then use RL to train a policy based that reward function (Abbeel & Ng, 2004).
8(0) cof(O)) + Σ (0)
via gradient ascent, where co 2 0 and e E IR for all j. where f the fitness (i.e. the cumulative reward) and mj are the measure functions. The coefficient vector e is sampled from distribution, which is optimized adaptively by the evolutionary strategy (ES) algorithm CMA-ES to optimize the archive improvement, which is based on the total sum of fitness in the archive, where the fitness refers to cumulative reward of the policy.
Early IRL. methods estimate rewards using the principle of maximum entropy (Ziebart et al., 2008; Wulfmeier et al., 2015; Finn et al., 2016). Recent adversarial IL methods treat IRL as a distribution-matching problem. For instance, Generative Adversarial Imitation Learning (GAIL) (Ho Ermon, 2016) trains discriminator to differentiate between the state-action distribution of the demonstrations and the state-action distribution induced by the agent's policy, and output reward to guide policy improvement. Improving on GAIL, Variational Adversarial Imitation Learning (VAIL) (Peng et al., 2018) applies a variational information bot- tleneck (VIB) (Alemi et al., 2016) to the discriminator, improving the stability of adversarial learning. Another technique for adversarial IL. is Adversarial Inverse Rein- forcement Learning (AIRL) (Fu et al., 2017). which learns robust reward function by training the discriminator via logistic regression to distinguish expert data from policy data. Generative Intrinsic Reward-driven Imitation Learn- ing (GIRIL) (Yu et al., 2020) computes offline rewards by pretraining a reward model using a conditional VAE (Sohn et al., 2015), combining action encoding with forward dy- namics, and has shown superior performance with limited demonstrations. Most recently, DiffAIL (Wang et al., 2024) adopts the loss function of a diffusion model as a distri- bution matching technique, enhancing the discriminator's performance and generalization capabilities.
In CMA-MEGA, the gradients are assumed to be analyti- cally given, i.c. the measures are directly differentiable as function, which is limitation. Several works in DQD approximate the gradients. Policy Gradient Assisted (PGA) MAP-Elites (Nilsson & Cully, 2021) incorporates TD3, which is suitable for off-policy learning based on deter- ministic policy gradient of the fitness. QD Policy Gradient (QD-PG) (Pierrot et al., 2021) introduces the diversity gra- dient, which uses both quality and diversity crities in TD3 algorithm, where the diversity reward is based on the dis- tance between state measures with similar policy parameters. Unfortunately, it is a challenge that this state measure is not directly related to the behavior, which can be based on the episode or directly on the parameters. The state-of-the-ar QD-RL algorithm, Proximal Policy Gradient Arborescence (PPGA), leverages a vectorized PPO (VPPO) architecture to parallelize gradient approximation: the objective fune- tion is assigned dedicated thread, while each measure function operates on its own thread, enabling efficient paral- lelized computation of both objectives and measures (Batra et al., 2023). PPGA introduces Markovian Measure Proxy (MMP), a state-based surrogate measure function that corre- lates strongly with the original measure and allows gradient approximation via policy gradient by treating it as a reward function. It approximates the gradients of both the objective and the & measure functions by subtracting the policy pa- rameters before and after multiple PPO updates. A further modification is its use of exponential evolution strategies (xNES) (Glasmachers et al., 2010), instead of CMA-ES, for updating the search policy, which has more favorable prop-
Inverse Reinforcement Learning (QD- IRL) follows the definition of QD-RL, optimizing the same objective, namely the cumulative reward. The key difference is that the true reward function is not accessible. and we infer rewards from the reward model A trained with the expert demonstration 9.
3

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
While existing work on Multi-domain Imitation Learning (Seyed Ghasemipour al., 2019; Yu et al., 2019; Chen et al., 2023) aims to learn confext-conditioned policies that gen- eralize across tasks, our approach differs in that we focus on constructing a large archive of policies that represent diverse behaviors. Similarly, skill or option discovery meth- ods (Zhang et al., 2021; Kipf et al., 2019; Jiang et al., 2022) also aim to "find diverse solutions" for policies exhibiting varied behaviors. However, these methods typically gen- erate low-level policies for use within a higher-level MDP (e.g,, an MDP with options), often linked to hierarchical RL, where low-level policies are triggered and terminated by specific sub- sub-goals. In contrast, our goal is not to create policies for a hierarchical setup, but rather to build a diverse and tigh-peerfrmiig set of policies independently of any hierarchical structure.
Sample haich Gpand ssig curmed pllc
R. Uptro morch pllcy
Uiptte ourth dalllvon by Eagooeacci
Bresc aadeer byo strar-
t
1
Joday data
11
-PAll
-Rettt
datta
11
Puittz
Talees rovul
2. Dompdd SAL, 20
1QC- BI--ILI
300-81)
Doogui (cmard beaaa
praket Ifal (P.m)
warth ditilati
Figure 1: Flow diagram of QD-IRL or QD-RL with EBC. The h in step 4 means the cumulative EBC reward.
Algorithm QD-IRL with PPGA as backbone
1: Input: Initial policy &. VPPO instance to approximate Vg, Vm and move the search policy, number of QD iterations No. number of VPPO iterations to estimate the objective-measure functions and gradients Ni, num- ber of VPPO iterations to move the search policy N2, branching population size 2, and an initial step size for xNES og. Initial reward model L. Expert data 9. 2: Initialize the search policy 0μ = Oo- Initialize NES parameters μt T2!
3. Extrinsic Behavioral Curiosity for QD-IRL
In this section, we will introduce Extrinsic Behavioral Cu- riosity (EBC), which provides the QD-IRL agent with a curiosity-driven reward bonus to encourage behavior space (a.k.a. policy archive) exploration and address the behavioral-overfftting issue of QD-IRI..
3: for iter + 1 to N do
4: f.8. Vg, m, Vm <- VPPO.jacooiann 0g. M,m,N D
We implement EBC on three different IRL. methods, namely GAIL, VAIL, DiffAIL. These IRL methods will learn a reward function from expert demonstration, and the QD- RL algorithm PPGA (Batra et al., 2023) is then applied on this learned reward train the policy archive. Notably,
approx grad using A8
5:
Vg + normalize Vg),
Vm f normalize(Vvi
< update archive (0u fu..f.m
7:
for 1 to 2 do
branching solutions
8:
c~ "(,,(,E)
P sample gradient coefficients V. Vg c
EBC could be easily integrated into various QD-IRI. algo- rithms and QD-RL algorithm PPGA by simply adding the reward bonus into the learned reward function or true reward function. Figure 1 shows the workflow of our framework. We first sample episodes to form the current search policy. For QD-IRL, we infer rewards from the reward model 2 pre-trained using the sampled episodes and expert demon- strations. While all of the optimisation is being done based on imitation rewards, as in (Yu et al., 2025), the solutions are added to the archive based on oracle fitness evaluations to avoid the additional layer of complexity introduced by accumulating effects of fitness estimation errors. For QD- RL, we always use the true rewards from the environment. Then in the third step, we compare the archive and use the measures of episodes to calculate the EBC reward bonus.
9:
10:
0 - 0y +V f "',,m', m' + rrolouut())
11:
12:
A + update_arreeae (atiate_rrchieeeee (rrcive(O',.f'.. ,,') > get archive
improvement of each solution. end for
13:
14:
Rank gradient coefficients V, by A
15: Adapt xNES parameters μ H Σ = E' based on
improvement ranking
16:
g' g((Og) == (p,) + E cu ((,)), where
Cμ ==
b construct EBC-QD objective function walk search
17:
0 <- VPPO..rr ..N2,
policy using reward model SP
18:
L .u.uate(22. 0%)
update reward model
19:
if there is no change in the archive then
The orange region of current archive indicates the occupied cells, and the white region indicates the empty cells. If the episode measure occupies the empty cell, the EBC re- ward bonus 'EbC will be applied to each step of this episode. Then VPPO uses the reward values after applying bonus to approximate gradients for the objective and measures. Sub- sequently, these gradients are used to produce new solutions, update the archive, update the search distribution, and the search policy based on the CMA-MAEGA paradigm.
20:
Restart xNES with μ 0,E =,E
21:
Set 0p to a randomly selected existing cell 0; from the archive
22: end if
23:
end for
We provide the pseudo-code of the general procedure of QD-IRL with PPGA in Algorithm 1, where different IRL methods differ from the "update reward model" part and
4

---

other parts requiring the reward model calculate learned reward (highlighted in red). Please refer to Appendix B for algorithms for updating the archive (Algorithm 2), updating the reward model, calculating the rewards with the EBC Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
Proof. (I). The objective of policy optimization is
h(0,,0))) -Resaa
Σ pr((s (() 1=0
bonus, and computing the gradients for the objective and measures (Algorithm 3).
Bg
I(mm' E Ae)
Σx
(3)
1=
3.1. Derivation of the EBC reward bonus
+1 Eenay
[I(m' E =xh))]
The objective of PPGA is based on Eq. 2. We observe that the fitness term f heavily influences PPGA's search policy update direction, as archive improvement is primarily driven by f. Therefore, PPGA frequently becomes stuck in local regions because f is calculated by cumulative reward and the policy tends to converge in local optima region while lacking exploration of other regions. Hence, PPGA often generates overlapping solutions with only marginal archive improvement. Additionally, a key challenge in QD-IRL is the conflict between imitation learning and diversity. Lim- ited expert demonstrations lead behaviorally localized and sometimes misleading reward functions, further exacer- bating the problem by restricting search policy updates.
where 7' is the episode length (rollout length). Since PPO is assumed to monotonically improve the policy, optimizing
h Gaa Ae) through multiple rounds of PPO results in a Onew with increased objective, i.e. h h(Onews A.) > h (oOoldd se). Therefore, we have
E Eg [I(m's E [aeeee
2 B [I(m' E df))
(4)
Since En [I(m's E <dce)] P E Ae(0) where m' is the measure of episode T; it follows that
P(m' ol Onew) 2 P(m' E dc(o)))
To encourage the search policy to find new behavior patterns (i.e., to explore the empty area in the policy archive), we formulate a reward function based on an indicator function over the archive, i.e. whether the behavior cells has already been explored or not. Since policies and environments are often stochastic, Lemma 3.1 provides a probabilistic guaran- tee for an EBC reward. More specifically, Lemma 3.1 states that using the indicator function T(m E .x)2) on the empty area of the current archive (denoted as ax) as the reward function in the standard PPO objective steadily increases the probability that the policy generates new behavior measures.
(2). Based on Bayes' rule with uniform prior on 0, we have
P(m' E ((0))(() x P(m' df))). P(0 E e.).) P(m' E dc)
Since P(0) and P(m's E .6) can be treated as constants when changes, we have
P(Onew |m' E ed)e 2 P(Oooll E de)).
Lemma 3.1. Suppose the reward function is given by r(s),d)= 100=[((m T(mm E d.), where s; and dt represent the state and action time step of episode de is the empty area of archive s4, and 1(m' E de) is the indicator function in- dicating whether the measure of the i'th episode is in de. Moreover, let O E O be parameter set with uniform prior over O. Then if one iteration of PPO successfully increases the objective value, the following inequalities hold:
Following Lemma 3.1, we define the EBC reward bonus for all time steps of episode as
IB (mmmm))):::))) = qI(mm E de),
:=)
(5)
where dle is the empty area of the current archive, m' is the measure of episode i, and the hyperparameter & controls the weight of the EBC reward. Note that the EBC reward is spisode-hased i.e. it is calculated at the end of each episode and added to each step of this episode. Essentially, we compute a policy's behavioral curiosity score at the trajectory's 's end and propagate this score back each step's s reward to credit the actions in that episode.
(1): Pmmp E Ed(00w))) 2 P(m' E de Omts) and (2): : P(0 m E c.o6)) 2 P( Gold m' E ale),
where P(m' E d.)0) is the likelihood, i.e. the probability under policy Ry that the measure of episode i belongs to the unoccupied area de, and P(0mm E d.) is the posterior, i.e. the probability that the policy that generated an episode- based measure m² E de is parametrized by 0.
By combining the EBC reward bonus with the imitation reward, the EBC algorithm adaptively balances exploration and exploitation; once a region in the archive has been explored, the bonus of this region is set to zero such that the emphasis is again more on exploitation.
5

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
3.2. Synergy of EBC reward bonus with PPGA
3.3.1. GAIL-EBC
We briefly discuss how using the EBC reward bonus syner- gizes with PPGA. The algorithm (see Algorithm 1) uses evolutionary strategy (ES) algorithm, namely xNES, as emitter to adaptively search for diverse and nigh-peeforming policies based on a gradient coefficient vector с sampled from a search distribution .A L),
Using GAIL (Ho & Ermon, 2016) as a backbone, GAIL- EBC trains discriminator Dy according to
minmax B(sa)) log( (((-DD((,a))) v 620
(7)
+ (s,a)= log(1 (s.
If we apply a reward bonus the original reward, the ob- jective of PPGA transforms into:
where D is the expert dataset and π is the learned policy, The reward function for GAIL-EBC is given by
af m', ..) = - log (1 Dv (s) a))) rEBC(m', Ad).
max 8
(col((0, + Σ com;(0),
(6)
(8)
where de is the empty region of the current policy archive.
3.3.2. VAIL-EBC
where (()) .f.) combines the cumula- tive EBC reward h h(0,ok) (ol, the empty area of current archive) and the fitness f())) The aim of the ES algorithm to guide the search policy towards the highest archive improvement, i.c. increased total fitness value of all policies in the archive (see Algorithm in Appendix B). Firstly, ES samples multiple coefficient vectors where each one leads to a new policy (in the sense that when the gradient of f and my are calculated using VPPO, the coefficients of these gradient determine the policy update). The ES algorithm then ranks sampled coefficient vectors based on the archive improvement of corresponding new policy and updates the search distribution M (1,, Σ) accordingly. The search policy ON is then updated using the mean of the posterior search distribution as gradient coefficient vector.
VAIL (Peng et al., 2018) modifies GAIL by using a varia- tional information bottleneck to the discriminator. Using VAIL as a backbone, VAIL-EBC trains a discriminator Dy according to
min max (r.) (owE((alaa) (ooraaaaaaaooggaaaa (wg(-Dw(<2)) Dy.E' $20
B(sa)~ ( | -log -v]][] (9)
+ +BE [dxr (F' (z|x; -c]]
where Ā is the mixture of expert policy and agent policy, and E' is the latent variable encoder. The reward function for VAIL-EBC is given by
r(s),af, of) -log (I DD((
The search policy then allows the population to explore from a promising location in policy space. First, VPPO computes the gradients for the objective and measures. Then, search distribution samples iid gradient coefficient vectors e'
(10)
+ MEaC (m', ,f),
where (s,, represents the mean of the encoded latent variable distribution.
N EH, to explore the policy space 0; +- 0μ Vü
3.3.3. DIFFALL-EBC
DiffALL (Wang et al., 2024) is a state-of of-the-artttttt adversarial II. method which adopts the distribution-matchiee capabil- ity of diffusion models to enhance the discriminator, The objective for training the discriminator based on the same principle as GAIL, but incorporates diffusion noise for more robust discrimination:
V co Vos Op A,) +
Vem
Making use of the gradient over in this manner allows to factor in the unoccupied areas to define appropriate gradient coefficients. In many cases, the archive improvement is higher when new cell is occupied than when replacing existing elite. Therefore, combined with Lemma 3.1, the resulting algorithm rapidly improves the archive, yielding diverse and high-perforning behaviors.
min max Be (0..ro000o----11))-))))))-- Ko Dy
Ex [ogg(D((xEE.))
+ Ex [log( -D((x..t)]]]
(11)
where (x, E,r) is the discriminator function at time and E E~ N (0,1). The specific formulation of Dy is given by
3.3. QD-IRL instances with EBC bonus
Having defined the EBC bonus and its connection to PPGA, we now discuss three implementations (GAIL-EBC, VAIL- EBC and DiCC,IL-EBC), which are based on standard imi- tation learning methods GAIL,, VAIL, and DIf
D Dw E.1) exp (-Diff (x,,E,1)
= cxx (-Ile [ Chx, + / -VIa,)))))
(12)
6

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
where Ey is a model parameterized by y to predict e from X, and I. This follows DDPM (Ho et al., 2020), where rather than approximating the process mean, the process noise is predicted. This is based on reparametrization trick which represents the forward process as X, v0,xo + vI - a,e, where a, = =1 (1 - B/) is the variance schedule B. The reward function for DiffAIL-EBC is:
settings for all baselines. Hyperparameter details are pro- vided in Appendix E. All the experiments are averaged with three random seeds. We also study the effect of hyperparam- eter which controls the weight of EBC reward bonus term in Eq. 5. We found q = 2 performs the best among 2,,,, and thus use q 2 for GAIL-EBC, VAIL-EBC and DiffAlL- EBC across the experiments. Please refer to Appendix E.3 for details.
1 T -
T Σ log (1 D ((((
Figure 2 compares the training curves across four metrics for three selected IL methods, their respective EBC-improved version, and the expert (PPGA with true reward function). By comparing the line with the same color, key observations include: 1) In Walker2d, EBC consistently improved GAIL, VAII. and DiffAlL. in terms of the most crucial QD metric -the QD score - by 38%. 32% and 71%, respectively. In Humanoid, EBC consistently improved GAIL, VAIL and DiffAIL in terms of the most crucial QD metric-QD score by 185%, 42% and 150%, respectively. In Halfcheetah, the performance of DiffAIL and its EBC-improved version is approximately the same, but EBC improves GAIL and VAIL in terms of QD-Score by 30% and 5%, respectively. 2) The coverage metric of all EBC-improved model consistently reaches near 100%, and significantly outperforms other the respective naive version and PPGA with true reward. No- tably, PPGA expert with true reward explored less than 50% of the cells in Humanoid. 3) In Humanoid, the performance of VAIL-EBC and DiffAIL-EBC reaches or even surpasses PPGA with true reward in terms of QD-Score by about 20%. It is also notable that the BestReward of EBC variants takes longer to converge, which is normal because the higher the coverage, the longer it will take to evolve the elite solutions for all the covered cells. For detailed quantitative results (including three IL baselines) on the different QD metrics, please refer to Table 8 in Appendix G. We also conducted experiments to report the results using an additional metric named Complementary Cumulative Distribution Function (CCDF) (Vassiliades et al., 2017), please refer to Appendix FF for details.
+ + (4)))
(13)
4. Experiments
4.1. Learning Diverse High-Perforring Skills from Limited Demonstrations
Experiment Setup We evaluate our framework on three popular MuJoCo (Todorov et al., 2012) environments: Halfcheetah, Humanoid, and Walker2d. The goal in each task is to maximize forward progress and robot stability while minimizing energy consumption, and the measure function maps the policy into a vector where each dimension indicates the proportion of time a leg touches the ground. We evaluate using four common QD-RL metries: 1) QD- Score, the sum of scores of all nonempty cells in the archive. QD-Score is the most important metric in QD-IRL as it aligns with the objective of QD-IRL as in equation (1); 2) Coverage, the percentage of nonempty cells, indicating the algorithm's 's ability to discover diverse behaviors; 3) Best Reward, the highest score found by the algorithm; and 4) Average Reward, the mean score of all nonempty cells, reflecting the ability to discover high-performing policies across the behavior space. We use the true reward functions to calculate these metrics. Please refer to Appendix C for hardware setup and more implementation details.
Demonstrations We use a policy archive obtained by PPGA to generate expert demonstrations. In line with real-world scenario with limited demonstrations, we first sample the top 500 perforrnrrre elites from the archive candidate pool, and then select few demonstrations such that they are as diverse as possible. This process results in 4 diverse demonstrations (episodes) per environment. Appendix D shows the statistical properties for selected demonstrations.
Policy Archive Visualization Figure 3 visualizes the policy archives learned by PPGA with true reward (True Reward), GAIL, GAIL-EBC in Halfcheetah and Humanoid. The archive produced by GAIL-EBC covers a much larger area, highlighting the importance of behavior exploration.
Extension to QD-RL Surprisingly, observe that VAIL- EBC achieves near-expert QD- and that DiffAIL-EBC even surpasses expert QD- in the Humanoid environ- ment in Figure 2. This is counter-intuitive because it is common belief that IL can hardly surpass expert perfor- mance. However, one may argue here for the benefit of IRL compared to traditional RL because the imitation reward function may be aligned with the true reward but potentially be casier to learn due to the function being less sparse and
Overall Performance As mentioned in section 3, we focus on applying EBC three IRL methods-GAIl VAIL and DiffAlL, to compare the performance improvement. Addi- tionally, we also include few widely-used and state-of-the- art IL methods as baselines: 1) Traditional IRL: Max-Ent IRL, 2) Online reward methods: AIRL, and 3) Data-driven methods: GIRIL. Each baseline learns a reward function, which is then used to train standard PPGA under identical
7

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
QD-Score
Coverage(%)
BestReward
AverageReward
Haltcheetah
6030
1003
40-
2030
533
592-
795
1210 1500 1Pa 2003
293
7in
124
1981
19o
2030
290
1866
2000
2.00
560
1838
2000
3.2
3.0
Walker2d
750
2038
530
05-
1830
2250 2790 2006
1230
2006
250
2006
2000
1006
3508
5200
00:00
4003
Humanold
6030
3323
2023
2030
3799
3239 Iterations
250
3506
3508
Iterations
Iherations
The Reward
GAIL_ EBC
GAIL
VAIL_ EBC
VAIL
DiffAIL_ES EBC
DiffAIL
Figure 2: EBC improves performance of QD-IRL instances with GAIL, VAIL and DiffAIL in three robot locomotion task. Lines indicate the mean and shaded areas indicate the standard deviations.
True Reward
GAIL
GAILEBC
QD-Score
Coverage(%))
Nolthvetah
212
710
2008
130
100
1000
230
180
700
2008
1500
100
100
Humanold
50000
AverageReward
1000
0m0
Figure 3: Visualization of well-trained policy archives.
continuous. To assess this potential benefit, add our EBC reward bonus to the true reward function of the QD-RL task. Figure 4 shows that the QD performance sig- nificantly improves after the EBC reward bonus is applied. The increase in coverage is particularly high, namely from 60% to near 100%, justifying the effectiveness of the cu- riosity module in exploring diverse policies. PPGA with true reward and EBC bonus significantly outperforms all IRL algorithms all metrics, with the exception of the coverage.
Thue Leword-EBC
VAIL-EBC
Figure 4: EBC improves PPGA with true reward (True Reward) in the Humanoid environment.
differ in both the type and the number of dimension of the QD measure Specifically, scalar that records the height of the lowest foot at each step; 2) Angle (dim=2), a 2-D vector & = (cos a, sin a) capturing the yaw orientation a of the torso about the z-axis; and 3) Feet Contact (dim=4), a 4-D vector that stores, for each of the ant's four legs, the fraction of time steps dur-
used: 1) Jump (dim=1) 1),
Experiments on Various Measure Metrics To further as- sess the generality our framework, we applied EBC to the GAIL baseline (GAIL-EBCC and evaluated it on tasks that
8

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
ing which the leg touches the ground. Table reports that GAIL-EBC improves QD-Sccre of GAIL throughout every scenario and significantly improve the Coverage of GAIL in humanoid environment.
Hurdles requires the agent to jump over hurdles of varying heights. Humanoid MMooor Failure requires the agent to adapt to different degrees of failure in the motor controlling its left knee. Finally, Walker2d - Friction requires the agent to adapt to varying levels of ground friction. Here, we eval- uate the agent's s ability to adjust its locomotion strategy to a new perturbed MDP. See Appendix J for more details.
Table 1: Performance of GAIL versus GAIL-EBC on tasks with measures of increasing dimensionality: Jump (1-D), Angle (2-D), and Feet Contact (4-D). Results are reported as mean = one standard deviation over three seeds.
Evaluation Metries Following (Grillotti al., 2024), we report the inter-uartill Mean (IQM) value for each metric with the estimated 95% Confidence Interval (CI) (Agarwal et al., 2021).
Enviromment Measure (dim)
Model
QD-Score
Coverago (%)
GAII.
64017 + 5.558
100.0 + 0.0
Hopper
Jump(() ())
GAIL-EBC
81 987 1 6 890
100.0 11 0.0
Angle (2)
GAIL
637771 113
53.01300
Results We show that we can harness the learned skills from the EBC-enhanced QD-IRL methods to adapt better than or comparable to its unenhanced counterpart in three different sets of perturbations each of which corresponds to 5 unseen MDPs (Fig. 5).
Humanoid
GAIL-EBC
1702733 19711
0.0
Jump (1)
GATI.
18812-46412
41.0 31.0
GBC
90631 1. 894
85.0 1 15.0
Ant
Feet Contact (4)
GAIL
484-468 + 3 264
75.84=000
GAI
52.511 + 383213
77.28 +:2
Statistical Analysis We use Tukey HSD test to evaluate the statistical significance of the QD-Scores (Table 8). Com- paring our EBC-enhanced algorithms to their counterparts, 7/9 effects are positive and 5 of these are significant with p S 0.05. See Appendix H for more details.
Humaroid- - Hurdias
Humranoid- Motor Falure
Walker2d- - Friction
Fitnass
4000
GAIL
GAIL-EBG
Filmess
2000
GNLL
GAIL-BBC
12000
1000
3002
GAL GAL-BBC
0.05
0.10 hardes height
0.1
0.2
dierage strength
fricion conffiiint
Effects of Number, Quality and Diversity of Demon- strations Here, we study how the quantity, quality, and diversity of demonstrations affect QD performance in QD- IRL. To isolate these factors we apply the EBC bonus a GAIL baseline (GAIL-EBC) on the Humanoid domain and report QD-Score and coverage in Table 10 (Appendix I). Vanilla GAIL shows no consistent benefit-and sometimes a decline- when more demonstrations are added, whereas GAIL-EBC benefits from a notable improvement as diverse demonstrations are added. Even as few as diverse demon- strations are beneficial to yield an observable QD-Score im- provement of about 15%, and not much if any improvement is observed by adding additional demonstrations. Imitation learning with GAIL-EBC from 4 non-diverse demonstra- tions actually reduces the QD-Score by roughly 30%. These results confirm the importance and scalability of imitation learning from diverse experts using GAIL-EBC. The results for GAIL indicate the trade-off between data quantity (num- ber of demonstrations), as well as the fitness and diversity of those demonstrations.
Figure 5: Performance for GAIL-EBC and GAIL in environ- ments with different levels of perturbations after few-shot adaptation. The lines and shaded area represent the IQM and the 95% CI based on exhaustive search over 2 archives.
5. Conclusion
In this work, we propose Extrinsic Behavioral Curiosity (EBC), a simple but effective approach which is highly extendable, and theoretically derived the rationale behind it. EBC can be seamlessly integrated into and can poten- tially improve any IRL. method in QD contexts, providing the generic QD-IRL framework for future research. Our framework follows the paradigm of IRL to learn a QD- enhanced reward function, and uses QD-RL algorithm to optimize a policy archive By encouraging behavior-level exploration, our framework addresses the key challenges of QD-IRL. Extensive experiments show that our framework can often improve QD-IRL methods and significantly out- perform baselines. Additionally, we empirically prove that EBC can also significantly improve the performance of the state-of offte-oofthe--r the-hetrtt art QD-RL algorithm. PPGA, highlighting the sub-optimality of the true reward function in QD-RL set- tings. With EBC, Gradient- Arborescence based QD-RL and QD-IRL algorithms can learn archives of diverse and high- performing policies more efficiently, pushing the boundary of diverse policy acquisition. We also provide potential limi- tations of our work in Appendix K and additional discussion in Appendix L.
4.2. Few-Shot Adaptation from the Imitation Archive
Experiment Setup We evaluate our method on few-shot adaptation scenarios with three types of perturbation (Gril- lotti et al., 2024). For each task, the reward is the same but the MDP's dynamics is perturbed. In few-shot adap- tion tasks, no re-training is allowed and we evaluate the top-performing skills for each method while varying the perturbation to measure the robustness of the different al- gorithms, in particular GAIL-ECB and GAIL.. Humanoid-
9

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
Acknowledgements
Bojarski, M. End end learning for self-driving cars, arXiv preprint arXiv: 160.07316, 2016.
We thank the mous reviewers and area chair for their helpful comments. XY, DB, YL. QG, FXF, YSO and IWT are supported by CFAR, Agency for Science, Technology and Research, Singapore. This research / project is sup- ported by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative, and the National Research Founda- tion, Singapore under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMI.P-2024 004). Any opinions, findings and conclusions or recom- mendations expressed in this material are those of the au- thor(s) and do not reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority.
Chatzilygeroudis, K., Cully, A., Vassiliades, V., and Mouret, J.-B. Quality-diversity optimization: a novel branch of stochastic optimization. In Black Box Optimization, Ma- chine Learning, and No-Free Lunch Theorems, pp. 109- 135. Springer, 2021.
Chen, J., Tamboli, D., Lan, T., and Aggarwal, V. Multi-task hierarchical adversarial inverse reinforcement learning. In International Conference on Machine Learning, pp. 4895-4920 PMLR, 2023.
Ding, L.,, Zhang, J., Clune, J., Spector, L., and Lehman, J. Quality diversity through human feedback: Towards open-ended diversity-driven optimization. 2024. URL https://arxiv: www.awwwwwww...oaaaooooooooooett abt./ass/.../ 221.2.211..1111 12103.
Impact Statement
Finn, C.,, Levine, S., and Abbeel, P. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (ICMI 2016), pp. 49-58. PMLR, 2016.
This research has the potential to significantly advance the field of robotics by enabling robots to autonomously develop wide range of adaptive and efficient locomotion strategies. By leveraging extrinsic behavioral curiosity, this work en- courages robots to explore and master diverse behaviors in complex, unstructured environments.
Fontaine, M. and Nikolaidis, S. Differentiable quality di- versity. In Advances in Neural Information Processing Systems (NeurIPS 2021), volume 34, pp. 10040-10052, 2021a.
The implications of this research are far-reaching. In real- world applications such as search and rescue, planetary exploration, and assistive robotics, robots equipped with this capability can adapt dynamic and unpredictable con- ditions, enhancing their utility and reliability. Furthermore, the emphasis on diversity in locomotion behaviors can lead to more robust and generalizable robotic systems, reducing the need for extensive manual tuning and domain-specific engineering.
Fontaine, M. and Nikolaidis, S. Differentiable quality di- versity. In Advances in Neural Information Processing Systems (NeurIPS 2021), volume 34, pp. 10040-10052, 2021b.
Fontaine, M. and Nikolaidis, S. Covariance matrix adapta- tion map-annealing. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2023), pp. 456-465, 2023.
References
Abbeel, P. and Ng, A. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Interna- tional Conference on Machine Learning (ICMI. 2004), pp. 1, 2004.
Fontaine, M. C., Togelius, J., Nikolaidis, S., and Hoover, A. K. Covariance matrix adaptation for the rapid illumina- tion of behavior space. In Proceedings of thegenetic and evolutionary computation conference (GECCO 2020)), pp. 94-102, 2020.
Agarwal, R., Schwarzer, M., Castro, S., Courville, A. C., and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances neural in- formation processing systems, 34:29304-29320, 2021.
Freeman, C. D., Frey, E., Raichuk, A., Girgin, S., Mor- datch, I., and Bachem, O. Brax-a differentiable physics engine for large scale rigid body simulation, 2021. URL http://github com/google/brax, 6, 2021.
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. arXiv preprint arXiv: 1612,00410, 10, 2016.
Fu, J., Luo, K., and Levine, S. Learning robust rewards with ad adversarial inverse reinforcement learning. arXiv preprint arXiv: 1710. 1248, 2017.
Batra, S., Tjanaka, B., Fontaine, M. C., Petrenko, A., Niko- laidis, S., and Sukhatme, G. Proximal policy gradient arborescence for quality diversity reinforcement learning. arXiv preprint arXiv:2305. 03.3005.379555 /3795, 2023.
Gao, Y., Vedula, S. S., Reiley, C. E., Ahmidi, N., Varadara- jan, B., Lin, H. C., Tao, L., Zappella, L., Béjar, B., Yuh, D. D., et al. Jhu-isi gesture and skill assessment working
10

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
set (jigsaws): A surgical activity dataset for human mo- tion modeling. In MICCAI workshop: M2cai, volume 3, pp. 3, 2014.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised predic- tion. In International Conference on Machine Learning (ICML 2017), pp. 2778 2787. PMLR, 2017.
Glasmachers. T., Schaul, T., Sun, Y., Wierstra, D., and Schmidhuber, J. Exponential natural evolution strategies. In Pelikan, M. and Branke, J. (eds.), Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, pp. 393-400, Portland, Oregon, USA, 2010. ACM.
Peng, X. B., Kanazawa, A., Toyer, S., Abbeel, P., and Levine, S. Variational discriminator bottleneck: Improv- ing imitation learning, inverse rl, and gans by constrain- ing information flow, arXiv preprint arXiv: 1810.00811 2018.
Grillotti, L., Faldor, M., León, B. G., and Cully, A. Quality- diversity actor-critic: learning high-performing and di- verse behaviors via value and successor features crities. arXiv preprint arXiv:2403.09933 2024.
Pierrot, T., Macé, V., Cideron, G., Beguir, K., Cully, A., Sigaud, O., and Perrin, N. Diversity policy gradient for sample efficient quality-diversity optimization.(2021) 2021.
Hansen, N. The ema evolution strategy: A tutorial. arXiv preprint arXiv: arXiv: /604 2016.
Pugh, J. K., Soros, L. B., and Stanley, K. O. Quality di- versity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:202845, 2016.
Hansen, N. and Ostermeier, A. Completely derandomized self-adaptation in evolution strategies. Evolutionary com- putation, 9(2):199-199)) 195, 2001.
Ross, S., Gordon, G., and Bagnell, D. A reduction of imita- tion learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635 JMLR Workshop and Conference Proceedings. 2011.
Ho, J. and Ermon, S. Generative adversarial imitation learn- ing. In Advances in neural information processing sys- tems (NeurIPS 2016), volume 29, 2016.
Ho, J., Jain, A., and Abbecl, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems (NeurIPS 2020), pp. 1-25, 2020.
Ross, S., Melik-Barkhudarge N., Shankar, K. S., Wendel, A., Dey, D., Bagnell, J. A., and Hebert, M. Learning monocular reactive uav control in cluttered natural envi- ronments. In IEEE international conference robotics and automation (ICRA 2013), pp. 1765-1772. IEEE, 2013. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXi:: /707.06347777 2017.
Huang, S., Dossa, R., and Cleanrl, C. Y. High-quality single-file file implementation of deep reinforcement learning algorithms. GitHub repository. 2020.
Jiang Y., Liu, E., Eysenbach. B., Kolter, J. Z., and Finn, C. Learning options via compression. Advances in Neural Information Processing Systems, 35:21 184-21999,, 2022.
Kipf, T.. Li, Y., Dai, H.. Zambaldi, V., Sanchez-Gonzaler A., Grefenstette, E., Kohli, P., and Battaglia, P. Com- pile: Compositional imitation learning and execution. In International Conference on Machine Learning, pp- 3418-3428. PMLR, 2019.
Seyed Ghasemipour, S. K., Gu, S. S., and Zemel, R. Smile: Scalable meta inverse reinforcement learning through context-conditional policies. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Sohn, K., Lee, H., and Yan, X. Learning structured output representation using deep conditional generative models. In Advances in Neural Information Processing Systems (NeurIPS 2015), 2015.
Lim, B., Allard, M., Grillotti, L., and Cully, A. Acceler- ated quality-diversie for robotics through massive paral- lelism. In ICLR Workshop on Agent Learning in Open- Endedness, 2022.
Tjanaka, B., Fontaine, M. C., Togelius, J., and Nikolaidis, S. Approximating gradients for differentiable quality diversity in reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2022), pp. 1102-1111, 2022
Mouret, J.-B. and Clune, J. Illuminating search spaces by mapping elites. arXiv preprint (54.:::500004999 201 5a.
Mouret, J.-B. and Clune, J. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.044 2015b.
Nilsson, O. and Cully, A. Policy gradient assisted map- elites. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2021). pp. 6668875, 2021.
Tjanaka, B., Fontaine, M. C., Lee, D. H., Zhang, Y., Balam, N. R., Dennler, N., Garlanka, S. S., Klapsis, N. D., and Nikolaidis, S. pyribs: A bare-bones python library for quality diversity optimization. In Proceedings of

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
the Genetic and Evolutionary Computation Conference (GECCO 2023), pp- 220-229, 2023.
Proceedings of the AAAI Conference Artificial Intelli- gence (AAAI 2008), volume 8, pp. 1433-1438 Chicago, IL, USA, 2008.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics en- gine for model-based control. In IEEE/RSJ international conference on intelligent robots and systems (IROS 2012), pp. 5026-5033. IEEE, 2012.
Vassiliades, V., Chatzilygeroudis, K., and Mouret, J.-B. Us- ing centroidal voronoi tessellations to scale up the multi- dimensional archive of phenotypic elites algorithm. IEEE Trunsactions Evolationary Computation, 22(4):623-
630, 2017.
Wang, B., Wu, G., Pang, T., Zhang, Y., and Yin, Diffail: Diffusion adversarial imitation learning. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 38, pp. 15447-15455, 2024.
Wulfmeier, M., Ondruska, P., and Posner, I. Maximum en- tropy deep inverse reinforcement learning. arXiv preprint arXiv: 150., 04888, 2015.
Yu, I.,, Yu, T., Finn, C.,, and Ermon, S. Meta-inverse rein- forcement learning with probabilistic context variables. Advances in neural information processing systems, 32, 2019.
Yu, X., Lyu, Y., and Tsang. I. Intrinsic reward driven imita- tion learning via generative model. International Con- ference on Machine Learning (ICML 2020)), pp. 10925- 10935. PMLR, 2020.
Yu, X., Wan, Z. Bossens, D. M., Lyu, Y., Guo, Q., and Tsang, I. W. Imitation from diverse behaviors: Wasser- stein quality diversity imitation learning with single-step archive exploration. In International Conference on Au- tonomous Agents and Multiagent Systems (AAMAS 2025), 2025.
Zare, M., Kebria, M., Khosravi, A., and Nahavandi, S. A survey of imitation learning: Algorithms, recent develop- ments, and challenges. IEEE Transactions on Cybernet- ics, 2024.
Zhang, J., Pertsch, K., Yang. J., and Lim, J. J. Minimum description length skills for accelerated reinforcement learning. In Self-Supervision for Reinforcement Learning Workshop-ICLR, volume 2021, 2021.
Zhu, Y., Wang, Z., Merel, J., Rusu, A., Erez, T., Cabi, S., Tunyasuvunakool, S., Kramár, J., Hadsell, R., Freitas, N., et al. Reinforcement and imitation learning for di- verse visuomotor skills. arXiv preprint arXiv:1802.09564 2018.
Ziebart, B. D., Maas, A. L., Bagnell, J. A., Dey, A. K., et al. Maximum entropy inverse reinforcement learning.
12

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
A. Background
A.1. Traditional QD Algorithms
Some traditional Quality Diversity optimization methods integrate Evolution Strategies (ES) with MAP-Elites (Mouret & Clune, 2015b), such as Covariance Matrix Adaptation MAP-Elites (CMA-ME) (Fontaine et al., 2020). CMA-ME uses CMA-ES (Hansen & Ostermeier, 2001) as ES algorithm generating new solutions that are inserted into the archive, and uses MAP-Elites retains the highest-performing solution in each cell. CMA-ES adapts its sampling distribution based on archive improvements from offspring solutions. However, traditional ES faces low sample efficiency, especially for high-dimensional parameters such as neural networks.
Differentiable Quality Diversity (DQD) improves exploration and fitness by leveraging the gradients of both objective and measure functions. Covariance Matrix Adaptation MAP-Elites via Gradient Arborescence (CMA-MEGA) (Fontaine & Nikolaidis, 2021a) optimizes both objective function f and measure functions m using gradients with respect to policy dm, parameters: ff af and Vm de
(0) + c)))),)) where dmp, The objective of CMA-MEGA is g (0) |col
de
de
the coefficients çj are sampled from a search distribution. CMA-MEGA maintains a search policy Tdz in policy parameter space, corresponding to a specific cell in the archive. CMA-MEGA generates local gradients by combining gradient vectors with coefficient samples from CMA-ES, creating branched policies ,..,..., Te,- These branched policies are ranked based on their archive improvement, which measures how much they improve the QD-Score (one QD metric, which will be discussed in the experiment section) of the archive. The ranking guides CMA-ES to update the search distribution, and yields a weighted linear recombination of gradients to step the search policy in the direction of greatest archive improvement. Covariance Matrix Adaptation MAP-Anneaing via Gradient Arborescence (CMA-MAEGA) (Fontaine & Nikolaidis, 2023), introduces soft archives, which maintain a dynamic threshold le for each cell. This threshold is updated by f + (1 - a) + (TTTT) when new policies exceed the cell's threshold, where a balances the time spent on exploring one region before exploring another region. This adaptive mechanism allows more flexible optimization by balancing exploration and exploitation.
A.2. Details about PPGA and Related Background
To help readers to better understand the background of QD-RI,, we begin with Covariance Matrix Adaptation MAP-Eliter via a Gradient Arborescence (CMA-MEGA) (Fontaine & Nikolaidis, 2021a).
For general OD-omiiizzation problem, the objective of CMA-MEGA is:
8(0) = (col + Σ c)m;((),
(14)
In this context, (0) represents the j-th measure of the solution 0, and k is the dimension of the measure space. The objective function of CMA-MEGA is dynamic because the coefficient for each measure, Cj. is updated adaptively to encourage diversity in m. For instance, if the algorithm has already found many solutions with high mi values, it may favor new solutions with low my values by making Ci negative, thus minimizing mi. However, the coefficient for the fitness function f will always be positive, as the algorithm always seeks to maximize fitness. This objective function ensures that CMA-MEGA simultaneously maximizes fitness f and encourages diversity across the measures m. We update 0 by differentiating objective (14) and use gradient descent based optimization approaches, since DQD assumes f and are differentiable.
Furthermore, the coefficients cy are sampled from a distribution, which is maintained using Covariance Matrix Adaptation Evolution Strategy (CMA-ES) (Hansen, 2016). Specifically, CMA-ES updates the coefficient distribution by iteratively adapting the mean μl and covariance matrix of the multivariate Gaussian distribution N(H, from which the coefficients Cj are sampled. At each iteration, CMA-MEGA ranks the solutions based on their archive improvement (i.e. How much they improve the existing solutions of occupied cell). The top-performing solutions are used to update μu, while Σ is adjusted to capture the direction and magnitude of successful steps in the solution space, thereby refining the search distribution over time.
In CMA-MAEGA (Fontaine & Nikolaidis, 2023), the concept of soft archives is introduced to improve upon CMA-MEGA. Instead of maintaining the best policy in each cell, the archive employs a dynamic threshold, denoted as f.. This threshold is updated using the following rule whenever a new policy ne surpasses the current threshold of its corresponding cell e:
13

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
+- (1- (T())))
Here, a is a hyperparameter called the archive learning rate, with 0 S a S 1. The value of a controls how much time is spent optimizing within a specific region of the archive hefore moving to explore a new region. Lower values of a result in slower threshold updates, emphasizing exploration in particular region, while higher values promote quicker transitions to different areas. The concept of soft archives offers several theoretical and practical advantages, as highlighted previous studies.
PPGA (Batra et al., 2023) is directly built upon CMA-MAEGA. We summarize the key synergies between PPGA and CMA-MAEGA follows:(1) In reinforcement learning (RL), the objective functions f and in Equation 14 are not directly differentiable. To address this, PPGA employs Markovian Measure Proxies (MMP), where single-step proxy (s,) is treated as the reward function of an MDP. PPGA utilizes k + parallel PPO instances to approximate the gradients of f and each measure m, where is the number of measures. Specifically, the gradient for each i-MDP is computed as the difference between the parameters nLee after multi-step PPO optimization and the previous parameters Ogold (2) Once the gradients are approximated, the problem transformed into standard DQD problem. PPGA then applies modified version of CMA-MAEGA perform quality diversity optimization. The key modifications include:
1. Replacing CMA-ES with xNES for Stability: To improve stability in noisy reinforcement learning environments, CMA-ES was replaced with Exponential Natural Evolution Strategy (xNES). While CMA-ES struggled with noisy, high-dimensional tasks due to its cumulative step-size adaptation mechanism, xNES provided more stable updates to the search distribution, especially in low-dimensional objective-measum spaces, and maintained search diversity.
2. Walking the Search Policy with VPPO: PPGA "walks" the search policy over multiple steps by optimizing new multi-objective reward function with VPPO (Vectorized Proximal Policy Optimization). This done by leveraging the mean gradient coefficient vector from xNES, ensuring stable and controlled movement toward greater archive improvement.
B. Algorithm Pseudo Code
Algorithm 2 shows the principle of updating the policy archive using MAP-Elites Annealing. Algorithm 3 explaining how our reward model calculates rewards and how the reward model being updated.
Algorithm 2 Update Archive
Input: Solution 0 to insert, fitness f, measures =< m)..... m, >, archive se,, archive learning rate a
Oine fine + of [m] if af [m] is nonempty else None, 0
A =0
<>> Sine then
insert into cell of [m]
fine (1 a)) fine + a
A fine
end if
return A
14

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
Algorithm 3 Reward Model E (using GAIL the backbone for example)
1: Initialize: Discriminator Dy
2:
3: Method: Reward Calculation for VPPO.compute jacobbinn)
4: def get_episode_ rewareeeeearrrssleee episode, current archive a/):
5: ".. t episode
6: 71. + Dv
GAIL batch reward
7: = sertode.ge_mmaaurre
8:
TEBC gm ml)
calculate curiosity-driven reward
9:
For 1=
10:
< + TEBC return 71.72,73.....
> calculate total reward
11:
12:
13:
Method: Update reward model
def eselt 2, Tg):
Sample batch of trajectories (s* a²) from 7
16:
Update discriminator Dy by minimizing:
LD(V) (aa)) (ogg)--s ((sa)sss)) (s.)))> (- (s, a))]
Repeat until the model converges the number of epochs is reached return Updated Dy
18:
C. Hardware Setup
Our experiments are based on the PPGA implementation using the Brax simulator (Freeman et al., 2021), enhanced with QDax wrappers for measure calculation (Lim et al., 2022). We leverage pyribs (Tjanaka et al., 2023) and CleanRL's PPO (Huang et al., 2020) for implementing the PPGA algorithm. The observation space sizes for these environments are 17, 18, and 227, with corresponding action space sizes of 6, 6, and 17. All Experiments are conducted on a system with four A40 48G GPUs, an AMD EPYC 7543P 32-core CPU, and Linux OS. Each single experiment only requires one A40 48G GPU and takes roughly days.
D. Demonstration Details
Figure 6 shows the MuJoCo environments used in our experiments. Table 2 shows the detailed information of the demonstrations in our experiment. Figure visualizes the selected policies in the behavior space.
(a) Halfcheetah
(b) Walker2d
Humanoid
Figure 6: MuJoCo Environments.
15

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
Table 2: Demonstrations are generated from top-500 high-performance elites.
Tasks
Demo number
Attributes
min
max
mean
std
Length
1000
1000
1000.0
0.0
Halfcheetah
Demonstration Return
3766.0
8405.4
5721.3
1927.6
Length 356.0
1000.0
625.8
254.4
Walker2d
4
Demonstration Return
1147.9
3721.8
2372.3
1123.7
Length
1000.0
1000.0
1000.0
0.0
Humanoid
4
Demonstration Return
7806.2
9722.6
8829.5
698.1
1.0
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.0
(a) Halfchectah
(b) Walker2d
(c) Humanoid
Figure 7: Visualization of the behavior space. Green indicates the full expert behavior space, blue indicates the selected top-500 elites, and red indicates the demonstrators. The x axis is the proportion of time Leg 1 touches the ground and the axis the proportion of time Leg 2 touches the ground.
E. Hyperparameter Setting
E.1. Hyperparameters for PPGA
Table 3 summarizes list of hyperparameters for PPGA policy updates.
Table 3: List of relevant hyperparameters for PPGA shared across all environments.
Hyperparameter
Value
Actor Network
[128, 128, Action Dim]
Critic Network
[256, 256, I]
N1 N2
10
10
PPO Num Minibatches
8
PPO Num Epochs
4
Observation Normalization
True
Reward Normalization
True
Rollout Length
128
Grid Size
50
Env Batch Size
3,000
Num iterations
2,000
E.2. Hyperparameters for IL
Table 4 summarizes list of hyperparameters for AIRL,, GAIL.
16

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
Table 4: List of relevant hyperparameters for AIRL, GAILs shared across all environments.
Hyperparameter
Value
Discriminator
[100, 100, I]
Learning Rate
3e-4
Discriminator Num Epochs
1
Table 5 summarizes list of hyperparameters for VAIL.
Table 5: List relevant hyperparameters for VAILs shared across all environments.
Hyperparameter
Value
Discriminator
[100, 100, (1, 50, 50)]
Learning Rate
3e-4
Information Constraint /
0.5
Diseriminator Num Epoch
Table 6 summarizes list of hyperparameters for GIRIL.
Table 6: List of relevant hyperparameters for GIRIL shared across all environments.
Hyperparameter
Value
Encoder
[100, 100, Action Dim]
Decoder
[100, 100, Observation Dim]
Learning Rate
3e-4
Batch Size
32
Num Pretrain Epochs
10,000
Table 7 shows the hyperparameter setting for DiffALL and DBB.
Table 7: List relevant hyperparameters for DiffAIL and DifTAIL-EBC.
Hyperparameter
Value
Variance schedule (B,)
linear from le 22-2
QD-Score
Coverage(%)
BestReward
AverageReward
7030
2000
Humanola
5030
60
4039
43
1000
2000
1000
500
Ieraatonss
htrrtionss
Iterations
q=1
q
q=
Figure 8: Study of Hyperparameter q
17

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
E.3. Hyperparameter Study of q
In our work, there is only one hyperparameter 4 which controls the weight of reward bonus, We select one environment, namely Humanoid, and compare the QD performance between q value 1,2 and 5. Figure 8 shows that q is the best in this study.
However, the best choice of varies according to the property of different environments, and it is not possible to find the exact optimal value for each potential environment since value is continuous. Hence, for simplicity, we select in our experiments.
F. Results on Additional Evaluation Metric
The complementary cumulative distribution function (CCDF) (Vassiliades al., 2017) is another important metric in QD family that plots, for every reward threshold R on the horizontal axis, the fraction of policies in the archive whose returns meet or exceed that value. By doing so, the CCDF conveys both the overall quality and the behavioral diversity of the archived policies, while also illustrating how performance is distributed across the archive (Batra et al., 2023).
Halfcheetah
Humanoid
Walker2d
CCDK
0.2
0.2
0.2
0.0
0.0
0.0
10000
Fitness
Fitness
Fitness
GAIL
GAIL-EBC
VAIL
VAIL-EBC
Figure 9: Plot of CCDF using GAIL, VAIL and their EBC variants across three environments.
As shown in Figure 9, we validate the effectiveness of EBC imitation learning using GAIL and VAIL as baseline IRL models, and report CCDF plot across three environments: Halfcheetah, Humanoid and Walker2d. Except for Halfcheetah, EBC consistently improves GAIL or VAIL with respect to the quality of learned policies, and we attribute this the EBC mechanism's capability unlock more potentially nigh-performing policies.
G. QD Metrics Performance Results
Table 8 summarizes the performance of the different algorithms on the QD metrics.
Table Quantitative performance comparison of QD-IRL methods (GAIL, VAIL, DiffAIL), their EBC-enhanced versions, PPGA with true reward, and three QD-IRL baselines. The mean performance across multiple random seeds is reported, with standard deviation indicated by-
Hakhema
WalleM
Homawii
QC- Spase
Cox(Q)
Beal
Arg
QD-Sexe
Box
Arg
QD-Sceee
Boi
Arg
TmuRtwaad
6.1%
41A31111 1.1
47311784
2364133
1/6
10"
37.011108
5,151
1882.12
7.76v
15.151
9706111
4,2031
EBC (Dup)
1.998125
VAILTRC
2:060 10*
1501-500
1.56633
$1.54162
3
1356-72
10
10
97 07.63-0.9
58344-009
2.915-145
VAIL
Rf 24.00 107
91,0525.0
51235401
1352243
2
10
71,4152.00
1
1.312273
65.6122117
1.865:290
3,094:3044
284% 1P
199
18.1111.9
17701440
1004225
3442x
20
$3,0011155
40321.50
1
5,31
M5.0011135
5,402
2,005-241
GAIL
2.0 (0
=8 36 10'
51154218
1,6544311
2.47
2.88
69.2744511
4031
1.45--73
16
10"
24 51
82 06.35449.0 30
6276--2245
9064-251
05
3/1611/118
60.32.A
6,921191
Mas IRL.
3.506710
553=153
5.17
68:810035
3,35563155
1.046:299
83.8466.00
4,55861.415
8025-192
18

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
H. Tukey HSD Test Results
Table 9 compares our EBC-enhanced algorithms to their counterparts, to estimate the statistical significance of the results in Table The test results show that 7/9 effects are positive and 5 of these are significant with p - 0.05.
Table Tukey HSD test of the QD Score for the QD-IRL comparisons (w/o vs w/ EBC) in Table + indicates a positive effect of EBC. The significant pairwise comparisons (p<=0.05) are boldfaced.
groupl
group2
HalfCheetah
Walker2d
Humanoid
DiffAIL
DifFAIL-EBC
1.0
0.0236
0.0
VAIL
VAIL-EBC
0.9994
0.1894
+ 0.0061
GAIL
GAIL-EBC
+ 1.0
+
0.0231
+ 0.0004
I. Effects of Number and Quality of Demonstrations
Table 10 shows the results of study number and quality of demonstrations.
Table 10: Effects of demonstration number, diversity and quality GAIL and GAIL-EBC in Humanoid. Values
mean one standard deviation over three seeds.
Demos
Models
QD-Score
Coverage (%)
10
GAIL
2,576 765 + 82 806
68.68 + 3.20
GAIL-EBC
5 582 + 254 060
97.00 = 0.16
4
GAIL
186 +:511 004
88.34 + 4.30
GAIL-EBC
1 150716
97.84 0.04
2
GAIL
2676 + 663
70.44+0...
GAIL-EBC
5803 908 + 120 888
97.30 0.30
1
GAIL
1718 577 + 834
75.24 = 1.12
GAIL-EBC
4948921 + 203 595
98.70 + 1.02
Non-diverse (top-4) demos
GAIL
2832 01:0078--1111 319 164
75.30
GAIL-EBC
4074729 + 306 686
98.12 0.40
J. Few-Shot Adaptation
For all adaptation tasks, the reward stays the same but the dynamics of the MDP is changed. The goal is to leverage the diversity of skills to adapt to unforeseen situations. Table 11 summarizes the details of the three adaptation tasks.
For all fho- adaptation tasks, we evaluate all skills for each replication each method and select the best one to solve the adaptation task.
On Humanoid- Hurdles, we use the jump features jump over hurdles varying in height from 0 to 50 cm.
On Humamoid Motor Failure, we use the feet contact features to find the best way to continue walking forward despite the damage. In this experiment, we scale the action corresponding to the torque of the left knee (actuator 10) by the damage strength ranging from 0.0 (no damage) to 1.0 (maximal damage).
On Walker2d Friction, we use the feet contact features to find the best way to continue walking forward despite the change in friction. In this experiment, we scale the friction by coefficient ranging from 0.0 (low friction) to 5.0 (high friction).
19

---

Diversifying Policy Behaviors via Extrinsic Behavior Curiosity
Table 11: Adaptation tasks
Perturbations Range
Hurdles height E [0,
Motor Failure
Friction
damage strength [0.0, 1.0]
coefficient E [0.0,5.0]
MuJoCo Task
Humanoid
Humanoid
Walker2d
Measures
Jump
Feet Contact
Feet Contact
K. Limitations
In this section, we discuss two potential limitations our work that we have identified.
The imitation reward functions learned by GAIL and VAIL are dynamically updated, and therefore it is still challenging to update the archives based on the imitation rewards since this error will accumulate over time. While we believe that the mechanism from MAP-Elites Annealing (Algorithm 2) alleviates the issue, we still observe significant performance declines in preliminary experiments and suggest future work is needed to ensure high-quality archives can be evolved.
Due to the MDP property of QD-RI and QD-IRI. and the mechanism of policy gradient, we can guide the search policy of PPGA towards the empty region of policy archive via the simple addition of curiosity-driver reward bonus. This is because the policy gradient methods will generate one gradient term which serves as improving the probability for the new policy to occupy empty cells. However, if the problem is not MDP-based or the policy optimization not based on policy gradient, then our work may be not directly applicable.
L. Discussion and Future Work
We first note the wide applicability of EBC. The measure m defining the behavior diversity of EBC is flexible, since it can be either pre-defined (as e.g. in the feet contact and angle measures in the paper) or based on a learning representation. As shown in our adaptation experiments, searching in this measure space can help to recover reasonable performance levels when the surface changes, the agent's limbs are damaged, and when new obstacles emerge. On the other hand, the measure could also be dynamically learned. For example, QDHF (Ding et al., 2024) adopts contrastive learning to learn the measure that best fits real-world human preferences. This fits particularly well in the extrinsie view curiosity, where human agent can serve critic of the behavioral diversity. In both cases, EBC could be applied improve the efficiency and performance of QD-RL and QD-IRL algorithms, providing more promising application landscape. Moreover, our framework can potentially be extended to other robot locomotion tasks besides MuJoCo, which forms our future research plan.
For the algorithm design aspect, it is true that there are existing methods which intend to address the lack-of lack-of-explae exploration issue in QD context, such as CMA-ME (Fontaine al., 2020). However, EBC allows a policy gradient algorithm to update the policy towards empty behavioral areas directly via incorporating the exploration incentive into gradients, which is more effective compared with the exploration of CMA-ME which doesn't utilize the gradient information. Moreover, adopt naive yet straightforward strategy to propagate the spisode-based reward bonus: simply add it uniformly to the reward each timestep. thereby crediting the action of every step in the episode. Designing more principled propagation method is a promising direction for further improving the performance of EBC. Furthermore, imitation tasks grounded in real-word datasets (e.g., human motion capture data) represent a compelling application domain for our method, with this direction emerging as priority for future research.
20


Reviews:
[{"review_id": "iuZjevBGxY", "reviewer": "Reviewer_Cxuk", "summary": "Existing imitation learning algorithms fail to learn diverse behaviors. To address this, the paper introduces the QD-IRL framework that applies QD optimization algorithms to IRL problems. To further improve the exploration of QD-IRL, the paper introduces Extrinsic Behavioral Curiosity (EBC) to encourage policies to explore areas that are not covered. Experimental results show that EBC achieves better performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thanks for the valuable insights and suggestions. \n- - -\n**Q1.** It would be better to discuss the relationship between Yu et al.'s paper.\n\n**A1.** The key difference between Yu’s work and our work lies in two aspects:\n- Yu’s work adopts a ​**single-step** reward bonus (calculated per (s,a) pair), while our method uses ​**episode-based reward** bonus aligned with whole-episode measurements.\n- Yu’s work focuses on QD-IL, whereas our framework extends to QD-RL.\n\nWe will add these discussions in our revised paper.\n\n- - -\n**Q2.** Why use EBC given CMA-ES already handles quality/diversity?  \n\n**A2.** The EBC reward can supplement the PPGA algorithm: while PPGA works by evolving random coefficients for each measure dimension, the EBC reward encourages the policy to unlock new areas of measure space by adding it to the fitness, which is independent of the local measure space, for a more direct, global, and efficient search of new behavioral areas. \n\n- - -\n**Q3.** More experiment needed on tasks with different measure dimensions and type of measures.\n\n**A3.** We tested EBC across tasks with varying measure dimensions (**m**):  \n- Jump (1D: lowest foot height)  \n- Angle (2D: body angle)  \n- Feet-contact (4D: leg-ground time)  \n\nGAIL-EBC consistently outperforms GAIL in all scenarios:\n\n| Game      | Measure          | Model     | QD-Score              | Coverage           |\n|-----------|------------------|-----------|-----------------------|--------------------|\n| hopper    | jump            | GAIL      | 64017±5558            | 100.0±0.0          |\n| hopper    | jump             | GAIL-EBC  | 81987±6890            | 100.0±0.0          |\n| humanoid  | angle  | GAIL      | 63771±7113            | 53.0±3.0           |\n| humanoid  | angle   | GAIL-EBC  | 170273±19711          | 96.0±0.0           |\n| humanoid  | jump           | GAIL      | 18812±6412            | 41.0±31.0          |\n| humanoid  | jump             | GAIL-EBC  | 90631±61894           | 85.0±15.0          |\n| ant       | feet_contact     | GAIL      | -484468±3264          | 75.84±0.0          |\n| ant       | feet_contact     | GAIL-EBC  | -52521±383213         | 77.28±3.2          |\n\n- - -\n**Q4.** How do demonstration(demo) quantity/diversity/quality impact performance?  \n\n**A4.** Experiments with varying demo counts in Humanoid environment:  \n\n| Demos | Model     | QD-Score         | Coverage      |\n|-------|-----------|------------------|---------------|\n| 10    | GAIL      | 2576765±82806    | 68.68±3.2     |\n|       | GAIL-EBC  | 5822582±254060   | 97.0±0.16     |\n| 4     | GAIL      | 1886725±551004   | 88.34±4.3     |\n|       | GAIL-EBC  | 5704650±150716   | 97.84±0.04    |\n| 2     | GAIL      | 2676218±360663   | 70.44±0.2     |\n|       | GAIL-EBC  | 5803908±1320888  | 97.3±0.3      |\n| 1     | GAIL      | 1718577±134816   | 75.24±1.12    |\n|       | GAIL-EBC  | 4948921±203595   | 98.7±1.02     |\n\n**Key observations:**  \n- With 1 demo, GAIL-EBC’s QD-Score drops by 15% vs. 10-demos.  \n- Using non-diverse elites (top4 best-reward):  \n\n| Algorithm     | QD-Score         | Coverage      |\n|----------------|--------|---------------|\n|     GAIL      |  2832078±319164    | 75.3±7.9   |\n|   GAIL-EBC  |  4074729±306686 (-30% vs 4-demos) | 98.12±0.4  |\n\n- - -\n**Q5**. It would be helpful to add the pure IRL algorithms (without PPGA) with and without EBC bonus as baselines.\n\n**A5**. Without the QD algorithm, there would be no QD-Score, coverage, and average reward metrics. So we would only be able to compare the maximal fitness, which is not the our objective.\n\n- - -\n**Q6**. It would be helpful to use statistical tests to show the significance of the results.\n\n**A6**. We perform a **Tukey HSD Test** on Table 7; comparing our EBC-enhanced algorithm to their counterparts, 7/9 effects are positive and 5 of these are significant with $p \\leq 0.05$.\n\n- - -\n**Q7**. The code is not available.\n\n**A7**. We will open-source the code and the seeds when the paper is published. \n\n- - -\n**Q8**. PPGA also optimizes behavior diversity and aims to maximize the QD-Score. Why does PPGA with EBC achieve a better QD-Score than PPGA without EBC?\n\n**A8**. We kindly refer you to A2. PPGA itself is not sufficient in exploration. However, our EBC bonus synergized with CMA-ES could facilitate exploration and exploitation more effectively (see our paper line 244).\n\n- - -\n**Q9.** Why does DiffAIL-EBC outperform PPGA with true reward in Humanoid?  \n\n**A9.** As shown in Fig.4, PPGA itself's performance improves significantly with EBC (improved exploration). However, DiffAIL-EBC does ​not surpass PPGA-EBC, aligning with expectations. See analysis in  l.420-428 (Page 8).  \n\n- - -\n**Q10.** Is comparing average rewards necessary?  \n\n**A10.** While QD-score is the primary metric, average reward provides insights into archive quality. We retain it for supplementary analysis but emphasize QD metrics as the core evaluation.\n\n- - -\n## We hope these answers address your concerns.", "claims_and_evidence": "The claim is overall clear. However, similar frameworks seem to have been proposed in existing papers, e.g., Yu et al. (2024) cited in the paper. It would be better to have a discussion about the relationship.\n\nThe claim of the limitation of PPGA is not convincing to me. The paper claims that *the fitness term $f$ heavily influences PPGA’s search policy update direction* and *PPGA frequently becomes stuck in local regions*. However, as is shown in Eq. (2) in the paper, PPGA optimizes a weighted sum of fitness term $f$ and behavior measure terms $m_j$. The weights $c$ are optimized by CMA-ES, which tries to find the best weights to maximize the QD-Score improvement. Therefore, as the QD-Score considers both quality and diversity, PPGA is expected to adapt the weights $c$ dynamically to encourage the policies to explore the areas with low quality or no policy to improve QD-Score, rather than heavily optimizing the quality. Thus, PPGA does not seem to have these issues. It is not fully clear to me why EBC performs well.", "methods_and_evaluation_criteria": "The proposed method overall makes sense for the problem.", "theoretical_claims": "The paper provides a slightly trivial lemma showing that optimizing the EBC reward helps cover more areas in the behavior space.", "experimental_designs_or_analyses": "- The experiments are not sufficient. The methods are only evaluated on three MuJoCo tasks with the same measure function (i.e., the proportion of time each leg touches the ground) on the same dimension (2). What are the performances on the tasks with higher or lower dimensions of measures (e.g., Ant and Hopper)? What are the performances on the tasks with other types of measure functions?\n- The impact of the demonstrations is not analyzed. It may be critical to the performance of the method. What is the impact of the number of the demonstrations? Do they need to be diverse? What is the impact of the quality of the demonstrations?\n- The advantages of the QD algorithms over the classical IRL algorithms are not analyzed. It would be helpful to add the pure IRL algorithms (without PPGA) with and without EBC bonus as baselines as well.\n- I am glad that the QD metric values and their standard deviations are presented in Table 7 in the appendix. However, as each experiment was only conducted with 3 seeds, the standard deviations (and the averaged performance) in both Figure 2 in the body and Table 7 in the appendix may not be valid. It would be helpful to run more (>4) seeds and utilize the statistical tests to show the significance of the results.\n- The seeds are not shown, and the code is not available currently, which may be harmful to the reproductivity of the results.", "relation_to_broader_scientific_literature": "According to the results, the proposed EBC method may also improve the performance (coverage) of classical QD-RL algorithms.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "The cross (x) markers in the figures are mixed up. It would be better to improve their presentation.", "questions_for_authors": "- PPGA also optimizes behavior diversity and aims to maximize the QD-Score. Why does PPGA with EBC achieve a better QD-Score than PPGA without EBC?\n- What is the relationship with Yu et al. (2024) cited in the paper? It would be better to have a discussion.\n- What are the performances on the tasks with other dimensions and other types of measure functions?\n- What is the impact of the demonstrations?\n- In Humanoid, DiffAIL-EBC even performs better than PPGA with true reward. Can you provide an analysis of it?\n- Is it necessary to compare the average rewards of the methods? According to the definition, when a policy with a new behavior but a relatively low reward is added to the QD archive (which is what we intend to do), the average reward will fall instead. This conflicts with the goal of QD.\n- Will the code and the seeds be open-sourced when the paper is published? This may be important for the reproducibility and validity of the experimental results.\n\nI would be happy to raise my score if the concerns and questions are answered.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aiSCXEXYqP", "reviewer": "Reviewer_Y5D4", "summary": "This paper proposes to combine Quality Diversity (QD) algorithms with Inverse Reinforcement Learning (IRL) problems. The authors introduce Quality Diversity Inverse Reinforcement Learning (QD-IRL), a method that uses rewards estimated from demonstrations (via GAIL, VAIL, and DiffAIL) with the PPGA quality diversity algorithm. The key contribution is Extrinsic Behavioral Curiosity (EBC), a reward mechanism that encourages exploration in unvisited areas of the measure space. Their experiments on three MuJoCo locomotion tasks (Halfcheetah, Walker2d, and Humanoid) to show that EBC significantly improves the performance of QD-IRL methods. The authors also demonstrate that EBC can enhance standard PPGA when applied to the true reward function.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Thank you for the valuable suggestions. Here is our response to questions and concerns.\n\n- - -\n**Q1**. This paper lacks evidence support for the significance of QD  (e.g. the ability of adaptation).\n\n**A1**. The claim that QD approaches help with adaptation in changing environments is reasonable and supported by prior works. We do notice that on l.46, the given reference does not adequately reflect the adaptation scenario, so we provide a more suitable few references. While the claim is not central to our work, we perform few-shot adaptation tasks for Humanoid-hurdles, and find that GAIL-EBC significantly outperforms GAIL on the return, and is comparable to the original QDAC performance of Grillotti et al. :\n\n|   Algorithm  | height 0.026  |  height 0.053    |  height 0.079     |  height 0.105          |\n|-- |------|---|---|---|\nGAIL |864.37135$\\pm$331 | 329$\\pm$65 | 198$\\pm$26 | 107 $\\pm$ 65 | \nGAIL-EBC | 2742$\\pm$2252 | 396$\\pm$105 | 275$\\pm$23 | 130$\\pm$89 |\n\n- - -\n**Q2**. More diverse IL tasks are needed to demonstrate the generalizability of the method.\n\n**A2**. We have conducted additional experiments with diverse measures. Please refer to our response to Q3 of reviewer Cxuk for details.\n\n- - -\n**Q3**. While GAIL and VAIL rewards are updated at every iteration, the finesses stored in the archive are not regularly updated (as acknowledged in the limitations section, lines 907-910).\n\n**A3**. That's indeed the limitation of our work that we already acknowledged, and we are happy to address this issue in our future work. Moreover, Algorithm 2, which dynamically adjust the fitness scores, mitigates the impact of this problem by computing a running average of past elite fitnesses.\n\n- - -\n**Q4**. EBC reward is non-Markovian and depends on whole trajectory, which may affect the theoretical guarantee.\n\n**A4**. The combination of non-markovian reward and PPO are explored and backed in prior work. For example,  [1]  validated that episode-based reward structures, when properly decomposed into each step, enhance PPO’s convergence and sample efficiency. Hence, we safely assume the convergence guarantee of PPO in our Lemma 3.1. Empirically, we also validated the effectiveness of EBC reward.\n\n[1] Arjona-Medina, Jose A., et al. \"Rudder: Return decomposition for delayed rewards.\" Advances in Neural Information Processing Systems 32 (2019).\n\n- - -\n**Q5**. BestReward of EBC variants takes longer to converge (which is normal, as PPGA now optimizes for fitness+q×EBC).\n\n**A5**. In general, the higher the coverage, the longer it will take to evolve the elite solutions for all the covered cells. We will mention this in the text on page 7, l.364, in the text about Fig.2.\n\n- - -\n**Q6**. The computational requirements seem overly demanding (48 hours on 4 A40 GPUs per experiment).\n\n**A6**.  We apologize that the text is not clearly written. We use 1 GPU per experiment but run multiple experiments at the same time. We will rewrite this part.\n\n- - -\n**Q7**. This method should not be named as a framework. As presented, the EBC reward seems limited to the Gradient Arborescence subfamily of QD approaches (like CMA-MEGA and PPGA) and may not be applicable to other QD algorithms.\n\n**A7**. This is a limitation that we have acknowledged in Appendix G. Please note that we have implemented the EBC reward on different imitation learning methods, showing why it is usefully conceived as a framework. While MAP-Elites does not have an RL problem, PGA-ME and DCRL seem compatible for our framework. For instance, the  EBC reward would allow to take a step in the direction of the policy which generates new measures. Given the wide range of experiments, we leave this to future work. \n\n- - -\n**Q8**. The connection of CMA-ME and EBC regarding the exploration should be discussed.\n\n**A8**. We claim that EBC allows a **policy gradient** algorithm to update the policy towards empty behavioral areas **directly**, which is more effective compared with the exploration of CMA-ME which doesn't utilize the gradient information. For more details, please refer to Q2 from Reviewer Cxuk.\n\n- - -\n**Q9**. Figure 4 is redundant with Figure 2 - the additional variant could simply be included in Figure 2.\n\n**A9**. Figure 4 is for more straightforward comparison to show specifically that EBC can improve PPGA with true reward on conditions without EBC. We separated these so it is more easy to discuss in dedicated paragraphs in the text.\n\n- - -\n**Q10**. Policy archives in Figure 3 have inconsistent colorbars.\n\n**A10**. Please note that Figure 3 is plotted only for comparing the coverage metric, not the fitness (l.392). Therefore, the color (which represents fitness) is not our focus.\n\n- - -\n**Q11**. Consider reporting CCDF plots to better characterize the distribution of episode returns in the final policy archives.\n\n**A11**: Thanks for suggestion. We will report the CCDF plot in our revised paper.\n\n- - -\n## We sincerely hope these responses fully address your concerns.", "claims_and_evidence": "The primary claim about EBC improving QD-IRL methods is supported by the experimental results. The performance improvements across metrics like QD-score, coverage, and reward are clearly demonstrated in the figures and tables. \n\nHowever, the claim that QD approaches help with \"adaptation in changing environments\" (line 46) and \"unpredictable real-world scenarios\" (line 14) lack supporting evidence.\nThe authors do not provide experiments showing how the learned diversity could be practically useful, e.g. for scenarios like damage adaptation or hierarchical control, as done in prior works [1,2]. \nSuch experiments would be useful to know if the EBC mechanism can really make agents more resilient.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally appropriate for the problem. The QD metrics (QD-Score, Coverage, Best Reward, Average Reward) are standard and suitable for evaluating quality diversity algorithms. \n\nHowever, the evaluation would be stronger if it included more diverse environments beyond the three locomotion tasks that all share similar objectives (maximizing forward progress) and measures (foot contact patterns). Adding tasks with different properties, such as an Ant Omni task from [5], would better demonstrate the generalizability of the approach.\n\nAlso, there is problem with the method design. While GAIL and VAIL rewards are updated at every iteration, the fitnesses stored in the archive are not regularly updated (as acknowledged in the limitations section, lines 907-910). This means that solutions in the archive are evaluated based on potentially outdated reward functions. This design choice could lead to poor performance because the archive might contain solutions that were highly rated under an old reward function but would score poorly under the current one. For example, AURORA [8] regularly updates the descriptors of the solutions stored in the archive (as the measure functions slightly changes every iteration).", "theoretical_claims": "While the proof of Lemma 3.1 appears to be correct, I find one Lemma assumption intriguing: the MDP reward appears to be non-Markovian as it depends on the full trajectory (episode measure).", "experimental_designs_or_analyses": "The authors appropriately test their approach with three different IRL methods (GAIL, VAIL, and DiffAIL) with and without EBC and compare against true reward baselines. \nThe experiments effectively show that EBC improves the QD-Score and Coverage in the 3 environments under study.\n\nIt would also be worth mentioning that the BestReward of EBC variants takes longer to converge (which is normal, as PPGA now optimizes for $fitness+q\\times EBC$).\n\nThe computational requirements seem overly demanding (48 hours on 4 A40 GPUs per experiment).", "relation_to_broader_scientific_literature": "To the best of my knowledge, this paper is the first to apply quality diversity to an inverse reinforcement learning setting. The authors implement their approach using the recent PPGA algorithm and introduce the EBC reward to enhance exploration. However, as presented, the EBC reward seems limited to the Gradient Arborescence subfamily of QD approaches (like CMA-MEGA and PPGA) and may not be applicable to other QD algorithms such as MAP-Elites, PGA-ME, and DCRL [3].", "essential_references_not_discussed": "A notable omission is the connection to improvement emitter mechanisms from CMA-ME [6], which use a similar approach to encourage exploration in empty areas of the archive.", "other_strengths_and_weaknesses": "Strengths:\n\n- The proposed EBC reward is clear, sound, and well-motivated\n- The results effectively show improvements in coverage and QD-score\n\nWeaknesses:\n\n- The authors describe their work as a \"framework\" (line 97 and abstract), but it's only applied to a specific QD algorithm (PPGA) and relies heavily on operations specific to gradient arborescence approaches. Indeed, the introduced EBC is only used when “branching solutions” and when “walking the search policy”, which are two steps very specific to PPGA and other gradient arborescence approaches like CMA-MEGA. Given the strong reliance on PPGA and gradient arborescence methods, the authors might consider renaming the algorithm to MEGA-IRL or PPGA-IRL rather than presenting it as a general framework\n- The lack of experiments demonstrating the practical utility of the learned diversity (e.g. damage adaptation, hierarchical control...)\n- Figure 4 is redundant with Figure 2 - the additional variant could simply have been included in Figure 2\n- policy archives in Figure 3 have inconsistent colorbars, making the results difficult to interpret.", "other_comments_or_suggestions": "- Consider reporting CCDF plots [4] (also called \"archive profiles\", see [5, 2]) to better characterize the distribution of episode returns in the final policy archives.\n- Line 45 2nd column, the reference is likely meant to be [7]\n- The reference to differential quality diversity (DQD) appears twice in the bibliography\n- The size of the markers in Figure 2 makes it difficult to read\n- Lines 377-379 should clarify that the baselines are only detailed in the Appendix\n- Line 279: absolute values appear to be missing around $c_0$​", "questions_for_authors": "- Lemma 3.1: the MDP reward appears to be non-Markovian, as it depends on the full trajectory. Is that normal? How does this affect the theoretical guarantees of your approach?\n- Each experiment takes 48 hours to run on 4 A40 GPUs (line 746). Why does it take so long to run? How much faster would it be to run with other QD algorithms like CMA-MEGA?\n- While you justify the usage of Quality-Diversity algorithm by the fact that it \"allows adaptation in changing environments\" (line 46), you do not provide any experiments showing why the learned diversity is useful. Could you demonstrate how the repertoires you obtain can be used (e.g., for damage adaptation, hierarchical control) as done in previous works [1,2]?\n- You describe your approach as a \"framework\" (line 97 and abstract), but it's only applied to PPGA and relies on steps specific to gradient arborescence methods. How would EBC be implemented in other QD algorithms that don't use the same branching and walking mechanisms?\n\n[1] Chalumeau, Felix, et al. \"Neuroevolution is a competitive alternative to reinforcement learning for skill discovery.\"\n\n[2] Grillotti, Luca, et al. \"Quality-diversity actor-critic: learning high-performing and diverse behaviors via value and successor features critics.\"\n\n[3] Faldor, Maxence, et al. \"Synergizing quality-diversity with descriptor-conditioned reinforcement learning.\"\n\n[4] Batra, Sumeet, et al. \"Proximal policy gradient arborescence for quality diversity reinforcement learning.\"\n\n[5] Flageat, Manon, et al. \"Benchmarking quality-diversity algorithms on neuroevolution for reinforcement learning.\"\n\n[6] Fontaine, Matthew C., et al. \"Covariance matrix adaptation for the rapid illumination of behavior space.\"\n\n[7] Cully, Antoine, et al. \"Robots that can adapt like animals.\" \n\n[8] Grillotti, Luca, and Antoine Cully. \"Unsupervised behavior discovery with quality-diversity optimization.\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WFNqPUQnBh", "reviewer": "Reviewer_U4H2", "summary": "This work proposes a new paradigm called Quality-Diversity Inverse Reinforcement Learning (QD-IRL) as well as a new component to encourage the acquisition of novel and diverse behaviors, called Extrinsic Behavioral Curiosity (EBC). The goal of the QD-IRL framework is to enable the agent to learn diverse and performant policies from limited demonstration data and can be integrated on top of existing imitation learning (IL) approaches. Thus, the authors can combine the best of both approaches, overcoming the problems of vanilla IL, which struggles to learn diverse behavioral patterns. This framework achieves that by using IRL to learn the reward function, and then it utilizes Differentiable QD, specifically, QD-RL, to optimize the solutions archive. The framework is evaluated on locomotion tasks based on the performance gains it offers when integrated into existing IL methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We appreciate the valuable insights and suggestions, and below is our detailed response to address your concerns.\n\n- - -\n**Q1**: More diverse tasks other than locomotion will be beneficial.\n\n**A1**: While we limit the scope of the experiments to robot locomotion tasks, we do agree with the inclusion of much more diverse tasks to demonstrate the capability of our algorithm. In particular, we have extended our method to more diverse tasks and will add these contents in the revised paper. It is the same table also mentioned to reviewer Cxuk in Q3 (reproduced here for convenience):\n\n\n| Game      | Measure          | Model     | QD-Score              | Coverage           |\n|-----------|------------------|-----------|-----------------------|--------------------|\n| hopper    | jump            | GAIL      | 64017±5558            | 100.0±0.0          |\n| hopper    | jump             | GAIL-EBC  | 81987±6890            | 100.0±0.0          |\n| humanoid  | angle  | GAIL      | 63771±7113            | 53.0±3.0           |\n| humanoid  | angle   | GAIL-EBC  | 170273±19711          | 96.0±0.0           |\n| humanoid  | jump           | GAIL      | 18812±6412            | 41.0±31.0          |\n| humanoid  | jump             | GAIL-EBC  | 90631±61894           | 85.0±15.0          |\n| ant       | feet_contact     | GAIL      | -484468±3264          | 75.84±0.0          |\n| ant       | feet_contact     | GAIL-EBC  | -52521±383213         | 77.28±3.2          |\n\n\n- - -\n**Q2**: The authors never refer to how they construct the archive for the QD algorithm. \n\n**A2**: We use the archive construction rules based on MAP-Elites, which is mentioned in section 2.1.\n\n- - -\n**Q3**: It would be helpful to introduce the behavior descriptor for each experiment, and the details of each tasks.\n\n**A3** : The information about the behavior descriptor is referenced at various locations in the paper under the terminology “measure” (which is the terminology that was used in the PPGA paper). Specifically, the measure used is highlighted in Section 4 as follows:\n “the measure function maps the policy into a vector where each dimension indicates the proportion of time a leg touches the ground.” Moreover, we have clarified definition of state (policy’s inputs) and action (policy’s outputs) of each task in our revised paper.\n\n- - -\n**Q4**: The general improvements in performance EBC has to offer, in some cases are very minimal. \n\n**A4**:  Actually, when comparing performance, we should compare the base method with this method plus EBC (e.g., GAIL vs. GAIL-EBC) in terms of the QD-score. In the QD-score plots, the authors can only observe 2 comparisons among 9 comparisons (3 pairs $\\times$ 3 environments), namely the Halfcheetah for DiffAIL-EBC vs DiffAIL and VAIL-EBC vs VAIL, that are not significantly better.  We highlight that EBC improves the QD score of GAIL, VAIL, and DiffAIL by up to 185%, 42%, and 150%, and that the standard deviations are very small. We have performed a **Tukey HSD Test** on Table 7; comparing our EBC-enhanced algorithm to their counterparts, and we find that 7/9 effects are positive and 5 of these are significant with $p \\leq 0.05$. \n\n- - -\n**Q5**: It would be helpful to provide videos to illustrate the learned diverse solutions.\n\n**A5**: Thanks for the suggestion. One of the benefits of diversity is that the archives can be used for adaptation to new environment. In the updated submission, we will include a video of adaptation to new environments which can demonstrate the types of behaviors in the evolved archives as well as how this diversity can be exploited.\n\n- - -\n**Q6**: Is the EBC values bounded?\n\n**A6**: Since the unscaled EBC reward is either 0 or 1, so  the scaled EBC rewards are bounded in $[0,q]$, where $q$ is the hyperparameter that controls the weight. \n\n- - -\n**Q7**: Could the authors explain how they did the visualization for Fig. 3 and elaborate a bit more on what this figure represents?\n\n**A7**: Fig.3 represents the overall QD performance of the archive using a heatmap. The colors represent the fitness levels (quality), while the spread across the two-dimensional measure/behavior space represents the diversity.\n\n- - -\n## We sincerely hope these responses fully address your concerns.", "claims_and_evidence": "Most of the claims are supported by clear evidence. However, the proposed framework is quite generic, and thus, I believe the authors should have conducted experiments on more diverse tasks rather than focusing solely on locomotion. The selection of a more diverse set of tasks would have contributed to a more constructive evaluation.", "methods_and_evaluation_criteria": "Aside from the comments above, which also apply here, I believe the evaluation criteria are on point. In general, for QD algorithms the coverage and QD score present a clear overview of the performance of such algorithms.", "theoretical_claims": "No problems were found in the theoretical claims of the manuscript.", "experimental_designs_or_analyses": "I believe there is some vital information missing from the manuscript. The authors never refer to how they construct the archive for the QD algorithm. Additionally, there is not a single reference to what behavior descriptor they utilize for each experiment, so the reader can understand how solutions diversify. Lastly, even though the locomotion tasks are well-known, I think it is important for the authors to add more information about the tasks such as the policy's inputs.", "relation_to_broader_scientific_literature": "This work is novel enough, and it presents a very interesting extension of IRL to the QD optimization paradigm. Moreover, the fact that the proposed approach can be implemented on top of existing algorithms is very positive. Nonetheless, I am a little concerned about the results in Fig. 2. The general improvements in performance EBC has to offer, in some cases are very minimal. An example would be the Halfcheetah and Walker2d experiments, where the QD increase in most of the algorithms is not that significant.", "essential_references_not_discussed": "The authors have provided all the essential references required for the reader to comprehend the proposed approach.", "other_strengths_and_weaknesses": "One other strong point of the manuscript is that it is well written and most of the figures are very intuitive. On the other hand, it would have been very positive if the authors had provided some videos of the optimal solutions in the archive for the reader to understand how diverse the solutions are, since for such high-dimensional tasks, it is hard to illustrate the outcome of each solution in the archive.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "* Is the EBC values bounded?\n* Could the authors explain how they did the visualization for Fig. 3 and elaborate a bit more on what this figure represents?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tvdEwNUfuA", "reviewer": "Reviewer_tEMb", "summary": "This paper introduces Quality Diversity Inverse Reinforcement Learning (QD-IRL), a framework that integrates quality-diversity (QD) optimization with inverse reinforcement learning (IRL) to enable robots to learn diverse locomotion behaviors from limited demonstrations. The key innovation is Extrinsic Behavioral Curiosity (EBC), which rewards agents for discovering novel behaviors, encouraging a broader exploration of the behavior space. The proposed method is tested on multiple robot locomotion tasks and and improves the performance of some QD-IRL instances. The results show that EBC can surpass even expert performance in some cases.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## We feel gratitude for the valuable insights and suggestions, and below is our detailed response.\n\n- - -\n**Q1**: More complex imitation tasks can enhance the expressiveness of the method.\n\n**A1**: We have added experiments with additional tasks and measure. Please refer to our response to Q3 of reviewer Cxuk.\n\n- - -\n**Q2**: It would be helpful to add sensitivity studies on hyperparameters.\n\n**A2**: We have conducted a hyperparameter study for the $q$ hyperparameter, the scale of the EBC reward; these results are mentioned in Appendix E.4. Other hyperparameters follow the settings in original implementations since these are not newly introduced by our method.\n\n- - -\n**Q3**: Source code not provided.\n\n**A3**: We will open-source the code and seeds when the paper is published. \n\n- - -\n**Q4**: Important concepts should be briefly introduced, e.g. MAP-Elites, VPPO, etc.\n\n**A4**: The details of MAP-Elites can be found in Section 2.1. For VPPO, we only briefly mention it in Section 2.1 as Vectorized PPO in the context of PPGA and in Appendix A2 l.671-673. We will expand the explanation both in the main text and in the appendix.\n- - -\n**Q5**: The measure function **m** defined in the experiments does not seem to reflect curiosity-driven exploration.\n\n**A5**: The measure value itself is not the curiosity. Instead, whether or not the diverse measure has been visited represents the behavioral-level curiosity.\n\n- - - \n**Q6**: Imitation tasks based on real-world data (e.g., human motion capture data) could better highlight the advantages of the proposed method.\n\n**A6**. We have conducted additional experiments on various imitation learning locomotion tasks (see our response to Q3 of reviewer Cxuk), and we are happy to extend our method to imitation tasks based on real-world data in our future work.\n\n- - -\n**Q7**: How is a sufficiently explored region defined?\n\n**A7**: The authors see that the “sufficiently explored region” is mentioned on l.240 page 5. We will remove the “sufficiently” from the text. Basically, as soon as the region is visited, the EBC reward will be zero for consequent visits, to facilitate exploitation of high-performing policy in this region.\n\n- - -\n**Q8**: From the perspective of imitation learning theory, how can the learned policy outperform the expert policy?\n\n**A8**: When we talk about performance we typically talk about the QD score. Hypothetically, it is possible that the expert policy has a higher cumulative reward since the expert would not be the optimal policy. However, we do not claim that the learned policy outperforms the expert. In Humanoid (see Fig.2), we do observe a higher QD score and coverage compared to PPGA with true reward. However, as shown in Fig.4,  none of the techniques can outperform PPGA with true reward with EBC.\n\n- - -\n**Q9**: Is there an optimal balance between imitation rewards and EBC rewards?\n\n**A9**: We studied the scale of the EBC reward, $q$ in Appendix E.4. $q=2$ appears to be the best choice. A sufficiently high $q$ is needed to for the behavioral level exploration to encourage higher QD score. It is impossible to find the exact optimal choice since the q value is continuous and there are infinite choices, and the optimal choice varies from tasks to tasks, so we choose q=2 in our experiments.\n\n- - -\n**Q10**: What happens if the expert demonstrations are of low quality?\n\n**A10**: Because the demonstrations are selected to be diverse, the expert data are already sub-optimal. If the performance is much lower than even a local optimum solution, then it seems difficult to achieve policy improvement. However, this can then be said for all imitation learning methods since it is assumed that the demonstrations are of desirable quality (otherwise there is limited point to imitating them).\n\n- - -\n**Q11**: The improvement in the Halfcheetah and Walker2d tasks does not seem significant.\n\n**A11**. We have performed a **Tukey HSD Test** on Table 7; comparing our EBC-enhanced algorithm to their counterparts, and we find that 7/9 effects are positive and  5 of these are significant with $p \\leq 0.05$. Also refer to our answer to Q4 of U4H2 for further  explanations.\n\n- - -\n**Q12**: Mutual-information-based methods also achieve diverse behavioral policies. What are the advantages of QD-IRL?\n\n**A12**: The mentioned references assume it is possible to sample skills from a skill distribution, and then condition policies based on the given skill variables. In our work, the measures are not known in advance and are determined from the evaluation. The key difference is that we can optimise diversity across a particular measure rather than be limited to a particular skill distribution. So typically, unless the distribution of said techniques is uniform across the entire feasible space, QD algorithms will have more emphasis on diversity. We will discuss this difference in the updated paper.\n\n- - -\n## We hope our response fully address your concerns.", "claims_and_evidence": "The claims made in the paper are well-supported by experimental results, particularly across three benchmark environments (Halfcheetah, Walker2d, Humanoid).", "methods_and_evaluation_criteria": "The proposed method is well-justified for the problem of diverse behavior generation in robotics. The criteria are appropriate and comprehensive for evaluating diverse locomotion behaviors.", "theoretical_claims": "The theoretical foundation is solid, particularly:\n- The derivation of the EBC reward bonus using an indicator function for novel behaviors.\n- Lemma 3.1, which provides a probabilistic guarantee that EBC increases the likelihood of discovering new behaviors.\n\nThe proofs appear sound, and the methodology aligns well with existing QD and IRL literature.", "experimental_designs_or_analyses": "The experimental setup is robust. Clear comparisons showing improvements with EBC. However, more complex imitation tasks can enhance the expressiveness of the method.", "relation_to_broader_scientific_literature": "The paper is well-positioned within the fields of Imitation Learning (IL), Inverse Reinforcement Learning (IRL), and Quality Diversity (QD) Optimization.", "essential_references_not_discussed": "The related work is well discussed.", "other_strengths_and_weaknesses": "- Strengths\n  - Novel combination of IL and QD for diverse behavior generation.\n  - Extrinsic Behavioral Curiosity (EBC) is a simple yet effective exploration mechanism.\n- Weaknesses\n  - The experiments in the paper need to be further strengthened.\n  - Source code not provided.", "other_comments_or_suggestions": "- Important concepts should be briefly introduced when first mentioned, such as MAP-Elites, VPPO, etc.\n- The measure function m defined in the experiments does not seem to reflect curiosity-driven exploration.\n- The experiments were conducted only in a limited set of MuJoCo environments. Imitation tasks based on real-world data (e.g., human motion capture data) could better highlight the advantages of the proposed method.", "questions_for_authors": "- How is a sufficiently explored region defined?\n- From the perspective of imitation learning theory, how can the learned policy outperform the expert policy?\n- Is there an optimal balance between imitation rewards and EBC rewards?\n- What happens if the expert demonstrations are of low quality?\n- In Figure 2, the improvement in reward for EBC in the Halfcheetah and Walker2d tasks does not seem significant. Is this related to the difficulty of the tasks?\n- Some more concise mutual-information-based methods[1-5] also achieve diverse behavioral policies. What are the advantages of QD-IRL compared to these approaches?\n\n[1] Eysenbach B, Gupta A, Ibarz J, et al. Diversity is all you need: Learning skills without a reward function[J]. arXiv preprint arXiv:1802.06070, 2018.\n\n[2] Li Y, Song J, Ermon S. Infogail: Interpretable imitation learning from visual demonstrations[J]. Advances in neural information processing systems, 2017.\n\n[3] Strouse D J, Baumli K, Warde-Farley D, et al. Learning more skills through optimistic exploration[J]. arXiv preprint arXiv:2107.14226, 2021.\n\n[4] Peng X B, Guo Y, Halper L, et al. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters[J]. ACM Transactions On Graphics (TOG), 2022.\n\n[5] Fu H, Tang K, Lu Y, et al. Ess-InfoGAIL: Semi-supervised imitation learning from imbalanced demonstrations[J]. Advances in Neural Information Processing Systems, 2023.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this submission, the authors introduce Quality Diversity Inverse Reinforcement Learning (QD-IRL), a new framework which integrates quality-diversity (QD) optimization with inverse reinforcement learning (IRL). QD-IRL enables agents to learn diverse behaviours from limited demonstrations. The paper also introduces Extrinsic Behavioral Curiosity (EBC) which provides agents with additional rewards from an external critic based on the novelty of the agent’s behavior. The authors provide experimental results validating the effectiveness of EBC in exploring diverse locomotion policies and also compare it with existing baselines.

The reviewers believe that the EBC reward and the application of IRL to QD algorithms are novel and important contributions. Furthermore, they agree that the claims presented in the paper are sufficiently supported by empirical evidence. That said, reviewers also raised several concerns, including the lack of diverse tasks (other than locomotion) and the lack of results demonstrating practical utilities of learned behaviour (e.g. adaption or hierarchical control). The reviewers would also believe that including videos of optimal solutions in the archive would strengthen the paper, as it is difficult to understand how diverse the solutions are based on this. In the rebuttal, the authors have addressed many of the concerns, which has led to increases in scores.

Overall Assessment: Based on the reviewers' comments and discussions during and after the rebuttal, I believe that this paper makes a strong technical contribution and is worthy of being accepted at ICML. I hope the reviewers’ comments can help the authors prepare an improved version of this submission for camera-ready.
Decision: Accept (poster)