AutoQD: Automatic Discovery of Diverse Behaviors with Quality-Diversity Optimization
Authors: Saeed Hedayatian
Stefanos Nikolaidis
Venue: ICLR 2026 Poster (2026)
Keywords: Quality Diversity Optimization
Policy Representation
Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We propose AutoQD, a method that automatically learns behavior descriptors for Quality-Diversity optimization by embedding policy occupancy measures, enabling unsupervised discovery of diverse policies in reinforcement learning.

Abstract: Quality-Diversity (QD) algorithms have shown remarkable success in discovering diverse, high-performing solutions, but rely heavily on hand-crafted behavioral descriptors that constrain exploration to predefined notions of diversity. Leveraging the equivalence between policies and occupancy measures, we present a theoretically grounded approach to automatically generate behavioral descriptors by embedding the occupancy measures of policies in Markov Decision Processes. Our method, AutoQD, leverages random Fourier features to approximate the Maximum Mean Discrepancy (MMD) between policy occupancy measures, creating embeddings whose distances reflect meaningful behavioral differences. A low-dimensional projection of these embeddings that captures the most behaviorally significant dimensions can then be used as behavioral descriptors for CMA-MAE, a state of the art blackbox QD method, to discover diverse policies. We prove that our embeddings converge to true MMD distances between occupancy measures as the number of sampled trajectories and embedding dimensions increase. Through experiments in multiple continuous control tasks we demonstrate AutoQD's ability in discovering diverse policies without predefined behavioral descriptors, presenting a well-motivated alternative to prior methods in unsupervised Reinforcement Learning and QD optimization. Our approach opens new possibilities for open-ended learning and automated behavior discovery in sequential decision making settings without requiring domain-specific knowledge.

Published as a conference paper at ICLR 2026
AUTOQD:
AUTOMATIC DISCOVERY OF DIVERSE BEHAVIORS WITH QUALITY-DIVERSITY OPTIMIZATION
Saeed Hedayatian' Stefanos Nikolaidis
UUniversity of Southern California
AArchimeees AI aeedhed, ni pikolaideeuss..ee. edu
ABSTRACT
Quality-Diversity (QD) algorithms have shown remarkable success in discovering diverse, high-performing solutions, but rely heavily on hand-crafted behavioral descriptors that constrain exploration to predefined notions of diversity. Lever- aging the equivalence between policies and occupancy measures, present theoretically grounded approach to automatically generate behavioral descriptors by embedding the occupancy measures of policies in Markov Decision Processes. Our method, AutoQD, leverages random Fourier features to approximate the Max- imum Mean Discrepancy (MMD) between policy occupancy measures, creating embeddings whose distances reflect meaningful behavioral differences. A low- dimensional projection of these embeddings that captures the most behaviorally significant dimensions can then be used as behavioral descriptors for CMA-MAL a state of the art blackbox QD method, to discover diverse policies. We prove that our embeddings converge to true MMD distances between occupancy measures the number of sampled trajectories and embedding dimensions increase. Through experiments in multiple continuous control tasks demonstrate AutoQD's abil- ity discovering diverse policies without predefined behavioral descriptors, pre- senting a rell-motivated alternative prior methods in unsupervised Reinforce- ment Learning and QD optimization. Our approach opens new possibilities for open-ended learning and automated behavior discovery in sequential decision making settings without requiring domain-specifie knowledge. Source code is available /tttp::/ /ttps//gitttt com/ wttp:///oa..ceeeeeeeee///oooooo.
INTRODUCTION
Traditional optimization methods, focused solely on finding optimal solutions, often fail to capture the rich diversity of possible solutions that could be valuable in different contexts. Quality-Diversity (QD) optimization addresses this limitation by generating collections of solutions that are both high-
performing and behaviorally diverse (Cully et al. 2015b Pugh al., 2016). This approach has demonstrated success across different domains including robot locomotion (Duarte et al.) 2017
Cully et al., 2015a). game level and scenario generation (Gravina al., 2019: Bhatt et al.) 2022
protein design (Boige et al. 2023), and even image generation (Fontaine et al. 2021).
Building on these successful applications, focus on sequential decision decision-making tasks where we seek diverse and high-quality policies, setting commonly referred to as Quality-Diversity Rein- forcement Learning (QD-RL. (Tjanaka et al., 2022b) Nilsson and Cully 2021: Pierrot al. 2022). Here, the importance of behavioral diversity stems from two key considerations. First, diverse poli- cics provide robustness against changing conditions-when one policy fails, alternatives with differ- behavioral characteristics might succeed. Second, diversity is crucial for open-ended learning, where the goal extends beyond solving predefined problems to continually discovering novel capa- bilities and behaviors (Lehman and Stanley 2011).
fundamental limitation of QD algorithms, particularly challenging in such tasks, is their reliance on hand-crafted behavior descriptors (BDs). Behavior descriptors are functions that map policies to

---

Published as a conference paper at ICLR 2026
A,
dese
(st,
Agt +
J(m) =
[(H)
Sample IT
Update CMA-ES
Figure I: Overview of AutoQD. Left: Policy parameters are sampled from a CMA-ES instance and evaluated in the environment. The collected trajectories embedded via random Fourier features map ó to produce the policy embedding 30x which is then projected to a www-iiennioonl descriptor using the affine map A b. The policy added the archive based its return J(T)) and descriptors desc(T), and CMA-ES updates its distribution based on the improvement made to the archive. Right: Periodically, embeddings from the archive are used update and b via ewPCA.
low-dimensional vectors characterizing their behavior. For example, when designing controllers for a bipedal robot, researchers typically define BDs based on foot contact patterns, which allows them to characterize behaviors such as walking, jumping, and hopping. Hand-crafting BDs require sub- stantial domain knowledge, which becomes increasingly difficult as task complexity grows. Further- more, they constrain the diversity of discovered policies to variations along predefined dimensions, potentially missing interesting behavioral variations (Grillotti and Cully 2022a).
In this paper, present theoretically principled approach to automatically generating behavior de- scriptors. Our method is based on the concept of occupancy measures, which captures the expected discounted visitation frequency of state-action pairs when following policy. Crucially, under stan- dard assumptions oully-obeervable environments, there exists a one-to-one correspondence be- tween policies and their occupancy measures (Puterman 2014), making them ideal representations of behaviors as they fully characterize a policy. This differentiates our method from prior work that human data (Ding al., 2024) proxy objectives such as state reconstruction (Grillotti and Cully) 2022a), to define BDs, and a wide range of other methods from the RL literature that typically use information theoretic objectives to train fixed number of policies to be maximally different or distinguishable (Eysenbach et al. 20199] Kumar et al. 2020).
Our key insight that by embedding occupancy measures into finite-dimensional vector spaces where distances approximate the Maximum Mean Discrepancy (MMD) between the occupancy measures, can create behaviorally meaningful representations. These representations be further reduced to liwer-dimensional behavior descriptors for QD optimization. Our approach, AutoQD, addresses several limitations existing QD methods. It does not require manual specifica- tion of behavior descriptors and can potentially discover unexpected behavioral variations. Further- more, when paired with a state-of-the- blackbox QD algorithm, enables us to discover thousands of policies covering continuous behavior space.
then
Our main contributions are: (1) Developing a method to efficiently embed occupancy measures
of policies from sampled trajectories (Sec. 3.I. (2) Formally showing how the distances between these embeddings approximate the MMD distances between occupancy measures (Theorem T). (3)
Proposing iterative algorithm that alternates between QD optimization and behavior descriptor refinement (Sec. 3.2). (4) Demonstrating empirically that our approach discovers diverse, high- performing policies without requiring hand-crafted descriptors (Sec.
2

---

Published as a conference paper at ICLR 2026
2 BACKGROUND
2.1 MARKOV DECISION PROCESSES AND POLICY OPTIMIZATION
Following the established terminology in RL, we consider Markov Decision Processes (MDPs), defined by the tuple (8, P. 7), where S is the state space, A is the action space, P(s'|s, a) is the transition probability, R(s,a) a) is the reward function, and E (0, 1) is the discount factor. A policy и is a function of the state, either deterministic (I - A) or stochastic (π S ((A) representing an agent. The goal in RL is to find policy that maximizes the expected discounted return J(x) = F 7 R((( a.]].
A key concept in RL is the occupancy measure, which arises naturally when studying solutions to MDPs, For policy æ, its occupancy measure p" a distribution over state-action pairs defined as:
oc > P(S s, A. = a)
= a=(s,a)= Σ
(1)
where P(S. = 8, A, a/ =| is the probability of visiting state-action pair (s, a) at time when fol- lowing policy л. Intuitively, p (s. a) represents the discounted visitation probability of (s, a) under policy т. The occupancy measure is fundamental to reinforcement learning as many quantities of interest, including the expected return J(T), can be expressed as expectations under this measure. Importantly, under standard assumptions in fully-observable MDPs, there exists one-to-one corre- spondence between Markovian policies and their occupancy measures (see Sec. 6.9.1 of Puterman 20144), making occupancy measures complete characterization of policy behavior.
2.2 QUALITY-DIVERSITY OPTIMIZATION
Diversity (QD) optimization aims discover collection of solutions that are both high- performing and behaviorally diverse. Unlike traditional optimization, which focuses on a single optimal solution, QD maintains an archive A of solutions, each associated with both performance measure and behavior descriptor. In QD reinforcement learning (QD-RL), a solution is the pa- rameters of a policy, typically represented as neural network. The performance of policy is its expected return, ( Ex 7 R(ss, at)), which we refer to as the filness. A behavior de- scriptor is a function desc: 11 + B that maps policies to a behavior space B C R² The goal of QD optimization is to find, for each behavior vector b E B, a policy Tb that satisfies (esc((T)=== b and maximizes the objective among all such policies. In practice, the behavior space B is divided into a finite number of cells, called an archive A with the QD goal being to fill each cell with the best solution. This objective is formalized by the QD score, defined as QDScore(A) which is the total fitness of all policies in the archive. QD algorithms employ various optimization techniques including random mutations (Cully et al. 20155), evolutionary strategies (Fontaine al., 2020), and gradient- based methods (Nilsson and Cully 2021) to maximize this score.
Σnea (T))
In this work, we use CMA-MAE (oontaine and Nikolaidis 2023), which applies the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) (Hansen 2016) to QD optimization. CMA-MAE runs multiple CMA-ES optimizers in parallel, each maintaining a Gaussian distribution over policy parameters. In each iteration, we sample batch of policies from the Gaussian, evaluate their fitness, and map them into the archive via their behavior descriptors. The algorithm then ranks the policies based on their improvement to the archive and uses this ranking to update the parameters of CMA- ES. This iterative update implicitly performs natural gradient ascent on (a reformulation of) the QD score (Fontaine and Nikolaidis 202I) enabling efficient optimization of both quality and diversity.
2.3 MAXIMUM MEAN DISCREPANCY
To quantify the differences between policy behaviors, we turn to the Maximum Mean Discrepancy (MMD), a metric for comparing probability distributions. Intuitively, MMD measures the difference of two distributions by comparing statistics of their samples. Given two distributions P and Q over space X', and a feature map á X' + RDD the MMD is defined as:
MMD P. Q) [Ex=p(o(X) Ey Ey>olo(Y)]l
(2)
3

---

Published as a conference paper at ICLR 2026
When the feature map corresponds characteristic kernel, such the Gaussian kernel k(z, y) = exp(-1læ - -yll= MMD defines a metric over the space of probability distribu- tions: it is non-negative, symmetric, satisfies the triangle inequality, and is zero if and only if the distributions are identical. The MMD can be computed using the "kernel trick" (Schölkopf and Smola 2002) with positive definite kernel k<x,,)) y), allowing for implicit feature maps in mlimit-dimensiona spaces.
While there are different ways of measuring distances between distributions, we chose MMD due to its desirable properties that allow us to obtain embeddings of the distributions in a computationally efficient manner. Notably, the MMD with Gaussian kernel can be efficiently approximated using random Fourier features (Rahimi and Recht [207]) providing fimite- snite-dimensionll embedding that preserves the geometry the original kernel space.
METHOD
Our method, AutoQD, automatically discovers behavior descriptors for quality-diversity optimiza- tion sequential decision-makinn domains. The key insight to use occupancy measures to char- acterize policy behaviors, and then extract low-dimensional BDs that capture the main variations in policy behavior. The method operates in three steps: (1) embedding policies into a space where distances approximate behavioral differences, (2) extracting low- low-dimensional BDs from these em- beddings, and (3) using these descriptors with standard QD algorithm (CMA-MAE) to discover diverse policies.
3.1 POLICY EMBEDDING VIA RANDOM FEATURES
((s,a)
(1),.)
(s,a) m p
1 p1 MMD p²)))
pin
(s,a) m pr
Figure 2: Overview of the proposed policy embedding. Each policy オ; induces an occupancy measure nº over state-action pairs. From sampled trajectories, a feature map & embeds the policies into a vector space. Theorem guarantees that the Euclidean distance between embeddings approx- imates the Maximum Mean Discrepancy (MMD) between the corresponding occupancy measures.
To explore diverse behavioral variations, we embed each policy into fiite-diimensional space where Euclidean distances approximate the MMD between the occupancy measures. Since oc- cupancy measures fully characterize policy behavior, and MMD with a Gaussian kernel defines a valid metric over them, this distance provides meaningful measure of behavioral similarity. The challenge is that the Gaussian kernel corresponds to an infinite dimensional feature map (Schölkopf and Smola 22002). 2m02) While the kernel trick allows pairwise MMD computation without explicitly structing the features (Gretton et al. 2012). it can only produce (nn)) distances and does not yield explicit embeddings. To overcome this, approximate the Gaussian kernel using random Fourier features (Rahimi and Recht 2007), which provide D-dimensional mapping that approximates the infinite dimensional feature space.
Concretely, given state s E S and action a E A, we define a D-dimensional random feature map
V
2 D
((s. a)
cos w/ [s: + b.)...., w al + bb)]
where w. ~ NO, a 21), bj ~ U(0, and [s; a] denotes the concatenation of state and action vectors. The kernel width determines the scale which state-action pairs considered similar.
4

---

We should also mention a subtlety regarding the practical computation of the policy embedding s" To compute d² we need i.i.d samples from the occupancy measure p". We can obtain these by collecting n independent rollouts of π and selecting one state-action pair from each trajectory ac- Published as a conference paper at ICLR 2026
Consider policy æ with occupancy measure p². With slight abuse of notation, let d denote the embedding of т,, defined as the empirical mean of the random Fourier features of n iid. samples from p". That is, o- = + E; d(s az) where (sT aF...... (5 a are ii.d. samples from p². This embedding (which under the policy's 's occupancy measure. The lg distance between embeddings of two policies ap- proximates their behavioral difference measured by MMD of their occupancy measures:
refer to policy embedding) approximates the expected feature map
| 1 Rs MMD()"). PF*))
The quality of this approximation characterized by the following theorem:
Theorem 1 (MMD Approximation), For any two policies Ti, R2 with occupancy measure P1 P2 and embeddings Φ1, 62 estimated by taking the mean of the D dimensional random Fourier features of i.idd samples from each occupancy measure,
Pr [116e1 -2ll2 - MMD(
3
2e nce² +0 to exp ( (
+ 6e nz?
644d + 2)
(5)
where d is the dimension of state-action vectors and 0 is a constant. A proof is provided in Appendix
This theorem establishes that the distance between our embeddings, | - 22 reliably approxi- mates the true MMD between occupancy measures with high probability. Hence, the geometry of occupancy measures is captured by their embeddings. The approximation error controlled by the number of samples n and the embedding dimension D. Importantly, the state-action dimension d ap- pears only once the denominator of an exponential term, suggesting that scaling to more complex domains requires D to grow only linearly with d.
cording Geometric distribution with parameter y. However, this leads to very inefficient use of the collected data as it discards all but one transition from each trajectory. Therefore, in practice,
attT as defined in Eq. 6 instead of d= as the policy embedding. Intuitively. this is justified by noting that uh" has the same expectation as as o but leverages all collected transitions, which potentially reduces the variance. More concretely, we show in Appendix B that the distance between these v embeddings also approximates the true MMD, with an approximation error that decays exponentially in n and D, albeit at a different rate. Consequently, we use 6tth to denote the policy embedding from this point on.
12
T
7)
Σ 1=0
7 (s{ at)
(6)
j=1
3.2 THE AUTOQD ALGORITHIM
Given policy embeddings that encode behavioral differences, we project them into a low- dimensional space (with h < D dimensions) to as behavior deseriptors for QD optimization. As explained in Sec. 2.2, this is needed because QD algorithms discretize each dimension of the behavior space, yielding an archive that grows exponentially with dimension. We perform this pro- jection using an affine transformation desc(T) = A + b. The parameters of this transformation, A & b I² are derived by performing Calibrated Weighted PCA (cwPCA), on the embed- dings of policies in the archive. cwPCA makes some small modifications to PCA (F.RS.. [901] to make more suitable for the specification of behavior descriptors. particular, it applies PCA to policy embeddings after weighting them by their finness so that better policies have greater influence on the principal directions. This biases the components toward capturing behavior variation among better policies, encouraging exploration among high-quality behaviors. Following this, we apply a simple calibration step: we scale each output axis so that most projected embeddings lie in the range [-1, 1]. This ensures stable and fixed archive bounds throughout the algorithm. Appendix C provides full details, including the precise form of the affine map, additional motivation, and an ablation study on the effect of the weighting mechanism.
5

---

Published as a conference paper at ICLR 2026
Putting these pieces together, Algorithm I presents method in its entirety. AutoQD combines the BDs described above with C.M--MAE to discover diverse and high segh-peforming policies. alternates between: (1) using the current descriptors discover diverse policies with QD optimiza- tion, and (2) refining the descriptors based on the expanded archive of policies. For clarity and conciseness, [ Jabstracts the internal mechanics of CMA-MAB omitting details of its ini- tialization and update step, Detailed pseudocodes for these components are provided in Appendix
Algorithm AutoQD
Input: MDP (S,A, P. R. 7). embedding dimension D, behavioral descriptor dimension k number of iterations n, Update schedule (tz. .2....)
2:
Output: Archive of diverse and high- nigh-performing policies &
3: Initialize:
CMA-MAE archive and parameters: A, QDState MMAMMEEIIni (k)
Affine map parameters: A,
4: Sample random features [wi)em ~v o(0,0=71) and offsets (b. ~ L((0 2T)
5: for E (1.2..... } do
6:
if E {11 (21,2.....) then
Time update descriptors
e Tm for म. E
Policy embeddings as defined Eq
8:
A, b t oWPCA(M, Update behavioral descriptors: desc(T)) = Anp + end if
9:
10:
QDState CMA.MAE. Mte. (A, QDState, desc)
Perform one step of QD optimization
end
13: return final archive
EXPERIMENTS
To empirically validate the effectiveness of AutoQD in discovering diverse and high perforrirf behaviors, we evaluated it on six standard continuous control tasks from the Gymnasium library (Towers et al. (0024), including five from the widely-used MuJoCo benchmark suite (Todorov al.) 2012). These environments are standard benchmarks for RL and remain challenging for many evo- lutionary approaches, despite recent progress the field.
4.1 BASELINES
We compare our method five baselines that have demonstrated strong performance prior work and represent distinct strategies for obtaining diverse and high-quality populations.
RegularQD applies standard QD algorithm using hand-crafted BDs specific each environment.
Aurora (Grillotti and Cully 2022a) learns behavior space by training autoencoder on the vis- ited states and uses the latent encoding of the last state rollout of the policy the BD.
LSTM-Aurora (Chalumeau et al.) 2023) extends AURORA by using LSTMs to encode full trajec- tories and using the hidden state of the encoder LSTM as the behavioral descriptor. DvD-ES Horkerr-Hlldr et al.) 2020) employs evolutionary strategies jointly optimize popula- tion of policies for both task performance and diversity.
SMERL (Kumar et al.] (2020) is RL-based algorithm that trains skill-conditioned policy using Soft Actor-Critic (Haarnoja et al. 2018) and uses an additional reward derived from a discriminator encourage diversity among skills.
We use CMA-MAE (Fontaine and Nikolaidis 2023) for all QD methods due to its simplicity and robustness across tasks. Additionally, following Choromanski
(2018), we Toeplitz matrices to parameterize the policies for these methods to reduce parameters and improve the performance of CMA-MAE. The full set of hyperparameters and more discussion about the implementation details are provided in Appendix
4.2 EVALUATION METRICS
To ensure a fair comparison, we employ three main metrics: the Ground- Truth QD Score (GT QD Score), the Vendi Score (VS), and the Quality-Weighte Vendi Score (qVS),

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of AutoQD and baseline methods across six environments. Each environment is evaluated using GT QD Score (QD) reported in units of 10 for readability, qVS, and VS metrics. Reported values are the mean = standard enror over evaluations with three different random seeds. Higher values indicate better performance for all metrics.
Metric
AutoQD
RegularQD QD
Aurora
LSTM-Aurora
DvD-ES
SMERL
Ant
QD (x10")
361.43 + 2.17
182.58
2.53
5.57 = 1.48
19.24 + 1.1
0.29 #.1
1.02 0.23
qVs
60.23 + 9.4
39.35 3.99
0.56 + 0.01
1.11 + 0.41
0.49 + 0.00
0.97 0.15
VS
72.37 + 10.63
39.49 + 3.93
1.11 = 0.01
1.9 + 0.54
1.00 + 0.00
1.29 + 0.18
HalfCheetah
QD (x10")
30.78 2.72
24.91 1 3.43
11.35 4.69
11.38 1: 2.02
0.85 0.23
1.61 0.37
qVS
1.35 + 0.6
2.07 + 0.13
2.39 + 0.42
1.71 + 0.21
1.15 +15++00 0.09
1.78 + 0.51
VS
5.29 1.59
3.44 + 0.34
5.8 +.881
4.83 + 0.16
1.19 +19++0.11 0.11
3.55 0.56
Hopper
QD (x10*)
1.84 1: 0.29
0.03
1.06 0.09
1: 0.01
0.18
0.15
qVS
1.94 1: 0.04
1.35 = 0.05
0.66 0.09
0.36 0.08
0.9 0.32
0.22
VS
4.5 0.2
2.85 0.04
2.67 0.09
2.13
1.27 0.13
3.34 0.24
Swimmer
QD (x10*)
21.31 4.57
11.09 + 0.08
8.05 = 0.58
10:26 + 0.72
0.22 + 0.02
0.02 + 0.00
qVS
6.04 + 0.66
3.17 + 0.19
3.09 + 0.15
3.82 + 0.77
1.16 + 0.1
0.24 0.06
VS
16.92 = 3.68
4.67 + 0.35
6.75 = 0.25
7.21 + 1.95
122 0.13
2.16 0.57
Walker2d
QD (x10*)
18.36 2.58
11.39 1: 0.55
7.71 + 1,26
12.99 0.77
0.61 0.11
1.17 0.14
qVS
7.22 2.08
9.08 + 0.53
1.11 + 0.08
2.12 + 0.07
1.47 0.26
2.74 0.42
VS
8.4 3.2
10.17 + 0.89
2.5 + 0.13
4.17 0.47
1.58 0.29
3.2 0.17
BipedalWalker
QD (x10*)
6.09 0.22
1.81 0.02
0.14 0.03
qVS
5.16
0.81 = 0.02
1.12 = 0.08
1.67 0.34
0.02
2.11 2.11
VS
12.17 0.52
1.57 0.03
2.88
0.21
3.36
1.06
5.54 0.42
GT QD Score the QD score of a population when its solutions are inserted into an archive that uses hand-designed BDs. It evaluates the quality and diversity of population using expert-defined behavior spaces. These are the same BDs that the RegularQD baseline uses and are commonly em- ployed in prior work. Vendi Score (VS) (Friedman and Dieng 2023) quantifies a population's ' diver- sity based on pairwise similarities between their occupancy embeddings. Given a population of size and a positive-definite kernel matrix K =x where Kij [0. 1] is the similarity of the i-th and j-th members of the population, the Vendi Score is defined VS(K) exp (- >i= 1 Xe log Å²). where 'ι ,2,...,,, are the normalized eigenvalues of K (i.e., they sum to one). Importantly, VS measures the effective population size and enables comparison between populations of varying sizes, as is the case with our baselines. Lastly, Quality -Weightdd Vendi Score (qVS) (Nguyen and Di- eng, 2024) extends the VS by incorporating solution quality: q VS(K) ( > (m:)) VS(K), where (m,)) is the fitness of the i-th individual, Since, VS requires all objectives to be positive, we scale all objectives to the (0, 1] range by adding constant offset and dividing each return by the highest mean return achieved by any of the algorithms in that environment, prior to computing qVS.
To construct the kernel matrix K used by VS and QVS, we use a Gaussian kernel applied to the inner product of the Random Fourier Feature (RFF) embeddings of policies. Although these embeddings are structurally similar to those used by AutoQD, we employ a separate, larger set RFFs solely for evaluation to ensure fair comparison. Our choice of embeddings is motivated by our theoretical results showing that distances between these embeddings asymptotically reflect distances between policy occupancy measures. For more detailed analysis of VS and its theoretical properties, refer the reader to Nguyen and Dieng (2024).
4.3
MAIN RESULTS: POLICY DISCOVERY
Table I compares AutoQD with the baseline algorithms across six tasks. For each combination, we report the mean and standard error across three random seeds. AutoQD consistently outperforms

---

Published as a conference paper at ICLR 2026
Performance with Friction Changes
Performance with Mass Changes
300
Policy
Algorithm
200
AutoOD
Aurora
LSTM Aurora
RegularOD
SMERL
DVD
-100
3
6
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
Priction Coefficient
Mass Scale
Figure 3: Performance of the best policy found by each algorithm under changing friction (left) or mass scale (right). The shaded regions represent the standard error across 32 evaluation seeds.
the baselines in most environments: The only exceptions being the Walker2d and HalfCheetah environments, where the best qVS and VS are achieved by RegularQD and Aurora, respectively.
In HalfCheetah, AutoQD was able to discover diverse policies, but the policies tended to be rela- tively low performing, reflected by its high VS and low qVS. Visual inspection showed that AutoQD discovered many policies that moved forward by "sliding" via subtle joint movements. While these behaviors were novel and diverse, they resulted in slow movement, and as result, lower overall rewards. Walker2d, AutoQD seemingly overemphasized the role of the bottom-mosi (feet) joints, missing out interesting behavioral variations that could be achieved, for instance, by fully lifting the legs. Nevertheless, AutoQD ranked second in this domain, outperforming all other baselines. Appendix F provides more fine-grained statistics and further analysis of AutoQD's 's lower perfor- mance in these two domains. Moreover, Appendix presents qualitative analysis and visualizations of the behaviors discovered by AutoQD.
4.4 APPLICATION: ADAPTATION TO DIFFERENT DYNAMICS
A key motivation for discovering diverse populations is adaptability, since collection of behav- iorally diverse policies is more likely to include one that performs well under altered environment conditions. To test this, we evaluated populations from AutoQD) and the baselines in the Bipedal- Walker environment with two types of variations: scaling the friction coefficient and altering the robot mass. Figure 3 shows the performance of each method's best policy under these changes, and their area under the curve (AUC) provides a scalar measure of robustness, with higher AUC indicating greater adaptability. As Table 2 shows, AutoQD's population maintains relatively high performance across both variations, and achieves the highest AUC.
Beyond measuring the performance of the single best policy, it is also helpful to analyze the perfor- mance of the top- por-fforornin subset and even the full distribution of returns across all policies within population. To quantify this, counted the number of policies in each population that maintained a significant fraction 0 < p < of the performance achieved by the best overall policy found in the original, unaltered environment. Specifically, if / represents the highest return achieved by any policy across all populations in the original environment, we counted policies whose mean return was least Rp. Intuitively, this would reflect the number of policies that successfully adapt the environmental changes by maintaining a high reward.
Figure 4 illustrates the count of these "successful" policies across different friction coefficients for two success thresholds: p = 0.9 and p = 0.7. At the strict threshold of p = 0.9, AutoQD's population consistently includes a larger number successfully adapted policies compared to the baselines. As we loosen the success criteria to 0.7, the number of successful policies increases across the board. Notably, the population of Aurrra is shown to contain many successful policies when the friction coefficient is in the range [1.3]. However, increases further, both AutoQD and DvD-ES prove to be more capable of finding successful policies. We observe similar general trends when varying the robot mass scale, with additional plots and the full distribution of returns provided in Appendix G Overall, these findings show that AutoQD's generated population not only contains single policy that adapt changing environmental
the friction coefficient
8

---

Published as a conference paper at ICLR 2026
p= 0.9
P=07
Number of Succestu Policies
b
begzarto 9vD
Number of Successhul Policles
100
LST
LETHiheses DvD
in
600
40
Prtction Coetficient
Prtction Coofficnt
Figure 4: Number of successfully adapting policies in each population under changing friction. A
policy is considered successful if its mean return at least Rp, where R is the highest overall return achieved in the unaltered environment. Results shown for two success thresholds: p 0.9 (left) and p = 0.7 (right).
Metric
AutoQD
RegularQD
Aurora
LSTM--urorr
DyD-ES
SMERL
Friction AUC
1429.66
30.27
1309.41 260.60
1226.29 271.83
1204.03
496.23
Mass AUC
295.65
12.8
113.68
71.38
Table 2: Comparison of Area Under the Curve (AUC) for each algorithm across friction and mass variations. Higher values indicate better adaptability to changing parameters.
conditions, but also includes many (either the most the second most among the baselines) policies that demonstrate substantial adaptability.
5 RELATED WORK
Quality-Diversity methods. Quality-Diverse algorithms discover collections of solutions that balance performance and diversity across specified behavioral dimensions (Pugh et al. 2016). MAP- Elites (Cully et al, 2015b) pioneered this approach by maintaining an archive of solutions organized by their behavioral characteristics. CMA-ME (Fontaine al. (0200) reformulated the QD problem as single objective optimization, enabling the use of powerful blackbox optimization methods like CMA-ES (Hansen 2016) instead of relying solely on random mutations. The more recent CMA- MAE (Fontaine and Nikolaidis 2023). which is used as the backbone QD algorithm in this paper, further improved this method by introducing the idea of soft archives. More recently, gradient-based variants like DQD (Fontaine and Nikolaidis 2021). PGA-MAP-Elinn (Nilsson and Cully 2021 and PPGA (Batra al. 2024) have made further progress by leveraging policy gradients.
Unsupervised QD approaches. Most prior work such as Aurora (Grillotti and Cully 2022a LSTM-Aurora (Chalumeau et al.] 2023), and TAXONS (Paolo et al. 2020) learn behavioral de- scriptors by training autoencoders states, relying on the hypothesis that representations capturing state information also reflect policy behavior. Grillotti and Cully (2022b) argues in favor of this ap- proach by showing that the entropy of the encoded trajectories lower-bounds the entropy of the full trajectories. However, their analysis assumes a discrete state space and does not formally link tra- jectory entropy to policy diversity. In contrast, AutoQD's embeddings are based on policy-induced occupancy measures, offering theoretically grounded representation of behavior.
Unsupervised RL for skill discovery. RL community has explored related approaches for learn- ing diverse behaviors. DIAYN (Eysenbach et al.] 2019) maximizes mutual information between skills and states, encouraging skills to visit distinct regions of the state space without using reward signals. DADS (Sharma et al. 2020) extends this by maximizing mutual information between skills and transitions, favoring predictable outcomes. However, both methods ignore the task reward.
SMERL (Kumar et al. 2020) and DoMiNo (Zahavy et al. 2023) incorporate task rewards into di- versity objectives. SMERL directl augments DIAYN's 's objective with task rewards, while DoMiNo frames the problem as a constrained MDP, maximizing diversity by encouraging distance between

---

Published as a conference paper at ICLR 2026
state occupancies of near-optimal policies. Both highlight the benefits of diverse, terh-perforning policies but require a fixed number of skills and tend to scale poorly with skill count. In this work, compared our method with SMERL, as its open-source implementation is readily available.
middle ground between QD methods and unsuper- vised RI. approaches, DvD (Parker-Holder et al. (220)) characterizes policies through their actions in (random) set of states, resembling the off-policy embeddings from Pacchiano et al. (2020). How- ever, these embeddings lack the theoretical backing that our method provides. Furthermore, like SMERL, their proposed algorithm requires specifying the number of policies in advance and faces stability issues this number increases. [hen et al. (2023) also share conceptual similarities with our approach, though in the context of transfer learning. They learn a Q-function basis by training policies on features from randomly initialized networks. In contrast, we use random Fourier features to embed occupancy measures directly, enabling QD optimization without prior RL training.
Policy embedding and representation.
Our use of Random Fourier Features (Rahimi and Recht (207)) to embed occupancy measures con- nects to theoretical work on kernel approximations (Rudi and Rosasco 2017 Rahimi and Recht 2008). key insight of our approach recognizing that these techniques can applied to rep- resent policy behaviors in theoretically principled way. By embedding occupancy measures and applying dimensionality reduction, we automatically generate behavioral descriptors that capture essential policy characteristics without manual specification.
6 CONCLUSION
We introduced AutoQD, novel approach for applying Quality-Diversin (QD) optimization quential decision-making tasks without handcrafted behavior descriptors. By embedding policies based their occupancy measures and projecting to compact behavior space, AutoQD can be integrated with CMA-MAE and achieves strong empirical performance.
Limitations. AutoQD has several limitations. First, in highly stochastic environments, accurately estimating policy embeddings may require many trajectories, which reduces sample efficiency. Sec- ond, as discussed in Appendix [F, when the behavior descriptor is low-dimensional, exploration may concentrate on stable yet simple behaviors, hindering the discovery of more complex ones. In ad- dition, the choice of kernel bandwidth influences the sensitivity of the embeddings; while we use fixed bandwidth in this work, dynamically adapting it during training could allow the embeddings to better capture behavioral distinctions at different stages of learning. In this study, we used AutoQD with CMA-MAE because of its simplicity and stability. However, AutoQD is in principle compati- ble with any standard QD algorithm. As result, inherits the scalability challenges of existing QD optimizers, particularly with large policy networks and high- -ige--iinnsional behavior spaces (Tjanaka ct al. 2023a), but can also benefit directly from future advances in QD algorithm design. Finally, al- though QD methods promote behavioral diversity, they may fall short of RI. methods in pure reward optimization. Nevertheless, we expect this gap narrow as QD algorithms continue to improve.
Future Work. A promising direction involves integrating AutoQD with gradient- -ased QD meth- ods such PGA-ME (Nilsson and Cully] 2021) and PPGA (Batra et al. (024)). While these methods typically offer better perlormance, their training objectives can become unstable as a result of the iterative refinement of the behavior space by AutoQD. By identifying the sources of these insta- bilities and mitigating them, future work can improve performance and increase sample efficiency. Furthermore, extending AutoQD to environments with image-based observations is also direction worth pursuing and could unlock exciting capabilities for autonomous agents. Lastly, the policy em- beddings produced by AutoQD could find applications beyond QD, including open-ended learning, imitation learning, and inverse RL. They may also prove useful for analyzing learned policies, for example through clustering and other forms of characterization.
7 ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
We would like to thank Varun Bhatt, Sophie Hsu, Aaquib Tabrez, Bryon Tjanaka, and Shihan Zhao for their feedback preliminary version of this work, as well as Saba Hashemi for assistance
10

---

Published as a conference paper at ICLR 2026
with the design of the visualizations. This work has been partially supported by the NSF CAREER #2145077, NSF NRI #2024949 and the DARPA EMHAT project.
REFERENCES
Sumeet Batra, Bryon Tjanaka, Matthew Christopher Fontaine, Aleksei Petrenko, Stefanos Niko- laidis, and Gaurav Sukhatme. Proximal policy gradient arborescence for quality diver- sity reinforcement learning. In The Twelfth International Conference Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: //openzeview. nwt. fort/forum? d=IFKI [hvdmZ
Varun Bhatt, Bryon Tjanaka, Matthew Fontaine, and Stefanos Nikolaidis. Deep surrogate assisted generation of environments. Advances in Neural Information Processing Systems, 35:37762 37777, 2022. URL http:::/apprsssspppp// nips. httpppp.cc/pppee....ee////eeee/e
Raphaël Boige, Guillaume Richard, Jérémie Dona, Thomas Pierrot, and Antoine Cully. Gradient- informed quality diversity for the illumination of discrete spaces. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1291228 2023.
Félix Chalumeau, Raphaël Boige, Bryan Lim, Valentin Macé, Maxime Allard, Arthur Flajolet, An- toine Cully, and Thomas Pierrot. Neuroevolution is a competitive alternative to reinforcement learning for skill discovery. In The Eleventh International Conference on Learning Representa- tions, ICLR, 2023. URL /tttps://operreveee ew net./
d= www.667777yyyyy
Boyuan Chen, Chuning Zhu, Pulkit Agrawal, Kaiqing Zhang, and Abhishek Gupta. Self-supervised reinforcement learning that transfers using random features. Advances in Neural Information Processing Systems, 36:56411-56436, 2023.
Krzysztof Choromanski, Mark Rowland, Vikas Sindhwani, Richard Turner, and Adrian Weller. Structured evolution with compact architectures for scalable policy optimization. In International Conference Machine Learning, pages 970-978. PMLR, 2018
Antoine Cully, Jeff Clune, Danesh Tarapore, and lean-Baptise Mouret. Robots that can adapt like animals. Nature, 521(7553):503-507 2015a.
Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean- Man-Baptisst Mouret. Robots that can adapt like animals. Nature, 521(7553),03-5077, 2015b.
Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, and Joel Lehman. Quality diversity through human feedback: Towards open-ended diversity-driven optimization. In Forty-first International Con- ference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL /tt.atts////ppeeeeeeesssssss wwtp://eeeetottttteeeeeee
Miguel Duarte, Jorge Gomes, Sancho Moura Oliveira, and Anders Lyhne Christensen. Evolution of repertoire-baser based control for robots with complex locomotor systems. IEEE Transactions on Evolutionary Compulation, 22(2):314-333 2017.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In 7th International Conference Learning Repre- sentations, ICLR, 2019. URL Ttl.hhtps:: httpt:tttppspetteele /ttp://///ppeeeeeeeeee net htttt/ffoummddd=Sx.....
Matthew Fontaine and Stefanos Nikolaidis. Differentiable quality diversity. Advances in Neural Information Processing Systems, 34:10040-1005,, 2021.
Matthew Fontaine and Stefanos Nikolaidis. Covariance matrix adaptation map-annealing. In Pro- ceedings of the genetic and evolutionary computation conference, pages 456-465, 2023.
Matthew C Fontaine, Julian Togelius, Stefanos Nikolaidis, and Amy K Hoover. Covariance matrix adaptation for the rapid illumination of behavior space. In Proceedings of the 2020 genetic and evolutionary computation conference, pages 94-102, 2020.

---

Published as a conference paper at ICLR 2026
Matthew C Fontaine, Ruilin Liu, Ahmed Khalifa, Jignesh Modi. Julian Togelius, Amy K Hoover, and Stefanos Nikolaidis. Illuminating mario scenes in the latent space of a generative adversarial network. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 5922-5930, 2021.
Dan Friedman and Adji Bousso Dieng, The vendi score: A diversity evaluation metric for machine learning. Trans. Mach. Learn. Res., 2023. URL attpss///penneeeiwwweeeeeeeeeeee
g970Hboyk1
Karl Pearson F.R.S. Lii. on lines and planes of closest to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-572, 1901. doi: 10. 00.10000147884400099977777
Damien Garreau, Wittawat Jitkrittum, and Motonobu Kanagawa. Large sample analysis of the median heuristic. arXiv preprint urXiv:1707.0766% 2017.
Daniele Gravina, Ahmed Khalifa, Antonios Liapis, Julian Togelius, and Georgios N Yannakakis. Procedural content generation through quality diversity. In 2019 IEEE Conference on Games (CoG), pages 1-8. IEEE, 2019.
Arthur Gretton, Karsten M Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773 2012.
Luca Grillotti and Antoine Cully, Unsupervised behavior discovery with quality-diverssi optimiza- tion. IEEE Transactions on Evolutionary Computation, 26(6):1599-152, 2022a.
Luca Grillotti and Antoine Cully. Discovering unsupervised behaviours from full state trajectories. In ICLR Workshop on Agent Learning in Open-Endednesss, 2022b.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pages 1861-1870 Pmlr, 2018.
Nikolaus Hansen. The ema evolution strategy: A tutorial. arXiv preprint arXiv:1604.00772, 2016.
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances in Neural Information Processing Systems, 33:81 198-8210, 2020.
Joel Lehman and Risto Miikkulanenn Enhancing divergent search through extinction events. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2015, Madrid, Spain, July 11-15, 2015, pages 951-958. ACM, 2015. 2139480 2146668
URL https::/dci..ooo 0ttp://///0...0/111111115...
Joel Lehman and Kenneth Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189-223, 2011.
Colin McDiarmid et al. On the method of bounded differences. Surveys in combinatorics, 141(1): 148-188, 1989.
Quan Nguyen and Adji Bousso Dieng. Quality-weighted vendi scores and their application di- verse experimental design. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024. URL httpttts///ppeeeviieet net/ forum?
(d=gDDMMAc9p)
Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 866-875, 2021.
Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Krzysztof Choromanski, Anna Choromanska, and Michael Jordan. Learning to score behaviors for guided policy optimization. In International Conference Machine Learning, pages 7445-7454. PMLR, 2020.
Paolo Pagliuca, Nicola Milano, and Stefano Nolfi. Efficacy of modern neuro-evolutionary strategies for continuous control optimization. Frontiers Robotics AI, 7:98, 2020. URL fattps://dom arq//10.389/ Frobt. 2020. 00098
12

---

Published as a conference paper at ICLR 2026
Giuseppe Paolo, Alban Laflaquiere, Alexandre Coninx, and Stephane Doncieux. Unsupervised learning and exploration of reachable outcome space. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 2379-23888 -2855 IEEE, 2020.
Jack BunkerHHoler, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances Neural Information Processing Systems, 13:18050-1806 2020.
Thomas Pierrot, Valentin Macé, Felix Chalumeau, Arthur Flajolet, Geoffrey Cideron, Karim Be- guir, Antoine Cully, Olivier Sigaud, and Nicolas Perrin-Gilbert. Diversity policy gradient for sample efficient quality-diversity optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1075-1083, 2022.
Iosif Pinelis. Optimum bounds for the distributions of martingales in banach spaces. arXiv preprint 1208. 2200, 2012.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley, Quality diversity: A new frontier for evolu- tionary computation. Frontiers Robotics and AI, 3:40, 2016.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. Advances in neural information processing systems, 21, 2008
Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. Advances in neural information processing systems, 30, 2017.
Bernhard Schölkopf and Alexander J Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In 8th International Conference on Learning Representations, ICLR, 2020. URL https: hettts::/opperevvieee
n forum? id HUgLHR
Bryon Tjanaka, Matthew C. Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradients for differentiable quality diversity in reinforcement learning. In GECCO '22: Genetic and Evolutionary Computation Conference, Boston, Massachusetts, USA, July 9 13, 2022, pages 1102-1111. ACM, 2022a. htt.ttttts//aaat 0000g//....1//3122291111 3512290. 3528705
1145
Bryon Tjanaka, Matthew C Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradients for differentiable quality diversity in reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1102-1111, 2022b.
Bryon Tjanaka, Matthew C Fontaine, David H Lee, Aniruddha Kalkar, and Stefanos Nikolaidis. Training diverse high- figh-dimensional controllers by scaling covariance matrix adaptation map- annealing. IEEE Robotics and Automation Letters, 8(10):6771-6778. 2023a.
Bryon Tjanaka, Matthew C Fontaine, David H Lec, Yulun Zhang, Nivedit Reddy Balam, Nathaniel Dennler, Sujay S Garlanka, Nikitas Dimitri Klapsis, and Stefanos Nikolaidis. Pyribs: bare- bones python library for quality diversity optimization. In Proceedings of the Genetic and Evo- lutionary Computation Conference, GECCO '23. page 220-229, New York, NY, USA, 2023b. Association for Computing Machinery. ISBN 9798400701191. doi: 10. 10.1145/3583131.3590877 3590374
URL hhttp::///o// http::::///oooor////0// 110.0...1443330
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
13

---

Published as a conference paper at ICLR 2026
Mark Towers, Ariel Kwiatkowski, Jordan Terr, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: standard interface for reinforcement learning environments. arXiv preprint arXlv:2407.17032, 2024.
Tom Zahavy, Yannick Schroecker, Feryal M. Behbahani, Kate Baumli, Sebastian Flennerhag, Shaobo Hou, and Satinder Singh. Discovering policies with domino: Diversity optimization maintaining near optimality. In The Eleventh International Conference on Learning Representa- tions, ICLR, 2023. URL https: /
bettps::/openevvllte opwpreveeeeeeeeeeeeeeeeeee net
14

---

Published as a conference paper at ICLR 2026
A PROOF OF THEOREM
As our main result, we show that the l2 distance between our embeddings of occupancy measures (estimated from samples) is good approximation of the true MMD between occupancy measures. Formally, let P and Q be two occupancy measures defined over dimensional state-action vectors. Let k be the Gaussian kernel and Φ IR + R. be the random Fourier features that map state-action vectors to a D dimensional embedding space. Given samples {I1, ......a In from P we define n Σ; ő(z,) as the embedding of P. Similarly, we define @Q as the embedding of Q obtained from samples y1," " : Vn- The claim is that | &2 ll is good approximation of MMD(P, Q)- The following are the steps we take to complete the proof.
Op 1
1. We start by showing that with high probability MMD*((,Q)) and ||ór - dQ l1 are close to one another. This is done in four steps where
(a) show that 16p ΦQ l2 is close to MMD,
(b) show that MMD is close MMD,
(c) show that MMD is close to MMD?
(d) conjoin the previous three bounds to show that 10pp Φell is close MMD
2. Then, we show that MMD Q) is close to 1lóp Φq 2.
Recall that from the definition MMD and using the kernel trick we have
MMD* = Ex.x (K(, x')] + Byyx'alk((XX")) 2Ex ((X,
(7)
Let us start from | - Φ2 1/ and step-by-step get closer to the quantity above. We have
| - 221ه = (òp - -0)--2))- (0r oQ) = =pOp + o500 20F
(8)
(9)
Examining each of these three terms more carefully, we see that
of
1 nt in
Σ
(II)) =
- n Σ o(xx) o(x;) + n
Σ
((z;))((Iz))
(10)
0000-
Σ
(f Po()s) = n
Σ
)())) ((()) +
Σ p(u)) o(1))
(11)
n 1
n
$5O4
1
Σ
(lx;)² (v))))
(12)
n²
1,j
Now, let MMD be defined as
MMD = 1 Σ o(x))) )(x) + - Σ (
006) - n Σ ((x)) T o(x)),
2
(13)
where Ze o/'' s are i.i.d. samples from P and yt, v/ are i.i.d. samples from Q. From this, we can see that the expectations of || o and MMD quite similar and in fact, they the same the limit n -> xc.
n2 (n=1)
1 -Ex-Pr-g [1 +l [
E [10P - S0ll
E MMD
+ n
(14)
Using the fact that, by the definition of random Fourier features, the entries of d(x) are bounded in [|0p ΦQ [2] and MMD at most (()).
|- -- v2
VD* VD we see that the difference between E
Similarly, for any x. x', we see that ((x)* ((x') E 2, Therefore, each of the n summands of MMD take values in [-8,8). 8).
15

---

Published as a conference paper at ICLR 2026
Now, application of Hoeffding's inequality yields that with probability at least - 8:
log
MMD
MMD
161
(15)
2n
Similarly, |1lóp - olll is a function of 2n independent samples and satisfies a bounded difference 16 + which at most 24
property: changing a single sample changes its value by at most for n 2 1. Therefore, application of McDiarmid's inequality (McDiarmid et al. 1989) shows that with probability at least 6,
log
|1lpp d -E 01 +al1?1
< 24h
(16)
Combining these with the triangle inequality and using the union bound we get that with probability at least 1 1 25
log + o
11 - dall -MMD| < 48
(17)
For large values of the first term the right hand side dominates, therefore say that with probability at least 1 - 26
(V log
1 dQll2 MMD O
(18)
Therefore, for some non-negative constant we have
Pr [10p 20l MMD
3 E
x 2e nec
(19)
We now move on to the next part of the proof. Define MMD follows.
MMM
n Σ k (x;,
1
Σ (vi, vi)
2
(xi, ve)
(20)
In words, MMD is just like MMD but with all of the inner products of random Fourier features replaced by kernel operations. We can see that
12
MMD MMD
(zx, x4) - ((x))
+(x [x{)]
(21)
n
[k (Ui, vi) - ((v:)* d()))
(22)
12
2
[k (a,, v:) o(x,)² -(xx)(((
(23)
Next, make use of the following lemma that guarantees the uniform convergence of Fourier features stated in section 3 Rahimi and Recht (2007):
Pr sup (d(x) d(y) k(z, v) < o (1 exp 2.
De²
(24)
4(d + 2)
where d is the dimensionality of the state-action vectors. This implies that each of the terms (sum-
mands) in 23 most & with probability at least 0
16 g
exp
)) Substituting & in
23 and using the triangle inequality, we see that
2
MMD MMD
<
E,
(25)
i=
16

---

Published as a conference paper at ICLR 2026
with probability at least O 0 exp (-
Dza 64(d+2)
)) Therefore,
16
))
MMD MMD]
3 c
< 0
exp 64(d + 2) (64(d+2))
(26)
This brings to the third step of the proof where connect MMD with (M, Q). This is more straight forward to show, since each term in the former is the Monte Carlo estimate of the corresponding expectation in the latter. More formally,
MMD - MMD (P,Q)= =
k x1) - -BP(((XX
(27)
+
k (yis v/ - -Bokk(X,
(28)
1=1
- 2 le (a,, v1) EpPq(K(XX)]]
(29)
Now note that in each of the three parentheses the first term is the empirical mean and the second term is the true mean. Combining this with the fact that k(x, y) is always between 0 and 1, we can apply Hoeffding's s inequality each term to get a tail bound for each of them. For example, for the first parenthesis we get
(122 n (a),))) Ex(x(X,X))
-nE²
Pr
> 4
ccxp
(30)
Applying the triangle inequality and the union bound we get
nn MMD MMD? > < 6xp
(31)
8
Now, we can combine
26 and to bound the difference between 11óp - - and
MMD?(P.Q) Q). Note that by triangle inequality
|| ق2lه MMD Q) < 11 20l12
MMD
MMD
MMD + MMD MMD3
(32)
Combining the bounds that we have for each of the terms on the right hand side, we get
Pr [llldp قاlه MMD > 3s] 2e -nce + + (² exp
(0A(a22))
-D 2) ))
nc?
+ Ge
(33)
This ensures that as we increase the number samples n and the number of features D, the proba- bility of enror decays exponentially. Lastly, we shall derive a bound on ||0e doll2 - MMD(P.Q) Note that
Óp - oq MMDD = | - dællz MMD(P.Q) (1 + MMD(P, Q))
(34)
|orr ølll2 NMD(P.Q)
(35)
|1 ΦQ
2222 MMD(P,Q))
(36)
Replacing this back into the bound in 33 get
Pr |lop dolla MMD(P, Q)
3
-
< 2e
nncs
+0
exp
-D=
)) + 6e nz2
4
64(d + 2)
(37)
Which is the result that sought. This ensures that increase and D, the distances between the embeddings of occupancy measures reflect the true MMD distance between them with a high probability.
17

---

Published as a conference paper at ICLR 2026
B JUSTIFICATION FOR THE EMPIRICAL TRAJECTORY EMBEDDINGS
we use a defined in Eq. to compute the policy embeddings in practice. Here, we connect this choice to the MMD approximation Theorem To show that using the practical, trajectory-baate -assed embeddings t/p and th in place of the theoretically analyzed occupancy-meatee embeddings óp and ΦQ (used in Appendix(A] is theoretically justified. To do so, show that, with high probability, the distance between 7pp and th is arbitrary close to that between Φp and Φo for any two occupancy measures P, Q. Hence, all of the results in the main proof remain valid when replace o with Rh*
As mentioned in Sec. 3.1
First, let define the true mean feature vector
Pr (( ((((, a));
(38)
where Pix is the occupancy measure of some policy л. The analysis in Appendix uses i.i.d samples 3 P and their empirical mean Φp Σ i= o(x;). However, in practice, obtain feature
vectors 2; from trajectories Ti by setting
24 = (1 7)
Σ
r a))),,a))
(39)
and using their empirical mean up 2im1 Zzz
e2ا12ه is small with high probability. This guarantees that replacing by db in the main proof does not affect any bounds, other than incurring an additional approximation error that will be derived below.
We will show that ||| whell2
11 -101-
First, let prove two simple lemmas regarding o(x;) and 2 vectors.
Lemma 1. Let 2 (1 7) Σ, yf ('sss, 0z) be feature vector obtained by sampling a trajectory from occupancy measure P and let x o(s, a) where the state action pair (s, a) is also sampled from P. Then, [p[2] [x).
Proof. Note that Ep(r} = pe by definition. Also,
B[z] = (1 7) Σ BloosS, Ad)
(40)
Σ (s..a)
(41)
8,0
pp. Therefore, and x have the same expectation.
(42)
Note (fiittehhorioon bias). The derivation above assumed an infinite-horizon setting (I' oo) so that a trajectory feature x = (1 7) Eizo 7 f((( 02) satisfies E| = pp. In practice we use truncated trajectories of length T' oc. Defining 2" (1 7) Et- 7 7 ac). the error (bias)
between the infinite and finite-horizon features is
by E
Ei2
7
* Sc.
(43)
Using a)]( & 22 (following lemma) we obtain
|b 1112 < (1 (1-2) 2 V2n
(44)
Thus, truncating trajectories at length T' introduces deterministic bias of most V2yT in the trajectory features. Consequently, when comparing two occupancy measures P. Q the truncation contributes at most ||by,e||2 2 + |ibr,ellz S 2/2y to the total enror and should be added to the sampling and RFF approximation terms in the final probabilistic bound (i.e., should be added to the He term in the final bound 56). For simplicity, we consider the infinite horizon setting throughout the rest of the proof.
18

---

Published as a conference paper at ICLR 2026
Lemma 2. For any trajectory and its corresponding featur vector defined according Equa-
tion 39 all coordinates of lie in [- DD VD
v2
This follows immediately by noting that each coordinate of Φ(5, a) is in the same range and each coordinate of z a geometric sum of these values.
Proof.
Hence, from the coordinate bound a))(s.)) & 22/D we have ||o(s, a)ll < v2 and likewise || & 22.
Now, we can define centered feature vectors U; := p(x;) - pr (for d-estimators) and U, == 2j - ur (for tj-estimators). For either case, using the triangle inequality yields that
||U [1x(z=)]l2 |p 2v2,
(45)
|| < || + lImell2 < 2v2.
(46)
We can now apply the following inequality which follows from Theorem 3.5 of Pinelis (2012): Lemma 3. Let U1...., Un be independent, zero-mean random vectors in IR satisfying ||U||| 2v2 almost surely. Then, for any 0,
<
-nn²
Pr
Up 2
exx
(47)
16
Proof. Follows from applying Theorem 3.5 of Pinelis (2012) with increments d, Uz- Since
|UBll2 & 222, we have 6% < (2V2)2 8n. Taking D 1, the theorem yields the bound above.
Setting r = and using the centered feature vectors U2 and Uj. this lemma shows that
Pr |lóp upllz
< 2 exp
(48)
2
64
|nn2
Pr [1bb Hrlle
< exp
(49)
2
64
By union bound, the probability that both deviations exceed e/2 is at most 4 exp((nne2/6A There- fore, with probability at least 4 4eee /64).
|| vr 2 < -1 prll2 | prl12 &
(50)
Similarly, can apply the same bound independently for another occupancy measure Q. Applying the union bound across both policies that with probability at least 8 eexp(-n=" / /64), both Opp
d and 1Í vell are most c. But note that this would imply that
11pp vQl1 1|pp Φ2ll
< 11(bp (4) (óp اq)هه
(51)
||bp dpll + 11 - d211
(52)
2s.
(53)
Putting everything together, we have shown
Pr [| - rioll - 110p ΦQ
2c]
< Se nec?
(54)
67 Combining this with the bound of Eq. 37 using the triangle inequality, we can see that 1|up v/2 la close to MMD(P, Q), with high probability. More concretely, if we use subscript
with 1.
of for the constant e in Eq. 37 and subscript of 2 for the one in Eq. 54 to avoid confusion, we see that
11 Pr ||br voll2 MMD(P.Q)
< 2e ncis= +0
(0( exp ( ( 2) )
De2
))
(55)
4
E2
+ 6e n²
=5 + 88 nooe??
(56)
19

---

Published as a conference paper at ICLR 2026
This completes the proof and shows that embeddings र! used in practice enjoy the same asymptotic guarantees as o" The only difference is that the approximation error is increased by an additional term, which, crucially, also decreases exponentially as the number of samples n increases.
To conclude, let us briefly compare the embeddings resulting from d² and शे Both are obtained by computing the empirical mean of RFF embeddings from independently sampled trajectories. The key difference lies in how they embed each trajectory: ø² embeds a single state-action pair, sam- pled according geometric distribution, from each trajectory, whereas vt computes a weighted average of the embedding of all state action pairs within trajectory. Both embeddings share the same expectation, but g potentially exhibits lower variance because it averages the state-action embeddings over the entire trajectory. This benefit comes at the cost of an added approximation eroor term the probabilistic bound obtained for y However, this term is very similar to the other terms already present in the bound and, like them, decreases exponentially with n.
C CALIBRATED WEIGHTED PCA
Our calibrated and weighted PCA variant addresses three critical requirements for effective spec- ification of behavioral descriptors in QD optimization: finding meaningful behavioral variations, ensuring compatibility with QD archives, and adapting to the evolving population of solutions. Be- low, detail each component and its motivation.
Given policies [TI,..., J with embeddings V = " and fitness scores (estimated returns) {fi.... fm] our algorithm proceeds as follows:
Step 1: Score normalization. We normalize fitness scores to form a weight distribution:
Ji min, fj 1
max
(57)
max, fy min, fy m
je Wi
(58)
where the weights sum policy.
1. The 1 term in Eq. 57 ensures minimum contribution from each m
Motivation:
While all policies provide information about the behavioral space, pegehperforming policies represent more successful strategies that we want to emphasize when discovering diverse behaviors. Low Low-performing policies often exhibit undesirable behaviors that should have less influ- ence on our descriptors. The normalization ensures all policies contribute at least minimally while prioritizing those with higher fitness.
Step 2: Weighted PCA. We compute:
(weighted mean)
(59)
i=
ST.
4
(centered embeddings)
(60)
G.
= i (weighted centered embeddings)
(61)
We perform SVD on the weighted centered embeddings to obtain the top k principal components P E Dxk
Motivation: PCA offers several advantages for our context:
It provides an affine transformation that preserves the geometry of the original embedding
space, maintaining relative distances between policies up scaling and translation.
Unlike non-linear dimensionality reduction techniques, doesn't 't introduce distortions that could misrepresent behavioral similarities.
It requires no additional hyperparameters or iterative training procedures.
The orthogonality of principal components ensures that each behavioral measure captures distinct aspect of policy behavior.
20

---

Published as a conference paper at ICLR 2026
By weighting the PCA computation, we focus capturing variations among high- performing policies.
Step 3: Calibration. We compute the 5th and 95th percentile quantiles of uncalibrated projections
desc(T) (*) (t) along each dimension:
Vlow (
(62)
Qhigh quantile 0.95)
(63)
The final transformation maps (loo: Uhigh] to :
S Qhigh Qlow
(64)
C s qlow
(65)
A diag(s)
(66)
c Au
(67)
(Operations in the first two lines are element-wise)
Motivation: Calibration addresses practical challenge in QD optimization:
PCA naturally produces dimensions with different scales based variance, which would require dimension-specifie archive bounds.
Calibration standardizes all dimensions to a fixed range [-1, 1]. allowing the QD algorithm use consistent archive bounds.
This standardization enables more uniform coverage of the archive along each dimension, preventing the QD algorithm from disproportionately exploring directions with naturally higher variance.
The 5th/95th percentile choice ensures that most solutions fall within the archive bounds.
Importantly, the calibration step preserves the affine nature of the transformation, combining the projection and scaling into single linear operation A with offset b. This results in computation- ally efficient mapping that maintains the essential geometric properties of the embedding space. The final behavioral descriptor desc(T) A + adaptively identifies and scales the most signifi- cant behavioral dimensions, focusing variations among high-performing policies while ensuring compatibility with fixed-bount QD archives.
We conclude this section by noting important caveat regarding the weighting strategy in ewPCA. In principle, weighting is designed to emphasize the contribution of high gigh-performing solutions, guid- ing exploration toward sometimes be counterproductive by placing excessive emphasis on behaviors that are useful but ulti- mately suboptimal. For example, robotics task, a simple stabilization strategy such as preventing the robot from falling represents an accessible local optimum. While stabilization is beneficial in the early stages, if more advanced locomotion patterns have not yet been discovered, the weighting mechanism in cwPCA may disproportionately highlight variations of this basic strategy. This can lead to the discovery of a diverse set of stabilization behaviors that remain confined and suboptimal region of the behavior space. Although this issue is less likely to happen in high- dimensional behavior spaces, can hinder the performance in constrained behavior spaces where early suboptimal variations may be amplified and prematurely lead the search into local optimum.
promising regions of the behavior space. However, this mechanism can
narrow
To assess the extent of this effect, we conducted an ablation study the Walker2d environment, comparing the performance of AutoQD with and without fitness weighting in PCA. Table 3 sum- marizes the results of this ablation. While weighting slightly improved the performance across all statistically significant according to double-sided Mann-Wii Whitey
metrics, the difference test (a-velue> > 0.7), suggesting that the impact of weighting may vary case by case, depending on the structure of the behavior space and difficulty of exploration.
21

---

Published as a conference paper at ICLR 2026
Table Comparison of AutoQD with and without weighting, Reported values are mean standard enror over evaluations with eight different random seeds.
Method
GT QD Score (x10*)
Mean Objective
Vendi Score
AutoQD (with weighting)
17.74 + 3.85
1162.8 + 116.8
8.35 + 4.14 1143.6 + 83.8
AutoQD (w/o weighting)
17.74 + 3.51
7.67 + 3.32
D DETAILS OF CMA-MAP
Here we provide the pseudocode for both the initialization of CMA-MAE (Algorithm 2 as well as its update step (Algorithm 33. These were abstracted as function calls in Algorithm the main paper for the sake of clarity.
In these pseudocodes, note that the internal parameters CMA-ES include Gaussian "search" distribution that are used to sample candidate policy parameters (Line 4 of Algorithm 3 and are updated through CMA-ES (Line[[7 Algorithm B). For a more detailed exposition of CMA-MAE, refer the reader to Fontaine and Nikolaidis 2023).
Algorithm 2 CMA.MA..Init
1: function CMA_MAE_INTT(h
2: Input: Behavior space dimension &
3: Output: Empty archive A, Optimization state QDState
4: Initialize CMA-ES internal parameters: CMA. CMA.ES.SHr ES_State
5: Initialize empty archive A
Uniform grid over [-1.2, 1.2]*
6: for all cells e in A do
7: Le min_moo min_objective
Acceptance threshold
8: end for
9: QDState (CMA.ES_State (te)eea)
10: return (A, QDState)
end function
Algorithm CMA.MAE_StEe
1: function CMA MAE SA QDState, desc)
2: Required Hyperparameeers: learning rate a, batch size À
3: for 1. do
4:
Sample candidate: 0, (OQnstatt, EQDState)
5:
trajectories + collect_roll rollouts (0₂)
6:
f f mean-retur:. (trajectories)
Fitness
7:
Compute & according to Eq. from trajectories
8:
BD < desc(v)
9:
+ calculate.cell (A, BD)
Locate corresponding cell from the archive >> Improvement over the cell's threshold
10:
A + f te if f Le then
11:
12:
Replace current occupant of cell in the archive with 0, te a)t af
13:
14:
end end for
15:
16:
Rank 0, by A;
Adapt CMA-ES parameters based on improvement rankings A
18:
return updated
end function
22

---

Published as a conference paper at ICLR 2026
E IMPLEMENTATION DETAILS AND HYPERPARAMETERS
E. ENVIRONMENTS
We use the latest versions of the environments available in Gymnasium (Towers al. 2024) in our experiments:
BipedalWaller-v3,
Ant-v5,
HalfCheetah-v55,
Hopper-v5,
Swimmer-v5,
Walker2d-v5.
E.2 NETWORK ARCHITECTURE
All QD-based methods (AutoQD, Aurora, LSTM-Aurora, and RegularQD) use identical policy ar- chitectures: a neural network with two hidden layers 128 units each and tanh activation functions. These networks employ a Toeplitz structure, which constrains the weight matrices such that all en- tries along each diagonal share the same value (Choromanski et al. 2018). This constraint enforces parameter sharing and reduces the search space.
SMERL uses a similar network architecture but with ReLU activations and without the Toeplitz constraint. Since SMERL employs gradient-based RL optimization, the Toeplitz structure is not necessary. We use ReLU activations to keep consistency with the author's 's hyperparameters and the open source implementations.
DvD-ES uses the authors' provided implementation, which employs MLPs with two hidden layers of size 32.
E.3 QD ALGORITIM CONRIGURATION
All methods utilize the standard Pyribs (Tjanaka et al. 2023b) implementation of CMA-MAE (Fontaine and Nikolaidis, (023) the underlying QD algorithm. They employ grid archives that are discretized to 10 cells along each dimension and use 5 emitters with different initial step sizes of (0.01 2 2 The rest of the configuration presented below.
Table 4: Common QD algorithm parameters shared across all methods
Parameter
Value
Number CMA-ES Instances
Initial Step Size (a%)
{0.01 2
Jael
Batch Size
64
Restart Rule
100 iterations
Archive Learning Rate
0.01
Total Iterations
500
Evaluations per Policy
5
E.4 AUTOQD
Our proposed method uses Random Fourier Features (RFF) to map trajectories/policcs into em- beddings and progressively refines measure map during optimization using calibrated weighted PCA to convert policy embeddings into low-dimensional behavior descriptors. The embedding map normalizes the observations based on the trajectories that it observes throughout its lifetime.
The update schedule indicates the iterations at which the measure map refined using the current archive.
23

---

Published as a conference paper at ICLR 2026
Table AutoOD-specific parameters
Parameter
Value
RFF Embedding
Embedding Dimension
100 True
State Normalization
Kernel Width
V state dim + action dim 0.999
Discount Factor (r)
Measure Map
Measures Dimension
Update Schedule
20, 50, 100, 200, 300]
E.5 AURORA
Aurora learns behavioral characterization using an autoencoder that reconstructs states.
Table 6: AURORA parameters
Parameter
Value
Encoder Architecture
Mapping
- R 664, 32]
Hidden Layers
Latent Dimension
Decoder Architecture
Mapping
Rd1 S
Hidden Layers
322 64)
AutoEncoder Training
Max Epochs
50
Learning Rate
0.001
Batch Size
64
Validation Split
0.2
Early Stopping Patience
10
Update Schedule
[20, 50, 100,,0000 300
At each iteration, the autoencoder is trained for a maximum of 50 epochs on 80% all data. A validation loss is computed using the remaining 20% and it does not decrease for 10 consecutive epochs, the training can stop earlier.
E.6 LSTM-AURORA
This variant of AURORA uses an LSTM-based architecture to encode full trajectories. The encoder maps sequences of states hidden states. The last hidden state of trajectory is mapped to latent vector (the hehavioral descriptor). The decoder maps this latent back to a hidden state vector and reconstructs the trajectory starting with this hidden state and using teacher forcing.
The trajectory sampling frequency of 10 means that every 10th state in trajectory is used for encoding, following the authors' implementation.
E.7 REGULARQD
The baseline RegularQD method uses handcrafted behavioral descriptors specific each environ- ment. For all of the environments except Swimmer, these are the foot-contact frequencies which are commonly used in literature. For Swimmer, we three descriptors that measure angular span (i.e., how much the joints bend), phase coordination (i.e., how well the joints coordinate), and straightness (i.e., how straight the trajectory is).
24

---

Published as a conference paper at ICLR 2026
Table 7: LSTM-Aurora parameters
Parameter
Value
Encoder-Decoder Architecture
Type
LSTM
Hidden Dimension
32
Latent Dimension
4
Hidden-to-Latent Map Type
Linear
Latent-to- to-dddeen Map Type
Linear
Teacher Foreing
True
Trajectory Sampling Frequency
10
AutoEncoder Training
Epochs
50
Learning Rate
0.001
Batch Size
64
Validation Split
0.2
Early Stopping Patience
10
Update Schedule
[00, 50, 100, 200, 300]
E.8 SMERL
Our implementation of SMERL based on an open source implementation, modified slightly to make it compatible with the latest version of the environments and to add parallelization. Other than increasing the size of network's ' hidden layers (two hidden layers of size 128), doubling the number of skills 10, and increasing the total training steps 1.6 10" total timesteps, keep the default hyperparameters.
E.9 DVD-ES
We use the DvD-ES implementation provided by the authros (link) with slight modifications make it compatible with the latest versions of the environments. Other than the number of policies (we use 10) we keep the default hyperparameters.
E. EVALUATION
In Sec. 4.2 stated that the Vendi Score (VS) relies positive-definite similarity kernel. For this purpose, we use the Gaussian (RBF) kernel, defined for a pair of embeddings x and y as:
K exp (->|ix y|l3)
(68)
To select an appropriate value of 7, we adopt a variant of the median heuristic GGarreau et al. 2017 which
In
(69)
median (||x; x, 112)
where the median is computed over the pairwise squared distances between all policy embeddings. This choice ensures that two embeddings separated by the median distance will have a similarity of K = 0.5, offering an intuitive scaling of the kernel. To reduce computational overhead for methods that generate large number of policies, we randomly subsample up 1000 embeddings when computing the median distance.
Finally, we emphasize that this similarity kernel is distinct from the one used construct the random Fourier feature (RFF) embeddings described in Sec. 3.1
E.11 COMPUTATIONAL RESOURCES
All of our experiments were conducted on local machines with an AMD Ryzen Threadripper PRO 5995WX 64-Cores CPU, 64GB of memory and either NVIDIA GeForce RTX 4090 NVIDIA
25

---

Published as a conference paper at ICLR 2026
GeForce RTX 3090 GPU, Each training rum for any of the algorithms took 3 hours except the experiments for the SMERL baseline which took around day each. Furthermore, the evaluation of each population took 1 hours.
F ADDITIONAL RESULTS FROM EXPERIMENTS
The results in the main paper primarily focus on aggregate measures of quality and diversity, namely GT QD Score and qVS. While these metrics capture the combined effect of quality and diversity, they do not reveal how each algorithm manages the inherent trade-off between the two. Achieving high diversity often comes at the expense of average quality, since generating novel behaviors re- quires deviating from the "optimal" behavior. For example, in bipedal locomotion, behaviors such as hopping on one leg or sliding forward are diverse but achieve lower quality compared to standard two-legged walking, as they result slower forward motion.
To better understand this trade-off, we compare the normalized mean quality and diversity across all six evaluation tasks. Quality is measured the mean fitness of all policies in a population, while diversity is measured by the Vendi Score. Both values are normalized to [0. 1] per task to enable cross-task comparison (raw values are reported Table [). Figure [ visualizes the trade-off scatter plot, with quality on the x-axis and diversity on the y-axis. Each point corresponds to the outcome of an algorithm on one task, averaged over three evaluation seeds. Ideally, populations would achieve both high quality and high diversity (top right corner.. The diagonal line represents a balanced 1-to-1 1 trade-off and points above it indicate more efficient quality- trade-offs.
Several observations can be drawn from Figure First, AutoQD achieves 5/6 points above the diagonal, highlighting its effectiveness balancing quality and diversity. The next-best method, RegularQD, achieves 4/6, while other baseline exceeds 2/6. Second, AutoQD tends sacrifice some quality to achieve higher diversity, often producing the most diverse populations. This sug- gests that its learned behavior descriptors particularly effective capturing diverse behavioral variations. By contrast, DvD-ES typically achieves the highest mean quality but shows very limited diversity, with points concentrated in the bottom right.
1.0
3.5
Diverstty
3.5
Regu awon
AutrOo
Autora
Auroo
DUD-ES
0.4-
QVERL
9.2
0/2
Quality
Figure 5: Quality-diversity trade-off of algorithms across domains. The x-axis shows normal- ized mean fitness (quality) and the y-axis shows normalized Vendi score (diversity). Each point corresponds the outcome of algorithm in one of the six domains. Points above the diagonal line exhibit more favorable trade-off of quality for diversity.
Fine-grained results. Table reports more detailed statistics for all methods: mean fitness, max- imum fitness, Vendi Score, and coverage of the ground-truth archive (i.e., the number of occupied cells in an archive with hand-crafted descriptors). While the first two metrics indicate quality and the latter two indicate diversity, interpreting them in isolation can be misleading. For example, mean objective tends to favor smaller populations (e.g., DvD-ES, SMERL) or those with correlated behav- ior spaces (e.g., Aurora), the fraction of hirf-pennonming policies in large, independent archive
26

---

Published as a conference paper at ICLR 2026
decreases exponentially (see ablations in Appendix H for more empirical evidence). Similarly, GT archive coverage is biased toward larger populations, since smaller populations may have too few policies to fill the descriptor space uniformly.
Even with these caveats, several patterns can be observed. First, SMERL, as an RL-based method, consistently achieves the highest maximum fitness in 5/6 tasks. This aligns with prior work (T [anaka
et al. 2022a), since reward functions in continuous-contro tasks are often shaped for RL optimiza-
tion, making them less suited to evolutionary strategies like CMA-MAE (Pagliuca
al.) 2020).
That said, advances in QD algorithms are rapidly narrowing this gap (Batra et al. 2024). Second, in HalfCheetah and Walker2d, the two domains where AutoQD underperforms compared to baselines, the causes differ. In HalfCheetah, AutoQD produces populations with significantly lower mean fit- ness. Rollout inspection revealed that many learned behaviors involved sliding close to the ground, propelled by small, rapid leg movements. Although these policies were diverse (as reflected in both Vendi Score and GT archive coverage), their quality remained too low to compete with the baselines. In Walker2d. the opposite occurred: AutoQD achieved high mean fitness but underperformed Reg- ularQD in terms of diversity. Here, AutoQD tended to focus on variations of gaits dominated by the lower joints, while neglecting behaviors involving opper-body movement.
In both domains, the issue appears to be linked with early convergence to specific behavioral modes. Sliding in HalfCheetah and lower-joint motions in Walker2d are highly stable and thus readily dis- covered, making them likely attractors early on during training. Given that we restrict the learned behavior space to low-dimensional (4-d) subspace, it is plausible that only variants of these be- haviors are captured, constraining further exploration. Future work may mitigate this by employing higher-dimensional descriptors pruning the learned descriptors, for example via extinction events (Lehman and Müikkulainen 2015).
The last column of Table g shows the performance of a gradient-based QD algorithm, PGA-ME (Nilsson and Cully 2021). when using ground truth behavior descriptors, similar to the RegularQD baseline. The performance of this algorithm, in particular its ability outperform RegularQD and its high maximum objective value, serves to illustrate the potential of replacing CMA-MAE with more powerful QD algorithms. As noted in the paper, combining AutoQD's learned descriptors with gradient-based QD methods such as PGA-ME is non-trivial. However, the results presented here indicate strong potential for the development of such algorithms. One primary area where AutoQD lagged behind some baselines is pure optimization capability, which manifested as lower maximum policy performance in some domains. PGA-ME's strong performance in this regard re- inforces our hypothesis that this limitation can be addressed by replacing CMA-MAE with stronger QD algorithms, highlighting a viable direction for future work.
G ADDITIONAL RESULTS FROM ADAPTATION EXPERIMENTS
Here, we provide further data and analysis derived from our adaptation experiments, discussed in
Section 4.4
Figure 6 is analogous to Figure 4 presented in the main paper and illustrates the number of poli- cies that successfully adapt to changes in the robot mass scale. It reveals similar pattern in that AutoQD's 's population consistently includes more successful policies under the strict threshold of p 0.9 and generally maintains the second-highest count (behind LSTM-Aurora) under the relaxed threshold of = 0.7.
We also include the full distribution of policy returns under five different friction coefficients and mass scales in Figure and Figure 8 respectively. To facilitate clearer comparison among the top- performing methods, we exclude RegularQD and SMERL from these figures, as they consistently exhibited poorer performance in this domain. These box plots show that AutoQD's population generally exhibits lower mean return, suggesting that many of its policies fail to adapt to the altered environmental conditions. Crucially, however, it also exhibits longer tails toward high returns, indicating that it has discovered a number of policies that are exceptionally effective at adapting to the environmental changes. This contrasts with, for instance, DvD-ES, which primarily displays the inverse behavior. Policies found by DvD-ES maintain a relatively high mean performance but exhibit much less variance and shorter tails. Consequently, while DvD-ES generally achieves the
and
27

---

Published as a conference paper at ICLR 2026
Table 8: Fine-grained results from the main experiments. Similar the results presented in the main paper, the reported values are the mean + standard error over evaluations with three different random seeds.
Metric
AutoQu
RegularQD
Aurora
LSTM-Aurera
D+D-ES
SMERL
PGA-ME
Mcan Objective
056.08
16.3G
15,60
112:27 88.69
3,50
100.20
121.00
1560.15
106.75
Coverage
12.06
1:00::::00
6.00
185.00
Score
72.37
10.63
3.90
1.50
1,00
1.28
8,55
Ohjontiv
1719.21
73.15
1422
46.00
1440.05
225.10
1898.91
302.00
-9.00
2461.71 033.22
3885.86
661.80
HallChrreah
Mean Otjective
803.33 1. 142.79
2528. 13 1: 254.52
1629.46
240
1422.47
206.38
4038.59 70.11
2032.38
401.47
3141.56 J: 74.26
GT Coverge
257.67 +28.14
96.00 2.65
06.33
4.31
76.33 + 5.36
2,000
0.58
6.38 + 1.45
868000 4.04
Vendi Scure
5.29 1: 1.50
1: 0.51 4421.82
3.64
5.80
0.81
4.83 1:0 0.1.1
1.19 1:::::0 0.10
3.55 1 0.58
5.07 1: 0.06
Max Objective
8892.08
564.88
370.81
4208.990
100.30
2843.77
d: 677.20
4114.53
43.11
6280.00
623.47
7853.81 d: 162.50
Hopper
Mean Objective
1002.51
+92.51
1105:51
527.87
05.83
321.75
30.94
1615.54
418.8
1302.92
229.27
2254 31 + -3.40
GT Coverage
12:33 0.60
9.67 0.53
18:30
0.35
19.00 0.58
3.35 L
7,38 0.33
18.53 0.30
Vendi Scase
0.200
0.04
=
1,27
3.34
3:24
Objective
478.91
1234.00
1344.32
120.88
47.02
Swimmer
Mean Dbjective
12.82 1321-77
162.21
194.67
345.35
1,41
242 #
Coveage
186.63
463.67 8.78
446.07
623.67 61.54
0.67
5.00 0.59
571.00 6.43
Vendi Score
16.92 1.068
4.67 0.33
6.75 0.25
7:21 1.98
1,20 0.13
2.18 0.57
5.94
Max Objeetive
330.76 0.86
249.12 1.38
361.71 0.227
350.60 01966
3666855 1.25
43.02 0.05
303.80
1.36
Walker2d
Mean Ofjective
1173.78 1: 102.76
1150 1: 64:45
510.84 10.85
025 01 1: 00
1106 18.53
1065.01 151.00
1747.66 1: 30.00
GI Coverage
156.67 2.2078
91.33 a: 0.33
151,00
12.06
215.65
8.88
4.117 2: (1.8
1967 = 0,39
#19.67 = 4,10
Vendi Score
8.40+33.20 +-320
10.17
+ 0.80
2.50
0.15
4.17 + +-0.77
1,58 + 0.229
3.20 + 0.17
0.16 + 1.10
Max Objective
2278.20
1315.46
2234,76
203.43
1900.61
308.75
2098. 40 223.70
1306.30 1. 00.28
3800.25 570.32
4845.52
325.05
BipedalWalker
Mean Objective
33.18-
8.74
182.07
-51.42
Coverage
$52.33
2.35
9:33
240.67
Score
0.52
3,30
1.06
5.64 0.42
4.17
Ohjjenive
239-43 18:47
6.95
250.54 11.63
198.44
100.52 46.35
331.00
0,9
p
Namber Successu polcees
12
Awswon
SHEILL
Namabor of Stocessdu Pobkokg
1000
8
60
40
4
20
0.6
0.8
1.0
1.4
1.6
0.6
0.8
1.0
3.2
1.4
1.6
Mux
Mas
Figure 6: Number of successfully adapting policies in each population under changing mass. A policy is considered successful if its mean return is at least Rp, where R is the highest overall return achieved in the unaltered environment. Results are shown for two success thresholds: p = 0.9 (left) and p 0.7 (right).
highest mean return in Figures 7 and it contains fewer policies that are close-to-best, as evidenced in Figures 4 [6] and
We note that this distribution of returns is not unexpected: when diverse collection of policies generated, it is natural that many of them will perform poorly when substantial changes are in- troduced to the environment. However, the critical requirement is that subset of these policies must maintain relatively high performance despite the changes in dynamics. Indeed, our expecta- tion from a diverse population is not overall robustness across all policies; rather, we seek to include some policies that are significantly better than others at adapting to the given changes.
H EFFECT OF EMBEDDING AND BEHAVIOR SPACE DIMENSIONS
We perform two ablation studies to investigate how the dimensionality of (i) the behavior descriptors and (ii) the random Fourier feature embeddings affects performance. All experiments conducted on the BipedalWalker environment using three different random seeds. We report the mean and standard error (shown error bars) for three key metrics:
28

---

Published as a conference paper at ICLR 2026
Distribution of Roburns
Aigorithm AutoQD
LSTM-Aurora
Return Palteios
150
50
0.20
0.80
1.00 Priction Confficlent
3.00
4,00
Figure 7: Distribution of policy returns for populations generated by each algorithm, evaluated under five different friction coefficients. Policies with negative returns are excluded.
Distrilutton of Retarns
8
Algortthm
250
AutsQp
PPlicies
050
LSTM Aurora
Aurora
50
0.70
0.90
1.20
1.40
1.66
Mass Scale
Figure 8: Distribution of policy returns for populations generated by each algorithm, evaluated under five different mass seales. Policies with negative returns are excluded.
Rffect of Embedding Dimension Metrics
Mean Objective
Vendi Score
QD Score
73030
10
10
Store
63000
Embedding Dimension
Embeddiing Dimensson
Embeddlng Dimsnssion
Figure 9: Ablating RFF embedding dimension on Wilker. The plot report mean values over
3 random seeds with bars indicating standard errors.
29

---

Published as a conference paper at ICLR 2026
Effect of Behavioral Descriptor Dimension on Metrics
Mean Objective
Vendi Score
QD Score
12
100
10
Score
53090
23030
13000
1.5
15
2.3
3,0
3.5
4.0
1.0
155
2.3
2.5
3,0
4.0
18
15
23
25
3.0
4.0
Descriptor Dimension
Descriptor Dimension
Descriptor Dimeusion
Figure 10: Ablating BD's dimension BipedalWalker. The plot report mean values over 3 random seeds with bars indicating standard errors.
Mean Objective. The average of the objective (fitness) values across all policies found by the algorithm with a given configuration. represents the overall quality discovered solutions.
Vendi Score. A measure of behavioral diversity. Recall that it can be interpreted as the effective population size.
QD Score. The Ground Truth QD score introduced in the main paper. It captures both the quality and the human-interpretable diversity the discovered policies.
Note that quality-weighted Vendi Score (qVS) used in the main paper is just the product of the mean objective, normalized and scaled to be in [0, 1], with the raw Vendi score. Here we report the raw Vendi score and the unnormalized mean objectives separately to provide clearer picture of quality and diversity independently.
For the behavior descriptors, vary the dimensionality of the measure space from to 4. Fig. 10 presents the results which show a consistent improvement in both the QD score and the Vendi score the dimensionality increases. In contrast, the mean objective value decreases with higher descrip- tor dimensionality. This suggests that as the behavior space becomes more complex, the fraction of high-quality solutions among all discovered solutions declines. This trend is intuitive: if we assume that a fixed proportion (0, 1) of behaviors along each axis are high-performing, then the overall solutions decreases exponentially with the number of descriptor dimensions.
fraction of Additionally, swwer-dimensional behavior spaces may be easier for the underlying CMA-ES opti- mizer to search effectively for high-quality solutions.
For the embeddings, we vary the number of random Fourier features from 10 to 1000, evaluating configurations with 10, 100, 500, and 1000 dimensions. Fig 9 presents the results which suggest that performance is relatively robust to this hyperparameter. While our main experiments use 100- dimensional embeddings, even with as few as 10 features, AutoQD achieves competitive perfor- mance in terms of both quality and diversity.
I STABILITY OF LEARNED BEHAVIOR DESCRIPTORS
In this section, we examine the stability of the behavior descriptors (BDs) generated by AutoQD and the relevant baselines.
AutoQD, along with other baselines that rely on BDs (RegularQD, Aurora, and LSTM-Aurora), estimates the behavior policy by averaging the descriptor over multiple independently sampled trajectories. More formally, the estimated descriptor is calculated as:
1
desc(m (T))
desc(Ts),
where Ti...., Tn trajectories sampled from the policy
In our main experiments, we used n = rollouts for all methods to estimate given policy's BD. A descriptor function is considered "stable" if the variance of the descriptor output is low, meaning
30

---

Published as a conference paper at ICLR 2026
Walker2d
Descriptor Vañaane (Log Scale)
100
10"4
10
10-4
Hopper
Bipedal_ wa lker
Halfcheetah
Descriptor Variance (Log Stale]
109
RegularQD
RegularQD
AuteQp
RegularQD
Figure 11: Distribution of variances of BDs assigned to 400 policies by different methods' descriptor function.
that different trajectories generated by the same policy are mapped to roughly the same behavior vector. This stability is desirable as it allows for accurate policy behavior estimate using only few sampled trajectories.
To investigate and compare the stability of the descriptor function learned by each method, we randomly sampled up to 100 policies from the final archives generated by AutoQD, RegularQD, Aurora, and LSTM-Aurora. We then collected 32 rollouts for each of these 400 policies. The trajectories were subsequently encoded as descriptors using the trained descriptor function of the four aforementioned algorithms. (For RegularQD, the descriptor function is hand-crafted rather than learned.) Since different BDs may inherently have different scales, we scaled the generated descriptors for each method to ensure they all lie within the range [0, 1]. Finally, we computed the variance (across the 32 rollouts) of each policy's 's embedding and plotted the distribution of these 400 variances.
Figure II illustrates these distributions across all six domains. The results demonstrate that Au- toQD's s descriptors are typically highly stable, achieving variances that are often orders of magnitude lower than those of the other methods in most environments. This finding suggests that AutoQD can reliably and with low variance encode given policy's 's behavior based on just few rollouts.
To further validate this point, we conducted an ablation study in the Bipedal Walker environment. We compared the baseline version of AutoQD (which estimates descriptors using rollouts) against two variants that use fewer (n 2) or more (n 10) rollouts for the estimate. As shown by the results Table increasing the number of rollouts to 10 yields only a marginal increase in performance, and the overall differences between these variants are relatively minor. Therefore, based the experiments in this section, we can conclude that AutoQD's embeddings are robust, allowing for an accurate BD estimate using small number of trajectories. This inherent stability enables AutoQD to maintain good performance even in constrained settings where as few as two trajectories are used estimate the BDs.
31

---

Published as a conference paper at ICLR 2026
Table Comparison of AutoQD's performance when it uses different number of rollouts to estimate the behavior descriptors of policies. Reported values are mean + standard error over evaluations with three different random seeds.
Method
GT QD Score (x10*)
Mean Objective
Vendi Score
AutoQD (n = 2)
5.99 0.52
-27.36 + 2.91
10.42 + 0.61
AutoQD (n
6.09 0.22
-33.18 + 7.66
12.17 = 0.52
AutoQD (n = 10)
6.12 + 0.39
-22.21 + 3.18
12.46 + 1.01
J QUALITATIVE ANALYSIS OF BEHAVIORS
In this section, we provide visual examples of the diverse behaviors discovered by AutoQD across different environments.
Since the learned behavior descriptors (BDs) generated by AutoQD do not necessarily align with simple, pre-defined behavioral variations (e.g.. they may capture complex combinations of varia- tions), we adopted the following procedure to select a representative sample of the variety present in the archive. First, filtered the top 20% of best-performing policies from the final archive. We then started by selecting the single best- best-performing policy and iteratively selected the policy that was farthest (in AutoQD's learned behavior space) from the current of selected policies, adding it to our selection pool. Using this procedure, we selected 10 diverse and high-performing policies from AutoQD's archive in each environment. We then collected rollouts from these policies and manually inspected them to choose those that exhibited the most noticeable differences.
Figures 12 and 13 showease five distinet behaviors found in the Swimmer and BipedalWalker en- vironments, respectively. Similarly, Figures [4 and 15 each showcase two behaviors from the HalfCheetah and Walker2d environments. A brief description of these behavioral variations is pro- vided below.
In the Swimmer environment, we observed multiple distinct locomotion strategies, including: (1) Smooth forward movement using all joints in an S-shaped pattern.
(2) Bending upwards to create U-shaped pattern followed by burst of upward motion. (3) Bending downwards to create n- shaped pattern followed by burst of forward motion. (4) Maintaining the body relatively straight while moving forward downward trajectory. (5) Moving forward by sharply bending the head and tail of the swimmer robot. Each row of Figure 12 depicts 8 sequential frames illustrating these five types of behavior.
From the WilkdalWalker environment, we also selected five distinct behaviors: (1) Regular forward motion characterized by smooth hopping. (2) Extending the front leg to take large steps forward. (3) Sitting primarily the back leg and crawling forward via small movements of the front leg. (4) Sitting on the back leg and moving forward through sharper, more aggressive movements of the front leg. (5) Standing tall and moving forward with minimal bending of the legs. These five behaviors are depicted sequentially in Figure 13
We also include behaviors from the two environments where AutoQD did not achieve the highest performance. In HalfCheetah, we showcase two behaviors where: (1) the cheetah moves forward by taking large steps using the tip of its front leg, and (2) the head of the cheetah tilts slightly down- wards such that a sizable part of the front leg makes contact with the ground during stepping. These two behaviors are presented in Figure [4] Overall, in this environment, the primary behavioral vari- ation observed revolves around the frequency and speed of joint movements, rather than significant variations in posture.
Lastly, Figure [5 [5suuee[5sssocaaaaa showcases two types of behaviors from Walker2d: (1) the robot keeps its legs close to one another and moves forward by "tip-toeing", and (2) the robot opens its legs widely and moves forward primarily through movements of the bottom joints. As noted in the paper, most policies in this environment rely heavily on movements from the bottom joints. While we do observe noticeable differences in posture (such as having the legs open or closed, shown in the two examples). none of the policies appear to have learned to walk forward by taking large steps that coordinate the upper and bottom joints.
32

---

Published as a conference paper at ICLR 2026
(1)
(2)
(3)
(4)
(5)
Figure 12: Demonstrating five different behaviors found by AutoQD in the Swimmer environment.
(1)
A
(2)
(3)
(4)
(5)
Figure 13: Demonstrating five different behaviors found by AutoQD in the BipedalWalker environ- ment.
33

---

Published as a conference paper at ICLR 2026
(1)
(2)
Figure 14: Demonstrating two types of behaviors found by AutoQD in the HalfCheetah environ- ment.
(L)
(2)
Figure 15: Demonstrating two types of behaviors found by AutoQD in the Walker2d environment.
K
STATEMENT GENERATIVE AI USAGE
Generative Al tools were used as an aid to improve clarity and style in the writing of this paper.
34


Reviews:
[{"review_id": "1IswWKfQ9o", "reviewer": "Reviewer_Xeky", "summary": "The paper introduces AutoQD, a method that automatically discovers behavioral descriptors for Quality-Diversity optimization. It proposes embedding a policy's unique state-action occupancy measure using Random Fourier Features, such that the L2 distance between embeddings approximates the Maximum Mean Discrepancy (MMD). A Calibrated Weighted PCA is then used to project these embeddings into a low-dimensional descriptor space, prioritizing variations among high-performing policies. This enables standard QD algorithms like CMA-MAE to discover diverse and effective behaviors without requiring hand-crafted, domain-specific features.", "questions": "See the Weaknesses points.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is theoretically sound, rigorously connecting occupancy measures, MMD, and Random Fourier Features to create a principled and efficient metric for behavioral distance.\n\n\n- The proposed policy embedding method is a versatile contribution with significant potential beyond QD. This technique for representing policy behavior could be applied to other RL tasks, making it a valuable tool for the broader community.\n\n\n- The experiments are extensive and includes a diverse set of environments even thought they are all low dimensional in the state space observations and clearly show that AutoQD is able to outperform sota algorithms in terms of GT QD Score.", "weaknesses": "- There exist a gap between the theoretical guarantees and the practical implementation of the policy embedding. Theorem 1 provides a powerful result for embeddings ($\\phi^\\pi$) constructed from i.i.d. samples drawn from the occupancy measure. However, the paper acknowledges that this sampling strategy is too inefficient for practical use. Instead, the algorithm uses a different estimator ($ \\psi^\\pi$ from Eq. 6) that averages features over all transitions in a trajectory. The paper lacks a formal analysis of how this practical estimator affects the approximation quality, leaving a gap in the theoretical justification of the implemented algorithm.\n\n- The empirical validation, while strong, is exclusively focused on environments with low-dimensional, vector-based state spaces. The scalability of the RFF-based embedding to high-dimensional observations, such as images, remains unproven. Such environments could pose significant challenges, as the curse of dimensionality might render the MMD approximation less effective or require an impractically large embedding dimension (D).\n\n- While the proposed method is interesting, it seems to completely disregard temporal information about the order of events. Two policies that produce the same state-action occupancy distribution but follow a different temporal order could lead to very different decision-making behaviors. Therefore, they should be accurately identified as distinct (or diverse) policies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5dStpCBoKK", "reviewer": "Reviewer_aLN2", "summary": "The paper introduces AutoQD, an algorithm for discovering diverse, high-performing policies without hand-crafted behavioral descriptors. The method combines a Quality-Diversity (QD) optimizer with an automated way to learn behavioral descriptors of policies.\n\nAutoQD represents each policy through its occupancy measure (the distribution over state-action pairs the policy visits). To compare the behavior of two different policies, one can measure the distance between their occupancy measures using the Maximum Mean Discrepancy (MMD). Computing MMD directly is costly, so the authors approximate it using random Fourier features, which transform each occupancy measure (and hence each policy) into a finite-dimensional behavioral embedding. In this embedding space, Euclidean distances approximate the true MMD distances between policies (Theorem 1). Because QD optimizers operate over low-dimensional behavior spaces, AutoQD applies Calibrated Weighted PCA to these embeddings (weighted by their performance) to project them into compact behavioral descriptors. These descriptors and their performances are then passed into a standard QD algorithm (CMA-MAE) to discover a diverse set of high-performing policies (see Figure 1 for an illustration of the entire pipeline).", "questions": "[General]\n\n- The geometry induced by MMD depends on the chosen kernel and might not reflect meaningful behavioral similarity. Do you think other more \"structrure-aware\" distances (e.g. Wasserstein/Energy) could be used instead and capture the geometry of the space better (or be more interpretable)? (I imagine that the compatibility of MMD and RFFs makes it hard to just switch).\n\n- In the introduction, the authors state: \"Crucially, there exists a one-to-one correspondence between policies and their occupancy measures.\"\nThis statement is not always true, as you need conditions like full observability and sufficient support over state-action pairs. I recommend rephrasing this to acknowledge the required assumptions (e.g., \"under standard assumptions in fully observable MDPs\") to avoid overclaiming and be precise about the problem scope. \n\n- In \"Future Work\", the authors mention integrating AutoQD with gradient-based QD methods, which implicitly suggests that such integration is non-trivial. It would be helpful to clarify why gradient-based QD methods might be harder to use with AutoQD. \nOne potential challenge is that cwPCA continually updates the affine mapping (A, b) used to compute behavioral descriptors. In a gradient-based setting, this would make the descriptor space a moving target, potentially destabilizing learning. \n\n- Scope clarification: The abstract states that the learned embeddings \"are then used as behavioral descriptors for off-the-shelf QD methods,\" which may overstate generality. The authors should check the wording both in the abstract and conclusion to specify that they focus on CMA-MAE methods and not imply that this is already compatible with the \"standard QD algorithms\".\n\n\n\n[Experiments]\n\n- Visualization of behavior space (what diversity looks like): Can you somehow visualize trajectories to show how policies differ across the learned descriptor space (to better connect the metrics of 4.3 to intuitive diveristy). Additionally, can you visualize what behavioral dimensions cwPCA captures? \n\n- Robustness and usefulness of diversity: Can the authors evaluate the diversity's benefit by testing population-level robustness? For example, measure how many policies in the archive (or the top-K) remain \"successful\" (or perform well) under dynamics changes (friction, mass). It would also help to report mean or quantile returns under these perturbations in Section 4.4, rather than only the best policy's performance.\n\n- Can you include experiments against gradient-based QD methods?\n\n\n[Theorem]\n\n- Is there any heuristic to determine when the number of trajectories or random features is sufficient for the MMD embedding to approximate occupancy-measure distances reliably? + In practice, how does QD performance vary with the number of random features or trajectories? (Ablation in Appendix E partially answers the second part of this).\n\n\nMy initial score reflects that I'm unsure whether this paper is ready for ICLR. This is a good piece of work, and I'm keen to increase my score if my concerns are addressed.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "[Originality] Proposes a new theoretically motivated connection between occupancy measures and QD behavior descriptors, moving beyond handwritten heuristics.\n\n[Quality] Method is modular and seems to be somewhat compatible with different QD optimization methods (authors only test CMA-MAE QD methods, and I have a question regarding gradient-based methods)\n\n[Quality] Implementation details are well-documented in Appendix D. The code is well structured (I haven't run it, but I have read some of it and was able to find the main parts of the algorithm).\n\n[Clarity] Introduction and motivation are clear and well-written.\n\n[Clarity] Figure 1 and Algorithm 1 make the method easy to understand. I would add some \"labels\" to the different components of Figure 1 (e.g. QD archive, behavioral embeddings, etc).\n\n[Significance] Addresses a limitation of QD methods - the need for hand-crafted behavior descriptors in a principled way.", "weaknesses": "[Method]\n\n- Computing accurate embeddings requires many trajectories, which might be very sample inefficient in long-horizon or high-variance tasks.\n\n- The kernel bandwidth and embedding dimension are fixed globally, but they could have a significant influence on performance. \n\n[Experiments]\n\n\n- Narrow experimental scope.\n\nThe experiments are mainly against QD methods that also use CMA-MAE (The experiments also include another evolutionary method (DvD-ES) and an RL method (SMERL) which are not very competitive in the tested domains). To clarify, there are no gradient-based QD baselines (e.g. DQD, PGA-MAP-Elites, PPGA). If AutoQD should be seen as an improvement to other CMA-MAE QD methods, then the paper should frame AutoQD as a CMA-MAE enhancement. Otherwise, the paper needs more broad experimental validation to support a claim of superiority.\n\n- Lack of failure analysis. (Minor)\n\nThe authors briefly acknowledge underperformance of AutoQD on Walker2d and HalfCheetah (section 4.3), but don't have any deep analysis or visual evidence. Exploring failure cases more systematically would strengthen the paper, e.g., by visualizing the learned behaviors or the structure of the descriptor space to provide intuition about what AutoQD fails to capture. Generally, it's unclear what the gained diversity looks like (see later point).\n\n- Misalignment between motivation and evaluation.\n   \nThe introduction motivates quality-diversity through robustness and open-ended learning (i.e., diverse policies should help under dynamic shifts or enable new capabilities).\n\n \"First, diverse policies provide robustness against changing conditions$\\textendash{}$when one policy fails, alternatives with different behavioral characteristics might succeed. Second, diversity is crucial for open-ended learning, where the goal extends beyond solving predefined problems to continually discovering novel capabilities and behaviors\".\n\nHowever, the experiments evaluate only internal QD metrics (QD Score, VS, and qVS), which capture diversity (mostly) and performance within the same environment. Even with experiment 4.4, the authors only examine the best policy. These policies weren't trained to be individually robust, so I'm not sure how meaningful/interesting of a comparison that is compared to looking at population-level robustness (specific suggestions in the Questions section, e.g. percentage of top K policies that achieve a certain reward under parameter shifts). The next point covers weaknesses of this experimental setup.\n\n\n- Overreliance on internal diversity metrics and lack of qualitative analysis.\n\nThe evaluation relies heavily on intrinsic metrics. VS and QD score capture internal notions of diversity but provide limited insight into behavioral / functional variety. qVS multiplies VS by the average return, yielding a crude diversity-quality product that can be misleading: a method with greater qVS may still have lower-quality policies overall. For instance, in Table 1, although AutoQD dominates the Ant environment according to these metrics, the qVS/VS ratio suggests that RegularQD achieves higher average policy performance (i.e., better quality but lower diversity). This raises the question of whether seeing only numerical gains on such metrics truly reflects meaningful improvements.\n\nMore broadly, the experiments don't convincingly demonstrate that AutoQD's learned diversity is \"useful\" (e.g., enhancing robustness or functional adaptability). The only robustness test (section 4.4) examines the best policy under environmental shifts rather than the population-level adaptability that Quality-Diversity methods are designed to provide. Visualizations of the learned descriptor space or representative behaviors would also strengthen the paper by clarifying whether the discovered diversity is interpretable and/or behaviorally meaningful.\n\n\n[Theorem 1]\n\n- Mismatch between theory and implementation.\n   \nTheorem 1 establishes a probabilistic bound under the assumption of independent samples drawn from the occupancy measure. However, the practical estimator in Eq. (6) uses correlated samples collected along trajectories, which violates this assumption. As a result, the theoretical guarantee does not strictly apply to the estimator used in AutoQD. The theorem thus serves primarily as \"theoretical motivation\" rather than a formal guarantee for the implemented method.\n\nAs far as I understand, under mild ergodicity assumptions, Eq. (4) holds in the limit, but the rate of convergence in Theorem 1 would be a lot slower. Part of the reason why this is important is that breaking the i.i.d. assumption is essential to make the method practically usable, it wasn't done just for convenience in the experiments. I would suggest making this clear in the main text of the paper.\n\n\n\n- Practical relevance of Theorem 1\n\nThere is no guidance on how many trajectories/features are needed for good approximation. The experiments primarily use D = 100, which is likely too small for the theoretical regime where the bound would hold anyway.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a0ys0pZUJ2", "reviewer": "Reviewer_cGSg", "summary": "This paper addresses the issue that Quality-Diversity (QD) optimization algorithms rely on hand-crafted Behavioral Descriptors (BDs) which limit exploration diversity, and proposes the AutoQD method. AutoQD leverages the one-to-one correspondence between policies and occupancy measures in MDPs, uses Random Fourier Features to approximate the Maximum Mean Discrepancy between occupancy measures for policy embedding, reduces high-dimensional embeddings to low-dimensional BDs via Calibrated Weighted PCA (cwPCA), and combines with the QD algorithm CMA-MAE to alternate between policy optimization and BD update. Theoretically, it proves that embeddings converge to the true MMD as the number of trajectories and embedding dimensions increase. Experimentally, on six continuous control tasks from the Gym library, AutoQD outperforms five baselines in most tasks in terms of GT QD Score, diversity, and adaptability to environmental changes (friction/mass variations), enabling the automatic discovery of diverse and high-performance policies without requiring domain knowledge.", "questions": "1. How to avoid missing complex behaviors beyond increasing BD dimensions?\n\n2. How to control computational complexity in high-dimensional observation tasks (e.g., images in Atari)?\n\n3. What’s the path to combine with gradient-based QD for better sample efficiency?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to follow.\n\n2. Solid theoretical support; Eliminates reliance on hand-crafted BDs; Compatible with existing QD algorithms and verified on mainstream tasks.", "weaknesses": "1. Low sample efficiency, needing many trajectories in stochastic environments.\n\n2. Low-dimensional BDs may miss complex behaviors (e.g., ignoring leg-lifting in Walker2d).\n\n3. Inferior maximum fitness compared to RL methods.\n\n4. Fixed kernel bandwidth; poor scalability with large policy networks.\n\n5. (Minor but Worth Discussing) A minor yet notable point worth discussing is the subjectivity of the concept of \"diversity\" itself. Diversity in QD optimization is inherently user-defined, as different users may prioritize distinct dimensions of behavioral variation for the same task. For example, in robot locomotion tasks, one user might focus on gait types (walking, hopping, sliding) as the core of diversity, while another could emphasize energy efficiency or adaptability to uneven terrain. However, AutoQD’s approach, which infers behavioral diversity from occupancy measure embeddings and MMD-based distances, essentially discovers implicit diversity definitions derived from data and mathematical metrics, rather than aligning with explicit user preferences. While this avoids the need for domain knowledge in hand-crafting BDs, it also raises the risk that the \"diverse\" policies AutoQD discovers may not fully match the specific diversity needs of practical users (e.g., a user caring about \"load-carrying capacity-related behaviors\" might find AutoQD’s focus on \"trajectory smoothness\" less relevant).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PFpEA9J7VY", "reviewer": "Reviewer_sHcp", "summary": "This paper introduces AutoQD, a new method that solves a major problem in Quality-Diversity (QD) optimization: the need for hand-crafted behavioral descriptors (BDs). Instead of simple proxies, it represents a policy's entire behavior using its occupancy measure and uses Random Fourier Features to create a high-dimensional embedding of this occupancy measure. By projecting these embeddings into a low-dimensional space, AutoQD provides meaningful BDs to an \"off-the-shelf\" QD algorithm. This enables the unsupervised discovery of diverse, high-performing policies in reinforcement learning tasks without needing any predefined, domain-specific knowledge.", "questions": "- Theorem 1 is based on i.i.d. samples, whereas the practical estimator ($\\psi^{\\pi}$) uses discounted, correlated samples from trajectories. Could you elaborate on how the theoretical bounds from Theorem 1 apply to this practical estimator, especially given the non-i.i.d. nature of the data?\n- Regarding the periodic updates of the BDs, could you clarify the mechanism for managing the archive? Since updating the cwPCA projection effectively changes the coordinate system of the behavior space, how are existing policies in the archive handled? Is there a re-mapping process, and if so, how does the algorithm manage the computational cost and potential instability or 'forgetting' of previously discovered niches?\n- The paper notes that AutoQD inherits the scalability challenges of the QD optimizer, but it also seems to introduce new computational steps, like storing all high-dimensional embeddings and periodically running PCA on the full archive matrix. Could you comment on the scalability of this new step as the archive size grows, and how significant this overhead is compared to the baseline QD algorithm?\n- In the baseline comparisons, AutoQD uses a 4-dimensional BD space, while many of the RegularQD baselines use 2-dimensional descriptors. Given the paper's ablation study showing that performance scales with descriptor dimensionality, how can we disentangle the contribution of the AutoQD method itself from the contribution of simply using a higher-dimensional behavior space?\nAutoQD's Dimension: Table 5 (Line 1029 on Page 20) explicitly states that AutoQD used a \"Measures Dimension\" of 4 for the main experiments.\nBaseline's Implied Dimension: Section D.7 (Line 1070 on Page 20) describes the handcrafted descriptors for RegularQD. For robots like Walker2d, Bipedal Walker, and Hopper, it uses \"foot-contact frequencies.\" These robots have two feet, which implies a 2-dimensional descriptor (one for each foot).", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- It addresses a widely acknowledged and significant bottleneck in QD optimization: the reliance on hand-crafted, domain-specific BDs.\n- The combination of the occupancy measures concept with Random Fourier Features to efficiently approximate the Maximum Mean Discrepancy is creative.", "weaknesses": "- In complex, high-dimensional state-action spaces (like a humanoid robot), a very large number of samples might be needed to get a stable and accurate embedding, potentially making the method computationally expensive or sample-inefficient.\n- A key advantage of hand-crafted BDs is their human-interpretability (e.g., \"walking speed,\" \"jump height\"). The BDs discovered by AutoQD are principal components of high-dimensional RFF embeddings, likely mathematically abstract and lacking any obvious physical meaning. This makes it difficult for researchers to understand what new behaviors are being discovered, which is often a primary goal of using QD.\n\nAuthors acknowledge these limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)