Generating Behaviorally Diverse Policies with Latent Diffusion Models
Authors: Shashank Hegde
Sumeet Batra
K.R. Zentner
Gaurav S. Sukhatme
Venue: NeurIPS 2023 poster (2023)
Keywords: Latent Diffusion
Quality Diversity
Reinforcement Learning
Graph Neural Networks
TL;DR: A latent diffusion model that generates a behaviorally diverse set of neural network policies for robotic control

Abstract: Recent progress in Quality Diversity Reinforcement Learning (QD-RL) has enabled learning a collection of behaviorally diverse, high performing policies. However, these methods typically involve storing thousands of policies, which results in high space-complexity and poor scaling to additional behaviors. Condensing the archive into a single model while retaining the performance and coverage of the
original collection of policies has proved challenging. In this work, we propose using diffusion models to distill the archive into a single generative model over policy parameters. We show that our method achieves a compression ratio of 13x while recovering 98% of the original rewards and 89% of the original humanoid archive coverage. Further, the conditioning mechanism of diffusion models allows
for flexibly selecting and sequencing behaviors, including using language. Project website: https://sites.google.com/view/policydiffusion/home.

Generating Behaviorally Diverse Policies with Latent Diffusion Models
Shashank Hegde
Sumeet Batra University of Southern California ssbatraQusc.cm edu
University of Southern California khegdeBusc edu
K.R. Zentner University of Southern California kzentnerQusc.
Gaurav S. Sukhatme' University of Southern California
gauravOusc.eds
Abstract
Recent progress in Quality Diversity Reinforcement Learning (QD-RL) has enabled learning collection of behaviorally diverse, high performing policies. However, these methods typically involve storing thousands of policies, which results high ppace-complexity and poor scaling to additional behaviors. Condensing the archive into a single model while retaining the performance and coverage of the original collection of policies has proved challenging. In this work, propose using diffusion models to distill the archive into a single generative model over policy parameters. We show that method achieves compression ratio of 13x while recovering 98% of the original rewards and 89% of the original humanoid archive coverage. Further, the conditioning mechanism of diffusion models allows for flexibly selecting and sequencing behaviors, including using language. Project website: https://ssites http://sites.ggogle..... wwwgle.coooo.oooo..o www..///////llllliiisusss/oooooo
Introduction
Quality Diversity (QD) is an emerging field in which collections of high performing. behaviorally diverse solutions are trained. QD methods perform what often referred to as illumination or divergent search, in that they attempt to illuminate the search space rather than optimizing towards single point. QD algorithms have shown success in learning robot controllers capable of adapting to damage, solving hard exploration problems, and generating diverse scenarios in the procedural
content generation (PCG) domain [5] 7 2 The foundational method, Map Elites (21). maintains an archive of solutions where each cell in the archive corresponds to solution with score given by the task objective f. and behavior specified by measure functions mp.. which map to a low dimensional behavior space. The measure functions m.... specify which cell each solution belongs to in the dimensional archive. New solutions are evolved using evolutionary methods and inserted into the archive only if they outperform existing ones with the same behavior.
promising subclass of methods (Quality Diversity Reinforcement Learning (QD-RL) combines the optimization capabilities of RL with the illumination capabilities of QD, to find high performing and diverse solutions. In the robotics domain where the environment and the objective-mentt measure functions ff and assumed be non-differentiable, RL can be leveraged to estimate the gradients of f and/or m and provide a powerful performance improvement operator on the solutions in the
Eqqual contribution
Sukhatme holds concurrent appointments as Professor at USC and as an Amazon Scholar. This paper describes work performed USC and is not associated with Amazon.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

archive. QD-RL methods that combine QD with on on-policy and off- policy RL algorithms have shown promising results on variety of locomotion tasks and are capable of finding plethora of high-quality One of several drawbacks of existing QD-RI methods is that they must maintain a collection of hundreds, if not thousands of policies, in order to cover the behavior space, which leads to poor space-complexity and difficulty in real-world deployment. Map-Flliess-basel QD methods show poor scaling properties and suffer from the curse of dimensionality, quite literally in that as the dimensionality & of the archive increases, the number of solutions one needs to store increases exponentially. Prior methods have attempted to scale Map-Elites to higher dimensional problems by using Centroidal Voronoi Tessellations to divide the archive into a small number of evenly spaced geometric regions [3I. However, these methods require recomputing the Voronoi cells periodically, resulting worse runtime performance, and try to keep the number of niches small in order to effectively exploit local competition. In order to smoothly interpolate between behaviors of different solutions with a discrete archive, one must upsample the archive resolution (tens of) thousands of policies, often resulting in level of discretization higher than the actual occurrence of distinct behaviors, while further worsening the space and time complexity of the algorithm.
gaits [2 [8 23
An alluring idea is to distill the archive into a single, expressive model that completely covers the behavior space and maintains high performance. A single model representing the archive reduces ppace-complexity and potentially allows for smooth interpolation in the behavior space, making it easier to deploy and use in downstream tasks. Prior methods have shown promising results in distilling the archive into a single policy [8) or by learning a generative model via adversarial training over the policies in the archive [16). We wish to improve these methods by maintaining, improving, the overall performance of the original archive during the distillation phase, and scale generative models be able to represent policies parameterized by deep neural networks rather than a low dimensional ID vector of parameters.
To this end, we utilize the powerful generative and conditioning mechanisms of diffusion models to distill the archive into a single, expressive model that can generate a policy with any behavior from the behavior space mapped by the original archive. This generative process can be conditioned on the desired behavior measures, and even language descriptions of the desired behavior. Diffusion models have shown great success in computer vision in image quality and diversity [15] (6. Latent diffusion models accelerate training by compressing the image datasct into a compact, expressive latent space and training a diffusion model on this lower dimensional space 25) They proceed in two stages, first by compressing imperceptible, righ-frequency details via a learned dimensionality reduction, and then by learning the semantic details of the images via the actual diffusion model itself. Similarly, here we show that one can compress a collection of policies parameterized by deep neural networks into lower dimensional space by using variational auto encoder (VAE), and then learn the semantic or behavioral details of the policy distribution via latent diffusion. Our experiments show evidence of the manifold hypothesis or the elite hypervolume 32] that all high performing policies lie on a now-dimensione manifold. We summarize our contributions below.
1. We compress an archive of policies parameterized by deep neural networks and trained via state of the art QD-RL method PPGA into a single, expressive model while maintaining performance of the policies in the original dataset.
2. We use the iterative conditioning mechanism of diffusion models to reconstruct policies with precise locations in measure space, and demonstrate how language conditioning can be used to flexibly generate policies with different behaviors.
3. We showcase our model's 's ability to sequentially compose completely different behaviors together, and additionally show that language conditioning can be used to dramatically improve the performance and consistency of sequential behavior composition.
2 Related Work
Quality Diversity QD Optimization attempts to illuminate the search space with high performing solutions. The optimization problem formulated as follows. Given an objective f(-) to maximize and k measure functions m =< my (-).)...m+++) > that map solution 6; to low dimensional behavior space, the QD problem is to find the highest performing solution 0, for every value of m. Since m is continuous variable, estimating a good solution for every point in behavior space requires infinite memory and intractable. The QD problem usually relaxed by discretizing m into finite number of cells M. represented as a k-dimensional archive A. The optimization

---

problem then becomes max Σ f(O,), where 6; is solution whose measures m m(f;) fall into cell i. Differentiable Quality Diversity (DQD) [9] considers the problem where the objective and measure functions are differentiable, which provides gradients V v(()) and Vm(-). Quality Diversity Reinforcement Learning (QD-RL) considers subelass of problems that can be framed as sequential decision making tasks with exploitable Markov Decision Process (MDP) structure. Instead of optimizing for a single optimal policy, the goal is to find a collection of high performing policies that are diverse with respect to embedding functions m that encode behavior information in low- dimensional space. QD-RL algorithms vary in implementation and leverage recent works in both on-policy and off-policy RL [22 on which we build here, is a state of the art QD-RL method that combines on on-policy reinforcement learning with DQD. It maintains a current search policy corresponding to some policy тe, in the archive. The objective and measure gradients / and m are estimated for this policy and used to branch off solutions into nearby cells. The information on which branched policies most improved the archive, where policies that land in new cells are favored, is used to derive a first-order gradient estimate of maximum archive improvement. On-policy RL is used to walk the search policy towards those promising new regions of the archive that have yet to be explored. PPGA has produced state of the art results on challenging locomotion tasks. A particularly nice property of this method is that the first-order approximation of the gradient w.r.t. archive improvement improves with higher archive resolution. Since training diffusion models require large datasets, upsampling the archive resolution in PPGA generally results in better performance and allows us to produce more data for the diffusion model.
30 29 23]. Proximal Policy Gradient Arborescence (PPGA) M.
Archive Distillation Archive distillation is the process by which collection of solutions is distilled into a single model. This is particularly useful in the QD-RL domain, since having a single policy with full coverage of the behavior space, the ability to interpolate between points in the behavior space, and compose different behaviors to produce new behaviors, makes the model more versatile, memory efficient, and easily deployable in downstream tasks. Prior works predict that form of the manifold hypothesis (Elite Hypervolume), exists because policies that map to the same low- dimensional behavioral space, despite occupying different niches, may share certain traits. Follow-up works attempt to either find such low-dimensional representations or illuminate them by searching over the manifold directly [11 24 Contemporary work in the QD-RL domain has shown success in archive distillation on difficult RL problems.
[8] jointly produces an archive using a state of the art QD-RL method Policy Gradient Assisted Map Elites (22) and distills the archive into
single becavior-conditioned model. [20] uses a variant of Map Elites to produce a collection of policies that perform well in uncertain environments, and distills these policies into single Decision Transformer [3]. Previous methods have also applied Generative Adversarial Networks to generate diverse collection of policies for robot ball-throwing task (16). Here, we aim to improve on generative models applied in the QD-RL domain by scaling the representational capacity of our model collections of deep neural networks, while simultaneously maintaining the performance and diversity of the original archive.
Diffusion models have become state of the art in image generation. Denoising Diffusion Probabilistic Models (DDPM) are a class of generative models that iteratively denoise noise sampled from an isotropic Gaussian. The iterative denoising process is a powerful mechanism that has been shown to produce state of the art results on computer vision benchmarks 66. Numerous methods have made improvements DDPMs that address some of the shortcomings of these models compared to other generative methods. 6 shows that classifier classifier-guidance guidance can be applied at test-time to improve the quality of the generated samples. (26) showed that, by relaxing the Markov assumption in the forward diffusion process, one can significantly improve inference-time by downsampling the number of diffusion steps while maintaining most of the sample quality. Multiple refined methods for sampling from a diffusion process have been proposed, including (27. [99 and [17] However, here we are not particularly concerned with sampling efficiency, and thus use the method proposed in 266 (25 showed that diffusion can be performed on the latent space of a pretrained Variational Autoencoder.
Diffusion
Graph Hypernetworks Hypernetwrks are models capable of estimating the weights of a sec- ondary network [12]. When conditioned on task identities, these can achieve continual learning by rehearsing task specific weight realizations (33. Graph hypernetworks were originally introduced for architecture search in image classification [18 and have been shown to be trainable with RL
to estimate variable architecture policies for locomotion and manipulation 14 and for quadrotor
control [13]
3

---

3 Method
Diftusion Process
Graph Hoper Naleork
Mi
GRU
Griph
π
πC
U-Nil
fii Latent Space
Encoder
Decoder
Parameter Space
Conditioning
Parameter Space
Policy Parameters
DI<
Encoder/Decoder
Composel
Netwarks
Representations
Decompose
Structure of model as encoder (left) and decoder (right). During encoding, policies are split into layers and encoded separately. The encodings are concatenated together and fed into final layer produce a latent representation. The conditional diffusion model samples latent code z from the latent representation. During decoding, graph hyper network jointly decodes the weights and bias parameters from 2, and the policy network architecture graph g, while normalization parameters are directly decoded from 2.
Figure 1:
Policy Compression Following [25]. we compress the archive A into lower dimensional space using a variational autoencoder (VAE). A policy consists of layers, each containing a weight matrix W; and bias vector b;. < i < I. In the encoder &, the features of each W; and b; are extracted using a convolutional neural network and fully connected network, respectively. These features concatenated together and fed into final fully connected layer, that produces intermediate latent representation Z h The decoder D contains a conditional graph hypernetwork and an observation normalizer decoder, d,,, which takes in the latent code and produces the reconstructed policy नं = D(2;). The conditional graph hypernetwork estimates the policy network's 's parameters while d,, estimates the observation normalizing mean and variance parameters. Our conditional graph hypernetwork is based on the implementation in [4]. While the original graph hypernetwork is meant estimate the parameters of variable architectures (represented graphs), freeze the input architecture graph This is set to be the architecture graph all the networks in the archive, which in our case represented {0, 128, 128, a). Here indicates the input node, the following 128's represent the hidden layer nodes and a is the output node and equals the action space dimension. Further, add latent encoder eg. and with the graph encoder e., a concatenated encoding is fed to the gated graph network (34). This mechanism lets us condition the parameter estimation on the latent Together, and D form the VAE architecture, which optimizes the objective
LVAE (T' (()s)) DEL [D((2))))I
(1)
Policy Diffusion We hypothesize that the denoising process can be used to produce high quality policies given dataset of behaviorally diverse policies parameterized by deep neural networks. We describe how the DDPM formulation can be applied to such datasets. Diffusion models consist of forward process g that iteratively applies noise e, to sample 2g from the original data distribution Fo ~ (x)). The noise e, at each timestep is sampled according to a variance schedule {3:)T=]
G = ( xT-1) = M(TT; 3,1
making the forward process Markovian. The reverse or generative process p(xT) reverts noise from an isotropic Gaussian into sample To in g(x) and contains similar Markov structure.
po(xo) p(xT) II po po 12)-) po(xe-1l2z) (( f), Eo Eo ())
Here, To is a latent code representing some policy (a)ss)) rather than the policy parameters itself. Thus, the diffusion model instead learns to capture the distribution the much lower dimensional 2, analogous to the Elite Hypervolume hypothesized in [32

---

15 makes a connection between the reverse process and Langevin Dynamics, where Po(2 (2t) is the learned gradient of the data density. When a represents neural network parameters, the diffusion model learns the score function of the optimal policy distribution, and iteratively refines the noisy policy parameters 2t towards this distribution. When conditioning the policy x to match a specific behavior m i.e. Pa (#t-1 [e; m), this gradient can be thought of as the gradient of the maximum posteriori (MAP) objective over the distribution of policies that exhibit behavior m with respect to policy parameters Zz. Thus, our diffusion formulation draws inspiration from Bayesian Methods, where p(ær, m) I Po (-+e- 22,, m) resembles iterative gradient descent from policy randomly initialized from an isotropic gaussian Dr ~ (0,, I) towards the mode of the posterior distribution over terh-performing policies with behavior m.
We first train an autoencoder according to the objective in Eq. T A random batch of policies and their observation normalizers (Ta. 11) are sampled from the archive and fed into the encoder & to produce latents z (TT, 7)). The decoder D then reconstructs the policies and their respective observation normalizers from the latents (T r1)) = (zz). To simplify training,
Training Procedure We follow the training procedure in 25
some tasks, we normalize the archive dataset by subtracting out the per-parameter mean and dividing by the per-parameter variance. This results in an autoencoder over parameter residuals relative to the original per-paramete mean and variance, which we keep for decoding. For training the latent diffusion model, we sample a batch of policies and their respective obs normalizers, measures, and text labels (TT. 17. m, y). The policies and measures first encoded into latent vectors and measure embeddings respectively z = E E(To. 1), Tem (m), where Tym is trainable encoder. These are subsequently fed into the diffusion model, where the latents are conditioned on the measure embeddings using the cross-attention mechanism. We uniformly
sample t from (1, ..., T for the batch and regress the predicted noise vectors according the latent diffusion training objective
LLDM == N0(111 [[le ca t, Tem (m)
In the case of language-conditionee diffusion, the measures replaced with text labels that are encoded using a Flan-T5-5 15-lll encoder (4),, which is fine-tuned end-to-end using the loss in Equation 4 to produce text embeddings c0(2t, t, Te, (u)) that condition the diffusion process.
Experiments
In our experiments, we wish to analyze our model's performance on the following: 1, archive com- pression while maintaining original performance, 2. measure and language conditioning to produce policies with specific behaviors, and sequential behavior composition to produce new behaviors. Since PPGA was evaluated on the Brax [10] environments Humanoid, Walker2D, Halfcheetah, and Ant, we evaluate our model on the same four environments. For each environment, the reward function encourages stable forward locomotion while minimizing energy consumption, and the obser- vation and action spaces are continuous. The measure functions are the proportion foot contact time of each leg of the robot over the entire trajectory and are thus bound to [O. 1]. where 0 indicates the foot never touched the ground, and I indicates the foot never left the ground. Policies with different measure functions correspond to different locomotion gaits. The dimensions for the (observation, action) spaces are: Humanoid (227, 17); Walker2d (17, 6): Halfchectah (18, 6); Ant (87, 8). Every policy in the archive has two hidden layers of 128 neurons each, followed by output layer matching the action space dimension. While there are recent works that perform archive distillation on these tasks 8 20 they produce very different dataset of policies using different underlying QD-RL methods. The Quality Diversity Transformer, for example, uses evolutionary strategies with explicit optimization towards policies with low-variance behaviors, whereas PPGA uses first-order gradient approximations and makes no such explicit optimization towards low behavior variance. As any comparison of distillation methods is relative the archive being distilled, we are unable to make any direct comparison to these methods.
Performance and Accuracy Experiments We evaluate our model's ability to reconstruct the performance and behavior of policies in the archive to high precision. Following (0), we first downsample our trained archives for each task into equally-spaced geometric regions using Centroidal Voronoi Tessalation Map Elites (CVT-ME) [31). Each region has policy no from
5

---

the original archive and corresponding behavior < mi.... my > for a d--imensionnal archive that lies in the center of that region. These policies' behaviors are used as conditions to produce 50 measure-conditione policies ; T0sa- Each policy is then rolled out 50 times, and the objective ,...
and measure functions (To m(Te,) ) are computed as the average over 50 episodes. These values are then used to compute the reward ratio, which is the average performance of the generated policies > f(%).) over the original ones: r 2 f(TT))
The reward ratio alone can be mislcading if all the generated policies have high performance but incorrect measures w.r.t. the measure conditions. For example, generating 50 pest-rerformin policies with the same measures despite sampling different measure conditions would lead to a large reward ratio but no diversity. Thus, also track the Jenson-Shannon (JS) Divergence between the distribution of measures produced by the generated policies and the original policies. We refer to this as the measure divergence. We report the JS divergence instead of the KL divergence because there is no clear preference between the KL divergence directions, and because some experiments produce policies with near-zero measure distribution overlap, for which the JS divergence is upper bounded by ln)2), and the KL divergence is arbitrarily large. We perform this once every 10 epochs over the course of training and present the results in Fig. 2 Humanoid, Walker2D, and Halfcheetah achieve a reward ratio of near ~ 1.0 while reaching a measure divergence of 10 Ant achieves reward ratio of ~ 0.75 with a measure divergence of ~ 0.1. We expect a higher measure divergence on Ant given that it is 4egged locomotor and thus has twice as many measures as the other environments.
Archive Reconstruction At test time, we analyze the ability of the latent diffusion model to reconstruct the entire original archive. We take the measure vector < m....., my > corresponding to cell c, and use it as condition to produce policy T.. Vc, E A. If the resolution of archive A is dh where d is the discretization level and k is the number of measures, this gives us collection of d policies. These are rolled out to compute f and m, and inserted into a new, empty archive A'' according to the standard insertion rules in QD optimization, where only the best solution for a cell i is stored when two solutions map to the same location in behavior space. The reconstructed archive
will thus have some number of unique solutions such that |A'| S d with a QD Score of E ((Tr.) ) To make an informative comparison between the reconstructed and original archives, we plot their cumulative distribution functions (CDF) (Fig- These not only encapsulate coverage and QD-score information, but also tell us how the policies are distributed with respect to the objective.
On all tasks, the policy distribution of the reconstructed archive closely matches that of the original one. On Halfcheetah, we are able to almost exactly reproduce the original archive. On all other tasks, we lose some density in the low performing regions of the archive, but consistently match policy density in the higher performing regions. Fig. 3 tells a strikingly similar story. in that our model first fills the central and often higher performing regions of the archive before moving on to the fringe regions to improve diversity. Both results seem to suggest that the diffusion model first learns common features corresponding to high performance across all policies, and then proceeds to learn aspects of individual policies that make them behaviorally unique. Table providee quantitative view of the CDF plots. We report the QD-scores and coverage on both the original and reconstructed archives for all tasks. Following (20), we report the Mean Error in Measure (MEM), which is the average L2 distance between all generated policies' and original policies' measures in archives A' and A, respectively: MEM- E [[]m(TTT))
m (1T
KL Coefficient Ablation A KL penalty (Eq. is used regularize the latent space. 25 used relatively small penalty coefficient of 10-6 to prevent information loss due to overly compacting the latent space. We wish to similarly quantify the information density of our dataset and the effects of stronger latent space regularization on VAE model performance. Fig-5) shows the VAE reward ratio and JS Divergence for larger values of the KL coefficient. Overall, our findings are in line with (25 stronger regularization results in loss of information, thus reducing our VAE's ability to reproduce policies from the original dataset. For all other experiments. we fix the KL coefficient to 10
GHN Size Ablation We examine the effect of model size on our model's ability to reproduce the archive. We chose three different values for the number of neurons in the hidden layers of the hypernetwork in the decoder and keep the diffusion model size fixed. The results are shown in Table for for the Humanoid environment. The QD Ratio indicates the QD score of the reconstructed archive over the original archive's s QD score. The compression ratio is calculated as the number of parameters in the decoder plus the number of parameters in the diffusion model, divided by the sum

---

Inmszok
1.00
1:00
1.00
1.0
0.10
2.3
0.50
100
0.400
1.5
150mmgg
0.10
0.60
10
012
100
150
110
200
150
200
0.0
30
100
200
Figure 2: Performance of our method four benchmark environments. Higher reward ratios correspond to better performing policies. Lower JS divergence corresponds to more precise measure reconstruction. See Section for description of the experimental method and limits of these performance metrics.
800
too
t003
nace
A20
4000
0.2
2000
0.2
2003
2000
800
0.55 3.50
6.75
1.00
8.00
0.5
0.50
6.75
1.00
0.00
0.25
6,50
0.75
1.00
0.00
0a5
0.50
0.75
1.00
Epoch 0
Epoch 1
Epoch 2
Epoch 3
Figure 3: Heatmaps of generated archives throughout training. We visualize the reconstructed archive generated by our model over the course of training. The axes correspond the measure values (proportion foot contact times). Each point corresponds to policy in the archive and its color represents performance (reward) for forward locomotion. Coverage and average reward increases rapidly in early epochs to approximately match the original archive. Reward ratios and JS divergence continue improve slowly with more epochs, as shown in Figure
Task
QD
Coverage Compression Reconstructed Reconstructed Coverage (%)
MEM
Score (x107)
(%)
Ratio
QD
Score
(x10²)
Humanoid
8.08
90.79
13:1
7.4 + 0.11
84.35 + 1.01
0.237 + 0.03
Walker2D
3.12
85.68
11:1
3.00 + 0.02
83.65 + 0.14
0.269 + 0.07
Halfcheetah 11.4
96.94
14:1
11.1 0.01
94.55 + 0.11
0.15 + 0.01
Ant
3.44
71.03
13:1
2.54 0.07
52.66 + 1.46 0.57 + 0.08
Table 1: Latent Diffusion QD Metrics. The QD Score and Coverage columns are calculated by
rolling out the policies in the original archive training dataset. The compression ratio is calculated as the number of parameters in the decoder plus the number parameters in the diffusion model, divided by the sum of parameters of policies in the original archive. The Reconstructed QD Score and Reconstructed Coverage are calculated by rolling out the policies that were generated by our model, conditioned on the measures from the original archive.
of parameters of all policies in the original archive. In general, we find that the MEM decreases and QD ratio increases with larger model size, the expense the compression ratio. Nonetheless, even the largest diffusion model (43.7 million parameters) achieves a compression ratio of 8 to 1, while reproducing 94% of the original archive with low measure error. In situations where model size is not a significant constraint, picking the largest model may be the best option, it nearly recovers the original archive's performance and covers all relevant parts of the behavior space covered by the original dataset.
Sequential Behavior Composition
test our model's ability to successfully compose an arbitrary chain of generated behavior policies, design an experiment where the episode, which naturally
7

---

Immacii
wolkrSi
balthhcchh
Resestected
Reseirette
Kille CET
Artiin CEV
Artiin CDP
Resocttonn hi Aor CDP Orighail krrltw CB1
240
62:00
9023
2400
4:30
9101
14 10.prti
DApectin
CAjpstion
Figure 4: CDF of policies for all tasks. For each task, the y axis represents the percentage of the archive grid that has at least the corresponding amount of returns on rollout (shown on the x axis).
Hyperparannter Seazch
1,130
Messure (18) Deveugrnee
1/"
Rewar Ratio
Coeffirien
1.075
10-1
16-2
Ic-2
1,050
Jo-]
10
Ic-1
Ic-6
1.025
10
1.000
100
203
300
300
0.0
70C
800
100
2.0
300
100
500
600
700
800
Epoch
Epoch
Figure 5: Reward Ratio and JS divergence of VAE reconstructed policy for different KL coefficients. A very large KL coefficient results our method producing policies very close to the average policy. This results in a reward ratio above one, at the cost of complete loss of precision in measure space, and a Measure (IS) Divergence near the maximal value of In(2). Smaller KL coefficients result in more precise policy reconstruction, while maintaining Reward Ratio near 1.
Decoder
Parameters
QD Ratio
Compression Ratio
MEM
GHN8
18.3M
0.77
19:1
0.267 + 0.091
GHN16
26.8M
0.87
13:1
0.242 + 0.101
GHN32
43.7M
0.94
8:1
0.129 0.005
Table 2: GHN size ablation on the humanoid archive. The Compression Ratio rounded to the nearest integer. QD Ratio and Mean Error in Measure improve with larger GHN sizes.
terminates T = 1000. is divided into four time intervals of 250 timesteps each. For each time inter- val, we randomly sample a measure condition m and use our conditional diffusion model combined with the decoder to produce
agent that exhibits behavior (0, 1). An experiment is successful if the episode terminates no earlier than t = 800, implying that model has produced all four behavior policies and successfully composed least three of them together. We consider the trajectory length for one experiment to be the average trajectory length over 50 parallel environments, and we repeat this experiment 10 times. Using measure conditioning, we found that the success rate is 48.5%, compared to the success rate of 51 % when choosing the policies with the appropriate measures directly from the archive dataset, closely matching the performance of our original archive. However, a surprising result emerges when using a "pruning strategy" with our model, where we essentially prune out policies with text labels that suggest they are low performing. For example, when sampling sequences of text labels, we filter out those that contain the word "fall", which indicates policy that falls over mid-trajectory. This allows to select better policies that don't 't fall over on their own, which increases the success rate of sequences to 59%. Finally, we sample sequences using only text labels that contain the term "quickly," which indicates policies that move forward quickly. This raises the success rate of sequencing four different behaviors to 79%, more than double the success rate of sampling text labels uniformly. An example of sequential behavior composition with our language-conditiones model, with corresponding frames demonstrating the motions, is shown in Figure 7 This pruning strategy is capable only with language and the emergent higher success rate demonstrates the importance of language-conditiong generative
TT(aas m) D(ea (2r, T, m)), 2T
language-condition
8

---

models and their capabilities for downstream tasks. This approach is similar to the use of "quality tags," used some diffusion models.
Compute Resources Each VAE and diffusion experiment was run on a SLURM cluster where each job was allocated 6 cores of Intel(R) (eon(R) Gold 6154 3.00GHz CPU, a NVIDIA GeForce RTX 2080 Ti GPU, and 108 GB of RAM. Each VAE experiment took about 16h; each measure conditioned diffusion model took about 3h train while the language conditioned diffusion runs took 4h.
1.9
Measure
foot
0.5
0.5
1.0
Time
Right foot
0.5
Mcasure
0.5
0.0
400
600
800
0.0
Time
Shutil tarears hop brwends [nigg iiht
Reconstructed
Desired
on lot. your right foct
Torwards os if
Fun lorward
right foot
gin fook
Figure 6: Visualization of the measure values resulting from sequence of four randomly selected desired measures (left) and text labels (right). The measure values from policy sequence (Section are shown on the left as function of time. These were run 10 times and the corresponding error plots are shown. The close match in measure values between the Desired values, which are used for conditioning, and the Reconstructed values show the effectiveness of our conditioning. On the right, the measure values from policy sequence as described in Section of time. The text labels used to produce the policy sequence are shown the bottom of the figure. Large changes in measure values show that text conditioning is able to produce and sequence highly diverse policies. The desired behavior both cases changed four times throughout the episode.
4 are shown as a function
Limitations
Scaling with measure dimension warrants further investigation (e.g., ant, high MEM). Further experimentation with diffusion hyperparameters might produce better policy reconstructions. currently limited and can be expanded to include much larger
The language descriptions corpus of descriptions. Language can occasionally under-describe the desired behavior, and therefore some descriptions can lead undesirable generated policies. For example, Figure shows higher variance in the reconstructed measures when conditioned on language. Finally, training the diffusion model requires us first construct the policy archive by using QD algorithm. An interesting
2003
WHILE KIOKING WITH FOOT
QUICKLY SHUFTLE FORWARD OR YOUR LEPT POOT
BUY HOP FORMARD ON LEFTT POOT WHILE RUSHIT FOOT
Figure 7: Temporal behavior sequencing from text labels. A humanoid (left) controlled by policy sequence beginning with "slide forward on your right foot while kicking with your left foot" (top left), then "run forward left foot while dragging right foot". then "quickly shuffle forward your left foot", and finally "wildly hop forward on left foot while lifting your right foot up" (bottom right). Heatmap of the archive (right) showing the sequence of text labels overlaid the measure space.

---

research direction would be to train this model directly, bypassing the archive construction step entirely.
6
Conclusion
We proposed method that uses diffusion models to distill the archive into a single generative model over policy parameters, achieving compression ratio of 13x while recovering 98% of the original reward and maintaining 89% of the original coverage. Our models can flexibly generate specific behaviors using measures text, and these behaviors can be sequenced together with surprising consistency. A pleasantly surprising find was additional evidence supporting the Elite Hypervolume hypothesis proposed in [22] From our training results and heatmap evolution over time, that the diffusion model first learns the general structure of what comprises a "good" policy across to different areas of behavior space, implying learning of what makes each policy behaviorally unique. As a potential future work, we can use equivariant layers such as those used in (35) instead of convolutional layers used encode the weights and biases of policy networks. Finally, we look forward to exploring the connections this work has to other subfields, such Meta-RL RL and Bayesian Learning
behavior space, and then proceeds to branch
References
[1] Sumeet Batra, Bryon Tjanaka, Matthew C Fontaine, Aleksei Petrenko, Stefanos Nikolaidis, and Gaurav Sukhatme. Proximal policy gradient arborescence for quality diversity reinforcement learning, arXiv preprint arXiv:2305.13799 2023.
[2] Varun Bhatt, Bryon Tjanaka, Matthew C. Fontaine, and Stefanos Nikolaidis. Deep surrogate assisted generation of environments. CoRR, abs/2206.04 04199, 2022.
[3] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems, 2021.
[4] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi. Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetune
language models, 2022.
[5] Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean- Baan-BBptsttee Mouret. Robots that can adapt like animals. Nat., 521(7553):503-507 2015.
[6] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans image syn- thesis. In Marc Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 8780-8794 8894,,94477 2021.
[7] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. Go-explore: a new approach for hard-exploration problems. CoRR, abs/1991.11099 2019.
[8] Maxence Faldor, Félix Chalumeau, Manon Flageat, and Antoine Cully. Map-elites with descriptor-conditione gradients and archive distillation into a single policy. CoRR, Abs/2303.00833. 2023.
[9] Matthew Fontaine and Stefanos Nikolaidis. Differentiable quality diversity. Advances in Neural Information Processing Systems, 34: 1404000000002, 2021.
[10] C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax differentiable physics engine for large scale rigid body simulation, 2021.
[11] Adam Gaier, Alexander Asteroth, and Jean- Can-Baptiste Mouret. Discovering representations for black-box optimization. Proceedings of the 2020 Genetic and Evolutionary Computation Conference, 2020.
10

---

[12] David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetworks. CoRR, htss1609.00906 2016.
[13] Shashank Hegde, Zhehui Huang, and Gaurav S Sukhatme. Hyperppo: A scalable method for finding small policies for robotic control. arXiv preprint arXiv:2309.16663 2023.
[14] Shashank Hegde and Gauray S Sukhatme. Efficiently learning small policies for locomotion and manipulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5909-5915. IEEE, 2023.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[16] Marija Jegorova, Stéphane Doncieux, and Timothy M. Hospedales. Generative adversarial policy networks for behavioural repertoire. CoRR, abs/ 10///1111 11881..000 1.02945, 2018.
[17] Tero Karras, Miika Aittala, Timo Aila, and Samuli I Laine. Elucidating the design space of diffusion-based generative models. ArXiv, abs/2206. 00052200.000644, 2022.
[18] Boris Knyazev, Michal Drozdzal, Graham W Taylor, and Adriana Romero. Parameter prediction for unseen deep architectures. In Advances in Neural Information Processing Systems, 2021.
[19] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. ArXiv, abs/2206.00927, 2022.
[20] Valentin Mac'e, Raphael Boige, Félix Chalumeau, Thomas Pierrot, Guillaume Richard, and Nicolas Perrin- Gilbert.. The quality- transformer: Generating behavior-conditioned trajectories with decision transformers. ArXiv, abs/2303. 16207, 2023.
[21] Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. CoRR, abs/1504.04909 2015.
[22] Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Francisco Chicano and Krzysztof Krawiec, editors, GECCO '1:: Genetic and Evolutionary Computation Conference, Lille, France, July 10-14, 2021, pages 866-875. ACM, 2021.
[23] Thomas Pierrot, Valentin Macé, Félix Chalumeau, Arthur Flajolet, Geoffrey Cideron, Karim Beguir, Antoine Cully, Olivier Sigaud, and Nicolas Perrin-Gilbert.. Diversity policy gradient optimization. In Jonathan F. Fieldsend and Markus
for sample efficient Wagner, editors, GECCO 22: Genetic and Evolutionary Computation Conference, Boston, Massachusetts, USA, July 9 - 13, 2022, pages 1075-1083, ACM, 2022.
[24] Nemanja Rakicevic, Antoine Cully, and Petar Kormushev. Policy manifold search: exploring the manifold hypothesis for biversity-based neuroevolution. Proceedings of the Genetic and Evolutionary Computation Conference, 2021.
[25] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 10674-10685. IEEE, 2022.
[26] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[27] Yang Song, Jascha Narain Sohl-Dickstein, Dicderik P. Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ArXiv,
2020.
[28) Bryon Tjanaka, Matthew C. Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradients for differentiable quality diversity in reinforcement learning. In Jonathan E. Fieldsend and Markus Wagner, editors, GECCO '22: Genetic and Evolutionary Computation Conference, Boston, Massachusetts, USA, July 9 13, 2022, pages 1102-1111. ACM, 2022.

---

[29] Bryon Tjanaka, Matthew C Fontaine, Julian Togelius, and Stefanos Nikolaidis. Approximating gradients for differentiable quality diversity in reinforcement learning. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1102-1111, 2002...
[30] Bryon Tjanaka, Matthew Christopher Fontaine, Aniruddha Kalkar, and Stefanos Nikolaidis. Scaling covariance matrix adaptation map-annealing to nigh-dimensional controllers. Deep Reinforcement Learning Workshop NeurlPS 2022.
[31] Vassilis Vassiliades,, Konstantinos I. Chatzilygeroudis, and Jean- Saan-Baptiste Mouret. Using cen- troidal voronoi tessellations to scale up the multidimensional archive of phenotypic elites algorithm. IEEE Trans. Evol. Comput., 22(():623-634) 2018.
[32] Vassilis Vassiliades and Jean-Baptiste Mouret. Discovering the elite hypervolume by leveraging interspecies correlation. Proceedings of the Genetic and Evolutionary Computation Conference, 2018.
[33] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin Grewe. Continual learning with hypernetworks. CoRR, 190990066000, 2019.
[34] Chris Zhang. Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
[35] Allan Zhou, Kaien Yang, Kaylee Burns, Yiding Jiang, Samuel Sokota, Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. arXiv preprint 140.2440..1040 2023.
12

---

Appendix A Model details and Hyper parameters
A.1
Variational Autoencoder
GHNx refers to diffusion model where the decoder (the GHN) has hidden layer with number of neurons. The main paper shows results across tasks with a GHN16 model. We normalize the archive
name
value
dimension
64
Encoder hidden dimension
64
Obs Normalizer encoder hidden dimension
64
KL coefficient
le-6
Gradient clipping
True
Learning rate
le-4
Training batch size
32
GHN hidden layer size
16
Table 3: Hyperparameters used to train the VAE
dataset by subtracting the per parameter mean and dividing by the per-parameter standard deviation for each policy the archive. This is equivalent to calculating "mean-policy" with parameters o, and subtracting it from each policy in the archive. The task for the VAE reduces to learning the parameter residuals. We enable archive-normalization on Humanoid and Ant where we saw the greatest improvement in performance and training stability, and leave disabled on Halfcheetah and Walker2d, where we observed negligent improvements.
A.2 Latent Diffusion Model
We utilize a UNet backbone as the architecture for latent diffusion model. We use a single ResNet block downsample the inputs into condensed embedding space. A spatial transformer used to perform cross-attention with condition-embeeddnngs in the embedding space. Finally, a single ResNet block used upsample the embeddings the same dimensionality as the inputs. A single ResNet block consists of two convolutional layers plus an embedding layer for sinusoidal position embeddings. The encoder and decoder used for latent diffusion remain the same as described above.
name
value
No. Resnet Blocks in U-Net
U-Net activation
SiLU
Transformer heads middle part of U-Net
Gradient clipping
True
Learning rate
1e-4
Training batch size
32
Table Hyperparameters used train the Latent Diffusion Model
Appendix B
Ablation of GHN size for the Ant archive
From the below table, we see that the performance of our method scales with the GHN size for the environment. For larger GHN sizes, the MEM reduces while the QD ratio increases.
Appendix C Rollout Videos
The entire list rollout videos for the results shown in the paper can found the project site.
13

---

Decoder
Parameters
QD
Compression
Reconstructed Coverage (%)
MEM
Ratio
Ratio
GHN8
12.0M
0.55
15:1
38.86 + 0.02
0.582 + 0.023
GHNI6
14.1M
0.74
13:1
52.66 + 1.46
0.571 + 0.081
GHN32
18.5M
0.85
10:1
58.72 1: 0.02
0.436 + 0.084
Table GHN size ablation on the ant archive. The Compression Ratio rounded to nearest integer. QD Ratio and Mean Error in Measure improve with larger GHN sizes.
Measure conditioned rollouts:
Language conditioned rollouts:
Behavior composition rollouts: httpp//wwww.oooooooooooooooooooa
Appendix D Heatmaps for all tasks
The following are the heatmaps for the original archive generated by PPGA and the reconstructed archive by the LDM model. Since the measure space for the Ant environment is 4 dimensional, it cannot be visualized with our current tools. Below are the plots for Halfcheetah, Walker2d and Humanoid. These heatmaps are obtained at the end of the 200th epoch of training.
Task
Original Archive
Reconstructed Archive
HalfCheetah
Walker2d
Humanoid
14


Reviews:
[{"review_id": "53NJ4229r3", "reviewer": "Reviewer_Usyj", "summary": "This work presents a novel framework using latent diffusion models to distill the archive of polices into a single generative model over policy parameters. The latent diffusion model with VAE backbone compresses the high dimensional neural network (NN) parameters into a compact, making it possible to reconstruct policies parameterized by NN. Further, the conditioning mechanism of diffusion models is used to flexibly generate policies with different behaviors.", "questions": "1.\tHow many policies are used as training dataset when conducting archive reconstruction experiments?\n2.\tDo the dimensions of the measures used in each environment have explicit semantic information? If so, can the author(s) further demonstrate the model's generalization ability on it? (For instance, assuming that a particular dimension of the metrics represents the robot's movement speed, if the value of that dimension is set to 1.2 in the condition of the diffusion model, will the generated strategies be able to achieve speeds beyond those observed in the training dataset?) If not, how could the users employ this model to generate policies of their desired behaviors?\n", "limitations": "1.\tThe author(s) only demonstrated the model's ability to reconstruct the original dataset and did not conduct further experimental demonstrations regarding its generalization capability. However, the generalization ability should be an important consideration when evaluating generative models.\n2.\tAt present, it appears that text conditioning does not show a sufficiently favorable influence within the model.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe author(s) use latent diffusion model to compress the high dimensional neural network (NN) parameters into a compact, making it possible to directly generate policies parameterized by NN.\n2.\tThe proposed framework recovers 98% of the original rewards and 89% of the original coverage while achieving a compression ratio of 13x.\n", "weaknesses": "1.\tThe performance and accuracy of the proposed method show significant discrepancies when applied under text conditions. Experiments show that the success rate of the method is influenced by the selection of text labels. This suggests that the model possesses limited understanding and generation capabilities in terms of language descriptions.\n2.\tThe proposed method shows poor performance on tasks with high-dimensional measure vector, such as Ant, suggesting its limited modeling capability in high-dimensional measure spaces. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The performance and accuracy of the proposed method show significant discrepancies when applied under text conditions. Experiments show that the success rate of the method is influenced by the selection of text labels. This suggests that the model possesses limited understanding and generation capabilities in terms of language descriptions.**\n\nThank you for your insights; we too found that the language-based generation capabilities are limited compared to measure-conditioned policy generation. Our hypothesis for this discrepancy lies in the fine-granularity of measure conditioning compared to the coarse-granularity of language conditioning. In order to produce large enough training data for the diffusion model, each measure is finely discretized into 100 equally spaced bins i.e. a 2D archive for a bipedal robot contains 10k cells and thus as many as 10k policies with unique measures. In practice, however, there are much fewer actual gaits than the 10,000 “unique” policies that can exist in the archive. Thus, a single language condition describing a specific high-level gait corresponds to a region of the archive, rather than a specific policy with a specific measure, as is the case with measure conditioning. This means that there is less training data for the language encoder to work with when compared with measure conditioning. Nonetheless, language conditioning is still useful since it is more intuitive than using measures, and it’s likely the case that many real world applications will not require measure-level granularity, instead opting for gait-level granularity. Improving on language-level conditioning, however, is an interesting direction of research we intend to pursue in the future. \n\n\n**The proposed method shows poor performance on tasks with high-dimensional measure vector, such as Ant, suggesting its limited modeling capability in high-dimensional measure spaces.**\n\nWhile we agree that the performance of ant is lacking compared to the other tasks, we do not believe that the dimensionality of ant’s measure space is affecting performance since the measure space of ant (4) is not significantly higher than the other tasks (2). In addition, the ant measures are discretized into only 10 bins as opposed to the 100 bins in the 2D archives of the other tasks, making the total maximum policy count in the datasets the same for all tasks. The CDF plots in Figure 4 show that the diffusion model recovers most, if not all, of the higher performing policies, and loses diversity where lower performing policies are concerned. If the proposed method struggled with the higher dimensionality of the measure space, we would expect an equivalent drop in diversity along all levels of policy performance. This suggests that the ant archive dataset itself contains many different policies that have parameter variations that are difficult to distill but achieve low reward.\nThe MEM is also not corrected by the dimensionality of the measure space. By dividing the MEM by the measure dimensionality, we can compute a MEM per dimension of Ant that has a similar value as the other environments. Finally, an ablation on GHN size on the Ant environment and provided in the added document shows that performance can be improved with larger decoders.\n\n\n### Questions:\n\n**How many policies are used as training dataset when conducting archive reconstruction experiments?**\n7470 policies are used for humanoid, 9815 for half-cheetah, 8192 for walker2d, and 6180 for ant.\n\n**Do the dimensions of the measures used in each environment have explicit semantic information? If so, can the author(s) further demonstrate the model's generalization ability on it? (For instance, assuming that a particular dimension of the metrics represents the robot's movement speed, if the value of that dimension is set to 1.2 in the condition of the diffusion model, will the generated strategies be able to achieve speeds beyond those observed in the training dataset?) If not, how could the users employ this model to generate policies of their desired behaviors?**\n\nThe dimensions do not have explicit semantic information. In our tasks, the measures are defined as the proportion of foot contact time for each leg of the robot over the entire trajectory. This inherently bounds the measure space to [0.0, 1.0] for each measure dimension. Consequently, there is no physical interpretation of a measure value of 1.2. The goal of each policy is the common RL objective for locomotion tasks, which is to move forward as fast as possible while minimizing energy consumption and avoiding termination conditions such as falling over. Thus, we cannot modulate the speed of the generated policies since the dataset contains policies that were trained to move as fast as possible. It would, however, be possible to modulate velocity, if forward progress was added as a measure function instead of as the objective. While it would be interesting to add forward progress as a measure function and experiment with out of distribution generalization in a separate QD paper, our focus in this work was archive distillation. Here, we achieve desired behaviors using measure or language-conditioning describing the behavior, where the behaviors in question correspond to different gaits. \n"}, {"review_id": "K7jW9BOH0f", "reviewer": "Reviewer_mRc8", "summary": "This paper tried to solve the high space complexity in Quality Diversity and proposes a method that uses diffusion model to distill the archive into a single generative model based on policy parameters.", "questions": "* Need more explanation and comparison in experiment\n* There are some environments for quality diversity like QD-GYM. What's the performance of the method in QD-GYM?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* This paper leverages the generation power of diffusion model and condenses one model instead of thousands of policies.\n* This paper is well structured.", "weaknesses": "* Quality Diversity is not well-known. It's better to include a background section instead of including basic knowledge in the related work. \n* It's confusing no baselines can be directly compared. The metrics can not only be rewards but also space efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Quality Diversity is not well-known. It's better to include a background section instead of including basic knowledge in the related work.**\nWe have included a brief background section in our related work section. Please see the global response for a detailed explanation of measures we used in Quality diversity. We would be happy to split into its own section and extend it where space permits.\n\n\n**It's confusing no baselines can be directly compared. The metrics can not only be rewards but also space efficiency.**\n\nUnfortunately, possible baselines are very recent work (within 3 months of the submission), and code is not available to easily reproduce them. We show space efficiency in Table 2. We would be happy to add a note to Table 1 to point to Table 2 for details on space efficiency.\n\n### Questions:\n\n**Need more explanation and comparison in experiment**\n\nThank you for your feedback. Please see the global response for more details. We will add additional analyses to the experiments section, space permitting, and hope this resolves any additional questions or points of confusion the reviewer may have. \n\n**There are some environments for quality diversity like QD-GYM. What's the performance of the method in QD-GYM?**\n\nWe use QDax[1], which re-implements all of the tasks available in QD-Gym in a GPU-accelerated environment. If our method were to be run on QD-Gym the results should be identical, but the experiments would take significantly longer to run.\n\n[1] Lim, Bryan, Maxime Allard, Luca Grillotti, and Antoine Cully. \"Accelerated Quality-Diversity through Massive Parallelism.\" arXiv preprint arXiv:2202.01258 (2022).\n"}, {"review_id": "xGZsnRP13n", "reviewer": "Reviewer_GgS4", "summary": "Quality-Diversity Reinforcement Learning generates a set of policies (here called the archive) that are learned to produce varied behaviors in the environment.  These archives can be large, and this paper aims to compress a previously learned archive into a single model by leveraging a conditional diffusion model.  Each of the policies in the archive is first represented in a latent space by using a variational auto-encoder that reconstructs the weights and biases of the policy's neural network.  Once the encoder and decoder of the policies have been trained, a diffusion model is then fit to the encoded latents, and conditioning information is used to help guide sampling.  The paper explores conditioning based on the measure of a policy (where the measure is a set of functions used to split the policies into different regions of behavior space), or text descriptions of the policies.  The paper shows that the learned diffusion model can generate policies that return similar rewards as the original archive in aggregate, and also have high overlap with the conditioning measure when the sampled policy is executed again in the environment.  ", "questions": "- I find the design of the network encoder to be strange.  Why are convolutional filters applied to the weight matrix of an MLP when encoding?  Is there some spatial structure there to be captured? \n- What dependence does the diffusion model have on the underlying archive to be compressed?\n- Where do text descriptions of behaviors come from? Are policies labeled after learning, or are the labels pre-specified with the desired measures?  Is it easy to create this conditioning data?", "limitations": "The authors have addressed the method's limitations.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper proposes an interesting application of powerful generative models to fit a Quality-Diversity archive of policies.  The ability to reconstruct the full archive from a single generative model increases the practicality of QD approaches to skill discovery in reinforcement learning.  The paper does a good job highlighting this contribution, and it is indeed an intriguing direction.  ", "weaknesses": "- Evaluation of the model is thorough, in that ablations and several domains are explored, however it is difficult to assess the quality of the approach given that no alternative approaches are attempted.  In line 210, the paper argues that other approaches to archive distillation are not comparable because the underlying archive is different.  I disagree: since the main contribution of this paper is a distillation method, it should be able to compared to other distillation methods when the archive is held fixed.  \n- I find the metrics used to evaluate the model difficult to interpret, perhaps related to the lack of baseline.  The Mean-Error in Measure (MEM) metric described by the paper is a reasonable one, but the paper does not describe what measures are used in the various tasks, so interpreting the scale of MEM is difficult.  QD score is similarly difficult to interpret.  Is a decrease of $0.6 \\times 10^7$ QD score a meaningful one?\n- The experiment on sequential behavior composition is similarly difficult to interpret.  Is 80% success of 4 consecutive behaviors good?  Perhaps including the success rate of the original archive would be a good start.  \n- Graphs in Figure 3 are not immediately interpretable without prior exposure to that form of QD visualization.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Evaluation of the model is thorough,...**\n\nThere are two archive distillation methods most similar to ours, DCG-Map-Elites [1] and the QD-Transformer [2]. However, both of those methods are concurrent with ours and the code was not available for either of them at the time of submission. There are also design differences that make direct comparison difficult. The algorithm proposed in [1] jointly fills the archive and distills the policies into a descriptor-conditioned actor policy. Replacing the descriptor-conditioned actor with our diffusion model would fundamentally be a different algorithm and produce different results. We understand the reviewer’s concerns for baselines. However, to the best of our knowledge, archive distillation is a relatively novel idea and only starting to be investigated. We hope to be a baseline for future methods.\n\n[1] Faldor, M., Chalumeau, F., Flageat, M., & Cully, A. (2023). MAP-Elites with Descriptor-Conditioned Gradients and Archive Distillation into a Single Policy. arXiv preprint arXiv:2303.03832.\n\n[2] Macé, Valentin, Raphaël Boige, Felix Chalumeau, Thomas Pierrot, Guillaume Richard, and Nicolas Perrin-Gilbert. \"The Quality-Diversity Transformer: Generating Behavior-Conditioned Trajectories with Decision Transformers.\" arXiv preprint arXiv:2303.16207 (2023).\n\n**I find the metrics used to evaluate the model difficult to interpret,..**\n\nPlease see the answer in the global response at the top.\n\n**The experiment on sequential behavior composition....**\n\nWe have performed sequential behavior composition evaluations on the original archive, and obtain the same results as reported in the paper by the generative model. There are a number of prior works [1,2] where the primary or secondary goal was to compose behaviors with explicit optimization on the distilled model. Therefore, we believe that 80% is an impressive success rate given that our method was not explicitly designed or optimized to compose policies sequentially.\n\n[1] Peng, Xue Bin, et al. \"Deepmimic: Example-guided deep reinforcement learning of physics-based character skills.\" ACM Transactions On Graphics (TOG) 37.4 (2018): 1-14.\n\n[2] Krishna, Lokesh, and Quan Nguyen. \"Learning Multimodal Bipedal Locomotion and Implicit Transitions: A Versatile Policy Approach.\" arXiv preprint arXiv:2303.05711 (2023).\n\n**Graphs in Figure 3 are not immediately interpretable...**\n\nPlease see the answer in the global response at the top.\n\n### Questions \n\n**I find the design of the network encoder to be strange...**\n\nDeconvolutional layers are typically used for parameter generation in hypernetworks [1][2]. For symmetry, we chose to use convolutional layers for encoding. Our experiments show that convolutional layers are sufficient to perform this encoding task, which indicates that useful policy encoding does not depend on the interactions of parameters that are distant from each other in the weight matrices, a type of loose spatial structure. The use of convolutional layers has the advantage of reducing the total parameter count when compared to an MLP-based encoder. The encoder is not used during policy generation, and consequently has not been a major focus in this work. In future work, encoding may be further improved by incorporating advancements from recent work in [3].\n\n[1] Knyazev, Boris, Michal Drozdzal, Graham W. Taylor, and Adriana Romero Soriano. \"Parameter prediction for unseen deep architectures.\" Advances in Neural Information Processing Systems 34 (2021): 29433-29448.\n\n[2] Hegde, Shashank, and Gaurav S. Sukhatme. \"Efficiently Learning Small Policies for Locomotion and Manipulation.\" In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5909-5915. IEEE, 2023.\n\n[3] Zhou, Allan, Kaien Yang, Kaylee Burns, Yiding Jiang, Samuel Sokota, J. Zico Kolter, and Chelsea Finn. \"Permutation equivariant neural functionals.\" arXiv preprint arXiv:2302.14040 (2023).\n\n**What dependence does the diffusion model have on ..**\n\nWe believe that in order to generalize and smoothly interpolate between behaviors as shown in the sequential behavior composition task, the diffusion model requires archives with higher resolutions, on the order of thousands of policies. The “Covariance Matrix Adaptation” line of work i.e. CMA-ME [1], CMA-MAEGA [2], etc. tend to favor larger archives, since having more cells results in higher quality gradient estimates in the evolutionary adaptation component of these algorithms. Methods such as PGA-ME [3] tend to favor lower resolution archives because they jointly optimize the entire archive and do not maintain a local search distribution that optimizes for where in the archive to explore next. We use PPGA [4] for our work, which under the hood runs CMA-MAEGA, thus favoring large archives, and has also produced SOTA results on the locomotion tasks, thus making it the ideal choice for use with diffusion models. \n\n[1] Fontaine, Matthew C., Julian Togelius, Stefanos Nikolaidis, and Amy K. Hoover. \"Covariance matrix adaptation for the rapid illumination of behavior space.\" In Proceedings of the 2020 genetic and evolutionary computation conference, pp. 94-102. 2020.\n\n[2] Fontaine, Matthew, and Stefanos Nikolaidis. \"Differentiable quality diversity.\" Advances in Neural Information Processing Systems 34 (2021): 10040-10052.\n\n[3] Lim, Bryan, Manon Flageat, and Antoine Cully. \"Understanding the Synergies between Quality-Diversity and Deep Reinforcement Learning.\" arXiv preprint arXiv:2303.06164 (2023).\n\n[4] Batra, Sumeet, Bryon Tjanaka, Matthew C. Fontaine, Aleksei Petrenko, Stefanos Nikolaidis, and Gaurav Sukhatme. \"Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning.\" arXiv preprint arXiv:2305.13795 (2023).\n\n**Where do text descriptions of behaviors come from?..**\n\nPlease see the answer in the global response at the top.\n"}, {"review_id": "SOBsU4t4lz", "reviewer": "Reviewer_s86A", "summary": "The paper presents a diffusion-based approach to compressing a policy archive discovered by a Quality-Diversity RL algorithm. The diffusion model operates in the latent space of a VAE and achieves high levels of compression together with a reasonable level of reconstruction. The algorithm is evaluated on a collection of 4 Brax environments.\n", "questions": "- The ablations on network capacity in Table 2 are only run on humanoid which works well, it would be interesting to understand if the large loss in diversity on the ant environment can be alleviated by higher network capacity or better representation learned by the VAE.\n- Can the diffusion model generalize to novel conditions and improve on the original archive?\n- Section 3: The convolutional layers used to encode the weights and biases of the network could be less appropriate than equivariant layers such as those used in [1].\n\n[1] Permutation Equivariant Neural Functionals. Allan Zhou, Kaien Yang, Kaylee Burns, Yiding Jiang, Samuel Sokota, J. Zico Kolter, Chelsea Finn.\n", "limitations": "The limitations of the method are discussed well in the paper.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Clearly written paper and description of the approach\n- Promising results showing high levels of compression on some Brax environments, together with good levels of reconstruction and coverage\n- Strong visualizations of behavior during training and different synthesized behaviors", "weaknesses": "- The approach studied in the paper is limited to compression of the original archive. Equally, the sequential behavior composition experiments only reproduce what was possible with the original archive. An interesting next step would be understanding if the diffusion model can generalize to novel measure vectors or language instructions.\n- The paper assumes very small 2-layer, 128-width MLPs trained by the QD algorithm, it is unclear if the algorithm can scale to larger and more representative networks\n- High loss in diversity particularly on the ant environment, in Table 1.\n- Line 8 in the Abstract, should clarify exactly what environments the authors see the compression ratio/coverage on\n\nMinor:\n- Line 42: typo in ‘uspample’\n- Scale is hard to see in Figure 3, axes should also be described\n- The idea of compressing policies into a single diffusion model is related to [1] which compresses offline RL datasets into a single diffusion model, [1] also achieves around 13x compression.\n- It would be helpful to also indicate the level of compression in Table 1.\n\n[1] Synthetic Experience Replay. Cong Lu, Philip J. Ball, Yee Whye Teh, Jack Parker-Holder.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The approach studied in the paper is limited to compression of the original archive...**\n\nWe agree that generalization to novel language instructions is an interesting direction for future work. However, generalizing to novel measure dimensions is not possible with the environments available. The measure functions in our policy datasets function similar to class categories in computer vision tasks i.e. it is not possible to generalize to novel measures in the same way that generative models trained on CIFAR10 cannot generalize to the CELEB-A dataset unless these datasets were combined apriori. However, similar to how vision models can generate images with different categories of objects drawn in the same image in different locations, we attempt to show with sequential behavior composition that different parts of a fixed-length trajectory can be filled in with arbitrarily different behaviors without catastrophic failure. Prior methods show the difficulty of this task and solve it by explicitly optimizing for sequential composition of different policies during training [1, 2]. \n\n[1] Peng, Xue Bin, et al. \"Deepmimic: Example-guided deep reinforcement learning of physics-based character skills.\" ACM Transactions On Graphics (TOG) 37.4 (2018): 1-14.\n\n[2] Krishna, Lokesh, and Quan Nguyen. \"Learning Multimodal Bipedal Locomotion and Implicit Transitions: A Versatile Policy Approach.\" arXiv preprint arXiv:2303.05711 (2023).\n\n**The paper assumes very small 2-layer, 128-width MLPs ...**\nTo address larger and different types of policy networks, we believe differences may arise in the encoding and decoding of these policies. For decoding, we believe that graph hypernetworks can scale well to different types of networks. It was shown in [1] that hypernetworks can estimate the weights to larger and more representative networks. [2] Further shows that these hypernetworks can estimate weights of larger MLP policies as well.  For encoding, an alternative to CNN based encoding can be using neural functional networks as described in [3]. These permutation equivariant neural functional networks seem to learn functions based on CNN networks weights very well.\n\n[1] Knyazev, Boris, Michal Drozdzal, Graham W. Taylor, and Adriana Romero Soriano. \"Parameter prediction for unseen deep architectures.\" Advances in Neural Information Processing Systems 34 (2021): 29433-29448.\n\n[2] Hegde, Shashank, and Gaurav S. Sukhatme. \"Efficiently Learning Small Policies for Locomotion and Manipulation.\" In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5909-5915. IEEE, 2023.\n\n[3] Zhou, Allan, Kaien Yang, Kaylee Burns, Yiding Jiang, Samuel Sokota, J. Zico Kolter, and Chelsea Finn. \"Permutation equivariant neural functionals.\" arXiv preprint arXiv:2302.14040 (2023).\n\n**High loss in diversity particularly on the ant environment, in Table 1.**\nWe believe that the loss in diversity on the ant environment is a result of there being a large number of poor performing policies that do not share parameters with other high performing policies in the dataset, and that this is not an issue with the proposed method itself. From the CDF plots in figure 4, we see that the diffusion model recovers most, if not all, of the higher performing policies, and loses diversity only where lower performing policies are concerned. If the proposed method struggled specifically on ant for some reason i.e. the higher dimensional measure space, we would expect an equivalent drop in diversity along all levels of policy performance. From our GHN size ablation, we see that coverage on Ant correlates strongly with model capacity. Thus, we expect to be able to recover most of the original archive’s coverage with larger models. \n\n**Line 8 in the Abstract,..**\nThank you for pointing this out. We will clarify this point in the revised manuscript.\n\n### Questions: \n\n**The ablations on network capacity in Table 2 are only run on humanoid...**\n\nThank you for your suggestion. We have conducted this ablation and have added it to the attached file. Scaling up the GHN size in the decoder does indeed alleviate the performance drop of our model on the Ant environment. Further, we see that the compression ratio is still reasonable at larger GHN sizes. \n\n\n**Can the diffusion model generalize to novel conditions and improve on the original archive?**\nWe believe that our model can interpolate well within the bounds of the measure space described in the global response above. Unfortunately, with the current specified measures, it is not possible to generalize outside these bounds because it is not physically possible. For example, a measure of 1.2 implies 120% foot contact time with the ground. Measure functions in QD behave as discrete categorizations of behavior, similar to how image categories function in computer vision tasks. However, an interesting research direction would be to add new measure functions online to the archive, filling it in with new policies while jointly using these new policies to train the diffusion model, in order to increase its expressivity and dimensionality of the behavior space. \n\n**Section 3: The convolutional layers used to encode the weights and biases of the network could be less appropriate than equivariant layers such as those used in [1].**\n\nThank you for this suggestion! We were not aware of this paper prior to submission and believe this would be a promising direction to improve our method’s representation capacity in the future. \n\n### Minor weaknesses:\n\nThe corrected figure has been added to the attached additional figures page.\n\n**The idea of compressing policies into a single diffusion model is related to [1] ..**\n\nThank you for bringing this recent work to our attention. We agree that compressing offline RL datasets and compressing policy datasets is related, so we will add a citation to this work.\n"}]

Author rebuttal:
We thank all the reviewers for their in-depth feedback. Attached here is a pdf document with additional tables and figures.

**Measures in Quality Diversity:**
Thank you for the reviewers' feedback. We see that our explanation of measures is lacking. Below is a brief explanation of what measures are used. The same edits will be made to the revised manuscript. The measure functions for all tasks in our experiments are the same – each measure function measures the proportion foot contact time for each foot on the robot. The measure functions are thus bounded to the [0, 1] interval, where 0 indicates the foot never touched the ground, and 1 indicates the foot never left the ground. We agree that QD-scores are difficult to interpret, and also have the weakness that they are sensitive to archive resolution and scale of the objective function. However, we feel it is necessary to include it in our evaluation to allow our results to be compared with prior works in QD that include the QD-score in their evaluations. A more interpretable description of our method’s performance can be found in the CDF plots in Figure 4. CDF plots show the percentage of policies that achieve an episodic reward of R or higher, for all possible R values on the x-axis. Thus, it captures aspects of both performance and diversity, as well as how the policies are distributed with respect to performance – data which is not well represented by a scalar value. We will revise the description of Table 1 to emphasize that Figure 4 provides more interpretable evaluation results.

**Visualization of the archive:**
Figure 3 visualizes an implicit archive over the course of training for the diffusion model. The axes of the figure describes the measures described above. As the diffusion model learns to better represent the policy distribution w.r.t. performance and behavior, it is capable of filling more cells (where each cell corresponds to a policy with that behavior) with high performing solutions. Near-optimal policies are represented in yellow. We appreciate the feedback and will update the caption and axes of Figure 3 to better explain this. The updated figure is in the attached additional figures pdf file.

**Text Labeling policies**
The training text labels were generated by manually labeling policies in the original archive after the archive is generated but before training the diffusion model. We labeled policies by repeatedly inspecting a rollout for the policy farthest in parameter space from all previously labeled policies. We continued this process until we had labeled 128 of the policies, and labeled the remaining policies using nearest neighbors in parameter space. Labeling took approximately 3-5 minutes per policy on average, for a rough total of 10 hours. Most of that time was spent searching for the next policy or waiting for policies to be rendered, which could be avoided with better labeling tools.


Meta-review: Four experts reviewed this paper with all accepted recommendations. The area chairs agree that this work makes a very important contribution by introducing the use of diffusion models to distill the archive into a single generative model over policy parameters. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper!
Decision: Accept (poster)