Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity
Authors: Robby Costales
Stefanos Nikolaidis
Venue: NeurIPS 2024 poster (2024)
Keywords: diversity
meta reinforcement learning
meta-RL
reinforcement learning
adaptation
adaptive
agents
open-endedness
genotypes
phenotypes
simulators
simulation
generalization
meta-reinforcement
Primary area: reinforcement_learning
TL;DR: DIVA is an evolutionary approach for generating meaningfully diverse meta-RL training tasks in truly open-ended simulators.

Abstract: The wider application of end-to-end learning methods to embodied decision-making domains remains bottlenecked by their reliance on a superabundance of training data representative of the target domain.
Meta-reinforcement learning (meta-RL) approaches abandon the aim of zero-shot *generalization*—the goal of standard reinforcement learning (RL)—in favor of few-shot *adaptation*, and thus hold promise for bridging larger generalization gaps.
While learning this meta-level adaptive behavior still requires substantial data, efficient environment simulators approaching real-world complexity are growing in prevalence.
Even so, hand-designing sufficiently diverse and numerous simulated training tasks for these complex domains is prohibitively labor-intensive.
Domain randomization (DR) and procedural generation (PG), offered as solutions to this problem, require simulators to possess carefully-defined parameters which directly translate to meaningful task diversity—a similarly prohibitive assumption.
In this work, we present **DIVA**, an evolutionary approach for generating diverse training tasks in such complex, open-ended simulators.
Like unsupervised environment design (UED) methods, DIVA can be applied to arbitrary parameterizations, but can additionally incorporate realistically-available domain knowledge—thus inheriting the *flexibility* and *generality* of UED, and the supervised *structure* embedded in well-designed simulators exploited by DR and PG.
Our empirical results showcase DIVA's unique ability to overcome complex parameterizations and successfully train adaptive agent behavior, far outperforming competitive baselines from prior literature.
These findings highlight the potential of such *semi-supervised environment design* (SSED) approaches, of which DIVA is the first humble constituent, to enable training in realistic simulated domains, and produce more robust and capable adaptive agents.
Our code is available at [https://github.com/robbycostales/diva](https://github.com/robbycostales/diva).

Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity
Robby Costales*
Stefanos Nikolaidis
Department of Computer Science University of Southern California
Abstract
The wider application of end-to-end learning methods to embodied decision- making domains remains bottlenecked by their reliance on a superabundance of training data representative of the target domain. Meta-reinforcement learn- ing (meta-RI.) approaches abandon the aim of zero-shot generalization goal of standard reinforcement learning (RL)-in favor of few-shot adaptation, and thus hold promise for bridging larger generalization gaps. While learning this meta-level adaptive behavior still requires substantial data, efficient environment simulators approaching real-world complexity are growing in prevalence. Even so, hand designing sufficiently diverse and numerous simulated training tasks for these complex domains is prohibitively labor-intensive Domain randomization (DR) and procedural generation (PG), offered as solutions this problem, re- quire simulators to possess carefully-defined parameters which directly translate to meaningful task diversity- a similarly prohibitive assumption. this work, we present DIVA, an evolutionary approach for generating diverse training tasks in such complex, open-ended simulators. Like unsupervised environment design (UED) methods, DIVA can be applied to arbitrary parameterizations, but can ad- ditionally incorporate realistically-vviilae the flexibility and generality of UED, and the supervised structure embedded in well-designed simulators exploited by DR and PG. Our empirical results showcase DIVA's unique ability to overcome complex parameterizations and successfully train adaptive agent behavior, far outperforming competitive baselines from prior literature. These findings highlight the potential of such semi-supervised environ- ment design (SSED) approaches, of which DIVA is the first humble constituent, to enable training in realistic simulated domains, and produce robust and capable adaptive agents. Our code is available at httpp://ithbb..cm//bbbyyttttettt
domain knowledge- inheriting
Introduction
Despite the broadening application of reinforcement learning (RL) methods to real-world problems [121. generalization to new scenarios-one, not explicitly supported by the training set set-remains fundamental challenge [3] Meta-reinforcement learning (meta-RI.), an extension the RI. frame- work, is formulated specifically for training adaptive agents, and is thus well-suited for overcoming these generalization gaps [4). One recent work has demonstrated that meta-RL agents can be trained at scale to achieve adaptation capabilities
par with human subjects [5]. However, learning this human-like adaptive behavior naturally requires a large amount of data representative of the down- stream (or target) distribution. For task distributions approaching real- world complexxty-ppecciii the ones of interest designing each scenario by hand is prohibitively expensive.
Correspondence to rscostalQusc. edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

random parameters
archive of levels
DIVA QD updates
01
axis of diversity
2
0
(idealistic) structured
freelistic) unstructured
(0)
enwronmment
E (
treat "black box"
exis of Sz
diversity
target diversity
low diversity
discovered diversity
Figure Highly structured environment simulators assume access to parameterizations Es (0) for which random seeds 0, directly produce meaningfully diverse features (e.g. RACING tracks with challenging turns). Open-ended environments with flexible, unstructured parameterizations (u(6))thour enabling more complex emergent features- lack direct control over high-level features of interest. We introduce DIVA, an approach that effectively creates a more workable parameterization (0) by evolving levels beyond the minimally diverse population from (0). By training on these discovered levels, DIVA enables superior performance on downstream tasks.
Prior works have explored the use of domain randomization (DR) and procedural generation (PG) techniques produce diverse training data for learning agents (6. Despite eliminating the need for hand-designing each task individually, human labor is still required to carefully design an environment generator that can produce diverse, high-quality tasks. As environments become more complex and open-ended, the ability to hand-design such a robust generator becomes increasingly infeasible. Some methods, like PLR 77 attempt ameliorate this limitation by learning curriculum over the generated levels, but these works still operate under the assumption that the generator produces meaningfully diverse levels with a high probability.
Unsupervised environment design (UED) g are a broad class of appproaches which use performance- based metrics to adaptively form currieulum of training levels. ACCEL (9] state-of-the-art UED method, uses an evolutionary process to discover more interesting regions of the simulator's parameter space (i.e. appropriately challenging tasks) than can be found by random sampling. While UED approaches are designed to be generally applicable and require little domain knowledge, they implicitly require very constrained environment generator-one in which all axes of difficulty correspond to meaningful learning potential for the downstream distribution. Moreover, when faced with complex open-ended open-ended environments with arbitrary parameterizations, even ACCEL is not able to efficiently explore the solution space, as it is still bottlenecked by the speed of agent evaluations.
In this work, introduce DIVA, approach for generating diverse training tasks in open-ended simulators to train adaptive agents. By using quality diversity (QD) optimization to efficiently explore the solution space, DIVA bypasses the problem of needing evaluate agents all generated levels. QD also enables fine-grained control over the axes of diversity to be captured the training tasks, allowing the flexible integration of task-related prior knowledge from both domain experts and learning approaches. We demonstrate that DIVA, with limited supervision in the form of feature samples from the target distribution, significantly outperforms state of the art UED approaches- despite the UED approaches being provided with significantly more interactions. We further show that UED techniques can be integrated into DIVA. Preliminary results with this combination (which we call DIVA+) are promising, and suggest an exciting avenue for future work.
2 Preliminaries
Meta-reinforremment learning. We use the meta-reinforcemmnt learning (meta-RI.) framework to train adaptive agents, which involves learning adaptive policy T& over distribution of tasks T.

---

begin stage
guide solutions to target
begin stage? 2
populate target
sample solutions
.V ((2.o)
target regian
sample mask
Density
(a)
(b)
(C)
(d)
(e)
Figure 2: DIVA archive updates on ALCHEMY. The first stage (a) begins with bounds that encapsulate initial solutions, and the target region. As the first stage progresses (b), and QD discovers more of the solution space, the sampling region for the emitters gradually shrinks towards the target region. The second stage begins by redefining the archive bounds be the target region and including some extra feature dimensions (c). QD fills out just the target region now (d), using sample weights from the target-derived prior (e), the same distribution used to sample levels during meta-training.
Each MM, E T is a Markov decision process (MDP) defined by a tuple A, R, Y, T", where S is the set of states, is the set of actions, P(s [st, ae) is the transition distribution between states given the current state and action, R(st, at) is the reward function, E [0, 1] is the discount factor, and T is the horizon. Meta-training involves sampling tasks M; ~ T, collecting trajectories D = { h== --=---1-- where H is the number of episodes in each trial T pertaining to the M,,and optimizing policy parameters & to maximize the expected discounted returns across all episodes.
-
VariBAD 10]
is a context variable-based meta-RL approach which belongs to the wider class of RNN- based methods 12 While prior methods [13 14 also context variables assist in task adap- tation, VariBAD uniquely learns within
MDP (BAMDP) (S, Z, P. R, 7, T) where the context variables z e Z encodes the agent's 's uncertainty about the task, promoting Bayesian exploration. VariBAD utilizes an RNN-based variational autoencoder (VAE) to model a posterior belief over possible tasks given the full agent trajectory, permitting efficient updates to prior beliefs.
Quality diversity. For a given problem, quality diversity (QD) optimization framework aims to generate a set of diverse, high-quality solutions. Formally, problem instance of QD [I5] specifies an objective function space formed by the range of f. where f R R is the joint feature vector. For each S. the QD objective is to find a solution E R where f(0) = s and (()) is maximized. Since R is continuous, an algorithm solving the QD problem definition above would require unbounded memory to store all solutions. QD algorithms in the MAP-Elites (16) family therefore discretize via a tessellation method, where G is a tessellation cells. In employing a MAP-Elites algorithm, we relax the QD objective to find set of solutions 0, í E {1,..., Ng), such that each 0, occupies one unique cell in G. We call the occupants 6, of all M cells, each with its own position f (0:) and objective value J(O)), the archive of solutions.
1n + R. and features ft : R" n R. Let = f fIX" be the feature
the continuous feature space S into Ng
3 Problem setting
One assumption underlying UED methods that random parameters- or parameter perturbations for ACCEL- -produce meaningfully different levels to justify the expense of computing objectives on each newly generated level. However, when the genotype not well-behaved-wve when meaningful di- versity is rarely generated through random sampling or mutations craatiions-theee algorithms waste significant time evaluating redundant levels. In our work, we discard the assumption of well-behaved genotypes in favor of making fewer, more realistic assumptions about complex environment generators. There are several assumptions we make about the simulated environments DIVA has access to.
Genotypes. We assume access to an unstructured environment parameterization function E (O). where each 0 is genotype (corresponding to the QD solutions 0,) describing parameters to be fed into the environment generator. QD algorithms can support both continuous and discrete genotype spaces, and in this work we evaluate on domains with both kinds. Crucially, we make no assumption of the quality of the training tasks produced by this random generator. We only assume that (1) there is some nonzero (and for practical purposes, nontrivial) probability that this generator will produce which success is possible and positive rewards are in reach; and (2) that it is computationally feasible to discover meaningful feature diversity through an intelligent search over the parameter space- an assumption implicit in all QD applications.
a valid level for training-on
3

---

Features. We assume access to a pre-defined set of features, S = f (RR), that capture axes of diversity which accurately characterize the diversity to be expected within the downstream task distribution. It is also possible to learn or select good environment features from a sample of tasks from the downstream distribution, which we discuss |Section 7, For the sake of simplicity, we use a grid archive as our tessellation G. where the & dimensions of the discrete archive correspond to the defined features. The number of bins for each feature is a hyperparameter, and can be learned or adapted over the course of training. We generally find it to be helpful to use moderately high resolutions to ease the search, since smaller leaps in feature-Ievel diversity are required to uncover new cells. By default, we use 100 sample feature values across all domains, but demonstrate in ablation studies that that significantly fewer may be used (see Appendix C).
4 DIVA
DIVA assumes access to a small set of feature samples representative of the target domain. It does not, however, require access to the underlying levels themselves. This is a key distinction, as the former is a significantly weaker assumption. Consider the problem of training in-home assistive robots in simulation with the objective of adapting to real-world houses. It is more likely we have access to publicly available data describing typical couusss-iimeniione stylistic features, etc.-than we have access to corresponding simulator parameters which produce those exact feature values.
Feature density estimation. DIVA begins by constructing a QD archive with appropriate bounds and resolution. Given set of specified features [f.}* and handful of downstream, feature samples, we first infer each feature's 's underlying distribution. These can be approximated with kernel density estimation (KDE), we can work with certain families of parameterized distributions. For our experiments, we assume each feature is either (independently) normally or uniformly distributed. We use statistical test3 to evaluate the fit of each distribution family, and select the best-fitting. Setting the resolution for discrete feature dimensions is straightforward, and depends only on the range, For continuous features, the resolution should enable enough signal for discovering new cells, while avoiding practical issues that arise with too many cell See Section for domain-specific details.
Two-stage QD updates. Once the feature-specific target distributions are determined, we can use these to set bounds for each archive dimension. A naive approach would be to set the archive ranges for each feature based on the confidence bounds of the target distribution. However, random samples from Ey may not produce feature values that fall within the target range. We found this to be a major issue in the ALCHEMY domain (see Figure 2). and for some features in RACING. We solve this problem by setting the initial archive bounds to include both randomly generated samples from Ev, as well as the full target region. As the updates progress, gradually update the sample mask which is used to inform the sampling of new solutions- towards the target region. We observe empirically that updating and applying this mask provides enormous speed-up in guiding solutions towards the target region (see (Figure 15). After this first stage, solutions are inserted into a new archive defined by the proper target bounds. See Appendix for more specifics on these two QD update stages.
Overview. DIVA consists of three stages. Stage (S1) begins by initializing the archive with bounds that include both the downstream feature samples (the target region), well as the initial population generated from (0). S1 then proceeds with alternating QD updates, to discover new solutions, and sample mask updates, to guide the population towards the target region. In Stage (S2), the archive is reinitialized with existing solutions, but is now bounded by the target region. QD updates continue to further diversify the population, now targeting the downstream feature values specifi- cally. The last stage is standard meta-training, where training task parameters are now drawn from Pa 10:
Algorithm 1 DIVA
Stage 1: discover target region
1: G + ceittallie_archiin ()
2: for i in range(Nsi) do
3:
G + QD QD_UPDATEEC,, J, M, Bop)
4: M <- hppdte_ssmpll Ma g)
Ssage 2: populate target region
5: G t uudatt__ update_archive___ archive_boundee
6: for i in range do
7:
G G- QD_UPDATE(G. J, 1.0, Bon)
Mata-learn over QD archive 8: for in ( do
9:
M + (o (o)
T T perform_rolloate (M,ooooootMM m
distribution over the feature space approximated using the downstream feature samples, discretized over the archive cells. See Appendix A detailed pseudocode.
11:
D2 + sstoreerooloutt (D,, T)
12:
neta_update(T,. D.)
2 We use Jalmogoroy-Smima test for features with continuous values and Chi-squared for discrete.
"Memory is one concer;; another is that optimizing objectives across all cells is slower with more cells.

---

5 Empirical results
Baselines. We implement the following baselines to evaluate their relative performance to DIVA. ODS is the "oracle" agent trained over the downstream environment distribution Es((), used for evaluation. With this baseline, we are benchmarking the upper bound in performance from the perspective of learning algorithm that has access to the underlying data distribution F DR is the meta-learner trained over a task distribution defined by performing domain randomization over the space of valid genotypes, 0, under the training parameterization, (u( Robust PLR (PLR-) [7 is the improved and theoretically grounded version of PLR 77. where agents' performance-basee PLR objectives are evaluated on each level before using them for training. ACCEL [9] is the same as PLR- but instead of randomly sampling over the genotype space to generate levels for evaluation, levels are mutated from existing solutions. All baselines use VariBAD [0] as their base meta-learner.
Experimental setup. The oracle agent (ODS) is first trained over the each environment's 's down- stream distribution to tune VariBAD's ' hyperparameters. These environment-specifee VariBAD settings then fixed while hyperparameters for DIVA and the other baselines are tuned. For faireess of comparison- since DIVA is allowed NOD QD update steps to fill its archive before meta- veeta-rainingg- we allow each UED approach (PLR- - and ACCEL) to use significantly more environment steps for agent evaluations (details discussed below per environment). All empirical results were run with 5 seeds unless otherwise specified, and error bars indicate a 95% confidence region for the metric in question. The QD archive parameters were set per environment, and for ALCHEMY and RACING, relied on some hand hond-tuning to find the right combinations of features and objectives. We leave it to future work perform deeper analysis on what constitutes good archive design, and how to better automate this process.
5.1 GRIDNAV
F
Eg
Eg
Ey
Our first evaluation domain is a modified version of GRIDNAV (Figure 3/. originally introduced to motivate and benchmark VariBAD [10]. The agent spawns at the center of the grid the start of each episode, and receives slight negative reward (r -0.1) each step until it discovers (inhabits) the goal cell, at which point it also receives a larger positive reward (r 1.0).
Parameterization. We parameterize the task space (i.e. the goal location) to reduce the likelihood of gen- erating meaningfully diverse goals. Specifically, each Fu, (or F&)) introduces k genes to the solution genotype which together define the final y location. Each gene j can assume the values 0, E (-1,0, 1), and the final y location is determined by summing these values, and performing a floor division to map the bounds back to the original range of the grid, As k increases, y values are increasingly biased towards 0, as shown on the right side of Figure 3 For more details the GRIDNAV domain, see Appendix B.1
Plray Peaa)(MM) > 0
75 (3)
Figure 3: Left: A GRIDNAV agent attempt- ing to locate the goal across two episodic rollouts. Right: The marginal probability of sampled goals inhabiting each y for dif- ferent complexities k of (x))).
QD updates. We define the archive features to be the x and y coordinates of the goal location.
The objective is set to the current iteration, that newer solutions are prioritized (additional details in Appendix B.I. DIVA is provided Ns2 = 8.0 x 10 (Nvi = 0) QD update iterations each provided with additional
for filling the archive. To compensate, PLR - and ACCEL 9.6 x 10th environment steps for evaluating PLR scores, which amounts to three times as many total interactions- since all methods provided Ng = 4.8 x 10 interactions for training. If each "reset" call counts as one environment stef[] the UED baselines are effectively granted 2.4x more additional step data than what DIVA additionally receives through its QD updates (details in Appendix E.I).
Results. From Figure 4a we see that increasing genotype complexity (i.e. larger k) reduces goal diversity for DR which is expected given the parameterization defined for Fgz. We can also see that DIVA, as a result of its QD updates, can effectively capture goal diversity, even as complexity
$ Teecnicallly, reweighting this distribution (e.g. via PLR) may produce a stronger oracle, but for the purposes of this work, we assume the unaltered downstream distribution can be efficiently trained over, sans curriculum. In general, rendering the environment (via "reset") is required to compute level features for DIVA.
5

---

Goal Diversity by 8 Complexity
UED Goal Diversity - 21)
Final Episade Returns
Success Rate by Episode
18-
PLR
Percent Coverage
Percent Coverage
ADCM
Lbogue
1005 DIVA
Value from Fagar Sa (1- 36
Success Rate
ACCEL
-3-
PLR
04:
DIVA DR
13-
00 05
16 Updates
34
DNA
ACOTL
P.
DR
Genotype Complexity (a)
Updates
Method
(b)
(c)
(d)
Figure 4: GRIDNAV analysis and results. (a) Target region coverage produced by DIVA and DR
over different genotype complexities k. DR represents the average coverage of batches corresponding to the size of the QD archive. DR represents the total number of unique levels discovered over the course of parameter randomization steps which equal in number to the additional environments PLR- is provided for evaluation. DR is thus an upper bound on the diversity that PLR- can capture. 500k iterations (QD or otherwise)
used across all results. (b) The diversity produced by PLR L and ACCEL over the course of training (later updates omitted due to no change in trend). (c) Final episode return curves for DIVA and baselines. (d) Final method success rates across each episode.
increases. When we fix the complexity (k: = 24) and train over the Eu distribution, we see that the UED approaches are unable to incidentally discover and capture diversity over the course of training FFigure 4b). DIVA's explicit focus on capturing meaningful level diversity enables it to significantly outperform these baselines in terms of episodic return (Figure 4ch and success rate [Figure 4d).
5.2 ALCHEMY
ALCHEMY [8] is an artificial chemistry environment with a combinatorially complex task distri- bution. Each task is defined by some latent chemistry, which influences the underlying dynamics, as well as agent observations. To successfully maximize returns over the course a trial, the agent must infer and exploit this latent chemistry. At the start of each episode, the agent is provided new set of (1-12) potions and (1-3) stones, where each stone has a latent state defined by a specific vertex of cube, i.e. ((0, 1), 1): {0, 1}), and each potion has latent effect, specific manner in which it transforms stone latent states (see Figure 5a). The agent observes only salient artifacts of this latent information, and must use interactions to identify the ground-truth mechanics. At each step, the agent can apply any remaining potion to any remaining stone. Each stone's ' value is maximized the closer its latent state is to (1, 1), and rewards are produced when stones are cast into the cauldron.
To make training feasible on academic resources, we perform evaluations on the symbolic version of ALCHEMY, opposed to the full Unity-based version. Symbolic ALCHEMY contains the same mechanistic complexity, minus the visuomotor challenges which are irrelevant to this project's aims.
Parameterization. F0)(O) is the downstream distribution containing maximal stone diversity. For training, implement Fu, where i controls the level of difficulty in generating diverse stones. Specifi- cally, we introduce larger set of coordinating genes fg E {0, } that together specify the initial stone latent states, similar to the mechanism we used in GRIDNAV to limit goal diversity. Each stone latent coordinate is specified with & genes, and only when all k genes are set to is the latent coordinate is set to 1. When any of the genes are O, the latent coordinate is O. For our experiments we set k = 8, and henceforth use Ex to signify Eug.
QD updates. We use features SATENTSTATEDDVERRRS of which target stone latent state diversity from different angles. See(Appendix B.2 for more specifies on these features and other details surrounding ALCHEMY's 's archive construction. Like GRIDNAV, the objective is set to bias new solutions. DIVA is provided with Ns11 = 8.0 x 10a and Nsz = 3.0 x 10 QD update iterations. PLR and ACCEL are compensated such that they receive 3.5x more additional step data than what DIVA receives via QD updates (see aeppendxxx E. for details).
and MANHATTANTOOFTIAAL both
Results Our empirical results demonstrate that DIVA is able to generate latent stone states with diversity representative of the target distribution. We see this both quantitatively in Figure 56) and qualitatively in Figure 6 In Figure 5c, we see this diversity translates to significantly better results

---

(0,
SANEAATAATOOPPINA
Final Episode Returns
Unique Genotypes
1001
Density
Returns
Quantity
DR
ob 2b de
1, 1)
Feature Values
Updates
Updates
(b)
(c)
(d)
Figure 5: ALCHEMY environment and results. (a) A visual representation of ALCHEMY'' structured stone latent space. Pi and P2 represent potions acting on stones. Only Pi results latent state change, because P2 would push the stone outside of the valid latent lattice. (b) Marginal feature distributions for Es (the structured target distribution), DIVA, and Ey (the unstructured distribution used directly for DR, and to initialize DIVA's 's archive). (c) Final episode return curves for DIVA and baselines. (d) Number of unique genolypes used by each method over the course meta- training.
Early samples
Archive after S1
Later samples
LATENTSTATTEIMERRR
MANHATTAN TOOPINAL
Figure 6: ALCHEMY level diversity. Early on in DIVA's QD updates (left), the levels in the archive do not posses much latent stone diversity-all all are close to (1, 1, 1). As samples begin populating the target region in later QD updates (right), stone diversity is significantly increased.
on Es over baselines. Despite generating roughly as many unique genotypes as DIVA Figure 5d), PLR and ACCEL are unable to generate training stone sets of significant phenotypical diversity to enable success on the downstream distribution.
5.3 RACING
Lastly, we evaluate DIVA on the RACING domain introduced by [17. In this environment, the agent controls race car via simulated steering and gas pedal mechanisms, and is rewarded for efficiently completing the track, M; E T. We adapt this RL environment to the meta-RL. setting by lowering the resolution of the observation space significantly. By increasing the challenge of perception, competent agents benefit from multiple episodes to better understand the underlying track. For all of H episodes per trial, and flattened 15 15 pixel observation space.
experiments,
Setup. We use three different parameterizations our experiments: (1) Es (0) is the downstream distribution we use for evaluating all methods, training ODS, and setting archive bounds for DIVA. Parameters 0 are used seed the random generation of control points which in turn parameterize sequence of Bézier curves designed to smoothly transition between the control locations. Track diversity further enforced by rejecting levels with control points that possess a standard deviation below a certain threshold. (2) Fx, (0) is a reparameterization of Es (0) that makes track diversity harder to generate, with the difficulty proportional to the value of & E N. For our experiments, we use A 32 (which we will denote simply as Eu (0)). which roughly means that meaningful diversity is 32x less likely to randomly occur than when
(which is equivalent to (0)). This is achieved by defining small region in the center, 32 (or kt in general) times smaller than the track boundaries, where all points outside the region are projected onto the unit square, and scaled to the track size. (3) Ep1 (0) uses 0 as RNG seed to select between set of 20 hand-crafted levels official Formula-1 tracks and is used to benchmark DIVA't 's zero-shot generalization to new target distribution.
QD updates. We define features the archive dimensions. Levels from Fy lack curvature (see Figure 8) so TAC, which is defined as the sum of angle changes between track segments, useful for directly targeting this desired curvature.
(TAC) and CENTEROFMASSX (CX) for

---

OHNTROOMMSSX
Final Episode Returns
Track Completion
10335
DIVA
02-
Success Rate
Density
Returns
DR
VAHASCEX
TOTA.ARGLECHANGGS
04-
DIVA
ACCFI
DIVA
P1-0
Fg:
12
65
45
0%%
201 205
Feature Values
Updates
Completion Threshald (c)
(a)
(h)
Figure 7: RACING features and main results, Left: Marginal feature distributions for Es (target distribution), Eş1 (human-dexigned F1 tracks), DIVA, and Eu (the unstructured distribution used for DR, the original levels that DIVA evolves) evolves)-croppee for readability. Center: Final episode return curves for DIVA and baselines on Es. Right: Track completion rates by method, evaluated on Es.
Archive f
Final archive
TOTALANNLEE CNGALANGGECIANGEE
TOTAL AOTAALAANCE CRREENCCII
CENTEROFMASSX
CENTEROFMASSX
Figure 8: RACING level diversity. We see that random Ey levels, used by DR, and which form the initial population of DIVA, are unable to produce qualitatively diverse tracks (left). After the two-stage QD-updates, DIVA is able produce tracks of high qualitative diversity (right).
CX, the average location of the segments, targets diversity in the location of these high-density (high-curvature) regions. We compute an alignment objective over features CENTEROFMASS and VARIANCEY further target downstream diversity. See Appendix B.3 for more details relevant to the archive construction process for RACING. DIVA provided with 2.5 x 10 initial QD updates on RACING. PLR and ACCEL are compensated with 4.0x more additional step data than what DIVA receives through QD updates (see Appendix E. for more details).
Main results. Results shown in Figure 7
DIVA outperforms all baselines, including the UED approaches, which have access to three times as many environment interactions. From Figure 8 we that final DIVA levels contain significantly more diversity than randomization over Eg.
Transfer to F1 tracks. Next, we evaluate the ability of these trained policies to zero-shot transfer human- designed F1 levels [7]. Eri. Though qualitative differ- ences apparent (see Digure 9), from [Figure 7a we can additionally see how these levels differ quantitatively. Even though DIVA uses feature samples from Es to define its archive, we see from the results in Figure that DIVA is not only able complete many of these tracks, but is also able to significantly outperform ODS. This result may seem unlikely, given that DIVA bases axes of diversity on Es. One possible explanation is that while DIVA success- fully matches its FOTALANGLECHANDE Es (see Figure 7). because it is less likely for all 12 control points be mutated to the iiversity-enablii just few control points with sharp angles, DIVA "opts" for the latter, and thus produces fewer, sharper angles, which is evidently useful for transferring to (these) human-designed tracks. This hypothesis matches what we see qualitatively from the DIVA alVA-produced levels in Figure 8
Levels
Track Completion on (0)
0.15
Success Rate
'O2S
0.10
DIVA
distribution to
0.05
region than
100%
Completion Threshold
Figure 9: Sample F1 levels (top), and track completion rates by methods tar- geting Es, evaluated on EFI (bottom).

---

Combining DIVA and UED. While PLR and ACCEL struggle on our evaluation domains, they still have utility of their own, which we hypothesize may be compatible with DIVA's. As a preliminary experiment to evaluate the potential of such combination, we introduce DIVA+ which still uses VA to generate diverse training samples via QD, but additionally uses PLR- to define a new distribution over these levels based on approxi- mate learning potential. Instead of randomly sampling levels from Ev. the PLR evaluation mechanism sam- ples levels from the DIVA-induced distribution over two differ- ent archives generated by DIVA: (1) an archive that is slightly misspecified (see Aspenndpp B.3 details). and (2) the archive used in our main results. From Fig ure T0] we that while performance does not sig- nificantly improve for (2), the combination of DIVA and PLR- is able to significantly improve performance on (1), and even statistically match the original DIVA results. These results highlight the potential of such hybrid (QD+UED) semi-supervised environment design (SSED) approaches, a promising area for future work.
Track Completion
0.5
DIVA,
DIVA,
Success Rate
0.4
DIVA,
the archive. We perform experiments
DIVA
0.3
0.2
0.1
0.0
80%
90%
100%
Completion Threshold
Figure 10: DIVA+ results compared to DIVA, for (1) misspecified, and (2) well- specified archives, evaluated on Es.
6 Related work
Meta-reinforcement learning. Meta-reinforcement learning methods range from gradient-based approaches (e.g. MAML) [19], RNN context- based approaches [12 II (e.g. RL²), and the slew of emerging works utilizing transformers 220 [ 21]. We use VariBAD [10), a state-of-the-art art context variable-based approach that extends RL 2 by using variational inference to incorporate task uncertainty into its beliefs. HyperX [22), extension that uses reward-bonuses, was not found to improve performance on our domains. In each of these works, the training distribution is given; none address the problem of generating diverse training scenarios in absence of such a distribution.
Procedural environment generation. Procedural (content) generation (PCG PG) (6) is vast field. Many RL and meta-RL domains themselves have PG baked-in (c.g. ProcGen [23). Meta-World 240, Alchemy [ and XLand (5)) Each of these works rely on human engineering produce levels with meaningfully diverse features. A related stream of works apply scenario generation to robotics- some works essentially perform PCG 25 26 while others integrate more involved search mechanics [27 22 29 33). One prior work [31 defines a formal but generic parameterization for applying PG generate meta-RL tasks. It is yet to be shown, however, if such an approach can scale to domains with vastly different dynamics, and greater complexity.
Unsupervised environment design. UED approaches-which use behavioral metrics to automati- cally define and adapt a curriculum of suitable tasks for agent training- the frontier of research on open-peeneete endennnss...s.. The recent stream of open-ended agent/environment co-evolution works (c.g. was kickstarted by the POET [35] 36 algorithm. The "UED" term itself originated in PAIRED which uses the performance of "antagonist" agent to define the curriculum for the main (protagonist) agent. PLR (7 introduces an approach for weighting training levels based on learning potential, using various proxy metrics to capture this high-level concept. [I7] introduces PLR which only trains on levels that have been previously evaluated, and thus enabling certain theoretical robustness guarantees. AdA 5 uses PLR as cornerstone of their approach for generating diverse training levels for adaptive agents in a complex, open-ended task space. ACCEL [9] borrows P scoring procedure, but the best-performing solutions instead mutated, the buffer not only collects and prioritizes levels of higher learning potential, but evolves them. We use ACCEL as our main bascline because it has demonstrated state-of-the art results on relevant domains, and like DIVA, evolves a population of levels. The main algorithmic differences between ACCEL and DIVA are that ACCEL (1) performs additional evaluation rollouts to produce scores during training and (2) uses 1-d buffer instead of DIVA's 's multi-dimmensioal archive. PLR serves as a secondary baseline in this work; its non-evolut nature makes it useful comparison to DR.
32
33 34]
Scenario generation via QD. A number of recent works apply QD to simulated environments in order to generate diverse scenarios, with distinct aims. Some works, like DSAGE 377, uses QD to develop diverse levels for the purpose of probing pretrained agent for interesting behaviors. In another line of work applies QD to human-robot interaction (HRI), and ranges from generating

---

diverse scenarios (38], to finding failure modes in shared autonomy systems [39 and human- numan-aware planners 44) DIVA's application of QD inspired by these approaches, as they produce meaningfully diverse environment scenarios, but no prior work exists which applies QD to define task distribution for agent training, much less adaptive agent training, or overcoming difficult parameterizations in open-ended environments.
7 Discussion
The present work enables adaptive agent training open-ended environment simulators by in- tegrating the unconstrained nature of unsupervised environment design (UED) approaches, with the implicit supervision baked into procedural generation (PG) and domain randomization (DR) methods. Unlike PG and DR, which requires domain knowledge to be carefully incorporated into the environment generation process, DIVA is able to flexibly incorporate domain knowledge, and can discover new levels representative of the downstream distribution. And instead of relying on behavioral metries to infer general, ungrounded form of "learning potential", like UED which becomes increasingly unconstrained and therefore less useful a signal as environments become more complex and open-ended DIVA is able to directly incorporate downstream feature samples to target specific, meaningful axes of diversity. With only a handful of downstream feature samples to set the parameters of the QD archive, our experiments (Section 5) demonstrate DIVA's ability to outperform competitive baselines compensated with three times as many environment steps during training.
In its current form, the most obvious limitation of DIVA is that, in addition to assuming access to downstream feature samples, the axes of diversity themselves must be specified. However, we imagine these axes of diversity could be learned automatically from set of sample levels, or selected from a larger set of candidate features; it may be possible to adapt existing QD works to automate this process in related settings [41]. The present work also lacks more thorough analysis of what constitutes good archive design. While some amount of heuristic decision-making is unavoidable when applying learning algorithms to specific domains, promising future direction would be to study how to approach DIVA's 's archive design from a more algorithmic perspective.
DIVA currently performs QD iterations over the environment parameter space defined by (O), where each component of the genotype represents some salient input parameter to the simulator. Prior works in other domains (e.g. [42) have demonstrated QD's s ability to explore the latent space of generative models. One natural direction for future work would therefore be to apply DIVA to neural environment generators (rather than algorithmic generators), where θ would instead correspond to the latent input space of the generative model. If the latent space of these models is more convenient to work with than the raw environment parameters- e.g. due to greater smoothness with respect to meaningful axes of diversity-thh may help QD more efficeintly discover samples within the target region. Conversely, DIVA's ability to discover useful regions of the parameter space means these neural environment generators do not need to be "well-behaved", or match a specific target distribution. Since these generative models are also likely to be differentiable, DIVA can additionally incorporate gradient-based QD works (e.g. DQD (T) to accelerate its search.
Preliminary results with DIVA+ demonstrate the additional potential of combining UED and DIVA approaches. The Fl transfer results (i.e. DIVA outperforming ODS trained directly on Es) further suggest that agents benefit from flexible incorporation of downstream knowledge. In future work, we hope to study more principle integrations of UED and DIVA-like approaches, and to more generally explore this exciting new area of semi-supervised environment design (SSED).
More broadly, now equipped with DIVA, researchers can develop more general-purpose, open-ended simulators, without concerning themselves with constructing convenient, well-behaved parameteri- zations. Evaluations in this work required constructing our own contrived paramterizations, since domains are rarely released without carefully designed parameterizations. It is no longer necessary to accomodate the assumption made my DR, PG, and UED approaches -thht either randomization over the parameter space should produce meaningful diversity, correspond to mcaningful learning potential. So long as diverse tasks are possible to generate, even sparsely distributed within the paramter space, QD may be used to discover these regions, and exploit them for agent training. Based on the promising empirical results presented in this work, we are hopeful that DIVA will enable future works tackle even more complicated domains, and assist researchers in designing more capable and behaviorally interesting adaptive agents.
that all forms of level difficulty ought to
10

---

8 Reproducibility statement
The source code, along with thorough documentation for reproducing each result this paper, is publicly available Githut Even without this code, researchers should be able to fully reproduce the algorithm from the details in the main body, the pseudocode provided in Appendii A] and training details (hyperparameters and hardware information) provided in Appendix E
Ethics statement
Like all fundamental technologies, this work has the potential to be misapplied for malicious purposes. The authors do not believe, however, that the methods introduced in this work present significant or unique risk for misuse or abuse. The authors intend for DIVA to be applied to use-cases that have the best interests of humanity (including concern for the carth and other sentient creatures) at heart.
10 Acknowledgements
This work was partially supported by NSF CAREER (#2145077) and the DARPA EMHAT project. We thank Tjanaka et al., the developers of pyribs [43. whose library served as the basis for our QD implementations. We thank Zintgraf et al., the authors of VariBAD [10], whose codebase served as the basis for our meta-RL agent. We thank Jiang et al. and Parker-Holder et al., the authors of PLR 7 and ACCEL [9], respectively, for their implementations which served as the basis for our UED baselines. We specifically thank Mingi Jiang for answering questions related to the PLR codebase in the early stages of development, and Varun Bhatt for helpful discussion at various stages of this work.
References
[1] X. Wang, S. Wang, X. Liang, D. Zhao, J. Huang, X. Xu, B. Dai, and Q. Miao, "Deep reinforcement learning: A survey," IEEE Trans. Neural Netw. Learn. Syst. vol. 35, pp. 5064-5078, Apr. 2024.
[2] K. Sivamayil, E. Rajasekar, B. Aljafari, S. Nikolovski, S. Vairavasundaram, and I. Vairavasundaram, "A systematic study on reinforcement learning based applications." Energies, vol. 16, 1512, Feb. 2023.
[3] R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktäschel, "A survey of zero-shot generalisation deep reinforcement lcarning," jair, vol. 76, pp. 201-264, Jan. 2023.
[4] Beck, R. Vuorio, Z. Liu, Z. Xiong, Zintgraf, C. Finn, and S. Whiteson, "A survey of meta- reinforcement learning." arXiv [cs. Jan. 2023.
[5] Bauer, K. Baumli, F. Behbahani, A. Bhoopchand, N. Bradley- -Shmicg, M. Chang, N. Clay, A. Collister, V. Dasagi, L. Gonzalez, K. Gregor, Hughes, S. Kashem, Thompsoo H. Openshaw, Parker- Holder, S. Pathak, N. Perez-Nieves, N. Rakicevic, T. Rocktäschel, Y. Schroecker, S. Singh, J. Sygnowski, K. Tuyls, S. York, A. Zacherl, and I.. M. Zhang, timeesete adaptation in open-ended task space," in Proceedings the 40th International Conference Machine Learning (A. Krause, Brunskill, K. Cho. Engelhardt, Sabato, and J. Scarlett, eds.), vol. 202 of Proceedings of Machine Learning Research, pp. 1887-1935, PMLR, 23-29 Jul 2023.
[6] N. Shaker, Togelius, and M. Nelson, Procedural Content Generation in Games. Computational Synthesis and Creative Systems, Springer, 2016.
M. Jiang, E. Grefenstetie, and T. Rocktäschel, "Prioritized Level Replay." in Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event (M. Meila and T. Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research, pp. 4940 49400 PMLR, 2021.
[8] M. Dennis, N. Jaques, Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine, "Emergent complexity and zero-shot transfer via unsupervised environment design," Advances neural information processing systems, vol. 33, pp. 13049-13061, 2020.
Parker-Holder, M. Jiang, Dennis, and others, "Evolving curricula with regret- based environment design," International Conference on Machine Learning, 2022.
[10] L. M. Zintgraf, K. Shiarlis, M. Igl. S. Schulze, Gal, K. Hofmann, and S. Whiteson, "VariBAD: Very Good Method for Bayes-adaptive Deep RL via Meta-learning," in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net, 2020.
'Codebase: https://gthb..cm///bbyyyttttl//t

---

[11] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever. and P. Abbeel, "RL²: Fast Reinforcement Learning via Slow Reinforcement Learning," Leamnnn, arXiv:1611. 1.02779 [cx, statj. Nov. 2016.
[12] J. Wang, Z. Kurth-Nelson, H. Soyer. J. Z. Leibo, D. Tirumala, R. Munos, C. Blundell, D. Kumaran, and M. M. Botvinick, "Leamning to reinforcement learn"" in Proceedings of the 39th Annual Meeting of the Cognitive Science Sociery, CogSei 2017, London, UK, 16-29 July 2017 (G. Gunzelmann, A. Howes, T. Tenbrink, and E. J. Davelaar, eds.), cogntiieseienncssiittyorgg 2017.
[13] L.. Zintgraf, K. Shiarli, V. Kurin, K. Hofmann, and S. Whiteson, "Fast context adaptation via meta-learning" in International Conference on Machine Learning, PP. 7693-7702, PMIR, 2019.
[14] K. Rakelly, A. 7hou, C. Finn, S. Levine, and D. Quillen, "Efficient Off-policy Meta-reinforcement I. .earning via Probabilistic Context Variables." in Proceedings of the 36th International Conference on Machine Learning, JCMI, 2019, 9-15 June 2019, Long Beach, California, USA (K. Chaudhuri and R. Salakhutdinov, eds.). vol. 97 of Proceedings of Machine Learning Research, pp. 5331-5340, PMLR, 2019.
[15] M. C. Fontaine and S. Nikolaidis, "Differentiable Quality Diversity," in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual (M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, eds.). pp. 10040-10052, 2021.
[16] J. Mouret and J. Clune, "Illuminating search spaces by mapping clites," " CoRR, vol. 0tt//504..04990 2015.
[17] Jiang, M. Dennis, Parker-Holder, N. Foerster, E. Grefenstette, and T. Rocktäschel, "Replay-guider Adversarial Environment Design," in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Procesing Systems 2021, NeurlPS 1021, December 6-14, 2021, virtual (M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, eds.), pp. 1884-1897, 2021.
[18] J. J. Wang, M. King, N. P. M. Porcel. Z. Kurth-Nelson, T. Zhu, C. Deck, P. Choy. M. Cassin, M. Reynolds. H. F. Song, al. "Alchemy: A benchmark and analysis toolkit for meta-reinforeement learning agents," in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[19] C. Finn, P. Abbeel, and S. Levine, "Mooel-agnostic Meta-lcarning for Fast Adaptation of Deep Networks," in Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW Australia, 6-11 August 2017 (D. Precup and Y. W. Tch, eds.), vol. 70 of Proceedings of Machine Learning Research, pp. 1126-1135, PMLR, 2017.
[20] L. C. Melo, "Transformers are Meta- RittaRRRiffrremeet Learners," in Proceedings of the 39th International Conference on Machine Learning (K. Chaudhuri. S. Jegelka, L. Song. C. Szepesvari, G. Niu, and S. Sabato, eds.), vol. 162 of Proceedings of Machine Learning Research, pp- 15340-15359, PMIR, 2022.
[21] J. Grigsby, L. Fan, and Y. Zhu, "AMAGO: Scalable in-context reinforcement learning for adaptive agents," in The Twelfth International Conference on Learning Representations, 2024.
[22] L. M. Zintgraf, L. Feng, C. Lu, M. Igl. K. Hartikainen, K. Hofmann, and S. Whiteson, "Exploration in Approximate Hyper-state Space for Meta Reinforcement Learning," " in Proceedings of the 38th International Conference on Mackine Learning. ICML 2021, 18-24 July 2021, Virtual Event (M. Mcila and T. Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research, pp.
PMLR, 2021.
[23] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman, "Leveraging Procedural Generation to Benchmark Reinforcement Learning." in Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, vol. 119 of Proceedings of Machine Learning Research, pp. 2048 2056, PMLR, 2020.
[24] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, "Meta- Merta-Wrld: A Benchmark and Evaluation for Multi-task and Meta Reinforcement Learning." 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings (L. P. Kaelbling. D. Kragic, and K. Sugiura, eds.), vol. 100 of Proceedings of Machine Learning Research, pp. 1094-1100, PMI.R, 2019.
[25] J. Arnold and R. Alexander, "Testing autonomous robot control software using procedural content gen- eration," in Lecture Notes in Computer Science, Lecture notes in computer seience, pp. 33-44, Berlin, Heidelberg:
Springer Berlin Hcidelberg, 2013.
[26] D. J. Fremont, T. Dreossi, S. Ghosh, X. Yue, A. L. Sangiovanni-Vincetllii Vincentelli and S. A. Seshia, "Scenie: language for scenario specification and scene generation." in Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2019, (New York, NY, USA),
p. 63 78, Association for Computing Machinery, 2019.
[27] G. E. Mullins, P. G. Stankiewicz, R. C. Hawthorne, and S. K. Gupta, "Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles." J. Syst. Softw, vol. 137, pp. 197 197-215, Mar. 2018.
12

---

[28] Y. Abeysirigoonawardenn, F. Shkurti, and G. Dudek, "Generating adversarial driving scenarios in high- fidelity simulators." in 2019 International Conference on Robotics and Automation (ICRA), pp. 8271-8277, IEEE, May 2019.
[29] A. Gambi, M. Mueller, and G. Fraser, "Automatically testing self alf-ddriing cars with search- based procedural content generation." " in Proceedings the 28th ACM SIGSOFT International Symposium Software Testing and Analysis, (New York, NY, USA), ACM, July 2019.
[30] Y. Zhou, S. Booth, N. Figueroa, and J. Shah, "Rocus: Robot controller understanding via sampling," in Proceedings of the 5th Conference Robot Learning (A. Faust, D. Hsu, and G. Neumann, eds.), vol. 164 of Proceedings of Machine Learning Research, pp. 850-860, PMLR, 08-11 Nov 2022.
[31] T. Miconi, "Procedural generation of meta-reinforcement learning tasks," Feb. 2023.
[32] T. Gabor, A. Sedlmeier, M. Kiermeier, T. Phan, M. Henrich, M. Pichlmair, B. Kempter, C. Klein, H. Sauer, R. S. Ag, and Wieghardt, "Scenario co-evolution for reinforcement learning on a grid world smart factory domain," in Proceedings of the Genetic and Evolutionary Computation Conference, (New York, NY, USA),
ACM, July 2019.
[33] D. M. Bossens and D. Tarapore, "QED: Using
to evolve resilient robot
swarms," IEEE Trans. Evol. Comput., vol. 25, pp. 346-357, Apr. 2021.
[34] A. Dharna, J. Togelius, and L. B. Soros, "Co-generation of game levels and game game-playing agents." in Pro- ceedings of the Sixteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AllDE' 20, AAAI Press, 2020.
[35] R. Wang. J. Lehman, J. Clune, and K. O. Stanley, "Paired open-ended trailblazer (POET); Endlessly generating incrcasingly complex and diverse learning environments and their solutions," " arXiv Jcs.NEl, Jan. 2019.
[36] R. Wang, J. Lehman, A. Rawal, J. Zhi, Y. J. Clune, and K. Stanley, "Enhanced POET: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions," " in Proceedings of the 37th International Conference on Machine Learning (H. D. III and A. Singh, eds.), vol. 1 19 of Proceedings of Machine Learning Research, pp. 9940-9951, PMIR, 13-18 Jul 2020.
[37] V. Bhatt, B. Tjanaka, M. Fontaine, and S. Nikolaidis, "Deep surrogate assisted generation of environments," Advances in Neural Information Processing Systems, vol. 35. pp. 37762-37777 2022.
[38] D. Gravina, A. Khalifa, A. Liapis, J. Togelius, and G. N. Yannakakis, "Procedural content generation through quality diversity," in 2019 IEEE Conference Games (CoG), pp. 1-8, feeexplore.icce.org Aug. 2019.
[39] M. C. Fontaine and S. Nikolaidis, "A Quality Diversity Approach to Automatically Generating Human- robot Interaction Scenarios in Shared Autonomy." in Robotics: Science and Systems XVII. Viriual Event, July 12-16, 202/ (D. A. Shell, M. Toussaint, and M. A. Hsich, eds.), 2021.
[40] M. C. Fontaine, Y. Hsu, Y. Zhang, B. Tjanaka, and S. Nikolaidis, "On the Importance of Environments in Human-robot Coordination," " in Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021 (D. A. Shell, M. Toussaint, and M. A. Hsieh, eds.), 2021.
[41] I.. Grillotti and A. Cully, "Unsupervised behavior discovery with quality-diversity optimization," IEEE
Trans. Evol. Comput., 2022.
[42] A. Khalifa, P. Bontrager, Earle, and J. Togelius, "PCGRL: Procedural Content Generation via Reinforce- ment Learning." in Proceedings of the Sixteenth AAAI Conference on Artificial Intelligence and Inteructive Digital Entertainennt, ALIDE 2020, viriual, October 19-23, 2020 (L. Lelis and D. Thue, eds.). pp. 95-101, AAAI Press, 2020.
[43] B. Tjanaka, M. C. Fontaine, Y. Zhang, S. Sommerer, and others, "pyribs: A bone- python library for quality diversity optimization." 2021.
13

---

Appendix
A
Algorithmic details
Algorithm. The pseudocode below walks through the entire training process for DIVA in abstract.
All of the new components that DIVA introduces is written in green, and all DIVA+ modifications in blue. Original VariBAD training steps are in black, and all inline comments are in orange.
Algorithm 2 DIVA (detailed)
1: # Initialize VariBAD and QD components
2: iiiit_toliic()>> (); Sme: faee init_vae
Initialize VariBAD components Initialize VariBAD buffers
3: D,, DvA
4: θ - {(0,) n -()))
& Sample initial solutions from space of valid genotypes
5: Fu =f(0)). [f(0)......fC
f(Om. + (
b Compute cnv. fcatures Compute env. objectives Init. archive to contain both Fo and target features Fş
6: Jo [(()),.... [m compute_oby objectives(
7: G initialize archive (Fo. Fb)
8: G insert_solutions (G, (0., Jo))
Add random solutions archive
9: stage 1: discover target region
for range(NNs) do
11: G f- QD AID_UPDATE(G, J, Bop)
b Perform QD update (batch size Bon) to populate archive Move mask gradually towards target region
12: M + ueddue__saaaaeee maaaammaassaa
stage 2: populate target region
14: G + update_ update_archive__
Create final archive (target region) before S2 updates
15: for 1 in range(NNs)) do
16: G QD_UPDATE(G, Bop)
P Perform QD update populate target region
17: Meta-learning over QD archive
18: for i in ravge(Nvv) do
19:
0' ~ Po(O)
& Sample solution 0' from QD archive using approximated target density from Fs
20: M generate_environnet environment("")
Generate training environment from solution 0'
# Produce meta-RL rollouts
22:
I + perform_policy. cttformppoiccy.roloott.....
wowplyrooollollleeeeeeeeeeeeeeee (M, TT)
23:
D., DvAE + tod_fto_buffers (D,, DVAE,
24:
# Update VAE and policy
25:
fen, faee varibad saepaddvvee_ppppe serribad_vaeeuudateeffff. fus, DvAR)
26:
if siter__ae__reetriinnng then
D.)
28:
# Perform DIVA+ QD updates
29:
if DIVA+ and (i % qd_ appate_intervee 0) then
for udd_uudates_pe__teee do G - QD CD__PDDATEC 0, Bon)
31:
b Perform QD update with PLR objective
Algorithm 3 QD update
1: # Perforr single QD update archive 9 with batch size B.
2: function q UPDATTE(G J, B)
3: Bxxn (.,...,
Get mutated batch of solutions
4:
FBxh = (0.)...... .0.)............... [00) t compute_featurest
P Compute env. features
5: Bx1 [J(O,), (00)] t compute_objectives((
Compute env. objectives P Add new solutions to archive if they elites
6: G' add coddd_solutiosdde F,,)))
7: return G'
Details on the two-stage QD updates Here we provide more details on the process described in Section /eccionn 4 Hyperparameters Nsi and Nş2 are set to define the number of QD updates to perform in each stage (see Appendix D). In proportion to how many updates in S1 have elapsed, if the sample mask is enabled, the mask moved a linear pace from encapsulating the full S1 archive, to covering only the target region. We also set a hyperparameter, NsM (see Appendix D), which
14

---

specifies the minimum number of solutions which must exist within the mask's s new bounds for it to be updated. This is to ensure the mask never outpaces the search process. The mask was only found to be necessary in the AL.CHEMY environment. In SI we sample solutions uniformly from within the mask. S2, we begin sampling from the discretized target density distribution approximated from the downstream feature samples. Two stages are used for RACING as well, since many initial samples fall outside of the target region, but masking was not found to be necessary. The sample mask has relatively straightforward implementation for MAP-Elites Elites, which we use for ALCHEMY'S discrete genotype (and GRIDNAV, where no mask required). Since MAP-Elite updates entail performing mutations solutions directly sampled from the archive, the mask is implemented to only consider solutions that fall within the mask bounds. However, since the CMA-ES- --seee emitter use for RACING operates by sampling from parameterized distribution, instead of sampling from the archive directly, the mask would need to be applied to these parameters instead of the archive.
B
Domain details
B.1 GRIDNAV
GRIDNAV features.
The following features are defined for the GRIDNAV environment:
Table I: GRIDNAV features.
Name
Abbr.
Description
XPOSITION
XP
I poxition of the goal.
YPOSITION
YP
y position of the goal.
B.2 ALCHEMY
ALCHEMY features.
We defined the following features for the ALCHEMY environment:
Table ALCHEMY features.
Name
Abbr.
Description
MANHATTANTOOPPTIMA
MTO
Average Manhattan distance between all stones (across all trials) to optimal state. Averuge Euclidean distance between all pairs of stones (across all trials).
STONETOSTTOEEDISAANEE
STSD
GRAPHNUMBOTTTLEEECKS LATENNSTAATEDDVVRSITYY
GNB
The number of bottlenecks the graph topology.
LSD
The 'diversity' of latent stone states (across all trials). Diversity is calculated as standard deviation of each latent state coordinate across all stones. First potion location (first trial, first potion), as parity measure.
PARITYFIRSTPOTION
PFP
PARITYFIRSTSTONE
PFS
First stone location (first trial, first stone), parity measure.
POTIONEITFECTDIIEESSTY
PED
The 'dversitty' of the potion effects (across all trials). Diversity is calculated as the standard deviation of each potion effect coordinate across all potions. Potion permutation.
POTIONPERMUTATION
PP
POTIONREFLECTION
PR
Potion reflection.
STONEREPLECTION
SRE
Stone reflection.
STONEROTATION
SRO
Stone rotation.
STONETTSSTONNEDSSTTGGVVAAAAAAAA
STSDV
Variance of the distances between stones (across all trials).
Figure 11 contains the feature distributions for the structured and unstructured environment param- eterizations on ALCHEMY, computed over 100 feature samples. [Figure 14 shows the covariance between feature values for RACING, computed over 100 feature samples.
15

---

AMTO
ASTSD
LSD
PFP
PFS
Density
PED
SRE
SRO
STSDV
Eis
Ey
Feature Values
Figure 11: ALCHEMY all feature distributions.
Covariance: Es
Covariance: Ev
AMTO
675
150
400
435
0.0
170
400
500
100
Figure 12: ALCHEMY measure covariances.
Archive hyperparameters for ALCHEMY determined based some knowledge about the domain, as well as the feature distributions (Figure 13). We noticed a major deviation between Eu and Es in the feature LATENTS DIVERIVERSTYY (LSDD), and even greater in MANHATTANTOOPTIMA. (MTO). These two constitute the initial dimensions of the archive, and we found the sample mask updates be crucial reach and fill the target region (see Figure 15). We use PARITYFIRSTSTONE (PFS) in the second stage to encourage more diversity once the target is reached; it is excluded from the first stage, which focused on simply reaching the target. The archive for the first stage of shape [100, 300, 17, corresponding to LSD, MTO, and PFS. The second stage shape is [150, 150. 5]. We found this archive produce diverse enough solutions, evidenced by the number and spread the target region, so we used this setting to train our DIVA agents. The only objective we found useful for ALCHEMY was a slight bias for newly generated solutions, which we also used for GRIDNAV. We hypothesize this prevents the archive from getting "stuck" with suboptimal set of solutions, in absense of other objectives.
B.3 RACING
RACING features. See Table 13 for all features defined on RACING. Figure 13 contains the feature distributions for the structured and unstructured environment parameterizations RACING, computed over 100 feature samples. Figure 14 shows the covariance between feature values for RACING, computed over 100 samples.
Archive hyperparameters for RACING were determined through trial and error, by viewing the samples produced by the archives at the end of the QD updates, as well the target coverage metrics. After few iterations, it became clear that Total Angle Change ToTALANGLECHANGER (TAC) was the most useful feature, and absolute covariance (see Figure 14).
tried pairing it with number of others, prioritizing other features with low
The best performing archive used by DIVA on RACING uses TAC and CX as its features, and used measure alignment objective over CY and VY. The measure alignment objective rewards solutions for having measure values over the specific measures that similar to the target distribution. We also
16

---

Table 3: RACING features.
Name
Abbr.
Description
AREATOLENGTHRATI
ATLR
The ratio of enclosed area to curve length.
AVERAGECURVATURE
The average curvature midpoints of Beziér segments.
CENTEROFMASSX
The of x position over the curve.
CENTEROFMASSY
CY
The center of mass y position over the curve.
CURVEDISTANCESS VARIIIS VARIANCE
CDV
The variability in distances between successive points.
CURVELENGTH
CL
The total length of the Beziér curve.
ENCLOSEDAREA
The enclosed by the Beziér
MEDIANX
MX
The median x position over the curve.
MEDIANY
MY
The median y position over the curve.
SIGNFFCANTANGLCCHNNEES
SAC
The sum of significant angle changes across the curve.
TOTALANGLECCANGES
TAC
The total change in angle across the curve.
TOTALCURVATURE
The total curvature
each segment and sum them up.
VARIANCEX
VX
The variance of the a poxitions over the curve.
VARIANCEY
VY
The variance of the positions over the curve.
Density
MX
MY
SAC
TAC
TC
VX
VY
Es
Eg
Feature Values
Figure 13: RACING all feature distributions.
found that randomly sampling these objective values according the target distribution provided some additional support covering the target region efficiently.
The slightly "misspecified" archive was chosen because its solutions generated some diversity, but not as much as aforementioned one. This archive uses TAC and ATLR as its features, and uses a measure diversity objective just CY. Instead of prioritizing alignment to the target distribution, the diversity objective samples a handful of solutions from the archive, and uses the current solutions deviation from these as its objective.
We use final archive dimensions of 500 x 500 for both S1 and $2
C Ablation analysis
Sample mask ablation. Figure 15 shows the benefit of updating the sample mask bounds during the first archive filling stage ALCHEMY. Not only does this approach produce significantly more total archive solutions, but more importantly, progress towards filling the target region specifically is accelerated.
D Hyperparameter sensitivity analysis
Varying QD mutation rate. We perform an ablation on the QD mutation rate, which is the probability that a given gene will be mutated (for MAP-elites). We perform this ablation on ALCHEMY because the its search is the most challenging of the three environments we consider (it is the sole environment that required a longer S1 and the sample mask trick for accelerating to accelerate the search). We see from Figure Th ALCHEMY results are not very sensitive the setting of the mutation rate.
17

---

Covariance: Es
Covariance: Eg
40
40
63V
Con
05-
10
-0.5
Figure 14: RACING measure covariances.
10
#
Target Solutions
x10
Archive Solutions
2.0
# of Solutions
of Solutions
1.5
1.0
2
0.5
Masking No masking
0.0
0-
2
QD Updates (51)
x10'
QD Updates (S1)
x10'
Figure 15: ALCHEMY sample mask ablation curves. This specific result is the result of two seeds instead of five, we found the variance to be very low for this ablation (validated across other parameter settings).
Varying number of QD updates. We perform a similar ablation to test how robust DIVA is to the number of QD updates performed. We from Figure 17 ALCIEMY results suffer somewhat from fewer updates (e.g. for only 10k in each stage), still significantly outperform baselines in QD updates produces solutions, which generally
each case. The trend is clear, however: translates to better performance, even slightly.
Varying number of downstream samples. Next we test how robust DIVA is to the number of downstream samples used compute the target distribution. In Figure 18 we see that, despite the errors increasing with fewer samples, DIVA still significantly outperlorms baselines with as few as five samples.
E
Training details
E.1 DIVA hyperparameters
Table displays the hyperparameters used for DIVA across all domains.
on NIRS computation The initial QD population (ng) is implemented such that the first set of QD updates simply generates Ro random levels from Ev, before performing the actual mutations (for ME) or intelligent sampling (for ES). Thus, the formula we use for computing NTRS. the total reset steps provided to DIVA (see Table 4), which we use to compare the extra steps provide PLR- and ACCEL (discussed in Section 5). does not include no; is simply the product of the batch size and the total number of QD iterations.
18

---

Final Return by Mutation Rate
Target Solutions by Mutation Rate
8000
Final Episode Return
Target Solutions
6000
4000
Method
DIVA
ODS
2000
PLR-
0.005
0.04
0.08
0.005
0.01
0.04
0.08
Mutation Rate
Mutation Rate
Figure 16: Effect of varying QD mutation rate in ALCHEMY. Left: The returns for the final episode by mutation rate, after training on archives produced with each mutation rate. Right: The final number of solutions in the archive after performing QD updates with each mutation rate. This result was produced by running three different seeds for each mutation rate.
Returns QD Updates
Target Solutions by # QD Updates
x10
Final Episode Returns
1.25
Solutions
1.00
1-
0.73
Method
DIVA
0.50
2-
0.25
PLR-
0.00
10000
20000
40000
120000
10000
20000
40000
120000
(log scole)
# QD Updates
[Bg scals)
# QD Updates
Figure 17: Effect of varying the number of QD updates in ALCHEMY. Left: The returns for
the final episode by number of QD updates in each stage (Ns1 Ns2). Right: The final number of solutions in the archive after performing each number of QD updates. This result was produced by running three different seeds for each setting.
E.2
VariBAD hyperparameters
Table 5 displays the hyperparameters used for VariBAD across domains.
E.3
Baseline hyperparameters
E.3.1
PLR
Table 6 displays the hyperparameters used for PLR across all domains.
E.3.2
ACCEL
ACCEL uses the same hyperparameters as PLR (see Table 6), combined with the same evolutionary hyperparameters used for DIVA's QD archive (see Table 4).
E.4
Computational details
All results were produced on handful of Titan or Xp GPUs. Environments were parallelized across multiple CPU cores to accelerate training. While the experiment time varies by method and environment, most experiments take less than a day to run to completion. PLR and ACCEL take longest, they required twice many environment steps the other methods-os the two latter domains, these methods take well over a day to run to completion.
19

---

Returns by Feature Samples
A
LATERTSSATTDIVBBBTTT
X10-
x10
Final Episode Retums
Sample Means MAE
Sample Varianoss MAE
Sample Meana MAE
2.0-
Sample Variamcss MAE
0.6
4-
2.0
Sample Variances
1.5
Method
1.3
DIVA
0.4
ODS
0.2
0.5
0.5
50
100
500
10
50
100
500
5
10
25
100
500
Dsg xcls
Feature Samples
Feature Samples
Feature Samples
Figure 18: Effect of varying number of samples in ALCHEMY. Left: DIVA evaluation returns for the final episode by number of downstream samples, after training on archives produced with by using each number of samples to produce archive bounds and prior. Center/Right: Errors for mean and variance parameters of the normal distribution based on number of samples used for computation; for MANHATTANTOOPT for each hyperparameter setting.
and LATENTSTTATEDDVERSSTII For all plots, five seeds were used
Table 4: DIVA settnng... settings.
Name
Description
Value
Nsi
Number of QD updates stage Number of QD updates in stage $2 (Effective) total QD updates: Nsi Nsa Initial population size Number of QD solution emitters Sampling batch size of each QD emitter
0 80,0000 50,000
Nsz
100,000* / 33,0000* 200,0000*
Ngp
100,000 110,000* 250.000
1,000 2,0000
B.
85
Bon
(Effective) QD butch size per update B.)
40 25*
Nys
(Effective) total reset steps (Nop x Bqp) Type of QD emitter
4.0 x 10 /2.75 x 10 11000111011111111111111111111111 /1.0 x 10
qd_emm emitter
MAP-Elites (ME)** CMA-ES (CMA)
Mutation percentage for ME emitter Initial sigma for ES emitter
0.1 /0.02
Es
0.1%
ABsM
Anneal sample mask bounds during S1
False True 40" 1000
NsM
Minimum solutions in sample mask
Jnew
Enable objective for slightly biasing new solutions
True t "/aalse
JMD
Enable measure diversity objective
False
JMA
Enable measure alignment objective Enable randomize objective
False True
Jand
False T* True
Ng
Total meta meta-training training env. steps (ref. from Table 5 Additional env. steps provided UED baselines (Effective) total UED emv. steps (Effective) Ratio of add. steps for UED vs DIVA
4.8 10t /2.0 x 107
NE-
9.6 10 /4.0 x 107
N E'
14.4 x 10 1 */.00 x 10
NE- /N
2.4 3.55* /4.0
GRIDNAY, ALCHEMY, RACING
20

---

Table 5: VariBAD hyperparameter settings.
Name
Description
Value
fx
Optimizer epsilon for policy
le-8
Discount factor for rewards State embedding dimension for policy Latent embedding dimension for policy Normalize state input
0.99
policy_state_ _dim
64
policy_latent_ carttt__m______ dim
64
policy_norm_state
True
policy_norm_latent
Normalize latent input Normalize belief input Normalize rewards for policy Hidden layers for policy network Activation function for policy Initializution method for policy Optimizer for policy Learning rate for policy Initial standard deviation for policy Value loss coefficient for policy Entropy coefficient for policy
True
policy.norm.beliet
True
policy_norm_rex
Truet False
policy_layers
[128, 128]** / [128]*
policy_activation
tanh
policy.init_method
norme
policy_optimizer
adam
policy_lr
0.0007
policy_init_sd
1.0
policy_val_loss_com _coff
0.5
policy_entropy_coo
0.01
policy_use_gae
Use generalized advantage estimation GAE parameter for policy Maximum gradient norm for policy
True
policy_tau
0.95
Baataaax_ra___oor_aeaaaaa norm volicy_max_grane
0.5
NvB
Total number of VariBAD learning updates
1.0 x 10* to 1 x 10*
besicy_numstepp
Number of environment steps per update
60* /40* /500
_um_processes Ng
Number of parallel environments
/4
Total env. steps (product of prior three)
4.8 x 10 ** 2.0 x 10"
ppo_num_epocce
Number of epochs per PPO update
18"
ppo_num_nee
hoummmiiibatch
Number of minibatches for PPO Use Huber loss for PPO Use clipped value loss for PPO Clipping parameter for PPO
4
ppo ppo_hueer__ 1osss ppo_
True
ppo_clip_val_lers
True
ppo_clip-ppraa
0.05 /0001
Learning rate for VAE
0.001
Now
Size of VAE buffer
5,000 100,0000 1
BAs
Number of trajectories
VAE update
25t* 10
precollect_ len
Frames to pre-collect before training
5,000
num num_vae_ updates
pom__ae__pddaaed
Number of VAE update steps per iteration
3
pretrain_len
Number of VAE pre-training updates Weight for KL term
0
kl_weight
0.01
action_emb_eet
Action emhedding size for VAE
8
state_ sittte__mb___t size
State embedding size for VAE Reward embedding size for GRU hidden size in encoder
16
sew_emb_size
enc_gru_.iiden_sizz
128
latent_dim
Latent dimension for VAE Reward coefficient Layers for reward decoder Use multihead for reward prediction Reward prediction rype
10 5
rew_loss_coeff
1.0
rew_dec_layers
[64. 32]
rew new_multihead
False
rew_pred_tyy
hemoulli
KL to Gaussian prior Backprop RL loss through encoder VAE loxs coefficient
False
lillos__hhru enc vae mae_loss_coef
False
1.0
GRIDNAY, ALCHEMY. RRACING
21

---

Table 6: PLR hyperparameter settings.
Name
Description
Value
NRn
Number of levels to store our buffer
45' / /122,500 / 8,000
Jpu
Level replay score transform function Level replay temperature start Level replay temperature end Level replay scoring function
power
Bg
1.0
Bg
1.0
score(r, m)
Positive value loss
Level replay epsilon for eps-greedy sampling
0.05
Papts
Probability of sampling replay vs. new level
0.5
OPe
Level score EWA smoothing factor
0.0
Po
Staleness coefficient
0.7
fe
Staleness normalization transform
power
Pc
Staleness normalization temperature
1.0
GRIDNAV. ALCHEMY. 'RACING
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The two main claims in the abstract are as follows: "Our empirical results demonstrate DIVA's [1] unique ability to leverage ill ti-parameterized simulators to train adaptive behavior in meta-RL. agents, [2] far outperforming competitive baselines. (I): This ability is indeed "unique". as far the authors are aware, and is supported by the discussion of related works in Section (2): The empirical results in this paper presented in [Section 5 support the claim that DIVA far outperforms other baselines for training meta-RL RL agents. Specific pieces of evidence include the GRIDNAV AV results in Figure 4 the ALCIEMY results in Figure 5] and the results in RACING, contained in Figure 7 and TO The final sentence in the abstract makes a more general, weaker claim about the presented method's potential: "These findings highlight the potential of approaches like DIVA to enable training in complex open-ended domains, and to produce more robust and adaptable agents." The authors believe the same set of evidence supports this claim as well, but is ultimately up to the reader-and more importantly, future work to decide.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The authors discuss limitations of the present work in Section 7 This Dis- cussion section covers (1) key takeaways from the paper, (2) limitations, and (3) promising avenues for future work. The authors combine these sections because the points are all interrelated, and having them in the same place preserves cohesion and flow of the writing.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
23

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does include any formalized theoretical results. The authors did not find reason to provide any theorems, formulas, or proofs to support the claims made.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: In addition the provided code (see Section which is self-contained and includes documentation for reproducing all empirical results in this paper, the authors additionally include all training details, including hyperparameter settings for all methods, in Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
24

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The code is available and well-documented (see Section 8), and contains sufficient instructions to faithfully reproduce the main experimental results. Additionally, all necessary details required independently reproduce the results are self contained in the paper, with hyperparameter settings and other details fully described in Appendixx
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.com wttpc///pppccguuueee Seciiccgguiee///ee Sttes//CoSSuummsssiic.. ssioiiiiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See (cppendix E for all relevant training and test details not covered in the main body. Additionally, details about each evaluation domain, if likewise not covered in the main body, are available in Amppeediix B Lastly, all of these details are self-contained and documented in the code (see (Section 8).
Guidelines:
The answer NA means that the paper does not include experiments.
25

---

The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All results (excepting visualizations, and the trend curve Figure (a), for which they are not needed) are accompanied with error bars that capture the factors of variability, mostly due to random seeding (affecting many different random settings of the algorithm, e.g. initial model weights, sampling, etc.). The significance of all error bars, and number of seeds used for each experiment, are detailed in [Section Other training details are available in Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All compute resources (workers, memory, time of execution) are detailed in Appendix E.4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
26

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics Sttiiigtttssss/pooopp..copppopop
Answer: [Yes]
Justification: The authors have read and understood the NeurlPS Code of Ethics, and the paper conforms to the code in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The authors specifically discuss the broader impacts of the present work in Section 9
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The authors feel that such safeguards are unnecessary for the present work, as the risk for misuse is estimated by the authors to be very low-it is unclear how this work and its artifacts can be directly used for malicious purposes. The authors are willing to adjust this position if members of the community feel otherwise.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
27

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All used resources are either cited properly in the paper and/or are documented and credited in the codebase (see [Section 8).
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwwpssspppppsitaeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The new asset introduced in this paper is the codebase (see Section 8), which is well-documented for the purpose of reproducibility, and contains details about training, the license, and limitations, etc.
Justification:
Guidelines:
The answer NA means that the paper does not relcase new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
28

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "PdHEs3MRpH", "reviewer": "Reviewer_7bxy", "summary": "This paper describes an approach for learning a QD-archive to be used as a proxy for samples of test environments, and shows that this results in improved performance in producing a set of meta-learning tasks over DR and UED baselines.", "questions": "How would this method compare to CLUTR or DRED?", "limitations": "See Weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The introduction of QD approaches into UED algorithms is a promising algorithm for improvement. The empirical results appear convincing, and the approach is quite natural.", "weaknesses": "The comparison between DIVA and PLR/ACCEL is a bit of an apples-to-oranges comparison. Regret-based UED methods like PLR and ACCEL are meant to be making decisions under ignorance, where there is no information known about the target distribution. There are other UED approaches designed for the case where there is some information that is known, and thus it is a decision under risk. Specifically, it would be better to compare against SAMPLR, CLUTER, or DRED. Since SAMPLR requires simulator access, the fairest comparison would be against CLUTER or DRED. \n\n[SAMPLR] Jiang, Minqi, et al. \"Grounding aleatoric uncertainty for unsupervised environment design.\" _Advances in Neural Information Processing Systems_ 35 (2022): 32868-32881.\n\n[CLUTER]Azad, Abdus Salam, et al. \"Clutr: Curriculum learning via unsupervised task representation learning.\" _International Conference on Machine Learning_. PMLR, 2023.\n\n[DRED] Garcin, Samuel, et al. \"DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design.\" _Forty-first International Conference on Machine Learning_. 2024.\n\nThat being said, the current results look to have more significant results than these other works (though it is hard to tell across domains), and this approach has not be tried in meta-learning before. It is important to note that UED has been used in meta-learning before, for instance in AdA.\n\n[AdA] Team, Adaptive Agent, et al. \"Human-timescale adaptation in an open-ended task space.\" _arXiv preprint arXiv:2301.07608_ (2023).\n\nIt occurs to me that DIVA + UED as discussed starting at line 259 could be used as an algorithm for decisions under ignorance, and would possibly be quite a good algorithm. It may be worth running the approach from 259, maybe without any test time data, on the traditional maze, F1, and bipedal walker environments and transfer tasks to check if it consistently outperforms existing methods.\n\nIt seems like the assumption about the probability of generating a useful level from the simulator underlying DIVA is similar to the assumption necessary for ACCEL or PLR. Both of these approaches should work if there is a ~0.01 % chance of generating a useful level. The boot-up time would just be a bit slower to get the initial buffer of levels. \n\nThe limitation of online agents evaluations is a real bottleneck for the UED approaches which DIVA avoids, but it is a bit of an odd comparison as DIVA generates the model once offline, and thus isn't aiming to be adaptive towards current agent performance. It seems like DIVA looses the benefits of adaptivity by completely removing online agent evaluations. I would be interested if a DIVA-like approach with limited online evaluations could keep the best of both. \n\n\nIn Figure 5d, \"unique genotypes is a quite bad metric for diversity, as completely random levels would score quite well even though most random levels are quite qualitatively to each other. \n\n### Clarity\n\nIn the abstract it is not clear to me what \"well-behaved simulator parameterisations\" means, what \"unscalable flexibility\" refers to, or what \"ill parameterised simulators\" means.\n\nIt would be good to have a citation on line 46 for how one could learn the axes of QD.\n\nIt's not immediately clear what \"genotype\" means, and it seems to be used in different ways on line 82 and 88. In the first case it may be more conventional to call it the \"level generator\" and in the second case it could be more conventional to call it the \"level parameters\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to write such a thorough review, and for not only appreciating the merits of this work, but for identifying points of weakness to strengthen our manuscript. We have responded to each of the weaknesses listed and questions posed below, and look forward to further discussion. \n\n> [W1] The comparison between DIVA and PLR/ACCEL is a bit of an apples-to-oranges comparison. [...] the fairest comparison would be against CLUTER or DRED.\n> \n> [Q1] How would this method compare to CLUTR or DRED?\n\nWe have added each of the suggested approaches (SAMPLR, CLUTR, and DRED) to the related works section, which will appear in our camera-ready draft. Our assessment of each of the individual works—see conclusions below, and **full assessments attached as \"official comments\"**—is that none are as appropriate a baseline choice as ACCEL, despite some similarities (especially DRED) to our problem setting. \n\n**SAMPLR**  We agree with the reviewer’s assessment that SAMPLR is less relevant for direct comparison to DIVA since SAMPLR requires access to the simulator itself (and the direct downstream sampling distribution, which we do not assume access to either). However, we believe SAMPLR’s relevance to the meta-RL setting (meta-RL potentially suffers even more from this bias) warrants mention in the related work section---an addition which will appear in the camera-ready draft.\n\n**CLUTR** By training a VAE on levels sampled from the generator distribution, which lack greatly in diversity in DIVA’s setting, CLUTR would fail to learn a generator that produces phenotypically diverse levels. ACCEL, by performing mutations to existing levels, is able evolve its levels towards greater diversity over time, making it a stronger baseline for our setting. Due to its relevance to future work, we have cited and added discussion of this work to the manuscript, which will appear in the camera-ready draft.\n\n**DRED**  The biggest difference between DRED and DIVA’s settings is that DIVA does not assume access to the underlying level parameters, but rather only the feature values---and these alone are used to define the archive. This difference makes DRED suffer from the same issue as CLUTR when faced with the setting that DIVA operates in: if the parameter space cannot produce diverse levels for the VAE training, then the resulting model will produce levels just as lacking in diversity. Resampling a la PLR will have a similar effect as our PLR baseline. Thus, as was mentioned of CLUTR above, ACCEL remains a stronger baseline than DRED for this reason. That being said, DRED’s close resemblance to our setting and possible applicability to future work (like CLUTR), we have cited and added discussion to our manuscript, which will appear in the camera-ready draft.\n\n> [W2] It occurs to me that DIVA + UED [...] could be used as an algorithm for decisions under ignorance, and would possibly be quite a good algorithm. It may be worth running [...] on the traditional maze, F1, and bipedal walker environments and transfer tasks to check if it consistently outperforms existing methods.\n>\n> [...]\n>\n> The limitation of online agents evaluations is a real bottleneck for the UED approaches which DIVA avoids, but it is a bit of an odd comparison as DIVA generates the model once offline, and thus isn't aiming to be adaptive towards current agent performance. It seems like DIVA looses the benefits of adaptivity by completely removing online agent evaluations. I would be interested if a DIVA-like approach with limited online evaluations could keep the best of both.\n\nWe would like to note that the DIVA+ algorithm (which demonstrating learning benefits when the archive is “misspecified”) is indeed a “DIVA-like approach with limited online evaluations”, so the reviewer is correct in identifying the promise of such a combination. And we agree that this kind of approach warrants more exploration; we include these preliminary results simply to showcase the promise of an approach of this nature as a potential future direction, and leave it to future work to both (1) provide a more principled integration of UED/QD (our DIVA+ algorithm is just one such combination), and to (2) produce more extensive results with such an approach on relevant domains, such as the ones the reviewer has listed. \n\n> [W3] It seems like the assumption about the probability of generating a useful level from the simulator underlying DIVA is similar to the assumption necessary for ACCEL or PLR. Both of these approaches should work if there is a ~0.01 % chance of generating a useful level. The boot-up time would just be a bit slower to get the initial buffer of levels.\n\nWhile the reviewer is correct in highlighting that this general assumption is shared between DIVA and ACCEL (and PLR), we demonstrate that DIVA is able to work with generators that produce smaller percentages of useful levels. In practice, the boot-up time for ACCEL (and PLR) is more than a bit slower. Because ACCEL must simulate the agent on each environment, and DIVA only requires rendering the first timestep, then ACCEL is $|\\tau| \\times H$ times slower if we wish to perform just as many evolutionary updates as DIVA (in practice rendering the first timestep to compute features may be slower depending on the environment, but the more general point still holds). Because DIVA can produce far more evolutionary updates in the same amount of time by avoiding simulating the agent on every level (and DIVA+ avoids this too by only simulating levels that constitute the final archive, after all the initial QD updates), DIVA can work more effectively with environments where generating a useful level is even rarer. This is a point we tried to get across in Section 3 (Problem setting), but for the camera ready we will also include the reasoning we’ve provided above—we thank the reviewer for prompting us to consider this point more closely in our discussion!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jXb3Lqr7BC", "reviewer": "Reviewer_34nq", "summary": "The paper identifies the limitation that hand-crafting a sufficiently diverse set of simulated training tasks to bridge any significant sim-to-real gap is labor-intensive. It then proposes DIVA, a new evolutionary approach for generating diverse training tasks in the absence of well-behaved simulator parameterizations. The paper demonstrates how DIVA outperforms proposed baselines such as ACCEL, PLR, and DR.", "questions": "- Why does DIVA want to evolve a population of minimally diverse solutions from the original parameterization? Does DIVA assume that the original parameterizations are what humans care about? If so, doesn’t that mean we still need to ensure that the original parameterizations are handcrafted enough to be close to what humans care about?\n- If the original parameterizations are chosen randomly or are not handcrafted to what humans might care about, how would DIVA perform?\n- To what extent and when do we consider an environment simulator to have structured or unstructured parameterizations?\n- In the GridNav experiment, why do we want to prevent the generation of diverse goals along the y-axis?\n- Is there a hypothesis on why “the domain randomization over increasingly complex genotypes diminishes diversity in terms of goal location” (lines 187-188)?\n- Line 188 claims that “DIVA, on the other hand, is able to capture this diversity.” However, doesn’t DIVA also decrease to the same level of percentage coverage as DR*? Why are there no confidence intervals for Figure 3a, like the others?", "limitations": "- Yes, the paper sufficiently covers the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper identifies a key limitation in current methods used to bridge the sim-to-real gap.\n- The proposed approach is novel.\n- There is a good spread of experimental results.", "weaknesses": "- I think the presentation of the paper can be improved. For example, Figure 1 could include more descriptions about what is happening, such as defining E_S(theta) as the structured environment simulator and E_U(theta) as the unstructured environment simulator.\n- In line 128, how was the number 80% chosen? And why is it described as “roughly”? More justifications for the choice of these parameters should be included.\n\nMinor things\n- Typo in line 184, “the final y location is determine by”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review of our work, and for engaging us with curiosity to better understand the paper. Below we address the two main weaknesses noted by the reviewer, and clarify certain aspects of the work in response to the reviewer’s questions.  \n\n> [W1] [...] The presentation of the paper can be improved. For example, Figure 1 could include more descriptions about what is happening [...]\n\nWe appreciate this constructive feedback. We have updated Figure 1 (see Visual A in the 1-page supplemental), which now includes more descriptions as suggested by the reviewer. \n\nAdditionally, we have taken other efforts to improve the presentation quality of the rest of the paper. These changes include adding algorithmic pseudocode to the Appendix (Visual E in the 1-page PDF) for the sake of clarity, a number of minor updates in wording for the sake of exposition clarity, switching out pixelated plots for raw vectorized PDFs for better image resolution, and some other minor aesthetic changes along these lines.\n\nWe welcome any other concrete suggestions the reviewer may have for improving the presentation.\n\n> [W2] In line 128, how was the number 80% chosen? And why is it described as “roughly”?\n\nWe thank the reviewer for pointing out this ambiguity. By “roughly” we were referring to the nature of samples versus population: we set this threshold using the 80th percentile of the samples, which encapsulates roughly 80% of the population. We understand this wording is unclear, and have therefore clarified what we mean precisely in the manuscript.  \n\n> […] More justifications for the choice of these parameters should be included.\n\n80% was chosen as an intuitive heuristic to balance the need for preserving the diversity contained within the DR samples used to initialize the archive (and the corresponding region that may be filled with useful mutations), and given a fixed archive size, maintaining as high a resolution as possible to enable the fastest propagation of solutions to and within the target region. \n\nWe have run experiments to demonstrate that any moderate setting of this value prevents the failure modes of either extreme—see Visual D in the 1-page PDF (which will be included in the camera-ready, along with relevant discussion). In Alchemy (left) we see that the failure mode preventing the generation of target solutions is when we set this too low—-i.e. where we take only a small part of the tail of the DR samples. In Racing (right) we see the opposite problem; when set too high, it has significantly reduced the number of target samples. In both cases, the any moderate settings of the hyperparameter seems to do the job; it’s really about preventing either of the extremes. We chose 80% to be on the conservative side, but from the experiments, it looks like a setting of around 50% might even be better (from the Racing results).\n\nAdditionally, we include robustness studies for other major hyperparameters in Visuals B, C, and F in the 1-page PDF (also for camera-ready). All attached figures correspond to Alchemy, but we can also provide similar ablations for Racing in the camera-ready draft. In Visual B we show robustness of final returns based on the mutation rate (Alchemy); in Visual C we see more QD updates is generally better, but our setting of 80k is not necessary to produce significant benefits over baselines; and in Visual F we see that while estimates for the normal parameters become less accurate with fewer sample features provided, DIVA is able to outperform baselines with as little as 5 downstream feature samples. \n\n> [W3] Typo in line 184, “the final y location is determine by”\n\nFixed, thanks!\n\nQuestions\n\n> [Q1] Why does DIVA want to evolve a population of minimally diverse solutions from the original parameterization? Does DIVA assume that the original parameterizations are what humans care about? If so, doesn’t that mean we still need to ensure that the original parameterizations are handcrafted enough to be close to what humans care about?\n\nWe believe there is a key misunderstanding here between the function of DIVA and the experimental setup. We choose parameterizations in our evaluation that do not provide the dials—so to speak—to tune the features that humans care about (like the dials on $E_S(\\theta)$ in Figure 1—-updated version in 1-page PDF response, Visual A). And these challenging parameterizations are what produce the “minimally diverse” solutions with which the archive is initialized. DIVA is designed to work with parameterizations like these that are explicitly not designed to produce diversity that humans care about. DIVA overcomes these difficult parameterizations by using QD to populate an archive of solutions that are diverse along certain features—which are hand-designed in this work, but may be learned automatically in future work (e.g. [1], a citation we’ve added thanks to the suggestion of Reviewer 7bxy).\n\nIt is alternatively possible the reviewer might be asking if we need access to $E_S(\\theta)$. DIVA does not need access to this generator/parameterization; all it needs is enough feature value samples from the downstream distribution to roughly estimate what our target distribution over the archive should be. It does not need the parameters that correspond to these scenarios. For example, if DIVA were tasked with generating house layouts for a house navigation task, it would need to know the target distribution for the number of rooms / the hallway width, etc. which could be determined from e.g. photos; but it would not need the parameters for constructing these sample houses in the simulator.\n\nIn short, it is the features that must be designed/specified to reflect what humans care about, not the parameterization. Most works assume the parameterization is well-behaved. We challenge this assumption, and replace it with an assumption we believe is more realistic for open-ended environment simulators.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t7EhfaTCvJ", "reviewer": "Reviewer_nCFY", "summary": "This paper introduces DIVA, a technique for exploring the parameter space of parametrisable environments. The technique uses a variant of MAP-Elites to explore the environment parameter space, finding exemplar points spread across the parameter space, as measured with respect to some user provided features. The authors show that this generates a usefully diverse collection of environment instances by training a meta-learning agent on the generated levels and comparing it to a number of baselines.", "questions": "L36-37: Nit: “a” refers to the singular, “autocurricula” is plural.\n\nL74: Nit: Add “are” between “algorithms” and “in”.\n\nL129-131: Sentence doesn’t parse. Maybe remove “but”, or perhaps something got accidentally removed?\n\nL159-161: I was confused by the experimental approach here. The downstream task distribution was first used to train the meta-learner’s hyperparameters, and then these hyperparameters were used to compare all of the approaches. But isn’t this essentially training on the test set? It feels like that the realistic setting would be one where the meta-learner does not know about the downstream tasks, and it would be important to demonstrate DIVA works under these conditions. I can appreciate that the authors might want to try to “factor” the behaviour of the meta-learner from the behaviour of DIVA, but it seems like a strong assumption that this is possible or useful.\n\nL245: Nit: Figures 8 & 9 are referenced before figure 7.\n\nL278: “Their approach”. Who’s approach?\n\nL306: Define “HRI”.\n\nL312: “DIVA’s” -> “DIVA”.\n\nL312: “incorporating” -> “incorporate”.", "limitations": "The primary limitation is the one regarding hand-tuned feature sets, which I have expanded on above, in the weaknesses section.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "On the whole the paper is well written and clear.\n\nThe authors choose a sensible selection of baseline algorithms to compare against, giving the reader an understanding of the strengths of their approach.", "weaknesses": "The primary weakness from my perspective is that domains that the experiments was conducted on are all very simple “toy” domains. In particular, these domains do not have all have intrinsically complex parameterisations, and for two of them the authors had to, in effect, obfuscate the parameter spaces so the algorithm had a challenge to work against. This leads one to worry that the authors’ results may not be representative of more realistic open-ended domains where the parameter space complexity may manifest in a different way.\n\nAnother weakness of the technique is the authors need to hand-select the features used by the algorithm, per domain. The authors mention that this could be automated, but the fact still stands that in this paper, quite extensive hand tuning and selection - as reported in the appendices - of the objectives was made to get their results. The impact of this technique is much more limited if feature sets need to be hand-tuned per domain, so without demonstrating that this is not the case, I think the authors do not demonstrate that this technique is likely to have wide impact.\n\nI have a further query about the way that the VariBAD hyperparameters were tuned, below in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s feedback and attention to detail. We have made all of the minor changes suggested under “Questions”, which will be present in the camera-ready draft. We address both major concerns below.\n\n**(1) On the domains being “toy”**\n\n> [W1] [...] domains [...] are all very simple [a] “toy” domains. In particular, these domains do not have all have [b] intrinsically complex parameterisations, and for two of them the authors had to, in effect, obfuscate the parameter spaces so the algorithm had a challenge to work against. [...] the authors’ results may [c] not be representative of more realistic open-ended domains where the parameter space complexity may manifest in a different way.\n\n**(1a)** For academic institutions with limited computational resources, this set of domains presents challenge enough to usefully benchmark methods, while keeping training runs cheap enough that many experimental trials can be run—enabling us to achieve statistically significant results.\n\n**(1b)** As discussed openly in Section 5, we recognize that the domains chosen require us to “obfuscate” the existing environment parameterization(s). We believe the reviewer may fundamentally misunderstand why adjusting the parameterization of these domains is necessary. \n\nMost meta-RL/RL domains are released with a parameterization/generator that can produce levels of meaningful diversity. Especially for domains with complex dynamics, designing these are not so straightforward. Because DIVA-like approaches are not widely studied at present, most domain designers do carefully hand-craft task distributions so that researchers will use their domains.\n\nAs meta-RL approaches improve, and domains become more complex and open-ended, well-structured environment parameterizations will become increasingly expensive to implement by hand. Approaches like DIVA will enable learning on these domains, without requiring carefully hand-designing a structured parameterization and corresponding task generator. We believe that demonstrating the existence of DIVA-like methods that can handle ill-parameterized domains is a necessary step to inspire researchers to build more open-ended domains with unstructured parameterizations in the first place.\n\n**(1c)** The main property that makes a parameter space “unstructured” is the low probability of producing meaningfully diverse levels. This is the guiding principle used to design the challenging parameterizations for our evaluation domains—which, as we have explained above in response to (1b), was a necessity, since most (if not all) current academic domains are released with convenient parameterizations. While we agree that “the parameter space complexity may manifest in a different way” for more complex open-ended domains, what will remain is the overarching principle of meaningfully diverse levels having a low probability of being generated. Without detailing some other specific property of open-ended domains we might have overlooked, this weakness is a “whataboutism” that can be applied in similar form to any set of finite empirical findings.\n\n**(2) On hand-selected features**\n\n> [W2] […] The authors need to hand-select the features used by the algorithm, per domain. […] The impact of this technique is much more limited if feature sets need to be hand-tuned per domain, so without demonstrating that this is not the case, I think the authors do not demonstrate that this technique is likely to have wide impact.\n\nFirst, only a few features specified (a subset of those detailed for each domain in the Appendix) were used in experiments, and the final designs for each were guided more by intuition (e.g. looking at the distributions and covariance matrices) than any \"extensive\" experimental tuning. We also demonstrate with DIVA+ (Line 259) that the misspecification of features/objectives can be made up for by performing a small number of online evaluations like UED works.\n\nWe believe that this criticism also overlooks the “extensive hand tuning” that is currently required to design parameterizations for complex domains. It is not that other approaches are able to sidestep the necessity of “hand-tun[ing]” feature sets per domain—it’s that this tuning is buried within the environment generation logic, which enabling a conveniently structured parameterization. \n\nThe assumption of having access to some features of useful diversity is much weaker assumption than the assumption of having pre-existing generators that can produce levels across these same features of diversity. Consider—how does one design a structured parameterization/generator without first determining what the resulting level features should look like?\n\n**Questions**\n\n> [Q1] L159-161: [...] The downstream task distribution was first used to train the meta-learner’s hyperparameters, and then these hyperparameters were used to compare all of the approaches. But isn’t this essentially training on the test set? [...] I can appreciate that the authors might want to try to “factor” the behaviour of the meta-learner from the behaviour of DIVA, but it seems like a strong assumption that this is possible or useful.\n\nWe recognize the ambiguity our wording may have caused here. The meta-learner is in fact “tuned” on the structured parameterizations. As can be seen in Appendix C.2, only a few VariBAD hyperparameters are adjusted per domain, which mostly pertain to pragmatic considerations such as network structure, and RAM/storage considerations. Because the parameterizations we consider produce poor training levels, we validate that the meta-RL component works by running the agent on the structured generator. This is indeed to “’factor’ the behavior of the meta-learner from the behavior of DIVA” and was a pragmatic decision so that we could more confidently study the effects of our method over the baselines. Once the meta-learner configuration was determined for each domain, it was held fixed for each method evaluated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vFE05gqTQ4", "reviewer": "Reviewer_kFbG", "summary": "This paper applies quality diversity (QD) optimization (evolutionary method) to the problem of diverse task generation for (meta) reinforcement learning (RL). It argues that QD could be used in settings where an open-ended simulator’s parameterization is unlikely to produce tasks that are diverse in high-level features. The method works by handcrafting a set of high-level task features that are relevant to the learning process of the RL agent, then running QD to collect a set of diverse parameterizations that cover the feature space distribution well. A (meta) RL agent is then trained on tasks sampled from a distribution based on the set of QD optimised tasks. The paper’s experiments on GridNav, Alchemy, and Racing tasks show significant improvement in agent performance over existing baselines such as robust prioritised level-replay.", "questions": "Does DIVA depend on any parameterizations at all? It seems like it might be possible (and very useful) to try DIVA on “weakly” parameterized tasks such as those generated by prompting an LLM or, say Genie [1].\n\n[1] Genie: Generative Interactive Environments. Bruce el al. https://arxiv.org/abs/2402.15391", "limitations": "Limitations have been adequately addressed.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Focuses on high-level features more relevant to downstream tasks that the meta-RL agent should learn to adapt in, rather than focusing on simulator parameters as in prior unsupervised environment design (UED) work.\n\nBy shifting the focus in task distribution design for meta-RL training, this paper showed significant improvements on representative existing UED solutions, on a diverse set of evaluation tasks (GridNav, Alchemy, Racing). The evaluations are well-designed with sufficient ablations. \n\nMakes a connection between QD and UED for meta-RL training, which is novel to the best of this reviewer’s knowledge.\n\nThe paper is very clearly written and illustrated. Relevant related works are discussed and contributions of the work are put into appropriate context. Content is well self-contained despite introducing new methods and tasks.", "weaknesses": "It is still necessary to hand-craft the high-level features, which needs expert knowledge (or at least quite high familiarity with downstream tasks) and can be heuristic. \n\nEvaluation only used one meta-RL method (VariBAD). It would be useful to see performance of other meta-RL methods such as RL^2 on the DIVA task distribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review our work and for appreciating its merits. We have addressed the noted weaknesses below, as well as their question about parameterizations. \n\n> [W1] It is still necessary to hand-craft the high-level features, which needs expert knowledge (or at least quite high familiarity with downstream tasks) and can be heuristic.\n\nWe agree that this is a limitation of our work, which we discuss in Section 7 (Line 319). In our response to [W1] from Reviewer 7bxy, we have included discussion of an example work [1] where QD features are learned automatically; it is possible we can draw inspiration from works in this vein to further automatize DIVA.\n\n[1] Ding, L., Zhang, J., Clune, J., Spector, L., & Lehman, J. (2024). Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization. In Forty-first International Conference on Machine Learning.\n\n> [W2] Evaluation only used one meta-RL method (VariBAD). It would be useful to see performance of other meta-RL methods such as RL^2 on the DIVA task distribution.\n\nIt is true that all evaluations are conducted using VariBAD the base meta-RL algorithm. However, we did perform some preliminary evaluations with RL2—which is the same as VariBAD, except (1) instead of using a VAE with a latent space, we directly pass the RNN hidden state to the policy, and (2) we backpropagate the policy loss through the RNN (VariBAD only applies the VAE loss)—however, we did not see a drastic difference in the relative performance—between DIVA and its baselines—and because of the RNN backpropagate via the policy loss, training is significantly slower (as originally noted in [2]). \n\nGiven these results, in absence of a performance differential, we decided on VariBAD, a state-of-the-art meta-RL algorithm. One reason we were drawn specifically to VariBAD is the harmony between the nature of task distributions, and VariBAD’s distributional nature. In DIVA’s case, task distributions are represented as a multivariate normal (some dimensions may be uniform) over a QD archive. We believe it may be possible for future works to exploit this connection; e.g. by using the location of a solution in the archive to ground the VariBAD latent space (a rough idea).\n\n[2] Zintgraf, L., Shiarlis, K., Igl, M., Schulze, S., Gal, Y., Hofmann, K., & Whiteson, S. (2019, September). VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. In International Conference on Learning Representations.\n\n> [Q1] Does DIVA depend on any parameterizations at all? It seems like it might be possible (and very useful) to try DIVA on “weakly” parameterized tasks such as those generated by prompting an LLM or, say Genie [1].\n\nThis is a great question, and certainly a direction worth exploring. DIVA in its current iteration can work with these weakly parameterized tasks, so long as (1) we have an environment generator that can accept this “weak” parameterization, and (2) we have access to some mutation method that can operate over this weak parameterization. For example, if the weak parameterization is language, we need some kind of language augmentation function (which prior literature has certainly explored). It may be even more effective to work with in the embedding space itself, since the augmentation function would be both simpler & smoother than language augmentation; in this case we would be essentially using QD to perform prefix/embedding tuning. We are excited to see these kinds of directions pursued in follow-up works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for the time they have taken to engage constructively with our work. Reviewer _kFbG_ appreciates the “novel [...] connection between QD and UED for meta-RL training” we have developed, along with our work’s explicit “focus on [relevant] high-level features”—in contrast to the implicit assumptions of diversity along these features in prior works—and the resulting “significant improvements [on] well-designed” evaluations. Reviewer _7bxy_ finds the approach “natural”, and the results “convincing”. Reviewer _34nq_ believes the paper “identifies a key limitation” in existing literature and contains a “good spread of experimental results”. Reviewer _nCFY_ finds the paper “well written and clear”, and that a “sensible selection of baseline algorithms” are used for evaluation. 

There are four main weaknesses in total that appear to be sticking points for Reviewers _34nq_ and _nCFY_, specifically. We have addressed each of these points at length in the individual responses, and summarize the holistic discussion of each point here.  

**(1) Specifying features.**  A commonly noted limitation among the reviewers, and one we address ourselves in Section 7 of the manuscript, is that the “axes of diversity must be specified” (Line 319). Reviewer _kFbG_ views DIVA’s explicit “focus on [relevant] high-level features” as a strength of our work, an aspect which contributes to our paper identifying a “key limitation” (Reviewer _34nq_) in existing literature. As we point out in a number of individual responses, while QD literature often assumes access to these high level features, some works (e.g. [1]) are able to determine these automatically. We view this as an interesting avenue for future work, but outside the scope of this paper, which, as previously noted is already a “novel [...] connection between QD and UED for meta-RL training” (Reviewer _kFbG_). For Reviewer _nCFY_ alone it is a major sticking point from which they conclude (in addition to other human involvement in feature/objective selection) that we have “not demonstrate[d] that this technique is likely to have wide impact”. In response, we remind the reviewer that, despite explicitly making this new assumption ourselves, we are removing a much stronger assumption implicit in prior works:

> The assumption of having access to some features of useful diversity is much weaker assumption than the assumption of having pre-existing generators that can produce levels across these same features of diversity. Consider—how does one design a structured parameterization/generator without first determining what the resulting level features should look like?

See our full response to Reviewer _nCFY_ for more discussion on this point.

**(2) Presentation.**  Reviewer _34nq_ expresses concern over the presentation of the paper as their main weakness. Specifically, the example they give for where clarity can be improved is Figure 1. We update Figure 1 along the lines of the reviewer’s suggestions (see **Visual A in the attached PDF**), add algorithmic pseudocode to the Appendix (see **Visual E in the attached PDF**) to further elucidate our approach, and make some other minor formatting changes to improve the presentation for the camera-ready draft. It is worthwhile noting that beyond typos and updates to specific cases of ambiguous wording, no other major presentation concerns were noted by Reviewer _34nq_ or the other reviewers. Reviewer _nCFY_ says “on the whole the paper is well written and clear”, and Reviewer _kFbG_ finds the paper “very clearly written and illustrated”. 

**(3) Hyperparameters.**  Reviewer _34nq_ expresses confusion over one of our hyperparameter choices (setting an archive bound based on 80% of samples), and expresses more generally that these decisions are not better justified in the submitted draft. We clarify our wording and justify our decision for the 80% bound hyperparameter, and provide a robustness study on both Alchemy and Racing domains to verify the intuition that any moderate setting of this hyperparameter avoids failure modes (see **Visual D in the attached PDF**). We additionally provide studies for other hyperparameter, demonstrating DIVA’s robustness to their settings (see **Visuals B, C, and F**). For more details, see our response to Reviewer _34nq_ [W2]. 

**(4) Evaluation domains.**  Reviewer _nCFY_’s main sticking point (unique among the reviewers; who otherwise find the evaluation sufficient and convincing) is that the domains are “toy”, specifically because we “obfuscate” the parameter spaces to produce challenging parameterizations. We first dispute the notion that the domains themselves are overly toy, both in terms of the unique challenges they pose for methods in our problem setting, and according to standards of academic research in meta-RL and UED. We then explain why it was necessary to reparameterize the tasks—namely, that most domains come pre-equipped with a parameterization that can generate levels of meaningful diversity. For open-ended domains of the future, with more complex dynamics, well-structured environment parameterizations will not be as feasible to implement by hand. Approaches like DIVA will enable learning on these domains, without requiring carefully hand-designing a structured parameterization or task generator. And crucially, we believe that demonstrating the existence of DIVA-like methods that can handle ill-parameterized domains is a necessary step to inspire researchers to build more open-ended domains with unstructured parameterizations in the first place.

[1] Ding, L., Zhang, J., Clune, J., Spector, L., & Lehman, J. (2024). Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization. In Forty-first International Conference on Machine Learning.

Meta-review: The paper proposes a method for generating actually diverse training tasks. 

Reviewers recommendations were Accept, Weak Accept, Reject, Strong Accept. There was some criticism of the fact that the proposed approach still requires some level of hand-design, and the simplicity of the experiments. 

Overall I am willing to accept due to the interest of the problem and the novelty of the method.

I would recommend the authors re-read the paper and fix the  typos. E.g. line 130 "but...",  line 194: "the each stone".
Decision: Accept (poster)