Title: Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity
Venue: NeurIPS 2024
Accepted: yes
Keywords: diversity

meta reinforcement learning

meta-RL

reinforcement learning

adaptation

adaptive

agents

open-endedness

genotypes

phenotypes

simulators

simulation

generalization

meta-reinforcement
Abstract: The wider application of end-to-end learning methods to embodied decision-making domains remains bottlenecked by their reliance on a superabundance of training data representative of the target domain.
Meta-reinforcement learning (meta-RL) approaches abandon the aim of zero-shot *generalization*—the goal of standard reinforcement learning (RL)—in favor of few-shot *adaptation*, and thus hold promise for bridging larger generalization gaps.
While learning this meta-level adaptive behavior still requires substantial data, efficient environment simulators approaching real-world complexity are growing in prevalence.
Even so, hand-designing sufficiently diverse and numerous simulated training tasks for these complex domains is prohibitively labor-intensive.
Domain randomization (DR) and procedural generation (PG), offered as solutions to this problem, require simulators to possess carefully-defined parameters which directly translate to meaningful task diversity—a similarly prohibitive assumption.
In this work, we present **DIVA**, an evolutionary approach for generating diverse training tasks in such complex, open-ended simulators.
Like unsupervised environment design (UED) methods, DIVA can be applied to arbitrary parameterizations, but can additionally incorporate realistically-available domain knowledge—thus inheriting the *flexibility* and *generality* of UED, and the supervised *structure* embedded in well-designed simulators exploited by DR and PG.
Our empirical results showcase DIVA's unique ability to overcome complex parameterizations and successfully train adaptive agent behavior, far outperforming competitive baselines from prior literature.
These findings highlight the potential of such *semi-supervised environment design* (SSED) approaches, of which DIVA is the first humble constituent, to enable training in realistic simulated domains, and produce more robust and capable adaptive agents.
Our code is available at [https://github.com/robbycostales/diva](https://github.com/robbycostales/diva).

Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity

Abstract

The wider application of end-to-end learning methods to embodied decisionmaking domains remains bottlenecked by their reliance on a superabundance of training data representative of the target domain. Meta-reinforcement learning (meta-RL) approaches abandon the aim of zero-shot generalization -the goal of standard reinforcement learning (RL)-in favor of few-shot adaptation , and thus hold promise for bridging larger generalization gaps. While learning this meta-level adaptive behavior still requires substantial data, efficient environment simulators approaching real-world complexity are growing in prevalence. Even so, hand-designing sufficiently diverse and numerous simulated training tasks for these complex domains is prohibitively labor-intensive. Domain randomization (DR) and procedural generation (PG), offered as solutions to this problem, require simulators to possess carefully-defined parameters which directly translate to meaningful task diversity-a similarly prohibitive assumption. In this work, we present DIVA , an evolutionary approach for generating diverse training tasks in such complex, open-ended simulators. Like unsupervised environment design (UED) methods, DIVA can be applied to arbitrary parameterizations, but can additionally incorporate realistically-available domain knowledge-thus inheriting the flexibility and generality of UED, and the supervised structure embedded in well-designed simulators exploited by DR and PG. Our empirical results showcase DIVA's unique ability to overcome complex parameterizations and successfully train adaptive agent behavior, far outperforming competitive baselines from prior literature. These findings highlight the potential of such semi-supervised environment design (SSED) approaches, of which DIVA is the first humble constituent, to enable training in realistic simulated domains, and produce more robust and capable adaptive agents. Our code is available at https://github.com/robbycostales/diva.

1 Introduction

Despite the broadening application of reinforcement learning (RL) methods to real-world problems [1, 2], generalization to new scenarios -ones not explicitly supported by the training set-remains a fundamental challenge [3]. Meta-reinforcement learning (meta-RL), an extension of the RL framework, is formulated specifically for training adaptive agents , and is thus well-suited for overcoming these generalization gaps [4]. One recent work has demonstrated that meta-RL agents can be trained at scale to achieve adaptation capabilities on par with human subjects [5]. However, learning this human-like adaptive behavior naturally requires a large amount of data representative of the downstream (or target ) distribution. For task distributions approaching real-world complexity-precisely the ones of interest-designing each scenario by hand is prohibitively expensive.

Figure 1

target diversity

f

2

f

1

structured

vs.

unstructured

environment

generators

f

3

f

2

axis of

diversity

axis of

diversity

f

3

(idealistic)

(realistic)

treat as

'black box'

archive of levels

random parameters

low diversity

discovered

diversity

DIVA

QD updates

01

02

03

Es(0)

9

K

Eu(0)

Prior works have explored the use of domain randomization (DR) and procedural generation (PG) techniques to produce diverse training data for learning agents [6]. Despite eliminating the need for hand-designing each task individually, human labor is still required to carefully design an environment generator that can produce diverse, high-quality tasks. As environments become more complex and open-ended, the ability to hand-design such a robust generator becomes increasingly infeasible. Some methods, like PLR [7], attempt to ameliorate this limitation by learning a curriculum over the generated levels, but these works still operate under the assumption that the generator produces meaningfully diverse levels with a high probability.

Unsupervised environment design (UED) [8] are a broad class of appproaches which use performancebased metrics to adaptively form a curriculum of training levels. ACCEL [9], a state-of-the-art UED method, uses an evolutionary process to discover more interesting regions of the simulator's parameter space (i.e. appropriately challenging tasks) than can be found by random sampling. While UED approaches are designed to be generally applicable and require little domain knowledge, they implicitly require a very constrained environment generator-one in which all axes of difficulty correspond to meaningful learning potential for the downstream distribution. Moreover, when faced with complex open-ended environments with arbitrary parameterizations, even ACCEL is not able to efficiently explore the solution space, as it is still bottlenecked by the speed of agent evaluations.

In this work, we introduce DIVA , an approach for generating diverse training tasks in open-ended simulators to train adaptive agents. By using quality diversity (QD) optimization to efficiently explore the solution space, DIVA bypasses the problem of needing to evaluate agents on all generated levels. QD also enables fine-grained control over the axes of diversity to be captured in the training tasks, allowing the flexible integration of task-related prior knowledge from both domain experts and learning approaches. We demonstrate that DIVA, with limited supervision in the form of feature samples from the target distribution, significantly outperforms state of the art UED approachesdespite the UED approaches being provided with significantly more interactions. We further show that UED techniques can be integrated into DIVA. Preliminary results with this combination (which we call DIVA+) are promising, and suggest an exciting avenue for future work.

2 Preliminaries

Meta-reinforcement learning. We use the meta-reinforcement learning (meta-RL) framework to train adaptive agents, which involves learning an adaptive policy π φ over a distribution of tasks T .

begin stage 1

guide solutions to target

begin stage 2

target

region

sample

mask

populate target

sample solutions

(a)

(b)

(c)

(d)

(e)

Density

N(μ,o)

f1

P

(1, 1, 1) y z 1 P 2 (a) (b) (c) (d) Each M i ∈ T is a Markov decision process (MDP) defined by a tuple 〈S , A , P, R, γ, T 〉 , where S is the set of states, A is the set of actions, P ( s t +1 | s t , a t ) is the transition distribution between states given the current state and action, R ( s t , a t ) is the reward function, γ ∈ [0 , 1] is the discount factor, and T is the horizon. Meta-training involves sampling tasks M i ∼ T , collecting trajectories D = { τ h } H h =0 -where H is the number of episodes in each trial τ pertaining to the M i -and optimizing policy parameters φ to maximize the expected discounted returns across all episodes.

VariBAD [10] is a context variable-based meta-RL approach which belongs to the wider class of RNNbased methods [11, 12]. While prior methods [13, 14] also use context variables to assist in task adaptation, VariBAD uniquely learns within a belief-augmented MDP (BAMDP) 〈S , A , Z , P, R, γ, T 〉 where the context variables z ∈ Z encodes the agent's uncertainty about the task, promoting Bayesian exploration. VariBAD utilizes an RNN-based variational autoencoder (VAE) to model a posterior belief over possible tasks given the full agent trajectory, permitting efficient updates to prior beliefs.

Quality diversity. For a given problem, quality diversity (QD) optimization framework aims to generate a set of diverse, high-quality solutions. Formally, a problem instance of QD [15] specifies an objective function J : R n → R and k features f i : R n → R . Let S = f ( R n ) be the feature space formed by the range of f , where f : R n → R k is the joint feature vector. For each s ∈ S , the QD objective is to find a solution θ ∈ R n where f ( θ ) = s and J ( θ ) is maximized. Since R k is continuous, an algorithm solving the QD problem definition above would require unbounded memory to store all solutions. QD algorithms in the MAP-Elites [16] family therefore discretize S via a tessellation method, where G is a tessellation of the continuous feature space S into N G cells. In employing a MAP-Elites algorithm, we relax the QD objective to find a set of solutions θ i , i ∈ { 1 , . . . , N G } , such that each θ i occupies one unique cell in G . We call the occupants θ i of all M cells, each with its own position f ( θ i ) and objective value J ( θ i ) , the archive of solutions.

3 Problem setting

One assumption underlying UED methods is that random parameters-or parameter perturbations for ACCEL -produce meaningfully different levels to justify the expense of computing objectives on each newly generated level. However, when the genotype is not well-behaved -when meaningful diversity is rarely generated through random sampling or mutations-these algorithms waste significant time evaluating redundant levels. In our work, we discard the assumption of well-behaved genotypes in favor of making fewer, more realistic assumptions about complex environment generators. There are several assumptions we make about the simulated environments DIVA has access to.

Genotypes. We assume access to an unstructured environment parameterization function E U ( θ ) , where each θ is a genotype (corresponding to the QD solutions θ i ) describing parameters to be fed into the environment generator. QD algorithms can support both continuous and discrete genotype spaces, and in this work we evaluate on domains with both kinds. Crucially, we make no assumption of the quality of the training tasks produced by this random generator. We only assume that (1) there is some nonzero (and for practical purposes, nontrivial) probability that this generator will produce a valid level for training-one in which success is possible and positive rewards are in reach; and (2) that it is computationally feasible to discover meaningful feature diversity through an intelligent search over the parameter space-an assumption implicit in all QD applications.

Features. We assume access to a pre-defined set of features, S = f ( R n ) , that capture axes of diversity which accurately characterize the diversity to be expected within the downstream task distribution. It is also possible to learn or select good environment features from a sample of tasks from the downstream distribution, which we discuss in Section 7. For the sake of simplicity, we use a grid archive as our tessellation G , where the k dimensions of the discrete archive correspond to the defined features. The number of bins for each feature is a hyperparameter, and can be learned or adapted over the course of training. We generally find it to be helpful to use moderately high resolutions to ease the search, since smaller leaps in feature-level diversity are required to uncover new cells. By default, we use 100 sample feature values across all domains, but demonstrate in ablation studies that that significantly fewer may be used (see Appendix C).

4 DIVA

DIVA assumes access to a small set of feature samples representative of the target domain. It does not, however, require access to the underlying levels themselves. This is a key distinction, as the former is a significantly weaker assumption. Consider the problem of training in-home assistive robots in simulation with the objective of adapting to real-world houses. It is more likely we have access to publicly available data describing typical houses-dimensions, stylistic features, etc.-than we have access to corresponding simulator parameters which produce those exact feature values.

Feature density estimation. DIVA begins by constructing a QD archive with appropriate bounds and resolution . Given a set of specified features { f i } k and a handful of downstream feature samples , we first infer each feature's underlying distribution. These can be approximated with kernel density estimation (KDE), or we can work with certain families of parameterized distributions. For our experiments, we assume each feature is either (independently) normally or uniformly distributed. We use a statistical test 2 to evaluate the fit of each distribution family, and select the best-fitting. Setting the resolution for discrete feature dimensions is straightforward, and depends only on the range. For continuous features, the resolution should enable enough signal for discovering new cells, while avoiding practical issues that arise with too many cells 3 . See Section 5 for domain-specific details.

Two-stage QD updates. Once the feature-specific target distributions are determined, we can use these to set bounds for each archive dimension. A naïve approach would be to set the archive ranges for each feature based on the confidence bounds of the target distribution. However, random samples from E U may not produce feature values that fall within the target range. We found this to be a major issue in the ALCHEMY domain (see Figure 2), and for some features in RACING. We solve this problem by setting the initial archive bounds to include both randomly generated samples from E U, as well as the full target region. As the updates progress, we gradually update the sample mask -which is used to inform the sampling of new solutions-towards the target region. We observe empirically that updating and applying this mask provides an enormous speed-up in guiding solutions towards the target region (see Figure 15). After this first stage, solutions are inserted into a new archive defined by the proper target bounds. See Appendix A for more specifics on these two QD update stages.

Overview. DIVA consists of three stages. Stage 1 (S1) begins by initializing the archive with bounds that include both the downstream feature samples (the target region ), as well as the initial population generated from E U ( θ ) . S1 then proceeds with alternating QD updates , to discover new solutions, and sample mask updates , to guide the population towards the target region. In Stage 2 (S2), the archive is reinitialized with existing solutions, but is now bounded by the target region. QD updates continue to further diversify the population, now targeting the downstream feature values specifically. The last stage is standard meta-training, where training task parameters are now drawn from P G ( θ ) , a distribution over the feature space approximated using the downstream feature samples, discretized over the archive cells. See Appendix A for detailed pseudocode.

5 Empirical results

Baselines. We implement the following baselines to evaluate their relative performance to DIVA . ODS is the 'oracle" agent trained over the downstream environment distribution E S ( θ ) , used for evaluation. With this baseline, we are benchmarking the upper bound in performance from the perspective of a learning algorithm that has access to the underlying data distribution. 4 DR is the meta-learner trained over a task distribution defined by performing domain randomization over the space of valid genotypes, θ , under the training parameterization, E U ( θ ) . Robust PLR ( PLR ⊥ ) [17] is the improved and theoretically grounded version of PLR [7], where agents' performance-based PLR objectives are evaluated on each level before using them for training. ACCEL [9] is the same as PLR ⊥ but instead of randomly sampling over the genotype space to generate levels for evaluation, levels are mutated from existing solutions. All baselines use VariBAD [10] as their base meta-learner.

Experimental setup. The oracle agent (ODS) is first trained over the each environment's downstream distribution to tune VariBAD's hyperparameters. These environment-specific VariBAD settings are then fixed while hyperparameters for DIVA and the other baselines are tuned. For fairness of comparison-since DIVA is allowed N QD QD update steps to fill its archive before meta-trainingwe allow each UED approach (PLR ⊥ and ACCEL) to use significantly more environment steps for agent evaluations (details discussed below per environment). All empirical results were run with 5 seeds unless otherwise specified, and error bars indicate a 95% confidence region for the metric in question. The QD archive parameters were set per environment, and for ALCHEMY and RACING, relied on some hand-tuning to find the right combinations of features and objectives. We leave it to future work to perform a deeper analysis on what constitutes good archive design, and how to better automate this process. Gridworld

5.1 GRIDNAV

Our first evaluation domain is a modified version of GRIDNAV (Figure 3), originally introduced to motivate and benchmark VariBAD [10]. The agent spawns at the center of the grid at the start of each episode, and receives a slight negative reward ( r = -0 . 1) each step until it discovers (inhabits) the goal cell, at which point it also receives a larger positive reward ( r = 1 . 0) .

Parameterization. We parameterize the task space (i.e. the goal location) to reduce the likelihood of generating meaningfully diverse goals. Specifically, each E U k (or E k ) introduces k genes to the solution genotype which together define the final y location. Each gene j can assume the values θ j ∈ {-1 , 0 , 1 } , and the final y location is determined by summing these values, and performing a floor division to map the bounds back to

the original range of the grid. As k increases, y values are increasingly biased towards 0 , as shown on the right side of Figure 3. For more details on the GRIDNAV domain, see Appendix B.1.

QD updates. We define the archive features to be the x and y coordinates of the goal location. The objective is set to the current iteration, so that newer solutions are prioritized (additional details in Appendix B.1). DIVA is provided N S2 = 8 . 0 × 10 4 ( N S1 = 0 ) QD update iterations for filling the archive. To compensate, PLR ⊥ and ACCEL are each provided with an additional 9 . 6 × 10 6 environment steps for evaluating PLR scores, which amounts to three times as many total interactions-since all methods are provided N E = 4 . 8 × 10 6 interactions for training. If each 'reset" call counts as one environment step 5 , the UED baselines are effectively granted 2.4 × more additional step data than what DIVA additionally receives through its QD updates (details in Appendix E.1).

Results. From Figure 4a, we see that increasing genotype complexity (i.e. larger k ) reduces goal diversity for DR-which is expected given the parameterization defined for E U. We can also see that DIVA, as a result of its QD updates, can effectively capture goal diversity, even as complexity

20

40

60

80

100

Genotype Complexity

k

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Percent Coverage

Goal Diversity by

θ

Complexity

DIVA

DR

∗

DR

(a)

0

1

2

3

Updates

×

10

2

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Percent Coverage

∗

values from Figure 3a (

k

= 24

)

∗

∗

∗

UED Goal Diversity (

k

= 24

)

PLR

⊥

ACCEL

Valid

Unique

(b)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Updates

×

10

4

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

1

.

75

Returns

×

10

1

Final Episode Returns

∗

ODS

DIVA

ACCEL

PLR

⊥

DR

DIVA

ACCEL

PLR

⊥

DR

Method

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Success Rate

Success Rate by Episode

τ

1

τ

2

(c)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Updates

×

10

4

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

1

.

75

Returns

×

10

1

Final Episode Returns

∗

ODS

DIVA

ACCEL

PLR

⊥

DR

DIVA

ACCEL

PLR

⊥

DR

Method

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

0

.

9

Success Rate

Success Rate by Episode

τ

1

τ

2

(d)

increases. When we fix the complexity ( k = 24 ) and train over the E U distribution, we see that the UED approaches are unable to incidentally discover and capture diversity over the course of training (Figure 4b). DIVA's explicit focus on capturing meaningful level diversity enables it to significantly outperform these baselines in terms of episodic return (Figure 4c) and success rate (Figure 4d).

5.2 ALCHEMY

ALCHEMY [18] is an artificial chemistry environment with a combinatorially complex task distribution. Each task is defined by some latent chemistry , which influences the underlying dynamics, as well as agent observations. To successfully maximize returns over the course of a trial, the agent must infer and exploit this latent chemistry. At the start of each episode, the agent is provided a new set of (1-12) potions and (1-3) stones , where each stone has a latent state defined by a specific vertex of a cube, i.e. ( { 0 , 1 } , { 0 , 1 } , { 0 , 1 } ) , and each potion has a latent effect , or specific manner in which it transforms stone latent states (see Figure 5a). The agent observes only salient artifacts of this latent information, and must use interactions to identify the ground-truth mechanics. At each step, the agent can apply any remaining potion to any remaining stone. Each stone's value is maximized the closer its latent state is to (1 , 1 , 1) , and rewards are produced when stones are cast into the cauldron .

To make training feasible on academic resources, we perform evaluations on the symbolic version of ALCHEMY, as opposed to the full Unity-based version. Symbolic ALCHEMY contains the same mechanistic complexity, minus the visuomotor challenges which are irrelevant to this project's aims.

Parameterization. E S ( θ ) is the downstream distribution containing maximal stone diversity. For training, implement E U k where k controls the level of difficulty in generating diverse stones. Specifically, we introduce a larger set of coordinating genes θ j ∈ { 0 , 1 } that together specify the initial stone latent states, similar to the mechanism we used in GRIDNAV to limit goal diversity. Each stone latent coordinate is specified with k genes, and only when all k genes are set to 1 is the latent coordinate is set to 1 . When any of the genes are 0 , the latent coordinate is 0. For our experiments we set k = 8 , and henceforth use E U to signify E U 8 .

QDupdates. We use features LATENTSTATEDIVERSITY and MANHATTANTOOPTIMAL -both of which target stone latent state diversity from different angles. See Appendix B.2 for more specifics on these features and other details surrounding ALCHEMY's archive construction. Like GRIDNAV, the objective is set to bias new solutions. DIVA is provided with N S1 = 8 . 0 × 10 4 and N S2 = 3 . 0 × 10 4 QD update iterations. PLR ⊥ and ACCEL are compensated such that they receive 3.5 × more additional step data than what DIVA receives via QD updates (see Appendix E.1 for details).

Results. Our empirical results demonstrate that DIVA is able to generate latent stone states with diversity representative of the target distribution. We see this both quantitatively in Figure 5b, and qualitatively in Figure 6. In Figure 5c, we see this diversity translates to significantly better results

(0, 0, 0)

(1, 1, 1)

y

z

x

P

1

P

2

(a)

LatentStateDiversity

Feature Values

Density

E

S

DIVA

E

U

(b)

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

Updates

×

10

4

0

1

2

3

4

5

6

7

Returns

Final Episode Returns

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

Updates

×

10

4

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

Quantity

×

10

4

∗

ODS

DIVA

ACCEL

PLR

⊥

DR

(c)

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

Updates

×

10

4

0

1

2

3

4

5

6

7

Returns

Final Episode Returns

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

Updates

×

10

4

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

Quantity

×

10

4

∗

ODS

DIVA

ACCEL

PLR

⊥

DR

(d)

x1

3

1

2

3

2

1

4

5

6

4

5

6

Recency

Early samples

Later samples

LatentStateDiversity

Archive after S1

on E S over baselines. Despite generating roughly as many unique genotypes as DIVA (Figure 5d), PLR ⊥ and ACCEL are unable to generate training stone sets of significant phenotypical diversity to enable success on the downstream distribution.

5.3 RACING

Lastly, we evaluate DIVA on the RACING domain introduced by [17]. In this environment, the agent controls a race car via simulated steering and gas pedal mechanisms, and is rewarded for efficiently completing the track, M i ∈ T . We adapt this RL environment to the meta-RL setting by lowering the resolution of the observation space significantly. By increasing the challenge of perception, even competent agents benefit from multiple episodes to better understand the underlying track. For all of our experiments, we use H = 2 episodes per trial, and a flattened 15 × 15 pixel observation space.

Setup. We use three different parameterizations in our experiments: (1) E S ( θ ) is the downstream distribution we use for evaluating all methods, training ODS, and setting archive bounds for DIVA. Parameters θ are used to seed the random generation of control points which in turn parameterize a sequence of Bézier curves designed to smoothly transition between the control locations. Track diversity is further enforced by rejecting levels with control points that possess a standard deviation below a certain threshold. (2) E U k ( θ ) is a reparameterization of E S ( θ ) that makes track diversity harder to generate, with the difficulty proportional to the value of k ∈ N . For our experiments, we use k = 32 (which we will denote simply as E U ( θ ) ), which roughly means that meaningful diversity is 32 × less likely to randomly occur than when k = 1 (which is equivalent to E S ( θ ) ). This is achieved by defining a small region in the center, 32 (or k , in general) times smaller than the track boundaries, where all points outside the region are projected onto the unit square, and scaled to the track size. (3) E F1 ( θ ) uses θ as an RNG seed to select between a set of 20 hand-crafted levels official Formula-1 tracks [17], and is used to benchmark DIVA's zero-shot generalization to a new target distribution.

QD updates. We define features TOTALANGLECHANGES (TAC) and CENTEROFMASSX (CX) for the archive dimensions. Levels from E U lack curvature (see Figure 8) so TAC, which is defined as the sum of angle changes between track segments, is useful for directly targeting this desired curvature.

CenterOfMassX

TotalAngleChanges

Feature Values

Density

E

S

E

F1

DIVA

E

U

(a)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Updates

×

10

4

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Returns

×

10

3

Final Episode Returns

∗

ODS

DIVA

ACCEL

PLR

⊥

DR

80%

90%

100%

Completion Threshold

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

Success Rate

Track Completion

∗

ODS

DIVA

ACCEL

PLR

⊥

DR

(b)

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Updates

×

10

4

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Returns

×

10

3

Final Episode Returns

∗

ODS

DIVA

ACCEL

PLR

⊥

DR

80%

90%

100%

Completion Threshold

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

Success Rate

Track Completion

∗

ODS

DIVA

ACCEL

PLR

⊥

DR

(c)

Figure 8 source

(took screenshots from here)

Figure 8 source

(took screenshots from here)

CenterOfMassX

TotalAngleChanges

Archive at

t

= 0

J

CenterOfMassX

TotalAngleChanges

Final archive

J

M

CX, or the average location of the segments, targets diversity in the location of these high-density (high-curvature) regions. We compute an alignment objective over features CENTEROFMASSY and VARIANCEY to further target downstream diversity. See Appendix B.3 for more details relevant to the archive construction process for RACING. DIVA is provided with 2 . 5 × 10 5 initial QD updates on RACING. PLR ⊥ and ACCEL are compensated with 4.0 × more additional step data than what DIVA receives through QD updates (see Appendix E.1 for more details).

Main results. Results are shown in Figure 7. DIVA outperforms all baselines, including the UED approaches, which have access to three times as many environment interactions. From Figure 8, we see that final DIVA levels contain significantly more diversity than randomization over E U. F1 results

Transfer to F1 tracks. Next, we evaluate the ability of these trained policies to zero-shot transfer to humandesigned F1 levels [17], E F1. Though qualitative differences are apparent (see Figure 9), from Figure 7a we can additionally see how these levels differ quantitatively. Even though DIVA uses feature samples from E S to define its archive, we see from the results in Figure 9 that DIVA is not only able to complete many of these tracks, but is also able to significantly outperform ODS. This result may seem unlikely, given that DIVA bases its axes of diversity on E S. One possible explanation is that while DIVA successfully matches its TOTALANGLECHANGES distribution to E S (see Figure 7), because it is less likely for all 12 control points to be mutated to the diversity-enabling region than just a few control points with sharp angles, DIVA 'opts" for the latter, and thus produces fewer, sharper angles, which is evidently useful for transferring to ( these ) human-designed tracks. This hypothesis matches what we see qualitatively from the DIVA-produced levels in Figure 8.

80%

90%

100%

Completion Threshold

0

.

00

0

.

05

0

.

10

0

.

15

Success Rate

Track Completion on

E

F1 (

θ

)

∗

ODS

DIVA

DR

Combining DIVA and UED. While PLR ⊥ and ACCEL struggle on our evaluation domains, they still have utility of their own, which we hypothesize may be compatible with DIVA's. As a preliminary experiment to evaluate the potential of such a combination, we introduce DIVA+ , which still uses DIVA to generate diverse training samples via QD, but additionally uses PLR ⊥ to define

80%

90%

100%

Completion Threshold

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Success Rate

Track Completion

DIVA

1

DIVA

1

+

DIVA

2

DIVA

2

+

6 Related work

Meta-reinforcement learning. Meta-reinforcement learning methods range from gradient-based approaches (e.g. MAML) [19], RNN context-based approaches [12, 11] (e.g. RL 2 ), and the slew of emerging works utilizing transformers [20, 5, 21]. We use VariBAD [10], a state-of-the-art context variable-based approach that extends RL 2 by using variational inference to incorporate task uncertainty into its beliefs. HyperX [22], an extension that uses reward-bonuses, was not found to improve performance on our domains. In each of these works, the training distribution is given; none address the problem of generating diverse training scenarios in absence of such a distribution.

Procedural environment generation. Procedural (content) generation (PCG / PG) [6] is a vast field. Many RL and meta-RL domains themselves have PG baked-in (e.g. ProcGen [23], Meta-World, [24], Alchemy [18], and XLand [5]). Each of these works rely on human engineering to produce levels with meaningfully diverse features. A related stream of works apply scenario generation to robotics-some works essentially perform PCG [25, 26], while others integrate more involved search mechanics [27, 28, 29, 30]. One prior work [31] defines a formal but generic parameterization for applying PG to generate meta-RL tasks. It is yet to be shown, however, if such an approach can scale to domains with vastly different dynamics, and greater complexity.

Unsupervised environment design. UED approaches-which use behavioral metrics to automatically define and adapt a curriculum of suitable tasks for agent training-form the frontier of research on open-endedness. The recent stream of open-ended agent/environment co-evolution works (e.g. [32, 33, 34]) was kickstarted by the POET [35, 36] algorithm. The 'UED' term itself originated in PAIRED [8], which uses the performance of an 'antagonist' agent to define the curriculum for the main (protagonist) agent. PLR [7] introduces an approach for weighting training levels based on learning potential , using various proxy metrics to capture this high-level concept. [17] introduces PLR ⊥ , which only trains on levels that have been previously evaluated, and thus enabling certain theoretical robustness guarantees. AdA [5] uses PLR as a cornerstone of their approach for generating diverse training levels for adaptive agents in a complex, open-ended task space. ACCEL [9] borrows PLR ⊥ 's scoring procedure, but the best-performing solutions are instead mutated, so the buffer not only collects and prioritizes levels of higher learning potential, but evolves them. We use ACCEL as our main baseline because it has demonstrated state-of-the art results on relevant domains, and like DIVA, evolves a population of levels. The main algorithmic differences between ACCEL and DIVA are that ACCEL (1) performs additional evaluation rollouts to produce scores during training and (2) uses a 1-d buffer instead of DIVA's multi-dimensional archive. PLR ⊥ serves as a secondary baseline in this work; its non-evolutionary nature makes it a useful comparison to DR.

Scenario generation via QD. A number of recent works apply QD to simulated environments in order to generate diverse scenarios, with distinct aims. Some works, like DSAGE [37], uses QD to develop diverse levels for the purpose of probing a pretrained agent for interesting behaviors. In another line of work applies QD to human-robot interaction (HRI), and ranges from generating a new distribution over these levels based on approximate learning potential. Instead of randomly sampling levels from E U, the PLR ⊥ evaluation mechanism samples levels from the DIVA-induced distribution over the archive. We perform experiments on two different archives generated by DIVA: (1) an archive that is slightly misspecified (see Appendix B.3 for details), and (2) the archive used in our main results. From Figure 10, we see that while performance does not significantly improve for (2), the combination of DIVA and PLR ⊥ is able to significantly improve performance on (1), and even statistically match the original DIVA results. These results highlight the potential of such hybrid (QD+UED) semi-supervised environment design (SSED) approaches, a promising area for future work.

diverse scenarios [38], to finding failure modes in shared autonomy systems [39] and human-aware planners [40]. DIVA's application of QD inspired by these approaches, as they produce meaningfully diverse environment scenarios, but no prior work exists which applies QD to define a task distribution for agent training , much less adaptive agent training, or overcoming difficult parameterizations in open-ended environments.

7 Discussion

The present work enables adaptive agent training on open-ended environment simulators by integrating the unconstrained nature of unsupervised environment design (UED) approaches, with the implicit supervision baked into procedural generation (PG) and domain randomization (DR) methods. Unlike PG and DR, which requires domain knowledge to be carefully incorporated into the environment generation process, DIVA is able to flexibly incorporate domain knowledge, and can discover new levels representative of the downstream distribution. And instead of relying on behavioral metrics to infer a general, ungrounded form of 'learning potential', like UED-which becomes increasingly unconstrained and therefore less useful a signal as environments become more complex and open-ended-DIVA is able to directly incorporate downstream feature samples to target specific, meaningful axes of diversity. With only a handful of downstream feature samples to set the parameters of the QD archive, our experiments (Section 5) demonstrate DIVA's ability to outperform competitive baselines compensated with three times as many environment steps during training.

In its current form, the most obvious limitation of DIVA is that, in addition to assuming access to downstream feature samples, the axes of diversity themselves must be specified. However, we imagine these axes of diversity could be learned automatically from a set of sample levels, or selected from a larger set of candidate features; it may be possible to adapt existing QD works to automate this process in related settings [41]. The present work also lacks a more thorough analysis of what constitutes good archive design. While some amount of heuristic decision-making is unavoidable when applying learning algorithms to specific domains, a promising future direction would be to study how to approach DIVA's archive design from a more algorithmic perspective.

DIVA currently performs QD iterations over the environment parameter space defined by E U ( θ ) , where each component of the genotype θ represents some salient input parameter to the simulator. Prior works in other domains (e.g. [42]) have demonstrated QD's ability to explore the latent space of generative models. One natural direction for future work would therefore be to apply DIVA to neural environment generators (rather than algorithmic generators), where θ would instead correspond to the latent input space of the generative model. If the latent space of these models is more convenient to work with than the raw environment parameters-e.g. due to greater smoothness with respect to meaningful axes of diversity-this may help QD more efficeintly discover samples within the target region. Conversely, DIVA's ability to discover useful regions of the parameter space means these neural environment generators do not need to be 'well-behaved", or match a specific target distribution. Since these generative models are also likely to be differentiable, DIVA can additionally incorporate gradient-based QD works (e.g. DQD [15]) to accelerate its search.

Preliminary results with DIVA+ demonstrate the additional potential of combining UED and DIVA approaches. The F1 transfer results (i.e. DIVA outperforming ODS trained directly on E S) further suggest that agents benefit from flexible incorporation of downstream knowledge. In future work, we hope to study more principle integrations of UED and DIVA-like approaches, and to more generally explore this exciting new area of semi-supervised environment design (SSED).

More broadly, now equipped with DIVA, researchers can develop more general-purpose, open-ended simulators, without concerning themselves with constructing convenient, well-behaved parameterizations. Evaluations in this work required constructing our own contrived paramterizations, since domains are rarely released without carefully designed parameterizations. It is no longer necessary to accomodate the assumption made my DR, PG, and UED approaches-that either randomization over the parameter space should produce meaningful diversity, or that all forms of level difficulty ought to correspond to meaningful learning potential. So long as diverse tasks are possible to generate, even if sparsely distributed within the paramter space, QD may be used to discover these regions, and exploit them for agent training. Based on the promising empirical results presented in this work, we are hopeful that DIVA will enable future works to tackle even more complicated domains, and assist researchers in designing more capable and behaviorally interesting adaptive agents.

8 Reproducibility statement

The source code, along with thorough documentation for reproducing each result in this paper, is publicly available on Github 6 . Even without this code, researchers should be able to fully reproduce the algorithm from the details in the main body, the pseudocode provided in Appendix A, and training details (hyperparameters and hardware information) provided in Appendix E.

9 Ethics statement

Like all fundamental technologies, this work has the potential to be misapplied for malicious purposes. The authors do not believe, however, that the methods introduced in this work present a significant or unique risk for misuse or abuse. The authors intend for DIVA to be applied to use-cases that have the best interests of humanity (including concern for the earth and other sentient creatures) at heart.

References

X. Wang, S. Wang, X. Liang, D. Zhao, J. Huang, X. Xu, B. Dai, and Q. Miao, 'Deep reinforcement learning: A survey,' IEEE Trans. Neural Netw. Learn. Syst. , vol. 35, pp. 5064-5078, Apr. 2024.

K. Sivamayil, E. Rajasekar, B. Aljafari, S. Nikolovski, S. Vairavasundaram, and I. Vairavasundaram, 'A systematic study on reinforcement learning based applications,' Energies , vol. 16, p. 1512, Feb. 2023.

R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktäschel, 'A survey of zero-shot generalisation in deep reinforcement learning,' jair , vol. 76, pp. 201-264, Jan. 2023.

J. Beck, R. Vuorio, E. Z. Liu, Z. Xiong, L. Zintgraf, C. Finn, and S. Whiteson, 'A survey of metareinforcement learning,' arXiv [cs.LG] , Jan. 2023.

J. Bauer, K. Baumli, F. Behbahani, A. Bhoopchand, N. Bradley-Schmieg, M. Chang, N. Clay, A. Collister, V. Dasagi, L. Gonzalez, K. Gregor, E. Hughes, S. Kashem, M. Loks-Thompson, H. Openshaw, J. ParkerHolder, S. Pathak, N. Perez-Nieves, N. Rakicevic, T. Rocktäschel, Y. Schroecker, S. Singh, J. Sygnowski, K. Tuyls, S. York, A. Zacherl, and L. M. Zhang, 'Human-timescale adaptation in an open-ended task space,' in Proceedings of the 40th International Conference on Machine Learning (A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, eds.), vol. 202 of Proceedings of Machine Learning Research , pp. 1887-1935, PMLR, 23-29 Jul 2023.

N. Shaker, J. Togelius, and M. J. Nelson, Procedural Content Generation in Games . Computational Synthesis and Creative Systems, Springer, 2016.

M. Jiang, E. Grefenstette, and T. Rocktäschel, 'Prioritized Level Replay,' in Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event (M. Meila and T. Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research , pp. 4940-4950, PMLR, 2021.

M. Dennis, N. Jaques, E. Vinitsky, A. Bayen, S. Russell, A. Critch, and S. Levine, 'Emergent complexity and zero-shot transfer via unsupervised environment design,' Advances in neural information processing systems , vol. 33, pp. 13049-13061, 2020.

J. Parker-Holder, M. Jiang, M. Dennis, and others, 'Evolving curricula with regret-based environment design,' International Conference on Machine Learning , 2022.

L. M. Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson, 'VariBAD: A Very Good Method for Bayes-adaptive Deep RL via Meta-learning,' in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 , OpenReview.net, 2020.

Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, 'RL 2 : Fast Reinforcement Learning via Slow Reinforcement Learning,' arXiv:1611.02779 [cs, stat] , Nov. 2016.

J. Wang, Z. Kurth-Nelson, H. Soyer, J. Z. Leibo, D. Tirumala, R. Munos, C. Blundell, D. Kumaran, and M. M. Botvinick, 'Learning to reinforcement learn,' in Proceedings of the 39th Annual Meeting of the Cognitive Science Society, CogSci 2017, London, UK, 16-29 July 2017 (G. Gunzelmann, A. Howes, T. Tenbrink, and E. J. Davelaar, eds.), cognitivesciencesociety.org, 2017.

L. Zintgraf, K. Shiarli, V . Kurin, K. Hofmann, and S. Whiteson, 'Fast context adaptation via meta-learning,' in International Conference on Machine Learning , pp. 7693-7702, PMLR, 2019.

K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen, 'Efficient Off-policy Meta-reinforcement Learning via Probabilistic Context Variables,' in Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA (K. Chaudhuri and R. Salakhutdinov, eds.), vol. 97 of Proceedings of Machine Learning Research , pp. 5331-5340, PMLR, 2019.

M. C. Fontaine and S. Nikolaidis, 'Differentiable Quality Diversity,' in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual (M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, eds.), pp. 10040-10052, 2021.

J. Mouret and J. Clune, 'Illuminating search spaces by mapping elites,' CoRR , vol. abs/1504.04909, 2015.

M. Jiang, M. Dennis, J. Parker-Holder, J. N. Foerster, E. Grefenstette, and T. Rocktäschel, 'Replay-guided Adversarial Environment Design,' in Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual (M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, eds.), pp. 1884-1897, 2021.

J. X. Wang, M. King, N. P. M. Porcel, Z. Kurth-Nelson, T. Zhu, C. Deck, P. Choy, M. Cassin, M. Reynolds, H. F. Song, et al. , 'Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents,' in Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) , 2021.

C. Finn, P. Abbeel, and S. Levine, 'Model-agnostic Meta-learning for Fast Adaptation of Deep Networks,' in Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017 (D. Precup and Y. W. Teh, eds.), vol. 70 of Proceedings of Machine Learning Research , pp. 1126-1135, PMLR, 2017.

L. C. Melo, 'Transformers are Meta-Reinforcement Learners,' in Proceedings of the 39th International Conference on Machine Learning (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, eds.), vol. 162 of Proceedings of Machine Learning Research , pp. 15340-15359, PMLR, 2022.

J. Grigsby, L. Fan, and Y. Zhu, 'AMAGO: Scalable in-context reinforcement learning for adaptive agents,' in The Twelfth International Conference on Learning Representations , 2024.

L. M. Zintgraf, L. Feng, C. Lu, M. Igl, K. Hartikainen, K. Hofmann, and S. Whiteson, 'Exploration in Approximate Hyper-state Space for Meta Reinforcement Learning,' in Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event (M. Meila and T. Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research , pp. 12991-13001, PMLR, 2021.

K. Cobbe, C. Hesse, J. Hilton, and J. Schulman, 'Leveraging Procedural Generation to Benchmark Reinforcement Learning,' in Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , vol. 119 of Proceedings of Machine Learning Research , pp. 2048-2056, PMLR, 2020.

T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, 'Meta-World: A Benchmark and Evaluation for Multi-task and Meta Reinforcement Learning,' in 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings (L. P. Kaelbling, D. Kragic, and K. Sugiura, eds.), vol. 100 of Proceedings of Machine Learning Research , pp. 1094-1100, PMLR, 2019.

J. Arnold and R. Alexander, 'Testing autonomous robot control software using procedural content generation,' in Lecture Notes in Computer Science , Lecture notes in computer science, pp. 33-44, Berlin, Heidelberg: Springer Berlin Heidelberg, 2013.

D. J. Fremont, T. Dreossi, S. Ghosh, X. Yue, A. L. Sangiovanni-Vincentelli, and S. A. Seshia, 'Scenic: a language for scenario specification and scene generation,' in Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation , PLDI 2019, (New York, NY, USA), p. 63-78, Association for Computing Machinery, 2019.

G. E. Mullins, P. G. Stankiewicz, R. C. Hawthorne, and S. K. Gupta, 'Adaptive generation of challenging scenarios for testing and evaluation of autonomous vehicles,' J. Syst. Softw. , vol. 137, pp. 197-215, Mar. 2018.

Y. Abeysirigoonawardena, F. Shkurti, and G. Dudek, 'Generating adversarial driving scenarios in highfidelity simulators,' in 2019 International Conference on Robotics and Automation (ICRA) , pp. 8271-8277, IEEE, May 2019.

A. Gambi, M. Mueller, and G. Fraser, 'Automatically testing self-driving cars with search-based procedural content generation,' in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis , (New York, NY, USA), ACM, July 2019.

Y. Zhou, S. Booth, N. Figueroa, and J. Shah, 'Rocus: Robot controller understanding via sampling,' in Proceedings of the 5th Conference on Robot Learning (A. Faust, D. Hsu, and G. Neumann, eds.), vol. 164 of Proceedings of Machine Learning Research , pp. 850-860, PMLR, 08-11 Nov 2022.

T. Miconi, 'Procedural generation of meta-reinforcement learning tasks,' Feb. 2023.

T. Gabor, A. Sedlmeier, M. Kiermeier, T. Phan, M. Henrich, M. Pichlmair, B. Kempter, C. Klein, H. Sauer, R. S. Ag, and J. Wieghardt, 'Scenario co-evolution for reinforcement learning on a grid world smart factory domain,' in Proceedings of the Genetic and Evolutionary Computation Conference , (New York, NY, USA), ACM, July 2019.

D. M. Bossens and D. Tarapore, 'QED: Using quality-environment-diversity to evolve resilient robot swarms,' IEEE Trans. Evol. Comput. , vol. 25, pp. 346-357, Apr. 2021.

A. Dharna, J. Togelius, and L. B. Soros, 'Co-generation of game levels and game-playing agents,' in Proceedings of the Sixteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , AIIDE'20, AAAI Press, 2020.

R. Wang, J. Lehman, J. Clune, and K. O. Stanley, 'Paired open-ended trailblazer (POET): Endlessly generating increasingly complex and diverse learning environments and their solutions,' arXiv [cs.NE] , Jan. 2019.

R. Wang, J. Lehman, A. Rawal, J. Zhi, Y. Li, J. Clune, and K. Stanley, 'Enhanced POET: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions,' in Proceedings of the 37th International Conference on Machine Learning (H. D. III and A. Singh, eds.), vol. 119 of Proceedings of Machine Learning Research , pp. 9940-9951, PMLR, 13-18 Jul 2020.

V. Bhatt, B. Tjanaka, M. Fontaine, and S. Nikolaidis, 'Deep surrogate assisted generation of environments,' Advances in Neural Information Processing Systems , vol. 35, pp. 37762-37777, 2022.

D. Gravina, A. Khalifa, A. Liapis, J. Togelius, and G. N. Yannakakis, 'Procedural content generation through quality diversity,' in 2019 IEEE Conference on Games (CoG) , pp. 1-8, ieeexplore.ieee.org, Aug. 2019.

M. C. Fontaine and S. Nikolaidis, 'A Quality Diversity Approach to Automatically Generating Humanrobot Interaction Scenarios in Shared Autonomy,' in Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021 (D. A. Shell, M. Toussaint, and M. A. Hsieh, eds.), 2021.

M. C. Fontaine, Y. Hsu, Y. Zhang, B. Tjanaka, and S. Nikolaidis, 'On the Importance of Environments in Human-robot Coordination,' in Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021 (D. A. Shell, M. Toussaint, and M. A. Hsieh, eds.), 2021.

L. Grillotti and A. Cully, 'Unsupervised behavior discovery with quality-diversity optimization,' IEEE Trans. Evol. Comput. , 2022.

A. Khalifa, P. Bontrager, S. Earle, and J. Togelius, 'PCGRL: Procedural Content Generation via Reinforcement Learning,' in Proceedings of the Sixteenth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AIIDE 2020, virtual, October 19-23, 2020 (L. Lelis and D. Thue, eds.), pp. 95-101, AAAI Press, 2020.

B. Tjanaka, M. C. Fontaine, Y. Zhang, S. Sommerer, and others, 'pyribs: A bare-bones python library for quality diversity optimization,' 2021.

Appendix

A Algorithmic details

Algorithm. The pseudocode below walks through the entire training process for DIVA in abstract. All of the new components that DIVA introduces is written in green , and all DIVA+ modifications are in blue . Original VariBAD training steps are in black , and all inline comments are in orange .

Algorithm 2 DIVA (detailed)

Algorithm 3 QD update

Details on the two-stage QD updates Here we provide more details on the process described in Section Section 4. Hyperparameters N S1 and N S2 are set to define the number of QD updates to perform in each stage (see Appendix D). In proportion to how many updates in S1 have elapsed, if the sample mask is enabled, the mask is moved at a linear pace from encapsulating the full S1 archive, to covering only the target region. We also set a hyperparameter, N SM (see Appendix D), which

specifies the minimum number of solutions which must exist within the mask's new bounds for it to be updated. This is to ensure the mask never outpaces the search process. The mask was only found to be necessary in the ALCHEMY environment. In S1 we sample solutions uniformly from within the mask. In S2, we begin sampling from the discretized target density distribution approximated from the downstream feature samples. Two stages are used for RACING as well, since many initial samples fall outside of the target region, but masking was not found to be necessary. The sample mask has a relatively straightforward implementation for MAP-Elites, which we use for ALCHEMY's discrete genotype (and GRIDNAV, where no mask is required). Since MAP-Elite updates entail performing mutations on solutions directly sampled from the archive, the mask is implemented to only consider solutions that fall within the mask bounds. However, since the CMA-ES-based emitter we use for RACING operates by sampling from a parameterized distribution, instead of sampling from the archive directly, the mask would need to be applied to these parameters instead of the archive.

B Domain details

B.1 GRIDNAV

GRIDNAV features. The following features are defined for the GRIDNAV environment:

B.2 ALCHEMY

ALCHEMY features. We defined the following features for the ALCHEMY environment:

Figure 11 contains the feature distributions for the structured and unstructured environment parameterizations on ALCHEMY, computed over 100 feature samples. Figure 14 shows the covariance between feature values for RACING, computed over 100 feature samples.

amto

astsd

lsd

pfp

pfs

ped

sre

sro

stsdv

Feature Values

Density

E

S

E

U

amto

astsd

lsd

pfp

pfs

ped

sre

sro

stsdv

amto

astsd

lsd

pfp

pfs

ped

sre

sro

stsdv

Covariance:

ES

amto

astsd

lsd

pfp

pfs

ped

sre

sro

stsdv

amto

astsd

lsd

pfp

pfs

ped

sre

sro

stsdv

Covariance:

EU

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

Archive hyperparameters for ALCHEMY were determined based on some knowledge about the domain, as well as the feature distributions (Figure 13). We noticed a major deviation between E U and E S in the feature LATENTSTATEDIVERSITY (LSD), and an even greater on in MANHATTANTOOPTIMAL (MTO). These two constitute the initial dimensions of the archive, and we found the sample mask updates to be crucial to reach and fill the target region (see Figure 15). We use PARITYFIRSTSTONE (PFS) in the second stage to encourage more diversity once the target is reached; it is excluded from the first stage, which is focused on simply reaching the target. The archive for the first stage is of shape [100 , 300 , 1] , corresponding to LSD, MTO, and PFS. The second stage shape is [150 , 150 , 5] . We found this archive to produce diverse enough solutions, evidenced by the number and spread in the target region, so we used this setting to train our DIVA agents. The only objective we found useful for ALCHEMY was a slight bias for newly generated solutions, which we also used for GRIDNAV. We hypothesize this prevents the archive from getting 'stuck' with a suboptimal set of solutions, in absense of other objectives.

B.3 RACING

RACING features. See Table 13 for all features defined on RACING. Figure 13 contains the feature distributions for the structured and unstructured environment parameterizations on RACING, computed over 100 feature samples. Figure 14 shows the covariance between feature values for RACING, computed over 100 samples.

Archive hyperparameters for RACING were determined through trial and error, by viewing the samples produced by the archives at the end of the QD updates, as well as the target coverage metrics. After a few iterations, it became clear that Total Angle Change TOTALANGLECHANGES (TAC) was the most useful feature, and so we tried pairing it with a number of others, prioritizing other features with low absolute covariance (see Figure 14).

The best performing archive used by DIVA on RACING uses TAC and CX as its features, and used a measure alignment objective over CY and VY. The measure alignment objective rewards solutions for having measure values over the specific measures that are similar to the target distribution. We also

Feature Values

Name

Abbr.

Description

AREATOLENGTHRATIO

ATLR

The ratio of enclosed area to curve length.

AVERAGECURVATURE

AC

The average curvature at midpoints of Beziér segments.

CENTEROFMASSX

CX

The center of mass x position over the curve.

CENTEROFMASSY

CY

The center of mass y position over the curve.

CURVEDISTANCESVARIANCE

CDV

The variability in distances between successive points.

CURVELENGTH

CL

The total length of the Beziér curve.

ENCLOSEDAREA

EA

The area enclosed by the Beziér curve.

MEDIANX

MX

The median

x

position over the curve.

MY

The median

y

position over the curve.

SIGNIFICANTANGLECHANGES

SAC

The sum of significant angle changes across the curve.

TOTALANGLECHANGES

TAC

The total change in angle across the curve.

TOTALCURVATURE

TC

The total curvature over each segment and sum them up.

VARIANCEX

VX

The variance of the

x

positions over the curve.

VARIANCEY

VY

The variance of the

y

positions over the curve.

atlr

ac

cx

cy

cdv

cl

ea

mx

my

sac

tac

tc

vx

vy

Density

E

S

E

U

found that randomly sampling these objective values according to the target distribution provided some additional support in covering the target region efficiently.

The slightly 'misspecified' archive was chosen because its solutions generated some diversity, but not as much as the aforementioned one. This archive uses TAC and ATLR as its features, and uses a measure diversity objective over just CY. Instead of prioritizing alignment to the target distribution, the diversity objective samples a handful of solutions from the archive, and uses the current solutions deviation from these as its objective.

We use final archive dimensions of 500 × 500 for both S1 and S2.

C Ablation analysis

Sample mask ablation. Figure 15 shows the benefit of updating the sample mask bounds during the first archive filling stage on ALCHEMY. Not only does this approach produce significantly more total archive solutions, but more importantly, progress towards filling the target region specifically is accelerated.

D Hyperparameter sensitivity analysis

Varying QD mutation rate. We perform an ablation on the QD mutation rate, which is the probability that a given gene will be mutated (for MAP-Elites). We perform this ablation on ALCHEMY because the its search is the most challenging of the three environments we consider (it is the sole environment that required a longer S1 and the sample mask trick for accelerating to accelerate the search). We see from Figure 16 that ALCHEMY results are not very sensitive to the setting of the mutation rate.

atlr

ac

cx

cy

cdv

cl

ea

mx

my

sac

tac

tc

vx

vy

atlr

ac

cx

cy

cdv

cl

ea

mx

my

sac

tac

tc

vx

vy

Covariance:

ES

atlr

ac

cx

cy

cdv

cl

ea

mx

my

sac

tac

tc

vx

vy

atlr

ac

cx

cy

cdv

cl

ea

mx

my

sac

tac

tc

vx

vy

Covariance:

EU

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

-

1

.

00

-

0

.

75

-

0

.

50

-

0

.

25

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

1

2

3

# QD Updates (S1)

10

4

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

# of Solutions

×

10

3

# Target Solutions

0

1

2

3

# QD Updates (S1)

10

4

0

1

2

3

4

# of Solutions

×

10

3

# Archive Solutions

Masking

No masking

×

×

Varying number of QD updates. We perform a similar ablation to test how robust DIVA is to the number of QD updates performed. We see from Figure 17 that ALCHEMY results suffer somewhat from fewer updates (e.g. for only 10k in each stage), still significantly outperform baselines in each case. The trend is clear, however: more QD updates produces more solutions, which generally translates to better performance, even if slightly.

Varying number of downstream samples. Next we test how robust DIVA is to the number of downstream samples used to compute the target distribution. In Figure 18 we see that, despite the errors increasing with fewer samples, DIVA still significantly outperforms baselines with as few as five samples .

E Training details

E.1 DIVA hyperparameters

Table 4 displays the hyperparameters used for DIVA across all domains.

A note on N TRS computation The initial QD population ( n 0 ) is implemented such that the first set of QD updates simply generates n 0 random levels from E U , before performing the actual mutations (for ME) or intelligent sampling (for ES). Thus, the formula we use for computing N TRS, the total reset steps provided to DIVA (see Table 4), which we use to compare the extra steps we provide PLR ⊥ and ACCEL (discussed in Section 5), does not include n 0 ; it is simply the product of the batch size and the total number of QD iterations.

0.005

0.01

0.04

0.08

Mutation Rate

0

2

4

6

Final Episode Return

Final Return by Mutation Rate

Method

DIVA

∗

ODS

PLR

⊥

0.005

0.01

0.04

0.08

Mutation Rate

0

2000

4000

6000

8000

# Target Solutions

# Target Solutions by Mutation Rate

10000

20000

40000

120000

# QD Updates

0

2

4

6

Final Episode Returns

(log scale)

Returns by # QD Updates

Method

DIVA

∗

ODS

PLR

⊥

10000

20000

40000

120000

# QD Updates

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

# Target Solutions

×

10

4

(log scale)

# Target Solutions by # QD Updates

E.2 VariBAD hyperparameters

Table 5 displays the hyperparameters used for VariBAD across all domains.

E.3 Baseline hyperparameters

E.3.1 PLR ⊥

Table 6 displays the hyperparameters used for PLR ⊥ across all domains.

E.3.2 ACCEL

ACCEL uses the same hyperparameters as PLR ⊥ (see Table 6), combined with the same evolutionary hyperparameters used for DIVA's QD archive (see Table 4).

E.4 Computational details

All results were produced on a handful of Titan X or Xp GPUs. Environments were parallelized across multiple CPU cores to accelerate training. While the experiment time varies by method and environment, most experiments take less than a day to run to completion. PLR ⊥ and ACCEL take the longest, as they required twice as many environment steps as the other methods-on the two latter domains, these methods take well over a day to run to completion.

5

10

25

50

100

500

0

2

4

6

Final Episode Returns

(log scale)

Method

DIVA

∗

ODS

PLR

⊥

5

10

25

50

100

500

2

4

6

Sample Means MAE

×

10

-

2

AverageManhattanToOptimal

Sample Means MAE

Sample Variances MAE

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

Sample Variances MAE

×

10

-

2

(log scale)

5

10

25

50

100

500

0

.

5

1

.

0

1

.

5

2

.

0

Sample Means MAE

×

10

-

2

LatentStateDiversity

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Sample Variances MAE

×

10

-

2

(log scale)

† GRIDNAV, glyph[star] ALCHEMY, glyph[diamondmath] RACING

† GRIDNAV, glyph[star] ALCHEMY, glyph[diamondmath] RACING

† GRIDNAV, glyph[star] ALCHEMY, glyph[diamondmath] RACING

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The two main claims in the abstract are as follows: 'Our empirical results demonstrate DIVA's [1] unique ability to leverage ill-parameterized simulators to train adaptive behavior in meta-RL agents, [2] far outperforming competitive baselines." (1) : This ability is indeed 'unique', as far as the authors are aware, and is supported by the discussion of related works in Section 6. (2) : The empirical results in this paper presented in Section 5 support the claim that DIVA far outperforms other baselines for training meta-RL agents. Specific pieces of evidence include the GRIDNAV results in Figure 4, the ALCHEMY results in Figure 5, and the results in RACING, contained in Figure 7, 9, and 10. The final sentence in the abstract makes a more general, weaker claim about the presented method's potential: 'These findings highlight the potential of approaches like DIVA to enable training in complex open-ended domains, and to produce more robust and adaptable agents." The authors believe the same set of evidence supports this claim as well, but is ultimately up to the reader-and more importantly, future work-to decide.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The authors discuss limitations of the present work in Section 7. This Discussion section covers (1) key takeaways from the paper, (2) limitations, and (3) promising avenues for future work. The authors combine these sections because the points are all interrelated, and having them in the same place preserves cohesion and flow of the writing.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include any formalized theoretical results. The authors did not find reason to provide any theorems, formulas, or proofs to support the claims made.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In addition to the provided code (see Section 8), which is self-contained and includes documentation for reproducing all empirical results in this paper, the authors additionally include all training details, including hyperparameter settings for all methods, in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is available and well-documented (see Section 8), and contains sufficient instructions to faithfully reproduce the main experimental results. Additionally, all necessary details required to independently reproduce the results are self contained in the paper, with hyperparameter settings and other details fully described in Appendix E.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Appendix E for all relevant training and test details not covered in the main body. Additionally, details about each evaluation domain, if likewise not covered in the main body, are available in Appendix B. Lastly, all of these details are self-contained and documented in the code (see Section 8).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All results (excepting visualizations, and the trend curve in Figure 4 (a), for which they are not needed) are accompanied with error bars that capture the factors of variability, mostly due to random seeding (affecting many different random settings of the algorithm, e.g. initial model weights, sampling, etc.). The significance of all error bars, and number of seeds used for each experiment, are detailed in Section 5. Other training details are available in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All compute resources (workers, memory, time of execution) are detailed in Appendix E.4.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have read and understood the NeurIPS Code of Ethics, and the paper conforms to the code in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The authors specifically discuss the broader impacts of the present work in Section 9.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The authors feel that such safeguards are unnecessary for the present work, as the risk for misuse is estimated by the authors to be very low-it is unclear how this work and its artifacts can be directly used for malicious purposes. The authors are willing to adjust this position if members of the community feel otherwise.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All used resources are either cited properly in the paper and/or are documented and credited in the codebase (see Section 8).

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The new asset introduced in this paper is the codebase (see Section 8), which is well-documented for the purpose of reproducibility, and contains details about training, the license, and limitations, etc.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper applies quality diversity (QD) optimization (evolutionary method) to the problem of diverse task generation for (meta) reinforcement learning (RL). It argues that QD could be used in settings where an open-ended simulator’s parameterization is unlikely to produce tasks that are diverse in high-level features. The method works by handcrafting a set of high-level task features that are relevant to the learning process of the RL agent, then running QD to collect a set of diverse parameterizations that cover the feature space distribution well. A (meta) RL agent is then trained on tasks sampled from a distribution based on the set of QD optimised tasks. The paper’s experiments on GridNav, Alchemy, and Racing tasks show significant improvement in agent performance over existing baselines such as robust prioritised level-replay.\n\nSTRENGTHS:\nFocuses on high-level features more relevant to downstream tasks that the meta-RL agent should learn to adapt in, rather than focusing on simulator parameters as in prior unsupervised environment design (UED) work.\n\nBy shifting the focus in task distribution design for meta-RL training, this paper showed significant improvements on representative existing UED solutions, on a diverse set of evaluation tasks (GridNav, Alchemy, Racing). The evaluations are well-designed with sufficient ablations. \n\nMakes a connection between QD and UED for meta-RL training, which is novel to the best of this reviewer’s knowledge.\n\nThe paper is very clearly written and illustrated. Relevant related works are discussed and contributions of the work are put into appropriate context. Content is well self-contained despite introducing new methods and tasks.\n\nWEAKNESSES:\nIt is still necessary to hand-craft the high-level features, which needs expert knowledge (or at least quite high familiarity with downstream tasks) and can be heuristic. \n\nEvaluation only used one meta-RL method (VariBAD). It would be useful to see performance of other meta-RL methods such as RL^2 on the DIVA task distribution.\n\nQUESTIONS:\nDoes DIVA depend on any parameterizations at all? It seems like it might be possible (and very useful) to try DIVA on “weakly” parameterized tasks such as those generated by prompting an LLM or, say Genie [1].\n\n[1] Genie: Generative Interactive Environments. Bruce el al. https://arxiv.org/abs/2402.15391\n\nLIMITATIONS:\nLimitations have been adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DIVA, a technique for exploring the parameter space of parametrisable environments. The technique uses a variant of MAP-Elites to explore the environment parameter space, finding exemplar points spread across the parameter space, as measured with respect to some user provided features. The authors show that this generates a usefully diverse collection of environment instances by training a meta-learning agent on the generated levels and comparing it to a number of baselines.\n\nSTRENGTHS:\nOn the whole the paper is well written and clear.\n\nThe authors choose a sensible selection of baseline algorithms to compare against, giving the reader an understanding of the strengths of their approach.\n\nWEAKNESSES:\nThe primary weakness from my perspective is that domains that the experiments was conducted on are all very simple “toy” domains. In particular, these domains do not have all have intrinsically complex parameterisations, and for two of them the authors had to, in effect, obfuscate the parameter spaces so the algorithm had a challenge to work against. This leads one to worry that the authors’ results may not be representative of more realistic open-ended domains where the parameter space complexity may manifest in a different way.\n\nAnother weakness of the technique is the authors need to hand-select the features used by the algorithm, per domain. The authors mention that this could be automated, but the fact still stands that in this paper, quite extensive hand tuning and selection - as reported in the appendices - of the objectives was made to get their results. The impact of this technique is much more limited if feature sets need to be hand-tuned per domain, so without demonstrating that this is not the case, I think the authors do not demonstrate that this technique is likely to have wide impact.\n\nI have a further query about the way that the VariBAD hyperparameters were tuned, below in the questions section.\n\nQUESTIONS:\nL36-37: Nit: “a” refers to the singular, “autocurricula” is plural.\n\nL74: Nit: Add “are” between “algorithms” and “in”.\n\nL129-131: Sentence doesn’t parse. Maybe remove “but”, or perhaps something got accidentally removed?\n\nL159-161: I was confused by the experimental approach here. The downstream task distribution was first used to train the meta-learner’s hyperparameters, and then these hyperparameters were used to compare all of the approaches. But isn’t this essentially training on the test set? It feels like that the realistic setting would be one where the meta-learner does not know about the downstream tasks, and it would be important to demonstrate DIVA works under these conditions. I can appreciate that the authors might want to try to “factor” the behaviour of the meta-learner from the behaviour of DIVA, but it seems like a strong assumption that this is possible or useful.\n\nL245: Nit: Figures 8 & 9 are referenced before figure 7.\n\nL278: “Their approach”. Who’s approach?\n\nL306: Define “HRI”.\n\nL312: “DIVA’s” -> “DIVA”.\n\nL312: “incorporating” -> “incorporate”.\n\nLIMITATIONS:\nThe primary limitation is the one regarding hand-tuned feature sets, which I have expanded on above, in the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper identifies the limitation that hand-crafting a sufficiently diverse set of simulated training tasks to bridge any significant sim-to-real gap is labor-intensive. It then proposes DIVA, a new evolutionary approach for generating diverse training tasks in the absence of well-behaved simulator parameterizations. The paper demonstrates how DIVA outperforms proposed baselines such as ACCEL, PLR, and DR.\n\nSTRENGTHS:\n- The paper identifies a key limitation in current methods used to bridge the sim-to-real gap.\n- The proposed approach is novel.\n- There is a good spread of experimental results.\n\nWEAKNESSES:\n- I think the presentation of the paper can be improved. For example, Figure 1 could include more descriptions about what is happening, such as defining E_S(theta) as the structured environment simulator and E_U(theta) as the unstructured environment simulator.\n- In line 128, how was the number 80% chosen? And why is it described as “roughly”? More justifications for the choice of these parameters should be included.\n\nMinor things\n- Typo in line 184, “the final y location is determine by”\n\nQUESTIONS:\n- Why does DIVA want to evolve a population of minimally diverse solutions from the original parameterization? Does DIVA assume that the original parameterizations are what humans care about? If so, doesn’t that mean we still need to ensure that the original parameterizations are handcrafted enough to be close to what humans care about?\n- If the original parameterizations are chosen randomly or are not handcrafted to what humans might care about, how would DIVA perform?\n- To what extent and when do we consider an environment simulator to have structured or unstructured parameterizations?\n- In the GridNav experiment, why do we want to prevent the generation of diverse goals along the y-axis?\n- Is there a hypothesis on why “the domain randomization over increasingly complex genotypes diminishes diversity in terms of goal location” (lines 187-188)?\n- Line 188 claims that “DIVA, on the other hand, is able to capture this diversity.” However, doesn’t DIVA also decrease to the same level of percentage coverage as DR*? Why are there no confidence intervals for Figure 3a, like the others?\n\nLIMITATIONS:\n- Yes, the paper sufficiently covers the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes an approach for learning a QD-archive to be used as a proxy for samples of test environments, and shows that this results in improved performance in producing a set of meta-learning tasks over DR and UED baselines.\n\nSTRENGTHS:\nThe introduction of QD approaches into UED algorithms is a promising algorithm for improvement. The empirical results appear convincing, and the approach is quite natural.\n\nWEAKNESSES:\nThe comparison between DIVA and PLR/ACCEL is a bit of an apples-to-oranges comparison. Regret-based UED methods like PLR and ACCEL are meant to be making decisions under ignorance, where there is no information known about the target distribution. There are other UED approaches designed for the case where there is some information that is known, and thus it is a decision under risk. Specifically, it would be better to compare against SAMPLR, CLUTER, or DRED. Since SAMPLR requires simulator access, the fairest comparison would be against CLUTER or DRED. \n\n[SAMPLR] Jiang, Minqi, et al. \"Grounding aleatoric uncertainty for unsupervised environment design.\" _Advances in Neural Information Processing Systems_ 35 (2022): 32868-32881.\n\n[CLUTER]Azad, Abdus Salam, et al. \"Clutr: Curriculum learning via unsupervised task representation learning.\" _International Conference on Machine Learning_. PMLR, 2023.\n\n[DRED] Garcin, Samuel, et al. \"DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design.\" _Forty-first International Conference on Machine Learning_. 2024.\n\nThat being said, the current results look to have more significant results than these other works (though it is hard to tell across domains), and this approach has not be tried in meta-learning before. It is important to note that UED has been used in meta-learning before, for instance in AdA.\n\n[AdA] Team, Adaptive Agent, et al. \"Human-timescale adaptation in an open-ended task space.\" _arXiv preprint arXiv:2301.07608_ (2023).\n\nIt occurs to me that DIVA + UED as discussed starting at line 259 could be used as an algorithm for decisions under ignorance, and would possibly be quite a good algorithm. It may be worth running the approach from 259, maybe without any test time data, on the traditional maze, F1, and bipedal walker environments and transfer tasks to check if it consistently outperforms existing methods.\n\nIt seems like the assumption about the probability of generating a useful level from the simulator underlying DIVA is similar to the assumption necessary for ACCEL or PLR. Both of these approaches should work if there is a ~0.01 % chance of generating a useful level. The boot-up time would just be a bit slower to get the initial buffer of levels. \n\nThe limitation of online agents evaluations is a real bottleneck for the UED approaches which DIVA avoids, but it is a bit of an odd comparison as DIVA generates the model once offline, and thus isn't aiming to be adaptive towards current agent performance. It seems like DIVA looses the benefits of adaptivity by completely removing online agent evaluations. I would be interested if a DIVA-like approach with limited online evaluations could keep the best of both. \n\n\nIn Figure 5d, \"unique genotypes is a quite bad metric for diversity, as completely random levels would score quite well even though most random levels are quite qualitatively to each other. \n\n### Clarity\n\nIn the abstract it is not clear to me what \"well-behaved simulator parameterisations\" means, what \"unscalable flexibility\" refers to, or what \"ill parameterised simulators\" means.\n\nIt would be good to have a citation on line 46 for how one could learn the axes of QD.\n\nIt's not immediately clear what \"genotype\" means, and it seems to be used in different ways on line 82 and 88. In the first case it may be more conventional to call it the \"level generator\" and in the second case it could be more conventional to call it the \"level parameters\".\n\nQUESTIONS:\nHow would this method compare to CLUTR or DRED?\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}