Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversity

Authors: robby  costales
stefanos  nikolaidis
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:00:16

Abstract: The wider application of end-to-end learning methods to embodied decision-making domains remains bottlenecked by their reliance on a superabundance of training data representative of the target domain.
Meta-reinforcement learning (meta-RL) approaches abandon the aim of zero-shot *generalization*—the goal of standard reinforcement learning (RL)—in favor of few-shot *adaptation*, and thus hold promise for bridging larger generalization gaps.
While learning this meta-level adaptive behavior still requires substantial data, efficient environment simulators approaching real-world complexity are growing in prevalence.
Even so, hand-designing sufficiently diverse and numerous simulated training tasks for these complex domains is prohibitively labor-intensive.
Domain randomization (DR) and procedural generation (PG), offered as solutions to this problem, require simulators to possess carefully-defined parameters which directly translate to meaningful task diversity—a similarly prohibitive assumption.
In this work, we present **DIVA**, an evolutionary approach for generating diverse training tasks in such complex, open-ended simulators.
Like unsupervised environment design (UED) methods, DIVA can be applied to arbitrary parameterizations, but can additionally incorporate realistically-available domain knowledge—thus inheriting the *flexibility* and *generality* of UED, and the supervised *structure* embedded in well-designed simulators exploited by DR and PG.
Our empirical results showcase DIVA's unique ability to overcome complex parameterizations and successfully train adaptive agent behavior, far outperforming competitive baselines from prior literature.
These findings highlight the potential of such *semi-supervised environment design* (SSED) approaches, of which DIVA is the first humble constituent, to enable training in realistic simulated domains, and produce more robust and capable adaptive agents.
Our code is available at [https://github.com/robbycostales/diva](https://github.com/robbycostales/diva).

Main paper: 
 introduction: 
 Despite the broadening application of reinforcement learning (RL) methods to real-world problems [1, 2] , generalization to new scenarios-ones not explicitly supported by the training set-remains a fundamental challenge [3] . Meta-reinforcement learning (meta-RL), an extension of the RL framework, is formulated specifically for training adaptive agents, and is thus well-suited for overcoming these generalization gaps [4] . One recent work has demonstrated that meta-RL agents can be trained at scale to achieve adaptation capabilities on par with human subjects [5] . However, learning this human-like adaptive behavior naturally requires a large amount of data representative of the downstream (or target) distribution. For task distributions approaching real-world complexity-precisely the ones of interest-designing each scenario by hand is prohibitively expensive. for which random seeds θ i directly produce meaningfully diverse features (e.g. RACING tracks with challenging turns). Open-ended environments with flexible, unstructured parameterizations E U (θ)-though enabling more complex emergent features-lack direct control over high-level features of interest. We introduce DIVA, an approach that effectively creates a more workable parameterization E QD (θ) by evolving levels beyond the minimally diverse population from E U (θ).By training on these discovered levels, DIVA enables superior performance on downstream tasks.Prior works have explored the use of domain randomization (DR) and procedural generation (PG) techniques to produce diverse training data for learning agents [6] . Despite eliminating the need for hand-designing each task individually, human labor is still required to carefully design an environment generator that can produce diverse, high-quality tasks. As environments become more complex and open-ended, the ability to hand-design such a robust generator becomes increasingly infeasible. Some methods, like PLR [7] , attempt to ameliorate this limitation by learning a curriculum over the generated levels, but these works still operate under the assumption that the generator produces meaningfully diverse levels with a high probability.Unsupervised environment design (UED) [8] are a broad class of appproaches which use performancebased metrics to adaptively form a curriculum of training levels. ACCEL [9] , a state-of-the-art UED method, uses an evolutionary process to discover more interesting regions of the simulator's parameter space (i.e. appropriately challenging tasks) than can be found by random sampling. While UED approaches are designed to be generally applicable and require little domain knowledge, they implicitly require a very constrained environment generator-one in which all axes of difficulty correspond to meaningful learning potential for the downstream distribution. Moreover, when faced with complex open-ended environments with arbitrary parameterizations, even ACCEL is not able to efficiently explore the solution space, as it is still bottlenecked by the speed of agent evaluations.In this work, we introduce DIVA, an approach for generating diverse training tasks in open-ended simulators to train adaptive agents. By using quality diversity (QD) optimization to efficiently explore the solution space, DIVA bypasses the problem of needing to evaluate agents on all generated levels. QD also enables fine-grained control over the axes of diversity to be captured in the training tasks, allowing the flexible integration of task-related prior knowledge from both domain experts and learning approaches. We demonstrate that DIVA, with limited supervision in the form of feature samples from the target distribution, significantly outperforms state of the art UED approachesdespite the UED approaches being provided with significantly more interactions. We further show that UED techniques can be integrated into DIVA. Preliminary results with this combination (which we call DIVA+) are promising, and suggest an exciting avenue for future work. 
preliminaries: 
 Meta-reinforcement learning. We use the meta-reinforcement learning (meta-RL) framework to train adaptive agents, which involves learning an adaptive policy π φ over a distribution of tasks T . Each M i ∈ T is a Markov decision process (MDP) defined by a tuple S, A, P, R, γ, T , where S is the set of states, A is the set of actions, P (s t+1 |s t , a t ) is the transition distribution between states given the current state and action, R(s t , a t ) is the reward function, γ ∈ [0, 1] is the discount factor, and T is the horizon. Meta-training involves sampling tasks M i ∼ T , collecting trajectories D = {τ h } H h=0 -where H is the number of episodes in each trial τ pertaining to the M i -and optimizing policy parameters φ to maximize the expected discounted returns across all episodes.VariBAD [10] is a context variable-based meta-RL approach which belongs to the wider class of RNNbased methods [11, 12] . While prior methods [13, 14] also use context variables to assist in task adaptation, VariBAD uniquely learns within a belief-augmented MDP (BAMDP) S, A, Z, P, R, γ, T where the context variables z ∈ Z encodes the agent's uncertainty about the task, promoting Bayesian exploration. VariBAD utilizes an RNN-based variational autoencoder (VAE) to model a posterior belief over possible tasks given the full agent trajectory, permitting efficient updates to prior beliefs.Quality diversity. For a given problem, quality diversity (QD) optimization framework aims to generate a set of diverse, high-quality solutions. Formally, a problem instance of QD [15] specifies an objective function J : R n → R and k features f i : R n → R. Let S = f (R n ) be the feature space formed by the range of f , where f : R n → R k is the joint feature vector. For each s ∈ S, the QD objective is to find a solution θ ∈ R n where f (θ) = s and J(θ) is maximized. Since R k is continuous, an algorithm solving the QD problem definition above would require unbounded memory to store all solutions. QD algorithms in the MAP-Elites [16] family therefore discretize S via a tessellation method, where G is a tessellation of the continuous feature space S into N G cells. In employing a MAP-Elites algorithm, we relax the QD objective to find a set of solutions θ i , i ∈ {1, . . . , N G }, such that each θ i occupies one unique cell in G. We call the occupants θ i of all M cells, each with its own position f (θ i ) and objective value J(θ i ), the archive of solutions.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The authors discuss limitations of the present work in Section 7. This Discussion section covers (1) key takeaways from the paper, (2) limitations, and (3) promising avenues for future work. The authors combine these sections because the points are all interrelated, and having them in the same place preserves cohesion and flow of the writing. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem setting: 
 One assumption underlying UED methods is that random parameters-or parameter perturbations for ACCEL -produce meaningfully different levels to justify the expense of computing objectives on each newly generated level. However, when the genotype is not well-behaved-when meaningful diversity is rarely generated through random sampling or mutations-these algorithms waste significant time evaluating redundant levels. In our work, we discard the assumption of well-behaved genotypes in favor of making fewer, more realistic assumptions about complex environment generators. There are several assumptions we make about the simulated environments DIVA has access to.Genotypes. We assume access to an unstructured environment parameterization function E U (θ), where each θ is a genotype (corresponding to the QD solutions θ i ) describing parameters to be fed into the environment generator. QD algorithms can support both continuous and discrete genotype spaces, and in this work we evaluate on domains with both kinds. Crucially, we make no assumption of the quality of the training tasks produced by this random generator. We only assume that (1) there is some nonzero (and for practical purposes, nontrivial) probability that this generator will produce a valid level for training-one in which success is possible and positive rewards are in reach; and(2) that it is computationally feasible to discover meaningful feature diversity through an intelligent search over the parameter space-an assumption implicit in all QD applications.Features. We assume access to a pre-defined set of features, S = f (R n ), that capture axes of diversity which accurately characterize the diversity to be expected within the downstream task distribution. It is also possible to learn or select good environment features from a sample of tasks from the downstream distribution, which we discuss in Section 7. For the sake of simplicity, we use a grid archive as our tessellation G, where the k dimensions of the discrete archive correspond to the defined features. The number of bins for each feature is a hyperparameter, and can be learned or adapted over the course of training. We generally find it to be helpful to use moderately high resolutions to ease the search, since smaller leaps in feature-level diversity are required to uncover new cells. By default, we use 100 sample feature values across all domains, but demonstrate in ablation studies that that significantly fewer may be used (see Appendix C).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: The paper does not include any formalized theoretical results. The authors did not find reason to provide any theorems, formulas, or proofs to support the claims made.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
diva: 
 DIVA assumes access to a small set of feature samples representative of the target domain. It does not, however, require access to the underlying levels themselves. This is a key distinction, as the former is a significantly weaker assumption. Consider the problem of training in-home assistive robots in simulation with the objective of adapting to real-world houses. It is more likely we have access to publicly available data describing typical houses-dimensions, stylistic features, etc.-than we have access to corresponding simulator parameters which produce those exact feature values.Feature density estimation. DIVA begins by constructing a QD archive with appropriate bounds and resolution. Given a set of specified features {f i } k and a handful of downstream feature samples, we first infer each feature's underlying distribution. These can be approximated with kernel density estimation (KDE), or we can work with certain families of parameterized distributions. For our experiments, we assume each feature is either (independently) normally or uniformly distributed. We use a statistical test2 to evaluate the fit of each distribution family, and select the best-fitting. Setting the resolution for discrete feature dimensions is straightforward, and depends only on the range. For continuous features, the resolution should enable enough signal for discovering new cells, while avoiding practical issues that arise with too many cells 3 . See Section 5 for domain-specific details.Two-stage QD updates. Once the feature-specific target distributions are determined, we can use these to set bounds for each archive dimension. A naïve approach would be to set the archive ranges for each feature based on the confidence bounds of the target distribution. However, random samples from E U may not produce feature values that fall within the target range. We found this to be a major issue in the ALCHEMY domain (see Figure 2 ), and for some features in RACING. We solve this problem by setting the initial archive bounds to include both randomly generated samples from E U , as well as the full target region. As the updates progress, we gradually update the sample mask-which is used to inform the sampling of new solutions-towards the target region. We observe empirically that updating and applying this mask provides an enormous speed-up in guiding solutions towards the target region (see Figure 15 ). After this first stage, solutions are inserted into a new archive defined by the proper target bounds. See Appendix A for more specifics on these two QD update stages. Overview. DIVA consists of three stages. Stage 1 (S1) begins by initializing the archive with bounds that include both the downstream feature samples (the target region), as well as the initial population generated from E U (θ). S1 then proceeds with alternating QD updates, to discover new solutions, and sample mask updates, to guide the population towards the target region. In Stage 2 (S2), the archive is reinitialized with existing solutions, but is now bounded by the target region. QD updates continue to further diversify the population, now targeting the downstream feature values specifically. The last stage is standard meta-training, where training task parameters are now drawn from P G (θ), a distribution over the feature space approximated using the downstream feature samples, discretized over the archive cells. See Appendix A for detailed pseudocode.Algorithm 1 DIVA # Stage 1: discover target region 1: G ← initialize_archive() 2: for i in range(NS1) do 3: G ← qd_update(G, J, M , BQD) 4: M ← update_sample_mask(M , G)Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: In addition to the provided code (see Section 8) , which is self-contained and includes documentation for reproducing all empirical results in this paper, the authors additionally include all training details, including hyperparameter settings for all methods, in Appendix E.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
empirical results: 
 Baselines. We implement the following baselines to evaluate their relative performance to DIVA. ODS is the "oracle" agent trained over the downstream environment distribution E S (θ), used for evaluation. With this baseline, we are benchmarking the upper bound in performance from the perspective of a learning algorithm that has access to the underlying data distribution. 4 DR is the meta-learner trained over a task distribution defined by performing domain randomization over the space of valid genotypes, θ, under the training parameterization, E U (θ). Robust PLR (PLR ⊥ ) [17] is the improved and theoretically grounded version of PLR [7] , where agents' performance-based PLR objectives are evaluated on each level before using them for training. ACCEL [9] is the same as PLR ⊥ but instead of randomly sampling over the genotype space to generate levels for evaluation, levels are mutated from existing solutions. All baselines use VariBAD [10] as their base meta-learner.Experimental setup. The oracle agent (ODS) is first trained over the each environment's downstream distribution to tune VariBAD's hyperparameters. These environment-specific VariBAD settings are then fixed while hyperparameters for DIVA and the other baselines are tuned. For fairness of comparison-since DIVA is allowed N QD QD update steps to fill its archive before meta-trainingwe allow each UED approach (PLR ⊥ and ACCEL) to use significantly more environment steps for agent evaluations (details discussed below per environment). All empirical results were run with 5 seeds unless otherwise specified, and error bars indicate a 95% confidence region for the metric in question. The QD archive parameters were set per environment, and for ALCHEMY and RACING, relied on some hand-tuning to find the right combinations of features and objectives. We leave it to future work to perform a deeper analysis on what constitutes good archive design, and how to better automate this process. Our first evaluation domain is a modified version of GRIDNAV (Figure 3 ), originally introduced to motivate and benchmark VariBAD [10] . The agent spawns at the center of the grid at the start of each episode, and receives a slight negative reward (r = -0.1) each step until it discovers (inhabits) the goal cell, at which point it also receives a larger positive reward (r = 1.0).Parameterization. We parameterize the task space (i.e. the goal location) to reduce the likelihood of generating meaningfully diverse goals. Specifically, each E U k (or E k ) introduces k genes to the solution genotype which together define the final y location. Each gene j can assume the values θ j ∈ {-1, 0, 1}, and the final y location is determined by summing these values, and performing a floor division to map the bounds back to the original range of the grid. As k increases, y values are increasingly biased towards 0, as shown on the right side of Figure 3 . For more details on the GRIDNAV domain, see Appendix B.1.QD updates. We define the archive features to be the x and y coordinates of the goal location.The objective is set to the current iteration, so that newer solutions are prioritized (additional details in Appendix B.1). DIVA is provided N S2 = 8.0 × 10 4 (N S1 = 0) QD update iterations for filling the archive. To compensate, PLR ⊥ and ACCEL are each provided with an additional 9.6 × 10 6 environment steps for evaluating PLR scores, which amounts to three times as many total interactions-since all methods are provided N E = 4.8 × 10 6 interactions for training. If each "reset" call counts as one environment step 5 , the UED baselines are effectively granted 2.4× more additional step data than what DIVA additionally receives through its QD updates (details in Appendix E.1).Results. From Figure 4a , we see that increasing genotype complexity (i.e. larger k) reduces goal diversity for DR-which is expected given the parameterization defined for E U . We can also see that DIVA, as a result of its QD updates, can effectively capture goal diversity, even as complexity increases. When we fix the complexity (k = 24) and train over the E U distribution, we see that the UED approaches are unable to incidentally discover and capture diversity over the course of training (Figure 4b ). DIVA's explicit focus on capturing meaningful level diversity enables it to significantly outperform these baselines in terms of episodic return (Figure 4c ) and success rate (Figure 4d ).ALCHEMY [18] is an artificial chemistry environment with a combinatorially complex task distribution. Each task is defined by some latent chemistry, which influences the underlying dynamics, as well as agent observations. To successfully maximize returns over the course of a trial, the agent must infer and exploit this latent chemistry. At the start of each episode, the agent is provided a new set of (1-12) potions and (1-3) stones, where each stone has a latent state defined by a specific vertex of a cube, i.e. ({0, 1}, {0, 1}, {0, 1}), and each potion has a latent effect, or specific manner in which it transforms stone latent states (see Figure 5a ). The agent observes only salient artifacts of this latent information, and must use interactions to identify the ground-truth mechanics. At each step, the agent can apply any remaining potion to any remaining stone. Each stone's value is maximized the closer its latent state is to (1, 1, 1), and rewards are produced when stones are cast into the cauldron.To make training feasible on academic resources, we perform evaluations on the symbolic version of ALCHEMY, as opposed to the full Unity-based version. Symbolic ALCHEMY contains the same mechanistic complexity, minus the visuomotor challenges which are irrelevant to this project's aims.Parameterization. E S (θ) is the downstream distribution containing maximal stone diversity. For training, implement E U k where k controls the level of difficulty in generating diverse stones. Specifically, we introduce a larger set of coordinating genes θ j ∈ {0, 1} that together specify the initial stone latent states, similar to the mechanism we used in GRIDNAV to limit goal diversity. Each stone latent coordinate is specified with k genes, and only when all k genes are set to 1 is the latent coordinate is set to 1. When any of the genes are 0, the latent coordinate is 0. For our experiments we set k = 8, and henceforth use E U to signify E U8 .QD updates. We use features LATENTSTATEDIVERSITY and MANHATTANTOOPTIMAL -both of which target stone latent state diversity from different angles. See Appendix B.2 for more specifics on these features and other details surrounding ALCHEMY's archive construction. Like GRIDNAV, the objective is set to bias new solutions. DIVA is provided with N S1 = 8.0 × 10 4 and N S2 = 3.0 × 10 4 QD update iterations. PLR ⊥ and ACCEL are compensated such that they receive 3.5× more additional step data than what DIVA receives via QD updates (see Appendix E.1 for details).Results. Our empirical results demonstrate that DIVA is able to generate latent stone states with diversity representative of the target distribution. We see this both quantitatively in Figure 5b , and qualitatively in Figure 6 . In Figure 5c , we see this diversity translates to significantly better results on E S over baselines. Despite generating roughly as many unique genotypes as DIVA (Figure 5d ), PLR ⊥ and ACCEL are unable to generate training stone sets of significant phenotypical diversity to enable success on the downstream distribution.Lastly, we evaluate DIVA on the RACING domain introduced by [17] . In this environment, the agent controls a race car via simulated steering and gas pedal mechanisms, and is rewarded for efficiently completing the track, M i ∈ T . We adapt this RL environment to the meta-RL setting by lowering the resolution of the observation space significantly. By increasing the challenge of perception, even competent agents benefit from multiple episodes to better understand the underlying track. For all of our experiments, we use H = 2 episodes per trial, and a flattened 15 × 15 pixel observation space.Setup. We use three different parameterizations in our experiments: (1) E S (θ) is the downstream distribution we use for evaluating all methods, training ODS, and setting archive bounds for DIVA. Parameters θ are used to seed the random generation of control points which in turn parameterize a sequence of Bézier curves designed to smoothly transition between the control locations. Track diversity is further enforced by rejecting levels with control points that possess a standard deviation below a certain threshold. (2) E U k (θ) is a reparameterization of E S (θ) that makes track diversity harder to generate, with the difficulty proportional to the value of k ∈ N. For our experiments, we use k = 32 (which we will denote simply as E U (θ)), which roughly means that meaningful diversity is 32× less likely to randomly occur than when k = 1 (which is equivalent to E S (θ)). This is achieved by defining a small region in the center, 32 (or k, in general) times smaller than the track boundaries, where all points outside the region are projected onto the unit square, and scaled to the track size. (3) E F1 (θ) uses θ as an RNG seed to select between a set of 20 hand-crafted levels official Formula-1 tracks [17] , and is used to benchmark DIVA's zero-shot generalization to a new target distribution.QD updates. We define features TOTALANGLECHANGES (TAC) and CENTEROFMASSX (CX) for the archive dimensions. Levels from E U lack curvature (see Figure 8 ) so TAC, which is defined as the sum of angle changes between track segments, is useful for directly targeting this desired curvature. Main results. Results are shown in Figure 7 . DIVA outperforms all baselines, including the UED approaches, which have access to three times as many environment interactions. From Figure 8 , we see that final DIVA levels contain significantly more diversity than randomization over E U .F1 results Transfer to F1 tracks. Next, we evaluate the ability of these trained policies to zero-shot transfer to humandesigned F1 levels [17] , E F1 . Though qualitative differences are apparent (see Figure 9 ), from Figure 7a we can additionally see how these levels differ quantitatively. Even though DIVA uses feature samples from E S to define its archive, we see from the results in Figure 9 that DIVA is not only able to complete many of these tracks, but is also able to significantly outperform ODS. This result may seem unlikely, given that DIVA bases its axes of diversity on E S . One possible explanation is that while DIVA successfully matches its TOTALANGLECHANGES distribution to E S (see Figure 7 ), because it is less likely for all 12 control points to be mutated to the diversity-enabling region than just a few control points with sharp angles, DIVA "opts" for the latter, and thus produces fewer, sharper angles, which is evidently useful for transferring to (these) human-designed tracks. This hypothesis matches what we see qualitatively from the DIVA-produced levels in Figure 8 .Combining DIVA and UED. While PLR ⊥ and ACCEL struggle on our evaluation domains, they still have utility of their own, which we hypothesize may be compatible with DIVA's.  
related work: 
 Meta-reinforcement learning. Meta-reinforcement learning methods range from gradient-based approaches (e.g. MAML) [19] , RNN context-based approaches [12, 11] (e.g. RL 2 ), and the slew of emerging works utilizing transformers [20, 5, 21] . We use VariBAD [10] , a state-of-the-art context variable-based approach that extends RL 2 by using variational inference to incorporate task uncertainty into its beliefs. HyperX [22] , an extension that uses reward-bonuses, was not found to improve performance on our domains. In each of these works, the training distribution is given; none address the problem of generating diverse training scenarios in absence of such a distribution.Procedural environment generation. Procedural (content) generation (PCG / PG) [6] is a vast field. Many RL and meta-RL domains themselves have PG baked-in (e.g. ProcGen [23] , Meta-World, [24] , Alchemy [18] , and XLand [5] ). Each of these works rely on human engineering to produce levels with meaningfully diverse features. A related stream of works apply scenario generation to robotics-some works essentially perform PCG [25, 26] , while others integrate more involved search mechanics [27, 28, 29, 30] . One prior work [31] defines a formal but generic parameterization for applying PG to generate meta-RL tasks. It is yet to be shown, however, if such an approach can scale to domains with vastly different dynamics, and greater complexity.Unsupervised environment design. UED approaches-which use behavioral metrics to automatically define and adapt a curriculum of suitable tasks for agent training-form the frontier of research on open-endedness. The recent stream of open-ended agent/environment co-evolution works (e.g. [32, 33, 34] ) was kickstarted by the POET [35, 36] algorithm. The "UED" term itself originated in PAIRED [8] , which uses the performance of an "antagonist" agent to define the curriculum for the main (protagonist) agent. PLR [7] introduces an approach for weighting training levels based on learning potential, using various proxy metrics to capture this high-level concept. [17] introduces PLR ⊥ , which only trains on levels that have been previously evaluated, and thus enabling certain theoretical robustness guarantees. AdA [5] uses PLR as a cornerstone of their approach for generating diverse training levels for adaptive agents in a complex, open-ended task space. ACCEL [9] borrows PLR ⊥ 's scoring procedure, but the best-performing solutions are instead mutated, so the buffer not only collects and prioritizes levels of higher learning potential, but evolves them. We use ACCEL as our main baseline because it has demonstrated state-of-the art results on relevant domains, and like DIVA, evolves a population of levels. The main algorithmic differences between ACCEL and DIVA are that ACCEL (1) performs additional evaluation rollouts to produce scores during training and (2) uses a 1-d buffer instead of DIVA's multi-dimensional archive. PLR ⊥ serves as a secondary baseline in this work; its non-evolutionary nature makes it a useful comparison to DR.Scenario generation via QD. A number of recent works apply QD to simulated environments in order to generate diverse scenarios, with distinct aims. Some works, like DSAGE [37] , uses QD to develop diverse levels for the purpose of probing a pretrained agent for interesting behaviors.In another line of work applies QD to human-robot interaction (HRI), and ranges from generating diverse scenarios [38] , to finding failure modes in shared autonomy systems [39] and human-aware planners [40] . DIVA's application of QD inspired by these approaches, as they produce meaningfully diverse environment scenarios, but no prior work exists which applies QD to define a task distribution for agent training, much less adaptive agent training, or overcoming difficult parameterizations in open-ended environments. 
discussion: 
 The present work enables adaptive agent training on open-ended environment simulators by integrating the unconstrained nature of unsupervised environment design (UED) approaches, with the implicit supervision baked into procedural generation (PG) and domain randomization (DR) methods. Unlike PG and DR, which requires domain knowledge to be carefully incorporated into the environment generation process, DIVA is able to flexibly incorporate domain knowledge, and can discover new levels representative of the downstream distribution. And instead of relying on behavioral metrics to infer a general, ungrounded form of "learning potential", like UED-which becomes increasingly unconstrained and therefore less useful a signal as environments become more complex and open-ended-DIVA is able to directly incorporate downstream feature samples to target specific, meaningful axes of diversity. With only a handful of downstream feature samples to set the parameters of the QD archive, our experiments (Section 5) demonstrate DIVA's ability to outperform competitive baselines compensated with three times as many environment steps during training.In its current form, the most obvious limitation of DIVA is that, in addition to assuming access to downstream feature samples, the axes of diversity themselves must be specified. However, we imagine these axes of diversity could be learned automatically from a set of sample levels, or selected from a larger set of candidate features; it may be possible to adapt existing QD works to automate this process in related settings [41] . The present work also lacks a more thorough analysis of what constitutes good archive design. While some amount of heuristic decision-making is unavoidable when applying learning algorithms to specific domains, a promising future direction would be to study how to approach DIVA's archive design from a more algorithmic perspective.DIVA currently performs QD iterations over the environment parameter space defined by E U (θ), where each component of the genotype θ represents some salient input parameter to the simulator. Prior works in other domains (e.g. [42] ) have demonstrated QD's ability to explore the latent space of generative models. One natural direction for future work would therefore be to apply DIVA to neural environment generators (rather than algorithmic generators), where θ would instead correspond to the latent input space of the generative model. If the latent space of these models is more convenient to work with than the raw environment parameters-e.g. due to greater smoothness with respect to meaningful axes of diversity-this may help QD more efficeintly discover samples within the target region. Conversely, DIVA's ability to discover useful regions of the parameter space means these neural environment generators do not need to be "well-behaved", or match a specific target distribution. Since these generative models are also likely to be differentiable, DIVA can additionally incorporate gradient-based QD works (e.g. DQD [15] ) to accelerate its search.Preliminary results with DIVA+ demonstrate the additional potential of combining UED and DIVA approaches. The F1 transfer results (i.e. DIVA outperforming ODS trained directly on E S ) further suggest that agents benefit from flexible incorporation of downstream knowledge. In future work, we hope to study more principle integrations of UED and DIVA-like approaches, and to more generally explore this exciting new area of semi-supervised environment design (SSED).More return G Details on the two-stage QD updates Here we provide more details on the process described in Section Section 4. Hyperparameters N S1 and N S2 are set to define the number of QD updates to perform in each stage (see Appendix D). In proportion to how many updates in S1 have elapsed, if the sample mask is enabled, the mask is moved at a linear pace from encapsulating the full S1 archive, to covering only the target region. We also set a hyperparameter, N SM (see Appendix D), which specifies the minimum number of solutions which must exist within the mask's new bounds for it to be updated. This is to ensure the mask never outpaces the search process. The mask was only found to be necessary in the ALCHEMY environment. In S1 we sample solutions uniformly from within the mask. In S2, we begin sampling from the discretized target density distribution approximated from the downstream feature samples. Two stages are used for RACING as well, since many initial samples fall outside of the target region, but masking was not found to be necessary. The sample mask has a relatively straightforward implementation for MAP-Elites, which we use for ALCHEMY's discrete genotype (and GRIDNAV, where no mask is required). Since MAP-Elite updates entail performing mutations on solutions directly sampled from the archive, the mask is implemented to only consider solutions that fall within the mask bounds. However, since the CMA-ES-based emitter we use for RACING operates by sampling from a parameterized distribution, instead of sampling from the archive directly, the mask would need to be applied to these parameters instead of the archive.B.1 GRIDNAV GRIDNAV features. The following features are defined for the GRIDNAV environment: Archive hyperparameters for ALCHEMY were determined based on some knowledge about the domain, as well as the feature distributions (Figure 13 ). We noticed a major deviation between E U and E S in the feature LATENTSTATEDIVERSITY (LSD), and an even greater on in MANHATTANTOOPTIMAL (MTO). These two constitute the initial dimensions of the archive, and we found the sample mask updates to be crucial to reach and fill the target region (see Figure 15 ). We use PARITYFIRSTSTONE (PFS) in the second stage to encourage more diversity once the target is reached; it is excluded from the first stage, which is focused on simply reaching the target. The archive for the first stage is of shape [100, 300, 1], corresponding to LSD, MTO, and PFS. The second stage shape is [150, 150, 5].We found this archive to produce diverse enough solutions, evidenced by the number and spread in the target region, so we used this setting to train our DIVA agents. The only objective we found useful for ALCHEMY was a slight bias for newly generated solutions, which we also used for GRIDNAV. We hypothesize this prevents the archive from getting "stuck" with a suboptimal set of solutions, in absense of other objectives.RACING features. See Table 13 for all features defined on RACING. Figure 13 contains the feature distributions for the structured and unstructured environment parameterizations on RACING, computed over 100 feature samples. Figure 14 shows the covariance between feature values for RACING, computed over 100 samples.Archive hyperparameters for RACING were determined through trial and error, by viewing the samples produced by the archives at the end of the QD updates, as well as the target coverage metrics. After a few iterations, it became clear that Total Angle Change TOTALANGLECHANGES (TAC) was the most useful feature, and so we tried pairing it with a number of others, prioritizing other features with low absolute covariance (see Figure 14 ).The best performing archive used by DIVA on RACING uses TAC and CX as its features, and used a measure alignment objective over CY and VY. The measure alignment objective rewards solutions for having measure values over the specific measures that are similar to the target distribution. We also The ratio of enclosed area to curve length. AVERAGECURVATURE found that randomly sampling these objective values according to the target distribution provided some additional support in covering the target region efficiently.The slightly "misspecified" archive was chosen because its solutions generated some diversity, but not as much as the aforementioned one. This archive uses TAC and ATLR as its features, and uses a measure diversity objective over just CY. Instead of prioritizing alignment to the target distribution, the diversity objective samples a handful of solutions from the archive, and uses the current solutions deviation from these as its objective.We use final archive dimensions of 500 × 500 for both S1 and S2.Sample mask ablation. Figure 15 shows the benefit of updating the sample mask bounds during the first archive filling stage on ALCHEMY. Not only does this approach produce significantly more total archive solutions, but more importantly, progress towards filling the target region specifically is accelerated.Varying QD mutation rate. We perform an ablation on the QD mutation rate, which is the probability that a given gene will be mutated (for MAP-Elites). We perform this ablation on ALCHEMY because the its search is the most challenging of the three environments we consider (it is the sole environment that required a longer S1 and the sample mask trick for accelerating to accelerate the search). We see from Figure 16 that ALCHEMY results are not very sensitive to the setting of the mutation rate. Varying number of QD updates. We perform a similar ablation to test how robust DIVA is to the number of QD updates performed. We see from Figure 17 that ALCHEMY results suffer somewhat from fewer updates (e.g. for only 10k in each stage), still significantly outperform baselines in each case. The trend is clear, however: more QD updates produces more solutions, which generally translates to better performance, even if slightly.Varying number of downstream samples. Next we test how robust DIVA is to the number of downstream samples used to compute the target distribution. In Figure 18 we see that, despite the errors increasing with fewer samples, DIVA still significantly outperforms baselines with as few as five samples.Table 4 displays the hyperparameters used for DIVA across all domains.A note on N TRS computation The initial QD population (n 0 ) is implemented such that the first set of QD updates simply generates n 0 random levels from E U , before performing the actual mutations (for ME) or intelligent sampling (for ES). Thus, the formula we use for computing N TRS , the total reset steps provided to DIVA (see Table 4 ), which we use to compare the extra steps we provide PLR ⊥ and ACCEL (discussed in Section 5), does not include n 0 it is simply the product of the batch size and the total number of QD iterations. ACCEL uses the same hyperparameters as PLR ⊥ (see Table 6 ), combined with the same evolutionary hyperparameters used for DIVA's QD archive (see Table 4 ).All results were produced on a handful of Titan X or Xp GPUs. Environments were parallelized across multiple CPU cores to accelerate training. While the experiment time varies by method and environment, most experiments take less than a day to run to completion. PLR ⊥ and ACCEL take the longest, as they required twice as many environment steps as the other methods-on the two latter domains, these methods take well over a day to run to completion. This ability is indeed "unique", as far as the authors are aware, and is supported by the discussion of related works in Section 6.(2): The empirical results in this paper presented in Section 5 support the claim that DIVA far outperforms other baselines for training meta-RL agents. Specific pieces of evidence include the GRIDNAV results in Figure 4 , the ALCHEMY results in Figure 5 , and the results in RACING, contained in Figure 7 , 9, and 10. The final sentence in the abstract makes a more general, weaker claim about the presented method's potential: "These findings highlight the potential of approaches like DIVA to enable training in complex open-ended domains, and to produce more robust and adaptable agents." The authors believe the same set of evidence supports this claim as well, but is ultimately up to the reader-and more importantly, future work-to decide. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: All results (excepting visualizations, and the trend curve in Figure 4 (a), for which they are not needed) are accompanied with error bars that capture the factors of variability, mostly due to random seeding (affecting many different random settings of the algorithm, e.g. initial model weights, sampling, etc.). The significance of all error bars, and number of seeds used for each experiment, are detailed in Section 5. Other training details are available in Appendix E.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: All compute resources (workers, memory, time of execution) are detailed in Appendix E.4.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All used resources are either cited properly in the paper and/or are documented and credited in the codebase (see Section 8) .Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The new asset introduced in this paper is the codebase (see Section 8), which is well-documented for the purpose of reproducibility, and contains details about training, the license, and limitations, etc.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.We use a Kolmogorov-Smirnov test for features with continuous values and Chi-squared for discrete.Memory is one concern; another is that optimizing objectives across all cells is slower with more cells.Technically, reweighting this distribution (e.g. via PLR) may produce a stronger oracle, but for the purposes of this work, we assume the unaltered downstream distribution can be efficiently trained over, sans curriculum.In general, rendering the environment (via "reset") is required to compute level features for DIVA.Codebase: https://github.com/robbycostales/diva† GRIDNAV, ALCHEMY, RACING 
 
 Appendix:  


Decision: The paper proposes a method for generating actually diverse training tasks. 

Reviewers recommendations were Accept, Weak Accept, Reject, Strong Accept. There was some criticism of the fact that the proposed approach still requires some level of hand-design, and the simplicity of the experiments. 

Overall I am willing to accept due to the interest of the problem and the novelty of the method.

I would recommend the authors re-read the paper and fix the  typos. E.g. line 130 "but...",  line 194: "the each stone".

Reviews: review: paper_summary: paper_summary: This paper describes an approach for learning a QD-archive to be used as a proxy for samples of test environments, and shows that this results in improved performance in producing a set of meta-learning tasks over DR and UED baselines.
strength_weakness: strength_weakness: The comparison between DIVA and PLR/ACCEL is a bit of an apples-to-oranges comparison. Regret-based UED methods like PLR and ACCEL are meant to be making decisions under ignorance, where there is no information known about the target distribution. There are other UED approaches designed for the case where there is some information that is known, and thus it is a decision under risk. Specifically, it would be better to compare against SAMPLR, CLUTER, or DRED. Since SAMPLR requires simulator access, the fairest comparison would be against CLUTER or DRED. 

[SAMPLR] Jiang, Minqi, et al. "Grounding aleatoric uncertainty for unsupervised environment design." _Advances in Neural Information Processing Systems_ 35 (2022): 32868-32881.

[CLUTER]Azad, Abdus Salam, et al. "Clutr: Curriculum learning via unsupervised task representation learning." _International Conference on Machine Learning_. PMLR, 2023.

[DRED] Garcin, Samuel, et al. "DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design." _Forty-first International Conference on Machine Learning_. 2024.

That being said, the current results look to have more significant results than these other works (though it is hard to tell across domains), and this approach has not be tried in meta-learning before. It is important to note that UED has been used in meta-learning before, for instance in AdA.

[AdA] Team, Adaptive Agent, et al. "Human-timescale adaptation in an open-ended task space." _arXiv preprint arXiv:2301.07608_ (2023).

It occurs to me that DIVA + UED as discussed starting at line 259 could be used as an algorithm for decisions under ignorance, and would possibly be quite a good algorithm. It may be worth running the approach from 259, maybe without any test time data, on the traditional maze, F1, and bipedal walker environments and transfer tasks to check if it consistently outperforms existing methods.

It seems like the assumption about the probability of generating a useful level from the simulator underlying DIVA is similar to the assumption necessary for ACCEL or PLR. Both of these approaches should work if there is a ~0.01 % chance of generating a useful level. The boot-up time would just be a bit slower to get the initial buffer of levels. 

The limitation of online agents evaluations is a real bottleneck for the UED approaches which DIVA avoids, but it is a bit of an odd comparison as DIVA generates the model once offline, and thus isn't aiming to be adaptive towards current agent performance. It seems like DIVA looses the benefits of adaptivity by completely removing online agent evaluations. I would be interested if a DIVA-like approach with limited online evaluations could keep the best of both. 


In Figure 5d, "unique genotypes is a quite bad metric for diversity, as completely random levels would score quite well even though most random levels are quite qualitatively to each other. 

### Clarity

In the abstract it is not clear to me what "well-behaved simulator parameterisations" means, what "unscalable flexibility" refers to, or what "ill parameterised simulators" means.

It would be good to have a citation on line 46 for how one could learn the axes of QD.

It's not immediately clear what "genotype" means, and it seems to be used in different ways on line 82 and 88. In the first case it may be more conventional to call it the "level generator" and in the second case it could be more conventional to call it the "level parameters".
questions: questions: How would this method compare to CLUTR or DRED?
limitations: limitations: See Weaknesses.
paper_summary: paper_summary: The paper identifies the limitation that hand-crafting a sufficiently diverse set of simulated training tasks to bridge any significant sim-to-real gap is labor-intensive. It then proposes DIVA, a new evolutionary approach for generating diverse training tasks in the absence of well-behaved simulator parameterizations. The paper demonstrates how DIVA outperforms proposed baselines such as ACCEL, PLR, and DR.
strength_weakness: strength_weakness: - I think the presentation of the paper can be improved. For example, Figure 1 could include more descriptions about what is happening, such as defining E_S(theta) as the structured environment simulator and E_U(theta) as the unstructured environment simulator.
- In line 128, how was the number 80% chosen? And why is it described as “roughly”? More justifications for the choice of these parameters should be included.

Minor things
- Typo in line 184, “the final y location is determine by”
questions: questions: - Why does DIVA want to evolve a population of minimally diverse solutions from the original parameterization? Does DIVA assume that the original parameterizations are what humans care about? If so, doesn’t that mean we still need to ensure that the original parameterizations are handcrafted enough to be close to what humans care about?
- If the original parameterizations are chosen randomly or are not handcrafted to what humans might care about, how would DIVA perform?
- To what extent and when do we consider an environment simulator to have structured or unstructured parameterizations?
- In the GridNav experiment, why do we want to prevent the generation of diverse goals along the y-axis?
- Is there a hypothesis on why “the domain randomization over increasingly complex genotypes diminishes diversity in terms of goal location” (lines 187-188)?
- Line 188 claims that “DIVA, on the other hand, is able to capture this diversity.” However, doesn’t DIVA also decrease to the same level of percentage coverage as DR*? Why are there no confidence intervals for Figure 3a, like the others?
limitations: limitations: - Yes, the paper sufficiently covers the limitations.
paper_summary: paper_summary: This paper introduces DIVA, a technique for exploring the parameter space of parametrisable environments. The technique uses a variant of MAP-Elites to explore the environment parameter space, finding exemplar points spread across the parameter space, as measured with respect to some user provided features. The authors show that this generates a usefully diverse collection of environment instances by training a meta-learning agent on the generated levels and comparing it to a number of baselines.
strength_weakness: strength_weakness: The primary weakness from my perspective is that domains that the experiments was conducted on are all very simple “toy” domains. In particular, these domains do not have all have intrinsically complex parameterisations, and for two of them the authors had to, in effect, obfuscate the parameter spaces so the algorithm had a challenge to work against. This leads one to worry that the authors’ results may not be representative of more realistic open-ended domains where the parameter space complexity may manifest in a different way.

Another weakness of the technique is the authors need to hand-select the features used by the algorithm, per domain. The authors mention that this could be automated, but the fact still stands that in this paper, quite extensive hand tuning and selection - as reported in the appendices - of the objectives was made to get their results. The impact of this technique is much more limited if feature sets need to be hand-tuned per domain, so without demonstrating that this is not the case, I think the authors do not demonstrate that this technique is likely to have wide impact.

I have a further query about the way that the VariBAD hyperparameters were tuned, below in the questions section.
questions: questions: L36-37: Nit: “a” refers to the singular, “autocurricula” is plural.

L74: Nit: Add “are” between “algorithms” and “in”.

L129-131: Sentence doesn’t parse. Maybe remove “but”, or perhaps something got accidentally removed?

L159-161: I was confused by the experimental approach here. The downstream task distribution was first used to train the meta-learner’s hyperparameters, and then these hyperparameters were used to compare all of the approaches. But isn’t this essentially training on the test set? It feels like that the realistic setting would be one where the meta-learner does not know about the downstream tasks, and it would be important to demonstrate DIVA works under these conditions. I can appreciate that the authors might want to try to “factor” the behaviour of the meta-learner from the behaviour of DIVA, but it seems like a strong assumption that this is possible or useful.

L245: Nit: Figures 8 & 9 are referenced before figure 7.

L278: “Their approach”. Who’s approach?

L306: Define “HRI”.

L312: “DIVA’s” -> “DIVA”.

L312: “incorporating” -> “incorporate”.
limitations: limitations: The primary limitation is the one regarding hand-tuned feature sets, which I have expanded on above, in the weaknesses section.
paper_summary: paper_summary: This paper applies quality diversity (QD) optimization (evolutionary method) to the problem of diverse task generation for (meta) reinforcement learning (RL). It argues that QD could be used in settings where an open-ended simulator’s parameterization is unlikely to produce tasks that are diverse in high-level features. The method works by handcrafting a set of high-level task features that are relevant to the learning process of the RL agent, then running QD to collect a set of diverse parameterizations that cover the feature space distribution well. A (meta) RL agent is then trained on tasks sampled from a distribution based on the set of QD optimised tasks. The paper’s experiments on GridNav, Alchemy, and Racing tasks show significant improvement in agent performance over existing baselines such as robust prioritised level-replay.
strength_weakness: strength_weakness: It is still necessary to hand-craft the high-level features, which needs expert knowledge (or at least quite high familiarity with downstream tasks) and can be heuristic. 

Evaluation only used one meta-RL method (VariBAD). It would be useful to see performance of other meta-RL methods such as RL^2 on the DIVA task distribution.
questions: questions: Does DIVA depend on any parameterizations at all? It seems like it might be possible (and very useful) to try DIVA on “weakly” parameterized tasks such as those generated by prompting an LLM or, say Genie [1].

[1] Genie: Generative Interactive Environments. Bruce el al. https://arxiv.org/abs/2402.15391
limitations: limitations: Limitations have been adequately addressed.

Comments: title: Comment
Response to Rebuttal by Authors
SAMPLR, CLUTR, and DRED Full Assessments
Additional Rebuttal Content
Additional Rebuttal Content (2)
Additional Rebuttal Content (1)
Additional Rebuttal Content
comment: Thank you for the detailed rebuttal. I maintain my original rating.
Thank you for the detailed response, the proposed changes do clarify the paper significantly. Unfortunately I don't see room to increase my score past "high impact on at least one sub-area", but I appreciate the work and do believe this paper should be accepted.
*[continued from above]*

**We would appreciate if the reviewer were able to provide some specific property of open-ended domains with unstructured parameterizations that we have not captured** with the principle of rarity in generating meaningfully diverse solutions. We do not claim that our evaluation covers every possible way in which open-ended domain parameterizations may express themselves, but this is not the burden of proof we place on methods for demonstrating their promise on a certain problem setting. Nor do we claim that DIVA can overcome shear statistical impossibilities (in practical terms) that some unstructured parameterizations may present. But based on our confidence in the experimental setup, and the convincingness of the results, we stand by the claim the reviewer has cited, that “our empirical results demonstrate DIVA’s unique ability to leverage ill-parameterized simulators [...]”. 

A brief note on **genetic techniques and QD**. It is possible that different QD algorithms will benefit more or less from whatever structure the genotype-phenotype mapping contains. For instance, in Alchemy, we use MAP-Elites over the discrete genotype, which does not exploit the structure at all. The random permutations uncover levels with new features periodically, and these levels, which have latched onto new diversity, are used as a basis for further mutations. DIVA can benefit from this in the ways discussed above, but it does not latch onto the structure in any way. For Racing, both DIVA and Alchemy use CMA-ES, a more sophisticated QD approach that works well with continuous action spaces—in this case, the covariance matrix guiding the mutations is adapted to more efficiently uncover the parameter space. As we mention in Section 7, it may be possible to even further inform training over complex action spaces by performing a pretraining step (or iterative updates)with some latent model, and then exploring the parameter space with a QD approach that can take advantage of gradients.

## (2) Specifying features

We appreciate the receptivity of the reviewer on the point about hand-crafted generators, and their clarification on the nature of their objection surrounding features. We are not quite sure what the reviewer is referring to by the “new scalability bottleneck”, but we believe this likely refers to the assumption we make about the availability of specified features. If this is the case, we are unsure how the claim oversells our approach, given our transparency surrounding this limitation in the manuscript—which as previously mentioned, is a limitation shared by most QD works. Would the reviewer like to see, for instance, a more precise mention of this fact in the abstract, specifically? We also wonder, more fundamentally, how this qualifies as a “new scalability bottleneck”, given our agreement that designing diverse generators presents a larger scalability bottleneck than DIVA’s requirement of specifying features. It is because DIVA does away with this more restrictive bottleneck that we find it accurate to say that our “findings highlight the potential of approaches like DIVA to enable training in complex open-ended domains [...]”. This claim does not imply that all complex open-ended domains are now in reach; instead, we are trying to get across that DIVA has demonstrated the *potential* of these kinds of approaches for bringing open-ended domains *more* within our collective reach.
We appreciate the reviewer’s continued engagement with our work, and for further clarifying their criticisms. We understand the reviewer’s desire to ensure the work’s claims are properly qualified and its significance is not overstated. However, we believe certain characterizations disputed by the reviewer are in fact accurate, which we address in detail below. We are happy to respond to any further questions or concerns the reviewer may have on these points or any others. 

## (1) On the structure of open-ended parameterizations

We would like to first address the reviewer’s “fear” that we might have “inadvertently introduced structure [that results] in compelling performance” for our algorithm specifically. **We can scrutinize the concrete facts of our experimental setup *directly* to determine if this fear is grounded in truth.** Let us remind ourselves of the parameterizations for both the Alchemy and Racing domains, which contain two very different types of open-ended structures. 

For the **Alchemy** domain, we effectively introduce a deeper layer of chemistry to the environment, whereby each latent stone dimension is represented by $k$ bits (of information) instead of just one. This increased resolution makes the environment more “open-ended”; greater diversity is created in the genotype space, but now it is less likely that random genotypes will produce the kind of diversity we care about. Thus, it is also a less “structured” parameterization, according to our definition; for stones with diverse phenotypes to be generated, the values of these bits must coordinate, which is made statistically less likely the higher $k$ is ($k = 8$ in our experiments; where the degenerate case is $k=1$). 

**DIVA is able to successfully coordinate these bits to produce diverse phenotypes *not* by exploiting the genotypical structure in some unique way that favors DIVA specifically**—-this can be seen in the mere fact that ACCEL uses the same mutation function as DIVA. Instead, *DIVA succeeds because (1) it is able to perform significantly more mutations and evaluations than ACCEL, by virtue of avoiding agent rollouts on every mutated level, and (2) by employing a QD archive, it captures all diversity encountered*, whereas approaches like ACCEL and PLR only store levels where learning potential is detected, meaning they will discard or ignore levels in the buffer where agent rollouts do not register learning potential. 

In **Racing**, a different kind of open-endedness challenge is introduced into the level parameter space. We define a small region of the parameter space that can maximally control the diversity of the racetracks. There is no structure or information about this region that is leaked to DIVA; the mutation function is simply fighting against the statistical improbability of landing in this region (which DIVA's QD archive is able to effectively capture). In Racing, as in Alchemy, **ACCEL and DIVA share the same mutation function**. 

To further justify these design choices, **consider what will naturally occur as simulated robotics domains become more “open-ended”**---we might make a robotic manipulation task more open-ended by enabling more granular objects to be represented and available for manipulation. But if we loosen the parameterization of the environment to allow a greater number of objects to be represented, it will be possible to specify all kinds of objects that either wouldn’t naturally occur in the real world, contain structural impossibilities, or otherwise would not be of much use to learn how to manipulate. Only a subset of the parameter space now will correspond to meaningfully diverse objects—-just like we have achieved for Alchemy and Racing. For Alchemy, we designed this diverse subset of configurations to correspond to a specific kind of improbable bit coordination, and in Racing, we designed this subset to correspond to a small region of the parameter space—-and in neither case does this structure specifically favor DIVA in any way. 

And to be clear why we have not obtained results on a robotics environment of the kind imagined above—-i.e. one with a “naturally” challenging open-ended parameterization: In short, this would have made for a prohibitively expensive initial exploration into this problem setting---especially given the limited resources of an academic institution, and the fact that meta-RL methods still find existing robotics domains challenging. We therefore stand by our decision to take domains that have proven useful for prior works UED and meta-RL research and bestow upon them the most important properties of domains with open-ended parameterizations. 

*[continued below]*
In response to the authors' detailed responses to my and other reviews, I have two comments:

On the question of evaluation domains and their parametrisation. The authors note that the reason they deliberately re-parametrise their domains is because most existing evaluation domains are built with meaningful parametrisations by design. I think this point was already understood. My original comment was not to question why this was done, but rather to highlight that this introduces a critical weakness to the paper's argument: that DIVA is likely to be a useful technique for domains where a meaningful parametrisation is naturally not available.

I don't think this criticism is "whataboutism" as the authors' poetically put it. It is well known - perhaps an "open secret" - that while genetic techniques are in principle very general, their efficacy in real-world scenarios often depends critically on the structure of the genotype-phenotype mapping. Indeed, if this map has no exploitable structure then genetic techniques generally reduce to random search. My fear is that by introducing their own simple "scrambling" of the genotype-phenotype mapping, that they have inadvertently introduced structure in this map which results in compelling performance from their algorithm. But it is certainly not obvious whether the required exploitable structures exist in more complex open-ended environments.

To put it succinctly, the authors' main claim is that "Our empirical results demonstrate DIVA's unique ability to leverage ill-parameterized simulators to ..." and I'm not sure this is well supported, as the environments the authors have used may well not be "ill-parametrized" in a relevant way, with respect to more interesting open-ended environments.

On specifying features. The authors' comment that while they appreciate hand-crafting diversity measuring features is a limitation, it is significantly less limiting than having to build a diverse environment generator. This comment is well taken. However, I think my objection is more that the authors make quite a big claim, that "These findings highlight the potential of approaches like DIVA to enable training in complex open-ended domains, and to produce more robust and adaptable agents" and I don't think that claim is well supported, given that they have *introduced* a new scalability bottleneck. This is less a comment on the research, and more a comment on how it is pitched. I would recommend softening the claim a little so that it is more consistent with what the authors have achieved.

Overall, the authors comments don't significantly change my opinion of the paper's suitability for NeurIPS.

[Edited for formatting only.]
Important typo fix: under **Questions** [Q1]: "the meta-learning is in fact *not* tuned" (the *not* is missing in the current response, and we are unable to edit the response directly at this time).
We appreciate the reviewer for engaging with our rebuttal and for helping strengthen our paper. We will follow through with the reviewer's feedback to include the explanations for the design choices (including the content contained in our rebuttal) in the camera-ready manuscript. This will be added along with the ablations and other updates noted, which have already been applied to our working manuscript draft. We thank the reviewer for updating their score, and are more than happy to address any remaining concerns.
Thanks to the authors for their detailed response and the work put into the new experiments. The new experiments do address many of the concerns I had. For the presentation of the paper, I hope the authors will put the explanations for many of the design choices (like what the authors wrote for rebuttals above) into the revised manuscript. I think that will significantly improve the paper's presentation. As such, I have increased my score.
> [SAMPLR] Jiang, Minqi, et al. "Grounding aleatoric uncertainty for unsupervised environment design." Advances in Neural Information Processing Systems 35 (2022): 32868-32881.

**SAMPLR** attempts to correct the “curriculum-induced covariate shift” (CICS)—with respect to the downstream task distribution—that results from learning an adaptive curriculum. SAMPLR mitigates this bias over aleatoric parameters while preserving the benefits of utilizing a curriculum (versus sampling directly from the downstream task parameters). 

Assessment: We agree with the reviewer’s assessment that this approach is less relevant for direct comparison to DIVA since SAMPLR requires access to the simulator itself (and the direct downstream sampling distribution, which we do not assume access to either). However, we believe SAMPLR’s relevance to the meta-RL setting (meta-RL potentially suffers even more from this bias) warrants mention in the related work section---an addition which will appear in the camera-ready draft.

> [CLUTR] Azad, Abdus Salam, et al. "Clutr: Curriculum learning via unsupervised task representation learning." International Conference on Machine Learning. PMLR, 2023.

The authors of **CLUTR** argue that many UED methods are burdened by the difficulty of simultaneously learning the task manifold (implicitly through RL, for PAIRED-based methods) and the curriculum over these tasks. CLUTR disentangles the task representation learning from curriculum learning by first pretraining a task manifold with a VAE in an unsupervised manner, and then learning a curriculum over this (fixed) task manifold via maximizing regret. 

Assessment: By training a VAE on levels sampled from the generator distribution, which lack greatly in diversity in DIVA’s setting, CLUTR would fail to learn a generator that produces phenotypically diverse levels. ACCEL, by performing mutations to existing levels, is able evolve its levels towards greater diversity over time, making it a stronger baseline for our setting. CLUTR may, however, be a useful future addition to a DIVA-like approach, e.g. by converting the archive to a learned manifold via a VAE. This could be potentially useful for storage considerations (i.e. if our archive is too large), or to extrapolate beyond levels that exist in the archive. Performing curriculum learning over this distilled archive would look something like our DIVA+ algorithm, which combines the benefits of DIVA (diversity) with UED (curriculum) approaches, but would have these storage/extrapolatory benefits. Due to its relevance to future work, we have cited and added discussion of this work to the manuscript, which will appear in the camera-ready draft.

> [DRED] Garcin, Samuel, et al. "DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design." Forty-first International Conference on Machine Learning. 2024.

The authors introduce data-regularized environment design (**DRED**), which combines adaptive sampling (like UED) with a level generator that approximates p(x). DRED assumes access not directly to p(x), but to a set of level parameters X with which a VAE can be trained. Once the VAE is trained, PLR is used to produce scores for sampling (other details omitted here for simplicity).

Assessment: The biggest difference between DRED and DIVA’s settings is that DIVA does not assume access to the underlying level parameters, but rather only the feature values---and these alone are used to define the archive. The QD search is then tasked with producing samples that match the feature distribution (in our work, for both simplicity and sample efficiency, making assumes of either independent Gaussian/Uniform distributions). DIVA’s archive is initialized with random parameters from the parameter space; it never sees the parameters corresponding to the feature values. This difference makes DRED suffer from the same issue as CLUTR when faced with the setting that DIVA operates in: if the parameter space cannot produce diverse levels for the VAE training, then the resulting model will produce levels just as lacking in diversity. Resampling a la PLR will have a similar effect as our PLR baseline. Thus, as was mentioned of CLUTR above, ACCEL remains a stronger baseline than DRED for this reason. That being said, DRED’s close resemblance to our setting and possible applicability to future work (like CLUTR), we have cited and added discussion to our manuscript, which will appear in the camera-ready draft. We would like to note, however, that DRED’s absense in our original discussion was due to its very recent publication (uploaded to ArXiv in February 2024; presented at ICML in July 2024).
> [W4] In Figure 5d, “unique genotypes[“] is a quite bad metric for diversity, as completely random levels would score quite well even though most random levels are quite qualitatively to each other.

We agree with the reviewer that unique genotypes is indeed a bad metric for diversity. We include this plot not to highlight diversity, but to make the same observation the reviewer notes (see Line 207)—-that despite DR (for example) producing more unique genotypes than DIVA, DIVA is able to produce greater meaningful diversity, which leads to better performance. We also note that in Line 208 we mistakenly referenced “Figure 5” (in general) instead of “Figure 5d”, and we have made this correction.

> [Q2] In the abstract it is not clear to me what "well-behaved simulator parameterisations" means, what "unscalable flexibility" refers to, or what "ill parameterised simulators" means.

We agree with the reviewer that the abstract should be self-contained, and that terminology should either be widely understood or defined in the abstract itself. We will modify these phrases in the camera-ready draft to be clear and unambiguous. We thank the reviewer for bringing our attention to these instances. 

> [Q3] It would be good to have a citation on line 46 for how one could learn the axes of QD.

Great suggestion; we have added one such example [1] to our manuscript to be present in the camera-ready draft.

[1] Ding, L., Zhang, J., Clune, J., Spector, L., & Lehman, J. (2024). Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization. In Forty-first International Conference on Machine Learning.

> [Q4] It's not immediately clear what "genotype" means, and it seems to be used in different ways on line 82 and 88. In the first case it may be more conventional to call it the "level generator" and in the second case it could be more conventional to call it the "level parameters".

The reviewer is correct that the first use of genotype in Line 82 is misleading; we will update to “level parameterization” as this is what we mean. In Line 88 we will use “level parameters” in as the reviewer suggests for clarity, but we will still introduce and use “genotype” as a shorthand/alternative for “level parameters”, in order to respect the connection to evolutionary approaches.
> [Q6] Line 188 claims that “DIVA, on the other hand, is able to capture this diversity.” However, doesn’t DIVA also decrease to the same level of percentage coverage as DR*? Why are there no confidence intervals for Figure 3a, like the others?

Figure 3a shows the ability of DR, DR*, and DIVA to “capture” diversity in generated levels with parameterizations of varying complexity. DR* reflects the upper bound of what a method like PLR can achieve with DR levels—it is the maximum diversity see by DR (i.e. the total number of unique levels seen over the entire generation process), whereas DR is just the final archive produced. DR has no memory mechanism to preserve diversity, whereas a method like PLR, with its priority buffer, can latch on to the diversity contained in DR* (but still may fail to preserve all discovered diversity).

The reviewer is correct that DIVA is shown to capture just as much as DR* at the most complex parameterization we tested. However, given the nature of the experimental design, this trailing off is destined to happen with a high enough complexity; it becomes statistically unlikely any algorithm to produce meaningful diversity. It was a purposeful decision to demonstrate that DIVA tails off in its ability to handle these highly challenging parameterizations. If we increased the number of update steps allowed for DIVA, or inserted a more sophisticated emitter algorithm (instead of MAP-Elites), DIVA would be able to benefit from this far more than DR* (the same trend would extend, but fall off just the same at a high enough complexity). 

The takeaway from this plot is that QD is able to uncover and preserve diversity in the archive as the complexity increases better than DR (or our invented oracle DR*). This is also why we have not included confidence intervals—-the point is to show the trend; each point is just a single run of QD updates. Given the cleanness of the trend, we found it unnecessary to run more seeds for this plot, but we can do this for the camera-ready draft. Either way, we have clarified this point in the paper. Thanks for your feedback!
> [Q2] If the original parameterizations are chosen randomly or are not handcrafted to what humans might care about, how would DIVA perform?

We believe this question likely reflects the same misunderstanding noted above; we refer the reviewer to our [Q1] response, and welcome any follow-up questions the reviewer may have on this point.

> [Q3] To what extent and when do we consider an environment simulator to have structured or unstructured parameterizations?

We introduce the idea of structured/unstructured parameterizations in Figure 1, and elaborate specifically on the unstructured parameterizations DIVA can work with starting on Line 88. We define structured parameterizations as ones that enable random sampling to produce diverse levels of interest, whereas increasingly unstructured parameterizations produce these useful levels with diminishing probability. These terms are not absolute, but are instead relative, and represent two poles of a continuum. Open-ended domains, with more degrees of freedom and greater complexity in environment dynamics, must either be carefully parameterized and provided with a sophisticated generator that can navigate these complexities (i.e. a structured parameterization), or they can be more flexibility parameterized with a simpler generator (an unstructured parameterization), where meaningful diversity is possible, but not guaranteed with high probability. DIVA can make use these more unstructured parameterizations, and some knowledge of some knowledge of what levels should look like, in order to produce an abundance of training levels that are diverse in these ways. Importantly, this saves the prohibitive effort of having to carefully craft structured parameterizations / generators for complex open-ended domains. 

> [Q4] In the GridNav experiment, why do we want to prevent the generation of diverse goals along the y-axis?

We believe this may pertain to the same misunderstanding addressed in [Q1], so we refer the reviewer to that response, but we also address this question more extensively below. 

The object of DIVA and the baselines are to produce meaningful diversity in the generated levels, and for the GridNav domain, this means producing goal diversity along both the x and y axis. 

In our GridNav experimental setup, we inject complexity into the parameterization that makes it difficult to generate goal diversity along the y-axis. We design the parameterization in such a way that we can vary the complexity (unstructuredness) of the parameterization. As we increase the complexity, diversity along the y-axis becomes less and less likely. Our results in GridNav show that DIVA is best able to capture this diversity by (1) evolving levels with the expressed purpose of discovering and preserving levels of meaningful diversity (via the QD archive), and (2) avoiding agent evals like ACCEL/PLR, which are expensive, and doesn’t guarantee all diversity discovered is preserved. 

> [Q5] Is there a hypothesis on why “the domain randomization over increasingly complex genotypes diminishes diversity in terms of goal location” (lines 187-188)?

This is by design (described in Line 178) rather than some emergent property requiring hypothesis, and is likely relevant to the misunderstanding addressed in [Q4]. Increasingly complex genotypes diminish diversity in terms of goal location by our design of the genotype; and we design the general genotype scheme in such a way that we can conveniently vary the complexity—in this case, the number of genes that must coordinate to produce diversity along the y-axis. The effect of this design is that diversity along the y-axis becomes less and less likely (with random genotypes) as the complexity increases, because more genes need to coordinate.
More on **(1a)**: GridNav is a standard navigation task which serves as a useful didactic domain (and serves this purpose in VariBAD [1] as well), the Racing domain is a control domain used to benchmark numerous UED works [2, 3], and symbolic Alchemy is a chemistry-inspired meta-RL domain that requires complex trial-and-error reasoning over numerous episodes, which nicely complements the navigation and control tasks.

More on **(2)**: We might also add that this is the first work to consider the combination of QD and meta-RL in this manner. In quality diversity (QD) literature, features are typically assumed to be pre-determined [4]. This pre-specification of features is indeed a limitation, but one that can be lifted in a similar fashion to QD works that learn features automatically [5]. Based on the contributions of our work, we decided it was appropriately out of scope to additionally consider the problem of learning features automatically. 

[1] Zintgraf, L., et al. (2019, September). VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning. In International Conference on Learning Representations.

[2] Parker-Holder, J., et al. (2022, June). Evolving curricula with regret-based environment design. In International Conference on Machine Learning (pp. 17473-17498). PMLR.

[3] Jiang, M., et al. (2021). Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34, 1884-1897.

[4] Pugh, Justin K. et al. (2016) “Quality Diversity: A New Frontier for Evolutionary Computation.” Frontiers in Robotics and AI 3:40.

[5] Ding, L. et al. (2024). Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization. ICML.