Enhancing the Hierarchical Environment Design via Generative Trajectory Modeling

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:22:09

Abstract: Unsupervised Environment Design (UED) is a paradigm that automatically generates a curriculum of training environments, enabling agents trained in these environments to develop general capabilities, i.e., achieving good zero-shot transfer performance. However, existing UED approaches focus primarily on the random generation of environments for open-ended agent training. This is impractical in resource-limited scenarios where there is a constraint on the number of environments that can be generated. In this paper, we introduce a hierarchical MDP framework for environment design under resource constraints. It consists of an upper-level RL teacher agent that generates suitable training environments for a lower-level student agent. The RL teacher can leverage previously discovered environment structures and generate environments at the frontier of the student's capabilities by observing the student policy's representation. Additionally, to alleviate the time-consuming process of collecting the experience of the upper-level teacher, we utilize recent advances in generative modeling to synthesize a trajectory dataset for training the teacher agent. Our method significantly reduces the resource-intensive interactions between agents and environments, and empirical experiments across various domains demonstrate the effectiveness of our approach.

Main paper: 
 introduction: 
 The advances of reinforcement learning (RL) [17] have promoted research into the problem of training autonomous agents that are capable of accomplishing complex tasks. One interesting, yet underexplored, area is training agents to perform well in unseen environments, a concept referred to as zero-shot transfer performance. To this end, Unsupervised Environment Design (UED) [3] has emerged as a promising paradigm to address this problem. The objective of UED is to automatically generate environments in a curriculum-based manner, and training agents in these sequentially generated environments can equip agents with general capabilities, enabling agents to learn robust and adaptive behaviors that can be transferred to new scenarios without explicit exposure during training.Existing approaches in UED primarily focus on building an adaptive curriculum for the environment generation process to train the generally capable agent. Dennis et al. [3] formalize the problem of finding adaptive curricula through a game involving an adversarial environment generator (teacher agent), an antagonist agent (expert agent), and the protagonist agent (student agent). The RL-based teacher is designed to generate environments that maximize regret, defined as the difference between the protagonist and antagonist agent's expected rewards. They show that these agents will reach a Nash Equilibrium where the student agent learns the minimax regret policy. However, since the teacher agent adapts solely based on the regret feedback, it is inherently difficult to adapt to student policy changes. Meanwhile, training such an RL-based teacher remains a challenge because of the high computational cost of training an expert antagonist agent for each environment.In contrast, domain randomization [19] based approaches circumvent the overhead of developing an RL teacher by training agents in randomly generated environments, resulting in good empirical performances. Building upon this, Jiang et al. [7] introduce an emergent curriculum by sampling randomly generated environments with high regret value 1 to train the agent. Parker-Holder et al. [10] then propose the adaptive curricula by manually designing a principled, regret-based curriculum, which involves generating random environments with increasing complexity. While these domain randomization-based algorithms have demonstrated good zero-shot transfer performance, they face limitations in efficiently exploring large environment design spaces and exploiting the inherent structure of previously discovered environments. Moreover, existing UED approaches typically rely on open-ended learning, necessitating a long training horizon, which is unrealistic in the real world due to resource constraints. Our goal is to develop a teacher policy capable of generating environments that are perfectly matched to the current skill levels of student agents, thereby allowing students to achieve optimal general capability within a strict budget for the number of environments generated and within a shorter training time horizon.In this paper, we address these challenges by introducing a novel, adaptive environment design framework. The core idea involves using a hierarchical Markov Decision Process (MDP) to simultaneously formulate the evolution of an upper-level teacher agent, tasked with generating suitable environments to train the lower-level student agent to achieve general capabilities. To accurately guide the generation of environments at the frontier of the student agent's current capabilities, we propose approximating the student agent's policy/capability by its performances across a set of diverse evaluation environments, which acts as the state abstraction for the teacher's decision-making process.The transitions in the teacher's state represent the trajectories of the student agent's capability after training in the generated environment. However, collecting experience for the upper-level teacher agent is slow and resource-intensive, since each upper-level MDP transition evolves a complete training cycle of the student agent on the generated environment. To accelerate the collection of upper-level MDP experiences, we utilize advances in diffusion models that can generate new data points capturing complex distribution properties, such as skewness and multi-modality, exhibited in the collected dataset [11] . Specifically, we employ diffusion probabilistic model [15, 6] to learn the evolution trajectory of student policy/capability and generate synthetic experiences to enhance the training efficiency of the teacher agent. Our method, called Synthetically-enhanced Hierarchical Environment Design (SHED), automatically generates increasingly complex environments suited to the current capabilities of student agents.In summary, we make the following contributions:• We develop a novel hierarchical MDP framework for UED that introduces a straightforward method to represent the current capability level of the student agent.• We introduce SHED, which utilizes diffusion-based techniques to generate synthetic experiences. This method can accelerate the training of the off-policy teacher agent.• We demonstrate that our method outperforms existing UED approaches (i.e., achieving a better general capability under resource constraints) in different task domains. 
preliminaries: 
 In this section, we provide an overview of two main research areas upon which our work is based.The objective of UED is to generate a sequence of environments that effectively train the student agent to achieve a general capability. Dennis et al. [3] first model UED with an Underspecified Partially Observable Markov Decision Process (UPOMDP), which is a tupleM =< A, O, Θ, S M , P M , I M , R M , γ >. The UPOMDP has a set Θ representing the free parameters of the environments, which are determined by the teacher agent and can be distinct to generate the next new environment. Further, these parameters are incorporated into the environment-dependent transition function P M : S × A × Θ → S. Here A represents the set of actions, S is the set of states. Similarly, I M : S → O is the environment-dependent observation function, R M is the reward function, and γ is the discount factor. Specifically, given the environment parameters ⃗ θ ∈ Θ, we denote the corresponding environment instance as M ⃗ θ . The student policy π is trained to maximize the cumulative rewards V M ⃗ θ (π) = T t=0 γ t r t in the given environment M ⃗ θ under a time horizon T , and r t are the collected rewards in M ⃗ θ . Existing works on UED consist of two main strands: the RL-based environment generation approach and the domain randomization-based environment generation approach.The RL-based generation approach was first formalized by Dennis et al. [3] as a self-supervised RL paradigm for generating environments. This approach involves co-evolving an environment generator policy (teacher) with an agent policy π (student), where the teacher's role is to generate environment instances that best support the student agent's continual learning. The teacher is trained to produce challenging yet solvable environments that maximize the regret measure, which is defined as the performance difference between the current student agent and a well-trained expert agent π * within the current environment:Regret M ⃗ θ (π, π * ) = V M ⃗ θ (π * ) -V M ⃗ θ (π).The domain randomization-based generation approach, on the other hand, involves randomly generating environments. Jiang et al. [7] propose to collect encountered environments with high learning potentials, which are approximated by the Generalized Advantage Estimation (GAE) [12] , and then the student agent can selectively train in these environments, resulting in an emergent curriculum of increasing difficulty. Additionally, Parker-Holder et al. [10] adopt a different strategy by using predetermined starting points for the environment generation process and gradually increasing complexity. They manually divide the environment design space into different difficulty levels and employ human-defined edits to generate similar environments with high learning potentials. Their algorithm, ACCEL, is currently the state-of-the-art (SOTA) in the field, and we use an edited version of ACCEL as a baseline in our experiments.Diffusion models [15] are a specific type of generative model that learns the data distribution.Recent advances in diffusion-based models, including Langevin dynamics and score-based generative models, have shown promising results in various applications, such as time series forecasting [18] ,robust learning [9] , anomaly detection [21] as well as synthesizing high-quality images from text descriptions [8, 11] . These models can be trained using standard optimization techniques, such as stochastic gradient descent, making them highly scalable and easy to implement.In a diffusion probabilistic model, we assume a d-dimensional random variable x 0 ∈ R d with an unknown distribution q(x 0 ). Diffusion Probabilistic model involves two Markov chains: a predefined forward chain q(x k |x k-1 ) that perturbs data to noise, and a trainable reverse chain p ϕ (x k-1 |x k ) that converts noise back to data. The forward chain is typically designed to transform any data distribution into a simple prior distribution (e.g., standard Gaussian) by considering perturb data with Gaussian noise of zero mean and a fixed variance schedule {β k } K k=1 for K steps:EQUATIONwhere k ∈ {1, . . . , K}, and 0 < β 1:K < 1 denote the noise scale scheduling. As K → ∞, x K will converge to isometric Gaussian noise: x K → N (0, I). According to the rule of the sum of normally distributed random variables, the choice of Gaussian noise provides a closed-form solution to generate arbitrary time-step x k through:EQUATIONHere α k = 1 -β k and ᾱk = k s=1 α s . The reverse chain p ϕ (x k-1 |x k ) reverses the forward process by learning transition kernels parameterized by deep neural networks. Specifically, considering the Markov chain parameterized by ϕ, denoising arbitrary Gaussian noise into clean data samples can be written as:EQUATION3) It uses the Gaussian form p ϕ (x k-1 |x k ) because the reverse process has the identical function form as the forward process when β t is small [15] . Ho et al. [6] consider the following parameterization ofEQUATION) ϵ ϕ is a trainable function to predict the noise vector ϵ from x k . Ho et al. [6] show that training the reverse chain to maximize the log-likelihood q(x 0 ) log p ϕ (x 0 )dx 0 is equivalent to minimizing re-weighted evidence lower bound (ELBO) that fits the noise. They derive the final simplified optimization objective:EQUATION)Once the model is trained, new data points can be subsequently generated by first sampling a random vector from the prior distribution, followed by ancestral sampling through the reverse Markov chain in Equation 3.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The limitations of this work is discussed in Appendix F.1.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
approach: 
 In this section, we formally describe our method, Synthetically-enhanced Hierarchical Environment Design (SHED), which is a novel framework for UED under resource constraints. The SHED incorporates two key components that differentiate it from existing UED approaches:• A hierarchical MDP framework to generate suitable environments,• A generative model to generate the synthetic trajectories.SHED uses a hierarchical MDP framework where an RL teacher leverages the observed student's policy representation to generate environments at the student's capabilities frontier. Such targeted environment generation process enhances the student's general capability by utilizing the underlying structure of previously discovered environments, rather than relying on the open-ended random generation. Besides, SHED leverages advances in generative models to generate synthetic trajectories that can be used to train the off-policy teacher agent, which significantly reduces the costly interactions between the agents and the environments. The overall framework is shown in Figure 1 , and the pseudo-code is provided in Algorithm 1.The objective is to generate a limited number of environments that are designed to enhance the general capability of the student agent. Inspired by the principles of PAIRED [3] , we adopt an RL-based approach for the environment generation process. To better generate suitable environments tailored to the current student skill level, SHED uses the hierarchical MDP framework, consisting of an upper-level RL teacher policy Λ and a lower-level student policy π. Specifically, the teacher policy, Λ : Π → Θ, maps from the space of all potential student policies Π to the space of environment parameters Θ. Existing RL-based methods (e.g., PARIED) rely solely on regret feedback and fail to effectively capture the nuances of the student policy. To address this challenge, SHED enhances understanding by encoding the student policy π into a vector that serves as the state abstraction for teacher Λ. Rather than compressing the knowledge in the student policy network, we approximate the embedding of the student policy π by assessing performance across a set of diverse evaluation environments. This performance vector, denoted as p(π), gives us a practical estimate of the student's current general capabilities, enabling the teacher to customize the next training environments accordingly. In our hierarchical framework, the environment generation process is governed by discrete-time dynamics. We delve into the specifics below.Upper-level teacher MDP. The upper-level teacher operates at a coarser layer of student policy abstraction and generates environments to train the lower-level student agent. This process can be formally modeled as an MDP by the tuple < S u , A u , P u , R u , γ u >:• S u represents the upper-level state space. Typically, s u = p(π) = [p 1 , . . . , p m ] denotes the student performance vector across m diverse evaluation environments. This vector serves as the representation of the student policy π and is observed by the teacher. end for 16: end for Output: Λ, π, D • A u is the upper-level action space. The teacher observes the abstraction of the student policy, s u and produces an upper-level action a u which is the environment parameters ⃗ θ. ⃗ θ (a u ) is then used to generate specific environment instances M ⃗ θ . Thus the upper-level action space A u is the environment parameter space Θ.• P u denotes the action-dependent transition dynamics of the upper-level state. The general capability of the student policy evolves due to training the student agent on the generated environments.• R u provides the upper-level reward to the teacher at the end of training the student on the generated environment. The design of R u will be discussed in Section 3.3.As shown in Figure 2 , given the student policy π, the teacher Λ first observes the representation of the student policy, s u = [p 1 , . . . , p m ]. Then teacher produces an upper-level action a u which corresponds to the environment parameters. These environment parameters are subsequently used to generate specific environment instances. The lower-level student policy π will be trained on the generated environments for C training steps. The upper-level teacher collects and stores the student policy evolution transition (s u , a u , r u , s u,′ ) every C times steps for off-policy training. The teacher agent is trained to maximize the cumulative reward giving the budget for the number of generated environments. The choice of the evaluation environments will be discussed in Section 3. The hierarchical framework enables the teacher agent to systematically measure and enhance the general capability of the student agent and to adapt the training process accordingly. However, it's worth noting that collecting student policy evolution trajectories (s u , a u , r u , s u,′ ) to train the teacher agent is notably slow and resource-intensive, since each transition in the upper-level teacher MDP encompasses a training horizon of C timesteps for the student in the generated environment. Thus, it is essential to reduce the need for costly collection of upper-level teacher experiences.In this section, we will formally introduce a generative model designed to ease the collection of upperlevel MDP experience. This will allow us to train our teacher policy more efficiently. In particular, we first utilize a diffusion model to learn the conditional data distribution from the collected experiencesτ = {(s u t , a u t , r u t , s p,′ t )}.Later we can use the reverse chain in the diffusion model to generate the synthetic trajectories that can be used to help train the teacher agent, thereby alleviating the need for extensive and time-consuming collection of upper-level teacher experiences. We deal with two different types of timesteps in this section: one for the diffusion process and the other for the upperlevel teacher agent, respectively. We use subscripts k ∈ 1, . . . , K to represent diffusion timesteps and subscripts t ∈ 1, . . . , T to represent trajectory timesteps in the teacher's experience.In the image domain, the diffusion process is implemented across all pixel values of the image. In our setting, we diffuse over the next state s u,′ conditioned the given state s u and action a u . We construct our generative model according to the conditional diffusion process:q(s u,′ k |s u,′ k-1 ), p ϕ (s u,′ k-1 |s u,′ k , s u , a u ) As usual, q(s u,′ k |s u,′ k-1) is the predefined forward noising process while p ϕ (s u,′ k-1 |s u,′ k , s u , a u ) is the trainable reverse denoising process. We begin by randomly sampling the collected experiences τ = {(s u t , a u t , r u t , s u,′ t )} from the real experience buffer B real . Giving the observed state s u and action a u , we use the reverse process p ϕ to represent the generation of the next state s u,′ :p ϕ (s u,′ 0:K |s u , a u ) = N (s u,′ K ; 0, I) K k=1 p ϕ (s u,′ k-1 |s u,′ k , s u , a u )At the end of the reverse chain, the sample s u,′ 0 , is the generated next state s u,′ . Similar to Ho et al. [6] , we parameterize p ϕ (s ′ k-1 |s ′ k , s u , a u ) as a noise prediction model with the covariance matrix fixed as Σ ϕ (s u,′ k , s u , a u , k) = β i I, and the mean isµ ϕ (s u,′ i , s u , a u , k) = 1 √ α k s u,′ k - β k √ 1 -ᾱk ϵ ϕ (s u,′ k , s u , a u , k) ϵ ϕ (s u,′ k , s u , a u , k)is the trainable denoising function, which aims to estimate the noise ϵ in the noisy input s u,′ k at step k.Training objective. We employ a similar simplified objective to train the conditional ϵmodel:EQUATIONWheres u,′ k = √ ᾱk s u,′ + √ 1 -ᾱk ϵ.The intuition for the loss function L(ϕ) is to predict the noise ϵ ∼ N (0, I) at the denoising step k, and the diffusion model is essentially learning the student policy involution trajectories collected in the real experience buffer B reals . Note that the reverse process necessitates a substantial number of steps K [15] . Recent research by Xiao et al. [22] has demonstrated that enabling denoising with large steps can reduce the total number of denoising steps K. To expedite the relatively slow reverse sampling process (as it requires computing ϵ ϕ networks K times), we use a small value of K. Similar to Wang et al. [20] , while simultaneously setting β min = 0.1 and β max = 10.0, we define:β k = 1 -exp β min × 1 K -0.5(β max -β min ) 2k -1 K 2This noise schedule is derived from the variance-preserving Stochastic Differential Equation by Song et al. [16] .Generate synthetic trajectories.Once the diffusion model has been trained, it can be used to generate synthetic experience data by starting with a draw from the prior s u,′ K ∼ N (0, I) and successively generating denoised next state, conditioned on the given s u and a u through the reverse chain p ϕ .Note that the giving condition action a can either be randomly sampled from the action space or use another diffusion model to learn the action distribution giving the initial state s u . This new diffusion model is essentially a behavior-cloning model that aims to learn the teacher policy Λ(a u |s u ). This process is similar to the work of Wang et al. [20] . We discuss this process in detail in the appendix.In this paper, we randomly sample a u as it is straightforward and can also increase the diversity in the generated synthetic experience to help train a more robust teacher agent.After obtaining the generated next state s u,′ conditioned on s u , a u , we compute reward r u using teacher's reward function R(s u , a u , s u,′ ). The specifics of how the reward function is chosen are explained in the following section.Selection of evaluation environments. The upper-level teacher generates environments tailoredfor the lower-level student to improve its general capability. Thus it is important to select a set of diverse suitable evaluation environments as the performance vector reflects the student agent's general capabilities and serves as an approximation of the policy's embedding. Fontaine and Nikolaidis [5] propose the use of quality diversity (QD) optimization to collect high-quality environments that exhibit diversity for the agent behaviors. Similarly, Bhatt et al. [1] introduce a QD-based algorithm for dynamically designing such evaluation environments based on the current agent's behavior. However, it's worth noting that this QD-based approach can be tedious and time-consuming, and the collected evaluation environments heavily rely on the given agent policy.Given these considerations, it is natural to take advantage of the domain randomization algorithm,as it has demonstrated compelling results in generating diverse environments and training generally capable agents. In our approach, we first discretize the environment parameters into different ranges, then randomly sample from these ranges, and combine these parameters to generate evaluation environments. This method can generate environments that may induce a diverse performance for the same policy, and it shows promising empirical results in the final experiments.Reward design. We define the reward function for the upper-level teacher policy as a parameterized function based on the improvement in student performance in the evaluation environments after training in the generated environment:R(s u , a u , s u,′ ) = m i=1 (p ′ i -p i )This reward function gives positive rewards to the upper-level teacher for taking action to create the right environment to improve the overall performance of students across diverse environments.However, it may encourage the teacher to obtain higher rewards by sacrificing student performance in one subset of evaluation environments to improve student performance in another subset, which conflicts with our objective to develop a student agent with general capabilities. Therefore, we need to consider fairness in the reward function to ensure that the generated environment can improve student's general capabilities. Similar to [4] , we build our fairness metric on top of the change in student's performance in each evaluation environment, denoted as ω i = p ′ i -p i , and we haveω = 1 m m i=1 ω i .We then measure the fairness of the teacher's action using the coefficient of variation of student performances:EQUATIONA teacher is considered to be fair if and only if the cv is smaller. As a result, our reward function is:EQUATIONHere η is the coefficient that balances the weight of fairness in the reward function (We set a small value to η). This reward function motivates the teacher to generate training environments that can improve student's general capability. The distance from the starting position to the end position also affects difficulty.-If the path from the start position to the end position requires fewer than 5 steps, the goal location is considered easy.-If the path from the start position to the end position requires 5 to 10 steps, the goal location is considered medium.-If the path from the start position to the end position requires more than 10 steps, the goal location is considered hard.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments: 
 In this section, we conduct experiments to compare SHED to other leading approaches on three domains: Lunar Lander, maze and a modified BipedalWalker environment. Experimental details and hyperparameters can be found in the Appendix. Specifically, our primary comparisons involve SHED and h-MDP (our proposed hierarchical approach without diffusion model aiding in training) against four baselines: domain randomization [19] , ACCEL, [10] , Edited ACCEL(with slight modifications that it does not revisit the previously generated environments), PAIRED [3] . In all cases, we train a student agent via Proximal Policy Optimization (PPO [13] , and train the teacher agent via Deterministic policy gradient algorithms(DDPG [14] ), because DDPG is an off-policy algorithm and can learn from both real experiences and the synthetic experiences.Setup. For each domain, we construct a set of evaluation environments and a set of test environments.The vector of student performances in the evaluation environments is used as the approximation of the student policy (as the observation to teacher agent), and the performances in the test environments are used to represent the student's zero-shot transfer performances (general capabilities). Note that in order to obtain a fair comparison of zero-shot transfer performance, the evaluation environments and test environments do not share the same environment and they are not present during training.Lunar Lander. This is a classic rocket trajectory optimization problem. In this domain, student agents are tasked with controlling a lander's engine to safely land the vehicle. Before the start of each episode, teacher algorithms determine the environment parameters that are used to generate environments in a given play-through, which includes gravity, wind power, and turbulence power. These parameters directly alter the difficulty of landing the vehicle safely. The state is an 8-dimensional vector, which includes the coordinates of the lander, its linear velocities, its angle, its angular velocity, and two booleans that represent whether each leg is in contact with the ground or not.We train the student agent for 1e6 environment time steps and periodically test the agent in test environments. The parameters for the test environments are randomly generated and fixed during training. We report the experiment results on the left side of Figure 3 . As we can see, student agents trained under SHED consistently outperform other baselines and have minimal variance in transfer performance. During training, the baselines, except h-MDP, show a performance dip in the middle. This phenomenon could potentially be attributed to the inherent challenge of designing the appropriate environment instance in the large environment parameter space. This further demonstrates the effectiveness of our hierarchical design (SHED and h-MDP), which can successfully create environments that are appropriate to the current skill level of the students.Bipedalwalker. We also evaluate SHED in the modified BipedalWalker from Parker-Holder et al. [10] . In this domain, the student agent is required to control a bipedal vehicle and navigate across the terrain, and the student receives a 24-dimensional proprioceptive state with respect to its lidar sensors, angles, and contacts. The teacher is tasked to select eight variables (including ground roughness, the number of stairs steps, min/max range of pit gap width, min/max range of stump height, and min/max range of stair height) to generate the corresponding terrain.We use similar experiment settings in prior UED works, we train all the algorithms for 1e7 environment time steps, and then evaluate their generalization ability on ten distinct test environments in Bipedal-Walker domain. The parameters for the test environments are randomly generated and fixed during training. As shown in Figure 3 , our proposed method SHED surpasses all other baselines and achieves performance levels nearly on par with the SOTA (ACCEL). Meanwhile, SHED maintains a slight edge in terms of stability and overall performance and PAIRED suffers from a considerable degree of variance in its performance.Partially observable Maze. Here we study navigation tasks, where an agent must explore to find a goal while navigating around obstacles. The environment is partially observable, and the agent's field of view is limited to a 3 × 3 grid area. Unlike the previously mentioned domains, maze environments are non-parametric and cannot be directly represented by compact parameter vectors due to their high complexity. To solve this challenge, we propose a novel method to generate maze by leveraging advances in large language models (e.g., ChatGPT). Specifically, we implement a retrieval-augmented generation (RAG) process to optimize the ChatGPT's output such that it can generate desired maze environments. This process ensures that large language models reference authoritative knowledge bases to generate feasible mazes. To simplify the teacher's action space, we extracted several key factors that constitute the teacher's action space (environmental parameters) for maze generation. The starting position can also affect the difficulty of the maze. The starting position is categorized into five levels:-If the start position is close to 1, it means it should be located as close to the top left of the maze.-If the start position is close to 2, it means it should be located as close to the top right of the maze.-If the start position is close to 3, it means it should be located as close to the bottom left of the maze.-If the start position is close to 4, it means it should be located as close to the bottom right of the maze.-If the start position is close to 5, it means it should be located as close to the center of the maze.Please note that the generated maze uses -1 to represent blocks, 0 to represent the feasible path, 1 to represent the start position, and 2 to represent the end position. Must ensure that there is a feasible path in the generated maze! A feasible path means that 1 and 2 are connected directly through 0s, or 1 and 2 are connected directly. For example:Feasible Maze:EQUATION[0, 0, 0, - [-] These second example does not have any feasible path. 
conclusion: 
 In this paper, we introduce an adaptive approach for efficiently training a generally capable agent under resource constraints. Our approach is general, utilizing an upper-level MDP teacher agent that can guide the training of the lower-level MDP student agent agent. The hierarchical framework can incorporate techniques from existing UED works, such as prioritized level replay (revisiting environments with high learning potential). Furthermore, we have described a method to assist the experience collection for the teacher when it is trained in an off-policy manner. Our experiment demonstrates that our method outperforms existing UED methods, highlighting its effectiveness as a curriculum-based learning approach within the UED framework. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The detailed configuration of the experiments is listed with required computational resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The broader impacts of our paper are presented in Section F.1.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All the assets, used in our paper, are properly credited and we explicitly mention and properly respect the license and terms of use.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.• The paper should discuss whether and how consent was obtained from people whose asset is used.• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Theorem 1 There exists a finite evaluation environment set that can capture the student's general capabilities and the performance vector [p 1 , . . . , p m ] is a good representation of the student policy.To prove this, we first provide the following Assumption:Assumption 1 Let p(π, ⃗ θ) denote the performance of student policy π in an environment ⃗ θ. For ∀i-th dimension of the environment parameters, denoted as θ i , when changing the θ i to θ ′ i to get a new environment ⃗ θ ′ while keeping other environment parameters fixed, there∃δ i > 0, if |θ ′ i -θ i | ≤ δ i , we have |p(π, ⃗ θ ′ ) -p(π, ⃗ θ)| ≤ ϵ i , where ϵ i → 0.If this is true, we then can construct a finite set of environments, and the student performances in We begin from the simplest case where we only consider using one environment parameter to generate environments, denoted as θ i . We can construct a finite environment parameter set for environment parameters, which is {θ min i + 1/2 * δ i , θ min i + 3/2 * δ i , θ min i + 7/2 * δ i , . . . , θ max i -δ i /2}.+ 3/2 * δ i ), (θ min i +1/2 * δ i , θ min i +5/2δ i ), (θ min i +5/2 * δ i , θ min i +9/2 * δ i ), . . . , (θ max i -3/2 * δ i , θ max i ]},which we call it as representative parameter interval set, also denoted as{(θ i -δ, θ i + δ)} Li i=1 .For any environment generated in those intervals, denoted as ⃗ θ ′ i , the performance p(π, ⃗ θ ′ i ) can always be represented by the p(π, ⃗ θ i ) which is in the same interval, as Then we extend to two environment parameter design space cases. Let's assume that the environment is generated by two-dimension environment parameters. Then, for each environment parameter,|p(π, ⃗ θ ′ i ) -p(π, ⃗ θ i )| ≤ ϵ i ,θ i ∈ {θ 1 , θ 2 }.We can find the same open interval set for each parameter. Specifically, for each θ i , there exists a δ i , such that if|θ ′ i -θ i | ≤ δ i , we have |p(π, ⃗ θ ′ ) -p(π, ⃗ θ)| ≤ ϵ i, where ϵ i → 0. Hence, we let δ = min{δ 1 , δ 2 } and ϵ = ϵ 1 + ϵ 2 . Thus the new representative environment set is the set that includes the any combination of{[θ 1 , θ 2 ]} where θ 1 ∈ { ⃗ θ i } L1 i=1 and θ 2 ∈ { ⃗ θ j } L2Table 1 : The teacher policies corresponding to the three approaches for UED. U (Θ) is a uniform distribution over environment parameter space, Dπ is a baseline distribution, θπ is the trajectory which maximizes regret of π, and v π is the value above the baseline distribution that π achieves on that trajectory, c π is the negative of the worst-case regret of π. Details are described in PAIRED [3] . UED Approaches Teacher Policy Decision Rule DR [19] Λ(π) = U (Θ) Randomly sample PARIED [3] Λ(π) = { θπ : cπ vπ , Dπ : otherwise} Minimax Regret SHED (ours)Λ(π) = arg max ⃗ θ∈Θ Q π (s = π, a = ⃗ θ) Maximize rewardopen parameter intervals combination. This set effectively characterizes the general performance capabilities of the student policy π.Similarly, we can show this still holds when the environment is constructed by a larger dimension environment parameters, where we set δ = min{δ i }, and ϵ = i ϵ i , and we have δ > 0, ϵ → 0. The overall logic is that we can find a finite set, which is called representative environment set, and we can use performances in this set to represent any performances in the environments generated in the representative parameter interval set, which is called representative performance vector set. Finally, we can show that representative parameter interval set fully covers the environment parameter space. Thus there exists a finite evaluation environment set that can capture the student's general capabilities and the performance vector, called representative performance vector set,[p 1 , . . . , p m ] is a good representation of the student policy.Here, we describe how to leverage the diffusion model to learn the conditional data distribution in the collected experiences τ = {(s u t , a u t , r u t , s u,′ t )}. Later we can use the trainable reverse chain in the diffusion model to generate the synthetic trajectories that can be used to help train the teacher agent, resulting in reducing the resource-intensive and time-consuming collection of upper-level teacher experiences. We deal with two different types of timesteps in this section: one for the diffusion process and the other for the upper-level teacher agent, respectively. We use subscripts k ∈ 1, . . . , K to represent diffusion timesteps and subscripts t ∈ 1, . . . , T to represent trajectory timesteps in the teacher's experience.In the image domain, the diffusion process is implemented across all pixel values of the image. In our setting, we diffuse over the next state s u,′ conditioned the given state s u and action a u . We construct our generative model according to the conditional diffusion process:q(s u,′ k |s u,′ k-1 ), p ϕ (s u,′ k-1 |s u,′ k , s u , a u )As usual, q(s u,′ k |s u,′ k-1 ) is the predefined forward noising process while p ϕ (s u,′ k-1 |s u,′ k , s u , a u ) is the trainable reverse denoising process. We begin by randomly sampling the collected experiences τ = {(s u t , a u t , r u t , s u,′ t )} from the real experience buffer B real .We drop the superscript u here for ease of explanation. Giving the observed state s and action a, we use the reverse process p ϕ to represent the generation of the next state s ′ :EQUATIONAt the end of the reverse chain, the sample s ′ 0 , is the generated next state s ′ . As shown in Section 2.2, p ϕ (s ′ k-1 |, s ′ k , s, a) could be modeled as a Gaussian distributionN (s ′ k-1 ; µ θ (s ′ k , s, a, k), Σ θ (s ′ k , s, a, k)). Similar to Ho et al. [6] , we parameterize p ϕ (s ′ k-1 |s ′ k , s, a) as a noise prediction model with the covariance matrix fixed asΣ θ (s ′ k , s, a, k) = β i I and mean is µ θ (s ′ i , s, a, k) = 1 √ α k s ′ k - β k √ 1 -ᾱk ϵ θ (s ′ k , s, a, k)Where ϵ θ (s ′ k , s, a, k) is the trainable denoising function, which aims to estimate the noise ϵ in the noisy input s ′ k at step k. Specifically, giving the sampled experience (s, a, s ′ ), we begin by sampling s ′ K ∼ N (0, I) and then proceed with the reverse diffusion chain p ϕ (s ′ k-1 |, s ′ k , s, a) for k = K, . . . , 1.The detailed expression for s ′ k-1 is as follows:EQUATIONwhere ϵ ∼ N (0, I). Note that ϵ = 0 when k = 1.Training objective. We employ a similar simplified objective, as proposed by Ho et al. [6] to train the conditional ϵmodel through the following process:EQUATIONWheres ′ k = √ ᾱk s ′ + √ 1 -ᾱk ϵ.U represents a uniform distribution over the discrete set {1, . . . , K}.The intuition for the loss function L(θ) tries to predict the noise ϵ ∼ N (0, I) at the denoising step k, and the diffusion model is essentially learning the student policy involution trajectories collected in the real experience buffer B reals . Note that the reverse process necessitates a substantial number of steps K, as the Gaussian assumption holds true primarily under the condition of the infinitesimally limit of small denoising steps [15] . Recent research by Xiao et al. [22] has demonstrated that enabling denoising with large steps can reduce the total number of denoising steps K. To expedite the relatively slow reverse sampling process outlined in Equation 3.2 (as it requires computing ϵ ϕ networks K times), we use a small value of K, while simultaneously setting β min = 0.1 and β max = 10.0.Similar to Wang et al. [20] , we define:β k = 1 -α k = 1 -exp β min × 1 K -0.5(β max -β min ) 2k -1 K 2This noise schedule is derived from the variance-preserving Stochastic Differential Equation by Song et al. [16] .Generate synthetic trajectories. Once the diffusion model has been trained, it can be used to generate synthetic experience data by starting with a draw from the prior s ′ K ∼ N (0, I) and successively generating denoised next state, conditioned on the given s and a through the reverse chain p ϕ in Equation 3.2. Note that the giving condition action a can either be randomly sampled from the action space (which is also the environment parameter space) or use another diffusion model to learn the action distribution giving the initial state s. In such case, this new diffusion model is essentially a behavior-cloning model that aims to learn the teacher policy Λ(a|s). This process is similar to the work of Wang et al. [20] . We discuss this process in detail in the appendix. In this paper, we randomly sample a as it is straightforward and can also increase the diversity in the generated synthetic experience to help train a more robust teacher agent.Once the diffusion model has been trained, it can be used to generate synthetic experience data by starting with a draw from the prior s ′ K ∼ N (0, I) and successively generating denoised next state, conditioned on the given s and a through the reverse chain p ϕ in Equation 3.2. Note that the giving condition action a can either be randomly sampled from the action space (which is also the environment parameter space) or we can train another diffusion model to learn the action distribution giving the initial state s, and then use the trained new diffusion model to sample the action a giving the state s. This process is similar to the work of Wang et al. [20] .In particular, We construct another conditional diffusion model as: As usual, q(a k |a k-1 ) is the predefined forward noising process while p ϕ (a k-1 |a k , s) is the trainable reverse denoising process. we represent the action generation process via the reverse chain of the conditional diffusion model asq(a k |a k-1 ), p ϕ (a k-1 |a k , s)EQUATIONAt the end of the reverse chain, the sample a 0 , is the generated action a for the giving state s.Similarly, we parameterize p ϕ (a k-1 |a k , s) as a noise prediction model with the covariance matrix fixed as Σ θ (a k , s, k) = β i I and mean isµ θ (a i , s, k) = 1 √ α k a k - β k √ 1 -ᾱk ϵ θ (a k , s, k)Similarly, the simplified loss function isEQUATIONWherea k = √ ᾱk a + √ 1 -ᾱk ϵ.U represents a uniform distribution over the discrete set {1, . . . , K}.The intuition for the loss function L a (θ) tries to predict the noise ϵ ∼ N (0, I) at the denoising step k, and the diffusion model is essentially a behavior cloning model to learn the student policy collected in the real experience buffer B reals .Once this new diffusion model is trained, the generation of the synthetic experience can be formulated as:• we first randomly sample the state from the collected real trajectories s ∼ τ ;• we use the new diffusion model discussed above to mimic the teacher's policy to generate the actions a;• giving the state s and action a, we use the first diffusion model presented in the main paper to generate the next state s ′ ;• we compute the reward r according to the reward function, and add the final generated synthetic experience (s, a, r, s ′ ) to the synthetic experience buffer B syn to help train the teacher agent. We begin by investigating SHED's ability to assist in collecting experiences for the upper-level MDP teacher. This involves the necessity for SHED to prove its ability to accurately generate synthetic experiences for teacher agents. To check the quality of these generated synthetic experiences, we employ a diffusion model to simulate some data for validation (even though Diffusion models have demonstrated remarkable success across vision and NLP tasks).We design the following experiment: given the teacher's observed states u = [p 1 , p 2 , p 3 , p 4 , p 5 ],where p i denotes the student performance on i-th evaluation environment. and given the teacher's action a u = [a 1 , a 2 , a 3 ], which is the environment parameters and are used to generate corresponding environment instances. We use a neural network f (s u , a u ) to mimic the involution trajectories of the student policy π. That is, with the input of the state s u and action a u into the neural network, it outputs the next observed states u,′ = [p ′ 1 , p ′ 2 , p ′ 3 , p ′ 4 , p ′ 5 ], indicating the updated student performance vector on the evaluation environments after training in the environment generated by a u . In particular, we add a noise ε into s u,′ to represent the uncertainty in the transition. We first train our diffusion model on the real dataset (s u , a u , s u,′ ) generated by neural network f (s u , a u ). We then set a fixed (s u , a u ) pair and input them into f (s u , a u ) to generate 200 samples of real s u,′ . The trained diffusion model is then used to generate 200 synthetic s u,′ conditioned on the fixed (s u , a u ) pair.The results are presented in Figure 6 , we can see that the generative model can effectively capture the distribution of real experience even if there is a large uncertainty in the transition, indicated by the value of ε. This provides evidence that the diffusion model can generate useful experiences conditioned on (s u , a u ). It is important to note that the marginal distribution derived from the reverse diffusion chain provides an implicit, expressive distribution, such distribution has the capability to capture complex distribution properties, including skewness and multi-modality.We further provide more results to show the ability of our generative model to generate synthetic trajectories where the noise is extremely small. In such cases, the actual next state s ′ will converge to a certain value, and the synthetic next state s syn,′ generated by the diffusion model should also be very close to that value, then the diffusion model has the ability to sample the next state s syn,′ 0 which can accurately represent the next state. We present the results in Figure 5 . Specifically, this figureshows when the noise is very small in the actual next state, which is 0.05 * ϵ, and ϵ ∼ N (0, 1). Giving any condition (s, a) pair, we selectively report on (s i , a i ), where x-axis is the a i value, and y-axis is the s i value. The student policy with initial performance vector s is trained on the environments generated by the teacher's action a. We report the new performance s ′ i of student policy on i-th environments after training in the z-axis. In particular, if two points s ′ i and s syn,′ i are close, it indicates that the diffusion model can successfully generate the actual next state. As we can see, when the noise is extremely small, our diffusion model can accurately predict the next state of s ′ i giving any condition (s, a) pair. We set the learning rate 1e-3 for actor, and 3e-3 for critic, we set gamma γ = 0.999, λ = 0.95, and set coefficient for the entropy bonus (to encourage exploration) as 0.01. For each environment, we conduct 50 PPO updates for the student agent, and We can train on up to 50 environments, including replay. For our diffusion model, the diffusion discount is 0.99, and batch size is 64, τ is 0.005, learning rate is 3e -4. The synthetic buffer size is 1000, and the ratio is 0.25.All the models were trained on a single NVIDIA GeForce RTX 3090 GPU and 16 CPUs.Here we provide the document shows the instruction to generate feasible maze environments.There are several factors that can affect the difficulty of a maze. Here are some key factors to consider:1. Maze Size: Larger mazes generally increase the complexity and difficulty as the agent has more states to explore. Typically, the maze size should be larger than 4x4 and smaller than 15*15.-If the size is 7*7 or smaller, the maze size is considered easy.-If the size is larger than 7*7 but smaller than 10*10, the maze size is considered medium.-If the maze size is larger than 10x10 but smaller than 15*15, the maze size is considered hard.2. Maze Structure: The complexity of the paths, including the number of twists, turns, and dead-ends, can significantly impact navigation strategies. The presence of narrow corridors versus wide-open spaces also plays a role.-If there are fewer than 2 turns in the feasible path from the start position to the end position, the maze structure is considered easy.-If there are more than 2 turns but fewer than 4 turns in the path from the start position to the end position, the maze structure is considered medium.-If there are 4 or more turns in the path from the start position to the end position, the maze structure is considered hard.We provide our prompt for the Retrieval Augmented Generation as follows:Please refer to the document, and generate a maze with feasible path. The difficulty level for the maze size is {maze_size_level}, and the difficulty level for the maze structure is {maze_structure_level}, he difficulty level for the goal location is {goal_location_level}, he difficulty level for the start location is {start_position_level}.We also provide ablation analysis to evaluate the impact of different design choices in Lunar lander domain, including (a) a larger evaluation environment set; (b) a bigger budget for constraint on the number of generated environments (which incurs a longer training time horizon). The results are reported in Figure 7 .We explore the impact of introducing the diffusion model in collecting synthetic teacher's experience and varying the size of the evaluation environment set. Specifically, as we can see from the right side of Figure 7 , the SHED consistently outperforms h-MDP, indicating the effectiveness of introducing the generative model to help train the upper-level teacher policy. Furthermore, we find that when increasing the size of the evaluation environment set, we can have a better result in the student transfer performances. The intuition is that a larger evaluation environment set, encompassing a more diverse range of environments, provides a better approximation of the student policy according to the Theorem 1. However, the reason why SHED with 30 evaluation environments slightly outperforms SHED with 40 evaluation environments is perhaps attributed to the increase in the dimension of the student performance vector, which amplifies the challenge of training an effective diffusion model with a limited dataset.We conduct experiments in Lunar lander under a longer time horizon. The results are provided on the right side of Figure 7 . As we can see, our proposed algorithm SHED can efficiently train the student agent to achieve the general capability in a shorter time horizon, This observation indicates that our proposed environment generation process can better generate the suitable environments for the current student policy, thereby enhancing its general capability, especially when there is a constraint on the number of generated environments. The limitation of this work comes from the UED framework, as UED is limited to the use of parameterized environments. This results in our experimental domain being relatively simple. Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We confirm that the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics, and all the authors preserve anonymity.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.They approximate the regret value by the Generalized Advantage Estimate[12]. 


Decision: This paper introduces a new environment design method. 

The reviewers were concerned both about the clarity of the writing, exaggerated claims, and a lack of experimental evaluations. 
Some of this was addressed during the rebuttal process, resulting in one reviewer increasing their score. However, other concerns were not addressed, for example the lack of ablations and the fact that the paper makes many changes at once.

While the paper is currently not ready for publication I believe that it will be a good contribution once all of the concerns from the reviewers have been addressed in a future version.

Reviews: review: paper_summary: paper_summary: The authors of this paper use hierarchical MDP formulation and a teacher agent trained by RL to perform curriculum learning. To address the sparse data available for the teacher agent, this paper uses diffusion models to synthesize datasets for training. This paper performs experiments on lunar lander and bipedal walker environments to validate their claim.
strength_weakness: strength_weakness: - This paper designs the teacher agent via hierarchical MDP to model the learning process of the student agent to perform curricular RL. However, Fingerprint Policy Optimization (Paul et al, 2019) also has a similar idea of modeling the learning process of the student agent. It would be interesting to explain more about how this paper's idea is related and contributes to this line of thought.

- A fully trained algorithm on the BipedalWalker should approach a cumulative reward of 300. Even the modified version used in the ACCEL paper is measured on a scale of 0 out of 300. However, from Figure 3, it appears all baselines perform less than 50 on the BipedalWalker benchmark. It is questionable whether all baselines were fully trained with the right settings. Also, the performance of the proposed algorithm and those of the baselines are statistically too similar to see whether SHED improves over the baselines in Lunar Lander and BipedalWalker benchmarks. Finally, other than the version of ACCEL in this paper not performing as well as the ACCEL in the original paper, I am curious whether ACCEL can be considered state-of-the-art in the benchmarks as written in line 324. Genetic Curriculum (Song et al, 2022) reports higher cumulative reward on the BipedalWalkerHardcore environment.

- Figure 4 has no scale on timestep and reward.
questions: questions: - To improve the scores on the review, I hope to see the authors explain how their algorithm works in relation to existing works listed above. I also hope to see clarifications regarding the experiment results stated above as well.
limitations: limitations: The authors has addressed the limitations of this paper.
paper_summary: paper_summary: This paper considers the Unsupervised Environment Design problem, where a teacher agent seeks to design environments to train a student. Methods such as PLR, PAIRED and ACCEL have recently shown promising performance for random, RL and evolutionary generators. This paper proposes a handful of modifications, using RL with a different objective vs. PAIRED (performance on held out set vs. regret) and also proposes to add synthetic data to accelerate the RL process.
strength_weakness: strength_weakness: * There appear to be two confounding features of the method, the new objective for PAIRED and then the synthetic data. Why do they make sense to combine in this way? It just feels like the authors tried to do "enough for a paper" rather than contribute something meaningful that people can build on. I say this because its unclear how these two independent features interact with other existing algorithms. Maybe we should just do ACCEL with synthetic data for instance? Did the authors try that? If it is in the Appendix already and I missed it then I will increase my score.
* The performance gains are fairly minor, and presented in an unclear fashion with just a bunch of curves on a single plot. Can we get some more rigorous analysis for example using the recommendations from Agarwal et al, "Deep Reinforcement Learning at the Edge of the Statistical Precipice"?
* The Maze experiment seems to have many inductive biases and seems distinct from the diffusion based approach for BipedalWalker and LunarLander. What happens if ACCEL has access to ChatGPT as an editor and then uses replay? This seems like a simpler extension that alone could be a strong paper - although it would resemble ELM (Lehman et al 2022) so it wouldn't be particularly novel.
* The related work is very light. This is disappointing since the paper builds on so many related areas, such as synthetic data, diffusion models, UED, language models for evolution, procedural content generation etc.
questions: questions: * The authors say "The hierarchical framework can incorporate techniques from existing UED works, such as prioritized level replay". Why did you not do this? Then it would be much clearer to see if it is state of the art.
limitations: limitations: Covered in the Appendix.
paper_summary: paper_summary: The paper presents a novel approach to Unsupervised Environment Design (UED) that addresses the challenges of efficiency by introducing a hierarchical MDP framework and using synthetic data. This framework involves an upper-level RL teacher agent that generates training environments tailored to a lower-level student agent's capabilities. The paper proposes the Synthetically-enhanced Hierarchical Environment Design (SHED) method, which uses generative modeling to create synthetic trajectory datasets, thereby reducing the resource-intensive interactions between agents and environments. The effectiveness of SHED is demonstrated through empirical experiments across various domains, showing superior performance compared to existing UED methods.
strength_weakness: strength_weakness: - The proposed method introduces significant complexity, particularly in the implementation of the hierarchical MDP and the generative modeling components. This might limit the accessibility and reproducibility of the approach.
- While the empirical results are promising, the evaluation is limited to a few specific domains. It would be beneficial to see broader applicability across more diverse and complex environments.
- Figure 4 is not properly formatted (no values on the axes).
questions: questions: - Why were diffusion models chosen over other generative models (e.g., GANs, VAEs)? Have other models been considered or tested?
- How well does the proposed method scale to real-world applications with significantly higher complexity and variability? Have any tests been conducted in such settings?
limitations: limitations: The limitations are discussed in Appendix F.1 but I think the authors should discuss the limitations in the main paper.
paper_summary: paper_summary: After rebuttal

While I think there are still some problems with this paper, e.g. the short training duration, and the slight exaggeration of claims (that SHED outperforms UED). I think, however, that the idea is nice, and getting RL environment design to work better is a good goal.


-----


This paper aims to improve Unsupervised Environment Design in two ways.
First, it introduces a hierarchical MDP formulation, where the top level corresponds to the teacher, and the lower level corresponds to the learning agent. Each transition in the top-level MDP involves training the lower level agent on generated levels. Related to this, they develop a state representation for the adversary, which is the performance of the agent on a fixed set of diverse levels.

Separately to this, they use a diffusion model to upsample the number of experiences for the teacher, effectively training on synthetic data, to improve sample efficiency.
strength_weakness: strength_weakness: - Major
	- The results do not very convincingly demonstrate that SHED is better than current SoTA. Looking at figure 3 particularly, I would say that ACCEL has about the same performance as SHED. However, comparing against the RL baseline, SHED does do better.
	- The method is limited in the types of environments it can generate. For instance, mazes are generated using an LLM instead of directly placing blocks. This method therefore is not quite as broad in scope as PAIRED or ACCEL, which can generate arbitrary environments.
	- Relatedly, in the minigrid experiments, do all methods generate the levels in the same way using an LLM, providing the difficulty numbers? It would be good to compare this against the standard version of ACCEL that directly places blocks in the maze level, as it does not have the same restriction as SHED.
- Minor
	- The figures can be improved:
		- Make the alpha value on the error bars a bit less
		- Keep colours consistent across figures, so the same method has the same colour
		- Keep capitalisation consistent across the figure labels.
	- line 80, the period ends on the line after the maths, it should end on the same line.
	- Footnote 1: Jiang et al. (2021) use (amongst others) the positive value loss, which is not quite the GAE, as it clips it at zero before summing.
	- equation one, you use $\beta_t$ but $t$ does not seem to be defined? Should this be $\beta_k$?
	- Line 159, PARIED should be PAIRED
	- There is no reward scale in figure 4
	- Figure 9's caption can be made clearer. I understand it to be the performance of each method in different testing environments. 
	- Line 718 does not link to a figure.
	- Figure 11's caption: zero-shot and not zeros-shot
	- Capitalise the first word in the title of appendix C.2
	- In terms of notation, in line 96, $\pi^*$ usually has a dependence on $\theta$ (e.g. $\pi^*_\theta$) to indicate it is optimal w.r.t. that particular level.
	- Line 217, maybe add a citation to the first sentence, as I thought that is what you do, which confused me for a second.
	- line 237 space after period.
	- Line 241 "given" instead of giving?
	- Lines 296 - 297 are a bit confusing, as the word environment is used three times.
	- The assumption in theorem 1 is pretty strong.
questions: questions: - It seems to me that the upsampling is very similar to [1], could you comment on that/cite the paper if indeed the technique is similar?
- Have you tried/have any intuition of what the effect would be if you set $\eta=0$?
- Furthermore, it seems like the specific evaluation environments you use can have a large impact. Have you looked at how variable results are when using different (still randomly generated) environment parameters. Relatedly, what happens if you just generate a few DR levels *without* explicitly discretising the environment parameters? 
- Are you using the high entropy version of PAIRED introduced by [2]?
- Do you have an indication of how easy it would be to generate larger environments, with much larger parameter vectors?
- Do you think it would be possible to incorporate the multi-step generation of PAIRED into this process? E.g., still generate a maze step-by-step but also have the H-MDP structure and policy performance vector?
- Why can't you sample the action in the diffusion process from the teacher's policy?
- A reward of 50 in bipedal walker seems very low, could you compare against other papers / explain why it is low? E.g. [2] had rewards around 150


[1] Lu, Cong, et al. "Synthetic experience replay." _Advances in Neural Information Processing Systems_ 36 (2024).

[2] Mediratta, Ishita, et al. "Stabilizing unsupervised environment design with a learned adversary." _Conference on Lifelong Learning Agents_. PMLR, 2023.
limitations: limitations: I think the authors can list a few more limitations. 
Primarily, the restriction on the type of environment that can be generated, i.e., it needs numerical parameters, and generating a maze outright is challenging. This is quite a large difference to prior settings.

Comments: title: Rebuttal by authors
Rebuttal by Authors
Concerns remain
Rebuttal by authors (2)
Rebuttal by authors (1)
Rebuttal by Authors
Discussion
comment: Thank you for taking the time to read our response. We’re glad that some of your concerns have been addressed. If there are any remaining issues or anything that still needs clarification, please let us know.
Thanks for all of your effort in the rebuttal. I have updated my score to 6, on the condition that all of the fixes/explanations are indeed added to the revised paper, and that the paper's claims be slightly softened. I don't think you outperform state of the art UED methods; however, you are competitive with them, and you greatly improve the performance of RL-based methods.
Thank the reviewer for the feedback. We understand your concerns and appreciate the opportunity to consider our original motivations.

Our primary motivation, as stated in the abstract and introduction, is to train a student agent with general capabilities more efficiently under resource constraints, specifically with a limited number of generated environments and a limited training horizon. Previous UED papers rely on open-ended training, requiring thousands of generated environments and billions of environment interactions. However, this approach is unrealistic in real-world scenarios due to the limited resources available to construct environments.

To address this challenge, we focus on restricting the number of generated environments (50 environments in our settings) and interaction steps (millions of interactions) to train the student agent to achieve general capabilities. This necessitates designing a teacher agent that can generate suitable environments tailored to the current student's capability level, as opposed to previous methods that relied on random environment generation. Our hierarchical framework addresses this by having the teacher take an approximation of the student policy as the observed state and output environment parameters to generate tailored environments.

However, training the teacher agent is challenging due to the costly collection of the teacher's experience, which requires extensive environment interactions. This conflicts with our original motivation. To mitigate this issue, we propose using a diffusion model to upsample the real collected teacher's experience, reducing the costly interactions needed to gather the teacher's experience.

Additionally, our hierarchical framework can incorporate techniques from existing UED works, such as prioritized level replay. Our proposed module is designed to be decoupled from other techniques. For example, if we have a larger budget for generated environments, we can implement a level buffer to store environments with high learning potential and revisit them in future training.

We hope this explanation clarifies our motivation.
Thank you for your detailed response! I'll update the scores accordingly.
The issue with this work is what I said in the original review - it is not clear what the motivation is, there are a million different things being compared and no clear take away from the work. The rebuttal hasn't changed anything for me.
**Q6.** Could you run a baseline comparing how well normal ACCEL does that places the blocks directly?

**A6.** Yes, we conducted an experiment using the maze generation method in ACCEL, which involves placing blocks directly to generate the maze. The maze generation process is as follows:

1. We randomly sample the maze size.
2. We randomly place the start and end positions.
3. We randomly sample the number of blocks.
4. We randomly place the blocks, ensuring that the blocks, start position, and end position are not the same.
Here are the rough results:

- Randomly sample maze width and height from {8, 9, 10, 11, 12, 13, 14, 15}, randomly sample the number of blocks from [20, 40]:
Results: ACCEL: -9.8

- Fix maze size at 13*13, randomly sample the number of blocks from [20, 60]:
Results: ACCEL: -12.7

- Fix the maze size at 13x13, and fix the number of blocks at 60:
Results: ACCEL: -15.3

For comparison, here are the ACCEL performances under our maze generation method:
ACCEL: -4.98

Additionally, we added a filter function to exclude mazes without a feasible solution until generate a feasible maze, and the results are:

- Randomly sample maze width and height from {8, 9, 10, 11, 12, 13, 14, 15}, randomly sample the number of blocks from [20, 40]:
Results: ACCEL: -5.3

- Fix maze size at 13x13, randomly sample the number of blocks from [20, 60]:
Results: ACCEL: -8.5

- Fix maze size at 13x13, fix the number of blocks at 60:
Results: ACCEL: -7.8


**Q7.** please rectify all of the minor points in your revised manuscript.

**A7.** Thank you for pointing out the minor issues. We will revise the paper to address these points accordingly.
Thank you for your interest in our work and your prompt response. We have done additional experiments to address your concerns.

**Q1.** SHED performs about the same as ACCEL on lunar lander and bipedalwalker. Shed does slightly better in maze, but that is using a nonstandard way of generating mazes.

**A1.** We conducted experiments in the LunarLander environment to exclude very challenging test environments when testing zero-shot transfer performance. This is because all approaches perform poorly in these challenging environments, which would narrow the performance gap among them. Because we cannot directly show detailed results, we roughly report the results: After removing the challenging environments and completing 1,000 PPO updates, the average performance of SHED, h-MDP, and ACCEL improved significantly on LunarLander. However, ACCEL remains only slightly weaker than SHED. Here are the results: SHED: -6.019, h-MDP: -12.92, ACCEL: -8.706, Random: -30.65, PAIRED: -126.2.

It is important to note that the training curves of ACCEL and Random show significant variance and fluctuations, while the learning curve of SHED is much smoother. This indicates that SHED can consistently provide suitable environments, enhancing robustness and stability in training.


**Q2.** Sampling actions from the teacher's policy during the diffusion process is feasible. Why did you not do this?

**A2.**  First, we provide detailed steps for sampling actions from the teacher's policy during the diffusion process in Appendix B.2 to upsample the teacher's experiences. However, we chose to randomly sample actions to generate synthetic trajectories because it is more straightforward, which is the method we used in our work. Second, using a diffusion model directly as a policy to obtain the teacher's actions would require further modifications to train the diffusion model. This approach is explored in works by Wang et al. (citation number 20) and Janner, Michael, et al. [1]. However, incorporating this into our work would significantly increase its complexity, and therefore, we leave this direction for future work.
[1] Janner, Michael, et al. "Planning with diffusion for flexible behavior synthesis.". International Conference on Machine Learning. PMLR, 2022.

**Q3.** Could you at least run evaluation on the same levels that ACCEL used? What is the computational demand of running Bipedal Walker for 30K PPO updates? (or at least something like 5k-10k, 50 doesn't seem enough to learn much)

**A3.** As reported in the paper, all models were trained on a single NVIDIA GeForce RTX 3090 GPU and 16 CPUs. Training ACCEL for 30K PPO updates can take approximately 5 days, which is quite long. There seems to be a misunderstanding: we trained on 50 environments, where each environment ran for 4 epochs, and each epoch included 5 PPO minibatches, resulting in a total of 20 PPO updates per environment. Across all 50 environments, there are 1,000 PPO updates. Also, We included some of the levels that ACCEL used, such as SmallCorridor and FourRooms. We will include more levels in the final version.

**Q4.** what details are lacking?

**A4.** Their work is generally well-explained. One difference between their work and ours is that they lack a more detailed technical illustration of how actions and next states are generated using the diffusion model.


**Q5.** The SynthER method is a way to upsample data for RL agents, and your teacher is still an RL agent. So I would still argue that it is very similar to what you are doing?

**A5.** SynthER utilizes a diffusion model to generate synthetic experiences from a limited dataset, which is similar to our approach. This work shares similar motivations with ours. However, the key difference is that SynthER focuses on directly synthesizing the experience dataset for the student agent and can be used for online/offline training. In contrast, our approach involves simultaneously upsampling the upper-level teacher's experiences to further assist in training the upper-level teacher agent efficiently, addressing drawbacks in our proposed hierarchical framework. Additionally, directly synthesizing student trajectories to help train the student agents would be unfair in our comparison, as it might induce extra training in terms of the number of PPO updates compared to other approaches. Overall, while we acknowledge that their work and ours share similar motivations and technologies, the application details are different, as we use synthetic trajectories in the upper-level teacher's experience within our hierarchical framework. We will include their work in related work and provide a discussion.
Thank you for your prompt response. 

**Q1.** Would it be OK to rephase the sentence as following? : ACCEL was trained with 1 billion timesteps to interact with the environment and PPO policy was updated 30,000 times. SHED interacted with the environment with 50 environments and had 1,000 PPO updates to the policy.

**A1.** Yes, that is true. We will revise the sentence accordingly to improve clarity.

**Q2.** If the above is true, how many steps did SHED spent interacting with the environment in total for the training?

**A2.** SHED interacted with 50 environments. Each environment ran for 4 epochs, and each epoch included 5 PPO minibatches, resulting in a total of 20 PPO updates per environment. Across all 50 environments, this amounted to 1,000 PPO updates. In the LunarLander environment, there were around 1,000,000 total environment steps, while the BipedalWalker environment and Maze environment involved around 10,000,000 and 400,000 total environment steps across all environments, respectively.

**Q3.**  Why is SHED's interaction measured in number of environments not in steps.

**A3.** ACCEL was trained on thousands of environments. We would like to emphasize the motivation of our work is to train a student agent with general capabilities more efficiently under resource constraints, i.e., with a limited number of generated environments. Please note that the number of generated environments is crucial for improving the agent's general capabilities as we only "shallowly" train the agent on each generated environment. Therefore, we use the number of environments as a metric, which provides a clearer understanding of the training conditions. Additionally, we report results in terms of the number of PPO updates to ensure a fair comparison across all approaches with the same number of updates, consistent with other works like PAIRED and ACCEL.

**Q4.** When you say 50 environments, are we referring to 50 different obstacle courses in BipedalWalker?

**A4.** This refers to 50 different instances of the environment that the student agent will be trained in to acquire the general capabilities.


**Q5.** What are the cases that there would be concerns about how many obstacle courses you load for training, instead of how many timesteps the agents spent interacting with the world?

**A5.** In general, these two terms are correlated. In this work, we focus on training a student agent with general capabilities more efficiently under resource constraints. The number, diversity, and complexity of the environments are crucial for improving the agent's general capabilities as we only "shallowly" train the agent on each environment. In such cases, the number of environments would be a concern. 

**Q6.** Why does figure 3 report only 1000 training steps when the baselines were trained with a scale of a billion steps?

**A6.** There is a misunderstanding, the 1000 training steps refer to the number of PPO updates, not the environment steps. All training algorithms for UED (PAIRED, PLR, ACCEL, and many others) report results in terms of the number of PPO updates. We also do the same in this paper.
Thank you for your response! I believe the Q1 and Q3 are now addressed.

However, I'm sorry if I missed the points the authors have already made, but I have some follow up questions regarding the answers to Q2.

"30k PPO updates (1b step), while in our setting, due to the limited generated environment and training horizon, for example, we use only about 1k PPO updates (50 environments) to train the student agent."
- Would it be OK to rephase the sentence as following? : ACCEL was trained with 1 billion timesteps to interact with the environment and PPO policy was updated 30,000 times. SHED interacted with the environment with 50 environments and had 1,000 PPO updates to the policy.

- If the above is true, how many steps did SHED spent interacting with the environment in total for the training? 

- Why is SHED's interaction measured in number of environments not in steps

- When you say 50 environments, are we referring to 50 different obstacle courses in BipedalWalker?

- What are the cases that there would be concerns about how many obstacle courses you load for training, instead of how many timesteps the agents spent interacting with the world?

- Why does figure 3 report only 1000 training steps when the baselines were trained with a scale of billion steps?
Thank you for your detailed response to my review. I will update my score accordingly.
Thank you for your response and your additional experiments.
I have some follow up questions and points to discuss

> A1. It is crucial to note that SHED demonstrates more consistent and superior performance in the more complex BipedalWalker and Maze, as depicted in Figure 3,4. This suggests that SHED offers better generalizability and robustness across different environments. Moreover, the less variability in performance with SHED underscores its stability and reliability, which are essential qualities in real-world applications where consistent performance is critical.


I am not sure I am convinced. SHED performs about the same as ACCEL on lunar lander and bipedalwalker. Shed does slightly better in maze, but that is using a nonstandard way of generating mazes.

> A7. Yes, sampling actions from the teacher's policy during the diffusion process is feasible.

Then why did you not do this?

> A8. Please refer to A2 to Reviewer AEHi

Could you at least run evaluation on the same levels that ACCEL used? What is the computational demand of running Bipedal Walker for 30K PPO updates? (or at least something like 5k-10k, 50 doesn't seem enough to learn much)

> A3

Thanks for that, it is interesting.

> A1. Thank you for pointing out this paper, which shares some similar motivations with our work. However, there are some key differences to note:
> While [1] also adopts a diffusion model to generate trajectories, it lacks specific technical details on how these trajectories are generated. In contrast, our paper thoroughly describes the generation process.

To me the paper seems pretty clear, what details are lacking?

> Additionally, [1] focuses on upsampling student trajectories, whereas our proposed hierarchical MDP framework emphasizes the challenge of collecting the teacher's experience. Our method specifically upsamples the teacher's experience to improve training efficiency for the teacher.

The SynthER method is a way to upsample data for RL agents, and your teacher is still an RL agent. So I would still argue that it is very similar to what you are doing?


> A2. In the maze domain, learning block placement is challenging due to the sequential decision process involved

Could you run a baseline comparing how well normal ACCEL does that places the blocks directly?

Also please rectify all of the minor points in your revised manuscript.