Title: Enhancing the Hierarchical Environment Design via Generative Trajectory Modeling
Venue: NeurIPS 2024
Accepted: no
Keywords: hierarchical MDP

environment design

general capability

generative model
Abstract: Unsupervised Environment Design (UED) is a paradigm that automatically generates a curriculum of training environments, enabling agents trained in these environments to develop general capabilities, i.e., achieving good zero-shot transfer performance. However, existing UED approaches focus primarily on the random generation of environments for open-ended agent training. This is impractical in resource-limited scenarios where there is a constraint on the number of environments that can be generated. In this paper, we introduce a hierarchical MDP framework for environment design under resource constraints. It consists of an upper-level RL teacher agent that generates suitable training environments for a lower-level student agent. The RL teacher can leverage previously discovered environment structures and generate environments at the frontier of the student's capabilities by observing the student policy's representation. Additionally, to alleviate the time-consuming process of collecting the experience of the upper-level teacher, we utilize recent advances in generative modeling to synthesize a trajectory dataset for training the teacher agent. Our method significantly reduces the resource-intensive interactions between agents and environments, and empirical experiments across various domains demonstrate the effectiveness of our approach.

Abstract

Unsupervised Environment Design (UED) is a paradigm that automatically generates a curriculum of training environments, enabling agents trained in these environments to develop general capabilities, i.e., achieving good zero-shot transfer performance. However, existing UED approaches focus primarily on the random generation of environments for open-ended agent training. This is impractical in resource-limited scenarios where there is a constraint on the number of environments that can be generated. In this paper, we introduce a hierarchical MDP framework for environment design under resource constraints. It consists of an upper-level RL teacher agent that generates suitable training environments for a lower-level student agent. The RL teacher can leverage previously discovered environment structures and generate environments at the frontier of the student's capabilities by observing the student policy's representation. Additionally, to alleviate the time-consuming process of collecting the experience of the upper-level teacher, we utilize recent advances in generative modeling to synthesize a trajectory dataset for training the teacher agent. Our method significantly reduces the resource-intensive interactions between agents and environments, and empirical experiments across various domains demonstrate the effectiveness of our approach.

1 Introduction

The advances of reinforcement learning (RL) [17] have promoted research into the problem of training autonomous agents that are capable of accomplishing complex tasks. One interesting, yet underexplored, area is training agents to perform well in unseen environments, a concept referred to as zero-shot transfer performance. To this end, Unsupervised Environment Design (UED) [3] has emerged as a promising paradigm to address this problem. The objective of UED is to automatically generate environments in a curriculum-based manner, and training agents in these sequentially generated environments can equip agents with general capabilities, enabling agents to learn robust and adaptive behaviors that can be transferred to new scenarios without explicit exposure during training.

Existing approaches in UED primarily focus on building an adaptive curriculum for the environment 28 generation process to train the generally capable agent. Dennis et al. [3] formalize the problem of 29 finding adaptive curricula through a game involving an adversarial environment generator (teacher 30 agent), an antagonist agent (expert agent), and the protagonist agent (student agent). The RL-based 31 teacher is designed to generate environments that maximize regret, defined as the difference between 32 the protagonist and antagonist agent's expected rewards. They show that these agents will reach 33 a Nash Equilibrium where the student agent learns the minimax regret policy. However, since the 34 teacher agent adapts solely based on the regret feedback, it is inherently difficult to adapt to student 35

policy changes. Meanwhile, training such an RL-based teacher remains a challenge because of the 36 high computational cost of training an expert antagonist agent for each environment. 37

In contrast, domain randomization [19] based approaches circumvent the overhead of developing 38 an RL teacher by training agents in randomly generated environments, resulting in good empirical 39 performances. Building upon this, Jiang et al. [7] introduce an emergent curriculum by sampling 40 randomly generated environments with high regret value 1 to train the agent. Parker-Holder et al. 41 [10] then propose the adaptive curricula by manually designing a principled, regret-based curriculum, 42 which involves generating random environments with increasing complexity. While these domain 43 randomization-based algorithms have demonstrated good zero-shot transfer performance, they face 44 limitations in efficiently exploring large environment design spaces and exploiting the inherent 45 structure of previously discovered environments. Moreover, existing UED approaches typically 46 rely on open-ended learning, necessitating a long training horizon, which is unrealistic in the real 47 world due to resource constraints. Our goal is to develop a teacher policy capable of generating 48 environments that are perfectly matched to the current skill levels of student agents, thereby allowing 49 students to achieve optimal general capability within a strict budget for the number of environments 50 generated and within a shorter training time horizon. 51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

In this paper, we address these challenges by introducing a novel, adaptive environment design framework. The core idea involves using a hierarchical Markov Decision Process (MDP) to simultaneously formulate the evolution of an upper-level teacher agent, tasked with generating suitable environments to train the lower-level student agent to achieve general capabilities. To accurately guide the generation of environments at the frontier of the student agent's current capabilities, we propose approximating the student agent's policy/capability by its performances across a set of diverse evaluation environments, which acts as the state abstraction for the teacher's decision-making process. The transitions in the teacher's state represent the trajectories of the student agent's capability after training in the generated environment. However, collecting experience for the upper-level teacher agent is slow and resource-intensive, since each upper-level MDP transition evolves a complete training cycle of the student agent on the generated environment. To accelerate the collection of upper-level MDP experiences, we utilize advances in diffusion models that can generate new data points capturing complex distribution properties, such as skewness and multi-modality, exhibited in the collected dataset [11]. Specifically, we employ diffusion probabilistic model [15, 6] to learn the evolution trajectory of student policy/capability and generate synthetic experiences to enhance the training efficiency of the teacher agent. Our method, called S ynthetically-enhanced H ierarchical E nvironment D esign ( SHED ), automatically generates increasingly complex environments suited to the current capabilities of student agents.

70

71

72

73

74

75

76

77

In summary, we make the following contributions:

We develop a novel hierarchical MDP framework for UED that introduces a straightforward method to represent the current capability level of the student agent.

We introduce SHED , which utilizes diffusion-based techniques to generate synthetic experiences. This method can accelerate the training of the off-policy teacher agent.

We demonstrate that our method outperforms existing UED approaches (i.e., achieving a better general capability under resource constraints) in different task domains.

2 Preliminaries

In this section, we provide an overview of two main research areas upon which our work is based. 78

2.1 Unsupervised Environment Design 79

The objective of UED is to generate a sequence of environments that effectively train the student agent to achieve a general capability. Dennis et al. [3] first model UED with an Underspecified Partially Observable Markov Decision Process (UPOMDP), which is a tuple

. The UPOMDP has a set Θ representing the free parameters of the environments, which are 80 determined by the teacher agent and can be distinct to generate the next new environment. Further, 81 these parameters are incorporated into the environment-dependent transition function P M : S × A × 82 Θ → S . Here A represents the set of actions, S is the set of states. Similarly, I M : S → O is the 83 environment-dependent observation function, R M is the reward function, and γ is the discount factor. 84 Specifically, given the environment parameters ⃗ θ ∈ Θ , we denote the corresponding environment 85 instance as M ⃗ θ . The student policy π is trained to maximize the cumulative rewards V M θ ⃗ ( π ) = 86 ∑ T t =0 γ t r t in the given environment M ⃗ θ under a time horizon T , and r t are the collected rewards 87 in M ⃗ θ . Existing works on UED consist of two main strands: the RL-based environment generation 88 approach and the domain randomization-based environment generation approach. 89

90

91

92

93

94

95

96

The RL-based generation approach was first formalized by Dennis et al. [3] as a self-supervised RL paradigm for generating environments. This approach involves co-evolving an environment generator policy (teacher) with an agent policy π (student), where the teacher's role is to generate environment instances that best support the student agent's continual learning. The teacher is trained to produce challenging yet solvable environments that maximize the regret measure, which is defined as the performance difference between the current student agent and a well-trained expert agent π ∗ within the current environment: Regret M θ ⃗ ( π, π ∗ ) = V M θ ⃗ ( π ∗ ) -V M θ ⃗ ( π ) .

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

The domain randomization-based generation approach, on the other hand, involves randomly generating environments. Jiang et al. [7] propose to collect encountered environments with high learning potentials, which are approximated by the Generalized Advantage Estimation (GAE) [12], and then the student agent can selectively train in these environments, resulting in an emergent curriculum of increasing difficulty. Additionally, Parker-Holder et al. [10] adopt a different strategy by using predetermined starting points for the environment generation process and gradually increasing complexity. They manually divide the environment design space into different difficulty levels and employ human-defined edits to generate similar environments with high learning potentials. Their algorithm, ACCEL, is currently the state-of-the-art (SOTA) in the field, and we use an edited version of ACCEL as a baseline in our experiments.

2.2 Diffusion Probabilistic Models

Diffusion models [15] are a specific type of generative model that learns the data distribution. Recent advances in diffusion-based models, including Langevin dynamics and score-based generative models, have shown promising results in various applications, such as time series forecasting [18], robust learning [9], anomaly detection [21] as well as synthesizing high-quality images from text descriptions [8, 11]. These models can be trained using standard optimization techniques, such as stochastic gradient descent, making them highly scalable and easy to implement.

In a diffusion probabilistic model, we assume a d -dimensional random variable x 0 ∈ R d with an unknown distribution q ( x 0 ) . Diffusion Probabilistic model involves two Markov chains: a predefined forward chain q ( x k | x k -1 ) that perturbs data to noise, and a trainable reverse chain p ϕ ( x k -1 | x k ) that converts noise back to data. The forward chain is typically designed to transform any data distribution into a simple prior distribution (e.g., standard Gaussian) by considering perturb data with Gaussian noise of zero mean and a fixed variance schedule { β k } K k =1 for K steps:

where k ∈ { 1 , . . . , K } , and 0 < β 1: K < 1 denote the noise scale scheduling. As K → ∞ , x K will converge to isometric Gaussian noise: x K → N (0 , I ) . According to the rule of the sum of normally distributed random variables, the choice of Gaussian noise provides a closed-form solution to generate arbitrary time-step x k through:

Here α k = 1 -β k and ¯ α k = ∏ k s =1 α s . The reverse chain p ϕ ( x k -1 | x k ) reverses the forward process by learning transition kernels parameterized by deep neural networks. Specifically, considering the Markov chain parameterized by ϕ , denoising arbitrary Gaussian noise into clean data samples can be written as:

It uses the Gaussian form p ϕ ( x k -1 | x k ) because the reverse process has the identical function form as the forward process when β t is small [15]. Ho et al. [6] consider the following parameterization of

p ϕ ( x k -1 | x k ) : 130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

ϵ ϕ is a trainable function to predict the noise vector ϵ from x k . Ho et al. [6] show that training the reverse chain to maximize the log-likelihood ∫ q ( x 0 ) log p ϕ ( x 0 ) dx 0 is equivalent to minimizing re-weighted evidence lower bound (ELBO) that fits the noise. They derive the final simplified optimization objective:

Once the model is trained, new data points can be subsequently generated by first sampling a random vector from the prior distribution, followed by ancestral sampling through the reverse Markov chain in Equation 3.

3 Approach

In this section, we formally describe our method, S ynthetically-enhanced H ierarchical E nvironment D esign ( SHED ), which is a novel framework for UED under resource constraints. The SHED incorporates two key components that differentiate it from existing UED approaches:

A hierarchical MDP framework to generate suitable environments,

A generative model to generate the synthetic trajectories.

SHED uses a hierarchical MDP framework where an RL teacher leverages the observed student's policy representation to generate environments at the student's capabilities frontier. Such targeted environment generation process enhances the student's general capability by utilizing the underlying structure of previously discovered environments, rather than relying on the open-ended random generation. Besides, SHED leverages advances in generative models to generate synthetic trajectories that can be used to train the off-policy teacher agent, which significantly reduces the costly interactions between the agents and the environments. The overall framework is shown in Figure 1, and the pseudo-code is provided in Algorithm 1.

3.1 Hierarchical Environment Design

The objective is to generate a limited number of environments that are designed to enhance the general capability of the student agent. Inspired by the principles of PAIRED [3], we adopt an RL-based approach for the environment generation process. To better generate suitable environments tailored to the current student skill level, SHED uses the hierarchical MDP framework, consisting of an upper-level RL teacher policy Λ and a lower-level student policy π . Specifically, the teacher policy, Λ : Π → Θ , maps from the space of all potential student policies Π to the space of environment parameters Θ . Existing RL-based methods (e.g., PARIED) rely solely on regret feedback and fail to effectively capture the nuances of the student policy. To address this challenge, SHED enhances understanding by encoding the student policy π into a vector that serves as the state abstraction for teacher Λ . Rather than compressing the knowledge in the student policy network, we approximate the embedding of the student policy π by assessing performance across a set of diverse evaluation environments. This performance vector, denoted as p ( π ) , gives us a practical estimate of the student's current general capabilities, enabling the teacher to customize the next training environments accordingly. In our hierarchical framework, the environment generation process is governed by discrete-time dynamics. We delve into the specifics below.

Upper-level teacher MDP . The upper-level teacher operates at a coarser layer of student policy abstraction and generates environments to train the lower-level student agent. This process can be formally modeled as an MDP by the tuple < S u , A u , P u , R u , γ u > :

S u represents the upper-level state space. Typically, s u = p ( π ) = [ p 1 , . . . , p m ] denotes the student performance vector across m diverse evaluation environments. This vector serves as the representation of the student policy π and is observed by the teacher.

Algorithm 1 SHED

Input: real data ratio ψ ∈ [0 , 1] , evaluate environment set θ eval , reward function R ;

1: Initialize: diffusion model D , teacher policy Λ , real and synthetic replay buffer B real , B syn = ∅ ;

2: for episode ep = 1 , . . . , K do

3: Initialize student policy π

4: Evaluate π on θ eval and get state s u = p ( π )

5: for Budget t = 1 , . . . , T do

6: generate ⃗ θ ∼ Λ , and create M ⃗ θ ( π )

8: evaluate π on θ eval and get next state s ′

7: train π on M ⃗ θ to maximize V ⃗ θ ( π )

9: compute teacher's reward r t according to R

10: add experience ( s u t , ⃗ θ, r u t , s u, ′ t ) to B real

11: train D with samples from B real

12: generate synthetic experiences from D and add them to B syn

13: train Λ on samples from B real ⋃ B syn mixed with ratio ψ

14: set s = s ′ ;

15: end for

16: end for

Output:

Λ , π , D

Train on the environment

Generate new environments

Student agent

Teacher agent

D

Evaluate on the

evaluation

SSss

environment set

Diffusion model

Train on Breat to

Breal

generate

Bsyu

teacher agent

Train the

Teacher agent experience (s, a, r, s)

p(π) = [P,.,Pm]

p(π) = [p,-,p'm]

Evaluation environment set

Teacher agent

Evaluation environment set

state s

Next state s,

state s

Generate

action

al#

Evaluate

Evaluate

New generated environments α

Train

Update

Train

Student agent

Train for C time steps

Student agent π

A u is the upper-level action space. The teacher observes the abstraction of the student policy, 174 s u and produces an upper-level action a u which is the environment parameters ⃗ θ . ⃗ θ ( a u ) is then 175 used to generate specific environment instances M ⃗ θ . Thus the upper-level action space A u is the 176 environment parameter space Θ . 177

178

179

P u denotes the action-dependent transition dynamics of the upper-level state. The general capability of the student policy evolves due to training the student agent on the generated environments.

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

R u provides the upper-level reward to the teacher at the end of training the student on the generated environment. The design of R u will be discussed in Section 3.3.

As shown in Figure 2, given the student policy π , the teacher Λ first observes the representation of the student policy, s u = [ p 1 , . . . , p m ] . Then teacher produces an upper-level action a u which corresponds to the environment parameters. These environment parameters are subsequently used to generate specific environment instances. The lower-level student policy π will be trained on the generated environments for C training steps. The upper-level teacher collects and stores the student policy evolution transition ( s u , a u , r u , s u, ′ ) every C times steps for off-policy training. The teacher agent is trained to maximize the cumulative reward giving the budget for the number of generated environments. The choice of the evaluation environments will be discussed in Section 3.3.

Lower-level student MDP . The generated environment is fully specified for the student, characterized by a Partially Observable Markov Decision Process (POMDP), which is defined by a tuple M ⃗ θ = < A,O,S ⃗ θ , P ⃗ θ , I ⃗ θ , R ⃗ θ , γ > , where A represents the set of actions, O is the set of observations, S ⃗ θ is the set of states determined by the environment parameters ⃗ θ , similarly, P ⃗ θ is the environmentdependent transition function, and I ⃗ θ : ⃗ θ → O is the environment-dependent observation function, R ⃗ θ is the reward function, and γ is the discount factor. At each time step t , the environment produces a state observation s t ∈ S ⃗ θ , the student agent samples the action a t ∼ A and interacts with environment ⃗ θ . The environment yields a reward r t according to the reward function R ⃗ θ . The student agent is trained to maximize their cumulative reward V ⃗ θ ( π ) = ∑ C t =0 γ t r t for the current environment under a finite time horizon C . The student agent will learn a good general capability from training on a sequence of generated environments.

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

The hierarchical framework enables the teacher agent to systematically measure and enhance the general capability of the student agent and to adapt the training process accordingly. However, it's worth noting that collecting student policy evolution trajectories ( s u , a u , r u , s u, ′ ) to train the teacher agent is notably slow and resource-intensive, since each transition in the upper-level teacher MDP encompasses a training horizon of C timesteps for the student in the generated environment. Thus, it is essential to reduce the need for costly collection of upper-level teacher experiences.

3.2 Generative Trajectory Modeling

In this section, we will formally introduce a generative model designed to ease the collection of upperlevel MDP experience. This will allow us to train our teacher policy more efficiently. In particular, we first utilize a diffusion model to learn the conditional data distribution from the collected experiences τ = { ( s u t , a u t , r u t , s p, ′ t ) } . Later we can use the reverse chain in the diffusion model to generate the synthetic trajectories that can be used to help train the teacher agent, thereby alleviating the need for extensive and time-consuming collection of upper-level teacher experiences. We deal with two different types of timesteps in this section: one for the diffusion process and the other for the upperlevel teacher agent, respectively. We use subscripts k ∈ 1 , . . . , K to represent diffusion timesteps and subscripts t ∈ 1 , . . . , T to represent trajectory timesteps in the teacher's experience.

In the image domain, the diffusion process is implemented across all pixel values of the image. In our setting, we diffuse over the next state s u, ′ conditioned the given state s u and action a u . We construct our generative model according to the conditional diffusion process:

As usual, q ( s u, ′ k | s u, ′ k -1 ) is the predefined forward noising process while p ϕ ( s u, ′ k -1 | s u, ′ k , s u , a u ) is the trainable reverse denoising process. We begin by randomly sampling the collected experiences τ = { ( s u t , a u t , r u t , s u, ′ t ) } from the real experience buffer B real . Giving the observed state s u and action a u , we use the reverse process p ϕ to represent the generation of the next state s u, ′ :

At the end of the reverse chain, the sample s u, ′ 0 , is the generated next state s u, ′ . Similar to Ho et al. [6], we parameterize p ϕ ( s ′ k -1 | s ′ k , s u , a u ) as a noise prediction model with the covariance matrix fixed as Σ ϕ ( s u, ′ k , s u , a u , k ) = β i I , and the mean is

ϵ ϕ ( s u, ′ k , s u , a u , k ) is the trainable denoising function, which aims to estimate the noise ϵ in the noisy input s u, ′ k at step k .

Training objective. We employ a similar simplified objective to train the conditional ϵ - model:

Where s u, ′ k = √ ¯ α k s u, ′ + √ 1 -¯ α k ϵ . The intuition for the loss function L ( ϕ ) is to predict the noise ϵ ∼ N (0 , I ) at the denoising step k , and the diffusion model is essentially learning the student policy involution trajectories collected in the real experience buffer B reals . Note that the reverse process necessitates a substantial number of steps K [15]. Recent research by Xiao et al. [22] has demonstrated that enabling denoising with large steps can reduce the total number of denoising steps K . To expedite the relatively slow reverse sampling process (as it requires computing ϵ ϕ networks K times), we use a small value of K . Similar to Wang et al. [20], while simultaneously setting β min = 0 . 1 and β max = 10 . 0 , we define:

This noise schedule is derived from the variance-preserving Stochastic Differential Equation by Song 235 et al. [16]. 236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

Generate synthetic trajectories. Once the diffusion model has been trained, it can be used to generate synthetic experience data by starting with a draw from the prior s u, ′ K ∼ N (0 , I ) and successively generating denoised next state, conditioned on the given s u and a u through the reverse chain p ϕ . Note that the giving condition action a can either be randomly sampled from the action space or use another diffusion model to learn the action distribution giving the initial state s u . This new diffusion model is essentially a behavior-cloning model that aims to learn the teacher policy Λ( a u | s u ) . This process is similar to the work of Wang et al. [20]. We discuss this process in detail in the appendix. In this paper, we randomly sample a u as it is straightforward and can also increase the diversity in the generated synthetic experience to help train a more robust teacher agent.

After obtaining the generated next state s u, ′ conditioned on s u , a u , we compute reward r u using teacher's reward function R ( s u , a u , s u, ′ ) . The specifics of how the reward function is chosen are explained in the following section.

3.3 Rewards and Choice of evaluate environments

Selection of evaluation environments. The upper-level teacher generates environments tailored for the lower-level student to improve its general capability. Thus it is important to select a set of diverse suitable evaluation environments as the performance vector reflects the student agent's general capabilities and serves as an approximation of the policy's embedding. Fontaine and Nikolaidis [5] propose the use of quality diversity (QD) optimization to collect high-quality environments that exhibit diversity for the agent behaviors. Similarly, Bhatt et al. [1] introduce a QD-based algorithm for dynamically designing such evaluation environments based on the current agent's behavior. However, it's worth noting that this QD-based approach can be tedious and time-consuming, and the collected evaluation environments heavily rely on the given agent policy.

Given these considerations, it is natural to take advantage of the domain randomization algorithm, as it has demonstrated compelling results in generating diverse environments and training generally capable agents. In our approach, we first discretize the environment parameters into different ranges, then randomly sample from these ranges, and combine these parameters to generate evaluation environments. This method can generate environments that may induce a diverse performance for the same policy, and it shows promising empirical results in the final experiments.

Reward design. We define the reward function for the upper-level teacher policy as a parameterized function based on the improvement in student performance in the evaluation environments after training in the generated environment:

This reward function gives positive rewards to the upper-level teacher for taking action to create the right environment to improve the overall performance of students across diverse environments. However, it may encourage the teacher to obtain higher rewards by sacrificing student performance in one subset of evaluation environments to improve student performance in another subset, which conflicts with our objective to develop a student agent with general capabilities. Therefore, we need to consider fairness in the reward function to ensure that the generated environment can improve student's general capabilities. Similar to [4], we build our fairness metric on top of the change in student's performance in each evaluation environment, denoted as ω i = p ′ i -p i , and we have ¯ ω = 1 m ∑ m i =1 ω i . We then measure the fairness of the teacher's action using the coefficient of variation of student performances:

A teacher is considered to be fair if and only if the cv is smaller. As a result, our reward function is:

Here η is the coefficient that balances the weight of fairness in the reward function (We set a small 279 value to η ). This reward function motivates the teacher to generate training environments that can 280 improve student's general capability. 281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

AveragePerformanceonLunarLander

AveragePerformanceonBipedalWalker

50

Performances

-550

Performances

0

-50

Random

Accel-Edit

Random

Accel

Accel

1200

PAIRED

-100

PAIRED

h-MDP

h-MDP

SHED

SHED

0

200

400

600

800

1000

0

200

400

600

800

1000

Trainingstep

Training step

4 Experiments

In this section, we conduct experiments to compare SHED to other leading approaches on three domains: Lunar Lander, maze and a modified BipedalWalker environment. Experimental details and hyperparameters can be found in the Appendix. Specifically, our primary comparisons involve SHED and h-MDP (our proposed hierarchical approach without diffusion model aiding in training) against four baselines: domain randomization [19], ACCEL, [10], Edited ACCEL(with slight modifications that it does not revisit the previously generated environments), PAIRED [3]. In all cases, we train a student agent via Proximal Policy Optimization (PPO [13], and train the teacher agent via Deterministic policy gradient algorithms(DDPG [14]), because DDPG is an off-policy algorithm and can learn from both real experiences and the synthetic experiences.

Setup. For each domain, we construct a set of evaluation environments and a set of test environments. The vector of student performances in the evaluation environments is used as the approximation of the student policy (as the observation to teacher agent), and the performances in the test environments are used to represent the student's zero-shot transfer performances (general capabilities). Note that in order to obtain a fair comparison of zero-shot transfer performance, the evaluation environments and test environments do not share the same environment and they are not present during training.

Lunar Lander. This is a classic rocket trajectory optimization problem. In this domain, student agents are tasked with controlling a lander's engine to safely land the vehicle. Before the start of each episode, teacher algorithms determine the environment parameters that are used to generate environments in a given play-through, which includes gravity, wind power, and turbulence power. These parameters directly alter the difficulty of landing the vehicle safely. The state is an 8-dimensional vector, which includes the coordinates of the lander, its linear velocities, its angle, its angular velocity, and two booleans that represent whether each leg is in contact with the ground or not.

We train the student agent for 1e6 environment time steps and periodically test the agent in test environments. The parameters for the test environments are randomly generated and fixed during training. We report the experiment results on the left side of Figure 3. As we can see, student agents trained under SHED consistently outperform other baselines and have minimal variance in transfer performance. During training, the baselines, except h-MDP, show a performance dip in the middle. This phenomenon could potentially be attributed to the inherent challenge of designing the appropriate environment instance in the large environment parameter space. This further demonstrates the effectiveness of our hierarchical design ( SHED and h-MDP), which can successfully create environments that are appropriate to the current skill level of the students.

Bipedalwalker. We also evaluate SHED in the modified BipedalWalker from Parker-Holder et al. [10]. In this domain, the student agent is required to control a bipedal vehicle and navigate across the terrain, and the student receives a 24-dimensional proprioceptive state with respect to its lidar sensors, angles, and contacts. The teacher is tasked to select eight variables (including ground roughness, the

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

number of stairs steps, min/max range of pit gap width, min/max range of stump height, and min/max range of stair height) to generate the corresponding terrain.

We use similar experiment settings in prior UED works, we train all the algorithms for 1e7 environment time steps, and then evaluate their generalization ability on ten distinct test environments in Bipedal-Walker domain. The parameters for the test environments are randomly generated and fixed during training. As shown in Figure 3, our proposed method SHED surpasses all other baselines and achieves performance levels nearly on par with the SOTA (ACCEL). Meanwhile, SHED maintains a slight edge in terms of stability and overall performance and PAIRED suffers from a considerable degree of variance in its performance.

Partially observable Maze. Here we study navigation tasks, where an agent must explore to find a goal while navigating around obstacles. The environment is partially observable, and the agent's field of view is limited to a 3 × 3 grid area. Unlike the previously mentioned domains, maze environments are non-parametric and cannot be directly represented by compact parameter vectors due to their high complexity. To solve this challenge, we propose a novel method to generate maze by leveraging advances in large language models (e.g., ChatGPT). Specifically, we implement a retrieval-augmented generation (RAG) process to optimize the ChatGPT's output such that it can generate desired maze environments. This process ensures that large language models reference authoritative knowledge bases to generate feasible mazes. To simplify the teacher's action space, we extracted several key factors that constitute the teacher's action space (environmental parameters) for maze generation. Details on maze generation are provided in Appendix D.3, and prompt are included in Appendix D.4.

The average zero-shot transfer performances are reported in Figure 4. Notably, SHED demonstrates the highest performance, consistently improving and achieving the highest cumulative rewards. The performance of h-MDP steadily improves but does not reach the highest levels, which further highlights the advantages of incorporating the generated synthetic datasets to train an effective RL teacher agent. Meanwhile, Accel-Edit and Accel show higher variances in performance, indicating that random teachers are less stable in finding a suitable environment to train student agents.

Ablation and additional Experiments In Appendix C, we evaluate

the ability of the diffusion model to generate the synthetic student policy involution trajectories. We further provide ablation studies to assess the impact of different design choices in Appendix E.1. Additionally, in Appendix E.2, we conduct experiments to show how the algorithm performs under different settings, including scenarios with a larger budget constraint on the number of generated environments or a larger weight assigned to CV fairness rewards. Notably, all results consistently demonstrate the effectiveness of our approach.

5 Conclusion

In this paper, we introduce an adaptive approach for efficiently training a generally capable agent under resource constraints. Our approach is general, utilizing an upper-level MDP teacher agent that can guide the training of the lower-level MDP student agent agent. The hierarchical framework can incorporate techniques from existing UED works, such as prioritized level replay (revisiting environments with high learning potential). Furthermore, we have described a method to assist the experience collection for the teacher when it is trained in an off-policy manner. Our experiment demonstrates that our method outperforms existing UED methods, highlighting its effectiveness as a curriculum-based learning approach within the UED framework.

DR

Accel-Edit

Accel

Paired

h-MDP

Shed

Reward

Timestep

371

References

Varun Bhatt, Bryon Tjanaka, Matthew Fontaine, and Stefanos Nikolaidis. Deep surrogate 372 assisted generation of environments. Advances in Neural Information Processing Systems , 35: 373 37762-37777, 2022. 374

Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, 375 Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative 376 interactive environments. arXiv preprint arXiv:2402.15391 , 2024. 377

Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew 378 Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised 379 environment design. Advances in neural information processing systems , 33:13049-13061, 380 2020. 381

382

383

384

Salma Elmalaki. Fair-iot: Fairness-aware human-in-the-loop reinforcement learning for harnessing human variability in personalized iot. In Proceedings of the International Conference on Internet-of-Things Design and Implementation , pages 119-132, 2021.

385

386

387

388

Matthew Fontaine and Stefanos Nikolaidis. Differentiable quality diversity. Advances in Neural Information Processing Systems , 34:10040-10052, 2021.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Interna389 tional Conference on Machine Learning , pages 4940-4950. PMLR, 2021. 390

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, 391 Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing 392 with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021. 393

Weili Nie, Brandon Guo, Yujia Huang, Chaowei Xiao, Arash Vahdat, and Anima Anandkumar. 394 Diffusion models for adversarial purification. arXiv preprint arXiv:2205.07460 , 2022. 395

396

397

398

Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. arXiv preprint arXiv:2203.01302 , 2022.

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning , pages 387-395. Pmlr, 2014.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Richard S Sutton, Andrew G Barto, et al. Introduction to reinforcement learning , volume 135. 417 MIT press Cambridge, 1998. 418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems , 34:24804-24816, 2021.

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) , pages 23-30. IEEE, 2017.

Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=AHvFDPi-FA .

Julian Wyatt, Adam Leach, Sebastian M Schmon, and Chris G Willcocks. Anoddpm: Anomaly detection with denoising diffusion probabilistic models using simplex noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 650-656, 2022.

Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.07804 , 2021.

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

A Theorem

Theorem 1 There exists a finite evaluation environment set that can capture the student's general capabilities and the performance vector [ p 1 , . . . , p m ] is a good representation of the student policy.

To prove this, we first provide the following Assumption:

Assumption 1 Let p ( π, ⃗ θ ) denote the performance of student policy π in an environment ⃗ θ . For ∀ i -th dimension of the environment parameters, denoted as θ i , when changing the θ i to θ ′ i to get a new environment θ ⃗ ′ while keeping other environment parameters fixed, there ∃ δ i > 0 , if | θ ′ i -θ i | ≤ δ i , we have | p ( π, ⃗ θ ′ ) -p ( π, ⃗ θ ) | ≤ ϵ i , where ϵ i → 0 .

If this is true, we then can construct a finite set of environments, and the student performances in those environments can represent the performances in all potential environments generated within the certain environment parameters open interval combinations, and the set of those open intervals combinations cover the environment parameter space Θ .

We begin from the simplest case where we only consider using one environment parameter to generate environments, denoted as θ i . We can construct a finite environment parameter set for environment parameters, which is { θ min i +1 / 2 ∗ δ i , θ min i +3 / 2 ∗ δ i , θ min i +7 / 2 ∗ δ i , . . . , θ max i -δ i / 2 } . Assume the set size is L i . We let the set { θ ⃗ i } L i i =1 denote the corresponding generated environments. This is served as the representative environment set . Then the student performances in those environments are denoted as { p ( π, ⃗ θ i ) } L i i =1 , which we call it as representative performance vector set . We can divide the space for θ i into a finite set of open intervals with size L i , which is { [ θ min i , θ min i +3 / 2 ∗ δ i ) , ( θ min i +1 / 2 ∗ δ i , θ min i +5 / 2 δ i ) , ( θ min i +5 / 2 ∗ δ i , θ min i +9 / 2 ∗ δ i ) , . . . , ( θ max i -3 / 2 ∗ δ i , θ max i ] } , which we call it as representative parameter interval set , also denoted as { ( θ i -δ, θ i + δ ) } L i i =1 . For any environment generated in those intervals, denoted as θ ⃗ ′ i , the performance p ( π, ⃗ θ ′ i ) can always be represented by the p ( π, ⃗ θ i ) which is in the same interval, as | p ( π, ⃗ θ ′ i ) -p ( π, ⃗ θ i ) | ≤ ϵ i , where ϵ i → 0 . In such cases, the finite set of environmental parameter intervals { θ min i +1 / 2 ∗ δ i , θ min i + 3 / 2 ∗ δ i , θ min i +7 / 2 ∗ δ i , . . . , θ max i -δ i / 2 } fully covers the entire parameter space Θ . We can find a representative environment set { θ ⃗ i } L i i =1 that is capable of approximating the performance of the student policy within the open parameter intervals combination. This set effectively characterizes the general performance capabilities of the student policy π .

Then we extend to two environment parameter design space cases. Let's assume that the environment is generated by two-dimension environment parameters. Then, for each environment parameter, θ i ∈ { θ 1 , θ 2 } . We can find the same open interval set for each parameter. Specifically, for each θ i , there exists a δ i , such that if | θ ′ i -θ i | ≤ δ i , we have | p ( π, ⃗ θ ′ ) -p ( π, ⃗ θ ) | ≤ ϵ i , where ϵ i → 0 . Hence, we let δ = min { δ 1 , δ 2 } and ϵ = ϵ 1 + ϵ 2 . Thus the new representative environment set is the set that includes the any combination of { [ θ 1 , θ 2 ] } where θ 1 ∈ { θ ⃗ i } L 1 i =1 and θ 2 ∈ { θ ⃗ j } L 2 j =1 . We can get the representative performance vector set as { p ( π, [ θ ⃗ i , ⃗ θ j ]) } i ∈ [1 ,L 1 ] ,j ∈ [1 ,L 2 ] . We then can construct the representative parameter interval set as { [( θ i -δ, θ i + δ ) , ( θ j -δ, θ j + δ )] } i ∈ [1 ,L 1 ] ,j ∈ [1 ,L j ] . As a result, for any new environments [ θ ⃗ ′ i , ⃗ θ ′ j ] , we can find the representative environment whose environment parameters are in the same parameter interval [ θ ⃗ i , ⃗ θ j ] , such that their performance difference is smaller than ϵ = ϵ 1 + ϵ 2 for all ∀ i ∈ [1 , L 1 ] , ∀ j ∈ [1 , L 2 ] :

In such cases, the finite set of environmental parameter intervals { [( θ i -δ, θ i + δ ) , ( θ j -δ, θ j + 473 δ )] } i ∈ [1 ,L 1 ] ,j ∈ [1 ,L j ] fully covers the entire parameter space Θ . We can find a representative environ474 ment set { θ ⃗ i } L i i =1 that is capable of approximating the performance of the student policy within the 475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

open parameter intervals combination. This set effectively characterizes the general performance capabilities of the student policy π .

Similarly, we can show this still holds when the environment is constructed by a larger dimension environment parameters, where we set δ = min { δ i } , and ϵ = ∑ i ϵ i , and we have δ > 0 , ϵ → 0 . The overall logic is that we can find a finite set, which is called representative environment set , and we can use performances in this set to represent any performances in the environments generated in the representative parameter interval set , which is called representative performance vector set . Finally, we can show that representative parameter interval set fully covers the environment parameter space. Thus there exists a finite evaluation environment set that can capture the student's general capabilities and the performance vector, called representative performance vector set , [ p 1 , . . . , p m ] is a good representation of the student policy.

B Details about the Generative model

B.1 Generative model to generate synthetic next state

Here, we describe how to leverage the diffusion model to learn the conditional data distribution in the collected experiences τ = { ( s u t , a u t , r u t , s u, ′ t ) } . Later we can use the trainable reverse chain in the diffusion model to generate the synthetic trajectories that can be used to help train the teacher agent, resulting in reducing the resource-intensive and time-consuming collection of upper-level teacher experiences. We deal with two different types of timesteps in this section: one for the diffusion process and the other for the upper-level teacher agent, respectively. We use subscripts k ∈ 1 , . . . , K to represent diffusion timesteps and subscripts t ∈ 1 , . . . , T to represent trajectory timesteps in the teacher's experience.

In the image domain, the diffusion process is implemented across all pixel values of the image. In our setting, we diffuse over the next state s u, ′ conditioned the given state s u and action a u . We construct our generative model according to the conditional diffusion process:

As usual, q ( s u, ′ k | s u, ′ k -1 ) is the predefined forward noising process while p ϕ ( s u, ′ k -1 | s u, ′ k , s u , a u ) is the trainable reverse denoising process. We begin by randomly sampling the collected experiences τ = { ( s u t , a u t , r u t , s u, ′ t ) } from the real experience buffer B real .

We drop the superscript u here for ease of explanation. Giving the observed state s and action a , we use the reverse process p ϕ to represent the generation of the next state s ′ :

At the end of the reverse chain, the sample s ′ 0 , is the generated next state s ′ . As shown in Section 2.2, p ϕ ( s ′ k -1 | , s ′ k , s, a ) could be modeled as a Gaussian distribution N ( s ′ k -1 ; µ θ ( s ′ k , s, a, k ) , Σ θ ( s ′ k , s, a, k )) . Similar to Ho et al. [6], we parameterize p ϕ ( s ′ k -1 | s ′ k , s, a ) as a noise prediction model with the covariance matrix fixed as

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

and mean is

Where ϵ θ ( s ′ k , s, a, k ) is the trainable denoising function, which aims to estimate the noise ϵ in the noisy input s ′ k at step k . Specifically, giving the sampled experience ( s, a, s ′ ) , we begin by sampling s ′ K ∼ N (0 , I ) and then proceed with the reverse diffusion chain p ϕ ( s ′ k -1 | , s ′ k , s, a ) for k = K,... , 1 . The detailed expression for s ′ k -1 is as follows:

where ϵ ∼ N (0 , I ) . Note that ϵ = 0 when k = 1 .

Training objective. We employ a similar simplified objective, as proposed by Ho et al. [6] to train the conditional ϵ - model through the following process:

Where s ′ k = √ ¯ α k s ′ + √ 1 -¯ α k ϵ . U represents a uniform distribution over the discrete set { 1 , . . . , K } . The intuition for the loss function L ( θ ) tries to predict the noise ϵ ∼ N (0 , I ) at the denoising step k , and the diffusion model is essentially learning the student policy involution trajectories collected in the real experience buffer B reals . Note that the reverse process necessitates a substantial number of steps K , as the Gaussian assumption holds true primarily under the condition of the infinitesimally limit of small denoising steps [15]. Recent research by Xiao et al. [22] has demonstrated that enabling denoising with large steps can reduce the total number of denoising steps K . To expedite the relatively slow reverse sampling process outlined in Equation 3.2 (as it requires computing ϵ ϕ networks K times), we use a small value of K , while simultaneously setting β min = 0 . 1 and β max = 10 . 0 . Similar to Wang et al. [20], we define:

This noise schedule is derived from the variance-preserving Stochastic Differential Equation by Song et al. [16].

Generate synthetic trajectories. Once the diffusion model has been trained, it can be used to generate synthetic experience data by starting with a draw from the prior s ′ K ∼ N (0 , I ) and successively generating denoised next state, conditioned on the given s and a through the reverse chain p ϕ in Equation 3.2. Note that the giving condition action a can either be randomly sampled from the action space (which is also the environment parameter space) or use another diffusion model to learn the action distribution giving the initial state s . In such case, this new diffusion model is essentially a behavior-cloning model that aims to learn the teacher policy Λ( a | s ) . This process is similar to the work of Wang et al. [20]. We discuss this process in detail in the appendix. In this paper, we randomly sample a as it is straightforward and can also increase the diversity in the generated synthetic experience to help train a more robust teacher agent.

B.2 Generative model to generate synthetic action

Once the diffusion model has been trained, it can be used to generate synthetic experience data by starting with a draw from the prior s ′ K ∼ N (0 , I ) and successively generating denoised next state, conditioned on the given s and a through the reverse chain p ϕ in Equation 3.2. Note that the giving condition action a can either be randomly sampled from the action space (which is also the environment parameter space) or we can train another diffusion model to learn the action distribution giving the initial state s , and then use the trained new diffusion model to sample the action a giving the state s . This process is similar to the work of Wang et al. [20].

In particular, We construct another conditional diffusion model as:

547

548

549

550

551

552

553

554

555

556

557

558

559

560

synthetic next state s

real next state s

synthetic next state s

real next state s

synthetic next state s

real next state s

real

5.0

action a1

2

3

4

6

2.0

2.5

3.0&0

3.5cale

action a2

2

3

5

6

2.0

2.5

4.05}

3

action a3

4

5

6

7

8

2.0

2.5

3.0&0

3.54ae

4.05

real next state s2

real next state s'3

real next state s

synthetic next state s2

synthetic next state s3

synthetic next state s

0.6

5.6.0

.1.

4.5

0

5.0

4.5

5.0

4.052

4.053

4.0SA

action a2

3

5

6

2.0

2.5

action a3

4

5

6

7

8

2.0

2.5

actiona3

5

6

8

2.0

2.5

As usual, q ( a k | a k -1 ) is the predefined forward noising process while p ϕ ( a k -1 | a k , s ) is the trainable 543 reverse denoising process. we represent the action generation process via the reverse chain of the 544 conditional diffusion model as 545

At the end of the reverse chain, the sample a 0 , is the generated action a for the giving state s . Similarly, we parameterize p ϕ ( a k -1 | a k , s ) as a noise prediction model with the covariance matrix fixed as

and mean is

Similarly, the simplified loss function is 546

Where a k = √ ¯ α k a + √ 1 -¯ α k ϵ . U represents a uniform distribution over the discrete set { 1 , . . . , K } . The intuition for the loss function L a ( θ ) tries to predict the noise ϵ ∼ N (0 , I ) at the denoising step k , and the diffusion model is essentially a behavior cloning model to learn the student policy collected in the real experience buffer B reals .

Once this new diffusion model is trained, the generation of the synthetic experience can be formulated as:

we first randomly sample the state from the collected real trajectories s ∼ τ ;

we use the new diffusion model discussed above to mimic the teacher's policy to generate the actions a ;

giving the state s and action a , we use the first diffusion model presented in the main paper to generate the next state s ′ ;

we compute the reward r according to the reward function, and add the final generated synthetic experience ( s, a, r, s ′ ) to the synthetic experience buffer B syn to help train the teacher agent.

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

real next state [p,P2. p']

real next state [p²,P2. P']

real next state [p²,P2.P']

synthetic next state [p².P2.P]

synthetic next state [p.P2. p]

synthetic next state [p.P2. P]

1.2

1.4

state

0

2

state

0

10

5

state

1.0

2

5

0.8

4

10

0.6

15

0.4

10

[°0000-

statep1/

0.20.3

0.4

0.5

0.6

0.7

-0.4

0.2

state p2/

0.0

0.2

-3

-2

statep/

-1

0

1

2

3

4

state

2

-1

0

P2

-10

-5

statep1/

0

5

10

15

-10

-5

state

0

P21

5

C Empirical analysis of generative model

C.1 Ability to generate good synthetic trajectories

We begin by investigating SHED 's ability to assist in collecting experiences for the upper-level MDP teacher. This involves the necessity for SHED to prove its ability to accurately generate synthetic experiences for teacher agents. To check the quality of these generated synthetic experiences, we employ a diffusion model to simulate some data for validation (even though Diffusion models have demonstrated remarkable success across vision and NLP tasks).

We design the following experiment: given the teacher's observed state s u = [ p 1 , p 2 , p 3 , p 4 , p 5 ] , where p i denotes the student performance on i -th evaluation environment. and given the teacher's action a u = [ a 1 , a 2 , a 3 ] , which is the environment parameters and are used to generate corresponding environment instances. We use a neural network f ( s u , a u ) to mimic the involution trajectories of the student policy π . That is, with the input of the state s u and action a u into the neural network, it outputs the next observed state s u, ′ = [ p ′ 1 , p ′ 2 , p ′ 3 , p ′ 4 , p ′ 5 ] , indicating the updated student performance vector on the evaluation environments after training in the environment generated by a u . In particular, we add a noise ε into s u, ′ to represent the uncertainty in the transition. We first train our diffusion model on the real dataset ( s u , a u , s u, ′ ) generated by neural network f ( s u , a u ) . We then set a fixed ( s u , a u ) pair and input them into f ( s u , a u ) to generate 200 samples of real s u, ′ . The trained diffusion model is then used to generate 200 synthetic s u, ′ conditioned on the fixed ( s u , a u ) pair.

The results are presented in Figure 6, we can see that the generative model can effectively capture the distribution of real experience even if there is a large uncertainty in the transition, indicated by the value of ε . This provides evidence that the diffusion model can generate useful experiences conditioned on ( s u , a u ) . It is important to note that the marginal distribution derived from the reverse diffusion chain provides an implicit, expressive distribution, such distribution has the capability to capture complex distribution properties, including skewness and multi-modality.

C.2 addition experiments on diffusion model

We further provide more results to show the ability of our generative model to generate synthetic trajectories where the noise is extremely small. In such cases, the actual next state s ′ will converge to a certain value, and the synthetic next state s syn, ′ generated by the diffusion model should also be very close to that value, then the diffusion model has the ability to sample the next state s syn, ′ 0 which can accurately represent the next state. We present the results in Figure 5. Specifically, this figure shows when the noise is very small in the actual next state, which is 0 . 05 ∗ ϵ , and ϵ ∼ N (0 , 1) . Giving any condition ( s, a ) pair, we selectively report on ( s i , a i ) , where x -axis is the a i value, and y -axis is the s i value. The student policy with initial performance vector s is trained on the environments generated by the teacher's action a . We report the new performance s ′ i of student policy on i -th environments after training in the z -axis. In particular, if two points s ′ i and s syn, ′ i are close, it indicates that the diffusion model can successfully generate the actual next state. As we can see, when the noise is extremely small, our diffusion model can accurately predict the next state of s ′ i giving any condition ( s, a ) pair.

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

AveragePerformanceonLunarlander

Performances

550

h-MDP

SHED

SHED with 30 evaluation environments

SHED with 40 evaluation environments

1200

0

200

400

600

800

1000

Training step

AveragePerformanceonLunarlander

-500

Performances

-1000

-1500

Random

-2000

Accel

h-MDP

SHED

0

500

1000

1500

2000

2500

3000

Trainingstep

D Additional Experiment Details

D.1 Hyperparameters

We set the learning rate 1 e -3 for actor, and 3 e -3 for critic, we set gamma γ = 0 . 999 , λ = 0 . 95 , and set coefficient for the entropy bonus (to encourage exploration) as 0 . 01 . For each environment, we conduct 50 PPO updates for the student agent, and We can train on up to 50 environments, including replay. For our diffusion model, the diffusion discount is 0.99, and batch size is 64, τ is 0.005, learning rate is 3 e -4 . The synthetic buffer size is 1000 , and the ratio is 0.25.

D.2 Experiments Compute Resources

All the models were trained on a single NVIDIA GeForce RTX 3090 GPU and 16 CPUs.

D.3 Maze document

Here we provide the document shows the instruction to generate feasible maze environments.

There are several factors that can affect the difficulty of a maze. Here are some key factors to consider:

Maze Size: Larger mazes generally increase the complexity and difficulty as the agent has more states to explore. Typically, the maze size should be larger than 4x4 and smaller than 15*15.

-If the size is 7*7 or smaller, the maze size is considered easy.

-If the size is larger than 7*7 but smaller than 10*10, the maze size is considered medium.

-If the maze size is larger than 10x10 but smaller than 15*15, the maze size is considered hard.

Maze Structure: The complexity of the paths, including the number of twists, turns, and dead-ends, can significantly impact navigation strategies. The presence of narrow corridors versus wide-open spaces also plays a role.

-If there are fewer than 2 turns in the feasible path from the start position to the end position, the maze structure is considered easy.

-If there are more than 2 turns but fewer than 4 turns in the path from the start position to the end position, the maze structure is considered medium.

-If there are 4 or more turns in the path from the start position to the end 627 position, the maze structure is considered hard. 628

Goal Location: The distance from the starting position to the end position 629 also affects difficulty. 630

-If the path from the start position to the end position requires fewer than 631

D.4 Prompt for RAG 676

We provide our prompt for the Retrieval Augmented Generation as follows: 677

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

E Additional experiments

E.1 Additional experiments about ablation studies

We also provide ablation analysis to evaluate the impact of different design choices in Lunar lander domain, including (a) a larger evaluation environment set; (b) a bigger budget for constraint on the number of generated environments (which incurs a longer training time horizon). The results are reported in Figure 7.

We explore the impact of introducing the diffusion model in collecting synthetic teacher's experience and varying the size of the evaluation environment set. Specifically, as we can see from the right side of Figure 7, the SHED consistently outperforms h-MDP, indicating the effectiveness of introducing the generative model to help train the upper-level teacher policy. Furthermore, we find that when increasing the size of the evaluation environment set, we can have a better result in the student transfer performances. The intuition is that a larger evaluation environment set, encompassing a more diverse range of environments, provides a better approximation of the student policy according to the Theorem 1. However, the reason why SHED with 30 evaluation environments slightly outperforms SHED with 40 evaluation environments is perhaps attributed to the increase in the dimension of the student performance vector, which amplifies the challenge of training an effective diffusion model with a limited dataset.

We conduct experiments in Lunar lander under a longer time horizon. The results are provided on the right side of Figure 7. As we can see, our proposed algorithm SHED can efficiently train the student agent to achieve the general capability in a shorter time horizon, This observation indicates that our proposed environment generation process can better generate the suitable environments for the current student policy, thereby enhancing its general capability, especially when there is a constraint on the number of generated environments.

E.2 Additional experiments on Lunar lander

we also conduct experiments to show how the algorithm performs under different settings, such 707 as a larger weight of cv fairness rewards ( η = 10 ). The results are provided in Figure 8. We 708 noticed an interesting finding: when fairness reward has a high weightage, our algorithm tends to 709 generate environments at the onset that lead to a rapid decline and subsequent improvement in student 710 performance across all test environments. This is done to avoid acquiring a substantial negative 711 fairness reward and thereby maximize the teacher's cumulative reward. Notably, the student's final 712 performance still surpasses other baselines at the end of training.

AveragePerformanceonLunarlander

250

-500

hances

750

erforma

-1000

Pe

-1250

1500

Random

Accel

1750

h-MDP

SHED

-2000

0

200

400

600

800

1000

Trainingstep

713

We further show in detail how the performance of different methods changes in each testing environ714 ment during training (see Figure 9 and Figure 10 ). 715

AveragePerformanceonLunarlander

AveragePerformanceonLunarlander

0

-1000

Performances

Performances

-200

-2000

-400

-3000

Random

Random

-600

Accel

Accel

h-MDP

-4000

h-MDP

SHED

SHED

0

200

400

600

800

1000

0

200

400

600

800

1000

Training step

Training step

AveragePerformanceonLunarlander

AveragePerformanceonLunarlander

0

-1000

-1000

Performances

Performances

-2000

-2000

-3000

000-

Random

Random

4000

Accel

-4000

Accel

h-MDP

h-MDP

SHED

SHED

-5000

0

200

400

600

800

1000

0

200

400

600

800

1000

Training step

Trainingstep

AveragePerformanceonLunarlander

AveragePerformanceonLunarlander

-1000

-1000

Performances

Performances

-2000

-2000

F0000-

-4000-

-3000

Random

-5000

Random

Accel

Accel

4000

h-MDP

F0009-

h-MDP

SHED

SHED

0

200

400

600

800

1000

0

200

400

600

800

1000

Training step

Trainingstep

E.3 Additional experiments on Maze 716

We selectively report some results of zero-shot transfer performances in maze environments. The 717 results are provided in Figure 718

F Discussion 719

F.1 Limitations 720

The limitation of this work comes from the UED framework, as UED is limited to the use of 721 parameterized environments. This results in our experimental domain being relatively simple. 722

AveragePerformanceonLunarlander

AveragePerformanceonLunarlander

Random

100

Accel

0

h-MDP

SHED

Performances

-200

Performances

-100

400

-200

-600

000-

Random

Accel

-800

h-MDP

-400

SHED

0

200

400

600

800

1000

0

200

400

600

800

1000

Trainingstep

Training step

AveragePerformanceonLunarlander

AveragePerformanceonLunarlander

200

0

-1000

-200

Performances

-2000

Performances

-400

0000-

F009-

-800

-4000

-1000-

Random

Random

-5000

Accel

1200

Accel

h-MDP

h-MDP

-6000

SHED

1400

SHED

0

200

400

600

800

1000

0

200

400

600

800

1000

Trainingstep

Training step

(f) small maze

(g) medium maze

(h) large maze

(i) four rooms maze

(j) corridor maze

出

coe-6R

Acoel

Peired

hKDP

DR

Acxel6et

Accel

Psied

hMOF

DR

Shed

Acoel

Pied

hMDP

Acce-6BR

OR

Shed

Acxce-6et

Pired

DR

Accel

hMDP

Acce-EfRt

Acoel

Pvired

h-MDP

Shed

OR

However, our work proposes a new hierarchical structure, and our policy representation is not only of 723 great help for UED, but also has certain inspirations for hierarchical RL. Additionally, in the world 724 model of UED (Genie [2]), the environment generator (teacher) focuses on creating video games, a 725 domain that is compatible with our proposed application of upsampling the teacher agent's experience 726 using a diffusion model (since the state is image-based). 727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of this work is discussed in Appendix F.1.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See the theoretical result in Appendix 1.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and conclusions of the paper, detailed in Section 3 and Appendix D.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is provided in the supplementary marterial.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the training and test details Section 3 and Appendix D.1 and Appendix D.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The proposed method is thoroughly evaluated on three domains, and the results are reported based on a statistical analysis in Section 4 and Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The detailed configuration of the experiments is listed with required computational resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We confirm that the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics, and all the authors preserve anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impacts of our paper are presented in Section F.1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the assets, used in our paper, are properly credited and we explicitly mention and properly respect the license and terms of use.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

986

987

988

989

990

991

992

993

994

995

996

997

998

999

1000

1001

1002

1003

1004

1005

1006

1007

1008

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

1037

1038

1039

1040

1041

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAfter rebuttal\n\nWhile I think there are still some problems with this paper, e.g. the short training duration, and the slight exaggeration of claims (that SHED outperforms UED). I think, however, that the idea is nice, and getting RL environment design to work better is a good goal.\n\n\n-----\n\n\nThis paper aims to improve Unsupervised Environment Design in two ways.\nFirst, it introduces a hierarchical MDP formulation, where the top level corresponds to the teacher, and the lower level corresponds to the learning agent. Each transition in the top-level MDP involves training the lower level agent on generated levels. Related to this, they develop a state representation for the adversary, which is the performance of the agent on a fixed set of diverse levels.\n\nSeparately to this, they use a diffusion model to upsample the number of experiences for the teacher, effectively training on synthetic data, to improve sample efficiency.\n\nSTRENGTHS:\n- I think the H-MDP formulation itself is very valuable; it moves away from treating the generation of environments as a black-box, sparse reward, multi-step generation process (as in PAIRED), and towards a more informed process, where the teacher gets feedback in terms of the state (i.e., the performance vector).\n- The analysis in the appendix investigating the ability of the model to generate good synthetic trajectories is useful.\n\nWEAKNESSES:\n- Major\n\t- The results do not very convincingly demonstrate that SHED is better than current SoTA. Looking at figure 3 particularly, I would say that ACCEL has about the same performance as SHED. However, comparing against the RL baseline, SHED does do better.\n\t- The method is limited in the types of environments it can generate. For instance, mazes are generated using an LLM instead of directly placing blocks. This method therefore is not quite as broad in scope as PAIRED or ACCEL, which can generate arbitrary environments.\n\t- Relatedly, in the minigrid experiments, do all methods generate the levels in the same way using an LLM, providing the difficulty numbers? It would be good to compare this against the standard version of ACCEL that directly places blocks in the maze level, as it does not have the same restriction as SHED.\n- Minor\n\t- The figures can be improved:\n\t\t- Make the alpha value on the error bars a bit less\n\t\t- Keep colours consistent across figures, so the same method has the same colour\n\t\t- Keep capitalisation consistent across the figure labels.\n\t- line 80, the period ends on the line after the maths, it should end on the same line.\n\t- Footnote 1: Jiang et al. (2021) use (amongst others) the positive value loss, which is not quite the GAE, as it clips it at zero before summing.\n\t- equation one, you use $\\beta_t$ but $t$ does not seem to be defined? Should this be $\\beta_k$?\n\t- Line 159, PARIED should be PAIRED\n\t- There is no reward scale in figure 4\n\t- Figure 9's caption can be made clearer. I understand it to be the performance of each method in different testing environments. \n\t- Line 718 does not link to a figure.\n\t- Figure 11's caption: zero-shot and not zeros-shot\n\t- Capitalise the first word in the title of appendix C.2\n\t- In terms of notation, in line 96, $\\pi^*$ usually has a dependence on $\\theta$ (e.g. $\\pi^*_\\theta$) to indicate it is optimal w.r.t. that particular level.\n\t- Line 217, maybe add a citation to the first sentence, as I thought that is what you do, which confused me for a second.\n\t- line 237 space after period.\n\t- Line 241 \"given\" instead of giving?\n\t- Lines 296 - 297 are a bit confusing, as the word environment is used three times.\n\t- The assumption in theorem 1 is pretty strong.\n\nQUESTIONS:\n- It seems to me that the upsampling is very similar to [1], could you comment on that/cite the paper if indeed the technique is similar?\n- Have you tried/have any intuition of what the effect would be if you set $\\eta=0$?\n- Furthermore, it seems like the specific evaluation environments you use can have a large impact. Have you looked at how variable results are when using different (still randomly generated) environment parameters. Relatedly, what happens if you just generate a few DR levels *without* explicitly discretising the environment parameters? \n- Are you using the high entropy version of PAIRED introduced by [2]?\n- Do you have an indication of how easy it would be to generate larger environments, with much larger parameter vectors?\n- Do you think it would be possible to incorporate the multi-step generation of PAIRED into this process? E.g., still generate a maze step-by-step but also have the H-MDP structure and policy performance vector?\n- Why can't you sample the action in the diffusion process from the teacher's policy?\n- A reward of 50 in bipedal walker seems very low, could you compare against other papers / explain why it is low? E.g. [2] had rewards around 150\n\n\n[1] Lu, Cong, et al. \"Synthetic experience replay.\" _Advances in Neural Information Processing Systems_ 36 (2024).\n\n[2] Mediratta, Ishita, et al. \"Stabilizing unsupervised environment design with a learned adversary.\" _Conference on Lifelong Learning Agents_. PMLR, 2023.\n\nLIMITATIONS:\nI think the authors can list a few more limitations. \nPrimarily, the restriction on the type of environment that can be generated, i.e., it needs numerical parameters, and generating a maze outright is challenging. This is quite a large difference to prior settings.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach to Unsupervised Environment Design (UED) that addresses the challenges of efficiency by introducing a hierarchical MDP framework and using synthetic data. This framework involves an upper-level RL teacher agent that generates training environments tailored to a lower-level student agent's capabilities. The paper proposes the Synthetically-enhanced Hierarchical Environment Design (SHED) method, which uses generative modeling to create synthetic trajectory datasets, thereby reducing the resource-intensive interactions between agents and environments. The effectiveness of SHED is demonstrated through empirical experiments across various domains, showing superior performance compared to existing UED methods.\n\nSTRENGTHS:\n- The use of diffusion models to generate synthetic trajectories is a novel approach that effectively reduces the computational burden of training the teacher agent.\n- The paper provides comprehensive experiments across different domains, demonstrating the effectiveness and robustness of the proposed method compared to state-of-the-art UED approaches.\n\nWEAKNESSES:\n- The proposed method introduces significant complexity, particularly in the implementation of the hierarchical MDP and the generative modeling components. This might limit the accessibility and reproducibility of the approach.\n- While the empirical results are promising, the evaluation is limited to a few specific domains. It would be beneficial to see broader applicability across more diverse and complex environments.\n- Figure 4 is not properly formatted (no values on the axes).\n\nQUESTIONS:\n- Why were diffusion models chosen over other generative models (e.g., GANs, VAEs)? Have other models been considered or tested?\n- How well does the proposed method scale to real-world applications with significantly higher complexity and variability? Have any tests been conducted in such settings?\n\nLIMITATIONS:\nThe limitations are discussed in Appendix F.1 but I think the authors should discuss the limitations in the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the Unsupervised Environment Design problem, where a teacher agent seeks to design environments to train a student. Methods such as PLR, PAIRED and ACCEL have recently shown promising performance for random, RL and evolutionary generators. This paper proposes a handful of modifications, using RL with a different objective vs. PAIRED (performance on held out set vs. regret) and also proposes to add synthetic data to accelerate the RL process.\n\nSTRENGTHS:\n* This is an interesting method in a relevant area of research. UED seems to be one of the most active areas of research with plenty of opportunities for impact.\n* The use of evaluation environments is sensible and novel.\n* The idea of combining this with Genie is incredibly exciting. It would be interesting to hear how this could be possible or could work. Is there any way to show a simple proof of concept?\n\nWEAKNESSES:\n* There appear to be two confounding features of the method, the new objective for PAIRED and then the synthetic data. Why do they make sense to combine in this way? It just feels like the authors tried to do \"enough for a paper\" rather than contribute something meaningful that people can build on. I say this because its unclear how these two independent features interact with other existing algorithms. Maybe we should just do ACCEL with synthetic data for instance? Did the authors try that? If it is in the Appendix already and I missed it then I will increase my score.\n* The performance gains are fairly minor, and presented in an unclear fashion with just a bunch of curves on a single plot. Can we get some more rigorous analysis for example using the recommendations from Agarwal et al, \"Deep Reinforcement Learning at the Edge of the Statistical Precipice\"?\n* The Maze experiment seems to have many inductive biases and seems distinct from the diffusion based approach for BipedalWalker and LunarLander. What happens if ACCEL has access to ChatGPT as an editor and then uses replay? This seems like a simpler extension that alone could be a strong paper - although it would resemble ELM (Lehman et al 2022) so it wouldn't be particularly novel.\n* The related work is very light. This is disappointing since the paper builds on so many related areas, such as synthetic data, diffusion models, UED, language models for evolution, procedural content generation etc.\n\nQUESTIONS:\n* The authors say \"The hierarchical framework can incorporate techniques from existing UED works, such as prioritized level replay\". Why did you not do this? Then it would be much clearer to see if it is state of the art.\n\nLIMITATIONS:\nCovered in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this paper use hierarchical MDP formulation and a teacher agent trained by RL to perform curriculum learning. To address the sparse data available for the teacher agent, this paper uses diffusion models to synthesize datasets for training. This paper performs experiments on lunar lander and bipedal walker environments to validate their claim.\n\nSTRENGTHS:\nData sparsity is one of the main limitations of using a teacher agent in curricular RL. This paper uses diffusion models to synthesize a dataset for the teacher agent.\n\nWEAKNESSES:\n- This paper designs the teacher agent via hierarchical MDP to model the learning process of the student agent to perform curricular RL. However, Fingerprint Policy Optimization (Paul et al, 2019) also has a similar idea of modeling the learning process of the student agent. It would be interesting to explain more about how this paper's idea is related and contributes to this line of thought.\n\n- A fully trained algorithm on the BipedalWalker should approach a cumulative reward of 300. Even the modified version used in the ACCEL paper is measured on a scale of 0 out of 300. However, from Figure 3, it appears all baselines perform less than 50 on the BipedalWalker benchmark. It is questionable whether all baselines were fully trained with the right settings. Also, the performance of the proposed algorithm and those of the baselines are statistically too similar to see whether SHED improves over the baselines in Lunar Lander and BipedalWalker benchmarks. Finally, other than the version of ACCEL in this paper not performing as well as the ACCEL in the original paper, I am curious whether ACCEL can be considered state-of-the-art in the benchmarks as written in line 324. Genetic Curriculum (Song et al, 2022) reports higher cumulative reward on the BipedalWalkerHardcore environment.\n\n- Figure 4 has no scale on timestep and reward.\n\nQUESTIONS:\n- To improve the scores on the review, I hope to see the authors explain how their algorithm works in relation to existing works listed above. I also hope to see clarifications regarding the experiment results stated above as well.\n\nLIMITATIONS:\nThe authors has addressed the limitations of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}