Title: Adversarial Environment Design via Regret-Guided Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: deep reinforcement learning

curriculum learning

environment design
Abstract: Training agents that are robust to environmental changes remains a significant challenge in deep reinforcement learning (RL). Unsupervised environment design (UED) has recently emerged to address this issue by generating a set of training environments tailored to the agent's capabilities. While prior works demonstrate that UED has the potential to learn a robust policy, their performance is constrained by the capabilities of the environment generation. To this end, we propose a novel UED algorithm, adversarial environment design via regret-guided diffusion models (ADD). The proposed method guides the diffusion-based environment generator with the regret of the agent to produce environments that the agent finds challenging but conducive to further improvement. By exploiting the representation power of diffusion models, ADD can directly generate adversarial environments while maintaining the diversity of training environments, enabling the agent to effectively learn a robust policy. Our experimental results demonstrate that the proposed method successfully generates an instructive curriculum of environments, outperforming UED baselines in zero-shot generalization across novel, out-of-distribution environments.

Adversarial Environment Design via Regret-Guided Diffusion Models

Abstract

Training agents that are robust to environmental changes remains a significant challenge in deep reinforcement learning (RL). Unsupervised environment design (UED) has recently emerged to address this issue by generating a set of training environments tailored to the agent's capabilities. While prior works demonstrate that UED has the potential to learn a robust policy, their performance is constrained by the capabilities of the environment generation. To this end, we propose a novel UED algorithm, adversarial environment design via regret-guided diffusion models (ADD). The proposed method guides the diffusion-based environment generator with the regret of the agent to produce environments that the agent finds challenging but conducive to further improvement. By exploiting the representation power of diffusion models, ADD can directly generate adversarial environments while maintaining the diversity of training environments, enabling the agent to effectively learn a robust policy. Our experimental results demonstrate that the proposed method successfully generates an instructive curriculum of environments, outperforming UED baselines in zero-shot generalization across novel, out-ofdistribution environments. Project page: https://rllab-snu.github.io/projects/ADD

1 Introduction

Deep reinforcement learning (RL) has achieved great success in various challenging domains, such as Atari [1], GO [2], and real-world robotics tasks [3, 4]. Despite the progress, the deep RL agent struggles with the generalization problem; it often fails in unseen environments even with a small difference from the training environment distribution [5, 6]. To train well-generalizing policies, various prior works have used domain randomization (DR) [7, 8, 9], which provides RL agents with randomly generated environments. While DR enhances the diversity of the training environments, it requires a large number of trials to generate meaningful structures in high-dimensional domains. Curriculum reinforcement learning [10, 11] has been demonstrated to address these issues by providing instructive sequences of environments. Since manually designing an effective curriculum for complicated tasks is challenging, prior works [12, 13] focus on generating curricula that consider the current agent's capabilities. Recently, unsupervised environment design (UED, [14]) has emerged as a scalable approach, notable for its advantage of requiring no prior knowledge. UED algorithms alternate between training the policy and designing training environments that maximize the regret of the agent. This closed-loop framework ensures the agent learns a minimax regret policy [15], assuming that the two-player game between the agent and the environment generator reaches the Nash equilibrium.

There are two main approaches for UED: 1) learning-based methods, which employ an environment generator trained via reinforcement learning, and 2) replay-based methods, which selectively replay among previously generated environments. The learning-based methods [14, 16, 17] utilize an adaptive generator that controls the parameters that fully define the environment configuration. The generator receives a regret of the agent as a reward and is trained via reinforcement learning to produce environments that maximize the regret. While the learning-based methods can directly generate meaningful environments, training the generator with RL is unstable due to the moving manifold [16]. Additionally, we observe that the RL-based generator has limited environment coverage, which limits the generalization capability of the trained agent. In contrast, the replay-based methods [18, 19, 20] employ a random generator and select environments to revisit among previously generated environments. Since the random generator can produce diverse environments without additional training, they outperform the learning-based methods in zero-shot generalization tasks [20]. However, the replay-based methods are sample inefficient as they require additional episodes to evaluate the regret on the randomly generated environments.

In this work, we propose a sample-efficient and robust UED algorithm by leveraging the strong representation power of diffusion models [21]. First, to make UED suitable for using a diffusion model as a generator, we introduce soft UED, which augments the regret objective of UED with an entropy regularization term, as done in maximum entropy RL [22]. By incorporating the entropy term, we can ensure the diversity of the generated environments. Then, we present adversarial environment design via regret-guided diffusion models (ADD), which guides a diffusion-based environment generator with the regret of the agent to produce environments that are conducive to the performance improvement of the agent. Enabling this regret guidance requires the gradient of the regret with respect to the environment parameter. However, since the true value of the regret is intractable and the regret estimation methods used in prior works on UED are not differentiable, a new form of regret estimation method is needed. To this end, we propose a novel method that enables the estimation of the regret in a differentiable form by utilizing an environment critic, which predicts a return distribution of the current policy on the given environment. This enables us to effectively integrate diffusion models within the UED framework, significantly enhancing the environment generation capability.

Since the regret-guided diffusion does not require an additional training of the environment generator, we can preserve the ability to cover the high-dimensional environment domain as the random generator of the replay-based method. Moreover, ADD can directly generate meaningful environments via regret-guided sampling as the learning-based methods. By doing so, ADD effectively combines the strengths of previous UED methods while addressing some of their limitations. Additionally, unlike other UED methods, ADD allows us to control the difficulty levels of the environments it generates by guiding the generator with the probability of achieving a specific return. It enables the reuse of the learned generator in various applications, such as generating benchmarks.

We conduct extensive experiments across challenging tasks commonly used in UED research: partially observable maze navigation and 2D bipedal locomotion over challenging terrain. Experimental results show that ADD achieves higher zero-shot generalization performance in unseen environments compared to the baselines. Furthermore, our analysis on the generated environments demonstrates that ADD produces an instructive curriculum with varying complexity while covering a large environment configuration space. As a result, it is shown that the proposed method successfully generates adversarial environments and facilitates the agent to learn a policy with solid generalization capabilities.

2 Related Work

2.1 Unsupervsied Curriculum Reinforcement Learning

While curriculum reinforcement learning [13, 23, 24] has been shown to enhance the generalization performance of the RL agent, Dennis et al. [14] first introduce the concept of the unsupervised environment design (UED). UED encompasses various environment generation mehods, such as POET [12, 25] and GPN[26]. In this work, we follow the original concept of UED, which aims to learn a minimax regret policy [15] by generating training environments that maximize the regret of the agent. Based on this concept, the learning-based methods train an environment generator via reinforcement learning. PAIRED [14] estimates the regret with a difference between returns

obtained by two distinct agents, and trains RL-based generator by utilizing the regret as a reward. Recently, CLUTR [16] and SHED [17] utilize generative models to improve the performance of PAIRED. CLUTR trains the environment generator on the learned latent space, and SHED supplies the environment generator with augmented experiences created by diffusion models. Despite the progress, training the generator via RL is unstable due to the moving manifold [16, 27] and often struggles to generate diverse environments. On the other hand, replay-based methods based on PLR [18] utilize a random environment generator and decide which environments to replay. ACCEL [20] combines the evolutionary approaches [12, 25] and PLR by taking random mutation on replayed environments. While these replay-based methods show scalable performance on a large-scale domain [28] and outperform the learning-based methods, they do not have the ability to directly generate meaningful environments. Unlike prior UED methods, we augment the regret objective of UED with an entropy regularization term and propose a method that employs a diffusion model as an environment generator to enhance the environment generation capability. Our work is also closely related to data augmentation for training robust policy. Particularly, DRAGEN [29] and ISAGrasp [30] augment existing data in learned latent spaces to train a policy that is robust to unseen scenarios. Our algorithm, on the other hand, focuses on generating curricula of environments without any prior knowledge and dataset.

2.2 Diffusion Models

Diffusion models [21, 31, 32] have achieved remarkable performance in various domains, such as image generation [33], video generation [34], and robotics [35, 36, 37]. Particularly, diffusion models effectively perform conditional generation using guidance to generate samples conditioned on class labels [38, 39] or text inputs [40, 41, 42]. Prior works also guide the diffusion models utilizing an additional network or loss functions, such as adversarial guidance to generate images to attack a classifier [43], safety guidance using pre-defined functions to generate safety-critical driving scenarios [44], and guidance using reward functions trained by human preferences to generate censored samples. [45]. We note that our implementation of the regret-guided diffusion model is based on the work of Dhariwal et al. [38] and Yoon et al. [45].

3 Background

3.1 Unsupervised Environment Design

Unsupervised environment design (UED, [14]) aims to provide an adaptive curriculum to learn a policy that successfully generalizes to various environments. The environments are represented using a Markov decision process (MDP), defined as ⟨ A,S, P , R , ρ 0 , γ ⟩ , where A is a set of actions, S is a set of states, P : S × A × S → [0 , 1] is a transition model, R : S × A → R is a reward function, ρ 0 : S → [0 , 1] is an initial state distribution and γ is a discount factor. UED employs an environment generator that designs environments by controlling free environment parameters of underspecified environments, which is represented using an underspecified Markov decision process (UMDP). UMDP is defined as M = ⟨ A,S, Θ , P M , R M , ρ M 0 , γ ⟩ , where Θ is a set of free environment parameters. Assigning a value to the free environment parameter θ ∈ Θ results in a specific MDP M θ with the environment configuration ( P θ = P M ( θ ) , R θ = R M ( θ ) , ρ θ 0 = ρ M 0 ( θ ) ). For example, when learning a mobile robot to navigate towards the goal point while avoiding obstacles, θ could represent the positions of obstacles, the position of the goal, and the start position of the robot.

UED algorithms alternate between designing a set of environments and training the agent on the generated environments. The environment generator first produces an environment parameter θ that maximizes the agent's regret. The regret of the policy π on environment M θ is defined as,

where Π is a set of policies and V ( π, θ ) := E ρ θ 0 ,π, P θ [ ∑ N n =0 r n γ n ] is an expected return where r n is a reward obtained by π at timestep n on M θ . Then, the agent is trained on the generated environment to maximize the expected return, resulting in minimizing the regret. This framework can be formulated with the following two-player minimax game:

It is ensured that the agent learns the minimax regret policy π ∗ ∈ argmin π ∈ Π max θ ∈ Θ REGRET ( π, θ ) by assuming the two-player game (2) reaches the Nash equilibrium [14, 19]. However, learning the minimax regret policy is challenging. Since the objective (2) does not guarantee the diversity of generated environments, the agent may not be trained on sufficiently various environments.

3.2 Diffusion Probabilistic Models

A diffusion probabilistic model [21] is a generative model that generates samples from noise via iterative denoising steps. Diffusion models start with perturbing data by progressively adding Gaussian noise, called the forward process . The forward process can be modeled with a valuepreserving stochastic differential equation (VP SDE, [31]):

where t ∈ [0 , T ] is a continuous diffusion time variable, β t > 0 is a variance schedule, and W t is a standard Brownian motion. Since the forward process (3) has tractable conditional marginal distributions p t ( X t | X 0 ) = N ( √ α t X 0 , (1 -α t ) I ) where α t = e -∫ t 0 β t dt , p T ( X T ) will be corrupted into N (0 , I ) when T →∞ .

Generating samples following the data distribution p data ( · ) requires a reverse process , a reverse-time SDE that has the same marginal distributions as the forward process (3). By Anderson's theorem [46], the reverse process can be formulated with a reverse-time SDE defined as,

Hence, learning a diffusion model means learning a score network s ϕ ( X t , t ) that approximates a score function ∇ X t log p t ( X t ) . The score network is trained via score matching [47], then plugged into the reverse process (4):

Indeed, we can generate samples by solving the approximated reverse process (5) with an initial condition X T ∼ N (0 , I ) .

To generate samples with label Y using the diffusion model, the score function of the conditional distribution p t ( X t | Y ) should be estimated. Since p t ( X t | Y ) ∝ p t ( X t , Y ) = p t ( X t ) p t ( Y | X t ) due to Bayes' rule, conditional samples can be generated by solving the reverse process with classifier guidance [38]:

where ˆ p t ( Y | X t ) is a time-dependent classifier network and ω > 0 is a guidance weight to scale classifier gradients.

4 Proposed Method

In this section, we describe our approach to employ a diffusion model as an environment generator to enhance the environment generation capability. We first introduce soft UED, which mutates UED to be more suitable for using a diffusion model as a generator by augmenting the original objective with the entropy regularization term. Then, we propose a novel soft UED algorithm, adversarial environment design via regret-guided diffusion models (ADD). ADD consists of two key components: 1) a diffusion-based environment generation by using the regret as a guidance, and 2) a method to estimate the regret in a differentiable form. We present these key components in detail and conclude the section with an explanation of the overall system and its advantages compared to prior UED methods.

4.1 Soft Unsupervised Environment Design

In this section, we introduce soft UED, designed to guarantee the diversity of environments by adding an entropy regularization term to the original UED objective (2). Soft UED is defined as the following minimax game between the agent and the environment generator:

where Λ is a distribution over Θ , D Λ is a set of distributions over Θ , H (Λ) := -∑ θ ∈ Θ Λ( θ ) log Λ( θ )

is an entropy of Λ , and ω is a regularization coefficient. Based on the fact that H is concave, we can show that the strong duality holds:

The proof is detailed in Appendix A.1. Proposition 4.1 implies that there exists a valid optimal point (˜ π, ˜ Λ) , and it is stationary for alternative optimization of π and Λ . Hence, the agent will learn a soft minimax regret policy ˜ π = argmin π ∈ Π max Λ ∈D Λ L ( π, Λ) if it reaches the optimal point. One of the most significant difference from the original UED is the role of the environment generator. Instead of finding a specific environment parameter that maximizes the regret, soft UED updates the environment generator to sample environment parameters from a distribution that maximizes the objective function of soft UED.

We note that the soft UED framework encompasses prior UED algorithms. In the learning-based methods, the generator is trained with RL using an entropy bonus, which is known to enhance performance [48] and plays a similar role to H (Λ) . The replay-based methods also consider the diversity of environments by sampling environment parameters from a probability distribution proportional to the regret, instead of selecting a parameter that maximizes the regret. Therefore, soft UED can be considered as a general framework that incorporates practical methods.

4.2 Regret-Guided Diffusion Models

Soft UED converts the problem of generating regret-maximizing environments into a problem of sampling the environment parameter θ from a desired distribution Λ π := argmax Λ ∈D Λ L ( π, Λ) . It is a well-known fact that Λ π has a closed-form solution as follows:

where C π is a normalizing constant, and u ( · ) denotes an uniform distribution over Θ . Inspired by the classifier guidance (6), we solve this sampling problem by guiding a pre-trained diffusion model with the regret. To this end, we decompose the score function of Λ π as follows:

where t is a diffusion time variable, θ t is an environment parameter perturbed by the forward process (3), u t ( · ) denotes a distribution of θ t when θ 0 ∼ u ( · ) , Λ π t ( · ) denotes a distribution of θ t when θ 0 ∼ Λ π ( · ) , and REGRET t ( π, θ t ) is a time-dependent regret on the noised environment θ t , which is equal to REGRET ( π, θ 0 ) . We approximate the first term ∇ θ t log u t ( θ t ) with a score network s ϕ ( θ t , t ) ≈ ∇ θ t log u t ( θ t ) that is learned by training a diffusion-based environment generator on the randomly generated environment dataset before the agent begins to learn. Then, we can formulate a regret-guided reverse process with a reverse-time SDE as follows:

s

π

ϕ

(

θ

t

, t

) =

s

ϕ

(

θ

t

, t

) +

ω

∇

θ

t

REGRET

t

(

π, θ

t

)

,

Hence, if a gradient of the regret is tractable, we can sample an environment parameter θ 0 from the desired distribution Λ π by solving the regret-guided reverse process (10) with an initial condition θ T ∼ N (0 , I ) . However, the regret (1) is intractable since we cannot access the environment-specific optimal policy. Prior works on UED propose various methods to estimate the regret using episodic returns or temporal difference errors, but none of them are differentiable w.r.t. θ t since agents cannot access the environment parameter and the reward function.

Episodicresults

Environments

Regretguidance

Environmentcritic

Diffusion-based environment generator

4.3 A Differentiable Regret Estimation

In order to estimate the regret in a differentiable form, we present a novel method based on a flexible regret [14], which is known to enhance the performance of the learning-based UEDs [16]. The main idea is to estimate the regret with a difference between the maximum and average episodic returns that can be achieved by the agent. To make it differentiable w.r.t. θ t , we utilize an environment critic that predicts the return of the agent in the given environment parameter, as done in DSAGE [49] and LPG [50]. The environment critic τ ψ learns to predict a distribution of returns, analogous to distributional RL [51], to better capture the stochasticity of the environment and policy. Based on a support defined as { z i = v min + i M -1 ( v max -v min ) } M -1 i =0 , which is a set of centers of M bins that evenly divide the return domain [ v min , v max ] , we obtain an estimated categorical return distribution from an environment critic output l ( θ t , t ; ψ ) ∈ R M as follows:

To align ˆ Z π with a true return distribution, we train the environment critic by gradient descent on the cross entropy loss between ˆ Z π ( θ t , t ) and a target distribution, which is constructed by projecting episodic returns that the agent achieves on the environment M θ 0 onto the support { z i } M -1 i =0 .

After the environment critic is updated, we estimate the regret (1) with a difference between the maximum return that the current agent can achieve and average of the predicted return distribution. However, the process of finding a maximum achievable return from the distribution is not differentiable. To address this issue, we further approximate the maximum with a conditional value at risk (CVaR), based on the fact that CVaR α ( ˆ Z ) converges to the maximum as a risk measure α goes zero. As a result, we estimate the regret of the agent as follows:

4.4 Adversarial Environment Design via Regret-Guided Diffusion Models

An overview of ADD is provided in Figure 1. First, a diffusion-based environment generator, which is pre-trained on the randomly generated environment dataset, produces a set of environments for the agent. After the agent interacts with the generated environments and is trained via reinforcement learning, the episodic results are utilized to update the environment critic. Then, the environment critic estimates the regret of the agent in a differentiable form (12) and guides the reverse process of the diffusion-based environment generator (10), resulting in environment parameters following the

distribution that maximizes the soft UED objective (7). By repeating this process, the agent learns the soft minimax regret policy, which is robust to the variations of environments. A pseudocode of the algorithm is shown in Appendix A.4.

Since ADD does not require an additional training of the pre-trained diffusion model, the ability to cover the high-dimensional environment domain can be preserved. Furthermore, ADD enables the generator to directly produce meaningful environments via regret-guided reverse process. Therefore, ADD can be seen as having both the advantage of the replay-based methods and learning-based methods while resolving some of their limitations. Additionally, we can control the difficulty level of the generated environments after the training of the RL agent is over. In detail, we can generate environments of difficulty level k ∈ { 1 , 2 , . . . , M } by replacing the regret in the regret-guided reverse process (10) with a log probability of achieving a specific return z M -k as follows:

It enables the reuse of the learned generator and environment critic in various applications, such as generating benchmarks with varying difficulties.

5 Experiments

Tasks We conduct extensive experiments with two challenging tasks. First, we evaluate the proposed method on a partially observable navigation task with a discrete action space and sparse rewards. Then, we assess the performance of our algorithm on a 2D bipedal locomotion task, which has a continuous action space while offering dense rewards.

Baselines We compare the proposed method against several UED baselines. For the learning-based method, we use PAIRED [14], which trains the environment generator via reinforcement learning. For the replay-based method, we use PLR ⊥ [19], which utilizes the random generator and updates the agent only with episodes from the replayed environments. To benchmark performance, we use ACCEL [20], a current state-of-the-art UED algorithm that applies random mutations to replayed environments. Among the two implementation methods of the ACCEL, we use the one that samples environment parameters from the full parameter range, rather than the one that restricts the sampling to a range that ensures simple environments are generated, as the latter could be seen as incorporating prior knowledge. Domain randomization (DR) is also included in baselines so that we can demonstrate the effectiveness of UED. Lastly, we use ADD w/o guidance to show whether the regret guidance induces the diffusion model to generate adversarial environments and enhances the performance of the agent.

Outline We first train a diffusion-based environment generator on the randomly generated environment dataset. Then, we use proximal policy optimization (PPO, [52]) to train the agent on the environments generated by UED methods. To evaluate the generalization capability of the trained agent, we measure the zero-shot transfer performance in challenging, human-designed environments. Additionally, to understand where the differences in performance originate, we conduct quantitative and qualitative analyses on the curriculum of the generated environments by tracking complexity metrics and drawing t-SNE plots. For space consideration, we elaborate on detailed experimental settings including environment parameterization methods in Appendix B.

5.1 Partially Observable Navigation Task

We first evaluate the proposed method on a maze navigation task [14], which is based on the Minigrid [53]. In this task, an agent is trained to take a discrete action using an observation from its surroundings to receive a sparse reward upon reaching a goal. For prior UED methods, we set the maximum number of blocks in a grid environment to 60, aligning with Parker-Holder et al. [20]. For the proposed method, we train the diffusion-based environment generator on 10M random environments whose number of blocks uniformly varies from zero to 60. Then, we train the LSTM-based policy for 250M environmental steps and evaluate the zero-shot performance on 12 challenging test environments from prior works [14, 19].

(a) Zero-shot performance

(b) Training curves

(c) Complexity metrics

(d) t-SNE plots

Mean solved rate

Labyrinth

Number of blocks

ADD

1e8

1.0

1.00

50 -

40

2.5

Solvedrate

0.8

0.75

40

20

30

2.0

0.6

10

1.5

Steps

0

0.4

20

-10

1.0

0.2

0.25

10

-20

0.5

0-

0.0

0.00

0.0

0.5

1.0

1.5

2.0

2.5

0+

0.0

0.5

1.0

1.5

2.0

2.5

-40

-40

-20

0

20

40

o.0

1e8

1e8

Boxplot of solvedrate

Maze

Shortestpathlength

PAIRED

1e8

1.0

1.00

40

2.5

Solved rate

0.8

0.75

50

40

30

20

2.0

0.6

Blocks

30

10

1.5

Steps

0

0.4

20

-10

1.0

0.2

0.25

10

20

0.5

FOE-

0.0

0.00

o

-40

0.0

0.0

0.5

1.0

1.5

2.0

2.5

0.0

0.5

1.0

1.5

2.0

2.5

-40

-20

0

20

40

Steps

1e8

Steps

1e8

DR

PAIRED

PLR

ACCEL

ADDw/oguidance

ADD (ours)

Performance. In Figure 2(a), we report the mean solved rate and box plot to compare the zero-shot performance of the learned policy on the test environments. The result demonstrates that ADD outperforms the baseline methods while achieving 85% of the mean solved rate, which is 18% higher compared to the ACCEL. Furthermore, ADD achieves the highest Q1, Q2, and Q3 values while its interquartile range, defined as Q3 - Q1, is 51% smaller than ACCEL. Therefore, we can infer that the proposed method consistently outperforms the baselines in the challenging test environments. In Figure 2(b), we report training curves on two test environments consisting of Maze and Labyrinth. The results demonstrate that ADD shows the monotonic performance improvement and achieves a higher solved rate compared to other baselines. While ACCEL shows 13% higher mean solved rate than DR, PAIRED and PLR ⊥ do not show notable performance improvement by applying UED techniques. Particularly, PAIRED shows 8% lower mean solved rate compared to DR, demonstrating the challenge of the learning-based UED methods. ADD w/o guidance shows 63% of mean solved rate similar to DR, demonstrating that the regret guidance is critical for training the robust policy. Please refer to Appendix C.1 for detailed per-environment results.

We note that the performance is measured after the fixed number of environmental steps, in line with some UED papers [19, 54] and traditional deep reinforcement learning research. In contrast, Parker-Holder et al. [20] recorded performance after the fixed number of policy updates. Since the replay-based methods require additional episodes without policy updates, using the number of environmental steps may be seen as unfair to PLR ⊥ and ACCEL. To address this issue, we also measured the performance of our method trained with only half the environmental steps, aligning the number of policy updates with PLR ⊥ and ACCEL. When using half the environmental steps, ADD achieves a 72% mean solved rate, which ties with ACCEL. This demonstrates that the proposed method remains competitive, even when using the number of policy updates as a metric.

Generated curriculum. In Figure 2(c), we report complexity metrics consisting of the number of blocks and shortest path length. The results demonstrate that complexity metrics of ADD w/o guidance and DR are almost consistent over time since they do not consider the policy. While PLR ⊥ eventually generates environments with a larger number of blocks compared to DR, ADD and PAIRED generate a curriculum with significantly increasing complexity. Specifically, ADD eventually generates environments with the second largest number of blocks and the longest shortest path. ACCEL also shows significantly increasing complexities despite being based on PLR ⊥ . This is because only up to 60 blocks exist on the 13X13 grid, so random mutation increases the expectation

BasicEnv

HardcoreEnv

StairsEnv

PitGapEnv

StumpEnv

RoughnessEnv

300

300

300

300

300

300

Testreturn

200

Test return

200

Testreturn

200

Testreturn

200

Test return

200

100

100

100

100

100

100

0

0

O

0

王

0

-100

-100

-100

-100

-100

-100

(a)Zero-shotperformance

StumpHeight

Stair Height

PitGap

StairSteps

GroundRoughness

Train Return

5

5

10

10

100

4

4

8

8

8

50

32

3

6

6

6

0

1

2

4

4

-100

-50

2

0

1

2

0

1

2

0

1

2

0

1

2

0

1

2

0

1

2

Steps

1e9

Steps

1e9

Steps

1e9

Steps

1e9

Steps

1e9

Steps

1e9

(b) Generated curriculum

DR

PAIRED

PLR

ACCEL

ADDw/oguidance

ADD (ours)

of the number of blocks. Therefore, it can be seen that ADD and PAIRED efficiently generate complex environments by adapting to the current policy while PLR ⊥ struggles to find environments with high complexity. To compare the distribution of generated environments, we report t-SNE [55] plots in Figure 2(d). While the environments generated by PAIRED eventually cover only a specific region, the environments generated by ADD cover a significantly larger region over time. The results demonstrate that ADD successfully generates adversarial environments while preserving diversity.

5.2 2D Bipedal Locomotion Task

We evaluate the proposed method on the 2D bipedal locomotion task, which is based on the BipedalWalker task in OpenAI Gym [56] and adopted by Parker Holder et al. [20]. In this task, an agent is trained to control its four motors using observation from its Lidar sensor and joints to walk over challenging terrain. UED methods, including the proposed algorithm, need to provide environment parameters consisting of stump height, stair height, pit gap, stair steps, and ground roughness. We note that each parameter increases the difficulty of the environment as its value increases. We train the RL agent for two billion environmental steps and evaluate the zero-shot performance on six test environments.

Performance. Figure 3(a) shows the average return on each test environment. The proposed algorithm achieves the highest return across all environments, with an average of 149.6. Even with half the environmental steps, it achieves a score of 127.4, still surpassing PLR ⊥ . ACCEL shows lower performance than PLR ⊥ , which can be attributed to the lower sample efficiency induced by the additional interaction between the environment and the agent to assess the modified environments. On the other hand, PAIRED achieves the lowest return in all test environments except the easiest Basic Environment. This shows that the learning-based methods struggle to train a robust policy in practice. We note that a recent work [48] stabilizes the training of PAIRED in this task by integrating the evolutionary concept of ACCEL. While applying the evolutionary approach to ADD is possible, we leave it for future work. Lastly, ADD w/o guidance demonstrates superior generalization performance compared to DR. Although these two methods are theoretically identical, this difference is presumably caused by the limited size of the dataset used for training the diffusion-based environment generator.

Generated curriculum Figure 3(b) presents the complexity metrics of the generated training environments and the episodic returns achieved by the RL agent. Unlike the partially observable navigation task, the complexity measure of the environments generated by ADD gradually decreases. This result arises since the randomly generated environments are excessively challenging for the current agent. As evidence, examining the returns achieved by the agent in the generated environments reveals that all methods, except for ADD, consistently yield returns of 0 or below. From these results, we can infer that the proposed algorithm generates environments that are not merely more difficult but are

Easy

Difficult

conducive to the agent's learning process. For detailed experimental results including a qualitative analysis on the generated environments, please refer to Appendix C.2.

5.3 Controllable Generation

To demonstrate the ability to control the difficulty level of the generated environments, we provide the example environment generation results in Figure 4. We control the difficulty level k by guiding the diffusion-based environment generator with a log probability of achieving a specific return z M -k , as described in (13). We vary k from zero to M -1 so that the difficulty level of generated environments increases. Environments generated with k = 0 , which are shown in the leftmost images, include fewer blocks and a close proximity between the agent's starting position and the goal. As k increases, environments are generated with a greater number of blocks and a larger distance between the starting position and the goal, resulting in the elimination of all possible paths when k = M -1 . The results demonstrate that we can effectively control the difficulty level of the environment using the diffusion-based environment generator and learned environment critic, without domain knowledge. We also present the results of controlling difficulty levels for the 2D bipedal locomotion task in Appendix C.2.

6 Limitation

While the proposed method is suitable for training a robust policy, there exist several limitations. First, despite the existence of the optimal point is proven in Proposition 4.1, convergence to such optimal point is not guaranteed. Furthermore, the difference between the true value of the regret and its estimate is not tightly bounded. Lastly, updating the environment critic using episodic results cannot exactly capture the current agent's capability since the policy is updated after the episode. Hence, exploring methods to estimate the regret with a rigorous theoretical background would be an interesting topic for future work.

7 Conclusion

In this work, we present a novel UED algorithm that exploits the representation power of diffusion models. We first introduce soft UED, which augments the original UED objective with an entropy regularization term to make it suitable for using a diffusion-based environment generator. Then, we propose ADD, which guides the pre-trained diffusion model with a novel regret estimator to produce environments that are conducive to train a robust policy. Our experimental results demonstrate that ADD is capable of training a policy that successfully generalizes to challenging environments. Moreover, it has been verified that ADD generates an instructive curriculum with varying complexity while covering large environment configuration spaces.

Acknowledgments and Disclosure of Funding

This work was partly supported by Institute of Information & Communications Technology Planning &Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2019-0-01190, [SW Star Lab] Robot Learning: Efficient, Safe, and Socially-Acceptable Machine Learning, 40%), Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (NRF-2022R1A2C2008239, General-Purpose Deep Reinforcement Learning Using Metaverse for Real World Applications, 40%), and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (NO.RS-2021-II211343, Artificial Intelligence Graduate School Program [Seoul National University], 20%).

References

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, Feb. 2015.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature , 529(7587):484-489, Jan. 2016.

Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of the Conference on Robot Learning . PMLR, Dec. 2022.

Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature , 620(7976):982-987, Aug. 2023.

Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Proceedings of the International Conference on Machine Learning . PMLR, Jun. 2019.

Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937 , Jun. 2018.

Nick Jakobi. Evolutionary robotics and the radical envelope-of-noise hypothesis. Adaptive Behavior , 6(2):325-368, Sep. 1997.

Fereshteh Sadeghi and Sergey Levine. CAD2RL: Real single-image flight without a single real image. In Proceedings of Robotics: Sicence and Systems , Jul. 2017.

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In Proceedings of the International Conference on Intelligent Robots and Systems . IEEE, Sep. 2017.

Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre-Yves Oudeyer. Automatic curriculum learning for deep rl: A short survey. In Proceedings of the International Joint Conference on Artificial Intelligence , Mar. 2020.

Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research , 21(1):7382-7431, Jan. 2020.

Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753 , Feb. 2019.

Seungjae Lee, Daesol Cho, Jonghae Park, and H Jin Kim. CQM: Curriculum reinforcement learning with a quantized world model. In Advances in Neural Information Processing Systems , Dec. 2024.

Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design. In Advances in Neural Information Processing Systems , Dec. 2020.

Leonard J Savage. The theory of statistical decision. Journal of the American Statistical Association , 46(253):55-67, Mar. 1951.

Abdus Salam Azad, Izzeddin Gur, Jasper Emhoff, Nathaniel Alexis, Aleksandra Faust, Pieter Abbeel, and Ion Stoica. CLUTR: Curriculum learning via unsupervised task representation learning. In Proceedings of the International Conference on Machine Learning . PMLR, Jul. 2023.

Dexun Li and Pradeep Varakantham. Enhancing the hierarchical environment design via generative trajectory modeling. arXiv preprint arXiv:2310.00301 , Feb. 2024.

Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In Proceedings of the International Conference on Machine Learning . PMLR, Jul. 2021.

Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. In Advances in Neural Information Processing Systems , Dec. 2021.

Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In Proceedings of the International Conference on Machine Learning . PMLR, Jul. 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems , Dec. 2020.

Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy . Carnegie Mellon University, Dec. 2010.

Peide Huang, Mengdi Xu, Jiacheng Zhu, Laixi Shi, Fei Fang, and Ding Zhao. Curriculum reinforcement learning using optimal transport via gradual domain adaptation. In Advances in Neural Information Processing Systems , Dec. 2022.

Daesol Cho, Seungjae Lee, and H Jin Kim. Outcome-directed reinforcement learning by uncertainty & temporal distance-aware curriculum goal generation. In Proceedings of International Conference on Learning Representations , Feb. 2023.

Rui Wang, Joel Lehman, Aditya Rawal, Jiale Zhi, Yulun Li, Jeffrey Clune, and Kenneth Stanley. Enhanced POET: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In Proceedings of the International Conference on Machine Learning . PMLR, Jul. 2020.

Philip Bontrager and Julian Togelius. Learning to generate levels from nothing. In Proceedings of the IEEE Conference on Games , Aug. 2021.

Eric Mazumdar, Lillian J Ratliff, and S Shankar Sastry. On gradient-based learning in continuous games. SIAM Journal on Mathematics of Data Science , 2(1):103-131, May 2020.

Jakob Bauer, Kate Baumli, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, et al. Humantimescale adaptation in an open-ended task space. In Proceedings of the International Conference on Machine Learning . PMLR, Jul. 2023.

Allen Z Ren and Anirudha Majumdar. Distributionally robust policy learning via adversarial environment generation. IEEE Robotics and Automation Letters , 7(2):1379-1386, Apr. 2022.

Zoey Chen, Karl Van Wyk, Yu-Wei Chao, Wei Yang, Arsalan Mousavian, Abhishek Gupta, and Dieter Fox. Learning robust real-world dexterous grasping policies via implicit shape augmentation. In Proceedings of the Conference on Robot Learning , Dec. 2022.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Proceedings of the International Conference on Learning Representations , May 2021.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Proceedings of the International Conference on Learning Representations , May 2020.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in neural information processing systems , Dec. 2022.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In Advances in Neural Information Processing Systems , Dec. 2022.

Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In Proceedings of the International Conference on Machine Learning , Jul. 2022.

Mineui Hong, Minjae Kang, and Songhwai Oh. Diffused task-agnostic milestone planner. In Advances in Neural Information Processing Systems , Dec. 2023.

Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Policy-guided diffusion. arXiv preprint arXiv:2404.06356 , Apr. 2024.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems , Dec. 2021.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , Jul. 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , Jun. 2022.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with CLIP latents. arXiv preprint arXiv:2204.06125 , Apr. 2022.

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , Jun. 2023.

Xinquan Chen, Xitong Gao, Juanjuan Zhao, Kejiang Ye, and Cheng-Zhong Xu. Advdiffuser: Natural adversarial example synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , Oct. 2023.

Chejian Xu, Ding Zhao, Alberto Sangiovanni-Vincentelli, and Bo Li. Diffscene: Diffusion-based safety-critical scenario generation for autonomous vehicles. In Proceedings of the International Conference on Machine Learning Workshop on New Frontiers in Adversarial Machine Learning , Jul. 2023.

TaeHo Yoon, Kibeom Myoung, Keon Lee, Jaewoong Cho, Albert No, and Ernest K Ryu. Censored sampling of diffusion models using 3 minutes of human feedback. In Advances in Neural Information Processing Systems , Dec. 2023.

Brian D.O. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications , 12(3):313-326, May 1982.

Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. In Advances in Neural Information Processing Systems , Dec. 2020.

Ishita Mediratta, Minqi Jiang, Jack Parker-Holder, Michael Dennis, Eugene Vinitsky, and Tim Rocktäschel. Stabilizing unsupervised environment design with a learned adversary. In Proceedings of the Conference on Lifelong Learning Agents . PMLR, Aug. 2023.

Varun Bhatt, Bryon Tjanaka, Matthew Fontaine, and Stefanos Nikolaidis. Deep surrogate assisted generation of environments. In Advances in Neural Information Processing Systems , Dec. 2022.

Matthew T Jackson, Minqi Jiang, Jack Parker-Holder, Risto Vuorio, Chris Lu, Greg Farquhar, Shimon Whiteson, and Jakob Foerster. Discovering general reinforcement learning algorithms with adversarial environment design. Dec. 2023.

Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In Proceedings of the International Conference on Machine Learning , Aug. 2017.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , Aug. 2017.

Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo Perez-Vicente, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and J Terry. Minigrid &amp; miniworld: Modular &amp; customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems , Dec. 2023.

Samuel Garcin, James Doran, Shangmin Guo, Christopher G Lucas, and Stefano V Albrecht. Dred: Zero-shot transfer in reinforcement learning via data-regularised environment design. In Proceedings of the International Conference on Machine Learning . PMLR, Jul. 2024.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research , 9(86):2579-2605, Aug. 2008.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv:1606.01540 , Jun. 2016.

Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear programming. In Proceedings of the International Conference on Machine Learning , Jul. 2008.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems , Dec. 2016.

Maurice Sion. On general minimax theorems. Pacific Journal of Mathematics , 8(1):171-176, Mar. 1958.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the Medical Image Computing and ComputerAssisted Intervention , Nov. 2015.

A Algorithm Details

A.1 Proof of Proposition 4.1

In this section, we show that the minimax problem of soft UED (7) has zero minimax duality gap. We assume that S, A, and Θ are finite to avoid the technical issues regarding compactness of a set of distributions. Following Section 3.1, we denote a reward function, transition probability, and initial state distribution of an environment M θ with R θ , P θ , and ρ θ 0 , respectively.

We first define an occupancy measure, for a policy π ∈ Π and an environment M θ , as ρ θ π ( s, a ) = π ( a | s ) ∑ N n =0 γ n Pr ( s n = s | π, θ ) . Then, there is a one-to-one correspondence between Π and a set of valid occupancy measures D θ := { ρ : ∑ a ρ ( s, a ) = ρ θ 0 ( s ) + γ ∑ s ′ ,a ′ P θ ( s | s ′ , a ′ ) ρ ( s ′ , a ′ ) } [57, 58].

If we define a global occupancy measure as ρ π := [ ρ θ i π ] | Θ | i =1 , where θ i is an ith element of Θ , it is obvious that there is a one-to-one correspondence between Π and a set of valid global occupancy measures D ⊂ D θ 1 ×···D θ | Θ | . Therefore, we can replace π in the objective function of soft UED (7) with ρ π , as in the following lemma:

Proof. Based on the definition of the regret (1), we have

Now, we can rewrite the objective function of soft UED with a global occupancy measure as follows:

Then, we can prove Proposition 4.1 by showing (15) has zero duality gap. However, we cannot apply minimax theorem [59] directly since D is not a convex set. To resolve this issue, we first augment the problem as follows:

where ¯ D := { ∑ K k =1 w k ρ k | K ∈ N , ∑ K k =1 w k = 1 , ∀ k ∈ { 1 , . . . , K } : w k ≧ 0 , ρ k ∈ D} is a convex hull of D . We will show that the augmented problem (16) has zero minimax duality gap, and end the proof by showing the optimal values of the augmented problem can also be reached by the original problem (15).

Proof. Since ¯ L ( ρ, Λ) is a linear combination of ρ θ , it is convex for all ρ . Furthermore, ¯ L ( ρ, Λ) is concave for Λ since the entropy H is concave. Therefore, based on the fact that ¯ D and D Λ are both convex and compact, the augmented problem has zero duality gap due to minimax theorem [59].

Lemma A.3. For every Λ ∈ D Λ and corresponding ρ ∗ (Λ) ∈ argmin ρ ∈ ¯ D ¯ L ( ρ, Λ) , there exists ρ ′ ∈ D such that ¯ L ( ρ ∗ (Λ) , Λ) = ¯ L ( ρ ′ , Λ) .

Proof. For every Λ ∈ D Λ and corresponding ρ ∗ (Λ) ∈ argmin ρ ∈ ¯ D ¯ L ( ρ, Λ) , there exist K ∈ N , w 1: K ≧ 0 , and ρ 1: K ∈ D such that ∑ K k =1 w k = 1 and ρ ∗ (Λ) = ∑ K k =1 w k ρ k . Then, following inequality holds:

where first inequality holds due to the definition of ρ ∗ (Λ) , equality holds since ¯ L is linear for ρ , and second inequality holds since ρ ∗ (Λ) is a convex combination of ρ 1: K . Then, ρ ′ ∈ argmin ρ ∈{ ρ k } K k =1 ¯ L ( ρ, Λ) is an element of D and satisfies ¯ L ( ρ ∗ (Λ) , Λ) = ¯ L ( ρ ′ , Λ) .

Lemma A.3 ensures the minimum value of ¯ L achieved over ¯ D is also achievable over D , implying the optimal value is the same for both the original and augmented problems. It confirms that strong duality holds for the original problem (15) as well.

Hence, using Lemma A.1 and Proposition A.4, we can prove Proposition 4.1:

A.2 Diffusion Models

In this section, we present implementation details on diffusion models. To solve the forward and reverse processes (3, 4), we follow the implementation of DDPM [21], which can be viewed as a discretization of VP SDE (3). As as result, the forward process (3) is implemented as follows:

where Z t ∼ N (0 , I ) . In the appendix, we make a slight abuse of notation by considering t as a discrete variable to provide detailed implementation specifics. To solve the reverse process (4), we utilize a error network ϵ ϕ ( θ t , t ) = - √ 1 -α t s ϕ ( θ t , t ) instead of using s ϕ , where α t = ∏ t t ′ =1 1 -β t ′ , and β t follows the linear noise schedule. The error network is trained to minimize the loss J ( θ 0 , t ; ϕ ) := E ϵ ∼N (0 ,I ) ∥ ϵ -ϵ ϕ ( √ α t θ 0 + √ 1 -α t ϵ ) ∥ 2 . To futher accelerate the sampling, we apply DDIM sampling [32] as follows:

Since DDIM sampling (20) is deterministic, we can sample the environment parameter θ with T ′ denoising steps, which is less than the original diffusion timestep T .

A.3 Environment Critic Update

In this section, we present details on updating the environment critic. After the interaction between the RL agent with a policy π and an environment M θ , we construct a target return distribution Z target π ( θ ) using episodic returns. Specifically, if episodic returns that the RL agent achieves are { v k } K -1 k =0 , we give equal probabilities to each return and project them into the support { z i } M -1 i =0 . As a result, the target return distribution is constructed as follows:

where [ · ] 1 0 bounds its argument in the range [0 , 1] , and ∆ := ( v max -v min ) M is a width of each bin. Then, we train the environment critic to produce the return distribution close to the target distribution

(21). To that end, we first sample t from 1 , . . . , T and obtain θ t by solving the forward process (3) with initial condition θ 0 = θ , then construct the estimated return distribution ˆ Z π ( θ t , t ) using an output of the environment critic τ ψ ( θ t , t ) . Then, ψ is updated via gradient descent to minimize the cross entropy loss between ˆ Z π ( θ t , t ) and Z target π ( θ ) . To prevent overfitting, we store episodic results in a buffer, then sample the environment parameters and their corresponding target return distribution from the buffer for training the environment critic.

A.4 Pseudocode of ADD

Algorithm 1 Adversarial Environment Design via Regret-Guided Diffusion Models

Input: Policy network π ξ , diffusion model s ϕ , and environment critic network τ ψ . Initialize network parameters ξ, ϕ, ψ .

Train the diffusion model s ϕ on a dataset of randomly generated environments.

for each epochs do

Sample a set of environment parameters { θ i } B i =1 via regret-guided reverse process (10), whose regret is estimated using the environment critic (12).

Run episode on a set of environments {M θ i } B i =1 and update the policy π ξ via RL.

Update the environment critic τ ψ using episodic returns (Appendix A.3). end for

B Experiment Details and Hyperparameters

In this section, we provide a comprehensive explanation of the experiments discussed in Section 5. We begin by detailing the two tasks: partially observable navigation and 2D bipedal locomotion. Then, we conclude the section by reporting the hyperparameters employed in the experiments.

B.1 Partially Observable Navigation Task

Environment details . In the partially observable navigation task, which is based on the Minigrid [53] and adopted for UED in prior works [14], the agent is trained to find and reach a goal in the grid maze environment. Each maze environment is a 15 × 15 grid whose cells on the edge are all walls, and the cells inside can contain walls, agents, or goals. When the agent reaches the goal, it receives a reward of 1 -N/N max , where N is a length of the episode and N max is a maximum length of each episode. If it does not reach the goal, it receives a reward of 0. The agent uses a 7 × 7 grid around itself and its direction as an observation and chooses one of the actions: turn left, turn right, or go forward.

Environment generation . Aligning with Parker-Holder et al. [20], we limit the number of walls that can exist in the 13 x 13 grid, excluding the walls on the edges, to 60. The RL generator of PAIRED selects locations to place walls over 60 steps, ensuring no changes if a wall already exists. After placing all the walls, it selects the starting position of the agent and the goal location. If a wall exists at those locations, it removes the wall and places the agent or the goal. On the other hand, the random generator used by PLR ⊥ and ACCEL uniformly samples the number of walls between 0 and 60 in advance and then choose the position to place the walls, agent and goal location randomly. This random generation is also used to create a dataset for training the diffusion-based environment generator of the proposed algorithm. Specifically, each environment parameter data is represented with a 13 × 13 × 3 image. The first channel represents the location of the walls, with a value of one if a wall is present in the cell, and zero if it is not. The second channel indicates the starting position of the agent, with a value of one for the starting cell, 0.5 for the cell after moving forward once, and zero for all other cells. Finally, the third channel has a value of one for the cell corresponding to the goal, and zero otherwise. After training the diffusion-based environment generator on the randomly generated dataset, we can produce the environment parameter by employing the trained generator to solve reverse process (5), as shown in Figure 5.

Training time . All methods are trained utilizing RTX 3090Ti. To train DR and ADD w/o guidance, they require almost 48 hours to run 250 million environment steps. ADD requires almost 56 hours, and PLR ⊥ and ACCEL requires almost 100 hours for each random seed.

0T

00

B.2 2D Bipedal Locomotion Task

Environment details . For the 2D bipedal locomotion task, we conduct the experiment using a modified version of the BipedalWalker environment from OpenAI Gym [56], as done in ParkerHolder et al. [20]. In this task, the agent is trained to walk over challenging terrains by controlling four joints, and action is decided using a 24-dimensional observation, which is consisting of Lidar measurements, linear and angular velocities of the robot, positions and speeds of joints, and contact information. The agent receives a positive reward for moving forward, and receives -100 as a reward if it falls to the ground. If the agent reaches the opposite end of the terrain, total reward it receives is over 300.

Environment generation . In the 2D bipedal locomotion task, we generate the environment by deciding the eight-dimensional environment parameter θ ∈ R 8 , which is consisting of a min / max stump height, min / max stair height, min / max pit gap , stair steps, and roughness of the terrain. The RL generator of PAIRED selects each parameter sequentially, and the random generator of PLR ⊥ and ACCEL randomly decide each parameter by sampling a real number from its domain, which is reported in Table 1. We employ the random generator to construct a dataset of environment parameters, and train the diffusion-based environment generator, which will be used to produce the environment parameter while training the agent using the proposed method. After the environment parameter is decided, the entire environment is generated by the procedural content generation algorithm.

Training time . All methods are trained utilizing RTX 3090Ti. To train DR and ADD w/o guidance, they require almost 80 hours to run 2 billion environment steps. ADD requires almost 92 hours, and PLR ⊥ and ACCEL requires almost 160 hours to run the experiment for each random seed.

B.3 Hyperparameters

To train RL agents using UED baselines, we follow the implementation and hyperparameters of ParkerHolder et al. [20], which is available at https://github.com/facebookresearch/dcd . The same parameters and network architecture were used to train the ADD agent, and the hyperparameters used are reported in the Table 2. To train the diffusion-based environment generator, we followed the implementation of Dhariwal et al. [38] and Yoon et al. [45], which is available at https: //github.com/tetrzim/diffusion-human-feedback , and the size of the randomly generated dataset is set to 10 million. For the partially observable navigation task, the architecture of the error network ϵ ϕ follows the UNet [60], which utilizes three residual blocks with channel multipliers [1 , 2 , 2 , 2] for each resolution. For the 2D bipedal locomotion task, the architecture of error network is four-layer MLP with a sinusoidal time embedding. We report detailed hyperparameters used to train diffusion models in Table 3. Lastly, we report detailed hyperparameters for regret-guided diffusion process and training the environment critic in Table 4. The network architecture of the environment critic is based on the UNet encoder for the partially observable navigation task and a four-layer MLP for the 2D bipedal locomotion task.

C Detailed Experimental Results

In this section, we present detailed experimental results in the partially observable navigation task and 2D bipedal locomotion task. For each task, we will provide specific zero-shot generalization performance, t-SNE plots of all baselines, and the training environments generated by the proposed algorithm. Additionally, we will show examples of environments generated with varying difficulty levels using the method described in Section 4.4.

C.1 Partially Observable Navigation Task

Zero-shot transfer test results. After training the RL agent in the partially observable navigation task, we evaluate the generalization capability of the learned policy by testing the agent in twelve unseen environments, which are shown in Figure 6. In each environment, the agent is evaluated for 100 independent episodes, and the full result is reported in Table 5. We note that the reported quartile values represent the average of the quartile values of the solved rates achieved in the test environments for each seed. The results demonstrate that the proposed method achieves the best performance in five out of 12 test environments, with particularly high mean and quartile values compared to the baselines. Therefore, we can infer that ADD successfully trains an agent that is robust to environmental changes and generalizes to various environments.

FourRooms.

16Rooms

16Rooms2

Labyrinth

Labyrinth2

Maze

尚

Maze2

Maze3

SimpleCrossing

SmallCorridor

LargeCorridor

PerfectMaze (M)

Generated training environments. To support the claim regarding the generated curriculum, we provide example scenes of generated training environments in Figure 7 and Figure 8. Comparing two figures reveals that the environments generated after training the agent with 200 million environmental steps are more complex and contain a larger number of blocks, aligning with the quantitative results shown in Figure 2(c). The reason for this difference is that as the agent is trained, it achieves near-optimal performance in simple environments. Consequently, the environment critic predicts that the agent's regret will be larger in environments with a greater number of blocks and increased complexity.

DR

1e8

PAIRED

1e8

PLR

1e8

ACCEL

1e8

ADDw/oguidance1e8

ADD

1e8

2.5

2.5

2.5

2.5

40

2.5

2.5

30

2.0

30

2.0

30

2.0

30

2.0

30

2.0

30

2.0

20

20

20

20

20

20

19

1.5

Steps

19

1.5

Steps

10

1.5

Steps

19

1.5

Steps

10

1.5

Steps

10

1.5

Steps

-10

1.0

-10

1.0

-10

0

1.0

10

1.0

-10

1.0

10

1.0

20

30

0.5

20

30

0.5

20

30

0.5

20

30

0.5

20

30

0.5

20

30

0.5

40

40

-20

20

40

0.0

40

40

20

0

20

40

0.0

40

40

-20

20

40

0.0

40

40

20

20

40

0.0

40

40

-20

20

40

0.0

40

40

20

20

40

0.0

t-SNE plots . To conduct a qualitative analysis on the generated environments, we visualize training environments generated from ADD and baselines. Since the parameter of the maze environment is high-dimensional and hard to define the meaningful distance, we first utilize the encoder of the environment critic to map environment parameters to the learned latent space, then train a t-SNE on the latent vectors. The t-SNE results are shown in Figure 9. The results demonstrate that the proposed method generate sufficiently diverse environments comparable to those produced by a random generator.

C.2 2D Bipedal Locomotion Task

Zero-shot transfer test results. For the 2D bipedal locomotion task, we assess the zero-shot performance of the trained RL agent in six unseen test environments shown in Figure 10. In each test environment, we evaluate the performance for 100 independent episodes. In this task, PAIRED struggles to train the RL-based environment generator since the value function often diverges. Therefore, we adopted the zero-shot performance result of PAIRED reported in Parker Holder et al. [20]. We report the full results in Table 6, and the results demonstrate that the proposed method outperforms baselines across all test environments.

Basic

Hardcore

Stair

PitGap.

Roughness

Generated training environments. To demonstrate the generated environments vary as the agent learns, we provide the generated environments in the early and later stages of the training in Figure 11 and Figure 12. It can be observed that the environments shown in Figure 12 are much simpler compared to those in Figure 11, which is consistent with the quantitative analysis provided in Figure 3(b). This difference occurs because randomly sampling environment parameters can result in overly challenging environments that hinder the agent's learning, and the environment critic guides the generator to produce simpler environments that are conducive to the agent's learning.

t-SNE plots . As in the previous task, we trained a t-SNE to visualize the generated environments for the 2D bipedal locomotion task. Since the environment parameters for this task have low dimensionality and the meaningful distance can be computed using L2 distance, we use the raw environment parameters without mapping them to a learned latent space. The results shown in Figure 13 demonstrate that the proposed method produces sufficiently diverse training environments. In contrast, training environments generated by PAIRED are concentrated in a small area. Due to this difference, the agent trained with the proposed algorithm demonstrates better performance across all test environments.

Controllable generation. As done in the partially observable navigation task, we provide the results of manipulating the difficulty of the generated environments in Figure 14. The results show that when we guide the generator to produce low-difficulty environments, nearly flat terrains are produced. As the desired difficulty increases, the environments become progressively more complex and challenging to walk over. This additional capability of ADD allows the trained environment critic and diffusion-based generator to be reused in applications such as benchmark generation.

DR

1e9

PAIRED

1e9

PLR

1e9

ACCEL

1e9

ADDw/oguidancele9

ADD

1e9

30

40

2.00

1.75

30

40

2.00

1.75

30

40

2.00

1.75

30

40

2.00

1.75

30

40

1.75

2.00

OE

40

2.00

1.75

20

1.50

20

1.50

20

1.50

20

1.50

20

1.50

20

1.50

-10

10

0.75

1.25

1.00

-10

10

0

1.25

1.00

0.75

-10

10

0

V.

0.75

1.25

1.00

10

10

0

1.25

1.00

0.75

Step

ps

10

10

9

1.00

0.75

1.25

-10

10

0

1.00

0.75

1.25

Steps

20

0.50

20

0.50

20

0.50

20

0.50

20

0.50

20

0.50

30

0.25

30

0.25

30

0.25

30

0.25

30

0.25

30

0.25

40

-40

-20

20

0.00

40

40

20

20

40

0.00

40

40

-20

20

0.00

40

40

=20

20

0.00

40

40

=20

20

40

0.00

40

40

=20

D

20

40

0.00

Easy

Difficult

C.3 Ablation Study

We conduct ablation studies to analyze the role of entropy regularization term and the number of environments used in diffusion pre-training. Frist, to show whether adding an entropy term to the original UED objective plays a critical role, we measure the zero-shot generalization performance of the trained agent with varying ω , which is defined in the soft UED objective E θ ∼ Λ [ REGRET ( π, θ )] + 1 ω H (Λ) . The experimental results are shown in Table 7. From the results, we observed that performance decreases as ω becomes large. Since the influence of the entropy term diminishes as ω increases, it can be seen that our experimental results highlight the importance of the entropy term.

Next, to analyze the influence of the number of samples used during the pre-training phase, we trained the diffusion model using one million samples, which is 100 times fewer than in the original experiment, and measured the performance of the proposed algorithm. The result is a mean success rate of 0.76 ± 0.07 in the partially observable navigation task. This is about 11% lower than the result reported in the original experiment. The result demonstrates that a larger number of samples used in pre-training would lead to better performance, which is quite trivial since the diffusion model can generate more diverse environments when trained with a larger number of samples. Additionally, we note that since we are dealing with an unsupervised setting and the samples used in pre-training are generated through random sampling, there is no need to worry about data scarcity.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The paper's contributions and scope are accurately described in the abstract and instruction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the work are discussed in Section 6.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide the full set of assumptions in Proposition 4.1, and the complete proof is presented in Appendix A.1

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide experiment details and hyperparameters in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We submit the code with a detailed instruction, and all experiments are reproducible.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the training and test details are provided in Appendix B and Appendix C. Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In Figure 2 and Figure 3, all the results are accompanied by error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report the type of compute workers and computation time in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research fully conform with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is not expected to cause direct societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work does not have a negative use case, so we don't need special safeguards.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite all the papers that produced the code package, and provide URLs in Appendix B.3.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the details about training in Appendix B.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on Unsupervised Environment Design (UED), a problem setting whereby a teacher designs environments for a student, learning to solve the task. This area of research has been in focus recently due to its ability to train more general agents in an open-ended setting. The authors look to build on recent work using generative models to generate environments, and propose a diffusion based approach ADD. The method seems sound and leads to gains in two of the canonical UED environments. I recommend acceptance as it incrementally improves UED literature.\n\nSTRENGTHS:\nThe main strengths are listed below. Since I recommend acceptance, the focus of the review is on weaknesses to improve the paper as much as possible for the camera ready version.\n* The method makes sense intuitively and seems to work as expected, especially with the curriculum results.\n* The paper is well written and clearly builds on previous works (e.g. in the related work discussion). \n* Experiments are well presented and show clear performance gains.\n\nWEAKNESSES:\n* The x axis for the plots is showing total steps and not student gradient updates. This puts replay methods at a huge disadvantage since they take ~2x the number of environment steps, but the policy isnt training so arguably it should not count when comparing the effectiveness of the curriculum. I think this should be changed to match the literature.\n* The baselines are fairly weak, aside from PLR and ACCEL (which have been weakened by the previous point). Can the authors compare against CLUTR and show why ADD produces a better curriculum? To me that is the obvious baseline. Further, note the improvements to PAIRED proposes in [1] which makes it a much stronger baseline. I would say the original PAIRED is more of a concept than a baseline at this point due to known deficiencies. \n* There is a clear limitation of your method that you require a pre-training phase with access to the environment, and do not count the number of steps as part of training because it is \"unsupervised\". This is fine, but then makes it wrong to show training steps for the replay methods. Please either 1) include the steps from this in the x axis for ADD (which will drastically weaken performance) or 2) switch the x axis to student gradient updates. Otherwise it is an apples to oranges comparison in your favour.\n* I would love to see examples of the generated environments in the main body, for me it is more useful than the theory part, but maybe that is subjective :)\n* More of a general comment, I find it strange that UED papers don't cite [2]. It is the largest scale demonstration of the value of UED and motivates research on these methods. It explicitly shows PLR is effective for curricula over a massive task space with a 500M parameter transformer based policy. This clearly shows that UED research can have impact on large scale AGI focused projects, so I would have expected the UED community to all be very excited about this.\n\n[1] Mediratta et al. \"Stabilizing Unsupervised Environment Design with a Learned Adversary\". CoLLAs 2023\n[2] Bauer et al. \"Human Timescale Adaptation in an Open-Ended Task Space\". ICML 2023\n\nQUESTIONS:\n* The meta data plots show ACCEL starting with the same number of stumps etc as the other baselines. However in the original implementation/paper it should be starting with very simple terrains. Was this the case in your work?\n\nLIMITATIONS:\nThe limitation of requiring additional data to pre train the diffusion model is not mentioned, I think this is actually the largest weakness by far. It could be interesting to see if pre trained foundation models work well here too.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work applies regret-guided diffusion models to the UED setting in order to generate adversarial environments that preserve diversity.\n\nSTRENGTHS:\n* The contributions are well motivated and appear to be novel.\n* The writing is generally clear and concise.\n* The paper is contextualized well within prior literature.\n\nWEAKNESSES:\n* Minor typos/grammatical issues:\n    * Line 253: “challenging”—This word alone does not hold any descriptive power. Because the nature of the tasks are described next in impartial terms, this word seems redundant/rhetorical.\n* The limitations do not mention the assumed structure of the environments; I am not convinced that diffusion can be applied to all environment parameterizations. \n* Why is shortest path length used as a complexity metric? It seems to describe the tail of the distribution; Why not average path length? Also, number of blocks does not seem to be a well-motivated complexity metric either.\n* While ADD “successfully generates adversarial environments while preserving diversity”, the behavior improvements seem to be minimal based on the training curves / zero shot performance. \n* A more general point: it is hard to tell what exactly the authors are trying to show from the experiments; the conclusion nicely sums up the analysis, but it would have been nice to have been told at the top of “experiments” what each experiment is trying to test.\n* Given the prevalence of UED domains in the literature, a third evaluation domain would really strengthen this work. Otherwise, I am concerned about the applicability of this method to other environments, especially since the authors have not addressed the implicit assumptions about workable environment parameterizations for their diffusion approach.\n\nQUESTIONS:\n* What are the assumptions on the structure of the environment parameterization? Can this be applied to all simulated environments? Or only a subset with certain properties?\n* Why were RGB channels used to describe the maze environment instead of e.g. a binary (continuized for the diffusion process); it seems odd to parameterize the environment based on the observation space, and then to extract these parameters themselves based on the observation values. Are there certain parameterizations for the diffusion process that work better/worse? What properties does this representation space need to have?\n* Are the t-SNE embeddings generated using all points, and then each individual plot displays just the relevant method’s points?\n\nLIMITATIONS:\n* The limitations are outlined accurately by the authors in Section 6. However, I would like the authors to address the first question above, which I believe is an additional limitation that would be worth discussing.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an approach for gradient directed, regret-based UED based on guiding a pre-trained diffusion model.\n\nSTRENGTHS:\nThis paper addresses a major shortcoming of prior UED approaches. In the past gradient-based UED approaches have been out-performed by sample-based or evolutionary approaches for searching for environments. While there was a general intuition that gradient based approaches would ultimately be more scalable, it was difficult to realise this intuition due to the fact that environment-design is a high-dimensional optimisation problem for which RL algorithms have struggled. Using insights from generative modelling is a natural approach to bridging this gap, and this paper provides a few tricks which seem to have been missing from prior attempts.\n\nSpecifically, pre-training the diffusion model on random levels, and using learned guidance seems to be a particularly powerful combination.\n\nWEAKNESSES:\nMy main concern is that It seems like the baseline results in the bipedal walker domain don't replicate the results from prior work, suggesting some sort of bug/lack of tuning in the implementation? It appears that if I compare the numbers of ADD to the results in the ACCEL paper it gets lower or equivalent performance to ACCEL itself? My main reason for not currently raising my score is questions about the accuracy of this evaluation.\n\nThere are some tricks that are rediscovered from prior work which should be attributed. Specifically, entropy regularisation for neural generators is studied in [1], and training a sort of critic to evaluate levels is studied in [2]. That being said, ADD puts a unique spin on both of these.\n\n[1] Mediratta, Ishita, et al. \"Stabilizing unsupervised environment design with a learned adversary.\" _Conference on Lifelong Learning Agents_. PMLR, 2023.\n\n[2] Bhatt, Varun, et al. \"Deep surrogate assisted generation of environments.\" _Advances in Neural Information Processing Systems_ 35 (2022): 37762-37777.\n\n\nFor the citation of environment design, it is correct to cite the UED formalism was from Dennis et al. but for the more general concept of designing environments it would be best to also cite contemporaries  POET[3] and GPN [4].\n\n[3] Wang, Rui, et al. \"Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions.\" _arXiv preprint arXiv:1901.01753_ (2019).\n\n[4] Bontrager, Philip, and Julian Togelius. \"Learning to generate levels from nothing.\" _2021 IEEE Conference on Games (CoG)_. IEEE, 2021.\n\n\nIt would also be best to be careful not to equate minimax regret based UED approaches with all UED approaches as is done on line 83, as there are many non minimax regret approaches to UED such as POET, SAMPLR, CLUTR, and DRED. \n\n\nPre-training on random levels seems like it leaves only a limited amount that the diffusion model could learn. It would be interesting to periodically fine-tune the diffusion model on newly generated levels to increase the power of the generator over time. This would generally be much more convincing as it could scale much further off of the distribution of random levels.\n\n#### Clarity\n\nIt would be helpful to include the per-transfer environment bar-plots as is traditional with UED papers, it seems like the same information is included in Table 5 and 6 but they are much harder to read in that format, and it is difficult to tell where error bars overlap.\n\nIt would also be helpful to include the bootstrapped CI's plot often in UED papers as recommended by [5]\n\nFigure 4 and Figure 9 would be amazing to include in the main body, and go a long way towards explaining and demonstrating the method.\n\n[5] Agarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" _Advances in neural information processing systems_ 34 (2021): 29304-29320.|\n|APA||\n\nHow do you arrive at the ADD acronym? It is a bit difficult for me to remember it and what it stands for.\n\nQUESTIONS:\nHave you tried replacing equation 12 with a PAIRED-style loss maximising the expectation between a protagonist and antagonist?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a diffusion model with differentiable regret estimate for unsupervised environment design. The authors write a diffusion process to model environment parameters where the process is described in terms of a scoring function and derivative of the regret. The scoring function is pre-trained on a set of random environments. The diffusion process is further fine-tuned with the regret to generate environments for curriculum learning of an agent. A critic, which is trained with cross-entropy loss using binned environment returns, is used to approximate the regret. By using the derivative of the critic w.r.t. environment parameters, diffusion process is fine-tuned with entropy augmented regret. Experimental results on continuous and discrete control domains show that the model is competitive with previous best models. Using regret guidance significantly improves the performance.\n\nSTRENGTHS:\nThe paper introduces a diffusion process which is well suited for modeling continuous parameter. It implements a differentiable regret approximation that better assigns credit for the diffusion process. The formulation through the optimal environment distribution with a trainable scoring function is also interesting.\n\nWEAKNESSES:\nThere are a few things that need more clarification and ablations.\n\n1. While the paper shows that the diffusion process with a differentiable regret estimate gives good results, it is not clear which component is the most critical. Is the diffusion process critical for the success? Can you train PAIRED with a differentiable regret?\n\n2. Similar to above, is the entropy term critical? Can you train the diffusion process without it and get comparable results?\n\n3. Can you explain the reason why you trained the critic model with binned returns rather than using actual returns in a regression objective?\n\n4. How critical is the pre-training for the scoring function? How does the performance change with less or more number of environment samples?\n\nQUESTIONS:\nPlease see above for specific questions.\n\nLIMITATIONS:\nThe paper addresses the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}